| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
Move sqlalchemy.ext.declarative imports to orm (#29862) | 3 年前 | |
Update Airflow version to 2.5.2 (#30111) * Update Airflow version to 2.5.2 * Update Chart airflow version | 3 年前 | |
Trigger Class migration to internal API (#29099) * Trigger Class migration to internal API --------- Co-authored-by: bugraozturk <bugra.ozturk@mollie.com> | 3 年前 | |
completed D400 for airflow/callbacks/* airflow/cli/* (#27721) | 3 年前 | |
Fix warning in airflow tasks test command regarding absence of data_interval (#27106) * Fix warning in airflow tasks test command regarding absence of data_interval. * Use manual data interval in task commands * Update tests/cli/commands/test_task_command.py --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> | 3 年前 | |
Switch to ruff for faster static checks (#28893) Gone are: - isort - pyupgrade - pydocstyle - yesqa - autoflake - flake8 All replaced with [ruff](https://github.com/charliermarsh/ruff). A chunk of the performance of ruff comes from the fact that it makes very good use of multiple cores. And since most docker virtual machines are only one or two core I have chosen to run it directly, not inside the breeze docker container so we get the full benefit of speed. * Work around namespace packages issue for providers Ruff is currently detecting "google" as a the name of the current package, so it thinks it goes in the "first" party import section | 3 年前 | |
Allow customized rules to check if a file has dag (#30104) * Allow customized rules to check if a file has dag Currently, Airflow uses its heuristic rules to check if a py file has DAGs defined or not by checking if dag or airflow in the file content. This generally works well however for cases with lots of py files, it could lead to lots of false positive. This change allows users to have more flexible rules to meet their needs. | 3 年前 | |
Align cncf provider file names with AIP-21 (#29905) Changes to import paths: operators.kubernetes_pod -> operators.pod triggers.kubernetes_pod -> triggers.pod This PR is backward compatible | 3 年前 | |
AIP-44 Migrate Dagbag.sync_to_db to internal API. (#29188) | 3 年前 | |
add some statsd metrics for dataset (#28907) ] | 3 年前 | |
Only allow setup/teardown tasks (#30150) We've decided to simply support only single setup and teardown tasks, so we can remove support for task groups. | 3 年前 | |
Only allow setup/teardown tasks (#30150) We've decided to simply support only single setup and teardown tasks, so we can remove support for task groups. | 3 年前 | |
try_number was not being passed to the get_task_log method, instead (#28817) ti.try_number was used for fetching log from k8s pod. it was causing incorrect log being returned for k8s pod. fixed by passing try_number from _read to get_task_log method use try_number argument instead of ti.try_number for selecting pod in k8s executor Co-authored-by: eladkal <45845474+eladkal@users.noreply.github.com> | 3 年前 | |
Upgrade Mypy to 1.0 (#29468) * Upgrade mypy to 0.991 Most of the changes are related to with x = None * Update airflow/dag_processing/processor.py Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Upgrade Mypy to 1.0 * Resolve newly found typing issues * Work around empty function in protocol Mypy seems to not be able to detect an empty default implementation in a protocol. Supplying 'return None' seems enough to tell Mypy the function is present. This is better than needing to implement an empty function everywhere. * Improve typing declarations * Fix pat reference in update-common-sql-api-stubs --------- Co-authored-by: Ash Berlin-Taylor <ash@apache.org> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> | 3 年前 | |
Add Pydantic-powered ORM models serialization for internal API. (#29776) Add basic serialization capabilities for the ORM SqlAlchemy models that we use on the client side of the Internal API. Serializing the whole ORM models is rather complex, therefore it seems much more reasonable to convert the ORM models into serializable form and use them - rather than the SQLAlchemy models. There are just a handful of those models that we need to serialize, and it is important to maintain typing of the fields in the objects for MyPy verification so we can allow some level of duplication and redefine the models as pure Python objects. We only need one-way converstion (from database models to Python models), because all the DB operations and modifications of the Database entries will be done in the internal API server, so the server side of any method will be able to use primary key stored in the serializable object, to retrieve the actual DB model to update. We also need to serialization to work both way - an easy way to convert such Python classees to json and back - including validation. We could serialize those models manually, but this would be quite an overhead to develop and maintain - therefore we are harnessing the power of Pydantic, that has already ORM mapping to plain Python (Pydantic) classes built in. This PR implements definition of the Pydantic classes and tests for the classes testing: * conversion of the ORM models to Pydantic objects * serialization of the Pydantic classes to json * deserialization of the json-serialized classes to Pydantic objects | 3 年前 | |
Add a new Airflow conf to specify a SSL ca cert for Kubernetes client (#30048) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> | 3 年前 | |
Fix and augment check-for-inclusive-language CI check (#29549) * Fix and augment check-for-inclusive-language CI check Related: #15994 #23090 This PR addresses a few items related to inclusive language use and the CI check: - There are several occurrences of "dummy" throughout documentation; however the current CI check for preventing non-inclusive language doesn't inspect docs/ files. Ideally, the docs also include inclusive language. - Even when removing docs/ from the exclusion list, non-inclusive language was still escaping pygrep. Upon inspection, the (?x) inline modifier was missing from the regex (although intended in #23090). Adding this modifier revealed these "dummy" instances and others related non-inclusive occurrences which were previously uncaught. - The exclusion list seemed too broad in places. There are still instances in which directories are excluded as a whole, but the list now is more tailored to non-inclusive occurrences that are beyond the purview of Airflow, history, dev/test files, etc. * Update links in get_pandas_df() of BigQueryHook * dummy-command -> placeholder-command in _KubernetesDecoratedOperator | 3 年前 | |
Allow datasets to be used in taskflow (#27540) * Allow datasets to be used in taskflow Datasets could not be passed as parameters to taskflow functions as they could not be serialized. This commit: 1) changes the xcom serializer so that it now can serialize objects that have attr, dataclass or a custom serializer 2) removes the need to for a custom serializer in lineage 3) adds a version check to the serializer/deserializer 4) registers any datasets as either inlets or outlets in the task 5) inlets or outlets that cannot be serialized now raise an error * Update docs/apache-airflow/concepts/taskflow.rst Co-authored-by: Bas Harenslak <BasPH@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Bas Harenslak <BasPH@users.noreply.github.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> | 3 年前 | |
Upgrade Mypy to 1.0 (#29468) * Upgrade mypy to 0.991 Most of the changes are related to with x = None * Update airflow/dag_processing/processor.py Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Upgrade Mypy to 1.0 * Resolve newly found typing issues * Work around empty function in protocol Mypy seems to not be able to detect an empty default implementation in a protocol. Supplying 'return None' seems enough to tell Mypy the function is present. This is better than needing to implement an empty function everywhere. * Improve typing declarations * Fix pat reference in update-common-sql-api-stubs --------- Co-authored-by: Ash Berlin-Taylor <ash@apache.org> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> | 3 年前 | |
Make json and yaml available in templates (#28930) * Add json_loads macro * Update __init__.py --------- Co-authored-by: eladkal <45845474+eladkal@users.noreply.github.com> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> | 3 年前 | |
Increase length of user identifier columns (#29061) Increase length of columns in ab_user and ab_register_user tables: - first_name and last_name from 64 to 256 characters - email and username from 256 to 512 characters | 3 年前 | |
Only allow setup/teardown tasks (#30150) We've decided to simply support only single setup and teardown tasks, so we can remove support for task groups. | 3 年前 | |
Add general-purpose "notifier" concept to DAGs (#28569) * Add general-purpose "notifier" concept to DAGs This makes it easy for users to setup notifications for their DAGs using on_*_callbacks It's extensible and we can add it to more providers. Implemented a SlackNotifier in this phase. In the course of this, I extracted a 'Templater' class from AbstractBaseOperator and have both the Notifier & ABO inherit from it. This is necessary in other to avoid code repetition. * Renames and a fixup not to require a call to super in subclasses * Raise compat exception and add docs * Ignore import error due to optional provider feature * fixup! Ignore import error due to optional provider feature * Apply suggestions from code review Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * fixup! Apply suggestions from code review Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> | 3 年前 | |
Don't use importlib.metadata to get Version for speed (#29723) As discovered by @uranusjr in other PRs, loading the Metadata info at runtime is surprisingly expensive. Recent versions of setuptools (including the one we already say we depend upon in pyproject.toml) have the ability to pull the version from an attribute using an "ast-eval" method so this keeps the property of there being a single source of truth for the Airflow version, it just moves that places to airflow/__init__.py. You might wonder why this particular case matters at runtime? In the grand scheme of things it likely doesn't, except that the airflow/operators/python.py imports this at the top level (or did before this PR) which made me look in to it and discover a quick win here. | 3 年前 | |
Fix the DLISparkCreateBatchJobOperator parameter | 2 年前 | |
Completed D400 for multiple folders (#27768) | 3 年前 | |
Completed D400 for multiple folders (#27722) | 3 年前 | |
Explicit skipped states list for ExternalTaskSensor (#29933) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> | 3 年前 | |
Don't separate setup/teardowns from normal task (#30008) Instead of keeping setup and teardown tasks separate from the normal task list in Task Groups, we will put them all into children together. Attributes on the tasks are all we need to support the setup/teardowns feature, and keeping them as normal-task-like as possible is desirable. | 3 年前 | |
| 3 年前 | ||
Add general-purpose "notifier" concept to DAGs (#28569) * Add general-purpose "notifier" concept to DAGs This makes it easy for users to setup notifications for their DAGs using on_*_callbacks It's extensible and we can add it to more providers. Implemented a SlackNotifier in this phase. In the course of this, I extracted a 'Templater' class from AbstractBaseOperator and have both the Notifier & ABO inherit from it. This is necessary in other to avoid code repetition. * Renames and a fixup not to require a call to super in subclasses * Raise compat exception and add docs * Ignore import error due to optional provider feature * fixup! Ignore import error due to optional provider feature * Apply suggestions from code review Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * fixup! Apply suggestions from code review Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> | 3 年前 | |
Fixes to how DebugExecutor handles sensors (#28528) * move fix to ready_to_reschedule * replace check on debug exec with new property Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> | 3 年前 | |
Adding ContinuousTimetable and support for @continuous schedule_interval (#29909) --------- Co-authored-by: Jarek Potiuk <jarek@potiuk.com> | 3 年前 | |
convert moment with timezone to UTC instead of raising an exception (#29606) * convert moments with timezone to utc instead of raising an exception * test DateTimeTrigger with different timezones | 3 年前 | |
Add Huawei Cloud provider | 3 年前 | |
Add task.trigger rule to grid_data (#30130) * add trigger rule to grid_data | 3 年前 | |
Don't use importlib.metadata to get Version for speed (#29723) As discovered by @uranusjr in other PRs, loading the Metadata info at runtime is surprisingly expensive. Recent versions of setuptools (including the one we already say we depend upon in pyproject.toml) have the ability to pull the version from an attribute using an "ast-eval" method so this keeps the property of there being a single source of truth for the Airflow version, it just moves that places to airflow/__init__.py. You might wonder why this particular case matters at runtime? In the grand scheme of things it likely doesn't, except that the airflow/operators/python.py imports this at the top level (or did before this PR) which made me look in to it and discover a quick win here. | 3 年前 | |
Fix import cycle starting from airflow/__main__.py (#29523) | 3 年前 | |
[AIRFLOW-6247] Fix sort order in Alembic Migration template (#6809) * [AIRFLOW-6247] Fix sort order in Alembic Migration template | 6 年前 | |
Change permissions of config/password files created by airflow (#29495) The permissions for files created by airflow when creating config and standalone files are now only limited to the owner. | 3 年前 | |
Remove redundant word (#13466) for for -> for | 5 年前 | |
Fix on_failure_callback when task receives a SIGTERM (#29743) This fixes on_failure_callback when task receives a SIGTERM by raising a different exception in the handler and catching the exception during task execution so we can directly run the failure callback. | 3 年前 | |
Completed D400 for multiple folders (#27767) | 3 年前 | |
Refactor serialization (#28067) Generalize serialization and deserialization Seralization and deserialization (derser) of objects that were not under the control of Airflow of the dag author was not possible. This implementation allows for extension of serialization of arbitrary objects by putting them in the namespace of airflow. Selection of serializer happens in order of registered serializer, custom serializer, attr/dataclass annottated object. | 3 年前 | |
Make the policy functions pluggable (#28558) Previously it was only possible to set "policy" functions via airflow_local_settings.py which is fine for "small clusters" but being able to control some of these policies from installed plugins/distributions is helpful in a few circumstances: it lets "platforms" (either of the SaaS variety, or internal platform teams) specify some common policies, but still let local Ariflow teams define other policies using airflow_local_settings This uses Pluggy[1] for plugin management (that we are already using for the listener interface), and out of the box that gives us the ability to have multiple implementations of the policy functions where it makes sense - on all the "mutation" hooks all implementations will be called. (i.e. for the policies that don't return a value) - for any policy function that returns a value (such as get_dagbag_import_timeout) then the plugins will be tried in LIFO order until a non-None value is returned. (No code needed for us -- Pluggy handles this out of the box with the firstresult=True property on the hookspec decorator). [1]: https://pluggy.readthedocs.io/en/latest/index.html | 3 年前 | |
Update provider.yaml.schema.json | 3 年前 | |
Add Documentation for notification feature extension (#29191) * Add Documentation for notification feature extension * fixup! Add Documentation for notification feature extension * Update docs/apache-airflow/howto/notifications.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * apply suggestions from code review --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> | 3 年前 | |
Aggressively cache entry points in process (#29625) | 3 年前 | |
Make airflow package type check compatible (#9791) | 5 年前 | |
AIP-51 - Misc. Compatibility Checks (#28375) * Remove pickling executor coupling from backfill_job.py (apache#27930) * Remove pickling executor coupling from scheduler_job.py (apache#27930) * Remove executor coupling from sentry.py, add tests for supports_sentry Executor attribute (apache#27930) * Use default executor helper method and rework pickling support attribute (apache#27930) | 3 年前 | |
Move sqlalchemy.ext.declarative imports to orm (#29862) | 3 年前 | |
Improve type hinting in stats.py (#30024) | 3 年前 | |
Completed D400 for multiple folders (#27767) | 3 年前 | |
Completed D400 for multiple folders (#27767) | 3 年前 | |
Don't use importlib.metadata to get Version for speed (#29723) As discovered by @uranusjr in other PRs, loading the Metadata info at runtime is surprisingly expensive. Recent versions of setuptools (including the one we already say we depend upon in pyproject.toml) have the ability to pull the version from an attribute using an "ast-eval" method so this keeps the property of there being a single source of truth for the Airflow version, it just moves that places to airflow/__init__.py. You might wonder why this particular case matters at runtime? In the grand scheme of things it likely doesn't, except that the airflow/operators/python.py imports this at the top level (or did before this PR) which made me look in to it and discover a quick win here. | 3 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 2 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 6 年前 | ||
| 3 年前 | ||
| 5 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 5 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 |