| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Wait for asset compilation to finish before starting airflow in Breeze (#28575) Asset compilation is performed in background and especially if it is run for the first time it might take some time. At the same time database intialization is done so usually it is not a problem but it some cases database might be initialized faster and webserver would start without assets compiled leading to nasty output. This change adds waiting for the compilation - tmux will not split screens and run webserver before it is done. | 3 年前 | |
Breeze help images are now consistently generated in pre-commit (#26032) Images generated for Breeze Help can slightly differ, however we want them to be generated always consistently. Therefore the images are now (when using pre-commit) generated inside the CI image rather than locally. This change also introduces much more selective image generation which will make the image generation much faster in case you change any of breeze code. Previously, any change in any breeze command parameters triggered regeneration of the images. With this change, only those commands that get changed get regenerated (and information about which images need to be regenerated is printed). This change removes one of the Breeze setup commands - the hash generation - as it is now done internally by the regenerate command and does not need to be externalized (--check-only) flag provides a way how to check if breeze images need regeneration. | 3 年前 | |
Add capability of iterating locally on constraint files for CI image (#30002) Whenever you want to upgrade a set of dependencies (for example all google packages) you might deal with a set of dependencies that are conflicting or difficult to upgrade all-at-once. And there might be multiple people working on it in parallel. This brings in the necessity of building the CI image of breeze with a different set of constraints than the ones in "main" branch of Airflow and share it with others working with you. This PR adds the capability of storing a temporary set of constraints in the repository (during PR iteration) and use this constraint file to build the CI image, rather than constraints from the repository. It also provides appropriate README.md file explaining how to iterate and regenerate such constraint files - even multiple times when iterating. | 3 年前 | |
Move LOAD_DEFAULT_CONNECTIONS env var to database config section in CI (#24536) | 4 年前 | |
Fix and augment check-for-inclusive-language CI check (#29549) * Fix and augment check-for-inclusive-language CI check Related: #15994 #23090 This PR addresses a few items related to inclusive language use and the CI check: - There are several occurrences of "dummy" throughout documentation; however the current CI check for preventing non-inclusive language doesn't inspect docs/ files. Ideally, the docs also include inclusive language. - Even when removing docs/ from the exclusion list, non-inclusive language was still escaping pygrep. Upon inspection, the (?x) inline modifier was missing from the regex (although intended in #23090). Adding this modifier revealed these "dummy" instances and others related non-inclusive occurrences which were previously uncaught. - The exclusion list seemed too broad in places. There are still instances in which directories are excluded as a whole, but the list now is more tailored to non-inclusive occurrences that are beyond the purview of Airflow, history, dev/test files, etc. * Update links in get_pandas_df() of BigQueryHook * dummy-command -> placeholder-command in _KubernetesDecoratedOperator | 3 年前 | |
Enable string normalization in python formatting (other) (#27206) | 3 年前 | |
Variables set in variables.env are automatically exported (#28633) The variables set in variables.env were not automatically exported. | 3 年前 | |
Fix output buffering for breeze testing test (#28433) Since we disabled warnings and redirected them to the output file, we lost continuous output of the progress of tests when they were run as breeze testing tests command. The whole summary of the tests appeared only at the end of tests. This turned out to be Python output buffering in "filter_out_warning" class. It is fixed by flushing sys.stdout after every printed line now. | 3 年前 | |
Enable Markdownlint rule - MD022/blanks-around-headings (#12225) https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md022---headings-should-be-surrounded-by-blank-lines | 5 年前 | |
Parallelize build of documentation. (#15062) This is far more complex than it should be because of autoapi problems with parallel execution. Unfortunately autoapi does not cope well when several autoapis are run in parallel on the same code - even if they are run in separate processes and for different packages. Autoapi uses common _doctree and _api directories generated in the source code and they override each other if two or more of them run in parallel. The solution in this PR is mostly applicable for CI environment. In this case we have docker images that have been already built using current sources so we can safely run separate docker containers without mapping the sources and run generation of documentation separtely and independently in each container. This seems to work really well, speeding up docs generation 2x in public GitHub runners and 8x in self-hosted runners. Public runners: * 27m -> 15m Self-hosted runners: * 27m -> < 8m | 5 年前 | |
Fix integration with codecov (#29145) | 3 年前 | |
Fix links to sources for examples (#24386) The links to example sources in exampleinclude have been broken in a number of providers and they were additionally broken by AIP-47. This PR fixes it. Fixes: #23632 Fixes: https://github.com/apache/airflow-site/issues/536 | 4 年前 | |
Parallelize build of documentation. (#15062) This is far more complex than it should be because of autoapi problems with parallel execution. Unfortunately autoapi does not cope well when several autoapis are run in parallel on the same code - even if they are run in separate processes and for different packages. Autoapi uses common _doctree and _api directories generated in the source code and they override each other if two or more of them run in parallel. The solution in this PR is mostly applicable for CI environment. In this case we have docker images that have been already built using current sources so we can safely run separate docker containers without mapping the sources and run generation of documentation separtely and independently in each container. This seems to work really well, speeding up docs generation 2x in public GitHub runners and 8x in self-hosted runners. Public runners: * 27m -> 15m Self-hosted runners: * 27m -> < 8m | 5 年前 | |
Filter out non-editable apache-airflow from constraints. (#25847) The constraint files should not contain apache-airflow. So far apache-airflow has been automatically filtered out by the fact that it has been an editable installation and we filtered out all editable and file installations from pip freeze. However as of ~12 August 2022, likely setuptools change triggered a slight behaviour change when eager-upgrade installation of providers from PyPi was run in the CI image. Previously airflow remained an editable install, but as of 12th of August, such an installation causes removal of editable airlfow install and reinstalls it in non-editable mode. While this change is somewhat confusing, we have to protect against it and remove apache-airflow from pip freeze also in non-editable mode. | 3 年前 | |
Add possibility to specify command to run at Breeze entry via env var (#25288) Breeze environment can be easily customized by specifying your own init.sh file in files/airflow-breeze-config/init.sh. However, there are certain scenarios (for example GitHub Codespaces), where you cannot do that, but you can easily specify an environment variable to set when you enter Breeze container - and you need to customize your environment there, becasue you need to make your airflow code re-linked from Github default "/workspace/airflow" to "/opt/airflow"). In such case it is much more convenient to provide the command to run via environment variable and eval the command at entering the image. This PR implements such a possibility. | 3 年前 | |
Don't use importlib.metadata to get Version for speed (#29723) As discovered by @uranusjr in other PRs, loading the Metadata info at runtime is surprisingly expensive. Recent versions of setuptools (including the one we already say we depend upon in pyproject.toml) have the ability to pull the version from an attribute using an "ast-eval" method so this keeps the property of there being a single source of truth for the Airflow version, it just moves that places to airflow/__init__.py. You might wonder why this particular case matters at runtime? In the grand scheme of things it likely doesn't, except that the airflow/operators/python.py imports this at the top level (or did before this PR) which made me look in to it and discover a quick win here. | 3 年前 | |
Re-enables MyPy in non-failure mode (#19890) This PR enables mypy back as pre-commit for local changes after the #19317 switched to Python 3.7 but also it separates out mypy to a separate non-failing step in CI. In the CI we will be able to see remaining mypy errors. This will allow us to gradually fix all the mypy errors and enable mypy back when we got all the problems fixed. | 4 年前 | |
Mark Airflow directory in container as safe for git commands (#29386) There is a new setting/version of git in GitHub Actions that started checking the ownership of the Git repository. Since in case of the provider commands we run them inside docker image as root user (this is in order to isolate the provider package building from the main CI environment), the owner of such directory is different (runner user) than the user that runs the git command (root). This change marks the current git directory for such commands as safe, regardles from the discrepancy. This config is global and run inside the image, so it is safe to leave it after methods complete as containers are torn-down after completing package preparation. This PR also improves diagnostics. Previously the git remote add output was redirected to dev null as there was no way it could fail, but this turned to be false - the output of the git remote add commnd is now also printed for diagnostics. | 3 年前 | |
Enable string normalization in python formatting (other) (#27206) | 3 年前 | |
Move provider issue generation to breeze (#28352) The exercise where another PMC performed a release process had proven useful to show the value of Breeze in release management commands - there were no problems with running the Breeze commands, but for scripts put elsewhere (docs building, dev) it was a bit problematic because each required own virtualenv. This is the first PR to simplify that - one less venv for the issue generation process - now it is moved to Breeze so no more setup is needed. | 3 年前 | |
Speed-up release management and shell Breeze commands on Mac (#27426) The release management and shell Breeze command on Mac should be now quite a bit faster: 1) The .pyc and __pycache__ are now deleted at entry of shell command and when docker commands are run - on the host. This will not remove all files sometimes - if they are root owned but we print warning in this case and suggest to use the breeze ci fix-ownership command 2) When preparing packages we are now using rsync rather than removing/copying sources. This should be way faster even on Mounted Mac system. | 3 年前 | |
Use MariaDB client binaries in arm64 image for support MySQL backend (#29519) | 3 年前 | |
Enable string normalization in python formatting (other) (#27206) | 3 年前 | |
Use inclusive words in apache airflow project (#23090) | 4 年前 | |
Improve 'start-airflow' experience for users and ui developers (#26341) Since we moved the "compile-www-assets" to host (and deferred it to pre-commit to prepare and manage appropriate node virtualenv), starting airflow via start-airflow command had become slower, adding quite a few seconds always when you run the command. Also it was a bit convoluted to start dev mode - you had to stop and restart webserver and run dev asset recompilation. This change improves the start-airflow experience in several ways: * the compile-www-assets pre-commit in start airflow runs in a background, daemon thread. This allows start-airflow command to continue even if asset compilation takes quite some time. By the time webserver starts and database initializes and the user launches browser it should be completed anyway. * the assets are not recompiled if none of the "www" files changed which for most people who run airflow without modifying the UI part will make it really fast to re-run start-airflow * we have a new flag --skip-assets-compilation added that will skip asset compilation even if the www files were modified * finally we have a new "--dev-mode" when starting airflow that will run the "dev" mode of assets compilation. And thanks to the background, daemon thread, assets compilation in dev mode will continue running as long as the start-airflow command is running All this should naturally plug in the exisitng start-airflow usage and add new capabilities for those who develop UI (you can run dev mode now without manually restarting anything). | 3 年前 | |
New Breeze command start-airflow, it replaces the previous flag (#11157) | 5 年前 | |
Improve "other" test category selection (#28630) The "Other" test category automatically selects all tests that are not included in any of the regular categories. That is to make sure that we do not forget to add any directory that has been added. However this led to a long directory selection for "Other" category including system tests that have been automatically added there. However those tests are always skipped in regular tests and collecting those tests during "Other" execution is not needed and slows it down. Similarly System tests changes were treated as "Other change" for incoming PRs. This means that any change to system tests would trigger "all tests" rather than selective subset of those - the same as any core change. However System tests are also part of the documentation, so any change in system tests should trigger docs builds. This change improves it in a few ways: * Other tests now do not include "System Tests" - they are treated the same way as other test categories (but not included in test category selection for now - until we get a good way of breeze-integration for System Tests * They are also excluded from treating them as "other" change when considering which tests to run. Changes to system tests will not trigger "all" tests, just those that accompanying changes would trigger. * The changes to system tests only, however, triggers docs build because those are triggered by any source change.a * The __pycache__ directories are removed from the list of "Other" packages to run. * In order to make sure system tests are pytest-collectable, we perform pytest collection for all tests right after downloading the CI images and verifying them. This makes sure that the tests are collectible before we even attempt to run them - this way we avoid unnecessary machine spin-up and breze installation for the multiple jobs that run the tests. This slows down feedback time a litle, but should increase overall robustness of the test suite. * an old, unused nosetest collection script doing the same in the past has been removed (it was discovered during implementation) Noticed in: #28319 | 3 年前 | |
Enable string normalization in python formatting (other) (#27206) | 3 年前 | |
Align cncf provider file names with AIP-21 (#29905) Changes to import paths: operators.kubernetes_pod -> operators.pod triggers.kubernetes_pod -> triggers.pod This PR is backward compatible | 3 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 4 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 5 年前 | ||
| 5 年前 | ||
| 3 年前 | ||
| 4 年前 | ||
| 5 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 4 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 4 年前 | ||
| 3 年前 | ||
| 5 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 |