| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Separate instruction to install OS dependencies in images (#25565) This change will allow to experiment with other base images (for example CentOS that our users highly demand) but also it has a few nice simplifications and improvements along the way: * no more runtime parameters for CI image (they only make sense for PROD image) * no more support for Buster image (it is end of life in August) * Dockerfile has now less embedded default values (most of them moved to inlined bash script) * configuration for yarn sources is removed (we do not need yarn any more in our images) * additional pure-dev dependencies in CI image are passed through ADDITIONAL_DEV_DEPS * dev installation does not remove installation cache, making the CI image slightly bigger but easier for devel use - to install new dependencies (no need for apt-get update before installation) * latest patchlevels of various tools we use for CI were bumped | 3 年前 | |
Converts Dockerfiles to be standalone (#22492) This change is one of the biggest optimizations to the Dockerfiles that from the very beginning was a goal, but it has been enabled by switching to buildkit and recent relase of support for the 1.4 dockerfile syntax. This syntax introduced two features: * heredocs * links for COPY commands Both changes allows to solve multiple problems: * COPY for build scripts suffer from permission problems. Depending on umask setting of the host, the scripts could have different group permissions and invalidate docker cache. Inlining the scripts (automatically by pre-commit) gets rid of the problem completely * COPY --link allows to optimize and parallelize builds for Dockerfile.ci embedded source code. This should speed up not only building the images locally but also it will allow to use more efficiently cache for the CI builds (in case no source code change, the builds will use pre-cached layers from the cache more efficiently (and in parallel) * The PROD Dockerfile is now completely standalone. You do not need to have any folders or files to build Airlfow image. At the same time the versatility and support for multiple ways on how you can build the image (as described in https://airflow.apache.org/docs/docker-stack/build.html is maintained (this was a goal from the very beginning of the PROD Dockerfile but it was not easily achievable - heredocs allow to inline scripts that are used for the build and the pre-commits will make sure that there is one source of truth and nicely editable scripts for both PROD and CI Dockerfile. The last point is really cool, because it allows our users to build custom dockerfiles without checking out the code of Airflow, it is enough to download the latest released Dockerfile and they can easily build the image. Overall - this change will vastly optimize build speed for both PROD and CI images in multiple scenarios. | 4 年前 | |
Separate instruction to install OS dependencies in images (#25565) This change will allow to experiment with other base images (for example CentOS that our users highly demand) but also it has a few nice simplifications and improvements along the way: * no more runtime parameters for CI image (they only make sense for PROD image) * no more support for Buster image (it is end of life in August) * Dockerfile has now less embedded default values (most of them moved to inlined bash script) * configuration for yarn sources is removed (we do not need yarn any more in our images) * additional pure-dev dependencies in CI image are passed through ADDITIONAL_DEV_DEPS * dev installation does not remove installation cache, making the CI image slightly bigger but easier for devel use - to install new dependencies (no need for apt-get update before installation) * latest patchlevels of various tools we use for CI were bumped | 3 年前 | |
Separate instruction to install OS dependencies in images (#25565) This change will allow to experiment with other base images (for example CentOS that our users highly demand) but also it has a few nice simplifications and improvements along the way: * no more runtime parameters for CI image (they only make sense for PROD image) * no more support for Buster image (it is end of life in August) * Dockerfile has now less embedded default values (most of them moved to inlined bash script) * configuration for yarn sources is removed (we do not need yarn any more in our images) * additional pure-dev dependencies in CI image are passed through ADDITIONAL_DEV_DEPS * dev installation does not remove installation cache, making the CI image slightly bigger but easier for devel use - to install new dependencies (no need for apt-get update before installation) * latest patchlevels of various tools we use for CI were bumped | 3 年前 | |
Separate instruction to install OS dependencies in images (#25565) This change will allow to experiment with other base images (for example CentOS that our users highly demand) but also it has a few nice simplifications and improvements along the way: * no more runtime parameters for CI image (they only make sense for PROD image) * no more support for Buster image (it is end of life in August) * Dockerfile has now less embedded default values (most of them moved to inlined bash script) * configuration for yarn sources is removed (we do not need yarn any more in our images) * additional pure-dev dependencies in CI image are passed through ADDITIONAL_DEV_DEPS * dev installation does not remove installation cache, making the CI image slightly bigger but easier for devel use - to install new dependencies (no need for apt-get update before installation) * latest patchlevels of various tools we use for CI were bumped | 3 年前 | |
Separate instruction to install OS dependencies in images (#25565) This change will allow to experiment with other base images (for example CentOS that our users highly demand) but also it has a few nice simplifications and improvements along the way: * no more runtime parameters for CI image (they only make sense for PROD image) * no more support for Buster image (it is end of life in August) * Dockerfile has now less embedded default values (most of them moved to inlined bash script) * configuration for yarn sources is removed (we do not need yarn any more in our images) * additional pure-dev dependencies in CI image are passed through ADDITIONAL_DEV_DEPS * dev installation does not remove installation cache, making the CI image slightly bigger but easier for devel use - to install new dependencies (no need for apt-get update before installation) * latest patchlevels of various tools we use for CI were bumped | 3 年前 | |
Converts Dockerfiles to be standalone (#22492) This change is one of the biggest optimizations to the Dockerfiles that from the very beginning was a goal, but it has been enabled by switching to buildkit and recent relase of support for the 1.4 dockerfile syntax. This syntax introduced two features: * heredocs * links for COPY commands Both changes allows to solve multiple problems: * COPY for build scripts suffer from permission problems. Depending on umask setting of the host, the scripts could have different group permissions and invalidate docker cache. Inlining the scripts (automatically by pre-commit) gets rid of the problem completely * COPY --link allows to optimize and parallelize builds for Dockerfile.ci embedded source code. This should speed up not only building the images locally but also it will allow to use more efficiently cache for the CI builds (in case no source code change, the builds will use pre-cached layers from the cache more efficiently (and in parallel) * The PROD Dockerfile is now completely standalone. You do not need to have any folders or files to build Airlfow image. At the same time the versatility and support for multiple ways on how you can build the image (as described in https://airflow.apache.org/docs/docker-stack/build.html is maintained (this was a goal from the very beginning of the PROD Dockerfile but it was not easily achievable - heredocs allow to inline scripts that are used for the build and the pre-commits will make sure that there is one source of truth and nicely editable scripts for both PROD and CI Dockerfile. The last point is really cool, because it allows our users to build custom dockerfiles without checking out the code of Airflow, it is enough to download the latest released Dockerfile and they can easily build the image. Overall - this change will vastly optimize build speed for both PROD and CI images in multiple scenarios. | 4 年前 | |
Separate instruction to install OS dependencies in images (#25565) This change will allow to experiment with other base images (for example CentOS that our users highly demand) but also it has a few nice simplifications and improvements along the way: * no more runtime parameters for CI image (they only make sense for PROD image) * no more support for Buster image (it is end of life in August) * Dockerfile has now less embedded default values (most of them moved to inlined bash script) * configuration for yarn sources is removed (we do not need yarn any more in our images) * additional pure-dev dependencies in CI image are passed through ADDITIONAL_DEV_DEPS * dev installation does not remove installation cache, making the CI image slightly bigger but easier for devel use - to install new dependencies (no need for apt-get update before installation) * latest patchlevels of various tools we use for CI were bumped | 3 年前 | |
Apache Airflow 2.3.4 has been released (#25913) | 3 年前 | |
Apache Airflow 2.3.4 has been released (#25913) | 3 年前 | |
Converts Dockerfiles to be standalone (#22492) This change is one of the biggest optimizations to the Dockerfiles that from the very beginning was a goal, but it has been enabled by switching to buildkit and recent relase of support for the 1.4 dockerfile syntax. This syntax introduced two features: * heredocs * links for COPY commands Both changes allows to solve multiple problems: * COPY for build scripts suffer from permission problems. Depending on umask setting of the host, the scripts could have different group permissions and invalidate docker cache. Inlining the scripts (automatically by pre-commit) gets rid of the problem completely * COPY --link allows to optimize and parallelize builds for Dockerfile.ci embedded source code. This should speed up not only building the images locally but also it will allow to use more efficiently cache for the CI builds (in case no source code change, the builds will use pre-cached layers from the cache more efficiently (and in parallel) * The PROD Dockerfile is now completely standalone. You do not need to have any folders or files to build Airlfow image. At the same time the versatility and support for multiple ways on how you can build the image (as described in https://airflow.apache.org/docs/docker-stack/build.html is maintained (this was a goal from the very beginning of the PROD Dockerfile but it was not easily achievable - heredocs allow to inline scripts that are used for the build and the pre-commits will make sure that there is one source of truth and nicely editable scripts for both PROD and CI Dockerfile. The last point is really cool, because it allows our users to build custom dockerfiles without checking out the code of Airflow, it is enough to download the latest released Dockerfile and they can easily build the image. Overall - this change will vastly optimize build speed for both PROD and CI images in multiple scenarios. | 4 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
| 4 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 4 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 3 年前 | ||
| 4 年前 |