| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
[rust png] Update cxx from 1.0.135 to 1.0.143. This CL updates the WORKSPACE.bazel dependency on cxx and cxxbridge-cmd crates to version 1.0.143. This CL supports migrating png crate to version 0.18, by providing support for additional Seek trait implementations from https://github.com/dtolnay/cxx/pull/1454 The following commands have been used to mirror the crates: $ go run ./bazel/gcs_mirror/gcs_mirror.go \ --url https://crates.io/api/v1/crates/cxxbridge-cmd/1.0.143/download \ --sha256 19c3062da294104183e1c34ea9887941d4d8c74f6195ce9fbb430ac4b5290ede \ --add_suffix=.tar.gz $ go run ./bazel/gcs_mirror/gcs_mirror.go \ --url https://crates.io/api/v1/crates/cxx/1.0.143/download \ --sha256 050906babad73f9b32a91cecc3063ff1e2235226dd2367dd839fd6fbc941c68a \ --add_suffix=.tar.gz Bug: chromium:400455848 Change-Id: Id8c35978edff9ca0b2978db6ede9930b4bc93413 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/959216 Reviewed-by: Kaylee Lubick <kjlubick@google.com> Commit-Queue: Łukasz Anforowicz <lukasza@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Roll Expat from 624da0f5 to 8e49998f (3866 commits) https://chromium.googlesource.com/external/github.com/libexpat/libexpat.git/+log/624da0f593bb8d7e146b9f42b06d8e6c80d032a3..8e49998f003d693213b538ef765814c7d21abada Change-Id: Ia4c881960042ca1a82056c5db46029e6cafd3979 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/994776 Reviewed-by: Florin Malita <fmalita@google.com> Commit-Queue: Ben Wagner <bungeman@google.com> Commit-Queue: Florin Malita <fmalita@google.com> Auto-Submit: Ben Wagner <bungeman@google.com> | 1 年前 | |
[Fontations] Roll Fontations Skrifa 0.34 -> 0.36 Read-fonts 0.32 -> 0.34 Roll process: Update versions in MODULE.bazel and bazel/external/fontations/Cargo.toml, build to ensure Bazel locks versions, for example: $ bazel build //tools/viewer Bug: skia:40045335 Cq-Include-Trybots: luci.skia.skia.primary:BazelBuild-fontations-release-linux_x64,Build-Mac-Clang-x86_64-Debug-Fontations,Build-Ubuntu24.04-Clang-x86_64-Debug-Fontations,Test-Mac14-Clang-MacMini8.1-CPU-AVX2-x86_64-Debug-All-NativeFonts_Fontations,Test-Mac15-Clang-MacBookPro15.1-CPU-AppleIntel-x86_64-Debug-All-NativeFonts_Fontations,Test-Ubuntu24.04-Clang-GCE-CPU-AVX2-x86_64-Debug-All-NativeFonts_Fontations Change-Id: I465f1516225d1c7854b85238b33c8411fb5e9da3 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/1050116 Auto-Submit: Dominik Röttsches <drott@google.com> Reviewed-by: Kaylee Lubick <kjlubick@google.com> Commit-Queue: Kaylee Lubick <kjlubick@google.com> | 10 个月前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Roll FreeType from 5d4e649f to 7172bd11 (112 commits) https://chromium.googlesource.com/chromium/src/third_party/freetype2.git/+log/5d4e649f740c675426fbe4cdaffc53ee2a4cb954..7172bd11badd468f6a86dba0b1769d624ead885c Change-Id: Id773f09fa44b1252e2a15a72a6e7a0cba91fc139 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/994816 Auto-Submit: Ben Wagner <bungeman@google.com> Reviewed-by: Julia Lavrova <jlavrova@google.com> Reviewed-by: Florin Malita <fmalita@google.com> Commit-Queue: Florin Malita <fmalita@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Roll HarfBuzz from ca3cd48f to 4e767271 (792 commits) https://chromium.googlesource.com/external/github.com/harfbuzz/harfbuzz.git/+log/ca3cd48fa3e06fa81d7c8a3f716cca44ed2de26a..4e76727183f388c3bc17b11db49c74e7e6c191d8 Change-Id: Ica056f83005b0a3ba86d5e85f817b899b77e8462 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/994716 Reviewed-by: Julia Lavrova <jlavrova@google.com> Commit-Queue: Ben Wagner <bungeman@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Fix fontations, icu4x, and vello rust builds This copies deps out of //bazel/external/icu4x:Cargo.toml //bazel/external/fontations:Cargo.toml //bazel/external/vello:Cargo.toml I had tried to use the crates.from_cargo but found odd errors with it erroring on unrelated .toml files. It seems that on linux, the rust libraries now need libc++ dynamically linked in. I had to run sudo apt-get install libc++-dev locally to get the new link commands to work. Our CI machines already have that installed. Getting rust to compile on x64 macs required a few changes: - rules_rust doesn't generate configs for intel macs by default, so I had to modify the from_specs in MODULE.bazel to incorporate the "x86_64-apple-darwin" target - I had to register a "compile for x64 mac on arm64 mac" toolchain (and this allows us to clean up the BUILD.gn commands). While doing this, I cleaned up some toolchain names. - I had to change the mac toolchain trampoline because Bazel's #include detection did not work as expected with the absolute paths, particularly when Rust was shelling out to C++. Using relative paths seems to have remedied this. Change-Id: I35053561039231c8ebc1f2062ff3004700e6d977 Bug: b/417974542 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/993638 Reviewed-by: Eric Boren <borenet@google.com> Commit-Queue: Kaylee Lubick <kjlubick@google.com> | 1 年前 | |
Make MODULE.bazel.lock updates easier for autorollers Everytime we updated deps.bzl, the MODULE.bazel.lock file needed to have a proprietary checksum re-calculated (via bazel mod tidy). This is problematic for autorollers, which strive to use find/replace (e.g. old revision with new revision) because that can be achieved w/o needing a full checkout (and are thus faster). I tried a few things to remove the checksum entirely, and eventually drew inspiration from how rules for go work [1], by returning a non-empty extension_metadata [2][3] that contains the reproducible flag. This tells Bazel "my generated rules are hermetic all by themselves" so it doesn't have to track the hash of the input files (neither cpp_modules.bzl nor deps.json). The result is simpler, as we don't have to generate hand-written Starlark code, but can instead just generate JSON data that is later parsed. Autorollers should be able to just keep doing find and replace on deps.json when updating. [1] https://github.com/bazel-contrib/bazel-gazelle/blob/4440fb7e8b6c29b7e0b69a77572bd91ed325398e/internal/bzlmod/go_deps.bzl#L336 [2] https://github.com/bazel-contrib/bazel-gazelle/blob/4440fb7e8b6c29b7e0b69a77572bd91ed325398e/internal/bzlmod/utils.bzl#L122-L137 [3] https://bazel.build/rules/lib/builtins/module_ctx#extension_metadata Change-Id: Ia36412a1d4ebec434bcf9986601d84481b1ee8aa Bug: b/418260494 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/995896 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Add simd/arm to includes for cc_library libjpeg_turbo This enables third party code to build against Skia using bzlmod. Without this there is this error: external/skia++cpp_modules+libjpeg_turbo/simd/arm/aarch64/jchuff-neon.c:36:10: fatal error: 'neon-compat.h' file not found 36 | #include "neon-compat.h" | Change-Id: I30803ae817bf47055b02b2c67b658db6262f3c70 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/1040796 Reviewed-by: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Florin Malita <fmalita@google.com> Commit-Queue: Brian Salomon <briansalomon@gmail.com> | 10 个月前 | |
Make MODULE.bazel.lock updates easier for autorollers Everytime we updated deps.bzl, the MODULE.bazel.lock file needed to have a proprietary checksum re-calculated (via bazel mod tidy). This is problematic for autorollers, which strive to use find/replace (e.g. old revision with new revision) because that can be achieved w/o needing a full checkout (and are thus faster). I tried a few things to remove the checksum entirely, and eventually drew inspiration from how rules for go work [1], by returning a non-empty extension_metadata [2][3] that contains the reproducible flag. This tells Bazel "my generated rules are hermetic all by themselves" so it doesn't have to track the hash of the input files (neither cpp_modules.bzl nor deps.json). The result is simpler, as we don't have to generate hand-written Starlark code, but can instead just generate JSON data that is later parsed. Autorollers should be able to just keep doing find and replace on deps.json when updating. [1] https://github.com/bazel-contrib/bazel-gazelle/blob/4440fb7e8b6c29b7e0b69a77572bd91ed325398e/internal/bzlmod/go_deps.bzl#L336 [2] https://github.com/bazel-contrib/bazel-gazelle/blob/4440fb7e8b6c29b7e0b69a77572bd91ed325398e/internal/bzlmod/utils.bzl#L122-L137 [3] https://bazel.build/rules/lib/builtins/module_ctx#extension_metadata Change-Id: Ia36412a1d4ebec434bcf9986601d84481b1ee8aa Bug: b/418260494 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/995896 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
[bazel] Add //gm/png_codec.cpp. This proof-of-concept CL shows what it would be like if we reimplemented DM's CodecSrcs as GMs. The motivation is that the CodecSrc class, and other such classes, are conceptually very similar to a GM: they do some work and produce an image as their output. So, rather than introducing the notion of various "sources" in our Bazel work, I thought I would try rewriting DM's codec tests as GMs. This particular CL focuses on PNG codec tests. Adding support for more file formats shouldn't be too different, and there is some potential for code reuse (e.g. a "BaseCodecGM" superclass). Rather than relying on multiple nested loops to iterate over all possible combinations of options like DM does, this CL pushes those decisions to Bazel. In //gm/png_codec.bzl, we define multiple Bazel targets where each target focuses on a single combination of codec mode, color type, alpha type, etc. In //gm/png_codec.cpp, we take those options as command-line flags, and we register exactly one GM per image (whereas DM registers multiple GMs per image in a triply-nested loop; one GM per combination of options). This hopefully reduces the complexity of the C++ code, and provides better sharding by breaking up tests into smaller Bazel targets. Notes to reviewers: - I recommend reading //gm/png_codec.cpp from top to bottom, and reading the skia.googlesource.com links in my comments alongside this CL while reviewing. A lot of the code in //gm/png_codec.cpp is copied verbatim from the CodecSrc class and related functions. - (Optional but recommended) See https://skia-review.googlesource.com/c/skia/+/728798 for an earlier attempt, where I tried to rewrite the entire CodecSrc class as a single CodecGM class. After chatting with kjlubick@, we decided it would be simpler to break that giant class into smaller ones focused e.g. on one file format per class. This CL is my attempt at isolating the parts of CodecSrc and related code that are only relevant to PNG files. Tested with: $ bazel test //gm:png_codec_tests \ --config=linux_rbe \ --config=debug \ --test_output=streamed \ --strategy=TestRunner=local Bug: b/40045301 Change-Id: I3e36c1ce03ceb892e7fae7fef5c0882e76b233e0 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/734379 Reviewed-by: Kevin Lubick <kjlubick@google.com> Reviewed-by: Brian Osman <brianosman@google.com> Commit-Queue: Leandro Lovisolo <lovisolo@google.com> | 2 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Fix fontations, icu4x, and vello rust builds This copies deps out of //bazel/external/icu4x:Cargo.toml //bazel/external/fontations:Cargo.toml //bazel/external/vello:Cargo.toml I had tried to use the crates.from_cargo but found odd errors with it erroring on unrelated .toml files. It seems that on linux, the rust libraries now need libc++ dynamically linked in. I had to run sudo apt-get install libc++-dev locally to get the new link commands to work. Our CI machines already have that installed. Getting rust to compile on x64 macs required a few changes: - rules_rust doesn't generate configs for intel macs by default, so I had to modify the from_specs in MODULE.bazel to incorporate the "x86_64-apple-darwin" target - I had to register a "compile for x64 mac on arm64 mac" toolchain (and this allows us to clean up the BUILD.gn commands). While doing this, I cleaned up some toolchain names. - I had to change the mac toolchain trampoline because Bazel's #include detection did not work as expected with the absolute paths, particularly when Rust was shelling out to C++. Using relative paths seems to have remedied this. Change-Id: I35053561039231c8ebc1f2062ff3004700e6d977 Bug: b/417974542 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/993638 Reviewed-by: Eric Boren <borenet@google.com> Commit-Queue: Kaylee Lubick <kjlubick@google.com> | 1 年前 | |
Manual Roll vulkan-deps from 1c3b39ad6dc0 to d5d3942f4b8f (14 revisions) Also updates the bazel/external/vulkan_headers/BUILD.bazel to reference the new headers. https://chromium.googlesource.com/vulkan-deps.git/+log/1c3b39ad6dc0..d5d3942f4b8f Also rolling transitive DEPS: https://skia.googlesource.com/external/github.com/KhronosGroup/SPIRV-Headers/+log/fd96661925488574fe247a779babe5d380b63635..3b9447dc98371e96b59a6225bd062a9867e1d203 https://skia.googlesource.com/external/github.com/KhronosGroup/SPIRV-Tools/+log/7dda3c01fb4c0f9941d3cb792947d57d896ac55f..90cfb3e96463b267e2a3b88181edf7bfde08d947 https://chromium.googlesource.com/external/github.com/KhronosGroup/Vulkan-Headers/+log/b11eecd68fb4b770f30fe2c9da522ff966f95b1e..1d6c53f65443ceeb97d3bdc695aaecc7ea6cc441 https://chromium.googlesource.com/external/github.com/KhronosGroup/Vulkan-Tools/+log/e4fb76dc08f139df0436e9c3031f75be5e1f6264..fbe722654b7173da961398cf78bd4a62d1839b65 https://chromium.googlesource.com/external/github.com/KhronosGroup/Vulkan-Utility-Libraries/+log/03e1445cc7cce22baeeef8eff7bb934362d040eb..e48ae20a7938b01aee62806bfcdafe8a0883b1e4 If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/vulkan-deps-skia-autoroll Please CC michaelludwig@google.com,skiabot@google.com on the revert to ensure that a human is aware of the problem. To file a bug in skia: https://bugs.chromium.org/p/skia/issues/entry To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md Cq-Include-Trybots: skia/skia.primary:Build-Ubuntu24.04-Clang-x86_64-Release-ANGLE;skia/skia.primary:Test-Win10-Clang-Golo-GPU-QuadroP400-x86_64-Debug-All-ANGLE Bug: None Tbr: michaelludwig@google.com Change-Id: Ie6f27cc3119bb95052c94622e6f4574e65033ace Reviewed-on: https://skia-review.googlesource.com/c/skia/+/1004719 Commit-Queue: Michael Ludwig <michaelludwig@google.com> Reviewed-by: Michael Ludwig <michaelludwig@google.com> Reviewed-by: Robert Phillips <robertphillips@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
Update Bazel to 8.2.1 and use bzlmod The update to bzlmod [1] changes how we handle third_party deps (dramatically) and how our toolchains work (minorly). To review and understand the changes, I suggest going language by language through the changes: C++ === We have Bazel check out git repositories listed in deps.bzl (like before), but this has been converted to a module_extension [2] which behaves basically the same. In MODULE.bazel, we load the extension and then specify all the dependencies that Skia directly depends on from there. If we can depend on versions from the Bazel registry [3], for example bazel_dep(name = "libpng", version = "1.6.47.bcr.1") we could remove the git repository versions and this might make things easier for external clients if there are version mismatches. A big change was required with how we customize our compilation of expat, FreeType, Harfbuzz, and ICU. Previously, we'd copied in (or cleverly referenced) files from the Skia repo in the third party copy of the code and this sort of worked. External clients had to use some helpers to load these configs and it was Fine (tm). This does not work with bzlmod's dependency graph. I tried several things to make this work cleanly and ran into road blocks with either RBE or external client usage (or both). See earlier patchsets for that, if interested. What *did* work (at least ok) was making a patch file per third party dep that needed it using tools/generate_patches.py to combine all those files and specify an output location. This has the downside of not being in sync, but I made the existing roll scripts for expat, FreeType, Harfbuzz shell out to the script to minimize that. Toolchains needed to be slightly updated to have a different path in the external/ output directory, which now includes some prefixing. To make this as robust as possible, I made the trampoline scripts use paths relative to where the script lives and use a glob to find the path no matter what the prefix is. Clang remains at 15.0.1 for Bazel builds. Rust ==== The rust toolchain has been updated to 1.86.0 (this was the primary impetus for this change). External crate dependences use crate_universe [4] and specify the versions we need directly in MODULE.bazel instead of using a toml file. I chose that approach because it seemed more direct when one wants to figure out what version our Bazel build uses. All crates except cxx worked "out of the box", meaning the rules_rust generated BUILD.bazel files for them in a way that just worked. I had to use crate.annotation to customize cxx to use bazel/external/cxx/BUILD.bazel.skia in addition to the generated rule so we can overwrite the include path. This was a bit fiddly and required me to spend a while looking through the bazel cache to experiment with what the output was. The only other major change is how we reference rust dependencies now. For example in src/ports/fontations/BUILD.bazel see that all rust dependencies are based out of @crates instead of all being top level dependencies. Again, poking into the bazel cache is helpful for figuring out what the generated target names are. Another upside of this update is the big long "splicing workspaces" seems to have gone away. Golang ====== The golang version is now 1.24.2. These were the most fiddly changes - particularly with some changes in the ecosystem related to protos. In MODULE.bazel, the go.mod file is read in to get both the go version and the direct golang dependencies. go_rules uses gazelle [5] to generate BUILD.bazel files for any deps if they don't have them. This includes generating BUILD.bazel files for proto files, which are a bit messy due to there being some conflicting ways to include these and thus resolve the deps. This [6] is why I ended up mothballing the upgrade of the Skia Infra repo in favor of a minimal [7] change. In this case, we only had to tell the go.chromium.org/luci repo not to autogenerate .go files for .proto files because there are pre-existing ones. While building task drivers, there was a problem where some go deps were not being brought in and thus our transitive dependencies were failing to build. This is why tools.go was added (inspired by [8] to force those to be included. Python3 ======= This was a straightforward upgrade to Python 3.13. MODULE.bazel still reads requirements.txt to figure out what to download. The BUILD.bazel file we build for Dawn uses these requirements, but we didn't have to change how that worked at all. Other Notes =========== Renames ------- There are many mechanical changes (e.g. io_bazel_rules_go->rules_go) throughout as a result of bzlmod having different (non prefixed) package names in some cases. bazel/buildrc ------------- Sandboxing seems much faster in this new Bazel version on Mac. Thus I deleted the no_sandbox config. This change also updates the version of emsdk we use to compile CanvasKit. I had to squelch some warnings that they have via our config because I didn't see another way to override their toolchain settings. buildifier ---------- Updated the version used by Bazel to 8.0.3 I suggest folks also update their local version too, so presubmits work [9] Not working (yet) / disabled ============================ - karma_tests (JS tests for CanvasKit). This was based on long-deprecated rules and I didn't have the time to fix this. We still have GN based tests for CanvasKit, so this was low enough priority to skip for now - codesize_test. git_common.WithGitFinder was not working. As this is a test for our task_driver and has been stable for a while, it seemed like a low priority to fix. - bazel run //:gofmt. This moved with new go_rules and I couldn't find an easy replacement. - bazel run //:errcheck. This needs to shell out to golang and doesn't work on systems that don't have go installed because of the hermeticity. Not sure what to do about that. - clang_ios and clang_windows_amd64 toolchains. We don't have any CI jobs that use these atm, and they are lower priority for now. I made some changes based on other toolchains, but didn't test these. [1] https://bazel.build/external/migration [2] https://bazel.build/external/extension [3] https://registry.bazel.build/modules/libpng [4] https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html [5] https://github.com/bazel-contrib/bazel-gazelle/blob/master/extensions.md#go_deps [6] https://skia-review.googlesource.com/c/buildbot/+/988440/11/MODULE.bazel [7] http://review.skia.org/990801 [8] https://github.com/google/skia-buildbot/blob/b4a7f41e01ea288eadd197f9eee0b1a8a12c5aa4/tools.go [9] https://github.com/bazelbuild/buildtools/releases/tag/v8.0.3 Change-Id: I68be15c5c4a5d2b14359fa460075591dafab336a Bug: b/413044303 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/984976 Commit-Queue: Kaylee Lubick <kjlubick@google.com> Reviewed-by: Dominik Röttsches <drott@google.com> Reviewed-by: Eric Boren <borenet@google.com> | 1 年前 | |
[bazel] Prefer includes to strip_include_prefix As described in bazel/external/README.md, includes will make the headers available via -isystem instead of -I, which means compile warnings in those will not be visible to dependents. Change-Id: Ie09b899d02d4011244678c1cfd4ef80645c6c8f8 Bug: skia:12541 Reviewed-on: https://skia-review.googlesource.com/c/skia/+/562311 Reviewed-by: Ben Wagner <bungeman@google.com> | 3 年前 |
This folder is where we put BUILD.bazel files for external (e.g. third party) dependencies.
If a dependency supports Bazel, we should use those rules, but if the dependency does not, we need to create our own rules in a subdirectory.
We generally compile third_party deps from source. If we do, we clone the repository and use
the given BUILD.bazel file to build it. This is specified in the WORKSPACE.bazel (e.g.
new_local_repository or new_git_repository) and we refer to those targets using labels like
@freetype, or @libpng.
Some third_party deps we only link against prebuilt versions. For those, we do not involve
WORKSPACE.bazel and link to them directly, e.g. //bazel/external/fontconfig.
Notes
Avoid strip_include_prefix
strip_include_prefix
causes the header path for the library to be added to the compiler include search path with -I,
which means Clang will treat it like a file in Skia proper. This means if those headers have
issues that Clang's diagnostic warnings catch (e.g. missing override), we will see those warnings
and the build will fail.
Generally, we do not want to have to fix third_party code's warnings, so instead of
using strip_include_prefix, use includes instead. This is more ergonomic, as it can let us
expose header files from multiple locations (e.g. freetype has its API in includes and the
customization headers in builds) and adds these to the search path with -isystem. Clang ignores
warnings in these "system" headers, which means our warnings will be focused to the Skia code base.