[ci] Add a legacy Android build-all test (#4005)
Adds the ability to replace portions of the flutter created app with saved copies, and adds a second build-all phase for Android that uses a Flutter 2.0.6-created android/ directory (AGP 4.1, Gradle 6.7) to catch issues like https://github.com/flutter/flutter/issues/125621 and https://github.com/flutter/flutter/issues/125482 prior to release.
Includes some incidental cleanup:
- Extracts a helper method for adjusting files, so that this doesn't add even more copies of basically identical code.
- (This was motivated by an earlier version of the PR that added modifications to several more files, which I ended up undoing, but the cleanup seemed worth keeping.)
- Adds missing unit test coverage.
- Reworks the unit tests to use a mock process manager and dummy files, instead of each test actually calling flutter create, which made each new test add several seconds to the unit test suite.
- While this reduces the integration-style coverage, in practice the integration tests of the repo tooling is the CI itself, so the unit tests should be true unit tests.
- Changes the non-legacy test to Kotlin; we were still testing with Java even though Kotlin has been the default for quite a while, so we weren't testing what most new users would actually be running. Since we now have a legacy test, I used Java there to cover both.
- Removes some dead code for modifying the AndroidManifest.xml; when trying to set up unit tests for it I discovered that it no longer matches anything in an actual project. It dates back to the original command, and seems to have been a camera-related hack, which we clearly no longer need since it wasn't working and camera still works in build-all.
This is captured somewhat in the README in the legacy project directory, but to document the approach here: originally I was going to add flags to change individual items (AGP version, Gradle version), but quickly ran into the fact that selective downgrading is extremely fragile. E.g.,:
- The Kotlin version set in current projects doesn't work when downgrading AGP and Gradle.
- The app template can unconditionally use anything (e.g., namespace) that the AGP version it uses supports, so arbitrary future breakage is possible.
It's also less useful as a real test of what a plugin client's project likely looks like. Starting with a complete platform directory, and doing whatever the minimal changes are to keep it working, will likely reflect a common real-world scenario. On the flip side, the reason this doesn't use a complete 2.0.6 project, but instead is based on specific platform directories, is that we don't want to waste time manually maintaining, e.g., old Dart code that is irrelevant to the goal of the test. For now this only uses Android because that's where we've seen problems in practice, but we can alway add other legacy platform tests later if we find a need.
Fixes https://github.com/flutter/flutter/issues/125689
[tool] Move changed file detection to base command class (#8730)
Consolidates the code to find all changed file paths into the PackageLoopingCommand class that is the base of almost all of the repo tooling commands. This in a preparatory PR for a future change to allow each command to define a list of files or file patterns that definitively *don't* affect that test, so that CI can be smarter about what tests to run (e.g., not running expensive integration tests for README changes).
A side effect of this change is that tests of almost all commands now need a mock GitDir instance. This would add a lot of copy/pasted boilerplate to the test setup, and there is already too much of that, so instead this refactors common test setup:
- Creating a memory file system
- Populating it with a packages directory
- Creating a RecordingProcessRunner to mock out process calls
- Creating a mock GitDir that forwards to a RecordingProcessRunner
into a helper method (using records and destructuring to easily return multiple values). While some tests don't need all of these steps, those that don't can easily ignore parts of it, and it will make it much easier to update tests in the future if they need them, and it makes the setup much more consistent which makes it easier to reason about test setup in general.
Prep for https://github.com/flutter/flutter/issues/136394
[tool] Move changed file detection to base command class (#8730)
Consolidates the code to find all changed file paths into the PackageLoopingCommand class that is the base of almost all of the repo tooling commands. This in a preparatory PR for a future change to allow each command to define a list of files or file patterns that definitively *don't* affect that test, so that CI can be smarter about what tests to run (e.g., not running expensive integration tests for README changes).
A side effect of this change is that tests of almost all commands now need a mock GitDir instance. This would add a lot of copy/pasted boilerplate to the test setup, and there is already too much of that, so instead this refactors common test setup:
- Creating a memory file system
- Populating it with a packages directory
- Creating a RecordingProcessRunner to mock out process calls
- Creating a mock GitDir that forwards to a RecordingProcessRunner
into a helper method (using records and destructuring to easily return multiple values). While some tests don't need all of these steps, those that don't can easily ignore parts of it, and it will make it much easier to update tests in the future if they need them, and it makes the setup much more consistent which makes it easier to reason about test setup in general.
Prep for https://github.com/flutter/flutter/issues/136394
[tool] Move changed file detection to base command class (#8730)
Consolidates the code to find all changed file paths into the PackageLoopingCommand class that is the base of almost all of the repo tooling commands. This in a preparatory PR for a future change to allow each command to define a list of files or file patterns that definitively *don't* affect that test, so that CI can be smarter about what tests to run (e.g., not running expensive integration tests for README changes).
A side effect of this change is that tests of almost all commands now need a mock GitDir instance. This would add a lot of copy/pasted boilerplate to the test setup, and there is already too much of that, so instead this refactors common test setup:
- Creating a memory file system
- Populating it with a packages directory
- Creating a RecordingProcessRunner to mock out process calls
- Creating a mock GitDir that forwards to a RecordingProcessRunner
into a helper method (using records and destructuring to easily return multiple values). While some tests don't need all of these steps, those that don't can easily ignore parts of it, and it will make it much easier to update tests in the future if they need them, and it makes the setup much more consistent which makes it easier to reason about test setup in general.
Prep for https://github.com/flutter/flutter/issues/136394
[tool] Add initial file-based command skipping (#8928)
Adds initial file-based filtering. This does not attempt to be comprehensive, just to get some low-hanging fruit, and to create a blueprint for anyone to follow in the future when adding more filtering. I expect that once this is in place, what will happen is that as we notice cases where PRs are hitting slow or flaky tests that they clearly don't need to, we can incrementally improve the filtering on demand.
Fixes https://github.com/flutter/flutter/issues/136394
[tool] Move changed file detection to base command class (#8730)
Consolidates the code to find all changed file paths into the PackageLoopingCommand class that is the base of almost all of the repo tooling commands. This in a preparatory PR for a future change to allow each command to define a list of files or file patterns that definitively *don't* affect that test, so that CI can be smarter about what tests to run (e.g., not running expensive integration tests for README changes).
A side effect of this change is that tests of almost all commands now need a mock GitDir instance. This would add a lot of copy/pasted boilerplate to the test setup, and there is already too much of that, so instead this refactors common test setup:
- Creating a memory file system
- Populating it with a packages directory
- Creating a RecordingProcessRunner to mock out process calls
- Creating a mock GitDir that forwards to a RecordingProcessRunner
into a helper method (using records and destructuring to easily return multiple values). While some tests don't need all of these steps, those that don't can easily ignore parts of it, and it will make it much easier to update tests in the future if they need them, and it makes the setup much more consistent which makes it easier to reason about test setup in general.
Prep for https://github.com/flutter/flutter/issues/136394
[tool] Move changed file detection to base command class (#8730)
Consolidates the code to find all changed file paths into the PackageLoopingCommand class that is the base of almost all of the repo tooling commands. This in a preparatory PR for a future change to allow each command to define a list of files or file patterns that definitively *don't* affect that test, so that CI can be smarter about what tests to run (e.g., not running expensive integration tests for README changes).
A side effect of this change is that tests of almost all commands now need a mock GitDir instance. This would add a lot of copy/pasted boilerplate to the test setup, and there is already too much of that, so instead this refactors common test setup:
- Creating a memory file system
- Populating it with a packages directory
- Creating a RecordingProcessRunner to mock out process calls
- Creating a mock GitDir that forwards to a RecordingProcessRunner
into a helper method (using records and destructuring to easily return multiple values). While some tests don't need all of these steps, those that don't can easily ignore parts of it, and it will make it much easier to update tests in the future if they need them, and it makes the setup much more consistent which makes it easier to reason about test setup in general.
Prep for https://github.com/flutter/flutter/issues/136394
[tool] Move changed file detection to base command class (#8730)
Consolidates the code to find all changed file paths into the PackageLoopingCommand class that is the base of almost all of the repo tooling commands. This in a preparatory PR for a future change to allow each command to define a list of files or file patterns that definitively *don't* affect that test, so that CI can be smarter about what tests to run (e.g., not running expensive integration tests for README changes).
A side effect of this change is that tests of almost all commands now need a mock GitDir instance. This would add a lot of copy/pasted boilerplate to the test setup, and there is already too much of that, so instead this refactors common test setup:
- Creating a memory file system
- Populating it with a packages directory
- Creating a RecordingProcessRunner to mock out process calls
- Creating a mock GitDir that forwards to a RecordingProcessRunner
into a helper method (using records and destructuring to easily return multiple values). While some tests don't need all of these steps, those that don't can easily ignore parts of it, and it will make it much easier to update tests in the future if they need them, and it makes the setup much more consistent which makes it easier to reason about test setup in general.
Prep for https://github.com/flutter/flutter/issues/136394
[tool] Move changed file detection to base command class (#8730)
Consolidates the code to find all changed file paths into the PackageLoopingCommand class that is the base of almost all of the repo tooling commands. This in a preparatory PR for a future change to allow each command to define a list of files or file patterns that definitively *don't* affect that test, so that CI can be smarter about what tests to run (e.g., not running expensive integration tests for README changes).
A side effect of this change is that tests of almost all commands now need a mock GitDir instance. This would add a lot of copy/pasted boilerplate to the test setup, and there is already too much of that, so instead this refactors common test setup:
- Creating a memory file system
- Populating it with a packages directory
- Creating a RecordingProcessRunner to mock out process calls
- Creating a mock GitDir that forwards to a RecordingProcessRunner
into a helper method (using records and destructuring to easily return multiple values). While some tests don't need all of these steps, those that don't can easily ignore parts of it, and it will make it much easier to update tests in the future if they need them, and it makes the setup much more consistent which makes it easier to reason about test setup in general.
Prep for https://github.com/flutter/flutter/issues/136394