Ash Pixel Unit Testing: Test Failure Triage

TL;DR: If you made a UI change that you expect to change ash_pixeltests, you need to increment the revision number of the test by 1 in the C++ test source.

1. Introduction

ash_pixeltests is the test suite of image-based testing. If your CL breaks ash_pixeltests, it means that the CL touches product UI features (including but not limited to colors or view locations) and generates the screenshots different from the benchmark images. This doc teaches you how to handle ash_pixeltests failure.

If you have any questions, please reach out to //ash/test/pixel/OWNERS.

2. Finding Failed Tests

When ash_pixeltests is broken, the Gerrit review UI will display a redbox with the message "X untriaged, Y new images on patchset Z". For example:


Fig 1: the red box

To view the details of the failed tests, click on the redbox. This will open a list of records, each of which corresponds to a failed test. Here is one example record:


Fig 2: an example record

Each record provides numerous information. As a CL author, you only need to focus on the benchmark image name and the images.

In one record, the string following Test specifies the benchmark image that your CL breaks. In the example record, the benchmark image name is DemoAshPixelDiffTest.VerifyTopLevelWidgets.check_widgets.rev_1.ash.

There are always three images in a record. The images from left to right are:

The new screenshot
The image difference
The original benchmark image.

If the new screenshot seems wrong to you, the pixel test could catch a bug in your CL.

3. Locating the Test Code Line

Each benchmark’s name follows this rule:

{Test Suite Name}.{Test Case Name}.{Screenshot Name}.{rev_#number}.
{Platform Suffix}

where:

{Test Suite Name}: it is the test class’s name. In the example record, it is “DemoAshPixelDiffTest”.
{Test Case Name}: In the example record, it is “VerifyTopLevelWidgets”.
{Screenshot Name}: it is the string specified when taking the screenshot. In the example record, it is “check_widgets”.
{rev_#number}: it is the concatenation between the constant “rev_” and the revision number given in the test code. We will revisit this revision number in the doc later.
{Platform Suffix}: it indicates the platform on which the screenshot is taken. For ash pixel tests, it is always “ash”. NOTE: this doc only covers ash pixel tests. If the platform suffix is not “ash”, you are reading the wrong doc.

To locate the test file, you can search {Test Suite Name} in the Chromium code base. In that test file, you can search {Test Case Name} to find the test case. In that test case, you can search {Screenshot Name} to find the code line that generates the benchmark image.

Use the previous record as an example. To locate the test code that generates DemoAshPixelDiffTest.VerifyTopLevelWidgets.check_widgets.rev_1.ash, you should:

First, search “DemoAshPixelDiffTest” in the Chromium code base. You will find ash/test/pixel/demo_ash_pixel_diff_test.cc; then
Second, search “VerifyTopLevelWidgets” in that file. You will find the test case defined by TEST_F(DemoAshPixelDiffTest, VerifyTopLevelWidgets); then
Third, search "check_widgets" in the test case. You will find the exact code line that generates the screenshot.

4. Locating the Test Code Line

Assume that the new screenshot generated by your CL is expected. Recall the revision number mentioned in Section 3. Add a one-line change in your CL: increase the revision number by one.

For example, the original test code could be

EXPECT_TRUE(GetPixelDiffer()->CompareUiComponentsOnPrimaryScreen(
      "check_widgets",
      /*revision_number=*/1, widget1.get(), widget2.get(), widget3.get(),
      widget4.get()));

Then the code after your change should be

EXPECT_TRUE(GetPixelDiffer()->CompareUiComponentsOnPrimaryScreen(
      "check_widgets",
      /*revision_number=*/2, widget1.get(), widget2.get(), widget3.get(),
      widget4.get()));

The only difference in the pixel test code should be the revision number.

Submit your new CL and wait for LGTM from the test owner.