Error Text Test Error

testAddProfileActivity_configChange_inputShortPin_create_pinLengthError.txt

// TODO(#3363): Test passes on Pixel3a sometimes and fails on Pixel3.
  @Test
  fun testAddProfileActivity_configChange_inputShortPin_create_pinLengthError() {
    launch(AddProfileActivity::class.java).use {
      testCoroutineDispatchers.runCurrent()
      onView(isRoot()).perform(orientationLandscape())
      onView(
        allOf(
          withId(R.id.add_profile_activity_user_name_edit_text),
          isDescendantOfA(withId(R.id.add_profile_activity_user_name))
        )
      ).perform(scrollTo()).perform(
        editTextInputAction.appendText("test"),
        closeSoftKeyboard()
      )
      testCoroutineDispatchers.runCurrent()
      onView(withId(R.id.add_profile_activity_pin_check_box)).perform(scrollTo())
      onView(withId(R.id.add_profile_activity_pin_check_box)).perform(click())
      onView(
        allOf(
          withId(R.id.add_profile_activity_pin_edit_text),
          isDescendantOfA(withId(R.id.add_profile_activity_pin))
        )
      ).perform(scrollTo())
      onView(
        allOf(
          withId(R.id.add_profile_activity_pin_edit_text),
          isDescendantOfA(withId(R.id.add_profile_activity_pin))
        )
      ).perform(
        editTextInputAction.appendText("12"),
        closeSoftKeyboard()
      )
      testCoroutineDispatchers.runCurrent()
      // S1 (very rare):Scrolling to view was attempted, but the view is not displayed
      // S2 (very rare): failed to perform scroll to (Scrolling to view was attempted, but the view is not displayed)
      onView(withId(R.id.add_profile_activity_create_button)).perform(scrollTo())
      // This line is to solve button not fully displayed which will failed the accessibility check
      onView(withId(R.id.add_profile_activity_scroll_view)).perform(swipeUp())
      waitForMatch(withId(R.id.add_profile_activity_pin), 200L)
      // S3(Most Common): Doesn't perform click
      onView(withId(R.id.add_profile_activity_create_button)).perform(click())
      waitForMatch(withId(R.id.add_profile_activity_pin), 200L)
      onView(withId(R.id.add_profile_activity_pin)).perform(scrollTo())
      // S3 result : Didn't scroll to add_profile_activity_pin check failed
//      androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'The expected error text is 'Your PIN should be 3 digits long.'' doesn't match the selected view.
//      Expected: The expected error text is 'Your PIN should be 3 digits long.'
//      Got: "TextInputLayout{id=2131296340, res-name=add_profile_activity_pin, visibility=VISIBLE, width=786,
      onView(withId(R.id.add_profile_activity_pin))
        .check(
          matches(
            hasErrorText(
              context.resources.getString(R.string.add_profile_error_pin_length)
            )
          )
        )
    }
  }

  private fun waitForMatch(viewMatcher: Matcher<View>, millis: Long): ViewAction {
    return object : ViewAction {
      override fun getDescription(): String {
        return "wait for a specific view with matcher <$viewMatcher> during $millis millis."
      }

      override fun getConstraints(): Matcher<View> {
        return isRoot()
      }

      override fun perform(uiController: UiController?, view: View?) {
        checkNotNull(uiController)
        uiController.loopMainThreadUntilIdle()
        val startTime = System.currentTimeMillis()
        val endTime = startTime + millis

        do {
          if (TreeIterables.breadthFirstViewTraversal(view).any { viewMatcher.matches(it) }) {
            return
          }
          uiController.loopMainThreadForAtLeast(50)
        } while (System.currentTimeMillis() < endTime)

        // Couldn't match in time.
        throw PerformException.Builder()
          .withActionDescription(description)
          .withViewDescription(HumanReadables.describe(view))
          .withCause(TimeoutException())
          .build()
      }
    }
  }

//Here is the updated test code
testCoroutineDispatchers.runCurrent()

//breakpoint A
//if I stop here and continue, then the error sometimes will be shown as failed to perform scroll and program stop. If it continues with no error and stops at B, from my observation on the emulator, the click seems not performed since the error message doesn't show.

onView(withId(R.id.add_profile_activity_create_button))
.perform(scrollTo())
.perform(click())

//breakpoint B

onView(withId(R.id.add_profile_activity_pin))
.perform(scrollTo())
.check(
matches(
hasErrorText(
context.resources.getString(R.string.add_profile_error_pin_length)
)
)
)

This is kind of hard to understand without a video. What do you mean by the click seems not performed since the error message doesn't show. which view is expected to be clicked on?

Just to take a step back, do you think that the way the test is set up, is an exact replica of the manual steps needed to test this functionality?

Thanks for the question, I will post a video later for better clarification.

Sorry for the confusion, this test situation is to inputs the profile information with a short pin, so that when click the add_profile_activity_create_button it shows the pin is too short.
In the test code ln4 above, it clicks the add_profile_activity_create_button so that the error text is supposed to appear in the emulator. But after the execution, the text doesn't appear.

To my understanding, the logic of the test code is a good replica. But may be due to some mechanism issue that I am not familiar with, it doesn't behave the same as the manual test in the emulator.

The behavior of the test is not consist, so it is a bit hard for me to present all scenarios.
Here is a case, that failed to scroll to add_profile_activity_create_button which basically acts as a submission for all info.
https://user-images.githubusercontent.com/74568012/278706334-31d7831e-9b65-4cdd-bfc7-c88af77243ee.mp4

Looked into this a bit, the issue here is that the create button is never fully displayed, therefore not clicked. This is probably a scrollview isue, and could be solved maybe using a nested scrollview, or a custom espresso function.

Here is the updated video: I did the scroll to make the button fully visible and even tried to add delay before and after the click action. But the test is still flaky.
https://gist.github.com/assets/74568012/898a3733-33e7-4610-894b-99953989aacd