Skip to content

Fix UI tests#25345

Draft
crazytonyli wants to merge 1 commit intotrunkfrom
uitest/fix-self-hosted-site-login
Draft

Fix UI tests#25345
crazytonyli wants to merge 1 commit intotrunkfrom
uitest/fix-self-hosted-site-login

Conversation

@crazytonyli
Copy link
Contributor

@crazytonyli crazytonyli commented Mar 4, 2026

Investigated and fixed by Claude Code. The blow is Claude Code's analysis.


UI Test Failure Root Causes

Documented: 2026-03-04

Issue 1: WireMock Not Running

Affected tests: All UI tests
Symptom: NSURLErrorDomain Code=-1004 "Could not connect to the server." or app crashes on launch.
Root cause: The UI tests depend on a WireMock mock server running on localhost:8282 for API mocking. Without it, the app cannot complete any network requests, causing test failures or crashes.

Fix: Start WireMock before running UI tests:

./API-Mocks/scripts/start.sh 8282

See also API-Mocks/README.md and docs/ui-tests.md (step 2: rake mocks).

Issue 2: iPad Sidebar Navigation Timeout

Affected tests: All iPad tests that use makeMainNavigationComponent() (AppSettingsTests, NotificationTests, ReaderTests, etc.)
Symptom: Tests timeout with repeated retries, each ~90 seconds apart.
Root cause: SidebarScreen (in UITestsFoundation) initializes with collectionViews["sidebar_list"] as its expected element. On iPad, this SwiftUI List with .listStyle(.sidebar) renders as a UICollectionView inside UISplitViewController's sidebar column. Due to a UIKit/SwiftUI accessibility quirk, the CollectionView reports isAccessibilityElement visibility as false (isVisible == 0), even though it is rendered on screen and its child elements are visible and interactive. Since ScreenObject waits for isHittable == true (which requires isVisible == true), the initialization times out after 45 seconds, exceeding the 60-second test timeout.

Fix: Changed SidebarScreen's expected element from collectionViews["sidebar_list"] to buttons["sidebar_me"], which is always present in the sidebar and correctly reports visibility.

Issue 3: Block Editor Text View Flakiness (Known, Handled by Retry)

Affected tests: EditorGutenbergTests_01 (testTextPostPublish, testUndoRedo)
Symptom: First attempt fails with "Failed to [tap] not hittable: TextView", but succeeds on retry.
Root cause: The Gutenberg block editor loads its content asynchronously via GutenbergKit (WebView-based). On the first cold launch in a test session, the WebView text views report isHittable == false even though they exist in the accessibility tree with valid frames. XCTest's doubleTap() requires hittability, so it fails. On retry, the WebView resources are cached and the text views become hittable quickly.

Attempted fixes (coordinate-based tapping, waitForIsHittable with various timeouts) either caused the test to hang at later steps or exceeded the test execution time allowance. The WebView text views may never report as hittable on cold start — this appears to be a fundamental limitation of XCTest with WebView-based editors.

Mitigation: The test plan uses retryOnFailure mode (up to 3 attempts). The tests reliably pass on the second attempt. No code change needed.

@dangermattic
Copy link
Collaborator

1 Message
📖 This PR is still a Draft: some checks will be skipped.

Generated by 🚫 Danger

@sonarqubecloud
Copy link

sonarqubecloud bot commented Mar 4, 2026

@wpmobilebot
Copy link
Contributor

App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number31309
VersionPR #25345
Bundle IDorg.wordpress.alpha
Commit3378454
Installation URL3866bi6qcpatg
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot
Copy link
Contributor

App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number31309
VersionPR #25345
Bundle IDcom.jetpack.alpha
Commit3378454
Installation URL7a8ujgo2dqeig
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot
Copy link
Contributor

🤖 Build Failure Analysis

This build has failures. Claude has analyzed them - check the build annotations for details.

@crazytonyli
Copy link
Contributor Author

@wordpress-mobile/apps-infrastructure If I remember it correctly, UI tests used to retry failures, right? It appears that's no longer the case?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants