Test planning and design deep dive

[boazsender]

On the ARIA-AT CG call today (2020 Nov 19th) we discussed a few different approaches for how to plan the test suite. This is timely as we are currently ramping up on generating new test material (@sinabahram and @jscholes), creating new automation APIs (@zcorpan et al), and designing the test results reports (@isaacdurazo and @spectranaut).

Two main approaches for the the scope of a test were discussed:

1. include specify key strokes to get to an interaction so that they can be automated
2. keep test conditions abstract and high level so that they can run across multiple screen readers.

A breakout session is being scheduled to hammer out the details. Outcomes should be posted here.

Additionally, the report design/implementation team requested that once we have that fleshed out we address the issues arsing in #336, and align on writing assertions and grouping test results in such a way that they can be reported on.

Notes document: https://docs.google.com/document/d/1jDf_gEQjRppLyEDEW0qPVbq1QaCT36bVDl9JkI-nvCU/edit

[boazsender]

As a place holder, I have a potential resolution which I think we could accommodate both of the above approaches by making our assertion model include a high level test assertion with key stroke mappings for each AT. I could imagine this being accomplished in with a couple of different solutions, which I think @spectranaut was starting to enumerate today on the CG call:

A) Each test is one file for the high level assertion, with different sub-tests/functions for that setup the state for each AT.

B) One file per AT, with some kind of keying system that lets a test author register a test onto a group. Or potentiall by filenaming convention, like my-test-name.nvda.js, my-test-name.narrator.js, etc.

[s3ththompson]

Just wanted to leave a summary of the call on this issue. Full notes here.

Attendees: Sina Bahram, James Scholes, Erika, Weston Thayer, Jes Daigle, Rebecca Merritz, Seth Thompson, Simon Pieters, Michael Fairchild, Boaz Sender

Agenda

1. Resolve how to make tests automatable (issue Test planning and design deep dive #337)
2. "Generic" or "comparable" tests across modes (issue "generic" or "comparable" tests -- should we be able to compare test results across NVDA, JAWS and VoiceOver? #336)
3. Structure the data around roles, states, properties (issue Add ability to distinguish between property, role, and state metadata #345) (cut for time)

Summary

1. Resolve how to make tests automatable (issue Test planning and design deep dive #337)

   The group cleared up terminology:

   A test can assert something (e.g., is a checkbox checked), and each assertion can be multiple “checks” (e.g., output contains the word “checked” and does not contain “not checked”).

   The main discussion was around the difference between:

   • asserting a whole string, with known good output
   • breaking an assertion down into multiple "checks" that might individually verify different parts of the output exist

   After confirming that the second approach (breaking an assertion down into more granular "checks") didn't preclude also checking for ordering and non-duplication, the group agreed that semantic sub-string granularity was better than opaque string matching.

   Next steps:

   • Finalize the automation test format, see followup issue Automation test format #349 driven by @zcorpan
   • If necessary, propose a second issue to discuss parametrization / templatization of tests (the way the assertions are written, the way the instructions are provided, and the way the instructions are specialized per AT) (as well as a recommendation on sequencing and cost / benefit analysis), assigned to @jscholes and @sinabahram

2. "Generic" or "comparable" tests across modes (issue "generic" or "comparable" tests -- should we be able to compare test results across NVDA, JAWS and VoiceOver? #336)

   The group determined that there is a need to clarify the correct semantic naming and grouping of individual tests:

   testID	title	appliesTo	mode	task
   1	Navigate to an unchecked checkbox in reading mode	JAWS,NVDA	reading	navigate to unchecked checkbox
   2	Navigate to an unchecked checkbox in interaction mode	JAWS,NVDA	interaction	navigate to unchecked checkbox
   3	Navigate to an unchecked checkbox	voiceover_macos	interaction	navigate to unchecked checkbox

   Currently we identify tests by their title, but there's a possibility that we should consider using task as the level of granularity presented in reports. In general, we need to align on how title is constructed and how the tests.csv data is collated for maximum readability in the reports page.

   Further discussion needed.

   Next steps:

   • Continue discussion on next CG call or schedule separate deep dive.

3. Structure the data around roles, states, properties (issue Add ability to distinguish between property, role, and state metadata #345)

   (Cut for time.)

[jscholes]

@s3ththompson

propose a second issue to discuss parametrization / templatization of tests

I've started a thread related to this in #364.