Add a test criterion for documentation or getting started

[ways2read]

When conducting the reading systems evaluations, some functions are possible but obscure or difficult to discover. This test could evaluate whether there is any information to help people get going with the reading system (with AT or other a11y related features).

[avneeshsingh]

The fundamental test book is designed to test accessibility of reading systems, using objective testing criteria. The issues like stability of reading systems, helpful documentation etc. can be captured in the short and long descriptions.
It is important to keep "accessibility testing methodology" focused on accessibility. In fact some people even expect us to make it completely aligned with EPUB Accessibility specs 1.0 and WCAG 2.0. Therefore s uch non-accessibility related information should be captured in comments.

Another way can be to present a short survey after the tester completes all the forms, and we can have a link on the reading systems grid, leading to the answers of the survey.
it can ask questions like:
Was sufficient documentation available to help you getting started?
Answer:
Was the user interface intuitive to use?
Answer:

Any other information that you like to provide about usability or stability.
Answer:

[ways2read]

The issue can be clarified that it relates to getting started from an accessibility perspective, not for general users. For example, some of the shortcuts for keyboard users, especially screen reader users, are hard to discover. This was raised as a requirement on one of the reading system calls. I agree that the guidance can be improved for the long description, this should not be a test criterion. Only a very small number of reading systems would pass today. Richard

…

________________________________ From: Avneesh Singh <notifications@github.com> Sent: Friday, June 29, 2018 5:39:34 PM To: daisy/epub-accessibility-tests Cc: Richard Orme; Author Subject: Re: [daisy/epub-accessibility-tests] Add a test criterion for documentation or getting started (#10) The fundamental test book is designed to test accessibility of reading systems, using objective testing criteria. The issues like stability of reading systems, helpful documentation etc. can be captured in the short and long descriptions. It is important to keep "accessibility testing methodology" focused on accessibility. In fact some people even expect us to make it completely aligned with EPUB Accessibility specs 1.0 and WCAG 2.0. Therefore such non-accessibility related information should be captured in comments. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIq--5ZE06WsD7WA7x7h_6sdh14R3rhnks5uBlhGgaJpZM4SqvEo>.

[avneeshsingh]

"I agree that the guidance can be improved for the long description, this should not be a test criterion. Only a very small number of reading systems would pass today."

There is another important principle that leads to this. The testing on epubtest.org is based on evaluating accessibility features using objective tests. And we decided to provide more information in reading systems descriptions at inclusivepublishing.org for example the reading system may be accessible but with poor usability, it may be accessible but highly unstable etc.
The easily available useful getting starting guide also fals in this category.
We capture information about these with help of comments on epubtest.org and show the information more predominatly on inclusivepublishing.org.
So, I think that we should continue with this principle, epubtest.org should capture objective accessibility functionality related information and the other helping information like usability, useful getting started guide, stability of the product etc. should be highlighted on inclusivepublishing.org.
This also ensures that information on inclusivepublishing.org is not confined to becoming just a narration of test results table of epubtest.org.

[GeorgeKerscher]

The ease of getting started with a Reading System is very interesting. We do not have this as part of our testing, but it may be something to consider and could be added to the Roundup.

Too late for the early 2024 titles, but perhaps good for an experimental title.

Related to this would be if the producer of the Reading System has an accessibility page and a mechanism to provide comments and report bugs.