-
Notifications
You must be signed in to change notification settings - Fork 25
User research findings
If you have any questions or curiosities about the research findings shared here, please reach out to CMS-EAPD@cms.hhs.gov for additional information.
- Future research ideas were previously stored here, but are now on the Roadmap Mural.
- When we have research studies that are actively being planned, they will appear here.
Lock and Submit Research Round 1
- Internal Review
- Lock and Export
- Submit
- Dashboard
- Read Only View
Validation/Required Fields Check Research Round 1
- Required Fields vs Optional Fields
- Administratively Complete APDs
- Multi-user editing
- APD Internal Review
- Required Fields Check
MESC Feedback Session Summer 2021
- FFY, Updates, Documents
MES Expansion Round 1 Spring 2021
- Prototype of Create New APD, APD Overview, Activity Overview, Activity Schedule and Milestones, Risks and Alternatives, Conditions for Enhanced Funding, Help Drawers
State Officer Preview Cake Layer 1 Jan 2021
- Full APD Experience
Beta Testing of Cake Layer 1 Dec 2020
- Full APD Experience Round 11.5 Usability Testing May 2020
- Tables
- Including the FFP and budget table
- Proposed budget summary page
- New state costs summary table
- Cost Allocation and Other Funding
- Data entry and interactions
Round 11 Usability Testing April 2020
- Sidebar updates
- Paginated view, previous/continue buttons
- Autosave implementation
- Objectives and key results
- New team - general topics
Round 10 Usability Testing Feb 2020
- Paginated view
- Tabbed view
- Autosave messaging placement
- Autosave error design
Round 9 Usability Testing Nov 2019
- Modal interaction
- IA of the section
- FFP and cost allocation section
- Editing an activity
- Reordering activities
- Editing the activity name
- Adding new activities
- Deleting an activity
- Full APD Experience
- How do states update their APDs?
In-person usability testing at State HIT March 2019
- Process of saving as PDF and submitting
- New left navigation
- New dashboard
- Removal of the accordions, addition of section numbers
Federal Reviewer Interviews Jan 2019
- Full APD Experience
Round 8 Usability Testing Nov 2018
- Tables: actual expenses, cost allocation (quarterly), summary schedule table, summary budget table, federal share by quarter, incentive payments
- Executive summary
- Certify and submit
Round 7 Usability Testing Sept 2018
- State dashboard updates
- Key personnel moving to the contact info section
- Program activities: Performance measures and benchmarks, In-house cost categories, Private contractor costs
- Commenting
Round 6 Usability Testing August 2018
- State dashboard
- Navigation
- Help text
- Submission experience
- Program activities: Personnel, Contract Resources, Non-Personnel Costs, Standards and Conditions
- Assurances and Compliances
Round 5 Usability Testing June 2018
- Tables: Summary, Quarterly, Incentive payments, Program budget table
- Executive summary
- A/B test the long form vs separating by sections
Round 4 Usability Testing May 2018
- Use a real example of an HIT/HIE activity
- Help text for Program Summary and Program Activities
- Activity breakout in the prototype
- Navigation between sections/subsections
Round 3 Usability Testing April 2018
- State dashboard V1, long scroll concept
Round 2 Usability Testing March 2018
- Revised content prototype
- Schedule, personnel, and expenses sections
- Help text clarity
- Form fields
In-depth user interviews with one state March 2018
- Full APD Experience
Round 1 Usability Testing Feb 2018
- Activities, goals, and objectives
Initial User Interviews Jan 2018
- How do states and CMS think about and work through the APD process?
- What are the challenges with the current process?
- What are the basic needs of our users?
In March of 2022, we designed an interactive prototype for our lock and submit feature and tested it with 5 participants in an usability study. We came into this research wondering if folks would understand locking down an APD and we came out of the research with reassurance of the concept, insights on folk’s submission process, and actionable recommendations to drive our upcoming designs.
- To figure out use cases for unlocking an APD
- To ask folks what are the anticipated touch-points with eAPD after an APD is submitted
- To validate the current flow of the lock and submit
- To assess whether preview, lock/unlock, read only, download/export language makes sense to folks
- Most States understand what locking down an APD means
- States agree that there should be restricted access to certain actions like locking, unlocking, and deleting an APD but believe the state administrator role should allow for 2 to 3 designated users. This finding neither validates the decision to have only state administrator lock an APD nor negates it. It is a further testament to allowing flexibility within eAPD, so that different users no matter how their state APD process is structured can use eAPD and incorporate it into their existing well functioning processes. Still, it is important to note that since most of the users do not currently use eAPD, they did not know who their state administrator would be. It may be possible that they would feel more comfortable with having one state administrator if they selected their point of contact.
- Preventing further edits has more benefits than we initially thought. States mentioned that it would be especially beneficial when MMIS and E&E come in to play because there would be more authors writing a single APD and the APDs are generally bigger.
- The final step to creating an APD on eAPD is unclear to users. Folks were confused to see copy that mentioned exporting and other call to actions, once they had locked down their APD. It led to questions on whether they had to export again or whether there was an additional step to upload documents and submit your APD through eAPD. So, design wise, there needs to be a clear indication that an APD process on eAPD is complete and some clear next steps in the review process. It also be helpful to transition folks to a different page since the Export APD title once they have locked was causing some confusion.
- Most states thought they could submit within an APD. This insight was not a surprise to us, but further validation that once submitting is available on eAPD, it would benefit users greatly and change how they prepare an APD for submission.
- To prepare to submit an APD, they have to pass the APD and supporting documents around. A couple participants recommended allowing leadership to review the APD within eAPD, possibly in a read only state. It may be helpful to consider whether we can allow leadership to review and comment as a guest or whether to prioritize this sort of work later on. Commenting during the internal review stage would also allow folks and eAPD to have an audit trial of changes. Still we found in the existing design that being able to distinguish between an APD that is in a draft state and one that is fit for CMS, as well as version numbers would be helpful to distinguish changes as an APD is being handed off through leadership.
- Folks want the ability to upload supporting documents into eAPD. In fact, many participants assumed that they could upload document, especially on the Export APD page. In order to enable this, we need to do more than provide the component to upload documents, we would need to design a way for folks to be able to upload and submit as a package. That package would consist of all the documents within their decision letter including associated contracts and RFPs, and the dashboard would organize related documents into a package.
- Although the “delete” action being on the dashboard was alarming to a majority of participants, it often led to further conversation on the update experience. Most participants stated that they would not want to delete an APD because they would use some of the information in an updated APD. This led to more conversation about how we could further utilize existing APDS for updates or new APDs.
- Folks asked for a guide on the different fields/content within the dashboard. A handful of participants wanted to know more information specifically about who last edited and/or locked the APD. They also wanted guidance on where the version number would be pulled from; system generated or CMS-generated.
- None of the participants clicked into the APD voluntarily, but found the read-only view to be helpful for things like multi-user editing and commenting for internal reviews. Participants who oversee processes and multiple APDs, expected to see this version when clicking “preview” on the dashboard so that they could quickly review APDs.
- States are likely to log out once they have locked and exported their APD. They only to return to eAPD, if they receive comments from their State Officer or an REI.
- Unlocking an APD on the “read-only” view was not apparent to participants. Most participants did not notice that they could unlock an APD once they clicked on it. Once they were prompted to click the APD, a few participants mentioned that they would look at the sections that warranted response/updates from CMS before unlocking.
- Those reviewing APDs have different needs from a “preview” than those who are authoring APDs. We found that APD reviewers preferred the “read-only” view of the APD so they could click through each section quickly. Those who author and/or contribute to the APD content, preferred to see the information laid out in PDF form.
- A PDF Preview may be helpful once an APD has been locked and can no longer be edited. Participants stated that they would like to see the “online” version of the APD so they could review to make edits. After locking an APD, the concept of “previewing” it as a PDF made sense.
- Present an indication that they are done with an APD on eAPD once user locks and exports the APD
- Allow users to confirm that they submitted APD as an intermediary confirmation until submitting within eAPD is available
- Consider moving the card for internal review in the Review APD page
- Move the “delete” action to the bottom of the list of links on the dashboard
- Provide a “last edited by” or “locked date” label
- Consider updating the dashboard to make room for expansion
- Consider adding statuses in preparation for multi-user editing (locked, read-only, editable draft, etc)
- Change Preview to say “Preview PDF”
- Determined whether to elevate the preview function on the export page so that it lives as its own function instead of attached to the “Export APD” cards or place Preview to the “Review APD” page instead of Export APD
The Validation/ Required Fields Research marks a progression for eAPD to include some more short form research sessions that would can happen frequently and would allow us to get insights quick. This research was triggered by the required fields check component design. It was first designed about a year ago and revised the component to be tested.
- To make sure the validation error check concept works for state authors
- To ask how many folks will be using the system or working on something at the same time.
-
Most states did not assume that all fields within the APD would be required. When asked this question, it was an overwhelming no. This is likely due to convention. They were used to leaving some fields blank or annotating not applicable fields and communicating with their State Officer to ensure that it can be left unanswered. Also, they might be used to the APD template having all fields included, regardless of their APD type, so they may expect to be deciding which fields are not pertinent to their funding request rather than having the system do that determination.
-
Examples of optional fields were mostly consistent and non-contradictory across the five states, observed. Folks had a relatively consistent idea of what they would consider required compared to what they would consider to be optional/not applicable depending on the type of APD they were filling out. Most states agreed that the budgets were required, regardless of conditions. On the other hand, most states agreed that Alternative Considerations should be optional in a iAPD; Cost benefit analysis should be optional if a project is federally mandated and poses no benefit to the state; and private contractor costs should be optional in a project that is internally funded. A couple of states differed on whether state costs should be required or not but that determination is dependent on whether they see state costs as part of the budget or part of the Key State Personnel. Largely, this finding invalidation our assumption that “States will not know which fields are needed until told to in the required fields check.” They did have a solid understanding of what is required, so the fields listed as unanswered may not come as a shock to them. As a result, we do not need to rely on inline validations throughout the APD to require that fields be filled out and we might not need to indicate that a field is optional. If a field is optional but they have that information on hand, it would be beneficial to encourage them to answer the field to give CMS a greater picture into their funding request rather than indicate that it is optional because of a system requirement.
-
Folks currently use their APD template as a tool to determine which fields are required. All participants mentioned that they go through the APD template that their state provides, section by section to make sure that all the required fields are complete. Additionally, they reviewed their APD for administrative completeness when they were at their final draft of their APD. This would lead us to believe that our assumption “[m]ost states will not use the required fields check in the beginning to only answer ‘required’ fields, largely because authors will not know it exists until they reach the Export and Submit page” is true. It suggests that the Required Fields Check is on the right page, but it needs to be tied to an action such as as Export so that it can fit into their process and cannot be avoided.
-
Another way states determine if an APD is administratively complete is implementing quality assurance processes. States might have contractors (responsible for writing/reviewing APDs), a SOP or any person(s) delegated to make sure that the APD is polished. Having a quality assurance process reduce the chances of an APD being sent back and it can be beneficial to maintain APD quality during staff turnover.
-
States also use previously approved APDs to determine if their current APD is administratively complete. They noted that they use previously approved APDs as the standard for their APDs because if the APD was deemed as complete and approvable the first time, replicating the previous APDs should ensure an APD’s approval. It is unclear from the research whether most states do this but it is likely that the APD template they are given by their state officer has everything, including not applicable fields and they have to trust the discretion of their state officer to know which fields are required. This is likely why folks had little trust that the eAPD would only have fields pertinent to their APD.
-
After the sessions, it became clear that the motivation for making sure an APD was administratively complete was that it would be approvable. Additionally, this idea of having an APD that is approvable and isn’t sent back repeatedly for clarification may have affected participants perception of the the required fields checks usefulness. While we have steered away from including that language in the eAPD, it is clear that we need to clearly express the benefits for completing certain fields. If we can foster trust within the system by making it clear that the contents of the eAPD are what CMS expects and that they would not be provided extraneous fields that are not applicable to their APD through conscious branching of information based on APD type and other factors, and the implementation of this required fields check which could aid in the quality control and review of the APD.
-
NV recommended a way to “bypass’ the system required validation when a field is not applicable is to attest that the author has seen the alert but the field does not apply.
-
States have varied organizational structures and behaviors when it comes to APD writing, but in terms of review, the process is often collaborative. Most of our participants were APD writers for their state. They were either the sole writer for a particular APD type or worked alongside other writers. Typically, folks who wrote the APD were also responsible for internal reviews, making sure that the APD was administratively complete and the budget and content is written in an approvable manner. Also, the amount of people doing the completion check depends. Most folks said it was multiple people and for states where they were one or two APD writers per state, the authors often clarified the content or sought review from the subject matter experts, either consultants or people actually implementing the the project. As for our assumption that “the author(s) completing the APD are the same authors who would be reviewing the APD before exporting it,” this was partly true. We did not consider the fact that there are two reviews at play: first, the internal review amongst authors and subject matter experts and second, the review the official submitter performs before sending an APD to CMS.
-
The number of authors depends on the state’s size, funding request amount and number of active APDs.
-
Multi-editing functionality can serve as a method to track changes during synchronous/asynchronous collaboration. Multiple users recommended having the ability to track changes because they currently hand off the APD documents to various people and knowing who edited the APD and when would help with knowing that the contents of the APD is up to date and possibly in its final draft form. Track changes would also be helpful when the eAPD team starts thinking about how to lock down a submitted APD. It is also important to note that this insight further validated previous insights on multi-user editing.
-
Often times, a team might have a person writing the narrative fields and another completing the budget component. This finding suggests that although there may be multiple people working on one APD at the same time, it is unlikely that they would be collaborating on the same sections or fields. The eAPD team can use this finding to assess how granular we have to be when alerting users that there are multiple people editing the same APD.
-
The validation research revalidated the need for comments. Related to track changes, folks wanted the ability to relay clarifications amongst authors, internal reviewers and CMS within the system, which would aid in the reviewing process. It would also minimize the need to resubmit an APD every time a clarification is needed or necessary information is missing.
-
Some states have an official submitter who is often somewhat removed from the project implementation and APD writing. Our participant was the official submitter for their state and noted that if there are issues, such as missing fields and incorrect content, they do not edit the APD, they send the APD back to the author for revision. However, they are the one ultimately responsible if there are issues with the APDs in her state. Since, the APD document is often administratively complete by the time the official submitter reviews it, the required fields check would be more beneficial for the state authors and team members participating in the internal review.
-
When asked “How would you use this system to check your APD to make sure it is administratively complete?” participants knew to click ‘Scan for Incomplete Fields,’ but struggled to see how they would use the Required Fields Check in their current process. Most states thought the Required Fields Check would be useful, but may have seen the check as a disjointed step when incorporated into their existing processes. Luckily, we did have a couple of folks who were official submitters for their state, mentioned that with the Required Fields Check in place, they might delegate the process of reviewing an APD for administrative completeness down to either the authors or an intermediary step.
-
Users knew how to use the Required Fields Check, but the copy confused a few users. During the pilot interview with a designer and SME, they mentioned that the copy within the required fields check stating “Exiting the required fields check will remove the error notifications from the APD until the next time you start the check. APDs cannot be submitted until all required fields are complete.” could be interpreted as users can disregard required fields and that’s not the case. We changed the text to “APD cannot be submitted until all required fields are complete. Exiting the required fields review will pause the review until you choose to restart the review or complete all required fields.” and moved the “stop check” information to the bottom of the help drawer and it was not commented on by participants. The button copy on the export and submit page also caused some confusion and although we weren’t testing the Submit to CMS per se, the research highlighted a concern we have had with the button copy that ‘Submit’ implies that they can submit within eAPD and possible attach additional documents to make their APD administratively complete. So, it is a bit misleading when a user clicks the button and an email is triggered. The submit button in the success required fields check help text was also easily misinterpreted because the button doesn’t submit but rather directs you to the Export and Submit page.
-
While they knew how to use the required fields check, a couple of folks wanted to find unanswered required fields through the side nav rather than clicking ‘go to page’ in the Required Fields Check. This could be a byproduct of checking each section by section to make sure the content is completed appropriately in the paper APD. A potential path we can take is encourage the use of the required fields check during the final review and allowing folks to go through section by section during the internal review. We could even implement a recommendation we received during the pilot test to put a check next to every complete section, so it is easier to scan through to see that an APD is administratively complete.
-
Folks assumed that the Required Fields Check does more that is intended to do. Folks see the biggest benefit to their review having the system review their math in their budgets and somehow aid in reviewing the content fields. The overwhelming focus on the budget as it relates to the the Required Fields Check surprised us at first. We later realized that States are concerned about the budget because that is a section prone to errors. They might have separate systems to calculate the budgets, such as MBES, and they need to make sure that that the numbers they gather from their state’s finance team matches with CMS. Related to the assumption that the Required Fields Check does more, one participant thought they could attach their draft APD and the system would scan their APD and tell them which fields need to completed. It is likely this misunderstanding would be resolved by being more acquainted with eAPD, but it does touch up the idea that states want to be able to attach documents to their APD.
-
There might be a need to limit access to the final submission to certain roles or people with certain job codes. The idea of a gatekeeper came up a couple of times. They would be conducting the final review and either authorizing the author to submit or would be the person tasked with ‘hitting the submit button’. If we were to implement this, it might make more sense to have the Required Fields Check tied to the submit button because the gatekeeper would not be editing the APD, they would simply be reviewing and submitting. Further discussion needs to be had about who is the person who will ultimately use the required fields check.
After presenting MES Expansion/Cake Layer 2 designs as MESC, we mostly received feedback that we’ve already addressed, or have heard before, but with some notable exceptions.
Users continue to ask where they let us know if an APD is an annual APD or an update. While we won’t do any fancy branching or metrics off of the indication of an update at this time, we will go ahead and allow the selection with the MES expansion.
Users want to know - ”When would a specific FFY check box (example FFY 2021) drop off the options with an APD submission?” The answer is October 1, 2021, or October 1 of whatever fiscal year is ending. Questions that need to be answered - is this for newly created APDs only? What happens to APDs that exist in the system but aren’t yet locked down? This is important before Oct 1 this year. We need to add an issue in zenhub to note this in the UI. We also need to start thinking about what the FFY roll over means for locking down an APD as the document of record. What does it mean for historical data.
Users asked if they could upload their budget documents into the system, either to be processed or instead of using the eAPD budget entry and tables. This is not something that we would allow in the future, and should make that clear. Documents are supporting and supplemental.
In the spring of 2021, our front-end dev created a prototype for us to test with users where they could actually type in answers into fields, since the purpose of the research was to observe states’ experiences with and understanding of a different order, and inclusion of fields in the eAPD system as they would appear in an MMIS iAPD. We tested the Create a New APD page, APD Overview (formerly known as the Program Summary), Activities Dashboard, Alternative Considerations and Risks, Activity Schedule and Milestones, and Conditions for Enhanced Funding (formerly known as Standards and Conditions.
Create a New APD
- Indicate the type of update (as needed vs annual)
- Allow folks to edit content after creating a new APD.
APD Overview
- Review the help text to either minimize what we are asking, or potentially break the Medicaid Program and Priorities question up into more than one field.
- Keep FFP at the APD level to reflect where users would be thinking about it.
Activities Structure
- Create an empty state help diagram for HITECH as a starting point where we can expand for MES. This will help us get the language of APDs and Activities in the context we know them, without being overwhelming.
Activity Overview
- Move business areas up because folks know which business areas they are going to select when they decide their APD type and funding source.
- Include definitions for the business areas because folks are unfamiliar with the business areas and might be inclined to select ‘other,’ as a result.
Alternative Considerations and Risks
- Improve text readability so that it is clear that they do not need to enter Alternative Considerations every time they write an APD.
- Consider whether the Alternative Considerations and Risks needs to be at the activity level
Activity Schedule and Milestones
- Allow folks to upload milestones or change design to a spreadsheet format.
Conditions for Enhanced Funding
- When users enter the FFP, the attestation for enhanced funding should be populated based on their FFP selection.
- Consider whether Conditions for Enhanced funding needs to be at the activity level. Folks mentioned that they determine this for the project as a whole, but the determination would also depend on how folks would organize their APD (combining APDs vs 1 project per APD).
- States differentiate between projects and activities in different ways.While many states had difficulty defining an activity before seeing it in the app, they understood what sort of questions they would have to answer within an activity.
- Alternatives and Risks are written in the iAPD because they are usually using the pAPD to request funding for this work. Also, they can either be written by the project team, an agency, or a consultant.
- There are situations where a project that is operational will have implementation costs or go back to the implementation phase..
- Some states wondered if previous work from the pAPD could be connected/populated when you start an iAPD.
- States might need to mix match rates in their problem statement because they combine projects in their APDs.
- States recognized the MES modules as MITA based.
- States would want to convert from HITECH to MMIS, but based off the folks we interviewed, they might not want to convert from an MMIS APD to an E&E APD.
- All states combined multiple projects into a APD at some point. Some states had implementation and operations work in one APD, as a result.
- There are situations where states would submit unconventional funding requests, such as for COVID, but there is no need to reflect them in the UI.
- States found the help text/drawer to be a useful reference but it might not be helpful at this stage if the state is viewing it as a decision making tool. If it is a took to help them write their answer to these questions, and it’s clear what the purpose is, it might be more useful.
More details on findings can be found in Dovetail. A summary for the wiki is forthcoming.
Early exposure to the eAPD system before release will help State Officers troubleshoot with state authors and orient themselves to shifts into the construction of the eAPD vs the current APD template.
- All State Officers were given early access.
- We had 8-10 folks participate.
- 3-4 folks tried out the eAPD system and gave short feedback via email or google form.
- We interviewed 4 State Officers, chosen by leadership, after their early access period was over to understand their experience.
For this study, we asked general questions about the State Officer experience, then drilled down to learn more about the export view’s readability, the structure of Activities, what content needed to be gathered for Results of Previous Activities, understanding of Outcomes and Metrics and Cost Allocation.
Overall the State Officers were able to read the sample APD and feel confident that with their own states’ information, they could make a decision on the APD with the information provided. They feel that the eAPD system will help streamline APDs, reduce extraneous information, and they appreciate the budget tables that are provided to reduce the number of calculations they have to do during a review.
This study has us thinking about the purpose of the export view and remembering that looking at it just from the perspective of the State Officers will not meet all users’ needs.
Author Needs
- A tangible document that I can file
- A document that I can distribute with my team to get feedback
Reviewer Needs
- To make a decision
- A document that I can review and mark up
It also has the team considering risks around the current export view for State Officers to make a decision, for instance:
- Are we going to add more time to the review with the new format. Will this discourage use?
- Will it contribute to more RAIs for a longer period of time than a transition to the eAPD system might warrant? Will this discourage use?
These are issues we’ll continue to keep in mind and monitor as we gain users after Cake Layer 1 Release.
Looking to the future around updates, we heard that reviewers would prefer that authors not copy and paste previous eADP information, they want current and relevant information in the new submission. State Officers also want to easily view the budget, what the project is about and later what was done/agreed upon from previous APDs, so this reinforces the value of the project overview.
When looking at the sample APD provided for this study, State Officers noted that they conduct relatively non-linear reviews of APDs, similar to states doing non-linear entry.
- Some State Officers look at the MDBT first.
- Some look at issues or sections of the APD that have prompted questions back to their state on past APDs.
- Some State Officers and other analysts look at budget tables and schedule to see if cost or time have changed. If scope changed one State Officer noted they go to activities to see what prompted the update.
Once they’ve done this first pass to determine if they can continue the review, or if they need to have a state clarify or fix information, State Officers then move on to reading the rest of the APD.
Recommendations for the eAPD system that will support this behavior:
-
This may require a “more robust summary” where they could drill down to details if they needed to.
-
Potentially, the proposed budget aggregate could be ahead of the activity details in the export view.
-
There was a request for the state share to be shown in the executive summary alongside the FFP amounts. It was also requested that Results of Previous Activities be moved closer to the executive summary.
-
Export view formatting - folks were not necessarily sure when the executive summary ended and APD began, “I kept thinking it was a summary report”. Explore the options? Our holy grail would be to export the APDs in a Word file.
-
Adding the state share details into the executive summary
-
A participant commented that they “didn’t see MMIS noted on an Activity until I was down into it”. This is referring to the project type in parenthesis on the Activity List that is not appearing on the export view. We need to determine if this remains an issue after expansion since there will not be multiple types of activities in the MES APDs.
-
Misunderstanding of milestones - give clarity around when the Milestones and Other Funding sections begin in export view and that they are still tied to specific activities.
-
Consider a review of the export view to determine where we’re inconsistent with headings (or where it looks like we’re inconsistent while trying to be consistent)
-
This may require access to export view and the builder view. From this it was noted that some State Officers and stakeholders believe they will have access to view the builder/state view. This could call for more navigation options or clearer headings on the export view.Potential Recommendation: explore a lite federal reviewer experience where they have read only access to the state view. Counter: As the export view becomes more familiar, will the SO use it to a higher percentage with confidence, or will it remain involved to navigate.
-
Explore how much work it would be to create our own PDF for the export view (2919)
State Officers were split on having states enter Standards and Conditions in each activity vs at the APD level. We will move forward in the release with S&Cs in each Activity, but will keep an eye on this as a potential future problem.
Cost allocation is generic and open enough now for states to answer however they need to in a narrative form. “At the end of the day the question is, ”what is the cost you need to take out of the budget” This is the question being answered here.” We can make this better with a review of the help text.
We will rework Cost Allocation in the MES expansion. Cost Allocation can be the same across all types and will work within the regulations. (2921)
Feedback we received on the help content and process:
- ”If Medicaid is not the sole funding source, they should be directed to include all other sources.”
- Direct the state to include the actual formula in the cost allocation to include all funding sources. - I believe this is in the instructions.
- About where it's a hundred percent Medicaid as it can be sometimes, especially with high-tech as opposed to the state just saying there's no cost allocation.
Outcomes and Metrics is ok for release, except for some very minor content changes. Two SOs had very strong reactions to the word “discrete” and felt like it was asking the Outcomes and Metrics to be secret or hidden, not out in the open. Language closer to “outcomes statements need to be brief and concise” or “a specific and measurable improvement.” would be clearer. (2920)
Additionally, we will spend some design time on Outcomes and Metrics as CMS solidifies their guidance. Perhaps there will be some meeting with folks setting the regulations/guidance. (2894)
This could result in something like seeing outcomes/results/goals before activities and drilling down to activities that include the O&Ms but are not where they are entered.
We still don’t have enough information to determine if Results of Previous Activities should definitely be moved into each Activity. This is something we should pay attention to, but possibly address in the update experience, alongside more information about Outcomes and Metrics.
A contractor statement is not required, but some State Officers think it is. Listen for more feedback on this topic and how to set up expectations. It was noted that the sole source information needs more detail under private contractor to describe how the contractor was chosen for this activity. (2917)
Allow some states to use and test Cake Layer 1 before release in order to test changes made to the UI after previous tests of Cake Layer 1 over the life of the project. This would be the last user testing check for any show stopper issues or bugs that need to be taken care of for release.
For beta testing, we recruited four states, large and small, single author and teams (including contractors), to spend two weeks in the staging environment in order to try and export an approval APD out of the system and provide feedback on their experience. They were provided with a notes document to capture any thoughts or problems they encountered along the way. After the two week beta period, we conducted 60 minute debriefs with each state to talk more about their experience.
“This tool really takes a lot of the guesswork out of this…you are helping States or understand exactly what needs to be there…getting rid of some of the fluff we have, and really just getting to the point of what the request is.”
“I feel like, just somebody…it's their first day on the job. I really feel like they could step into this tool and it really explains to them what they need to include.”
“The team was really happen [with] how easy it was just to pop the information in.”
“One of the strengths of making it activity based…is that in theory, we should be able to say less, have occasional light summaries, and then everything rolls up gracefully. “
“I think there's no love lost for us trying to coordinate and tabulate a bunch of different tables as part of our revision of the EAPD document itself. So that is obviously excellent. There should be improvements there, but the tool can only improve it so much of the process. Of course, some of it is the process in our end of, of learning what we need to know and getting what we need to get, to inform the document electronic or otherwise. And so some of some of that will still remain unimproved.”
In conducting this beta testing, we were searching for any show stopper issues that would prevent us from releasing. We did find some bugs that needed to be addressed, but no showstopper usability issues.
-
Two states (out of four) were confident that they could submit an approvable APD through the new system. One additional state believed they could, but noted concerns about transitioning the process, not using the system. Recommendation: Keep leadership aware of the need to transition a massive process across many departments before requiring APDs be created in the eAPD system. There will be some transition in thinking for states as they move to an activities based idea from an APD wide idea. They can get used to this, but will need more time than we might be think to allow to switch over to a new method of reporting.
-
Even after changes were made from previous testing, participants were still having issues with knowing what key personnel should include and whether to enter contract lengths vs funding request lengths. Additional changes were made to those sections to be more clear about what CMS needs.
-
States were concerned about the change to needing to report the standards and conditions per Activity vs per APD.
-
Minor usability issues were found and when the balance of small changes vs greater impact to users was weighed and determined to be ok, an issue was added for each usability issue. (See list of issues created for details.)
Bugs
- 2421 - Estimated Quarterly Expenditure table is overwriting across Activities
- 2708 - User can not access "Cost Allocation and Other Funding Page' for Activities after 1
- 2715 - In the Other Funding area, an extra FFY is showing in the export view
- 2725: Outcomes and metrics interactions were inconsistent between activities
Auth issues
- 2702 - Authorization is expiring while users are in the system and they are not being alerted
Usability Issues created
- 2531 - Standards and conditions fields should match in builder and export views Fix: see issue details
- 2722: Program summary and executive summary are confusing users Fix: changed the language on navigation to not infer that the program summary was indeed a summary, but an overview
- 2723: Results of Previous Activities Actual Expenditure Table has confusing headings Fix: Headings were simplified, see issue for screenshots and details
- 2724: The outcomes and metrics fields were too small for the type of content being entered Fix: The outcomes and metrics fields should be longer than one line, further adjustments TBD after more use
- 2726: Private contractor cost page field labels are confusing to users Fix: made minor changes to the field labels to be more clear on what content is needed as this could be an area where the wrong answer could render an APD not approvable
- 2730: help text around FTE number or percent is confusing or unclear, people seemed unsure they could enter a decimal or percentage Fix: showed a decimal in the example text instead of a round number
- 2733: Determine helpful changes to Key Personnel help text and fields Fix: simplified the help text and examples around Key Personnel and changed field labels to make it clearer what information needed to be added in this area. Additional changes will be made when we move Key Personnel to the project level.
- 2728 - Be clear in the export view about what was chosen in Assurances and Compliance Fix: not a priority for release, but will monitor for additional feedback to determine priority
Further research:
- The export view doesn’t always show when information has not been entered - needs a usability/readability review
- With changing reporting standard and conditions to each activity, we’re asking State Officers to review that section in Early Access and will think about how to communicate that new need to states.
Feature suggestions:
- Provide a RTE that allows folks to create tables in the eAPD system so they don’t have to use an outside table creator
- (Priority) Provide the ability to see who made the last edit to what section/field Provide the ability to see where someone left off in editing, or what remains to be edited
- (Priority) Links back to sections where information is collected from summaries and budgets (Users are still looking for an “edit” option on tables to take them back to where they enter corresponding info.)
- Collect details about date/timeframe for expenditures in Previous Activities
- (Priority) Contractors should not be allowed to submit APDs - permissions
Design Thoughts from Team
- After expansion we're going to need to have a way to move APDs that were entered before expansion into new projects. It would be a good test of attaching documents to various projects.
Who we talked to: Delaware, Idaho, Reviewers from Idaho What they did: Successfully submitted their FFY 2020 HITECH APD using an export from the eAPD system. Reviewed a submitted APD. What we did: reviewed their APD submissions for issues both technical and regulatory, met with the participants for a kickoff and a series of debriefs to discuss their experiences.
Through allowing two users (who requested access unsolicited!) to return to complete their updates before release, we uncovered a number of issues that might not have come to light without real users doing real tasks. These kind of findings are usually not uncovered until beta testing, or until after release, so we were able to make many improvements before the release. This will make the ramp up into the new system smoother and easier for all users!
“There were places in the old template and filling it in myself where I kind of thought or expected that I needed to put a lot of content in. Where on this one it’s scaled back to make it less general. ‘We need to know about this. We need to know about this.’ I idn’t have to guess on the information they (CMS) were looking for.”
“It takes a lot less time because I’m not formatting a bunch of Word documents. Just not needing to do tables saves at least a day’s worth of work. [I saved] hours of time because I didn’t have to make my own calculations and go back and check them. The tables are fantastic.“
-
People want to use our system and are going to have an “update experience” no matter if we have built a specific experience or not. Users shared how their “update experience” with us in debriefs and follow up sessions after they submitted their APD.
-
- I used italics and/or bold and blanks (______) to show myself what changes needed to be made in the narrative boxes. I was expecting redlining opportunities in narrative boxes, or more formatting for text to be able to highlight changes. All I need to update in the narrative is “what is in italics”, I put it in italics so I remember I need to update that.
-
- I highlighted and made myself a little note, used bold, and then would come back and look for those to make edits to update.
-
Formatting consistency in TinyMCE is difficult when someone is copy/pasting in to the app from multiple sources. At the time of this test, we weren’t allowing any formatting options in the editor and the copy/paste function is messy as it’s not possible to remove or set formatting on text that is pasted in. We are looking into paid TinyMCE features to allow for better formatting.
-
One state would still check their Excel files and match them to our system. One state was ready to trust the system and let the tables do the work.
-
The Activity Schedule Summary was well received and was copy/pasted and used as a milestone chart outside of the system. Notices she needs to change the activity date range when they don’t have dates listed, for instance. Could use this “chart” to close out HITECH.
-
Some unexpected downtime during the testing window showed a need for a contingency and communication plan is needed for unexpected downtime.
- 2385 - Bug: FFP and budget page data from Activity 1 is being shown in every Activity in the export view
- 2403 - Give users the ability to choose FFYs before entering their Key Personnel - if FFY isn’t entered before KP, it’s impossible to capture the right data about KP unless you know to go back a section after selecting the FFY
- 2416 - Deleting a year deletes all data. Warn users about lost data before they uncheck a FFY.
- 2394 - add TinyMCE editor to two fields it was missing from (participants noted they needed to make updates in this section and weren’t able to do any formatting)
- 2528 - Participants note having trouble finding where to “add an activity”. We’ve noticed too in navigating the app that we seem to easily lose “Program Activities”. Update Program Activities headers, labels, and help text.
- 2521, 2404 - TinyMCE formatting to accommodate updates formatting
- 2431 - Add help text to Contract Term field - Participants expressed confusion over the amount to add - is it the total $ of the project or total $ needed for the fiscals years noted in the APD?
- 2439 - Appears there is no option to add a next year FFY to show in the Actual Expenditures table - made the decision to show the Current FFY and 2 years before it in Results of Previous Activities to make it clearer what expenditures are needed
- 2529 - Conduct Dev Ops Contingency Plan Execution Table Top Exercise
-
The participant sends out a template to other teams so they can provide the information for the APD to the author for formatting.
-
Participant was concerned about fields that were not filled out coming across as “missing information” to the federal reviewer.
-
Assurances and Compliance - “I might skip that and think it’s another table - was expecting it to be in Activities.” It was interpreted that the participant was not expecting more things to fill out at this point in the app.
-
Key Personnel budget was still hard to understand, but there are already plans to fix that in Cake Layer 2.
-
There were some minor concerns with the export as PDF and the ability to comment on a PDF for reviewers.
-
Users are still looking for an “edit” option on tables to take them back to where they enter corresponding info.
-
We tested user understanding of existing, updated, and new tables in the application including the FFP and budget table, the proposed budget summary page and we showed folks the new state costs summary table design as a mockup
-
We gained a little info about how folks interpret what we’re asking about Cost Allocation and Other Funding. Initially we wanted to observe data entry in the application, but that direction resulted in stifled conversations as people tried to comprehend a complicated system while typing or copying/pasting from complicated spreadsheets.
-
After gathering a couple of bugs and questions about data entry and interactions in the system we pivoted the protocol to favor conversation over data entry. This turned out to be the right move to gather really great information about how people use tables and what they are looking for when looking at the tables.
-
Folks are often looking at data in the tables: 1) to ensure they added the data in correctly 2) to get internal state approvals 3) to see the state share. And in further convos with Jerome that folks can be looking at tables to connect the information in their approval letter to the data they entered to see what was approved and if it differs from their proposal.
-
People were generally able to read the tables, but we did see them looking for data sooner than they’d actually gotten to the page that data was on
-
There was a clear expectation of seeing the combined activity costs first on the state cost summary, then seeing the breakdown by activity
-
They also seemed to look for the split information often, so we need to answer the Q - “When people are looking for the split, what and where are they looking for?”
-
For data entry, we did see a couple of bugs and a couple of clunky interactions like copying and pasting into the wrong field, or using arrows on the date fields instead of the simpler/faster action of typing in dates.
-
The team saw that we needed to expedite our conversations about cost allocation and other funding to help the team understand what we’re working with.
-
Were able to make changes in 2162 (content review) based on observing users in this round (bonus!)
-
As we’re reviewing issues for release and as new things are built, ensure that we’re noting any concerns with interactions in the app as we’re reviewing for release.
- Take a look at the cost allocation and FFP and budget tables pages for potential content and field order changes to help understanding #2217
- Do a review of the navigation labels #2231
- Rename the State Costs Summary table #2233
- State Costs Summary isn't a medicaid term - lean on regulation and user understanding for table names #2234
-
- Change Medicaid Share and Total Medicaid Costs on the state costs summary table to "Total computable medicaid cost" for all totals
- Move the combined activity costs above the break down by activity in the state costs summary table and take a look at more established sectioning #2235
- Add an edit option to the state costs summary table (hold for future discussions)
-
- Ensure consistency with the rest of the application
-
- Returning to Activity per Activity would be easy, but what would we do on the combined activity costs?
-
- Might be per section taking you back to the top of the page where the information is entered
-
- Should we even do this?
- FTE Costs in KP and state staff sections (hold for future discussions, see Mural link above)
- FTE count arrows allow the number to go below 0 #2229
- The placeholder 0 in number input fields doesn't go away when you paste over it until you tab out #2230
- Hourly resource under private contractor wiping out numbers #2236
-
- Same area - yes/no can be selected way out to the right of the radio buttons #2232
- Will trust increase with usage? Would be neat to track when/if folks stop feeling like they need to crosswalk or double check the math in certain areas.
- What is the use case of comparing the approval letter to the APD submission?
- Sidebar updates
- Paginated view, previous/continue buttons
- Autosave implementation
- Objectives and key results
- Leave space for conversation with new team for general topics
Side bar updates and Paginated view, previous/continue buttons
- In general folks were able to use the sidebar navigation and the previous/continue buttons in ways that make sense to them.
- Folks noted that the previous and continue buttons would be the navigation they use at the start, to make sure they move through all the sections, and at the end, to check their work, again moving through all sections. They noted they would use the sidebar navigation when they needed to go to a certain section, or jump around through the app.
- Some minor improvements could be made to the hierarchy of the side navigation for faster understanding, but nothing show stopping was present in this round of testing.
- People did comment that when they clicked on a section in the side bar, they expected to see the side bar headers matching to the heads in the workspace.
Recommendations
- Add an arrow to the appropriate locations on the side bar to indicate a sub nav | #2188
- Align the headers in the workspace to clicking on the side bar nav | #2204
- We could remove the words under the previous/continue buttons, but might reserve that as a formal recommendation until we look at the activities list navigation
- If we have any FED time, consider design/font hierarchy to make sure folks don’t think that all the sections are a sub of the top section, or that sections and sub nav sections are very clear in the side nav.
Autosave implementation
- Folks noticed the autosave once we asked about it, but didn't generally notice it before that.
- When asked how the app saves, they commented that it might save when you use the previous/continue buttons, click on done, or move to another section, as well as assuming it would save at some interval.
- Once they noticed the autosave they saw it had been a while since the app saved (since they typed anything in). This is probably just a remnant from testing protocol that focused on navigation and not on data entry.
Recommendations
- Slow the autosave and make the animation slightly more noticeable for folks | #2193
Activities Section Navigation In the activities section, we dug into, a little later in testing, the idea of a mental model of the Activities being a deeper section. Currently there are 3 ways to enter an Activity 1) Edit 2) Side Nav 3) Continue Button.
- One user mentioned that the Continue Button would go into the Activities.
- Two users mentioned that the continue button from the Activities List should go to the next main nav section.
- One user mentioned that at the end of the last existing Activity, the continue button could go back to the Activities List for review.
Recommendations
- We are exploring multiple design concepts for the Activity List | 2116
General Findings and Recommendations
- Do we need help text that tells the user that the calculation occurs (90/10 split)? | #2217
- Revisit the label as yearly costs with benefits | #2162
OKRs
- OKRs may be moving to an Outcomes and Measures framework.
- We’ve been advised to wait for word from leadership before moving on this information and can revisit findings from this section at this time.
Further research needed
- People have their own systems for tracking what they have completed and what needs attention. How could we allow for this in the update experience?
- Paginated view
- Tabbed view
- Autosave messaging placement
- Autosave error design
Paginated view
- Overall, most users prefer the linear, paginated view
“The ideal is the left navigation model where it automatically toggles for you and you see your progress on the left bar.”
“I like that you are in a discrete section and when you are finished with it, you can just go to the next section. It's more of a discrete chunk that you can go into and work on and then you can know when you have completed all for the parts. I used to get really lost with all of the scrolling. I like this better.”
- Most users could tell where they were when the nav was in view (it wasn’t always visible when they had it expanded and scrolled)
- Bug with highlighting the current section caused some confusion
“I would think the menu should stay where it is and they each scroll independently – like a split screen.”
- Users looked for and expected buttons at the bottom of the section to progress forward and back
“It would be helpful to be guided through by clicking next, next, next.”
“Having the next/back buttons AND left nav would be helpful to have.”
Recommendations
- Implement the paginated concept for each section to be on its own page
- Add secondary navigation for progressing back and next
- Refine the left navigation
- Fix bug for highlighting current position
- Should scroll independently - fixed position sidebar
- Only one section open at a time
- Do we need to show subsections in the nav?
- Current position always visible
- Scroll to the top when switching pages
Tabbed view
- Some users prefer the long scroll
“Would rather be able to scroll through everything at once rather than have to click over to the left to go to another one. I guess as long as you can get to them either way, it doesn’t really matter.“
“You can see the big picture of what the expectation is -- I could get a sneak preview (scrolls through the app). I like to see it all beforehand. It shows more details”
- Some don’t prefer the long scroll
“This just scrolls through everything. Feels like it is going to go on forever.”
“The length of text boxes seem gigantic. Might be better to be a smaller area that you scroll within.”
“It’s easy to get lost”
“When I have to keep scrolling down through things it’s confusing to me.”
- Most people did not prefer the tabbed version over the paginated view
“I feel like some things could potentially get missed. It is easier to miss stuff.” — speaking about when it’s not in a “linear setup” “It’s easier to glance to the left.”
- One user thought that if the tabs stuck to the top of the page, they prefer it
“I like having the detail sections at the top. Visually, your eye goes there. I think I like the tabs across the top”
Recommendations
- No changes — don’t implement the tabbed view
- Keep the finding in our back pocket that having subsection labels stuck to the top of the Activity section could work.
Autosave errors (generally)
- The phrasing of the error message and saved notification seemed confusing and contradictory
“It says last saved at 12:07, then it says some data was not saved. Those two statements seem incongruent to me.”
“Why would it say “some data is not saved” if saving is not my responsibility?”
“Out of the range of what? I think you should be more specific than that. Say what you mean.”
“I think “save failed” feels like a total meltdown failure -- something really bad. Something like a data range was off -- is there less sensitive verbiage that could be used?”
- Users expected to see inline errors
“In an ideal situation it would highlight the error which would help me to take the next step to fix it.” *Mixed expectations for continuing to save with errors “Saving everything but the error makes the most sense to me.”
Recommendations
- Allow saving all the time even for invalid information
- No change to the current inline error messages
- Explore options for informing users about errors prior to submitting
- Refine the error message text so that it’s clearer what caused a problem and how to resolve it
Autosave at the top
- Most users prefer to see the autosave information at the top
“The first one (top) is most natural.”
“I think I prefer it up at the top. That feels a little more prominent. You would be more apt to see it at the top.”
- The blue header in a fixed position at the top of the page was appealing
“The nav sticking to the top anchors the document and gives it more structure”
“I like having everything up there stuck to that. It looks neater.”
Recommendations
- Implement the fixed position header and autosave messaging in it
- Explore the phrasing of the save messaging (e.g. “last saved 30 minutes ago” vs timestamp)
Skinny error messages
- The slimmer version of the error message was preferred over the alternatives
“I do like that full banner. It didn’t cover up any material on the page”
“The first one was a little less threatening to me.”
Recommendations
- Refine the skinny error design (if needed)
- Implement the skinny error message (consult with dev)
Square error message
- Users were worried that the error was covering important information
“This just covered up my stuff and I can’t see it and what if that is where the error is?”
“The only thing I don’t like is that it is covering up some of your data.”
- “This is big and clunky looking. It's not very visually pleasing.”
Recommendations
- No change - this error type is useful for when you can’t move forward at all, not best when you can continue moving forward
Autosave on left
- Most people did not prefer the autosave information in the left nav
“Not as obvious and it’s not as helpful to me.”
“Gets a little bit buried at the bottom”
- One person did prefer it “the lower area felt more natural”
Recommendations
- No change
Other findings
- Users would find version control useful, especially for knowing what changes certain people made
“If others were using the app, I would want to know when it was saved, but also who saved it”
- Multi-user support is a frequently requested feature that would allow states to use the app as their single source of truth
“Multi-user support would be a tremendous value to replace the existing process with this new process. And this would be the only process.”
- Want to be able to link people across activities
“If I’m allocated 50% here.. There is only 50% for me in another activity. I don’t want to be more than 100% allocated. If you go to the person level, it would be nice to have consistent rates and not over-allocated.”
“If I was over allocated to 110%, then I would need to go through all the affected APDs. And this is being done using Word and Excel, it’s a very labor intensive process.”
- Most people expected a new activity being added to append to the bottom of the list (as it works now), one person expected it to automatically pop up
- The update experience continues to be top of mind for users, and they expect it to be automatically populated from the previously submitted APD
- Modal interaction: how does this compare to the current version?
- IA of the section: does it flow well? Does the order of information make sense?
- FFP and cost allocation section: is it easy to input the amounts?
- Editing an activity
- Reordering activities
- Editing the activity name
- Adding new activities
- Deleting an activity
Modal experience
- The modal isn’t necessarily what users expect to happen, but they don’t seem to be put off by it
- One person thought edit link would open up in the same interaction as “add new activity”
- Another person thought it’d just go to the “next screen” instead of popping up a modal
- Still not 100% clear they are in the context of a single activity with the modal
“There was too much scrolling in the old version. It was hard to keep track of where you are scrolling. The new window – I like that a lot.”
Recommendations
- Implement the modal
- Change “Activity list” to just “Activities”
- Make the modal bigger or touching the edges
- Make sure people can still see where they are in the context of the rest of the app (it’s not a separate screen)
- For future sessions, preference test the two options (modal vs scrolling)
Overview
- People weren’t sure if day is a required field because they often don’t know the exact date
- Expected longer activity description first and then the shorter summary next with instructions saying it’s only 2-3 sentences
Recommendations
- Change “Activity summary” to “Activity description”
- Change “High level summary” to “Short summary”
- Swap back the order of short summary and description
- Add some help text to the short summary explaining that it should only be 2-3 sentences
Standards and Conditions
- The standards and conditions question seems in-line with how they answer now
- The external link is helpful for people who need more information
- Two people expected this to just be a checkbox where you only need to explain if you’re not complying
Objectives and Key Results
- The list of objectives that were already entered wasn’t obvious
- Expected “add another” interaction to be same as adding new activity
- Users think it might be hard to come up with percentages
- States want to make the measure as generic and non-meaningful as they can. CMS should want it to be as specific as possible.
- One person didn’t think the name of the milestone is descriptive enough without adding additional detail
Recommendations
- Add “Objective” label for the list of already-entered objectives
- Add help text around key results to make it easier to enter percentages
In-house cost, private contractor, cost allocation
- People are struggling with in-house cost versus private contractor cost
- One person was unsure about the difference between personnel and non-personnel and thinks one might be a contractor
Recommendations
- Change ‘In-house’ to “State personnel” - think more about this. Can add help text saying in-house = state personnel and non-personnel associated with state
- Spell out full time employees instead of FTE
FFP and Cost Allocation
- The order of first table feels backwards. People expected to see the total, followed by other funding and then the Medicaid share.
- Confused by the table labels
- People expected the second table to lead with the total, then split, then the federal and state shares
- One person didn’t know what
other fundingmeant
“In addition to what Medicaid is contributing, there could be other funds? I am a little confused about this. I didn’t think that was allowed. We have gotten denied by CMS when we have tried to put in anything other than the 10%. Maybe a link or explanation about what this is according to regulations or give an example.”
Recommendations
- Update the first table to be total, other funding, then Medicaid share to reflect the math that’s happening
- Label first table as “Calculated Medicaid share”
- Label second table as “Calculated Medicaid federal and state share”
- Update the second table to be total, split, then the fed/state shares
- Add more help text about what other funding means and what counts
- Remove placeholder percentages in the quarterly expenditure table
- Consider adding a short narrative to explain cost calculation in paragraph format
Delete and save
- Wanted to be able to save at any time and expected save button to be visible
- Feel like delete makes more sense from the activity list page
- Since deleting the whole activity is rare, expected it to require two clicks
Recommendations
- Continue to use hoving save button for the activity modal until auto-save is implemented
- Put “Save and close” in the top right
- Add delete option from activity list page
Editing the name
- People are looking in different places to edit the name. Some are expecting to do that from the main activity list screen or from the top of the modal
Recommendations
- Add edit activities button next to add new activity
- Editing the name should happen in one place
Reordering
- Some expect drag and drop to work for reordering, others weren’t sure how to do the reordering
- Expected to be able to edit the list separate from just adding the activities.
Recommendation
- Enable drag and drop for reordering
- Accessibility concerns with drag and drop?
See the full report and findings
- How do states update their APDs?
- What do they need in order to begin the update?
- How do they know when they are ready to start updating their APD?
- What is their process?
- Who is involved?
- What materials do they use?
- What are the pain points with regards to updating?
Each state has their own APD update process with built-in workarounds.
Their exact process is unique, but there are a few commonalities across all states:
Delegation
- There is generally one person who is in charge of the process who delegates the work of collecting and inputting information into the document.
- Sometimes the delegator is the person who merges the edits into the final version.
Collaboration
- The APD is passed back and forth between many people via email.
- States often take a “divide and conquer” approach where certain people are responsible for specific sections.
- When someone is done with their section, they send it to the next person.
Version control
- States have created their own type of version control by saving a copy of the document with their initials added to the end of the Word doc file name.
Custom spreadsheets
- Spreadsheets are heavily relied on for tracking the numbers.
- Most states use their spreadsheets as the source of truth.
- Every state has a different spreadsheet format that they’ve refined over time.
States try to make their APD updates easy for CMS to follow.
They think they know what CMS is looking for and do what they can to highlight the differences with the updated APD since their last submission.
What states think CMS wants
- At a high level, they think CMS wants to know the status of how states are progressing against their deliverables and what they plan to accomplish next.
- At an individual level, states have varying assumptions about what CMS is looking for when they review the APD.
Track changes
- States use track changes internally and when submitting their final APD to CMS.
- Track changes is messy, but it’s the only workaround they’ve found that works without being able to use other tools like Google Docs.
The nature of APD updating is complex — some sections change every time, some remain consistent.
The manual process of updating an APD leaves room for error because of the reliance on memory for what’s been done vs. what to do and is generally inefficient.
Checking what to update
- The first time states create the APD is the most important because it’s what sets the structure.
- States rely on their eyes to catch what needs to be updated as they skim over the previously approved APD looking for dates, dollar amounts, vendor names, etc.
- States like to keep track of what’s been edited/reviewed and what’s still to-do. One person did this with a checklist (in Excel) to mark off which sections are complete and another state utilizes comments in Word.
Budget tables
- Budget tables are the biggest pain point — the same data needs to be filled in manually and spliced several different ways to complete an APD.
- States have a workaround for budget tables where they maintain the data in their own spreadsheet and copy it over to the APD.
- These workarounds are better than nothing but have shortcomings when it comes to human error of copying/pasting and the inconsistencies of rounding.
State user needs for updating
In scope
- Some sort of version control
- Keeping track of what’s been edited/reviewed and any to-dos
- Some way to validate tables and share out with stakeholders
- Guidance and guardrails for what to update
Maybe/worth exploring
- Multi-user editing. Roles based permissions
- Ability to restrict what people can do
- A way to interface with Excel
Out of scope (for now)
- Internal approval mechanism
- State-side workflow so they can better coordinate through their process
- Process of saving as PDF and submitting
- New left navigation
- New dashboard (prototype)
- Removal of the accordions, addition of section numbers (prototype)
Overall feedback
- Would like to be able to add the person writing the APD to the State Profile Contact info
- Would like to be able to add an additional peer to the Medicaid Director State Profile Contact info
- It seems too early to be entering costs for key personnel as a part of the Profile Contact Section. Users were expecting to just list points of contact, and provide costs for key personal in a subsequent section (probably near or just prior to the activities list).
- Help text box expected to be a hover, kind of cluttering the view. If user knows what it is, they don’t need to keep seeing it.
Dashboard prototype
- Can we get a status update on the dashboard? Where it is in the CMS process? Should be somewhere near “Last Edited”
- What is “manage account”? I would want it to be to manage my own account, but if I am the administrator for the Division’s APD, I might want to be able to set security settings for multiple users.
- When we get an approved APD, would anything show up that says it’s approved? (seems okay with the plans for the pilot being minimal in this regard)
Removal of accordion
- It feels a little redundant having both the accordion body and the left nav. It does the same thing. User is more accustomed to the left menu to navigate.
- Sticky nav means you don’t need the accordions here, this is actually awesome.
- Misses the accordion, “you can put more stuff on the screen”
Saving as PDF and submitting
- Assumed a left hand nav submit button.
- Would prefer to just submit the eAPD, and not create a PDF and email to CMS.
“It’s a little long, and keeps telling me to print, so I’m going to switch to the eAPD (non-PDF) version.”
- Some sort of print preview function is important - comparing how the eAPD looks on screen versus how it exports to PDF.
- State reported no access to Adobe Pro (will be unable to get an editable version) of the eAPD
- No easy way to convert from PDF to DOCX
- A lot of white space on the export.
- Could not find “Export” button, didn’t realize Save as PDF is “export”
- What do CMS analysts need to know/see in order to complete the review process?
- How can we make this process easier and more efficient for them?
- What are the pain points in the review process, generally?
- How can the eAPD alleviate these?
- How will the eAPD app fit into the current workflow of the reviewers?
- What are people most looking excited about with the eAPD?
- Are there any concerns with using the app?
Workflow and review process
- No standard workflow
- Received via email (eAPD/centralized mailbox)
- 60 day review period
- Multiple levels of review
- Outreach for more information via email – calls when necessary
Checklist or review in order
- Not one that exists for primary reviewers.
- Process of reviewing varies by analyst and by type of APD (Regular, MMIS, HIE)
Which docs are received
- Everything is via email; “the rules are to send all documents to the HITECH mailbox (officially)”
- Contract, amendments, SOW, RFP – These are usually embedded into the PDF.
- Use of SharePoint for storing documents for reference throughout the review process (secondary review, etc.)
Roles and handoffs
- Concurrent reviews by SMEs
- Collectively issue comments to the state via email and they respond with any comments
Slowest/longest step
- Waiting for comments back from the state
- Looking at their previous approved activities and see what they were going to do and then compare it to what they did. Seeing how they connect is a challenge.
Variation
- Some states have larger Medicaid populations, which translates to more complex requests.
- Format/Template – “No one state provides information in the same way”
What to look for when reviewing
- “Allowability” of what they are requesting.
- Reasonableness of the project
- Cost allocated
- Funding according to timelines
- Alternate analysis/Cost Benefit
Rely on certain tables/sections
- Section III, V and VII
- Financial Tables
Full review of all content
- It would be great to see section V (contractor resources).
- Executive Summary is good place to start but look at all.
- One comment indicates instruction to state not to submit sections not updated, but to indicate “See previous”.
Communication with state
- All Email when possible
- Calls only when necessary
Frequent questions that come up
- Reporting on previous expenditures
- Correcting inconsistencies
- Cost allocation
- Scope of Work info
- Staffing
- Contracts
- Project Timeline
Top issues that states need to address most often
- Cost Allocation
- Timeline (consistency)
- SOW
Pain points
- 60 (45) day timeline for review – getting everyone to provide feedback
- Reiteration – having to reference old APDs for a current APD
If you had a magic wand, what would you change?
- Timeline (allow full 60 days for approval)
- System for communication and comments with states and CMS
- Section 2 –easier to determine progress; see a snap shot of the project history
- Comparing costs –hub for cost comparison; maybe not in the eAPD, but somewhere
Most looking forward to
- Not allowing states to submit unless they have completed everything
- Structure/standardization for easy navigation
Concerns about the app?
- Don’t want to lose the 1-on-1 communication –don’t want to lose the relationship
- Allow flexibility for states to do something different and innovative
- Test thoroughly
Physical setup
- It varies- Sometimes two monitors. The larger the more complex, the more screens I need because it requires more comparison (i.e. old APDs vs new APDs). (Maybe 50% of the time)
- Mostly just one screen and toggle/paste
Not currently required but would like to see
- The SMHP should cross references the activity
- Analytics information would be helpful (identify early if state will be over budget or over timeline)
What should be kept consistent with current version
- Keep the template sections; making sure they are seeing the same content / what we are asking for doesn’t change
- Don’t restrict state flexibility
- Tables: actual expenses, cost allocation (quarterly), summary schedule table, summary budget table, federal share by quarter, incentive payments
- What information are state users looking for/interpreting from the tables?
- What do state users need from the tables? How are they interacting with each of the tables?
- Executive summary: how are people using this section? What details are they looking for?
- Certify and submit: how do people expect to certify and submit / how does it match their expectations?
Program Activities
- A couple people were inclined to add a completed date in addition to start and end for the activities
“We have a planned end date, but can we add a completed date? It might make it easier to review some of our projects.”
- There was confusion with what the in-house cost categories salary included
“So you are looking for salary amount and not benefits for personnel? Or is that included into salary?”
Cost Allocation
*The other funding amount section prompted additional questions
“Add more to this sentence — “other funding amount” + due to cost allocation or after cost allocation. Make this clear that this is the cost allocation part.”
Summary Budget table
- Familiar to the template and what folks are comfortable with
“It’s not taking away from the template we’ve come to love. To me, it modernizes it and streamlines the approach.” *Having a view of the breakdown per activity “I like this, what I’m seeing here, if I have 4-5 people, I don’t need it separately identified, I just need to know how much money is identified for the project.”
- The + expanding interaction needs some work
Incentive Payments by FFY Quarter table
*Some confusion around the EH and EP counts
“Is the EH or EP count totals or unique values? I found it difficult to determine unique counts for EP in a given fiscal year. Our EPs come in on different years. I’m assuming this here is not a unique count, this is the EH payments made.”
Executive Summary
- The green stands out and doesn’t match the rest of the palette
“Don’t totally know why this is green, this seems like a new color.”
- The rounding caused some confusion
“I don’t know why we’re rounding all of a sudden. It looks pretty, but wouldn’t be accepted.”
- Confusion around what these numbers consist of
“Is this number the state and fed together or just the federal side?”
Print to PDF
- People used print and save when talking about converting to PDF *The print preview is very long
“Oh it’s just a short 125 pages. It’s so big.”
- Need to define the unnecessary text
- Sections that have scrollable content are cut off
“That’s hard (pointing to tables)” “If there are scrolling tables, you can’t scroll through, I want to see all of those.”
- People would prefer to have their APD in Word format
“If you could get this in Word instead of PDF, that would be best case scenario.”
State dashboard
- The dashboard clears up communication bottlenecks
“This system will help address the lack of communication with the states and CMS.” *Green checkmarks are misleading when it’s not a favorable outcome “Disapproved has a big green checkmark which is not very happy. Congratulations, you were disapproved! Maybe it could be a red x.”
- When CMS sends the approval letter, the approval timeframes can be complicated for state users
“They are approved for different timeframes. Would like to know the dates that each of them were approved for”
State profile contact info
- There was confusion with the reasoning behind asking for key personnel
“The question is just do you care about who the people are for contact reasons or do you care because you’re paying their salary? The word choice will show which is more important.”
- The percent commitment to the project wasn’t clear to everyone
“I would move the percent commitment to project down next to the 2019 cost boxes”
- State dashboard updates
- Key personnel moving to the contact info section
- Program activities: Performance measures and benchmarks, In-house cost categories, Private contractor costs
- Commenting (sketches)
State dashboard
What people liked about it
- Linear timeline
- Knowing the status without having to email people
- “Seems pretty easy”
Opportunities
- There was some confusion about “pending comments” and “clearance”
- It wasn’t obvious for state users who needs to take action at each step in the process
- Consideration: put the steps of the process along the top and more specifics below for each APD
- For the events section, could add reminders about meetings, annual reports, due dates, all-state calls, regional meetings
Key personnel
Opportunities
- The instructions/help text for key personnel wasn’t clear for everyone — someone had questions about costs and not knowing for sure if they should leave it blank or not
- One person was confused about what this section was asking for and thought it would make more sense to call it “contact” instead of “personnel”
Program overview
Opportunities
- Still seems to be some confusion about what to put in overview — need to make it clear that it’s high-level
- Size of the text boxes can indicate the expected level of detail
Program activities
Opportunities
- Someone talked about grouping multiple activities into one because of the way their contractor contract is written. Is it okay to do or would you prefer they not do it that way?
- How do we approach having contracts that span multiple activities?
Performance measures and benchmarks
Opportunities
- A couple people referred to these as “goals and objectives” rather than “performance measures and benchmarks” — should we switch that back to what is was?
In-house cost categories
What people liked about it
- Separation of in-house and contract — “That’s pretty obvious”
Opportunities
- One person mentioned changing “salary” to “loaded salary” for clarity
- Wasn’t clear the app will do the math. How can we make this clear?
Commenting
What people liked about it
- The fact that this commenting is a thing and is an improvement than emailing back and forth
- Commenting on specific sections
Opportunities
- One person pointed out a need to be able to resolve comments — would expect CMS to be the one to resolve
- Confusion around read/unread comment indicators
General feedback
Opportunities
- Marking sections as complete (for state users)
- Someone mentioned uploading an attachment that was just text. Do we want to add guidelines around what types of things should be uploaded so people understand it’s for more for pictures and diagrams?
- State dashboard
- Navigation
- Help text
- Submission experience
- Program activities: Personnel, Contract Resources, Non-Personnel Costs, Standards and Conditions (understand how people progress through this section)
- Assurances and Compliances (understand how people progress through this section)
State Dashboard
What people liked about it
- One-stop shop for everything
- Really liked “recent activity with dashboard”
Opportunities
- Clarify what ‘due date’ means
- A clearer queue of items submitted
- More detail in status bar
- What are the status labels?
- Add “in clearance” to received, under reviewed, approved, completed.
- Track internal submissions status/review vs. submit to CMS
- Review date/Submit date to CMS in status section
- More at-a-glance information in dashboard
- Reconsider dashboard table header names re: users
Program Activities
What people liked about it
- Really like budget summary table view (though concerns about data entry)
- Sticky header works!
Opportunities
- Personnel assigned to activity in one fell swoop, rather than separately.
- Rethink nomenclature around contract personnel sections
- Better explain HIT overview, listing different projects by funding
- What makes ‘key’ personnel?
- How to differentiate text inputted by different people (Track changes)
- Different activities are reviewed by different people, how to account for this?
- Export to Word, as well as PDF.
- Tables: Summary, Quarterly, Incentive payments, Program budget table
- Executive summary
- A/B test the long form vs separating by sections
Program Summary / Results of Previous Activities
What people liked about it
- Ability to format text with the rich text editor
- Instructions formatting for HIE Overview
Opportunities
- In Program Overview, clarify what years the APD covers vs what years funding is being requested for
- There was a tendency to paste in a chunk of information in the Program Introduction – need to let people know they will have the opportunity to add in more detail in the following sections
- Add in placeholder numbers under actual (could just be the approved amount if we can’t pull in the number from MBES)
Program Activities
What people liked about it
- Appreciated the Activity Schedule Summary table
Opportunities
- 280 character limit prompted questions – need to re-work the phrasing of that character limitation
- There was a tendency to paste in a chunk of information in the Brief Activity Summary – need to let people know they will have the opportunity to add in more detail in the following sections
- Text in alternative considerations should specify “previous versions of the IAPD update”
- Add rich text formatting for objectives for people who want to add in a bulleted list
- States don’t always have the names of then vendors when they are making the APD request, rethink help text around the names in case people don’t know the names yet or anticipate that the names might change
- Language around key personnel percentages needs clarification
- Should be salary and benefits and %FTE attributable to this activity
- Could put a box under salary amount that shows the math
Summary Budget table
What people liked about it
- Autosaving and seeing the tables update when a number changes in another section
- Auto-filling of the table
Opportunities
- Add staff names under +
- Ability to export table into Excel, then edit it there and import it back into the tool
- Quarterly table - could specify when/which quarter(s) the activity will be funded while entering the activity
Executive Summary
What people liked about it
- It’s not a lot of detail, but if it’s enough for the reviewer, it’s enough for states
- Likes this because it clearly shows what is going to be approved
- Using this as a final review section to double check everything
Opportunities
- Link activities to the activity section
Single page scroll
What people liked about it
- The ability to enter something in and then scroll down to see the effect
- It’s close to the current IAPD structure
- Having everything in one place
Opportunities
- Easy for people to get “lost in the scroll” – need an intuitive way to navigate and indicate where you are
Multi-page scroll
What people liked about it
- The navigation on the left makes it easy to navigate to the different sections
- Can see what needs to be done in each section
- If you have a question about a particular section or want to point someone to a specific place, you can do so without scrolling
- Use a real example of an HIT/HIE activity
- Help text for Program Summary and Program Activities
- Activity breakout in the prototype
- Navigation between sections/subsections
Program Summary
- Better clarity around what’s needed (and not needed) in summary sections
- Decide level of narrative required for summary sections
- Current entries will be inputted from print documents (often with many years of history) — need transition strategy
Personnel
- Need to standardize dollar amount fields
- Indicate required text — “I don’t know if any of these fields are required or not.”
- Add help text to differentiate key staff from activity state staff
- Need clearer help text to differentiate additional non-personnel expenses — “Contracted services are all widely different, they don’t fit into anything standard. I think you have to have a way for people to explain them.
Expenses & Contracts
- A drop-down menu for consulting services. — Would want to see consulting services here too in the dropdown, you could get rid on the contracting section if you add it in the dropdown here
- Contracting section needs to better explained. — “When you say do you really mean a name of a person or a title?”
Executive Summary
- Some confusion around whether the Executive Summary should be first or last — those who complete it last, liked having it at the end. Still reported that it was confusing at first.
State dashboard V1, long scroll concept
State dashboard
What people are excited about
- Having the timeline and expected approval date
- Seeing the steps on the CMS side
- Having the dates and timelines up front helps PMs
Considerations
- Add a way to track the dates for the contracts and subcontracts in HIE space
- Add a card for tracking current tasks that need their attention
- Allow people to add their own dates to the calendar
- Add the approval letter to the dashboard
- Wants to see what dates the contracts and APDs are approved through
- Wants the structure for the timeline to be: APD > Contract > RPF > Contract amendment
Long scroll
What people are excited about
- Auto-calculations of the budget
- Proposed budget pulling details from the activities that were entered
- Being able to save as a pdf
- Love getting rid of appendix D
Considerations
- Grouping HIT and HIE together in the narrative
- Auto-generate the executive summary based on the activities that have been entered already
- Help text that says what would kick the APD back to the state
- A way to expand and collapse all sections and when a state side reviewer logs in, they want to see all the sections auto-expanded
- The ability to highlight something and add a comment letting someone know it needs to be updated
- Help text that gives examples of what the classic activities are and what are not activities. Could give a range for the average amount of activities
Live prototype
Considerations
- A final review that flags the areas that need to be fixed before submitting
- Clearly show which activity a subsection relates to - maybe a sticky nav for each activity
Revised content prototype
- Do the schedule, personnel, and expenses sections make sense to state users with how they view APDs and their mental models?
- Where do states need more clarity (help text)?
- How do state users understand the instructions and what they are being asked?
- Do the form fields provide enough space for users to input necessary information? If not, what do they need?
- How are they filling in the information (copying, pasting, etc)?
Activity Schedule
- For the Activity Schedule, it's helpful to indicate what’s required and what's optional
“Asteriks or other indicators of what’s required would be helpful. When completing APD, there’s information you know from the outset, whereas other information (Status, start date) comes later. This enable you to know what information you’d need to have ready to complete the APD.”
- Need to increase the input field for milestones
“The milestone section isn’t big enough, there are not enough characters.”
Personnel
- Change “Contracting personnel” to “Contracting resources”
“I have state personnel and allocated personnel. On the contract part, it should read contract resources because they’re not personnel”
- Are job descriptions and names necessary?
“How much are you going to hold people on these estimates? You’re being so granular? Me to report back against this is going to be problematic.”
-
Sometimes the name is just TBD, sometimes they know the name. Title and description are important and required and name as an optional field
-
Need to increase the input field for name and title
“Name/Title, etc. fields are not big enough”
Select Expenses
- Add help text to give examples of these expenses
“Facilities? What is that? Is that rent? We’re put it in training and outreach, if we had to find somewhere to hold a symposium, it might be a training & outreach.”
Expense List
- Test breaking down into separate categories
“We just give a one line expense number that we report back with all expenses - everything from travel, memberships, etc., Leaving it higher level is okay, so there’s less constraint for how we spend.” “How do you distinguish equipment from hardware?” Should expenses be in a narrative format instead of list? “We outline the current numbers in the narrative form of the document rather than in a table. Use the narrative form to contextualize what we put in our contractor/staffing level. (e.g. contract firm, not the individual resource.)”
Program Summary
- Link summaries to earlier sections so they can be edited
“I think an activity summary and total summary would be helpful, so you could go back and tweak sections.”
- Allow people to download the full report and information entered as a pdf
“When you download the PDF does it come in report form or a summary? People are going to want to read the entire report in PDF format through multiple rounds of reviews. I would be inclined to think download PDF would give me a printable version of all of the information I just completed.”
The overall concept of an online APD creation and submission tool
- The APD is used as a tool for facilitating conversations and presenting to leadership. They try to make it look professional by including graphics (which are not required) in order to supplement the narrative.
- Writing an APD is writing the story of what they are trying to accomplish and how. They are afraid they won’t be able to tell the full narrative in less than 100 pages (they also seem to be proud of how long their document is).
- There is no transparency into the review process and they don’t understand how decisions are being made. If we veer away from the 100-page free-form documents, states are afraid they won’t be able to fully tell their story. * They are scared that the decisions will be political and based on the numbers rather than their words.
Opportunities:
- Doing the math and Excel templates/importing spreadsheets
- Insight into the decision-making process on CMS’ side
- What are the inputs that feed into this process?
- Set expectations about who the audience is for the APD (Is it just CMS? Is it CMS + internal teams?)
- Based on the audience, need to identify what is necessary to include (and what gets in the way for CMS to * efficiently approve)
- Setting expectations about what happens after submitting
- Communication back and forth
Focus on:
- Single page (less wizard-like)
- More instructions about what CMS is expecting
- Allow for more free-form (less constrained and choosing from a list)
- Here are some common ones, and here are some ways to add in others
- How is narrative captured?
- A way to upload graphics
Content prototype
- How do state users react to the create activity, goals, and objectives section?
- Does the goal and objective structure make sense to state users? Do users see a connection between activities, goals, and objectives?
- What activities, goals, and objectives are states planning? How do state users describe the activities, goals, and objectives they’re planning?
As we continue working on this APD submission experience, we will make many changes to the design. What is one thing that we should not change?
- Workflow
P4: the flow and the way it is working aligns with how one would intuitively write and APD. Start with contact info, dig into the activities, and then add details. This matches with how we do APDs.
P2: Don’t change the functionality that it autosaves your work. When I go back and it’s still there, don’t change that.
- Core concept
P1: The concept itself is good. I like having the ability to streamline it, but I definitely think there is room for improvement with the concept of having it online, I like it.
What is one thing we definitely should change?
- WYSIWYG
P1: I think it might be helpful to have at the beginning, like TurboTax, where this is going to take your answers and put in format, have a sidebar menu where you can jump around and see what’s this all leading to? Where is this getting populated into an IAPD document? How is this getting integrated? You could have something at the beginning saying how this tool works. Will it literally take your answers and drop it into something in the end? Set the understanding of how this works.
- Language and form descriptions
P4: having more narrative to describe what needs to be entered or what needs to be typed into each of the various sections.
P2: The specificity of the highest level words. I can tell you’re building this hierarchical structure of what the words administration means, I assume you have approaches and projects next, the top level of the hierarchy doesn’t make sense to me. Maybe it’s because we started with administration and I would understand the others. *Vision and goals P3: I would avoid asking a lot of those questions, like vision and goals and stuff like that. [...] If there was a worksheet for the financial part that everyone did the same and was understandable, that would be great. Visual design
P4: is it possible to have a lot less white screen! Really unpleasant to the eyes.
- How do states and CMS think about and work through the APD process?
- What are the challenges with the current process?
- What are the basic needs of our users?
- Doesn’t know how to do it wrong because CMS is having an ongoing conversation
- Communication break after submission
- Differences in the communication process
- Differences between HITECH, E&E, and MMIS — pertinent points
- Two-year scale of funding is consistent with HITECH from the beginning, but new level of modularity for other funding streams
- Because of review timing, states make updates each year to continue funding cushion
- MMIS and E&E are centralized but not standardized; HITECH is more standardized
- More the norm across the U.S. to update an APD than create a new one
- CMS letters and guidelines theme
- States not feeling like communication and letter response time is fast enough
- HITECH checks that submitted budget matches letter budget within $1
- States aren’t sure how to allocate folks across programs and activities; methodologies are unclear
- Team Working Agreement
- Team composition
- Workflows and processes
- Testing and bug filing
- Accessing eAPD
- Active Documentation:
- Sandbox Environment
- Glossary of acronyms
- APDs 101
- Design iterations archive
- MMIS Budget calculations
- HITECH Budget calculations
- Beyond the APD: From Paper to Pixels
- UX principles
- User research process
- Visual styling
- Content guide
- User research findings
- eAPD pilot findings
- User needs
- Developer info
- Development environment
- Coding Standards
- Development deployment
- Infrastructure Architecture
- Code Architecture
- Tech 101
- Authentication
- APD Auto Saving Process
- Resetting an Environment
- Hardware Software List
- Deploying Staging Production Instances Using Scripts
- Terraform 101 for eAPD
- Provisioning Infrastructure with Terraform
- WebSocket basics
- Operations-and-Support-Index
- Single Branch Deployment Strategy
- Ops and Support Overview
- Service Level AOI
- Incident Response Plan
- On-Call Policy
- Infrastructure Contingency Plan
- Updating CloudFront Security Headers
- Requesting and Installing TLS Certificates