Beyond the APD: From Paper to Pixels

APD Creation Process

The eAPD APD creation process has been heavily shaped by our research findings, user feedback via email and demos. One might conclude that the eAPD would be simply digitizing the paper APDs, Advanced Planning Documents, but that oversimplification couldn’t have been farther from the truth. There are a number of factors to consider when creating eAPD, namely how the information architecture would look like in this new user interface, how we can incorporate and improve state author processes in the eAPD workflow, and address State and CMS pain points in the design. All of which necessitated testing designs, communicating the findings, and turning the findings into actionable tasks that would improve eAPD.

Information Architecture (IA)

There has not been a standard APD template across all of MES for some time. While the HITECH and the E&E templates were approved by the Office of Management and Budget under the Paperwork Reduction Act requirements, a bulk of APDs submitted to CMS are the form of MMIS APDs. In all APD types, States have relied on past APDs and feedback from State Officers for what information is important to add to their APDs. This lack of standardization led to discrepancies in how APDs were reviewed. eAPD strived to bring a standard to APD writing, and this started with the HITECH IAPD OSM template and a new information architecture. The information architecture refers to the organization of a website.

Logical Structure

Federal regulation and the HITECH OSM template gave us insights to what sorts of fields to require, but eAPD had free reign to request that information into a logical structure, while also holding the required HITECH template. The structure of the eAPD had to make sense to ensure that key features were intuitive and easy to find. Within the builder, we needed to organize fields in a manner so that users could enter related information in one place and that primary information such as Results of Previous Activities preceded Proposed Budget information.

Asynchronous APD Authoring and Field Structure

Another example of information architecture we had to consider very early on was how to structure the fields. 18F tested how most users entered in fields for the paper APD and found out that they entered in information as they received it and did not follow a particular sequence, every time. With this revelation the team decided to structure the builder in a way that you can jump around the form and answer questions when available and the form would not bar you from proceeding. This revelation brought forth our autosave function, which is unique from other CMS products, and shaped how we did validations. Although the paper template had to have all relevant fields, enumerated by regulation, establishing a concise information architecture that considered future expansion humanized the required fields and made it easier to create an APD in a sequential manner. In the future, we had planned on improving the IA in regards to how the dashboard is organized.

User Process

Understanding users’ processes was a key motivator for conducting research. We needed to understand how users currently create APDs, how state officers review APDs, and in what ways we could improve on their existing processes through eAPD. We would first draft a user workflow, design it, and test.

Lock and Submit

A good example of this is our Lock and Submit Research. The idea behind the Lock and Submit functionality was to allow states to submit APDs to CMS and prevent (“lock down”) states from making changes when a submission occurred. With the lock and submit workflow, we asked users about how they review APDs for submission, how they collaborated on the review, who submitted their APD, and what were their next steps. Asking these questions allowed us to validate our existing user workflow and improve on the potential next steps involved when a user locks an APD for submission.

Central themes in eAPD Research

State User Pain Points

During our various research sessions, office hours, and demos, we found a commonality between users based on their roles. Through research we identify seven main themes in state users voiced were: Users want the process of creating an APD to be easier. Within this priority, content wise, we crafted our help text to be succinct, accessible, and understandable. A common response to our research prompting about what States wanted to put in a particular portion of the template was, “What does CMS want here?” The help text was designed to answer this question. We also provided example boxes to help users have specific examples for what was expected in their responses. We also tested the use of help drawers which would present relevant regulations when users needed a reference.

Users find the budget fields in the paper APD to be complicated. With this revelation during research, we decided to do the math for users, whenever possible. Users will simply enter in values for key state personnel, state staff and personnel expenses, private contractor costs, other funding, and would narrow the parameters with the Federal Fiscal Year (FFY) and match rate and then the builder will calculate the totals within the Budget and FFP, Proposed Budget, and exported APD. Future research for this work was to add the MBES line items to each part of the budget so we could automatically generate the requisite MDBT tables and/or submit the data directly to MACFin through an API.

Users want the process of creation and submission to be centralized. Since the user workflow had traditionally been decentralized (internal state processes, CMS email box, CMS internal processes, and sending communications to the State), it was a challenge to centralize the process of creating an APD, APD submission, CMS review and CMS certification within eAPD. We reached out to the Sharepoint team to gather information on the process after the user has submitted their APD and a state officer reviews it, in turn we planned to provide information on our process as well because if a state officer had clarifying questions or revisions, the user would have to return to eAPD to resolve it.

Users want to be able to collaborate with others in their state. From the Required Fields Research, we deduced that states can be structured very differently, partly because of the states size. Some states have one person writing all the APD and another reviewing it. Some have three different authors specializing on one funding source. All required some sort of collaboration among their budget team, calculating costs, authors, and the people in charge of reviewing it before submission. With this information, the plan was to create an intermediate step to prevent data loss, then allow multi-user editing and track changes. These changes would help translate the in person collaboration into a digital space. Users want to minimize rework.To make the APD creation process easier, we worked towards data population in the app in general and within the builder. Projects can span decades, so as long as you created an APD within the system, your project’s APD for the following year would populate entries from the previous APD. Additionally, we would populate users' entry into a consolidated budget page.

State Officer Pain Points

State officers want an APD to be administratively complete (all the required fields are done and relevant documents are submitted) before a final review. Historically, there had been a lot of back and forth between the state officers and state authors to complete required fields. Some of that confusion stemmed from there being less guidance on what fields were absolutely necessary to submit an APD. We gleaned this from user research and conversations with your product owner, Jerome Lee, who is a state officer. With this information, we came up with a document, identifying required fields, then created the administrative check, which would scan through your APD once you were ready to export and point out incomplete required fields. Once a user was complete with the administrative check, there would be an attestation in the exported APD, announcing to the State authorthat the APD was administratively complete, thus minimizing this introductory review of required fields and documents by a state officer. State officers want to quickly scan through an APD and see the projected costs, past costs (Results of Previous Activities), and a justification for a project at the forefront of an APD. During our State Officers Early Access Feedback sessions, we tested state officers and asked them what they looked at first. Among other changes, we learned that we needed to focus on the content of the executive summary. Executive summaries should not exceed 1-2 pages, but we discovered that many of the executive summaries in our sample set exceed that length and included more detail than was strictly necessary. The eAPD required states to be concise and automatically generated the executive summary so we could (1) provide the State Officer a concise view of the scope of the APD and (2) a concise view of the budget of the APD. State officers want to receive a concise APD that is easy to review, historically they have been several pages long and would expand when a new APD was created. During a follow up interview with a state officer who participated in our HITECH Beta Testing, we gathered that eAPD has dramatically decreased the size of APDs, making it easier to review. Their APDs page size decreased from 82 pgs in the previous year to 19 pgs. Creating a concise APD form design within eAPD led to less time spent reviewing and more time spent doing strategic thinking on the content of the APD.

CMS Pain Points

There is a reasonable justification for a project and the APD used to fund it. As noted in the state officer pain points, this is one of the factors, CMS looks to first, in addition to the dollar amount requested. As a result, we structured the builder in the way that most of the justification is answered in the beginning of the APD in the APD Overview and Activity Overview pages, then in the activities sections, users enumerate their costs. This is the critical factor in whether an APD is approved, returned for clarification, or rejected, so it is important that it is in the forefront of our page designs. The Funding requests and project complies with regulations. eAPD is not built with a list of requirements, but key capabilities outlined from the product owner/manager (who is also a MES state officer) checked against user insights and federal regulations. The product and design team is constantly measuring the UI against a set of APD specific regulations. Similarly, we require that users attest they comply with federal regulations in the Assurances and Compliance and Conditions for Enhanced Funding pages to meet CMS’ requirements

eAPD Past Research Accomplishments

We were able to test UI in different stages of development. For instance, in the Lock and Submit research since we needed quick feedback about the direction of our workflow, we made our figma designs interactive and tested within Figma. This low lift research method allowed us to get quick feedback without putting in a lot of effort in development to find out we were headed in the wrong path. At the same time, design often collaborated with development to test features in our staging environment, which gave us greater insights on usability, findability, and accessibility. We kept the entire team engaged in every research session. Design would create the prototypes with annotations on how the components were supposed to behave. With that development would develop the prototypes in a testing environment. During the research planning, we involved the team in hypothesis and research goal writing portion of the research planning. This allowed the entire team to understand what we were testing for and the expectations for the research. The entire team would sit in on the research sessions and we would debrief afterwards. This allowed the team to be on the same page on the research findings before the design team shared the research reports. Research became a motive for our design decisions. Design heavily referenced research findings on decision decisions. The research finds helped us create accessible, inclusive, and intuitive designs for our users. In addition to using design findings and user goals to shape decisions, we used it to decide which decision tasks to prioritize first. While synthesizing research, we would make a list of action items based on findings, create tickets from these findings, and prioritize based on severity, impact, and how often the finding came up in the research. Therefore, research played a huge role in shaping our roadmap.

Current Research Opportunities for eAPD

The eAPD system has advanced from ideation, design, development, and user testing to now providing digitally simplified features for MMIS IAPD creation, eliminating the need to create APDs on paper. Iterative research and user testing studies are at the core of how we effectively validate assumptions and test the success rate of our current 2023 OKRs:

• Increase the number of states leveraging the eAPD product to create their APDs
• Ensuring that at least 20 states/territories have registered for the eAPD and completed the Emergency Use Authorization (EUA) process.
• Having 75% of registered states submit an eAPD.
• Having 50% of registered states submit at least 2 eAPDs, demonstrating that users find value in utilizing the eAPD.
• Streamline MES (Medical Enterprise System) Business Processes and Operations
• Achieving a 75% first-cycle approval rate for eAPDs, demonstrating that the submitted documents are regulatory and administratively complete.
• Aiming for a 30% decrease in average review times for eAPDs to improve the utilization of State Officer time and oversight of projects.

Hypothesis-driven Process

To validate assumptions and track metrics that correlate with the system's targeted OKRs, the team planned to conduct a Pilot Launch for MMIS IAPD integration during the second quarter of 2023. In addition to launching the Pilot, the eAPD team planned to facilitate remote usability testing with a few state users, to assess the effectiveness, efficiency, and user satisfaction of the eAPD system. We utilize a hypothesis-driven process as an essential and iterative way to understand user needs, explore concepts that align with both user needs and business objectives, conceptualize designs for proposed solutions, develop forward-thinking capabilities that link HCD with business processes, and test hypotheses with actual users to verify them. These are a few key areas highlighted in the MMIS IAPD Pilot Launch that have been implemented to facilitate a more streamlined APD creation process:

• MMIS IAPD: New funding source type
• The APD Overview: Displays a frame of reference that reflects the user's funding source type: HITECH or MMIS IAPD, FFY, and Medicaid Enterprise Systems Business Area(s).
• MDBT (Medicaid Detailed Budget Table) inclusion for MMIS: Support a designation for capturing DDI (Design Development & Installation funding) and M&O (Maintenance & Operations) Costs.
• Administrative Check for APD Completeness: Simplifies the validation process for APD authors and reviewers
• Aesthetic modifications to the user interface:
• Updated tabular data structure to accommodate efficient readability and accessibility standards
• Clear, informative, and descriptive help text for form fields
• Improving the overall usability of the user interface by applying accessibility standards
• Export View: Provide the capability for users to review and download a copy of their APD as the first step in submitting a completed APD to CMS

Modernizing and Improving the APD Submission Process

Overall, the eAPD system offers state users a more efficient, accurate, accessible, transparent, collaborative, and cost-effective way to submit APDs, making it an important tool for modernizing and improving the APD submission process.

• Efficiency: The eAPD system streamlines the APD submission process by providing a digital platform where state users can submit, review, and track APDs online. This eliminates the need for paper documents to be physically mailed or delivered, reducing administrative overhead and processing time.
• Accuracy: The eAPD system allows state users to input APD details directly into the system, reducing the risk of errors that can occur during manual data entry. It also provides validation checks and prompts for required fields, helping to ensure that complete and accurate information is submitted.
• Accessibility: The eAPD system can be accessed online from anywhere, at any time, providing convenient access for state users. This allows for greater flexibility and accessibility, especially for users who may be geographically remote or have limited mobility.
• Transparency: The eAPD system provides a clear and transparent process for submitting and tracking APDs. State users can easily monitor the status of their APDs, track any updates or changes, and receive notifications on the progress of their submissions.
• Collaboration: The eAPD system facilitates collaboration among state users by allowing multiple users to access and work on APDs simultaneously. It also provides features for comments, document attachments, and version tracking, which enables efficient communication and collaboration among team members.
• Cost savings: The eAPD system can potentially result in cost savings for state agencies by reducing printing, mailing, and storage costs associated with paper-based systems. It also helps to minimize the risk of lost or misplaced documents, saving time and effort in tracking and retrieval.