Supplemental content crossrefs findings and recommendations II
- Adding a “See more” link to make sidebar content less visually overwhelming is helpful but not necessary for the first state of implementation. The LOE is relatively straightforward given that this kind of link does not change the structure of the information.
- Scratch the “email us” link in the sidebar for now. Add in the same place something like “For additional Medicaid and CHIP resources, visit Medicaid.gov” and link to https://www.medicaid.gov/medicaid/index.html.
- Change the Federal Policy Guidance link to “Search Medicaid and CHIP Federal Policy Guidance on Medicaid.gov”.
- Change label of the supplemental sidebar from “Subpart Resources” to “Subpart X Resources” to match whatever subpart is displayed.
- Link to the State Medicaid Manual under Implementation Resources
- Include sidebar definitions under the Term category (with abbreviations and formally defined terms) since it is helpful to users (and less problematic and labor intensive than inline definitions).
- Include a cross-reference link to Part 400 in the same category as an additional source of definitions.
- Make no changes to Related Regulations right now, and plan on doing a content design pass later to make it more specific and easy to understand
- As part of content authoring, try linking to both sources for the Social Security Act (SSA.gov and House.gov).
- Continue with the plan to link to government offices and departments
- Switch to linking to the Department of Labor page with the Rehabilitation Act for cross-references to that Act, because it is clearly labeled.
- For older rules, go with the Federal Register for now. Later, we can enhance with handmade LOC links. For newer rules, also use the Federal Register.
In this study, we evaluated whether the current supplemental sidebar content meets the needs of policy experts, and identified which cross-reference sources (and format) are most useful and trustworthy for policy experts in their research.
Overall, participants had favorable impressions of eRegulations:
P1: Good to have the “one-stop” shopping - being able to find different resources without needing to open multiple tabs and bookmarks.
P2: It can be sometimes taxing to keep up with [guidance]. Having to go back to multiple sites is very time-consuming. Going back to something like this, to reference those actions, would be really useful."
P3: coming here instead of googling for FR notices ... that would be fantastic.
Key findings include:
- People’s perception of reliability over time will determine trustworthiness: while most participants felt that eRegulations should or would be trustworthy given its origin (CMCS), the need to know who will keep content regularly updated (and how) is the most significant factor in eRegulations being trustworthy.
- Generally, people found it beneficial to have rules go back to the 1970s because it could save time and effort during research.
- Linking out to Medicaid.gov is useful, but it needs to be clear when and where we are linking and that it’s contextually appropriate.
- Largely, sidebar categories are mostly understandable and complete, except for a few areas. Related Regulations is still unclear to some, and the sidebar likely needs additional labeling to communicate the relationship of information between the sidebar and the regulations text itself.
- Participants also gave useful feedback on the sources we’re linking to via the cross-references in the regulations.
The focus of this part was to gather feedback from policy experts around overall content suitability.
One of our key questions was how trustworthy eRegulations felt to the participants. Four out of five participants found the content trustworthy. Most gave reasons why:
- P2: “the terminology mirrors the .gov sites and sites I go to. Seems like it's compiled accurately.”
- P3: “it's linking to other federal resources. I trust my employer.”
- P5: “if this is something that CMS is providing and is behind, I would consider it trustworthy”
For P5, however, they said it would depend on who published the content and how often it would be updated, and that would inform how they would regard the trustworthiness of eRegulations.
From this, we can infer that users will likely feel comfortable initially with the trustworthiness of eRegulations, but trustworthiness will only really be established with an agreed-upon (and supported) governance process.
Recommendations
- Continue working on the content governance planning in collaboration with CMS: both the governance workflow as well as messaging on the eRegulations website around who, how, and when content is updated.
Another key question was around the importance of including rules as far back as the 1970s. While people do not have to check historical information on a regular basis, the consensus among participants was that it was better to have them because of how time-consuming it is to find them when they are needed.
P4 keeps a desktop folder with rules from the 80s because they needed to know where the requirements came from for state plan amendments. Additionally, some rules from the 70s are still in effect. P3 summed it up by saying that:
For rules, I would go back as far as we can. They are not that useful, but it’s good to know when they’re not that useful. Sometimes you really need to know what thing changed when, [such as] how long we have been using a term without defining it?
In terms of level of effort for the eRegs team, it is not a significant addition of research work to include all the subreg guidance for the supplemental information. For the final rules and NPRMs, the Federal Register automated information only goes back to 1994, so anything earlier than that would need to be manual, which would have implications on development.
Recommendations
- Continue with the plan to go back as far as the 70s. To keep the content less visually overwhelming, identify a cut-off point (such as anything older than 5 years) and use a “see more” link or similar to remove that content from the default view.
- Given that the Federal Register only goes back to 1994, the team will need to discuss how to include older rules manually (in addition to the automatic upload).
The next questions involved understanding how useful it would be to link to Medicaid.gov, and whether it would help establish the relationship of eRegulations with other resources.
When linking out to Medicaid.gov’s Federal Policy Guidance page under the subregulatory guidance tab (via the link text “Search all Federal Policy Guidance”), all participants expected that link would take them to Medicaid.gov “Federal Medicaid Policy Guidance” page. However, 4 out of 5 participants stated that it would be helpful for those less familiar to clarify that this was linking to Medicaid.gov. Both P3 and P4 specifically stated that including this link would be helpful to them as it would save them time and effort finding subregulatory guidance.
When linking out to Medicaid.gov’s Medicaid page (via the link text “Browse additional materials”), all participants expected that link would take them somewhere on Medicaid.gov, but no one was sure where, given that “some of our guidance is pretty scattered,” according to P3. Interestingly, P2 was the only participant to correctly guess where the link would go, but the participant’s mental model was actually of the page’s dropdown navigation menu, not the page itself. This is unsurprising given that the dropdown navigation menu has more information than the page itself, as the menu includes the subitems for the policies and program topics (unlike the page itself). P5 recommended adding clarification on where this link would take users.
It is difficult to state conclusively at this point whether linking to Medicaid.gov helps establish the relationship between the category of implementation resources on the eRegulations site and other resources on Medicaid.gov, because related resources are housed on many different pages on Medicaid.gov. Clarifying where eRegulations is linking to on Medicaid.gov is a good practice and will help establish eRegulations’ place within and among Medicaid.gov resources.
Recommendations
- Keep the Federal Policy Guidance link, but amend the link text to say something like “Search Medicaid and CHIP Federal Policy Guidance on Medicaid.gov” .
- Keep the link to Medicaid.gov’s Medicaid page, but consider moving it from the Implementation resources category sidebar category to the Subpart Resources sidebar home. Use a clear phrase like “For additional resources, visit Medicaid.gov” to point to the wider world of resources.
Regarding the order of sidebar categories, only P5 stated a different preferred order: statutes first, then subreg guidance, implementation resources, rules, and lastly, related regs. P1 didn’t mention a preference. The remaining three participants thought the order of categories was appropriate:
Nothing jumps out at me as needing to be reordered. [It’s] really intuitive to have the sections broken out like this, being able to use the dropdowns and the implementation guidance. -P2
Order makes sense: from least to most malleable guidance. Terms at the end are like definitions at the end of a textbook. - P3
Order of topics is exactly how I would have done it: in order of precedence, most important, then sub-sub-reg guidance. -P4
We also wanted to probe people’s understanding of category labels and the content within those categories. Most participants found the category labels informative, with the exception of one category that is still proving to be confusing (Related Regulations).
Two participants (P1 and P3) would have preferred having the State Medicaid Manual (SMM) included as a Subpart resource. P1 only noticed the lack of SMM in the sidebar after seeing that we link the mention of SMM in the reg text itself. P3 specifically stated the only resource the Subpart Resources sidebar was missing was the SMM, and that they would probably place it in the Subregulatory Guidance category. Even though it’s out of date, it’s still a useful resource for their work. They said:
It would be helpful if [eRegs] was able to tell me to go to a specific section of the SMM. It would be nice to know which zip file to download so I can find the right TOC [table of contents].
All participants regarded terms in the sidebar as being a useful category for the public, especially state staff. Even CMCS staff learning policy would be helped: P4 noted that “everyone started with a list in their notebook for what you had to ask somebody else for and had to refer back until we memorized them.” Other interesting comments from participants:
- P1 would want to see the whole list of definitions while looking at the reg, possibly in a panel you can view while viewing the reg. They said even if the terms list was long, it was okay because it’d be alphabetical.
- P2 found the terms category helpful and wanted to be able to click the links and go to that part.
- P4: terms: helpful if direct quotes from the statute, reg, or an interpretation (but would need to know that)
- P5 shared that they often have so many confusing terms or overlapping citations, so to the extent that [eRegs] can help define or connect those dots for people that's good.
The one category that is still unclear for participants is Related Regulations. While it’s possible that this might be a limitation of the Figma prototype, there is still a question around whether we need to continue with the current way we’re defining Related Regulations, or change it to something else. P4 thought relevant might mean relevant only to the application reg, vs the entire regulation. Similarly, P3 guessed that “related means these are pieces I would use to help interpret 435.907.” If we keep the current definition, there are a few techniques we can try to test this again: reorder Related Regulations in order of section to which they relate, and offer a tooltip to describe further what we mean by related.
This points to a larger issue we’ve been observing with participants. We have hypothesized that eRegulations content (regulation, cross-reference, and supplemental content) will help our primary users complete their work confidently and efficiently. However, in testing eRegulations, we have observed a problem: it is easy to misunderstand the extent to which the content in the sidebar relates to the section they are looking at. Again, this may be a limitation of the Figma prototype, but we can’t take for granted that it’s the only reason why people are not yet grasping how the sidebar content as a whole relates to the regulations in the main content area. The solution is to more accurately convey how the sidebar content maps to the regulation they are looking at. There are a few possible approaches (not mutually exclusive):
- Label the sidebar Subpart X Resources
- Organize content in order of relevant reg section (vs the current: by date)
- Label the documents themselves with the relevant section or subpart when it applies to all sections in a subpart
Recommendations
- Within the reg text itself, the SMM is not referenced often.
- Until SMM is either updated or deprecated, we need to make a call about whether we think CMCS will be served better by linking to SMM, including to save people time when they do need to find it.
- Participants found terms to be beneficial. While we identified issues with the previously tested inline definitions (high LOE, risk of possible over-interpretation), definitions in the sidebar might be more feasible. Consider the level of effort for sidebar terms and weigh it against the prioritized features.
- In the meantime, it might be helpful to have a cross-reference link to Part 400 in the terms section to mitigate the lack of (initial) robust support for terms.
- If we keep the current definition of Related Regulations, consider reordering Related Regs in section by section order. Also consider adding a tooltip to describe further what we mean by related and test both changes again.
- Similar to the ordering of Related Regulations, consider reordering Subregulatory Guidance by relevant reg section instead of date.
- Consider labeling the documents themselves with the relevant reg section or subpart when it applies to all sections in a subpart
- Consider relabeling the sidebar to Subpart X Resources (the ‘X’ being whatever subpart is displayed)
The second half of the session involved sharing a link to a “content wireframe” of sorts where participants clicked on various cross-reference links to help us evaluate the sources we link to. Overall, the feedback was summed up best by P2: including links like SMM makes for a more "illustrative, interactive way to read through regs and analyze the pieces that go into them." This feedback can be seen as extending to the practice of linking government offices and departments as well.
Linking to the State Medicaid Manual Definitely helpful to link to SMM within the reg text. P4 said they “love that direct link – I never know where to find it – it would be great if it's right there in what I'm reading.”
SSA.gov vs House.gov for the Social Security Act 3 out of 5 participants preferred SSA.gov vs house.gov, though two participants noted that SSA.gov is not always up to date. Only one participant (P2) preferred house.gov because of its readability (they found SSA.gov difficult to navigate). P3 suggested that while they themself preferred SSA, if "one tends to be more up to date than the other, [they] would link to that one." P4 thought it would be ideal to somehow link to both.
eCFR vs beta eCFR for regulations outside CMCS Regulations None of our participants were familiar with beta eCFR (but are experienced users of classic eCFR). Most participants seemed intrigued by the offered features in the beta (sorting, searching, historical versions) and seemed open to trying it.
Linking to government offices and departments 4 out of 5 participants thought it was fine to link to government offices and departments (we did not get feedback from P2 on this specific kind of link). Only P5 remarked that it’s a question of “whether it’s too busy to have all the links. I prefer more links, it doesn’t bother me.” Providing links seemed useful for participants in scenarios where a name has changed or if a person is new and didn’t know (for instance) what the office of the national coordinator was. P1 said it was a good practice to link to a cited resource.
Rehabilitation Act Some participants found this source for the Rehabilitation Act confusing. Because we linked to the text in the reg “section 508 of the Rehabilitation Act”, participants seemed to expect to see that section heading in the US Code. However, that section number is not evident on the page, which caused confusion around whether they were being linked to the right place.
Affordable Care Act Participants thought the option we provided (HHS.gov PDF) was fine.
Older and newer rules Both options for older rules had issues: page load timeouts with the Library of Congress (LOC) scans vs Federal Register’s less readable scans. Participants vastly preferred being linked to the Library of Congress PDFs for older rules instead of the Federal Register citation url, even though they ran the risk of page load timeouts. For newer rules, most participants preferred govinfo.gov PDFs to the Federal Register webpage. However, some said that in thinking what the overall preference would be, it would be best to link to the Federal Register, and then people can choose the PDF from there.
Range of reg content in eRegs Participants were mostly fine with how this was linked, however P4 said they might prefer to be taken to the top of the provision (section) to double check that they were in 433.116 because they can’t see the top – despite there being a section indicator in the upper left corner. When this indicator was pointed out to them, P4 said they did not notice this.
Recommendations
- Continue linking to the State Medicaid Manual within the reg text when mentioned.
- Participants mostly prefered SSA.gov despite not being as up to date. There are a few options here: stick with SSA.gov because it’s the common preference among CMCS staff, offer both links somehow, or ask CMCS to weigh in and make a content decision about which they would prefer staff to use. An example of offering both links could be to consistently use SSA.gov within the text, but offering both House.gov and SSA.gov in the sidebar list of related statutes.
- Consider linking to beta eCFR instead of eCFR, as this would be a good introduction to eCFR’s eventual replacement, including offering helpful history comparison features for reg parts outside of the scope of eRegs.
- Since participants generally accepted these implicit cross-reference links to government offices and departments, as it saves time for those that might need it
- Given that the lack of heading within the linked site was confusing for the Rehabilitation Act, use an alternative that includes titles confirming the reader is at the right place, such as:
- Don’t change how we link to the Affordable Care Act
- Use the Federal Register web pages for new rules. For older rules, both options had issues, but participants preferred LOC, so discuss as a team which might prove to be the better user experience.
- Continue linking to the range of reg content in eRegs the way we are (though we will need to further study how noticeable the section wayfinding indicator is in the upper left corner).
- Evaluate whether the current supplemental sidebar content meets the needs of policy experts
- Identify which cross-reference sources (and format) are most useful and trustworthy for policy experts in their research
Overall content suitability
- How well does the sidebar content meet the expectations of policy experts?
- Do they find the content trustworthy?
- Does the content go back to a far enough date, or is it too far?
Category labels and descriptions
- How understandable are our current category descriptions?
- How well do they hint at the correct materials that will be under those categories?
- Are the existing categories the right ones?
- How helpful is it to have a category for key definitions?
- Are there additional categories that would be helpful to include?
- How well does the inclusion of the title (or truncated language from the section) facilitate browsing?
Content governance, feedback, and relationship to Medicaid.gov
- Do participants find it useful to have a link out to Medicaid.gov Federal Policy Guidance?
- Do participants find it useful to have a link out to other reference materials on Medicaid.gov?
Cross-reference sources and format
- Which cross-reference sources are preferred?
Participants P1-5 were recruited based on extensive policy experience in their jobs.
We conducted a moderated formative evaluation using the think-aloud protocol in a two-part format. We used a limited prototype for the first part, and an HTML content wireframe for the second part.