From 967c76b582a520bb9c21efde46ea47c63359f832 Mon Sep 17 00:00:00 2001 From: Ben Cuff Date: Wed, 4 Sep 2024 15:23:29 +0100 Subject: [PATCH] Rolling back earlier changes --- docs/quality-assurance.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/quality-assurance.md b/docs/quality-assurance.md index b5e1082..d43864a 100644 --- a/docs/quality-assurance.md +++ b/docs/quality-assurance.md @@ -8,8 +8,8 @@ layout: guidance-page >- Quality assurance should be planned in advance, with roles, responsibilities, and the extent of assurance activities understood fully by the entire team. >- Quality assurance should be appropriate and proportionate to the scope, risks, methodology, and data source of a piece of analysis. >- Understanding user needs is important when measuring the quality of your data. Perfect data quality may not always be achievable and therefore focus should be given to ensuring the data is as fit for purpose as it can be. ->- Teams should hold discussions to review quality and plan quality assurance on a semi-regular basis, identifying gaps in current procedures (our [quality review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) can help facilitate this). ->- A quality assurance log should be created and maintained (see our [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx)). This is a crucial tool to help teams plan, record, and sign off on quality; it provides an audit trail of everything that was checked and what the outcomes were. +>- Teams should hold discussions to review quality and plan quality assurance on a semi-regular basis, identifying gaps in current procedures (our [quality review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) can help facilitate this). +>- A quality assurance log should be created and maintained (see our [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx)). This is a crucial tool to help teams plan, record, and sign off on quality; it provides an audit trail of everything that was checked and what the outcomes were. >- The quality assurance process and any outstanding quality concerns should be clearly communicated alongside the final results of the analysis. @@ -27,7 +27,7 @@ The quality of an analytical output may be thought of in simple terms as its “ 4. Comparability and Coherence 5. Accessibility and Clarity -All quality assessments of official statistics should be conducted in line with these dimensions and both our [QA review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx) sets out these dimensions for you to consider. +All quality assessments of official statistics should be conducted in line with these dimensions and both our [QA review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx) sets out these dimensions for you to consider. See inside the expandable sections below for example questions you might consider as part of each dimension. @@ -136,7 +136,7 @@ It is our role as analysts to explain how any limitations in our outputs might i -These key stages along the data journey, alongside the 5 Dimensions of Quality of the European Statistical System (ESS), form the basis of our [QA review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx). +These key stages along the data journey, alongside the 5 Dimensions of Quality of the European Statistical System (ESS), form the basis of our [QA review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx). ## Roles and responsibilities @@ -204,18 +204,18 @@ If the work is being used for a new purpose, the analytical assurer should obtai 2. QA log template
- -
### What is the QA review conversation tool? -Our [QA review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) was created to provide a useful starting point for thinking about the QA of statistical outputs in line with the five European Statistical System (ESS) Quality Dimensions. It has been designed to facilitate team discussions about quality at either the start of a new project or at regular intervals for cyclical releases to help teams plan QA, identifying gaps in current procedures. We have compiled discussion questions that teams can use to reflect on the quality of their analysis, identify areas for improvement, and consider how to communicate quality to users. The outcomes of these discussions should inform QA plans and QA logs which teams should be using during QA activities. +Our [QA review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) was created to provide a useful starting point for thinking about the QA of statistical outputs in line with the five European Statistical System (ESS) Quality Dimensions. It has been designed to facilitate team discussions about quality at either the start of a new project or at regular intervals for cyclical releases to help teams plan QA, identifying gaps in current procedures. We have compiled discussion questions that teams can use to reflect on the quality of their analysis, identify areas for improvement, and consider how to communicate quality to users. The outcomes of these discussions should inform QA plans and QA logs which teams should be using during QA activities. ### What is the QA log template? -To help teams record and sign off on quality, we have also created a [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx). This approach is standard practice across government statistics and should ensure that a QA mindset is present when conducting analytical projects. A QA log should be used every time you are producing a publication. +To help teams record and sign off on quality, we have also created a [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx). This approach is standard practice across government statistics and should ensure that a QA mindset is present when conducting analytical projects. A QA log should be used every time you are producing a publication. A QA log will: @@ -248,9 +248,9 @@ The table below gives some considerations when completing this section of the re | Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |-------------------|-------------------------|-------------------------------------------|-------------------------| -| What is the need for this analysis or statistical release? | Does the output contribute something new that cannot be found in existing research or literature?

Has there been a clear user need for this analysis shown through user consultation? | Understanding why the analysis is needed and what it will be used for is critical for understanding whether what you have done is fit for purpose.

If you are responsible for part of an analytical process, understanding the end use will help you to make sure that your part of the output does what is needed to meet user needs. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)


[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) | +| What is the need for this analysis or statistical release? | Does the output contribute something new that cannot be found in existing research or literature?

Has there been a clear user need for this analysis shown through user consultation? | Understanding why the analysis is needed and what it will be used for is critical for understanding whether what you have done is fit for purpose.

If you are responsible for part of an analytical process, understanding the end use will help you to make sure that your part of the output does what is needed to meet user needs. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) | | Who uses your analysis or statistical release? | Does the work meet user needs appropriately?

Have you addressed feedback obtained from recent user consultations?

What are the expectations of my users?

Are the length and content of the publication appropriate for its users?

Are terms defined for your less-technical users? | Understanding who uses your analytical output will help you to make sure that it meets their needs.

It also helps you to tailor your outputs to make sure all your users are fully supported in using the outputs effectively. | [Guidance: User engagement top tips (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/user-engagement-top-tips/) | - | What analytical question you are addressing? | Are questions suitably explicit and do they clearly reflect the evidence gaps the analysis is intended to fill?

Will the output align with overarching departmental or programme goals? | Having a clear understanding of the problem your team is trying to solve ensures that the analysis you design is fit for purpose.

If you do not know how your work is contributing to answering an analytical need, you may be unaware of important requirements or limitations for your part of the work. | [Guidance: The Aqua Book (HM Treasury) ](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)


[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2)


[UKHSA Data Strategy (UK Health Security Agency)](https://intranet.ukhsa.gov.uk/sites/data-analytics-and-surveillance/SitePageModern/49668/data-strategy) | + | What analytical question you are addressing? | Are questions suitably explicit and do they clearly reflect the evidence gaps the analysis is intended to fill?

Will the output align with overarching departmental or programme goals? | Having a clear understanding of the problem your team is trying to solve ensures that the analysis you design is fit for purpose.

If you do not know how your work is contributing to answering an analytical need, you may be unaware of important requirements or limitations for your part of the work. | [Guidance: The Aqua Book (HM Treasury) ](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2)

[UKHSA Data Strategy (UK Health Security Agency)](https://intranet.ukhsa.gov.uk/sites/data-analytics-and-surveillance/SitePageModern/49668/data-strategy) | | Is everyone aware of their role and responsibility in the process? | Is a Senior Responsible Officer (SRO) required as the analysis is business critical?

What is the role of the commissioner in your project? Are there any key changes to the publications that need flagging?

Does the assurer understand the analytical methods used in the project? Are there any skills gaps? | For effective quality assurance, you need to be clear on roles and responsibilities throughout the chain of production. It is not enough to say that an individual will carry out some QA.

Assigning the roles set out in the AQUA book provide a useful framework for considering the different ways QA should be built into the life cycle of a project.

For more information, see Roles and responsibilities section of this guidance. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf) {% endcapture %} @@ -266,7 +266,7 @@ The table below gives some considerations when completing this section of the re |Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Is all of your documentation about the project easy to find? | Is there a single location (for example, on GitHub or SharePoint) where all project documentation is either stored or linked?

Have you got a link to your project code repository within the documentation? | Your analysis must be well documented so that somebody new can understand it and pick it up. Poor documentation means that other people will not understand why the process is configured as it is, how the process works or how to run the process safely, potentially leading to errors. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: QA of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | -| Have you got a consistent project structure set up? | Does the project structure align with other analytical projects in your organisation?

Is there a clear system to track version control within your project? | All projects should have appropriate version control including a clear system for version labelling and a version control log.

It is recommended to use a project template such as the [UKHSA Project Template](https://confluence.collab.test-and-trace.nhs.uk/display/QA/UKHSA+Project+Template). A template provides a consistent project structure and some built-in automated QA. | [UKHSA Project Template (GitHub link)](https://confluence.collab.test-and-trace.nhs.uk/display/QA/UKHSA+Project+Template)

[Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world) | +| Have you got a consistent project structure set up? | Does the project structure align with other analytical projects in your organisation?

Is there a clear system to track version control within your project? | All projects should have appropriate version control including a clear system for version labelling and a version control log.

It is recommended to use a project template such as [govcookiecutter](https://github.com/best-practice-and-impact/govcookiecutter). A template provides a consistent project structure and some built-in automated QA. | [govcookiecutter (GitHub link)](https://github.com/best-practice-and-impact/govcookiecutter)

[Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world) | | Is there a recorded plan for the analysis? | Has an appropriate plan been agreed with adequate consideration of time, resource quantity and skills required?

Will the proposed time frames allow for adequate quality assurance?

Where are the highest risk points for errors in the process? What measures do you or could you take to mitigate risk at these points? | Analysis will often involve a trade off between time, resource and quality.

Discussions about the desired and achievable levels of QA should take place at the very start of a project. The reality is that most analysis will be carried out under time and resource pressure and we will not be able to carry out all the QA activity we would ideally like to. In these circumstances, QA activities will need to be prioritised based on the risk of not carrying them out. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[The Government Data Quality Framework (Government Data Quality Hub)](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)

[Guidance: Tips for urgent quality assurance of data (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/tips-for-urgent-quality-assurance-of-data) | | Can you summarise and explain the end-to-end process of your analysis? | How does your work feed in to the bigger analytical picture? | Having an overview of the analysis (especially if you only work on part of it) ensures that you and your team understand how your work feeds into the wider product.

It can help you to identify potential quality risks or issues, both upstream and downstream of your own work as well as how your activity supports and underpins downstream processing. | [Generic Statistical Business Process Model (GSBPM; United Nations Economic Commision for Europe)](https://unece.org/statistics/modernstats/gsbpm) @@ -284,7 +284,7 @@ The table below gives some considerations when completing this section of the re | Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |-----------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Is the input data quality well-understood and sufficient for project needs? | Are the source, collection methods, size, accuracy, and precision of the data known? Have you checked for missing and duplicated data? Have you addressed these? Have the data been checked for any extreme or implausible values? Do any data quality concerns pose a risk for the final outcomes of the analysis? | The quality of input data directly affects the validity of any results or conclusions drawn from it. Understanding the data quality helps you assess the reliability and trustworthiness of findings. You should assess your datasets for missing values to understand which variables and records are affected, how much is missing and whether there might be bias in the missingness. Always tell your users about the prevalence and treatment of missing data. | [Guidance: Missing data (internal link)](missing-data) [Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/) | +| Is the input data quality well-understood and sufficient for project needs? | Are the source, collection methods, size, accuracy, and precision of the data known? Have you checked for missing and duplicated data? Have you addressed these? Have the data been checked for any extreme or implausible values? Do any data quality concerns pose a risk for the final outcomes of the analysis? | The quality of input data directly affects the validity of any results or conclusions drawn from it. Understanding the data quality helps you assess the reliability and trustworthiness of findings. You should assess your datasets for missing values to understand which variables and records are affected, how much is missing and whether there might be bias in the missingness. Always tell your users about the prevalence and treatment of missing data. | [Guidance: Missing data (internal link)](missing-data)

[Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/) | | How do you know the method you are using is appropriate? | Are the methods agreed to be the most suited for the project needs compared to other potential alternatives? Does the methodology deliver the required degree of certainty and precision? When changes have been made to methodologies, are they justified, well-documented, and clearly communicated? | You should be able to explain why the method(s) you use are suitable for this analysis and be able to support your choice with evidence. This might include reference to academic peer-reviewed publications or other projects that are similar. If you cannot explain why you chose the methods you use and why they are right for your analysis and the data you are using, you cannot be sure that your approach is sound. | | | Is the code or software functioning as expected? | Have an appropriate range of automated checks been built into the analytical pipeline? Has the code or other implementation of the analysis been checked by an independent analyst? Have intermediate outputs from every stage of analysis been checked? | You need to be sure that your analysis produces the outputs that you think it should and that the processes you run work as expected. If you cannot demonstrate that code and processes you have set up are functioning correctly, you cannot confirm the quality of the results. | [Guidance: RAP Framework - Bronze, Silver and Gold standards (internal link)](rap-framework) | | Do the outputs appear realistic? | Are there any anomalies or unexpected trends in the data? Have they been sufficiently investigated? Does the output appear realistic for the real-world behaviour that is being analysed? Are outputs consistent with results from historical data or other data from other regions? Have the methodology and outputs been peer reviewed to ensure they appear reasonable? | Stakeholders rely on analytical outputs to make informed decisions. You need to ensure that these decisions are based on accurate and plausible data, and so if the data looks different to previous trends this should be investigated and communicated clearly to users. | | @@ -302,10 +302,10 @@ The table below gives some considerations when completing this section of the re | Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Has the code been properly documented? | Has version control software such as Git and GitHub been used to track and manage changes? Is code documentation embedded in the project rather than saved elsewhere? Have functions been documented and wrapped up in a package where useful to do so? | Your code must be well documented so that somebody new can understand it and pick it up. Poor documentation means that other people will not understand why the process is configured as it is, how the process works or how to run the process safely, potentially leading to errors. Good version control ensures that you have a full understanding of when, why and how changes were made to your analysis process. If it is hard to track changes, this makes it hard to retrace steps if there is a problem and means you do not fully understand the process. | [Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world) [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | -| Is the process from initial data extraction to final report fully reproducible? | Without any input from the analytical project team, could a third party reproduce our results? Have manual steps in data extraction, analysis, and report creation been automated where possible? If mathematical formulas have been used to describe methodology, have these been presented with all terms and steps defined? Where manual steps were involved, have these been thoroughly documented? Has a standard operating procedure (SOP) been produced if necessary? Have the documentation and quality assurance logs been peer reviewed to ensure that the process is reproducible? If we repeat our processes with different software, will we get the same results? | If the process from initial extraction to the final report contains manual steps, then these steps need to be documented so that the team can understand the process. Manual steps can lead to a higher risk of human error, so they need to be clearly understood. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf) [Guidance: Reproducible Analytical Pipelines (RAP): an introduction (internal link)](rap) | -| Have best practices for coding been followed? | Has an agreed-upon style guide been followed in all of the code written for analysis? Does code adhere to best practice standards? Are there plans in place to ensure there is sufficient capability within the team to use and modify the pipeline as needed in the future? | If best practices have not been followed, there may be a higher risk of error. By understanding the team's agree-upon styles and standards, you can ensure consistency when modifying the pipeline. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf) [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | -| Do you consistently use peer review to check scripts and code, documentation, implementation of methods, processes and outputs? | Have the methodology and outputs been peer reviewed to ensure they appear reasonable? Is your code or analysis ever peer reviewed by someone outside your team? | Peer review is a standard part of analysis best practice. It is helpful because it helps to identify where steps are unclear, documents are hard to understand or there might be problems with calculations or implementation of methods. Routine peer review helps to improve the quality of processes and to reduce risk by identifying potential problems. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) [Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/#quality-assurance) | +| Has the code been properly documented? | Has version control software such as Git and GitHub been used to track and manage changes? Is code documentation embedded in the project rather than saved elsewhere? Have functions been documented and wrapped up in a package where useful to do so? | Your code must be well documented so that somebody new can understand it and pick it up. Poor documentation means that other people will not understand why the process is configured as it is, how the process works or how to run the process safely, potentially leading to errors. Good version control ensures that you have a full understanding of when, why and how changes were made to your analysis process. If it is hard to track changes, this makes it hard to retrace steps if there is a problem and means you do not fully understand the process. | [Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world)

[Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | +| Is the process from initial data extraction to final report fully reproducible? | Without any input from the analytical project team, could a third party reproduce our results? Have manual steps in data extraction, analysis, and report creation been automated where possible? If mathematical formulas have been used to describe methodology, have these been presented with all terms and steps defined? Where manual steps were involved, have these been thoroughly documented? Has a standard operating procedure (SOP) been produced if necessary? Have the documentation and quality assurance logs been peer reviewed to ensure that the process is reproducible? If we repeat our processes with different software, will we get the same results? | If the process from initial extraction to the final report contains manual steps, then these steps need to be documented so that the team can understand the process. Manual steps can lead to a higher risk of human error, so they need to be clearly understood. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html)

[Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Reproducible Analytical Pipelines (RAP): an introduction (internal link)](rap) | +| Have best practices for coding been followed? | Has an agreed-upon style guide been followed in all of the code written for analysis? Does code adhere to best practice standards? Are there plans in place to ensure there is sufficient capability within the team to use and modify the pipeline as needed in the future? | If best practices have not been followed, there may be a higher risk of error. By understanding the team's agree-upon styles and standards, you can ensure consistency when modifying the pipeline. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html)

[Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | +| Do you consistently use peer review to check scripts and code, documentation, implementation of methods, processes and outputs? | Have the methodology and outputs been peer reviewed to ensure they appear reasonable? Is your code or analysis ever peer reviewed by someone outside your team? | Peer review is a standard part of analysis best practice. It is helpful because it helps to identify where steps are unclear, documents are hard to understand or there might be problems with calculations or implementation of methods. Routine peer review helps to improve the quality of processes and to reduce risk by identifying potential problems. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html)

[Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/#quality-assurance) | {% endcapture %} @@ -397,4 +397,4 @@ Below is a summary of five key documents as they relate to QA. This is not a com 1. [Government Analysis Function: Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) 1. [Office for Statistics Regulation: Regulatory guidance - thinking about quality when producing statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-thinking-about-quality-when-producing-statistics/pages/2/) 1. [The Aqua Book: guidance on producing quality analysis for government](https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government) -1. [UK Statistics Authority: Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/the-code/) \ No newline at end of file +1. [UK Statistics Authority: Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/the-code/)