Add validation and transformation ADRs (#1142)

* wip: add adrs for rule engines

* Updated template with Impact section

- added Impact section for revision improvement

* Updated template format for clarity

* Modified ADR #20

- modified wording and format
- added additional context for Impact

* Modified ADR #21

- reworded and modified format to include Impact

* Modified ADR #22

- Updated format
- Added content

* Updated validation engine ADR

* Updated transformation engine ADR

* Fixed a few typos

* Added another plus

* Fixed formatting and added impact considerations for ADR 001

---------

Co-authored-by: tjohnson7021 <tjohnson@flexion.us>
Co-authored-by: Tiffini Johnson <86614374+tjohnson7021@users.noreply.github.com>

adr/001-architecture-decision-records.md

@@ -12,16 +12,35 @@ Accepted.
 
 ## Context
 
-ADRs are originally described by
-[Michael Nygard in an article](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions)
-and more robustly detailed and demonstrated [on GitHub](https://adr.github.io).
+ADRs are originally described by [Michael Nygard in an article](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) and more robustly detailed and demonstrated [on GitHub](https://adr.github.io).
 
 We want to record our architectural decisions so that...
 
 - New team members who join can see why we made the decisions we made.
 - The team can revise or revisit decisions with more confidence and context.
 
-### Related Issues
+## Impact
+
+_The outcomes of the decision, both positive and negative. This section explains the impact of the decision, such as trade-offs, risks, and what needs to be done to implement it._
+
+### Positive
+
+- **Transparency**: ADRs make decision-making more transparent, helping current and future team members understand the rationale behind decisions.
+- **Historical Context**: They provide valuable historical context, aiding in future decision-making and avoiding repeated mistakes.
+- **Onboarding**: ADRs speed up the onboarding process by quickly familiarizing new team members with architectural decisions.
+- **Consistency**: A standardized format ensures consistent documentation, making records easier to maintain and reference.
+
+### Negative
+
+- **Overhead**: Maintaining ADRs requires time and effort.
+- **Outdated Records**: If not regularly updated, ADRs can become outdated and misleading.
+
+### Risks
+
+- **Incomplete Documentation**: Not all decisions may be documented, leading to gaps in the record.
+- **Misalignment**: ADRs may not always match the actual implementation, causing confusion.
+
+## Related Issues
 
 - #1
 - #13

adr/020-azure-alerts.md

@@ -1,10 +1,10 @@
-# 1. Architecture Decision Records
+# 20. Azure Alerts
 
 Date: 2024-05-17
 
 ## Decision
 
-We will use Azure alerts to notify our team of issues with our applications.
+We will implement Azure Alerts to notify our team of application issues.
 
 ## Status
 
@@ -14,12 +14,33 @@ Accepted.
 
 As part of our CI/CD infrastructure, we need notifications when failures occur.
 
-Current alert is configure to be:
-- [Azure Log Search Alerts](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-types#log-alerts) for HikariCP Connection failures
-  - Trigger on any logged failures with database connections as Azure metrics weren't capturing the issues found in our logs.
-  - Stateful (Auto-mitigation), to keep alerts in a `fired` status until resolved and for less noise on frequent/duplicate alerts.
-  - Configured to a Slack channel email until our Pagerduty is set up.
+To ensure rapid response to application failures within our CI/CD infrastructure, we require real-time notifications for critical issues. The current alert setup focuses on:
 
-### Related Issues
+- **Type:** [Azure Log Search Alerts](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-types#log-alerts) for HikariCP connection failures.
+
+
+- **Trigger:** Any logged failures with database connections.
+
+
+- **Configuration:** Alerts are stateful (auto-mitigation); set to `fired` status to reduce noise from frequent or duplicate alerts.
+
+
+- **Notification:** Alerts sent to a Slack channel via email until PagerDuty is operational.
+
+## Impact
+
+### Positive
+
+- Immediate awareness of critical issues, reducing downtime
+
+### Negative
+
+- Possible alert fatigue if not fine-tuned
+
+### Risks
+
+- None
+
+## Related Issues
 
 - #1001

adr/021-validation-engine.md

@@ -0,0 +1,38 @@
+# 21. Validation Engine
+
+Date: 2024-06-11
+
+## Decision
+
+We will implement a Validation Engine using the [Rules Engine Pattern](https://deviq.com/design-patterns/rules-engine-pattern) to validate incoming FHIR messages, based on given conditions. Each validation will be represented as a rule, specifying conditions that must be met, and the corresponding failure message if validation fails.
+
+Initially, rules will be stored in a JSON resource file, but the plan is to migrate them to a database or or other type of external storage in the future, which would allow to update the rules without the need for deploying the application.
+
+## Status
+
+Accepted.
+
+## Context
+
+The intermediary needs to validate FHIR messages based on SME research and partner-specific requirements, which are subject to change. The Rules Engine Pattern enables flexible, scalable, and maintainable validation management.
+
+## Impact
+
+### Positive
+
+- Easy to update and scale validations. Easy to refine scope for validations (general or partner-specific)
+- Validation rules could be reused by multiple partners.
+- It should make it easier to add a UI in the future, which could potentially allow partners to self-serve and add their own validations.
+- The framework can be leveraged to implement transformations as well.
+- Separation of concerns and code reusability.
+- It's modular enough that in the future we could decide to spin-off the Validation Engine into its own microservice that could be used to validate any FHIR (and potentially HL7) message.
+
+### Negative
+
+- There's added overhead code for the engine, but this code should seldom be required to maintain compared to the actual rules which should be a lot easier to maintain.
+- We'll need to deploy the application to have any changes to the validations available in production, at least until we migrate to a database or external storage.
+
+### Risks
+
+- The engine will need to iterate over all rules to decide which ones to apply, which could impact performance if the rules grow substantially.
+- If the conditions are not well defined there could be potential leakage (validations misapplied), but this should be identified in staging while onboarding.

adr/022-transformation-engine.md

@@ -0,0 +1,41 @@
+# 22. Transformation Engine
+
+Date: 2024-06-11
+
+## Decision
+
+As with the [Validation Rule Engine](021-validation-engine.md), we will use a variation of the [Rules Engine Pattern](https://deviq.com/design-patterns/rules-engine-pattern) to design and implement a transformation rule engine that can transform incoming FHIR messages, given a condition. Each transformation will be represented as a rule, specifying conditions that must be met, with a name and arguments. The name of the rule will match a Java function in [/etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/ruleengine/transformation/custom/](/etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/ruleengine/transformation/custom/) that will apply the transformation, using the given arguments.
+
+Failed transformations will not interrupt the flow of the message. Any transformations that have errors will be skipped instead, making sure to log the error.
+
+Initially, rules will be stored in a JSON resource file, but the plan is to migrate them to a database or external storage in the future, which would allow to update the rules without the need for deploying the application.
+
+## Status
+
+Accepted.
+
+## Context
+
+The intermediary needs to transform incoming FHIR messages based on partner-specific requirements, which are subject to change. The Rules Engine Pattern enables flexible, scalable, and maintainable management for transformations.
+
+## Impact
+
+### Positive
+
+- Easy to update and scale transformations. Easy to refine scope for transformations (general or partner-specific)
+- Transformation rules could be reused by multiple partners.
+- It should make it easier to add a UI in the future, which could potentially allow partners to self-serve and add their own transformations.
+- Leverages on existing code from the Validation Rules Engine.
+- Separation of concerns and code reusability.
+- It's modular enough that in the future we could decide to spin-off the Transformation Engine into its own microservice that could be used to transform any FHIR (and potentially HL7) message.
+
+### Negative
+
+- There could potentially be conflicts between rules. Order of execution matters.
+- There's added overhead code for the engine, but this code should seldom be required to maintain compared to the actual rules which should be a lot easier to maintain.
+- We'll need to deploy the application to have any changes to the transformations available in production, at least until we migrate to a database or external storage.
+
+### Risks
+
+- The engine will need to iterate over all rules to decide which ones to apply, which could impact performance if the rules grow substantially.
+- If the conditions are not well defined there could be potential leakage (transformations misapplied), but this should be identified in staging while onboarding.