Add External Validation Rules for Analysis Properties Based on Dynamic Schema

[Azher2Ali]

This PR introduces external validation capabilities for analysis properties, allowing for dynamic schemas to specify validation rules that are stored in the database. Each rule specifies a jsonPath to identify the property for validation and a URL to an external service for validation logic. During analysis submission, the system will validate specific property values by querying the external service, confirming that the values (e.g., donor IDs) meet the criteria defined by the external registry or ontology.

• Adds support for configurable external validation rules per analysis type.

• Enables database storage of validation rules tied to specific analysis types.

• Implements web request calls to validate property values according to the rules during submission.

This update also introduces external validation checks for analyses during creation and update in SONG. These checks are applied after JSON schema validation and file type validation. For each configured external validation rule on an analysis type, the system will:

Check Analysis Properties: Determine if the property at the specified path has a value. If no value exists, the validation is considered passed.

Validate Using External Source: If a value is present, SONG constructs a GET request to the external validation URL. The URL includes query parameters:

• studyId: the study identifier the analysis is associated with.

• value: the value of the property being validated.

Process the External Validation Response:

• If the value is valid, the external service returns an HTTP 200 status with no response body.

• If invalid, the external service returns an HTTP 400 status and optionally includes a JSON error message explaining why the value is invalid. This message is provided to the submitter for clarity.

• Other HTTP errors indicate a validation failure, and a message is returned to the submitter stating that external validation was unsuccessful, preventing the analysis submission/update.

This update sets the groundwork for calling external validation services, but the actual call to the external service is pending final implementation .We added this feature with an exception that the actual call to the external service is skipped as the service is not up and running and if we start invoking it, this will fail all the validations #872 #875

[leoraba]

So far so good! Just left some comments to consider 📝

[joneubank]

👍

[joneubank]

Remove commented code.

[joneubank]

What about external validation URLs when they are validating against a service that isn't Lyric? We need to define a schema for writing a URL that isn't specific to Lyric. This is about defining string template rules with variables we will replace with the studyId and entity name.

Perhaps ${studyId}, ${entityName}, and ${entityId}. So an external URL value could be https://mydomain.example.org/whatever/path/to/validate/${studyId}/${entityName}/${entityId}.

[Azher2Ali]

The code will take any URL apart from lyric, but for the path and path params should be well defined in order to have a common logic

[joneubank]

Have we published this version with file types yet? We would need to migrate all the file_types that have already been set into the new analysis schema format.

Did you consider storign analysis_schema in a separate column instead of storing options as a JSON?