docs: update Campaign management documentation

README.md

@@ -1,122 +1,9 @@
-# Focus Group on AI for Health (FG-AI4H) - Annotation Platform
-The Focus Group on Artificial Intelligence for Health (FG-AI4H) was established under ITU-T
-SG16 and operated as a partnership of the International Telecommunication Union (ITU) and the
-World Health Organization (WHO) to develop a standardized assessment framework for the
-evaluation of AI-based methods for health, diagnosis, triage or treatment decisions. Participation in
-the FG-AI4H was free and open to all.
+# Data Annotation Package
 
-As a consequence, ITU, WHO and WIPO agreed to launch a successor collaboration platform in the
-form of a global initiative that would enable, facilitate use and promote implementation of AI for
-health, expected to start operations in 2nd quarter of 2024. The Global Initiative on AI for Health
-(GI-AI4H) is expected to grandfather the work developed by the FG-AI4H and further leverage and
-expand the community it developed.
+Data annotation is one of the most dependable factors on model performance, it serves as an important aspect of data quality control on Artificial Intelligence for health. This document is committed to give a general guideline of data annotation specification, including definition, background and goals, framework, standard operating procedure, scenario classifications and corresponding consistency criteria, as well as recommended metadata, etc.
 
-## Introduction
-The Open Code Initiative (OCI) of the FG-AI4H developed a running platform with the proof-of-
-concept for the assessment and benchmarking of AI for health solutions using the concepts
-identified in the various FG-AI4H Deliverables. The OCI developed the digital building blocks of
-the FG-AI4H assessment platform, which can support the end-to-end development and assessment
-of AI for health algorithms under consideration of regulatory guidelines and the needs of all AI for
-health stakeholders.
+This annotation module is built according to [FG-AI4H-DEL5.3](https://www.itu.int/en/ITU-T/focusgroups/ai4h/Pages/deliverables.aspx) (Proposed Standard for Data Annotation in Health) which specifies general requirements and guidelines for data annotation to ensure consistency and quality of the annotations. It applies to various types of data including, but not limited to, images, text, audio, and video. The standard covers both manual and automated methods of annotation and applies to all stages of the annotation process, including design, creation, quality assurance, and maintenance. The standard does not address specific annotation tasks or health domains.
 
-The platform is intended to help in developing and assessing AI4H products, and to provide
-guidance to implementers developing their own applications and it can also be used by other
-stakeholders, such as regulatory bodies and medical professionals. OCI's open code nature aims to
-allow reuse in a variety of contexts, as well as for transparency and easier verification.
-The OCI contains five main working packages:
-- Core package: Provisions the common services to all packages, e.g., storage, authentication
-and authorization to access resources.
-- Data acquisition and storage package: Provides safe and secure storage of medical data;
-serves as an interface for other packages to access this data using the FHIR standard; and
-offers data governance that complies with data protection laws. Facilitates data compilation
-for many modalities; registers data and metadata; offers a federated data catalogue and data
-governance; ingests data; and manages patient consent information.
-- Data annotation package: Provides an annotation campaign management API and annotation
-interfaces for many modalities; includes collaboration features; develops a network of
-annotation experts; and creates notifications of pending annotation tasks. OCI partners
-provide annotation tools integrated with the FG-AI4H platform, e.g., AI-based pre-annotation
-platform or 3D image annotation user interface.
-- Training package: Provides a Privacy Preserving Federated Learning solution enabling
-training on datasets that need to stay at their location. This solution guarantees secure sharing
-of models and model aggregation.
-- Evaluation (audit) package: Offers testing measures and methods for different quality
-dimensions including interpretation, bias, uncertainty, and robustness; questionnaires provide
-qualitative evaluation addressing ethical and governance considerations.
-- Reporting package: A customizable reporting interface that presents the results of the
-Evaluation package.
-
-Over 40 developers, regulators, and medical professionals from five continents were involved in the
-OCI development during the FG-AI4H lifetime.
-
-To timely deliver resilient software capability the OCI is implementing strategies that focus on
-cybersecurity and survivability across the development process. The OCI adopted the DevSecOps
-approach, which is a widely-adopted industry best practice that enables the delivery of resilient
-software in a short development cycle. The DevSecOps software lifecycle approach creates cross-
-functional teams that unify historically disparate evolutions - development (Dev), cybersecurity
-(Sec), and operations (Ops). As a unified team, they follow agile design principles and embrace a
-culture that recognizes resilient software is only possible at the intersection of quality, stability, and
-security.
-
-The platform prototype was tested in different proof-of concepts and a whitepaper on the
-assessment platform has been developed. The integration process of Topic Group TG-Symptoms,
-TG-Fall, and TG-Snakes with the OCI platform yielded significant enhancements to the Data
-Acquisition and Data Annotation packages. Notably, their inputs were instrumental in refining these
-packages. The Validation of the Assessment Platform and its associated processes was conducted
-with active participation from several TGs (TG- Dermatology, TG- Ophthalmology, TG- Malaria,
-TG- Symptoms, TG- Psychology, TG- Snakes, TG-Radiology, TG-Neuro, TG-Outbreaks) during
-the trial audits, which main objectives were:
-- To facilitate the conduct of trial audit series and to provide a platform for the submission,
-discussion and publication of FG-AI4H audit methods and reports under the special collection
-titled “Machine Learning for Health: Algorithm Auditing & Quality Control” in the Journal of
-Medical Systems (JOMS)
-- To provide basic training to FG-AI4H Topic Groups on how to verify and validate the
-technical, clinical and regulatory requirements of their ML4H tool by following the audit
-workflow over the AI4H Assessment Platform.
-- 
-The collaboration between the OCI platform and the Topic Groups (TGs) has been mutually
-beneficial, extending beyond platform enhancement to include improvements in TGs' Topic
-Description Documents (TDDs). Through the process of running trial audits and engaging in
-detailed discussions, the OCI experience has enabled TGs to refine their understanding of their
-data's nature and characteristics. This, in turn, has empowered them to describe their datasets in a
-standardized and comprehensive manner. By gaining clarity on data attributes and standardizing
-their documentation, TGs have enhanced the quality and utility of their TDDs. This collaborative
-exchange underscores the synergistic relationship between the OCI platform and the TGs, driving
-continuous improvement and innovation across the FG-AI4H ecosystem.
-In parallel, the OCI team collaborated with various FG-AI4H Working Groups, especially WG-
-DAISAM and WG-Clinal Evaluation, to implement a range of features aimed at enhancing the
-platform's functionality. These features include:
-- Data annotation campaign management: This feature facilitates the streamlined management
-of data annotation campaigns, allowing for efficient organization and tracking of tasks within
-the platform.
-- Health Metadata management: Enabling comprehensive management of health metadata
-ensures that relevant information is effectively captured and utilized across various aspects of
-the platform, enhancing data accuracy and usability.
-- FHIR standard implementation for Data acquisition: Integration of the HL7 Fast Healthcare
-Interoperability Resources (FHIR) standard for data acquisition ensures compatibility with
-existing healthcare systems, promoting seamless data exchange and interoperability.
-- Providing an integration layer for data annotation UI tools: This feature establishes a flexible
-integration layer, allowing for the seamless incorporation of various data annotation user
-interface (UI) tools into the platform, catering to diverse user preferences and requirements.
-For example, integration with Visian, a student project from HPI Potsdam, offers a flexible
-human-in-the-loop AI solution tailored for effortless utilization of ML models on medical
-image data. It also facilitates comfortable project management for data and model versioning,
-enhancing overall workflow efficiency and collaboration.
-- Collaboration with industry on privacy-preserving encryption: The OCI team has partnered
-with leading industry players such as Inpher.io to implement state-of-the-art privacy-
-preserving encryption techniques. By leveraging homomorphic encryption, the platform
-ensures robust protection of sensitive data through advanced cryptographic methods, fostering
-trust and compliance with privacy regulations.
-
-Furthermore, as part of the development process, comprehensive documentation was created and
-hosted on GitHub (https://github.com/fg-ai4h). This documentation serves as a valuable resource
-for users, providing guidance on platform usage, feature implementation, and troubleshooting.
-Feedback from stakeholders was actively solicited and utilized to refine several FG-AI4H
-deliverables, ensuring alignment with user needs and preferences.
-
-Particularly interesting from an ITU perspective is that the OCI of ITU/WHO FG-AI4H offers a
-large potential for the development of technical standards in the emerging field of AI that are
-directly informed by and tested using actual software code. One good example is the
-standardization of data annotation processes.
 
 # Contribution & Forking
 

documentation/Campaign-management.md

@@ -0,0 +1,29 @@
+# Campaign Management
+
+The `Campaign Management` module is located in the `domain/campaign` package. This module is responsible for creating, managing, and monitoring campaigns.
+
+## Overview
+
+Campaigns define a set of annotators, reviewers and supervisors acting on annotation tasks. Datasets are linked to the campaign and used to define the annotation tasks. The `Campaign Management` module provides the necessary functionalities to manage these campaigns effectively.
+
+## Features
+
+### Campaign Creation
+
+Campaigns can be created by the Manager. 
+
+### Campaign Management
+
+Campaigns can be managed by the Manager. This includes starting and closing campaigns, assigning tasks to annotators, and monitoring the progress of the campaign.
+
+### Campaign Monitoring
+
+The progress of the campaigns can be monitored. This includes tracking the completion of tasks, the quality of annotations, and the overall progress of the campaign.
+
+## Classes
+
+### Campaign
+
+The `Campaign` class represents a campaign. It contains information about the campaign, such as the annotators assigned to the campaign, and the status of the campaign.
+
+For technical details, see the [Campaign class](../src/main/java/org/fgai4h/ap/domain/campaign/entity/CampaignEntity.java)

documentation/Contributing.md

@@ -52,6 +52,21 @@ git clone https://github.com/<your-username>/open-code-initiative.git
 
 1. **Create a Branch**: Always create a new branch for your changes. This keeps the project history clean and your changes isolated.
 
+**Branching strategy**
+
+In order to standardize the branching strategy and leverage automation, we decided to implement Trunk based development on all projects.
+
+**Branch naming convention**
+
+We have defined a naming convention for tagging branches and artifact versioning, consistent branch naming promotes clarity and organization. OWT follows these naming conventions:
+
+- Main branch: master
+- Feature branches: feature/branch-name
+- Development branch (default branch for Git-flow): develop
+- Release branches: release/branch-name
+- Bug fix branches: bugfix/branch-name
+- Hotfix branches: hotfix/branch-name
+
 ```bash
 git checkout -b <branch-name>
 ```

documentation/Data-annotation.md

@@ -17,6 +17,13 @@ The scope of this annotation tool is to provide a platform for annotators to ann
 - Continuous learning
 - Panel decision
 
+# Annotation UI
+Annotation UIs are not developed by the FG-AI4H OCI team itself, but are provided by third-party vendors. The FG-AI4H OCI team is responsible for integrating the annotation UIs into the platform and ensuring that they meet the requirements of the FG-AI4H-DEL5.3 standard.
+
+The Annotation Platform is providing an integration layer for data annotation UI tools: This feature establishes a flexible integration layer, allowing for the seamless incorporation of various data annotation user interface (UI) tools into the platform, catering to diverse user preferences and requirements.
+
+For example, integration with Visian, a student project from HPI Potsdam, offers a flexible human-in-the-loop AI solution tailored for effortless utilization of ML models on medical image data. It also facilitates comfortable project management for data and model versioning, enhancing overall workflow efficiency and collaboration.
+
 # Definitions
 - **Data Annotation**: The process of labeling data points with metadata to provide context and meaning to the data.
 - **Data Point**: A single unit of data that is annotated.
@@ -42,3 +49,5 @@ The scope of this annotation tool is to provide a platform for annotators to ann
 - **HL7**: Health Level Seven International, a standard for exchanging healthcare information electronically.
 
 Additional definitions about entities can be found in the [Entities](Entities.md).
+
+

pom.xml

@@ -150,6 +150,12 @@
             <artifactId>junit</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.jetbrains</groupId>
+            <artifactId>annotations</artifactId>
+            <version>17.0.0</version>
+            <scope>compile</scope>
+        </dependency>
 
     </dependencies>
 

src/main/java/org/fgai4h/ap/domain/campaign/entity/CampaignEntity.java

@@ -13,66 +13,126 @@
 import java.util.List;
 import java.util.UUID;
 
+/**
+ * Represents a campaign in the system.
+ * A campaign is a set of tasks that are to be annotated by a group of annotators.
+ */
 @Data
 @Builder
 @AllArgsConstructor
 @NoArgsConstructor
 @Entity
-@Table(name="campaign")
-public class CampaignEntity implements Serializable
-{
+@Table(name = "campaign")
+public class CampaignEntity implements Serializable {
     private static final long serialVersionUID = 1L;
 
+    /**
+     * The unique identifier for the campaign.
+     */
     @Id
     @GeneratedValue(generator = "uuid2")
     @GenericGenerator(name = "uuid2", strategy = "uuid2")
     @Column(columnDefinition = "BINARY(16)")
     private UUID campaignUUID;
 
+    /**
+     * The name for the campaign.
+     */
     private String name;
+    /**
+     * The description for the campaign.
+     */
     private String description;
+    /**
+     * The status of the campaign.
+     */
     private String status;
+    /**
+     * The kind of annotation that is to be performed in the campaign.
+     */
     private String annotationKind;
+    /**
+     * The tool that is to be used for annotation.
+     */
     private String annotationTool;
+    /**
+     * The model that is to be used for annotation.
+     */
     private String preAnnotationTool;
+    /**
+     * The model that is to be used for annotation.
+     */
     private String preAnnotationModel;
+    /**
+     * The method that is to be used for annotation.
+     */
     private String annotationMethod;
+    /**
+     * The instructions for annotation.
+     */
     private String annotationInstructions;
+    /**
+     * The quality assurance for annotation.
+     */
     private String qualityAssurance;
+    /**
+     * The kind of quality assurance that is to be performed in the campaign.
+     */
     private Boolean isInstanceLabel;
+    /**
+     * The kind of quality assurance that is to be performed in the campaign.
+     */
     private Integer minAnnotation;
 
+    /**
+     * The class labels for the campaign.
+     */
     @OneToMany(mappedBy = "campaignEntity")
     private List<ClassLabelEntity> classLabels = new java.util.ArrayList<>();
 
-    @ManyToMany(cascade=CascadeType.MERGE)
+    /**
+     * The annotators for the campaign.
+     */
+    @ManyToMany(cascade = CascadeType.MERGE)
     @JoinTable(
             name = "campaign_annotator",
             joinColumns = @JoinColumn(name = "campaignUUID"),
             inverseJoinColumns = @JoinColumn(name = "userUUID"))
     private List<UserEntity> annotators;
 
-    @ManyToMany(cascade=CascadeType.MERGE)
+    /**
+     * The reviewers for the campaign.
+     */
+    @ManyToMany(cascade = CascadeType.MERGE)
     @JoinTable(
             name = "campaign_reviewer",
             joinColumns = @JoinColumn(name = "campaignUUID"),
             inverseJoinColumns = @JoinColumn(name = "userUUID"))
     private List<UserEntity> reviewers;
 
-    @ManyToMany(cascade=CascadeType.MERGE)
+    /**
+     * The supervisors for the campaign.
+     */
+    @ManyToMany(cascade = CascadeType.MERGE)
     @JoinTable(
             name = "campaign_supervisor",
             joinColumns = @JoinColumn(name = "campaignUUID"),
             inverseJoinColumns = @JoinColumn(name = "userUUID"))
     private List<UserEntity> supervisors;
 
-    @ManyToMany(cascade=CascadeType.MERGE)
+    /**
+     * The datasets for the campaign.
+     */
+    @ManyToMany(cascade = CascadeType.MERGE)
     @JoinTable(
             name = "campaign_dataset",
             joinColumns = @JoinColumn(name = "campaignUUID"),
             inverseJoinColumns = @JoinColumn(name = "datasetUUID"))
     private List<DatasetEntity> datasets;
 
+    /**
+     * The constructor for a campaign.
+     */
     public CampaignEntity(UUID uuid, String name, String description) {
         this.campaignUUID = uuid;
         this.name = name;