Proposal: Enable Federated Analytics using OpenFL

[tanwarsh]

Summary

Federated Analytics is a data analysis approach where a querier answers a query through collaboration with multiple data owners (clients) who retain their local raw data. Instead of exchanging raw data, intermediate query responses are transferred and aggregated by the querier to answer the query. This documentation provides a high-level design for implementing Federated Analytics using OpenFL

Motivation

FA is a distributed approach to data analysis that enables multiple parties to collaboratively analyze data without sharing the actual data. This approach is particularly valuable in scenarios where data privacy and security are critical, such as in healthcare, finance, and other sensitive fields. Key points about FA include:

• Data Privacy: FA ensures that data remains with the data owner and is not shared with other parties. Only analysis results or model updates are shared.
• Collaboration: Multiple parties can collaborate on data analysis without exposing their data to each other, enabling joint analysis on a larger dataset while maintaining data privacy.
• Security: By keeping data local and only sharing necessary information, FA reduces the risk of data breaches and unauthorized access.
• Efficiency: FA leverages the computational resources of multiple parties, reducing the need for centralized data processing and potentially increasing efficiency.
• Applications: FA is used in various fields, including healthcare for multi-institutional collaborations, finance for fraud detection, and other areas where data privacy is critical.

High-Level Design

Scope

• Showcase Federated Analytics with OpenFL.
• Integrate with existing federated learning frameworks to leverage distributed data for analytics.
• Implement methods for the Aggregator to aggregate query results and for Collaborators to execute queries on their local datasets and return the results to the Aggregator.
• Set up a Taskrunner API workspace to showcase FA.

Technical Details

TaskRunner API Workspace 

1.  runner_fa.py, loader_fa.py, and task_fa.yaml will support all FA workspaces. 
   • runner_FA.py: Task runner class responsible for managing tasks for collaborators and query and aggregation function itself will be added by users themself. This will facilitate adding support for new queries.
   • loader_FA.py: For federated data loader to handle data for collaborator.
   • tasks_FA.yaml: would only realistically need the function setting to be updated to a standardised name (i.e. query_task  for collaborator and aggregate_query_result_task for aggregator).
2. Task Group : analysis will be used for FA.
3. Aggregation type (AggregationFunction) will be used by aggregator to aggregate query results.
4. taskrunner.py: It will be responsible for different types of query for FA.
5. Similar to how collaborators share model weights, with FA collaborators will share query results which will be aggregated using the custom aggregation function written by users. eg. LocalTensor(col_name='collaborator1_sepal length (cm)', tensor=array([2.2 , 2.42, 2.64, 2.86, 3.08, 3.3 , 3.52, 3.74, 3.96, 4.18, 4.4 ], dtype=float32))
6. Sequence diagram
7. Sample Plan.

aggregator :
  defaults : plan/defaults/aggregator.yaml
  template : openfl.component.Aggregator
  settings :
    save_path : save/result.csv
 
collaborator :
  defaults : plan/defaults/collaborator.yaml
  template : openfl.component.Collaborator
 
data_loader :
  defaults : plan/defaults/data_loader.yaml
  template : src.dataloader.IRISInMemory
  settings :
    collaborator_count : 2
    data_group_name    : iris
    batch_size         : 150
 
task_runner :
  defaults : plan/defaults/task_runner.yaml
  template : src.taskrunner.IrisHistogram
 
network :
  defaults : plan/defaults/network.yaml
 
assigner :
  template : openfl.component.RandomGroupedAssigner
  settings :
    selected_task_group : analysis
    task_groups  :
      - name       : analysis
        percentage : 0
        tasks      :
          - query
 
tasks :
  query:
    function : query
    aggregation_type: src.taskrunner.histogram
    kwargs   :
      columns: ['sepal length (cm)', 'sepal width (cm)']

Dataset
Calculate histogram (an estimate of the probability distribution of a continuous variable) on specific columns of Iris dataset.

Types of Supported Queries
1.Statistical queries
2.Set-based queries
3.Matrix transformation queries

Documentation

• Update Federate Analytics related documentation in OpenFL.

Open Questions

• Handling data and data schema? - Not in scope.
• Handling timeouts? - Not in scope.

Phase 2

1. Use Secure Aggregation or other methods like encryption on intermediate results which will help preserve data privacy and security.
2. Basic set of aggregation functions

[teoparvanov]

Thanks for starting this discussion, @tanwarsh! Federated Analytics is a promising field, but it presents a new set of challenges, esp. regarding the wide spectrum of possible analytics queries.

I have a couple of questions from my first read-through of this proposal:

1. Could you elaborate more on the type of aggregation functions that would be supported? Would there need to be a custom one per query type - implemented by the OpenFL user? Alternatively, would OpenFL provide a basic set of aggregation functions specifically for FA?
2. It looks like that for each type of query (histogram, SQL, K/V, …), you plan to have a separate task runner. Is there any common functionality that can be extracted in an abstract FA_TaskRunner base class, to facilitate adding support for new types of queries (possibly even by OpenFL users)?
3. Regarding the example FL plan, why do you need learning and evaluation task groups in a FL plan focused around analytics?
4. Could you elaborate a bit more in the design how the query results would be encoded as tensors by the collaborator?

[teoparvanov]

@tanwarsh, maybe some inspiration (esp. regarding the aggregation aspect) can be drawn from MapReduce as a pattern for the first iteration of Federated Analytics v1.0 for OpenFL?

[tanwarsh]

Thank you @teoparvanov, for the detailed review.

1. Query and Aggregation of result of queries depend on use-case and it will be defined by the users. There will be need of custom aggregation function which will be unique to the query. As of the first iteration OpenFL will not provide a basic set of aggregation function but in future iteration we can add basic aggregation functions for all types of query. eg. Statistical, Set-based and Matrix transformation queries.
2. There will be FA_TaskRunner base class with common functionality but the query and aggregation function itself will be added by users themself. This will facilitate adding support for new queries.
3. We only need analysis task group for this. Similar to FedEval only analysis task group task will be executed. selected_task_group : analysis.
4. Similar to how collaborators share model weights, with FA collaborators will share query results which will be aggregated using the custom aggregation function written by users.
   eg. LocalTensor(col_name='collaborator1_sepal length (cm)', tensor=array([2.2 , 2.42, 2.64, 2.86, 3.08, 3.3 , 3.52, 3.74, 3.96, 4.18, 4.4 ], dtype=float32))

[teoparvanov]

Thanks for the clarifications - please update the proposal accordingly, so we can continue the discussion.

[tanwarsh]

@teoparvanov, I have updated proposal.

[teoparvanov]

Regarding supplementing FA with SecureAggregation, you should take into account that SecAgg mainly works for cases where the local query results are summed arithmetically (as in FedAvg). It will likely not be applicable in other cases where other types of query aggregation is used, such as concatenation, finding maximum or minimum, or any non-linear operation.

I would therefore suggest that you add a section in the design where you illustrate the types of federated queries that will be supported, and more importantly - how they will be aggregated.

[tanwarsh]

Thank you for pointing this out. I agree that SecureAggregation will only work where the local query results are summed arithmetically. However, it does provide stronger privacy and security for the results and protects the data. Therefore, decided to include it in the second iteration although it will only support query with arithmetic results. Added a section that lists the types of federated queries for the first iteration.

[teoparvanov]

I would like to raise an additional question/suggestion. In the current proposal, the query is defined as part of the FL plan. This entails that for each new query, the FL plan would have to be re-distributed among re-approved the participants. For a more streamlined experience, could we consider an approach where the query can be passed as an aggregator CLI parameter? This could follow a similar pattern to how we can currently switch the aggregator's mode between learning and evaluation:

fx aggregator start --task_group evaluation

In the case of FA, the query could be entirely removed from the FL plan, and passed via command line. For example:

fx aggregator start --task_group analytics --query query.json # ... or directly inline

Now, although convenient, this means that the aggregator would be able to run pretty much any query without preliminary approval by the collaborators. These may need to be additionally restricted.

@ishaileshpant , what would be your thoughts and suggestions in this regard?