Skip to content

Latest commit

 

History

History
469 lines (345 loc) · 33.8 KB

BACKUP-ALL-IN-ONE.md

File metadata and controls

469 lines (345 loc) · 33.8 KB
Raw

Maintenance Last Commit Ask Me Anything ! Awesome GitHub license

Awesome Federated Learning

This repository maintains a collection of papers, articles, videos, frameworks, etc of federated learing, for the purpose of learning and research.

Note:

  • Italic contents are the ones worth reading
  • Last Update: 16 Aug, 2020

Introduction

  • Federated Learning Comic [Google Blog]
  • Federated Learning: Collaborative Machine Learning without Centralized Training Data [Google Blog]
  • GDPR, Data Shotrage and AI (AAAI-19) [Video]
  • Federated Learning: Machine Learning on Decentralized Data (Google I/O'19) [Youtube]
  • Federated Learning White Paper V1.0 [Paper]
  • Federated learning: distributed machine learning with data locality and privacy [Blog]

Survey

  • Federated Machine Learning: Concept and Applications [Paper]
  • Federated Learning: Challenges, Methods, and Future Directions [Paper]
  • Advances and Open Problems in Federated Learning [Paper]
  • Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection [Paper]
  • Federated Learning in Mobile Edge Networks: A Comprehensive Survey [Paper]
  • Federated Learning for Wireless Communications: Motivation, Opportunities and Challenges [Paper]
  • A Review of Applications in Federated Learning [Paper]

Benchmarks

  • LEAF: A Benchmark for Federated Settings [Paper] [Github] [Recommend]
  • The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems [Paper]
  • A Performance Evaluation of Federated Learning Algorithms [Paper]
  • Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking [Paper]

Model Aggregation

  • One-Shot Federated Learning [Paper]
  • Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating [Paper] [NIPS]2019 Workshop)
  • Bayesian Nonparametric Federated Learning of Neural Networks [Paper] (ICML 2019)
  • Agnostic Federated Learning [Paper] (ICML 2019)
  • Federated Learning with Matched Averaging [Paper] (ICLR 2020)
  • Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications [Paper]

Statistical Heterogeneity

  • Federated Learning with Non-IID Data [Paper]
  • The Non-IID Data Quagmire of Decentralized Machine Learning [Paper]
  • Robust and Communication-Efficient Federated Learning from Non-IID Data [Paper] [IEEE transactions on neural networks and learning systems]
  • FedMD: Heterogenous Federated Learning via Model Distillation [Paper] [NIPS 2019 Workshop]
  • First Analysis of Local GD on Heterogeneous Data [Paper]
  • SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning [Paper] [ICML20]
  • Federated Optimization for Heterogeneous Networks [Paper]
  • On the Convergence of FedAvg on Non-IID Data [Paper] [OpenReview]
  • Agnostic Federated Learning [Paper] (ICML 2019)
  • Local SGD Converges Fast and Communicates Little [Paper]
  • Adaptive Gradient-Based Meta-Learning Methods [Paper] [NIPS 2019 Workshop]
  • Federated Adversarial Domain Adaptation [Paper] (ICLR 2020)
  • LoAdaBoost: Loss-Based AdaBoost Federated Machine Learning on Medical Data [Paper]
  • On Federated Learning of Deep Networks from Non-IID Data: Parameter Divergence and the Effects of Hyperparametric Methods [Paper] [Rejected in ICML 2020]
  • Overcoming Forgetting in Federated Learning on Non-IID Data [Paper] [NIPS 2019 Workshop]
  • FedMAX: Activation Entropy Maximization Targeting Effective Non-IID Federated Learning [Video] [NIPS 2019 Workshop]
  • Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification [Paper] [NIPS 2019 Workshop]
  • Fair Resource Allocation in Federated Learning [Paper]
  • Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data [Paper]
  • Think Locally, Act Globally: Federated Learning with Local and Global Representations [Paper] [NIPS 2019 Workshop]

Personalization

  • Improving Federated Learning Personalization via Model Agnostic Meta Learning [Paper] [NIPS 2019]Workshop)
  • Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach [Paper] [NeurIPS 2020]
  • Lower Bounds and Optimal Algorithms for Personalized Federated Learning [Paper] [NeurIPS 2020]
  • Personalized Federated Learning with Moreau Envelopes [Paper] [NeurIPS 2020]

Meta Learning

  • Federated Meta-Learning with Fast Convergence and Efficient Communication [Paper]
  • Federated Meta-Learning for Recommendation [Paper]
  • Adaptive Gradient-Based Meta-Learning Methods [Paper]

Multi-task Learning

  • MOCHA: Federated Multi-Task Learning [Paper] [NIPS 2017] [Slides]
  • Variational Federated Multi-Task Learning [Paper]
  • Federated Kernelized Multi-Task Learning [Paper]
  • Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints [Paper] [NIPS 2019 Workshop]

Convergence

  • A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication [Paper] [NIPS 2018]
  • SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning [Paper]
  • Federated Optimization for Heterogeneous Networks [Paper]
  • On the Convergence of FedAvg on Non-IID Data [Paper] [OpenReview]
  • Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent [Paper] [NIPS 2017]
  • Communication Efficient Decentralized Training with Multiple Local Updates [Paper]
  • First Analysis of Local GD on Heterogeneous Data [Paper]
  • MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling [Paper]
  • Local SGD Converges Fast and Communicates Little [Paper]
  • SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum [Paper]
  • Adaptive Federated Learning in Resource Constrained Edge Computing Systems [Paper] [IEEE Journal on Selected Areas in Communications, 2019]
  • Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning [Paper] [AAAI 2018]
  • On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization [Paper] [ICML 2019]
  • Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data [Paper]
  • Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data [Paper] [NIPS 2019 Workshop]

System

  • Towards Federated Learning at Scale: System Design [Paper] [Must Read]
  • Demonstration of Federated Learning in a Resource-Constrained Networked Environment [Paper]
  • Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection [Paper]
  • Applied Federated Learning: Improving Google Keyboard Query Suggestions [Paper]
  • Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy [Paper] (Startup)
  • When Edge Meets Learning: Adaptive Control for Resource-Constrained Distributed Machine Learning [Paper] [IEEE INFOCOM 2018]

Communication Efficiency

  • Communication-Efficient Learning of Deep Networks from Decentralized Data [Paper] [Github] [Google] [Must Read]
  • Two-Stream Federated Learning: Reduce the Communication Costs [Paper] [2018 IEEE VCIP]
  • Client-Edge-Cloud Hierarchical Federated Learning [Paper]
  • PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization [Paper] [NIPS 2019], Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi.
  • Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training [Paper] [ICLR 2018] Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally
  • The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication [Paper] Sebastian U Stich and Sai Praneeth Karimireddy, 2019.
  • A Communication Efficient Collaborative Learning Framework for Distributed Features [Paper] [NIPS 2019 Workshop]
  • Active Federated Learning [Paper] [NIPS 2019 Workshop]
  • Communication-Efficient Distributed Optimization in Networks with Gradient Tracking and Variance Reduction [Paper] [NIPS 2019 Workshop]
  • Gradient Descent with Compressed Iterates [Paper] [NIPS 2019 Workshop]

Compression

  • Robust and Communication-Efficient Federated Learning from Non-IID Data [Paper], 2019
  • Expanding the Reach of Federated Learning by Reducing Client Resource Requirements [Paper] Sebastian Caldas, Jakub Konecny, H Brendan McMahan, and Ameet Talwalkar, 2018
  • Federated Learning: Strategies for Improving Communication Efficiency [Paper] [NIPS2016 Workshop] [Google]
  • Natural Compression for Distributed Deep Learning [Paper] Samuel Horvath, Chen-Yu Ho, Ludovit Horvath, Atal Narayan Sahu, Marco Canini, and Peter Richtarik, 2019.
  • FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization [Paper], 2019
  • ATOMO: Communication-efficient Learning via Atomic Sparsification [Paper] [NIPS 2018], H. Wang, S. Sievert, S. Liu, Z. Charles, D. Papailiopoulos, and S. Wright.
  • vqSGD: Vector Quantized Stochastic Gradient Descent [Paper] Venkata Gandikota, Raj Kumar Maity, and Arya Mazumdar, 2019.
  • QSGD: Communication-efficient SGD via gradient quantization and encoding [Paper] [NIPS 2017], Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic.
  • cpSGD: Communication-efficient and differentially-private distributed SGD [Paper]
  • Federated Optimization: Distributed Machine Learning for On-Device Intelligence [Paper] [Google]
  • Distributed Mean Estimation with Limited Communication [Paper] [ICML 2017], Ananda Theertha Suresh, Felix X. Yu, Sanjiv Kumar, and H Brendan McMahan.
  • Randomized Distributed Mean Estimation: Accuracy vs Communication [Paper] Frontiers in Applied Mathematics and Statistics, Jakub Konecny and Peter Richtarik, 2016
  • Error Feedback Fixes SignSGD and other Gradient Compression Schemes [Paper] [ICML 2019], Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi.
  • ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning [Paper] [ICML 2017], H. Zhang, J. Li, K. Kara, D. Alistarh, J. Liu, and C. Zhang.

Important-Based Updating

  • eSGD: Communication Efficient Distributed Deep Learning on the Edge [Paper] [USENIX 2018 Workshop (HotEdge 18)]
  • CMFL: Mitigating Communication Overhead for Federated Learning [Paper]

Decentralization

  • Communication Compression for Decentralized Training [Paper] [NIPS 2018], H. Tang, S. Gan, C. Zhang, T. Zhang, and J. Liu.
  • 𝙳𝚎𝚎𝚙𝚂𝚚𝚞𝚎𝚎𝚣𝚎: Decentralization Meets Error-Compensated Compression [Paper] Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce Zhang, Tong Zhang, and Ji Liu, 2019

Resource Allocation

Participants Selection

  • Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge [Paper] (FedCS)

  • Hybrid-FL for Wireless Networks: Cooperative Learning Mechanism Using Non-IID Data [Paper]

    • Ask to upload some data from client to server

  • Efficient Training Management for Mobile Crowd-Machine Learning: A Deep Reinforcement Learning Approach [Paper]

    • Reward function: accumulated data, energy consumption, training accuracy

  • Fair Resource Allocation in Federated Learning [Paper]

  • Low-latency Broadband Analog Aggregation For Federated Edge Learning [Paper]

  • Federated Learning over Wireless Fading Channels [Paper]

  • Federated Learning via Over-the-Air Computation [Paper]

Adaptive Aggregation

  • Asynchronous Federated Learning for Geospatial Applications [Paper] [ECML PKDD Workshop 2018]
  • Asynchronous Federated Optimization [Paper]
  • Adaptive Federated Learning in Resource Constrained Edge Computing Systems [Paper] [IEEE Journal on Selected Areas in Communications, 2019]

Incentive Mechanism

  • Incentive Mechanism for Reliable Federated Learning: A Joint Optimization Approach to Combining Reputation and Contract Theory [Paper]
  • Motivating Workers in Federated Learning: A Stackelberg Game Perspective [Paper]
  • Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach [2019] [Paper]

Vertical Federated Learning

  • A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression [Paper] [NIPS 2019 Workshop]

Adversarial Attacks

  • Can You Really Backdoor Federated Learning? [Paper]
  • Model Poisoning Attacks in Federated Learning [Paper] [NIPS workshop 2018]

Decentralized Federated Learning

  • Decentralized Federated Learning: A Segmented Gossip Approach [Paper]
  • Peer-to-peer Federated Learning on Graphs [Paper]

Data Privacy and Confidentiality

  • Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning [Paper] [NIPS 2019 Workshop]
  • Quantification of the Leakage in Federated Learning [Paper]

Courses

  • Applied Cryptography [Udacity]
    • Cryptography basics

Differential Privacy

  • A Brief Introduction to Differential Privacy [Blog]
  • Deep Learning with Differential Privacy [Paper]
    • Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.
  • Learning Differentially Private Recurrent Language Models [Paper]
  • Federated Learning with Bayesian Differential Privacy [Paper] [NIPS 2019 Workshop]
  • Private Federated Learning with Domain Adaptation [Paper] [NIPS 2019 Workshop]
  • cpSGD: Communication-efficient and differentially-private distributed SGD [Paper]
  • Federated Learning with Bayesian Differential Privacy [Paper] [NIPS 2019 Workshop]

PATE

  • Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data [Paper]
    • Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar.
    • Private Aggregation of Teacher Ensembles (PATE)
  • Scalable Private Learning with PATE [Paper]
    • Extension of PATE

Secure Multi-party Computation

Secret Sharing

  • Simple Introduction to Sharmir's Secret Sharing and Lagrange Interpolation [Youtube]
  • Secret Sharing, Part 1 [Blog]: Shamir's Secret Sharing & Packed Variant
  • Secret Sharing, Part 2 [Blog]: Improve efficiency
  • Secret Sharing, Part 3 [Blog]

SPDZ

  • Basics of Secure Multiparty Computation [Youtube]: based on Shamir's Secret Sharing

  • What is SPDZ?

    • Part 1: MPC Circuit Evaluation Overview [Blog]
    • Part 2: Circuit Evaluation [Blog]

  • The SPDZ Protocol [Blog]: implementation codes included

Advance (Not Recommended For Beginners)
  • Multiparty Computation from Somewhat Homomorphic Encryption [Paper]
    • SPDZ introduction
  • Practical Covertly Secure MPC for Dishonest Majority – or: Breaking the SPDZ Limits [Paper]
  • MASCOT: Faster Malicious Arithmetic Secure Computation with Oblivious Transfer [Paper]
  • Removing the crypto provider and instead letting the parties generate these triples on their own
  • Overdrive: Making SPDZ Great Again [Paper]
  • Safetynets: Verifiable execution of deep neural networks on an untrusted cloud

Build Safe AI Series

  • Building Safe A.I. [Blog]

    • A Tutorial for Encrypted Deep Learning
    • Use Homomorphic Encryption (HE)

  • Private Deep Learning with MPC [Blog]

    • A Simple Tutorial from Scratch
    • Use Multiparty Compuation (MPC)

  • Private Image Analysis with MPC [Blog]

    • Training CNNs on Sensitive Data
    • Use SPDZ as MPC protocol

MPC related Paper

Helen: Maliciously Secure Coopetitive Learning for Linear Models [Paper] [NIPS 2019 Workshop]

Privacy Preserving Machine Learning

  • Privacy-Preserving Deep Learning [Paper]
  • Privacy Partition: A Privacy-Preserving Framework for Deep Neural Networks in Edge Networks [Paper]
  • Practical Secure Aggregation for Privacy-Preserving Machine Learning [Paper] (Google)
    • Secure Aggregation: The problem of computing a multiparty sum where no party reveals its update in the clear—even to the aggregator
    • Goal: securely computing sums of vectors, which has a constant number of rounds, low communication overhead, robustness to failures, and which requires only one server with limited trust
    • Need to have basic knowledge of cryptographic algorithms such as secret sharing, key agreement, etc.
  • Practical Secure Aggregation for Federated Learning on User-Held Data [Paper] (Google)
    • Highly related to Practical Secure Aggregation for Privacy-Preserving Machine Learning
    • Proposed 4 protocol one by one with gradual improvement to meet the requirement of secure aggregation propocol.
  • SecureML: A System for Scalable Privacy-Preserving Machine Learning [Paper]
  • DeepSecure: Scalable Provably-Secure Deep Learning [Paper]
  • Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications [Paper]
  • Contains several MPC frameworks

Other Learning

  • Federated Reinforcement Learning [Paper]
  • Secure Federated Transfer Learning [Paper]

Frameworks

Workshops

  • NIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 1 [Video]
  • NIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 2 [Video]
  • NIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality 3 [Video]

Applications

  • Federated Learning Approach for Mobile Packet Classification [Paper]
  • Federated Learning for Ranking Browser History Suggestions [Paper] [NIPS 2019 Workshop]

Healthcare

  • HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalography [Paper] [NIPS 2019 Workshop]
  • Learn Electronic Health Records by Fully Decentralized Federated Learning [Paper] [NIPS 2019 Workshop]
  • Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records [Paper] [News]
    • MIT CSAI, Harvard Medical School, Tsinghua University
  • Federated learning of predictive models from federated Electronic Health Records. [Paper]
    • Boston University, Massachusetts General Hospital
  • FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare [Paper]
    • Microsoft Research Asia
  • Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation [Paper]
    • Intel
  • NVIDIA Clara Federated Learning to Deliver AI to Hospitals While Protecting Patient Data [Blog]
    • Nvidia
  • What is Federated Learning [Blog]
    • Nvidia
  • Split learning for health: Distributed deep learning without sharing raw patient data [Paper]
  • Two-stage Federated Phenotyping and Patient Representation Learning [Paper] [ACL 2019]
  • Federated Tensor Factorization for Computational Phenotyping [Paper] SIGKDD 2017
  • FedHealth- A Federated Transfer Learning Framework for Wearable Healthcare [Paper] [ICJAI19 workshop]
  • Multi-Institutional Deep Learning Modeling Without Sharing Patient Data- A Feasibility Study on Brain Tumor Segmentation [Paper] [MICCAI'18 Workshop]
  • Federated Patient Hashing [Paper] [AAAI'20]
  • Federated Learning in Distributed Medical Databases: Meta-Analysis of Large-Scale Subcortical Brain Data [Paper]
  • Confederated Machine Learning on Horizontally and Vertically Separated Medical Data for Large-Scale Health System Intelligence [Paper]
  • Privacy-Preserving Deep Learning Computation for Geo-Distributed Medical Big-Data Platform [Paper]
  • Institutionally Distributed Deep Learning Networks [Paper]

Natual Language Processing

Google

  • Federated Learning for Mobile Keyboard Prediction [Paper]
  • Applied Federated Learning: Improving Google Keyboard Query Suggestions [Paper]
  • Federated Learning Of Out-Of-Vocabulary Words [Paper]
  • Federated Learning for Emoji Prediction in a Mobile Keyboard [Paper]

Snips

Computer Vision

  • Performance Optimization for Federated Person Re-identification via Benchmark Analysis [Paper] [ACMMM20] [Github]
  • Real-World Image Datasets for Federated Learning [Paper]
    • Webank & Extreme Vision
  • FedVision- An Online Visual Object Detection Platform Powered by Federated Learning [Paper] [IAAI20]
  • Federated Learning for Vision-and-Language Grounding Problems [Paper] [AAAI20]

Recommendation

  • Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System [Paper]
    • Huawei
  • Federated Meta-Learning with Fast Convergence and Efficient Communication [Paper]
    • Huawei

Industrial

  • Turbofan POC: Predictive Maintenance of Turbofan Engines using Federated Learning [Github]
  • Turbofan Tycoon Simulation by Cloudera/FastForwardLabs [Web]
  • Firefox Search Bar [Blog] [Github] [Github]
    • Detail explaination of their implementationn of Federated Learning in production.

Finance

  • FFD: A Federated Learning Based Method for Credit Card Fraud Detection Paper International Conference on Big Data 2019

Company