The goal of federated learning is to design algorithms in which several
agents communicate with a central node, in a privacy-protecting manner, to
minimize the average of their loss functions. In this approach, each node not
only shares the required computational budget but also has access to a larger
data set, which improves the quality of the resulting model. However, this
method only develops a common output for all the agents, and therefore, does
not adapt the model to each user data. This is an important missing feature
especially given the heterogeneity of the underlying data distribution for
various agents. In this paper, we study a personalized variant of the federated
learning in which our goal is to find a shared initial model in a distributed
manner that can be slightly updated by either a current or a new user by
performing one or a few steps of gradient descent with respect to its own loss
function. This approach keeps all the benefits of the federated learning
architecture while leading to a more personalized model for each user. We show
this problem can be studied within the Model-Agnostic Meta-Learning (MAML)
framework. Inspired by this connection, we propose a personalized variant of
the well-known Federated Averaging algorithm and evaluate its performance in
terms of gradient norm for non-convex loss functions. Further, we characterize
how this performance is affected by the closeness of underlying distributions
of user data, measured in terms of distribution distances such as Total
Variation and 1-Wasserstein metric.
Description
[2002.07948] Personalized Federated Learning: A Meta-Learning Approach
%0 Journal Article
%1 fallah2020personalized
%A Fallah, Alireza
%A Mokhtari, Aryan
%A Ozdaglar, Asuman
%D 2020
%K federated meta-learning
%T Personalized Federated Learning: A Meta-Learning Approach
%U http://arxiv.org/abs/2002.07948
%X The goal of federated learning is to design algorithms in which several
agents communicate with a central node, in a privacy-protecting manner, to
minimize the average of their loss functions. In this approach, each node not
only shares the required computational budget but also has access to a larger
data set, which improves the quality of the resulting model. However, this
method only develops a common output for all the agents, and therefore, does
not adapt the model to each user data. This is an important missing feature
especially given the heterogeneity of the underlying data distribution for
various agents. In this paper, we study a personalized variant of the federated
learning in which our goal is to find a shared initial model in a distributed
manner that can be slightly updated by either a current or a new user by
performing one or a few steps of gradient descent with respect to its own loss
function. This approach keeps all the benefits of the federated learning
architecture while leading to a more personalized model for each user. We show
this problem can be studied within the Model-Agnostic Meta-Learning (MAML)
framework. Inspired by this connection, we propose a personalized variant of
the well-known Federated Averaging algorithm and evaluate its performance in
terms of gradient norm for non-convex loss functions. Further, we characterize
how this performance is affected by the closeness of underlying distributions
of user data, measured in terms of distribution distances such as Total
Variation and 1-Wasserstein metric.
@article{fallah2020personalized,
abstract = {The goal of federated learning is to design algorithms in which several
agents communicate with a central node, in a privacy-protecting manner, to
minimize the average of their loss functions. In this approach, each node not
only shares the required computational budget but also has access to a larger
data set, which improves the quality of the resulting model. However, this
method only develops a common output for all the agents, and therefore, does
not adapt the model to each user data. This is an important missing feature
especially given the heterogeneity of the underlying data distribution for
various agents. In this paper, we study a personalized variant of the federated
learning in which our goal is to find a shared initial model in a distributed
manner that can be slightly updated by either a current or a new user by
performing one or a few steps of gradient descent with respect to its own loss
function. This approach keeps all the benefits of the federated learning
architecture while leading to a more personalized model for each user. We show
this problem can be studied within the Model-Agnostic Meta-Learning (MAML)
framework. Inspired by this connection, we propose a personalized variant of
the well-known Federated Averaging algorithm and evaluate its performance in
terms of gradient norm for non-convex loss functions. Further, we characterize
how this performance is affected by the closeness of underlying distributions
of user data, measured in terms of distribution distances such as Total
Variation and 1-Wasserstein metric.},
added-at = {2020-03-04T14:20:52.000+0100},
author = {Fallah, Alireza and Mokhtari, Aryan and Ozdaglar, Asuman},
biburl = {https://www.bibsonomy.org/bibtex/20d94999a58a02787d6a04c1ac1770688/kirk86},
description = {[2002.07948] Personalized Federated Learning: A Meta-Learning Approach},
interhash = {cfb3ca45d1f5121ff8b39eb7e4870529},
intrahash = {0d94999a58a02787d6a04c1ac1770688},
keywords = {federated meta-learning},
note = {cite arxiv:2002.07948},
timestamp = {2020-03-04T14:20:52.000+0100},
title = {Personalized Federated Learning: A Meta-Learning Approach},
url = {http://arxiv.org/abs/2002.07948},
year = 2020
}