Incentivized Communication for Federated Bandits

We consider a learning system consisting of (1) $N$ clients that directly interact with the environment by taking actions and receiving the corresponding rewards, and (2) a central server that coordinates the communication among the clients to facilitate their learning collectively. The clients can only communicate with the central server, but not with each other, resulting in a star-shaped communication network. At each time step $t\in[T]$, an arbitrary client $i_{t}\in[N]$ becomes active and chooses an arm $\mathbf{x}_{t}$ from a candidate set $\mathcal{A}_{t}\subseteq\mathbb{R}^{d}$, and then receives the corresponding reward feedback $y_{t}=f(\mathbf{x}_{t})+\eta_{t}\in\mathbb{R}$. Note that $\mathcal{A}_{t}$ is time-varying, $f$ denotes the unknown reward function shared by all clients, and $\eta_{t}$ denotes zero mean sub-Gaussian noise with known variance $\sigma^{2}$.

The performance of the learning system is measured by the cumulative (pseudo) regret over all $N$ clients in the finite time horizon $T$, i.e., $R_{T}=\sum_{t=1}^{T}r_{t}$, where $r_{t}=\max_{\mathbf{x}\in\mathcal{A}_{t}}\mathbf{E}[y|\mathbf{x}]-\mathbf{E}[y_{t}|\mathbf{x}_{t}]$ is the regret incurred by client $i_{t}$ at time step $t$. Moreover, under the federated learning setting, the system also needs to keep the communication cost $C_{T}$ low, which is measured by the total number of scalars (Wang et al., 2020b) being transferred across the system up to time $T$.

With the linear reward assumption, i.e., $f(\mathbf{x})=\mathbf{x}^{\top}\theta_{\star}$, where $\theta_{\star}$ denotes the unknown parameter, a ridge regression estimator $\hat{\theta}_{t}=V_{t}^{-1}b_{t}$ can be constructed based on sufficient statistics from all $N$ clients at each time step $t$, where $V_{t}=\sum_{s=1}^{t}\mathbf{x}_{s}\mathbf{x}_{s}^{\top}$ and $b_{t}=\sum_{s=1}^{t}\mathbf{x}_{s}y_{s}$ (Lattimore and Szepesvári, 2020). Using $\hat{\theta}_{t}$ under the Optimism in the Face of Uncertainty (OFUL) principle (Abbasi-Yadkori et al., 2011), one can obtain the optimal regret $R_{T}=O(d\sqrt{T})$. To achieve this regret bound in the federated setting, a naive method is to immediately share statistics of each newly collected data sample to all other clients in the system, which essentially recovers its centralized counterpart. However, this solution incurs a disastrous communication cost $C_{T}=O(d^{2}NT)$. On the other extreme, if no communication occurs throughout the entire time horizon (i.e., $C_{T}=0$), the regret upper bound can be up to $R_{T}=O(d\sqrt{NT})$ when each client interacts with the environment at the same frequency, indicating the importance of timely data/model aggregation in reducing $R_{T}$.

To balance this trade-off between regret and communication cost, prior research efforts centered around designing communication-efficient protocols for federated bandits that feature the “delayed update” of sufficient statistics (Wang et al., 2020b; Li and Wang, 2022a; He et al., 2022). Specifically, each client $i$ only has a delayed copy of $V_{t}$ and $b_{t}$, denoted as $V_{i,t}=V_{t_{\text{last}}}+\Delta V_{i,t},b_{i,t}=b_{t_{\text{last}}}+\Delta b_{i,t}$, where $V_{t_{\text{last}}},b_{t_{\text{last}}}$ is the aggregated sufficient statistics shared by the server in the last communication, and $\Delta V_{i,t},\Delta b_{i,t}$ is the accumulated local updates that client $i$ obtain from its interactions with the environment since $t_{\text{last}}$. In essence, the success of these algorithms lies in the fact that $V_{t},b_{t}$ typically changes slowly and thus has little instant impact on the regret for most time steps. Therefore, existing protocols that only require occasional communications can still achieve nearly optimal regret, despite the limitation on assuming clients’ willingness on participation as we discussed before.

Different from the prior works in this line of research, where all clients altruistically share their data with the server whenever a communication round is triggered, we are intrigued in a more realistic setting where clients are self-interested and thus reluctant to share data with the server if not well motivated. Formally, each client in the federated system inherently experiences a cost ¹¹1Note that if the costs are trivially set to zero, then clients have no reason to opt-out of data sharing and our problem essentially reduces to the standard federated bandits problem (Wang et al., 2020b). of data sharing, denoted by $\widetilde{D}^{p}_{i}\in\mathbb{R}$, due to their individual consumption of computing resources in local updates or concerns about potential privacy breaches caused by communication with the server. Moreover, as the client has nothing to lose when there is no local update to share in a communication round at time step $t$, in this case we assume the cost is 0, i.e., $D^{p}_{i}=\widetilde{D}^{p}_{i}\cdot\mathbb{I}(\Delta V_{i,t}\neq\mathbf{0})$. As a result, the server needs to motivate clients to participate in data sharing via the incentive mechanism $\mathcal{M}:\mathbb{R}^{N}\times\mathbb{R}^{d\times d}\rightarrow\mathbb{R}^{N}$, which takes as inputs a collection of client local updates $\Delta V_{i,t}\in\mathbb{R}^{d\times d}$ and a vector of cost values $D^{p}=\{D^{p}_{1},\cdots,D^{p}_{N}\}\in\mathbb{R}^{N}$, and outputs the incentive $\mathcal{I}=\{\mathcal{I}_{1,t},\cdots,\mathcal{I}_{N,t}\}\in\mathbb{R}^{N}$ to be distributed among the clients. Specifically, to make it possible to measure gains and losses of utility in terms of real-valued incentives (e.g., monetary payment), we adopt the standard quasi-linear utility function assumption, as is standard in economic analysis (Allais, 1953; Pemberton and Rau, 2007).

At each communication round, a client decides whether to share its local update with the server based on the potential utility gained from participation, i.e., the difference between the incentive and the cost of data sharing. This requires the incentive mechanism to be individually rational:

The server coordinates with all clients and incentivizes them to participate in the communication to realize its own objective (e.g., collective regret minimization). This requires $\mathcal{M}$ to be sufficient:

Typically, under different application scenarios, the server may have different objectives, such as regret minimization or best arm identification. In this work, we set the objective of the server to minimize the regret across all clients; and ideally the server aims to attain the optimal $\widetilde{O}(d\sqrt{T})$ regret in the centralized setting via the incentivized communication. Therefore, we consider an incentive mechanism is sufficient if it ensures that the resulting accumulated regret is bounded by $\widetilde{O}(d\sqrt{T})$.

Our framework comprises three main steps: 1) client’s local update; 2) communication trigger; and 3) incentivized data exchange among the server and clients. Specifically, after initialization, an active client performs a local update in each time step and checks the communication trigger. If a communication round is triggered, the system performs incentivized data exchange between clients and the server. Otherwise, no communication is needed.

Formally, at each time step $t=1,\dots,T$, an arbitrary client $i_{t}$ becomes active and interacts with its environment using observed arm set $\mathcal{A}_{t}$ (Line 5). Specifically, it selects an arm $\mathbf{x}_{t}\in\mathcal{A}_{t}$ that maximizes the UCB score as follows (Line 6):

where $\hat{\theta}_{i_{t},t-1}(\lambda)=V^{-1}_{i_{t},t-1}(\lambda)b_{i_{t},t-1}$ is the ridge regression estimator of $\theta_{\star}$ with regularization parameter $\lambda>0$, $V_{i_{t},t-1}(\lambda)=V_{i_{t},t-1}+\lambda I$, and $\alpha_{i_{t},t-1}=\sigma\sqrt{\log{\frac{\det({V_{i_{t},t-1}(\lambda))}}{\det{(\lambda I)}}}+2\log{1/\delta}}+\sqrt{\lambda}$. $V_{i_{t},t}(\lambda)$ denotes the covariance matrix constructed using data available to client $i_{t}$ up to time $t$. After obtaining a new data point $(\mathbf{x}_{t},y_{t})$ from the environment, client $i_{t}$ checks the communication event trigger $\Delta t_{i_{t},t}\cdot\log\frac{\det(V_{i_{t},t}(\lambda))}{\det(V_{i_{t},t_{\text{last}}}(\lambda))}>D_{c}$ (Line 9), where $\Delta t_{i_{t},t}$ denotes the time elapsed since the last time $t_{\text{last}}$ it communicated with the server and $D_{c}\geq 0$ denotes the specified threshold.

As mentioned in Section 1, in federated bandit learning, clients can reduce their regret by using models constructed via shared data. Denote $\widetilde{V}_{t}$ as the covariance matrix constructed by all available data in the system at time step $t$. Based on Lemma 5 and 7, the instantaneous regret of client $i_{t}$ is upper bounded by:

where the determinant ratio reflects the additional regret due to the delayed synchronization between client $i_{t}$’s local sufficient statistics and the global optimal oracle. Therefore, minimizing this ratio directly corresponds to reducing client $i_{t}$’s regret. For example, full communication keeps the ratio at 1, which recovers the regret of the centralized setting discussed in Section 3.1.

Therefore, given the client’s desire for regret minimization, the data itself can be used as a form of incentive by the server. And the star-shaped communication network also gives the server an information advantage over any single client in the system: a client can only communicate with the server, while the server can communicate with every client. Therefore, the server should utilize this advantage to create incentives (i.e., the LHS of Eq. (1)), and a natural design to evaluate this data incentive is:

where $D_{i,t}(S_{t})=\sum_{j:\{\Delta V_{j,t}\in S_{t}\}\wedge\{j\neq i\}}\Delta V_{j,t}+\Delta V_{-i,t}$ denotes the data that the server can offer to client $i$ during the communication at time $t$ (i.e., current local updates from other participants that have not been shared with the server) and $\Delta V_{-i,t}$ is the historically aggregated updates stored in the server that has not been shared with client $i$. Eq. (4) suggests a substantial increase in the determinant of the client’s local data is desired by the client, which results in regret reduction.

With the above data valuation in Eq. (4), we propose the payment-free incentive mechanism that motivates clients to share data by redistributing data collected from participating clients. We present this mechanism in Algorithm 2, and briefly sketch it below. First, we initiate the participant set $S_{t}=\tilde{S}_{t}$, assuming all clients agree to participate. Then, we iteratively update $S_{t}$ by checking the willingness of each client $i$ in $S_{t}$ according to Eq. (1). If $S_{t}$ is empty or all clients in it are participating, then terminate; otherwise, remove client $i$ from $S_{t}$ and repeat the process.

While this payment-free incentive mechanism is neat and intuitive, it has no guarantee on the amount of data that can be collected. To see this, we provide a theoretical negative result with rigorous regret analysis in Theorem 3 (see proof in Appendix C).

Recall the discussion in Section 3.1, when there is no communication $R_{T}$ is upper bounded by $O(d\sqrt{NT})$. Hence, in the worst-case scenario, the payment-free incentive mechanism might not motivate any client to participate. It is thus not a sufficient mechanism.

To address the insufficiency issue, we further devise a payment-efficient incentive mechanism that introduces additional monetary incentives to motivate clients’ participation:

where $\mathcal{I}^{d}_{i,t}$ is the data incentive defined in Eq. (4), and $\mathcal{I}^{m}_{i,t}$ is the real-valued monetary incentive, i.e., the payment assigned to client for its participation. Specifically, we are intrigued by the question: rather than trivially paying unlimited amounts to ensure everyone’s participation, can we devise an incentive mechanism that guarantees a certain level of client participation such that the overall regret is still nearly optimal but under acceptable monetary incentive cost?

Inspired by the determinant ratio principle discussed in Eq. (3), we propose to control the overall regret by ensuring that every client closely approximates the oracle after each communication round, which can be formalized as $\det(V_{g,t})/\det(\widetilde{V}_{t})\geq\beta$, where $V_{g,t}=V_{g,t-1}+\Sigma(S_{t})$ is to be shared with all clients and $\Sigma(S_{t})=\sum_{j:\{\Delta V_{j,t}\in S_{t}\}}\Delta V_{j,t}$. The parameter $\beta\in[0,1]$ characterizes the chosen gap between the practical and optimal regrets that the server commits to. Denote the set of clients motivated by $\mathcal{I}^{d}_{i,t}$ at time $t$ as ${S}^{d}_{t}$ and those motivated by $\mathcal{I}^{m}_{i,t}$ as ${S}^{m}_{t}$, and thus ${S}_{t}={S}^{m}_{t}\cup{S}^{d}_{t}$. At each communication round, the server needs to find the minimum $\mathcal{I}^{m}_{i,t}$ such that pooling local updates from $S_{t}$ satisfies the required regret reduction for the entire system.

Algorithm 2 maximizes $\mathcal{I}^{d}_{i,t}$, and thus the servers should compute $\mathcal{I}^{m}_{i,t}$ on top of optimal $\mathcal{I}^{d}_{i,t}$ and resulting ${S}^{d}_{t}$, which however is still combinatorially hard. First, a brute-force search can yield a time complexity up to $O(2^{N})$. Second, different from typical optimal subset selection problems (Kohavi and John, 1997), the dynamic interplay among clients in our specific context brings a unique challenge: once a client is incentivized to share data, the other uninvolved clients may change their willingness due to the increased data incentive, making the problem even more intricate.

To solve the above problem, we propose a heuristic ranking-based method, as outlined in Algorithm 3. The heuristic is to rank clients by the marginal gain they bring to the server’s determinant, as formally defined in Eq. (6), which helps minimize the number of clients requiring monetary incentives, while empowering the participation of other clients motivated by the aggregated data. This forms an iterative search process: First, we rank all $m$ non-participating clients (Line 2-3) by their potential contribution to the server (with participant set $S_{t}$ committed); Then, we segment the list by $\beta$, anyone whose participation satisfies the overall $\beta$ gap constraint is an immediately valid choice (Line 4). The first client $i_{\alpha}$ in the valid list and its payment $\mathcal{I}^{m}_{\text{last}}$ ($\infty$ if not available) will be our last resort (Line 5); Lastly, we check if there exist potentially more favorable solutions from the invalid list (Line 6). Specifically, we try to elicit up to $k=\alpha-1$ ($k=m$ if $i_{\alpha}$ is not available) clients from the invalid list in $n\leq k$ rounds, where only one client will be chosen using the same heuristic in each round. If having $n$ clients from the invalid list also satisfies the $\beta$ constraint and results in a reduced monetary incentive cost compared to $\mathcal{I}^{m}_{\text{last}}$, then we opt for this alternative solution. Otherwise, we will adhere to the last resort.

This Heuristic Search is detailed in Appendix A, and it demonstrates a time complexity of only $O(N)$ in the worst-case scenarios, i.e., $n=m=N$. Theorem 4 guarantees the sufficiency of this mechanism w.r.t communication and payment bounds.

The proof of theorem 4 can be found in Appendix D.

We first empirically compared the performance of the payment-free mechanism (named as Inc-FedUCB-PF) and the payment-efficient mechanism Inc-FedUCB in Figure 1. It is clear that the added monetary incentives lead to lower regret and communication costs, particularly with increased $D^{p}_{\star}$. Lower regret is expected as more data can be collected and shared; while the reduced communication cost is contributed by reduced communication frequency. When less clients can be motivated in one communication round, more communication rounds will be triggered as the clients tend to have outdated local statistics.

To investigate the impact of different components in our heuristic search, we compare the full-fledged model Inc-FedUCB with following variants on various environments: (1) Inc-FedUCB (w/o PF): without payment-free incentive mechanism, where the server only use money to incentivize clients; (2) Inc-FedUCB (w/o IS): without iterative search, where the server only rank the clients once. (3) Inc-FedUCB (w/o PF + IS): without both above strategies.

In Figure 2, we present the averaged learning trajectories of regret and communication cost, along with the final payment costs (normalized) under different $D^{p}_{\star}$. The results indicate that the full-fledged Inc-FedUCB consistently outperforms all other variants in various environments. Additionally, there is a substantial gap between the variants with and without the PF strategy, emphasizing the significance of leveraging server’s information advantage to motivate participation.

We further explored diverse $\beta$ hyperparameter settings for Inc-FedUCB in various environments with varying $D^{p}_{\star}$, along with the comparison with DisLinUCB (Wang et al., 2020b) (only comparable when $D^{p}_{\star}=0$).