PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

Each training group $g\in G_{\text{train}}$ has a preference dataset: $D_{g}=\{(x_{1}^{g},y_{1}^{g}),\dots,(x_{n}^{g},y_{n}^{g})\}$ where $x_{i}^{g}$ represents the embedding of LLM concatenated prompt-response pair: $x_{i}^{g}=\omega_{\text{emb}}(q_{i}^{g},r_{i}^{g})$ and $\omega_{\text{emb}}$ is a language model embedding function. The preference $y_{i}^{g}$ represents the group preference probability for the generated response $r_{i}^{g}$ to the query $q_{i}^{g}$. For instance, in Figure 1, the query represents the question, while the response corresponds to the aggregated probability distribution of answers within a group. This preference data is then processed by the Alpaca model (LLM) to generate embeddings, following the approach in (Zhao et al., 2023).

The dataset for each group is randomly divided into $m$ context samples $(x_{1}^{g},y_{1}^{g}),...,(x_{m}^{g},y_{m}^{g})$ and $n-m$ target samples $(x_{m+1}^{g},y_{m+1}^{g}),..,(x_{n}^{g},y_{n}^{g})$. The model is trained to predict the target preferences given the context examples, optimizing the following loss function as introduced in (Zhao et al., 2023):

where $p_{\theta}$ denotes the target points predicted preference distribution conditioned on the context points. At the end of local training, each client $g$ transmits its updated model parameters $\theta_{g}$ to the central aggregation server, which combines the received updates to train the global model.

We employ the FedAvg technique to aggregate updates from multiple groups (McMahan et al., 2017). The aggregation process is designed to minimize the global optimization function of FL.

where $p_{g}$ is the weight assigned to each training group $g$, defined as: $p^{g}=\frac{|D_{g}|}{\sum_{g^{\prime}}|D_{g^{\prime}}|}$. Here, $D_{g}$ represents the size of the preference dataset for group $g$. The term $p_{g}$ ensures that each group’s contribution to the global objective is weighted by the proportion of its dataset size relative to the total dataset across all training groups. The local objective function for group $g$ is defined as: $F_{g}(\theta)=\mathbb{E}_{(x,y)\sim D_{g}}\left[\mathcal{L}(f_{\theta}(x),y)\right]$, where $\mathcal{L}$ represents the loss function defined at Equation 1 and $f_{\theta}(x)$ is the model output for input $x$. To approximate the global objective, model updates from training groups are aggregated as follows:

where $\theta_{g}^{t}$ represents the locally updated model parameters at the training group $g$ in round $t$. The aggregated model is then redistributed to training and evaluation clients.

Our experiments utilize a transformer-based preference predictor model (GPO  (Zhao et al., 2023)), originally designed to train groups sequentially in an ordered manner. However, in PluralLLM, we adapt it to a FL setting, employing FedAvg as a completely different learning paradigm. The primary goal of our evaluation is to determine whether our approach in PluralLLM impacts the alignment score and group fairness across different groups compared to the original centralized learning.

Experiments are conducted on machines equipped with one NVIDIA A30 GPU, an Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz, and 256GB RAM. All results reported in this study are averaged over four different runs with varying random seeds to ensure stability.

We used the dataset from the Pew Research Center’s Global Attitudes Surveys (PewResearch), which collects public opinions on a wide range of social, political, and economic issues (Durmus et al., 2023b). These surveys capture diverse perspectives from various demographic and geographic groups, providing a rich foundation for analyzing preference alignment. To ensure a fair comparison with GPO (Zhao et al., 2023), we use the same subset of groups as in the original GPO training setup. In both FL and centralized learning, groups are partitioned into $60\%$ and $40\%$ groups for training and evaluation, respectively.

• •

  Federated Learning: We trained the transformer-based preference predictor model (GPO) for $1,300$ communication rounds in the FL setup, assuming that all clients participate in each communication round. Each local training step consists of 6 local epochs, where, in each epoch, we randomly sample context questions and corresponding preferences, then the target questions that we wish to predict its preferences to train the model. Adam optimizer used with a learning rate of $3\times 10^{-4}$.

• •

  Centralized Learning: We trained the transformer-based preference predictor model (GPO) for $1,300$ epochs, iterating over all training groups in each epoch. During the training epoch, each group samples a random selection of context questions and their corresponding preferences and target questions. Unlike FL, where model updates are aggregated after each communication round, the centralized approach updates the model sequentially for each group within a single epoch.

• •

  Preferences Embedding: We use Alpaca-7B, a fine-tuned version of LLaMA-7B, as the embedding model to represent preference data, which is then passed to transformer input (Zhao et al., 2023). The embedding step is done once over all the preference data for each group at the beginning of training.

To assess alignment performance, evaluation is conducted every $10$ communication rounds (in the FL setting) or every $10$ epochs (in the centralized setting). The alignment score is computed over randomly sampled data from all the evaluation groups to measure how well the trained model adapts to new group preferences over time.

To quantify the impact of PluralLLM on alignment performance, we evaluate:

• •

  Alignment Score(AS): We assess the degree of alignment between $2$ opinion distributions $P_{1}$ and $P_{2}$ by calculating the Alignment Score $AS(P_{1},P_{2};Q)$ over a set of questions $Q$ as used in  (Zhao et al., 2023). Jensen-Shannon Distance ²²2The Jensen–Shannon divergence (JSD) is a symmetric measure of similarity between two probability distributions, always non-negative, with $0$ denoting identical distributions and any value above $0$ indicating differences., denoted as JSD, is used to assess preference distribution similarity shifts.

  (4)

  $$AS(P_{1},P_{2};Q)=\frac{1}{|Q|}\sum_{q\in Q}\text{{JSD}}(P_{1}(q),P_{2}(q);Q)$$

• •

  Convergence Speed of Loss Function: We measure how quickly the model optimizes preference alignment by tracking the average loss across all clients at each communication round. This is compared to the centralized training loss per epoch. Convergence speed is defined as the point where the model reaches $95\%$ of its final loss value.

• •

  Fairness Metrics: We analyze the effect of PluralLLM on group fairness in preference alignment across different groups compared to centralized learning. Numerous studies have demonstrated that FL can inadvertently introduce unfairness into the trained models (Shi et al., 2023). This unfairness arises primarily from data heterogeneity across clients, which leads to disparate performance results and challenges in achieving equitable model accuracy across all participants. Furthermore, these fairness issues have the potential to show disparity in privacy leakage risks, as adversaries can potentially exploit shared model parameters to infer sensitive information (Zhao et al., 2025). We assess the fairness by adapting Coefficient of Variation (CoV) and Fairness Index(FI) to measure the disparity of alignment scores across distinct groups. These metrics are used to measure the relative perception of fairness in human-centered systems (Zhao and Elmalaki, 2024; Zhao et al., 2024). For $K$ groups, we define the alignment score of group $i$ as $AS_{i}$. The average alignment score across groups is calculated as $\mu=\frac{1}{K}\sum_{i=1}^{K}\text{AS}_{i}$. The CoV of alignment scores $CoV(AS)$ is calculated as:

  (5)

  $\displaystyle CoV(AS)=\frac{\sigma}{\mu}=\frac{\sqrt{\frac{1}{K}\sum_{i=1}^{K}(\text{AS}_{i}-\mu)^{2}}}{\mu},$

  where $\sigma$ is the standard derivation of $AS$. A lower CoV indicates a more equitable distribution of alignment scores among groups, suggesting better fairness in aligning distinct groups. We apply the Fairness Index (FI) transformation to interpret the fairness in percentage between $0$ and $1$. A higher FI indicates greater fairness, where $1$ represents perfect fairness.

  (6)

  $\displaystyle FI(AS)=\frac{1}{1+CoV^{2}(AS)}$

  In the context of group fairness in Machine Learning (ML) classification task, the principle of equal opportunity implies that individuals who are eligible for a favorable outcome have an equal chance of being correctly classified by the prediction model, regardless of their group membership. Similar to the classification task, we investigate the equal opportunity in LLM alignment task. To adapt the definition of equal opportunity in the probabilistic setting of LLM alignment, we use our definition of $CoV$ (Equation 5). In this context, equal opportunity would imply that the variability of alignment scores between groups is minimum or the $FI$ (Equation 6) is close to $1$.

PluralLLM converges at communication round $634$, whereas the centralized approach requires significantly more steps, converging at iteration $1180$ epoch as seen in Figure 2. Hence, PluralLLM achieves convergence 46% faster than the centralized learning approach, highlighting its efficiency in accelerating model training. Additionally, PluralLLM maintains a lower loss throughout training as observed in Figure 2, demonstrating improved stability compared to the centralized approach. The faster convergence of PluralLLM suggests that it is well-suited for distributed learning scenarios where reducing communication rounds is critical for efficiency.

Figure 4 demonstrates that PluralLLM achieves a $\approx 4$% improvement in the mean evaluation alignment score compared to the centralized approach. While the centralized method shows slower improvements and remains at a lower score, PluralLLM maintains a more stable progression with lower fluctuations, indicating that the distributed training approach enhances robustness in achieving better alignment and improves generalization across diverse data distributions. Additionally, Figure 3 highlights that for two evaluation groups, PluralLLM more accurately represents the baseline distribution for a given Q/A task compared to the centralized approach.

As observed in  Figure 5, PluralLLM improves the $FI$ by $0.04\%$ on average before converging at a round $634$ compared to the centralized GPO. PluralLLM maintains a comparable $FI$ across training steps till round $1300$ achieving an equal opportunity with $FI\approx 1$.