The Curse of Depth in Large Language Models

Methods: We evaluate Pre-LN and Post-LN models by assessing the impact of layer pruning at different depths. Our hypothesis is that Pre-LN models exhibit diminishing effectiveness in deeper layers, whereas Post-LN has less effective early layers. To verify this, we empirically quantify the contribution of individual layers to overall model performance across a diverse set of LLMs.

LLMs: We conduct experiments on multiple widely adopted LLMs: BERT-Large (Devlin, 2019), Mistral-7B (Jiang et al., 2023), LLaMA2-7B/13B (Touvron et al., 2023), DeepSeek-7B (Bi et al., 2024), and Qwen-7B (Bai et al., 2023). These models were chosen to ensure architectural and application diversity. BERT-Large represents a Post-LN model, whereas the rest are Pre-LN-based. This selection enables a comprehensive evaluation of the effects of layer normalization across varying architectures and model scales.

Evaluation Metric: To empirically assess the impact of deeper layers in LLMs, we adopt the Performance Drop metric $\Delta P^{(\ell)}$, inspired by Li et al. (2024b). This metric quantifies the contribution of each layer by measuring the degradation in model performance following its removal. Specifically, it is defined as:

where $P_{\text{original}}$ represents the performance of the unpruned model, and $P_{\text{pruned}}^{(\ell)}$ denotes the performance after removing layer $\ell$. A lower $\Delta P^{(\ell)}$ suggests that the pruned layer plays a minor role in the model’s overall effectiveness. For BERT-Large, we evaluate performance on the SQuAD v1.1 dataset (Rajpurkar, 2016), which measures reading comprehension. For Mistral-7B, LLaMA2-13B, and Qwen-7B, we assess model performance on the MMLU benchmark (Hendrycks et al., 2021), a widely-used dataset for multi-task language understanding.

Figure 2 presents the performance drop ($\Delta P^{(\ell)}$) across different layers for six LLMs, including one Post-LN model (BERT-Large) and five Pre-LN models (Mistral-7B, LLaMA2-13B, Qwen-7B, DeepSeek-7B and LLaMA2-7B).

As shown in Figure 2 (a), pruning deeper layers in BERT-Large leads to a significant decline in accuracy on SQuAD v1.1, while pruning earlier layers has minimal impact. The performance drop $\Delta P^{(\ell)}$ becomes particularly severe beyond the 10th layer, highlighting the crucial role of deeper layers in maintaining overall performance in Post-LN models. In contrast, removing layers in the first half of the network results in negligible changes, indicating their limited contribution to the final output.

However, as shown in Figure 2 (b)-(f), Pre-LN models exhibit a contrast pattern, where deeper layers contribute significantly less to the overall model performance. For instance, as shown in Figure 2 (b) and (c), pruning layers in the last third of Mistral-7B and Qwen-7B results in a minimal performance drop on MMLU, indicating their limited contribution to overall accuracy. In contrast, pruning the first few layers leads to a substantial accuracy degradation, highlighting their crucial role in feature extraction. Similarly, Figure 2 (d) and (e) show that DeepSeek-7B and LLaMA2-7B follow a similar pattern, where deeper layers have little impact on performance, while earlier layers play a more significant role. Finally, as shown in Figure 2 (f), more than half of the layers in LLaMA2-13B can be safely removed. This suggests that as model size increases, the contrast between shallow and deep layers becomes more pronounced, with earlier layers playing a dominant role in representation learning. This observation underscores the need for the community to address the Curse of Depth to prevent resource waste.

This paper primarily focuses on Pre-LN Transformer (Baevski and Auli, 2019; Dai et al., 2019). Let $x_{\ell}\in\mathbb{R}^{d}$ be the input vector at the $\ell$-th layer of Transformer, where $d$ denotes the feature dimension of each layer. For simplicity, we assume all layers to have the same dimension $d$. The layer output $y$ is calculated as follows:

where LN denotes the layer normalization function. In addition, the feed-forward network (FFN) and the multi-head self-attention (Attn) sub-layers are defined as follows:

where $\mathcal{F}$ is an activation function, $\mathrm{concat}$ concatenates input vectors, $\mathrm{softmax}$ applies the softmax function, and $W_{1}\in\mathbb{R}^{d_{\mathrm{ffn}}\times d}$, $W_{2}\in\mathbb{R}^{d\times d_{\mathrm{ffn}}}$, $W_{Qi}\in\mathbb{R}^{d_{\mathrm{head}}\times d}$, $W_{Ki}\in\mathbb{R}^{d_{\mathrm{head}}\times d}$, $W_{Vi}\in\mathbb{R}^{d_{\mathrm{head}}\times d}$, and $W_{O}\in\mathbb{R}^{d\times d}$ are parameter matrices, and $d_{\mathrm{FFN}}$ and $d_{\mathrm{head}}$ are the internal dimensions of FFN and multi-head self-attention sub-layers, respectively. $X\in\mathbb{R}^{d\times s}$, where $s$ is the input sequence length.

The derivatives of Pre-Ln Transformers are:

where $f$ here represents either the multi-head attention function or the FFN function. If the term $\frac{\partial f(\text{LN}(x))}{\partial\text{LN}(x)}\frac{\partial\text{LN}(x)}{\partial x}$ becomes too small, the Pre-LN layer $\frac{\partial\text{Pre-LN}(x)}{\partial x}$ behaves like an identity map. Our main objective is to prevent identity map behavior for very deep Transformer networks. The first step in this process is to compute the variance $\sigma^{2}_{x_{\ell}}$ of vector $x_{\ell}$.

Here, the notation $\Theta$ means: if $f(x)\in\Theta\bigl{(}g(x)\bigr{)}$, then there exist constants $C_{1},C_{2}$ such that $C_{1}\,|g(x)|\leq|f(x)|\leq C_{2}\,|g(x)|$ as $x\to\infty$. The lower bound $\Theta(L)\leq\sigma^{2}_{x_{\ell}}$ indicates that $\sigma^{2}_{x_{\ell}}$ grows at least linearly, while the upper bound $\sigma^{2}_{x_{\ell}}\leq\Theta(\exp(L))$ implies that its growth does not exceed an exponential function of $L$.

Based on Assumption 1 and the work of (Takase et al., 2023b), we obtain the following:

The detailed description of $A$ and $B$, as well as the complete proof, are provided in Appendix A.2.

From Theorem 1, we observe that when the variance grows exponentially, as the number of layers $L\to\infty$, the norm $\left\|\frac{\partial y_{L}}{\partial x_{1}}\right\|_{2}$ is bounded above by a fixed constant $M$. This result implies that even an infinitely deep Transformer remains stable, and by the Weierstrass Theorem, the network is guaranteed to converge. Consequently, this implies that for very large $L$, deeper layers behave nearly as an identity map from $x_{\ell}$ to $y_{\ell}$, thereby limiting the model’s expressivity and hindering its ability to learn meaningful transformations.

This outcome is undesirable, therefore, we would instead prefer the variance to increase more gradually—e.g., linearly—so that $\left\|\frac{\partial y_{L}}{\partial x_{1}}\right\|_{2}$ exhibits linear growth. This observation highlights the necessity of appropriate variance control mechanisms, such as scaling strategies, to prevent excessive identity mappings and enhance network depth utilization.

For Post-LN Transformers, we continue to adopt Assumption 1. In this setting, each layer is followed by a layer normalization (LN) step, ensuring that the variances $\sigma^{2}_{x_{\ell}}$ and $\sigma^{2}_{x^{\prime}_{\ell}}$ remain fixed at 1 across all layers. Consequently, the norm $\left\|\frac{\partial y_{\ell}}{\partial x_{\ell}}\right\|_{2}$ exhibits minimal variation from one layer to the next, indicating stable gradient propagation.

Since the variance is effectively controlled by LN in Post-LN Transformers, an explicit variance‐based analysis becomes less critical. Nonetheless, there remain other important aspects to investigate in deeper Post-LN architectures, such as the evolution of feature mappings and the behavior of covariance kernels over deep layers. These directions will be pursued in future work.

From Lemma 2, we can conclude that our scaling method effectively slows the growth of the variance upper bound, reducing it from exponential to polynomial growth. Specifically, it limits the upper bound to a quadratic rate instead of an exponential one. Based on Theorem 1, after scaling, we obtain the following:

The detailed descriptions of $A$ and $B$, and $\epsilon$, along with the full proof, are provided in Appendices A.3 and A.4.

By comparing Theorem 1 (before scaling) with Theorem 17 (after scaling), we observe a substantial reduction in the upper bound of variance. Specifically, it decreases from exponential growth $\Theta(\exp(L))$ to at most quadratic growth $\Theta(L^{2})$. In fact, this growth is even slower than quadratic expansion, as it follows $\Theta(L^{(2-\epsilon)})$ for some small $\epsilon>0$.

When we select a reasonable upper bound for this expansion, we find that $\left\|\frac{\partial y_{L}}{\partial x_{1}}\right\|_{2}$ no longer possesses a strict upper bound. That is, as the depth increases, $\left\|\frac{\partial y_{L}}{\partial x_{1}}\right\|_{2}$ continues to grow gradually. Consequently, fewer layers act as identity mappings compared to the original Pre-LN where nearly all deep layers collapsed into identity transformations. Instead, the after-scaled network effectively utilizes more layers, even as the depth approaches infinity, leading to improved expressivity and trainability.

To evaluate the effectiveness of LayerNorm Scaling, we follow the experimental setup of Li et al. (2024b), using the same model configurations and training conditions to compare it with widely used normalization techniques, including Post-LN (Nguyen and Salazar, 2019), DeepNorm (Wang et al., 2024), and Pre-LN (Dai et al., 2019). In line with Lialin et al. (2023) and Zhao et al. (2024), we conduct experiments using LLaMA-based architectures with model sizes of 130M, 250M, 350M, and 1B parameters, ensuring consistency in architecture and training settings.

The architecture incorporates RMSNorm (Shazeer, 2020) and SwiGLU activations (Zhang and Sennrich, 2019), which are applied consistently across all model sizes and normalization methods. For optimization, we use the Adam optimizer (Kingma, 2015) and adopt size-specific learning rates: $1\times 10^{-3}$ for models up to 350M parameters, and $5\times 10^{-4}$ for the 1B parameter model. All models share the same architecture, hyperparameters, and training schedule, with the only difference being the choice of normalization method. Unlike Mix-LN (Li et al., 2024b), which introduces an additional hyperparameter $\alpha$ manually set to 0.25, LayerNorm Scaling requires no extra hyperparameters, making it simpler to implement. Table 1 shows that LayerNorm Scaling consistently outperforms other normalization methods across different model sizes. While DeepNorm performs comparably to Pre-LN on smaller models, it struggles with larger architectures like LLaMA-1B, showing signs of instability and divergence in loss values. Similarly, Mix-LN outperforms Pre-LN in smaller models but faces convergence issues with LLaMA-350M, indicating its sensitivity to architecture design and hyperparameter tuning due to the introduction of Post-LN. Notably, Mix-LN was originally evaluated on LLaMA-1B with 50,000 steps (Li et al., 2024b), while our setting extends training to 100,000 steps, where Mix-LN fails to converge, highlighting its instability in large-scale settings caused by the usage of Post-LN.

In contrast, LayerNorm Scaling solves the Curse of Depth without compromising the training stability thanks to its simplicity. LayerNorm Scaling achieves the lowest perplexity across all tested model sizes, showing stable performance improvements over existing methods. For instance, on LLaMA-130M and LLaMA-1B, LayerNorm Scaling reduces perplexity by 0.97 and 1.31, respectively, compared to Pre-LN. Notably, LayerNorm Scaling maintains stable training dynamics for LLaMA-1B, a model size where Mix-LN fails to converge. These findings demonstrate that LayerNorm Scaling provides a robust and computationally efficient normalization strategy, enhancing large-scale language model training without additional implementation complexity.

Comparison with Other Layer Normalization. In addition, we conducted comparisons using LLaMA-130M to evaluate LayerNorm Scaling against recently proposed normalization methods, including Admin (Liu et al., 2020), Sandwich-LN (Ding et al., 2021), and Group-LN (Wu and He, 2018; Ma et al., 2024). Table 2 shows that Admin and Group-LN degrade performance. Sandwich-LN slightly outperforms Pre-LN. Both Mix-LN and LayerNorm Scaling improve over Pre-LN by good margins. However, Mix-LN fails to reduce perplexity under 26, falling short of LayerNorm Scaling.

We believe that LayerNorm Scaling allows deeper layers in LLMs to contribute more effectively during supervised fine-tuning by alleviating gradient vanishing associated with increasing depth. Compared to models trained with Pre-LN, the deeper layers with LayerNorm Scaling maintain stable output variance, preventing uncontrolled growth and ensuring effective feature representation. As a result, deeper layers contribute more effectively to feature transformation, enhancing representation learning and improving generalization on complex downstream tasks.

To verify this, we follow the fine-tuning methodologies in Li et al. (2024b) and Li et al. (2024a), applying the same optimization settings as pre-training. We fine-tune models from Section 5.1 on the Commonsense170K dataset (Hu et al., (Hu et al., 2023)) across eight downstream tasks. The results, presented in Table 3, demonstrate that LayerNorm Scaling consistently surpasses other normalization techniques in all evaluated datasets. For the LLaMA-250M model, LayerNorm Scaling improves average performance by1.80% and achieves a 3.56% gain on ARC-e compared to Mix-LN. Similar trends are observed with the LLaMA-1B model, where LayerNorm Scaling outperforms Pre-LN, Post-LN, Mix-LN, and DeepNorm on seven out of eight tasks, with an average gain of 1.86% over the best baseline. These results confirm that LayerNorm Scaling, by improving gradient flow and deep-layer representation quality, achieves better fine-tuning performance, demonstrating robustness and enhanced generalization on diverse downstream tasks.

As LayerNorm Scaling aims to reduce output variance, we validate this by comparing it with two scaling approaches: LayerScale (Touvron et al., 2021) and Scaled Initialization (Shoeybi et al., 2020). LayerScale applies per-channel weighting using a diagonal matrix, diag$(\lambda_{1},\ldots,\lambda_{d})$, where each weight $\lambda_{i}$ is initialized to a small value (e.g., $\lambda_{i}=\epsilon$). Unlike LayerNorm Scaling, LayerScale learns the scaling factors automatically, which does not necessarily induce a down-scaling effect. Scaled Initialization scales the initialization of $W_{0}$ and $W_{2}$ to small values by $\frac{1}{\sqrt{2L}}$ where $L$ is the total number of transformer layers. Since scaling is applied only at initialization, we argue that Scaled Initialization may not effectively reduce variance throughout training. We further verify this in Figure 1, where we can see the output variance of Scaled Initialization is as large as Pre-LN. Table 4 presents the results of LLaMA-130M and LLaMA-250M. First, we can see that LayerScale degrades performance. While Scaled Initialization slightly improves over Pre-LN, it falls short of LayerNorm Scaling and the gap becomes larger for the larger model.

To evaluate how LayerNorm Scaling improves deep layer effectiveness, we conduct a layer pruning experiment on LLaMA-130M, systematically removing individual layers and measuring the performance drop ($\Delta P^{(\ell)}$) on the ARC-e benchmark (Clark et al., 2018). Figure 4 compares the pruning effects between standard Pre-LN and LayerNorm Scaling. In the Pre-LN, removing deep layers results in minimal performance degradation, indicating their limited role in representation learning. In contrast, with LayerNorm Scaling, pruning deeper layers leads to a more pronounced drop, suggesting they now play a more active role in learning. While early layers remain critical in both models, the performance degradation in the LayerNorm Scaling is more evenly distributed across layers, reflecting a more balanced learning process. These findings confirm that LayerNorm Scaling mitigates the Curse of Depth by ensuring deeper layers contribute effectively to training.