Fractional Chern Insulator in Twisted Bilayer MoTe₂

Introduction.—Fractional Chern insulators (FCI), the analog of the fractional Quantum Hall effect [2, 3] in lattice systems, feature fractional excitations with anyonic statistics [4, 5, 6, 7]. Due to its exotic nature and potential applications in topological quantum computing [8, 9], the FCI has long been sought after in condensed matter experiments. To find FCI, the widely-accepted approach is to find a Chern band with significantly quenched kinetic energy [10, 11, 12, 13, 14]. This can be realized in two-dimensional (2D) moiré superlattices, which have been shown to be a fruitful and tunable platform to explore electronic correlation [15, 16]. Indeed, theoretical proposals of FCI have been put forward in twisted bilayer graphene [17, 18, 19, 20, 21, 22] and twisted bilayer transition metal dichalcogenide (TMD) [23, 24, 25]. Experimental evidences for FCI have also been found in graphene-based superlattices [26, 27], albeit at a finite magnetic field. Recently, the first observation of the FCI in the absence of magnetic field (i.e., the fractional quantum anomalous Hall effect) has been reported in twisted bilayer MoTe₂ at hole fillings $\nu=-2/3$ and $\nu=-3/5$ [1]. The observation is at an unexpected large twist angle ($\sim 3.7^{\circ}$), for which the Chern bands had been commonly believed to be too dispersive to stabilize the FCI. The experimental observation calls for a better understanding of single-particle band structure of twisted TMD bilayer as well as the emergence of the FCI in real materials.

In this work, we perform large-scale density functional theory (DFT) calculation for the twisted bilayer MoTe₂. In contrast to previous theoretical studies, we find that the band structure features an isolated flat Chern band that favors FCI. By exact diagonalization (ED) calculations, we confirm the existence of FCI at $\nu=-2/3$. We also find that the ferromagnetism at $\nu=-1/3$ is much weaker than that at $\nu=-2/3$, explaining the absence of the $\nu=-1/3$ FCI in the experiment. We further investigate the fate of FCI under an external electric field and find that FCI become unstable at $E=1.26$ mV/Å, which is consistent with experimental observation. The suppression of FCI by external electric field is attributed to the deterioration of the flatness of band geometry. We establish phase diagrams with respect to the twist angle and electronic interaction, revealing an optimal twist angle of $3.5^{\circ}$ for the observation of FCI. Finally, we show evidences of the $\nu=-3/5$ FCI state in this system. Our research highlights the importance of accurate single particle band structure in the quest for strong correlated electronic states and provides insights into engineering FCI in moiré superlattices.

Single-particle electronic structure.— Moiré superlattices formed by twisting a bilayer introduce a long wavelength periodic structure characterized by the moiré lattice constant $a_{M}=a/\theta$, where $a$ is the lattice constant of the original 2D layer and $\theta$ is the twist angle. This large length scale makes it possible to model the low-energy electronic structure with a continuum model. The valence band edge of monolayer MoTe₂ is located at the corners of its Brillouin zone, i.e., $K$ and $K^{\prime}$ points. The two points are separated by a large momentum such that $K$ and $K^{\prime}$ valleys can be considered independently. Following the experiment [1], we consider $R$ stacking twisted bilayer and the continuum model Hamiltonian for $K$ valley reads [28, 29]

Here, $H_{b/t}=-\hbar^{2}\left(\bm{k}-\bm{K}_{b/t}\right)^{2}/2m^{*}+\Delta_{b/t}(\bm{r})$ is the bottom ($b$) and top ($t$) layer Hamiltonian subjected to a moiré potential $\Delta_{b/t}(\bm{r})=2v\sum_{j=1,3,5}\cos\left(\bm{G}_{j}\cdot\bm{r}\pm\psi\right),$ where the bottom (top) layer corresponds to the positive (negative) sign. $\bm{K}_{b/t}$ is the $K$ point for the bottom and top layer and $\bm{G}_{j}$ is the moiré reciprocal lattice vectors defined by $\bm{G}_{j}=\frac{4\pi}{\sqrt{3}a_{M}}(\cos(\frac{\pi(j-1)}{3}),\sin(\frac{\pi(j-1)}{3}))$. The interlayer tunneling is dictated by three-fold rotational symmetry as $\Delta_{T}(\bm{r})=w\left(1+e^{-i\bm{G}_{2}\cdot\bm{r}}+e^{-i\bm{G}_{3}\cdot\bm{r}}\right)$. $m^{*}=0.6m_{e}$ is the effective mass ($m_{e}$ is the bare electron mass) and $(v,\psi,w)$ are the free parameters in the continuum model. The continuum Hamiltonian for the $K^{\prime}$ valley can be obtained by applying time reversal symmetry to $\mathcal{H}_{K}$. Inside each valley, the electrons are fully spin polarized due to the large spin-valley coupling, and opposite valleys have opposite spin due to time-reversal symmetry [30, 31].

The parameters can be fixed by various approaches, among which the simplest one is to fit from DFT calculations for bilayer MoTe₂ at various stackings. This approach is adopted by Ref. 28 and the parameters are reproduced in Table 1. The valence band structure with this set of parameters is shown in Fig. 1(b) at twist angle $3.89^{\circ}$. The topmost valence band is dispersive with band width larger than 20 meV. In addition, the two topmost bands overlap each other in energy. Both features are unfavorable for the emergence of FCI.

In this work, we seek to establish a better understanding of the single particle band structure by performing large-scale DFT calculations to take into account atomic relaxation, layer corrugation and interlayer electric polarization (details in the Supplemental Material [32]). We choose the closest commensurate twist angle (3.89^∘) to the experimental value and construct the moiré superlattice of MoTe₂ using its monolayer unit cell with the optimized lattice constant $a=3.52$ Å. The band structure of the moiré superlattice is presented in Fig. 1(a) as red dots. The DFT result shows significant lattice reconstruction in both in-plane and out-of-plane direction [Fig. 1(c-d)]. We then fit the continuum model parameters to the DFT band structure, and the result is presented in Table 1. Compared to the parameters from Ref. 28, our parameters features a much larger moiré potential and interlayer tunneling, which is likely caused by the significant lattice reconstruction [33, 34, 35], resulting in an isolated Chern band with band width of roughly 9 meV [cf. Fig. 1(a)].

Fractional Chern insulator at $\nu=-2/3$.—Having established the existence of an isolated, relatively flat Chern band, we investigate whether FCI can be stabilized. We adopt the following form of the Coulomb interaction:

where $V(\bm{q})=e^{2}\text{tanh}(|q|d)/2\epsilon_{0}\epsilon|q|$ is the Coulomb interaction with dual-gate screening, $A$ is the area of the system (proportional to the number of $k$ points in the calculations), $d$ is the distance between the twisted bilayer MoTe₂ and two symmetric metal gates, $\epsilon_{0}$ is the vacuum permittivity and $\epsilon$ is the relative dielectric constant. Here, $c_{l\tau\bm{k}}^{\dagger}$ creates a plane wave with momentum $\bm{k}$ at valley $\tau$ and layer $l$. Due to spin-valley locking, $\tau$ can also be understood as the spin label. We project the interaction onto the topmost moiré band and carry out ED calculations. We choose $\epsilon=15$ to make the characteristic interaction strength smaller than the averaged energy gap. For smaller $\epsilon$, we present the phase diagram in the Supplemental Material [32]. While we left a more accurate treatment to include the band mixing for future studies, there are evidences that FCI can still be stablized even when the interaction exceeds the band gap [36, 37].

The precursor to the FCI is spontaneous time-reversal symmetry breaking. We first perform ED calculations taking both valleys into account with a system size of $3\times 4$ unit cells. We find that over a broad range of twist angles, the ground state for both $\nu=-1/3$ and $\nu=-2/3$ is fully valley-polarized, with holes occupying only one valley [cf. Fig. 2(a)]. Since the spin and valley indices are coupled, full valley polarization implies full spin polarization. The spin gap, defined as the energy difference between the lowest-energy state that does not exhibit full valley-polarization and the fully valley-polarized ground state, is shown in Fig. 2(b) for both $\nu=-1/3$ and $\nu=-2/3$. The spin gap for $\nu=-1/3$ is much smaller than that of $\nu=-2/3$, indicating much weaker ferromagnetism of the former. The difference in the spin gap is consistent with the experimental observation that ferromagnetism appears at $\sim$4.5 K at $\nu=-2/3$, whereas no ferromagnetism is observed at $\nu=-1/3$ down to base temperature of 1.6 K [38].

Given the large spin gap and strong ferromagnetism at $\nu=-2/3$, we further carry out ED calculations for a single valley, which allows us to consider a larger system with $4\times 6$ unit cells.

The most important signature of the FCI is the ground state degeneracy when the system is put on a torus [39, 40]. Indeed, the ED energy spectrum shows three nearly degenerate states, separated by an energy gap from other states, as shown in Fig. 2(c). Under flux insertion, the three ground states evolve into each other, exhibiting a $6\pi$ periodicity [Fig. 2(d)]. We also calculate many-body Chern number [39] at this filling to be $-2/3$, which is consistent with the experimental observation [1]. The single particle occupation number, defined as $n(\bm{k})=\langle c_{\bm{k}}^{\dagger}c_{\bm{k}}\rangle$ is presented in the Supplemental Material [32]. The uniformity of $n(\bm{k})$ is a strong indicator favoring FCI over charge density wave (CDW) states, one of FCI’s competing phases. The above evidences provide strong evidence to the existence of FCI in twisted MoTe₂ at $\nu=-2/3$.

Phase diagram.—The emergence of the FCI state depends on the dominance of electron-electron interaction energy over single particle kinetic energy. This ratio between the two energy scales can be adjusted by two factors: the single-particle band width, and the environmental dielectric screening of electron-electron interactions. The most experimentally accessible knob to tune the band width is changing the twist angle. For example, in twisted bilayer graphene, flat band emerges at a series of magic angles [41]. For twisted bilayer TMD systems, the band width is less sensitive to twist angles and we find relative isolated flat bands for twist angle $3^{\circ}-4^{\circ}$. The screening of the electron-electron interactions can be tuned by changing $\epsilon$, as well as the distance between the sample and the metal gate $d$: larger $d$ leads to weaker screening of the electron-electron interactions.

The phase diagram, as a function of $\theta$ and either $\epsilon$ or $d$, is presented in Fig. 3. Within the range of the twist angle presented in Fig. 3, the band width of the topmost valence band increases monotonically with $\theta$. There are three distinct phases: the FCI phase, the valley-polarized (VP) phase, and the non-valley-polarized (NVP) phase. The NVP phase emerges in the weak electron-electron interaction regime, characterized by large $\theta$ and large $\epsilon$ [see Fig.3(a)], or small $d$ [see Fig.3(b)]. For stronger interactions, all holes occupy the same valley, which is shown as the VP phase in Fig.3. Our numerical evidence suggests that the VP states are most likely metal states with fully polarized spin, meaning they represent a half-metal state [24]. The nature of these VP phases is left for further studies. FCI emerges from the VP phases at even stronger interactions. The many-body gap for FCI phase has a peak at $\theta=3.5^{\circ}$, indicating an optimal twist angle for the observation of FCI. This optimal angle is close to the twist angle ($\sim 3.7^{\circ}$) of the device in which FCI is observed [1].

The effect of electric field.—An out-of-plane electric field generates potential differences between the top and bottom layers. Experimentally, it is observed that the FCI at $\nu=-2/3$ can be suppressed by this out-of-plane electric field. This observation is not necessarily surprising, since layer potential differences will induce a topological phase transition at the single-particle level, making the topmost valence band topologically trivial. However, our calculations show that FCI is suppressed well before the single-particle topological phase transition. In Fig. 4(a), we show that the many-body gap closes at electric field $E=1.26$ meV/Å. In Fig. 4(b), we present the single particle band structure for $E=1.39$ meV/Å, where an isolated flat Chern band can still be observed. The band width of the Chern band at $E=1.39$ meV/Å is comparable to that at $E=0.0$ meV/Å, but the FCI is already destroyed. In experiment, the ferromagnetism disappears at $E\sim 5$ meV/Å [1], implying FCI is destroyed at a smaller electric field. Our critical electric field $E=1.26$ meV/Å is consistent with this observation.

It is well-known that completely quenched kinetic energy (i.e. vanishing band width) does not ensure the existence of FCI, and a number of proposals [42, 43, 44, 45, 46] are put forward to identify the conditions for FCI to emerge. Many of these proposals aim to design wave functions in a flat Chern band such that they closely resemble the wave functions of a Landau level. For example, since the Berry curvature $\Omega$ and quantum metric tensor $g$ is constant for Landau levels, the flatness of these two quantities [42, 43, 44] in the reciprocal space is heuristically viewed as a promising indicator for the emergence of FCI. In Fig. 4(c), we present the distributions of $\Omega$ and $\mathrm{tr}(g)$ for $E=0.0$ meV/Å and $E=1.39$ meV/Å. A serious deterioration of the flatness of the Berry curvature and quantum metric tensor can be observed when the FCI is destroyed. Both $\Omega$ and $\mathrm{tr}(g)$ at $E=1.39$ meV/Å are concentrated at the moire $K$ valley [Fig. 4(c)], which explains the suppression of the FCI state.

Fractional Chern insulator at $\nu=-3/5$.—In addition to the $\nu=-1/3$ state, the experimental observation also includes the $\nu=-3/5$ state [1]. However, in our calculations, we do not observe a clear many-body gap at $\nu=-3/5$ for the dielectric constant of $\epsilon=15$. Nonetheless, when the dielectric constant is increased to $\epsilon=8$, which is closer to the experimental value, we find evidence for the FCI [32]. The characteristic interaction at $\epsilon=8$ is larger than the energy gap between the two topmost bands, but the result can still be viewed as supporting evidences of the existence of FCI at $\nu=-3/5$. We also performed ED calculations for $\nu=-1/5$, $-2/5$ and $-4/5$ and find no clear evidence of FCI for $\nu=-1/5$ and $-4/5$. For $\nu=-2/5$, we find evidence of a FCI phase with the many-body energy gap roughly half of that at $\nu=-3/5$.

In summary, we have presented a comprehensive theoretical study of the recent experimental observation of fractional Chern insulator at zero magnetic field in a twisted MoTe₂ bilayer. Density functional theory calculations reveal the existence of an isolated flat Chern band, which allows us to confirm the existence of a $\nu=-2/3$ fractional Chern insulator state using exact diagonalization. Phase diagrams are presented to guide future experimental study of fractional Chern insulator in this system, and the suppression of fraction Chern insulator by external electric field is studied. Our findings offer valuable insights into the nature and properties of fractional Chern insulators in moiré superlattices.

Note added.—We recently became aware of an independent work on similar topics [47].