Helical superfluid in a frustrated honeycomb Bose-Hubbard model

Frustrated systems exhibit novel emergent phenomena as they often host extensive degeneracy that is sensitive to even weak perturbations Moessner and Chalker (1998); Balents (2010). This manifests itself in rich low-temperature phenomenology, as in the frustrated magnetism, which commonly remains disordered down to temperatures low compared to the Curie-Weiss temperature set by the exchange interaction Ramirez et al. (1990); Gaulin et al. (1992); Ramirez (1994); Uemura et al. (1994); Dunsiger et al. (1996). At even lower temperature, generically order develops through the so-called “order-by-disorder” process Villain et al. (1980); Henley (1989), where fluctuations (e.g., entropically) select ordered states among many competing nearly free-energetically degenerate phases. This contrasts with an even more exotic possibility of a putative quantum liquid, that fails to order down to zero temperature, as believed to be exemplified by some 2d Kagome magnetic materials with a flat-band dispersion Sachdev (1992); Hastings (2000); Ran et al. (2007); Hermele et al. (2008); Evenbly and Vidal (2010); Yan et al. (2011); Depenbrock et al. (2012); Jiang et al. (2012); Iqbal et al. (2013, 2014); He et al. (2014); Gong et al. (2015); Norman (2016); Mei et al. (2017); He et al. (2017); Zhu et al. (2018).

Another rich class of “codimension-one” frustrated systems Bergman et al. (2007); Lee and Balents (2008); Mulder et al. (2010); Matsuda et al. (2010); Gao et al. (2017); Biswas and Damle (2018); Niggemann et al. (2019); Shimokawa and Kawamura (2019); Gao et al. (2020); Liu et al. (2020); Balla et al. (2019, 2020); Yao et al. (2021); Bordelon et al. (2021); Gao et al. (2022), characterized by a bare dispersion that is nearly degenerate along $d-1$ dimensions ($d$ the spatial dimension), develop even in the bipartite lattice materials, frustrated by competing interactions. Examples include spinel materials such as $\textrm{MnSc}_{2}\textrm{S}_{4}$ on 3d diamond lattice Bergman et al. (2007); Lee and Balents (2008); Gao et al. (2017); Bordelon et al. (2021) and $\textrm{FeCl}_{3}$ on layered honeycomb lattice Gao et al. (2022). Theoretically, they are well described by classical spin models with nearest-neighbor and next-nearest-neighbor exchange interactions. Within certain regime of frustration, the dispersion minima form a degenerate manifold with codimension-one in the reciprocal space. However, at nonzero temperature a set of wavevectors is selected entropically based on their lowest free energy.

The phenomenon of order-by-disorder, however, is elusive in magnetic systems, often obscured by further-neighbor interactions, which instead can lead to a low-temperature multi-step ordering sequence of phases Gao et al. (2017). Alternatively, the codimension-one bare dispersion and the associate phenomenology can be realized in bosonic systems with frustrated hoppings Varney et al. (2012); Sedrakyan et al. (2014) or Rashba spin-orbital coupling Sedrakyan et al. (2012), engineered in a controlled way in cold atom experiments Wintersperger et al. (2020); Zhai (2015). More exotic constructions involve ring exchange interactions that induce an emergent Bose metal Paramekanti et al. (2002); Tay and Motrunich (2011).

It is thus of interest to elucidate the nature of a state that emerges from a nonzero density of interacting frustrated bosons condensed at nonzero momentum on a dispersion minimum contour, and in particular to explore its stability to a putative quantum liquid state Sur and Yang (2019); Lake et al. (2021). To this end, we take two complementary approaches, a paradigmatic two-dimensional Bose-Hubbard model on honeycomb lattice and a continuum field theory that we derive from it. The model is frustrated by competing nearest-neighbor and next-nearest-neighbor hoppings, $t_{1}$ and $t_{2}$, and thereby (for a broad range of parameters) exhibits a bare dispersion with a minimum on a closed contour at nonzero momentum $k_{0}$ set by the hopping matrix elements. We thus explore in detail the rich phenomenology of the resulting helical superfluid state. In particular, as summarized in detail in Sec. II, we analyze the helical superfluid within the lattice Bogoliubov approximation, computing its dispersion, condensate depletion, momentum distribution, structure factor, and equation of state, all of which different qualitatively from that of a conventional superfluid. We complement this honeycomb lattice analysis by deriving from it (guided by generalized dipolar symmetry Gromov (2019); Radzihovsky (2020); Yuan et al. (2020); Gorantla et al. (2022); Lake et al. (2022)) a superfluid smectic field theory Radzihovsky (2011, 2020) to more generally explore the properties of the helical condensate. By going beyond the Bogoliubov approximation, we analyze the quantum and thermal order-by-disorder phenomenon that sets in at low energies and leads to a crossover to a more conventional but highly anisotropic XY superfluid.

The outline of this paper is as follows. In Sec II, we give a summary of our primary results. In Sec. III, we study a model of bosons hopping on a frustrated honeycomb lattice and its superfluidity within the Bogoliubov approximation. In Sec. IV, we construct a smectic field theory for the helical superfluid, and use it to study its stability to quantum and thermal fluctuations and its properties in the isotropic continuum, thereby complementing the microscopic lattice model analysis in Sec III. Sec. V is devoted to the quantum and thermal order-by-disorder phenomenon, that reduces the degeneracy of the dispersion minimum contour down to a discrete set of six minima, and thereby stabilizes the helical superfluid state. We conclude in Sec. VI with a discussion of results in the contexts of current experiments and remaining open questions.

Before turning to a detailed analysis, we summarize the key results of this work. We study a Bose-Hubbard model with frustrated nearest-neighbor (NN) and next-nearest-neighbor (NNN) hoppings, $t_{1}$ and $t_{2}$. For intermediate frustration $1/6<\rho\equiv t_{2}/t_{1}<1/2$, its band structure features a closed contour minimum centered around the $\Gamma$ point (${\bf k}=0$). We then consider a bosonic condensate with a macroscopic wavefunction $\Phi=\sqrt{n_{0}+\pi}e^{i{\bf k}_{0}\cdot{\bf r}+i\phi}$ at a single momentum ${\bf k}_{0}$ on the contour minimum – a “helical condensate”, where $n_{0}$ is the condensate density, $\phi$ and $\pi$ are the phase and density fluctuations, respectively. As illustrated in Fig. 1, the helical superfluid is a stable phase on the high symmetry points of the contour with its transition temperature to the normal state (below that of a conventional superfluid in the unfrustrated regime $\rho<1/6$) strongly suppressed by quantum and thermal fluctuations. Quantum Lifshitz transitions at $\rho=1/6,1/2$¹¹1The quantum Lifshitz transition at $\rho=1/2$ only happens at the vertices of the hexagonal contour (see Fig. 5), which is energetically stable at one-loop order as shown in Fig. 1(b). Higher-order fluctuations may shift the positions of the Lifshitz transitions, or make the transition at $\rho=1/2$ split into two Lifshitz points characterized by smectic-like Goldstone modes with Lagrangian (44b), as allowed by the $\text{C}_{\text{6}}$ lattice symmetry. are characterized by “soft” Goldstone modes that exhibit an anisotropic quartic dispersion, which suppresses the transition temperature to zero.

As summarized in Fig. 2, at zero temperature ($T=0$) and on scale shorter than quantum order-by-disorder length $\lambda_{\perp Q}$, the highly anisotropic superfluid exhibits an unconventional smectic-like Goldstone mode (GM) with a low-energy dispersion [for full dispersion, see Eq. (31)]

where the subscript $\parallel$ ($\perp$) denotes the direction that is parallel (perpendicular) to condensate momentum ${\bf k}_{0}$, located at the high symmetry points of the dispersion contour. At temperatures higher than the interaction $U$, the crossover scale is modified to $\lambda_{\perp T}$. On longer scales the so-called order-by-disorder sets in, driven by quantum, thermal and lattice effects, and the system crossovers to a more conventional superfluid with highly anisotropic linear dispersion. In the weakly-interacting limit, the order-by-disorder length is long, given by

To further explain the order-by-disorder phenomenology (discussed in detail in Sec. V), we first note that the thermodynamic potential of the helical state near the condensate momentum ${\bf k}_{0}$ [see the right panel of Fig. 2(a)] exhibits a property

where the coefficients $b({\bf k}_{0})$ and $b_{4}({\bf k}_{0})$ are dominated by the bare dispersion (see Appendix B), while $b_{\perp}({\bf k}_{0})$ is generated perturbatively by the lattice order-by-disorder via interaction $U$. In the above, $N_{0}$ is the condensate number. The corresponding low-energy Goldstone mode Lagrangian density is given by

which when treated at a quantum level leads to the dispersion (1) for $a\ll q_{\perp}^{-1}\ll\lambda_{\perp}=\sqrt{K/B_{\perp}}$ ($a$ the lattice constant). We observe that a difference choice of the helical condensation, ${\bf k}_{0}\to{\bf k}_{0}+{\bf q}$, equivalently corresponds to a linear in ${\bf r}$ phase rotation, $\phi\to\phi+{\bf q}\cdot{\bf r}$. By applying the two transformations to the thermodynamic potential and the Goldstone mode Hamiltonian respectively, and requiring the shifted energies to be identical, we obtain the following Ward identities,

We expect the relations (7) to hold at all order of a perturbation theory (see Sec. V). In particular, at the zeroth order, the thermodynamic potential [see the left panel of Fig. 2(a)] is featured by a degenerate contour minimum, and thus $b_{\perp}=B_{\perp}=0$. At one-loop (first) order, the correction to the thermodynamic potential is given by the zero-point energy of the Bogoliubov quasiparticles together with entropic contributions, which in turn gives a $b_{\perp}\propto U^{5/4}$ ($b_{\perp}\propto U^{1/4}T$) for $T\ll Un_{0}$ ($T\gg Un_{0}$) in the weakly-interacting limit. We also perform a complementary calculation of $B_{\perp}$ verifying (7). In contrast, while $b_{4}=K$ at zeroth-order, the above symmetry does not constrain it to hold generically.

The Goldstone mode theory (6) predicts the low-energy properties of the helical superfluid. In particular, the off-diagonal order of the superfluid $\langle\Psi^{\ast}({\bf r})\Psi(0)\rangle\propto e^{-C({\bf r})}$, where $C({\bf r})=\langle[\phi({\bf r})-\phi(0)]^{2}\rangle$. As shown in Fig. 2(b), the two-point correlation function $C({\bf r})$ exhibits several qualitatively distinct limits. Here we focus on the behavior along the $\perp$ direction, leaving detailed discussions in Sec. IV.1. At zero temperature (see the blue-dotted curve), for either $B_{\perp}=0$ or $B_{\perp}\neq 0$, $C(r_{\parallel}=0,r_{\perp}\to\infty)\to\text{const}$, indicating a long-range order of the helical superfluid. At nonzero temperature, as illustrated by the black (red-dashed) curve, $C(0,r_{\perp}\to\infty)\sim\log(r_{\perp})$ ($\sim r_{\perp}$) for $B_{\perp}\neq 0$ ($B_{\perp}=0$), which signifies a quasi-long-range (short-range) order as a consequence of the XY (smectic) GM fluctuations. For a general case $T\neq 0$ and $B_{\perp}\neq 0$, there are two important crossover scales, $r_{\perp T}$ and $\lambda_{\perp}$. The former separates the low-temperature quantum and higher-temperature classical regimes, while the latter, as discussed above, separates the short-distance smectic and long-distance XY regimes.

Consequently, the helical state is characterized by an anisotropic XY superfluidity at the longest scale. However, within a large (for small $U$) order-by-disorder crossover scale, $(r_{\parallel}^{*},r_{\perp}^{*})=(\lambda_{\perp}^{2}/\xi,\lambda_{\perp})$ ($\xi$ the coherence length), the system exhibits soft ($B_{\perp}\approx 0$) quantum and thermal fluctuations, with its physical properties qualitatively distinct from that of a conventional superfluid. Abstracting from the Bose-Hubbard lattice model, we develop a field theory that captures the key qualitative features of the helical superfluid within this regime, and calculate a number of its physical observables, with the zero-temperature results summarized in Table 1. At nonzero temperature, thermal fluctuations set in at scales beyond $(r_{\parallel T},r_{\perp T})=(\frac{Un_{0}}{T}\xi,\sqrt{\frac{Un_{0}}{T}}\sqrt{\lambda\xi})$ ($\lambda=\sqrt{K/B}$), as illustrated in Fig. 2(b). Such quantum-to-classical crossover is observable in real space- or momentum-resolved quantities, as exemplified by the structure factor in Sec. IV.2.1. We now turn to detailed analysis that leads to the results enumerated above.

To explore the phenomenon of nonzero momentum helical superfluidity, we study a frustrated Bose-Hubbard model on honeycomb lattice with Hamiltonian $H=H_{0}+H_{\text{int}}$, where

is the usual on-site interactions with the subscripts $i$ and $s$ labeling the Bravais lattice sites and the two sublattices respectively. $n_{i,s}=a^{\dagger}_{i,s}a_{i,s}$ is the boson number operator with $a^{\dagger}_{i,s}$ and $a_{i,s}$ the corresponding creation and annihilation operators. The kinetic part of the Hamiltonian is given by

where $t_{1}$ and $t_{2}$ are the nearest-neighbor (NN) and the next-nearest-neighbor (NNN) hopping parameters, respectively. We focus on the case of $t_{1}>0$, taking the advantage of the symmetry of the model: $t_{1}\rightarrow-t_{1}$, $a_{2}\rightarrow-a_{2}$. In contrast to the NN hopping, we take the NNN hopping, $-t_{2}$, to be “frustrated antiferromagnetic”, $-t_{2}<0$. This has been engineered in atomic systems in an optical lattice through Floquet techniques Wintersperger et al. (2020). Throughout our analysis below, we use dimensionless parameter $\rho=t_{2}/t_{1}>0$ to quantify the frustration in this model.

We now turn to a complementary continuum field-theoretic description of the helical superfuild state, characterized by Bose condensation on a closed contour minimum in momentum space. In the isotropic case, neglecting lattice effects, this physics can be encoded through a quartic dispersion

with minima on a contour ${\bf k}^{2}=\bar{k}_{0}^{2}$. As we demonstrated in Sec. III, such dispersion is indeed realized for bosons on a honeycomb lattice for $\rho\approx 1/6^{+}$, with

We can encode this physics in a noninteracting Hamiltonian $H_{0}=\int_{\bf r}\hat{\Phi}_{\bf r}^{\dagger}\hat{\varepsilon}_{{\bf r}}\hat{\Phi}_{\bf r}$ with

We then extend this to an interacting field theory and study it in a grand canonical ensemble, with the corresponding imaginary-time action $S=\int d^{2}rd\tau(\mathcal{L}_{0}+\mathcal{L}_{1})$,

$\Phi({\bf r},\tau)$ a complex bosonic field and $\mu$ the chemical potential. We also defined an energy offset $\tilde{\varepsilon}_{0}=J\bar{k}_{0}^{4}+\varepsilon_{0}$ and interaction $U$ parameter inherited from the Hubbard model.

As with the lattice model in Sec. III, here we focus on the helical superfluid state – a condensate at a single point ${\bf k}_{0}$ on the dispersion contour minimum – encoded in $\varepsilon_{\bf k}$, (38), and $\mathcal{L}_{0}$ in (41). In the density-phase representation, the state is characterized by

with a nonzero condensate density (at mean-field level) $n_{0}=(\mu-\varepsilon_{0})/U$ and momentum $k_{0}=\bar{k}_{0}$, for $\mu>\varepsilon_{0}$ (vanishing otherwise), and density and phase fluctuations, $\pi$ and $\phi$, respectively. To quadratic order, the helical superfluid is then described by a Goldstone mode Lagrangian density ⁶⁶6 Each term in Eq. (41) can be rewritten in terms of $\pi$ and $\phi$ as $\displaystyle\partial_{\tau}\Phi=$ $\displaystyle\frac{e^{i{\bf k}_{0}\cdot{\bf r}+i\phi}}{\sqrt{n}}\biggl{(}\frac{1}{2}\partial_{\tau}\pi+in\partial_{\tau}\phi\biggr{)},$ $\displaystyle\nabla\Phi=$ $\displaystyle\frac{e^{i{\bf k}_{0}\cdot{\bf r}+i\phi}}{\sqrt{n}}\biggl{[}\frac{1}{2}\nabla\pi+in({\bf k}_{0}+\nabla\phi)\biggr{]},$ $\displaystyle\nabla^{2}\Phi=$ $\displaystyle\frac{e^{i{\bf k}_{0}\cdot{\bf r}+i\phi}}{n^{3/2}}\biggl{[}\frac{1}{2}n\nabla^{2}\pi-n^{2}({\bf k}_{0}+\nabla\phi)^{2}-\frac{1}{4}(\nabla\pi)^{2}$ $\displaystyle+in(\nabla\pi\cdot{\bf k}_{0}+\nabla\pi\cdot\nabla\phi+n\nabla^{2}\phi)\biggr{]}.$

where the subscripts $\parallel$ and $\perp$ denote axes parallel and perpendicular to ${\bf k}_{0}$, respectively. In Sec. V, we will consider higher-order fluctuations that renormalize the parameters $n_{0}$, ${\bf k}_{0}$ and reduce the symmetry of the Lagrangian by $C_{6}$ lattice effects. We note that the phase $\phi$ is compact with $2\pi$ periodicity, and thereby allows vortex configurations – dislocation in the helical pattern of the state. However, because we are primarily interested in the ordered helical superfluid at zero temperature, where these defects are confined, we will neglect them in most of our analysis. They are however important for the finite temperature Kosterlitz-Thouless superfluid-to-normal transition in Fig. 1(a).

Integrating out $\pi$ field in Eq. (43), we obtain the effective theory of $\phi$ described by a quantum smectic Lagrangian (consistent with the honeycomb lattice model analysis in Sec. III)

where Fourier transform of the operators $\hat{\mathcal{E}}$ and $\hat{\mathcal{E}}_{2}$ are given by

with the couplings⁷⁷7At $T=0$ and weak interaction, we neglect the difference between the boson and condensate densities, $n$ and $n_{0}$, with $(n-n_{0})/n\sim U^{3/4}$, see Eq. (83).

The final form in Eq. (44b) corresponds to the asymptotic long wavelength limit. The superfluid mode $\phi$ exhibits an unusual dispersion given by

consistent with our lattice analysis, Eq. (31) and the features illustrated in Fig. 7. Two length scales emerge from the zero-sound dispersion (49). The crossover scale $\xi=\sqrt{BB_{\tau}}$ is the coherence length along ${\bf k}_{0}$, beyond which the dispersion is controlled by interaction. The length $\lambda=\sqrt{K/B}=1/(2k_{0})$ characterizes the anisotropy along ($\parallel$) and transverse ($\perp$) to ${\bf k}_{0}$. The corresponding momentum ${\bf q}_{c}=(q_{\parallel c},q_{\perp c})\equiv(\xi^{-1},(\lambda\xi)^{-1/2})$ separates the noninteracting regime, $E_{\bf q}\approx\Big{(}\sqrt{B}q_{\parallel}+\sqrt{K}q^{2}\Big{)}^{2}$, for ${\bf q}\gg{\bf q}_{c}$ from the long-wavelength interacting form, $E_{\bf q}\approx B_{\tau}^{-1/2}\sqrt{Bq_{\parallel}^{2}+Kq^{4}}$, for ${\bf q}\ll{\bf q}_{c}$. The latter agrees with the predictions of the lattice model, Eq. (III.2), if the lattice symmetry breaking effects are neglected.

Importantly, we note that, in the absent of lattice effects, $(\nabla_{\perp}\phi)^{2}$ is absent in the expansion of Eq. (44b) at all orders. This is enforced by the underlying rotational symmetry that is spontaneously broken by the helical superfluid state. More explicitly, an infinitesimal shift of the condensate momentum ${\bf k}_{0}\rightarrow{\bf k}_{0}+\delta{\bf k}_{0}$ along the dispersion minimum contour is a symmetry⁸⁸8This is closely related to the spontaneous breaking of dipole conservation (in addition to charge) of Bose condensation, which prohibits linear derivative terms, as recently studied in the dipolar Bose-Hubbard model by Lake et al. Lake et al. (2022) Here, the helical superfluid is characterized by an emerged dipole conservation perpendicular to ${\bf k}_{0}$, which however, only holds infinitesimally.. According to (42), this transforms $\phi_{\bf r}\to\phi_{\bf r}+\delta{\bf k}_{0}\cdot{\bf r}$, which then enforces the exact vanishing of the coefficient of $(\nabla_{\perp}\phi)^{2}$.