Flavor Constraints in a Generational Three Higgs Doublet Model

We augment the SM with two additional Higgs doublets. The associated gauge representations under $SU(3)_{c}\times SU(2)_{L}\times U(1)_{Y}$ are analogous to the SM. All three Higgs doublets transform as $\Phi_{a}\sim({\mathbf{1}},{\mathbf{2}},\frac{1}{2})$, with $a=1,2,3$ labeling the three Higgs fields.

The terms in the model’s Lagrangian that contain the Higgs fields can be written as

$$\mathcal{L}_{\text{3HDM}}\supset\sum\limits_{a=1}^{3}\lvert D_{\mu}\Phi_{a}\rvert^{2}-V_{\text{3HDM}}+\mathcal{L}_{\text{3HDM}}^{\text{Yuk}}\leavevmode\nobreak\ .$$

(1)

In addition to the kinetic terms, the Lagrangian contains a potential for the Higgs fields and Yukawa interactions with the SM fermions. The most general Yukawa Lagrangian is

$$-\mathcal{L}_{\text{3HDM}}^{\text{Yuk}}=\sum\limits_{i,j}\left(\lambda_{u_{1}}^{ij}\bar{q}_{L_{i}}\tilde{\Phi}_{1}u_{R_{j}}+\lambda_{d_{1}}^{ij}\bar{q}_{L_{i}}\Phi_{1}d_{R_{j}}+\lambda_{\ell_{1}}^{ij}\bar{\ell}_{L_{i}}\Phi_{1}e_{R_{j}}\right)+\text{h.c.}\\ +\sum\limits_{i,j}\left(\lambda_{u_{2}}^{ij}\bar{q}_{L_{i}}\tilde{\Phi}_{2}u_{R_{j}}+\lambda_{d_{2}}^{ij}\bar{q}_{L_{i}}\Phi_{2}d_{R_{j}}+\lambda_{\ell_{2}}^{ij}\bar{\ell}_{L_{i}}\Phi_{2}e_{R_{j}}\right)+\text{h.c.}\\ +\sum\limits_{i,j}\left(\lambda_{u_{3}}^{ij}\bar{q}_{L_{i}}\tilde{\Phi}_{3}u_{R_{j}}+\lambda_{d_{3}}^{ij}\bar{q}_{L_{i}}\Phi_{3}d_{R_{j}}+\lambda_{\ell_{3}}^{ij}\bar{\ell}_{L_{i}}\Phi_{3}e_{R_{j}}\right)+\text{h.c.}\leavevmode\nobreak\ ,$$

(2)

where $\tilde{\Phi}_{a}\equiv i\sigma_{2}\Phi_{a}^{*}$ and $i,j$ are flavor indices which run from 1 to 3. $\overline{q}_{L}$ and $\overline{\ell}_{L}$ represent the left-handed quark and lepton doublets, while $u_{R}$, $d_{R}$, $e_{R}$ are the quark and lepton singlets. The $\lambda$ matrices denote the Yukawa couplings. Neutrino masses and mixing are beyond the scope of this paper.

The most general gauge invariant $n$HDM potential may contain up to $n^{2}$ mass parameters and $\frac{n^{2}}{2}(n^{2}+1)$ quartic interactions (see e.g. [28, 33])

$$V_{\text{3HDM}}=\sum\limits_{a,b=1}^{3}m_{ab}^{2}(\Phi_{a}^{{\dagger}}\Phi_{b})+\sum\limits_{a,b,c,d=1}^{3}\lambda_{ab,cd}(\Phi_{a}^{{\dagger}}\Phi_{b})(\Phi_{c}^{{\dagger}}\Phi_{d})\leavevmode\nobreak\ ,$$

(3)

where $\lambda_{ab,cd}=\lambda_{cd,ab}$ and, by hermiticity, $m_{ab}^{2}=(m_{ba}^{2})^{*}$ while $\lambda_{ab,cd}=\lambda^{*}_{ba,dc}$. In a 3HDM, this corresponds in general to 54 terms in the Higgs potential.

As discussed in more detail in section II.4 and appendix B, we are interested in a scenario in which the three Higgs fields couple preferentially to a single generation. This can be most readily achieved if the Higgs fields are charged under softly broken $U(1)$ flavor symmetries. We thus consider it plausible that the Higgs potential respects to a good approximation a softly broken $U(1)^{3}$ symmetry, with the three $U(1)$ factors acting separately on a single Higgs field. This reduces significantly the possible 54 potential terms. Explicitly, we are left with¹¹1This setup is sometimes referred to as the $U(1)\times U(1)$ symmetric 3HDM in the literature, see e.g. [32]. In fact, the third $U(1)$ is automatically guaranteed by hypercharge gauge invariance.

	$\displaystyle V_{\text{3HDM}}$	$\displaystyle=m_{11}^{2}(\Phi_{1}^{{\dagger}}\Phi_{1})+m_{22}^{2}(\Phi_{2}^{{\dagger}}\Phi_{2})+m_{33}^{2}(\Phi_{3}^{{\dagger}}\Phi_{3})$
		$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ -\left[m_{12}^{2}(\Phi_{1}^{{\dagger}}\Phi_{2})+m_{23}^{2}(\Phi_{2}^{{\dagger}}\Phi_{3})+m_{13}^{2}(\Phi_{1}^{{\dagger}}\Phi_{3})+\text{h.c.}\right]$
		$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ +\lambda_{1}(\Phi_{1}^{{\dagger}}\Phi_{1})^{2}+\lambda_{2}(\Phi_{2}^{{\dagger}}\Phi_{2})^{2}+\lambda_{3}(\Phi_{3}^{{\dagger}}\Phi_{3})^{2}+\lambda_{4}(\Phi_{1}^{{\dagger}}\Phi_{1})(\Phi_{2}^{{\dagger}}\Phi_{2})+\lambda_{5}(\Phi_{1}^{{\dagger}}\Phi_{1})(\Phi_{3}^{{\dagger}}\Phi_{3})$
		$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ +\lambda_{6}(\Phi_{2}^{{\dagger}}\Phi_{2})(\Phi_{3}^{{\dagger}}\Phi_{3})+\lambda_{7}(\Phi_{1}^{{\dagger}}\Phi_{2})(\Phi_{2}^{{\dagger}}\Phi_{1})+\lambda_{8}(\Phi_{1}^{{\dagger}}\Phi_{3})(\Phi_{3}^{{\dagger}}\Phi_{1})+\lambda_{9}(\Phi_{2}^{{\dagger}}\Phi_{3})(\Phi_{3}^{{\dagger}}\Phi_{2})\leavevmode\nobreak\ .$		(4)

The three diagonal mass parameters $m_{aa}^{2}$ as well as all nine quartic interactions $\lambda_{i}$ of the potential are real. The three off-diagonal mass parameters, $m^{2}_{ab}$ with $a\neq b$, softly break the $U(1)^{3}$ symmetry. They can be complex and give rise to CP violation. Note that one can in principle use the freedom to re-phase the Higgs fields and remove the imaginary parts of two of the off-diagonal mass parameters, leaving a single physical CP violating phase.

The quartic couplings are not fully arbitrary but are constrained by bounded from below and vacuum stability conditions (see e.g. [16, 20, 32, 39]), as well as by perturbativity considerations (see e.g. [25]).

We assume that the three Higgs fields acquire vacuum expectation values such that spontaneous symmetry breaking (SSB) occurs: $SU(2)_{L}\times U(1)_{Y}\rightarrow U(1)_{\text{em}}$. In particular, we assume that the vacuum expectation values (vevs) of the Higgs fields are aligned such that $U(1)_{\text{em}}$ remains unbroken.

Based on the study of bounded from below constraints in 3HDMs that include charged runaway directions (see e.g. [32, 39]) and the study of electroweak symmetry breaking in multi-Higgs doublet models (see e.g. [17, 23]), we expect that an order 1 amount of parameter space does not break $U(1)_{\text{em}}$, as is common in the 3HDM literature.

In this case, we can parameterize the scalar doublets in terms of charged and neutral components in the usual way

$$\Phi_{a}=\begin{pmatrix}\phi^{+}_{a}\\ \frac{1}{\sqrt{2}}\left(v_{a}+\varphi_{a}+ia_{a}\right)\end{pmatrix}\leavevmode\nobreak\ .$$

(5)

The three vevs $v_{a}$ can, in principle, be complex. We find it convenient to work in a phase convention in which all three vevs are real. Without loss of generality, we label the Higgs fields such that $v_{1}<v_{2}<v_{3}$. As discussed below in section II.4, we will construct the Yukawa sector such that the doublets $\Phi_{1}$, $\Phi_{2}$, and $\Phi_{3}$ provide mass mainly to the first, second, and third generation of fermions, respectively. The ordering of the three vevs thus follows the mass ordering of the three generations of fermions, and we will later focus mainly on the limit $v_{1}\ll v_{2}\ll v_{3}$, which partially addresses the hierarchies in the fermion spectrum of the SM.

In order to reproduce the masses of the $W$ and $Z$ bosons, the individual vevs must sum in quadrature to the SM Higgs vev

$$v_{\text{SM}}^{2}\equiv v^{2}=v_{1}^{2}+v_{2}^{2}+v_{3}^{2}\simeq(246\leavevmode\nobreak\ \text{GeV})^{2}\leavevmode\nobreak\ .$$

(6)

This condition is satisfied by construction if we work with the convenient parameterization

$$v_{1}=v\,\cos\beta^{\prime}=vc_{\beta^{\prime}}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ v_{2}=v\,\sin\beta^{\prime}\,\cos\beta=vs_{\beta^{\prime}}c_{\beta}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ v_{3}=v\,\sin\beta^{\prime}\,\sin\beta=vs_{\beta^{\prime}}s_{\beta}\leavevmode\nobreak\ ,$$

(7)

where we used the notation $s_{x}=\sin x$, and $c_{x}=\cos x$. The above definitions imply

$$\tan\beta^{\prime}=t_{\beta^{\prime}}=\frac{\sqrt{v_{2}^{2}+v_{3}^{2}}}{v_{1}}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \tan\beta=t_{\beta}=\frac{v_{3}}{v_{2}}\leavevmode\nobreak\ ,$$

(8)

which may be viewed as an extension of the 2HDM definition for $\tan\beta$. In effect, we may trade $v_{1}$, $v_{2}$, $v_{3}$ for $v$, $\tan{\beta}$, $\tan{\beta^{\prime}}$.

Working with real vevs implies in general that $m^{2}_{12}$, $m^{2}_{13}$, and $m^{2}_{23}$ all have imaginary parts. We find that these imaginary parts are related by the minimization conditions of the potential in the following way

$$\text{Im}(m_{13}^{2})=-\frac{v_{2}}{v_{3}}\leavevmode\nobreak\ \text{Im}(m_{12}^{2})\leavevmode\nobreak\ ,\qquad\text{Im}(m_{23}^{2})=\frac{v_{1}}{v_{3}}\leavevmode\nobreak\ \text{Im}(m_{12}^{2})\leavevmode\nobreak\ .$$

(9)

The remaining minimization conditions allow us to express the mass parameters $m_{11}^{2}$, $m_{22}^{2}$, and $m_{33}^{2}$ in terms of the real vacuum expectation values

	$\displaystyle m^{2}_{11}$	$\displaystyle=\text{Re}(m^{2}_{12})\frac{v_{2}}{v_{1}}+\text{Re}(m^{2}_{13})\frac{v_{3}}{v_{1}}-\lambda^{2}_{1}v^{2}_{1}-\frac{1}{2}\big{[}(\lambda_{4}+\lambda_{7})v_{2}^{2}+(\lambda_{5}+\lambda_{8})v_{3}^{2}\big{]}\leavevmode\nobreak\ ,$		(10)
	$\displaystyle m^{2}_{22}$	$\displaystyle=\text{Re}(m^{2}_{12})\frac{v_{1}}{v_{2}}+\text{Re}(m^{2}_{23})\frac{v_{3}}{v_{2}}-\lambda^{2}_{2}v^{2}_{2}-\frac{1}{2}\big{[}(\lambda_{4}+\lambda_{7})v_{1}^{2}+(\lambda_{6}+\lambda_{9})v_{3}^{2}\big{]}\leavevmode\nobreak\ ,$		(11)
	$\displaystyle m^{2}_{33}$	$\displaystyle=\text{Re}(m^{2}_{13})\frac{v_{1}}{v_{3}}+\text{Re}(m^{2}_{23})\frac{v_{2}}{v_{3}}-\lambda^{2}_{3}v^{2}_{3}-\frac{1}{2}\big{[}(\lambda_{5}+\lambda_{8})v_{1}^{2}+(\lambda_{6}+\lambda_{9})v_{2}^{2}\big{]}\leavevmode\nobreak\ .$		(12)

In what follows, we will make the simplifying assumption that the Higgs potential respects CP invariance and set $\text{Im}(m_{12}^{2})=\text{Im}(m_{13}^{2})=\text{Im}(m_{23}^{2})=0$. The general case with CP violation in the Higgs potential and the possible implications will be discussed elsewhere.

In the absence of CP violation, $n$HDM models will contain $n$ physical neutral CP-even Higgs bosons, $n-1$ physical neutral CP-odd Higgs bosons, and $2(n-1)$ physical charged Higgs bosons. The remaining CP-odd and charged degrees of freedom are Goldstone bosons ($G^{0},G^{\pm}$) that provide the longitudinal components of the $Z$ and $W$ bosons of the SM. In our 3HDM case, after SSB, we expect a total of 3 neutral CP-even Higgs bosons ($h$, $H$, $H^{\prime}$), 2 neutral CP-odd Higgs ($A$, $A^{\prime}$), and 2 pairs of charged Higgs bosons ($H^{\pm}$, $H^{\pm\,\prime}$).

The mass matrix of the CP-odd scalars is independent of the quartic interactions and given by

$$\hat{m}^{2}_{a}=\begin{pmatrix}m_{12}^{2}\leavevmode\nobreak\ \frac{v_{2}}{v_{1}}+m_{13}^{2}\leavevmode\nobreak\ \frac{v_{3}}{v_{1}}&-m_{12}^{2}&-m_{13}^{2}\\ -m_{12}^{2}&m_{12}^{2}\leavevmode\nobreak\ \frac{v_{1}}{v_{2}}+m_{23}^{2}\leavevmode\nobreak\ \frac{v_{3}}{v_{2}}&-m_{23}^{2}\\ -m_{13}^{2}&-m_{23}^{2}&m_{13}^{2}\leavevmode\nobreak\ \frac{v_{1}}{v_{3}}+m_{23}^{2}\leavevmode\nobreak\ \frac{v_{2}}{v_{3}}\end{pmatrix}\leavevmode\nobreak\ .$$

(13)

For the charged and CP-even scalar mass matrices, we find

	$\displaystyle\hat{m}^{2}_{\pm}$	$\displaystyle=$	$\displaystyle\hat{m}^{2}_{a}+\frac{1}{2}\begin{pmatrix}-v_{2}^{2}\lambda_{7}-v_{3}^{2}\lambda_{8}&v_{1}v_{2}\lambda_{7}&v_{1}v_{3}\lambda_{8}\\ v_{1}v_{2}\lambda_{7}&-v_{1}^{2}\lambda_{7}-v_{3}^{2}\lambda_{9}&v_{2}v_{3}\lambda_{9}\\ v_{1}v_{3}\lambda_{8}&v_{2}v_{3}\lambda_{9}&-v_{1}^{2}\lambda_{8}-v_{2}^{2}\lambda_{9})\end{pmatrix}\leavevmode\nobreak\ ,$		(14)
	$\displaystyle\hat{m}^{2}_{\varphi}$	$\displaystyle=$	$\displaystyle\hat{m}^{2}_{a}+\begin{pmatrix}2\,v_{1}^{2}\lambda_{1}&v_{1}v_{2}(\lambda_{4}+\lambda_{7})&v_{1}v_{3}(\lambda_{5}+\lambda_{8})\\ v_{1}v_{2}(\lambda_{4}+\lambda_{7})&2\,v_{2}^{2}\lambda_{2}&v_{2}v_{3}(\lambda_{6}+\lambda_{9})\\ v_{1}v_{3}(\lambda_{5}+\lambda_{8})&v_{2}v_{3}(\lambda_{6}+\lambda_{9})&2\,v_{3}^{2}\lambda_{3}\end{pmatrix}\leavevmode\nobreak\ .$		(15)

We rotate into the physical Higgs mass-eigenstate basis through orthogonal, 3-parameter rotations for each of the three Higgs sectors

$$\begin{pmatrix}A^{\prime}\\ A\\ G^{0}\end{pmatrix}=O_{A}\begin{pmatrix}a_{1}\\ a_{2}\\ a_{3}\end{pmatrix}\leavevmode\nobreak\ ,\qquad\begin{pmatrix}H^{\pm\prime}\\ H^{\pm}\\ G^{\pm}\end{pmatrix}=O_{\pm}\begin{pmatrix}\phi_{1}^{\pm}\\ \phi_{2}^{\pm}\\ \phi_{3}^{\pm}\end{pmatrix}\leavevmode\nobreak\ ,\qquad\begin{pmatrix}H^{\prime}\\ H\\ h\end{pmatrix}=O_{H}\begin{pmatrix}\varphi_{1}\\ \varphi_{2}\\ \varphi_{3}\end{pmatrix}\leavevmode\nobreak\ .$$

(16)

The diagonalization matrices $O_{A}$, $O_{\pm}$, and $O_{H}$ can be parameterized as products of three rotation matrices. We find for the pseudoscalar and charged Higgs sectors

$$O_{A}=\begin{pmatrix}\cos\gamma&-\sin\gamma&0\\ \sin\gamma&\cos\gamma&0\\ 0&0&1\end{pmatrix}\begin{pmatrix}\sin{\beta^{\prime}}&0&-\cos{\beta^{\prime}}\\ 0&1&0\\ \cos{\beta^{\prime}}&0&\sin{\beta^{\prime}}\end{pmatrix}\begin{pmatrix}1&0&0\\ 0&\sin\beta&-\cos\beta\\ 0&\cos\beta&\sin\beta\end{pmatrix}\leavevmode\nobreak\ ,$$

(17)

$$O_{\pm}=\begin{pmatrix}\cos{\gamma_{\pm}}&-\sin{\gamma_{\pm}}&0\\ \sin{\gamma_{\pm}}&\cos{\gamma_{\pm}}&0\\ 0&0&1\end{pmatrix}\begin{pmatrix}\sin{\beta^{\prime}}&0&-\cos{\beta^{\prime}}\\ 0&1&0\\ \cos{\beta^{\prime}}&0&\sin{\beta^{\prime}}\end{pmatrix}\begin{pmatrix}1&0&0\\ 0&\sin\beta&-\cos\beta\\ 0&\cos\beta&\sin\beta\end{pmatrix}\leavevmode\nobreak\ ,$$

(18)

where $\beta$ and $\beta^{\prime}$ where already introduced in (7). After applying the $\beta$ and $\beta^{\prime}$ rotations, one obtains partially diagonalized forms of the respective mass matrices, with the massless Goldstone bosons $G^{0}$ and $G^{\pm}$ already appearing as eigenstates. The final rotations by the angels $\gamma$ and $\gamma_{\pm}$ fully diagonalize the mass matrices and define the massive eigenstates $A$, $A^{\prime}$ and $H^{\pm}$, $H^{\pm\,\prime}$.

The pseudoscalar masses $m_{A}$, $m_{A^{\prime}}$ and the mixing angle $\gamma$ are determined by

	$\displaystyle m_{A}^{2}m_{A^{\prime}}^{2}$	$\displaystyle=$	$\displaystyle\frac{m_{23}^{2}}{c_{\beta^{\prime}}s_{\beta^{\prime}}}\left(\frac{m_{13}^{2}}{c_{\beta}}+\frac{m_{12}^{2}}{s_{\beta}}\right)+\frac{m_{12}^{2}m_{13}^{2}}{c_{\beta}s_{\beta}s_{\beta^{\prime}}^{2}}\leavevmode\nobreak\ ,$		(19)
	$\displaystyle m_{A}^{2}+m_{A^{\prime}}^{2}$	$\displaystyle=$	$\displaystyle\frac{m_{23}^{2}}{c_{\beta}s_{\beta}}+\frac{1}{c_{\beta^{\prime}}s_{\beta^{\prime}}}\left[m_{12}^{2}\left(\frac{s_{\beta}^{2}c_{\beta^{\prime}}^{2}}{c_{\beta}}+c_{\beta}\right)+m_{13}^{2}\left(\frac{c_{\beta}^{2}c_{\beta^{\prime}}^{2}}{s_{\beta}}+s_{\beta}\right)\right]\leavevmode\nobreak\ ,$		(20)
	$\displaystyle(m_{A}^{2}-m_{A^{\prime}}^{2})\frac{1}{2}\sin(2\gamma)$	$\displaystyle=$	$\displaystyle m_{13}^{2}\frac{c_{\beta}}{s_{\beta^{\prime}}}-m_{12}^{2}\frac{s_{\beta}}{s_{\beta^{\prime}}}\leavevmode\nobreak\ .$		(21)

These relations can be used to trade the Lagrangian parameters $m_{12}^{2}$, $m_{13}^{2}$, $m_{23}^{2}$ for $m_{A}$, $m_{A^{\prime}}$, $\gamma$. The charged Higgs masses $m_{H^{\pm}}$, $m_{H^{\pm\,\prime}}$ and the corresponding mixing angle $\gamma_{\pm}$ can be determined analogously. We find

	$\displaystyle m_{H^{\pm}}^{2}m_{H^{\pm\,\prime}}^{2}$	$\displaystyle=$	$\displaystyle m_{A}^{2}m_{A^{\prime}}^{2}-\frac{v^{2}}{2}\Bigg{[}\frac{\lambda_{7}}{s_{\beta}}\left(m_{23}^{2}c_{\beta}+m_{13}^{2}/t_{\beta^{\prime}}\right)$		(22)
			$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ +\frac{\lambda_{8}}{c_{\beta}}\left(m_{23}^{2}s_{\beta}+m_{12}^{2}/t_{\beta^{\prime}}\right)+\lambda_{9}t_{\beta^{\prime}}\left(m_{12}^{2}c_{\beta}+m_{13}^{2}s_{\beta}\right)\Bigg{]}$
			$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ +\frac{v^{4}}{4}\Bigg{[}\lambda_{7}\lambda_{8}c_{\beta^{\prime}}^{2}+\lambda_{7}\lambda_{9}c_{\beta}^{2}s_{\beta^{\prime}}^{2}+\lambda_{8}\lambda_{9}s_{\beta}^{2}s_{\beta^{\prime}}^{2}\Bigg{]}\leavevmode\nobreak\ ,$
	$\displaystyle m_{H^{\pm}}^{2}+m_{H^{\pm\,\prime}}^{2}$	$\displaystyle=$	$\displaystyle m_{A}^{2}+m_{A^{\prime}}^{2}-\frac{v^{2}}{2}\Bigg{[}\lambda_{7}\left(c_{\beta}^{2}+s_{\beta}^{2}c_{\beta^{\prime}}^{2}\right)+\lambda_{8}\left(s_{\beta}^{2}+c_{\beta}^{2}c_{\beta^{\prime}}^{2}\right)+\lambda_{9}s_{\beta^{\prime}}^{2}\Bigg{]}\leavevmode\nobreak\ ,$		(23)
	$\displaystyle(m_{H^{\pm}}^{2}-m_{H^{\pm\,\prime}}^{2})\frac{1}{2}\sin(2\gamma_{\pm})$	$\displaystyle=$	$\displaystyle(m_{A}^{2}-m_{A^{\prime}}^{2})\frac{1}{2}\sin(2\gamma)+\frac{v^{2}}{2}\left(\lambda_{7}-\lambda_{8}\right)c_{\beta}s_{\beta}c_{\beta^{\prime}}\leavevmode\nobreak\ .$		(24)

In the case of the CP-even Higgs bosons, we write

$$O_{H}=\begin{pmatrix}\cos{\gamma_{H}}&-\sin{\gamma_{H}}&0\\ \sin{\gamma_{H}}&\cos{\gamma_{H}}&0\\ 0&0&1\end{pmatrix}\begin{pmatrix}\cos{\alpha^{\prime}}&0&\sin{\alpha^{\prime}}\\ 0&1&0\\ -\sin{\alpha^{\prime}}&0&\cos{\alpha^{\prime}}\end{pmatrix}\begin{pmatrix}1&0&0\\ 0&\cos\alpha&\sin\alpha\\ 0&-\sin\alpha&\cos\alpha\end{pmatrix}\leavevmode\nobreak\ ,$$

(25)

with all three angles $\alpha$, $\alpha^{\prime}$, and $\gamma_{H}$, as new parameters. As already anticipated in the introduction, the stringent limits from meson mixing generically suggest that the BSM Higgs bosons may be considerably heavier than the electroweak scale. It is thus motivated to consider the decoupling limit in which $v_{1}^{2},v_{2}^{2},v_{3}^{2}\ll m_{12}^{2},m_{13}^{2},m_{23}^{2}$. In this limit, one finds to first approximation

$$m_{H}^{2}\simeq m_{A}^{2}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ m_{H^{\prime}}^{2}\simeq m_{A^{\prime}}^{2}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ O_{H}\simeq O_{A}\leavevmode\nobreak\ .$$

(26)

The eigenstate $h$ has a mass of the order of $v$ and has at leading order precisely SM-like couplings. It is thus identified with the 125 GeV Higgs. As we will see in section III, in the approximation (26), an exact cancellation of new physics contributions to the meson mixing amplitudes can occur. To assess the constraints from meson mixing, it thus becomes important to systematically include corrections beyond the leading order of the decoupling limit.

While it is possible to systematically expand the masses and mixing angles in the CP even scalar sector in powers of $v_{k}^{2}/m_{ij}^{2}$, the resulting expressions are rather lengthy and will not be presented here. Instead, we will focus on the scenario of large $\tan\beta$ and $\tan\beta^{\prime}$ in which the expressions simplify considerably. In fact, one motivation to consider the generational 3HDM model is the possibility of partially addressing the hierarchies in the SM fermion masses. We thus consider a scenario with $v_{1}\ll v_{2}\ll v_{3}$, corresponding to $1\ll\tan\beta\ll\tan\beta^{\prime}$.

First of all, assuming no particular hierarchy in the mass parameters $m_{12}^{2}\sim m_{13}^{2}\sim m_{23}^{2}$, we find that the pseudoscalar $A^{\prime}$ is parametrically heavier than $A$ ²²2According to the minimization conditions in eqs. (10) - (12), this corresponds to a hierarchical set of diagonal masses $m^{2}_{11}\gg m^{2}_{22}\gg m^{2}_{33}\sim O(v^{2})$. Alternatively, one could entertain a scenario with $m^{2}_{11}\sim m^{2}_{22}\gg m^{2}_{33}\sim O(v^{2})$, which gives $m_{A^{\prime}}^{2}\sim m_{A}^{2}$, but requires $m_{13}^{2}\ll m_{23}^{2}$. In our numerical analysis, we do not assume any particular hierarchy in the masses of the heavy Higgs bosons.

$$m_{A^{\prime}}^{2}\simeq m_{13}^{2}\tan\beta^{\prime}\gg m_{A}^{2}\simeq m_{23}^{2}\tan\beta\leavevmode\nobreak\ .$$

(27)

Moreover, the mixing angle in the pseudoscalar sector, $\gamma$, turns out to be small and is of the order of $\gamma\sim 1/\tan\beta^{\prime}\ll 1$. More precisely, we find

$\displaystyle\gamma_{A}\simeq\frac{m_{12}^{2}}{m_{13}^{2}}\,\frac{1}{\tan\beta^{\prime}}\leavevmode\nobreak\ .$

(28)

In the charged Higgs sector, the masses and mixing angles are given at leading order by the corresponding pseudoscalar quantities. Including next-to-leading order corrections, we find

$$m_{H^{\pm}}^{2}\simeq m_{A}^{2}-\frac{v^{2}}{2}\lambda_{9}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ m_{H^{\pm\,\prime}}^{2}\simeq m_{A^{\prime}}^{2}-\frac{v^{2}}{2}\lambda_{8}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \gamma_{\pm}\simeq\gamma+\frac{\gamma v^{2}}{2m_{A^{\prime}}^{2}}(\lambda_{8}-\lambda_{9})\leavevmode\nobreak\ .$$

(29)

Similarly, for the masses of the scalar Higgs bosons we find

	$\displaystyle m_{h}^{2}\simeq 2v^{2}\lambda_{3}\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ m_{H}^{2}$	$\displaystyle\simeq$	$\displaystyle m_{A}^{2}+\frac{2v^{2}}{\tan^{2}\beta}\left(\lambda_{2}+\lambda_{3}-\lambda_{6}-\lambda_{9}\right)=m_{A}^{2}+\frac{2v^{2}}{\tan^{2}\beta}\lambda_{H}\leavevmode\nobreak\ ,$		(30)
	$\displaystyle m_{H^{\prime}}^{2}$	$\displaystyle\simeq$	$\displaystyle m_{A^{\prime}}^{2}+\frac{2v^{2}}{\tan^{2}\beta^{\prime}}\left(\lambda_{1}+\lambda_{3}-\lambda_{5}-\lambda_{8}\right)=m_{A^{\prime}}^{2}+\frac{2v^{2}}{\tan^{2}\beta^{\prime}}\lambda_{H^{\prime}}\leavevmode\nobreak\ .$		(31)

For the diagonalization matrix in the scalar Higgs sector, we find it convenient to write

$$O_{H}=(1\!\!1+\Delta)O_{A}\leavevmode\nobreak\ ,\quad\Delta\simeq\begin{pmatrix}0&\Delta_{12}&\Delta_{13}\\ -\Delta_{12}&0&\Delta_{23}\\ -\Delta_{13}&-\Delta_{23}&0\end{pmatrix}\leavevmode\nobreak\ ,$$

(32)

where the matrix $\Delta=O_{H}O_{A}^{\text{T}}-1\!\!1$ captures the departure from the decoupling limit. We find

	$\displaystyle\Delta_{12}$	$\displaystyle=$	$\displaystyle\frac{v^{2}}{m_{A^{\prime}}^{2}}\frac{1}{t_{\beta}t_{\beta^{\prime}}}\left(2\lambda_{3}+\lambda_{4}-\lambda_{5}-\lambda_{6}+\lambda_{7}-\lambda_{8}-\lambda_{9}\right)\equiv\frac{v^{2}}{m_{A^{\prime}}^{2}}\frac{1}{t_{\beta}t_{\beta^{\prime}}}\lambda_{12}\leavevmode\nobreak\ ,$		(33)
	$\displaystyle\Delta_{13}$	$\displaystyle=$	$\displaystyle-\frac{v^{2}}{m_{A^{\prime}}^{2}}\frac{1}{t_{\beta^{\prime}}}\left(2\lambda_{3}-\lambda_{5}-\lambda_{8}\right)\equiv-\frac{v^{2}}{m_{A^{\prime}}^{2}}\frac{1}{t_{\beta^{\prime}}}\lambda_{13}\leavevmode\nobreak\ ,$		(34)
	$\displaystyle\Delta_{23}$	$\displaystyle=$	$\displaystyle-\frac{v^{2}}{m_{A}^{2}}\frac{1}{t_{\beta}}\left(2\lambda_{3}-\lambda_{6}-\lambda_{9}\right)\equiv-\frac{v^{2}}{m_{A}^{2}}\frac{1}{t_{\beta}}\lambda_{23}\leavevmode\nobreak\ .$		(35)

We consider the following set of Yukawa matrices, with textures that are an extension of the textures previously explored in the context of so-called flavorful 2HDMs [13, 12, 14]

	$\displaystyle\lambda_{u_{1}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{1}}\begin{pmatrix}m_{u}&m_{u}&m_{u}\\ m_{u}&m_{u}&m_{u}\\ m_{u}&m_{u}&m_{u}\end{pmatrix},$	$\displaystyle\lambda_{u_{2}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{2}}\begin{pmatrix}0&0&0\\ 0&m_{c}&m_{c}\\ 0&m_{c}&m_{c}\end{pmatrix},$	$\displaystyle\lambda_{u_{3}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{3}}\begin{pmatrix}0&0&0\\ 0&0&0\\ 0&0&m_{t}\end{pmatrix},$		(36a)
	$\displaystyle\lambda_{d_{1}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{1}}\begin{pmatrix}m_{d}&m_{s}\lambda&m_{b}\lambda^{3}\\ m_{d}&m_{d}&m_{d}\\ m_{d}&m_{d}&m_{d}\end{pmatrix},$	$\displaystyle\lambda_{d_{2}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{2}}\begin{pmatrix}0&0&0\\ 0&m_{s}&m_{b}\lambda^{2}\\ 0&m_{s}&m_{s}\end{pmatrix},$	$\displaystyle\lambda_{d_{3}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{3}}\begin{pmatrix}0&0&0\\ 0&0&0\\ 0&0&m_{b}\end{pmatrix},$		(36b)
	$\displaystyle\lambda_{\ell_{1}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{1}}\begin{pmatrix}m_{e}&m_{e}&m_{e}\\ m_{e}&m_{e}&m_{e}\\ m_{e}&m_{e}&m_{e}\end{pmatrix},$	$\displaystyle\lambda_{\ell_{2}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{2}}\begin{pmatrix}0&0&0\\ 0&m_{\mu}&m_{\mu}\\ 0&m_{\mu}&m_{\mu}\end{pmatrix},$	$\displaystyle\lambda_{\ell_{3}}$	$\displaystyle\sim\frac{\sqrt{2}}{v_{3}}\begin{pmatrix}0&0&0\\ 0&0&0\\ 0&0&m_{\tau}\end{pmatrix}.$		(36c)

With $\sim$ we indicate the order of the entries, keeping in mind that non-zero entries in the same matrix can differ by complex $\mathcal{O}(1)$ factors. We assume that all of the Yukawa couplings are rank-1 matrices, but otherwise do not contain any specific structure. Because of the assumed rank-1 structure, each Higgs doublet couples only to a single linear combination of the three generations and we dub such a scenario a “generational” 3HDM (or G3HDM). The rank-1 structure of the Yukawas can be realized in a straight-forward way by having the SM fermions mix with three separate generations of vector-like fermions that each couple to only one of the Higgs doublets. More details are given in appendix B.

An interesting feature of the generational 3HDM setup is that one may choose to generate part of the SM fermion mass hierarchies by adjusting the vevs of the three doublets relative to each other. This offers the option to either generate the large mass differences purely in the Yukawa sector, as seen in the SM, or to shift some of the burden over to the Higgs sector. In the limit of a large relative vev hierarchy, $v_{1}\ll v_{2}\ll v_{3}$, one may begin with a set of rank-1 and $\mathcal{O}(1)$ Yukawa couplings, and proceed to have the large parts of the quark and lepton mass hierarchies be generated in the Higgs sector alone.

Without loss of generality, we have expressed the Yukawa couplings in a flavor basis in which the couplings of $\Phi_{3}$ are diagonal, emphasizing the role of $\Phi_{3}$ in providing the dominant component of the mass for the third generation of quarks and leptons. Note that the couplings of $\Phi_{3}$ preserve a $SU(2)^{5}$ flavor symmetry that acts on the first two generations. The couplings of $\Phi_{2}$ and $\Phi_{1}$ are in general misaligned in flavor space and both break the $SU(2)^{5}$ symmetry. However, we can use the remaining freedom in choosing a flavor basis to bring the couplings of $\Phi_{2}$ into the shown block-diagonal form.

The non-minimal breaking of the SM flavor symmetries by the above set of Yukawa couplings is expected to give large contributions to flavor-changing neutral current (FCNC) processes, kaon and $D$ meson mixing in particular. The corresponding flavor constraints on the model will be analyzed in detail in section III. In order to soften the constraints one may entertain the possibility that some of the off-diagonal entries in $\lambda_{f_{1}}$ and $\lambda_{f_{2}}$ are suppressed compared to what is shown in (36a) - (36c). This could for example be achieved by spontaneously broken flavor symmetries. However, we emphasize that not all off-diagonal entries in the Yukawa couplings can be arbitrarily suppressed. The entries in $\lambda_{d_{1}}$ and $\lambda_{d_{2}}$ of order $m_{s}\lambda$, $m_{b}\lambda^{3}$, and $m_{b}\lambda^{2}$, where $\lambda\simeq 0.23$ is the sine of the Cabibbo angle, are required to reproduce the CKM matrix, once one rotates in the fermion mass eigenstate basis. Note that it is an assumption that the CKM matrix originates from the diagonalization of the down-quark mass matrix. Other Yukawa textures can also be viable (see also footnote 3 below.)

In the fermion mass eigenstate basis, we define the following set of mass parameters which we will use below to write the couplings of the physical Higgs mass eigenstates to fermion mass eigenstates

$$m^{f_{1}}_{ff^{\prime}}=\frac{v_{1}}{\sqrt{2}}\langle f_{L}|\lambda_{f_{1}}|f_{R}^{\prime}\rangle\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ m^{f_{2}}_{ff^{\prime}}=\frac{v_{2}}{\sqrt{2}}\langle f_{L}|\lambda_{f_{2}}|f_{R}^{\prime}\rangle\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ m^{f_{3}}_{ff^{\prime}}=\frac{v_{3}}{\sqrt{2}}\langle f_{L}|\lambda_{f_{3}}|f_{R}^{\prime}\rangle\leavevmode\nobreak\ .$$

(37)

These parameters obey the relationship

$$m^{f_{3}}_{ff^{\prime}}+m^{f_{2}}_{ff^{\prime}}+m^{f_{1}}_{ff^{\prime}}=m_{f}\delta_{ff^{\prime}}\leavevmode\nobreak\ ,$$

(38)

with $m_{f}$ (without superscripts) representing the physical masses of the quarks or leptons.

We obtain the following set of explicit mass parameters after expanding each entry to leading order in ratios of first to second and second to third generation masses

$$\frac{m^{u_{1}}_{qq^{\prime}}}{m_{u}}\simeq\begin{pmatrix}1&x_{uc}&x_{ut}\\ x_{cu}&x_{cu}x_{uc}&x_{cu}x_{ut}\\ x_{tu}&x_{tu}x_{uc}&x_{tu}x_{ut}\end{pmatrix},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \frac{m^{u_{2}}_{qq^{\prime}}}{m_{c}}\simeq\begin{pmatrix}\frac{m_{u}^{2}}{m_{c}^{2}}x_{uc}x_{cu}&-\frac{m_{u}}{m_{c}}x_{uc}&-\frac{m_{u}}{m_{c}}x_{uc}y_{ct}\\ -\frac{m_{u}}{m_{c}}x_{cu}&1&y_{ct}\\ -\frac{m_{u}}{m_{c}}y_{tc}x_{cu}&y_{tc}&y_{tc}y_{ct}\end{pmatrix}\leavevmode\nobreak\ ,$$

(39)

$$\frac{m^{u_{3}}_{qq^{\prime}}}{m_{t}}\simeq\begin{pmatrix}\frac{m_{u}^{2}}{m_{t}^{2}}(x_{ut}-x_{uc}y_{ct})(x_{tu}-y_{tc}x_{cu})&\frac{m_{u}m_{c}}{m_{t}^{2}}(x_{ut}-x_{uc}y_{ct})y_{tc}&-\frac{m_{u}}{m_{t}}(x_{ut}-x_{uc}y_{ct})\\ \frac{m_{u}m_{c}}{m_{t}^{2}}y_{ct}(x_{tu}-y_{tc}x_{cu})&\frac{m_{c}^{2}}{m_{t}^{2}}y_{ct}y_{tc}&-\frac{m_{c}}{m_{t}}y_{ct}\\ -\frac{m_{u}}{m_{t}}(x_{tu}-y_{tc}x_{cu})&-\frac{m_{c}}{m_{t}}y_{tc}&1\end{pmatrix}\leavevmode\nobreak\ ,$$

(40)

$$\frac{m^{d_{1}}_{qq^{\prime}}}{m_{d}}\simeq\begin{pmatrix}1&\frac{m_{s}}{m_{d}}V_{ud}^{*}V_{us}&\frac{m_{b}}{m_{d}}V_{ud}^{*}V_{ub}\\ x_{sd}&x_{sd}\frac{m_{s}}{m_{d}}V_{ud}^{*}V_{us}&x_{sd}\frac{m_{b}}{m_{d}}V_{ud}^{*}V_{ub}\\ x_{bd}&x_{bd}\frac{m_{s}}{m_{d}}V_{ud}^{*}V_{us}&x_{bd}\frac{m_{b}}{m_{d}}V_{ud}^{*}V_{ub}\end{pmatrix},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \frac{m^{d_{2}}_{qq^{\prime}}}{m_{s}}\simeq\begin{pmatrix}-\frac{m_{d}}{m_{s}}V_{cd}^{*}V_{cs}x_{sd}&V_{cd}^{*}V_{cs}&\frac{m_{b}}{m_{s}}V_{cd}^{*}V_{cb}\\ -\frac{m_{d}}{m_{s}}x_{sd}&1&\frac{m_{b}}{m_{s}}V_{cs}^{*}V_{cb}\\ -\frac{m_{d}}{m_{s}}y_{bs}x_{sd}&y_{bs}&y_{bs}\frac{m_{b}}{m_{s}}V_{cs}^{*}V_{cb}\end{pmatrix}\leavevmode\nobreak\ ,$$

(41)

$$\frac{m^{d_{3}}_{qq^{\prime}}}{m_{b}}\simeq\begin{pmatrix}-\frac{m_{d}}{m_{b}}V_{td}^{*}V_{tb}(x_{bd}-y_{bs}x_{sd})&-\frac{m_{s}}{m_{b}}V_{td}^{*}V_{tb}y_{bs}&V_{td}^{*}V_{tb}\\ -\frac{m_{d}}{m_{b}}V_{ts}^{*}V_{tb}(x_{bd}-y_{bs}x_{sd})&-\frac{m_{s}}{m_{b}}V_{ts}^{*}V_{tb}y_{bs}&V_{ts}^{*}V_{tb}\\ -\frac{m_{d}}{m_{b}}(x_{bd}-y_{bs}x_{sd})&-\frac{m_{s}}{m_{b}}y_{bs}&1\end{pmatrix}\leavevmode\nobreak\ .$$

(42)

The lepton mass parameters are completely analogous to the ones in the up quark sector. In the above expressions, the $x_{ij}$, $y_{ij}$ are free, in general complex, $\mathcal{O}(1)$ parameters. These parameterize new sources of flavor and CP violation. Note that not all entries of the mass matrices are independent. This is due to two reasons: first, they need to reproduce the quark and lepton masses as well as the CKM matrix; second, we have assumed that they originate from rank-1 Yukawa couplings.