Unveiling the Unexplored Decay Mode of a Light Charged Higgs Boson to an Off-Shell Top Quark and a Bottom Quark

The milestone discovery of the Higgs boson at the LHC ATLAS:2012yve ; CMS:2012qbp seemingly completes the Standard Model (SM), yet the quest for a new particle physics theory beyond the SM (BSM) continues. This pursuit is driven by unresolved fundamental questions of the Universe, such as the naturalness problem, fermion mass hierarchy, baryogenesis, non-zero neutrino masses, and the identity of dark matter. High-energy collider experiments are indispensable in this quest, offering the ability to directly study fundamental particles in a highly controlled environment and providing complementary insights to cosmological and dark matter searches.

One of the most promising BSM signals at high-energy colliders is the presence of a light charged Higgs boson with a mass below the top quark mass $m_{t}$. This possibility remains viable under current constraints within the type-I and type-X¹¹1In type-II and type-Y 2HDM, the charged Higgs boson is tightly constrained to be as heavy as $M_{H^{\pm}}\gtrsim 800{\;{\rm GeV}}$ due to the measurements of the inclusive $B$-meson decay into $s\gamma$ Misiak:2020vlo . two-Higgs-doublet model (2HDM) Aoki:2009ha ; Branco:2011iw ; Craig:2013hca ; Wang:2022yhm ; Shen:2022yuo ; Kanemura:2022ldq ; Lee:2022gyf , three-Higgs doublet model Akeroyd:2018axd , next-to-2HDM Abouabid:2021yvw , lepton-specific Inert doublet model Han:2021gfu , and scalar-triplet model Ferreira:2021bdj .

Significant efforts have been made to search for the light charged Higgs boson at the LHC and future lepton colliders. For the decay $H^{\pm}\rightarrow\tau^{\pm}\nu$, various production channels have been explored, such as $t\rightarrow H^{\pm}b$ Abbaspour:2018ysj ; Demir:2018iqo ; ATLAS:2018gfm ; Sanyal:2019xcp ; Sirunyan:2019hkq ; Ghosh:2022wbe , $pp\to H^{\pm}\varphi^{0}$ Kim:2023lxc , $pp\rightarrow H^{\pm}A$ Kanemura:2011kx , $pp\rightarrow H^{+}H^{-}$ Duarte:2024zeh , $cs/cb\rightarrow H^{\pm}$ Hernandez-Sanchez:2012vxa ; Hernandez-Sanchez:2020vax , and $W^{\pm*}W^{\pm*}\rightarrow H^{\pm}H^{\pm}$ Aiko:2019mww . Here, $\varphi^{0}$ denotes a new CP-even neutral Higgs boson. For the $H^{\pm}\rightarrow cb/cs$ mode, the production channel of $t\rightarrow H^{\pm}b$ has been considered ATLAS:2013uxj ; CMS:2015yvc ; Akeroyd:2018axd ; CMS:2018dzl ; ATLAS:2021zyv ; CMS:2020osd ; Akeroyd:2022ouy . The $H^{\pm}\rightarrow W^{\pm}\varphi^{0}/W^{\pm}A$ modes have been extensively studied for production channels such as $t\rightarrow H^{\pm}b$ Dermisek:2012cn ; Arhrib:2020tqk ; Hu:2022gwd ; Fu:2023sng , $pp\rightarrow H^{\pm}\varphi^{0}$ Arhrib:2017wmo ; Mondal:2021bxa ; Arhrib:2022inj ; Kim:2022hvh ; Kim:2022nmm ; Bhatia:2022ugu ; Li:2023btx ; Mondal:2023wib , $pp\rightarrow H^{\pm}A$ Arhrib:2021xmc ; Arhrib:2021yqf , $e^{+}e^{-}\rightarrow H^{+}H^{-}$ Kausar:2020ims , $pp\rightarrow H^{+}H^{-}$ Arhrib:2021xmc ; Arhrib:2021yqf ; Shen:2022yuo , $W^{\pm*}W^{\pm*}\rightarrow H^{\pm}H^{\pm}$ Arhrib:2019ywg , $pp\rightarrow H^{\pm}W^{\mp}$ Krab:2022lih , and $pp\rightarrow H^{\pm}hh$ Kang:2022mdy .

However, one important decay mode has been largely overlooked: the decay of a light charged Higgs boson into an off-shell top quark (denoted as $t^{*}$) and a bottom quark. In the type-I 2HDM, where the Yukawa couplings of $H^{\pm}$ are inversely proportional to $\tan\beta$, the decay $H^{\pm}\rightarrow t^{*}b$ becomes the leading mode for $135{\;{\rm GeV}}\lesssim M_{H^{\pm}}\lesssim m_{t}$, with $H^{\pm}\rightarrow\tau\nu$ as the second leading mode. Thus, it is of great significance to investigate the discovery potential of future high-energy colliders for this decay mode. For the production of charged Higgs bosons, we consider pair production, yielding the final state of $H^{+}H^{-}\rightarrow t^{*}b\tau\nu\rightarrow bbjj\tau\nu$. This approach is necessitated by the stringent constraints from searches for $t\rightarrow bH^{\pm}(\rightarrow\tau\nu)$, which limit the branching ratio $\text{Br}(t\rightarrow bH^{\pm})$ to below $\mathcal{O}(10^{-4})$ in the type-I 2HDM.

We will rigorously investigate the discovery potential of the HL-LHC and a 100 TeV $pp$ collider for the signal $H^{+}H^{-}\rightarrow t^{*}b\tau\nu\rightarrow bbjj\tau\nu$, employing both cut-flow strategies and the Boosted Decision Tree (BDT) method. The analysis reveals that these efforts are unfortunately unsuccessful due to the softness of the $b$ jets. Given these challenges, the focus of our research then shifts to the multi-TeV Muon Collider (MuC). This facility promises to offer a higher boost for the $b$ jets, potentially enhancing their detectability and opening new avenues for exploration.

The MuC stands out as a powerful tool for BSM searches Capdevilla:2020qel ; Bandyopadhyay:2021pld ; Sen:2021fha ; Asadi:2021gah ; Huang:2021nkl ; Choi:1999kn ; Han:2020uak ; Han:2021udl ; Jueid:2021avn ; Han:2022edd ; Black:2022qlg ; Belfkir:2023vpo ; Jueid:2023zxx , thanks to its clean collision environment, higher energy reach, reduced beamstrahlung, and efficient energy use. The prospects of the MuC program have been significantly enhanced by recent advancements in addressing critical challenges, such as cooling muon beams Antonelli:2013mmk ; Antonelli:2015nla and reducing beam-induced backgrounds (BIB) Collamati:2021sbv ; Ally:2022rgk .

Our study will conduct a signal-to-background analysis for two collider configurations: $\sqrt{s}=3\text{ TeV}$ with a total integrated luminosity of $1\text{ ab}^{-1}$ and $\sqrt{s}=10\text{ TeV}$ with a total integrated luminosity of $10\text{ ab}^{-1}$. The analysis aims to demonstrate that the entire mass range of $M_{H^{\pm}}\in[130,170]\text{ GeV}$ can achieve a high signal significance, surpassing the $5\sigma$ discovery threshold. These findings represent our main contributions to the study of the light charged Higgs boson.

The paper is organized as follows. In Sec. II, we briefly review the type-I 2HDM with CP invariance and softly broken $Z_{2}$ parity. Based on the results of random scans incorporating theoretical and experimental constraints, we investigate the characteristic features of the allowed parameters and suggest the golden channel to probe the unexplored $H^{\pm}\rightarrow t^{*}b$ decay mode, $H^{+}H^{-}\rightarrow t^{*}b\tau\nu\rightarrow bbjj\tau\nu$. Section III deals with the signal-to-background analysis at the HL-LHC and a 100 TeV $pp$ collider, incorporating comprehensive cut-based analysis and the BDT. In Sec. IV, we turn to the multi-TeV MuC and perform the signal-to-background analysis. Conclusions are presented in Sec. V.

The 2HDM introduces two complex $SU(2)_{L}$ Higgs doublet scalar fields, $\Phi_{1}$ and $\Phi_{2}$, with hypercharge $Y=+1$ Branco:2011iw :

where $v_{1}$ and $v_{2}$ denote the non-zero vacuum expectation values of $\Phi_{1}$ and $\Phi_{2}$, respectively. The ratio of $v_{2}$ to $v_{1}$ defines the mixing angle $\beta$ through $\tan\beta=v_{2}/v_{1}$. The electroweak symmetry is spontaneously broken by $v=\sqrt{v_{1}^{2}+v_{2}^{2}}\approx 246{\;{\rm GeV}}$.

To prevent flavor-changing neutral currents at the tree level, a discrete $Z_{2}$ symmetry is imposed, under which $\Phi_{1}\rightarrow\Phi_{1}$ and $\Phi_{2}\rightarrow-\Phi_{2}$ Glashow:1976nt ; Paschos:1976ay . Assuming CP invariance and allowing for softly broken $Z_{2}$ parity, the scalar potential is defined as follows:

In 2HDM, there are five distinct physical Higgs bosons: the lighter CP-even scalar $h$, the heavier CP-even scalar $H$, the CP-odd pseudoscalar $A$, and a pair of charged Higgs bosons $H^{\pm}$. The relationships between these Higgs states and the weak eigenstates described in Equation 3 are determined by two mixing angles, $\alpha$ and $\beta$, which can be found in Ref. Song:2019aav . The SM Higgs boson ${h_{\rm SM}}$ is a linear combination of $h$ and $H$, specifically as ${h_{\rm SM}}=\sin(\beta-\alpha)h+\cos(\beta-\alpha)H$.

According to the assignment of $Z_{2}$ parity to the right-handed fermions, the model has four variants, type-I, type-II, type-X, and type-Y. The mass of $H^{\pm}$ in type-II and type-Y is heavily constrained by the measurements of the inclusive $B$-meson decay into $s\gamma$, requiring $M_{H^{\pm}}\gtrsim 800{\;{\rm GeV}}$ Misiak:2020vlo . Only type-I and type-X allow for the existence of charged Higgs bosons lighter than the top quark. Therefore, we focus on type-I in this study.

We employ two popular conditions: the Higgs alignment limit for the SM-like Higgs boson Carena:2013ooa ; Celis:2013rcs ; Cheung:2013rva ; Bernon:2015qea ; Chang:2015goa ; Das:2015mwa ; Kanemura:2021dez and the mass degeneracy of $H$ and $A$ for the electroweak precision data Kanemura:2011sj ; Chang:2015goa ; Chen:2019pkq . The Higgs alignment limit precludes the decay channel $H^{\pm}\to W^{\pm}h$. Furthermore, we restrict our analysis to scenarios where the charged Higgs boson is lighter than $H$ and $A$. Under this condition, the charged Higgs boson decays exclusively into fermion pairs, ensuring that the $H^{\pm}\rightarrow t^{*}b$ mode maintains a significant branching ratio. Conversely, if $M_{H/A}<M_{H^{\pm}}$, the decay modes $H^{\pm}\rightarrow W^{\pm(*)}H/A$ become accessible²²2For a detailed phenomenological study of the scenario where $M_{H/A}<M_{H^{\pm}}$, see Ref. Sanyal:2023pfs ., suppressing the branching ratio of $H^{\pm}\rightarrow t^{*}b$.

In summary, our model configuration is as follows:

The Yukawa couplings of the charged Higgs boson to the SM fermions in type-I 2HDM are given by

where $P_{\pm}=(1\pm\gamma^{5})/2$. Because these Yukawa couplings have a common factor of $1/\tan\beta$, the branching ratio of $H^{\pm}\rightarrow f\bar{f}$ is independent of $\tan\beta$.

We also present the gauge interactions of a pair of charged Higgs bosons, crucial for the pair production at high-energy colliders Branco:2011iw :

where $s_{W}=\sin\theta_{W}$, $c_{W}=\cos\theta_{W}$, and $\theta_{W}$ is the electroweak mixing angle.

To study the characteristics of the permissible parameter space for the light charged Higgs boson that predominantly decays into $t^{*}b$, we set $M_{H^{\pm}}=150{\;{\rm GeV}}$ and perform a random scan within the following parameter ranges:

The scan is conducted while imposing both theoretical requirements and experimental constraints.

For the theoretical requirements, we enforce conditions ensuring vacuum stability Ivanov:2008cxa ; Barroso:2012mj ; Barroso:2013awa , a bounded-from-below Higgs potential Ivanov:2006yq , tree-level unitarity in scalar-scalar scatterings Branco:2011iw ; Arhrib:2000is , and perturbativity of the Higgs quartic couplings Chang:2015goa . These conditions are evaluated using the public code 2HDMC Eriksson:2009ws . Additionally, we require that the cutoff scale exceed $10{\;{\rm TeV}}$, where the cutoff scale is defined as the energy level at which any of the conditions for tree-level unitarity, perturbativity, or vacuum stability is violated Kim:2023lxc . The evolution of the model parameters via the renormalization group equations is facilitated using the public code 2HDME Oredsson:2018yho ; Oredsson:2018vio .

For the experimental constraints, we incorporated measurements at the 95% confidence level, encompassing inclusive $B$-meson decay into $X_{s}\gamma$ Arbey:2017gmh ; Sanyal:2019xcp ; Misiak:2017bgg and direct search bounds from LEP, Tevatron, and LHC experiments. For the direct search constraints, we have employed the public code HiggsBounds-v5.10.2 Bechtle:2013wla . Notably, our adoption of the Higgs alignment limit ensures that the Higgs precision data at the LHC are inherently satisfied.

In Figure 1, we present the allowed parameter points in the $(M_{H/A},\tan\beta)$ plane for the given $M_{H^{\pm}}=150{\;{\rm GeV}}$. The color code denotes $m_{12}^{2}$. A notable feature is the upper bound on the masses of $H$ and $A$, with $M_{H/A}\lesssim 236{\;{\rm GeV}}$. Although our focus in this paper is on the charged Higgs boson, the relatively low upper bounds on $M_{H/A}$ suggest promising discovery prospects for the heavy neutral Higgs bosons at high-energy colliders. Additionally, we observe lower bounds on $\tan\beta$, specifically $\tan\beta\gtrsim 6$.

Now, let us identify the optimal production mechanism at the LHC for the light charged Higgs boson. LHC searches have primarily focused on its production through top quark decay, $t\rightarrow bH^{\pm}$, followed by $H^{\pm}\rightarrow\tau\nu$ ATLAS:2018gfm ; Sirunyan:2019hkq because the Yukawa couplings of $H^{\pm}$ are proportional to the fermion mass with a common coupling modifier $1/\tan\beta$. Despite comprehensive searches, no new signals have been observed, leading to stringent upper limits on the product of the two branching ratios, $\text{Br}(t\rightarrow bH^{\pm})\text{Br}(H^{\pm}\rightarrow\tau\nu)$. As $\text{Br}(t\rightarrow bH^{\pm})$ depends solely on $\tan\beta$ for the given $M_{H^{\pm}}$, the observed upper bound strictly limits $\text{Br}(t\rightarrow bH^{\pm})$. For $M_{H^{\pm}}\sim 100{\;{\rm GeV}}$, $\text{Br}(t\rightarrow bH^{\pm})$ must be below $\mathcal{O}(10^{-4})$. Consequently, leveraging top quark decay for $H^{\pm}$ production proves ineffective.

Given this constraint, the production channels of $H^{\pm}$ via the decay of heavier Higgs states, such as $H/A\rightarrow H^{\pm}W^{\mp(*)}$, have been extensively investigated Arhrib:2017wmo ; Mondal:2021bxa ; Arhrib:2021xmc ; Cheung:2022ndq ; Bhatia:2022ugu ; Kim:2022hvh ; Kim:2022nmm ; Arhrib:2022inj ; Kim:2023lxc ; Li:2023btx ; Mondal:2023wib . The production of $H$ or $A$ occurs through gluon fusion ($gg\rightarrow H/A$) or associated production ($gg\rightarrow H\rightarrow AZ$, $q\bar{q}\rightarrow Z\rightarrow HZ$). However, the signal rates are sensitive to model parameters, such as $M_{H/A}$ and $\tan\beta$. Furthermore, the allowed parameter space shown in Figure 1 does not permit the on-shell decay $H/A\rightarrow H^{\pm}W^{\mp}$, preventing the effective suppression of backgrounds by exploiting the $W$ boson mass constraint.

A more promising production mechanism is the pair production of charged Higgs bosons via the Drell-Yan process. This channel offers a straightforward and model-independent avenue for $H^{\pm}$ production, as the production cross section is exclusively determined by the mass of the charged Higgs boson. Although this production channel has been studied for the decays $H^{\pm}\rightarrow\tau\nu$ Duarte:2024zeh and $H^{\pm}\rightarrow W^{\pm(*)}\varphi^{0}/A$ Arhrib:2021xmc ; Arhrib:2021yqf ; Shen:2022yuo , it has not been explored for our target decay mode $H^{\pm}\rightarrow t^{*}b$. Therefore, we focus our analysis on the pair production channel for the $H^{\pm}\rightarrow t^{*}b$ decay mode.

Let us delve into the decay of the light charged Higgs boson in the mass range between 130 GeV and 170 GeV. There are two dominant decay channels, $H^{\pm}\rightarrow t^{*}b$ and $H^{\pm}\rightarrow\tau\nu$. Figure 2 depicts their branching ratios as functions of $M_{H^{\pm}}$, comparing scenarios with a single charged Higgs boson (left panel) and a pair of charged Higgs bosons (right panel). Note that these results are independent of $\tan\beta$, $m_{12}^{2}$, or $M_{H/A}$. For the $H^{\pm}\rightarrow t^{*}b$ decay, we incorporate QCD radiative corrections to order $\alpha_{s}^{2}$ in the $\overline{\text{MS}}$ scheme, employing the 2HDMC Eriksson:2009ws . This includes adjustments for running fermion masses in the Higgs couplings, applying leading logarithmic corrections across all orders with the renormalization scale $\mu_{R}=M_{H^{\pm}}$.

The decay mode $H^{\pm}\rightarrow t^{*}b$ exhibits significant branching ratios throughout the target $M_{H^{\pm}}$ range and becomes dominant if $M_{H^{\pm}}\gtrsim 135$ GeV. For $M_{H^{\pm}}\lesssim 135$ GeV, $H^{\pm}\rightarrow t^{*}b$ becomes the subleading decay channel, with $H^{\pm}\rightarrow\tau\nu$ being the leading one. In the pair production of charged Higgs bosons, $H^{\pm}\rightarrow t^{*}b$ plays a more significant role. The $H^{+}H^{-}\rightarrow t^{*}b\,t^{*}b$ mode is leading for $M_{H^{\pm}}\gtrsim 143$ GeV. Interestingly, the process $H^{+}H^{-}\rightarrow t^{*}b\tau\nu$ emerges as the most prominent for $M_{H^{\pm}}\lesssim 143$ GeV and remains the second most dominant for $M_{H^{\pm}}\gtrsim 143$ GeV, which benefits from a combinatorial factor of two. In contrast, the $\tau\nu\tau\nu$ final state, heavily emphasized in prior studies for lighter $H^{\pm}$, exhibits markedly reduced branching ratios.

The results in Figure 2 strongly support investigating the off-shell $t^{*}b$ mode as a potential discovery channel for the light charged Higgs boson within the mass range in 130 to $170{\;{\rm GeV}}$. Since the $t^{*}b\,t^{*}b$ final state faces challenges from the combinatoric complications and the larger QCD backgrounds at the LHC, our investigation targets the following discovery channel for the light charged Higgs boson in the mass range of 130 to 170 GeV:

In the preceding section, we identified the pair production of charged Higgs bosons, followed by $H^{+}H^{-}\rightarrow t^{*}b\tau\nu$, as a key channel for probing the light charged Higgs boson in the mass range of $[130,170]{\;{\rm GeV}}$. This section explores the discovery potential of the HL-LHC and a prospective 100 TeV $pp$ collider, focusing on the case with $M_{H^{\pm}}=150{\;{\rm GeV}}$. For the decay of the off-shell top quark, we consider the hadronic decay channel of the $W$ boson. Our signal process is summarized as:

where $\tau=\tau^{+},\tau^{-}$, $j$ denotes a light quark jet, and the particles in a square bracket represent decay products originating from the same parent particle. The resultant final state includes two jets, two $b$ jets, a tau lepton, and missing transverse energy. The primary background originates from top quark pair production:

For the simulation of the signal and background, we followed a comprehensive procedure. We first calculated parton-level cross sections at the 14 TeV LHC using MadGraph5-aMC@NLO Alwall:2011uj version 3.5.0 with the PDF set to NNPDF31_nlo_as_0118, setting both the renormalization and factorization scales as $\mu_{R}=\mu_{F}=\sum_{i}\sqrt{p_{T,i}^{2}+m_{i}^{2}}$. We generated $3.2\times 10^{6}$ signal events and $1.2\times 10^{7}$ background events. Next, we applied NLO corrections for the signal and approximate N³LO corrections for the background by incorporating the $K$-factor Kim:2024ppt . The parton-level cross sections were determined to be $1.593\times 10{\;{\rm fb}}$ for the signal and $1.028\times 10^{2}{\;{\rm pb}}$ for the background.

Parton showering and hadronization were performed using Pythia version 8.309 Bierlich:2022pfr . For the detector-level analysis, a fast detector response simulation was employed with Delphes deFavereau:2013fsa using the high-luminosity LHC card, delphes_card_HLLHC.tcl. Jet clustering was conducted with FastJet version 3.3.4 Cacciari:2011ma using the anti-$k_{T}$ algorithm with a jet radius of $R=0.4$.

Accurate identification of the final state in Equation 10 critically depends on $b$-tagging and $\tau$-tagging procedures. A jet is designated as a $b$ jet if a $B$ hadron with $p_{T}>5{\;{\rm GeV}}$ is detected within a $\Delta R=0.3$ radius of the jet. Candidate $b$ jets are required to meet the threshold of $p_{T}>25{\;{\rm GeV}}$ and $|\eta|<2.5$, after which the $b$-tagging efficiency is applied. Charm and other light quark jets can be mistagged as $b$ jets. The efficiencies for $b$ tagging and mistagging depend on the jet’s kinematics and are approximately ATL-PHYS-PUB-2016-026 ; ATL-PHYS-PUB-2017-001 :

Identification of the tau lepton is feasible when it decays hadronically, denoted as ${\tau_{\rm h}}$, marked by a collimated jet with a sparse number of hadrons CMS:2007sch ; Bagliesi:2007qx ; CMS:2018jrd . The Delphes default settings for $\tau$ tagging and mistagging efficiencies were applied, which are approximately

After completing the detector simulation, the following basic selection criteria are imposed:

• •

  $N_{j}\geq 2$, $N_{b}\geq 2$, and $N_{\tau_{\rm h}}\geq 1$, where $j$, $b$, and ${\tau_{\rm h}}$ satisfy $p_{T}>25$ GeV and $|\eta|<2.5$;

• •

  $E^{\rm miss}_{T}>25$ GeV.

The presence of neutrinos in the decay chain of $H^{\pm}\rightarrow\tau\nu$ necessitates a minimum threshold for the missing transverse energy ${E_{T}^{\rm miss}}$.

Despite the relatively loose selection criteria, the basic selection results in an exceedingly low acceptance rate for the signal, approximately 1%. In contrast, the background acceptance rate is substantially higher, around 2.7%. With the suppressed signal cross section after the basic selection being only $1.852\times 10^{-1}{\;{\rm fb}}$, even the final projected luminosity of $3{\;{\rm ab}^{-1}}$ at the HL-LHC would result in merely a few hundred signal events. This scarcity of signal events severely limits the ability to devise an effective strategy using kinematic cuts to disentangle the signal from the backgrounds, irrespective of their efficiency.

A primary factor contributing to the low signal acceptance after basic selection is the low transverse momentum (softness) of the $b$ jets originating from the decay $H^{\pm}\rightarrow t^{*}b$. This softness is due to the small mass difference between $M_{H^{\pm}}$ and $m_{t}$, which is crucial for ensuring a substantial branching ratio for the $H^{\pm}\rightarrow t^{*}b$ decay mode, as depicted in Figure 2. Additionally, the $b$ quark from the decay of the off-shell top quark, $t^{*}\rightarrow Wb$, also exhibits lower transverse momentum compared to its on-shell counterpart. The challenge with these soft $b$ jets is that the majority of the subleading $b$ jets fail to meet the minimum transverse momentum threshold ($p_{T}>25{\;{\rm GeV}}$) required for jet clustering.

To illustrate the softness of the $b$ jets, we present in Figure 3 the parton-level $p_{T}$ distributions of the leading and subleading $b$ jets for the signal process. We order jets by their descending $p_{T}$. The left panel shows the results at the 14 TeV LHC, while the right panel displays the results for a 100 TeV $pp$ collider. It is evident that approximately 70% of the subleading $b$ jets fail to surpass the $p_{T}>25{\;{\rm GeV}}$ threshold. This minimum $p_{T}$ threshold for $b$ jets cannot be relaxed because it plays a pivotal role in jet clustering algorithms, primarily aimed at reducing noise from low-energy particles, suppressing background, and enhancing computational efficiency. Moreover, lowering this $p_{T}$ threshold is counterproductive for the signal-to-background analysis, as the background acceptance would increase more than the signal acceptance.

Even at a higher collision energy of 100 TeV, as shown in the right panel, both the leading and subleading $b$ jets remain as soft as those at the HL-LHC. The $b$ jets do not receive a substantial boost because parton-parton collisions at hadron colliders do not fully utilize the beam energy.

Given the limited number of signal events after the basic selection, we need to devise a strategic approach using kinematic cuts that retain as many signal events as possible while effectively suppressing the backgrounds. With this goal in mind, we examined various kinematic distributions and identified key variables that could discriminate the signal from the background.

One set of crucial discriminating variables are the angular separations, defined by $\Delta R\equiv\sqrt{(\Delta\eta)^{2}+(\Delta\phi)^{2}}$, among $b$ jets and light jets. These variables are efficient because the signal channel $pp\rightarrow H^{+}H^{-}\rightarrow[jjbb][\tau\nu]$ results in a smaller angular separation within the $[jjbb]$ grouping. In contrast, the background channel $pp\rightarrow t\bar{t}\rightarrow[bjj][b\tau\nu]$ forms a back-to-back topology between $[bjj]$ and $[b\tau\nu]$, leading to a larger $\Delta R$ between two $b$ jets as well as a larger $\Delta R$ between $j$ and the $b$ in the $[b\tau\nu]$ system.

In Figure 4, we present normalized distributions of two representative angular separations: $\Delta R(b_{1},b_{2})$ in the left panel and $\Delta R(j_{1},b_{1})$ in the right panel. For the signal $pp\rightarrow H^{+}H^{-}\rightarrow[jjbb][\tau\nu]$, the $\Delta R(b_{1},b_{2})$ distribution peaks near 0.4, indicating the close proximity of the two $b$ jets originating from the same parent particle $H^{\pm}$. In contrast, the background $pp\rightarrow t\bar{t}\rightarrow[bjj][b\tau\nu]$ exhibits a broader $\Delta R(b_{1},b_{2})$ distribution with a dominant peak near 3. This reflects the back-to-back motion of two $b$ jets, each originating from a different parent top quark. Moreover, the $\Delta R(j_{1},b_{1})$ distributions further highlight differences between the signal and background. The signal’s $\Delta R(j_{1},b_{1})$ distribution peaks at a lower value of approximately 0.8, whereas the background peaks at higher values near 3.

Based on these distinct features in the angular separation distributions, we impose the following $\Delta R$ cuts to suppress the background while retaining a significant fraction of the signal events:

Other crucial discriminating variables pertain to the reconstruction of the charged Higgs boson mass $M_{H^{\pm}}$. In the signal process $H^{+}H^{-}\rightarrow[bbjj][\tau\nu]$, $M_{H^{\pm}}$ can be measured in two complementary ways: through the invariant mass of the $[bbjj]$ system and the transverse mass derived from the $[\tau\nu]$ system. The transverse mass $M_{T}(X)$ is a useful variable defined for a visible particle $X$ and the missing transverse energy $\vec{E}_{T}^{\rm miss}=-\sum_{i}\vec{p}_{T}^{,i}$, where $i$ covers all observed particles:

where $E_{T}^{X}=\sqrt{m_{X}^{2}+(p_{T}^{X})^{2}}$.

For the signal process $H^{\pm}\rightarrow\tau\nu$, with the visible particle being the tau lepton, $M_{T}(\tau)$ is expected to peak at the charged Higgs boson mass $M_{H^{\pm}}$. Since both the invariant mass $M_{bbjj}$ and the transverse mass $M_{T}(\tau)$ should reconstruct the same $M_{H^{\pm}}$ for the signal events, we define an asymmetry variable $\mathcal{A}_{M}$ to quantify the difference between these two mass observables:

By imposing an appropriate upper bound on $\mathcal{A}_{M}$, we can efficiently separate the signal from the background.

In Figure 5, we present the normalized distributions of $\mathcal{A}_{M}$ (left) and the invariant mass $M(j_{1}j_{2}b_{1}b_{2})$ (right) for the signal with $M_{H^{\pm}}=150{\;{\rm GeV}}$ (blue) and the $t\bar{t}$ background (orange), after imposing the $\Delta R$ cuts in Equation 14. Two distinct features are evident. First, as expected, the $\mathcal{A}_{M}$ distribution for the signal peaks sharply at $\mathcal{A}_{M}\simeq 0$, while the background prefers larger values above 0.2. Imposing an upper bound on $\mathcal{A}_{M}$ will efficiently separate the signal events from the background.

The second notable feature is observed in the invariant mass distribution $M(j_{1}j_{2}b_{1}b_{2})$. For the signal, it exhibits a resonance peak around 130 GeV, which is lower than the true charged Higgs boson mass of 150 GeV. This discrepancy is attributed to several factors, including neutrinos in $B$ meson decays, imperfect jet energy resolution, and gluon radiation not fully captured within the jet clustering cone. Despite this shift, the resonance peak in the $M(j_{1}j_{2}b_{1}b_{2})$ distribution provides a distinct signature for the signal process.

To evaluate the discovery potential for this channel, we present in Table 1 the cut-flow for the signal process $pp\rightarrow H^{+}H^{-}\rightarrow t^{*}b\tau\nu\rightarrow jjbb\tau\nu$ at the 14 TeV LHC. The table shows the cross sections of the signal and background after each cut, as well as the significance considering a 10% background uncertainty for an integrated luminosity of $3{\;{\rm ab}^{-1}}$. The significance is defined as:

where $N_{\rm s}$ denotes the number of signal events, $N_{\rm b}$ the number of background events, and $\delta_{\rm bg}=\Delta_{\rm B}N_{\rm b}$ the background uncertainty yield.

Despite applying the key kinematic cuts, the final significance reaches only 0.19. While our final selection cut boosts the signal significance by approximately 300-fold relative to the basic selection phase, the significance remains substantially below a detectable level. Moreover, the signal cross section after the final selection, approximately $4.65\times 10^{-3}{\;{\rm fb}}$, precludes any further refinements through cuts. This cut-based analysis highlights the challenges in probing the charged Higgs boson through the $H^{+}H^{-}\rightarrow t^{*}b\tau\nu$ channel at the HL-LHC using kinematic cuts alone, due to the extremely small signal cross section and overwhelming backgrounds.

To assess if the BDT method can enhance sensitivity to the $t^{*}b$ decay mode of the light charged Higgs boson, we employed the Extreme Gradient Boosting (XGBoost) package Chen:2016btl . XGBoost has seen increasing use in the particle physics community for a variety of analyses, including studies on the SM Higgs boson ATLAS:2017ztq ; CMS:2020tkr ; ATLAS:2021ifb ; CMS:2020cga ; CMS:2021sdq , dark matter ATLAS:2021jbf , vectorlike quarks Dasgupta:2021fzw , a composite pseudoscalar Cornell:2020usb , and innovative strategies for faster event generation Bishara:2019iwh . In our study, XGBoost was used as a binary classifier, aimed at more effectively distinguishing between signal and background events.

We initialized the XGBoost classifier with the objective set to binary and the evaluation metric set to logloss. The learning rate was configured at 0.1. For training the model, we generated $3.7\times 10^{4}$ signal events and $3.3\times 10^{5}$ background events, all of which met the basic selection criteria. We divided the dataset into three parts: 50% for training, 20% for validating the algorithm, and 30% for testing. As inputs for the XGBoost model, we used the following 50 variables:

1. 1.

   Angular distance: $\Delta R(b_{1},b_{2})$, $\Delta R(b_{1},j_{1}j_{2})$, $\Delta R(b_{1},j_{1}j_{2}b_{2})$, $\Delta R(b_{1},{\tau_{\rm h}})$, $\Delta R(b_{1}{\tau_{\rm h}},j_{1}j_{2})$, $\Delta R(b_{2},j_{1}j_{2})$, $\Delta R(b_{2},j_{1}j_{2}b_{1})$, $\Delta R(b_{2},{\tau_{\rm h}})$, $\Delta R(b_{2}{\tau_{\rm h}},j_{1}j_{2})$, $\Delta R(j_{1},b_{1})$, $\Delta R(j_{1},b_{2})$, $\Delta R(j_{1},j_{2})$, $\Delta R(j_{1},{\tau_{\rm h}})$, $\Delta R(j_{2},b_{1})$, $\Delta R(j_{2},b_{2})$, $\Delta R(j_{2},{\tau_{\rm h}})$, $\Delta R({\tau_{\rm h}},j_{1}j_{2})$, $\Delta R({\tau_{\rm h}},j_{1}j_{2}b_{1})$, $\Delta R({\tau_{\rm h}},j_{1}j_{2}b_{1}b_{2})$, $\Delta R({\tau_{\rm h}},j_{1}j_{2}b_{2})$.

2. 2.

   Invariant mass $M$: $M(j_{1}j_{2})$, $M(j_{1}j_{2}b_{1})$, $M(j_{1}j_{2}b_{1}b_{2})$, $M(j_{1}j_{2}b_{2})$.

3. 3.

   Transverse mass of $M_{T}$: $M_{T}(b_{1}{\tau_{\rm h}})$, $M_{T}(b_{2}{\tau_{\rm h}})$, $M_{T}({\tau_{\rm h}})$.

4. 4.

   Four momentum of $\left[p_{T},\eta,\phi,m\right]$ for each of $j_{1}$, $j_{2}$, $b_{1}$, $b_{2}$, ${\tau_{\rm h}}$.

5. 5.

   Missing transverse energy and Missing energy azimuthal angle: $E_{T}^{\rm miss}$, $\phi^{\rm miss}$.

6. 6.

   Mass asymmetry: $\mathcal{A}_{M}$.

Here, the multiple particle symbol represents the system consisting of the constituent particles. For example, the term $j_{1}j_{2}$ refers to a system whose momentum is the vector sum of the momenta of $j_{1}$ and $j_{2}$.

In Figure 6, we illustrate the distributions of BDT scores for both the signal (in blue) and the background (in red). These results are derived exclusively from the testing dataset, which the model did not encounter during its training and validation phases. The BDT score distributions reveal a discernible separation between the signal and background. Applying a threshold of 0.98 for the XGBoost score results in cross sections of $1.42\times 10^{-2}{\;{\rm fb}}$ for the signal and $8.53\times 10^{-2}{\;{\rm fb}}$ for the background. Considering a 10% background uncertainty and an integrated luminosity of $3{\;{\rm ab}^{-1}}$, the signal significance reaches 1.35.

Although the signal significance in the BDT analysis shows a roughly sevenfold increase compared to the significance in the cut-based analysis, it remains below the threshold for a confident detection. This challenge is primarily due to the soft $b$ jets in the signal events, which fail to satisfy the basic selection.

To explore whether a 100 TeV $pp$ collider could offer higher discovery potential for the $H^{\pm}\to t^{*}b$ signal, we conducted signal-to-background analyses for the same process, $pp\to H^{+}H^{-}\to t^{*}b\tau\nu\to bbjj\tau\nu$, using the Delphes card FCChh.tcl. Unfortunately, the issue of excessively soft $b$-jets persists even at a 100 TeV $pp$ collider, failing to yield a significance above the detection threshold.

Our cut-based analysis, employing sequential kinematic cuts of $N_{\tau_{\rm h}}\geq 1$ (with $p_{T}^{\tau_{\rm h}}>60{\;{\rm GeV}}$), $N_{b}\geq 2$, $N_{j}\geq 2$, $\Delta R(b_{1},b_{2})<1.5$, ${E_{T}^{\rm miss}}\geq 100{\;{\rm GeV}}$, $\mathcal{A}_{M}<0.7$, and $M(j_{1}j_{2}b_{1}b_{2})<200{\;{\rm GeV}}$, results in a significance of only 0.38 at a 100 TeV $pp$ collider. Moreover, the BDT analysis underperforms compared to the HL-LHC, with the signal significance reaching merely about 0.75.

In conclusion, high-energy hadron colliders, such as the HL-LHC and a prospective 100 TeV $pp$ collider, cannot effectively probe the $t^{*}b$ decay mode of the light charged Higgs boson due to the inherent softness of the $b$-jets in this channel. This limitation highlights the need for alternative approaches or collider technologies to investigate this particular decay mode of the light charged Higgs boson.

In the previous section, we demonstrated that the proposed signal process $pp\rightarrow H^{+}H^{-}\rightarrow t^{*}b\tau\nu$ at the HL-LHC yields very low significance, reaching only 1.35 even with BDT analysis. This low significance is primarily due to the $b$ jets from the decay $H^{\pm}\rightarrow t^{*}b$ being too soft for effective reconstruction. Additionally, a 100 TeV $pp$ collider cannot adequately boost the $b$ jets as the beam energy is not fully transferred to the parton-parton collision. Consequently, we shift our focus to a multi-TeV MuC, which fully harnesses the beam energy in collisions between two fundamental particles. We target two configurations: $\sqrt{s}=3{\;{\rm TeV}}$ with an integrated luminosity of $1{\;{\rm ab}^{-1}}$ and $\sqrt{s}=10{\;{\rm TeV}}$ with an integrated luminosity of $10{\;{\rm ab}^{-1}}$ Han:2021udl ; AlAli:2021let .

For the production of charged Higgs bosons, we consider their pair production with the following final state:

At the MuC, there are three different production channels relevant to this final state:

where $\nu$ and $\bar{\nu}$ include all three neutrino flavors, and $\mu_{\rm f}$ denotes a forward muon with $|\eta|>2.5$.

The first process in Equation 19 is the Drell-Yan process mediated by the $Z$ boson and photon. The second process in Equation 20 involves $H^{\pm}$ pair production associated with a pair of neutrinos. Since these additional neutrinos manifest as missing transverse energy, they create the same phenomenological signature as the final state with a single neutrino, making it indistinguishable from the first process. For this process, there are numerous Feynman diagrams, including those involving $Z^{*}/\gamma^{*}\rightarrow H^{+}H^{-}$ and $Z^{*}\rightarrow\nu\bar{\nu}$. Additionally, Vector Boson Scattering (VBS) processes such as $W^{+}W^{-}\rightarrow\gamma^{*}/Z^{*}/h^{*}\rightarrow H^{+}H^{-}$, VBS processes through the quartic vertex $W^{+}$-$W^{-}$-$H^{+}$-$H^{-}$ in Equation 7, and VBS processes through the $t$-channels mediated by $H$ and $A$ also contribute.

The third process in Equation 21 involves $H^{\pm}$ pair production associated with two forward muons. In the conventional detector design of a multi-TeV MuC, forward muons are not detectable³³3Recently, the integration of forward muon detectors into the MuC’s design has been initiated within the MuC community, as the energetic forward muons can penetrate the tungsten nozzles. This forward muon detector is highly expected to play a crucial role in probing various BSM models Accettura:2023ked ; Ruhdorfer:2023uea ; Forslund:2023reu ; Bandyopadhyay:2024plc . because they fall outside the pseudorapidity coverage limit of $|\eta|<2.5$. This limitation is due to tungsten nozzles designed to shield the detector from BIB particles Collamati:2021sbv ; Ally:2022rgk . This neutral-current VBS process occurs through $ZZ\to h^{*}\to H^{+}H^{-}$, the quartic vertex $Z/\gamma$-$Z/\gamma$-$H^{+}$-$H^{-}$, and the $t$-channels mediated by $H$ and $A$.

Let us discuss the dependence of cross sections for ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\nu\bar{\nu}$ and ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\mu_{\rm f}^{+}\mu_{\rm f}^{-}$ on model parameters. Unlike the Drell-Yan process, these two associated production processes include contributions mediated by $H$ and $A$, making their cross sections potentially sensitive to $M_{H/A}$. Additionally, VBS contributions through $W^{+}W^{-}\rightarrow h^{*}\rightarrow H^{+}H^{-}$ to ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\nu\bar{\nu}$ and $ZZ\to h^{*}\to H^{+}H^{-}$ to ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\mu_{\rm f}^{+}\mu_{\rm f}^{-}$, though not dominant, introduce dependences on $\tan\beta$.

Figure 7 illustrates parton-level cross sections as a function of $M_{H/A}$ for the processes ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\nu\bar{\nu}$ and ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\mu_{\rm f}^{+}\mu_{\rm f}^{-}$ with $M_{H^{\pm}}=150{\;{\rm GeV}}$ at $\sqrt{s}=10{\;{\rm TeV}}$. The color codes denote $\tan\beta$. The analysis considers parameter points allowed by theoretical and experimental constraints discussed in Sec. II. For forward muons $\mu_{\rm f}$, we apply the condition $2.5<|\eta_{\mu}|<4.7413$. The lower bound of $|\eta|>2.5$ ensures that forward muons fall outside the conventional pseudorapidity coverage limit. The upper bound of $|\eta|<4.7413$ is implemented to manage cross section divergence that occurs as the scattering angle $\theta$ approaches 0 or $\pi$, a consequence of $t$-channel photon-mediated diagrams. Typically, this divergence is canceled by higher-order QED corrections, particularly from soft photon emissions. In practice, such divergences are often handled by imposing a minimum scattering angle cut during data analysis. Considering the high collision energy at the MuC, we constrain the scattering angle to $1^{\circ}<\theta<179^{\circ}$, which corresponds to $|\eta|<4.7413$.

Figure 7 clearly demonstrates that both processes exhibit modest dependence of cross sections on the model parameters $M_{H/A}$ and $\tan\beta$. Notably, the cross section for ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\mu_{\rm f}^{+}\mu_{\rm f}^{-}$ remains nearly constant across the allowed parameter space, varying by only about 10%. In contrast, the cross section for ${\mu^{+}\mu^{-}}\to H^{+}H^{-}\nu\bar{\nu}$ shows a more pronounced dependence on the model parameters, with variations of approximately 70%. For a given $\tan\beta$, the cross section initially decreases as $M_{H/A}$ increases, reaches a minimum at $M_{H/A}\simeq 200{\;{\rm GeV}}$, and then rises again.

Given these observations, we adopt a conservative approach by selecting $M_{H}=M_{A}=200{\;{\rm GeV}}$, $\tan\beta=10$, and $m_{12}^{2}=3.84\times 10^{3}{\;{\rm GeV}}^{2}$ for our signal-to-background analysis. This choice represents a pessimistic scenario, ensuring that if we achieve signal significance above the discovery potential, it would guarantee the accessibility of the entire parameter space through our target signal $H^{\pm}\to t^{*}b$.