Measurement of branching fractions and search for $C\!P$ violation in $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$, $D^{0}\rightarrow K^{+}K^{-}\eta$, and $D^{0}\rightarrow\phi\eta$ at Belle

Singly Cabibbo-suppressed (SCS) decays of charmed mesons provide a promising opportunity to study $C\!P$ violation in the charm sector. Within the Standard Model, $C\!P$ violation in charm decays is expected to be of the order of $10^{-3}$ or smaller bib:PRD85o034036 ; bib:PRD75d036008 , and thus challenging to observe. SCS decays are of special interest, as interference that includes a new physics amplitude could lead to large $C\!P$ violation. The $C\!P$ asymmetry between $D^{0}\rightarrow f$ and $\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow\bar{f}$ decays ($A_{C\!P}$) is defined as

The only observation of $C\!P$ violation in the charm sector to date is from the LHCb experiment, where a difference in $A_{C\!P}$ between the SCS $D^{0}\rightarrow K^{+}K^{-}$ and $D^{0}\rightarrow\pi^{+}\pi^{-}$ decays bib:CPVobservation was observed: $\Delta A_{C\!P}=(-15.4\pm 2.9)\times 10^{-4}$. In this paper, we investigate two analogous SCS decays, $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$ and $D^{0}\rightarrow K^{+}K^{-}\eta$. A search for a $C\!P$ asymmetry in the first decay was performed by the BESIII experiment; the resulting precision was 6% bib:PRD101d052009 . There have been no results for $D^{0}\rightarrow K^{+}K^{-}\eta$ decays to date. Theoretically, it is difficult to predict $C\!P$ asymmetries for three-body decays, while some predictions exist for intermediate two-body processes: $A_{C\!P}(D^{0}\rightarrow\rho^{0}\eta)$ is predicted to be $-0.53\times 10^{-3}$ from tree amplitudes alone, and $-0.23\times 10^{-3}$ after considering QCD-penguin and weak penguin-annihilation bib:PRD85o034036 . The asymmetry $A_{C\!P}(D^{0}\rightarrow\phi\eta)$ is predicted to be zero in several theoretical models bib:PRD85o034036 . A precise measurement of branching fractions ($\mathcal{B}$) for these three-body decays is an important step towards searching for $C\!P$ violation in these channels.

In this paper we utilize the full Belle data sample of 980 $\mbox{\,fb}^{-1}$ to measure $\mathcal{B}$ and $A_{C\!P}$ for three SCS decays: $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$, $D^{0}\rightarrow K^{+}K^{-}\eta$, and $D^{0}\rightarrow\phi\eta$. All $\mathcal{B}$ measurements are performed relative to the Cabibbo-favored (CF) decay $D^{0}\rightarrow K^{-}\pi^{+}\eta$, which has been well-measured (with a fractional uncertainty $\delta\mathcal{B}/\mathcal{B}\sim 3\%$ bib:PDG2020 ) by both Belle bib:PRD102d012002 and BESIII bib:PRL124d241803 . The current world average for $\mathcal{B}(D^{0}\rightarrow\pi^{+}\pi^{-}\eta)$ has a fractional uncertainty $\delta\mathcal{B}/\mathcal{B}\sim 6\%$ bib:PDG2020 . The branching fraction for $D^{0}\rightarrow\phi\eta$ was previously measured by Belle with 78 $\mbox{\,fb}^{-1}$ of data bib:PRL92d101803 ; the measurement reported here uses an order of magnitude more data and supersedes that result. BESIII found evidence for $D^{0}\rightarrow\phi\eta$ ($4.2\sigma$) bib:PLB798d135017 and observed a non-$\phi$ $D^{0}\rightarrow K^{+}K^{-}\eta$ component ($5.2\sigma$) bib:PRL124d241803 .

To identify the flavor of the neutral $D$ meson when produced, we reconstruct $D^{*+}\rightarrow D^{0}\pi^{+}_{s}$ and $D^{*-}\rightarrow\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}\pi^{-}_{s}$ decays; the charge of the daughter $\pi^{\pm}_{s}$ (which has low momentum and is referred to as the “slow” pion) identifies whether the $D$ meson is $D^{0}$ or $\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}$. The raw asymmetry measured ($A_{\rm raw}$) receives contributions from several sources:

where $A_{C\!P}^{D^{0}\rightarrow f}$ is the $C\!P$ asymmetry for $D^{0}\rightarrow f$; $A_{\rm FB}^{D^{*+}}$ is the forward-backward asymmetry due to $\gamma$-$Z^{0}$ interference and higher-order QED effects bib:BROWN1973403 in $e^{+}e^{-}\rightarrow c\bar{c}$ collisions; and $A_{\varepsilon}^{\pi_{s}}$ is the asymmetry resulting from a difference in reconstruction efficiencies between $\pi^{+}_{s}$ and $\pi^{-}_{s}$. This asymmetry depends on the transverse momentum $p_{T}(\pi_{s})$ and polar angle $\theta(\pi_{s})$ of the $\pi_{s}$ in the laboratory frame. We correct for this by weighting signal events by a factor $[1-A_{\varepsilon}^{\pi_{s}}(p_{T},\cos\theta)]$ for $D^{0}$ decays, and by a factor $[1+A_{\varepsilon}^{\pi_{s}}(p_{T},\cos\theta)]$ for $\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}$ decays. After this weighting, we are left with the $\pi_{s}$-corrected asymmetry

Since $A_{\rm FB}$ is an odd function of the cosine of the $D^{*+}$ polar angle $\theta^{*}$ in the $e^{+}e^{-}$ center-of-mass (CM) frame, and $A_{C\!P}$ is independent of $\cos\theta^{*}$, we extract $A_{C\!P}$ and $A_{\rm FB}(\cos\theta^{*})$ via

Fitting the values of $A_{C\!P}$ for different $\cos\theta^{*}$ bins to a constant gives our final measurement of $A_{C\!P}$ for $D^{0}\rightarrow f$.

This measurement is based on the full data set of the Belle experiment, which corresponds to a total integrated luminosity of 980 $\mbox{\,fb}^{-1}$ bib:BelleDetector2 collected at or near the $\Upsilon(nS)$ ($n=1$, 2, 3, 4, 5) resonances. The Belle experiment ran at the KEKB energy-asymmetric collider bib:KEKB ; bib:KEKB2 . The Belle detector is a large-solid-angle magnetic spectrometer consisting of a silicon vertex detector (SVD), a $50$-layer central drift chamber (CDC), an array of aerogel threshold Cherenkov counters (ACC), a barrel-like arrangement of time-of-flight scintillation counters (TOF), and an electromagnetic calorimeter comprising CsI(Tl) crystals located inside a superconducting solenoid coil providing a $1.5$ T magnetic field. An iron flux-return located outside the coil is instrumented to detect $K_{L}^{0}$ mesons and to identify muons. A detailed description of the detector is given in Refs. bib:BelleDetector ; bib:BelleDetector2 .

We use Monte Carlo (MC) simulated events to optimize selection criteria, study backgrounds, and evaluate the signal reconstruction efficiency. Signal MC events are generated by EVTGEN bib:evtgen and propagated through a detector simulation based on GEANT3 bib:geant3 . Final-state radiation from charged particles is simulated using the PHOTOS package bib:PHOTOS . Three-body decays are generated according to phase space. An MC sample of “generic” events, corresponding to an integrated luminosity four times that of the data, is used to develop selection criteria. It includes $B\overline{B}$ events and continuum processes $e^{+}e^{-}\rightarrow q\bar{q}$, where $q=u,d,s,c$. At the $\Upsilon(5S)$ resonance, the MC includes $B^{(*)0}_{s}\,\overline{B}{}^{(*)0}_{s}$ events. Selection criteria are optimized by maximizing a figure-of-merit $N_{\rm sig}/\sqrt{N_{\rm sig}+N_{\rm bkg}}$, where $N_{\rm sig}$ and $N_{\rm bkg}$ are the numbers of signal and background events, respectively, expected in a two-dimensional signal region in variables $M$ and $Q$. The variable $M$ is the invariant mass of the $h^{+}h^{-}\eta$ ($h=\pi,K$) combination, and $Q=[M(h^{+}h^{-}\eta\,\pi^{+}_{s})-M(h^{+}h^{-}\eta)-m_{\pi^{+}_{s}}]\cdot c^{2}$ is the kinetic energy released in the $D^{*+}$ decay.

We reconstruct the signal decays $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$ and $D^{0}\rightarrow K^{+}K^{-}\eta$, and the reference decay $D^{0}\rightarrow K^{-}\pi^{+}\eta$, in which the $D^{0}$ originates from $D^{*+}\rightarrow D^{0}\pi^{+}$, as follows.¹¹1Throughout this paper, charge-conjugate modes are implicitly included unless stated otherwise. Charged tracks are identified as $K^{\pm}$ or $\pi^{\pm}$ candidates using a likelihood ratio $\mathcal{R}_{K}\equiv\mathcal{L}_{K}/(\mathcal{L}_{K}+\mathcal{L}_{\pi})$, where $\mathcal{L}_{K}$ ($\mathcal{L}_{\pi}$) is the likelihood that a track is a $K^{\pm}$ ($\pi^{\pm}$) based on the photon yield in the ACC, $dE/dx$ information in the CDC, and time-of-flight information from the TOF bib:PID . Tracks having $\mathcal{R}_{K}>0.60$ are identified as $K^{\pm}$ candidates; otherwise, they are considered as $\pi^{\pm}$ candidates. The corresponding efficiencies are approximately $90\%$ for kaons and $95\%$ for pions. Tracks that are highly electron-like ($\mathcal{R}_{e}>0.95$) or muon-like ($\mathcal{R}_{\mu}>0.95$) are rejected, where the electron and muon likelihood ratios $\mathcal{R}_{e}$ and $\mathcal{R}_{\mu}$ are determined mainly using information from the ECL and KLM detectors, respectively bib:NIMA485d490 ; bib:NIMA491d69 . Charged tracks are required to have at least two SVD hits in the $+z$ direction (defined as the direction opposite that of the positron beam), and at least two SVD hits in the $x$-$y$ (transverse) plane. The nearest approach of the $\pi_{s}^{+}$ track to the $e^{+}e^{-}$ interaction point (IP) is required to be less than 1.0 cm in the $x$-$y$ plane, and less than 3.0 cm along the $z$ axis.

Photon candidates are identified as energy clusters in the ECL that are not associated with any charged track. The photon energy ($E_{\gamma}$) is required to be greater than 50 MeV in the barrel region (covering the polar angle $32^{\circ}<\theta<129^{\circ}$), and greater than 100 MeV in the endcap region ($12^{\circ}<\theta<31^{\circ}$ or $132^{\circ}<\theta<157^{\circ}$). The ratio of the energy deposited in the $3\times 3$ array of crystals centered on the crystal with the highest energy, to the energy deposited in the corresponding $5\times 5$ array of crystals, is required to be greater than 0.80.

Candidate $\eta\rightarrow\gamma\gamma$ decays are reconstructed from photon pairs having an invariant mass satisfying $500~{\rm MeV}/c^{2}<M(\gamma\gamma)<580~{\rm MeV}/c^{2}$. This range corresponds to about $3\sigma$ in $M(\gamma\gamma)$ resolution. The absolute value of the cosine of the $\eta\rightarrow\gamma_{1}\gamma_{2}$ decay angle, defined as $\cos\theta_{\eta}\equiv E(\eta)/p(\eta)\cdot(E_{\gamma_{1}}-E_{\gamma_{2}})/(E_{\gamma_{1}}+E_{\gamma_{2}})$, is required to be less than 0.85. This retains around 89% of the signal while reducing backgrounds by a factor of two. To further suppress backgrounds, we remove $\eta$ candidates in which both photon daughters can be combined with other photons in the event to form $\pi^{0}\rightarrow\gamma\gamma$ candidate decays satisfying $|M_{\gamma\gamma}-m_{\pi^{0}}|<10$ MeV/$c^{2}$, where $m_{\pi^{0}}$ is the nominal $\pi^{0}$ mass bib:PDG2020 . This veto requirement has an efficiency of 95% while reducing backgrounds by a factor of three ($D^{0}\rightarrow K^{+}K^{-}\eta$) and four ($D^{0}\rightarrow\pi^{+}\pi^{-}\eta$).

Candidate $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$, $D^{0}\rightarrow K^{+}K^{-}\eta$, and $D^{0}\rightarrow K^{-}\pi^{+}\eta$ decays are reconstructed by combining $\pi^{\pm}$ and $K^{\pm}$ tracks with $\eta$ candidates. A vertex fit is performed with the two charged tracks to obtain the $D^{0}$ decay vertex position; the resulting fit quality is labeled $\chi^{2}_{v}$. To improve the momentum resolution of the $\eta$, the $\gamma$ daughters are subjected to a fit in which the photons are required to originate from the $D^{0}$ vertex position, and the invariant mass is constrained to be that of the $\eta$ meson bib:PDG2020 . The fit quality of this mass constraint ($\chi_{m}^{2}$) is required to satisfy $\chi_{m}^{2}<8$, and the resulting $\eta$ momentum is required to be greater than 0.70 GeV/$c$. For $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$ candidates, we veto events in which $|M(\pi^{+}\pi^{-})-m_{K^{0}_{\scriptscriptstyle\rm S}}|<10$ MeV/$c^{2}$, where $m_{K^{0}_{\scriptscriptstyle\rm S}}$ is the nominal $K^{0}_{\scriptscriptstyle\rm S}$ mass bib:PDG2020 , to suppress background from CF $D^{0}\rightarrow K^{0}_{\scriptscriptstyle\rm S}\,\eta$ decays. This veto range corresponds to about $3\sigma$ in resolution. The $D^{0}$ invariant mass $M$ is required to satisfy $1.850~{\rm GeV}/c^{2}<M<1.878~{\rm GeV}/c^{2}$ for $D^{0}\rightarrow K^{+}K^{-}\eta$ candidates; $1.840~{\rm GeV}/c^{2}<M<1.884~{\rm GeV}/c^{2}$ for $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$ candidates; and $1.842~{\rm GeV}/c^{2}<M<1.882~{\rm GeV}/c^{2}$ for $D^{0}\rightarrow K^{-}\pi^{+}\eta$ candidates. These ranges correspond to about $2\sigma$ in resolution.

Candidate $D^{*+}\rightarrow D^{0}\pi_{s}^{+}$ decays are reconstructed by combining $D^{0}$ candidates with $\pi_{s}^{+}$ tracks. We first fit for $D^{*+}$ decay vertex using the $D^{0}$ momentum vector and decay vertex position, and the IP as a constraint (i.e., the $D^{*+}$ nominally originates from the IP). The resulting goodness-of-fit is labeled $\chi^{2}_{\rm IP}$. To improve the resolution in $Q$, another vertex fit is performed: in this case we constrain the $\pi_{s}^{+}$ daughter to originate from the $D^{*+}$ decay vertex, and the resulting fit quality is labeled $\chi^{2}_{s}$. The sum of the above three fit qualities, $\sum\chi^{2}_{\rm vtx}=\chi^{2}_{v}+\chi^{2}_{\rm IP}+\chi^{2}_{s}$, is required to be less than 50; this requirement has a signal efficiency of about 97%. Those $D^{*+}$ candidates satisfying $0<Q<15$ MeV are retained for further analysis. To eliminate $D^{*+}$ candidates originating from $B$ decays, and to also suppress combinatorial background, the $D^{*+}$ momentum in the CM frame is required to be greater than 2.70 GeV/$c$.

After the above selection criteria are applied, about 2.1% of $D^{0}\rightarrow\pi^{+}\pi^{-}\eta$ events, 1.3% of $D^{0}\rightarrow K^{-}\pi^{+}\eta$ events, and $<0.1$% of $D^{0}\rightarrow K^{+}K^{-}\eta$ events have two or more $D^{*+}$ candidates. For such multi-candidate events, we choose a single candidate: that which has the smallest value of the sum $\sum\chi^{2}_{\rm vtx}+\chi_{m}^{2}(\eta)$. This criterion, according to MC simulation, identifies the correct candidate 54% of the time.