Loading [MathJax]/jax/element/mml/optable/SuppMathOperators.js

Heavy neutral leptons in gauged U(1)LµLτ at muon collider

  • Heavy neutral leptons N are the most appealing candidates to generate tiny neutrino masses. We studied the signature of heavy neutral leptons in gauged U(1)LμLτ at a muon collider. Charged under the U(1)LμLτ symmetry, the heavy neutral leptons can be pair produced via the new gauge boson Z at the muon collider as μ+μZNN and μ+μZ()γNNγ. We then performed a detailed analysis on the lepton number violation signature μ+μNNμ±μ±WW and μ+μNNγμ±μ±WWγ at the 3 TeV muon collider, where the hadronic decays of W boson are treated as fat-jets J. These lepton number violation signatures have quite clean backgrounds at the muon collider. Our simulation shows that a wide range of viable parameter space is within the reach of the 3 TeV muon collider. For instance, with new gauge coupling g=0.6 and an integrated luminosity of 1000 fb1, the μ±μ±JJ signal could probe mZ13 TeV. Meanwhile, if the gauge boson mass satisfies 2mN<mZ<s, the μ±μ±JJγ signature would be more promising than the μ±μ±JJ signature.
  • Heavy neutral leptons N are well motivated to explain the origin of tiny neutrino masses. Due to the singlet nature of heavy neutral leptons under the standard model gauge group, we can write a Majorana mass term mN¯NcN, which results in light Majorana neutrino masses as mνm2D/mN via the type-I seesaw mechanism [1, 2]. To generate the sub-eV neutrino masses, the heavy neutral lepton masses should be at very high scale mN1014 GeV if mD is at the electroweak scale, which is far beyond current experimental reach. Alternatively, an electroweak scale mN is usually assumed for phenomenological studies [3], which is then determined by the mixing parameter VN.

    To confirm the Majorana nature of neutrinos, lepton number violation signatures are expected. The most sensitive experiment is the neutrinoless double beta decay, which could probe the inverted mass ordering scenario in next-generation experiments [4]. Meanwhile, the detection of the lepton number violation signature at colliders could unravel the explicit mechanism of tiny neutrino masses [5]. For heavy neutral leptons in the type-I seesaw, the distinct signature at the hadron collider is ppW±±N±±jj [6]. However, the sensitive region of this signature heavily depends on a relatively large mixing parameter VN [7, 8], which is usually much higher than the natural seesaw prediction VNmν/mN.

    Besides the canonical seesaw, there are many theories with extended gauge groups, such as the U(1)BL model [9, 10] and left-right symmetric model [11, 12]. In these models, the heavy neutral leptons are charged under the extended gauge group, which opens new production mechanisms for heavy neutral leptons. Pair production of heavy neutral leptons via the decay of the Z boson in the framework of the left-right symmetric model has been searched for recently by the CMS collaboration [13]. The region within mN1.4 TeV and mZ4.4 TeV has been excluded by the dimuon channel. Meanwhile, the ATLAS collaboration searches for the right-handed W boson decaying to heavy neural lepton N and leptons , which can exclude the region with mN3.8 TeV and mW6.4 TeV [14].

    The construction of a multi-TeV muon collider was proposed recently [15, 16]. Since then, searches for heavy neutral leptons at the multi-TeV muon collider have drawn increasing interest [1721]. One promising signature is μ+μNν, which could probe the mixing parameter |VμN|2106 at the 10 TeV muon collider [2224]. The Majorana nature of heavy neutral leptons can be confirmed with double peaks of the rapidity distribution of the reconstructed N [25]. Another interesting signature is the lepton number violation signal via vector boson scattering process W±Z/γ±N [26], via associated production process μ+μNW± [27], or at the same-sign muon collider via μ+μ+W+W+ process [28]. These studied signatures also require that the mixing parameter VN is not too small. Meanwhile, under the tight constraints from current experimental searches [13, 14], the first stage of a 3 TeV muon collider [29] is not promising to probe prompt heavy neutral leptons in the U(1)BL and left-right symmetric models.

    The multi-TeV muon collider is a perfect machine to test the muon-philic forces. One attractive option is the gauged U(1)LμLτ model [3034]. Compared to U(1)BL or left-right symmetry, U(1)LμLτ symmetry is less constrained due to the lack of direct couplings to electrons and quarks. For instance, there is only a loose constraint from neutrino trident production when mZ100 GeV [35]. Therefore, this model has been extensively studied to explain the anomaly of the muon magnetic moment [36, 37], B meson anomaly [38, 39], dark matter [40, 41], and neutrino masses [42, 43]. It is shown that a 3 TeV muon collider is powerful enough to probe a relatively large parameter space with mZ10 TeV [44, 45]. In this paper, we consider the gauged U(1)LμLτ model with three heavy neutral leptons Ne,Nμ,Nτ. We then study the pair production of heavy neutral lepton N via Z boson at a 3 TeV muon collider. To test the Majorana nature of neutrinos, we focus on the lepton number violation process μ+μZNNμ±μ±WW and μ+μZγNNγμ±μ±WWγ with the hadronic decay of W.

    The remainder of this paper is organized as follows. In Section II, we review the gauged U(1)LμLτ model with three heavy neutral leptons and discuss relevant experimental constraints. The decay properties of gauge boson Z and heavy neutral lepton N are also considered in this section. In Section III, we study the pair production of heavy neutral lepton at the 3 TeV muon collider. Analysis of the lepton number violation signatures is performed in Section IV. Finally, Section V summarizes our conclusions.

    In this paper, we consider the anomaly-free gauged U(1)LμLτ extension of the type-I seesaw mechanism. Three heavy neutral leptons Ne,Nμ,Nτ with U(1)LμLτ charge (0,1,1) are introduced to generate tiny neutrino masses. Predicting type CR of the two-zero minor [46, 47], the minimal model with only one scalar singlet Φ1, which carries U(1)LμLτ charge +1, is now tightly constrained by the neutrino oscillation data and the sum of light neutrino masses [48, 49]. Thus, the second scalar singlet Φ2 with U(1)LμLτ charge +2 is also employed to reduce the above conflict [50]. The Yukawa interactions and mass terms relevant to neutrino masses are given by [51]

    LyeˉLe˜HNeyμˉLμ˜HNμyτˉLτ˜HNτ12Mee¯NceNeMμτNcμNτyeμΦ1¯NceNμyeτΦ1¯NceNτ12yμμΦ2¯NcμNμ12yττΦ2¯NcτNτ+h.c.,

    (1)

    where Le,Lμ,Lτ are the lepton doublets, H is the standard model Higgs doublet, and ˜H=iτ2H. After the spontaneous symmetry breaking, we can denote the vacuum expectation values of scalars as H=v0,Φ1=v1,Φ2=v2. Then, the Dirac neutrino mass matrix and heavy neutral lepton mass matrix are given by

    MD=(yev0000yμv0000yτv0),MN=(Meeyeμv1yeτv1yeμv1yμμv2Mμτyeτv1Mμτyττv2).

    (2)

    Light neutrino masses are generated via the type-I seesaw formula

    MνMDM1NMTD.

    (3)

    Without any specific structure of heavy neutral lepton mass matrix MN, the obtained light neutrino mass matrix Mν is a general symmetric matrix; thus, it can easily fit the neutrino oscillation data [52]. Conversely, we can use Eq. (3) to express the mass matrix MN as [53]

    MN=MTDM1νMD,

    (4)

    where Mν=Udiag(mν1,mν2,mν3)U, and U is the light neutrino mixing matrix. In this way, the mass matrix MN is determined by the light neutrino oscillation data and the Yukawa coupling ye,yμ,yτ. The mass matrix MN can be diagnalized by a unitarity matrix Ω as

    ΩTMNΩ=diag(mN1,mN2,mN3).

    (5)

    Considering the mixing with light neutrinos, the heavy neutral leptons interact with the W±,Z gauge boson and Higgs boson as

    Lg2ˉNkVkγμPLWμg2cosθWˉNkVkγμPLνZμgmN2mWˉNkVkPLνh+h.c.,

    (6)

    where the mixing matrix VMDM1N. For electroweak-scale heavy neutral leptons, we assume a seesaw-induced mixing parameter VNmν/mN106, which is allowed by current experimental limits [3].

    The interactions of the new boson Z with fermions are

    Lg(ˉμγμμˉτγμτ+ˉνμγμPLνμˉντγμPLντ+ˉNμγμPRNμˉNτγμPRNτ)Zμ.

    (7)

    In terms of mass eigenstates, the new neutral current of neutral leptons can be rewritten as

    Lg[ˉνi(UμiUμjUτiUτj)γμPLνj+ˉNi(ΩμiΩμjΩτiΩτj)γμPRNj]Zμ,

    (8)

    From Eqs. (6) and (8), it is obvious that the collider phenomenology of heavy neutral leptons depends on the mixing patterns, which makes the results model-dependent. In the following collider simulation, we consider a muon-philic heavy neutral lepton N, which couples exclusively to muons [54]. We also take the mixing parameter ΩμNΩμNΩτNΩτN=1 in the following calculation, so the general model-dependent results can be obtained by a simple rescaling.

    In this section, we briefly summarize the constraints on the gauged U(1)LμLτ model. One important motivation of this symmetry is to explain the anomaly of the muon magnetic moment Δaμ=(251±59)×1011 [36]. The new gauge boson Z contributes to Δaμ at one loop level, which is evaluated as [37]

    Δaμ=g24π210dxm2μx(1x)2m2μ(1x)2+m2Zx.

    (9)

    The viable parameter space to interpret Δaμ is shown in Fig. 1. For very light mZ, we have Δaμg2/8π2, which requires g4×104 to explain Δaμ. In the heavy Z limit, Δaμg2m2μ/12π2m2Z; thus, the experimental discrepancy requires g/mZ5×103GeV1.

    Figure 1

    Figure 1.  (color online) Constraints on the gauged U(1)LμLτ. The blue band can explain the Δaμ result [36]. The gray, orange, red, purple, and pink regions are excluded by BBN [55], CCFR [35], NA64μ [56], BABAR [57], and LHC [58, 59], respectively.

    For the gauge boson Z, one tight constraint is the dimuon production from the inelastic neutrino-nucleus scattering νμNuνμNuμ+μ, where Nu denotes the nucleus [60]. The CCFR experiment measured the neutrino trident production cross section, which is consistent with the standard model prediction as σ/σSM=0.82±0.28 [35]. This result has excluded the Δaμ favored region with mZ0.3 GeV. Recently, the NA64μ experiment has excluded g6×104 with mZ0.1 GeV by the process of μNuμNuZμNuˉνν [56], which makes mZ0.01 GeV the Δaμ-favored region. A more stringent constraint comes from the e+eμ+μZ4μ searches at BABAR in the mass region of 0.212GeV<mZ<10 GeV [57]. For Z mass in the range of [5, 81] GeV, the LHC experiment has excluded g0.003 to 0.2 (depending on mZ) by the process ppZμ+μ4μ with 139 fb1 data [58]. Within the same mass region, a complement search by the process ppZμ±ν3μν with 140 fb1 data was also performed [59], which has better sensitivity in the light Z region. In this paper, we consider the combined limit of the 3μ and 4μ channels at the LHC [59]. Meanwhile, for Z lighter than approximately 6 MeV, it is disfavored by ΔNeff constraints from big-bang nucleosynthesis (BBN) [55], because decays of Z could heat the neutrino population and delay the process of neutrino decoupling. In the following study, we consider mZ100 GeV to avoid these tight constraints.

    For the heavy neutral lepton N, the promising signature at the LHC is ppW±()±N, which depends on the mixing parameter VN. When mN<mW, N is produced from on-shell W boson decay; then, LHC could exclude |VN|22.9×106 via the fully-leptonic decay channel Nν [61]. Meanwhile, for light N with proper VN, it becomes long-lived and leads to displaced vertex signature. Currently, the LHC has excluded |VμN|2>4×107 for mN=10 GeV [62]. Above mW, production of N is mediated by off-shell W boson, so the LHC limit becomes much weaker, e.g., |VμN|2=1.1×103 for mN=100 GeV [61]. In this paper, we also consider mN>100 GeV with seesaw-induced mixing |VN|21012 to satisfy current LHC limits.

    Before studying the explicit signatures, we first review the decay properties of new gauge boson Z and heavy neutral lepton N. The partial decay widths of Z are calculated as

    Γ(Z+)=g212πmZ,

    (10)

    Γ(Zνν)=g224πmZ,

    (11)

    Γ(ZNN)=g224πmZ(14m2Nm2Z),

    (12)

    where =μ,τ, and we have assumed vanishing masses for ,ν in the above calculations. For simplicity, we assume that the tree-level kinetic mixing between boson Z and photon γ is absent. However, mixing between Z and γ in the form of ϵ2ZμνFμν appears at one-loop level, where the kinetic mixing factor ϵ is calculated as [55]

    ϵ=eg12π2log(m2τm2μ)g70.

    (13)

    In this way, Z also couples to electron e via mixing with γ. The Ze+e decay width is suppressed by the kinetic mixing factor ϵg/70, so it is not taken into account in this study.

    After the production of heavy neutral lepton N, it decays via mixing with light neutrinos. When the heavy neutral lepton is heavier than the standard model gauge bosons, the two-body decays are the dominant modes. The partial decay widths are given by

    Γ(N±W)=|VN|216π(m2Nm2W)2(m2N+2m2W)m3Nv20,

    (14)

    Γ(NνZ)=|VN|232π(m2Nm2Z)2(m2N+2m2Z)m3Nv20,

    (15)

    Γ(Nνh)=|VN|232π(m2Nm2h)2mNv20.

    (16)

    In Fig. 2, we show the branching ratio (BR) of gauge boson Z and heavy neutral lepton N. The dilepton mode Zμ+μ,τ+τ is always the dominant decay channel of Z. Neglecting the final state's phase space effect, we have BR(Zνν)= BR(ZNN)=1/4 for two degenerate heavy neutral leptons. In the heavy mN limit, we have BR(N±W): BR(NνZ): BR(Nνh)=2:1:1. To reconstruct the heavy neutral lepton mass, we focus on the decay mode N±W with Wqˉq in the collider simulation.

    Figure 2

    Figure 2.  (color online) Branching ratio of gauge boson Z (left) and heavy neutral lepton N (right).

    Through mixing with light neutrinos, the pair production of heavy neutral leptons at the muon collider can occur through s-channel Z exchange and t-channel W exchange. However, the production cross section is proportional to the fourth power of the mixing parameter |VN|4. Provided natural seesaw predicted value VN106, the predicted cross section is negligible tiny.

    The heavy neutral lepton N is charged under U(1)LμLτ symmetry, which induces the pair production of heavy neutral lepton via Z at the muon collider. With fixed collision energy, the direct pair production process μ+μNN via the s-channel off-shell Z is more promising when mZ>s. Meanwhile, when mZ<s, the new gauge boson Z can be produced on-shell with an associated photon from the initial legs as μ+μZγNNγ, which also requires mN<mZ/2 to make the decay ZNN kinematically allowed.

    We first consider the pair production of heavy neutral leptons without associated photon. The production cross section of μ+μZNN with a center of mass energy s is calculated as

    σ(μ+μZNN)=g424πs(sm2Z)2+m2ZΓ2Z(14m2Ns)3/2,

    (17)

    where ΓZ is the total decay width of Z.

    The theoretical cross section of μ+μZNN at a 3 TeV muon collider for specific scenarios is shown in Fig. 3, where we directly use Eq. (17) without considering the initial state radiation effect. It is obvious that when mZs, the on-shell production of Z can greatly enhance the cross section. The maximum value is approximately 12πBR(Zμ+μ)BR(ZNN)/m2Z3π/(8m2Z) [45], which could be over 50 pb. For a light Z, the cross section reduces to

    Figure 3

    Figure 3.  (color online) Cross section of μ+μZNN at a 3 TeV muon collider. We have fixed mN=100 GeV in the left panel and mZ=2000 GeV in the right panel.

    σ(μ+μZNN)g424π1s(14m2Ns)3/2,

    (18)

    which is independent of Z mass. Typically, for g=0.3 and mN=1000 GeV, we have σ(μ+μZNN)4.7 fb. Meanwhile, a heavy Z leads to

    σ(μ+μZNN)g424πsm4Z(14m2Ns)3/2.

    (19)

    In this scenario, the cross section is suppressed by the large value of mZ. While single production could probe mNs, the pair production of heavy neutral lepton is kinematically allowed when mN<s/2. Thus, the 3 TeV muon collider could only probe mN1500 GeV. For heavier mN, we require a more energetic muon collider.

    The photon from initial state radiation carries part of the collision energy, which results in the gauge boson Z produced on-shell when mZs. In this case, there is a resonance peak in the invariant mass spectrum of the dilepton [44]. Here, we study the pair production of heavy neutral leptons with associated photon process as

    μ+μZ()γNNγ.

    (20)

    Here, the scenario with off-shell Z contribution is also considered when mZ or m_{Z'}\geqslant \sqrt{s} .

    In Fig. 4, we show the cross section of \mu^+\mu^-\to NN\gamma at a 3 TeV muon collider, where the numerical results are calculated by MadGraph. There is also a sharp peak around m_{Z'}\simeq \sqrt{s} . To avoid the soft photon singularity, the detected photon is required to satisfy the following pre-selection cuts:

    Figure 4

    Figure 4.  (color online) Cross section of \mu^+\mu^-\to NN\gamma at a 3 TeV muon collider. We have fixed m_N=100 GeV in the left panel and m_{Z'}=2000 GeV in the right panel.

    \begin{aligned}[b] P_T(\gamma)>20\; \text{GeV},\; |\eta(\gamma)|<2.5. \end{aligned}

    (21)

    When m_{Z'}>2 m_N , the on-shell production of Z' followed by the cascade decay Z'\to NN can notably enhance the cross section of \mu^+\mu^-\to NN\gamma . Using the narrow-width approximation, the cross section can be expressed as [63]

    \begin{aligned}[b] \sigma(\mu^+\mu^-\to NN\gamma)\simeq \sigma(\mu^+\mu^-\to Z'\gamma) \times \text{BR}(Z'\to NN). \end{aligned}

    (22)

    Because the branching ratio of Z'\to NN is independent of the new gauge coupling g' , the cross section of \mu^+\mu^-\to Z'\gamma\to NN\gamma is proportional to e^2g^{\prime 2} , while the cross section of the off-shell process \mu^+\mu^-\to Z ^{\prime (*)}\gamma \to NN\gamma is proportional to e^2g^{\prime 4} . Thus, we observe that the enhancement effect becomes larger when the gauge coupling g' is smaller. For g'=0.1,\; m_N=100 GeV, and m_{Z'}=1000 GeV, we have \sigma(\mu^+\mu^-\to Z'\gamma\to NN\gamma)\sim 1 fb, while \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN) is less than 0.1 fb; therefore, the former process is expected to be more promising at the muon collider.

    Pair production of heavy neutral leptons followed by various cascade decay modes could lead to many interesting signatures, such as monolepton, dilepton, and trilepton signals [64, 65]. In this paper, we focus on the most distinct lepton number violation, same-sign dimuon signal, which also has a much cleaner background compared with the lepton number conserving one.

    The simulation procedure is as follows. The FeynRules package [66] is used to implement the model. The parton level events are simulated with MadGraph5_aMC@NLO [67]. We then use PYTHIA 8 [68] for parton shower. The detector effects are included by employing Delphes 3 [69] with the detector card of the muon collider. The W boson from the heavy neutral lepton decay could be highly boosted, so the two jets from cascade W decay merge into one fat-jet J. We use the Valencia algorithm [70] with R=1.2 to reconstruct the fat-jets.

    The full production process of the same-sign dimuon signature without associated photon is

    \begin{aligned}[b] \mu^+\mu^-\to Z^{\prime *}\to N N\to \mu^\pm W^\mp +\mu^\pm W^\mp, \end{aligned}

    (23)

    with the hadronic decays of W. The standard model backgrounds come from processes such as

    \begin{aligned}[b] \mu^+\mu^-\to \mu^+\mu^- W^+W^- , \mu^+\mu^-W^+ W^-\gamma/Z, W^\pm W^\pm W^\mp \mu^\mp \nu. \end{aligned}

    (24)

    The contributions of \mu^+\mu^- W^+W^- and \mu^+\mu^-W^+ W^-\gamma/Z are from lepton charge misidentification, which are suppressed by the misidentification rate 0.1% [17]. The W^\pm W^\pm W^\mp \mu^\mp \nu is dominant by the vector boson fusion process, where the two same-sign W bosons decay leptonically. There is also one possible background process \mu^+\mu^-\to W^+W^-jj with the light jets mistagged. However, by requiring the light jet masses close to the W mass, this background can easily be suppressed [17]. Meanwhile, the t\bar{t}W^\pm \mu^\mp\nu process also contributes to the same-sign dimuon signature, which can be further reduced by cut on the opposite-sign lepton [26]. Thus, we do not include contributions of the \mu^+\mu^-\to W^+W^- jj and t\bar{t}W^\pm \mu^\mp\nu processes in this paper.

    We first apply the following pre-selection cuts on the transverse momentum and pseudorapidity of the muon and fat-jets:

    \begin{aligned}[b] P_T (\mu^\pm)>50\; \text{GeV}, |\eta (\mu^\pm)|<2.5,P_T (J)>50\; \text{GeV}, |\eta (J)|<2.5. \end{aligned}

    (25)

    In Fig. 5, we show the distributions of some variables for the \mu^\pm\mu^\pm JJ signal and backgrounds after applying the pre-selection cuts. We have set m_{N}=1000 GeV, m_{Z'}= 2500 GeV, and g'=0.6 as the benchmark point of the signal. The cut flow for the \mu^\pm\mu^\pm JJ signature and backgrounds are summarized in Table 1.

    Figure 5

    Figure 5.  (color online) Normalized distribution of transverse momentum of muon P_T(\mu^\pm) (up-left panel), transverse momentum of fat-jet P_T(J) (up-right panel), fat-jet mass m_J (down-left panel), and invariant mass of muon and fat-jet m_{\mu J} (down-right panel) for the \mu^\pm\mu^\pm JJ signature and corresponding backgrounds.

    Table 1

    Table 1.  Cut flow table for the \mu^\pm\mu^\pm JJ signal at the 3 TeV muon collider and corresponding backgrounds. The significance is calculated with Eq. (29) by assuming an integrated luminosity of 1000 fb ^{-1} .
    σ(fb) \mu^\pm\mu^\pm W^\mp W^\mp \mu^+\mu^- W^+W^- \mu^+\mu^-W^+ W^-\gamma/Z W^\pm W^\pm W^\mp \mu^\mp \nu
    Pre-selection 68.19 21.57 2.13 3.11
    N_{\mu^\pm}=2 41.38 2.6\times10^{-2} 2.5\times10^{-3} 1.8\times10^{-1}
    N_J=2 24.87 1.1\times10^{-2} 1.3\times10^{-3} 7.1\times10^{-2}
    50 GeV \leqslant m_J\leqslant 100 GeV 14.41 3.2\times10^{-3} 2.1\times10^{-4} 2.0\times10^{-2}
    0.8 m_N<m_{\mu J}<1.2 m_N 13.62 3.4\times10^{-4} 2.7\times10^{-5} 8.1\times10^{-4}
    Significance 476.6 Total Background 1.2\times10^{-3}
    DownLoad: CSV
    Show Table

    The lepton number violation signature is satisfied for events with two same-sign muons. In the single production of the heavy neutral lepton process [26, 27], only one boosted W is expected from N decay. To distinguish from the single production and also suppress the background from vector boson fusion processes, we require exactly two fat-jets in the final states

    \begin{aligned}[b] N_{\mu^\pm}=2, \; N_J=2, \end{aligned}

    (26)

    although tagging one fat-jet has greater efficiency for the signal. To be identified as W bosons, the fat-jet masses are also required in the following range:

    \begin{aligned}[b] 50 \; \text{GeV} \leqslant m_J \leqslant 100 \; \text{GeV}. \end{aligned}

    (27)

    Because there are two heavy neutral leptons in the signal, we reconstruct their masses through the two-muon and two-fat-jet system by minimizing \chi^2= (m_{\mu_1 J_1} - m_N)^2+(m_{\mu_2 J_2} -m_N)^2 [17]. We require that both the invariant mass of muon and fat-jet m_{\mu J} from reconstructed N satisfy

    \begin{aligned}[b] 0.8 m_N<m_{\mu J}<1.2 m_N. \end{aligned}

    (28)

    According to the distributions in Fig. 5, we may tighten the cuts on P_T(\mu^\pm) and P_T(J) to suppress the background. However, we can see from Table 1 that the cut on the invariant mass m_{\mu J} is relatively efficient to reduce the background. After applying all these cuts, the total cross section of the background is approximately 1.2\times10^{-3} fb. Thus, with an integrated luminosity of 1000 fb ^{-1} , there are only 1.2 background events, while the signal events are over 10^4 . In this manner, the significance will be 476.6 for the benchmark point with 1000 fb ^{-1} data, where the significance is calculated as [71]

    \begin{aligned}[b] {\cal{S}}=\sqrt{2\left[(N_S+N_B)\ln\left(1+\frac{N_S}{N_B}\right)-N_S\right]}. \end{aligned}

    (29)

    Here, N_S and N_B are the event numbers of signal and background, respectively. To reach the 5\sigma discovery limit, we require only 0.11 fb ^{-1} data.

    Based on the above analysis, we then explore the sensitivity of the \mu^\pm \mu^\pm JJ signature at the 3 TeV muon collider. The results are shown in Fig. 6. There are three free parameters g',m_{Z'} , and m_N related to the collider phenomenology. With fixed mass relation m_{N} = m_{Z'}/3 , we could probe m_{Z'} approximately in the range of [300, 4500] GeV. Around the resonance region m_{Z'}\sim3000 GeV, the gauge coupling g' could be down to approximately 0.025 with 1000 fb ^{-1} data. We also find that for m_N\lesssim300 GeV, the cut efficiency decreases quickly as m_N becomes smaller. One main reason for this is that the heavy neutral leptons are also highly boosted at the 3 TeV muon collider for such light m_N , so the muons and W-jets from boosted N decays are mostly non-isolated [72]. One may use the substructure-based variables as lepton sub-jet fraction (LSF) and lepton mass drop (LMD) [73] to probe the light m_N , which is beyond the scope of this study.

    Figure 6

    Figure 6.  (color online) Sensitivity of the \mu^\pm \mu^\pm JJ signature at the 3 TeV muon collider. In the upper-left panel, we fix the mass relation with m_N=m_{Z'}/3 . In the upper-right panel, we set m_N=1000 GeV. In the lower panel, we fix the gauge coupling g'=0.6 . The dark-green region corresponds to sensitivity with 1 fb ^{-1} data, while the light-green region corresponds to sensitivity with 1000 fb ^{-1} data. The orange region is excluded by neutrino trident production at CCFR [35]. The magenta region is not allowed kinematically. The gray line is the projected sensitivity at the HL-LHC [74]. The red line is the combined sensitivity of the 3 TeV muon collider via processes \mu^+\mu^-\to \ell^+\ell^-(\ell=\mu\; \text{or}\; \tau), \ell\bar{\ell}\gamma(\ell=\mu\; \text{or}\; \tau\; \text{or}\; \nu_{\mu/\tau}) [44].

    For pair production of heavy neutral leptons without associated photon signature, the gauge boson Z' does not need to be heavier than 2 m_N . In the upper-right panel of Fig. 6, we have fixed m_N=1000 GeV for illustration. The production cross section of \mu^+\mu^-\to NN is approximately constant for light Z' , so g'\sim0.13 is nearly independent of m_{Z'} when Z' is lighter than 1 TeV. This channel is most sensitive for the large m_{Z'} region. For instance, we may probe m_{Z'}\sim10 TeV with g'\gtrsim0.4 and m_N\sim1000 GeV at the 3 TeV muon collider. Near the resonance region with m_{Z'}\sim \sqrt{s} , we can even have a promising signature when g'\gtrsim0.14 with only 1 fb ^{-1} data. Compared with the direct dilepton channels

    \mu^+\mu^-\to \ell^+\ell^-(\ell=\mu\; \text{or}\; \tau), \; \ell\bar{\ell}\gamma(\ell=\mu\; \text{or}\; \tau\; \text{or}\; \nu_{\mu/\tau})

    the sensitivity region of lepton number violation signal from heavy neutral lepton pair at the muon collider is smaller due to lower tagging efficiency of N.

    We then fix the gauge coupling g'=0.6 and explore the sensitivity region on the m_N-m_{Z'} plane. With 1 fb ^{-1} data, a large parameter space near the resonance region with 2 TeV \lesssim m_{Z'}\lesssim4 TeV can be probed. The most sensitive heavy neutral lepton mass is around 500 GeV. This is because a lighter m_N leads to smaller acceptance efficiency, while a larger m_N has a smaller production cross section. With an integrated luminosity of 1000 fb ^{-1} , the region with m_{Z'}\lesssim 13 TeV and 150 GeV \lesssim m_N<1500 GeV is within the reach of the 3 TeV muon collider.

    The full production process of the same-sign dimuon signature with associated photon is

    \begin{aligned}[b] \mu^+\mu^-\to Z^{\prime (*)} \gamma \to N N \gamma\to \mu^\pm W^\mp +\mu^\pm W^\mp+\gamma, \end{aligned}

    (30)

    with the hadronic decays of W. The standard model backgrounds come from processes such as

    \begin{aligned}[b] \mu^+\mu^-\to \mu^+\mu^- W^+W^-\gamma , W^\pm W^\pm W^\mp \gamma \mu^\mp \nu . \end{aligned}

    (31)

    During the simulation, the pre-selection cuts on associated photon in Eq. (21) are also applied. Based on the cut-flow in Table 1, we expect that the cross section of the process \mu^+\mu^-W^+ W^-\gamma\gamma/Z with one additional \gamma/Z gauge boson is approximately an order of magnitude smaller than that of the process \mu^+\mu^- W^+W^-\gamma . Therefore, \sigma(\mu^+\mu^-W^+ W^-\gamma\gamma/Z) is approximately 10^{-5} fb after all cuts, which will not be considered in the analysis of the \mu^\pm\mu^\pm JJ\gamma signal.

    The cross sections of the background processes with associated photons are approximately one order of magnitude smaller than those without associated photons. If we apply exactly the same selection cuts as the previous \mu^\pm\mu^\pm JJ signal, the total cross section of background is expected to be of the order of 10^{-4} fb, which will have approximately 0.1 background events, even with a total of 1000 fb ^{-1} data. Therefore, we can loosen the selection cut to keep more signal events. In the analysis of the \mu^\pm\mu^\pm JJ\gamma signal, we consider that at least one fat-jet is detected in the final states

    \begin{aligned}[b] N_J\geqslant 1, \end{aligned}

    (32)

    while we further require one detected photon in this signal.

    In Fig. 7, we show the distributions of some variables for the \mu^\pm\mu^\pm JJ\gamma signal and backgrounds after applying the pre-selection cuts. The benchmark point for the \mu^\pm\mu^\pm JJ\gamma signal is the same as for the previous \mu^\pm\mu^\pm JJ signal. With relatively similar distributions of P_T(\mu^\pm) , P_T(J) , m_J , and m_{\mu J} , we then apply the same selection cuts for these variables as the \mu^\pm\mu^\pm JJ signal. The cut flows for the \mu^\pm\mu^\pm JJ\gamma signature and backgrounds are summarized in Table 2.

    Figure 7

    Figure 7.  (color online) Normalized distribution of transverse momentum of muon P_T(\mu^\pm) (upper-left panel), transverse momentum of fat-jet P_T(J) (upper-right panel), fat-jet mass m_J (middle-left panel), and invariant mass of muon and fat-jet m_{\mu J} (middle-right panel). Energy of photon E_\gamma (lower-left panel) and invariant mass of reconstructed heavy neutral lepton pair m_{NN} (lower-right panel) for the \mu^\pm\mu^\pm JJ \gamma signature and corresponding backgrounds.

    Table 2

    Table 2.  Cut flow table for the \mu^\pm\mu^\pm JJ\gamma signal at the 3 TeV muon collider and corresponding backgrounds. The significance is calculated by Eq. (29), assuming an integrated luminosity of 1000 fb ^{-1} .
    σ/fb \mu^\pm\mu^\pm W^\mp W^\mp \gamma \mu^+\mu^- W^+W^- \gamma W^\pm W^\pm W^\mp\gamma \mu^\mp \nu
    Pre-selection 24.91 1.15 0.14
    N_{\gamma}=1,N_{\mu^\pm}=2 13.28 1.1\times10^{-3} 7.4\times10^{-3}
    N_J\geq1 13.25 1.0\times10^{-3} 7.2\times10^{-3}
    50 GeV \leqslant m_J\leqslant 100 GeV 9.61 5.7\times10^{-4} 4.5\times10^{-3}
    0.8 m_N<m_{\mu J}<1.2 m_N 8.86 2.6\times10^{-4} 1.2\times10^{-3}
    Significance 369.0 Total Background 1.5\times10^{-3}
    DownLoad: CSV
    Show Table

    Although with one additional photon, the cross section of the \mu^\pm\mu^\pm JJ\gamma signal is of the same order as the \mu^\pm\mu^\pm JJ signal. After all selection cuts, the cross section of the \mu^\pm\mu^\pm JJ\gamma process is 8.86 fb. With a loose cut on the fat-jet number, the total cross section of the background is approximately 1.5\times10^{-3} fb after all cuts. With an integrated luminosity of 1000 fb ^{-1} , the significance of the \mu^\pm\mu^\pm JJ\gamma signal could be approximately 369 for the benchmark point. Meanwhile, 0.19 fb ^{-1} data is sufficient to discover the benchmark point at the 5\sigma level.

    Besides the invariant mass of dilepton m_{\ell^+\ell^-}(\ell=\mu,\tau) , there are two more pathways to confirm the on-shell production of gauge boson Z' in the \mu^\pm \mu^\pm JJ\gamma signature. For the two-body process \mu^+\mu^-\to Z' \gamma , the photon energy is connected to the gauge boson mass as [44]

    \begin{aligned}[b] E_\gamma=\frac{s-m_{Z'}^2}{2\sqrt{s}}. \end{aligned}

    (33)

    For the benchmark point with m_{Z'}=2500 GeV at the 3 TeV muon collider, the peak value of E_\gamma is approximately 458 GeV. The sharp bump above the background is clearly shown in the lower-left panel of Fig. 7. Meanwhile, we can also measure m_{Z'} through the invariant mass of two reconstructed heavy neutral leptons m_{NN} , which also has a resonance near m_{Z'} . From the distribution of m_{NN} in the lower-right panel of Fig. 7, the background processes \mu^+\mu^-W^+W^-\gamma and W^\pm W^\pm W^\mp \gamma \mu^\mp \nu have peaks around 3 TeV and 1.5 TeV, respectively. Besides, with less than two background events after all cuts, the backgrounds have little impact on the distribution of m_{NN} for the signal.

    The sensitivity of the \mu^\pm \mu^\pm JJ\gamma signature at the 3 TeV muon collider is explored with the cuts in Table 2, and Fig. 8 depicts the results. The upper-left panel shows the sensitivity region with fixed mass relation m_N=m_{Z'}/3 , so the decay channel Z'\to NN is always allowed. In this way, we might probe m_{Z'} in the range of [300, 4300] GeV. Compared with the \mu^\pm \mu^\pm JJ signature, we find that this signal is more promising approximately in the region of [400, 3000] GeV. For example, with m_{Z'}=2000 GeV, the \mu^\pm \mu^\pm JJ\gamma signature could probe g'\gtrsim0.015 , while the \mu^\pm \mu^\pm JJ signature could only reach g'\sim0.1 . Near m_{Z'}\sim \sqrt{s}=3 TeV, we can even reveal g'\gtrsim0.005 . When m_{Z'} is larger than the collision energy \sqrt{s} , the associated photon with the off-shell contribution of Z' makes the \mu^\pm \mu^\pm JJ\gamma signature less detectable than the \mu^\pm \mu^\pm JJ signal.

    Figure 8

    Figure 8.  (color online) Same as Figure 6 but for the sensitivity of the \mu^+\mu^-JJ\gamma signal.

    The upper-right panel of Fig. 8 shows the sensitivity region with fixed m_N=1000 GeV. When m_{Z'}\leqslant 2 m_N , the decay channel Z'\to NN is not allowed, so production of NN\gamma is through the off-shell Z' . With nearly a constant cross section, the \mu^\pm \mu^\pm JJ\gamma signature could test g'\gtrsim0.23 for m_{Z'}\leqslant 2000 GeV, which is clearly less promising than the \mu^\pm \mu^\pm JJ signature. However, once the decay mode Z'\to NN is open, the cross section of NN\gamma is enhanced, so the \mu^\pm \mu^\pm JJ\gamma signal becomes quite promising. According to our simulation, we find that this signature could probe g'\gtrsim0.11 with only 1 fb ^{-1} data when 2000 GeV \lesssim m_{Z'}\lesssim\sqrt{s} .

    The sensitivity region in m_N-m_{Z'} with fixed g'=0.6 is shown in the lower panel of Fig. 8. With 1 fb ^{-1} data, a large portion of the parameter space with 2m_N\lesssim m_{Z'}\lesssim \sqrt{s} could be tested. With a similar reason as the \mu^\pm \mu^\pm JJ signature, the most sensitive m_N of the \mu^\pm \mu^\pm JJ\gamma signature is also approximately 500 GeV. Outside the above resonance Z' region, the significance of the \mu^\pm \mu^\pm JJ\gamma signature decreases due to the associated photon. Although less promising than the \mu^\pm \mu^\pm JJ signature, the \mu^\pm \mu^\pm JJ\gamma signal could probe the region with m_{Z'}\lesssim8.3 TeV and 120 GeV \lesssim m_N\lesssim1470 GeV at the 3 TeV muon collider with an integrated luminosity of 1000 fb ^{-1} .

    Heavy neutral leptons N are introduced in the type-I seesaw mechanism to generate tiny neutrino masses. While electroweak scale N is extensively studied at colliders, a relatively large mixing parameter V_{\ell N} is usually required. In this paper, we considered the gauged U(1)_{L_\mu-L_\tau} extension of the type-I seesaw mechanism, where three heavy neutral leptons N_e,N_\nu,N_\tau with U(1)_{L_\mu-L_\tau} charge (0,1,-1) were introduced. Charged under the new gauged U(1)_{L_\mu-L_\tau} symmetry, there are new production channels of the heavy neutral lepton at colliders. We then investigated the lepton number violation signatures of heavy neutral lepton N in the gauged U(1)_{L_\mu-L_\tau} at the 3 TeV muon collider.

    Mediated by the new gauge boson Z' , the heavy neutral lepton N can be pair produced at the muon collider via the processes \mu^+\mu^-\to Z^{\prime *}\to NN and \mu^+\mu^-\to Z^{\prime (*)}\gamma\to NN \gamma. Cross sections of these processes are determined by the new gauge coupling g' , gauge boson mass m_{Z'} , and heavy neutral lepton mass m_N , which are independent of the mixing parameter V_{\ell N} . While both processes have a maximum cross section at m_{Z'}\simeq \sqrt{s} , the former is more promising when m_{Z'}>\sqrt{s} . For lighter new gauge bosons satisfying 2m_N<m_{Z'}<\sqrt{s} , Z' is produced on-shell with an associated photon, and the decay mode Z'\to NN is also allowed. In this way, the cross section of \mu^+\mu^-\to NN\gamma can be enhanced.

    Cascade decays of heavy neutral leptons can raise various interesting signatures. Provided the Majorana nature of heavy neutral leptons in the seesaw model, we focus on the lepton number violation signatures in this study. For illustration, we further assume that the heavy neutral lepton N preferentially couples to muons via mixing with light neutrinos. Thus, the dominant decay mode of heavy neutral leptons is N\to \mu^\pm W^\mp . The hadronic decays of W bosons are considered to reconstruct m_N , which are treated as one fat-jet J. The explicit signatures are \mu^+\mu^-\to NN\to \mu^\pm\mu^\pm JJ and \mu^+\mu^-\to NN\gamma\to \mu^\pm\mu^\pm JJ\gamma .

    With relatively clean backgrounds, a cut-based analysis was then performed, which indicates that a large part of the viable parameter space is within the reach of the 3 TeV muon collider. For example, \mu^\pm\mu^\pm JJ could probe m_{Z'} in the range of [300, 4500] GeV with fixed mass relation m_{N}=m_{Z'}/3 and an integrated luminosity of 1000 fb ^{-1} . Around the resonance region m_{Z'}\sim\sqrt{s} , the gauge coupling g' could be down to approximately 0.025. We can probe g'\gtrsim0.13 with m_N=1000 GeV and light Z' . Otherwise, this signal could probe m_{Z'}\lesssim 13 TeV with g'=0.6 . Meanwhile, if the gauge boson mass satisfies 2 m_N<m_{Z'}<\sqrt{s} , the \mu^\pm\mu^\pm JJ\gamma signature would be more promising than the \mu^\pm\mu^\pm JJ signature. It is also notable that around the resonance region m_{Z'}\sim\sqrt{s} , both signatures could probe a sizable parameter space even with only 1 fb ^{-1} data for g'=0.6 .

    [1] P. Minkowski, Phys. Lett. B 67, 421 (1977) doi: 10.1016/0370-2693(77)90435-X
    [2] R. N. Mohapatra and G. Senjanovic, Phys. Rev. Lett. 44, 912 (1980) doi: 10.1103/PhysRevLett.44.912
    [3] A. M. Abdullahi, P. B. Alzas, B. Batell et al., J. Phys. G 50(2), 020501 (2023), arXiv:2203.08039 [hep-ph] doi: 10.1088/1361-6471/ac98f9
    [4] M. Agostini, G. Benato, J. A. Detwiler et al., Rev. Mod. Phys. 95(2), 025002 (2023), arXiv:2202.01787 [hep-ex] doi: 10.1103/RevModPhys.95.025002
    [5] Y. Cai, T. Han, T. Li and R. Ruiz, Front. in Phys. 6, 40 (2018), arXiv:1711.02180 [hep-ph] doi: 10.3389/fphy.2018.00040
    [6] T. Han and B. Zhang, Phys. Rev. Lett. 97, 171804 (2006), arXiv:hep-ph/0604064 [hep-ph] doi: 10.1103/PhysRevLett.97.171804
    [7] A. M. Sirunyan et al. [CMS], JHEP 01 , 122 (2019), arXiv:1806.10905 [hep-ex]
    [8] G. Aad et al. [ATLAS], JHEP 10 , 265(2019), arXiv:1905.09787 [hep-ex]]
    [9] A. Davidson, Phys. Rev. D 20, 776 (1979) doi: 10.1103/PhysRevD.20.776
    [10] R. N. Mohapatra and R. E. Marshak, Phys. Rev. Lett. 44(2), 1316 (1980)[erratum: Phys. Rev. Lett. 44 , 1643 (1980)] doi: 10.1103/PhysRevLett.44.1316
    [11] W. Y. Keung and G. Senjanovic, Phys. Rev. Lett. 50, 1427 (1983) doi: 10.1103/PhysRevLett.50.1427
    [12] K. A. Urquía-Calderón, arXiv:2310.17406 [hep-ph]
    [13] A. Tumasyan et al. [CMS], JHEP 11 , 181 (2023), arXiv:2307.06959[hep-ex]]
    [14] G. Aad et al. [ATLAS], Eur. Phys. J. C 83 (12), 1164 (2023), arXiv:2304.09553 [hep-ex]
    [15] J. P. Delahaye, M. Diemoz, K. Long et al., arXiv:1901.06150 [physics.acc-ph]
    [16] K. Long, D. Lucchesi, M. Palmer et al., Nature Phys. 17(3), 289 (2021), arXiv:2007.15684 [physics.acc-ph] doi: 10.1038/s41567-020-01130-x
    [17] W. Liu, K. P. Xie and Z. Yi, Phys. Rev. D 105(9), 095034 (2022), arXiv:2109.15087 [hep-ph] doi: 10.1103/PhysRevD.105.095034
    [18] T. Li, H. Qin, C. Y. Yao et al., Phys. Rev. D 106(3), 035021 (2022), arXiv:2205.04214 [hep-ph] doi: 10.1103/PhysRevD.106.035021
    [19] I. Chakraborty, H. Roy and T. Srivastava, arXiv:2206.07037 [hep-ph]
    [20] O. Mikulenko and M. Marinichenko, JHEP 01, 032 (2024), arXiv:2309.16837 [hep-ph]
    [21] Z. Wang, X. H. Yang and X. Y. Zhang, arXiv:2311.15166 [hep-ph].
    [22] K. Mekala, J. Reuter and A. F. Żarnecki, Phys. Lett. B 841, 137945 (2023), arXiv:2301.02602 [hep-ph] doi: 10.1016/j.physletb.2023.137945
    [23] T. H. Kwok, L. Li, T. Liu et al., arXiv:2301.05177 [hep-ph].
    [24] P. Li, Z. Liu and K. F. Lyu, JHEP 03, 231 (2023), arXiv:2301.07117 [hep-ph]
    [25] Q. H. Cao, K. Cheng and Y. Liu, arXiv:2403.06561 [hep-ph]
    [26] T. Li, C. Y. Yao and M. Yuan, JHEP 09, 131 (2023), arXiv:2306.17368 [hep-ph]
    [27] E. Antonov, A. Drutskoy and M. Dubinin, JETP Lett. 118(7), 461 (2023), arXiv:2308.02240 [hep-ph] doi: 10.1134/S0021364023602518
    [28] R. Jiang, T. Yang, S. Qian et al, arXiv:2304.04483 [hep-ph].
    [29] J. de Blas et al. [Muon Collider], arXiv:2203.07261[hep-ph]]
    [30] X. G. He, G. C. Joshi, H. Lew et al., Phys. Rev. D 43, 22 (1991) doi: 10.1103/PhysRevA.43.22
    [31] A. Das, T. Nomura and T. Shimomura, Eur. Phys. J. C 83(9), 786 (2023), arXiv:2212.11674 [hep-ph] doi: 10.1140/epjc/s10052-023-11955-4
    [32] J. Sun, F. Huang and X. G. He, Phys. Lett. B 845, 138121 (2023), arXiv:2307.00531 [hep-ph] doi: 10.1016/j.physletb.2023.138121
    [33] S. K.A., A. Das, G. Lambiase, T. Nomura and Y. Orikasa, arXiv:2308.14483 [hep-ph]
    [34] A. Das and Y. Orikasa, arXiv:2401.00696 [hep-ph]
    [35] S. R. Mishra et al., Phys. Rev. Lett. 66, 3117 (1991) doi: 10.1103/PhysRevLett.66.3117
    [36] B. Abi et al., Phys. Rev. Lett. 126, 141801 (2021), arXiv:2104.03281 [hep-ex] doi: 10.1103/PhysRevLett.126.141801
    [37] S. Baek, N. G. Deshpande, X. G. He et al., Phys. Rev. D 64, 055006 (2001), arXiv:hep-ph/0104141 [hep-ph] doi: 10.1103/PhysRevD.64.055006
    [38] W. Altmannshofer, S. Gori, M. Pospelov et al., Phys. Rev. D 89, 095033 (2014), arXiv:1403.1269 [hep-ph] doi: 10.1103/PhysRevD.89.095033
    [39] R. Aaij et al. [LHCb], Nature Phys. 18 (3), 277 (2022), arXiv:2103.11769 [hep-ex]
    [40] S. Baek and P. Ko, JCAP 10, 011 (2009), arXiv:0811.1646 [hep-ph]
    [41] I. Holst, D. Hooper and G. Krnjaic, Phys. Rev. Lett. 128(14), 141802 (2022), arXiv:2107.09067 [hep-ph] doi: 10.1103/PhysRevLett.128.141802
    [42] S. Baek, H. Okada and K. Yagyu, JHEP 04, 049 (2015), arXiv:1501.01530 [hep-ph]
    [43] K. Asai, K. Hamaguchi and N. Nagata, Eur. Phys. J. C 77(11), 763 (2017), arXiv:1705.00419 [hep-ph] doi: 10.1140/epjc/s10052-017-5348-x
    [44] G. Y. Huang, F. S. Queiroz and W. Rodejohann, Phys. Rev. D 103(9), 095005 (2021), arXiv:2101.04956 [hep-ph] doi: 10.1103/PhysRevD.103.095005
    [45] A. Dasgupta, P. S. B. Dev, T. Han, et al., JHEP 12, 011 (2023), arXiv:2308.12804 [hep-ph]
    [46] L. Lavoura, Phys. Lett. B 609, 317 (2005), arXiv:hep-ph/0411232 [hep-ph] doi: 10.1016/j.physletb.2005.01.047
    [47] T. Araki, J. Heeck and J. Kubo, JHEP 07, 083 (2012), arXiv:1203.4951 [hep-ph]
    [48] K. Asai, K. Hamaguchi, N. Nagata et al., Phys. Rev. D 99(5), 055029 (2019), arXiv:1811.07571 [hep-ph] doi: 10.1103/PhysRevD.99.055029
    [49] K. Asai, K. Hamaguchi, N. Nagata et al., JCAP 11, 013 (2020), arXiv:2005.01039 [hep-ph]
    [50] D. Borah, A. Dasgupta and D. Mahanta, Phys. Rev. D 104(7), 075006 (2021), arXiv:2106.14410 [hep-ph] doi: 10.1103/PhysRevD.104.075006
    [51] S. Patra, S. Rao, N. Sahoo and N. Sahu, Nucl. Phys. B 917, 317 (2017), arXiv:1607.04046 [hep-ph] doi: 10.1016/j.nuclphysb.2017.02.010
    [52] P. F. de Salas, D. V. Forero, S. Gariazzo et al., JHEP 02, 071 (2021), arXiv:2006.11237 [hep-ph]
    [53] A. Granelli, K. Hamaguchi, N. Nagata, et al., JHEP 09, 079 (2023), arXiv:2305.18100 [hep-ph]
    [54] M. Drewes, J. Klarić and J. López-Pavón, Eur. Phys. J. C 82(12), 1176 (2022), arXiv:2207.02742 [hep-ph] doi: 10.1140/epjc/s10052-022-11100-7
    [55] M. Escudero, D. Hooper, G. Krnjaic and M. Pierre, JHEP 03, 071 (2019), arXiv:1901.02010 [hep-ph]
    [56] Y. M. Andreev, D. Banerjee, B. B. Oberhauser, et al., arXiv:2401.01708 [hep-ex]
    [57] J. P. Lees et al. [BaBar], Phys. Rev. D 94 (1), 011102 (2016), arXiv:1606.03501[hep-ex]
    [58] G. Aad et al. [ATLAS], JHEP 07 , 090(2023), arXiv:2301.09342 [hep-ex]
    [59] G. Aad et al. [ATLAS], arXiv:2402.15212 [hep-ex]
    [60] W. Altmannshofer, S. Gori, M. Pospelov and I. Yavin, Phys. Rev. Lett. 113, 091801 (2014), arXiv:1406.2332 [hep-ph] doi: 10.1103/PhysRevLett.113.091801
    [61] A. Hayrapetyan et al. [CMS], arXiv:2403.00100 [hep-ex]
    [62] A. Hayrapetyan et al. [CMS], JHEP 03 , 105 (2024), arXiv:2312.07484 [hep-ex]
    [63] T. Appelquist, B. A. Dobrescu and A. R. Hopper, Phys. Rev. D 68, 035012 (2003), arXiv:hep-ph/0212073 [hep-ph] doi: 10.1103/PhysRevD.68.035012
    [64] J. Nakajima, A. Das, K. Fujii, et al., arXiv:2203.06929[hep-ex]
    [65] M. T. Arun, A. Chatterjee, T. Mandal et al., Phys. Rev. D 106(9), 095035 (2022), arXiv:2204.02949 [hep-ph] doi: 10.1103/PhysRevD.106.095035
    [66] A. Alloul, N. D. Christensen, C. Degrande, et al., Comput. Phys. Commun. 185, 2250 (2014), arXiv:1310.1921 [hep-ph] doi: 10.1016/j.cpc.2014.04.012
    [67] J. Alwall, R. Frederix, S. Frixione et al., JHEP 07, 079 (2014), arXiv:1405.0301 [hep-ph] doi: 10.1007/JHEP07(2014)079
    [68] T. Sjöstrand, S. Ask, J. R. Christiansen et al., Comput. Phys. Commun. 191, 159 (2015), arXiv:1410.3012 [hep-ph] doi: 10.1016/j.cpc.2015.01.024
    [69] J. de Favereau et al. [DELPHES 3], JHEP 02 , 057 (2014), arXiv:1307.6346 [hep-ex]
    [70] M. Boronat, J. Fuster, I. Garcia et al., Phys. Lett. B 750, 95 (2015), arXiv:1404.4294 [hep-ex] doi: 10.1016/j.physletb.2015.08.055
    [71] G. Cowan, K. Cranmer, E. Gross et al., Eur. Phys. J. C 71, 1554 (2011), arXiv:1007.1727 [physics.data-an] doi: 10.1140/epjc/s10052-011-1554-0
    [72] A. Dey, R. Rahaman and S. K. Rai, arXiv:2207.06857 [hep-ph]
    [73] C. Brust, P. Maksimovic, A. Sady et al., JHEP 04, 079 (2015), arXiv:1410.0362 [hep-ph] doi: 10.1007/JHEP04(2015)079
    [74] F. del Aguila, M. Chala, J. Santiago et al., JHEP 03, 059 (2015), arXiv:1411.7394 [hep-ph] doi: 10.1007/JHEP03(2015)059
  • [1] P. Minkowski, Phys. Lett. B 67, 421 (1977) doi: 10.1016/0370-2693(77)90435-X
    [2] R. N. Mohapatra and G. Senjanovic, Phys. Rev. Lett. 44, 912 (1980) doi: 10.1103/PhysRevLett.44.912
    [3] A. M. Abdullahi, P. B. Alzas, B. Batell et al., J. Phys. G 50(2), 020501 (2023), arXiv:2203.08039 [hep-ph] doi: 10.1088/1361-6471/ac98f9
    [4] M. Agostini, G. Benato, J. A. Detwiler et al., Rev. Mod. Phys. 95(2), 025002 (2023), arXiv:2202.01787 [hep-ex] doi: 10.1103/RevModPhys.95.025002
    [5] Y. Cai, T. Han, T. Li and R. Ruiz, Front. in Phys. 6, 40 (2018), arXiv:1711.02180 [hep-ph] doi: 10.3389/fphy.2018.00040
    [6] T. Han and B. Zhang, Phys. Rev. Lett. 97, 171804 (2006), arXiv:hep-ph/0604064 [hep-ph] doi: 10.1103/PhysRevLett.97.171804
    [7] A. M. Sirunyan et al. [CMS], JHEP 01 , 122 (2019), arXiv:1806.10905 [hep-ex]
    [8] G. Aad et al. [ATLAS], JHEP 10 , 265(2019), arXiv:1905.09787 [hep-ex]]
    [9] A. Davidson, Phys. Rev. D 20, 776 (1979) doi: 10.1103/PhysRevD.20.776
    [10] R. N. Mohapatra and R. E. Marshak, Phys. Rev. Lett. 44(2), 1316 (1980)[erratum: Phys. Rev. Lett. 44 , 1643 (1980)] doi: 10.1103/PhysRevLett.44.1316
    [11] W. Y. Keung and G. Senjanovic, Phys. Rev. Lett. 50, 1427 (1983) doi: 10.1103/PhysRevLett.50.1427
    [12] K. A. Urquía-Calderón, arXiv:2310.17406 [hep-ph]
    [13] A. Tumasyan et al. [CMS], JHEP 11 , 181 (2023), arXiv:2307.06959[hep-ex]]
    [14] G. Aad et al. [ATLAS], Eur. Phys. J. C 83 (12), 1164 (2023), arXiv:2304.09553 [hep-ex]
    [15] J. P. Delahaye, M. Diemoz, K. Long et al., arXiv:1901.06150 [physics.acc-ph]
    [16] K. Long, D. Lucchesi, M. Palmer et al., Nature Phys. 17(3), 289 (2021), arXiv:2007.15684 [physics.acc-ph] doi: 10.1038/s41567-020-01130-x
    [17] W. Liu, K. P. Xie and Z. Yi, Phys. Rev. D 105(9), 095034 (2022), arXiv:2109.15087 [hep-ph] doi: 10.1103/PhysRevD.105.095034
    [18] T. Li, H. Qin, C. Y. Yao et al., Phys. Rev. D 106(3), 035021 (2022), arXiv:2205.04214 [hep-ph] doi: 10.1103/PhysRevD.106.035021
    [19] I. Chakraborty, H. Roy and T. Srivastava, arXiv:2206.07037 [hep-ph]
    [20] O. Mikulenko and M. Marinichenko, JHEP 01, 032 (2024), arXiv:2309.16837 [hep-ph]
    [21] Z. Wang, X. H. Yang and X. Y. Zhang, arXiv:2311.15166 [hep-ph].
    [22] K. Mekala, J. Reuter and A. F. Żarnecki, Phys. Lett. B 841, 137945 (2023), arXiv:2301.02602 [hep-ph] doi: 10.1016/j.physletb.2023.137945
    [23] T. H. Kwok, L. Li, T. Liu et al., arXiv:2301.05177 [hep-ph].
    [24] P. Li, Z. Liu and K. F. Lyu, JHEP 03, 231 (2023), arXiv:2301.07117 [hep-ph]
    [25] Q. H. Cao, K. Cheng and Y. Liu, arXiv:2403.06561 [hep-ph]
    [26] T. Li, C. Y. Yao and M. Yuan, JHEP 09, 131 (2023), arXiv:2306.17368 [hep-ph]
    [27] E. Antonov, A. Drutskoy and M. Dubinin, JETP Lett. 118(7), 461 (2023), arXiv:2308.02240 [hep-ph] doi: 10.1134/S0021364023602518
    [28] R. Jiang, T. Yang, S. Qian et al, arXiv:2304.04483 [hep-ph].
    [29] J. de Blas et al. [Muon Collider], arXiv:2203.07261[hep-ph]]
    [30] X. G. He, G. C. Joshi, H. Lew et al., Phys. Rev. D 43, 22 (1991) doi: 10.1103/PhysRevA.43.22
    [31] A. Das, T. Nomura and T. Shimomura, Eur. Phys. J. C 83(9), 786 (2023), arXiv:2212.11674 [hep-ph] doi: 10.1140/epjc/s10052-023-11955-4
    [32] J. Sun, F. Huang and X. G. He, Phys. Lett. B 845, 138121 (2023), arXiv:2307.00531 [hep-ph] doi: 10.1016/j.physletb.2023.138121
    [33] S. K.A., A. Das, G. Lambiase, T. Nomura and Y. Orikasa, arXiv:2308.14483 [hep-ph]
    [34] A. Das and Y. Orikasa, arXiv:2401.00696 [hep-ph]
    [35] S. R. Mishra et al., Phys. Rev. Lett. 66, 3117 (1991) doi: 10.1103/PhysRevLett.66.3117
    [36] B. Abi et al., Phys. Rev. Lett. 126, 141801 (2021), arXiv:2104.03281 [hep-ex] doi: 10.1103/PhysRevLett.126.141801
    [37] S. Baek, N. G. Deshpande, X. G. He et al., Phys. Rev. D 64, 055006 (2001), arXiv:hep-ph/0104141 [hep-ph] doi: 10.1103/PhysRevD.64.055006
    [38] W. Altmannshofer, S. Gori, M. Pospelov et al., Phys. Rev. D 89, 095033 (2014), arXiv:1403.1269 [hep-ph] doi: 10.1103/PhysRevD.89.095033
    [39] R. Aaij et al. [LHCb], Nature Phys. 18 (3), 277 (2022), arXiv:2103.11769 [hep-ex]
    [40] S. Baek and P. Ko, JCAP 10, 011 (2009), arXiv:0811.1646 [hep-ph]
    [41] I. Holst, D. Hooper and G. Krnjaic, Phys. Rev. Lett. 128(14), 141802 (2022), arXiv:2107.09067 [hep-ph] doi: 10.1103/PhysRevLett.128.141802
    [42] S. Baek, H. Okada and K. Yagyu, JHEP 04, 049 (2015), arXiv:1501.01530 [hep-ph]
    [43] K. Asai, K. Hamaguchi and N. Nagata, Eur. Phys. J. C 77(11), 763 (2017), arXiv:1705.00419 [hep-ph] doi: 10.1140/epjc/s10052-017-5348-x
    [44] G. Y. Huang, F. S. Queiroz and W. Rodejohann, Phys. Rev. D 103(9), 095005 (2021), arXiv:2101.04956 [hep-ph] doi: 10.1103/PhysRevD.103.095005
    [45] A. Dasgupta, P. S. B. Dev, T. Han, et al., JHEP 12, 011 (2023), arXiv:2308.12804 [hep-ph]
    [46] L. Lavoura, Phys. Lett. B 609, 317 (2005), arXiv:hep-ph/0411232 [hep-ph] doi: 10.1016/j.physletb.2005.01.047
    [47] T. Araki, J. Heeck and J. Kubo, JHEP 07, 083 (2012), arXiv:1203.4951 [hep-ph]
    [48] K. Asai, K. Hamaguchi, N. Nagata et al., Phys. Rev. D 99(5), 055029 (2019), arXiv:1811.07571 [hep-ph] doi: 10.1103/PhysRevD.99.055029
    [49] K. Asai, K. Hamaguchi, N. Nagata et al., JCAP 11, 013 (2020), arXiv:2005.01039 [hep-ph]
    [50] D. Borah, A. Dasgupta and D. Mahanta, Phys. Rev. D 104(7), 075006 (2021), arXiv:2106.14410 [hep-ph] doi: 10.1103/PhysRevD.104.075006
    [51] S. Patra, S. Rao, N. Sahoo and N. Sahu, Nucl. Phys. B 917, 317 (2017), arXiv:1607.04046 [hep-ph] doi: 10.1016/j.nuclphysb.2017.02.010
    [52] P. F. de Salas, D. V. Forero, S. Gariazzo et al., JHEP 02, 071 (2021), arXiv:2006.11237 [hep-ph]
    [53] A. Granelli, K. Hamaguchi, N. Nagata, et al., JHEP 09, 079 (2023), arXiv:2305.18100 [hep-ph]
    [54] M. Drewes, J. Klarić and J. López-Pavón, Eur. Phys. J. C 82(12), 1176 (2022), arXiv:2207.02742 [hep-ph] doi: 10.1140/epjc/s10052-022-11100-7
    [55] M. Escudero, D. Hooper, G. Krnjaic and M. Pierre, JHEP 03, 071 (2019), arXiv:1901.02010 [hep-ph]
    [56] Y. M. Andreev, D. Banerjee, B. B. Oberhauser, et al., arXiv:2401.01708 [hep-ex]
    [57] J. P. Lees et al. [BaBar], Phys. Rev. D 94 (1), 011102 (2016), arXiv:1606.03501[hep-ex]
    [58] G. Aad et al. [ATLAS], JHEP 07 , 090(2023), arXiv:2301.09342 [hep-ex]
    [59] G. Aad et al. [ATLAS], arXiv:2402.15212 [hep-ex]
    [60] W. Altmannshofer, S. Gori, M. Pospelov and I. Yavin, Phys. Rev. Lett. 113, 091801 (2014), arXiv:1406.2332 [hep-ph] doi: 10.1103/PhysRevLett.113.091801
    [61] A. Hayrapetyan et al. [CMS], arXiv:2403.00100 [hep-ex]
    [62] A. Hayrapetyan et al. [CMS], JHEP 03 , 105 (2024), arXiv:2312.07484 [hep-ex]
    [63] T. Appelquist, B. A. Dobrescu and A. R. Hopper, Phys. Rev. D 68, 035012 (2003), arXiv:hep-ph/0212073 [hep-ph] doi: 10.1103/PhysRevD.68.035012
    [64] J. Nakajima, A. Das, K. Fujii, et al., arXiv:2203.06929[hep-ex]
    [65] M. T. Arun, A. Chatterjee, T. Mandal et al., Phys. Rev. D 106(9), 095035 (2022), arXiv:2204.02949 [hep-ph] doi: 10.1103/PhysRevD.106.095035
    [66] A. Alloul, N. D. Christensen, C. Degrande, et al., Comput. Phys. Commun. 185, 2250 (2014), arXiv:1310.1921 [hep-ph] doi: 10.1016/j.cpc.2014.04.012
    [67] J. Alwall, R. Frederix, S. Frixione et al., JHEP 07, 079 (2014), arXiv:1405.0301 [hep-ph] doi: 10.1007/JHEP07(2014)079
    [68] T. Sjöstrand, S. Ask, J. R. Christiansen et al., Comput. Phys. Commun. 191, 159 (2015), arXiv:1410.3012 [hep-ph] doi: 10.1016/j.cpc.2015.01.024
    [69] J. de Favereau et al. [DELPHES 3], JHEP 02 , 057 (2014), arXiv:1307.6346 [hep-ex]
    [70] M. Boronat, J. Fuster, I. Garcia et al., Phys. Lett. B 750, 95 (2015), arXiv:1404.4294 [hep-ex] doi: 10.1016/j.physletb.2015.08.055
    [71] G. Cowan, K. Cranmer, E. Gross et al., Eur. Phys. J. C 71, 1554 (2011), arXiv:1007.1727 [physics.data-an] doi: 10.1140/epjc/s10052-011-1554-0
    [72] A. Dey, R. Rahaman and S. K. Rai, arXiv:2207.06857 [hep-ph]
    [73] C. Brust, P. Maksimovic, A. Sady et al., JHEP 04, 079 (2015), arXiv:1410.0362 [hep-ph] doi: 10.1007/JHEP04(2015)079
    [74] F. del Aguila, M. Chala, J. Santiago et al., JHEP 03, 059 (2015), arXiv:1411.7394 [hep-ph] doi: 10.1007/JHEP03(2015)059
  • 加载中

Figures(8) / Tables(2)

Get Citation
Ru-Yi He, Jia-Qi Huang, Jin-Yuan Xu, Fa-Xin Yang, Zhi-Long Han and Feng-Lan Shao. Heavy Neutral Leptons in Gauged U(1)LµLτ at Muon Collider[J]. Chinese Physics C. doi: 10.1088/1674-1137/ad4d61
Ru-Yi He, Jia-Qi Huang, Jin-Yuan Xu, Fa-Xin Yang, Zhi-Long Han and Feng-Lan Shao. Heavy Neutral Leptons in Gauged U(1)LµLτ at Muon Collider[J]. Chinese Physics C.  doi: 10.1088/1674-1137/ad4d61 shu
Milestone
Received: 2024-03-26
Article Metric

Article Views(1568)
PDF Downloads(18)
Cited by(0)
Policy on re-use
To reuse of Open Access content published by CPC, for content published under the terms of the Creative Commons Attribution 3.0 license (“CC CY”), the users don’t need to request permission to copy, distribute and display the final published version of the article and to create derivative works, subject to appropriate attribution.
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Email This Article

Title:
Email:

Heavy neutral leptons in gauged U(1)LµLτ at muon collider

  • 1. School of Physics and Technology, University of Jinan, Jinan 250022, China
  • 2. School of Physics and Physical Engineering, Qufu Normal University, Qufu 273165, China

Abstract: Heavy neutral leptons N are the most appealing candidates to generate tiny neutrino masses. We studied the signature of heavy neutral leptons in gauged U(1)_{L_\mu-L_\tau} at a muon collider. Charged under the U(1)_{L_\mu-L_\tau} symmetry, the heavy neutral leptons can be pair produced via the new gauge boson Z' at the muon collider as \mu^+\mu^-\to Z^{\prime *}\to NN and \mu^+\mu^-\to Z^{\prime (*)} \gamma\to NN\gamma . We then performed a detailed analysis on the lepton number violation signature \mu^+\mu^-\to NN\to \mu^\pm\mu^\pm W^\mp W^\mp and \mu^+\mu^-\to NN \gamma\to \mu^\pm\mu^\pm W^\mp W^\mp \gamma at the 3 TeV muon collider, where the hadronic decays of W boson are treated as fat-jets J. These lepton number violation signatures have quite clean backgrounds at the muon collider. Our simulation shows that a wide range of viable parameter space is within the reach of the 3 TeV muon collider. For instance, with new gauge coupling g'=0.6 and an integrated luminosity of 1000 fb ^{-1} , the \mu^\pm\mu^\pm JJ signal could probe m_{Z'}\lesssim 13 TeV. Meanwhile, if the gauge boson mass satisfies 2 m_N<m_{Z'}<\sqrt{s} , the \mu^\pm\mu^\pm JJ\gamma signature would be more promising than the \mu^\pm\mu^\pm JJ signature.

    HTML

    I.   INTRODUCTION
    • Heavy neutral leptons N are well motivated to explain the origin of tiny neutrino masses. Due to the singlet nature of heavy neutral leptons under the standard model gauge group, we can write a Majorana mass term m_N \overline{N^c}N , which results in light Majorana neutrino masses as m_\nu\sim m_D^2/m_N via the type-I seesaw mechanism [1, 2]. To generate the sub-eV neutrino masses, the heavy neutral lepton masses should be at very high scale m_N\gtrsim10^{14} GeV if m_D is at the electroweak scale, which is far beyond current experimental reach. Alternatively, an electroweak scale m_N is usually assumed for phenomenological studies [3], which is then determined by the mixing parameter V_{\ell N} .

      To confirm the Majorana nature of neutrinos, lepton number violation signatures are expected. The most sensitive experiment is the neutrinoless double beta decay, which could probe the inverted mass ordering scenario in next-generation experiments [4]. Meanwhile, the detection of the lepton number violation signature at colliders could unravel the explicit mechanism of tiny neutrino masses [5]. For heavy neutral leptons in the type-I seesaw, the distinct signature at the hadron collider is pp\to W^{\pm*}\to \ell^\pm N\to\ell^\pm\ell^\pm jj [6]. However, the sensitive region of this signature heavily depends on a relatively large mixing parameter V_{\ell N} [7, 8], which is usually much higher than the natural seesaw prediction V_{\ell N}\sim\sqrt{m_\nu/m_N} .

      Besides the canonical seesaw, there are many theories with extended gauge groups, such as the U(1)_{B-L} model [9, 10] and left-right symmetric model [11, 12]. In these models, the heavy neutral leptons are charged under the extended gauge group, which opens new production mechanisms for heavy neutral leptons. Pair production of heavy neutral leptons via the decay of the Z' boson in the framework of the left-right symmetric model has been searched for recently by the CMS collaboration [13]. The region within m_N\lesssim1.4 TeV and m_{Z'}\lesssim4.4 TeV has been excluded by the dimuon channel. Meanwhile, the ATLAS collaboration searches for the right-handed W' boson decaying to heavy neural lepton N and leptons \ell , which can exclude the region with m_N\lesssim3.8 TeV and m_{W'}\lesssim6.4 TeV [14].

      The construction of a multi-TeV muon collider was proposed recently [15, 16]. Since then, searches for heavy neutral leptons at the multi-TeV muon collider have drawn increasing interest [1721]. One promising signature is \mu^+\mu^-\to N\nu , which could probe the mixing parameter |V_{\mu N}|^2\gtrsim10^{-6} at the 10 TeV muon collider [2224]. The Majorana nature of heavy neutral leptons can be confirmed with double peaks of the rapidity distribution of the reconstructed N [25]. Another interesting signature is the lepton number violation signal via vector boson scattering process W^\pm Z/\gamma\to \ell^\pm N [26], via associated production process \mu^+\mu^-\to N W^\pm \ell^\mp [27], or at the same-sign muon collider via \mu^+\mu^+\to W^+W^+ process [28]. These studied signatures also require that the mixing parameter V_{\ell N} is not too small. Meanwhile, under the tight constraints from current experimental searches [13, 14], the first stage of a 3 TeV muon collider [29] is not promising to probe prompt heavy neutral leptons in the U(1)_{B-L} and left-right symmetric models.

      The multi-TeV muon collider is a perfect machine to test the muon-philic forces. One attractive option is the gauged U(1)_{L_\mu-L_\tau} model [3034]. Compared to U(1)_{B-L} or left-right symmetry, U(1)_{L_\mu-L_\tau} symmetry is less constrained due to the lack of direct couplings to electrons and quarks. For instance, there is only a loose constraint from neutrino trident production when m_{Z'}\gtrsim100 GeV [35]. Therefore, this model has been extensively studied to explain the anomaly of the muon magnetic moment [36, 37], B meson anomaly [38, 39], dark matter [40, 41], and neutrino masses [42, 43]. It is shown that a 3 TeV muon collider is powerful enough to probe a relatively large parameter space with m_{Z'}\lesssim10 TeV [44, 45]. In this paper, we consider the gauged U(1)_{L_\mu-L_\tau} model with three heavy neutral leptons N_e,N_\mu,N_\tau . We then study the pair production of heavy neutral lepton N via Z' boson at a 3 TeV muon collider. To test the Majorana nature of neutrinos, we focus on the lepton number violation process \mu^+\mu^-\to Z^{\prime *}\to NN\to \mu^\pm\mu^\pm W^\mp W^\mp and \mu^+\mu^-\to Z^{\prime} \gamma\to NN \gamma \to \mu^\pm\mu^\pm W ^\mp W^\mp \gamma with the hadronic decay of W.

      The remainder of this paper is organized as follows. In Section II, we review the gauged U(1)_{L_\mu-L_\tau} model with three heavy neutral leptons and discuss relevant experimental constraints. The decay properties of gauge boson Z' and heavy neutral lepton N are also considered in this section. In Section III, we study the pair production of heavy neutral lepton at the 3 TeV muon collider. Analysis of the lepton number violation signatures is performed in Section IV. Finally, Section V summarizes our conclusions.

    II.   THE MODEL
    • In this paper, we consider the anomaly-free gauged U(1)_{L_\mu-L_\tau} extension of the type-I seesaw mechanism. Three heavy neutral leptons N_e,N_\mu,N_\tau with U(1)_{L_\mu-L_\tau} charge (0,1,-1) are introduced to generate tiny neutrino masses. Predicting type C ^R of the two-zero minor [46, 47], the minimal model with only one scalar singlet \Phi_1 , which carries U(1)_{L_\mu-L_\tau} charge +1 , is now tightly constrained by the neutrino oscillation data and the sum of light neutrino masses [48, 49]. Thus, the second scalar singlet \Phi_2 with U(1)_{L_\mu-L_\tau} charge +2 is also employed to reduce the above conflict [50]. The Yukawa interactions and mass terms relevant to neutrino masses are given by [51]

      \begin{aligned}[b] {\cal{L}}\supset & - y_e \bar{L}_e \tilde{H} N_e - y_\mu \bar{L}_\mu \tilde{H} N_\mu - y_\tau \bar{L}_\tau \tilde{H} N_\tau \\ & -\frac{1}{2} M_{ee} \overline{N^c_e}N_e-M_{\mu\tau}{N^c_\mu}N_\tau -y_{e\mu} \Phi_1^\dagger \overline{N^c_e}N_\mu \\ & -y_{e\tau} \Phi_1 \overline{N^c_e}N_\tau-\frac{1}{2}y_{\mu\mu} \Phi_2^\dagger \overline{N^c_\mu}N_\mu -\frac{1}{2}y_{\tau\tau} \Phi_2 \overline{N^c_\tau}N_\tau+ \text{h.c.}, \end{aligned}

      (1)

      where L_e,L_\mu,L_\tau are the lepton doublets, H is the standard model Higgs doublet, and \tilde{H}=i\tau_2 H^* . After the spontaneous symmetry breaking, we can denote the vacuum expectation values of scalars as \langle H \rangle=v_0,\langle \Phi_1 \rangle=v_1,\langle \Phi_2 \rangle=v_2 . Then, the Dirac neutrino mass matrix and heavy neutral lepton mass matrix are given by

      \begin{aligned}[b] & M_D=\left( \begin{array}{c c c} y_e v_0 & 0 & 0\\ 0 & y_\mu v_0&0 \\ 0 & 0 & y_\tau v_0 \end{array} \right), \\ & M_N = \left( \begin{array}{c c c} M_{ee} & y_{e\mu}v_1 & y_{e\tau}v_1\\ y_{e\mu}v_1 & y_{\mu\mu} v_2& M_{\mu\tau} \\ y_{e\tau}v_1 & M_{\mu\tau} & y_{\tau\tau} v_2 \end{array}\right). \end{aligned}

      (2)

      Light neutrino masses are generated via the type-I seesaw formula

      \begin{aligned}[b] M_\nu\simeq - M_D M_N^{-1} M_D^T. \end{aligned}

      (3)

      Without any specific structure of heavy neutral lepton mass matrix M_N , the obtained light neutrino mass matrix M_\nu is a general symmetric matrix; thus, it can easily fit the neutrino oscillation data [52]. Conversely, we can use Eq. (3) to express the mass matrix M_N as [53]

      \begin{aligned}[b] M_N=-M_D^T M_\nu^{-1} M_D, \end{aligned}

      (4)

      where M_\nu=U^*\text{diag}(m_{\nu_1},m_{\nu_2},m_{\nu_3}) U^\dagger , and U is the light neutrino mixing matrix. In this way, the mass matrix M_N is determined by the light neutrino oscillation data and the Yukawa coupling y_e,y_\mu,y_\tau . The mass matrix M_N can be diagnalized by a unitarity matrix Ω as

      \begin{aligned}[b] \Omega^T M_N \Omega = \text{diag}(m_{N_1},m_{N_2},m_{N_3}). \end{aligned}

      (5)

      Considering the mixing with light neutrinos, the heavy neutral leptons interact with the W^\pm,Z gauge boson and Higgs boson as

      \begin{aligned}[b] {\cal{L}}\supset & -\frac{g}{\sqrt{2}} \bar{N}_k V_{\ell k}^{*} \gamma^\mu P_L \ell W_\mu -\frac{g}{2\cos\theta_W} \bar{N}_k V_{\ell k}^{*} \gamma^\mu P_L \nu_\ell Z_\mu \\ & - \frac{g m_N}{2 m_W} \bar{N}_k V_{\ell k}^{*} P_L \nu_\ell h + \text{h.c.}, \end{aligned}

      (6)

      where the mixing matrix V\simeq M_D M_N^{-1} . For electroweak-scale heavy neutral leptons, we assume a seesaw-induced mixing parameter V_{\ell N}\sim \sqrt{m_\nu/m_N}\sim10^{-6} , which is allowed by current experimental limits [3].

      The interactions of the new boson Z' with fermions are

      \begin{aligned}[b] {\cal{L}}\supset & g'(\bar{\mu}\gamma^\mu\mu-\bar{\tau}\gamma^\mu\tau+\bar{\nu}_\mu\gamma^\mu P_L\nu_\mu-\bar{\nu}_\tau\gamma^\mu P_L\nu_\tau\\ & +\bar{N}_\mu\gamma^\mu P_R N_\mu-\bar{N}_\tau\gamma^\mu P_R N_\tau)Z'_\mu. \end{aligned}

      (7)

      In terms of mass eigenstates, the new neutral current of neutral leptons can be rewritten as

      \begin{aligned}[b] {\cal{L}}\supset & g'[\bar{\nu}_i (U_{\mu i}^{*}U_{\mu j}-U_{\tau i}^{*}U_{\tau j})\gamma^\mu P_L\nu_j \\ & +\bar{N}_i (\Omega_{\mu i}^{*}\Omega_{\mu j}-\Omega_{\tau i}^{*}\Omega_{\tau j})\gamma^\mu P_R N_j]Z'_\mu, \end{aligned}

      (8)

      From Eqs. (6) and (8), it is obvious that the collider phenomenology of heavy neutral leptons depends on the mixing patterns, which makes the results model-dependent. In the following collider simulation, we consider a muon-philic heavy neutral lepton N, which couples exclusively to muons [54]. We also take the mixing parameter \Omega_{\mu N}^{*}\Omega_{\mu N}-\Omega_{\tau N}^{*}\Omega_{\tau N}=1 in the following calculation, so the general model-dependent results can be obtained by a simple rescaling.

    • A.   Constraints

    • In this section, we briefly summarize the constraints on the gauged U(1)_{L_\mu-L_\tau} model. One important motivation of this symmetry is to explain the anomaly of the muon magnetic moment \Delta a_\mu=(251\pm59)\times10^{-11} [36]. The new gauge boson Z' contributes to \Delta a_\mu at one loop level, which is evaluated as [37]

      \begin{aligned}[b] \Delta a_\mu=\frac{g^{\prime 2}}{4\pi^2}\int_0^1{\rm{d}}x\frac{m_\mu^2 x(1-x)^2}{m_\mu^2(1-x)^2+m_{Z'}^2 x}. \end{aligned}

      (9)

      The viable parameter space to interpret \Delta a_\mu is shown in Fig. 1. For very light m_{Z'} , we have \Delta a_\mu\simeq g^{\prime 2}/8\pi^2 , which requires g'\sim4\times10^{-4} to explain \Delta a_\mu . In the heavy Z' limit, \Delta a_\mu\simeq g^{\prime 2} m_\mu^2/12\pi^2 m_{Z'}^2 ; thus, the experimental discrepancy requires g'/m_{Z'}\sim 5\times10^{-3}\; \text{GeV}^{-1} .

      Figure 1.  (color online) Constraints on the gauged U(1)_{L_\mu-L_\tau} . The blue band can explain the \Delta a_\mu result [36]. The gray, orange, red, purple, and pink regions are excluded by BBN [55], CCFR [35], NA64μ [56], BABAR [57], and LHC [58, 59], respectively.

      For the gauge boson Z' , one tight constraint is the dimuon production from the inelastic neutrino-nucleus scattering \nu_\mu N_u \to \nu_\mu N_u \mu^+\mu^- , where N_u denotes the nucleus [60]. The CCFR experiment measured the neutrino trident production cross section, which is consistent with the standard model prediction as \sigma/\sigma_\text{SM}=0.82\pm0.28 [35]. This result has excluded the \Delta a_\mu favored region with m_{Z'}\gtrsim0.3 GeV. Recently, the NA64μ experiment has excluded g'\gtrsim6\times10^{-4} with m_{Z'}\lesssim0.1 GeV by the process of \mu N_u\to \mu N_u Z' \to \mu N_u \bar{\nu} \nu [56], which makes m_{Z'}\sim0.01 GeV the \Delta a_\mu -favored region. A more stringent constraint comes from the e^+e^-\to \mu^+\mu^- Z' \to 4\mu searches at BABAR in the mass region of 0.212\; { \rm GeV}<m_{Z'}<10 GeV [57]. For Z' mass in the range of [5, 81] GeV, the LHC experiment has excluded g'\gtrsim0.003 to 0.2 (depending on m_{Z'} ) by the process pp\to Z'\mu^+\mu^-\to4\mu with 139 fb ^{-1} data [58]. Within the same mass region, a complement search by the process pp\to Z'\mu^\pm\nu\to 3\mu \nu with 140 fb ^{-1} data was also performed [59], which has better sensitivity in the light Z' region. In this paper, we consider the combined limit of the 3\mu and 4\mu channels at the LHC [59]. Meanwhile, for Z' lighter than approximately 6 MeV, it is disfavored by \Delta N_\text{eff} constraints from big-bang nucleosynthesis (BBN) [55], because decays of Z' could heat the neutrino population and delay the process of neutrino decoupling. In the following study, we consider m_{Z'}\gtrsim 100 GeV to avoid these tight constraints.

      For the heavy neutral lepton N, the promising signature at the LHC is pp\to W^{\pm(*)}\to \ell^\pm N , which depends on the mixing parameter V_{\ell N} . When m_N<m_W , N is produced from on-shell W boson decay; then, LHC could exclude |V_{\ell N}|^2\gtrsim2.9\times10^{-6} via the fully-leptonic decay channel N\to \ell \ell \nu [61]. Meanwhile, for light N with proper V_{\ell N} , it becomes long-lived and leads to displaced vertex signature. Currently, the LHC has excluded |V_{\mu N}|^2 > 4\times10^{-7} for m_N=10 GeV [62]. Above m_W , production of N is mediated by off-shell W boson, so the LHC limit becomes much weaker, e.g., |V_{\mu N}|^2=1.1\times10^{-3} for m_N=100 GeV [61]. In this paper, we also consider m_N>100 GeV with seesaw-induced mixing |V_\ell N|^2\lesssim10^{-12} to satisfy current LHC limits.

    • B.   Decay properties

    • Before studying the explicit signatures, we first review the decay properties of new gauge boson Z' and heavy neutral lepton N. The partial decay widths of Z' are calculated as

      \Gamma(Z'\to \ell^+\ell^-) = \frac{g'^2}{12\pi}m_{Z'},

      (10)

      \Gamma(Z'\to \nu_\ell \nu_\ell) = \frac{g'^2}{24\pi}m_{Z'},

      (11)

      \Gamma(Z'\to N_\ell N_\ell ) = \frac{g'^2}{24\pi}m_{Z'}\left(1-4\frac{m_{N_\ell }^2}{m_{Z'}^2}\right),

      (12)

      where \ell=\mu,\tau , and we have assumed vanishing masses for \ell,\nu_\ell in the above calculations. For simplicity, we assume that the tree-level kinetic mixing between boson Z' and photon γ is absent. However, mixing between Z' and γ in the form of -\dfrac{\epsilon}{2} Z'_{\mu\nu} F^{\mu\nu} appears at one-loop level, where the kinetic mixing factor \epsilon is calculated as [55]

      \begin{aligned}[b] \epsilon=-\frac{e g'}{12\pi^2}\log\left(\frac{m_\tau^2}{m_\mu^2}\right) \simeq - \frac{g'}{70}. \end{aligned}

      (13)

      In this way, Z' also couples to electron e via mixing with γ. The Z'\to e^+e^- decay width is suppressed by the kinetic mixing factor \epsilon\simeq -g'/70 , so it is not taken into account in this study.

      After the production of heavy neutral lepton N, it decays via mixing with light neutrinos. When the heavy neutral lepton is heavier than the standard model gauge bosons, the two-body decays are the dominant modes. The partial decay widths are given by

      \Gamma(N\to \ell^\pm W^\mp) = \frac{|V_{\ell N}|^2}{16\pi } \frac{(m_N^2-m_W^2)^2(m_N^2+2 m_W^2)}{m_N^3 v_0^2},

      (14)

      \Gamma(N\to \nu Z) = \frac{|V_{\ell N}|^2}{32\pi } \frac{(m_N^2-m_Z^2)^2(m_N^2+2 m_Z^2)}{m_N^3 v_0^2},

      (15)

      \Gamma(N\to \nu h) = \frac{|V_{\ell N}|^2}{32\pi } \frac{(m_N^2-m_h^2)^2}{m_N v_0^2 }.

      (16)

      In Fig. 2, we show the branching ratio (BR) of gauge boson Z' and heavy neutral lepton N. The dilepton mode Z'\to \mu^+\mu^-,\tau^+\tau^- is always the dominant decay channel of Z' . Neglecting the final state's phase space effect, we have BR (Z'\to \nu_\ell \nu_\ell)= BR (Z'\to N_\ell N_\ell )=1/4 for two degenerate heavy neutral leptons. In the heavy m_N limit, we have BR (N \to \ell^\pm W^\mp) : BR (N \to \nu Z) : BR (N \to \nu h)= 2:1:1 . To reconstruct the heavy neutral lepton mass, we focus on the decay mode N\to \ell^\pm W^\mp with W\to q \bar{q}' in the collider simulation.

      Figure 2.  (color online) Branching ratio of gauge boson Z' (left) and heavy neutral lepton N (right).

    III.   PAIR PRODUCTION OF HEAVY NEUTRAL LEPTON
    • Through mixing with light neutrinos, the pair production of heavy neutral leptons at the muon collider can occur through s-channel Z exchange and t-channel W exchange. However, the production cross section is proportional to the fourth power of the mixing parameter |V_{\ell N}|^4 . Provided natural seesaw predicted value V_{\ell N}\lesssim 10^{-6}, the predicted cross section is negligible tiny.

      The heavy neutral lepton N is charged under U(1)_{L_\mu-L_\tau} symmetry, which induces the pair production of heavy neutral lepton via Z' at the muon collider. With fixed collision energy, the direct pair production process \mu^+\mu^-\to NN via the s-channel off-shell Z' is more promising when m_{Z'}>\sqrt{s} . Meanwhile, when m_{Z'}<\sqrt{s} , the new gauge boson Z' can be produced on-shell with an associated photon from the initial legs as \mu^+\mu^-\to Z' \gamma \to NN \gamma , which also requires m_N<m_{Z'}/2 to make the decay Z'\to NN kinematically allowed.

    • A.   Without associated photon

    • We first consider the pair production of heavy neutral leptons without associated photon. The production cross section of \mu^+\mu^-\to Z^{\prime *}\to NN with a center of mass energy \sqrt{s} is calculated as

      \begin{aligned}[b] & \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN)\\ = & \frac{g^{\prime 4}}{24\pi}\frac{s}{(s-m_{Z'}^2)^2+m_{Z'}^2\Gamma_{Z'}^2}\left(1-4\,\frac{m_N^2}{s}\right)^{3/2}, \end{aligned}

      (17)

      where \Gamma_{Z'} is the total decay width of Z' .

      The theoretical cross section of \mu^+\mu^-\to Z^{\prime *}\to NN at a 3 TeV muon collider for specific scenarios is shown in Fig. 3, where we directly use Eq. (17) without considering the initial state radiation effect. It is obvious that when m_{Z'}\simeq \sqrt{s} , the on-shell production of Z' can greatly enhance the cross section. The maximum value is approximately 12\pi \text{BR}(Z'\to \mu^+\mu^-) \text{BR}(Z'\to NN)/m_{Z'}^2\approx 3\pi/(8m_{Z'}^2) [45], which could be over 50 pb. For a light Z' , the cross section reduces to

      Figure 3.  (color online) Cross section of \mu^+\mu^-\to Z^{\prime *}\to NN at a 3 TeV muon collider. We have fixed m_N=100 GeV in the left panel and m_{Z'}=2000 GeV in the right panel.

      \begin{aligned}[b] \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN)\simeq \frac{g^{\prime 4}}{24\pi} \frac{1}{s} \left(1-4\,\frac{m_N^2}{s}\right)^{3/2}, \end{aligned}

      (18)

      which is independent of Z' mass. Typically, for g'=0.3 and m_N=1000 GeV, we have \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN)\simeq4.7 fb. Meanwhile, a heavy Z' leads to

      \begin{aligned}[b] \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN)\simeq \frac{g^{\prime 4}}{24\pi} \frac{s}{m_{Z'}^4} \left(1-4\,\frac{m_N^2}{s}\right)^{3/2}. \end{aligned}

      (19)

      In this scenario, the cross section is suppressed by the large value of m_{Z'} . While single production could probe m_N\lesssim \sqrt{s} , the pair production of heavy neutral lepton is kinematically allowed when m_N<\sqrt{s}/2 . Thus, the 3 TeV muon collider could only probe m_N\lesssim1500 GeV. For heavier m_N , we require a more energetic muon collider.

    • B.   With associated photon

    • The photon from initial state radiation carries part of the collision energy, which results in the gauge boson Z' produced on-shell when m_{Z'}\lesssim \sqrt{s} . In this case, there is a resonance peak in the invariant mass spectrum of the dilepton [44]. Here, we study the pair production of heavy neutral leptons with associated photon process as

      \begin{aligned}[b] \mu^+\mu^-\to Z^{\prime (*)}\gamma\to NN\gamma. \end{aligned}

      (20)

      Here, the scenario with off-shell Z' contribution is also considered when m_{Z'}\leqslant 2 m_N or m_{Z'}\geqslant \sqrt{s} .

      In Fig. 4, we show the cross section of \mu^+\mu^-\to NN\gamma at a 3 TeV muon collider, where the numerical results are calculated by MadGraph. There is also a sharp peak around m_{Z'}\simeq \sqrt{s} . To avoid the soft photon singularity, the detected photon is required to satisfy the following pre-selection cuts:

      Figure 4.  (color online) Cross section of \mu^+\mu^-\to NN\gamma at a 3 TeV muon collider. We have fixed m_N=100 GeV in the left panel and m_{Z'}=2000 GeV in the right panel.

      \begin{aligned}[b] P_T(\gamma)>20\; \text{GeV},\; |\eta(\gamma)|<2.5. \end{aligned}

      (21)

      When m_{Z'}>2 m_N , the on-shell production of Z' followed by the cascade decay Z'\to NN can notably enhance the cross section of \mu^+\mu^-\to NN\gamma . Using the narrow-width approximation, the cross section can be expressed as [63]

      \begin{aligned}[b] \sigma(\mu^+\mu^-\to NN\gamma)\simeq \sigma(\mu^+\mu^-\to Z'\gamma) \times \text{BR}(Z'\to NN). \end{aligned}

      (22)

      Because the branching ratio of Z'\to NN is independent of the new gauge coupling g' , the cross section of \mu^+\mu^-\to Z'\gamma\to NN\gamma is proportional to e^2g^{\prime 2} , while the cross section of the off-shell process \mu^+\mu^-\to Z ^{\prime (*)}\gamma \to NN\gamma is proportional to e^2g^{\prime 4} . Thus, we observe that the enhancement effect becomes larger when the gauge coupling g' is smaller. For g'=0.1,\; m_N=100 GeV, and m_{Z'}=1000 GeV, we have \sigma(\mu^+\mu^-\to Z'\gamma\to NN\gamma)\sim 1 fb, while \sigma(\mu^+\mu^-\to Z^{\prime *}\to NN) is less than 0.1 fb; therefore, the former process is expected to be more promising at the muon collider.

    IV.   LEPTON NUMBER VIOLATION SIGNATURES
    • Pair production of heavy neutral leptons followed by various cascade decay modes could lead to many interesting signatures, such as monolepton, dilepton, and trilepton signals [64, 65]. In this paper, we focus on the most distinct lepton number violation, same-sign dimuon signal, which also has a much cleaner background compared with the lepton number conserving one.

      The simulation procedure is as follows. The FeynRules package [66] is used to implement the model. The parton level events are simulated with MadGraph5_aMC@NLO [67]. We then use PYTHIA 8 [68] for parton shower. The detector effects are included by employing Delphes 3 [69] with the detector card of the muon collider. The W boson from the heavy neutral lepton decay could be highly boosted, so the two jets from cascade W decay merge into one fat-jet J. We use the Valencia algorithm [70] with R=1.2 to reconstruct the fat-jets.

    • A.   Without associated photon

    • The full production process of the same-sign dimuon signature without associated photon is

      \begin{aligned}[b] \mu^+\mu^-\to Z^{\prime *}\to N N\to \mu^\pm W^\mp +\mu^\pm W^\mp, \end{aligned}

      (23)

      with the hadronic decays of W. The standard model backgrounds come from processes such as

      \begin{aligned}[b] \mu^+\mu^-\to \mu^+\mu^- W^+W^- , \mu^+\mu^-W^+ W^-\gamma/Z, W^\pm W^\pm W^\mp \mu^\mp \nu. \end{aligned}

      (24)

      The contributions of \mu^+\mu^- W^+W^- and \mu^+\mu^-W^+ W^-\gamma/Z are from lepton charge misidentification, which are suppressed by the misidentification rate 0.1% [17]. The W^\pm W^\pm W^\mp \mu^\mp \nu is dominant by the vector boson fusion process, where the two same-sign W bosons decay leptonically. There is also one possible background process \mu^+\mu^-\to W^+W^-jj with the light jets mistagged. However, by requiring the light jet masses close to the W mass, this background can easily be suppressed [17]. Meanwhile, the t\bar{t}W^\pm \mu^\mp\nu process also contributes to the same-sign dimuon signature, which can be further reduced by cut on the opposite-sign lepton [26]. Thus, we do not include contributions of the \mu^+\mu^-\to W^+W^- jj and t\bar{t}W^\pm \mu^\mp\nu processes in this paper.

      We first apply the following pre-selection cuts on the transverse momentum and pseudorapidity of the muon and fat-jets:

      \begin{aligned}[b] P_T (\mu^\pm)>50\; \text{GeV}, |\eta (\mu^\pm)|<2.5,P_T (J)>50\; \text{GeV}, |\eta (J)|<2.5. \end{aligned}

      (25)

      In Fig. 5, we show the distributions of some variables for the \mu^\pm\mu^\pm JJ signal and backgrounds after applying the pre-selection cuts. We have set m_{N}=1000 GeV, m_{Z'}= 2500 GeV, and g'=0.6 as the benchmark point of the signal. The cut flow for the \mu^\pm\mu^\pm JJ signature and backgrounds are summarized in Table 1.

      Figure 5.  (color online) Normalized distribution of transverse momentum of muon P_T(\mu^\pm) (up-left panel), transverse momentum of fat-jet P_T(J) (up-right panel), fat-jet mass m_J (down-left panel), and invariant mass of muon and fat-jet m_{\mu J} (down-right panel) for the \mu^\pm\mu^\pm JJ signature and corresponding backgrounds.

      σ(fb) \mu^\pm\mu^\pm W^\mp W^\mp \mu^+\mu^- W^+W^- \mu^+\mu^-W^+ W^-\gamma/Z W^\pm W^\pm W^\mp \mu^\mp \nu
      Pre-selection 68.19 21.57 2.13 3.11
      N_{\mu^\pm}=2 41.38 2.6\times10^{-2} 2.5\times10^{-3} 1.8\times10^{-1}
      N_J=2 24.87 1.1\times10^{-2} 1.3\times10^{-3} 7.1\times10^{-2}
      50 GeV \leqslant m_J\leqslant 100 GeV 14.41 3.2\times10^{-3} 2.1\times10^{-4} 2.0\times10^{-2}
      0.8 m_N<m_{\mu J}<1.2 m_N 13.62 3.4\times10^{-4} 2.7\times10^{-5} 8.1\times10^{-4}
      Significance 476.6 Total Background 1.2\times10^{-3}

      Table 1.  Cut flow table for the \mu^\pm\mu^\pm JJ signal at the 3 TeV muon collider and corresponding backgrounds. The significance is calculated with Eq. (29) by assuming an integrated luminosity of 1000 fb ^{-1} .

      The lepton number violation signature is satisfied for events with two same-sign muons. In the single production of the heavy neutral lepton process [26, 27], only one boosted W is expected from N decay. To distinguish from the single production and also suppress the background from vector boson fusion processes, we require exactly two fat-jets in the final states

      \begin{aligned}[b] N_{\mu^\pm}=2, \; N_J=2, \end{aligned}

      (26)

      although tagging one fat-jet has greater efficiency for the signal. To be identified as W bosons, the fat-jet masses are also required in the following range:

      \begin{aligned}[b] 50 \; \text{GeV} \leqslant m_J \leqslant 100 \; \text{GeV}. \end{aligned}

      (27)

      Because there are two heavy neutral leptons in the signal, we reconstruct their masses through the two-muon and two-fat-jet system by minimizing \chi^2= (m_{\mu_1 J_1} - m_N)^2+(m_{\mu_2 J_2} -m_N)^2 [17]. We require that both the invariant mass of muon and fat-jet m_{\mu J} from reconstructed N satisfy

      \begin{aligned}[b] 0.8 m_N<m_{\mu J}<1.2 m_N. \end{aligned}

      (28)

      According to the distributions in Fig. 5, we may tighten the cuts on P_T(\mu^\pm) and P_T(J) to suppress the background. However, we can see from Table 1 that the cut on the invariant mass m_{\mu J} is relatively efficient to reduce the background. After applying all these cuts, the total cross section of the background is approximately 1.2\times10^{-3} fb. Thus, with an integrated luminosity of 1000 fb ^{-1} , there are only 1.2 background events, while the signal events are over 10^4 . In this manner, the significance will be 476.6 for the benchmark point with 1000 fb ^{-1} data, where the significance is calculated as [71]

      \begin{aligned}[b] {\cal{S}}=\sqrt{2\left[(N_S+N_B)\ln\left(1+\frac{N_S}{N_B}\right)-N_S\right]}. \end{aligned}

      (29)

      Here, N_S and N_B are the event numbers of signal and background, respectively. To reach the 5\sigma discovery limit, we require only 0.11 fb ^{-1} data.

      Based on the above analysis, we then explore the sensitivity of the \mu^\pm \mu^\pm JJ signature at the 3 TeV muon collider. The results are shown in Fig. 6. There are three free parameters g',m_{Z'} , and m_N related to the collider phenomenology. With fixed mass relation m_{N} = m_{Z'}/3 , we could probe m_{Z'} approximately in the range of [300, 4500] GeV. Around the resonance region m_{Z'}\sim3000 GeV, the gauge coupling g' could be down to approximately 0.025 with 1000 fb ^{-1} data. We also find that for m_N\lesssim300 GeV, the cut efficiency decreases quickly as m_N becomes smaller. One main reason for this is that the heavy neutral leptons are also highly boosted at the 3 TeV muon collider for such light m_N , so the muons and W-jets from boosted N decays are mostly non-isolated [72]. One may use the substructure-based variables as lepton sub-jet fraction (LSF) and lepton mass drop (LMD) [73] to probe the light m_N , which is beyond the scope of this study.

      Figure 6.  (color online) Sensitivity of the \mu^\pm \mu^\pm JJ signature at the 3 TeV muon collider. In the upper-left panel, we fix the mass relation with m_N=m_{Z'}/3 . In the upper-right panel, we set m_N=1000 GeV. In the lower panel, we fix the gauge coupling g'=0.6 . The dark-green region corresponds to sensitivity with 1 fb ^{-1} data, while the light-green region corresponds to sensitivity with 1000 fb ^{-1} data. The orange region is excluded by neutrino trident production at CCFR [35]. The magenta region is not allowed kinematically. The gray line is the projected sensitivity at the HL-LHC [74]. The red line is the combined sensitivity of the 3 TeV muon collider via processes \mu^+\mu^-\to \ell^+\ell^-(\ell=\mu\; \text{or}\; \tau), \ell\bar{\ell}\gamma(\ell=\mu\; \text{or}\; \tau\; \text{or}\; \nu_{\mu/\tau}) [44].

      For pair production of heavy neutral leptons without associated photon signature, the gauge boson Z' does not need to be heavier than 2 m_N . In the upper-right panel of Fig. 6, we have fixed m_N=1000 GeV for illustration. The production cross section of \mu^+\mu^-\to NN is approximately constant for light Z' , so g'\sim0.13 is nearly independent of m_{Z'} when Z' is lighter than 1 TeV. This channel is most sensitive for the large m_{Z'} region. For instance, we may probe m_{Z'}\sim10 TeV with g'\gtrsim0.4 and m_N\sim1000 GeV at the 3 TeV muon collider. Near the resonance region with m_{Z'}\sim \sqrt{s} , we can even have a promising signature when g'\gtrsim0.14 with only 1 fb ^{-1} data. Compared with the direct dilepton channels

      \mu^+\mu^-\to \ell^+\ell^-(\ell=\mu\; \text{or}\; \tau), \; \ell\bar{\ell}\gamma(\ell=\mu\; \text{or}\; \tau\; \text{or}\; \nu_{\mu/\tau})

      the sensitivity region of lepton number violation signal from heavy neutral lepton pair at the muon collider is smaller due to lower tagging efficiency of N.

      We then fix the gauge coupling g'=0.6 and explore the sensitivity region on the m_N-m_{Z'} plane. With 1 fb ^{-1} data, a large parameter space near the resonance region with 2 TeV \lesssim m_{Z'}\lesssim4 TeV can be probed. The most sensitive heavy neutral lepton mass is around 500 GeV. This is because a lighter m_N leads to smaller acceptance efficiency, while a larger m_N has a smaller production cross section. With an integrated luminosity of 1000 fb ^{-1} , the region with m_{Z'}\lesssim 13 TeV and 150 GeV \lesssim m_N<1500 GeV is within the reach of the 3 TeV muon collider.

    • B.   With associated photon

    • The full production process of the same-sign dimuon signature with associated photon is

      \begin{aligned}[b] \mu^+\mu^-\to Z^{\prime (*)} \gamma \to N N \gamma\to \mu^\pm W^\mp +\mu^\pm W^\mp+\gamma, \end{aligned}

      (30)

      with the hadronic decays of W. The standard model backgrounds come from processes such as

      \begin{aligned}[b] \mu^+\mu^-\to \mu^+\mu^- W^+W^-\gamma , W^\pm W^\pm W^\mp \gamma \mu^\mp \nu . \end{aligned}

      (31)

      During the simulation, the pre-selection cuts on associated photon in Eq. (21) are also applied. Based on the cut-flow in Table 1, we expect that the cross section of the process \mu^+\mu^-W^+ W^-\gamma\gamma/Z with one additional \gamma/Z gauge boson is approximately an order of magnitude smaller than that of the process \mu^+\mu^- W^+W^-\gamma . Therefore, \sigma(\mu^+\mu^-W^+ W^-\gamma\gamma/Z) is approximately 10^{-5} fb after all cuts, which will not be considered in the analysis of the \mu^\pm\mu^\pm JJ\gamma signal.

      The cross sections of the background processes with associated photons are approximately one order of magnitude smaller than those without associated photons. If we apply exactly the same selection cuts as the previous \mu^\pm\mu^\pm JJ signal, the total cross section of background is expected to be of the order of 10^{-4} fb, which will have approximately 0.1 background events, even with a total of 1000 fb ^{-1} data. Therefore, we can loosen the selection cut to keep more signal events. In the analysis of the \mu^\pm\mu^\pm JJ\gamma signal, we consider that at least one fat-jet is detected in the final states

      \begin{aligned}[b] N_J\geqslant 1, \end{aligned}

      (32)

      while we further require one detected photon in this signal.

      In Fig. 7, we show the distributions of some variables for the \mu^\pm\mu^\pm JJ\gamma signal and backgrounds after applying the pre-selection cuts. The benchmark point for the \mu^\pm\mu^\pm JJ\gamma signal is the same as for the previous \mu^\pm\mu^\pm JJ signal. With relatively similar distributions of P_T(\mu^\pm) , P_T(J) , m_J , and m_{\mu J} , we then apply the same selection cuts for these variables as the \mu^\pm\mu^\pm JJ signal. The cut flows for the \mu^\pm\mu^\pm JJ\gamma signature and backgrounds are summarized in Table 2.

      Figure 7.  (color online) Normalized distribution of transverse momentum of muon P_T(\mu^\pm) (upper-left panel), transverse momentum of fat-jet P_T(J) (upper-right panel), fat-jet mass m_J (middle-left panel), and invariant mass of muon and fat-jet m_{\mu J} (middle-right panel). Energy of photon E_\gamma (lower-left panel) and invariant mass of reconstructed heavy neutral lepton pair m_{NN} (lower-right panel) for the \mu^\pm\mu^\pm JJ \gamma signature and corresponding backgrounds.

      σ/fb \mu^\pm\mu^\pm W^\mp W^\mp \gamma \mu^+\mu^- W^+W^- \gamma W^\pm W^\pm W^\mp\gamma \mu^\mp \nu
      Pre-selection 24.91 1.15 0.14
      N_{\gamma}=1,N_{\mu^\pm}=2 13.28 1.1\times10^{-3} 7.4\times10^{-3}
      N_J\geq1 13.25 1.0\times10^{-3} 7.2\times10^{-3}
      50 GeV \leqslant m_J\leqslant 100 GeV 9.61 5.7\times10^{-4} 4.5\times10^{-3}
      0.8 m_N<m_{\mu J}<1.2 m_N 8.86 2.6\times10^{-4} 1.2\times10^{-3}
      Significance 369.0 Total Background 1.5\times10^{-3}

      Table 2.  Cut flow table for the \mu^\pm\mu^\pm JJ\gamma signal at the 3 TeV muon collider and corresponding backgrounds. The significance is calculated by Eq. (29), assuming an integrated luminosity of 1000 fb ^{-1} .

      Although with one additional photon, the cross section of the \mu^\pm\mu^\pm JJ\gamma signal is of the same order as the \mu^\pm\mu^\pm JJ signal. After all selection cuts, the cross section of the \mu^\pm\mu^\pm JJ\gamma process is 8.86 fb. With a loose cut on the fat-jet number, the total cross section of the background is approximately 1.5\times10^{-3} fb after all cuts. With an integrated luminosity of 1000 fb ^{-1} , the significance of the \mu^\pm\mu^\pm JJ\gamma signal could be approximately 369 for the benchmark point. Meanwhile, 0.19 fb ^{-1} data is sufficient to discover the benchmark point at the 5\sigma level.

      Besides the invariant mass of dilepton m_{\ell^+\ell^-}(\ell=\mu,\tau) , there are two more pathways to confirm the on-shell production of gauge boson Z' in the \mu^\pm \mu^\pm JJ\gamma signature. For the two-body process \mu^+\mu^-\to Z' \gamma , the photon energy is connected to the gauge boson mass as [44]

      \begin{aligned}[b] E_\gamma=\frac{s-m_{Z'}^2}{2\sqrt{s}}. \end{aligned}

      (33)

      For the benchmark point with m_{Z'}=2500 GeV at the 3 TeV muon collider, the peak value of E_\gamma is approximately 458 GeV. The sharp bump above the background is clearly shown in the lower-left panel of Fig. 7. Meanwhile, we can also measure m_{Z'} through the invariant mass of two reconstructed heavy neutral leptons m_{NN} , which also has a resonance near m_{Z'} . From the distribution of m_{NN} in the lower-right panel of Fig. 7, the background processes \mu^+\mu^-W^+W^-\gamma and W^\pm W^\pm W^\mp \gamma \mu^\mp \nu have peaks around 3 TeV and 1.5 TeV, respectively. Besides, with less than two background events after all cuts, the backgrounds have little impact on the distribution of m_{NN} for the signal.

      The sensitivity of the \mu^\pm \mu^\pm JJ\gamma signature at the 3 TeV muon collider is explored with the cuts in Table 2, and Fig. 8 depicts the results. The upper-left panel shows the sensitivity region with fixed mass relation m_N=m_{Z'}/3 , so the decay channel Z'\to NN is always allowed. In this way, we might probe m_{Z'} in the range of [300, 4300] GeV. Compared with the \mu^\pm \mu^\pm JJ signature, we find that this signal is more promising approximately in the region of [400, 3000] GeV. For example, with m_{Z'}=2000 GeV, the \mu^\pm \mu^\pm JJ\gamma signature could probe g'\gtrsim0.015 , while the \mu^\pm \mu^\pm JJ signature could only reach g'\sim0.1 . Near m_{Z'}\sim \sqrt{s}=3 TeV, we can even reveal g'\gtrsim0.005 . When m_{Z'} is larger than the collision energy \sqrt{s} , the associated photon with the off-shell contribution of Z' makes the \mu^\pm \mu^\pm JJ\gamma signature less detectable than the \mu^\pm \mu^\pm JJ signal.

      Figure 8.  (color online) Same as Figure 6 but for the sensitivity of the \mu^+\mu^-JJ\gamma signal.

      The upper-right panel of Fig. 8 shows the sensitivity region with fixed m_N=1000 GeV. When m_{Z'}\leqslant 2 m_N , the decay channel Z'\to NN is not allowed, so production of NN\gamma is through the off-shell Z' . With nearly a constant cross section, the \mu^\pm \mu^\pm JJ\gamma signature could test g'\gtrsim0.23 for m_{Z'}\leqslant 2000 GeV, which is clearly less promising than the \mu^\pm \mu^\pm JJ signature. However, once the decay mode Z'\to NN is open, the cross section of NN\gamma is enhanced, so the \mu^\pm \mu^\pm JJ\gamma signal becomes quite promising. According to our simulation, we find that this signature could probe g'\gtrsim0.11 with only 1 fb ^{-1} data when 2000 GeV \lesssim m_{Z'}\lesssim\sqrt{s} .

      The sensitivity region in m_N-m_{Z'} with fixed g'=0.6 is shown in the lower panel of Fig. 8. With 1 fb ^{-1} data, a large portion of the parameter space with 2m_N\lesssim m_{Z'}\lesssim \sqrt{s} could be tested. With a similar reason as the \mu^\pm \mu^\pm JJ signature, the most sensitive m_N of the \mu^\pm \mu^\pm JJ\gamma signature is also approximately 500 GeV. Outside the above resonance Z' region, the significance of the \mu^\pm \mu^\pm JJ\gamma signature decreases due to the associated photon. Although less promising than the \mu^\pm \mu^\pm JJ signature, the \mu^\pm \mu^\pm JJ\gamma signal could probe the region with m_{Z'}\lesssim8.3 TeV and 120 GeV \lesssim m_N\lesssim1470 GeV at the 3 TeV muon collider with an integrated luminosity of 1000 fb ^{-1} .

    V.   CONCLUSION
    • Heavy neutral leptons N are introduced in the type-I seesaw mechanism to generate tiny neutrino masses. While electroweak scale N is extensively studied at colliders, a relatively large mixing parameter V_{\ell N} is usually required. In this paper, we considered the gauged U(1)_{L_\mu-L_\tau} extension of the type-I seesaw mechanism, where three heavy neutral leptons N_e,N_\nu,N_\tau with U(1)_{L_\mu-L_\tau} charge (0,1,-1) were introduced. Charged under the new gauged U(1)_{L_\mu-L_\tau} symmetry, there are new production channels of the heavy neutral lepton at colliders. We then investigated the lepton number violation signatures of heavy neutral lepton N in the gauged U(1)_{L_\mu-L_\tau} at the 3 TeV muon collider.

      Mediated by the new gauge boson Z' , the heavy neutral lepton N can be pair produced at the muon collider via the processes \mu^+\mu^-\to Z^{\prime *}\to NN and \mu^+\mu^-\to Z^{\prime (*)}\gamma\to NN \gamma. Cross sections of these processes are determined by the new gauge coupling g' , gauge boson mass m_{Z'} , and heavy neutral lepton mass m_N , which are independent of the mixing parameter V_{\ell N} . While both processes have a maximum cross section at m_{Z'}\simeq \sqrt{s} , the former is more promising when m_{Z'}>\sqrt{s} . For lighter new gauge bosons satisfying 2m_N<m_{Z'}<\sqrt{s} , Z' is produced on-shell with an associated photon, and the decay mode Z'\to NN is also allowed. In this way, the cross section of \mu^+\mu^-\to NN\gamma can be enhanced.

      Cascade decays of heavy neutral leptons can raise various interesting signatures. Provided the Majorana nature of heavy neutral leptons in the seesaw model, we focus on the lepton number violation signatures in this study. For illustration, we further assume that the heavy neutral lepton N preferentially couples to muons via mixing with light neutrinos. Thus, the dominant decay mode of heavy neutral leptons is N\to \mu^\pm W^\mp . The hadronic decays of W bosons are considered to reconstruct m_N , which are treated as one fat-jet J. The explicit signatures are \mu^+\mu^-\to NN\to \mu^\pm\mu^\pm JJ and \mu^+\mu^-\to NN\gamma\to \mu^\pm\mu^\pm JJ\gamma .

      With relatively clean backgrounds, a cut-based analysis was then performed, which indicates that a large part of the viable parameter space is within the reach of the 3 TeV muon collider. For example, \mu^\pm\mu^\pm JJ could probe m_{Z'} in the range of [300, 4500] GeV with fixed mass relation m_{N}=m_{Z'}/3 and an integrated luminosity of 1000 fb ^{-1} . Around the resonance region m_{Z'}\sim\sqrt{s} , the gauge coupling g' could be down to approximately 0.025. We can probe g'\gtrsim0.13 with m_N=1000 GeV and light Z' . Otherwise, this signal could probe m_{Z'}\lesssim 13 TeV with g'=0.6 . Meanwhile, if the gauge boson mass satisfies 2 m_N<m_{Z'}<\sqrt{s} , the \mu^\pm\mu^\pm JJ\gamma signature would be more promising than the \mu^\pm\mu^\pm JJ signature. It is also notable that around the resonance region m_{Z'}\sim\sqrt{s} , both signatures could probe a sizable parameter space even with only 1 fb ^{-1} data for g'=0.6 .

Reference (74)

目录

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return