Role of Quantum Fluctuations in Ba$_3$CoSb$_2$O$_9$ Revealed by Excitations within the 1/3-Magnetization Plateau

2.8 MB·

by  Y. Kamiya

NOTICE OF COPYRIGHT This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-
00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the
article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-
widelicensetopublishorreproducethepublishedformofthismanuscript,orallowotherstodoso,forUnitedStatesGovernment
purposes. TheDepartmentofEnergywillprovidepublicaccesstotheseresultsoffederallysponsoredresearchinaccordance
withtheDOEPublicAccessPlan(http://energy.gov/downloads/doe-public-access-plan).
7
1
0
2

r
a
M

1
2

]
l
e
-
r
t
s
.
t
a
m
-
d
n
o
c
[

2
v
1
7
9
7
0
.
1
0
7
1
:
v
i
X
r
a
RoleofQuantumFluctuationsinBa CoSb O RevealedbyExcitationswithinthe
3 2 9
1/3-MagnetizationPlateau
Y. Kamiya,1,∗ L. Ge,2 Tao Hong,3 Y. Qiu,4 D. L. Quintero-Castro,5 H. B. Cao,3
M. Matsuda,3 C. D. Batista,6,7 M. Mourigal,2 H. D. Zhou,6,8 and J. Ma9,10,6,†
1Condensed Matter Theory Laboratory, RIKEN, Wako, Saitama 351-0198, Japan
2School of Physics, Georgia Institute of Technology, Atlanta, GA 30332, USA
3QuantumCondensedMatterDivision,OakRidgeNationalLaboratory,OakRidge,Tennessee37831,USA
4NISTCentreforNeutronResearch,NationalInstituteofStandardsandTechnology,Gaithersburg,MD20899,USA
5Helmholtz-Zentrum Berlin fu¨r Materialien und Energie, D-14109 Berlin, Germany
6DepartmentofPhysicsandAstronomy, UniversityofTennessee, Knoxville, Tennessee37996, USA
7Quantum Condensed Matter Division and Shull-Wollan Center,
Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA
8NationalHighMagneticFieldLaboratory, FloridaStateUniversity, Tallahassee, FL32310, USA
9Key Laboratory of Artificial Structures and Quantum Control,
DepartmentofPhysicsandAstronomy, ShanghaiJiaoTongUniversity, Shanghai200240, China
10CollaborativeInnovationCenterofAdvancedMicrostructures, Nanjing, Jiangsu210093, China
(Dated:March22,2017)
A recent experiment revealed that the spin-1/2 triangular-lattice Heisenberg antiferromagnet Ba CoSb O
3 2 9
is a magnetically ordered material with an anomalous excitation spectrum, comprising a high-intensity ex-
citation continuum and short-lived magnons, which cannot be explained with spin wave theory [Ma et al.,
Phys.Rev.Lett.116,087201(2016)]. Weuseneutronscatteringmeasurementsinhighmagneticfieldtomap
themagneticexcitationsofBa CoSb O inits1/3-magnetizationplateau,whichenablesaninvestigationonthe
3 2 9
preciseroleofquantumfluctuations. Themeasuredexcitationspectrumoftheup-up-downphaseinducedbya
magneticfieldof(cid:38)9.8Tisquantitativelyexplainedbynonlinearspinwavetheory,confirmingthevalidityof
thespinHamiltonianpreviouslyobtainedforthismaterial.Mostimportantly,ournewfindingsindicatethatthe
anomalouszero-fieldspindynamicshasanintrinsicquantummechanicalorigin.
Highly frustrated quantum magnets [1] currently offer the theMajoranafermionsoftheKitaevQSL[14].
main venue in the search for quantum spin liquids (QSLs), Given the abundance of magnetically ordered frustrated
alongwithavarietyofunusualorderedphases[2]. QSLsare magnets which are potentially close to their QMP, it is rele-
novelstatesofmatterof“disordered”nature,whichhostfrac- vanttolookforfingerprintsofthenearbyliquidstateintheir
tional excitations, such as spinons, and fluctuations of emer- magneticexcitationspectrum. Arecentreport[22]shedlight
gent gauge degrees of freedom, such as visons [3, 4]. While on anomalous observations in an inelastic NS experiment on
itisestablishedthatspinonsexistinquantumspinchains[5], singlecrystalsoftheequilateraltriangular-latticeHeisenberg
their counterparts in higher dimensions are much more elu- antiferromagnet(TLHAF)Ba CoSb O [23–30]. Thisquan-
3 2 9
sive. Besidesthefundamentalinterestinmeasuringtheseex- tum magnet comprises triangular layers of Co2+ ions carry-
oticstatesofmatter,identifyingQSLsinrealmaterialsisrel- ing effective spin-1/2 magnetic moments. Deviations from a
evantfortopologicalquantuminformationprocessing[6]. pure 2D Heisenberg model due to anisotropies and 3D cou-
WhileidentifyingagenuineQSLinarealmaterialshould plings are rather small [22, 26] and Dzyaloshinskii-Moriya
await further investigations, there are multiple reports of interactions are essentially precluded because of the highly
magnetically ordered materials with anomalous spin dynam- symmetric crystal structure (space group P6 /mmc). Thus,
3
ics[7,8]. Aconceivableworkinghypothesisisthatthesema- Ba CoSb O isanalmostidealrealizationofthequantumTL-
3 2 9
terials are not far from the “quantum melting point” (QMP) HAF,aparadigmaticmodelforfrustratedmagnets[31].
that signals the transition into a QSL phase. The proxim- The ground state of Ba CoSb O is magnetically ordered
3 2 9
ity to a QMP implies a potentially strong influence of the into the 120◦ state [32, 33]. However, its spin dynamics
nearby QSL phases on the magnetic excitation spectrum [7– still retains various anomalous features [22]. The high fre-
12]. For example, magnons are expected to become weakly quency ((cid:38) 2meV) excitation continuum has a large spectral
bound pairs of spinons near the QMP [9, 13]. This phe- weight, almost comparable to that of the single-magnon ex-
nomenon is clearly beyond the reach of large-S theory be- citations. Furthermore, magnon spectral lines are broadened
causeofthefractionalizednatureofspinons[8]. Theseideas throughouttheentireBrillouinzone. Magnondecays,nonlin-
are realized in weakly coupled spin chains [5, 9] and, more eareffectsinspin-wave(SW)theorystemmingfromthenon-
recently,intheilluminatingcontextoftheKitaevmodel[14] collinear magnetic structure [34–38], are natural contenders
anditsmultipleextensions[15–18]. Forinstance,recentRa- to explain these qualitative features in the dynamical struc-
manscattering[19]andinelasticneutronscattering(NS)mea- ture factor S(q,ω). Surprisingly, neither of the anomalous
surements [20, 21] on α-RuCl suggested that this magneti- experimentalobservationscanbeexplainedbyaconventional
3
callyorderedmaterialhostsfermionicexcitations,resembling 1/S expansion [22]. It was found that the small, but non-
2
negligible, easy-plane anisotropy renders the decay process diffractionat(1/3,1/3,1)andthelackofintensityat(1/3,1/3,
fromonemagnontotwo-magnoncontinuum[34–38]ineffec- 0)forthe120◦stateatH =0.
tive (the kinematic conditions are not satisfied), because one The dynamical properties of Ba CoSb O in the UUD
3 2 9
ofthetwoNambu-GoldstonemodesoftheisotropicTLHAF phase were investigated with the Multi Axis Crystal Spec-
is gapped out by the anisotropy. In addition, SW theory sig- trometer(MACS)[50]atNISTCenterforNeutronResearch
nificantlyoverestimatesthein-planemagnonbandwidth[22]. (NCNR),NIST,andthecoldneutrontriple-axisspectrometer
Given that impurity effects have been carefully ex- (V2-FLEXX)[51]atHelmholtz-ZentrumBerlin(HZB).The
cluded[22],theproximitytoaQMPisonepossibleexplana- finalenergieswerefixedat3.5meVand5meVontheMACS
tionforthefailureofSWtheory.Inthemeantime,theordered and3.0meVonV2-FLEXX.Figures2(a)–2(c)showthescat-
momentofm =1.53(2)µ [39]isaround79(1)%ofthefull tering intensity I(q,ω) ≡ k/k (d2σ/dΩdE ) along high-
Q B i f f
momentm =1.93µ perCo2+[26],whichisratherinclose symmetrydirectionsinthereciprocalspaceatµ H = 10.5T
full B 0
agreement with SW theory applied to this material [22] pre- and T = 0.5K; this temperature is low enough compared to
dictingm /m =73%. Thus,thisratioisnotsmallenough thetransitiontemperatureT ≈5KtotheUUDphaseinthis
Q full N
toconcludetheproximitytoaQMP[40],whileprobablynot fieldregime[29]. Weobservedtwogappedmodesinthein-
largeenoughtosafelyvalidatethe1/S expansiontodescribe plane dispersion along with a third one, for which it is not
thespindynamicsoftheTLHAF.Itishencecrucialtovalidate clearwhetherthegapisnonzeroornotduetothelimitedin-
thespinHamiltonianofBa CoSb O inanindependentfash- strumentalresolution[Fig.2(a)];ifthismodeisalsogapped,
3 2 9
ion. Itisalsodesiredtoexcludeotherextrinsicexperimental theenergygapshouldbe(cid:46) 0.3meV.Thedispersionisnearly
effectssuchasmagnon-phononcoupling[41]. flatintheout-of-planedirection[Figs.2(b)and2(c)].
InthisLetter,wedemonstratethatthedeviationsfromSW To compare against the above inelastic NS results in the
theoryobservedforBa CoSb O atzerofield[22]areanin- UUD phase, we consider the same spin Hamiltonian used in
3 2 9
trinsic effect induced by quantum fluctuations. To this end, Ref. 22 for H parallel to the a-axis. This is a spin-1/2 XXZ
westudythespindynamicsofBa CoSb O inhighmagnetic modelonthestackedtriangularlatticeshowninFig.1(b),with
3 2 9
fields. Thehighfieldssuppressquantumfluctuationsandcre- in-planenearest-neighbor(NN)exchangeJandinter-layerex-
atefavorableconditionsforacomparisonagainstSWtheory. changeJ :
c
While SW theory becomes exact for single-magnon excita- (cid:88)(cid:16) (cid:17)
tionsabovethesaturationfieldH ,itistoohighforcontem- H = J SxSx+SySy+∆SzSz
sat i j i j i j
porary NS facilities (µ H = 31.9T for H (cid:107) ab and 32.8T (cid:104)ij(cid:105)
0 sat
forH (cid:107) c[26]). However,itturnsouttobesufficienttomea- +J (cid:88)(cid:18)SxSx +SySy +∆SzSz (cid:19)−H˜(cid:88)Sx, (1)
sure the spin dynamics within the 1/3 magnetization plateau c i i+cˆ i i+cˆ i i+cˆ i
i 2 2 2 i
withtheup-up-down(UUD)spinstructure[24,26,28],which
extends between µ H = 9.7T and µ H = 15T for H (cid:107) whereH˜ =g µ Histhereducedmagneticfieldandg isthe
0 c1 0 c2 ⊥ B ⊥
ab [29]. In fact, although this phase itself is stabilized by in-plane component of the g-tensor. S is the effective spin-
i
quantumfluctuations[28,42–44], itsgappedspectrumisex- 1/2 operator describing the ground state Kramers doublet of
pected to be well described by SW theory because of the Co2+ and (cid:104)ij(cid:105) denotes the in-plane NN sites. The displace-
smaller reduction of the ordered moment. Specifically, we ment vector along the c-axis is cˆ/2 because the unit cell of
will demonstrate that the spectrum within the plateau can be Ba CoSb O containstwoverticallyalignedCo2+.
3 2 9
reproduced by the spin Hamiltonian in Ref. 22 with almost
identical coupling constants. Our work thus establishes that
tsoethhilrcueedceTaairndnLindaoHgtmimAntagooFlostamtuhselQaitkzsMeeeprlriiPyoan.l-,cIdfinwayeulnhtdshaielemsedpmisecbucesyctahirntnuhptmietlmhapetoeerf,qoatuuxBhaiiaimnss3tCkiiutsnyomoStohwbUfen2UOifittsoDr9semtipxsaahitisgatntenstameerlitsnpiinoc-t (aIntensity (arb. unit))1284000 (1/3, 1/3, 0) 100 TT (1/3, 1/3, 1) 100 TT 1842000Intensity (arb. unit)
inCs CuBr [45–49]. 25 26 27 28 30 31 32 33
2 4 2θ (deg.) 2θ (deg.)
AsinglecrystalofBa CoSb O wasgrownbythefloating-
3 2 9 (b) J (c)
zone technique and oriented in the (h,h,l) scattering plane.
Theneutrondiffractiondataundermagneticfieldsappliedin
the [1,-1,0] direction was obtained by using the CG-4C cold Jc
triple-axisspectrometer(CTAX)withtheneutronenergyfixed
at5.0meVatHighFluxIsotopeReactor(HFIR),OakRidge
NationalLaboratory. ThemagneticBraggpeaksat(1/3, 1/3,
1) and (1/3, 1/3, 0) were measured at 1.5K. As depicted in FIG.1. (a)NSdiffractiondataatµ H =0Tand10TatT =1.5K.
Fig.1(a), thestrongdiffractionatboth(1/3, 1/3, 1)and(1/3, Errorbarsinallfiguresrepresentone0standarddeviation.(b)Stacked
1/3,0)confirmsstabilizationoftheUUDstateatµ H ≥9.8T, triangular lattice of Co2+ carrying effective spin-1/2 magnetic mo-
0
which can be contrasted with the combination of the strong ments.(c)SpinstructureoftheUUDstate.
3
(a) (b) (c) (g) (1/3, 1/3, -2)
4
Exp.
I
V)3 Itot 4
e ||
(m2 4 1
E 2 3
2
1 3 (qI
,
0 2 ) (a 0
(d) (e) (f) rb (h) (1/2, 1/2, -2)
34 1 . unit) EItoxtp. 4
V) I
e 1 ||
m 0
2
E ( & 2
2 3
1
0 0
0 1/6 1/3 1/2 2/3 -2 -3/2 -1-2 -3/2 -1 0 1 2 3 4
(h, h, -2) (1/2, 1/2, l) (1/3, 1/3, l) E (meV)
FIG. 2. (a)–(c) Experimental scattering intensity at µ H = 10.5T and T = 0.5K with momentum transfer (a) in the triangular plane
0
q = (h,h,−2),andalongthecaxiswith(b)q = (1/2,1/2,l)and(c)q = (1/3,1/3,l). TheenergyoftheoutgoingneutronsisE = 5meV
f
(3meV)above(below)thedashedline. (d)–(f)Calculatedtransversepartofthescatteringintensity,I (q,ω),obtainedbythenonlinearSW
⊥
theoryandevaluatedalongthesamemomentumcutsasin(a)–(c)forparametersJ=1.77meV,∆=0.85,J /J=0.05,andg =4.0.Thesolid
c ⊥
linesshowtheSWpoles. (g)and(h)Energydependenceofthescatteringintensity,I (q,ω)(solidline),comparedtothedataformomenta
tot
(g)q = (1/3,1/3,−2)and(h)q = (1/2,1/2,−2). Thelongitudinalcontributiontothescatteringintensity,I(q,ω),isplottedseparatelyasa
(cid:107)
shadedarea.
As first predicted by Chubukov and Golosov for large ble over a finite volume of the phase diagram. The resulting
S [42] and confirmed later on by numerical studies for S = nonlinearSWHamiltonian,H ,istheBogoliubovHamil-
NLSW
1/2 [43, 44, 52–54], the UUD state is expected to be the tonian for the six-sublattice magnon fields, with the magnon
ground state of this model (1) for ∆ (cid:46) 1 and J /J (cid:28) 1 interactionsincludedatthemean-fieldlevel. TheUUDphase
c
over a finite window around H˜ = 3JS, i.e., 1/3 of the sat- becomeslocallyunstablewhenthegapcloses.
uration field in the 2D limit (J = 0). However, in deriv- Figures 2(d)–2(h) show the calculated scattering intensity
c
ingthecorrespondingSWspectrum,theusual1/S expansion I(q,ω) ≡ (γr /2)2|F(q)|2(cid:80) (1 − qˆαqˆα)g2Sαα(q,ω) where
0 α α
machineryisunapplicablebecausetheUUDstateistheclas- F(q)denotesthemagneticformfactor,(γr /2)2isaconstant,
0
sicalgroundstateonlyonthespecialline J = 0, H˜ = 3JS, qˆα = qα/|q|, and Sαα(q,ω) are the diagonal components of
c
and ∆ ≤ 1 [55]. Thus, the magnons of the UUD phase de- thedynamicalstructurefactorevaluatedatµ H = 10.5Tand
0
rived in the conventional way become unstable away from T = 0. ThisresultiscomparedagainstourinelasticNSmea-
this line. Fortunately, this instability can be avoided by us- surements along different directions of the Brillouin zone in
ing a modified approach that was applied to a similar prob- Figs.2(a)–2(c). DefiningtheUUDorderasalongthe xaxis,
lemforthedistortedtriangularlattice[56]. Thebasicideais transverse spin fluctuations related to single-magnon excita-
thatmagnon-magnoninteractionsshouldenlargethestability tions appear in Syy and Szz, while longitudinal spin fluctua-
region for the UUD phase around the classical line. To out- tions corresponding to the two-magnon continuum appear in
lineouranalysis(seeSupplementalMaterialfordetails[55]), S , denoting the inelastic part of Sxx [55]. Accordingly, the
(cid:107)
we first expand H in 1/S on the special line by assuming scatteringintensityI(q,ω)canbeseparatedintotransverseI
⊥
thesix-sublatticeUUDspinconfigurationshowninFig.1(c). andlongitudinalI contributions. Tocomparewithourexper-
(cid:107)
Thepurposeofthispreliminarycalculationistoevaluatethe iments, the theoretical intensity is convoluted with empirical
leading-order magnon self-energies arising from decoupling instrumentalenergyresolutionandmomentumbinningeffects
thequartictermsinthe1/S expansion. Infact,thelinearSW (onlytheenergyresolutionisconvolutedforI ).
(cid:107)
Hamiltonian at this stage has two relativistic modes, both of Three bands of scattering associated with six distinct dis-
whichbecomegappedafterincorporatingtheself-energycon- persion branches are expected: the lowest band is well iso-
tributionsevaluatedat J = 0andH˜ = 3JS foragivenvalue lated, while there is a significant hybridization between the
c
of ∆ ≤ 1 [55]. The fully gapped spectrum is robust against other two around q = (1/2,1/2,l) with arbitrary l and re-
a finite J > 0 and H˜ (cid:44) 3JS, implying that the UUD is sta- lated momenta [see Figs. 2(a) and 2(b)]. This implies that
c
4
showthetheconstant-qscansat(1/3, 1/3, 1)atµ H = 10.5,
(a) (b) 0
200 12,and13.5TintheUUDphase,fromwhichweextractthe
13.5 T
unit) 150 V) 3 ω3 3 bHr-adnecpheensd,ennecgeleoctfintghethSeWsplpitetainkgs dωu1e–ωto3Jo,fwthheicthhriesebemyaojnodr
y (arb. 100 12.0 T gy (me 2 12 oduepreinnsdternucmeenagtarlereessowlueltliown.itAhctchoerdnionnglitnoeFacrigS.W3(bt)h,ethoeryir.fiWelde
ntensit 50 10.5 T Ener 1 ω2 0 10 11 12 13 10 11 12 13 fi(dnedcrtheaasteωs)3ifinrcsrte,aasnedsltihneenardlyecinreHas,ewsh(einrecaresaωse1s()ω, 2re)flineccrteinagseas
I ω
1 level-crossinginducedbyH.
ω ω ω
0 1 2 3 0 Thus,thespinexcitationspectrumintheUUDphaseisvery
0 1 2 3 10 11 12 13 14 15
well reproduced by the nonlinear SW calculation. This con-
Energy (meV) Field (T)
firms the validity of the minimal Hamiltonian [Eq. (1)]. SW
FIG.3. (a)Constant-qscansat(1/3,1/3,1)fortheselectedvaluesof theory works well for describing the UUD state because the
themagneticfieldintheUUDphaseat0.1K. Thesolid-linesshow collinearorderingreducestheamplitudeofthequantumfluc-
theresultsofthefittingtoGaussianfunctions. Thearrowsindicate tuations.Indeed,thecalculatedreductionofthemagneticmo-
theSWpeaksω1–ω3. (b)ComparisonofthefittedSWfrequencies mentineachsublatticeµ,δSµ =S −|(cid:104)Sj(cid:105)|with j∈µ,isvery
ω1 (circles),ω2 (triangles),andω3 (squares)againstthesimulation small: δS = δS = 0.0826, δS = δS = 0.0767, and
resultsofpoles(lines)basedonthenonlinearSWtheoryforg = A A¯ B C¯
⊥ δS =δS =0.145fortheUUDstateillustratedinFig.1(c).
4.0.Theinsetshowsthecasesforg =3.84[26]andg =4.45[30]. C B¯
⊥ ⊥
This is consistent with the very weak intensity of the two-
magnoncontinuum[Figs.2(g)and2(h)].
More importantly, the estimated Hamiltonian parameters
µ H = 10.5T corresponds to H˜ (cid:46) 3JS [55]. The best fit almost coincide with those in Ref. 22. This observation is
0
withtheexperimentsisobtainedfor J =1.77meV,∆=0.85, particularlyimportantbecauseitconfirmstheconjecturetrig-
and J /J = 0.05. In addition, we assume g = 4.0, which geredbythezero-fieldinelasticNSexperiments[22]: thedis-
c ⊥
is slightly larger than g = 3.84 [26] measured with elec- crepancybetweenthemeasuredS(q,ω)andthesemiclassical
⊥
tronicspinresonance(ESR)atT =20Kwhereasitissmaller theory at H = 0 is not rooted in an inadequate spin Hamil-
than the recent claim g = 4.45 [30] based on susceptibil- tonian. Moreover,itisnoteworthythattheUUDstatebreaks
⊥
ity measurements [57]. This is to improve the fit to the field bothtranslationalandrotationalsymmetriesoftheunderlying
dependence of the SW modes, as discussed below. Remark- lattice [see Fig. 1(c)]. Unlike the zero field case, the bonds
ably, the estimated coupling constants are not very different become inequivalent in the UUD state: two thirds of the in-
fromthosebasedontheinelasticNSmeasurementsatH = 0 plane NN spin-pairs are aligned antiferromagnetically, while
probingexcitationsofthe120◦ state, J = 1.7meV,∆ = 0.89, theotherthirdareferromagneticallyaligned.Hence,theUUD
and J /J = 0.05 [22]. However, unlike the H = 0 case, the stateismoresusceptibletomagnon-phononcouplingthanthe
c
agreementbetweentheoryandexperimentisexcellentforthe zero-field 120◦ state. The fact that Eq. (1) reproduces the
UUD phase. First of all, the calculated bandwidths for the measuredexcitationspectrumoftheUUDstatesuggeststhat
majorthreebandsagreeverywellwiththeobservedones,in themagnon-phononcouplingissmallinthismaterialandthis
contrast to the nonlinear SW theory at H = 0 which overes- mustholdtruedowntoH =0.
timatestheoverallbandwidthby≈35%[22]. Also,thechar- To conclude, our high-field NS experiments mapped
acteristicsofthein-planedispersion[Fig.2(d)]andthenearly the excitations within the 1/3-magnetization plateau of
dispersionlessfeatureintheout-of-planedirections[Figs.2(e) Ba3CoSb2O9, amodelquantumtriangular-latticeHeisenberg
and2(f)]arewellreproduced. Furthermore,theobservedline antiferromagnet,andconfirmedthevalidityofthespinHamil-
broadeningisreproduced.Thebroadeningarisesfromthefact tonianconsideredinRef.22. Itwillbeinterestingtoinvesti-
that each of the three “bands” actually comprise two poles gatetheapplicabilityrangeofSWtheoryinamagneticfield
with comparable amplitudes, which are split by the rather for both H (cid:107) ab and H (cid:107) c [58]. Our study also excluded
smallinter-layercoupling J . Thecorrespondingpeaksover- magnon-phononinteractionasthepossiblesourceforthedis-
c
lapwitheachotherbecauseofthelimitedenergy-momentum crepancybetweenthezero-fieldinelasticNSexperimentsand
resolution. In more details, the linewidths of ω and ω in SWtheory,thuscallingforanewtheoreticalapproachcapable
2 3
Figs. 2(g) and 2(h) are slightly broader than the resolution. of describing the anomalous spin dynamics of magnetically
This broadening can be due to single-to-triple magnon de- ordered materials in the proximity of their quantum melting
cays[38],whicharekinematicallyallowed. point.
The lowest-energy branch [Fig. 2(a)] is a massive excita- Acknowledgments.—WethankH.Tanaka,A.Chernyshev,
tionwithasmallenergygap≈0.37meVatq = (1/3,1/3,1) and O. Starykh for valuable discussions. We are grateful for
that can be guessed from the experimental data. The small- the access to the neutron beam time at the neutron facilities
nessofthegapresultsfromtheproximitytothelowerquan- atNCNR,BER-2atHelmholtz-ZentrumBerlinandHFIRop-
tumcriticalpointatµ H = 9.7T[29],separatingtheUUD erated by ORNL. The research at HFIR at ORNL was spon-
0 c1
phasefromthelow-fieldcoplanarphase[28]. InFig.3(a),we soredbytheScientificUserFacilitiesDivision(T.H.,H.B.C.,
5
and M.M.), Office of Basic Energy Sciences, U.S. DOE. Lett.116,087201(2016).
This work utilized MACS supported in part by the National [23] Y.Doi,Y.Hinatsu, andK.Ohoyama,J.Phys.:Condens.Matter
Science Foundation under Agreement No. DMR-1508249. 16,8923(2004).
[24] Y. Shirata, H. Tanaka, A. Matsuo, and K. Kindo, Phys. Rev.
J.M. acknowledges the support of the Ministry of Science
Lett.108,057205(2012).
andTechnologyofChina(2016YFA0300500).Y.K.acknowl-
[25] H.D.Zhou,C.Xu,A.M.Hallas,H.J.Silverstein,C.R.Wiebe,
edges the financial support by JSPS Grants-in-Aid for Sci-
I.Umegaki,J.Q.Yan,T.P.Murphy,J.-H.Park,Y.Qiu,J.R.D.
entific Research under Grant No. JP16H02206. The work at Copley, J. S. Gardner, and Y. Takano, Phys. Rev. Lett. 109,
Georgia Tech was supported by ORAU’s Ralph E. Powe Ju- 267206(2012).
nior Faculty Enhancement Award (M.M.). H.D.Z. acknowl- [26] T.Susuki,N.Kurita,T.Tanaka,H.Nojiri,A.Matsuo,K.Kindo,
edges support from NSF-DMR-1350002 and from NHMFL andH.Tanaka,Phys.Rev.Lett.110,267201(2013).
[27] K. Naruse, T. Kawamata, M. Ohno, Y. Matsuoka, H. Sudo,
throughNSF-DMR-1157490andtheStateofFlorida.
H. Nagasawa, Y. Hagiya, T. Sasaki, and Y. Koike, J. Phys.:
Conf.Series568,042014(2014).
[28] G.Koutroulakis, T.Zhou, Y.Kamiya, J.D.Thompson, H.D.
Zhou,C.D.Batista, andS.E.Brown,Phys.Rev.B91,024410
(2015).
∗ [email protected] [29] G.Quirion,M.Lapointe-Major,M.Poirier,J.A.Quilliam,Z.L.
† [email protected] Dun, andH.D.Zhou,Phys.Rev.B92,014414(2015).
[1] C.Lacroix,P.Mendels, andF.Mila,eds.,IntroductiontoFrus- [30] A. Sera, Y. Kousaka, J. Akimitsu, M. Sera, T. Kawamata,
tratedMagnetism: Materials, Experiments, Theory(Springer- Y.Koike, andK.Inoue,Phys.Rev.B94,214408(2016).
Verlag,BerlinHeidelberg,2011). [31] P.Anderson,Mater.Res.Bull.8,153 (1973).
[2] O.A.Starykh,Rep.Prog.Phys.78,052502(2015). [32] L.Capriotti,A.E.Trumper, andS.Sorella,Phys.Rev.Lett.82,
[3] L.Balents,Nature464,199(2010). 3899(1999).
[4] L.SavaryandL.Balents,Rep.Prog.Phys.80,016502(2017). [33] S.R.WhiteandA.L.Chernyshev,Phys.Rev.Lett.99,127004
[5] M.Mourigal,M.Enderle,A.Klopperpieper,J.-S.Caux,A.Stu- (2007).
nault, andH.M.Ronnøw,Nat.Phys9,435(2013). [34] A.L.ChernyshevandM.E.Zhitomirsky,Phys.Rev.Lett.97,
[6] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and 207202(2006).
S.DasSarma,Rev.Mod.Phys.80,1083(2008). [35] O.A.Starykh,A.V.Chubukov, andA.G.Abanov,Phys.Rev.
[7] R.Coldea, D.A.Tennant, A.M.Tsvelik, andZ.Tylczynski, B74,180403(2006).
Phys.Rev.Lett.86,1335(2001). [36] A. L. Chernyshev and M. E. Zhitomirsky, Phys. Rev. B 79,
[8] B.DallaPiazza,M.Mourigal,N.B.Christensen,G.J.Nilsen, 144416(2009).
P.Tregenna-Piggott,T.G.Perring,M.Enderle,D.F.McMor- [37] M. Mourigal, W. T. Fuhrman, A. L. Chernyshev, and M. E.
row,D.A.Ivanov, andH.M.Ronnøw,Nat.Phys11,62(2015). Zhitomirsky,Phys.Rev.B88,094407(2013).
[9] M. Kohno, O. A. Starykh, and L. Balents, Nat. Phys 3, 790 [38] M.E.ZhitomirskyandA.L.Chernyshev,Rev.Mod.Phys.85,
(2007). 219(2013).
[10] Y.Kamiya,Y.Kato,J.Nasu, andY.Motome,Phys.Rev.B92, [39] TheorderedmomentisestimatedfromtheNSdiffractiondata
100403(2015). presentedinRef.22.Here,themagneticformfactoriscorrected
[11] X.-Y. Song, Y.-Z. You, and L. Balents, Phys. Rev. Lett. 117, withtheorbitalcontribution.
037209(2016). [40] Wealsonotethattheorderedmomentperseisnotaverysensi-
[12] M. Gohlke, R. Verresen, R. Moessner, and F. Pollmann, tivemeasureoftheproximitytoaQMP,asitcanchangerather
arXiv:1701.04678 (unpublished). rapidlynearaQMP.
[13] A. V. Chubukov and O. A. Starykh, Phys. Rev. B 52, 440 [41] K.Park,J.Oh,J.C.Leiner,J.Jeong,K.C.Rule,M.D.Le, and
(1995). J.-G.Park,Phys.Rev.B94,104421(2016).
[14] A.Kitaev,Ann.Phys.(N.Y.)321,2 (2006). [42] A.V.ChubukovandD.I.Golosov,J.Phys.: Condens.Matter
[15] G. Jackeli and G. Khaliullin, Phys. Rev. Lett. 102, 017205 3,69(1991).
(2009). [43] D.Yamamoto,G.Marmorini, andI.Danshita,Phys.Rev.Lett.
[16] J. Chaloupka, G. Jackeli, and G. Khaliullin, Phys. Rev. Lett. 112,127203(2014).
110,097204(2013). [44] D.Yamamoto,G.Marmorini, andI.Danshita,Phys.Rev.Lett.
[17] J.G.Rau,E.K.-H.Lee, andH.-Y.Kee,Phys.Rev.Lett.112, 114,027201(2015).
077204(2014). [45] T.Ono,H.Tanaka,H.ArugaKatori,F.Ishikawa,H.Mitamura,
[18] J. Nasu, Y. Kato, J. Yoshitake, Y. Kamiya, and Y. Motome, andT.Goto,Phys.Rev.B67,104431(2003).
arXiv:1610.07343 (unpublished). [46] T. Ono, H. Tanaka, O. Kolomiyets, H. Mitamura, T. Goto,
[19] J.Nasu,J.Knolle,D.L.Kovrizhin,Y.Motome, andR.Moess- K. Nakajima, A. Oosawa, Y. Koike, K. Kakurai, J. Klenke,
ner,Nat.Phys.12,912(2016). P. Smeibidle, and M. Meiner, J. Phys.: Condens. Matter 16,
[20] A. Banerjee, C. A. Bridges, J. Q. Yan, A. A. Aczel, L. Li, S773(2004).
M.B.Stone,G.E.Granroth,M.D.Lumsden,Y.Yiu,J.Knolle, [47] T.Ono,H.Tanaka,O.Kolomiyets,H.Mitamura,F.Ishikawa,
S.Bhattacharjee,D.L.Kovrizhin,R.Moessner,D.A.Tennant, T. Goto, K. Nakajima, A. Oosawa, Y. Koike, K. Kakurai,
D.G.Mandrus, andS.E.Nagler,Nat.Mater.15,733(2016). J.Klenke,P.Smeibidle,M.Meiner,R.Coldea,A.D.Tennant,
[21] A. Banerjee, J. Yan, J. Knolle, C. A. Bridges, M. B. Stone, andJ.Ollivier,Prog.Theor.Phys.Suppl.159,217(2005).
M.D.Lumsden,D.G.Mandrus,D.A.Tennant,R.Moessner, [48] H.Tsujii,C.R.Rotundu,T.Ono,H.Tanaka,B.Andraka,K.In-
andS.Nagler,arXiv:1609.00103 (unpublished). gersent, andY.Takano,Phys.Rev.B76,060406(2007).
[22] J.Ma,Y.Kamiya,T.Hong,H.B.Cao,G.Ehlers,W.Tian,C.D. [49] N. A. Fortune, S. T. Hannahs, Y. Yoshida, T. E. Sherline,
Batista, Z.L.Dun, H.D.Zhou, andM.Matsuda,Phys.Rev. T.Ono,H.Tanaka, andY.Takano,Phys.Rev.Lett.102,257201
6
(2009). isaclassicalgroundstate. Inparticular,atH˜ =3JS,theclas-
[50] J. A. Rodriguez, D. M. Adler, P. C. Brand, C. Broholm, sicalgroundstatemanifoldincludestheUUDstate. Toshow
J. C. Cook, C. Brocker, R. Hammond, Z. Huang, P. Hundert- thattheUUDstateisnottheclassicalgroundstateforJ >0,
mark, J.W.Lynn, N.C.Maliszewskyj, J.Moyer, J.Orndorff, we work in the subspace with fixed Mx = N−1 (cid:80) Ωc x =
D.Pierce,T.D.Pike,G.Scharfstein,S.A.Smee, andR.Vi- spin j j
1/3 (N is the total number of spins) and consider a one-
laseca,Meas.Sci.Technol.19,034023(2008). spin
parameterdeformationofthespinconfigurationparametrized
[51] M.Le,D.Quintero-Castro,R.Toft-Petersen,F.Groitl,M.Sk-
oulatos, K. Rule, and K. Habicht, Nucl. Instrum. Methods by 0 ≤ θ ≤ π, as illustrated in Fig. S1(b). As this transfor-
Phys.Res.Sect.A729,220 (2013). mation does not change Mx = 1/3, the first term in Eq. (S1)
[52] D. J. J. Farnell, R. Zinke, J. Schulenburg, and J. Richter, J. doesnotchangeunderthistransformation. Ontheotherhand,
Phys.:Condens.Matter21,406002(2009). we find that the classical energy for the inter-layer coupling
[53] T.SakaiandH.Nakano,Phys.Rev.B83,100405(2011).
changesas
[54] R. Chen, H. Ju, H.-C. Jiang, O. A. Starykh, and L. Balents,
Phys.Rev.B87,165123(2013). E (θ)=2J S2(cos2θ−2cosθ) (S3)
c c
[55] SeeSupplementalMaterial.
[56] J.Alicea,A.V.Chubukov, andO.A.Starykh,Phys.Rev.Lett. permagneticunitcell,whichisminimizedatθ=π/3forJ >
c
102,137201(2009). 0insteadoftheθ = 0valuecorrespondingtotheUUDstate.
[57] Thediscrepancyaboutg⊥ hastoberesolvedwitheitherhigh- Since the UUD state has Mx = 1/3 for H (cid:107) aˆ, this analysis
fieldmagnetizationorESRmeasurementsatT (cid:29) Jtoexclude
showsthatitisnottheclassicalgroundstateforJ >0.
effectsofshort-rangespincorrelations. c
[58] P. A. Maksimov, M. E. Zhitomirsky, and A. L. Chernyshev,
Phys.Rev.B94,140407(2016).
II. SPINWAVETHEORYFORTHEUP-UP-DOWNPHASE
INTHEQUASI-2DSYSTEM
—
A. NonlinearspinwaveHamiltonian
Asmentionedinthemaintext,itisexpectedthatmagnon-
SupplementalMaterial
magnon interactions can stabilize the UUD state, in spite of
thefactthatitisnotagroundstateofH . Thephysical
classical
reasoning behind this statement is that the UUD state is the
I. CLASSICALGROUNDSTATE
ground state of the antiferromagnetic triangular Heisenberg
modeloverafinitewindowofmagneticfieldvalue. Giventhe
TheUUDstate[Fig.S1(a)]isnottheclassicalgroundstate
gapped nature of the excitation spectrum, it is expected that
oftheeasy-planeXXZmodelonthestackedtriangular-lattice
this phase should extend up to a nonzero value of J (quasi-
c
in a magnetic field applied along the a-axis. To show this,
2Dsystem).
we take the classical version of the Hamiltonian defined in
To derive the corresponding SW spectrum, we first work
Eq. (1) in the main text, and assume that every spin is lying on the 2D limit, J = 0, for a given value of ∆ ≤ 1 exactly
c
in the ab plane, as favored by the anisotropy and the exter- at H˜ = 3JS, which are the conditions for the UUD state to
nal field [28]. By introducing the unit length classical spin
be the classical ground state. The purpose of our analysis at
Ω = (Ωx,Ωy,Ωz) at each site j, the classical Hamiltonian
j j j j this stage is to derive the quadratic Hamiltonian with which
apartfromirrelevantconstanttermscanbewrittenas
weevaluatemean-field(MF)parameters. Weapplythelocal
1 (cid:88)(cid:32) H˜ (cid:33)2 transformation,
H = JS2 Ω +Ω +Ω −
classical 2 (cid:52),1 (cid:52),2 (cid:52),3 3JS Sx =S˜z, Sy =S˜y, Sz =−S˜x, (S4)
(cid:52) j j j j j j
(cid:88)(cid:16) (cid:17)
+J S2 ΩxΩx +ΩyΩy , (S1)
c i i+cˆ/2 i i+cˆ/2
i (a) (b) H
under the assumption of Ωz = 0, ∀j, where H˜ = g µ H is
j (cid:80) ⊥ B θ
in the ab plane. The summation of means that we have
(cid:52) θ
dividedthein-planebondsineachtriangularlatticelayerinto
upwardtriangles((cid:52))and j = ((cid:52),1),((cid:52),2),and((cid:52),3)arethe
three sites on each triangle (a given site is shared by three θ
neighboring upward triangles, but this does not cause any θ
complicationinthefollowinganalysis). For J = 0,anyspin
c
configurationsatisfying
FIG.S1. (a)SpinconfigurationoftheUUDstate.(b)One-parameter
Ω +Ω +Ω = H˜ , ∀(cid:52), (S2) deformationoftheUUDstate. ThespinsofsublatticesBandC¯ are
(cid:52),1 (cid:52),2 (cid:52),3 3JS fixed.
7
forthesublatticeswithup-spins(namely,A,B,A¯,andC¯)and with
fsotartethsohsoewwniitnhFdSioxgjw.=nS-−1s(Sp˜aiz)jn,.sTS(hCyje=aHnSo˜dlyjs,Bt¯e)iSnbz-jyP=raisSm˜sxjua,mkoiffngbothseonUs(USa5rDe) a¯k =aaaAB¯¯,,kk≡aaa45,,kk, a¯†−k =aaa†††AB¯¯,,−−kk≡aaa†4†5†,,−−kk, (S12)
definedrelativetothepolarizedstateofS˜ : C¯,k 6,k C¯,−k 6,−k
j
S˜z =S −a† a , andthecoefficientmatricescanberelatedtothosefortheeven
j µ,j µ,j
layerssimplybypermutingindices:
SS˜˜−+jj ==(cid:16)S˜S˜xj+j+(cid:17)†iS,˜yj ≈ √2S 1− a†µ4,jSaµ,jaµ,j, (S6) H¯101,k = H¯202,k =001 100 010H101,k010 001 100,
   
fA¯o,rB¯e,acahndsuC¯bliantttihcies1ord≤er.µW≤e 6hacvoerrtreusnpcoantdeidngthetoexAp,aBns,ioCn, H¯102,k = H¯201,k =00 10 01H102,k01 00 10. (S13)
the square roots in the transformation at the order relevant 1 0 0 0 1 0
for deriving the quartic term. By the Fourier transformation,
aµ,k = (1/Nu.c.)1/2(cid:80)j∈µe−ik·rjaµ,j with Nu.c. = Nspin/6, the Under the special condition of Jc = 0, H˜ = 3JS, and
quadraticpartoftheSWHamiltonianisgivenby ∆<1,thequadraticHamiltonianinEq.(S7)ispositivesemi-
definite. Hence we can define the following MF parameters
H0 =H0 +H0 , (S7)
LSW LSW,even LSW,odd symmetrizedbyusingtranslationalandrotationalinvariance:
wheretheconstanttermhasbeendroppedand
1 (cid:88)
HL0SW,even = S2 (cid:88) (cid:16)(a†k)T (a−k)T(cid:17)HH1001,k HH1002,kaa†k, ρµ = Nu.c. j∈µ(cid:104)a†µ,jaµ,j(cid:105)0,
k∈RBZ 21,k 22,k −k 1 (cid:88)
(S8) δ = (cid:104)(a )2(cid:105) ,
µ N µ,j 0
u.c. j∈µ
with
1 (cid:88)(cid:88)
ak =aaaCAB,,,kkk≡aaa123,,,kkk, a†−k =aaaC†††AB,,,−−−kkk≡aaa†1†2†3,,,−−−kkk, (S9) ξζµµνν == 33NN1uu..cc.. (cid:88)jj∈∈µµ (cid:88)ηηˆˆµµνν(cid:104)(cid:104)aa†µµ,,jjaaνν,,jj++ηηˆˆµµνν(cid:105)(cid:105)00,, (S14)
is the contribution from the even layers (sublattices A–C).
Here,H101,k = H202,k,H102,k = H201,k,and whereηˆµν representsthein-planedisplacementvectorforsite
j∈µtoreachaNNsiteinsublatticeν. Themeanvalues(cid:104)...(cid:105)
H101,k =323JJ((1S1+−−1∆H∆˜))γγ−k 332JJ((1S1−−+1∆H∆˜))γγk 32236JJJ((11−−−S∆∆−)1)γHγ˜−kk, aMfrroeFmepvtaahrloaumsaeteeftdoerrwseivfthoerntho(edoddgdr(o)euvlaneydne)srtslaaatyeseorsfHcaLn0SWbe. Wobetaninoetedtshiamtpthlye0
2 k 2 −k
H102,k =−−323JJ((110−+∆∆))γγ−k −−323JJ((110+−∆∆))γγk −−2323JJ((110++∆∆))γγ−kk, ρδξAAA¯¯¯B¯===δρξBBB,,C,δρB¯B¯ξ=B=¯C¯δρ=CC,,ξCδρACC¯,¯==ξCδ¯ρAA¯A,,=ξAB,
2 k 2 −k
(S10) ζ =ζ , ζ =ζ , ζ =ζ . (S15)
A¯B¯ BC B¯C¯ CA C¯A¯ AB
where γ = 1(eik·a +eik·b +e−ik·(a+b)). The summation over
k 3 The excitation spectrum of H0 retains two relativistic
kinEq.(S8)isconfinedtothereducedBrillouinzone(RBZ) modesatk=0[Fig.S2(a)].ToturLnSWonthedeviationsfromthe
correspondingtothemagneticunitcelloftheUUDstate(cid:80)(be- conditionofJc =0andH˜ =3JS,weincludenonlinearterms,
low, this type of summations will be simply denoted as ).
k which indeed will gap out these modes [Fig. S2(b)]. At this
Also, the contribution from the odd layers (sublattices A¯–C¯)
stage,thenonlineartermscorrespondtothedecouplingofthe
is
intra-layer quartic terms. The inter-layer coupling will also
HL0SW,odd = S2 (cid:88)(cid:16)(a¯†k)T (a¯−k)T(cid:17)HH¯¯1001,k HH¯¯1002,ka¯a¯†k, bdeecionucplulidnegdolfattehreoqnu,arwtihcetreermbostwhitlhlebelinadeadredSWtooteurrmnoannldintehaer
k 21,k 22,k −k SWHamiltonianH . Bycombiningallthecontributions,
(S11) NLSW
weobtain,
8
(a) Linear SW theory :
2 2
1.5 1.5
1 1
0.5 0.5
(b) Nonlinear SW theory :
0 0
0 1/3 2/3 1 0 1/3 2/3 1
q = (h, h) q = (h, h)
(c) Nonlinear SW theory :
2 2
1.5 1.5
1 1
0.5 0.5
(d) Nonlinear SW theory :
0 0
0 1/3 2/3 1 0 1/3 2/3 1
q = (h, h) q = (h, h)
FIG.S2. (a)MagnonfrequenciesinthelinearSWapproximationatH˜ =3JS withtwogaplessmodesatq=(0,0),(1/3,1/3),and(2/3,2/3).
MagnonfrequenciesinthenonlinearSWapproximationat(b)H˜ = 3JS,(c)H˜ = 0.9×3JS,and(d)H˜ = 1.1×3JS. Alltheresultsarefor
J =0and∆=0.85.TheconditionofJ (cid:44)0leadstothesplittingofthesethreebranchesintosix.
c c
HNLSW = S2 (cid:88)k (cid:16)(a†k)T (a¯†k)T (a−k)T (a¯−k)T(cid:17)(cid:0)(cid:0)(cid:0)HHHHe(cid:48)e(cid:48)eeo,e−,k,kk(cid:1)(cid:1)(cid:1)††∗ (cid:0)(cid:0)HHHHe(cid:48)(cid:48)oeo,oo−,,kkk(cid:1)(cid:1)∗∗ (cid:0)(cid:0)(cid:0)HHHHe(cid:48)eoee(cid:48),,e−−,kkk(cid:1)(cid:1)(cid:1)TT∗ (cid:0)(cid:0)HHHHeoeo(cid:48)(cid:48),oo−,,kkk(cid:1)(cid:1)∗∗aa¯aa¯†−†kkk, (S16)
eo,k oo,−k eo,−k oo,−k −k
with
 (cid:16) (cid:17)∗ 
Hee,k = H101,k+−200Jc 200Jc 200Jc+S−1(cid:16)ttCMAµABFMA(cid:17)γ∗F−γk ttMABµBCFMBγFγk (cid:16)ttMBMCµCFAFC(cid:17)γ∗−γkk+S−12J00cρA¯ −2J00cρB¯ −2J00cρC¯,
MF k MF −k MF
Hoo,k = H¯101,k+−200Jc 200Jc 200Jc+S−1(cid:16)ttCMA¯¯µABF¯¯MA(cid:17)γ¯∗F−γk (cid:16)ttMAB¯¯µBCF¯¯MB(cid:17)¯γ∗Fγk (cid:16)ttMBMC¯¯µCFAF¯¯C(cid:17)γ¯∗−γkk+S−12J00cρA −2J00cρB −2J00cρC,
MF k MF −k MF
He(cid:48)e,k = H102,k+S−1ggMAΓCMBFAMAFγγF−kk ggMBΓMACFBMBFγγF−kk ggCMΓMBAFCCMFγγF−kk, Ho(cid:48)o,k = H¯102,k+S−1ggMAΓ¯CM¯BF¯AMAF¯¯γγF−kk ggMB¯ΓMA¯CF¯BMBF¯¯γγF−kk ggCMΓ¯MB¯AFC¯CMF¯¯γγF−kk,
Heo,k =cosk3Jc(100+∆) Jc(100−∆) Jc(100−∆)+S−1cosk3(cid:16)tMA00AF¯(cid:17)∗ (cid:16)tMB00BF¯(cid:17)∗ (cid:16)tC00C¯(cid:17)∗,
MF
9
He(cid:48)o,k =cosk3−Jc(100−∆) −Jc(100+∆) −Jc(100+∆)+S−1cosk3gM00AAF¯ gM00BBF¯ gC00C¯. (S17)
MF
Here,
µA =3J(cid:34)ρ −ρ − 1+∆(cid:0)Reξ −Reζ (cid:1)− 1−∆(cid:0)Reξ −Reζ (cid:1)(cid:35),
MF B C 2 AB CA 2 CA AB
µB =3J(cid:34)ρ −ρ − 1+∆(cid:0)Reξ −Reζ (cid:1)− 1−∆(cid:0)Reξ −Reζ (cid:1)(cid:35),
MF A C 2 AB BC 2 BC AB
µC =3J(cid:34)−ρ −ρ + 1+∆(cid:0)Reζ +Reζ (cid:1)− 1−∆(cid:0)Reξ +Reξ (cid:1)(cid:35),
MF A B 2 BC CA 2 BC CA
tAB =3J(cid:34)ξ − 1+∆(cid:0)ρ +ρ (cid:1)+ 1−∆(cid:0)δ∗ +δ (cid:1)(cid:35), gAA¯ = J (cid:34)−1+∆(cid:16)δ +δ (cid:17)+ 1−∆(cid:16)ρ +ρ (cid:17)(cid:35),
MF AB 4 A B 8 A B MF c 4 A A¯ 2 A A¯
tBC =3J(cid:34)−ξ + 1+∆(cid:0)δ∗ +δ (cid:1)− 1−∆(cid:0)ρ +ρ (cid:1)(cid:35), gBB¯ = J (cid:34)1+∆(cid:16)ρ +ρ (cid:17)− 1−∆(cid:16)δ +δ (cid:17)(cid:35),
MF BC 8 B C 4 B C MF c 2 B B¯ 4 B B¯
tCA =3J(cid:34)−ξ + 1+∆(cid:0)δ∗ +δ (cid:1)− 1−∆(cid:0)ρ +ρ (cid:1)(cid:35), gCC¯ = J (cid:34)1+∆(cid:16)ρ +ρ (cid:17)− 1−∆(cid:16)δ +δ (cid:17)(cid:35), (S20)
MF CA 8 C A 4 C A MF c 2 C C¯ 4 C C¯
ΓA = 3J (cid:34)1+∆(cid:0)ξ −ζ (cid:1)+ 1−∆(cid:0)ξ∗ −ζ (cid:1)(cid:35), aretherecombinedMFparametersassociatedwiththeinter-
MF 2 2 CA AB 2 AB CA
layercoupling.
ΓMBF = 32J (cid:34)1+2∆(cid:0)ξB∗C −ζAB(cid:1)+ 1−2∆(cid:0)ξAB−ζBC(cid:1)(cid:35), eteFrsigfuorresSe3leschteodwsvathlueecsaflcourla∆te≤dv1a.lueWseofctohnefisremMedFtphaartamal-l
ΓC = 3J (cid:34)1+∆(cid:0)ξ −ξ∗ (cid:1)− 1−∆(cid:0)ζ +ζ (cid:1)(cid:35), ocofetffihecsieenMtmFaptarirxamineEteqrs.(aSr1e6r)ehaalsvtahlueefdo;llothwisinmgefaonrms :that the
MF 2 2 BC CA 2 BC CA
gMABF =3J(cid:34)ζAB− 1+8∆(cid:0)δA+δB(cid:1)+ 1−4∆(cid:0)ρA+ρB(cid:1)(cid:35), HNLSW =(cid:32)QPkk QPkk(cid:33), (S21)
gBC =3J(cid:34)−ζ + 1+∆(cid:0)ρ +ρ (cid:1)− 1−∆(cid:0)δ +δ (cid:1)(cid:35), where
MF BC 4 B C 8 B C
gCMAF =3J(cid:34)−ζCA+ 1+4∆(cid:0)ρC +ρA(cid:1)− 1−8∆(cid:0)δC +δA(cid:1)(cid:35), Pk =HHeeeo,,kk HHeooo,,kk, Qk =HHee(cid:48)(cid:48)eo,,kk HHeo(cid:48)(cid:48)oo,,kk (S22)
(S18)
are6×6matrices. ThiscanbediagonalizedbyaBogoliubov
with transformation,
µA¯ =µB , µB¯ =µC , µC¯ =µA , a     
tΓMA¯MMA¯BF¯FF==tMΓBCMFMB,FF,tMB¯ΓCF¯MMB¯=FF =tMCAFΓ,MCMFFt,MC¯AF¯ΓM=CM¯FFtMA=BF,ΓMMAFF, aa¯a¯†−†kkk= UVkk UVkk  αα†−kk, (S23)
gA¯B¯ =gBC, gB¯C¯ =gCA, gC¯A¯ =gAB. (S19) −k
MF MF MF MF MF MF
whereα (α† )isthe6-componentvectorcomprisingthean-
are the recombined MF parameters for the intralayer terms. k −k
nihilation(respectively,thecreation)operatorsofBogoliubov
Similarly, bosons. The transformation matrices satisfy Uµκ = (cid:0)Uµκ(cid:1)∗
(cid:34) 1+∆(cid:16) (cid:17) 1−∆(cid:16) (cid:17)(cid:35) and Vµκ = (cid:0)Vµκ(cid:1)∗. The poles, ω , can be obtkained as−tkhe
tMAAF¯ = Jc − 2 ρA+ρA¯ + 4 δ∗A+δA¯ , squarek-rootsof−ktheeigenvaluesofκS,k2(Pk+Qk)(Pk−Qk). As
(cid:34)1+∆(cid:16) (cid:17) 1−∆(cid:16) (cid:17)(cid:35) demonstrated in Figs. S2(c) and S2(d), the in-plane disper-
tBB¯ = J δ∗ +δ − ρ +ρ , sionhasdifferentcharacteristicsdependingon H˜ (cid:46) 3JS and
MF c 4 B B¯ 2 B B¯ H˜ (cid:38) 3JS regardingthebandhybridization; theexperimental
(cid:34)1+∆(cid:16) (cid:17) 1−∆(cid:16) (cid:17)(cid:35) observationinBa CoSb O at10.5Tcorrespondstothecase
tCC¯ = J δ∗ +δ − ρ +ρ , 3 2 9
MF c 4 C C¯ 2 C C¯ ofH˜ (cid:46)3JS.