Analyzing Rater Agreement: Manifest Variable Methods

196 Pages·2004·7 MB·English

by  Alexander von Eye

Analyzing
Rater
Agreement
8
AnalyziRnagt eArg reement
ManifeVsatr iabMleet hods
This page intentionally left blank
AnalyziRnagt eArg reement
ManifeVsatr iabMleet hods
Alexandevro nE ye
MichigSatanUt nei versity
EunY oungM un
UniverosfAi ltya baamat B irmingham
LAWRENCE ERLBAUM ASSOCIATEPSU,B LISHERS
2005 MahwahN,e wJ ersey London
Camera ready copy for this book was provided by the authors.
Copyright © 2005 by Lawrence Erlbaum Associates, Inc.
All rights reserved. No part of this book may be reproduced in any form,
by photostat, microform, retrieval system, or any other means, with-
out prior written permission of the publisher.
Lawrence Erlbaum Associates, Inc., Publishers
10 Industrial Avenue
Mahwah, New Jersey 07430
Cover design by Kathryn Houghtaling Lacey
Library of Congress Cataloging-in-Publication Data
Eye, Alexander von.
Analyzing rater agreement: manifest variable methods / Alexander
von Eye, Eun Young Mun.
p. cm.
Includes bibliographical references and index.
ISBN 0-8058-4967-X (alk. paper)
1. Multivariate analysis. 2. Acquiescence (Psychology)—Statistical
methods. I. Mun, Eun Young. II. Title.
QA278.E94 2004
519.5'35—dc22 2004043344
CIP
Books published by Lawrence Erlbaum Associates are printed on acid-
free paper, and their bindings are chosen for strength and durability.
Printed in the United States of America
10 9 8 7 6 5 4 3 21
Disclaimer:
This eBook does not include the ancillary media that was
packaged with the original printed version of the book.
Contents
Preface IX
1. CoefficieonfRt ast eArg reement 1
1.1 Cohen'1Cs ( kappa)
1.11. 1C asa SummaryS tatemefnotrt heE ntirAeg reemenTtab le 2
1.12. Conditilo1C n a 8
1.2 Weightde 1C 10
1.3 Raw AgreemenBtr,e nnaann dP redigerK'.,sa nda
Compariswoint hC ohen'1Cs 13
1.4 TheP owero f1C 17
1.5 KendallW' fs orO rdinaDla ta 19
1.6 MeasuriAnggr eemenatm ongT hreoer M oreR aters 22
1.7 ManyR aterosr M anyC omparisoOnb jects 25
1.8 Exercises 27
2. Log-LineMaord elosf R ateArg reement 31
2.1 A Log-LineBaars eM odel 32
2.2 A Familoyf L og-iLneaMro delsf orR ateArg reement 34
2.3 SpecifLiocg -LineMaord elsfo rR ateArg reement 35
2.3.1 TheE qual-WeiAgghrte emenMto del 35
2.3.2 TheW eight-bye-sRpnose-CategAogrrye emenMto del 40
2.3.3 Modelwsi thC ovariates 41
2.3..3I Modelfso rR ateArg reemenwti thC ategoriCcoavla riates 42
2.3.3.M2o delfso rR ateArg reemenwti thC ontinouusC ovariates 48
2.3.4 RateArg reemenptl uLsi near-by-LAisnseoacri atifoonr
OrdinaVla riables 54
2.3.5 DifferenWteiiaglh Atg reemenMto delw ithL inear-by-Linear
InteractpilouCnso variates 59
2.4 Extensions 63
2.4.1 ModelinAgg reemenatm ongM oret hanTw o Raters 63
2.4.1.E1s timatioofRn a ter-Pair-SpPeacriafmiect ers 64
2.4.1.A2g reemenatm ongT hreRea ters 67
2.4.2 Rater-SpecTirfeincd s 68
2.4.3 GeneraliCzoeedf ficie1C nts 70
2.5 Exercises 75
3. ExploriRnagt eArg reement 79
3.1 ConfiguraF)re quencAyn alysiAs T:u torial 80
3.2 CFA BaseM odelfso rR ateArg reemenDtat a 85
1. CoefficieonftR sa terA greement
Then umbero fc oefficiefnotrsr a tearg reemenits l argHeo.w evetrh,e
numbeorfc oefficietnhtasit sa ctuaulsleyid n e mpirirceasle airscs hm all.
Thep resesnetc tidoins cusfisveeos ft hmeo rfer equeunstelcdyo efficients.
Thefi rsitsC ohen('1s 96K0 )( kappoan)e,o ft hem ostw ideleym ployed
coefficieinntt sh es ocisacli encKe si.sa coefficiefnotrn ominlaelv el
variabTlheess .e conadn dt het hircdo efficieanrtesv ariaonftK s.T he
seconcdo efficiewneti,g htKe,d a llowtsh es tattiiscaanl alytsotp lace
differewnetiighatolsn d iscrerpaatnitn Tghsic.so efficiernetq uiorredsi nal
ratisncga lTehse.t hicrode fficieinsBt r ennaannd P redigKe"r,a' v sa riant
ofK thauts eas d iffercehnatn cmeo detlh atnh eo riginKa.lT hef ourth
coefficiiesrn atwa geremenwth icehx prestsheeds e greoef a greemeanst
percentoafgj eu dgemeinntw sh icrha teargrse eT.h efi ftcho efficient
discushseerdie s K endalWl.' sT hicso efficiiesdn etf infeodro rdinal
variables.
1.1 CohenK' s( Kappa)
Cleartlhyme o sftr equeunstelcdyoe fficieonfrt a taegrr eemenitsC ohen's
(196k0a)p pKa.,I ni tosr iginfaolr mw,h icihsp resteenidn t hisse ctitohni,s
coefficiceannbt e a ppliteods quacrreo ss-clcaastsiioofinfts w or aters'
judgeme(nvtasr iafnottrhs r oerem orer atearrspe r esenint Seedc ti1o.n6 ).
Thescer oss-classiafircaeal tscioao lnlsae gdr eemetnatb lCeosn.s idtehre
twor ateAr asn dB whou setdh et hrecea tego1r;i2ea,sn d3 toe valuate
2 CoefficioefnR tast eArg reement
studenptesr'f ormanicnEne g lisThh.e c ross-classifiocft ahteisroean t ers'
judgemecnatsnbe depictaesgd i veinnT abl1e. 1.
Thei nterpretoaftt ihofenr equenmci;eiins't , h cer oss-classification
giveinn T abl1e. 1i ss traightforCwealr1ld1 .d isplatyhsen umbeorf
instanicnew sh icbho tRha teAr andR ateBr u sedC atego1r.yC el1l2
contaitnhsen umbeorf i nstanicnew sh icRha teAr u seCda tego1r ayn d
RateBr u seCda tego2r,ay n ds of ortThh.ec elwlist ihn dexie= sj d isplay
then umberosfi ncidenicnwe hsi cthhe twor ateursse tdh sea mec ategory.
Thescee lalrse a lscoa lltehdea greemecnetl lTsh.e scee lalrse s hadeidn
Tabl1e. 1.
Tabl1e .1: Cross-ClassifiocfaT tiwoon R aterJsu'd gements
RateBr
Ratinagt g ories
2 3
Rater m, mu m,..
Rating
Categories 2 111:1 m!J m2J
3 m., mJJ mH
Thef ollowtiwnogs ectifiornssit n trodKu acsea coefficietnhta t
alloownse t od escrribaeta egrre emenitnt hef oromf a s umma rys tatement
fora ne ntitraeb lSee.c oncdo,n ditiKo nisai ln troducTehdi.sm easure
alloownse t od escrriabteae grr eemesnetp araftoeerla yc rha ticnagt egory.
1.1.1K asa SummaryS tatemefnotrt heE ntirAeg reemenTta ble
Toi ntrodCuochee n'K,sl eptiJ b et hep robabiolfiC teylij l. T hec eltlhsa t
Pw
indicraatteae grre ementth,ai ts t,h ea greemecnetl lhsa,v per obability
Thep aramettehret a1,
I
LPzi
e,= z�l
descritbheeps r oportoifio nns tanicnwes h icthh et wor ateargsr eweh,e re
Ii st hen umbero fc ategorTioe hsa.v ea referewnictewh h ic8h1 c anb e
comparewde,a ssumien dependeonftc heet wor ateIrnso .t hewro rdsw,e
Cohen'Ks 3
assumteh atth er atedrisdn oti nflueenacceho thewrh ene valuattihneg
studen(tLsa.t ienrt hitse xwte,w ilsle et hatth iass sumptcioornr esponds
tom aine ffecmtosd elisnl og-lianneaalry saindsC onfigurFar!e quency
AnalysiBsa.s)e odn t hiass usmptiowne, c ane stimattheep roportoifo n
instanicnwe hsi cht het wor ateargrse eb yc hancues intgh et<Lz,
I
LP;P ';
82
=
1=1
wheraep eriiondd ictahtemae r ginaslu mmeadc rosMso.r es pecificail.l y,
indicattheies t rho wt otaanld . ii ndicattheies t cho lumtno taSlu.b tracting
82fr om 81r esuilnta sm easurofer ataegrr eemecnotr,r ecbtyec dh ancIef.
thed iffereen1c- ee 2i sp osititvheet, w or ateargsr emeo reo ftetnh an
expectbeads eodn c hancIefe. 1 - 82i sn egatitvhee,ay g relee sosf tetnh an
expectbeads eodn c hance.
Thel argepsots sidbilsec repabnectyw e8en1a nd8 2i s1 - 82T•h is
discreparnecsyu wlhtesn a ljlu dgemeanptpse airnt hseh adecde lolfst he
cross-classitfihcaiatst t,ih oean g,r eemecnetl olfst hec ross-classification
(seTea bl1e. 1In)t .h icsa saeg,r eemenitsp erfeWceti.g htitnhgde i fference
81 -82b y1 - 82y ielCdosh enK' s( kpapa),
81- 82
K =
1 - 82
K indicattheeps r oportoifio nnc idenicnwe hsi cthw or ateursset hes ame
categortioee vsla uataen umbeorf o bjecctosr,r ecbtyec dh ance.
A generianlt erpreotfaK t fioocnu soenst hec haracteorfiK s atsi c
a measuroef p roportrieodnuactitenie ornr (oPrR EFl;e is1s9,7 5K) .
inspetchtcese lilnst hmea ind iagonoaftl h Iex Ic ross-classoiffit cwaot ion
raterjsu'd gemenTthse.q uestiaosnk eids h owt heo bservferde quency
distribduitfifofenrr ostm h eex pecteodrc, h ancdeis tribuitnti hodeni agonal
cellIsft. h eo bservdeids tribcuotnitoanim nosr ec aseosfa greemeonnte,
exprestsheers e suilntt e rmosf t hep roportiroendautcet iinoe nr roTrh.i s
reductiinodni cattheatsth eo bservferde quednicsyt ribcuotnitoanim nosre
caseosf a greemeanntd f ewecra seosf d isagreemtehnatn t hec hance
distrtiiboIunn.d ifferteenrtm Ks i,s a ne xamploef a PREm easuroeft he
form
81 -82
PRE= ----­
ma.-x1()8 -82
Them aximumv aluteh a8t1 c ant akies 1 .T hisw ouldi ndicatthaeat l l
4 CoefficientosfR ateArg reement
responsaersel ocatiendt hem aind iagonoarl,t hatth eraer en o
1
disagreemTehneat bso.v dee finitoifKo nu seesl inth e denominator.
=
ThusK, c anb ei dentiafisea Pd RE measure.
An estimoaftK e u ndear m ultinomsiaamlp lisncgh emcea nb e
obtainbeykd a pphaa t,
"'
N m. -"'m. m.
� ll � l. .l
iC =
wherie= 1,. .,.Ii ndextehsce a tegoursieebdsy t hrea teNr iss,t hneu mber
ofd ecisimoandseb yt hrea tearnsdm, indicattheoesb servferde quencies.
Thimse asuhraebes e nu seadn dd iscusesxetde nsivealnyo (vfeorrv iseewe ,
2002; 1989).
e.gA.g,r esti, Wickens, HistoriKc aclalnby et, racbeadc tko
1954;
.l.,.a measuroefa symmetsriimci la(rGiotoyd ma&n ,Kr uskal, cf.
1985).
Froma&n Llabre,
Thec haracteroifKs itnicclsu de
(1) 1;
Ther angoef Ki s-oo < K 5: thsem allepsots sivballeuo ef Ki sf,o r
E
1
as amploefs izeN,- 1 . ,w herme; i;s t hfer equeinnc y
m
- 11
Celili t,h aits a,n a greemceenlt(l i nt hmea ind iagonPaols)i.t ive
valuoefsK indicaagtree emebnett ttehra cnh ancaen,dn egative
valuoefsK indicaagtree emleenstt sh acnh ance.
(2) 0
K ift hper obabiolfdi itsya gree(moefnft- diacgeolnlaislst) h e
=
samea st hper obabiolfai gtrye em(ednita gocnealllK s c)a;nb ez ero
eveinf t hera tejrusd'eg mentasr eno ti ndependent.
1
(3) K onliyft hep robabiilnti htedy i sagreemceenltil ssz ero.
=
4)
( K isd efineodn liyfa tl eatswto c ategoarrieue sse bdy b otrha ters,
thaits i,ft hep robabiplijii,ts gy r,e attehranz erfoo art l eatswto
cells.
(5)
Ift hep robabiilnti htoeyf f-diagiosnn oanl-sze rtoh,em aximum
valuoef K decreaasset sh em argindaelvsi aftreo ma uniform
distrib(usteiteoh nen otioofnp revalednecpee ndeonfcc yh ance­
1995).
correcatgerde emeGnutg;g enmoos-mannHol,z
6)
( Whent hper obabiolfdi itsya greemdeenctr eaasnedis ss malltehra n
thneu mbeorfa greemeKn itnsc,r eamsoenso tonic(asleFleiy g ure
1,
beloww)h;e nt hep robabiolfdi itsya gretei mnecnreaasnedis s