Exploiting Convolutional Neural Network for Risk Prediction with Medical Feature Embedding

0.22 MB·

by  Zhengping Che

Exploiting Convolutional Neural Network for Risk
Prediction with Medical Feature Embedding
ZhengpingChe YuCheng
UniversityofSouthernCalifornia IBMT.J.WatsonResearchCenter
[email protected] [email protected]
7
1
ZhaonanSun YanLiu
0
IBMT.J.WatsonResearchCenter UniversityofSouthernCalifornia
2
[email protected] [email protected]
n
a
J

5 1 Introduction
2

The widespread availability of electronic health records (EHRs) promises to usher in the era of
]
G personalized medicine. However, the problem of extracting useful clinical representations from
L longitudinalEHRdata,sometimescalledcomputationalphenotyping,remainschallenging,owingto
theheterogeneous,longitudinallyirregular,noisyandincompletenatureofsuchdata.
.
s
c Inthispaper,ourfocusisontheproblemsofhighdimensionalityandtemporality. Weexploredeep
[ neuralnetworkmodelswithlearnedmedicalfeatureembeddingtodealwiththeseissues. Specifically,

we use a multi-layer convolutional neural network (CNN) to parameterize the model and is thus
1
abletocapturecomplexnon-linearlongitudinalevolutionofEHRs. Differentfromrecentproposed
v
4 deeplearningapproachessuchasstackedauto-encoders[11]andrecurrentneuralnetwork[5,2],our
7 modelcaneffectivelycapturelocal/shorttemporaldependencyinEHRs,whichisbeneficialforrisk
4 prediction. Toaccountforhighdimensionality,insteadofusingtherawEHRdataastheinput,we
7 usetheembeddingmedicalfeaturesintheCNNmodel. Basedonthemedicalcontext,eachmedical
0 eventiscompressedintoagivenlengthvectorwithmedicalfeatureembedding. Similartotheword
1. embedding[8],theeventembeddingpresentedinourmodelholdsitsnaturalmedicalconcept. Our
0 initialexperimentsproducepromisingresults,anddemonstratetheeffectivenessofboththemedical
7 featureembeddingandtheproposedconvolutionalneuralnetworkinriskpredictiononcohortsof
1 congestiveheartfailureanddiabetespatients,comparedwithseveralstrongbaselines.
:
v
i
X 2 Methodology
r
a
2.1 MedicalFeatureEmbeddingLearning
Weextendword2vecmodel[8,3]tolearnmedicalfeatureembeddingsfromEHRdata. Word2vec
takesacorpusoftextsequencesandlearnsthevectorrepresentations,calledembeddings,foreach
word. The words with similar contexts usually have close embeddings in the vector space. In
healthcaredomain,Word2vechasalsobeenusedformedicaltextsandclaims[4]. SinceourEHRs
arealsosequentialdatawiththousandsofdifferentmedicalevents(e.g.,diagnosis,medications,lab
test,etc.),itisnaturaltogeneralizetheideasinword2vectolearnlow-dimensionandmeaningful
embeddingsformedicaleventsgivenalargecorpusofEHRdata. Wetakealltherecordsforone
patientasasequenceofmedicalevents. Foreacheventinthesequence,theword2vecmodeltakes
other events within a local window of current event into consideration. We train word2vec with
continuousbag-of-words(CBOW)method.
NIPS2016WorkshoponMachineLearningforHealth(NIPSML4HC).
EHR matrix with Convolution Pooled Full-connected
embedding feature maps representation softmax
F2
X
Tp p K2+K3
F3
D K3 K2
Figure1: Convolutionalneuralnetworkpredictionmodel(withfiltersofsize2and3).
2.2 CNNRiskPredictionModel
Convolutionalneuralnetwork(CNN)isonetypeofneuralnetworkmodelswhichisgoodatcapturing
localstructureandhasmoderatemodelsize. AvariaityofCNNmodelsareshowntobeeffective
inhandlingimages[7],sequentialdata[6],andrecentlyonmedicallabtests[9]. Inriskprediction
tasksonEHRdataofdiagnosesandmedications,thetemporalstructuresandlocaldependencies
across features also provide lots of useful information. In order to effectively learn these useful
characteristicsfromEHRdataandmakebetterpredictions,weuseaCNNwithone-dimensional
convolutional operation over the temporal dimension. The input to our CNN model is the EHR
recordsofpatientp, whichisrepresentedasatemporalembeddingmatrixXp ∈ RTp×D, where
T , which is usually different among the patients, is the number of medical events in patient p’s
p
record, and D is the learned embedding dimension. Each row x ∈ RD,i = 1,...,T in X is
i p p
theembeddingvectorofitheventofthatpatient. Noticingthatnospatialortemporalrelationship
inherentlyexistsalongtheembeddingdimensionintheEHRmatrix,weapplyconvolutionaloperation
notoverembeddingdimensionbutonlyoverthetemporaldimension. AssumethattheCNNmodel
hasK convolutionalfiltersofsizeF,wewillhaveK differentvectoroutputsoflengthT −F +1.
p
Afilterofsize2capturespairwisetemporaldependency,whileafilteroflargersizemightcapture
longertemporaldependency. Thatsaid,usingacombinationoffilterswithdifferentlengthsbenefits
capturingdependenciesinmultiplelevelsandthusimprovestheprediction,whichisvalidatedin
ourexperiments. Afterconvolutionalstep,weapplyamax-poolingoperationalongthetemporal
dimensionstokeepthemostimportantfeaturesacrossthetime. Thistemporalpoolingproduces
outputvectoroflengthK andmoreoverconvertstheinputswithdifferenttemporallengthsintoa
fixedlengthvectorinherently. Finallyafullyconnectedsoftmaxlayerisusedtogettheprediction.
An illustration of the proposed CNN prediction model is shown in Figure 1. Our convolutional
networkmodelissimilartothebasicmodelshownin[1]butwithmultiplescalefilters. Furthermore,
theinputtoourmodelistemporalembeddingmatrixofeventsinsteadofraweventmatrix. Wecan
alsojointlyfine-tuneatask-specificembeddingmatrixtogetherwithtrainingtheCNNmodel,oruse
bothfixedandfine-tunedembeddingtogetherasinputmatrix.
3 Experiments
3.1 ExperimentalSetup
3.1.1 MedicalFeatureEmbedding
WeappliedmedicalfeatureembeddinglearningmethoddescribedinSection2.1fromaprivateEHR
dataset. Thisdatasetcontains218680patientsand14969489observationsof14690uniquemedical
events, whichareICD-9diagnosisandmedications. Weonlycomputetheembeddingsfor8627
events that show at least 5 times in this dataset. We set the local window size to be 20 for each
event,whichmeansatmost40eventsnearbyareconsideredtobetheneighborsofthatevent. Weset
theembeddingdimensionis200inallthefollowingexperiments,ifnotmentionedexplicitly. Our
medicalfeatureembeddingmodelisimplementedbasedonpythonGensim[10]package.
3.1.2 RiskPrediction
Weapplytheconvolutionalneuralnetworkandthelearnedembeddingstopredicttwoimportant
diagnoses,diabetesandcongestiveheartfailure,astwobinaryclassificationtasksseparately. Wetake
thesameprivateEHRdatasetusedinmedicalembeddinglearning,butonlyasmallportionofpatients
inthedatasethavethetargetdiagnoses. Inordertohandlethisclassimbalanceissueandcompare
allmethodsrobustly,werespectivelyextractsubsetsforthesetwodiagnosis. First,allpatientswith
the target diagnosis are selected to build the case group. We take both the ICD-9 diagnoses and
2
Table1: Predictionperformancecomparisonofconvolutionalneuralnetworkandbaselines.
Method Input HeartFailure Diabetes
Accuracy AUROC Accuracy AUROC
CNN W2V 0.8630 0.9329 0.9844 0.9989
CNN RAND 0.8337 0.8999 0.9815 0.9959
CNN RAW 0.8511 0.9203 0.9748 0.9953
LR BOFW 0.8094 0.8652 0.9066 0.9681
SVM BOFW 0.7906 0.8397 0.9044 0.9687
RF BOFW 0.8620 0.9311 0.9029 0.9683
LR W2V 0.8625 0.9289 0.9266 0.9802
SVM W2V 0.8476 0.9140 0.9148 0.9753
RF W2V 0.8650 0.9269 0.8955 0.9654
medicationsasinputfeaturesforheartfailureprediction,butonlydiagnosesfordiabetessincesome
medications may apparently indicate diabetes. If there are multiple events at the same time, we
generatetheinputsequencebyorderingthemascendinglybytheeventindex. Aspatientsusually
havemedicalrecordsindifferentlengths,itmightimpactthepredictionperformanceandpreventus
fromlearningrobustmodelandcomparingmodelsconsistently. Weremoveallpatientswithlessthan
50medicaleventrecords,andkeepthefirstupto250eventsbeforethetimetheygettheirfirsttarget
diagnosis. Foreachpatientinthecasegroup, twootherpatientswithoutthetargetdiagnosisbut
withsimilardemographicinformationandEHRrecordlengthareselectedtoformacontrolgroup.
Similarlywelimitthelengthofeachrecordsequencebetween50and250. Finallywehave2248and
4496patientsincaseandcontrolgroupfordiabetes,and3357and6714patientsincaseandcontrol
groupforheartfailure. Wesplitallthedataintotraining,validation,andtestsubsetsby7:1:2. The
CNNmodelweusedinexperimentshasoneconvolutionallayerwith100filtersofsize3,4and5,
followedbyonepoolinglayerandonefullyconnectedpredictionlayer. Wetaketanhfunctionas
activationinconvolutionalandfullyconnectedlayersinceitperformedbetterinourexperiments.
WetrainthenetworkusingAdaDelta[12]withdefaultsettings.
3.2 ExperimentalResults
Inourexperiments,wetakeclassificationaccuracy(Accuracy),theareaunderreceiveroperating
characteristiccurve(AUROC),theareaunderprecision-recallcurve(AUPRC),andmaximumF1
score(MaxF1)tocomparetheperformanceofallmethodsintworiskpredictiontasks.
3.2.1 RiskPredictionComparison
Firstwewanttoshowtheperformanceofourproposedpredictmodel.Wecomparelogisticregression
(LR), linear support vector machine (SVM) and random forest (RF) as baselines. We apply L2
regularizers in LR and SVM. We use early stopping for RF with at most 50 trees. We evaluate
baselineswitheitherrawinputsorlearnedembeddings. Forrawinput(BOFW),wetakebagofwords
tokeeponlythefrequenciesofallevents, thustheinputvectorisoflength8627. Forword2vec
embedding input (W2V), we take the embeddings of all events in that record. In order to make
the patient records in different temporal lengths to be the same length. we take the columnwise
aggregationvaluesalongthetemporaldimension,includingsummation,maximum,andminimumof
allembeddingsintherecord,whichisequivalenttoW2V-ALLshowninSection3.2.2.Inthisway,the
inputlengthisrelatedtoonlytheembeddingdimensionandmuchsmallerthanrawinputlength. For
CNNmodel,besidesthelearntmedicalfeatureembeddings(W2V),wealsoevaluatetheperformance
withrawfeatureindexasinput(RAW),anduserandomvectortoinitializeembeddingandjointly
trainedembeddingstogetherwithCNNmodel(RAND). Table1showstheclassificationaccuracy
andAUROConthetwopredictiontasks. TheproposedCNNmethodswithlearnedembeddingsis
amongthebestmethodsinheartfailurepredictiontaskandsignificantlyoutperformsbaselinesin
diabetespredictiontask. Theperformanceimprovementmainlycomesfromthelearnedembeddings
forheartfailuretask,butfromCNNmodelstructuresfordiabetestask. Thelearnedembeddinghelps
LRandSVMalot,butRFbarelybenefitsfromtheit,whichisprobablybecauseRFselectsdiscrete
inputfeatureateachtreenode.
3
Table2: Learnedembeddingsv.s. rawinputsonlogisticregressionpredictionmodel.
HeartFailure Diabetes
Accuracy AUROC AUPRC MaxF1 Accuracy AUROC AUPRC MaxF1
BOFW 0.8094 0.8652 0.7590 0.7302 0.9066 0.9681 0.9386 0.8654
RAND-SUM 0.8005 0.8784 0.7818 0.7382 0.8488 0.9317 0.8869 0.8030
W2V-AVE 0.8471 0.9119 0.8133 0.7792 0.8977 0.9583 0.8848 0.8639
W2V-SUM 0.8407 0.9142 0.8389 0.7797 0.9266 0.9802 0.9638 0.8950
W2V-MAX 0.8422 0.9157 0.8267 0.7878 0.8695 0.9431 0.8654 0.8168
W2V-ALL 0.8625 0.9289 0.8668 0.8056 0.9162 0.9740 0.9530 0.8763
3.2.2 FeatureEmbeddingEvaluation
Next,wecomparethelearnedmedicalfeatureembeddingsandrawrepresentationsbylogisticregres-
sionpredictionmodels. ComparingwithrawfeatureindexBOFW,wetakeseveralrepresentations
basedonlearnedembeddings: W2V-AVE,W2V-SUM,W2V-MAXtakestheaverage,summation,and
maximumvaluesofthecolumnwiseembeddinginputs,respectively. Theinputsizeis200forthese
threemethods. Wealsotakeacombinationofsummation,minimumandmaximumvalues,whichis
usedinSection3.2.1,asW2V-ALL. Additionally,wetakethesummationofrandomembeddings
(RAND-SUM)ascomparison. AsshowninTable2,ourlearnedword2vecembeddingsoutperform
therawinputandrandomembedding,especiallyforheartfailureprediction.
3.2.3 EarlyPrediction
Makingbetterpredictioninadvancecanhelpdoctormaketimelydecisionsanddiagnosediseases
intheirearlystage. Wealsotestourmodelinasimulatedearlystoppingsetting. Foreachpatient
inthecasegroup,weonlytaketheobservationsbeforeaperiodoftimefromthetimethatthefirst
target diagnosis is confirmed to the patient. We set the hold-off period to be 90/180 days in our
experiments,topredictwhetherthepatientwillhavethetargetdiagnosisin90/180dayslater. Some
ofthepatientsdonothaveenoughrecordsbeforethefirsttargetdiagnosisandthusareremovedfrom
thedataset. Thusthelongerhold-offperiodweselect,thefewerpatientswillbetakeninthecase
groupwhichmakestheproblemmoredifficult. Wealsoremovesomepatientsincontrolgroupto
makesurethesizeofcontrolgroupisalwaysdoubleofthatofcasegroup. Thenumberofpatients
andearlypredictionperformanceisshownTable3. Asexpected,theperformancedecreaseswhenthe
durationgoesup,butit’sstillcomparablewithbaselinesshowninTable1. Forinstance,our180-day
earlypredictionondiabetesachievesAccuracyof0.9277,whichisevenhigherthanthethatofbest
baselinewithfullobservation,whichaccuracyof0.9266fromLR+W2V-ALL.
4 Discussion
In this work, we proposed a general framework for risk prediction tasks by convolutional neural
networkandlearnedmedicalfeatureembeddingonEHRdata. Wefoundthatthelearnedembeddings
ofarebetterthanrawrepresentationsinbothbaselinesandourproposedCNNmodels. Wealso
evaluated our framework in details and demonstrated its capability to capture complex temporal
structuresandmakereasonableearlypredictions. Inthefutureweplantoconsidertemporalgapsof
eventsinembeddingandpredictionmodel,andfindclinicalinterpretationsofthelearnedembeddings
andfiltersinthenetwork.
Table3: EarlypredictionresultsforCNNpredictionmodel.
HeartFailure Diabetes
#ofCase Accuracy AUROC #ofCase Accuracy AUROC
0days 3357 0.8630 0.9329 2248 0.9844 0.9989
90days 2573 0.8329 0.8889 1616 0.9835 0.9988
180days 2105 0.8274 0.8995 1221 0.9277 0.9716
4
References
[1] Y.Cheng,F.Wang,P.Zhang,andJ.Hu. Riskpredictionwithelectronichealthrecords:Adeeplearning
approach. SDM,2016.
[2] K.Cho,B.VanMerriënboer,D.Bahdanau,andY.Bengio. Onthepropertiesofneuralmachinetranslation:
Encoder-decoderapproaches. arXivpreprintarXiv:1409.1259,2014.
[3] E. Choi, M. T. Bahadori, E. Searles, C. Coffey, M. Thompson, J. Bost, J. Tejedor-Sojo, and J. Sun.
Multi-layerrepresentationlearningformedicalconcepts. InProceedingsofthe22NdACMSIGKDD
InternationalConferenceonKnowledgeDiscoveryandDataMining,KDD’16,pages1495–1504,New
York,NY,USA,2016.ACM.
[4] Y.Choi,C.Y.-I.Chiu,andD.Sontag. Learninglow-dimensionalrepresentationsofmedicalconcepts.
AMIASummitsonTranslationalScienceProceedings,2016:41,2016.
[5] S.HochreiterandJ.Schmidhuber. Longshort-termmemory. Neuralcomputation,9(8):1735–1780,1997.
[6] N.Kalchbrenner,E.Grefenstette,andP.Blunsom.Aconvolutionalneuralnetworkformodellingsentences.
arXivpreprintarXiv:1404.2188,2014.
[7] A.Krizhevsky,I.Sutskever,andG.E.Hinton. Imagenetclassificationwithdeepconvolutionalneural
networks. InAdvancesinneuralinformationprocessingsystems,pages1097–1105,2012.
[8] T.MikolovandJ.Dean. Distributedrepresentationsofwordsandphrasesandtheircompositionality.
Advancesinneuralinformationprocessingsystems,2013.
[9] N.RazavianandD.Sontag. Temporalconvolutionalneuralnetworksfordiagnosisfromlabtests. arXiv
preprintarXiv:1511.07938,2015.
[10] R.Rˇehu˚ˇrekandP.Sojka. SoftwareFrameworkforTopicModellingwithLargeCorpora. InProceedingsof
theLREC2010WorkshoponNewChallengesforNLPFrameworks,pages45–50,Valletta,Malta,May
2010.ELRA. http://is.muni.cz/publication/884893/en.
[11] P.Vincent,H.Larochelle,I.Lajoie,Y.Bengio,andP.-A.Manzagol. Stackeddenoisingautoencoders:
Learningusefulrepresentationsinadeepnetworkwithalocaldenoisingcriterion. JournalofMachine
LearningResearch,11(Dec):3371–3408,2010.
[12] M.D.Zeiler. Adadelta:anadaptivelearningratemethod. arXivpreprintarXiv:1212.5701,2012.
5