Table Of ContentEmploying computational linguistics techniques to identify limited patient health literacy:
Findings from the ECLIPPSE study
Employing computational linguistics techniques to identify limited patient health literacy:
Findings from the ECLIPPSE study
Dean Schillinger1,2, Renu Balyan3, Scott A. Crossley4, Danielle S. McNamara3,
and Jennifer Y. Liu2, Andrew J. Karter1,2
1 Zuckerberg San Francisco General Hospital and Trauma Center
2 Kaiser Permanente Northern California
3 Arizona State University
4 Georgia State University
DOI: 10.1111/1475-6773.13560
Health Services Research
(cid:33)(cid:11)(cid:34)(cid:11)(cid:3)(cid:33)(cid:7)(cid:15)(cid:2)(cid:3)(cid:33)(cid:36)(cid:17)(cid:7)(cid:21)(cid:11)
(cid:11)(cid:108)(cid:114)(cid:1140)(cid:111)(cid:139)(cid:98)(cid:109)(cid:93)(cid:2)(cid:49)(cid:111)(cid:108)(cid:114)(cid:134)(cid:124)(cid:45)(cid:124)(cid:98)(cid:111)(cid:109)(cid:45)(cid:1140)(cid:2)(cid:1140)(cid:98)(cid:109)(cid:93)(cid:134)(cid:98)(cid:118)(cid:124)(cid:98)(cid:49)(cid:118)(cid:2)(cid:124)(cid:59)(cid:49)(cid:95)(cid:109)(cid:98)(cid:116)(cid:134)(cid:59)(cid:118)(cid:2)(cid:124)(cid:111)(cid:2)(cid:98)(cid:55)(cid:59)(cid:109)(cid:124)(cid:98)(cid:61)(cid:139)(cid:2)
(cid:1140)(cid:98)(cid:108)(cid:98)(cid:124)(cid:59)(cid:55)(cid:2)(cid:114)(cid:45)(cid:124)(cid:98)(cid:59)(cid:109)(cid:124)(cid:2)(cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:313)(cid:2)(cid:13)(cid:98)(cid:109)(cid:55)(cid:98)(cid:109)(cid:93)(cid:118)(cid:2)(cid:61)(cid:117)(cid:111)(cid:108)(cid:2)(cid:124)(cid:95)(cid:59)(cid:2)(cid:11)(cid:7)(cid:21)(cid:17)(cid:30)(cid:30)(cid:34)(cid:11)(cid:2)
(cid:118)(cid:124)(cid:134)(cid:55)(cid:139)
(cid:9)(cid:59)(cid:45)(cid:109)(cid:2)(cid:34)(cid:49)(cid:95)(cid:98)(cid:1140)(cid:1140)(cid:98)(cid:109)(cid:93)(cid:59)(cid:117)(cid:2)(cid:24)(cid:9)(cid:400)(cid:311)(cid:401)(cid:311)(cid:402)(cid:1354)|(cid:2)(cid:2)(cid:2)(cid:33)(cid:59)(cid:109)(cid:134)(cid:2)(cid:6)(cid:45)(cid:1140)(cid:139)(cid:45)(cid:109)(cid:2)(cid:30)(cid:95)(cid:9)4 (cid:1354)|(cid:2)(cid:2)(cid:2)(cid:34)(cid:49)(cid:111)(cid:124)(cid:124)(cid:2)(cid:3)(cid:314)(cid:2)(cid:7)(cid:117)(cid:111)(cid:118)(cid:118)(cid:1140)(cid:59)(cid:139)(cid:2)(cid:30)(cid:95)(cid:9)5(cid:1354)|(cid:2)(cid:2)(cid:2)(cid:2)
(cid:9)(cid:45)(cid:109)(cid:98)(cid:59)(cid:1140)(cid:1140)(cid:59)(cid:2)(cid:34)(cid:314)(cid:2)(cid:24)(cid:49)(cid:25)(cid:45)(cid:108)(cid:45)(cid:117)(cid:45)(cid:2)(cid:30)(cid:95)(cid:9)6(cid:1354)|(cid:2)(cid:2)(cid:2)(cid:19)(cid:59)(cid:109)(cid:109)(cid:98)(cid:61)(cid:59)(cid:117)(cid:2)(cid:43)(cid:314)(cid:2)(cid:21)(cid:98)(cid:134)(cid:2)(cid:24)(cid:30)(cid:15)(cid:401)(cid:1354)|(cid:2)(cid:2)(cid:2)(cid:3)(cid:109)(cid:55)(cid:117)(cid:59)(cid:137)(cid:2)(cid:19)(cid:314)(cid:2)(cid:20)(cid:45)(cid:117)(cid:124)(cid:59)(cid:117)(cid:2)(cid:30)(cid:95)(cid:9)(cid:400)(cid:311)(cid:401)
1UCSF Division of General Internal
(cid:3)(cid:48)(cid:118)(cid:124)(cid:117)(cid:45)(cid:49)(cid:124)
Medicine, Zuckerberg San Francisco General
Hospital and Trauma Center, San Francisco, (cid:27)(cid:48)(cid:102)(cid:59)(cid:49)(cid:124)(cid:98)(cid:136)(cid:59)(cid:313)(cid:2)To develop novel, scalable, and valid literacy profiles for identifying lim-
California, USA
ited health literacy patients by harnessing natural language processing.
2Division of Research, Kaiser Permanente
Northern California, Oakland, California, USA (cid:9)(cid:45)(cid:124)(cid:45)(cid:2)(cid:34)(cid:111)(cid:134)(cid:117)(cid:49)(cid:59)(cid:313)(cid:2)With respect to the linguistic content, we analyzed 283 216 secure
3UCSF Health Communications Research messages sent by 6941 diabetes patients to physicians within an integrated system's
Program, Center for Vulnerable Populations,
electronic portal. Sociodemographic, clinical, and utilization data were obtained via
Zuckerberg San Francisco General Hospital
and Trauma Center, San Francisco, questionnaire and electronic health records.
California, USA
(cid:34)(cid:124)(cid:134)(cid:55)(cid:139)(cid:2)(cid:9)(cid:59)(cid:118)(cid:98)(cid:93)(cid:109)(cid:313)(cid:2)Retrospective study used natural language processing and machine
4Ira A. Fulton School of Engineering, Arizona
learning to generate five unique “Literacy Profiles” by employing various sets of lin-
State University, Mesa, Arizona, USA
5Department of Applied Linguistics/ESL, guistic indices: Flesch-Kincaid (LP_FK); basic indices of writing complexity, including
College of Arts and Sciences, Georgia State
lexical diversity (LP_LD) and writing quality (LP_WQ); and advanced indices related
University, Atlanta, Georgia, USA
6Department of Psychology, Arizona State to syntactic complexity, lexical sophistication, and diversity, modeled from self-re-
University, Tempe, Arizona, USA ported (LP_SR), and expert-rated (LP_Exp) health literacy. We first determined the
performance of each literacy profile relative to self-reported and expert-rated health
(cid:7)(cid:111)(cid:117)(cid:117)(cid:59)(cid:118)(cid:114)(cid:111)(cid:109)(cid:55)(cid:59)(cid:109)(cid:49)(cid:59)
Dean Schillinger, MD, UCSF Division of literacy to discriminate between high and low health literacy and then assessed
General Internal Medicine, Zuckerberg San
Literacy Profiles’ relationships with known correlates of health literacy, such as pa-
Francisco General Hospital and Trauma
Center, San Francisco, CA 94110, USA. tient sociodemographics and a range of health-related outcomes, including ratings
Email: [email protected]
of physician communication, medication adherence, diabetes control, comorbidities,
(cid:13)(cid:134)(cid:109)(cid:55)(cid:98)(cid:109)(cid:93)(cid:2)(cid:98)(cid:109)(cid:61)(cid:111)(cid:117)(cid:108)(cid:45)(cid:124)(cid:98)(cid:111)(cid:109) and utilization.
National Institutes of Health, Grant/Award
(cid:30)(cid:117)(cid:98)(cid:109)(cid:49)(cid:98)(cid:114)(cid:45)(cid:1140)(cid:2)(cid:13)(cid:98)(cid:109)(cid:55)(cid:98)(cid:109)(cid:93)(cid:118)(cid:313)(cid:2)LP_SR and LP_Exp performed best in discriminating between high
Number: NLM R01 LM012355 ; National
Institute of Child Health and Human and low self-reported (C-statistics: 0.86 and 0.58, respectively) and expert-rated
Development, Grant/Award Number: R01
health literacy (C-statistics: 0.71 and 0.87, respectively) and were significantly associ-
HD46113 ; Institute of Education Sciences,
Grant/Award Number: R305A180261; ated with educational attainment, race/ethnicity, Consumer Assessment of Provider
Office of Naval Research, Grant/Award
and Systems (CAHPS) scores, adherence, glycemia, comorbidities, and emergency
Number: N00014-17-1-2300; National
Institute of Diabetes and Digestive and department visits.
Kidney Diseases, Grant/Award Number: P30
(cid:7)(cid:111)(cid:109)(cid:49)(cid:1140)(cid:134)(cid:118)(cid:98)(cid:111)(cid:109)(cid:118)(cid:313)(cid:2)Since health literacy is a potentially remediable explanatory factor
DK092924 and R01 DK065664
in health care disparities, the development of automated health literacy indicators
Dean Schillinger and Renu Balyan are co-first authors.
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in
any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
© 2020 The Authors. Health Services Research published by Wiley Periodicals LLC on behalf of Health Research and Educational Trust
|
Health Serv Res. 2020;00:1–13. wileyonlinelibrary.com/journal/hesr (cid:1354)(cid:1354)(cid:2)(cid:400)
(cid:401)(cid:1354)(cid:2)|(cid:1354)(cid:2)(cid:1354)(cid:1357) SCHILLINGER Et aL.
Health Services Research
represents a significant accomplishment with broad clinical and population health
applications. Health systems could apply literacy profiles to efficiently determine
whether quality of care and outcomes vary by patient health literacy; identify at-risk
populations for targeting tailored health communications and self-management sup-
port interventions; and inform clinicians to promote improvements in individual-level
care.
(cid:20)(cid:11)(cid:43)(cid:41)(cid:27)(cid:33)(cid:9)(cid:34)
communication, diabetes, health literacy, machine learning, managed care, natural language
processing, secure messaging
(cid:400)(cid:1354)|(cid:1354)(cid:17)(cid:25)(cid:36)(cid:33)(cid:27)(cid:9)(cid:38)(cid:7)(cid:36)(cid:17)(cid:27)(cid:25)
Patient-physician communication is a fundamental pillar of care that (cid:41)(cid:15)(cid:3)(cid:36)(cid:2)(cid:17)(cid:34)(cid:2)(cid:20)(cid:25)(cid:27)(cid:41)(cid:25)(cid:2)(cid:27)(cid:25)(cid:2)(cid:36)(cid:15)(cid:17)(cid:34)(cid:2)(cid:36)(cid:27)(cid:30)(cid:17)(cid:7)
influences patient satisfaction and health outcomes,1 particularly in
diabetes mellitus.2 More than 30 million US adults are living with • Limited health literacy is associated with untoward and
diabetes,3 and one quarter to one third of them has limited health costly health outcomes that contribute to health dispari-
literacy skills. Limited health literacy is associated with untoward ties, and poor communication exchange is an important
and costly diabetes outcomes that contribute to health disparities.4-6 mediator in the relationship between limited health lit-
Limited health literacy impedes physician-patient communication, eracy and health outcomes.
as well as imparts a barrier to patients’ learning and understanding • Given the time and personnel demands intrinsic to cur-
across numerous communication domains.7-10 rent health literacy instruments, combined with the sen-
Being able to assess patients’ health literacy is of interest to sitive nature of screening, measuring health literacy is
clinicians, delivery systems, and the public health community.11 both challenging and controversial.
Clinicians often are unaware of the health literacy status of their • Electronic patient portals are an increasingly popular
patients and have been found to both be receptive to receiving channel for patients and providers to communicate via
this information as well as responsive.12 Ignoring differences secure messaging, and secure messages contain linguis-
in health literacy in population management has been shown to tic content that could be anlayzed to measure patient
amplify health literacy-related disparities.6 To date, identifying health literacy.
limited health literacy patients has proven painstaking and infea-
sible to scale.13 Because “big data”—in this case data derived from (cid:41)(cid:15)(cid:3)(cid:36)(cid:2)(cid:36)(cid:15)(cid:17)(cid:34)(cid:2)(cid:34)(cid:36)(cid:38)(cid:9)(cid:43)(cid:2)(cid:3)(cid:9)(cid:9)(cid:34)
patients’ written secure messages sent via patient portals—are
• Two valid literacy profiles from patients’ secure mes-
increasingly available, we sought to determine whether natural
sage were generated by applying computational linguis-
language processing tools and machine learning approaches can
tics approaches to “big linguistic data”, creating a novel,
be utilized to identify patients with limited health literacy. An au-
feasible, automated, and scalable strategy to identify
tomated process, if it could generate health literacy estimates with
patients and subpopulations with limited health literacy.
sufficient accuracy, would provide an efficient means to identify
• Literacy profiles can provide a health IT tool to enable
patients with limited health literacy, with a number of implications
tailored communication support and other targeted
for improving health services delivery. The few formulas used in
interventions with potential to reduce health literacy-
prior health literacy studies of written text (eg, Flesch-Kincaid,
related disparities.
SMOG) depend on surface-level lexical and sentential features,
have not examined secure messages, and have not used natural
language processing and machine learning. We are aware of only
two studies that attempted to identify patient health literacy using To advance methods for identifying patients’ health literacy auto-
secure message content.14,15 Both studies developed predictive matically, we built on this prior work, and developed and compared
models of health literacy using natural language processing based five literacy profiles based on distinct theoretical models and asso-
on linguistic features extracted from secure messages. These ciated natural language processing (NLP) tools and machine learning
“Literacy Profiles” were generated from patients’ self-reported techniques. The primary goal of the current study was to compare the
health literacy14 and expert ratings of health literacy based on se- relative performance of these literacy profiles with respect to their
cure message quality15 and showed promising results. ability to discriminate between limited vs. adequate health literacy in
SCHILLINGER Et aL. (cid:1354)(cid:1357)(cid:2)(cid:1354)|(cid:2)(cid:1354)(cid:402)
Health Services Research
(cid:13)(cid:17)(cid:14)(cid:38)(cid:33)(cid:11)(cid:2)(cid:400)(cid:1354)Patient and secure messages inclusion/exclusion flowchart*. *MRN#: Patient ID; msg_date: Date of message sent; Svy: survey;
SM#: number of secure messages; LP: literacy profile; PCP_ID: primary care provider ID; proxy_pct: % of proxy messages; TOFROM_PAT_C:
SM sent by the patient
a large sample of diabetes patients based on a large written corpus patterns that mirror previous research in terms of their relationships
of patients’ secure messages. The secondary goal was to assess the with patient sociodemographics, ratings of physician communication,
extent to which these different literacy profiles are associated with and a range of diabetes-related health outcomes.
(cid:403)(cid:1354)(cid:2)|(cid:1354)(cid:2)(cid:1354)(cid:1357) SCHILLINGER Et aL.
Health Services Research
(cid:401)(cid:1354)|(cid:1354)(cid:24)(cid:11)(cid:36)(cid:15)(cid:27)(cid:9)(cid:34) (cid:401)(cid:314)(cid:401)(cid:1354)|(cid:1354)(cid:15)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:318)(cid:14)(cid:111)(cid:1140)(cid:55)(cid:2)(cid:34)(cid:124)(cid:45)(cid:109)(cid:55)(cid:45)(cid:117)(cid:55)(cid:118)(cid:319)
(cid:401)(cid:314)(cid:400)(cid:1354)|(cid:1354)(cid:9)(cid:45)(cid:124)(cid:45)(cid:2)(cid:118)(cid:111)(cid:134)(cid:117)(cid:49)(cid:59)(cid:118)(cid:2)(cid:45)(cid:109)(cid:55)(cid:2)(cid:114)(cid:45)(cid:117)(cid:124)(cid:98)(cid:49)(cid:98)(cid:114)(cid:45)(cid:109)(cid:124)(cid:118) DISTANCE survey included three validated health literacy items
that measure self-efficacy in specific health literacy competencies
Our sampling frame included over one million secure messages using a 5-point Likert scale in which a response of 1 referred to
generated by >150 000 ethnically diverse diabetes patients in the “Always” and a response of 5 to “Never”.21 Questions include self-
Kaiser Permanente Northern California (KPNC) Diabetes Registry, reported confidence in filling out medical forms, problems under-
and >9000 primary care physicians. KPNC is a fully integrated health standing written medical information, and frequency of needing
system that provides care to ~4.4 million patients and supports a help in reading and understanding health materials. We combined
well-developed and mature patient portal (kp.org). these items to create a self-reported health literacy variable to
The current study includes the subset of the KPNC registry pa- compare performance of the linguistic models,14 by averaging
tients who completed a 2005-2006 survey as part of the Diabetes scores across the health literacy items. Average scores were di-
Study of Northern California (DISTANCE) and responded to the chotomized to create binary data, with scores <4.5 indicating lim-
self-reported health literacy items on the associated question- (cid:98)(cid:124)(cid:59)(cid:55)(cid:2)(cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:45)(cid:109)(cid:55)(cid:2)(cid:446)(cid:403)(cid:314)(cid:404)(cid:2)(cid:98)(cid:109)(cid:55)(cid:98)(cid:49)(cid:45)(cid:124)(cid:98)(cid:109)(cid:93)(cid:2)(cid:45)(cid:55)(cid:59)(cid:116)(cid:134)(cid:45)(cid:124)(cid:59)(cid:2)(cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:314)14
naire (N = 14 357).2,16,17 DISTANCE surveyed diabetes patients, The threshold was determined based on the distribution of these
oversampling minority subgroups to assess the role of sociodemo- average scores to maintain the appropriate balance between the
graphic factors on quality and outcomes of care. The average age health literacy categories and is consistent with prior studies that
of the study population at the time was 56.8 (±10); 54.3 percent have employed these measures.10,16,21
were male; and 18.4 percent Latino, 16.9 percent African American, We also generated health literacy scores based on expert rat-
22.8 percent Caucasian, 11.9 percent Filipino, 11.4 percent Asian ings of the quality of patients’ secure messages. These ratings
(Chinese or Japanese), 7.5 percent South Asian/Pacific Islander/ used a subset of the DISTANCE sample, comprised of aggregated
Native American/Eskimo, and 11.0 percent multi-racial. Variables secure messages written by 512 patients purposively sampled
were collected from questionnaires completed via telephone, on- to represent a balance of self-reported health literacy, as well
line or paper and pencil (62 percent response rate). Details of the as a range of age, race/ethnicity, and socio-economic status.15 A
DISTANCE Study have been reported previously.17 health literacy scoring rubric was used to holistically assess the
We first extracted all secure messages (N = 1 050 577) ex- perceived health literacy of the patients based on their secure
changed from 01/01/2006 through 12/31/2015 between diabe- messages, adapting an established rubric used to score the writ-
tes patients and all clinicians from KPNC’s patient portal. For the ing abilities of high school students entering college.22 An ordinal
current analyses, only those secure messages that a patient sent scale ranging from 1 to 6 15 assessed the extent to which patients’
to his or her primary care physician were included. We excluded secure messages demonstrated mastery of written English, orga-
all secure messages: from patients who did not have matching nization and focus, and a varied, accurate, and appropriate health
DISTANCE survey data; written in a language other than English; vocabulary to enable clear access to the health-related content
and written by proxy caregivers (determined by the KP.org proxy and ideas the patient intended to express to their physician.
check-box or by a validated NLP algorithm18). The study flowchart Because of limited relevance to the construct of health literacy,
(Figure 1) shows details about inclusion/ exclusion of patients we removed aspects of the rubric related to length, developing
and associated secure messages. The final dataset consisted of point of views, and discourse-related elements important in ar-
283 216 secure messages sent by 6941 patients to their primary gumentative writing including the use of examples, reason, and
care physicians. evidence. Two raters experienced in linguistics and health liter-
The number of individual SMs sent by a patient to their physi- acy research were trained twice on 25 separate aggregated se-
cian(s) ranged between 2 and 205, and the mean number of SMs cure messages not included in the 512 messages used in the final
sent was 39.88. For each patient, all secure messages were then analysis. After reaching a satisfactory inter-rater reliability (IRR,
collated into a single file. The length of patients’ aggregated SMs r > .70), raters independently scored the 512 messages. Secure
ranged from 1 word and 16 469 words, with a mean length of messages were categorized into two groups: limited health liter-
2058.95 words. To provide appropriate linguistic coverage to de- acy (scores < 4, n =(cid:2)(cid:401)(cid:399)(cid:399)(cid:337)(cid:2)(cid:45)(cid:109)(cid:55)(cid:2)(cid:45)(cid:55)(cid:59)(cid:116)(cid:134)(cid:45)(cid:124)(cid:59)(cid:2)(cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:336)(cid:118)(cid:49)(cid:111)(cid:117)(cid:59)(cid:118)(cid:2)(cid:446)(cid:2)(cid:403)(cid:311)(cid:2)
velop literacy profiles, we excluded patients whose aggregated se- n = 312).
cure messages lacked sufficient words (<50 words, see Figure 1), We examined for existence of any associations between the
a threshold based on previous NLP text research in learning ana- self-reported and expert-rated health literacy measures before
lytics domains.19,20 these two measures were employed to train literacy profiles. The
This study was approved by the KPNC and UCSF Institutional Cramer's V and chi-squared tests were used to measure the strength
Review Boards (IRBs). All analyses involved secondary data and all of association and significance of that association (effect size). The
data were housed on a password-protected secure KPNC server two variables were significantly different (P = .01) and only weakly
that could only be accessed by authorized researchers. correlated (r = 0.118, P = .001).
SCHILLINGER Et aL. (cid:1354)(cid:1357)(cid:2)(cid:1354)|(cid:2)(cid:1354)(cid:404)
Health Services Research
(cid:36)(cid:3)(cid:6)(cid:21)(cid:11)(cid:2)(cid:400)(cid:1354)Linguistic indices used in five
(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:114)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59) (cid:21)(cid:98)(cid:109)(cid:93)(cid:134)(cid:98)(cid:118)(cid:124)(cid:98)(cid:49)(cid:2)(cid:98)(cid:109)(cid:55)(cid:98)(cid:49)(cid:59)(cid:118) (cid:9)(cid:59)(cid:118)(cid:49)(cid:117)(cid:98)(cid:114)(cid:124)(cid:98)(cid:111)(cid:109)
literacy profilesa
LP_FK Readability The length of words (ie, number of letters or
syllables) and length of sentences (ie, number
of words)
LP_LD Lexical Diversity The variety of words used in a text based on D
LP_WQ Word Frequency Frequency of word in a reference corpus
Syntactic Complexity Number of words before the main verb in a
sentence
Lexical Diversity The variety of words used in a text based on
MTLD
LP_SR Concreteness The degree to which a word is concrete
Lexical diversity The variety of words used in a text based on
two measures of lexical diversity: MTLD,
and D
Present tense Incidence of present tense
Determiners Incidence of determiners (eg, a, the)
Adjectives Incidence of adjectives
Function words Incidence of function words such as
prepositions, pronouns etc
LP_Exp Age of Exposure The estimated age at which a word first
appears in a child's vocabulary
Lexical decision The time it takes for a human to judge a string
response time of characters as a word
Attested lemmas Number of attested lemmas used per verb
argument construction
Determiner per Number of determiners in each noun phrase
nominal phrase
Dependents per Number of structural dependents for each
nominal subject subject in a noun phrase
Number of associations Number of words strongly associated with a
single word
Abbreviations: LP_Exp, Literacy Profile Expert-Rated Health Literacy; LP_FK, Literacy Profile
Flesch-Kincaid; LP_LD, Literacy Profile Lexical Diversity; LP_SR, Literacy Profile Self-Reported
Health Literacy; LP_WQ, Literacy Profile Writing Quality.
aWe present examples of linguistic indices for LP_SR (n = 185) and LP_Exp (n = 8).
(cid:401)(cid:314)(cid:402)(cid:1354)|(cid:1354)(cid:25)(cid:45)(cid:124)(cid:134)(cid:117)(cid:45)(cid:1140)(cid:2)(cid:1140)(cid:45)(cid:109)(cid:93)(cid:134)(cid:45)(cid:93)(cid:59)(cid:2)(cid:114)(cid:117)(cid:111)(cid:49)(cid:59)(cid:118)(cid:118)(cid:98)(cid:109)(cid:93)(cid:2)(cid:336)(cid:25)(cid:21)(cid:30)(cid:337)(cid:2)(cid:124)(cid:111)(cid:111)(cid:1140)(cid:118) University International Language Database frequency norms,33 and
Wordnet.34 These tools have been developed using Python and Java.
The linguistic features we examined were derived from the pa- To generate word frequencies for medical terminology, we used
tients’ secure messages using several NLP tools that measure differ- medical corpora from HIstory of MEdicine coRpus Annotation35 and
ent language aspects, such as text level information (eg, number of Informatics for Integrating Biology and the Bedside.36-39
words in the text, type-token ratio), lexical sophistication (eg, word
frequency, concreteness), syntactic complexity (embedded clause
and phrasal complexity), and text cohesion (eg, connectives, word (cid:401)(cid:314)(cid:403)(cid:1354)|(cid:1354)(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:114)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:118)(cid:2)(cid:55)(cid:59)(cid:136)(cid:59)(cid:1140)(cid:111)(cid:114)(cid:59)(cid:55)
overlap). These tools were selected because they measure linguistic
features that are important aspects of literacy, including text com- Using the patients’ secure messages, we applied natural language
plexity, readability, and cohesion. The tools included the Tool for the processing and machine learning techniques to develop five sepa-
Automatic Assessment of Lexical Sophistication,23,24 the Tool for rate literacy profile prototypes for categorizing both patients’ self-
the Automatic Analysis of Cohesion,25 the Tool for the Automatic reported and expert-rated health literacy. As a result, the literacy
Assessment of Syntactic Sophistication and Complexity,26,27 the profiles differed based on the dependent health literacy variable
SEntiment ANalysis and Cognition Engine,28 and Coh-Metrix.29 (self-reported or expert-rated) and the linguistic features that were
These open-access tools rely on several NLP packages to pro- used as independent variables to develop these literacy profiles.
cess text including the Stanford Parser,30 the British National Each literacy profile is briefly discussed below, and the component
Corpus,31 the MRC psycholinguistic database,32 Collins Birmingham linguistic indices used for each are summarized in Table 1.
(cid:1141)(cid:1354)(cid:2)|(cid:1354)(cid:2)(cid:1354)(cid:1357) SCHILLINGER Et aL.
Health Services Research
(cid:401)(cid:314)(cid:403)(cid:314)(cid:400)(cid:1354)|(cid:1354)(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:30)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:2)(cid:13)(cid:1140)(cid:59)(cid:118)(cid:49)(cid:95)(cid:330)(cid:20)(cid:98)(cid:109)(cid:49)(cid:45)(cid:98)(cid:55)(cid:2)(cid:336)(cid:21)(cid:30)(cid:333)(cid:13)(cid:20)(cid:337) bigrams association strength, and dependency structures, were
used as independent variables to predict human ratings of health lit-
As a “baseline” literacy profile, we calculated Flesch-Kincaid read- eracy from the purposively sampled subset of 512 secure messages
ability scores40 for the secure messages. We used Flesch-Kincaid as used in the LP_SR analysis (Table 1). Additional details related to the
a baseline measure because it is one of the most commonly used and development and experimental design of LP_Exp have been previ-
widely available readability formulas in medical domain, including ously reported15 and can also be found in the Health Literacy “Gold
assessing the readability and comprehensibility of a broad range of Standards” section.
medical information.41-46 Flesch-Kincaid is based on average number
of words per sentence and average number of syllables per word.
(cid:401)(cid:314)(cid:404)(cid:1354)|(cid:1354)(cid:3)(cid:118)(cid:118)(cid:59)(cid:118)(cid:118)(cid:98)(cid:109)(cid:93)(cid:2)(cid:114)(cid:59)(cid:117)(cid:61)(cid:111)(cid:117)(cid:108)(cid:45)(cid:109)(cid:49)(cid:59)(cid:2)(cid:111)(cid:61)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:114)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:118)(cid:2)
(cid:45)(cid:93)(cid:45)(cid:98)(cid:109)(cid:118)(cid:124)(cid:2)(cid:93)(cid:111)(cid:1140)(cid:55)(cid:2)(cid:118)(cid:124)(cid:45)(cid:109)(cid:55)(cid:45)(cid:117)(cid:55)(cid:118)
(cid:401)(cid:314)(cid:403)(cid:314)(cid:401)(cid:1354)|(cid:1354)(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:30)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:2)(cid:21)(cid:59)(cid:138)(cid:98)(cid:49)(cid:45)(cid:1140)(cid:2)(cid:9)(cid:98)(cid:136)(cid:59)(cid:117)(cid:118)(cid:98)(cid:124)(cid:139)(cid:2)(cid:336)(cid:21)(cid:30)(cid:333)(cid:21)(cid:9)(cid:337)
We compared the performance of the five literacy profiles using
We used lexical diversity as an additional baseline measure because several supervised machine learning classification algorithms: lin-
it is a commonly used and a straightforward method for assessing ear discriminant analysis (LDA), random forests, support vector
writing proficiency in the linguistics domain and it captures both machine (SVM), naïve Bayes, and neural networks. In a supervised
lexical richness and text cohesion.47,48 Both these features are con- machine learning model, the algorithm learns from a labeled data-
sistent predictors of text sophistication and writing quality.29,49 We set, providing an answer key that the algorithm can use to classify
calculated lexical diversity based on a type-token ratio (TTR) meas- unseen data and evaluate its accuracy. There are two main areas
ure. TTR measures assess lexical variety based on the number of where supervised machine learning is useful: classification and
words produced (tokens) divided by the number of unique words regression. Classification problems ask the algorithm to predict a
produced (types), to evaluate writers’ lexical production. TTR meas- discrete value, identifying the input data as the member of a class,
ure D47 was used because it controls for text length by calculating or group.51-56 The models in this study were trained and tested
probability curves that mathematically model how new words are using Weka (version 3.8.1) and R (version 3.3.2) implementations.
introduced into increasingly large language samples. We first examined performance between the self-reported health
literacy and other literacy profiles, and then between the expert
ratings of health literacy and all other literacy profiles. We report
(cid:401)(cid:314)(cid:403)(cid:314)(cid:402)(cid:1354)|(cid:1354)(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:30)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:2)(cid:41)(cid:117)(cid:98)(cid:124)(cid:98)(cid:109)(cid:93)(cid:2)(cid:32)(cid:134)(cid:45)(cid:1140)(cid:98)(cid:124)(cid:139)(cid:2)(cid:336)(cid:21)(cid:30)(cid:333)(cid:41)(cid:32)(cid:337) results for the models using support vector machines, because it
yielded the best results for all the literacy profiles. Using a ran-
We used a previously validated model50 to classify secure messages as domly allocated split-sample approach, we report discriminatory
either low or high in terms of writing quality. The model, derived from performance results using c-statistics (area under the receiver
three linguistic indices of word frequency, syntactic complexity, and lex- operator [ROC] curves), sensitivity, specificity, and positive and
ical diversity,29,48 reveals that higher-level writers use more infrequent negative predictive values (PPV and NPV).
and lexical diverse words and more syntactically complex structures.
(cid:401)(cid:314)(cid:1141)(cid:1354)|(cid:1354)(cid:3)(cid:118)(cid:118)(cid:59)(cid:118)(cid:118)(cid:98)(cid:109)(cid:93)(cid:2)(cid:49)(cid:117)(cid:98)(cid:124)(cid:59)(cid:117)(cid:98)(cid:111)(cid:109)(cid:330)(cid:117)(cid:59)(cid:1140)(cid:45)(cid:124)(cid:59)(cid:55)(cid:2)(cid:136)(cid:45)(cid:1140)(cid:98)(cid:55)(cid:98)(cid:124)(cid:139)(cid:2)(cid:61)(cid:111)(cid:117)(cid:2)
(cid:401)(cid:314)(cid:403)(cid:314)(cid:403)(cid:1354)|(cid:1354)(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:30)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:2)(cid:34)(cid:59)(cid:1140)(cid:61)(cid:330)(cid:33)(cid:59)(cid:114)(cid:111)(cid:117)(cid:124)(cid:59)(cid:55)(cid:2)(cid:15)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2) (cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:114)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:118)
(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:336)(cid:21)(cid:30)(cid:333)(cid:34)(cid:33)(cid:337)
We examined associations between the health literacy classifica-
A set of 185 linguistic features was calculated from the patients’ tions generated by the literacy profiles and known correlates of
secure messages and used to predict patients’ self-reported health health literacy including patients’ educational attainment, race/
literacy scores. A subset of the linguistic indices used for develop- ethnicity, and age. Because of the known association between
ing this literacy profile are provided in Table 1. The rationale, de- limited health literacy and suboptimal patient-provider communi-
velopment, and experimental design for LP_SR have been briefly cation,7-10 we also examined relationships with patients’ reports
discussed in the Health Literacy “Gold Standards” section, and the of physician communication using an adapted version of the most
details have also been previously reported.14 health literacy-relevant item from the 4-item CAHPS survey7: “In
the last one year, how often have your physician and health care
providers explained things in a way that you could understand?”.
(cid:401)(cid:314)(cid:403)(cid:314)(cid:404)(cid:1354)|(cid:1354)(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:30)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:2)(cid:11)(cid:138)(cid:114)(cid:59)(cid:117)(cid:124)(cid:330)(cid:33)(cid:45)(cid:124)(cid:59)(cid:55)(cid:2)(cid:15)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2) We defined communication as “poor” if the patient reported that
(cid:336)(cid:21)(cid:30)(cid:333)(cid:11)(cid:138)(cid:114)(cid:337) his or her doctor and health care team “never” or “sometimes”
explained things in a way that he/she could understand.2 We
A set of eight linguistic indices, including lexical decision latencies, also examined the extent to which each literacy profile was as-
age of exposure, word naming response times, academic word lists, sociated with diabetes-related outcomes previously found to
SCHILLINGER Et aL. (cid:1354)(cid:1357)(cid:2)(cid:1354)|(cid:2)(cid:1354)(cid:405)
Health Services Research
be associated with health literacy. These included adherence to (cid:402)(cid:1354)|(cid:1354)(cid:33)(cid:11)(cid:34)(cid:38)(cid:21)(cid:36)(cid:34)
cardio-metabolic medications based on continuous medication
gaps (CMG),57,58 a validated measure based on percent time with (cid:402)(cid:314)(cid:400)(cid:1354)|(cid:1354)(cid:7)(cid:117)(cid:98)(cid:124)(cid:59)(cid:117)(cid:98)(cid:111)(cid:109)(cid:330)(cid:117)(cid:59)(cid:1140)(cid:45)(cid:124)(cid:59)(cid:55)(cid:2)(cid:136)(cid:45)(cid:1140)(cid:98)(cid:55)(cid:98)(cid:124)(cid:139)(cid:2)(cid:111)(cid:61)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:114)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59)(cid:118)(cid:2)
insufficient medication supply; hemoglobin A1c (HbA1c), an inte- (cid:48)(cid:45)(cid:118)(cid:59)(cid:55)(cid:2)(cid:111)(cid:109)(cid:2)(cid:114)(cid:59)(cid:117)(cid:61)(cid:111)(cid:117)(cid:108)(cid:45)(cid:109)(cid:49)(cid:59)(cid:2)(cid:45)(cid:93)(cid:45)(cid:98)(cid:109)(cid:118)(cid:124)(cid:2)(cid:93)(cid:111)(cid:1140)(cid:55)(cid:2)(cid:118)(cid:124)(cid:45)(cid:109)(cid:55)(cid:45)(cid:117)(cid:55)(cid:118)
grated measure of blood sugar control, measured both as optimal
(cid:336)(cid:15)(cid:48)(cid:3)(cid:400)(cid:49)(cid:2)(cid:445)(cid:2)(cid:405)(cid:2)(cid:114)(cid:59)(cid:117)(cid:49)(cid:59)(cid:109)(cid:124)(cid:337)(cid:2)(cid:45)(cid:109)(cid:55)(cid:2)(cid:114)(cid:111)(cid:111)(cid:117)(cid:2)(cid:49)(cid:111)(cid:109)(cid:124)(cid:117)(cid:111)(cid:1140)(cid:2)(cid:336)(cid:15)(cid:48)(cid:3)(cid:400)(cid:49)(cid:2)(cid:446)(cid:2)(cid:406)(cid:2)(cid:114)(cid:59)(cid:117)(cid:49)(cid:59)(cid:109)(cid:124)(cid:337)(cid:312)(cid:2)(cid:446)(cid:400)(cid:2) (cid:402)(cid:314)(cid:400)(cid:314)(cid:400)(cid:1354)|(cid:1354)(cid:30)(cid:59)(cid:117)(cid:61)(cid:111)(cid:117)(cid:108)(cid:45)(cid:109)(cid:49)(cid:59)(cid:2)(cid:137)(cid:98)(cid:124)(cid:95)(cid:2)(cid:117)(cid:59)(cid:118)(cid:114)(cid:59)(cid:49)(cid:124)(cid:2)(cid:124)(cid:111)(cid:2)(cid:118)(cid:59)(cid:1140)(cid:61)(cid:330)(cid:117)(cid:59)(cid:114)(cid:111)(cid:117)(cid:124)(cid:59)(cid:55)(cid:2)
clinically relevant hypoglycemic episodes (an adverse drug event (cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)
associated with diabetes treatment)59; and comorbidities, using
the Charlson index60,61 (Deyo version).62 HbA1c reflected the When self-reported health literacy was the dependent variable,
value collected after the first secure message was sent. CMG, LP_SR performed well in terms of its ability to discriminate between
hypoglycemia, and Charlson index were measured the year be- those with limited vs. adequate health literacy, with a c-statistic of
fore the first secure message. The occurrence of one or more hy- 0.86; sensitivity was high, specificity was modest (0.67), and PPV
poglycemia-related ED visits or hospitalizations in the year prior and NPV were acceptable. All other literacy profiles performed
was based on a validated algorithm that uses specific diagnostic poorly (Figure 2). LP_Exp, while performing slightly better than LP_
codes.63 Finally, we explored relationships between each literacy FK, LP_LD, and LP_WQ, yielded a c-statistic of 0.58 and sensitivity
profile and outpatient, emergency room, and hospitalization utili- in the intermediate range; specificity, PPV, and NPV values were all
zation data 12 months prior to the first secure message date. For low.
all analyses, we examined bivariate associations between each of
the literacy profiles and sociodemographics, the single CAHPS
item, and health outcomes using a two-sided p-value at the 0.05 (cid:402)(cid:314)(cid:400)(cid:314)(cid:401)(cid:1354)|(cid:1354)(cid:30)(cid:59)(cid:117)(cid:61)(cid:111)(cid:117)(cid:108)(cid:45)(cid:109)(cid:49)(cid:59)(cid:2)(cid:137)(cid:98)(cid:124)(cid:95)(cid:2)(cid:117)(cid:59)(cid:118)(cid:114)(cid:59)(cid:49)(cid:124)(cid:2)(cid:124)(cid:111)(cid:2)(cid:59)(cid:138)(cid:114)(cid:59)(cid:117)(cid:124)(cid:330)(cid:117)(cid:45)(cid:124)(cid:59)(cid:55)(cid:2)
level. Categorical variables such as education, race, adherence,64 (cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)
HbA1c levels, and hypoglycemia were analyzed using chi-square
analysis. For comorbidity and health care utilization rates, mean When expert-rated health literacy was the dependent variable,
comparisons were conducted using t tests. LP_Exp performed best in terms of its ability to discriminate
(cid:13)(cid:17)(cid:14)(cid:38)(cid:33)(cid:11)(cid:2)(cid:401)(cid:1354)ROCs and performance metrics for the literacy profiles relative to self-reported health literacy. AUC: Area Under Curve;
LP_Exp: Literacy Profile Expert-Rated Health Literacy; LP_FK: Literacy Profile Flesch-Kincaid; LP_LD: Literacy Profile Lexical Diversity;
LP_SR: Literacy Profile Self-Reported Health Literacy; LP_WQ: Literacy Profile Writing Quality; ML: Machine Learning; SVM: Support Vector
Machine [Color figure can be viewed at wileyonlinelibrary.com]
(cid:1142)(cid:1354)(cid:2)|(cid:1354)(cid:2)(cid:1354)(cid:1357) SCHILLINGER Et aL.
Health Services Research
between those with limited vs. adequate health literacy, with a and LP_Exp each were associated with lower education, with the
c-statistic of 0.87, high sensitivity, moderate specificity, and PPV, strongest effects observed for LP_SR and LP_Exp. Only LP_SR was
and NPV in an acceptable range. LP_FK and LP_LD each per- associated with older patient age.
formed poorly (Figure 3). LP_WQ performed better, with all the
performance metrics > 0.70 except for specificity. Performance
metrics for LP_SR were sub-optimal, with a c-statistic of 0.71, (cid:402)(cid:314)(cid:401)(cid:314)(cid:401)(cid:1354)|(cid:1354)(cid:30)(cid:117)(cid:111)(cid:136)(cid:98)(cid:55)(cid:59)(cid:117)(cid:2)(cid:49)(cid:111)(cid:108)(cid:108)(cid:134)(cid:109)(cid:98)(cid:49)(cid:45)(cid:124)(cid:98)(cid:111)(cid:109)
intermediate sensitivity, moderate specificity, and PPV, but low
NPV. The proportion of patients identified as having limited or adequate
health literacy and who reported poor physician communication is
shown in Table 3. Those patients predicted to have limited health
(cid:402)(cid:314)(cid:401)(cid:1354)|(cid:1354)(cid:30)(cid:117)(cid:59)(cid:55)(cid:98)(cid:49)(cid:124)(cid:98)(cid:136)(cid:59)(cid:2)(cid:136)(cid:45)(cid:1140)(cid:98)(cid:55)(cid:98)(cid:124)(cid:139)(cid:2)(cid:48)(cid:45)(cid:118)(cid:59)(cid:55)(cid:2)(cid:111)(cid:109)(cid:2)(cid:45)(cid:118)(cid:118)(cid:111)(cid:49)(cid:98)(cid:45)(cid:124)(cid:98)(cid:111)(cid:109)(cid:118)(cid:2)(cid:137)(cid:98)(cid:124)(cid:95)(cid:2) literacy by LP_SR and LP_Exp only were significantly more likely to
(cid:118)(cid:111)(cid:49)(cid:98)(cid:111)(cid:55)(cid:59)(cid:108)(cid:111)(cid:93)(cid:117)(cid:45)(cid:114)(cid:95)(cid:98)(cid:49)(cid:118)(cid:311)(cid:2)(cid:49)(cid:111)(cid:108)(cid:108)(cid:134)(cid:109)(cid:98)(cid:49)(cid:45)(cid:124)(cid:98)(cid:111)(cid:109)(cid:2)(cid:117)(cid:45)(cid:124)(cid:98)(cid:109)(cid:93)(cid:118)(cid:311)(cid:2)(cid:45)(cid:109)(cid:55)(cid:2) rate their health care providers as “poor” on the CAHPS item, with
(cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:330)(cid:117)(cid:59)(cid:1140)(cid:45)(cid:124)(cid:59)(cid:55)(cid:2)(cid:111)(cid:134)(cid:124)(cid:49)(cid:111)(cid:108)(cid:59)(cid:118) somewhat more robust findings for LP_SR.
(cid:402)(cid:314)(cid:401)(cid:314)(cid:400)(cid:1354)|(cid:1354)(cid:34)(cid:111)(cid:49)(cid:98)(cid:111)(cid:55)(cid:59)(cid:108)(cid:111)(cid:93)(cid:117)(cid:45)(cid:114)(cid:95)(cid:98)(cid:49)(cid:118)
(cid:402)(cid:314)(cid:401)(cid:314)(cid:402)(cid:1354)|(cid:1354)(cid:15)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:111)(cid:134)(cid:124)(cid:49)(cid:111)(cid:108)(cid:59)(cid:118)
We found patterns that mirrored previously observed health liter-
acy-related relationships, with considerable variation across patient Limited health literacy as categorized only by the three literacy pro-
characteristics and literacy profile type. Table 2 shows the educa- files (LP_FK, LP_SR, and LP_Exp) was associated with poor cardio-
tional attainment (% with college degree vs. less), race (% white vs. metabolic medication adherence, serious hypoglycemia and greater
non-white), and the mean age among patients predicted to have ade- comorbidity. Poor medication adherence was most robustly associ-
quate vs. inadequate health literacy for each of the five literacy pro- ated with LP_FK and LP_Exp. Limited health literacy as measured
files. All literacy profiles generated classifications in which limited only by LP_FK and LP_Exp was associated with both optimal and
health literacy was associated with non-white race. LP_LD, LP_SR, poor diabetes control.
(cid:13)(cid:17)(cid:14)(cid:38)(cid:33)(cid:11)(cid:2)(cid:402)(cid:1354)ROCs and performance metrics for the literacy profiles relative to expert-rated literacy. AUC: Area Under Curve; LDA: Linear
Discriminant Analysis; LP_Exp: Literacy Profile Expert-Rated Health Literacy; LP_FK: Literacy Profile Flesch-Kincaid; LP_LD: Literacy Profile
Lexical Diversity; LP_SR: Literacy Profile Self-Reported Health Literacy; LP_WQ: Literacy Profile Writing Quality; ML: Machine Learning;
SVM: Support Vector Machine [Color figure can be viewed at wileyonlinelibrary.com]
SCHILLINGER Et aL. (cid:1354)(cid:1357)(cid:2)(cid:1354)|(cid:2)(cid:1354)(cid:406)
Health Services Research
(cid:36)(cid:3)(cid:6)(cid:21)(cid:11)(cid:2)(cid:401)(cid:1354)Prevalence of sociodemographic characteristics by literacy profile
(cid:11)(cid:55)(cid:134)(cid:49)(cid:45)(cid:124)(cid:98)(cid:111)(cid:109)(cid:332)(cid:7)(cid:111)(cid:1140)(cid:1140)(cid:59)(cid:93)(cid:59)(cid:2)(cid:55)(cid:59)(cid:93)(cid:117)(cid:59)(cid:59)(cid:2)(cid:1143) (cid:33)(cid:45)(cid:49)(cid:59)(cid:332)(cid:41)(cid:95)(cid:98)(cid:124)(cid:59)(cid:2)(cid:1143) (cid:3)(cid:93)(cid:59)(cid:2)(cid:45)(cid:124)(cid:2)(cid:34)(cid:134)(cid:117)(cid:136)(cid:59)(cid:139)(cid:332)(cid:24)(cid:59)(cid:45)(cid:109)(cid:2)(cid:336)(cid:34)(cid:9)(cid:337)
(cid:21)(cid:98)(cid:108)(cid:98)(cid:124)(cid:59)(cid:55)(cid:2) (cid:3)(cid:55)(cid:59)(cid:116)(cid:134)(cid:45)(cid:124)(cid:59)(cid:2) (cid:21)(cid:98)(cid:108)(cid:98)(cid:124)(cid:59)(cid:55)(cid:2) (cid:3)(cid:55)(cid:59)(cid:116)(cid:134)(cid:45)(cid:124)(cid:59)(cid:2)
(cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2) (cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2) P(cid:330) (cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2) (cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2) P(cid:330) (cid:21)(cid:98)(cid:108)(cid:98)(cid:124)(cid:59)(cid:55)(cid:2)(cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2) (cid:3)(cid:55)(cid:59)(cid:116)(cid:134)(cid:45)(cid:124)(cid:59)(cid:2) P(cid:330)
(cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:114)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59) (cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139) (cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139) (cid:136)(cid:45)(cid:1140)(cid:134)(cid:59) (cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139) (cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139) (cid:136)(cid:45)(cid:1140)(cid:134)(cid:59) (cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139) (cid:95)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:1140)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139) (cid:136)(cid:45)(cid:1140)(cid:134)(cid:59)
LP_FK 66.3 60.4 .076 22.0 33.7 <.001 56.60 (11.4) 57.29 (9.89) .305
LP_LD 68.9 59.9 .002 21.4 32.6 <.001 56.72 (9.73) 56.74 (10.0) .966
LP_WQ 60.4 57.4 .070 31.7 34.8 .056 56.71 (9.89) 57.44 (10.2) .032
LP_SR 72.3 53.7 <.001 23.9 36.5 <.001 58.88 (9.98) 55.74 (9.74) <.001
LP_Exp 71.2 57.4 <.001 23.1 33.1 <.001 56.70 (10.2) 57.80 (10.0) <.001
Abbreviations: LP_Exp, Literacy Profile Expert-Rated Health Literacy; LP_FK, Literacy Profile Flesch-Kincaid; LP_LD, Literacy Profile Lexical Diversity;
LP_SR, Literacy Profile Self-Reported Health Literacy; LP_WQ, Literacy Profile Writing Quality; SD, Standard Deviation.
(cid:36)(cid:3)(cid:6)(cid:21)(cid:11)(cid:2)(cid:402)(cid:1354)Associations between five literacy profiles and single-item CAHPS ratings of poor physician communication, diabetes-related
health outcomes (%), and annual health care service utilization—mean visits (SD)
(cid:15)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:111)(cid:134)(cid:124)(cid:49)(cid:111)(cid:108)(cid:59)(cid:118) (cid:21)(cid:98)(cid:124)(cid:59)(cid:117)(cid:45)(cid:49)(cid:139)(cid:2)(cid:114)(cid:117)(cid:111)(cid:61)(cid:98)(cid:1140)(cid:59) (cid:21)(cid:30)(cid:333)(cid:13)(cid:20) (cid:21)(cid:30)(cid:333)(cid:21)(cid:9) (cid:21)(cid:30)(cid:333)(cid:41)(cid:32) (cid:21)(cid:30)(cid:333)(cid:34)(cid:33) (cid:21)(cid:30)(cid:333)(cid:11)(cid:138)(cid:114)
Poor Physician Limited health literacy 12.2 10.6 9.2 13.8 15.5
Communication (%) Adequate health literacy 8.8 9.6 10.7 7.3 11.3
P-value .0919 .5610 .1372 <.001 <.001
Poor medication Limited health literacy 29.0 26.6 23.9 25.6 27.9
adherence (%) Adequate health literacy 22.9 23.8 25.3 23.4 22.9
P-value .043 .277 .364 .047 <.001
(cid:446)(cid:400)(cid:2)(cid:34)(cid:59)(cid:136)(cid:59)(cid:117)(cid:59)(cid:2) Limited health literacy 8.9 4.3 2.9 5.1 4.3
Hypoglycemia (%) Adequate health literacy 3.1 3.3 3.9 2.0 3.4
P-value <.001 .318 .119 <.001 .02
(cid:15)(cid:48)(cid:3)(cid:400)(cid:49)(cid:2)(cid:445)(cid:2)(cid:405)(cid:1143) Limited health literacy 40.2 44.8 47.4 45.9 43.3
Adequate health literacy 48.8 48.5 45.2 47.7 47.4
P-value .011 .202 .193 .141 <.001
(cid:15)(cid:48)(cid:3)(cid:400)(cid:49)(cid:2)(cid:446)(cid:2)(cid:406)(cid:1143) Limited health literacy 19.1 13.3 13.8 14.6 16.4
Adequate health literacy 13.7 13.5 14.6 13.5 13.0
P-value .02 .91 .499 .24 <.001
Charlson Index Limited health literacy 2.61 (1.84) 2.36 (1.69) 2.20 (1.61) 2.65 (1.91) 2.42 (1.79)
Adequate health literacy 2.28 (1.68) 2.31 (1.69) 2.40 (1.72) 2.02 (1.41) 2.32 (1.70)
P-value .004 .636 <.001 <.001 .006
Outpatient clinic Limited health literacy 9.10 (7.37) 9.01 (8.98) 9.45 (9.75) 10.29 (10.7) 9.83 (10.6)
visits Adequate health literacy 9.53 (9.33) 9.61 (10.3) 9.42 (9.53) 9.01 (9.16) 9.68 (9.57)
P-value .479 .301 .931 <.001 .499
ED visits Limited health literacy 0.48 (1.00) 0.47 (1.15) 0.38 (0.94) 0.53 (1.20) 0.47 (1.14)
Adequate health literacy 0.39 (0.94) 0.38 (0.88) 0.43 (0.96) 0.31 (0.76) 0.42 (1.01)
P-value .170 .102 .085 <.001 .016
Hospitalization Limited health literacy 0.23 (0.71) 0.21 (0.61) 0.17 (0.60) 0.25 (0.73) 0.20 (0.65)
Adequate health literacy 0.18 (0.62) 0.19 (0.65) 0.19 (0.67) 0.13 (0.54) 0.20 (0.67)
P-value .243 .604 .503 <.001 .713
Abbreviations: Exp, Expert-Rated; FK, Flesch-Kincaid; LD, Lexical Diversity; LP, Literacy Profile; SD, Standard Deviation; SR, Self-Reported; WQ,
Writing Quality.
(cid:402)(cid:314)(cid:401)(cid:314)(cid:403)(cid:1354)|(cid:1354)(cid:15)(cid:59)(cid:45)(cid:1140)(cid:124)(cid:95)(cid:2)(cid:49)(cid:45)(cid:117)(cid:59)(cid:2)(cid:134)(cid:124)(cid:98)(cid:1140)(cid:98)(cid:140)(cid:45)(cid:124)(cid:98)(cid:111)(cid:109) literacy, LP_SR was the only model that associated inadequate
health literacy with higher rates of outpatient visits and hospi-
Utilizations rates associated with each of the five literacy profiles talizations. Higher annual emergency room utilization rates were
are given in Table 3. For those classified as having limited health observed for limited health literacy when assessed by both LP_SR
