Table Of ContentInt. J. Human-Computer Studies (2000) 53, 827}850
doi:10.1006/ijhc.2000.0419
Available online at http://www.idealibrary.comon
Turning pictures into numbers: extracting and
generating information from complex visualizations
J. GREGORYTRAFTON
HCI Laboratory, Naval Research Laboratory, Code 5513, 4555 Overlook Av. S.W.,
Washington, DC 20375-5337, USA. email:[email protected].
SUSANS. KIRSCHENBAUM
Naval ;ndersea =arfare Center Division, Newport, RI, ;SA
TEDL. TSUI
Marine Meteorology Division, Naval Research ‚aboratory, Monterey, CA, ;SA
ROBERTT. MIYAMOTO
Applied Physics ‚aboratory, ;niversity of =ashington, Seattle, =A, ;SA
JAMESA. BALLAS
HCI ‚aboratory, Naval Research ‚aboratory, Code 5513, 4555 Overlook Av. S.=.,
=ashington, DC 20375-5337,;SA
PAULAD. RAYMOND
Center for Applied Research, Great Falls, <A, ;SA
(Received 14 February 2000, and accepted in revised form 31 May 2000)
Wepresentastudyofcomplexvisualizationusagebyexpertmeteorologicalforecasters.
Weperformedaprotocolanalysisandexaminedthetypesofvisualizationstheyexam-
ined.Wepresentevidenceforhowexpertsareabletomakeuseofcomplexvisualizations.
Our"ndingssuggestthatusersofcomplexvisualizationscreatequalitativementalmodels
from which they can then generate quantitative information. In order to build their
qualitative mental models, forecasters integrated information across multiple visuali-
zations and extracted primarily qualitative information from visualizations in a goal-
directedmanner.Wediscussboththeoreticalandpracticalimplicationsofthisstudy.
(2000AcademicPress
1. Introduction
Everywhere you look, newspapers and banners declare this the &&Information Age.’’
Scientists,engineersandweatherforecasters(justtonameafew)examineextremelylarge
amounts of data on a daily basis. The visualizations that they examine have so much
data that it is obvious that they cannot understand all the data all the time. How do
expertsdealwiththislargeamountofinformation?Thispaperpresentsastudyshowing
1071-5819/00/110827#24$35.00/0 ( 2000AcademicPress
Report Documentation Page Form Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it
does not display a currently valid OMB control number.
1. REPORT DATE 3. DATES COVERED
2000 2. REPORT TYPE 00-00-2000 to 00-00-2000
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
Turning pictures into numbers: extracting and generating information
5b. GRANT NUMBER
from complex visualizations
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
Naval Research Laboratory,HCI Laboratory,4555 Overlook Avenue REPORT NUMBER
SW,Washington,DC,20375
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution unlimited
13. SUPPLEMENTARY NOTES
14. ABSTRACT
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF
ABSTRACT OF PAGES RESPONSIBLE PERSON
a. REPORT b. ABSTRACT c. THIS PAGE Same as 24
unclassified unclassified unclassified Report (SAR)
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std Z39-18
828 J.GREGORYTRAFTONE„A‚.
how expert weather forecasters extract information from complex visualizations,
integratethatinformationintoaqualitativementalmodel,andthenusethatinformation
to make a (mostly) quantitative forecast.
Howdopeoplecomprehendandusecomplexvisualizations?Verylittletheoreticalor
empirical work has been done on complex visualization comprehension, but there is
asizableresearchbodyongraphcomprehension.Bertin(1983)presentedataskanalysis
ofhowgraphsarecomprehended.Agreatdealofthepastresearchongraphcomprehen-
sion,however,hasfocusedontheencodingandperceptionofthegraph.Clevelandand
McGill,forexample,haveexaminedthepsychophysicalaspectsofgraphicalperception
(Cleveland & McGill, 1984, 1986). Similarly, Pinker’s theory of graph comprehension,
whilequitebroad,focusesontheencodingandunderstandingofgraphs(Lewandowsky
& Spence,1990; Pinker, 1990). Kosslyn’swork emphasizesthe cognitive processesthat
make a graph more or less di$cult to read, while focusing on the encoding and
perception of graphs (Kosslyn, 1989).
Unfortunately,thegeneralizabilityofpastresearchongraphcomprehensiontomore
complex visualizations is unclear. Most of the past research has focused on arti"cial
situationsandarti"cialdomainswheretheonlyusableinformationavailabletopartici-
pantsis the graph itself. For example, a traditionaltask given to participants in graph
comprehensionexperimentsis to readinformationdirectlyo! ofagraphofa made-up
company or product (Carter, 1947; Sparrow, 1989; Pinker, 1990; Lohse, 1993). In
contrast,real-worldusersofcomplexvisualizationsnotonlyhaveagreatdealofdomain
knowledgebuthavetodecidehowandwhatinformationtovisualize,whatinformation
toextractandwhattodowiththatknowledge.Usersofcomplexvisualizationsalsohave
particular reasons for extracting information. They use that information for deducing,
generalizing, extrapolating and predicting (to name just a few goals).
Additionally,moststudieshavefocusedondata spacesthatare purposelylimited to
2 or 3 variables (Simkin & Hastie, 1987; Casali & Gaylin, 1988; Sparrow, 1989).
Experimenters limit their data spaces to 2 or 3 variables to retain experimental
control. This experimental control is not only necessary for performing good science,
but it is critical to understanding how encoding and perception of graphs occurs. If
complexgraphs and visualizationswereused, the standard participantsin experiments
(undergraduates), would probably be very confused. In contrast, many scienti"c and
engineering disciplines use complex data spaces that have many variables over a wide
range of scales and across time. For example, it is not uncommon for a typical
meteorologicalforecasting chart (shown in Figure1) to contain "ve or more variables.
Figure1,forexample,representsatleastsixvariables:(1)Thelatitude/longitude,(2)the
geographicalregion(southernCalifornia),(3)windspeed,(4)winddirection,(5)temper-
atureand(6)barometricpressure.Ifthisinformationisexaminedacrosstimeormultiple
altitudes are examined (as happened in this study), the number of variable greatly
increases.
Tostudyhowcomplexvisualizationsarecomprehendedandused,webelievethatitis
necessary to study use of complex visualizations &&in the wild’’ (e.g. Hutchins, 1995)
or &&in vivo’’ (e.g. Dunbar, 1995, 1996; Trickett, Trafton & Schunn, 2000; Trickett, Fu,
Schunn & Trafton, 2000) in addition to studying situations where there is a great
dealoflaboratorycontrol(e.g.Lohse,1993;Shah&Carpenter,1995;Carpenter&Shah,
1998).
TURNINGPICTURESINTONUMBERS 829
FIGURE1. Asamplescreenshotofatypicalvisualizationthatforecastersused.Thisvisualizationwasmade
usingaCOAMPSmodelrunandshowswindspeedanddirection,temperatureandpressureatthesurface.The
originalpictureisincolor.
Weproposethatusersofcomplexvisualizationsuseheuristicstohelpthemdealwith
thelargeamountofdatatheyhaveaccessto.Webelievethatthreeoftheseheuristicsare
(1)attendingtoinformationinagoal-drivenmanner;(2)integratinginformationacross
visualizationsand(3)buildingaqualitativementalmodelfromwhichtheycanabstract
quantitative information.
Weatherforecastingisanappropriatedomaininwhichtostudythesephenomenafor
several reasons. First, weather forecasting is extremely complex and the visualizations
thatforecastersuseoftenhaveatleast"vedi!erentvariablesin additiontospatialand
temporalinformation.However,forecastersareamongthebestdecisionmakers,making
accurateforecastsamajorityofthetime(e.g.itrains70%ofthetimesthattheyforecast
a 70% chance of rain; Murphy & Winkler, 1992). There are computational weather
models that make forecasts, but their interpretation requires a great deal of training,
knowledge and information. In addition, the weather models need to be updated with
currentdatafrequentlyandre-run.Weathermodelsrarelyusedatathatisasuptodate
as forecasters would like.
2. Method
The experiment was an exploratory investigation with some of the characteristics of
a"eldstudy.Wewereunabletocontroltheactualweatherandtheoccasionalcomputer
problemssuchasWorldWideWeb(WWW)sitesbeingdown,computercrashesandthe
830 J.GREGORYTRAFTONE„A‚.
like. These are realistic problems normally encountered by METOC (METeorological
and OCeanographic)forecasters.
2.1. TASK
The task assigned to the forecasters was to prepare a written information brief for an
airplane #own from an aircraft carrier at a given o!shore location 12h in the future.
Theyweregiven#ightpath,altitudes,destinations,alternateairportsandexpectedtimes
ofarrival.ThedestinationwasWhidbeyIsland,Washington,DC.Thebriefwastocover
theentireroundtripwithprimaryemphasisoverthedestination.Theintendedaudience
was the #ight coordinator and the pilots, several hours before takeo!. The forecasters
were requested to provide speci"c data for departure, enroute, destination, tanker
refueling(ifapplicable),recoveryandalternateair"eld.Forecastersalsoprovidedastro-
nomicaldataforsolarandlunarconditions.Thus,theforecastersweretoldtodetermine
detailedqualitativeandquantitativeinformationonwhattheweatherwouldbe12hin
thefutureandtowritethatinformationinastandardbrie"ngpackage.Thistypeoftask
isroutineforNavyandMarineforecasters;theywerenotbeingaskedtodoanythingout
of the ordinary, and all forecasters felt comfortable with their task. It should also be
noted that not every forecaster "lled in every piece of information (due to time con-
straints). Table 1 shows the information requested as well as a sample of the kind of
information that was written down on the brief.
2.2. PARTICIPANTS
Theparticipantsamplewasrepresentativeof therangeofexpertiseandtrainingwithin
the population. Participants were selected on the basis of scheduling availability. Two
individuals participated in each session. All forecasters were Naval or Marine Corps
forecastersandforecasters-in-training.Allhadcompletedatleastthe"rst-levelweather
school. They ranged in forecasting experience from 1 to 16 years with the more
experiencedindividualineachsessionservingasleadforecasterandthelessexperienced
serving as technician. All forecasters had signi"cant operational experience. Table
2 shows the quali"cations of the forecasters and technicians who participated in this
study.Becauseoftheforecasters’trainingandexperience,wecategorizedthemasexperts
(Hayes, 1985; Ericsson, Krampe & Tesch-Roemer, 1993). In addition, two senior
METOCO$cersplayedtheroleoftheregionalcenterandofanintelligenceo$cer.The
regionalcenteristypicallysta!edwithexperiencedforecasters;itistheirjobtohelpout
thelocalforecastersontheshipandprovideadvancedand/orspecializedvisualizations.
In this case, the center was available to provide this help to the forecasters in the
study.Theintelligenceo$cerprovideddetailedinformationaboutothermilitary-related
questions.
Boththeforecasterandthetechnicianprovidedtalk-aloudprotocols(Ericsson&Sim-
on, 1993). Participants were asked to talk aloud as they were working through their
forecasting task. If they needed to speak to someone else (usually their partner), they
stopped the talk-aloud protocol; when they went back to working on their own, they
continuedtheirverbalprotocol.Threepairsofforecastersperformedtheforecastingand
brie"ng tasks.
TURNINGPICTURESINTONUMBERS 831
TABLE 1
A partial description of weather information that participants were asked to provide for
a brief to pilots. „his information comes from several di+erent briefs
Area Informationto"llin Example
Weathersynopsis Mostlyclear
Destination Winddirectionandspeed
(altitudetosource) 080:24010050:28005(etc.)
weather Temperaturemax/min Max:173/Min:113
Cloudbases/heights/amounts 0
Sunrise 0544
Sunset 2055
Astronomical Sunangle !203
data Azimuth 3403
Moonrise 2112
Moonset 0556
Illumination 99%
Skies Mostlycloudy;bases010}020@
Sourcewind 290/04KTS
Departure Visibility Unrestricted/07
weather Turbulence NIL
Icing NIL
weatherwarnings/advisory MWA27C
Flightlevel050 300/07KTS
Flightlevel100 250/26KTS
Flightlevel150
Enroute Flightlevel200 240/30KTS
weather Flightlevel250
Flightlevel300 260/19KTS
Turbulence Lgtsfc-140
Icing NIL
Tankerweather Synopsis Bases:BKN010Tops:025
Recovery Skies BKN-OVC010/015CASEIII
weather Winds 32005KTS
Visibility 5
Alternateair"eld
weather Synopsis Mostlyclear
Water Withsurvivalsuit 5.3h
survivability Withoutsurvivalsuit 1.3h
General Remarks
832 J.GREGORYTRAFTONE„A‚.
TABLE 2
Forecaster and technician quali,cations
Session Job YearsMETOCexperience No.ofbriefs/years
1 Forecaster 9 100
1 Technician 3 50
2 Forecaster 11 180
2 Technician 5 180
3 Forecaster 16 600
3 Technician 1 200
FIGURE2. Asimpleschematicofhowtheroomwheredatawerecollectedwasarranged.Boththeforecaster’s
and the technician’s monitors could display all visualizations. , video camera; , computer terminal;
,SMEwithnotepad.
2.3. SETTINGANDAPPARATUS
Theexperimentalsessionstookplaceinalargeroomwithtablesacrossfromeachother.
On one side of the room two Windowns-NTworkstations were arranged side by side.
Theforecasterandtechnicianworkedonthesetwocomputers.Inaddition,therewasone
subject matter expert (SME) standing behind the forecaster and one SME standing
behindthetechnician.ThetwoSMEstooknotesaboutwhattheythoughtthepairwas
doing.Ontheoppositesideoftheroomwerethreeworkstationswhichwereusedasthe
regionalcenter.OneoftheseniorMETOCo$cerswasinchargeoftheregionalcenter.
One of the center’s computers was connected to the forecaster’s workstation by the
securemilitarylinkcalledSIPERNET.Theotherworkstationswere(1) anunclassi"ed
workstation to draw information from the unclassi"ed Internet and (2) a workstation
withthemeso-scaleforecastingtool,TAMS/RTloadedonit.TAMS/RTisdescribedin
more detail below. All communications between the simulated shipboard and the
simulatedRegionalCenterwere carried out overa chat tool familiarto all, called IRC
Chat. Figure 2 shows a simple schematic of this layout.
TURNINGPICTURESINTONUMBERS 833
2.4. FORECASTINGBACKGROUND
There were two primary programs that forecasters used: Joint METOC Viewer (JMV)
and various satellite images viewed over the World Wide Web. JMV allows weather
model data to be visualized in a variety of ways. There are several types of weather
model data. The two weather models that were used most often by the forecasters
in this study were COAMPS and NOGAPS. COAMPS and NOGAPS describe and
predict di!erent atmospheric features in terms of their scales. NOGAPS resolves
large-scale weather features (e.g. cold fronts and warm fronts), while COAMPS can
resolve "ne scale (or meso-scale) weather features (e.g. thunderstorms). Earth terrain,
of course, a!ects the weather signi"cantly. Local terrain features can in#uence the
meso-scale weather features, and large mountain ranges can modify the large-scale
weather events. It typically takes hours to run each of these weather models, so no
model is perfectly up to date, since it will be based on data that is several hours
old (at least).
Afterthetypeofweathermodeldatahasbeendownloaded,JMVprovidesagraphical
userinterface(GUI)toinspect, examineandanimatethedata. A user mayselectup to
"ve variables at a time (wind speed and direction, pressure or temperature at various
heights, etc.) and view those variables over a particular geographical area. It is also
possible to extract exact information on those variables at a particular latitude and
longitude by moving the mouse over a location and reading o! text information that
appearsoverthevisualization.JMValso allowsuserstocyclethroughtimetoseehow
di!erentvariableschangeovertime.Figure1showsascreensnapshotofatypicalJMV
visualization.Figures3and4showtwokindsofvisualizationsthatforecastersexamined
to compare the di!erences between winds at di!erent heights.
Satellitepicturesareavailableinthreedi!erentkinds:visual(essentiallyabigcamera
lookingdownontheearth),Infraredradiation(IR)and6.75lmchannel(theamountof
water vapor in the atmosphere). It should be emphasized that JMV provides weather
modeldatafromwhichpredictionsofthefutureweatheraretobemade.However,JMV
isonlyasgoodastheinputsandtheweathermodelbeingrun.Sometimes,ofcourse,the
weathermodelscanbedramaticallywrong.Satelliteimagesshow&&truth’’asitwassome
timeintherecentpast,butthereisnocapabilityforpredictioninsatellitevisualizations.
In additionto JMV andvarious satellite pictures,a numberof special-purposetools
were available to the forecasters.
fEOTDA, winEOTDA and TAWS, used to determine the Electro-Optical ranges for
a variety of sensors.
fAREPS, an electro-magnetic (EM) e!ects decision aid.
fSolar/Lunar Almanac Program (SLAP) for sunrise, sunset and illumination calcu-
lations.
fTAMS/RT,theon-scenemeso-scalenumericalweatherpredictionsystem.Itiscapable
of resolving meso-scale features and to be run locally at a very "ne resolution.
Video recorders were positioned to capture the forecaster’s and the technician’s
computer screens. A third video recorder captured interactions between the two
forecasters.
834 J.GREGORYTRAFTONE„A‚.
FIGURE3. Asamplescreenshotofatypicalvisualizationthatforecastersused.Thisvisualizationwasmade
using a COAMPS model run and shows wind speed and direction and relative humidity at 850mb. The
originalpictureisincolor.
2.5. PROCEDURE
Each trial was run with slightly di!erent task parameters. The task di!ered slightly
because the daily weather conditions at the destination location di!ered because real
weather data and weather changes were used (i.e. a forecast was made using current
weather, not a simulation or past data).
Each session began with a description of the task by the Intelligence O$cer, giving
destination,timesandotherinformationasneeded.Thisinformationwasalsorecorded
onawhiteboardthatwasvisibletoboththeforecasterandthetechnicianforreference
whenneeded.ForecasterscreatedbriefsusingeitherPowerPointorsimplywrotedown
the information on paper.
The forecaster served as leader during the session, requesting information from the
technician as needed. As the technicians di!ered widely in experience, the forecasters
sometimesgavedetailedinstructionson whereto "ndspeci"cinformation,whichtools
touse and howto use the tools. In other groups,the forecasterand technicianworked
relatively independently. This type of interaction was in keeping with the way that
METOC o$cers operate ship-board.
For example, all forecasters would ask their technician to perform simple tasks like
"nding a good satellite image that showed the Whidbey Island area. The technician
would work on that task for a while, and then after a good satellite image had been
found, the forecaster would look over the technician’s shoulder and discuss di!erent
TURNINGPICTURESINTONUMBERS 835
FIGURE4. Asamplescreenshotofatypicalvisualizationthatforecastersused.Thisvisualizationwasmade
using a COAMPS model run and shows wind speed and direction and relative humidity at 925mb. The
originalpictureisincolor.
features of the image or forecasting task. The communicative aspects of this task are
discussed more fully in a separate report (McFarlane & Trafton, in preparation).
Forecasterswere given 2h to complete their forecast and brief. Everyone completed
thetaskwithinthetimeallotted.Whentimeallowed,theforecastergavehisbriefatthe
end of the session. All sessions were concludedwith a debrief during which the experi-
mentershadanopportunitytoaskquestionsandtheparticipantshadanopportunityto
give feedback to the experimenters.
2.6. COGNITIVETASKANALYSIS
Acognitivetaskanalysis(CTA)wasperformed.Theoverallimportanceofthestructure
of the task is re#ected in later analyses: all later analyses will be structured around
the CTA.
For the sake of brevity, only a high-level description will be provided.
Forecasters went through four stages to complete their task: (1) initialize their un-
derstandingofthelargescale(global,wholeUS,etc.)weather;(2)buildaqualitativemental
model (QMM) of the weather; (3) verify and adjust their QMM and (d) write the brief.
A brief description of this process follows.
2.6.1. Initializationoflarge-scaleweatherpicture. First,theforecasterlooksatthelarge-
scale weather to see if there are any fronts, anomalies, etc. that they should be
