Slides - Department of Linguistics - The Ohio State University

30 Pages·2013·40.96 MB·English

Describing objects in visual scenes
Is visual salience like conversational salience?
Micha Elsner
Hannah Rohde, Alasdair Clarke
DepartmentofLinguistics
TheOhioStateUniversity
UniversityofEdinburgh
“Describe the person in
the box so that someone
could find them”
2
(cid:73) To the right of the men
smoking a woman wearing
a yellow top and red skirt.
(cid:73) woman in yellow shirt, red
skirt in the queue leaving
the building
(cid:73) the woman in a yellow
short just behind the spray
of the hose
(cid:73) Between the yellow and white airplanes there is a red
vehicle spraying people with a hose. The people getting
sprayed have a small line behind them. In the line there is
a woman with brownish red hair, a yellow shirt and a red
skirt holding a purse. She is standing behind a man
dressed in green.
3
Relational descriptions
“The woman standing
near the jetway”
(cid:73) Overall target:
(cid:73) “the woman”
(cid:73) Landmark:
(cid:73) “the jetway”
(cid:73) relative to “woman”
4
Motivation:
(cid:73) Information structure via discourse salience:
(cid:73) Familiar / important / in common ground
(cid:73) Image understanding via visual salience:
(cid:73) Perceptually apparent / attracts attention
(cid:73) What do they have in common?
This study:
(cid:73) Complex information structure of relational
descriptions
(cid:73) Visual features matter...
(cid:73) Visual salience is like discourse salience
5
Overview
Ordering strategies in the corpus
“Where’s Wally”: the dataset
Learning to use visual features
Experiments: predicting the order
6
Ordering strategies: direction
right
The woman standing near the jetway
left
Near the hut that is burning, there is a man...
inter
Man... next to railroad tracks wearing a white coat
(cid:73) Orders defined WRT first mention
(cid:73) Information structure, not syntax
7
Basic ordering
(cid:73) RIGHT default for landmarks (40%)
(cid:73) LEFT default for image regions (57%)
(cid:73) “On the left is a woman”...
(cid:73) Other orders are marked:
(cid:73) LEFT landmarks (33%)
(cid:73) INTER landmarks (27%)
8
Non-relational mentions
Look at the plane. This man is holding a box
that he is putting on the plane.
(cid:73) First mention isn’t relational
(cid:73) “There is”, “look at”, “find the”...
(cid:73) Annotated as ESTABLISH construction
(cid:73) Usually occurs with LEFT ordering
9
Where’s Wally: the dataset
By Martin Handford: Walker Books, London
(cid:73) Published in US as “Where’s
Waldo”
(cid:73) Series of childrens’ books: a
game based on visual search
(cid:73) Gathered referring
expressions through
Mechanical Turk
(cid:73) Each subject saw a single
target in each image
10