Automatic evaluation of Machine Translation (MT)

131 Pages·2013·1.87 MB·English

Automatic evaluation of Machine
Translation (MT): BLEU, its
shortcomings and other
evaluation metrics
Presented by:
Guide:
(As a part of CS 712)
Prof. Pushpak Bhattacharyya,
Aditya Joshi IIT Bombay
Kashyap Popat
Shubham Gautam,
IIT Bombay
Part I: Introduction and
formulation of BLEU
Presented by:
Guide:
(As a part of CS 712)
Prof. Pushpak Bhattacharyya,
Aditya Joshi IIT Bombay
Kashyap Popat
Shubham Gautam,
IIT Bombay
PPooeettrryy iiss wwhhaatt ggeettss
lloosstt iinn ttrraannssllaattiioonn..””
Robert Frost
Poet (1874 – 1963)
Wrote the famous poem ‘Stopping by woods on a snowy
evening’ better known as ‘Miles to go before I sleep’
Motivation
How do we judge a good translation?
Can a machine do this?
Why should a machine do this? Because humans take time!
Outline
• Evaluation
• Formulating BLEU Metric
• Understanding BLEU formula
Part I
• Shortcomings of BLEU Part II
• Shortcomings in context of English-Hindi MT
K. Papineni, S. Roukos, T. Ward, and W. Zhu. Bleu: a method for automatic
R. AnDaonuthga Akrrnisohlnda, nL,o Puuissah SpaadkleBrh, aatntadc Rh.a Lreyyea H, uMm. pSahsriekyusm, Eavraalunadt iRointe: sahnM .
evaluation of machine translation, IBM research report rc22176 (w0109-
Shah, Saossmeess Imsseunets. iMn aAcuhtionme aTtriacn Esvlaatluioant,i oVno loufm Een g8l,i spha-gHeins d1i– M27T.: 1M9o9r3e. Blues
022). Technical report, IBM Research Division, Thomas, J. Watson Research
for BLEU, ICON 2007, Hyderabad, India, Jan, 2007.
Center. 2001.
BLEU - I
The BLEU Score rises
EEvvaalluuaattiioonn
[[11]]
Of NLP systems
Of MT systems
Evaluation in NLP: Precision/Recall
Precision:
How many results returned were correct?
Recall:
What portion of correct results were returned?
Adapting precision/recall to NLP tasks
Evaluation in NLP: Precision/Recall
• Document Retrieval • Classification
Precision =
Precision =
|True Positives|
|Documents relevant and
|True Positives + False
retrieved|
Positives|
|Documents retrieved|
Recall=
Recall=
|True Positives|
|Documents relevant and
| True Positives + False
retrieved|
Negatives|
| Documents relevant|
Evaluation in MT [1]
• Operational evaluation
– “Is MT system A operationally better than MT system B?
Does MT system A cost less?”
• Typological evaluation
– “Have you ensured which linguistic phenomena the MT
system covers?”
• Declarative evaluation
–
“How does quality of output of system A fare with respect
to that of B?”