Analyzing the Differences Between Human and Machine Translation of Selected Texts

291 Pages·2017·21.7 MB·English

by  Nicole Howell

SSyyrraaccuussee UUnniivveerrssiittyy
SSUURRFFAACCEE
Syracuse University Honors Program Capstone Syracuse University Honors Program Capstone
Projects Projects
Spring 5-2016
AAppppllyyiinngg LLiinngguuiissttiiccss:: AAnnaallyyzziinngg tthhee DDiiffffeerreenncceess BBeettwweeeenn HHuummaann
aanndd MMaacchhiinnee TTrraannssllaattiioonn ooff SSeelleecctteedd TTeexxttss
Nicole Howell
Follow this and additional works at: https://surface.syr.edu/honors_capstone
Part of the Spanish and Portuguese Language and Literature Commons
RReeccoommmmeennddeedd CCiittaattiioonn
Howell, Nicole, "Applying Linguistics: Analyzing the Differences Between Human and Machine Translation
of Selected Texts" (2016). Syracuse University Honors Program Capstone Projects. 940.
https://surface.syr.edu/honors_capstone/940
This Honors Capstone Project is brought to you for free and open access by the Syracuse University Honors Program
Capstone Projects at SURFACE. It has been accepted for inclusion in Syracuse University Honors Program Capstone
Projects by an authorized administrator of SURFACE. For more information, please contact [email protected].
Applying Linguistics: Analyzing the Differences Between Human and Machine Translation of
Selected Texts

A Capstone Project Submitted in Partial Fulfillment of the
Requirements of the Renée Crown University Honors Program at
Syracuse University

Nicole Howell

Candidate for Bachelor of Arts
and Renée Crown University Honors
May 2016

Honors Capstone Project in Spanish Language, Literature, and Culture

Capstone Project Advisor: _______________________
M. Emma Ticio Quesada,
Associate Professor

Capstone Project Reader: _______________________
Catherine M. Nock, Senior
Lecturer

Honors Director: _______________________
Stephen Kuusisto, Director

ii
© (Nicole Howell, 2016)
iii
Abstract
Machine translation systems experience a trade-off between efficiency and efficacy:
while quicker and easier than human translation, machine translations are less accurate and
comprehensible.
This study analyzes two sets of English to Spanish translations to compare the differences
between human and machine translation and evaluate the performance of machine translation.
Google Translate—the best-performing machine translator—represents machine translation as a
whole in producing English to Spanish translations of the given text. The primary investigator—
a non-native speaker and graduating student of Spanish Language, Literature & Culture at
Syracuse University—serves as the benchmark for the average non-native speaker of Spanish at
the advanced-mid level of proficiency. Findings are divided into four groups: lexical findings,
sentential findings, phrasal findings, and stylistic findings.
From these findings, the study is able to establish that machine translation struggles to
decipher context, which results in errors in word choice and lack of proper technical language;
displays inconsistency in its errors, and lacks the capability to learn. Such individual findings
led to the overall conclusion that Google Translate—and thus machine translation—performs
below the level of an average non-native speaker of Spanish.

iv
Executive Summary

The following Capstone Project compares and evaluates the functionality and efficacy of
human and machine translation when translating from English to Spanish. This study attempts to
determine if Google Translate—the machine translation engine deemed superior among its
competitors—operates as effectively as a human translator for English to Spanish translations.
The human translations—generated by the primary investigator—serve as a benchmark for
translations carried out by a non-native speaker at an advanced level of proficiency. According
to the American Council on the Teaching of Foreign Languages (ACTFL), a non-native speaker
at this level can not only speak about a variety of topics, but also delivers information clearly,
and with little confusion. This study compares the understandability and the comprehensibility
of the translations by Google with the human translations to determine if the popular search
engine operates below, at, or above this level.
Previous studies have focused on the relative accuracies of different machine translation
systems, the strength of particular language pairings, and the relationship between accuracy and
understandability of the target language to native speakers. Such research is meant to contribute
to the understanding of the functionality of machine translation, as well as advance its
development. The motivation for this Capstone Project, however, has many origins: to answer
the overarching question about the accuracy of machine translation as it relates to the needs of
the general public (such as travelers or business people), to use new methods to evaluate machine
translation, and to extrapolate pedagogical conclusions that could ultimately impact students of
foreign language.
The need to translate between languages quickly and easily is also evident outside the
academic setting. In an increasingly connected world, language barriers have become more
v
prevalent. Students, travelers, and business people alike require ways to overcome this obstacle
quickly and easily. More often than not, machine translation is used instead of human translation
to hurdle these language barriers. Between the two methods, we observe a trade-off between
quality and quantity: human translation, while more accurate, cannot translate large volumes of
text as quickly as a machine can. As the demand for accurate machine translation has grown, so
has research and development. For more than forty years, computer scientists and linguists have
dedicated their research to diagnosing the weaknesses of machine translators, and creating
systems that will perform more quickly and with higher accuracy. Today, machine translation is
widely considered an acceptable means to garner the “gist” of a given text, even though it cannot
yet generate a translation whose accuracy and quality is equal to that of a human translator.
Among the present systems, the results of previous studies have favored Google Translate over
other machine translation platforms.
Overall, computer scientists and linguistic researchers have come to the conclusion that
machine translation does not yet match the level of a human translator. Previous studies have
focused largely on statistical analyses of a variety of factors of machine translation, such as the
efficacy of one language pair against another (i.e. Chinese/English compared to Spanish/English)
or the efficacy of one translation engine versus another (i.e. Google Translate versus
SYSTRAN). Such studies analyzed their findings quantitatively, generally using a smaller
sample of translated texts as their corpora. Presently, the linguistic field lacks research that
focuses on the qualitative differences of human translation and machine translation of a larger
corpus. Comparing these translations at the qualitative level provides a more comprehensive
understanding not only of how well machine translation performs, but in what grammatical areas
vi
it excels or struggles. Given the frequency with which students use machine translation in
academic settings, it is crucial to also investigate the pedagogical implications of its use.
For the purposes of comparison, this project utilizes a relatively large corpus:
approximately 60 pages of material. Animal Fact Sheets from the Rosamond Gifford Zoo in
Syracuse, New York—the selected texts for this study—fulfilled the need for a large corpus
while also providing a good example of technical writing. Additionally, the 29 human Spanish
translations of the animal fact sheets will benefit the zoo and their visitors, as they will be made
available to the public on the Zoo’s website.
The corpora were translated twice: first by the primary investigator and then by Google
Translate, from English to Spanish. A set of revised translations, which served as the basis for
comparison, were provided by Professor María Emma Ticio Quesada, Associate Professor of
Spanish and Linguistics at Syracuse University. The primary investigator edited the machine
translations based on the revised translations, and used the revisions to analyze the discrepancies
between human and machine translation as a whole. The qualitative findings were divided into
four linguistic categories—lexical, sentential, phrasal, and stylistic—and analyzed per the
grammatical parameters of each category.
The results of this study led to the following conclusions. First and foremost, this study
reinforces the idea that Google Translate, and similar machine translators, are best used to get the
general idea of a selected passage, and should not be relied on to generate completely
understandable or professional-quality translations. Primarily, the results show that Google
struggles to identify context, which result in errors in word choice that affect understandability.
Additionally, and surprisingly, Google seems unable to apply some of the most basic
vii
grammatical rules of the target language—such as inverse noun/adjective word order—that most
students learn in their introductory courses.
The results also highlight an important difference between human and machine
translation: the ability to learn. As the human translations progress, it is evident that the primary
investigator was learning while translating; becoming more comfortable with the target language
and its grammar, as well as with the technical language required to communicate effectively
about zoo animals. Google, by contrast, showed no such improvements. What is more, Google
lacked consistency in its shortcomings throughout the translations, deeming its functionality not
only inaccurate, but also unreliable and unpredictable. Finally, these findings led to the
determination that Google Translate performs below the average level of a human translator. In
other words, Google is not able to communicate about a variety of topics effectively and clearly.
Ultimately, it is the goal of the primary investigator to both make a significant
contribution to the preexisting research in the linguistic field regarding machine translation, and
to form applicable pedagogical conclusions that will benefit students of foreign language. For
these students, the most important conclusion to draw from this study is as follows: machine
translation, after decades of research and development, continues to perform at a level of
accuracy below that of a human. It is a system that cannot learn, and when it fails, it does not do
so consistently or with any apparent predictability. While human translation cannot be as easily
accessed as machine translation, it is important not to sacrifice quality for convenience. With
further studies in this field, it is possible that one day students will not have to choose between
the two.

viii
Table of Contents
Abstract……………………………………….……………….………….. iv

Executive Summary………………………….……………….………….. v

Acknowledgements …………..…………………………………………… x

Chapter 1: Introduction ……………………………………………… 1
State of the Art ……………………………………. 1
Scope of the Project ……………………………………. 7
Research Questions ……………………………………. 7
Expectations ……………………………………. 8

Chapter 2: Methods ……………………………………………… 9
Corpora ……………………………………. 9
Translation Methods ……………………………………. 10

Chapter 3: Results ……………………………………………… 12
Lexical Findings ……………………………………. 12
Sentential Findings ……………………………………. 14
Phrasal Findings ……………………………………. 17
Stylistic Findings ……………………………………. 20

Chapter 4: Conclusions ……………………………………………… 22
General Discussion ……………………………………. 22
Pedagogical Implications ……………………………………. 27
Future Research ……………………………………. 28

Works Cited .……………………………………………………………… 31
Appendices …………………………………………………………………
Appendix A: Original Fact Sheets ………………………………. 33
Appendix B: Human Translations ………………………………. 97
Appendix C: Machine Translations ………………………………. 180

Appendix D: Clean Translations …………………………………. 264

Appendix E: Revised Human Translations ………………………. 282
Appendix F: Corrected Machine Translations ……………………. 300

ix
Acknowledgements

This Capstone Project was made possible through the collective efforts of the Renée
Crown University Honors Program, the Spanish Department within the Languages, Literatures
and Linguistics Department at Syracuse University, and the Rosamond Gifford Zoo at Burnett
Park.
I would like to specially thank my Capstone Advisor, Professor Emma Ticio, and
Capstone Reader, Professor Catherine Nock for their professional insight and support. I would
also like to extend my sincerest gratitude to my Honors Capstone Coordinator and long-time
academic advisor, Karen Hall, for her advice and patience, and for allowing me to change my
mind until the last possible second.
Additional thanks to the Rosamond Gifford Zoo and their Small Mammals Department,
especially Adrienne Whiteley, for their permission to use their Animal Fact Sheets as the corpora
for this study.

x