Neural Machine Translation

409 Pages·2020·6.056 MB·English

by  Philipp Koehn

Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
Neural Machine Translation
Deep learning is revolutionizing how machine translation systems are
built today. This book introduces the challenge of machine translation
and evaluation – including the historical, linguistic, and applied con-
text – then develops the core deep learning methods used for natural
language applications. Code examples in Python give readers a hands-
on blueprint for understanding and implementing their own machine
translationsystems.Thebookalsoprovidesextensivecoverageofmachine
learning tricks, issues involved in handling various forms of data, model
enhancements, and current challenges and methods for analysis and
visualization. Summaries of the current research in the (cid:30)eld make this
astate-of-the-arttextbookforundergraduateandgraduateclasses,aswell
asanessentialreferenceforresearchersanddevelopersinterestedinother
applications of neural methods in the broader (cid:30)eld of human language
processing.
PHILIPP KOEHN isaleadingresearcherinthe(cid:30)eldofmachinetransla-
tionandprofessorofcomputerscienceatJohnsHopkinsUniversity.In
2010, he authored the textbook Statistical Machine Translation (Cam-
bridge University Press). He received the Award of Honor from the
InternationalAssociationforMachineTranslationandwasoneofthree
(cid:30)nalistsfortheEuropeanInventorAwardoftheEuropeanPatentOf(cid:30)ce
in2013.KoehnalsoworksactivelyinthetechnologyindustryasChief
ScientistforOmniscienTechnologiesandasaconsultantforFacebook.
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
Neural Machine Translation
Philipp Koehn
JohnsHopkinsUniversity
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,
NewDelhi–110025,India
79AnsonRoad,#06–04/06,Singapore079906
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781108497329
DOI:10.1017/9781108608480
©PhilippKoehn2020
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2020
PrintedintheUnitedKingdombyTJInternational,PadstowCornwall
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
LibraryofCongressCataloging-in-PublicationData
Names:Koehn,Philipp,author.
Title:Neuralmachinetranslation/PhilippKoehn,CenterforSpeechand
LanguageProcessing,DepartmentofComputerScience,JohnsHopkinsUniversity.
Description:Firstedition.|NewYork:CambridgeUniversityPress,2020.|
Includesbibliographicalreferencesandindex.
Identi(cid:30)ers:LCCN2019046120(print)|LCCN2019046121(ebook)|
ISBN9781108497329(hardback)|ISBN9781108608480(epub)
Subjects:LCSH:Machinetranslation.|Neuralnetworks(Computerscience)
Classi(cid:30)cation:LCCP308.K6382020(print)|LCCP308(ebook)|
DDC418/.020285–dc23
LCrecordavailableathttps://lccn.loc.gov/2019046120
LCebookrecordavailableathttps://lccn.loc.gov/2019046121
ISBN978-1-108-49732-9Hardback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracy
ofURLsforexternalorthird-partyinternetwebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain,
accurateorappropriate.
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
forLeo,Phianna,andTrishann
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
Contents
Preface pagexi
ReadingGuide xiii
Part I Introduction 1
1 TheTranslationProblem 3
1.1 GoalsofTranslation 4
1.2 Ambiguity 5
1.3 TheLinguisticView 8
1.4 TheDataView 11
1.5 PracticalIssues 15
2 UsesofMachineTranslation 19
2.1 InformationAccess 19
2.2 AidingHumanTranslators 20
2.3 Communication 23
2.4 NaturalLanguageProcessingPipelines 26
2.5 MultimodalMachineTranslation 27
3 History 29
3.1 NeuralNetworks 30
3.2 MachineTranslation 33
4 Evaluation 41
4.1 Task-BasedEvaluation 41
4.2 HumanAssessments 45
4.3 AutomaticMetrics 52
4.4 MetricsResearch 59
Part II Basics 65
5 NeuralNetworks 67
5.1 LinearModels 67
5.2 MultipleLayers 68
vii
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
viii Contents
5.3 Nonlinearity 69
5.4 Inference 70
5.5 Back-PropagationTraining 72
5.6 ExploitingParallelProcessing 79
5.7 HandsOn:NeuralNetworksinPython 80
6 ComputationGraphs 89
6.1 NeuralNetworksasComputationGraphs 89
6.2 GradientComputations 91
6.3 HandsOn:DeepLearningFrameworks 95
7 NeuralLanguageModels 103
7.1 Feed-ForwardNeuralLanguageModels 103
7.2 WordEmbeddings 107
7.3 NoiseContrastiveEstimation 109
7.4 RecurrentNeuralLanguageModels 110
7.5 LongShort-TermMemoryModels 112
7.6 GatedRecurrentUnits 115
7.7 DeepModels 116
7.8 HandsOn:NeuralLanguageModels
inPyTorch 118
7.9 FurtherReadings 123
8 NeuralTranslationModels 125
8.1 Encoder–DecoderApproach 125
8.2 AddinganAlignmentModel 126
8.3 Training 130
8.4 DeepModels 133
8.5 HandsOn:NeuralTranslationModels
inPyTorch 136
8.6 FurtherReadings 141
9 Decoding 143
9.1 BeamSearch 143
9.2 EnsembleDecoding 147
9.3 Reranking 148
9.4 OptimizingDecoding 154
9.5 DirectingDecoding 155
9.6 HandsOn:DecodinginPython 158
9.7 FurtherReadings 163
© in this web service Cambridge University Press www.cambridge.org
Cambridge University Press
978-1-108-49732-9 — Neural Machine Translation
Philipp Koehn
Frontmatter
More Information
Contents ix
Part III Reinements 169
10 MachineLearningTricks 171
10.1 FailuresinMachineLearning 171
10.2 EnsuringRandomness 174
10.3 AdjustingtheLearningRate 176
10.4 AvoidingLocalOptima 179
10.5 AddressingVanishingandExplodingGradients 182
10.6 Sentence-LevelOptimization 186
10.7 FurtherReadings 189
11 AlternateArchitectures 193
11.1 ComponentsofNeuralNetworks 193
11.2 AttentionModels 199
11.3 ConvolutionalMachineTranslation 203
11.4 ConvolutionalNeuralNetworkswithAttention 205
11.5 Self-Attention:Transformer 207
11.6 FurtherReadings 211
12 RevisitingWords 213
12.1 WordEmbeddings 214
12.2 MultilingualWordEmbeddings 220
12.3 LargeVocabularies 224
12.4 Character-BasedModels 229
12.5 FurtherReadings 233
13 Adaptation 239
13.1 Domains 239
13.2 MixtureModels 245
13.3 Subsampling 249
13.4 Fine-Tuning 253
13.5 FurtherReadings 257
14 BeyondParallelCorpora 263
14.1 UsingMonolingualData 264
14.2 MultipleLanguagePairs 269
14.3 TrainingonRelatedTasks 272
14.4 FurtherReadings 274
15 LinguisticStructure 281
15.1 GuidedAlignmentTraining 282
15.2 ModelingCoverage 284
© in this web service Cambridge University Press www.cambridge.org