MATLAB Text Analytics Toolbox™ User's Guide

230 Pages·2020·4.018 MB·English

by  The MathWorks, Inc.

Text Analytics Toolbox™
User's Guide
R2020a
How to Contact MathWorks
Latest news: www.mathworks.com
Sales and services: www.mathworks.com/sales_and_services
User community: www.mathworks.com/matlabcentral
Technical support: www.mathworks.com/support/contact_us
Phone: [phone]
The MathWorks, Inc.
1 Apple Hill Drive
Natick, MA 01760-2098
Text Analytics Toolbox™ User's Guide
© COPYRIGHT 2017–2020 by The MathWorks, Inc.
The software described in this document is furnished under a license agreement. The software may be used or copied
only under the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form
without prior written consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through
the federal government of the United States. By accepting delivery of the Program or Documentation, the government
hereby agrees that this software or documentation qualifies as commercial computer software or commercial computer
software documentation as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014.
Accordingly, the terms and conditions of this Agreement and only those rights specified in this Agreement, shall pertain
to and govern the use, modification, reproduction, release, performance, display, and disclosure of the Program and
Documentation by the federal government (or other entity acquiring for or through the federal government) and shall
supersede any conflicting contractual terms or conditions. If this License fails to meet the government's needs or is
inconsistent in any respect with federal procurement law, the government agrees to return the Program and
Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See
www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be
trademarks or registered trademarks of their respective holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for
more information.
Revision History
March 2018 Online Only New for Version 1.1 (Release 2018a)
September 2018 Online Only Revised for Version 1.2 (Release 2018b)
March 2019 Online Only Revised for Version 1.3 (Release 2019a)
September 2019 Online Only Revised for Version 1.4 (Release 2019b)
March 2020 Online Only Revised for Version 1.5 (Release 2020a)
Contents
Text Data Preparation
1
Extract Text Data from Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Prepare Text Data for Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-10
Parse HTML and Extract Text Content . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-17
Correct Spelling in Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-21
Create Extension Dictionary for Spelling Correction . . . . . . . . . . . . . . . . 1-23
Create Custom Spelling Correction Function Using Edit Distance
Searchers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-27
Data Sets for Text Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-33
Modeling and Prediction
2
Create Simple Text Model for Classification . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Analyze Text Data Using Multiword Phrases . . . . . . . . . . . . . . . . . . . . . . . . 2-7
Analyze Text Data Using Topic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-13
Choose Number of Topics for LDA Model . . . . . . . . . . . . . . . . . . . . . . . . . 2-19
Compare LDA Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-23
Create Co-occurrence Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-28
Analyze Text Data Containing Emojis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-32
Create Simple Preprocessing Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-38
Train a Sentiment Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-41
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-48
Classify Text Data Using Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 2-49
iii
Classify Text Data Using Convolutional Neural Network . . . . . . . . . . . . . 2-57
Multilabel Text Classification Using Deep Learning . . . . . . . . . . . . . . . . . 2-66
Sequence-to-Sequence Translation Using Attention . . . . . . . . . . . . . . . . 2-86
Classify Out-of-Memory Text Data Using Deep Learning . . . . . . . . . . . . 2-106
Pride and Prejudice and MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-112
Word-By-Word Text Generation Using Deep Learning . . . . . . . . . . . . . . 2-118
Classify Out-of-Memory Text Data Using Custom Mini-Batch Datastore
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-124
Display and Presentation
3
Visualize Text Data Using Word Clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Visualize Word Embeddings Using Text Scatter Plots . . . . . . . . . . . . . . . . 3-8
Language Support
4
Language Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Language-Independent Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Japanese Language Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
Part of Speech Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
Named Entity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-6
Stop Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7
Lemmatization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Language-Independent Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Analyze Japanese Text Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10
German Language Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-20
Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-20
Sentence Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-20
Part of Speech Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-21
Named Entity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-22
Stop Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23
Stemming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23
Language-Independent Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24
Analyze German Text Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-25
iv Contents
Korean Language Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
Part of Speech Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
Named Entity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
Stop Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
Lemmatization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
Language-Independent Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
Language-Independent Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-38
Word and N-Gram Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-38
Modeling and Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-38
Glossary
5
Text Analytics Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Documents and Tokens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3
Modeling and Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3
Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5
v

1
Text Data Preparation
• “Extract Text Data from Files” on page 1-2
• “Prepare Text Data for Analysis” on page 1-10
• “Parse HTML and Extract Text Content” on page 1-17
• “Correct Spelling in Documents” on page 1-21
• “Create Extension Dictionary for Spelling Correction” on page 1-23
• “Create Custom Spelling Correction Function Using Edit Distance Searchers” on page 1-27
• “Data Sets for Text Analytics” on page 1-33
1
Text Data Preparation
Extract Text Data from Files
This example shows how to extract the text data from text, HTML, Microsoft® Word, PDF, CSV, and
Microsoft Excel® files and import it into MATLAB® for analysis.
Usually, the easiest way to import text data into MATLAB is to use the extractFileText function.
This function extracts the text data from text, PDF, HTML, and Microsoft Word files. To import text
from CSV and Microsoft Excel files, use readtable. To extract text from HTML code, use
extractHTMLText. To read data from PDF forms, use readPDFFormData.
Text File
Extract the text from sonnets.txt using extractFileText. The file sonnets.txt contains
Shakespeare's sonnets in plain text.
filename = "sonnets.txt";
str = extractFileText(filename);
View the first sonnet by extracting the text between the two titles "I" and "II".
start = " I" + newline;
fin = " II";
sonnet1 = extractBetween(str,start,fin)
sonnet1 =
"
From fairest creatures we desire increase,
That thereby beauty's rose might never die,
But as the riper should by time decease,
His tender heir might bear his memory:
But thou, contracted to thine own bright eyes,
Feed'st thy light's flame with self-substantial fuel,
Making a famine where abundance lies,
Thy self thy foe, to thy sweet self too cruel:
Thou that art now the world's fresh ornament,
And only herald to the gaudy spring,
Within thine own bud buriest thy content,
And tender churl mak'st waste in niggarding:
Pity the world, or else this glutton be,
To eat the world's due, by the grave and thee.

"
Microsoft Word Document
Extract the text from sonnets.docx using extractFileText. The file exampleSonnets.docx
contains Shakespeare's sonnets in a Microsoft Word document.
filename = "exampleSonnets.docx";
str = extractFileText(filename);
View the second sonnet by extracting the text between the two titles "II" and "III".
start = " II" + newline;
fin = " III";
sonnet2 = extractBetween(str,start,fin)
1-2
Extract Text Data from Files
sonnet2 =
"
When forty winters shall besiege thy brow,

And dig deep trenches in thy beauty's field,

Thy youth's proud livery so gazed on now,

Will be a tatter'd weed of small worth held:

Then being asked, where all thy beauty lies,

Where all the treasure of thy lusty days;

To say, within thine own deep sunken eyes,

Were an all-eating shame, and thriftless praise.

How much more praise deserv'd thy beauty's use,

If thou couldst answer 'This fair child of mine

Shall sum my count, and make my old excuse,'

Proving his beauty by succession thine!

This were to be new made when thou art old,

And see thy blood warm when thou feel'st it cold.

"
The example Microsoft Word document uses two newline characters between each line. To replace
these characters with a single newline character, use the replace function.
sonnet2 = replace(sonnet2,[newline newline],newline)
sonnet2 =
"
When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gazed on now,
Will be a tatter'd weed of small worth held:
Then being asked, where all thy beauty lies,
Where all the treasure of thy lusty days;
To say, within thine own deep sunken eyes,
Were an all-eating shame, and thriftless praise.
How much more praise deserv'd thy beauty's use,
If thou couldst answer 'This fair child of mine
Shall sum my count, and make my old excuse,'
Proving his beauty by succession thine!
This were to be new made when thou art old,
And see thy blood warm when thou feel'st it cold.
"
1-3
1
Text Data Preparation
PDF Files
Extract text from PDF documents and data from PDF forms.
PDF Document
Extract the text from sonnets.pdf using extractFileText. The file exampleSonnets.pdf
contains Shakespeare's sonnets in a PDF.
filename = "exampleSonnets.pdf";
str = extractFileText(filename);
View the third sonnet by extracting the text between the two titles "III" and "IV". This PDF has a
space before each newline character.
start = " III " + newline;
fin = "IV";
sonnet3 = extractBetween(str,start,fin)
sonnet3 =
"
Look in thy glass and tell the face thou viewest
Now is the time that face should form another;
Whose fresh repair if now thou not renewest,
Thou dost beguile the world, unbless some mother.
For where is she so fair whose unear'd womb
Disdains the tillage of thy husbandry?
Or who is he so fond will be the tomb,
Of his self-love to stop posterity?
Thou art thy mother's glass and she in thee
Calls back the lovely April of her prime;
So thou through windows of thine age shalt see,
Despite of wrinkles this thy golden time.
But if thou live, remember'd not to be,
Die single and thine image dies with thee.

"
PDF Form
To read text data from PDF forms, use readPDFFormData. The function returns a struct containing
the data from the PDF form fields.
filename = "weatherReportForm1.pdf";
data = readPDFFormData(filename)
data = struct with fields:
event_type: "Thunderstorm Wind"
event_narrative: "Large tree down between Plantersville and Nettleton."
HTML
Extract text from HTML files, HTML code, and the web.
1-4