MATLAB Data Import and Export - MathWorks

444 Pages·2016·4.42 MB·English

®
MATLAB
Data Import and Export
R2020a
How to Contact MathWorks
Latest news: www.mathworks.com
Sales and services: www.mathworks.com/sales_and_services
User community: www.mathworks.com/matlabcentral
Technical support: www.mathworks.com/support/contact_us
Phone: [phone]
The MathWorks, Inc.
1 Apple Hill Drive
Natick, MA 01760-2098
®
MATLAB Data Import and Export
© COPYRIGHT 2009–2020 by The MathWorks, Inc.
The software described in this document is furnished under a license agreement. The software may be used or copied
only under the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form
without prior written consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through
the federal government of the United States. By accepting delivery of the Program or Documentation, the government
hereby agrees that this software or documentation qualifes as commercial computer software or commercial computer
software documentation as such terms are used or defned in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014.
Accordingly, the terms and conditions of this Agreement and only those rights specifed in this Agreement, shall pertain
to and govern the use, modifcation, reproduction, release, performance, display, and disclosure of the Program and
Documentation by the federal government (or other entity acquiring for or through the federal government) and shall
supersede any conficting contractual terms or conditions. If this License fails to meet the government's needs or is
inconsistent in any respect with federal procurement law, the government agrees to return the Program and
Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See
www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be
trademarks or registered trademarks of their respective holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for
more information.
Revision History
September 2009 Online only New for MATLAB 7.9 (Release 2009b)
March 2010 Online only Revised for Version 7.10 (Release 2010a)
September 2010 Online only Revised for Version 7.11 (Release 2010b)
April 2011 Online only Revised for Version 7.12 (Release 2011a)
September 2011 Online only Revised for Version 7.13 (Release 2011b)
March 2012 Online only Revised for Version 7.14 (Release 2012a)
September 2012 Online only Revised for Version 8.0 (Release 2012b)
March 2013 Online only Revised for Version 8.1 (Release 2013a)
September 2013 Online only Revised for Version 8.2 (Release 2013b)
March 2014 Online only Revised for Version 8.3 (Release 2014a)
October 2014 Online only Revised for Version 8.4 (Release 2014b)
March 2015 Online only Revised for Version 8.5 (Release 2015a)
September 2015 Online only Revised for Version 8.6 (Release 2015b)
October 2015 Online only Rereleased for Version 8.5.1 (Release 2015aSP1)
March 2016 Online only Revised for Version 9.0 (Release 2016a)
September 2016 Online only Revised for Version 9.1 (Release 2016b)
March 2017 Online only Revised for Version 9.2 (Release 2017a)
September 2017 Online only Revised for Version 9.3 (Release 2017b)
March 2018 Online only Revised for Version 9.4 (Release 2018a)
September 2018 Online only Revised for Version 9.5 (Release 2018b)
March 2019 Online only Revised for Version 9.6 (Release 2019a)
September 2019 Online only Revised for Version 9.7 (Release 2019b)
March 2020 Online only Revised for Version 9.8 (Release 2020a)
Contents
File Opening, Loading, and Saving
1
Supported File Formats for Import and Export . . . . . . . . . . . . . . . . . . . . . . 1-2
Methods for Importing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Tools that Import Multiple File Formats . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Importing Specifc File Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Importing Data with Low-Level I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-6
Import Images, Audio, and Video Interactively . . . . . . . . . . . . . . . . . . . . . . 1-7
Viewing the Contents of a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Specifying Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Generating Reusable MATLAB Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-9
Import or Export a Sequence of Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-10
Save and Load Parts of Variables in MAT-Files . . . . . . . . . . . . . . . . . . . . . 1-11
Save and Load Using the matfle Function . . . . . . . . . . . . . . . . . . . . . . . 1-11
Load Parts of Variables Dynamically . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-12
Avoid Inadvertently Loading Entire Variables . . . . . . . . . . . . . . . . . . . . . 1-13
Partial Loading and Saving Requires Version 7.3 MAT-Files . . . . . . . . . . . 1-13
MAT-File Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-15
Overview of MAT-File Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-15
Save to Nondefault MAT-File Version . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-16
Data Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-16
Accelerate Save and Load Operations for Version 7.3 MAT-Files . . . . . . . 1-17
Growing Arrays Using matfle Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-18
Unexpected Results When Loading Variables Within a Function . . . . . . 1-20
Create Temporary Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-22
Text Files
2
Import Text Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Import Text Files Using the Import Tool . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Import Text Files Using readtable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Import Data from Text Files as Other Data Types . . . . . . . . . . . . . . . . . . . 2-3
iii
Read Text File Data Using Import Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
Select Data Interactively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
Import Data from Multiple Text Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-6
Import Dates and Times from Text Files . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
Import Numeric Data from Text Files into Matrix . . . . . . . . . . . . . . . . . . 2-12
Import Comma-Separated Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-12
Import Delimited Numeric Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-12
Import Mixed Data from Text File into Table . . . . . . . . . . . . . . . . . . . . . . 2-14
Import Block of Mixed Data from Text File into Table or Cell Array . . . . 2-17
Write Data to Text Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-21
Export Table to Text File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-21
Export Cell Array to Text File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-22
Export Numeric Array to Text File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-23
Write to a Diary File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-24
Read Collection or Sequence of Text Files . . . . . . . . . . . . . . . . . . . . . . . . 2-25
Import Block of Numeric Data from Text File . . . . . . . . . . . . . . . . . . . . . . 2-28
Spreadsheets
3
Import Spreadsheets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Import Spreadsheet Data Using the Import Tool . . . . . . . . . . . . . . . . . . . . 3-2
Import Spreadsheet Data Using readtable . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Import Spreadsheet Data as Other Data Types . . . . . . . . . . . . . . . . . . . . . 3-3
Read Spreadsheet Data Using Import Tool . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Select Data Interactively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Import Data from Multiple Spreadsheets . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
Paste Data from Clipboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6
Read Spreadsheet Data into Array or Individual Variables . . . . . . . . . . . . 3-7
Read Spreadsheet Data into Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9
Read Collection or Sequence of Spreadsheet Files . . . . . . . . . . . . . . . . . . 3-12
Write Data to Excel Spreadsheets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-14
Write Tabular Data to Spreadsheet File . . . . . . . . . . . . . . . . . . . . . . . . . . 3-14
Write Numeric and Text Data to Spreadsheet File . . . . . . . . . . . . . . . . . . 3-14
Disable Warning When Adding New Worksheet . . . . . . . . . . . . . . . . . . . . 3-15
Format Cells in Excel Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-15
Defne Import Options for Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17
iv Contents
Low-Level File I/O
4
Import Text Data Files with Low-Level I/O . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Reading Data in a Formatted Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Reading Data Line-by-Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4
Testing for End of File (EOF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5
Opening Files with Diferent Character Encodings . . . . . . . . . . . . . . . . . . 4-7
Import Binary Data with Low-Level I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Low-Level Functions for Importing Data . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Reading Binary Data in a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Reading Portions of a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10
Reading Files Created on Other Systems . . . . . . . . . . . . . . . . . . . . . . . . 4-12
Export to Text Data Files with Low-Level I/O . . . . . . . . . . . . . . . . . . . . . . 4-13
Write to Text Files Using fprintf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13
Append To or Overwrite Existing Text Files . . . . . . . . . . . . . . . . . . . . . . . 4-14
Open Files with Diferent Character Encodings . . . . . . . . . . . . . . . . . . . . 4-17
Export Binary Data with Low-Level I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18
Low-Level Functions for Exporting Data . . . . . . . . . . . . . . . . . . . . . . . . . 4-18
Write Binary Data to a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18
Overwrite or Append to an Existing Binary File . . . . . . . . . . . . . . . . . . . . 4-19
Create a File for Use on a Diferent System . . . . . . . . . . . . . . . . . . . . . . 4-20
Write and Read Complex Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-21
Internet of Things (IoT) Data
5
Aggregate Data in ThingSpeak Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Regularize Irregularly Sampled Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3
Plot Data Read from ThingSpeak Channel . . . . . . . . . . . . . . . . . . . . . . . . . 5-4
Read ThingSpeak Data and Predict Battery Discharge Time with Linear
Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5
Images
6
Importing Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-2
Getting Information About Image Files . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-2
Reading Image Data and Metadata from TIFF Files . . . . . . . . . . . . . . . . . 6-3
v
Exporting to Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-5
Exporting Image Data and Metadata to TIFF Files . . . . . . . . . . . . . . . . . . 6-5
Scientifc Data
7
Import CDF Files Using Low-Level Functions . . . . . . . . . . . . . . . . . . . . . . . 7-2
Represent CDF Time Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-4
Import CDF Files Using High-Level Functions . . . . . . . . . . . . . . . . . . . . . . 7-5
Export to CDF Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-8
Map NetCDF API Syntax to MATLAB Syntax . . . . . . . . . . . . . . . . . . . . . . . 7-11
Import NetCDF Files and OPeNDAP Data . . . . . . . . . . . . . . . . . . . . . . . . . 7-13
MATLAB NetCDF Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-13
Read from NetCDF File Using High-Level Functions . . . . . . . . . . . . . . . . 7-13
Find All Unlimited Dimensions in NetCDF File . . . . . . . . . . . . . . . . . . . . 7-15
Read from NetCDF File Using Low-Level Functions . . . . . . . . . . . . . . . . 7-16
Resolve Errors Reading OPeNDAP Data . . . . . . . . . . . . . . . . . . . . . . . . . . 7-19
Export to NetCDF Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-20
MATLAB NetCDF Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-20
Create New NetCDF File From Existing File or Template . . . . . . . . . . . . 7-20
Merge Two NetCDF Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-21
Write Data to NetCDF File Using Low-Level Functions . . . . . . . . . . . . . . 7-23
Importing Flexible Image Transport System (FITS) Files . . . . . . . . . . . . 7-26
Importing HDF5 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-27
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-27
Using the High-Level HDF5 Functions to Import Data . . . . . . . . . . . . . . . 7-27
Using the Low-Level HDF5 Functions to Import Data . . . . . . . . . . . . . . . 7-32
Exporting to HDF5 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-33
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-33
Using the MATLAB High-Level HDF5 Functions to Export Data . . . . . . . 7-33
Using the MATLAB Low-Level HDF5 Functions to Export Data . . . . . . . . 7-34
Working with Non-ASCII Characters in HDF5 Files . . . . . . . . . . . . . . . . . 7-40
Create Dataset and Attribute Names Containing Non-ASCII Characters . 7-40
Create Variable-Length String Data Containing Non-ASCII Characters . . 7-41
Import HDF4 Files Programmatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-43
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-43
Using the MATLAB HDF4 High-Level Functions . . . . . . . . . . . . . . . . . . . 7-43
Map HDF4 to MATLAB Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-46
vi Contents
Import HDF4 Files Using Low-Level Functions . . . . . . . . . . . . . . . . . . . . . 7-47
About HDF4 and HDF-EOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-50
Export to HDF4 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-51
Write MATLAB Data to HDF4 File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-51
Manage HDF4 Identifers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-52
Audio and Video
8
Read and Write Audio Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2
Record and Play Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-4
Record Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-4
Play Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-6
Record or Play Audio within a Function . . . . . . . . . . . . . . . . . . . . . . . . . . 8-6
Read Video Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-8
Supported Video and Audio File Formats . . . . . . . . . . . . . . . . . . . . . . . . . 8-12
Video Data in MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-12
Audio Data in MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-14
Convert Between Image Sequences and Video . . . . . . . . . . . . . . . . . . . . . 8-16
XML Documents
9
Importing XML Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-2
What Is an XML Document Object Model (DOM)? . . . . . . . . . . . . . . . . . . . 9-2
Example — Finding Text in an XML File . . . . . . . . . . . . . . . . . . . . . . . . . . 9-3
Exporting to XML Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-5
Creating an XML File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-5
Updating an Existing XML File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-6
Memory-Mapping Data Files
10
Overview of Memory-Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2
What Is Memory-Mapping? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2
Benefts of Memory-Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2
When to Use Memory-Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-3
Maximum Size of a Memory Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-4
Byte Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-4
vii
Map File to Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-5
Create a Simple Memory Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-5
Specify Format of Your Mapped Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-6
Map Multiple Data Types and Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-6
Select File to Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-8
Read from Mapped File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-9
Write to Mapped File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-14
Write to Memory Mapped as Numeric Array . . . . . . . . . . . . . . . . . . . . . 10-14
Write to Memory Mapped as Scalar Structure . . . . . . . . . . . . . . . . . . . 10-15
Write to Memory Mapped as Nonscalar Structure . . . . . . . . . . . . . . . . . 10-15
Syntaxes for Writing to Mapped File . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-16
Work with Copies of Your Mapped Data . . . . . . . . . . . . . . . . . . . . . . . . 10-17
Delete Memory Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-19
Ways to Delete a Memory Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-19
The Efect of Shared Data Copies On Performance . . . . . . . . . . . . . . . . 10-19
Share Memory Between Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-20
Internet File Access and JSON
11
Server Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2
Server Authentication For RESTful Web Services . . . . . . . . . . . . . . . . . . 11-2
Server Authentication For HTTP Web Services . . . . . . . . . . . . . . . . . . . . 11-2
Proxy Server Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-4
RESTful Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-4
HTTP Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-4
Use MATLAB Web Preferences For Proxy Server Settings . . . . . . . . . . . . 11-4
Use System Settings For Proxy Server Settings . . . . . . . . . . . . . . . . . . . 11-5
MATLAB and Web Services Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-6
MATLAB Does Not Verify Certifcate Chains . . . . . . . . . . . . . . . . . . . . . . 11-6
Download Data from Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-7
Convert Data from Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-10
Download Web Page and Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-13
Example — Use the webread Function . . . . . . . . . . . . . . . . . . . . . . . . . 11-13
Example — Use the websave Function . . . . . . . . . . . . . . . . . . . . . . . . . 11-13
Call Web Services from Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-14
Error Messages Concerning Web Service Options . . . . . . . . . . . . . . . . . 11-15
Send Email . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-16
Perform FTP File Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-17
viii Contents
Display Hyperlinks in the Command Window . . . . . . . . . . . . . . . . . . . . . 11-19
Create Hyperlinks to Web Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-19
Transfer Files Using FTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-19
Customize JSON Encoding for MATLAB Classes . . . . . . . . . . . . . . . . . . . 11-20
Large Data
12
Getting Started with MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-3
What Is MapReduce? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-3
MapReduce Algorithm Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-3
Example MapReduce Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-4
Write a Map Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-9
Role of Map Function in MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-9
Requirements for Map Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-10
Sample Map Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-10
Write a Reduce Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-13
Role of the Reduce Function in MapReduce . . . . . . . . . . . . . . . . . . . . . 12-13
Requirements for Reduce Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-14
Sample Reduce Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-14
Speed Up and Deploy MapReduce Using Other Products . . . . . . . . . . . 12-17
Execution Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-17
Running in Parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-17
Application Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-17
Build Efective Algorithms with MapReduce . . . . . . . . . . . . . . . . . . . . . 12-18
Debug MapReduce Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-20
Set Breakpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-20
Execute mapreduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-20
Step Through Map Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-21
Step Through Reduce Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-22
Analyze Big Data in MATLAB Using MapReduce . . . . . . . . . . . . . . . . . . 12-25
Find Maximum Value with MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . 12-32
Compute Mean Value with MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . 12-35
Compute Mean by Group Using MapReduce . . . . . . . . . . . . . . . . . . . . . . 12-38
Create Histograms Using MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-43
Simple Data Subsetting Using MapReduce . . . . . . . . . . . . . . . . . . . . . . . 12-50
Using MapReduce to Compute Covariance and Related Quantities . . . 12-56
ix
Compute Summary Statistics by Group Using MapReduce . . . . . . . . . . 12-61
Using MapReduce to Fit a Logistic Regression Model . . . . . . . . . . . . . . 12-67
Tall Skinny QR (TSQR) Matrix Factorization Using MapReduce . . . . . . 12-73
Compute Maximum Average HSV of Images with MapReduce . . . . . . . 12-78
Getting Started with Datastore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-84
What Is a Datastore? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-84
Create and Read from a Datastore . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-85
Select Datastore for File Format or Application . . . . . . . . . . . . . . . . . . . 12-88
Datastores for Standard File Formats . . . . . . . . . . . . . . . . . . . . . . . . . . 12-88
Datastores for Specifc Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-88
Custom File Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-90
Nondeterministic Datastores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-90
Work with Remote Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-91
Amazon S3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-91
Microsoft Azure Storage Blob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-92
Hadoop Distributed File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-94
Read and Analyze Large Tabular Text File . . . . . . . . . . . . . . . . . . . . . . . . 12-96
Read and Analyze Image Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-98
Read and Analyze MAT-File with Key-Value Data . . . . . . . . . . . . . . . . . 12-102
Read and Analyze Hadoop Sequence File . . . . . . . . . . . . . . . . . . . . . . . 12-105
Develop Custom Datastore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-107
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-107
Implement Datastore for Serial Processing . . . . . . . . . . . . . . . . . . . . . 12-108
Add Support for Parallel Processing . . . . . . . . . . . . . . . . . . . . . . . . . . 12-110
Add Support for Hadoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-111
Add Support for Shufing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-112
Add Support for Writing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-113
Validate Custom Datastore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-115
Testing Guidelines for Custom Datastores . . . . . . . . . . . . . . . . . . . . . . 12-116
Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-116
Workfow Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-122
Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-123
Develop Custom Datastore for DICOM Data . . . . . . . . . . . . . . . . . . . . . 12-124
Developing Custom Datastores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-124
Class Defnition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-124
Using the DICOMDatastore Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-128
Set Up Datastore for Processing on Diferent Machines or Clusters . 12-130
Save Datastore and Load on Diferent File System Platform . . . . . . . . 12-130
Process Datastore Using Parallel and Distributed Computing . . . . . . . 12-131
x Contents