Description:

Table of Contents

Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

1. Enterprise Analytics Fundamentals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

The Analytics Data Pipeline 1

Data Lakes 2

Lambda Architecture 3

Kappa Architecture 5

Choosing Between Lambda and Kappa 6

The Azure Analytics Pipeline 6

Introducing the Analytics Scenarios 9

Example Code and Example Data Sets 11

What You Will Need 11

Broadband Internet Connectivity 11

Azure Subscription 11

Visual Studio 2015 with Update 1 11

Azure SDK 2.8 or Later 15

Summary 16

2. Getting Data into Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Ingest Loading Layer 17

Bulk Data Loading 19

Disk Shipping 19

End User Tools 35

Network-Oriented Approaches 52

Stream Loading 74

Stream Loading with Event Hubs 75

iii

Summary 76

3. Storing Ingested Data in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

File-Oriented Storage 77

Blob Storage 79

Azure Data Lake Store 84

HDFS 90

Queue-Oriented Storage 94

Blue Yonder Scenario: Smart Buildings 95

Event Hubs 96

IoT Hub 111

Summary 122

4. Real-Time Processing in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Stream Processing 123

Consuming Messages from Event Hubs 125

Tuple-at-a-Time Processing in Azure 129

Introducing HDInsight 129

Storm on HDInsight 129

EventProcessorHost 170

Azure Machine Learning 174

Summary 174

5. Real-Time Micro-Batch Processing in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

Micro-Batch Processing in Azure 175

Spark Streaming on HDInsight 175

Storm on HDInsight 192

Azure Stream Analytics 199

Summary 206

6. Batch Processing in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

Batch Processing with MapReduce on HDInsight 209

Apache Hadoop MapReduce 210

Batch Processing with Hive on HDInsight 213

Internal and External Tables 214

Partitioning Tables 214

Views 215

Indexes 215

Databases 216

Using Hive on HDInsight 216

Storage on HDInsight 218

Batch Processing Blue Yonder Airports Data 219

iv | Table of Contents

Creating an External Table 220

Creating an Internal Table 225

Batch Processing with Pig on HDInsight 228

Batch Processing with Spark on HDInsight 229

Batch Processing Blue Yonder Airports Data 232

Creating an External Table 233

Batch Processing with SQL Data Warehouse 237

Using SQL Data Warehouse 240

Batch Processing Blue Yonder Airports Data 240

Storing the Credentials to Azure Storage 241

Batch Processing with Data Lake Analytics 247

Using Data Lake Analytics 249

Batch Processing Blue Yonder Airports Data 250

Processing with U-SQL 250

Batch Processing with Azure Batch 258

Orchestrating Batch Processing Pipelines with Azure Data Factory 259

Summary 260

7. Interactive Querying in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

Interactive Querying with Azure SQL Data Warehouse 263

Partitions and Distributions 263

Indexes 265

Interactive Exploration of the Blue Yonder Airports Data 266

Interactive Querying with Hive and Tez 269

Indexes 271

Partitions 271

Interactive Exploration of the Blue Yonder Airports Data 271

Interactive Querying with Spark SQL 278

Indexes 278

Partitions 278

Interactive Exploration of the Blue Yonder Airports Data 279

Interactive Querying with USQL 283

Interactive Exploration of the Blue Yonder Airports Data 283

Summary 285

8. Hot and Cold Path Serving Layer in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Azure Redis Cache 290

Redis in the Speed Serving Layer 291

Document DB 296

Document DB in the Speed Serving Layer 299

Document DB in the Batch Serving Layer 302

SQL Database 303

Table of Contents | v

SQL Database in the Speed Serving Layer 305

SQL Database in the Batch Serving Layer 311

SQL Data Warehouse 311

HBase on HDInsight 312

Azure Search 317

Summary 318

9. Intelligence and Machine Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319

Azure Machine Learning 322

R Server on HDInsight 324

SQL R Services 325

Microsoft Cognitive Services 326

Summary 338

10. Managing Metadata in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339

Managing Metadata with Azure Data Catalog 339

Data Catalog in the Blue Yonder Airports Scenario 342

Add an Azure Data Lake Store Asset 344

Add Azure Storage Blobs 347

Add a SQL Data Warehouse 352

Summary 355

11. Protecting Your Data in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

Identity and Access Management 357

Data Protection 359

Auditing 361

Summary 362

12. Performing Analytics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

Analytics with Power BI 363

Real-Time Power BI in the Blue Yonder Scenario 365

Batch Analytics Reporting with Power BI in the Blue Yonder Scenario 374

A Look Ahead 378

Real Time 378

Lower Batch Latencies 379

IoT 379

Security 379

More Linux 379

ebook img

Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark PDF

412 Pages·2019·31.3467 MB·english
by  Zoiner Tejada
Save to my drive
Quick download
Download

Download Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark PDF Free - Full Version

by Zoiner Tejada| 2019| 412 pages| 31.3467| english

Download Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark by Zoiner Tejada in PDF format completely FREE. No registration required, no payment needed. Get instant access to this valuable resource on PDFdrive.to!

Free Download PDF

About Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark

<p>Table of Contents</p><p>Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii</p><p>Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix</p><p>1. Enterprise Analytics Fundamentals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1</p><p>The Analytics Data Pipeline 1</p><p>Data Lakes 2</p><p>Lambda Architecture 3</p><p>Kappa Architecture 5</p><p>Choosing Between Lambda and Kappa 6</p><p>The Azure Analytics Pipeline 6</p><p>Introducing the Analytics Scenarios 9</p><p>Example Code and Example Data Sets 11</p><p>What You Will Need 11</p><p>Broadband Internet Connectivity 11</p><p>Azure Subscription 11</p><p>Visual Studio 2015 with Update 1 11</p><p>Azure SDK 2.8 or Later 15</p><p>Summary 16</p><p>2. Getting Data into Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17</p><p>Ingest Loading Layer 17</p><p>Bulk Data Loading 19</p><p>Disk Shipping 19</p><p>End User Tools 35</p><p>Network-Oriented Approaches 52</p><p>Stream Loading 74</p><p>Stream Loading with Event Hubs 75</p><p>iii</p><p>Summary 76</p><p>3. Storing Ingested Data in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77</p><p>File-Oriented Storage 77</p><p>Blob Storage 79</p><p>Azure Data Lake Store 84</p><p>HDFS 90</p><p>Queue-Oriented Storage 94</p><p>Blue Yonder Scenario: Smart Buildings 95</p><p>Event Hubs 96</p><p>IoT Hub 111</p><p>Summary 122</p><p>4. Real-Time Processing in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123</p><p>Stream Processing 123</p><p>Consuming Messages from Event Hubs 125</p><p>Tuple-at-a-Time Processing in Azure 129</p><p>Introducing HDInsight 129</p><p>Storm on HDInsight 129</p><p>EventProcessorHost 170</p><p>Azure Machine Learning 174</p><p>Summary 174</p><p>5. Real-Time Micro-Batch Processing in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175</p><p>Micro-Batch Processing in Azure 175</p><p>Spark Streaming on HDInsight 175</p><p>Storm on HDInsight 192</p><p>Azure Stream Analytics 199</p><p>Summary 206</p><p>6. Batch Processing in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207</p><p>Batch Processing with MapReduce on HDInsight 209</p><p>Apache Hadoop MapReduce 210</p><p>Batch Processing with Hive on HDInsight 213</p><p>Internal and External Tables 214</p><p>Partitioning Tables 214</p><p>Views 215</p><p>Indexes 215</p><p>Databases 216</p><p>Using Hive on HDInsight 216</p><p>Storage on HDInsight 218</p><p>Batch Processing Blue Yonder Airports Data 219</p><p>iv | Table of Contents</p><p>Creating an External Table 220</p><p>Creating an Internal Table 225</p><p>Batch Processing with Pig on HDInsight 228</p><p>Batch Processing with Spark on HDInsight 229</p><p>Batch Processing Blue Yonder Airports Data 232</p><p>Creating an External Table 233</p><p>Batch Processing with SQL Data Warehouse 237</p><p>Using SQL Data Warehouse 240</p><p>Batch Processing Blue Yonder Airports Data 240</p><p>Storing the Credentials to Azure Storage 241</p><p>Batch Processing with Data Lake Analytics 247</p><p>Using Data Lake Analytics 249</p><p>Batch Processing Blue Yonder Airports Data 250</p><p>Processing with U-SQL 250</p><p>Batch Processing with Azure Batch 258</p><p>Orchestrating Batch Processing Pipelines with Azure Data Factory 259</p><p>Summary 260</p><p>7. Interactive Querying in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261</p><p>Interactive Querying with Azure SQL Data Warehouse 263</p><p>Partitions and Distributions 263</p><p>Indexes 265</p><p>Interactive Exploration of the Blue Yonder Airports Data 266</p><p>Interactive Querying with Hive and Tez 269</p><p>Indexes 271</p><p>Partitions 271</p><p>Interactive Exploration of the Blue Yonder Airports Data 271</p><p>Interactive Querying with Spark SQL 278</p><p>Indexes 278</p><p>Partitions 278</p><p>Interactive Exploration of the Blue Yonder Airports Data 279</p><p>Interactive Querying with USQL 283</p><p>Interactive Exploration of the Blue Yonder Airports Data 283</p><p>Summary 285</p><p>8. Hot and Cold Path Serving Layer in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287</p><p>Azure Redis Cache 290</p><p>Redis in the Speed Serving Layer 291</p><p>Document DB 296</p><p>Document DB in the Speed Serving Layer 299</p><p>Document DB in the Batch Serving Layer 302</p><p>SQL Database 303</p><p>Table of Contents | v</p><p>SQL Database in the Speed Serving Layer 305</p><p>SQL Database in the Batch Serving Layer 311</p><p>SQL Data Warehouse 311</p><p>HBase on HDInsight 312</p><p>Azure Search 317</p><p>Summary 318</p><p>9. Intelligence and Machine Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319</p><p>Azure Machine Learning 322</p><p>R Server on HDInsight 324</p><p>SQL R Services 325</p><p>Microsoft Cognitive Services 326</p><p>Summary 338</p><p>10. Managing Metadata in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339</p><p>Managing Metadata with Azure Data Catalog 339</p><p>Data Catalog in the Blue Yonder Airports Scenario 342</p><p>Add an Azure Data Lake Store Asset 344</p><p>Add Azure Storage Blobs 347</p><p>Add a SQL Data Warehouse 352</p><p>Summary 355</p><p>11. Protecting Your Data in Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357</p><p>Identity and Access Management 357</p><p>Data Protection 359</p><p>Auditing 361</p><p>Summary 362</p><p>12. Performing Analytics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363</p><p>Analytics with Power BI 363</p><p>Real-Time Power BI in the Blue Yonder Scenario 365</p><p>Batch Analytics Reporting with Power BI in the Blue Yonder Scenario 374</p><p>A Look Ahead 378</p><p>Real Time 378</p><p>Lower Batch Latencies 379</p><p>IoT 379</p><p>Security 379</p><p>More Linux 379</p>

Detailed Information

Author:Zoiner Tejada
Publication Year:2019
Pages:412
Language:english
File Size:31.3467
Format:PDF
Price:FREE
Download Free PDF

Safe & Secure Download - No registration required

Why Choose PDFdrive for Your Free Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark Download?

  • 100% Free: No hidden fees or subscriptions required for one book every day.
  • No Registration: Immediate access is available without creating accounts for one book every day.
  • Safe and Secure: Clean downloads without malware or viruses
  • Multiple Formats: PDF, MOBI, Mpub,... optimized for all devices
  • Educational Resource: Supporting knowledge sharing and learning

Free Books Similar to Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark

Frequently Asked Questions

Is it really free to download Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark PDF?

Yes, on https://PDFdrive.to you can download Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark by Zoiner Tejada completely free. We don't require any payment, subscription, or registration to access this PDF file. For 3 books every day.

How can I read Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark on my mobile device?

After downloading Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark PDF, you can open it with any PDF reader app on your phone or tablet. We recommend using Adobe Acrobat Reader, Apple Books, or Google Play Books for the best reading experience.

Is this the full version of Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark?

Yes, this is the complete PDF version of Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark by Zoiner Tejada. You will be able to read the entire content as in the printed version without missing any pages.

Is it legal to download Mastering Azure Analytics : Architecting in the Cloud with Azure Data Lake, HDInsight, and Spark PDF for free?

https://PDFdrive.to provides links to free educational resources available online. We do not store any files on our servers. Please be aware of copyright laws in your country before downloading.

The materials shared are intended for research, educational, and personal use in accordance with fair use principles.

We use cookies

We use cookies to ensure you get the best browsing experience on our website. By clicking "Accept Cookies", you agree that we can store cookies on your device in accordance with our Terms and Privacy.