HBase: The Definitive Guide, 2nd Edition: Random Access to Your Planet-Size Data

667 Pages·2015·11.81 MB·English

by  Lars George

www.it-ebooks.info
SECOND EDITION
HBase - The Definitive
Guide - 2nd Edition
Lars George
www.it-ebooks.info
HBase - The Definitive Guide - 2nd Edition, Second Edition
by Lars George
Copyright © 2010 Lars George. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol,
CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional
use. Online editions are also available for most titles (http://safaribookson
line.com). For more information, contact our corporate/institutional sales depart‐
ment: 800-998-9938 or <[email protected]>.
Editor: Ann Spencer Proofreader: FIX ME!
Production Editor: FIX ME! Indexer: FIX ME!
Copyeditor: FIX ME! Cover Designer: Karen Montgomery
Interior Designer: David Futato
Illustrator: Rebecca Demarest
January -4712: Second Edition
Revision History for the Second Edition:
2015-04-10 Early release revision 1
2015-07-07 Early release revision
See http://oreilly.com/catalog/errata.csp?isbn=0636920033943 for release details.
Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are regis‐
tered trademarks of O’Reilly Media, Inc. !!FILL THIS IN!! and related trade dress
are trademarks of O’Reilly Media, Inc.
Many of the designations used by manufacturers and sellers to distinguish their
products are claimed as trademarks. Where those designations appear in this
book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations
have been printed in caps or initial caps.
While every precaution has been taken in the preparation of this book, the publish‐
er and authors assume no responsibility for errors or omissions, or for damages re‐
sulting from the use of the information contained herein.
ISBN: 063-6-920-03394-3
[?]
www.it-ebooks.info
Table of Contents
Foreword: Michael Stack. . . . . . . . . . . . . . . . . . . . . . . . . . ix
Foreword: Carter Page. . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
The Dawn of Big Data 1
The Problem with Relational Database Systems 7
Nonrelational Database Systems, Not-Only SQL or NoSQL? 10
Dimensions 13
Scalability 15
Database (De-)Normalization 16
Building Blocks 19
Backdrop 19
Namespaces, Tables, Rows, Columns, and Cells 21
Auto-Sharding 26
Storage API 28
Implementation 29
Summary 33
HBase: The Hadoop Database 34
History 34
Nomenclature 37
Summary 37
2. Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Quick-Start Guide 39
Requirements 43
Hardware 43
Software 51
Filesystems for HBase 67
Local 69
HDFS 70
iii
www.it-ebooks.info
S3 70
Other Filesystems 72
Installation Choices 73
Apache Binary Release 73
Building from Source 76
Run Modes 79
Standalone Mode 79
Distributed Mode 79
Configuration 85
hbase-site.xml and hbase-default.xml 87
hbase-env.sh and hbase-env.cmd 88
regionserver 88
log4j.properties 89
Example Configuration 89
Client Configuration 91
Deployment 92
Script-Based 92
Apache Whirr 94
Puppet and Chef 94
Operating a Cluster 95
Running and Confirming Your Installation 95
Web-based UI Introduction 96
Shell Introduction 98
Stopping the Cluster 99
3. Client API: The Basics. . . . . . . . . . . . . . . . . . . . . . . . . 101
General Notes 101
Data Types and Hierarchy 103
Generic Attributes 104
Operations: Fingerprint and ID 104
Query versus Mutation 106
Durability, Consistency, and Isolation 108
The Cell 112
API Building Blocks 117
CRUD Operations 122
Put Method 122
Get Method 146
Delete Method 168
Append Method 181
Mutate Method 184
Batch Operations 187
Scans 193
Introduction 193
The ResultScanner Class 199
iv Table of Contents
www.it-ebooks.info
Scanner Caching 203
Scanner Batching 206
Slicing Rows 210
Load Column Families on Demand 213
Scanner Metrics 214
Miscellaneous Features 215
The Table Utility Methods 215
The Bytes Class 216
4. Client API: Advanced Features. . . . . . . . . . . . . . . . . . 219
Filters 219
Introduction to Filters 219
Comparison Filters 223
Dedicated Filters 232
Decorating Filters 252
FilterList 256
Custom Filters 259
Filter Parser Utility 269
Filters Summary 272
Counters 273
Introduction to Counters 274
Single Counters 277
Multiple Counters 278
Coprocessors 282
Introduction to Coprocessors 282
The Coprocessor Class Trinity 285
Coprocessor Loading 289
Endpoints 298
Observers 311
The ObserverContext Class 312
The RegionObserver Class 314
The MasterObserver Class 334
The RegionServerObserver Class 340
The WALObserver Class 342
The BulkLoadObserver Class 344
The EndPointObserver Class 344
5. Client API: Administrative Features. . . . . . . . . . . . . . 347
Schema Definition 347
Namespaces 347
Tables 350
Table Properties 358
Column Families 362
HBaseAdmin 375
Basic Operations 375
Table of Contents v
www.it-ebooks.info
Namespace Operations 376
Table Operations 378
Schema Operations 391
Cluster Operations 393
Cluster Status Information 411
ReplicationAdmin 422
6. Available Clients. . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
Introduction 427
Gateways 427
Frameworks 431
Gateway Clients 432
Native Java 432
REST 433
Thrift 444
Thrift2 458
SQL over NoSQL 459
Framework Clients 460
MapReduce 460
Hive 460
Mapping Existing Tables 469
Mapping Existing Table Snapshots 473
Pig 474
Cascading 479
Other Clients 480
Shell 481
Basics 481
Commands 484
Scripting 497
Web-based UI 503
Master UI Status Page 504
Master UI Related Pages 521
Region Server UI Status Page 532
Shared Pages 551
7. Hadoop Integration. . . . . . . . . . . . . . . . . . . . . . . . . . 559
Framework 559
MapReduce Introduction 560
Processing Classes 562
Supporting Classes 575
MapReduce Locality 581
Table Splits 583
MapReduce over Tables 586
Preparation 586
Table as a Data Sink 603
vi Table of Contents
www.it-ebooks.info
Table as a Data Source 610
Table as both Data Source and Sink 614
Custom Processing 617
MapReduce over Snapshots 620
Bulk Loading Data 627
A. Upgrade from Previous Releases. . . . . . . . . . . . . . . . 633
Table of Contents vii
www.it-ebooks.info
www.it-ebooks.info
Foreword: Michael Stack
The HBase story begins in 2006, when the San Francisco-based start‐
up Powerset was trying to build a natural language search engine for
the Web. Their indexing pipeline was an involved multistep process
that produced an index about two orders of magnitude larger, on aver‐
age, than your standard term-based index. The datastore that they’d
built on top of the then nascent Amazon Web Services to hold the in‐
dex intermediaries and the webcrawl was buckling under the load
(Ring. Ring. “Hello! This is AWS. Whatever you are running, please
turn it off!”). They were looking for an alternative. The Google Bigta‐
ble paper1 had just been published.
Chad Walters, Powerset’s head of engineering at the time, reflects
back on the experience as follows:
Building an open source system to run on top of Hadoop’s Distribut‐
ed Filesystem (HDFS) in much the same way that Bigtable ran on
top of the Google File System seemed like a good approach be‐
cause: 1) it was a proven scalable architecture; 2) we could lever‐
age existing work on Hadoop’s HDFS; and 3) we could both contrib‐
ute to and get additional leverage from the growing Hadoop ecosys‐
tem.
After the publication of the Google Bigtable paper, there were on-
again, off-again discussions around what a Bigtable-like system on top
of Hadoop might look. Then, in early 2007, out of the blue, Mike Ca‐
farela dropped a tarball of thirty odd Java files into the Hadoop issue
tracker: “I’ve written some code for HBase, a Bigtable-like file store.
It’s not perfect, but it’s ready for other people to play with and exam‐
1. “Bigtable: A Distributed Storage System for Structured Data” by Fay Chang et al.
ix
www.it-ebooks.info