Table Of ContentContents Contents
Parallel Computing
Architectures and APIs
IoT Big Data Stream Processing
References
Parallel Computing
Architectures and APIs
IoT Big Data Stream Processing
Vivek Kale
CRC Press
Taylor & Francis Group
52 Vanderbilt Avenue,
New York, NY 10017
© 2020 by Vivek Kale
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
International Standard Book Number-13: 978-1-138-55391-0 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the
validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copy-
right holders of all material reproduced in this publication and apologize to copyright holders if permission to publish
in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know
so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or
utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including pho-
tocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission
from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA
01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users.
For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been
arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
To my family friends Charudutta (“Charu”) and Shobha Palkar
with whom we reconnected after a gap of almost 30 years
References
Contents
Preface ..........................................................................................................................................xvii
Acknowledgments ....................................................................................................................xxiii
Author ..........................................................................................................................................xxv
1 Uniprocessor Computers .......................................................................................................1
1.1 Types of Computers ......................................................................................................1
1.1.1 Microcomputers ...............................................................................................2
1.1.2 Midrange Computers ......................................................................................2
1.1.3 Mainframe Computers ....................................................................................2
1.1.4 Supercomputers ...............................................................................................3
1.2 Computer System ..........................................................................................................3
1.2.1 Hardware ..........................................................................................................3
1.2.2 Software ............................................................................................................5
1.2.3 Network .............................................................................................................6
1.3 Hardware and Software Logical Equivalence ..........................................................7
1.4 Ladder of Abstraction ...................................................................................................8
1.4.1 Modeling-Level Architecture .........................................................................9
1.4.2 Algorithm-Level Architecture .......................................................................9
1.4.3 High-Level Architecture ...............................................................................11
1.4.4 Assembly-Level Architecture.......................................................................12
1.4.5 System or Instruction Set Architecture-Level Architecture ....................12
1.4.6 Machine or Microarchitecture-Level Architecture ...................................12
1.4.7 Control or Logic-Level Architecture ...........................................................13
1.4.8 Device-Level Architecture ............................................................................13
1.5 Application Programming Interfaces ......................................................................14
1.6 Summary ......................................................................................................................15
2 Processor Physics and Moore’s Law..................................................................................17
2.1 Speed of Processing and Power Problem ................................................................17
2.2 Area, Delay, and Power Consumption .....................................................................19
2.2.1 Area Consumption ........................................................................................19
2.2.2 Delay Consumption .......................................................................................20
2.2.3 Power Consumption ......................................................................................22
2.3 Area, Latency, and Power Trade-offs .......................................................................24
2.3.1 Area versus Delay Trade-off ........................................................................24
2.3.2 Delay versus Power Trade-off ......................................................................26
2.3.3 Area versus Delay versus Power Trade-off ................................................27
2.4 Moore’s Law .................................................................................................................28
2.4.1 Leveraging Moore’s Law ...............................................................................30
2.4.1.1 Reconfigurable Computing ...........................................................31
2.5 Performance Wall ........................................................................................................31
2.5.1 Power Wall ......................................................................................................32
2.5.2 Memory Wall ..................................................................................................32
vii
viii Contents
2.5.3 Instruction-Level Parallelism Wall ..............................................................32
2.6 Summary ......................................................................................................................32
Section I Genesis of Parallel Computing
3 Processor Basics ....................................................................................................................37
3.1 Processor ......................................................................................................................37
3.2 Aspects of Processor Performance ...........................................................................38
3.2.1 Potential for Speedup ....................................................................................38
3.2.2 Scalability ........................................................................................................40
3.2.3 Speedup versus Communication Overhead ..............................................40
3.3 Enhancing Uniprocessor Performance ....................................................................42
3.3.1 Improving CPU Performance ......................................................................43
3.3.2 Increasing Processor Clock Frequency .......................................................43
3.3.3 Parallelizing Arithmetic Logic Unit (ALU) Structure ..............................44
3.3.4 Pipelining ........................................................................................................45
3.3.5 Memory Hierarchy ........................................................................................46
3.3.5.1 Cache Memory ................................................................................46
3.3.6 Very Long Instruction Word (VLIW) Processors ......................................46
3.3.7 Superscalarity .................................................................................................47
3.3.8 Instruction-Level Parallelism .......................................................................47
3.3.9 Multicore Architectures ................................................................................48
3.3.10 Multithreading ...............................................................................................50
3.4 Summary ......................................................................................................................50
4 Networking Basics................................................................................................................51
4.1 Network Principles .....................................................................................................51
4.1.1 Protocol ............................................................................................................51
4.1.2 Protocol Layers ...............................................................................................52
4.1.3 Protocol Suite ..................................................................................................52
4.1.4 Datagram .........................................................................................................54
4.2 Types of Networks ......................................................................................................54
4.2.1 Personal Area Networks ...............................................................................54
4.2.2 Local Area Networks ....................................................................................55
4.2.3 Metropolitan Area Networks .......................................................................56
4.2.4 Wide Area Networks .....................................................................................56
4.3 Network Models ..........................................................................................................56
4.3.1 OSI Reference Model .....................................................................................57
4.3.2 TCP/IP Reference Model...............................................................................59
4.3.2.1 Link Layer .......................................................................................60
4.3.2.2 Internet Layer ..................................................................................60
4.3.2.3 Transport Layer ..............................................................................60
4.3.2.4 Application Layer ...........................................................................61
4.4 Interconnection Networks .........................................................................................61
4.4.1 Ethernet ...........................................................................................................62
4.4.2 Switches ...........................................................................................................63
4.5 Summary ......................................................................................................................64
Contents ix
5 Distributed Systems Basics ................................................................................................65
5.1 Distributed Systems ....................................................................................................65
5.1.1 Distributed Computing .................................................................................66
5.1.1.1 System Architectural Styles ..........................................................68
5.1.1.2 Software Architectural Styles.......................................................69
5.1.1.3 Technologies for Distributed Computing ...................................74
5.2 Distributed System Benefits.......................................................................................77
5.3 Distributed Computation Systems ...........................................................................78
5.4 Summary ......................................................................................................................79
Section II Road to Parallel Computing
6 Parallel Systems ....................................................................................................................83
6.1 Flynn’s Taxonomy for Parallel Computer Architectures .......................................83
6.2 Types of Parallel Computers ......................................................................................86
6.2.1 Shared Memory Multiprocessor Systems...................................................86
6.2.2 Distributed Memory Multicomputers ........................................................88
6.2.2.1 Interconnection Network (IN) ......................................................88
6.3 Characteristics of Parallel Systems ...........................................................................92
6.3.1 Coupling, Parallelism, Concurrency, and Granularity ............................92
6.3.2 Shared Memory Systems versus Message-Passing Systems ...................93
6.3.3 Distributed Communication ........................................................................94
6.3.3.1 Blocking/Non-blocking, Synchronous/Asynchronous
Primitives ........................................................................................94
6.3.3.2 Processor Synchrony......................................................................96
6.3.4 Synchronous versus Asynchronous Executions .......................................96
6.4 Summary ......................................................................................................................97
7 Parallel Computing Models ...............................................................................................99
7.1 Shared Memory Models .............................................................................................99
7.1.1 Theoretical Models ........................................................................................99
7.1.1.1 RAM Model .....................................................................................99
7.1.1.2 PRAM Model ..................................................................................99
7.1.2 Practical Models ...........................................................................................101
7.1.2.1 Bulk Synchronous Parallel (BSP) Model ...................................101
7.1.2.2 LogP Model ...................................................................................103
7.2 Interconnection Network Models ...........................................................................104
7.2.1 Theoretical Models ......................................................................................105
7.2.1.1 Mesh ...............................................................................................105
7.2.1.2 Mesh of Trees ................................................................................106
7.2.1.3 Cube ...............................................................................................107
7.2.1.4 Cube-Connected Cycles ..............................................................108
7.2.1.5 Butterfly .........................................................................................108
7.2.2 Practical Models ...........................................................................................109
7.2.2.1 Cluster ............................................................................................109
7.2.2.2 Grids ...............................................................................................110
7.3 Dataflow Model .........................................................................................................111
7.4 Summary ....................................................................................................................113
