Table Of ContentSwitch/Router
Architectures
Switch/Router
Architectures
Systems with Crossbar Switch Fabrics
James Aweya
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2020 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
International Standard Book Number-13: 978-0-367-40785-8 (Hardback)
This book contains information obtained from authentic and highly regarded sources.
Reasonable efforts have been made to publish reliable data and information, but the author and
publisher cannot assume responsibility for the validity of all materials or the consequences
of their use. The authors and publishers have attempted to trace the copyright holders of all
material reproduced in this publication and apologize to copyright holders if permission
to publish in this form has not been obtained. If any copyright material has not been
acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted,
reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means,
now known or hereafter invented, including photocopying, microfilming, and recording, or in
any information storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access
www.copyright.com (www.copyright.com/) or contact the Copyright Clearance Center, Inc.
(CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit
organization that provides licenses and registration for a variety of users. For organizations that
have been granted a photocopy license by the CCC, a separate system of payment has been
arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks,
and are used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
www.taylorandfrancis.com
and the CRC Press Web site at
www.crcpress.com
Contents
Preface.................................................................................................................xv
Author ...............................................................................................................xvii
PART 1 Characteristics of Switch/Routers
with Crossbar Switch Fabrics
Chapter 1 The Switch/Router: Integrated OSI Layers 2 and 3
Forwarding on a Single Platform ....................................................3
1.1 Introduction ..........................................................................3
1.2 Flow-Based Layer 3 Forwarding ..........................................4
1.3 Network Topology-Based Layer 3 Forwarding ....................5
1.4 Using Centralized or Distributed Forwarding Engines ........8
1.4.1 Forwarding Using a Centralized Forwarding
Engine .......................................................................8
1.4.2 Forwarding Using Distributed Forwarding
Engines .....................................................................9
1.5 Building the Layer 2 Forwarding Tables ............................10
1.5.1 MAC Address Aging ..............................................10
1.5.2 Synchronizing MAC Address Tables in a
Distributed Forwarding Architecture .....................11
1.5.3 History of MAC Address Changes .........................11
1.6 Memory Architectures for Storing Forwarding
Databases ............................................................................12
1.6.1 Content Addressable Memory (CAM) ...................12
1.6.2 Ternary Content Addressable Memory (TCAM) ...14
1.7 Physical and Logical Interfaces on a Switch/Router ..........16
1.7.1 Layer 2 (Switch) Interfaces .....................................17
1.7.1.1 Access Ports ............................................17
1.7.1.2 Access Ports and Tagged Packets ............18
1.7.1.3 Trunk Ports ..............................................19
1.7.1.4 Port Channels or Port Groups .................20
1.7.2 Layer 3 (Routed) Interfaces ....................................21
1.7.2.1 Physical Layer 3 (Routed) Interfaces ......21
1.7.2.2 Logical Layer 3 (Routed) Interfaces
(VLAN Interfaces or Switch Virtual
Interfaces (SVIs)) .....................................23
1.7.3 Subinterfaces ..........................................................25
v
vi Contents
1.7.4 Loopback Interfaces ...............................................27
1.7.4.1 Local Communications ...........................28
1.7.4.2 Testing and Performance Analysis ..........28
1.7.4.3 Device Identification ...............................28
1.7.4.4 Routing Information Maintenance ..........29
1.7.5 Tunnel Interfaces ....................................................31
1.7.5.1 Layer 3 Tunnel Interfaces/Ports
Examples .................................................32
1.7.5.2 Layer 2 Tunnel Interfaces/Ports
Examples .................................................36
1.7.6 Port Channels .........................................................40
1.7.6.1 Layer 2 and Layer 3 Port Channels .........41
1.7.6.2 Load Balancing .......................................43
1.7.6.3 Dynamic Configuration of Port
Channels ..................................................44
1.8 Challenges ..........................................................................44
References .....................................................................................46
Chapter 2 Understanding Crossbar Switch Fabrics .......................................47
2.1 Introduction ........................................................................47
2.2 The Crossbar Switch Fabric ...............................................49
2.2.1 Implementing a Crossbar Switch ............................50
2.2.2 Building Multistage Switches .................................51
2.2.3 Challenges in Building Larger Crosspoint Arrays .55
2.2.4 Designing Today’s System Interconnects ...............58
2.3 Logical Architecture and Improving Data Transfer
Throughput .........................................................................61
2.4 Components of a Practical Crossbar Switch Fabric
System ................................................................................65
2.5 Traffic Scheduling in the Crossbar Switch .........................67
2.5.1 The Scheduling Problem ........................................69
2.5.2 Parallel Iterative Matching (PIM) ..........................70
2.5.3 Round-Robin Matching (RRM) .............................72
2.5.4 Iterative Round-Robin Matching with Slip
(iSLIP) ....................................................................73
2.5.5 Other Considerations in the Design of
Schedulers ...............................................................75
2.6 Handling Multicast Traffic .................................................75
2.6.1 Multicast Packet Replication ..................................77
2.6.2 Multicast Traffic Scheduling ..................................79
2.6.2.1 Full versus Partial Multicast
Scheduling across the Crossbar
Switch Fabric ...........................................79
2.6.2.2 Scheduling in Internally Unbuffered
Crossbar Switches ...................................80
Contents vii
2.6.2.3 Scheduling in Internally Buffered
Crossbar Switches ...................................82
2.6.2.4 Special Focus: The ESLIP
Scheduling Algorithm .............................83
2.7 Data Transfer Process over the Switch Fabric ....................86
2.8 System Monitoring and Control .........................................87
2.8.1 Main MBus Functions ............................................89
2.8.2 Alarm Functionality ...............................................90
2.9 Scalability ...........................................................................92
2.10 Fault Tolerance ...................................................................95
2.11 Generic Switch/Router with Crossbar
Switch Fabric ......................................................................99
References ...................................................................................104
Chapter 3 Introduction to Switch/Routers with Crossbar
Switch Fabrics .............................................................................109
3.1 The Crossbar Switch Fabric .............................................109
3.2 Architectures with Crossbar-Based Switch Fabrics
and Centralized Forwarding Engines ................................110
3.2.1 Architectures with Forwarding Using a Flow/
Route Cache in Centralized Processor ..................110
3.2.2 Architectures with Forwarding Using an
Optimized Lookup System in Centralized
Processor ...............................................................111
3.3 Architectures with Crossbar-Based Switch Fabrics
and Distributed Forwarding Engines ................................112
3.3.1 Architectures with Forwarding Engine and
Flow/Route Cache in Line Cards ..........................113
3.3.2 Architectures with Fully Distributed
Forwarding Engines in Line Cards .......................114
3.4 Relating Architectures to Switch/Router Types ................116
PART 2 Design Examples and Case Studies
Chapter 4 Cisco Catalyst 6500 Series Switches with Supervisor
Engines 1A and 2 ........................................................................123
4.1 Introduction ......................................................................123
4.2 Main Architectural Features of the Catalyst
6500 Series .......................................................................124
4.3 Catalyst 6500 Switch Fabric Architecture .......................128
4.4 Catalyst 6500 Line Cards Architectures ..........................132
4.4.1 Fabric-Enabled Line Cards ...................................132
4.4.2 Fabric-Only Line Cards ........................................133
viii Contents
4.5 Catalyst 6500 Control Plane Implementation and
Forwarding Engines—Supervisor Engines ......................133
4.5.1 Supervisor Engine 1A Architecture .....................133
4.5.1.1 Supervisor Engine 1A with Only a
PFC1 ......................................................133
4.5.1.2 Supervisor Engine 1A with a PFC1
and MSFC1/MSFC2 ..............................135
4.5.2 Supervisor Engine 2 Architecture ........................137
4.5.2.1 Supervisor Engine 2 with a PFC2 .........137
4.5.2.2 Supervisor Engine 2 with a PFC2
and MSFC2 ...........................................140
4.5.3 Supporting High Availability with Dual
Supervisor Engines ................................................141
4.5.4 Distributed Forwarding Card ...............................142
4.5.4.1 Packet Forwarding in the DFC ...............143
4.6 Packet Flow in the Catalyst 6500 .....................................145
4.6.1 Packet Flow in the Catalyst 6500 with
Centralized Forwarding ........................................145
4.6.2 Packet Flow in the Catalyst 6500 with
Distributed Forwarding ........................................148
References ...................................................................................156
Chapter 5 Avaya P580 and P882 Routing Switch Architecture with
80-Series Media Module .............................................................157
5.1 Introduction ......................................................................157
5.2 Basic Architecture ............................................................157
5.3 Data Flow through the Avaya 80-Series Switch ...............158
5.4 Quality of Service (QoS) Mechanisms ..............................161
5.4.1 Classification Precedence ......................................161
5.4.2 DiffServ Mapping Table .......................................163
5.4.3 Queuing and Scheduling ......................................164
5.4.4 Traffic Management .............................................164
5.4.5 IEEE 802.1p/Q and DSCP or ToS Standards in
the Avaya 80-Series ..............................................167
5.4.5.1 DiffServ’s Per-Hop
Behaviors (PHB)....................................167
5.4.5.2 Packet Loss Priority (PLP) ....................168
5.4.5.3 Avaya 80-Series Recommendations
for IEEE 802.1p/Q, DSCP Code
Point, and Queues ..................................169
5.5 Designing the High-Performance Switch/Router..............170
References ...................................................................................172
Contents ix
Chapter 6 Foundry Networks Multilayer Switches with IronCoreTM
Network Interface Module ...........................................................175
6.1 Introduction .......................................................................175
6.2 Switch Chassis Overview ..................................................175
6.3 Chassis Crossbar Switch Fabric .......................................177
6.4 IronCoreTM Network Interface Module
Architecture ......................................................................179
6.4.1 Network Interface Module Components ..............180
6.4.1.1 Physical Ports ........................................180
6.4.1.2 Multi-Port MAC ....................................182
6.4.2 Packet Processor—The Forwarding Engine of
IronCore ................................................................182
6.4.3 Route Processor Components ...............................183
6.4.3.1 System Management Module/
Board—The Route Processor
Module...................................................183
6.4.3.2 System Management
Interface/CPU ........................................183
6.4.3.3 CPU Path and Data Transfer
Engine Path ...........................................184
6.4.3.4 Management Bus ...................................184
6.4.4 Shared Memory and Switch Fabric Interface
Components ..........................................................184
6.4.4.1 Shared-Memory Switch Fabric and
Buffer Pool ............................................184
6.4.4.2 Data Transfer Engine .............................185
6.4.4.3 Crossbar Backplane Connection—
Module Connection to Crossbar
Switch Fabric .........................................185
6.4.4.4 Multiple Destination Output Priority
Queues ...................................................185
6.4.4.5 Multiple Input Source Buffers per
Output Port ............................................186
6.5 BigIron 4000 Complete Crossbar Switch System ............187
6.6 Packet Processing Overview ............................................188
6.6.1 Role of the Data Transfer Engine and Packet
Flow for Unicast and Multicast Traffic .................189
6.6.1.1 Forwarding Unicast Traffic ...................190
6.6.1.2 Forwarding Multicast Traffic ................190
6.7 Important Attributes of the Shared-Memory Based
IronCore Architecture .......................................................191
References ...................................................................................192
