Fibre Channel for SANs

342 Pages·2001·1.72 MB·English

by  Alan Frederic Benner

Source: Fibre Channel for SANs
1
Chapter
Fibre Channel and
Storage Area Networks
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
2
Chapter 1
Introduction
Fibre Channel technology is over a decade old. How successful has it been?
Here is an illustration. The first edition of this book included a section
called “The Unification of LAN and Channel technologies,” which
described how Fibre Channel would be part of a trend towards convergence
between LANs and channels. LANs (Local Area Networks) are used for
computer-to-computer communications, and channels are high-efficiency,
high-performance links between computers and their long-term storage
devices (disk and tape drives), and other I/O devices.
Since then, the prediction has come true, in three quite different ways.
• Most important has been the introduction and widespread use of the term
“StorageAreaNetwork,” or SAN, describing a network which is highly
optimized for transporting traffic between servers and storage devices.
• At the physical layer, the LAN and Fibre Channel technologies have
become nearly identical — Gigabit Ethernet and Fibre Channel share com-
mon signaling and data encoding mechanisms, and the future 10 Gb/s
Ethernet and Fibre Channel are expected to share nearly the same data rate.
• The management methods for Fibre Channel SANs have steadily
approached the traditional methods used for LAN management, although
the current level of management effort required for Fibre Channel SANs is
still higher than for LANs.
Interestingly, however, although the LAN and SAN types of computer
data communications have converged at a technology level, they have so far
stayed quite different in how they are used and how they are managed. That
is, systems are usually built with the SAN storage traffic separated on sepa-
rate networks from the LAN traffic, so that the management, topologies, and
provisioning of each network can be optimized for the types of traffic tra-
versing them.
The trends that originally motivated the creation of Fibre Channel have
continued or accelerated. The speed of processors, the capacities of memory,
disks, and tapes, and the use of switched communications networks have all
been doubling every 18 to 24 months, and the doubling period has in many
cases even been steadily shortening slightly. However, the rate of I/O
improvement has been much slower, so that devices are even more I/O lim-
ited. The continuing observation is that computers usually appear nearly
instantaneous, except when doing I/O (e.g., downloading web pages), or
managing stored data (e.g., backing up file systems).
Fibre Channel, and Storage Area Networks, are focused at (a) optimizing
the movement of data between server and storage systems, and (b) managing
the data and the access to the data, so that communications are optimized as
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
3
Fibre Channel and Storage Area Networks
much as possible, while continuously and reliably providing access to data,
for whoever needs it.
Fibre Channel Features
Following is a list of the major features that Fibre Channel provides:
• Unification of networking and I/O channel data communications: This was
described in detail above, and allows storage to be decoupled from servers
and managed separately. Similarly, many servers can directly access the
data as if it were their own, as long as they are coordinated to manage it
coherently.
• Bandwidth: The base definition of Fibre Channel provides better than 100
MBps for I/O and communications on current architectures, with speeds
defined up to 4 times this rate, for implementation as market and applica-
tions dictate.
• Inexpensive implementation: Fibre Channel uses an 8B/10B encoding for
all data transmission, which, by limiting low-frequency components,
allows design of AC-coupled gigabit receivers using inexpensive CMOS
VLSI technology
• Low overhead: The very low 10-12 bit error rate achievable using a combi-
nation of reliable hardware and 8B/10B encoding allows very low extra
overhead in the protocol, providing efficient usage of the transmission
bandwidth and saving effort in implementation of low-level error recovery
mechanisms.
• Low-level control: Local operations depend very little on global informa-
tion. This means, for example, that the actions that one Port takes are only
minimally affected by actions taking place on other Ports, and that individ-
ual computers need to maintain very little information about the rest of the
network. This feature minimizes the amount of work to do at the higher
levels.
• For example, hardware-controlled flow control alleviates the host pro-
cessors from the burden of managing much of the flow control overhead.
• Similarly, the low-level hardware does sophisticated error detection and
deletion, so that it can assure delivery of data intact or not at all. Upper
layer protocols don’t have to do as much error detection, and can be
more efficient.
• Flexible topology: Physical connection topologies are defined for (1)
point-to-point links, (2) shared-media loop topologies, and (3) packet-
switching network topologies. Any of these can be built using the same
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
4
Chapter 1
hardware, allowing users to match physical topology to the required con-
nectivity characteristics.
• Distance: 50 m in a room simplifies wiring, more important is 10 km,
which allows remote copy without WAN infrastructure. Consider a high
performance disk drive attached to a computer over an optical fiber. The
access time for the disk drive (to rotate the disk and move the head over the
data) would be roughly 5 ms. The speed of light in optical fiber is about
124 mi/ms. This means that the time to reach an optically connected disk
drive located a mile away would be only 0.008 ms more than the time to
reach a disk drive in the same enclosure.
• Availability: More capability to attach to multiple servers allows the data
to be accessed through many paths, which enhances availability in case
one of those paths fails.
• Flexible transmission service: Mechanisms are defined for multiple
Classes of services, including (1) dedicated bandwidth between Port pairs
at the full hardware capacity, (2) multiplexed transmission with multiple
other source or destination Ports, with acknowledgment of reception, and
(3) best-effort multiplexed datagram transmission without acknowledg-
ment, for more efficient transmission in environments where error recov-
ery is handled at a higher level, (4) dedicated connections with
configurable quality of service guarantees on transmission bandwidth and
latency, and (5) reliable multicast, with a dedicated connection at the full
hardware capacity.
• Standard protocol mappings: Fibre Channel can operate as a data transport
mechanism for multiple Upper Level Protocols, with mappings defined for
IP, SCSI-3, IPI-3 Disk, IPI-3 Tape, HIPPI, the Single Byte Channel Com-
mand set for ESCON, the AAL5 mapping of ATM for computer data, and
VIA or Virtual Interface Architecture. The most commonly used of these
currently are the mapping to SCSI-3, which is termed “FCP,” and the map-
ping to ESCON, which is termed either “FICON,” or “SBCON,” depend-
ing on context.
• Wide industry support: Most major computer, disk drive, and adapter man-
ufacturers are currently developing hardware and/or software components
based on the Fibre Channel ANSI standard.
These improvements to traditional channels don’t actually provide much
real benefit when a single server is used to process the data on a single stor-
age device. However, when multiple servers act together (for better reliabil-
ity, or higher throughput, or better pipelining, etc.) to work with the data on
multiple storage devices of different types, then the advantages of Fibre
Channel can become very important.
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
5
Fibre Channel and Storage Area Networks
Storage Area Networks
What is a Storage Area Network, and how is it different from the various
other types of networks that are built?
Here is a definition of a Storage Area Network, from one of the leaders in
the industry:
A Storage Area Network (SAN) is a dedicated, centrally managed, secure
information infrastructure, which enables any-to-any interconnection of
servers and storage systems.
This definition is unfortunately not particularly instructive as to, for
example, the difference between SANs and LANs, or MANs, or even
WANs, all of which, in some applications, could fit this description.
The difference between SANs and other types of networks can perhaps
best be understood by considering the difference between the storage and
networking ports on a desktop computer. Every computer has access to some
kind of long-term storage, and almost every computer has access to some
way of communicating with other computers. The storage interface is highly
optimized, tightly controlled (in laptops and most desktop machines, it may
not even be visible outside the box), and not shared with any other comput-
ers — which helps make it highly predictable, efficient, and fast. Network
interfaces, on the other hand, are much slower, less efficient (you have to
wait for them), and have higher overhead, but they allow access to any other
machine that it knows how to communicate with.
Storage Area Networks are built to incorporate the best of both storage
and networking interfaces: fast, efficient communications, optimized for
efficient movement of large amounts of data, but with access to a wide range
of other servers and storage devices on the network.
The primary difference then between a Storage Area Network and the
other types of networks mentioned is that, in a SAN, communication within
the network is well-managed, very well-controlled, and predictable. There-
fore, each entity on the network can almost operate is if it has sole access to
whichever partner on the network that it is currently communicating with.
A primary reason for this has been the idea of decoupling the servers
from their storage, and allowing multiple servers to access the same data at
the same time. The key here is that client systems often access their through
servers, which assure consistency, security, and authorization for the data
access. Clients, however, don’t particularly care which server is used to
access the data, and the data is the same no matter which server is accessing
it. This three-tiered system of clients displaying the data, servers processing
and managing the data, and storage subsystems holding the data, is tied
together with networks — LANs and SANs — between each layer.
Fibre Channel overlaps very little with Ethernet, except in very specific
applications. For general-purpose communications, Ethernet is very difficult
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
6
Chapter 1
to compete with (particularly since the Ethernet community tends to adopt
the best networking innovations every time there is a new generation, which
is regularly).
Fibre Channel does, however, overlap very closely with the storage tech-
nologies such as IDE and SCSI. In fact, to a file system or higher-level
device, Fibre Channel may appear almost exactly like SCSI — the SCSI
command set is transported across a Fibre Channel link, just as it would be
across a SCSI bus.
The preceding picture is generally valid for on mid-range machines. On
high-end machines, the networking interface is usually still Ethernet
(although Token Ring, FDDI, HiPPI, and others have all been important),
but the storage interface has, for the last 10 years or so, been a channel proto-
col. The primary one in the early ’90s was called ESCON, for Enterprise
System Connections. ESCON was the first real SAN, since it allowed multi-
ple servers to access multiple storage units through a high-performance,
switched fabric. In fact, currently the ESCON protocols are still transmitted
over a high-performance, switched fabric, but now the fabric is Fibre Chan-
nel, and the name has changed to FICON or SBCON.
SAN topologies
A typical topology for a large-scale system using both a Fibre Channel-
based Storage Area Network and a Local Area Network is shown in Figure
1.1.
This configuration allows a number of advantages, vs. a system with stor-
age devices tightly integrated with each separate server.
• Networked Access: All servers have direct access to all disk and tape
arrays through the SAN, once authorization has been established at the net-
work and the data level.
• Storage Consolidation: Since the client, server, and storage units can be
scaled separately, and storage units can be shared, fewer units are neces-
sary. This is especially important for expensive, large tape libraries.
• Remote Mirroring and Archiving: Since the SAN links may be up to 10
km. long, disk and tape drives can be remotely located, for disaster recov-
ery.
• LAN-free backup. The servers can move the data between disk and tape
arrays over the SAN — so the LAN between server and clients is not
impacted by the backups, and is always available.
• Server-free backup. In the ideal case, the disk array and the tape array have
enough intelligence to let the servers command 3rd-party transfers, so that,
for example, data would flow directly between a disk array and tape library
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
7
Fibre Channel and Storage Area Networks
Desktops
Figure 1.1
or
Example of an Enterprise
Laptops
or Service Provider To
SAN+LAN Topology
WAN
Router
LAN Switch
Servers
with local
storage
SAN Switch
SAN Switch
Disk Array
Tape Library
Storage
Devices
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
8
Chapter 1
across the SAN, without loading any servers.
These capabilities are getting steadily more important. In 1999, roughly
3/4 of the storage sold in the world was attached directly to servers, while
the remaining part was attached directly to the network. In 2003, over 3/4 of
storage is expected to be directly attached to the networks, either as SAN or
NAS storage.
SANs, LANs, and NAS
A major issue in the design of complex installations such as this involves the
set of difference between LANs and SANs, particularly, since there are a
large number of storage devices, termed “Network Attached Storage,” that
attach to Ethernet LANs.
In general, the fact is that SAN traffic is faster and more efficient than
LAN traffic. Getting over 80% throughput on SAN links is expected, while
getting over 30% on a sustained basis on LAN links is doing well. More
importantly, the processor overhead for communications is generally much
higher on LANs, than in SANs. Some estimates are that the processor over-
head for TCP/IP on a LAN is 1,000 MIPS to receive data at 1 Gb/s, and that
the processor overhead running TCP/IP over Ethernet is 30 times higher than
running the same data rate over Fibre Channel.
The 30X performance difference is quite amazing — what could possibly
cause two networks with the same line speed to use 30X difference in proc-
essor protocol-processing overhead? The following sections attempt to
explain this in some detail.
A caution on this section. Many of these factors (1) are extremely
dependent on implementation, and (2) are changing extremely quickly — so
don’t expect them to be always true everywhere. The main reason for listing
them here is to help people understand how to optimize design of networks
and network interfaces.
LANs vs. SANs: Differences in Network Design
Some of the efficiency advantages of Fibre Channel compared to Ethernet
relate directly to the design of the network. In an environment of steady
innovation, any real design advantages get quickly adopted in all following-
generation designs, so these are only short-term advantages.
• Low-level (hardware-based) link-level and end-to-end flow control, so the
higher levels don’t have to manage flow control and congestion control.
High-level flow control and congestion control (e.g., the TCP window
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
9
Fibre Channel and Storage Area Networks
mechanism, slow start and congestion avoidance) can require significant
overhead, especially on heavily-loaded networks.
• Switch-based transmission (vs. shared medium), so the quality of service
for a particular connection can be higher.
• Upper-level protocol information defined in the network-level headers, so
low-level hardware can effectively assist higher-level protocol processing.
Again, the network layer for Fibre Channel is not much different than
modern Ethernet on a switched fabric (i.e., not shared medium), with link-
level backpressure flow control. There are some advantages to the Fibre
Channel network vs. Gigabit Ethernet, but not a 30X difference.
LANs vs. SANs: Differences in Protocol Design
The more important advantages in SAN efficiency vs. LAN efficiency
and performance relate to the higher levels of protocol design, and have to
do with the fact that LANs are, in general, accessed through a TCP/IP (or
UDP/IP) protocol stack, where SANs are accessed through a simpler SCSI
protocol stack with less overhead on the host processor. This include the fol-
lowing factors.
• Lower-lever error checking. The channels deliver the data to the server
intact, or not at all (data corruption, or pulled cable) — so the processors
do less checksum calculation or validation of header fields, for example.
• Predictable network performance
• Ordered transmission — assume no re-ordering of traffic on the network,
so the extra overhead associated with checking for correct delivery order,
and resource allocation to compensate if you don’t have it, are gone.
• Well-defined network round-trip times, so that the protocol doesn’t have
to include code to handle the “did the packet get lost, or is it just badly
delayed?” problem.
• Request/Response network — the server makes requests to the disk sub-
system for reads or writes, so all incoming packets to the server are
expected packets. This means:
• Less header parsing and less handling of special cases, since all packets
coming in are expected, and resources for dealing with them have been
pre-allocated.
• Less overhead for flow control — no need to allocate buffer space or do
buffer management processing for traffic which may or may not come in.
• Message-based transport: TCP is a sockets stream protocol, where SCSI
works in command or data blocks, or messages, with memory space pre-
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.
Fibre Channel and Storage Area Networks
10
Chapter 1
allocated, so less buffer management, and less data copying, are required
in many cases.
• Higher granularity of transfers — Ethernet adapters typically work at the
level of Ethernet packets, with all higher-level segmentation and reassem-
bly into IP datagrams, or TCP-level sockets, requires host processor inter-
vention. Fibre Channel adapters typically do reassembly of Frames into
Sequences, and deliver the full Sequence to the ULP for processing by the
host processor. This means, for example, that there may be fewer processor
interrupts, and less context switching.
• Real address operations — SCSI protocols work in the kernel, so there’s no
switching from user context to kernel context, and real addresses can be
used in all the operations, so may be less translation between virtual and
physical addresses.
Network-attached Storage (NAS) and Storage Area
Networks (SAN)
An area that is closely tied to this difference between LANs and SANs is the
difference between NAS and SANs. It is sometimes difficult to be sure of the
function difference between the two, partly because they nearly share an
acronym, and partly because they both allow networked access to stored
data. However, they really are quite different from each other, both in func-
tionality and how they are used.
Part of the difference between Network-attached Storage, and a Storage-
Area Network has to do with the network and protocol stack used. Network
attached storage emphasizes the network: Ethernet networks and TCP/IP or
UDP/IP protocol stacks), where Storage Area Networks use Fibre Channel
and a SCSI protocol stack.
The hardware difference is less important than the higher layer differ-
ences, however, particularly if both networks operate at nearly the same
speed and topology.
A more important key to the difference between NAS and SAN is the dis-
tinction in which kind of traffic crosses the network. In NAS, the traffic
crossing the network is high-level requests and responses for files, independ-
ent of how they are arranged on disks. In SAN, however, the traffic is
requests and responses for blocks of data at specific locations on specific
disks.
The difference here is that NAS operates above the file system level,
where SANs operate below the file system level, at the data block level.
A network-attached storage device is a dedicated file server which holds
files, and exports to the clients a picture of a file system. The clients request
reads or writes to files, and the network-attached storage device does the
Downloaded from Digital Engineering Library @ McGraw-Hill (www.digitalengineeringlibrary.com)
Copyright © 2004 The McGraw-Hill Companies. All rights reserved.
Any use is subject to the Terms of Use as given at the website.