Digital Video Processing for Engineers

217 Pages·2012·4.921 MB·English

DIGITAL VIDEO PROCESSING
FOR ENGINEERS
A Foundation for
Embedded Systems Design
MICHAEL PARKER, SUHEL DHANANI
AMSTERDAM(cid:1)BOSTON(cid:1)HEIDELBERG(cid:1)LONDON
NEWYORK(cid:1)OXFORD(cid:1)PARIS(cid:1)SANDIEGO
SANFRANCISCO(cid:1)SINGAPORE(cid:1)SYDNEY(cid:1)TOKYO
NewnesisanImprintofElsevier
NewnesisanimprintofElsevier
TheBoulevard,LangfordLane,Kidlington,OxfordOX51GB,UK
225WymanStreet,Waltham,MA02451,USA
Firstedition2013
Copyright(cid:1)2013ElsevierInc.Allrightsreserved
Nopartofthispublicationmaybereproduced,storedinaretrievalsystemortransmittedinany
formorbyanymeanselectronic,mechanical,photocopying,recordingorotherwisewithouttheprior
writtenpermissionofthepublisher
PermissionsmaybesoughtdirectlyfromElsevier’sScience&TechnologyRightsDepartmentin
Oxford,UK:phone[phone];fax[phone];email:[email protected].
AlternativelyyoucansubmityourrequestonlinebyvisitingtheElsevierwebsiteathttp://elsevier.com/
locate/permissions,andselectingObtainingpermissiontouseElseviermaterial
Notice
Noresponsibilityisassumedbythepublisherforanyinjuryand/ordamagetopersonsorproperty
asamatterofproductsliability,negligenceorotherwise,orfromanyuseoroperationofany
methods,products,instructionsorideascontainedinthematerialherein.Becauseofrapidadvances
inthemedicalsciences,inparticular,independentverificationofdiagnosesanddrugdosagesshould
bemade
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressCataloging-in-PublicationData
AcatalogrecordforthisbookisavailabefromtheLibraryofCongress
ISBN:978-0-12-415760-6
ForinformationonallNewnespublicationsvisit
ourwebsiteatbooks.elsevier.com
PrintedandboundintheUSA
1213141516 10987654321
1
VIDEO IN THE MODERN WORLD
Videobeganasapurelyanalogtechnology.Successiveimages
werecapturedonfilmstreamingthroughthecamera.Themovie
was played by using flashes of light to illuminate each frame on
the moving film, at rates sufficient to show continual motion.
Flicker, however, was easilyseen.
Animprovedsystemforearlybroadcasttelevisionutilizedthe
luminance (or light intensity) information represented as an
analog signal. To transmit an image, the luminance information
was sent in successive horizontal scans. Sufficient horizontal
scansbuiltupatwo-dimensionalimage.Televisionsandmonitors
used cathode ray guns that shot a stream of electrons to excite
aphosphorus-coatedscreen.The slowlyfadingphosphorus ten-
ded to eliminate flicker. The cathode gun scanned in successive
rows,each rowjustbelow the previousrow,guidedby magnetic
circuits. Thishappenedsorapidlythat imageswere“painted”at
a rate of 25 to 30 frames per second (fps). The luminance signal
wasusedtocontroltheintensityoftheelectronstream.
A horizontal synchronization signal is used to separate the
horizontal scan periods. The horizontal sync period is a short
pulseattheendofeachscanline.Thishastobelongenoughto
allowtheelectronguntomovebacktotheleftsideofthescreen,
in preparation for the next scan line. Similarly, the vertical
synchronization signal occurs at the end of the last or bottom
scan, and is used to separate each video frame. The vertical
synchronizationintervalismuchlonger,and allows the electron
guntomovebackupfromthelowerrightcornerofthescreento
the upper leftcorner,to begin anew frame.
Later,colorinformationintheformofredandbluehueswas
added, known as chrominance information. This was super-
imposed on the luminance signal, so that the color system is
backwards compatible to the black-and-white system.
Modern video signals are represented, stored and trans-
mitted digitally. Digital representation has opened up all sorts
of new usages of video. Digital video processing is of grow-
ing importance in a variety of markets such as video surveil-
lance; video conferencing; medical imaging; military imaging
DigitalVideoProcessingforEngineers.http://dx.doi.org/10.1016/B978-0-12-415760-6.00001-5 1
Copyright(cid:1)2013ElsevierInc.Allrightsreserved.
2 Chapter 1 VIDEOIN THE MODERNWORLD
Figure 1.1. Videorasterscan.
including UAVs (Unmanned Aerial Vehicles), weapons sights
and night vision; broadcast; digital cinema; industrial displays
and consumer electronics. All these sectors are embarking on
a massive decade-long upgrade cycle from standard definition
to HD-and higher than HD-resolution video processing. In
some cases the old analog video-processing equipment is
being replaced by systems using digital techniques.
In many cases, industries that have not traditionally been
involvedinvideoprocessingmustnowintegratethistechnology
into their products. Examples are rear cameras, entertainment
centers, “lane departure” and “head-up” displays in automo-
tives; video data handling in networking servers and routers;
sharingandmergingofvideotoprovidesituationalawarenessin
military systems; surveillance and guidance in military and
commercial airborne systems; robotic systems; facial and other
features recognition (such as license plates) in security surveil-
lance systems and myriad other applications. This trend is
placing new requirements on system designers as well as
implementationengineerstounderstandvideotechnology.This
book is designed for those individuals who need to understand
basic concepts and applications, so that they can either build
their own video systems, or integrate third-party video tech-
nology into their products.
Another target audience is those involved in technical
marketing and sales and executives from the many industries
Chapter 1 VIDEOIN THE MODERNWORLD 3
requiring video technology. Again, the need is to understand
basic concepts and applications, without being overwhelmed by
the details and implementation complexity.
Themarketsizesofthesenewvideoapplicationsaregrowing
rapidly. For example, here are some publically available
projections:
• ABI Research believes that the video surveillance market is
poised for explosive growth, which the firm forecasts to
expandfromrevenueofabout$13.5Bin2006toaremarkable
$46B in 2013. Those figures include cameras, computers and
storage, professional services, and hardware infrastructure:
everything that goes into anend-to-end security system.
• According to Wainhouse Research, the overall endpoint
market for video conferencing will grow from $1.3B in 2007
to over $4.9B in 2013. Videoconferencing infrastructure
product revenues, including MCUs, gateways, and gate-
keepers, are forecast to grow to $725Mover the same period.
• HD penetration rates: there is still a lot of work to be done to
develop, store, edit and transmit HD signals within both the
USA and Europe.
Digitalcinemaisrampingupwithinthenextfiveyearse10,000
US theaters are to be upgraded in 2010e2011. Digital cinemas
drivesignificantdesignactivityinHDand4Kvideoprocessing.
The$16BUS medical-imaging productindustrywillgrow six
percent annually in the course of 2010 based on technological
advances, aging demographics and changing health care
approaches. Equipment will outpace consumables, led by CT
scanners and by MRI and PET machines.
Allofthesetrendsandmanymoreleadustobelievethatthere
isatremendousandgrowingdemandforabookthatdemystifies
video processing. Professional engineers, marketers, executives
and students alikeneed to understand:
• What is video – in terms ofcolors, bits and resolutions?
• What are the typical ways that video is transported?
• Whatfunctionsaretypicalinvideoprocessing–scaling,dein-
terlacing, mixing?
• What are the typical challenges involved in building video-
processing designs – frame buffering, line buffering, memory
bandwidth, embedded control, etc.?
• What is video compression?
• How is video modulated,encoded and transmitted?
These concepts provide a solid theoretical foundation upon
which the reader can build their video knowledge. This book
intends to be the first text on this subject for these engineers/
students.
2
INTRODUCTION TO VIDEO
PROCESSING
CHAPTER OUTLINE
2.1 DigitalVideo: Pixels andResolution 5
2.2 DigitalVideo: Pixels andBits 6
2.3 DigitalVideo: Color Spaces 8
2.4 Video Processing Performance 9
Videoprocessingethemanipulationofvideotoresize,clarify
or compress it e is increasingly done digitally and is rapidly
becomingubiquitousinbothcommercialanddomesticsettings.
This book looks at video in the digital form e so we will talk
aboutpixels,colorspaces,etc.Westartwiththeassumptionthat
video is made of pixels, that a row of pixels makes a line, and
a collection of lines makes a video frame. In some chapters we
will briefly discuss the older analog format but mainly in the
context of displayingit ona digital display.
Since this is an introductory text, and is meant to serve as
a first book that clarifies digital video concepts, digital video is
explained primarily through pictures, with little mathematics.
2.1 Digital Video: Pixels and Resolution
Digitalvideoismadeofpixelsethinkofapixelasasmalldot
onyourtelevision screen.Therearemanypixels inoneframeof
video and many frames within one second e commonly60fps.
WhenyoulookatyourTVtherearevariousresolutionssuchas
standarddefinition(SD),highdefinition(HD)with720porhigh-
definition with 1080p. The resolution determines how many
pixelsyourTVshowsyou.Figure2.1showsthenumberofpixels
for these different resolutions e as you can see the same video
frame for a 1080p TV is represented by a little over two million
pixels compared to only about 300,000 pixels for standard defi-
nition.No wonderHDlooks sogood.
DigitalVideoProcessingforEngineers.http://dx.doi.org/10.1016/B978-0-12-415760-6.00002-7 5
Copyright(cid:1)2013ElsevierInc.Allrightsreserved.
6 Chapter 2 INTRODUCTION TO VIDEOPROCESSING
1080p
1920 ×1080
2,073,600 pixels
~2 ×more pixels
720p
1280 ×720
921,600 pixels
~3 ×more pixels
480p
720 ×480
345,600 pixels
~6 ×more pixels
Figure 2.1. Increasingnumberof
pixelsineachframeofvideo
It might be interesting to note that the old cathode ray tube
(CRT)TVshadonlyhalfofthepixelsofevenSDresolutionshown
hereesogoingfromaCRTTVtoanew1080pTVjustgaveyour
eyes a feast of 12 times more pixels for each video frame.
The number ofpixels makesa huge difference.
Takeanother exampleewhen Applecreatedthe new‘retina’
display on the iPhone 4 it proved extremely popular with
consumers.ThenewiPhone4hadaresolutionof940(cid:1)640pixels
compared to the old iPhone 3, which had a resolution of
320(cid:1)480.SoApplefoundawaytoincreasethenumberofpixels
on the same size screen by afactor offour.
The number of pixels also determines the complexity of the
hardwareusedtomanipulatethesepixels.Sinceallmanipulation
is in termsof bits, let’s see how pixelstranslate to bits.
2.2 Digital Video: Pixels and Bits
Each pixel has a unique color which is a combination of the
primarycolors:red,blueandgreen.Howmuchofred,howmuch
ofblueandhowmuchofgreenisthekey.Andthis“howmuch”is
describedpreciselybythevalueofthepixel.Thevalueofthepixel
is represented by bits and the more bits are available, the more
accurate the representation. Bear in mind however, that bits are
expensivetostore,tomanipulateandtotransmitfromonedevice
to the other.So a happy balance mustbe realized.
Chapter2 INTRODUCTION TOVIDEOPROCESSING 7
Each pixel has a red (R), green (G) and blue (B) component.
There are other ways to describe this as well, but we will look at
red,blueandgreenfirst.Let’ssaythatyouuseeightbitstostore
thevalueofred,eightbitsforblueandeightbitsforgreen.With
eightbitsyoucanhave28or256differentpossiblevaluesforred,
blueandgreeneach.Whenthisisthecase,peoplerefertothisas
a color depth ofeight, oran 8-bit colordepth.
Some HD video will be encoded with 10-bit color depth or
even12-bitcolordeptheeachRGBcomponentisencodedwith
10 or12 bits.
While more is better, remember that these bits add up.
Consider 8-bit color depth. Each pixel requires 8(cid:1)3¼24 bits to
represent its value.
Nowthinkaboutaflat-panelTVinyourhouse.Youprobably
remember that this TV is 1080p e the salesperson probably also
talkedabout1920(cid:1)1080resolution.Whatthismeansisthateach
video frame shown on this flat-panel TV has 1080 lines and that
eachlinehas1920pixels.Soyouwerealreadytalkingaboutpixels
all the time e even thoughit may nothave registered.
Let’s put it together. Since each pixel requires 24 bits, and
thereare1920pixelsperlineandthereare1080linesinoneframe
of video, this means that your hard-working flat-panel TV is
showing you information that is 24(cid:1)1920(cid:1)1080¼49,766,400
bitsineachframe.Approximately50millionbits;alsoreferredto
as50Mbits.AndremembermostTVsgothrough60framesinone
second.Someof the newer ones even go through 120fps.
Sotogiveyoutheviewingpleasureforonesecondwehaveto
manipulate 3 billion bits, also referred to as 3Gbits. And this is
with 60fps with acolordepth of8. It could be higher.
Table2.1showsthenumberofbitsrequiredforeachframeat
differentresolutions.Herewehaveused30bitsperpixelandalso
Table 2.1
Frame Size: (Total# of Frame Size: (Assume
Image Size Pixels) 30Bits per Pixel)
1920(cid:1)1080p 1920(cid:1)1080=2Mpixels 60Mbits
1920(cid:1)1080i 1920(cid:1)1080(cid:1)0.5=1M 30Mbits
pixels
1280(cid:1)720p 1280(cid:1)720=900Kpixels 27Mbits
SD720(cid:1)480p 720(cid:1)480(cid:1)0.5=173K 5.19Mbits
pixels
8 Chapter 2 INTRODUCTION TO VIDEOPROCESSING
showntheeffectofinterlacedvideoefornowjustrememberthat
theresolutionishalvedwhenthevideoisinterlaced.Thetableis
meanttomakeyouawareoftheamountofbitsthatareprocessed
when working with digital video. Digital video processing is
ademandingcomputationaltaskeespeciallyatHDresolutions.
Andtheprimaryreasonisthesheernumberofpixels(andhence
bits) involved.
2.3 Digital Video: Color Spaces
Acolorspaceisamethodbywhichwecanspecify,createand
visualize color. Each pixel has a certain color, which in simple
termscanbedescribedasacertaincombinationofred,blueand
green. Let’s representeach valueofthe color byeight bits.Ifthe
pixel is completely red, the R component of the pixel would be
11111111andtheothertwocomponents(blueandgreen)would
be 0000 0000.
When these values are added together all we see is red. If the
other two color values are not zero then the resultant color is
acombination ofred and some blueand some green. This color
space is additive e the resultant pixel color is the sum of the
intensities ofeach ofthe colors. See Figure 2.2.
The RGB color model is used to display colors on older CRT
TVsaswellastoday’sLCDTVs.Eachvaluedrivestheexcitations
of red, green and blue phosphors on the CRT faceplate. And for
digital TVs the resultant pixel value stored in hardware is con-
vertedtovoltagethatfiresthatpixelonthescreen.Thereismore
to this, including accounting for gamma correction, but we will
look at that later.
Printers describe a color stimulus in terms of the reflectance
and absorbance of cyan, magenta, yellow and black inks on the
paper. So they work in adifferent colorspace.
R
G
PIXEL VALUE
B
Figure2.2.
Chapter2 INTRODUCTION TOVIDEOPROCESSING 9
Figure 2.3.
There are many color spaces e one of the more interesting
onesistheYCrCbcolorspace.Thisisacolorspacerepresentation
in which you code the pixel value in terms of its brightness
(luminance), and Cr, Cb which is a combination of RGB. This
methodofrepresentingcolorisveryusefulsincethehumaneyeis
verysensitivetobrightnessorluminance,andmuchlesssensitive
tocolor.Whenthepixelvalueisbrokendownintoluminanceand
color,wecangetawaywithusingfewerbits(lowerresolution)to
encodethecolorinformationasthehumaneyecannotdetectthe
difference.
YCrCbisanotherwayofencodingtheRGBcolorseandusing
fewer bits in the process e but before the video is displayed we
mustreconvert everything to RGB.
Thewayyouconvertapixelvaluefromonecolorspace(RGB)
to another (YCrCb) is to multiply each color component in the
RGB space with a fixed constant e see Figure 2.3.
Intermsofhardwareallyouneedismultipliersandaddersto
implement the operation. Any decent processor can do this
FPGAs (field-programmable gate arrays) of course can do this
elegantly and very fast given their inherent DSP (digital signal
processing) capabilities.
Whenyoustartconvertingapixelvaluefromonecolorspace
to another there are multipleconversions in each stage.
For example:
ConvertRGBto YCrCb/TRANSMIT
/Convert back to RGB /PROCESS THE VIDEO
/Convert back to YCrCb/TRANSMIT
/Convert back to RGB
/DISPLAY
2.4 Video Processing Performance
Anyvideoprocessingsignalchainisboundtohavemanycolor
space conversions along the way. These conversions have to be
done at the pixel rate, whichfor HDvideo is very high.
Consider 1920(cid:1)1080 with 60fps. 1920(cid:1)1080(cid:1)60 pixels are
comingineachsecond.Whichmeans124.4millionpixelsineach