Table Of ContentAUTOMATICTEXTAREASEGMENTATIONINNATURALIMAGES
SyedAliRazaJafri,MireilleBoutin,andEdwardJ.Delp
SchoolofElectricalandComputerEngineering
PurdueUniversity
465NorthwesternAv.
WestLafayette,IN47907
8
0
ABSTRACT itself. Findingbackgroundsisalotsimplerthanfindingtext
0
2 directly. Itcanbeaccomplishedrobustlybyextractingsome
We present a hierarchical method for segmenting text areas
well chosen texture features. Once a potential background
n innaturalimages. Themethodassumesthatthetextiswrit-
a area has been selected, we then use a combination of shape
tenwithacontrastingcoloronamoreorlessuniformback-
J andcolorfeaturestodetectwhethertextispresentinsidethe
ground. But no assumption is made regarding the language
1 area. Havingpre-identifiedthebackgroundprovidesuswith
or character set used to write the text. In particular, the text
3 asampleofthebackgroundcolorandtexture, andthussim-
can contain simple graphics or symbols. The key feature of
plifiestheproblemofdeterminingwhetherthereistextonthe
] ourapproachisthatwefirstconcentrateonfindingtheback-
V background. Thesearchforthetextareaisperformedhierar-
groundofthetext,beforetestingwhetherthereisactuallytext
chicallyinatop-downfashion: ifnotextisfoundatagiven
C on the background. Since uniform areas are easy to find in
scale,thenwelookfortextatasmallerscale. Thisallowsus
. naturalimages,andsincetextbackgroundsdefineareaswhich
s to find the text without making prior assumptions regarding
c contain ”holes” (where the text is written) we thus look for
thefontandareasizes.
[
uniformareascontaining”holes”andlabelthemastextback-
1 groundscandidates.Eachcandidateareaisthenfurthertested There is an extensive literature on the problem of seg-
v forthepresenceoftextwithinitsconvexhull. Wetestedour mentingtextareasinimages. Manymethodsarespecifically
7 method on a database of 65 images including English and designed to deal with special types of images, such as doc-
0
Urdu text. The method correctly segmented all the text ar- uments [1], webpages [2], or pictures of newspaper [3]. For
8
eas in 63 of these images, and in only 4 of these were areas eachtypeofdocuments,assumptionsaremaderegardingthe
4
. thatdonotcontaintextalsosegmented. type of text present in the image and its surrounding. For
1
example, for newspapers it can be assumed that the text is
0 IndexTerms— textsegmentation,uniformtexture.
writteninblackonawhitebackground,fortextwrittenusing
8
0 theromanalphabetitcanbeassumedthatthecharactersare
: 1. INTRODUCTION writtenatregularintervals, etc. Oneproblemthatresembles
v
theproblemoffindingtextinnaturalareaisthatofsegment-
i
X Finding areas of text in a natural image is a difficult prob- inglicenseplatesonnaturalimagescontainingcars[4,5],on
r lem. Onereasonforthisisthat,fromatextureandgeometric whichalotofefforthasbeenput.
a
standpoint,manyobjects(e.g.,treebranchesorelectricalwire
Thegeneralproblemoffindingtextinnaturalimageshas
on a blue sky) resemble text. In addition, the text size, font,
also been considered [6, 7, 8]. Many text area segmenta-
color, orientation and skew are generally unpredictable. In
tion methods formulate the problem as a two-class decision
the following, we introduce an efficient method for finding
problem: is the given block/pixel part of a text area or not?
text areas in natural images. The only assumption we make
Then the problem is reduced to finding good features for
regarding the text is that it is written on a more or less uni-
formulatingthisdecisionproblemandfindingagoodclassi-
formbackgroundusingacontrastingcolor. Inparticular,the
fierformakingdecisions. Ingeneral, threetypesoffeatures
textcouldbewrittenincursivelettersorwithforeigncharac-
are used for identifying the text: texture based, edge based
tersandevencontainsimplegraphics. Weproceedbyusing
or connected-component based. Sometimes, three or more
a combination of texture and shape features in order to se-
classes are considered (e.g., text, background, and other) in
quentiallyruleoutregionsoftheimagewhichdonotcontain
ordertoincreasetheaccuracyoftheclassification. Butfalse
text. Oneofkeyelementsofourapproachisthatwebeginby
positives always constitute a nuisance and the parameters
focusing on finding the text background, rather than the text
of the classifier must be carefully tuned in order to avoid
ThisworkwassupportedbyagrantfromNextWaveSystems,LLC. them as much as possible. In contrast, we do not look text
directly, but rather find the text by first finding likely text
contextsandstudyingthefeaturesofeachpotentialtextcon-
texttodecidewhetherornotitcontainstext. Falsepositives
in the early stages thus do not constitute a problem, and so
we can conservatively estimate the thresholds of the early
decisionparameters. Thedetailsofourapproacharegivenin
the next section. In Section 3, we present our experimental
methodologyandresultsbeforeconcludinginSection4.
2. PROPOSEDSEGMENTATIONMETHOD
All the text areas that can be found in natural images have
littleincommon. Theycanbewrittenindifferentlanguages
(e.g.,English,Korean,Arabic)withdifferentfonts,usingdif-
ferent colors, and on different backgrounds. Shadows and
occlusions can also be present. The image acquisition pro-
cess adds further variability such as camera skew, distance,
lighting and motion blur. But almost all the text that is eas-
ily noticeable by humans has an important characteristic: it
is written on a background that is more or less uniform, at
least piecewise. Another common characteristic is that the
textcolortypicallycontrastswiththebackgroundcolor.
In order to look for text, we thus begin by looking for
image areas where the color is more or less uniform. So we
dividetheimageintoblocksofequalsizeandtesteachblock
foruniformity. Theinitialblocksizeissettoonequarterthe
image size. Once uniform blocks have been identified, we
groupthembasedoncolorsimilarityandproximity. Groups
whose shape is incompatible with that of a text background
are then ruled out. The interior of the convex hull of each
group that remains is then tested for the presence of text. If Fig.1.SchematicRepresentationoftheStepsoftheProposed
no text is found, then a smaller block size is chosen and the AreaSegmentationMethod.
above steps are repeated. A schematic representation of this
process is given in Figure 1. We now describe each step in
matrix
moredetails.
p r g b
1 1 1 1
p2 r2 g2 b2
2.1. UniformBlockDetection . = . . . .
. . . .
. . . .
Oncetheimageisdividedintoblocksofsizek×k,weselect p r g b
k2 k2 k2 k2
the blocks that have a uniform texture. One way to do this
Then each row of the above matric (which corresponds
consistsinlookingatthecolordistributionofthepixelscon-
to a k ×k block of a single color) is projected onto a basis
tained in the block, and in checking to what extent this 3-D
distributionisscattered: alowamountofscatteringisanin- {v1,v2,...,vk2} of Rk2 which satisfies the following con-
dicationofauniformcolor. However,thiswouldnotaccount ditions:
forthefactthatsomebackgroundsaremadeofpixelsofdif-
1. theentriesofthebasisvectorsconsistof+1’sand−1’s
ferentcolorsthatarearrangedinsuchawaythattheydonot
form any visible higher level structures. Such backgrounds
2. the sum of the entry of any basis element is equal to
dolookuniform,eventhoughtheircolordistributionmaybe
zero.
veryscattered.
We therefore use an alternative approach to quantify the In other words, the value of the projection of a row of the
uniformity of a given block. We consider each k ×k color matrixontoanyoneofthesebasisvectorsquantifiesthedif-
blockasasumofthreeone-colorblocks: aredone, agreen ferencebetweentheamountofcolorontworegionsofequal
one and a blue one. More precisely, after concatenating the size within the block. Some elements of such a basis are il-
rows of pixels of the block, we can represent it as a k2 ×3 lustrated in Figure 2. The basis elements can be viewed as
filterswhichquantifyuniformityofthecoloroftheblock:the 2.2. UniformBlockGrouping
smallertheprojectionofthethreecolorblocksontotheseba-
Oncetheuniformblockshavebeenselected,wegroupthem
siselements,themoreuniformthetextureofthegivencolor
together in order to form uniform regions. We begin by
block.
grouping sets of connected blocks based on color similarity.
Wecanwrite
Moreprecisely,wegrouptogetherconnecteduniformblocks
ifthedistancebetweentheirmeancolorvectorislessthan45.
r g Again,thisthresholdvaluewaschosenempirically. Abetter
1 1
r2 (cid:88)k2 g2 (cid:88)k2 value could be obtained from a training set. Once we have
.. = αivi, .. = βivi, obtainedconnecteduniformregions,wemergetheseregions
. i=1 . i=1 basedoncolorsimilarityandbasedonthevariationofcolor
r g
k2 k2 inthespacebetweenthem. Moreprecisely,wemergeregions
b suchthat
1
b2 (cid:88)k2
.. = γivi, 1. thedistancebetweentheirmeancolorvectorislessthan
. i=1 45;
b
k2
2. thedistributionofcolorsbetweenthetworegionsisei-
therunimodalorbimodal. Ifitisbimodalwelookfor
whereα isequaltothedotproductbetween(r ,...,r )T
i 1 k2
atleastadistanceof100betweenthetwopeaks.
and the basis vector v divided by the norm of v , and sim-
i i
ilarly for β end γ . Observe that a block has a completely
i i
uniformcolorifandonlyifallthecoefficientsα ’s,β ’sand 2.3. ShapeTestforGroupsofUniformBlocks
i i
γ ’sarezero.
i
Since the image areas containing the text itself are not uni-
Inourmethod,wetakeeachblockoftheimageandobtain
form, then any uniform region corresponding to the back-
thecoefficientsα ,β andγ oftheexpansionforeachcolor
i i i ground of a sign must have ”holes”. In other words, we as-
of the block. We then choose the blocks for which the L
2 sumethatthetextisatleastpartiallysurroundedbyauniform
normofthesecoefficientsislessthanonestandarddeviation
area. Anyselecteduniformareawhichisconnectedandcon-
below the mean of the other blocks inside the picture. This
vexisthuseliminated. Thissimplesteprulesoutmostofthe
thresholdwaschosenempirically. Forbetterperformance,it
uniform regions identified with the previous steps. The few
couldbelearnedfromatrainingset.
remainingregions(ifany)gothroughthenextandfinalstep
of our method. Note that one often needs to reach a small
scale before a uniform region with an appropriate shape is
identified.
2.4. TestforTextInsideConvexHullofGroup
Given a uniform area with holes, we obtain the convex hull
ofthatareaandtestwhethertextispresentinside. Thereare
manysophisticatedwaystotestforthepresenceoftext. But
many of these ways are language specific. Fortunately, we
found that considering the color distribution of the convex
hull of the uniform region was enough to discriminate be-
tween text and non-text in most circumstances. Our test is
based on the fact that text areas contain few and contrasting
colors. We thus look for color distribution which are multi-
modal and with a large distance between the modes. More
precisely,weexpandouruniformregionsusingsmallerblock
sizes until we cover as much of the background as possible.
We then determine the average color intensity of the set of
points within the convex hull that are not part of the back-
Fig. 2. Some Basis Vectors used as Filters to Quantify the
ground. The color distance between background and non-
Uniformity of the Blocks. The colors black and white are
backgroundpointsprovidesanaccuratefeaturetodetermine
usedtorepresent+1and−1entriesrespectively.
whichregionscontaintext. Currentlyweuseacolordistance
of100asaconservativeminimumdistancebetweentextand
(b) (c) (d)
(a) (e) (f) (g)
Fig.3. AFewSamplesofourExperimentalResults. (a)Textofvaryingsizesandcolor(includinggraphics). (b)Streetsignin
frontofasmoothbackground(sky). (c)Non-rectangulartextarea. (d)TextwritteninEnglishandUrdubotharesuccessfully
segmented. (e)Streetsigninfrontofatexturedbackground(cement). (f)Textprintedonanirregularsurface. (g)Shopdisplay.
background. Again, training could be used to select a better based) in order to improve the quality of the last step of our
threshold. method.
5. REFERENCES
3. EXPERIMENTALRESULTS
[1] S.; Hirogaki Y. Amamoto, N.; Torigoe, “Block segmentation
Wetestedourmethodonadatabaseof65(threemegapixel) andtextareaextractionofvertically/horizontallywrittendocu-
images of outdoor signs and shop displays. Ten of these ment,”DocumentAnalysisandRecognition,1993.,Proceedings
images contained outdoor signs written in both English and oftheSecondInternationalConferenceon,pp.739–742,20-22
Oct1993.
Urdu. The rest (55 images) contained English signs only,
butsomeincludedsimplegraphicsaswell. Allthetextarea [2] T.Park,D.Kim,andK.Chung,“Orientationandscaleinvariant
was correctly segmented in 63 (i.e., 97%) of these 65 im- textregionextractioninwwwimages,” 1998,pp.xx–yy.
ages. Infourofthese63images,someotherareaswerealso [3] Q.YuanandC.L.Tan, “Textextractionfromgrayscaledocu-
segmented as well. However, these areas all contain highly mentimagesusingedgeinformation,” 2001,pp.302–306.
contrasting high level structures on a uniform background [4] Javier Cano and Juan Carlos Pe´rez-Cortes, “Vehicle license
whichinmanywaysresembletext(forexample,acapital”i” plate segmentation in natural images,” in IbPRIA, Francisco
letter)butcouldberuledoutfromasemanticpointofview. J. Perales Lo´pez, Aure´lio C. Campilho, Nicolas Pe´rez de la
Blanca, andAlbertoSanfeliu, Eds.2003, vol.2652ofLecture
NotesinComputerScience,pp.142–149,Springer.
4. CONCLUSION [5] G.Shapiro,V.;Gluhchev, “Multinationallicenseplaterecogni-
tion system: segmentation and classification,” Pattern Recog-
Wehavepresentedatop-downhierarchicalmethodsforfind- nition,2004.ICPR2004.Proceedingsofthe17thInternational
ingtextareasinnaturalimages. Thekeypointofthismethod Conferenceon,vol.4,pp.352–355Vol.4,23-26Aug.2004.
is that it begins by looking for text background areas before [6] YuZhong,K.Karu,andA.K.Jain, “Locatingtextincomplex
testingforthepresenceoftextinsidetheselectedareas. The colorimages,” icdar,vol.01,pp.146,1995.
methodcorrectlysegmentedallthetextin97%oftheimages [7] X.Garcia,C.;Apostolidis, “Textdetectionandsegmentationin
in a small database of outdoor signs and shop displays. In complexcolorimages,” Acoustics,Speech,andSignalProcess-
future work, we will test the method on a larger database of ing, 2000.ICASSP’00.Proceedings.2000IEEEInternational
Conferenceon,vol.6,pp.2326–2329vol.4,2000.
natural images. To improve the results, we will use training
tochoosetheoptimalparametersforallthedecisionsweper- [8] M.PietkainenandO.Okun, “Edge-basedmethodfortextde-
tectionfromcomplexdocumentimages,” 2001,pp.286–291.
form. Wewillalsoinvestigatetheuseofmoresophisticated
text presence test (e.g., edge based or connected component
