Model-based dewarping method and apparatus

41 Pages·2013·1.93 MB·English

US008406476B2
(12) United States Patent (10) Patent N0.: US 8,406,476 B2
Wu et a]. (45) Date of Patent: Mar. 26, 2013
(54) MODEL-BASED DEWARPING METHOD AND (56) References Cited
APPARATUS
U.S. PATENT DOCUMENTS
(75) Inventors: Minghui Wu, Beijing (CN); Rongfeng
Li, Beijing (CN); WenXin Li, Beijing 4,783,833 A 11/1988 Kawabata et al.
(CN); Edward P. Heaney, Jr., Long 5,280,367 A * 1/1994 Zuniga ........................ .. 358/462
5,280,546 A 1/1994 Machida et al.
Beach, CA (US); Karl M. Chan, 5,377,019 A 12/1994 Okisu et al.
Huntington Beach, CA (US); Kurt A. 5,515,181 A 5/1996 Iyoda et al.
Rapelje, Hawthorne, CA (US); Bin Fu, 5,585,962 A 12/1996 Dixon
Beijing (CN); Bin Feng, Bellevue, WA 5,677,776 A 10/1997 Matsuda et al.
5,742,354 A 4/1998 Vlahos et al.
(Us)
5,764,228 A 6/1998 Baldwin
(73) Assignee: Compulink Management Center, Inc., (Continued)
Long Beach, CA (US)
FOREIGN PATENT DOCUMENTS
Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 EP 1571820 A2 9/2005
USC 154(b) by 1034 days.
OTHER PUBLICATIONS
(21) Appl. No .: 12/2s1,315
Daniel S. Le, George R. Thomas, Harry Wechsler, “Automated Page
(22) PCT Filed: Mar. 2, 2007
Orientation and Skew Angle Detection for Binary Document
(86) PCT No.: PCT/US2007/005492 Images” Pattern Recognition, vol. 27, No. 10, pp. 1325-1344, 1994,
Great Britain.
§ 371 (0X1)’
(2), (4) Date: Mar. 27, 2009 (Continued)
(87) PCT Pub. No.: WO2007/103251
Primary Examiner * Anand Bhatnagar
PCT Pub. Date: Sep. 13, 2007 (74) Attorney, Agent, or Firm * Dickstein Shapiro LLP
(65) Prior Publication Data
US 2010/0239165 A1 Sep. 23, 2010 (57) ABSTRACT
Related US. Application Data An apparatus and method for processing a captured image
(63) Continuation-in-part of application No. 11/368,260, and, more particularly, for processing a captured image com
?led on Mar. 2, 2006, now Pat. No. 7,330,604. prising a document. In one embodiment, an apparatus com
prising a camera to capture documents is described. In
(51) Int. Cl.
another embodiment, a method for processing a captured
G06K 9/00 (2006.01)
image that includes a document comprises the steps of dis
(52) US. Cl. ...................................... .. 382/112; 702/150 tinguishing an imaged document from its background, adjust
(58) Field of Classi?cation Search ................ .. 382/ 100, ing the captured image to reduce distortions created from use
382/103,101,112,181,182,190,195,199, of a camera and properly orienting the document is described.
382/206, 276, 281, 2894292, 286, 287; 356/138;
702/150, 151
See application ?le for complete search history. 11 Claims, 27 Drawing Sheets
TEXT AREA SEGMENTATION
1
LOCAL DIRECTION
ENHANCEMENT
1
TEXT LINE EXTRACTION
l
TEXT LINE SELECTION
US 8,406,476 B2
Page 2
U.S. PATENT DOCUMENTS 2006/0045379 A1 3/2006 Heaney et a1.
2007/0206877 A1 9/2007 Wu et al.
5,764,383 6/1998 Saund et al.
2007/0253031 A1 11/2007 Fan
5,831,750 11/1998 Okisu et al.
5,848,183 12/1998 Farrell OTHER PUBLICATIONS
6,011,635 1/2000 Bungo et al.
6,014,470 1/2000 Matsuda Aditya Vailaya, HongJiang Zhang, Changjiang Yang, Feng-I Liu,
6,064,778 * 5/2000 Pasco et al. ................. .. 382/289 Anil K. Jain, “Automatic Image Orientation Detection”, IEEE Trans
6,134,346 10/2000 Berman et al. actions on Image Processing, vol. 11 No. 7, pp. 746-755, Jul. 2002.
6,282,326 * 8/2001 Lee et a1. .................... .. 382/289 MartinA. Fischler, Robert C. Bolles, “Random Sample Consensus: A
6,304,313 * 10/2001 Honma 355/18 Paradigm for Model Fitting With Applications to Image Analysis and
6,310,984 * 10/2001 Sansom-Wai et al. ...... .. 382/289 Automated Cartography”, Communications of the ACM vol. 24 No.
6,385,347 5/2002 Matsuda
6, pp. 381-395, Jun. 1981.
6,430,320 8/2002 Jia et a1.
6,453,056 9/2002 Laumeyer et al. Robert S. Caprari, “Algorithm for text p. up/doWm orientation deter
6,525,741 2/2003 Klassen et al. mination”, Patten Recognition Letters 2, Elsevier Science B.V., pp.
6,630,938 10/2003 Nanni 311-317, 1999 Australia.
6,633,332 10/2003 Nay et al. Document Copy Stand and Digital Camera, Sky Mall Magazine, Late
6,687,420 2/2004 Matsuda et al. Spring 2004, p. 5, American Airlines.
6,716,175 4/2004 Geiser et a1. Digital Presenter, NEC Projectors Web Site: DV11, http://WWWnec
6,806,903 10/2004 Okisu et al. pj.com/products/dv/, (2005).
6,839,463 1/2005 Blake et al. Patented Xerox Technology Could Turn Camera Phone into Portable
6,873,732 3/2005 Dance Scanner, PhoneContent.com, http://WWW.phonecontent.com/bm/
6,954,290 10/2005 Braudaway et al. neWs/gneWs/584.shtml, (2005).
7,092,548 8/2006 Laumeyer et al.
Document Imaging With Cameras, Xerox Website, http://WWW.
7,121,469 10/2006 Dorai et al.
xeroxtechnolo gy.com/ ip 1 .nsf/ sedan 1 ?readform
7,330,604 2/2008 Wu et al.
&unid:6192C61923FED63885256FCC0069lEEF, (2005).
7,602,995 * 10/2009 Araki et a1. ................. .. 382/290
2002/0044681 4/2002 Fujimoto et al. Document Imaging With Cameras, Xerox Research Centre Europe,
2003/0026482 2/2003 Dance Mar. 2005.
Milan Sonka, Vaclav Hlavac, Roger Boyle: Image Processing, Analy
2003/0048271 3/2003 Liess et al.
2003/0053692 3/2003 Hong et a1. sis, and Machine Vision, Mar. 19, 2007, CL Engineering,
2004/0012679 1/2004 Fan XP002633279, p. 221.
2004/0022451 2/2004 Fujimoto et al. Extended European Search Report dated Apr. 29, 2011 issued to
2004/0099741 5/2004 Dorai et al. European Application No. 087674966.
2005/0053304 3/2005 Frei
2005/0175255 8/2005 Fujimoto et al. * cited by examiner

US. Patent Mar. 26, 2013 Sheet 2 6f 27 US 8,406,476 B2
v% 7\
9;
8t539o803mi0
E5 &:
)8%5%g\2651&5m\5
GmE2
E5‘5
50223026 0005002 2
(925K
US. Patent Mar. 26, 2013 Sheet 3 0f 27 US 8,406,476 B2
Receive
220 Image
230‘- Segmentation
i
240‘ Distortion
Removal
250w Orientation
Z60
FIG. 2
US. Patent Mar. 26, 2013 Sheet 4 6f 27 US 8,406,476 B2
305
J70 Receive Image
I/—500
.3 75 -—L Convert To Device Independent Bitmap_I
320% Edge-Based Segmentation _I
325\-LCalculate Edge-Based Segmentation Con?dence_I
5.30
Content
Con?dence Based
H'gh? No Segmentation
Yes
Calculate Comers From Edge
.340‘- Ll. ne intersecti. on
350-—I Distortion Removal I
360-I Create New Cropped And Un-Warped Image I
355\-I Threshold Process I
370% Lines or text 1
375-I Determine Landscape/Portrait Orientation I
580~—-I Determine Up/Dowm Orientation _I
J85~—-I Rotate Image I
J90-—I Output Image I
395 FIG. 3
US. Patent Mar. 26, 2013 Sheet 5 6f 27 US 8,406,476 B2
Input
I
I_ — * — — — — — — — ~ — — — — — — — — ~ — — — — — — — — '" — _I
I l
I |
l l
I l
I l
: From Center- From Outside-In- :
Find Horizontal Find Horizontal
: 470 And Vertical And Vertical 420 i
I Edge Points Edge Points I
l I
I I
I l
I I
I I
I
| Random :
I 430“ Sample |
| Consensus |
I l
I l
I I
I I
I l
l Outlier '
: 440 Removal :
l I
I l
I l
I I
|__ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ _ _1
Output
US. Patent Mar. 26, 2013 Sheet 6 6f 27 US 8,406,476 B2
Input
\
Pick Sample
Group Of
570
Random Edge
Point Pairs
Plot Lines
Between Edge
520
Point Pairs In
Sample Group
530
Reached
Sample
No
Number’)
Locate Max
540 Bins In Line
Space
Extract Edge
550 Set
US. Patent Mar. 26, 2013 Sheet 7 6f 27 US 8,406,476 B2
Input
6 70~—- Calculate
Best Fit Line
From Edge
Points
620
All Edges
Suf?ciently Close
To Best Fit Line?
Remove Edge
Point Furthest
From Best
Fit Line
Output
FIG. 6
US. Patent Mar. 26, 2013 Sheet 8 6f 27 US 8,406,476 B2
I
I
I
770 Connected I
Components I
I
I
I
l
I
I
Find Nearest
720 Neighbors I
I
I
I
I
I
I
Connect :
7,30‘- Neighbo rs I
Into Lines I
I
I
I I
I
Use Lines To I
740w Determine I
Border I
I
I
I
I
I
____ Add Margins I
750 From Borders I
I
I
I
Output
FIG. 7