Low power parallel rolling shutter artifact removal

123 Pages·2014·23.02 MB·English

Low Power Parallel Rolling Shutter
Artifact Removal
by
Nick Stupich, B.Eng.
A thesis submitted to the Faculty of Graduate and Postdoctoral Affairs in
partial fulfillment of the requirements for the degree of
Master of Applied Science in Electrical and Computer Engineering
Ottawa-Carleton Institute for Electrical and Computer Engineering
Department of Electronics
Carleton University
Ottawa, Ontario
(cid:13)c 2014
Nick Stupich
Abstract
Thisworkpresentsanalgorithmcapableofmodelingandcorrectingvideoartifacts
caused by movements of a rolling shutter video camera. A distortion model is fit
to feature points extracted from pairs of frames to quantity camera movements
acrossimagescanlines. Anaffinetransformationisusedtomodelfullframecamera
movements, andsinusoidsmodelhighfrequencycameramovementsandvibrations
in the x and y directions, as well as rotations. The model parameters that fit to
the extracted feature points are robust to outliers using an m–estimator solution
thatisefficientlyoptimizedbyiterativelydecreasingthem–estimatorkernelwidth.
An exponential moving average filter is used to produce smooth output camera
motion before the distortion in individual frames is removed. Automated code
optimization is applied to inner model fitting loops to improve performance. An
implementation suitable for a low power parallel processing platform is presented.
The distortion model was found to be capable of accurately modeling rolling shut-
ter distortions, especially those caused by high frequency camera vibrations. The
m-estimator solution was found to accurately discount outlier features, and com-
bined with the iteratively decreasing kernel width the global optimum solution is
reliably and efficiently found. Automated code optimization decreased model pa-
rameter calculation time by 49 times by factoring out common terms from matrix
element computations.
Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
Contents ii
List of Tables v
List of Figures vi
List of Acronyms and Symbols viii
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Introduction to rolling shutter sensors . . . . . . . . . . . . . . . . . 3
1.2.1 CMOS pixel circuits . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Pixel scanning circuit . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Background 8
2.1 Types of distortions . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.1 Skew . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.2 Wobble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.3 Smear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.4 Partial Exposure . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.5 Scope of Work . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.6 Importance of Work . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Previous work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Full frame video stabilization . . . . . . . . . . . . . . . . . 12
2.2.2 Electronic global shutters . . . . . . . . . . . . . . . . . . . 15
2.2.3 Gyroscope based software solutions . . . . . . . . . . . . . . 16
2.2.4 Software rolling shutter solutions . . . . . . . . . . . . . . . 17
2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3 Algorithm Design 20
3.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2 Extracting feature pairs . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2.1 KLT feature extractor . . . . . . . . . . . . . . . . . . . . . 22
3.2.1.1 Extracting features . . . . . . . . . . . . . . . . . . 23
ii
iii
3.2.1.2 KLT feature tracking . . . . . . . . . . . . . . . . . 25
3.2.1.3 Sub-pixel resolution of features . . . . . . . . . . . 28
3.2.2 Alternatives to the KLT algorithm . . . . . . . . . . . . . . 28
3.2.3 Division of images into sub-images . . . . . . . . . . . . . . 29
3.3 Distortion transfer function . . . . . . . . . . . . . . . . . . . . . . 36
3.3.1 Full frame movement . . . . . . . . . . . . . . . . . . . . . . 38
3.3.2 Row based x-translation . . . . . . . . . . . . . . . . . . . . 39
3.3.3 Row based y-translation . . . . . . . . . . . . . . . . . . . . 41
3.3.4 Row based rotation . . . . . . . . . . . . . . . . . . . . . . . 41
3.3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4 Outlier rejection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4.1 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.2 RANSAC & variants . . . . . . . . . . . . . . . . . . . . . . 46
3.4.3 Least Median of Squares . . . . . . . . . . . . . . . . . . . . 47
3.4.4 M-estimators . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.4.5 Use of m-estimators . . . . . . . . . . . . . . . . . . . . . . . 49
3.4.5.1 Kernel choice . . . . . . . . . . . . . . . . . . . . . 50
3.4.5.2 Optimization method . . . . . . . . . . . . . . . . 53
3.4.5.3 Levenberg-Marquardt Optimization . . . . . . . . . 57
3.4.5.4 Ideal final kernel width parameter computation . . 60
3.5 Motion Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.6 Image Warping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.7 Overall Model Fitting Algorithm . . . . . . . . . . . . . . . . . . . 67
3.7.1 Model Fitting to Feature Pairs . . . . . . . . . . . . . . . . . 69
3.7.2 Row-based Translation Model Fitting . . . . . . . . . . . . . 70
4 Implementation 75
4.1 Automated code optimization . . . . . . . . . . . . . . . . . . . . . 75
4.1.1 Sub-expression selection strategy . . . . . . . . . . . . . . . 79
4.1.2 Additional Optimizations . . . . . . . . . . . . . . . . . . . 80
4.1.3 Stopping Criteria . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2 Parallelization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.2 Feature Matching . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.3 Distortion Transfer Function Computation . . . . . . . . . . 82
4.2.4 Motion Filtering and Image Warping . . . . . . . . . . . . . 83
4.3 Fixed Point Implementation . . . . . . . . . . . . . . . . . . . . . . 84
5 Results 88
5.1 Code optimization speedup . . . . . . . . . . . . . . . . . . . . . . . 88
5.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . 89
5.2.2 Distortion Transfer Function Calculation . . . . . . . . . . . 89
5.2.3 Motion Smoothing and Image Warping . . . . . . . . . . . . 90
iv
5.3 Artificially distorted, noisy video . . . . . . . . . . . . . . . . . . . 90
5.4 Real world video results . . . . . . . . . . . . . . . . . . . . . . . . 92
5.4.1 Parliament Video . . . . . . . . . . . . . . . . . . . . . . . . 92
5.4.2 Driving Video . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.4.3 Walking Video . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.4.4 House Video . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.4.5 Street Video . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.5 Fixed Point Implementation . . . . . . . . . . . . . . . . . . . . . . 97
6 Conclusion 98
6.1 Contributions to Research . . . . . . . . . . . . . . . . . . . . . . . 98
6.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.2.1 3D Camera Motion Model . . . . . . . . . . . . . . . . . . . 100
6.2.2 Bicubic Pixel Interpolation . . . . . . . . . . . . . . . . . . . 100
6.2.3 M-estimator width reduction . . . . . . . . . . . . . . . . . . 101
6.2.4 Motion Smoothing . . . . . . . . . . . . . . . . . . . . . . . 102
6.2.5 Removal of Motion Blur . . . . . . . . . . . . . . . . . . . . 103
6.2.6 Fisheye lens correction . . . . . . . . . . . . . . . . . . . . . 104
List of References 106
List of Tables
3.1 χ2 values, counts of extracted features and average minimum eigen-
values for different sub-image grid dimensions . . . . . . . . . . . . 32
3.2 Results of the recursive sub-image feature extraction algorithm . . . 35
3.3 Model parameter symbols and descriptions . . . . . . . . . . . . . . 44
3.4 Several popular m-estimator kernels . . . . . . . . . . . . . . . . . . 49
3.5 Kernel treatment of inliers . . . . . . . . . . . . . . . . . . . . . . . 50
3.6 Kernel treatment of extreme outliers . . . . . . . . . . . . . . . . . 51
4.1 Model Parameter scaling for fixed point implementation . . . . . . . 86
4.2 Calculation of optimal fixed point variable scaling . . . . . . . . . . 87
5.1 Speedup of Levenberg-Marquardt (LM) iterations from code opti-
mizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.2 Artificial test video results . . . . . . . . . . . . . . . . . . . . . . . 92
v
List of Figures
1.1 Active complementary metal oxide semiconductor (CMOS) pixel
with basic circuitry . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 CMOS image sensor circuit architecture . . . . . . . . . . . . . . . 6
2.1 Skew effect of a panning camera . . . . . . . . . . . . . . . . . . . . 9
2.2 Wobble effect of a vibrating camera . . . . . . . . . . . . . . . . . . 10
2.3 An example of frame smearing . . . . . . . . . . . . . . . . . . . . . 11
2.4 A pixel circuit for an electronic global shutter . . . . . . . . . . . . 16
3.1 Sub-image feature extraction algorithm pseudocode . . . . . . . . . 30
3.2 χ2 values, counts of extracted features and average minimum eigen-
values for different sub-image grid dimensions . . . . . . . . . . . . 33
3.3 Comparison of standard feature extraction and grid based extraction 34
3.4 Visualization of distortion and desired correction . . . . . . . . . . . 36
3.5 Fitting errors with different numbers of sinusoid terms . . . . . . . 38
3.6 x-Translation between video frames . . . . . . . . . . . . . . . . . . 40
3.7 Fitting of x-translation by sinusoids . . . . . . . . . . . . . . . . . . 40
3.8 Y shifts due to camera rotation . . . . . . . . . . . . . . . . . . . . 42
3.9 Failure of a least squares fit . . . . . . . . . . . . . . . . . . . . . . 45
3.10 Welsch kernel response . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.11 Comparison of Welsch kernel and least squares . . . . . . . . . . . . 53
3.12 Optimization surfaces as w decreases. . . . . . . . . . . . . . . . . . 56
3.13 Change in model parameters as w decreases . . . . . . . . . . . . . 57
3.14 Monte carlo simulation to find an ideal w(x) . . . . . . . . . . . . . 63
3.15 Monte carlo simulation to find an ideal full frame w(x) . . . . . . . 64
3.16 Motion smoothing using an EMA filter . . . . . . . . . . . . . . . . 65
3.17 Frequency response of the motion EMA filter . . . . . . . . . . . . . 66
3.18 Overall fitting process flowchart . . . . . . . . . . . . . . . . . . . . 68
3.19 Frequency analysis of row based x-translation . . . . . . . . . . . . 73
4.1 Sub-expression generation for automated code optimization. . . . . 77
4.2 Automated optimization flowchart . . . . . . . . . . . . . . . . . . . 78
4.3 Sub-expression frequencies in Hessian and Jacobian computations . 79
4.4 Number of temporary variables vs LM iteration time . . . . . . . . 81
5.1 Screenshot of the Parliament video . . . . . . . . . . . . . . . . . . 92
5.2 Screenshot of the Driving video . . . . . . . . . . . . . . . . . . . . 94
vi
vii
5.3 Screenshot of the Walking video . . . . . . . . . . . . . . . . . . . . 95
5.4 Screenshot of the House video . . . . . . . . . . . . . . . . . . . . . 96
5.5 Screenshot of the Street video . . . . . . . . . . . . . . . . . . . . . 96
6.1 Maximum Relative Error vs Grid spacing . . . . . . . . . . . . . . . 103
6.2 Fisheye distortion caused by stabilization . . . . . . . . . . . . . . . 105
List of Acronyms and Symbols
G Image derivative taken in the x-direction.
x
G Image derivative taken in the y-direction.
y
I An Image.
I(x,y) A pixel at location (x,y) in Image I.
I drain to source current.
ds
M Covariance matrix of a sub image used for feature extraction.
cov
M Matrix of the minimum eigenvalue of the covariance matrix of each pixel
min(λ)
in an image.
V gate to source voltage.
gs
λ ParameterusedinLMoptimizationtointerpolatebetweengradientdescentand
the Newton Raphson method.
ψ(x) Influence function of an m-estimator.
ρ(x) Kernel function of an m-estimator.
σ Standard deviation of inlier data points.
in
σ Standard deviation of outlier data points.
out
f frequency of distortion.
distort
viii
ix
f frame rate.
frame
f row scanning frequency.
row
n number of columns.
cols
n Maximum number of features to be extracted from a single image.
features
n Number of inlier data points.
in
n Number of pyramid layers used in the KLT tracking algorithm.
layers
n Number of outlier data points.
out
n number of rows.
rows
n Number of sub-images used for feature extraction.
subImages
qualityLevel Minimum eigenvalue to be considered for a feature point, relative to
the max(M ).
min(λ)
w(x) Weight function of an m-estimator.
winSize Window size used to extract sub-pixel accurate features.
centerX The center x value, about which the image is allowed to rotate.
ADC analog to digital converter.
APS active pixel sensor.
ASIC application specific integrated circuit.
CMOS complementary metal oxide semiconductor.
CPU Central Processing Unit.
DFT discrete Fourier transform.