Variable Selection in Spline ANOVA Models

27 Pages·2012·0.21 MB·English

Variable Selection in Spline ANOVA Models
Variable Selection in Spline
ANOVA Models
Hao Helen Zhang
North Carolina State University
1
Variable Selection in Spline ANOVA Models
Regression Problems
Continuous Responses
(cid:0)
(cid:1)(cid:2) (cid:3) (cid:4) (cid:5)(cid:7)(cid:6) (cid:2) (cid:8)(cid:10)(cid:9) (cid:2)(cid:11) (cid:12) (cid:13) (cid:3) (cid:14) (cid:12)(cid:16)(cid:15) (cid:15) (cid:15) (cid:12)(cid:18)(cid:17)
,
(cid:19) (cid:20) (cid:20)
(cid:3) (cid:16)(cid:15) (cid:15) (cid:15)
(cid:6) (cid:2) (cid:5)(cid:7)(cid:6) (cid:2) (cid:12) (cid:12)(cid:6) (cid:2) (cid:8)(cid:22)(cid:21) (cid:23)
is the unknown regression function
(cid:7)(cid:6)
(cid:4) (cid:5) (cid:8)
i.i.d. noise with mean and variance
(cid:27)(cid:26)
(cid:11) (cid:24) (cid:25)
(cid:2)
Discrete Responses – Binary case
(cid:0) (cid:1)(cid:2) (cid:21) (cid:28) (cid:24) (cid:12) (cid:14) (cid:29)
, the logit function
(cid:6) (cid:3) (cid:31) ! " $# (cid:3) (cid:14) $& (cid:3) (cid:6)
(cid:30) (cid:5) (cid:8) (cid:5) % (cid:8)
(cid:7)(cid:6)
(cid:30) (cid:5) (cid:8)
(cid:6) (cid:3) ’ (
(cid:4) (cid:5) (cid:8) ) * ,+ .-
(cid:14) (cid:6)
(cid:30) (cid:5) (cid:8)
– To estimate on the product domain ,
0
(cid:0) (cid:7)(cid:6) / (cid:3) /
(cid:4) (cid:5) (cid:8) 01 (cid:19) 0
– To select the important ’s.
(cid:6)
2
Variable Selection in Spline ANOVA Models
Variable Selection in Linear Models
Based on the ordinary least squares (OLS) estimates
Best subset regression (exhaustive search)
(cid:0)
– expensive computation
– the leaps and bounds by Furnival(1971) efficient for .
(cid:0) (cid:1) (cid:2) (cid:24)
Sequential search methods
(cid:0)
– forward selection; backward elimination
– stepwise regression by Efroymson (1960)
Criteria for inclusion/deletion
(cid:0)
– adjusted , Mean Squared Error, Mallow’s
(cid:26)
(cid:23) (cid:3)
(cid:4)
– -statistic, ,
(cid:5) (cid:6) (cid:7) (cid:3) (cid:8) (cid:7) (cid:3)
3
Variable Selection in Spline ANOVA Models
Shrinkage Methods – Penalized Least Squares Estimates
Bridge regression by Frank & Friedman (1993)
(cid:0)
(cid:4) (cid:20) (cid:20)
(cid:14)
+ + (cid:9)(cid:8) (cid:7) (cid:10) (cid:26) (cid:13)(cid:12)
(cid:0)(cid:1) (cid:2) (cid:5) (cid:2)(cid:1) (cid:5)(cid:6) (cid:5) (cid:7) (cid:6) (cid:2) (cid:8) (cid:9) (cid:11) (cid:5)% (cid:7) % (cid:12) (cid:14)(cid:15) (cid:14)
(cid:3) (cid:17)
(cid:2) 1 (cid:19) 1 (cid:19) 1 (cid:19)
(cid:7) (cid:7)
– LASSO by Tibshirani (1996) ; Ridge regression .
(cid:3) (cid:14) (cid:3) (cid:16)
(cid:14) (cid:14)
Nonnegative garrote by Breiman (1995)
(cid:0)
(cid:4) (cid:20) (cid:20)
(cid:14) (cid:19) (cid:26) subject to
+ (cid:21)(cid:20) (cid:22)(cid:23) (cid:8) (cid:7) (cid:10)
(cid:0)(cid:1) (cid:2) (cid:17) (cid:2)(cid:1) (cid:18) (cid:7) (cid:5) (cid:7) (cid:6) (cid:2) (cid:24) (cid:18) (cid:15)(cid:7) (cid:24) (cid:12) (cid:18) % (cid:7) (cid:25) % (cid:26)(cid:27)
(cid:3) (cid:17)
(cid:2) 1 (cid:19) 1 (cid:19) 1 (cid:19)
(cid:7) (cid:7)
Smoothly clipped absolute deviation (SCAD) by Fan & Li (2001)
(cid:0)
(cid:4) (cid:20) (cid:20)
(cid:14)
+ + (cid:28)(cid:8) (cid:7) (cid:10) (cid:26)
(cid:0)(cid:1) (cid:2) (cid:5) (cid:2)(cid:1) (cid:6)(cid:5) (cid:5) (cid:7) (cid:6) (cid:2) (cid:8) (cid:9) (cid:11) (cid:30) (cid:7) (cid:5) (cid:5)% (cid:7) % (cid:8)
(cid:3) (cid:17)
(cid:2) 1 (cid:19) 1 (cid:19) 1 (cid:19)
(cid:7) (cid:7)
4
Variable Selection in Spline ANOVA Models
Nonparametric Variable Selection
Linear models
(cid:0)
– simple, easy to implement, easy to interprete, ...
– lack of flexibility
Nonlinear models
(cid:0)
– Classification and Regression Tree (CART)
– Multivariate Adaptive Regression Spline (MARS)
–
(cid:15) (cid:15) (cid:15)
5
Variable Selection in Spline ANOVA Models
Smoothing Spline ANOVA Models
In a reproducing kernel Hilbert space (RKHS),
(cid:4)
min (cid:14)
(cid:0)(cid:1) (cid:2) (cid:3) (cid:4) (cid:5) (cid:2)(cid:1) (cid:12) (cid:4) (cid:5)(cid:7)(cid:6) (cid:2) (cid:8) (cid:8)(cid:9) (cid:11) (cid:5) (cid:5) (cid:4) (cid:8)
(cid:17)
(cid:2) 1 (cid:19)
provides a rigorous nonparametric framework for multivariate
(cid:0)
functional estimate
how to add variable selection features?
(cid:0)
– Likelihood basis pursuit (LBP) for regression
– SVM basis pursuit for classification
6
Variable Selection in Spline ANOVA Models
Various Penalties in Regularization Framework
is the fit to the data (e.g. least squares, likelihood, other loss
(cid:0) (cid:4)
functions)
is the penalty of the estimator
(cid:0) (cid:5)
(cid:5) (cid:4) (cid:8)
– Ordinary smoothing spline model uses the squared RKHS norm
– Likelihood basis pursuit uses the norm of the basis
(cid:0)
(cid:19)
coefficients
– COSSO by Lin & Zhang (2002) uses the sum of component
RKHS norms
is the tuning parameter
(cid:0) (cid:11)
7
Variable Selection in Spline ANOVA Models
Basis Pursuit
1. decompose a signal into an overcomplete set of basis functions
2. choose the optimal decomposition: the smallest norm of the
(cid:0)
(cid:19)
coefficients
Wavelet smoothing
Donoho & Johnstone (1994) – SURE shrinkage (Stein Unbiased Risk
(cid:0)
Estimation)
Chen & Donoho (1998), Sardy (1997), etc.
(cid:0)
8
Variable Selection in Spline ANOVA Models
Algorithm in Solving the LBP
Original problem
(cid:0)
– Objective: likelihood plus absolute values of the coefficients
– The second part is non-differentiable at the origin
Transformed into a constrained nonlinear optimization
(cid:0)
– Objective: convex and differentiable, nonlinear
– Polyhedral constraints
MATLAB, GAMS, MINOS,
(cid:15) (cid:15) (cid:15)
(cid:0)
9
Variable Selection in Spline ANOVA Models
Incorporating Categorical Variables
Examples: sex, race, drinking/smoking history, marital status
(cid:0)
Categorical predictors:
(cid:19) (cid:2)(cid:1) (cid:1)
(cid:0) (cid:0) (cid:3) (cid:0) (cid:27) (cid:27) (cid:27) (cid:0) (cid:22)(cid:21) (cid:23)
(cid:5) (cid:12) (cid:12) (cid:8)
Assume each has categories.
(cid:0) (cid:0) (cid:3)
The simplest case . Binary responses T, F
(cid:0) (cid:3) (cid:3) (cid:16)
(cid:28) (cid:29)
– Define a mapping by
(cid:19) (cid:19)
+ +
(cid:3) (cid:4) (cid:6)(cid:5) (cid:5) (cid:7)
(cid:28) (cid:12) (cid:29) (cid:26)(cid:28) (cid:12) (cid:26) (cid:29)
if
(cid:19)
(cid:3) (cid:9)(cid:8) (cid:3) (cid:8) (cid:3) (cid:5)
(cid:5) (cid:8) (cid:26)
if
(cid:19)
(cid:3) + (cid:8) (cid:3) (cid:5)
(cid:26)
For , we need mappings.
+
(cid:0) (cid:3) (cid:10) (cid:16) (cid:3) (cid:14)
10