Model comparison

Prof. Eric Friedlander

term	estimate	std.error	statistic	p.value
(Intercept)	-0.170	0.366	-0.465	0.643
Party	1.837	0.124	14.758	0.000
AgeMiddle	1.009	0.408	2.475	0.014
AgeSenCit	1.388	0.485	2.862	0.005

term	df	sumsq	meansq	statistic	p.value
Party	1	1188.64	1188.64	285.71	0.00
Age	2	38.03	19.01	4.57	0.01
Residuals	165	686.44	4.16	NA	NA

ANOVA

Main Idea: Decompose the total variation of the outcome into:
- the variation that can be explained by the each of the variables in the model
- the variation that can’t be explained by the model (left in the residuals)
$S S_{T o t a l}$ : Total sum of squares, variability of outcome, $\sum_{i = 1}^{n} (y_{i} - \bar{y})^{2}$
$S S_{E r r o r}$ : Residual sum of squares, variability of residuals, $\sum_{i = 1}^{n} (y_{i} - {\hat{y}}_{i})^{2}$
$S S_{M o d e l} = S S_{T o t a l} - S S_{E r r o r}$ : Variability explained by the model, $\sum_{i = 1}^{n} ({\hat{y}}_{i} - \bar{y})^{2}$

Complete Exercise 3.

term	df	sumsq	meansq	statistic	p.value
Party	1	1188.63588	1188.635880	285.711511	0.000000
Age	2	38.02783	19.013916	4.570361	0.011699
Residuals	165	686.44389	4.160266	NA	NA

term	df	sumsq	meansq	statistic	p.value
Party	1	1188.64	1188.64	285.71	0
Age	2	38.03	19.01	4.57	0.01
Residuals	165	686.44	4.16
Total	168	1913.11

term	df	sumsq
Party	1	1188.64
Age	2	38.03
Residuals	165	686.44
Total	168	1913.11

term	df	sumsq
Party	1	1188.64
Age	2	38.03
Residuals	165	686.44
Total	168	1913.11

term	df	sumsq
Party	1	1188.64
Age	2	38.03
Residuals	165	686.44
Total	168	1913.11

term	df	sumsq
Party	1	1188.64
Age	2	38.03
Residuals	165	686.44
Total	168	1913.11

F-Test: Testing the whole model at once

Hypotheses:

$H_{0} : β_{1} = β_{2} = \dots = β_{k} = 0$ vs. $H_{A} :$ at least one $β_{i} \neq 0$

Test statistic: F-statistics

$F = \frac{M S M o d e l}{M S E} = \frac{S S M o d e l / k}{S S E / (n - k - 1)}$

p-value: Probability of observing a test statistic at least as extreme (in the direction of the alternative hypothesis) from the null value as the one observed

$p-value = P (F > test statistic),$

calculated from an $F$ distribution with $k$ and $n - k - 1$ degrees of freedom.

Nested Models

We say one model is nested inside another model if all of its TERMS are present in the other model

Consider three different models:
- Model 1: $Y = β_{0} + β_{1} X_{1} + β_{2} X_{2} + β_{3} X_{3} + ϵ$
- Model 2: $Y = β_{0} + β_{1} X_{1} + β_{2} X_{2} + ϵ$
- Model 3: $Y = β_{0} + β_{1} X_{1} + β_{2} X_{2} + β_{3} X_{1} X_{2} + ϵ$

Model 2 is nested inside both Model 1 and Model 3.
Why isn’t Model 3 nested in Model 1?
Smaller model is called the Reduced Model
Larger model is called the Full Model (be careful, this term depends on context)

Complete Exercises 5-7.

Recall: ANOVA, F-Test

Hypotheses:

$H_{0} : β_{1} = β_{2} = \dots = β_{p} = 0$ vs. $H_{A} :$ at least one $β_{i} \neq 0$

Test statistic: F-statistic

$F = \frac{M S M o d e l}{M S E} = \frac{S S M o d e l / p}{S S E / (n - p - 1)}$

p-value: Probability of observing a test statistic at least as extreme (in the direction of the alternative hypothesis) from the null value as the one observed

$p-value = P (F > test statistic),$

calculated from an $F$ distribution with $p$ and $n - p - 1$ degrees of freedom.

Nested F-Test

Suppose $k$ is the number of $β$ ’s in the nested model and $p$ is the full number of predictors in the larger model. I.e. $β_{k + 1}, \dots, β_{p}$ are the new $β$ ’s

Hypotheses:

$H_{0} : β_{k + 1} = β_{k + 2} = \dots = β_{p} = 0$ vs. $H_{A} :$ at least one $β_{i} \neq 0$ for $i > k + 1$

Test statistic: F-statistic

$F = \frac{(S S M o d e l_{f u l l} - S S M o d e l_{r e d u c e d}) / (p - k)}{S S E_{f u l l} / (n - p - 1)}$

p-value: Probability of observing a test statistic at least as extreme (in the direction of the alternative hypothesis) from the null value as the one observed

$p-value = P (F > test statistic),$

calculated from an $F$ distribution with $p - k$ (the number of predictors being tested) and $n - p - 1$ degrees of freedom.

Note: Same as regular F-test if reduced model is just $Y = β_{0}$ .

Nested F-Test in R

tip_fit_1 <- lm(Tip ~ Party + Age + Meal, data = tips)
tip_fit_2 <- lm(Tip ~ Party + Age + Meal + Day, data = tips)

anova(tip_fit_1, tip_fit_2) |> # Enter reduced model first
  tidy() |> 
  kable()

term	df.residual	rss	df	sumsq	statistic	p.value
Tip ~ Party + Age + Meal	163	622.9793	NA	NA	NA	NA
Tip ~ Party + Age + Meal + Day	158	607.3815	5	15.59778	0.8114993	0.543086

$F = \frac{(S S M o d e l_{f u l l} - S S M o d e l_{r e d u c e d}) / (p - k)}{S S E_{f u l l} / (n - p - 1)} = \frac{15.59778 / 5}{6073815 / 158} = 0.8114993$

Let’s interpret this together.

Complete Exercise 8.

R-squared, $R^{2}$ , Overfitting

$R^{2}$ will always increase as we add more variables to the model
- If we add enough variables, we can usually achieve $R^{2} = 100 %$
- Eventually our model will over-align to the noise in our data and become worse at predicting new data… this is called overfitting
If we only use $R^{2}$ to choose a best fit model, we will be prone to choosing the model with the most predictor variables

Comparing models with $R_{a d j}^{2}$

tip_fit_1:

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.6743626	0.6643738	1.954983	67.51136	0	5	-350.0405	714.0811	735.9904	622.9793	163	169

tip_fit_2:

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.6825157	0.6624218	1.96066	33.96625	0	10	-347.898	719.7959	757.3547	607.3815	158	169

Which model would we choose based on $R^{2}$ ?
Which model would we choose based on Adjusted $R^{2}$ ?
Which statistic should we use to choose the final model - $R^{2}$ or Adjusted $R^{2}$ ? Why?

AIC	BIC
714.0811	735.9904

AIC	BIC
719.7959	757.3547

Forward Selection

Start with a model with no predictors and find the best single predictor (the largest correlation with the response gives the biggest initial).
Add the new predictor to the model, run the regression, and find its individual p-value:
1. If p-value is small (say, less than 5%), add predictor which would produce the most benefit (biggest increase in $R^{2}$ ) when added to the existing model.
2. If the p-value is large (over 5%), stop and discard this predictor. At this point, no (unused) predictor should be significant when added to the model and we are done.

Stepwise Selection

Forward, stepwise selection

Start with a model with no predictors and find the best single predictor (the largest correlation with the response gives the biggest initial).
Add the new predictor to the model, run the regression, and find its individual p-value:
1. If p-value is small (say, less than 5%), run backward elimination, then add predictor which would produce the most benefit (biggest increase in $R^{2}$ ) when added to the existing model.
2. If the p-value is large (over 5%), stop and discard this predictor. At this point, no (unused) predictor should be significant when added to the model and we are done.

Why? Sometimes variables that were significant early on, can become insignificant after other new variables are added to the model.

Backward, stepwise selection is the same, except you perform forward selection every time you delete a term from the model.

Model comparison Prof. Eric Friedlander

Model comparison
Announcements
Topics
Computational setup
Introduction
Data: Restaurant tips
Variables
Outcome: Tip
Predictors
Relevel categorical predictors
Predictors, again
Outcome vs. predictors
Fit and summarize model
Another model summary
Analysis of variance (ANOVA)
Analysis of variance (ANOVA)
ANOVA
ANOVA output in R1
ANOVA output, with totals
Sum of squares
Sum of squares: $S S_{T o t a l}$
Sum of squares: $S S_{E r r o r}$
Sum of squares: $S S_{M o d e l}$
F-Test: Testing the whole model at once
F-test in R
Comparing sets of predictors
Nested Models
Recall: ANOVA, F-Test
Nested F-Test
Nested F-Test in R
Model comparison
R-squared, $R^{2}$
R-squared, $R^{2}$ , Overfitting
Adjusted $R^{2}$
$R^{2}$ and Adjusted $R^{2}$
Using $R^{2}$ and Adjusted $R^{2}$
Comparing models with $R_{a d j}^{2}$
AIC & BIC
AIC & BIC
AIC & BIC
AIC & BIC
Using AIC & BIC
Comparing models with AIC and BIC
Commonalities between criteria
Parsimony and Occam’s razor
In pursuit of Occam’s razor
Alternate views
Other concerns with our approach
Model Selection
Model Selection
AIC, BIC, Mallows’ $C_{p}$
Best Subset Selection
Best Subset Selection in R
Backward Elimination
Forward Selection
Stepwise Selection
CAUTION
Recap