Linear Regression 6 | Sum of Squared Errors, Sum of Squares, and Type 1,2,3 ANOVA for MLR
Linear Regression 6 | Sum of Squared Errors, Sum of Squares, and Type 1,2,3 ANOVA for MLR
- Sum of Squared Errors for MLR
(1) The Definition of the Sum of Squared Errors (SSE)
The sum of squared error terms, which is also the residual sum of squares, is by its definition, the sum of squared residuals.
(2) Formula #1 of the Sum of Squared Errors
Proof:
By the model of MLR,
then,
By the definition of the residual,
then,
then,
then,
because H-bar is a symmetric matrix, then,
Also, because H-bar is a idempotent matrix, then,
(3) Formula #2 of the Sum of Squared Errors
Proof:
Because H is a projection matrix transfer a vector onto the column space of X, then we must have the conclusion that,
then,
this is to say that,
thus, in conclusion,
By formula #1, we can have,
By the real model of MLR,
then,
then,
then,
then,
then,
By the fact that,
then,
(4) Formula #3 of the Sum of Squared Errors
Proof:
By the definition of the SSE,
then,
By the solution of the OLS estimator,
then,
then,
then,
then,
(5) Formula #4 of the Sum of Squared Errors
Proof:
By the definition of the SSE,
then,
then,
By the solution of the OLS estimator,
then,
then,
then,
then,
then,
then,
(5) Trace of the Hat Matrix
Suppose we are given k independent (explanatory) variables, then, by the definition of the matrix X, X is going to be a n × k matrix. In order to get our OLS estimator, we have to make an assumption that the matrix is k ranked. This means we have enough observations of this model. Then,
Then, by the definition of the hat matrix, which is the projection matrix onto the column space of X, that is,
Note that H is an n × n matrix.
then,
By the commutative property of the trace (see a proof from here),
By definition of the inverse matrix,
So the trace of the hat matrix equals k.
(6) Trace of the Hat-bar Matrix
Based on our following discussion, the trace of the H-bar matrix is,
Proof:
By definition,
By property of the matrix trace,
By our discussions above,
(7) Cochran’s Theorem
Suppose we are given a vector x of i.i.d. random variables x1, x2, …, xn follow the same Gaussian distribution. Also, we are given a matrix A with a trace of p. Then we can have,
We are not going to prove this conclusion here (this is a conclusion of the quadratic forms), it will be fine to have it as our conclusion.
(8) The Expectation of SSE
Since,
then,
Hence, by Cochran’s theorem,
Thus, the expectation of SSE is,
Because the expectation of SSE is not σ², then SSE is a biased estimator for σ².
(9) The Definition of MSE
To find the unbiased estimator of σ², we have to define a new estimator. Based on the discussions above,
then,
then,
Suppose we define that,
then, MSE is an unbiased estimator of σ².
2. Sum of Squares for MLR
(1) The Definition of the Total Sum of Squares of MLR
The total sum of squares is defined by,
(2) The Quadratic Form of the Total Sum of Squares of MLR
The quadratic form of SST is,
Where Jn is an n × n matrix of ones.
Proof:
By the definition of the SST,
then,
then,
By the fact that,
then,
thus,
(3) Distribution of the SST of MLR
By Cochran’s theorem,
then,
then,
thus,
(4) The Quadratic Form of the Regression Sum of Squares of MLR (SSR)
The quadratic form of SSR is,
Proof:
By the fact that,
thus,
then,
By formula #3 of SSE,
then,
By the fact of OLS that,
so,
then, replace the vector β-hat in SSR,
then, by definition of the hat matrix,
thus,
(5) Distribution of the SSR of MLR
Similarly, by Cochran’s theorem,
then,
then,
thus,
3. Type 1,2,3 ANOVA for MLR
Suppose we are given k-1 dependent variables (k including the constant), then
(1) Global ANOVA for MLR
(2) Type 1 (Sequential) ANOVA for MLR
Suppose we define,
then,
The type 1 ANOVA should be,
(3) Type 2 (Partial) ANOVA for MLR
Suppose we define,
then,
The type 2 ANOVA should be,
(4) Type 3 (Partial with a Test on Intercept) ANOVA for MLR
Similar to type 2, the type 3 ANOVA for MLR should be,
