Linear Regression 15 | Logistic Regression Inference

Series: Linear Regression

Linear Regression 15 | Logistic Regression Inference

  1. Wald Test of Coefficient

(1) Hypotheses for Wald Test

Similar to SLR and MRL, to tell whether or not a predictor of logistic regression is significant or not, we have to test the parameter of this predictor. The hypotheses of the Wald test is,

(2) Assumption for Wald Test

The Wald test can only be valid when the sample size is large enough. If we assume that the logistic model is rely on a large sample size, then the MLE is of predicor p follows,

(3) Wald Test

Based on our discussion, we should conduct Z test for the parameter,

Here, we don’t Then, if

then we can reject H0 and reach the conclusion that the p predictor is significant.

(4) Confidence Interval based on Wald Test

  • The confidence level for an estimate
  • The confidence level for the odds ratio of an estimate

2. Likelihood Ratio Test

(1) Hypotheses for Likelihood Ratio Test

The likelihood ratio test is quite similar to a likelihood version F test with OLSE. Commonly. The null hypothesis and the alternate hypothesis are,

(2) The Definition of Deviance

We don’t have to fully understand this and we only have to know the meaning of this model in a general way. The deviance is defined as the difference of the likelihood value between the fitted model and the saturated model.

Then, because the second term of this equation is a constant, then the deviance can be simplified as,

Denote G² as the deviance of the model, then

then for the test above, define the test statistics as,

if,

then reject H0, the full model is significantly improved in the likelihood.

3. Deviance Residuals

If we want to evaluate the residuals for the logistic regression model, we have to modify the residuals because they are not the same as the OLS residuals. Basically, there are two commonly used ways for modification,

  • Pearson Residuals: this is similar to OLS studentized residuals

we should observe the person residuals to be distributed horizontally around the ei = 0 line.

  • Deviance Residuals

The deviance residuals are defined by,

The square of each deviance residual measures the contribution of each response to the deviance of the fitted model.

4. Pseudo R²

Because there’s no OLS principle for the logistic model, then we don’t have the regular R² for us to explain the variance of the model. But there’s actually some pseudo-R²’s that can suggest the goodness of fit for us.

  • Efron’s pseudo-R²
  • Mc Fadden’s pseudo-R²

See a more detailed explanation here (from UCLA).

To select the best logistic model, we can use the AIC or the BIC because these concepts hold for the logistic model.