Logistic Regression

Logistic Regression is a method of classification using the regression framework. In logistic regression, the output (or target, or dependent) variable y_i is a binary variable, taking values of either 0,1. The predictor (or input, or independent) variables x_{i,j} are not limited in this way, and can take any value.

Logistic regression is based on the logistic function, which always takes values between 0 and 1. Replacing the dependent variable of the logistic function with a linear combination of dependent variables we intend to use for regression, we arrive at the formula for logistic regression.

    \begin{equation*} \pi(X_i) = \frac{1}{1+ e^{-(\beta'X_i)}} \hspace{1cm} \beta'X_i = \beta_0 + \beta_1x_{i,1} + \beta_2x_{i,2} + \ldots + \beta_px_{i,p} \end{equation*}

Where X_i is the vector of independent variable observations for subject i with a preceding 1 to match \beta_0. \beta is the vector of coefficients. We can interpret \pi(x) as the probability p(y_i=1|X_i). I.e., the probability that the dependent variable is of class 1, given the independent variables.

To get an idea of what’s happening here, we’re going to introduce an idea called “odds”. Odds are a ratio of probabilities. Let’s say P is the probability of an event occuring. 1-P is the probability of that event not occuring. Odds is defined as \frac{P}{1-P}. IE, the ratio of probabilities of that event occuring over that event not occuring.

The Logit function is given

    \begin{equation*} \logit(p) = \log(\frac{P}{1-P}) \end{equation*}

or, the log-odds ratio. The logit function has the interesting property that \logit(1-p) = -\logit(p). So now, we declare the logit equal to the linear combination of covariates.

    \begin{equation*} \logit(\pi(x)) = \beta'X \hspace{1cm}\Rightarrow\hspace{1cm}\pi(x) = \frac{1}{1+e^{-(\beta'X)}} \end{equation*}

But now we’re solving for the \logit(\pi(x)). We can solve this using the least squared error solution from linear regression.

    \begin{align*} \logit(\pi(X)) &= X\beta\\ X'Y &= X'X\beta\\ \beta &= (X'X)^{-1}X'Y \end{align*}

This solves for the \beta that minimizes the squared error. We should specify that in Logistic Regression, the errors are assumed to be binomial.