# DATA 5600: Introduction to Regression and Machine Learning for Analytics

## __Bayesian Econometrics - Chapter 2 (Koop): The Normal Linear Regression Model__

Author:  Tyler J. Brough

Updated: November 29, 2021

---

```python
import numpy as np
import pandas as pd
from scipy import stats
import matplotlib.pyplot as plt
import statsmodels.api as sm

plt.rcParams['figure.figsize'] = [10, 8]
```

---

## __Introduction__

These notes are based upon Chapter 2: The Normal Linear Regression Model with Natural Conjugate Prior and a Single Explanatory Variable from the book _Bayesian Econometrics_ by Gary Koop.

* In this chapter and notebook we will look at how to carry out a simple linear regression from the Bayesian perspective

* The model consider here has a natural conjugate prior so there is an analytical solution

* This will pave the way to move on to the case of multiple linear regression from the Bayesian perspective (the subject of Chapter 3)

## __Section 2.2: The Likelihood Function__

To simplify the analysis we will not consider an intercept, so the model we will work with becomes the following:

$$
\Large{y_{i} = \beta x_{i} + \varepsilon_{i}}
$$

* where $\varepsilon_{i}$ is an error term

* The error term might reflect measurement error or that the true relationship between $x$ and $y$ is only an approximate one

* You can also think graphically in terms of the $XY$-plot where we seek to draw a best fitting line by minimizing the squared errors

* Assumptions about $\varepsilon_{i}$ and $x_{i}$ determine the form of the likelihood function

1. $\varepsilon_{i} \sim N(0,\sigma^{2})$ (Normally distributed) and further that $\varepsilon_{i}$ and $\varepsilon_{j}$ are independent for $i \ne j$ (iid)

2. The $x_{i}$ are either fixed (i.e. not random variables) or, if they are random variables, they are independent of $\varepsilon_{i}$ with a probability density function, $p(x | \lambda)$ where $\lambda$ is a vector of parameters that does not include $\beta$ and $\sigma^{2}$

The likelihood function then becomes: $\large{p(y, x | \beta, \sigma^{2}, \lambda)}$

The second assumption implies that we can write the likelihood function as:

$$
\Large{p(y, x | \beta, \sigma^{2}, \lambda) = p(y | x, \beta, \sigma^{2}) p(x | \lambda)}
$$

* We are not directly interested in modeling $x$ so we can work with the likelihood function conditional on $x$

* For simplicity then we will not usually even write $x$ in the model

* __NB:__ regression modeling (Bayesian or frequentist) works with the conditional distribution of $y$ given $x$ and not their joint distribution

We can use the above to work out a form for the likelihood function:

1. $p(y_{i} | \beta, \sigma^{2})$ is Normal

2. $E(y_{i} | \beta, \sigma^{2}) = \beta x_{i}$

3. $var(y_{i} | \beta, \sigma^{2}) = \sigma^{2}$

$$
\Large{p(y_{i} | \beta, \sigma^{2}) = \frac{1}{\sqrt{2 \pi \sigma^{2}}} exp\left[-\frac{y_{i} - \beta x_{i})^{2}}{2 \sigma^{2}} \right]}
$$

* Since for $i = j$, $\epsilon_{i}$ and $\epsilon_{j}$ are independent of one another it follows that $y_{i}$ and $y_{j}$ are also independent thus

$$
\Large{p(y | \beta, \sigma^{2}) = \prod_{i=1}^{N} p(y_{i} | \beta, \sigma^{2})}
$$

Hence, the likelihood function can be summarized as follows:

$$
\Large{p(y | \beta, \sigma^{2}) = \frac{1}{(2 \pi)^{\frac{N}{2}} \sigma^{N}}} exp\left[-\frac{1}{2\sigma^{2}} \sum_{i=1}^{N} (y_{i} - \beta x_{i})^{2} \right]
$$

For convenience, we will want to slightly rewrite the likelihood function as follows:

$$
\Large{\sum_{i=1}^{N} (y_{i} - \beta x_{i})^{2} = \nu s^{2} + (\beta - \hat{\beta})^{2} \sum_{i=1}^{N} x_{i}^{2}}
$$

where

$$
\Large{\begin{align}
\nu &= N - 1 \\
    & \\
\hat{\beta} &= \frac{\sum x_{i} y_{i}}{\sum x_{i}^{2}} \\
    & \\
s^{2} &= \frac{\sum (y_{i} - \hat{\beta} x_{i})^{2}}{\nu}
\end{align}}
$$

* __NB:__ $\hat{\beta}$, $s^{2}$ and $\nu$ are the frequentist OLS estimator for $\beta$, standard error, and degrees of freedom respectively

* Here they are given the interpretation of sufficient statistics

* In many cases it is easier (algebraically) to work with the precision rather than the variance

* The precision of the error term is defined as $\large{h = \frac{1}{\sigma^{2}}}$

* This just helps to ease the mathematical manipulations by reparameterizing things

### __Sufficient Statistic__

---

A statistic that, in a certain sense, summarizes all the information contained in a sample of observations about a particular parameter. In more formal terms this can be expressed using the conditional distribution ofo the sample given the statistic and the parameter $f(y | s, \theta)$,
in the sense that $s$ is sufficient for $\theta$ if this conditional distribution does not depend
on $\theta$. As an example consider a random variable $x$ having the following gamma distribution;

$$
f(x) = x^{\gamma - 1} exp(-x)/\Gamma(\gamma)
$$

The likelihood for $\gamma$ given a sample $x_{1}, x_{2}, \ldots, x_{n}$ is given by

$$
exp\left(-\sum_{i=1}^{n}x_{i} \right)\left[\prod_{i=1}^{n}x_{i}\right]^{\gamma - 1} / \left[\Gamma(\gamma)\right]^{n}
$$

Thus in this case the geometric mean of the observations is sufficient for the parameter. Such a statistic, which is a function of all other such statistics, is referred to as a _minimal sufficient statistic_.

---

Here is the Wikipedia article on [Sufficient Statistics](https://en.wikipedia.org/wiki/Sufficient_statistic)

Using these results we can write the likelihood function as:

$$
\Large{p(y | \beta, h) \frac{1}{(2\pi)^\frac{N}{2}}} \{h^{\frac{1}{2}} exp \left[-\frac{h}{2} (\beta - \hat{\beta})^{2}\sum_{i=1}^{N} x_{i}^{2}\right] \} \{h^{\frac{\nu}{2}} exp \left[-\frac{h\nu}{2 s^{-2}} \right] \}
$$

## __Section 2.3: The Prior__

* Priors are meant to reflect any information the researcher/analyst has before seeing the data

* Priors can take any form

* It is common though to choose a functional form for the prior that eases computation and interpretation

* _Natural conjugate priors_ have these properties

* A conjugate prior is one what when combined with the likelihood yields a posterior that falls in the same class of distributions

* A _natural conjugate prior_ has the additional property that it also has the same form as the likelihood function

* This means that the prior can be interpreted in the same way as the likelihood

* In other words, the prior can be thought of as arising from a fictitious data set from the same process that generated the sample data

For the simple linear regression model we must elicit a prior for $\beta$ and $h$, which we denote by $p(\beta, h)$

* __NB:__ we are not conditioning on the data

* The posterior will condition on the data: $p(\beta, h, | y)$

* It will be convenient to decompose the prior as follows (assuming independence of $\beta$ and $h$): $p(\beta, h) = p(\beta | h) p(h)$

* Then we can write a prior for $\beta | h$ and one for $h$

* Because we selected a Normal distribution for the likelihood function, a natural conjugate prior will also be Normal.

* $\beta | h$ will have a Normal distribution

* But $h$ will have a Gamma distribution

* Taken together we will call this the _Normal-Gamma_ prior

$$
\Large{\begin{align}
\beta | h & \sim N(\underline{\beta}, h^{-1} \underline{V})
& \\
& \\
h & \sim G(\underline{s}^{-2}, \underline{\nu})
\end{align}}
$$

This is all denoted by:

$$
\Large{\beta, h \sim NG(\underline{\beta}, \underline{V}, \underline{s}^{-2}, \underline{\nu})}
$$

* $\underline{\beta}$, $\underline{V}$, $\underline{s}^{-2}$, $\underline{\nu}$ are called _hyperparameters_

* We will need to select values for these to reflect the prior information

* The specific role of these parameters becomes more clear when we see how they show up in the posterior

## __Section 2.4: The Posterior__

* The posterior summarizes the information that we have (both prior and data-based) about the unknown parameters $\beta$ and $h$

* It is proportional to the likelihood times the prior

* For the sake of simplicity we will not work out the gory algebraic details

* After the messy mathematical manipulations it can be shown that the posterior is also _Normal-Gamma_

Formally, we have

$$
\Large{\beta, h | y \sim NG(\bar{\beta}, \bar{V}, \bar{s}^{-2}, \bar{\nu})}
$$

Where

$$
\Large{\begin{align}
\bar{V} &= \frac{1}{\underline{V}^{-1} + \sum x_{i}^{2}} \\
& \\
& \\
\bar{\beta} &= \bar{V}(\underline{V}^{-1}\underline{\beta} + \hat{\beta} \sum x_{i}^{2}) \\
& \\
& \\
\bar{\nu} &= \underline{\nu} + N
\end{align}}
$$

Credit where credit is due: https://github.com/tonyduan/conjugate-bayes/blob/master/conjugate_bayes/models.py

```python
class NIGLinearRegression(object):
    """
    The normal inverse-gamma prior for a linear regression model with unknown
    variance and unknown relationship. Specifically,
        1/σ² ~ Γ(a, b)
        β ~ N(0, σ²V)
    Parameters
    ----------
    mu: prior for N(mu, v) on the model β
    v:  prior for N(mu, v) on the model β
    a:  prior for Γ(a, b) on the inverse sigma2 of the distribution
    b:  prior for Γ(a, b) on the inverse sigma2 of the distribution
    """
    def __init__(self, mu, v, a, b):
        self.__dict__.update({"mu": mu, "v": v, "a": a, "b": b})

    def fit(self, x_tr, y_tr):
        m, _ = x_tr.shape
        mu_ast = np.linalg.inv(np.linalg.inv(self.v) + x_tr.T @ x_tr) @ \
                 (np.linalg.inv(self.v) @ self.mu + x_tr.T @ y_tr)
        v_ast = np.linalg.inv(np.linalg.inv(self.v) + x_tr.T @ x_tr)
        a_ast = self.a + 0.5 * m
        b_ast = self.b + 0.5 * (y_tr - x_tr @ self.mu).T @ \
                np.linalg.inv(np.eye(m) + x_tr @ self.v @ x_tr.T) @ \
                (y_tr - x_tr @ self.mu.T)
        self.__dict__.update({"mu": mu_ast, "v": v_ast, "a": a_ast, "b": b_ast})

    def predict(self, x_te):
        scales = np.array([x.T @ self.v @ x for x in x_te]) + 1
        scales = (self.b / self.a * scales) ** 0.5
        return sp.stats.t(df=2 * self.a, loc=x_te @ self.mu, scale=scales)

    def get_conditional_beta(self, sigma2):
        return sp.stats.multivariate_normal(mean=self.mu, cov=sigma2 * self.v)

    def get_marginal_sigma2(self):
        return sp.stats.invgamma(self.a, scale=self.b)
```

```python

```


An explicit function is of the form that should be the most familiar, such as:

$$ f(x) = x^2 + 3 $$
$$ y = \sin{x} $$

Whereas an *implicit function* defines an algebraic relationship between variables. These functions have a form similar to the following:

$$ x^2 + y^2 = 25 $$
$$ y^5 + xy = 3 $$

In these examples and the following exercises, it is assumed that $y$ is implicitly defined and is a differentiable function of $x$.

Luckily, implicitly defined equations do not need to be solved for $y$ in terms of $x$ (or any other variable) to calculate the derivative of the function. We can employ *implicit differentiation* to find the derivatives of implicitly defined equations.

Implicit differentiation is performed by differentiating both sides of the equation with respect to $x$ and then solving for the resulting equation for the derivative of $y$.

As an example, consider the function $y^3 + x^3 = 1$. We can apply implicit differentiation to this equation to find its derivative. Begin by differentiating both sides of the equation with respect to $x$.

$$ \frac{d}{dx} (y^3 + x^3) = \frac{d}{dx} (1) $$

$$ \frac{d}{dx} (y^3) + \frac{d}{dx} (x^3) = 0 $$

Remembering that $y$ is implicitly defined as a function of $x$, we apply the Chain Rule for finding the derivative of $y^3$.

$$ 3y^2 \frac{dy}{dx} + 3x^2 = 0 $$

Then, solving for $\frac{dy}{dx}$:

$$ 3y^2 \frac{dy}{dx} = -3x^2 $$

$$ \frac{dy}{dx} = -\frac{3x^2}{3y^2} = -\frac{x^2}{y^2} $$

When dealing with more complex equations that feature more terms, it can be helpful to use Newton notation such as $y^\prime$ to help keep the intermediate calculations easier to read.

We can confirm our result by using the [SymPy package](http://docs.sympy.org/latest/index.html) as before. Import the needed functions and set $x$ and $y$ as variables to be recognized by SymPy.

```python
from sympy import symbols, idiff, sin, cos, tan, sqrt, init_printing, plot_implicit

init_printing()
x = symbols('x')
y = symbols('y')
```

As we are finding the derivatives of implicitly defined functions, we use the `idiff` function rather than `diff` as seen in previous examples.

```python
idiff(y ** 3 + x ** 3 - 1, y, x)
```

## Examples

### Example 1: Find the derivative of the implicit function $x^2 + xy - y^2 + 4$.

We begin by taking the derivative of both sides of the equation after subtracting $4$ from both sides.

$$ \frac{d}{dx} (x^2 + xy - y^2) = \frac{d}{dx} (-4) $$

Then taking each derivative in turn:

$$ \frac{d}{dx} (x^2) + \frac{d}{dx} (xy) - \frac{d}{dx} (y^2) = 0 $$

Remembering that $y$ is implicitly defined as a function of $x$, we apply the Product Rule when finding the derivative $\frac{d}{dx} (xy)$.

$$ y \frac{dx}{dy} x + x \frac{dy}{dx} = yx^\prime + y^\prime x $$

Finding the remaining derivatives of the function:

$$ 2x + yx^\prime + y^\prime x - 2yy^\prime = 0 $$

The derivative of $x$ is $1$. The derivative $\frac{d}{dx}$ is found using the Chain Rule:

$$ \frac{d}{dx} (y^2) = \frac{d}{dy} (y^2) \frac{dy}{dx} = 2y \frac{dy}{dx} = 2yy^\prime $$

Finish finding the derivative of the equation:

$$ 2x + y + y^\prime x - 2yy^\prime = 0 $$

Isolate $y^\prime$ to one side by subtracting $2x + y$ from both sides.

$$ y^\prime x - 2y y^\prime = -2x - y $$

Factor $y^\prime$ on the left side:

$$ y^\prime (2y - x) = -2x - y $$

Divide both sides by $x - 2y$.

$$ y^\prime = \frac{dy}{dx} = - \frac{2x - y}{x - 2y} $$

Check our answer using SymPy:

```python
idiff(x ** 2 + x * y - y ** 2 + 4, y, x)
```

For those interested, the implicit equations and their derivatives can be plotted easily using SymPy's [`plot_implicit()`](http://docs.sympy.org/latest/modules/plotting.html) function.

```python
plot_implicit(x ** 2 + x * y - y ** 2 + 4, title = 'graph of $x^2 + xy - y^2 + 4$')
plot_implicit(-(2 * x + y) / (x - 2 * y))
```

### Example 2: Find the derivative of the implicit equation $x^4 (x + y) = y^2 (3x - y)$

Focus on the left-hand side of the equation for now. Differentiate both sides of the equation and apply the Product Rule to the left-hand side.

$$ x^4 \frac{d}{dx} (x + y) + (x + y) \frac{d}{dx} x^4 = \frac{d}{dx} (y^2 (3x - y)) $$

$$ x^4 \frac{d}{dx} (x) + \frac{d}{dx} (y) + (x + y) \frac{d}{dx} x^4 = \frac{d}{dx} (y^2 (3x - y)) $$

Calculate the derivatives on the left-hand side of the equation:

$$ x^4(1 + y^\prime) + 4x^3(x + y) = y^2 \frac{d}{dx}(3x - y) + (3x - y) \frac{d}{dx} y^2 $$

Shift focus to the right-hand side of the equation:

$$ y^2 (\frac{d}{dx} 3x - \frac{d}{dx} y) + (3x - y) \frac{d}{dx} y^2 $$
$$ = y^2(3 - y^\prime) + 2yy^\prime (3x - y) $$

Thus, we now have:

$$ x^4(1 + y^\prime) + 4x^3 (x + y) = y^2(3 - y^\prime) + 2yy^\prime (3x - y) $$

Expand and clean up the terms:

$$ x^4 + x^4 y^\prime + 4x^4 + 4x^3 y = 3y^2 - y^2 y^\prime + 6xyy^\prime - 2y^2 y^\prime $$

$$ 5x^4 + x^4 y^\prime + 4x^3 y = 3y^2 + 6xyy^\prime - 3y^2 y^\prime $$

We then want to isolate $y^\prime$ to one side of the equation.

$$ 5x^4 + x^4 y^\prime + 4x^3 y - 6xyy^\prime + 3y^2 y^\prime = 3y^2 $$

$$ x^4 y^\prime - 6xyy^\prime + 3y^2 y^\prime = 3y^2 - 5x^4 - 4x^3 y $$

Finish calculating the derivative by factoring $y^\prime$ on the left-hand side of the equation and dividing:

$$ y^\prime (x^4 - 6xy + 3y^2) = 3y^2 - 5x^4 - 4x^3 y $$

$$ y^\prime = \frac{3y^2 - 5x^4 - 4x^3 y}{x^4 - 6xy + 3y^2} $$

Using SymPy as before, we can verify our answer is correct.

```python
idiff(x ** 4 * (x + y) - y ** 2 * (3 * x - y), y, x)
```

Also, for fun, we can plot the implicit function and its derivative as we did previously. Implicit functions, such as the one we just worked with, often make very appealing curves and graphs. For example, the previous implicit equation makes a unique bowtie like shape when plotted.

```python
plot_implicit(x ** 4 * (x + y) - y ** 2 * (3 * x - y))
```

The derivative of the equation can be plotted similarly.

```python
plot_implicit((-5 * x ** 5 - 4 * x ** 3 * y + 3 * y ** 2) / (x ** 4 - 6 * x * y + 3 * y ** 2))
```

### Example 3: Derivative of the equation $4 \cos{x} \space \sin{y} = 1$

Plotting the implicit equation $4 \cos{x} \space \sin{y} = 1$ reveals a rather unique and interesting graph.

```python
plot_implicit(4 * cos(x) * sin(y) - 1, (x, -10, 10), (y, -10, 10))
```

The derivative of the implicit equation can be found as in the previous examples. Start by differentiating both sides of the equation:

$$ 4 (\cos{x} \frac{d}{dx} \sin{y} + \sin{y} \frac{d}{dx} \cos{x}) = \frac{d}{dx} \space 1 $$

Then we calculate each derivative in turn, recalling that $y$ is implicitly defined as a function of $x$ and thus need to apply the Chain Rule to find the derivative $\frac{d}{dx} \sin{y}$.

$$ 4(\cos{x} \space y^\prime \cos{y} - \sin{y} \space \sin{x}) = 0 $$

Expanding the terms:

$$ 4 \cos{x} \space y^\prime \cos{y} - 4 \sin{y} \space \sin{x} = 0 $$

Isolate $y^\prime$ to one side of the equation.

$$ 4 y^\prime \cos{x} \space \cos{y} = 4 \sin{y} \sin{x} $$

$$ \frac{dy}{dx} = \frac{4 \sin{y} \sin{x}}{4 \cos{x} \space \cos{y}} $$

The derivative can be simplified further knowing that $\tan{x} = \frac{\sin{x}}{\cos{x}}$

$$ \frac{dy}{dx} = \tan{x} \tan{y} $$

Confirm our derivative is correct by checking against SymPy.

```python
idiff(4 * cos(x) * sin(y) - 1, y, x)
```

The derivative of the implicit equation also makes for an interesting graph and creates a grid-like appearance.

```python
plot_implicit(tan(x) * tan(y), (x, -10, 10), (y, -10, 10))
```

## References

[Stewart, J. (2007). Essential calculus: Early transcendentals. Belmont, CA: Thomson Higher Education.](https://amzn.to/38dnRV0)

[Strang, G. (2010). Calculus. Wellesley, MA: Wellesley-Cambridge.](https://amzn.to/2vVY0SZ)

Weisstein, Eric W. "Implicit Function." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/ImplicitFunction.html


# Data Analysis and Machine Learning: Logistic Regression

**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

Date: **Oct 18, 2018**

Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license

## Logistic Regression

In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable $y_i$ is based on some
independent variables $\hat{x}_i$. Linear regression resulted in
analytical expressions (in terms of matrices to invert) for several
quantities, ranging from the variance and thereby the confidence
intervals of the parameters $\hat{\beta}$ to the mean squared
error. If we can invert the product of the design matrices, linear
regression gives then a simple recipe for fitting our data.

Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients' brains, figure out if there is a tumor or
not; or given a specific physical system, we'd like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.

The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.

## Optimization and Deep learning

Logistic regression will also serve as our stepping stone towards neural
network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters $\hat{\beta}$. The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms.

We note also that many of the topics discussed here
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.

## Basics

We consider the case where the dependent variables, also called the
responses or the outcomes, $y_i$ are discrete and only take values
from $k=0,\dots,K-1$ (i.e. $K$ classes).

The goal is to predict the
output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
made of $n$ samples, each of which carries $p$ features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.

Let us specialize to the case of two classes only, with outputs $y_i=0$ and $y_i=1$. Our outcomes could represent the status of a credit card user who could default or not on her/his credit card debt. That is

$$
y_i = \begin{bmatrix} 0 & \mathrm{no}\\  1 & \mathrm{yes} \end{bmatrix}.
$$

## Linear classifier

Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if $y_i > 0.5$ and the no default case $y_i \leq 0.5$.

We would then have our
weighted linear combination, namely

$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} +  \hat{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
$$

where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.

## Some selected properties

The main problem with our  function is that it
takes values on the entire real axis. In the case of
logistic regression, however, the labels $y_i$ are discrete
variables.

One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values $\{0,1\}$,
$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft" classifier that outputs
the probability of a given category. This leads us to the logistic function.

The code for plotting the perceptron can be seen here. This si nothing but the standard [Heaviside step function](https://en.wikipedia.org/wiki/Heaviside_step_function).

## The logistic function

The perceptron is an example of a ``hard classification" model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a "soft"
classifier that outputs the probability of a given category rather
than a single value. For example, given $x_i$, the classifier
outputs the probability of being in a category $k$.  Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point $x_i$
belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,

$$
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$

Note that $1-p(t)= p(-t)$.
The following code plots the logistic function.

## Two parameters

We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities

$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
\end{align*}
$$

where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.

Note that we used

$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
$$

## Maximum likelihood

In order to define the total likelihood for all possible outcomes from a
dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels
$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is

$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$

from which we obtain the log-likelihood and our **cost/loss** function

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
$$

## The cost function rewritten

Reordering the logarithms, we can rewrite the **cost/loss** function as

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that

$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.

## Minimizing the cross entropy

The cross entropy is a convex function of the weights $\hat{\beta}$ and,
therefore, any local minimizer is a global minimizer.

Minimizing this
cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$

and

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$

## A more compact expression

Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
derivative of cost function as

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
$$

If we in addition define a diagonal matrix $\hat{W}$ with elements
$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as

$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
$$

## Extending to more predictors

Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors

$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$

Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to

$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$

## Including more classes

Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to $K$ classes.
Let us for the sake of simplicity assume we have only two predictors. We have then following model

1
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
$$

and so on till the class $C=K-1$ class

$$
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
$$

and the model is specified in term of $K-1$ so-called log-odds or **logit** transformations.

## The Softmax function

In our discussion of neural networks we will encounter the above again in terms of the so-called **Softmax** function.

The softmax function is used in various multiclass classification
methods, such as multinomial logistic regression (also known as
softmax regression), multiclass linear discriminant
analysis, naive Bayes classifiers, and artificial neural networks.
Specifically, in multinomial logistic regression and linear
discriminant analysis, the input to the function is the result of $K$
distinct linear functions, and the predicted probability for the $k$-th
class given a sample vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two predictors):

$$
p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
$$

It is easy to extend to more predictors. The final class is

$$
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
$$

and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations.

To find the optimal parameters we would typically use a gradient descent method.
Newton's method and gradient descent methods are discussed in the material on [optimization methods](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html).

## A **scikit-learn** example

```
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
iris = datasets.load_iris()
list(iris.keys())
['data', 'target_names', 'feature_names', 'target', 'DESCR']
X = iris["data"][:, 3:] # petal width
y = (iris["target"] == 2).astype(np.int) # 1 if Iris-Virginica, else 0

from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression()
log_reg.fit(X, y)

X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
y_proba = log_reg.predict_proba(X_new)
plt.plot(X_new, y_proba[:, 1], "g-", label="Iris-Virginica")
plt.plot(X_new, y_proba[:, 0], "b--", label="Not Iris-Virginica")
plt.show()
```

## A simple classification problem

```
import numpy as np
from sklearn import datasets, linear_model
import matplotlib.pyplot as plt

def generate_data():
    np.random.seed(0)
    X, y = datasets.make_moons(200, noise=0.20)
    return X, y

def visualize(X, y, clf):
    # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)
    # plt.show()
    plot_decision_boundary(lambda x: clf.predict(x), X, y)
    plt.title("Logistic Regression")

def plot_decision_boundary(pred_func, X, y):
    # Set min and max values and give it some padding
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole gid
    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
    plt.show()

def classify(X, y):
    clf = linear_model.LogisticRegressionCV()
    clf.fit(X, y)
    return clf

def main():
    X, y = generate_data()
    # visualize(X, y)
    clf = classify(X, y)
    visualize(X, y, clf)

if __name__ == "__main__":
    main()
```

## The two-dimensional Ising model, Predicting phase transition of the two-dimensional Ising model

The Hamiltonian of the two-dimensional Ising model without an external field for a constant coupling constant $J$ is given by

$$
\begin{equation}
    H = -J \sum_{\langle ij\rangle} S_i S_j,
\label{_auto2} \tag{2}
\end{equation}
$$

where $S_i \in \{-1, 1\}$ and $\langle ij \rangle$ signifies that we only iterate over the nearest neighbors in the lattice. We will be looking at a system of $L = 40$ spins in each dimension, i.e., $L^2 = 1600$ spins in total. Opposed to the one-dimensional Ising model we will get a phase transition from an **ordered** phase to a **disordered** phase at the critical temperature

$$
\begin{equation}
    \frac{T_c}{J} = \frac{2}{\log\left(1 + \sqrt{2}\right)} \approx 2.26,
\label{_auto3} \tag{3}
\end{equation}
$$

as shown by Lars Onsager.

Here we use **logistic regression** to predict when a phase transition
occurs. The data we will look at is a set of spin configurations,
i.e., individual lattices with spins, labeled **ordered** `1` or
**disordered** `0`. Our job is to build a model which will take in a
spin configuration and predict whether or not the spin configuration
constitutes an ordered or a disordered phase. To achieve this we will
represent the lattices as flattened arrays with $1600$ elements
instead of a matrix of $40 \times 40$ elements. As an extra test of
the performance of the algorithms we will divide the dataset into
three pieces. We will do a conventional train-test-split on a
combination of totally ordered and totally disordered phases. The
remaining "critical-like" states will be used as test data which we
hope the model will be able to make good extrapolated predictions on.

```
import pickle
import os
import glob
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.model_selection as skms
import sklearn.linear_model as skl
import sklearn.metrics as skm
import tqdm
import copy
import time
from IPython.display import display

%matplotlib inline

sns.set(color_codes=True)
```

## Reading in the data

Using the data from [Mehta et al.](https://physics.bu.edu/~pankajm/ML-Review-Datasets/isingMC/) (specifically the two datasets named `Ising2DFM_reSample_L40_T=All.pkl` and `Ising2DFM_reSample_L40_T=All_labels.pkl`) we have to unpack the data into numpy arrays.

```
filenames = glob.glob(os.path.join("..", "dat", "*"))
label_filename = list(filter(lambda x: "label" in x, filenames))[0]
dat_filename = list(filter(lambda x: "label" not in x, filenames))[0]

# Read in the labels
with open(label_filename, "rb") as f:
    labels = pickle.load(f)

# Read in the corresponding configurations
with open(dat_filename, "rb") as f:
    data = np.unpackbits(pickle.load(f)).reshape(-1, 1600).astype("int")

# Set spin-down to -1
data[data == 0] = -1
```

This dataset consists of $10000$ samples, i.e., $10000$ spin
configurations with $40 \times 40$ spins each, for $16$ temperatures
between $0.25$ to $4.0$. Next we create a train/test-split and keep
the data in the critical phase as a separate dataset for
extrapolation-testing.

```
# Set up slices of the dataset
ordered = slice(0, 70000)
critical = slice(70000, 100000)
disordered = slice(100000, 160000)

X_train, X_test, y_train, y_test = skms.train_test_split(
    np.concatenate((data[ordered], data[disordered])),
    np.concatenate((labels[ordered], labels[disordered])),
    test_size=0.95
)
```

## Logistic regression

Logistic regression is a linear model for classification. Recalling
the cost function for ordinary least squares with both L2 (ridge) and
L1 (LASSO) penalties we will see that the logistic cost function is
very similar. In OLS we wish to predict a continuous variable
$\hat{y}$ using

$$
\begin{equation}
    \hat{y} = X\omega,
\label{_auto4} \tag{4}
\end{equation}
$$

where $X \in \mathbb{R}^{n \times p}$ is the input data and $\omega^{p
\times d}$ are the weights of the regression. In a classification
setting (binary classification in our situation) we are interested in
a positive or negative answer. We can thus define either answer to be
above or below some threshold. But, in order to limit the size of the
answer and also to get a probability interpretation on how sure we are
for either answer we can compute the sigmoid function of OLS. That is,

$$
\begin{equation}
    f(X\omega) = \frac{1}{1 + \exp(-X\omega)}.
\label{_auto5} \tag{5}
\end{equation}
$$

We are thus interested in minizming the following cost function

$$
\begin{equation}
    C(X, \omega) = \sum_{i = 1}^n \left\{
        - y_i\log\left( f(x_i^T\omega) \right)
        - (1 - y_i)\log\left[1 - f(x_i^T\omega)\right]
    \right\},
\label{_auto6} \tag{6}
\end{equation}
$$

where we will restrict ourselves to a value for $f(z)$ as the sigmoid
described above. We can also tack on a L2 (Ridge) or L1 (LASSO)
penalization to this cost function in the same manner we did for
linear regression.

## Exploring the logistic regression

The penalization factor $\lambda$ is inverted in the case of the
logistic regression model we use. We will explore several values of
$\lambda$ using both L1 and L2 penalization. We do this using a grid
search over different parameters and run a 3-fold cross validation for
each configuration. In other words, we fit a model 3 times for each
configuration of the hyper parameters.

```
lambdas = np.logspace(-7, -1, 7)

param_grid = {
    "C": list(1.0/lambdas),
    "penalty": ["l1", "l2"]
}
clf = skms.GridSearchCV(
    skl.LogisticRegression(),
    param_grid=param_grid,
    n_jobs=-1,
    return_train_score=True
)
t0 = time.time()
clf.fit(X_train, y_train)
t1 = time.time()

print (
    "Time spent fitting GridSearchCV(LogisticRegression): {0:.3f} sec".format(
        t1 - t0
    )
)
```

We can see that logistic regression is quite slow and using the grid
search and cross validation results in quite a heavy
computation. Below we show the results of the different
configurations.

```
logreg_df = pd.DataFrame(clf.cv_results_)

display(logreg_df)
```

## Accuracy of a classification model

To determine how well a classification model is performing we count
the number of correctly labeled classes and divide by the number of
classes in total. The accuracy is thus given by

$$
\begin{equation}
    a(y, \hat{y}) = \frac{1}{n}\sum_{i = 1}^{n} I(y_i = \hat{y}_i),
\label{_auto7} \tag{7}
\end{equation}
$$

where $I(y_i = \hat{y}_i)$ is the indicator function given by

$$
\begin{equation}
    I(x = y) = \begin{array}{cc}
    1 & x = y, \\
    0 & x \neq y.
    \end{array}
\label{_auto8} \tag{8}
\end{equation}
$$

This is the accuracy provided by Scikit-learn when using **sklearn.metrics.accuracyscore**.

Below we compute the accuracy of the best fit model on the training data (which should give a good accuracy), the test data (which has not been shown to the model) and the critical data (completely new data that needs to be extrapolated).

```
train_accuracy = skm.accuracy_score(y_train, clf.predict(X_train))
test_accuracy = skm.accuracy_score(y_test, clf.predict(X_test))
critical_accuracy = skm.accuracy_score(labels[critical], clf.predict(data[critical]))

print ("Accuracy on train data: {0}".format(train_accuracy))
print ("Accuracy on test data: {0}".format(test_accuracy))
print ("Accuracy on critical data: {0}".format(critical_accuracy))
```

We can see that we get quite good accuracy on the training data, but gradually worsening accuracy on the test and critical data.

## Analyzing the results

Below we show a different metric for determining the quality of our
model, namely the **reciever operating characteristic** (ROC). The ROC
curve tells us how well the model correctly classifies the different
labels. We plot the **true positive rate** (the rate of predicted
positive classes that are positive) versus the **false positive rate**
(the rate of predicted positive classes that are negative). The ROC
curve is built by computing the true positive rate and the false
positive rate for varying **thresholds**, i.e, which probability we
should acredit a certain class.

By computing the **area under the curve** (AUC) of the ROC curve we get an estimate of how well our model is performing. Pure guessing will get an AUC of $0.5$. A perfect score will get an AUC of $1.0$.

```
fig = plt.figure(figsize=(20, 14))

for (_X, _y), label in zip(
    [
        (X_train, y_train),
        (X_test, y_test),
        (data[critical], labels[critical])
    ],
    ["Train", "Test", "Critical"]
):
    proba = clf.predict_proba(_X)
    fpr, tpr, _ = skm.roc_curve(_y, proba[:, 1])
    roc_auc = skm.auc(fpr, tpr)

    print ("LogisticRegression AUC ({0}): {1}".format(label, roc_auc))

    plt.plot(fpr, tpr, label="{0} (AUC = {1})".format(label, roc_auc), linewidth=4.0)

plt.plot([0, 1], [0, 1], "--", label="Guessing (AUC = 0.5)", linewidth=4.0)

plt.title(r"The ROC curve for LogisticRegression", fontsize=18)
plt.xlabel(r"False positive rate", fontsize=18)
plt.ylabel(r"True positive rate", fontsize=18)
plt.axis([-0.01, 1.01, -0.01, 1.01])
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
plt.legend(loc="best", fontsize=18)
plt.show()
```

We can see that this plot of the ROC looks very strange. This tells us
that logistic regression is quite inept at predicting the Ising model
transition and is therefore highly non-linear. The ROC curve for the
training data looks quite good, but as the testing data is so far off
we see that we are dealing with an overfit model.