```python
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
import scipy.stats
import statsmodels.api as sm
import pandas as pd
import plotly.express as px

plt.style.use('ggplot')
```

    E:\Anaconda\lib\site-packages\statsmodels\compat\pandas.py:65: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
      from pandas import Int64Index as NumericIndex

In order to follow this notes with least pain, it is strongly advised to study in linear algebra, statistics, probability theory and basic econometrics.

Click the hyperlink, you will find all course notes in my GitHub pages.

Advanced econometrics aims to provide thorough mathematical foundation to econometric theory, which will be extremely useful in handling complicated data science projects.

#  The Geometry of Vector Space

Our starting point of **Ordinary Least Square** is to understand that linear regression model usually takes the matrix form

$$
\boldsymbol{y} = \boldsymbol{X\beta}+\boldsymbol{u}\tag{1}\label{1}
$$

where $\boldsymbol{y}$ and $\boldsymbol{u}$ are $n\times 1$ matrices, $\boldsymbol{X}$ is a $n \times k$ matrix and $\boldsymbol{\beta}$ is a $k \times 1$ matrix.
$$
\boldsymbol{y}=\left[\begin{array}{c}
y_{1} \\
y_{2} \\
\vdots \\
y_{n}
\end{array}\right], \quad \boldsymbol{u}=\left[\begin{array}{c}
u_{1} \\
u_{2} \\
\vdots \\
u_{n}
\end{array}\right], \quad \boldsymbol{X}_{n\times k}=\left[\begin{array}{cccc}
1 & X_{11} & X_{12} & \ldots & X_{1k}  \\
1 & X_{21} & X_{22} & \ldots & X_{1k}\\
\vdots & \vdots & \vdots& \ddots& \vdots\\
1 & X_{n1} & X_{n2} & \ldots& X_{nk}
\end{array}\right], \quad \text { and } \quad \boldsymbol{\beta}=\left[\begin{array}{c}
\beta_{1} \\
\beta_{2} \\
\vdots\\
\beta_k
\end{array}\right]
$$

##  Two Inequalities

There are two inequalities useful for further illustration in vector space. Define $\boldsymbol{x}$ and $\boldsymbol{y}$ as vectors in $\mathbb{R}^n$.

### The Cauchy-Schwarz Inequality

$$
|\boldsymbol{x}^T\boldsymbol{y}| \leq \|\boldsymbol{x}\|\|\boldsymbol{y}\| \tag{2}\label{2}
$$
$\|\|$ notation means the **modulus** or length of vector.

The Cauchy-Schwarz inequality holds because of another famous formula dervied from law of cosine, recall that $-1 \leq \cos{\vartheta} \leq 1$

\begin{equation}
\boldsymbol{x}^T\boldsymbol{y} = \|\boldsymbol{x}\|\|\boldsymbol{y}\| \cos{\vartheta} \label{3}\tag{3}
\end{equation}

The most important implication is that when $\vartheta = 90^{\circ}$, i.e. $\boldsymbol{x}\perp\boldsymbol{y}$, then $\boldsymbol{x}^T\boldsymbol{y} = 0$.

### The Triangle Inequality

This is the vector version of a primary school concept: the length of the third edge is at least as long as the sum of other two edges.

$$
\|\boldsymbol{x}+\boldsymbol{y}\| \leq \|\boldsymbol{x}\| +  \|\boldsymbol{y}\|
$$

## Basis of Subspace

A set of vectors $\boldsymbol{X}_{n\times k} = (\boldsymbol{x}_1, ..., \boldsymbol{x}_k)$ which spans a subspace $W$ is denoted as

$$
W = \text{Span}\left(\boldsymbol{x}_{1}, \ldots, \boldsymbol{x}_{k}\right) \equiv\left\{\boldsymbol{z} \in \mathbb{R}^{n} \bigg| \boldsymbol{z}=\sum_{i=1}^{k} b_{i} \boldsymbol{x}_{i}, \quad b_{i} \in \mathbb{R}\right\}
$$

The orthogonal complement of $W$, is denoted as

$$
W^{\perp} = \text{Span}^\perp(\boldsymbol{x}_{1}, \ldots, \boldsymbol{x}_{k}) = \left\{\boldsymbol{w} \in \mathbb{R}^{n} | \boldsymbol{w}^{\top} \boldsymbol{z}=0 \text { for all } \boldsymbol{z} \in \text{Span}(\boldsymbol{X})\right\}
$$

The $\text{dim}W = k$ and $\text{dim}W^\perp = n-k$.

#  The Geometry of OLS Estimation

Take a look at the graph below, which is the geometric concept of OLS. The vectors are arbitrarily chosen only serving purpose of visualization in the codes, though demonstration is 3D, you shall imagine them exist in higher dimension, i.e. in $\mathbb{R}^n$ rather than $\mathbb{R}^3$.

```python
s = np.linspace(0, 10, 10)
t = np.linspace(0, 10, 10)
S, T = np.meshgrid(s, t)
X = S
Y = T
Z = np.zeros((10, 10))

fig, ax = plt.subplots(figsize = (10, 10))
ax = fig.add_subplot(projection='3d')
ax.plot_surface(X, Y, Z, alpha = .4)

y = np.array([6, 6, 5])
y_vec = np.array([[0, 0, 0, y[0], y[1], y[2]]])
X, Y, Z, U, V, W = zip(*y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'black', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

yhat = np.array([y[0], y[1], 0])
yhat_vec = np.array([[0, 0, 0, yhat[0], yhat[1], yhat[2]]])
X, Y, Z, U, V, W = zip(*yhat_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'red', alpha = .6,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

yhat_vec = np.array([[0, 0, 0, 0, 0, y[2]]])
X, Y, Z, U, V, W = zip(*yhat_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'red', alpha = .6,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

Xbeta = np.array([6, 3, 0])
y_vec = np.array([[0, 0, 0, Xbeta[0], Xbeta[1], Xbeta[2]]])
X, Y, Z, U, V, W = zip(*y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'purple', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

u = y - Xbeta
u_vec = np.array([[Xbeta[0], Xbeta[1], Xbeta[2], u[0], u[1], u[2]]])
X, Y, Z, U, V, W = zip(*u_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'Aqua', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

point1 = [y[0], y[1], y[2]]
point2 = [yhat[0], yhat[1], yhat[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.5, ls = '--')

ax.text(x = y[0], y = y[1], z = y[2], s = '$y$', size = 16)
ax.text(9, 9, 0, '$Col\ X$', size = 16)
ax.text(x = y[0], y = y[1], z = 0, s = r'$\hat{y}=X\hat{\beta}$', size = 16)
ax.text(x = 0, y = 0, z = y[2], s = r'$\hat{u} = y-X\hat{\beta}$', size = 16)
ax.text(x = Xbeta[0], y = Xbeta[1], z = Xbeta[2], s = r'$X\beta$', size = 16)
ax.text(x = 5.6, y = 4.1, z = 2.4, s = r'$u$', size = 16)

for i in ['x', 'y', 'z']:
    exec('ax.set_'+i+'lim3d(0, 10)')

ax.set_title('Geometric Mechanism of OLS', fontsize = 17)
ax.set_xlabel('X-axis'),ax.set_ylabel('Y-axis'),ax.set_zlabel('Z-axis')
plt.show()
```

In words, OLS algorithm is to find a special linear combination with basis of $\text{Col}\boldsymbol{X}$ or $\text{Span}\boldsymbol{X}$.

And this linear combination is the orthogonal projection $\boldsymbol{\hat{y}}$ of $\boldsymbol{y}$ onto $\text{Span}\boldsymbol{X}$, which means the distance $\|\boldsymbol{y}-\boldsymbol{\hat{y}}\|$ is the shortest among all other possible $\|\boldsymbol{y}-\text{proj}_{\text{Col}X}\boldsymbol{y}\|$, where $\text{proj}_{\text{Col}X}\boldsymbol{y}$ means a projection of $\boldsymbol{y}$ onto column space of $\boldsymbol{X}$ i.e. $\text{Col}\boldsymbol{X}$, which is depicted as a transparent plane.

And also note that orthogonal complement $\boldsymbol{\hat{u}}$ most possibly will smaller than $\boldsymbol{u}$ itself, which is decided by the nature of OLS algorithm. In the graph, we give $\boldsymbol{\hat{u}}$ red color and aqua color to $\boldsymbol{u}$.

To formulate mathematically the verbal description above,

$$
\boldsymbol{X}^T(\boldsymbol{y-X\hat{\beta}})=\boldsymbol{X}^T\boldsymbol{\hat{u}}= \boldsymbol{0}\label{4}\tag{4}
$$

which is called **orthogonality condition of OLS**.

With orthogonality condition, we achieve the most famous OLS algorithm of $\hat{\boldsymbol{\beta}}$

$$
\boldsymbol{\hat{\beta}} = (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\label{5}\tag{5}
$$

##  Test Trials of OLS Algorithm

Construct $\boldsymbol{X}$ matrix

```python
const = np.ones(100)
const = const[np.newaxis, :]

X_inde = np.random.randn(3, 100)

X = np.concatenate((const.T, X_inde.T), axis=1)
```

Set true $\boldsymbol{\beta} = [2, 3, 4, 5]^T$

```python
beta_array = np.array([2, 3, 4, 5])
beta_array = beta_array[np.newaxis, :].T
```

Generate disturbance term $\boldsymbol{u}$ with a standard normal distribution

```python
u = np.random.randn(100)
u = u[np.newaxis, :].T
```

Generate $\boldsymbol{y}$

```python
y = X@beta_array + u
```

Estimate parameters with OLS algorithm

```python
beta_hat = np.linalg.inv(X.T@X)@X.T@y
for i in range(len(beta_array)):
    print('beta' + str(i) + ':' + str(beta_hat[i]))
```

    beta0:[1.85918885]
    beta1:[2.92793357]
    beta2:[3.94075377]
    beta3:[5.103734]

Verify the results with ```statsmodels```

```python
ols_obj = sm.OLS(y, X)
```

```python
ols_obj_fit = ols_obj.fit()
beta_array_sm = ols_obj.fit().params
for i in range(len(beta_array_sm)):
    print('beta' + str(i) + ':' + str(beta_array_sm[i]))
```

    beta0:1.8591888465944915
    beta1:2.9279335723753905
    beta2:3.940753769943618
    beta3:5.1037339975308464

##  Monte Carlo Simulation of OLS Algorithm

```python
beta_hat = []
for i in range(3000):
    u = np.random.randn(100)
    u = u[np.newaxis, :].T
    y = X@beta_array + u
    beta_hat.append(np.linalg.inv(X.T@X)@X.T@y)
beta_hat = np.array(beta_hat).T
```

Compare with true parameters
$$
\boldsymbol{\beta} = [2, 3, 4, 5]^T
$$

```python
fig, ax = plt.subplots(figsize=(20, 5), nrows=1, ncols=4)
for i in range(len(beta_hat[0])):
    ax[i].hist(beta_hat[0][i], bins=100)
    ax[i].axvline(x=np.mean(beta_hat[0][i]), color='red', label ='mean {}'.format(np.round(np.mean(beta_hat[0][i]),3)))
    ax[i].set_title(r'$\beta_{}$'.format(i+1))
    ax[i].legend()
plt.show()
```

##  Goodness of Fit in Pythagorean Theorem

One important implication from orthogonality condition is that $\sum_{t=1}^n\hat{u}_t=0$.

To see why this holds, we denote the first column of $\boldsymbol{X}$ as $\iota$ (iota), which contains all $1$'s, this vector is surly in the $\text{Col}\boldsymbol X$, thus

$$
\iota^T \boldsymbol{\hat{u}} = \sum_{t=1}^n\hat{u}_t=0
$$

Using the residual of last round generated data, numerical value essentially equals $0$

```python
np.sum(ols_obj_fit.resid)
```

    2.220446049250313e-13

The fitted value $\boldsymbol{y}=\boldsymbol{X\hat{\beta}}$ is the linear combination that we are seeking for, and it is orthogonal to OLS residuals $ \boldsymbol{\hat{u}}$

$$(\boldsymbol{X} \boldsymbol{\hat{\beta}})^{T} \boldsymbol{\hat{u}}= \boldsymbol{\hat{\beta}}^{T} \boldsymbol{X}^{T} \boldsymbol{\hat{u}}=\mathbf{0}$$

$ \boldsymbol{\hat{u}}$ can be seen as a function of $ \boldsymbol{\beta}$, denoted $\boldsymbol{\hat{u}(\beta)}$, and $\boldsymbol{\hat{\beta}}$ minimizes $\|\boldsymbol{\hat{u}(\beta)}\|$.

Since we have known that $\boldsymbol{\hat{u}}\perp \text{Col}X$, then the length of three vectors can be represented by Pythagoras' Theorem.

$$
\|\boldsymbol{y}\|^2= \|\boldsymbol{X\hat{\beta}}\|^2+\|\boldsymbol{\hat{u}}\|^2\tag{5}
$$

This is geometric version of the classical property of OLS

$$
TSS = ESS + RSS
$$

Numerical version is
\begin{equation}
\underbrace{\sum_{i=1}^n(y_i-\bar{y})^2}_{TSS}=\underbrace{\sum_{i=1}^n(\hat{y}_i-\bar{y})^2}_{ESS}+\underbrace{\sum_{i=1}^n\hat{u}^2_i}_{RSS}
\end{equation}

```python
const = np.ones(100)
const = const[np.newaxis, :]
X_inde = np.random.randn(3, 100)
X = np.concatenate((const.T, X_inde.T), axis=1)

beta_array = np.array([2, 3, 4, 5])
beta_array = beta_array[np.newaxis, :].T

beta_hat = np.linalg.inv(X.T@X)@X.T@y
```

```python
TSS = np.linalg.norm(y)
ESS = np.linalg.norm(X@beta_hat)
RSS = np.linalg.norm(y-X@beta_hat)
print('TSS : {:.4f}'.format(TSS**2))
print('ESS : {:.4f}'.format(ESS**2))
print('RSS : {:.4f}'.format(RSS**2))
print('ESS + RSS : {:.4f}'.format(ESS**2+RSS**2))
```

    TSS : 6527.2487
    ESS : 230.7970
    RSS : 6296.4518
    ESS + RSS : 6527.2487

Compare with ```statsmodel``` results, we can notice the $\text{ESS}$ is different, because it is computed by subtracting $\text{RSS}$ from centered $\text{TSS}$. The difference between centered or uncentered values actually have little practical significance, i.e. in practice we report one of them and stick to it with further explanation. But we will explain later in this chapter.

```python
ols_obj_fit = sm.OLS(y, X).fit()

print('Uncentered TSS from Statsmodels : {:.4f}'.format(ols_obj_fit.uncentered_tss))
print('ESS from Statsmodels : {:.4f}'.format(ols_obj_fit.ess))
print('RSS from Statsmodels : {:.4f}'.format(ols_obj_fit.ssr))
print('ESS + RSS, from Statsmodels : {:.4f}'.format(ols_obj_fit.ess+ols_obj_fit.ssr))
```

    Uncentered TSS from Statsmodels : 6527.2487
    ESS from Statsmodels : 127.9961
    RSS from Statsmodels : 6296.4518
    ESS + RSS, from Statsmodels : 6424.4479

##  Projection Matrix

We have known that $\boldsymbol{X\hat{\beta}}$ is the orthogonal projection of $\boldsymbol{y}$ , here we substitute OLS solution back into it

$$
\boldsymbol{X\hat{\beta}}=\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}= \boldsymbol{P}_{\boldsymbol{X}}\boldsymbol{y}
$$

where $ \boldsymbol{P}_{\boldsymbol{X}}=\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T$ is a projection matrix which projects $\boldsymbol{y}$ onto $\text{Col}\boldsymbol{X}$.

If you are familiar with linear algebra, you would have seen formula of orthogonal projection, below is a vector presentation of $\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}$

$$\hat{\mathbf{y}}=\frac{\mathbf{y} \cdot \mathbf{x}_{1}}{\mathbf{x}_{1} \cdot \mathbf{x}_{1}} \mathbf{x}_{1}+\cdots+\frac{\mathbf{y} \cdot \mathbf{x}_{k}}{\mathbf{x}_{k} \cdot \mathbf{x}_{x}} \mathbf{x}_k
=\frac{\mathbf{x}_1^T \mathbf{y}}{\mathbf{x}_{1}^T  \mathbf{x}_{1}} \mathbf{x}_{1}+\cdots+\frac{\mathbf{x}_k^T \mathbf{y}}{\mathbf{x}_{k}^T  \mathbf{x}_{k}} \mathbf{x}_{k}
$$

Another projection matrix $\boldsymbol{M}_{\boldsymbol{X}}=\left(\mathbf{I}-\boldsymbol{X}\left(\boldsymbol{X}^{T} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{T}\right)= \mathbf{I} - \boldsymbol{P}_{\boldsymbol{X}}$ is constructed from

$$
\boldsymbol{\hat{u}}=\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\hat{\beta}} =\boldsymbol{y}-\boldsymbol{P}_{\boldsymbol{X}} \boldsymbol{y}=\left(\mathbf{I}-\boldsymbol{X}\left(\boldsymbol{X}^{T} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{T}\right) \boldsymbol{y}=\boldsymbol{M}_{\boldsymbol{X}} \boldsymbol{y}
$$

$\boldsymbol{M}_X$ can project any vector into the subspace of $\perp \boldsymbol{X}$.

```python
def proj_mat_P(X):
##computing projection matrix P. Input X represents regressors.##
    proj_matP = X@np.linalg.inv(X.T@X)@X.T
    return proj_matP

def proj_mat_M(X):
##computing projection matrix M. Input X represents regressors.##
    proj_matM =  np.eye(X.shape[0]) - X@np.linalg.inv(X.T@X)@X.T
    return proj_matM
```

Let's verify if our projection matrix $\boldsymbol{P}_{\boldsymbol{X}}\boldsymbol{y}$ yields the same result as $\boldsymbol{\hat{y}}$ from ```statsmodels```, calculate the difference then print the first $10$ entries.

```python
fitted_discrp = proj_mat_P(X)@y - ols_obj_fit.fittedvalues[np.newaxis,:].T
fitted_discrp[:10]
```

    array([[-2.22044605e-16],
           [-5.55111512e-16],
           [ 0.00000000e+00],
           [-2.22044605e-16],
           [ 8.88178420e-16],
           [-1.33226763e-15],
           [-6.66133815e-16],
           [-1.16573418e-15],
           [ 0.00000000e+00],
           [ 1.44328993e-15]])

Similarly, verify if $\boldsymbol{M}_{\boldsymbol{X}}\boldsymbol{y} = \boldsymbol{\hat{u}}$, also print the first $10$ entries

```python
resid_discrp = proj_mat_M(X)@y - ols_obj_fit.resid[np.newaxis,:].T
resid_discrp[:10]
```

    array([[ 8.88178420e-16],
           [ 2.66453526e-15],
           [-8.88178420e-16],
           [-1.77635684e-15],
           [-8.88178420e-16],
           [ 1.33226763e-15],
           [ 5.55111512e-16],
           [ 8.88178420e-16],
           [-7.10542736e-15],
           [-1.44328993e-15]])

###  Properties of Projection Matrix

Geometrically, it is easy to visualize properties of projection matrices in your mind
$$
\boldsymbol{P}_{\boldsymbol{X}}\boldsymbol{P}_{\boldsymbol{X}}=\boldsymbol{P}_{\boldsymbol{X}}\quad \text{and}\quad \boldsymbol{M}_{\boldsymbol{X}}\boldsymbol{M}_{\boldsymbol{X}}=\boldsymbol{M}_{\boldsymbol{X}}
$$
which are called **idempotent**.

We can verify numerically, if multiplication equal zero matrix
$$
\boldsymbol{P}_{\boldsymbol{X}}\boldsymbol{P}_{\boldsymbol{X}}-\boldsymbol{P}_{\boldsymbol{X}} = \boldsymbol{O}\\
\boldsymbol{M}_{\boldsymbol{X}}\boldsymbol{M}_{\boldsymbol{X}}-\boldsymbol{M}_{\boldsymbol{X}} = \boldsymbol{O}
$$

```python
Proj_P_X = proj_mat_P(X)
Proj_P_X@Proj_P_X - Proj_P_X
```

    array([[ 2.77555756e-17, -1.30104261e-18,  5.20417043e-18, ...,
             0.00000000e+00, -2.16840434e-18,  0.00000000e+00],
           [-2.16840434e-18,  6.93889390e-18, -3.46944695e-18, ...,
             1.04083409e-17,  0.00000000e+00, -3.46944695e-18],
           [-1.73472348e-18, -5.20417043e-18,  6.93889390e-18, ...,
            -6.93889390e-18, -1.73472348e-18,  1.38777878e-17],
           ...,
           [ 6.93889390e-18,  1.38777878e-17, -6.93889390e-18, ...,
             2.08166817e-17, -8.67361738e-18, -5.20417043e-18],
           [-2.16840434e-18,  1.73472348e-18, -3.46944695e-18, ...,
            -6.93889390e-18,  3.46944695e-18,  2.60208521e-18],
           [-1.04083409e-17, -1.73472348e-18,  1.04083409e-17, ...,
            -1.04083409e-17,  2.60208521e-18,  6.93889390e-18]])

```python
Proj_M_X = proj_mat_M(X)
Proj_M_X@Proj_M_X - Proj_M_X
```

    array([[-3.33066907e-16,  8.67361738e-19,  1.21430643e-17, ...,
            -6.93889390e-18, -2.60208521e-18,  3.46944695e-18],
           [-2.60208521e-18, -1.11022302e-16, -1.04083409e-17, ...,
             0.00000000e+00,  1.73472348e-18,  1.73472348e-18],
           [ 0.00000000e+00, -1.21430643e-17,  0.00000000e+00, ...,
            -1.04083409e-17,  1.73472348e-18,  3.46944695e-18],
           ...,
           [-6.93889390e-18,  3.46944695e-18, -1.21430643e-17, ...,
             0.00000000e+00, -8.67361738e-18, -5.20417043e-18],
           [-1.73472348e-18,  3.46944695e-18,  0.00000000e+00, ...,
            -6.93889390e-18,  1.11022302e-16,  1.73472348e-18],
           [-1.73472348e-18, -1.73472348e-18,  6.93889390e-18, ...,
            -1.04083409e-17,  1.73472348e-18,  0.00000000e+00]])

$\boldsymbol{P}_{\boldsymbol{X}}$ and $\boldsymbol{M}_{\boldsymbol{X}}$ are _symmetric matrices_, can be easily shown with rules of transpose.

$$
\boldsymbol{P}_{\boldsymbol{X}}^T = \boldsymbol{X}\left[(\boldsymbol{X}^T\boldsymbol{X})^{-1}\right]^{T}\boldsymbol{X}^T = \boldsymbol{X}\left[(\boldsymbol{X}^T\boldsymbol{X})^{T}\right]^{-1}\boldsymbol{X}^T =\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T=\boldsymbol{P}_{\boldsymbol{X}}
$$

$$
\boldsymbol{M}_{\boldsymbol{X}}^T = (\mathbf{I} - \boldsymbol{P}_{\boldsymbol{X}})^T = \mathbf{I} - \boldsymbol{P}_{\boldsymbol{X}}^T =  \mathbf{I} - \boldsymbol{P}_{\boldsymbol{X}} = \boldsymbol{M}_{\boldsymbol{X}}
$$

$\boldsymbol{P}_{\boldsymbol{X}}$ and $\boldsymbol{M}_{\boldsymbol{X}}$ are called **complementary projections**, because

$$
\boldsymbol{P}_{\boldsymbol{X}}+\boldsymbol{M}_{\boldsymbol{X}} = \mathbf{I}
$$

$\boldsymbol{P}_{\boldsymbol{X}}$ and $\boldsymbol{M}_{\boldsymbol{X}}$ **annihilate each other**,

$$
\boldsymbol{P}_{\boldsymbol{X}}\boldsymbol{M}_{\boldsymbol{X}} = \mathbf{O}
$$

```python
Proj_P_X + Proj_M_X
```

    array([[1., 0., 0., ..., 0., 0., 0.],
           [0., 1., 0., ..., 0., 0., 0.],
           [0., 0., 1., ..., 0., 0., 0.],
           ...,
           [0., 0., 0., ..., 1., 0., 0.],
           [0., 0., 0., ..., 0., 1., 0.],
           [0., 0., 0., ..., 0., 0., 1.]])

```python
Proj_P_X@Proj_M_X
```

    array([[-2.03287907e-19,  1.65340831e-18, -1.08420217e-19, ...,
             7.04731412e-19,  2.12774676e-18,  0.00000000e+00],
           [ 2.61563774e-18,  2.18195687e-18, -5.80048162e-18, ...,
            -1.09233369e-17,  5.82758668e-19,  3.46944695e-18],
           [ 5.42101086e-20, -4.06575815e-18, -5.31259065e-18, ...,
             5.47522097e-18,  9.21571847e-19, -1.38777878e-17],
           ...,
           [ 8.21283146e-18, -3.55076211e-18,  1.05709712e-17, ...,
            -1.94885341e-17,  8.87690529e-18,  5.20417043e-18],
           [ 1.32814766e-18, -2.37169225e-18, -1.21972744e-18, ...,
             6.58652820e-18, -2.40557357e-18, -2.60208521e-18],
           [ 2.00577402e-18,  4.33680869e-19, -5.96311195e-18, ...,
             8.99887803e-18, -1.65340831e-18, -6.93889390e-18]])

An  _orthogonal decomposition_ of $\boldsymbol{y}$ by Pythagoras' Theorem

$$
\|\boldsymbol{y}\|^2 = \|\boldsymbol{P}_{\boldsymbol{X}}\boldsymbol{y}\|^2+\| \boldsymbol{M}_{\boldsymbol{X}}\boldsymbol{y}\|^2
$$

However, the projection matrix is not an efficient algorithm. In computation-wise, you shall either use OLS formula or QR decomposition.

The residuals and fitted values are invariant to linear transformation of regressors (explanatory variables), which is exact reason that OLS algorithm holds even we have performed data handling, such as changing unit, in terms of projection matrix, $\boldsymbol{XA}$ is the linear combination of $\boldsymbol{X}$

$$
\boldsymbol{P}_{\boldsymbol{XA}} = \boldsymbol{XA}(\boldsymbol{A}^T\boldsymbol{X}\boldsymbol{XA})^{-1}\boldsymbol{A}^T\boldsymbol{X}^T= \boldsymbol{X}\boldsymbol{A}\boldsymbol{A}^{-1}(\boldsymbol{X}^T\boldsymbol{X})^{-1}(\boldsymbol{A}^T)^{-1}\boldsymbol{A}^T\boldsymbol{X}^T = \boldsymbol{P}_{\boldsymbol{X}}
$$

Because $\boldsymbol{XA}$ is still in $\text{Col}\boldsymbol{X}$.

#  The Frisch-Waugh-Lovell Theorem

Now consider a single linear model

$$
\boldsymbol{y}= \beta_1\boldsymbol{\iota}+\beta_2\boldsymbol{x}+\boldsymbol{u}\label{6}\tag{6}
$$

And mostly, that $\boldsymbol{x}$ and $\boldsymbol{\iota}$ are not completely independent. Geometrically speaking, they might not be orthogonal to each other.

However, we can use one-step Gram-Schmidt process to orthogonalize them. If we choose to find the orthogonal complement of $\text{proj}_{\iota}\boldsymbol{x}$, we can formulate a vector subtraction (if you don't know what's happening here, check my notebook of linear algebra.)

$$
\boldsymbol{z} = \boldsymbol{x}- \frac{\boldsymbol{x}\cdot\boldsymbol{\iota}}{\boldsymbol{\iota}\cdot\boldsymbol{\iota}}\boldsymbol{\iota} = \boldsymbol{x}-\frac{n\bar{x}}{n}\boldsymbol{\iota}=\boldsymbol{x}- \bar{x}\boldsymbol{\iota}
$$

Now we get $\boldsymbol{z}\perp \boldsymbol{\iota}$, in econometric term it is called **centering** the variable by subtracting the mean of itself. Substituting $\boldsymbol{x}= \boldsymbol{z}+\bar{x}\boldsymbol{\iota}$ back in

$$
\boldsymbol{y}=\beta_1\boldsymbol{\iota}+\beta_2(\boldsymbol{z}+\bar{x}\boldsymbol{\iota})+u = (\beta_1+\bar{x}\beta_2)\boldsymbol{\iota}+\beta_2\boldsymbol{z}+u = \alpha_1\boldsymbol{\iota}+\alpha_2\boldsymbol{z}+\boldsymbol{u}
$$

Now the we are sure that $\boldsymbol{\iota}^T \boldsymbol{z} =0$, because of orthogonality, however note that coeffcients are no longer $\beta$'s any more.

It might not be clear at this moment why we want this property, read on.

##  Multivariate Regression Model Visualization

For multivariate regression model, we can partition $\boldsymbol{X}$ in to $[\boldsymbol{X}_1, \boldsymbol{X}_2]$, where $\boldsymbol{X}_1$ is $n \times k_1$ and $\boldsymbol{X}_2$ is $n \times k_2$, $k_1+k_2 =k$.

It is impossible to visualize any multivariate regression model which is in higher dimension, but we can visualize by imagining the subspace spanned by $\boldsymbol{X}_1$ or $\boldsymbol{X}_2$ each represented by a line.

The regression model becomes

$$
\boldsymbol{y}=  \boldsymbol{X}_1\beta_1 + \boldsymbol{X}_2\beta_2 +\boldsymbol{u}\label{7}\tag{7}
$$

We denote $\boldsymbol{M}_1 = \boldsymbol{M}_{\boldsymbol{X}_1}=\mathbf{I}-\boldsymbol{P}_1$ where $\boldsymbol{P}_1 = \boldsymbol{P}_{\boldsymbol{X}_1} = \boldsymbol{X}_1(\boldsymbol{X}_1^T\boldsymbol{X}_1)^{-1}\boldsymbol{X}_1^T$.

$\boldsymbol{X}_1$ and $\boldsymbol{X}_2$ are not orthogonal, we can project $\boldsymbol{X}_2$ off $\boldsymbol{X}_1$ to obtain $\boldsymbol{Z} = \boldsymbol{M}_1\boldsymbol{X}_1$, where $\boldsymbol{Z}\perp \boldsymbol{X}_1$.

Once we orthogonalize $\boldsymbol{X}_2$, the model becomes

$$
\boldsymbol{y} = \boldsymbol{X}_1\alpha_1+\boldsymbol{M}_1\boldsymbol{X}_1\alpha_2+\boldsymbol{u}\label{8}\tag{8}
$$

From the graph below, we can see that actually $\hat{\alpha}_2=\hat{\beta}_2$, because $\boldsymbol{Z}$ and $\boldsymbol{X}_2$ form two similar triangles that

$$
\frac{\|\boldsymbol{Z}\hat{\alpha}_2\|}{\|\boldsymbol{Z}\|}=\frac{\|\boldsymbol{X}_2\hat{\beta}_2\|}{\|\boldsymbol{X}_2\|}\Longrightarrow \frac{\|\boldsymbol{Z}\|\|\hat{\alpha}_2\|}{\|\boldsymbol{Z}\|}=\frac{\|\boldsymbol{X}_2\|\|\hat{\beta}_2\|}{\|\boldsymbol{X}_2\|}\Longrightarrow \hat{\alpha}_2 =\hat{\beta}_2
$$

And also because $\boldsymbol{Z}=\boldsymbol{M}_1\boldsymbol{X}_2$ and $\boldsymbol{X}_1$ are mutally orthogonal, consequently two models

\begin{align}
\boldsymbol{y} &= \boldsymbol{X}_1\alpha_1+\boldsymbol{M}_1\boldsymbol{X}_2\beta_2 + \boldsymbol{u}\\
\boldsymbol{y} &= \boldsymbol{M}_1\boldsymbol{X}_2\beta_2 +\boldsymbol{v}
\end{align}

must yield the same estimates of $\hat{\beta}_2$. However the error terms are different, that's why we use $\boldsymbol{v}$ in the second regression model. This is shown in the graph below with two different residual vectors $\boldsymbol{\hat{u}}$ and $\boldsymbol{\hat{v}}$.

However, if we want to have the same residual, we can project $\boldsymbol{y}$ off $\boldsymbol{X}_1$, i.e. to orthogonalize $\boldsymbol{y}$ to $\boldsymbol{X}_1$.

$$
\boldsymbol{M}_1\boldsymbol{y}=\boldsymbol{M}_1\boldsymbol{X}_2\beta_2+\boldsymbol{u}\label{9}\tag{9}
$$

This model is called $\text{FWL}$ regression.

And here is the visualization of $\text{FWL}$ theorem, please walk through the plot with description above together.

```python
s = np.linspace(0, 10, 10)
t = np.linspace(0, 10, 10)
S, T = np.meshgrid(s, t)
X = S
Y = T
Z = np.zeros((10, 10))

fig, ax = plt.subplots(figsize = (10, 10))
ax = fig.add_subplot(projection='3d')
ax.plot_surface(X, Y, Z, alpha = .3)

y = np.array([6, 6, 5])
y_vec = np.array([[0, 0, 0, y[0], y[1], y[2]]])
X, Y, Z, U, V, W = zip(*y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'black', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 5)

yhat = np.array([y[0], y[1], 0])
yhat_vec = np.array([[0, 0, 0, yhat[0], yhat[1], yhat[2]]])
X, Y, Z, U, V, W = zip(*yhat_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'red', alpha = .6,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

# yhat_vec = np.array([[0, 0, 0, 0, 0, y[2]]])
# X, Y, Z, U, V, W = zip(*yhat_vec)
# ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'red', alpha = .6,arrow_length_ratio = .04, pivot = 'tail',
#           linestyles = 'solid',linewidths = 3)

################################ regressors ########################################

X1 = np.array([10, 0, 0])
X2 = np.array([3, 10, 0])
X1_vec = np.array([[0, 0, 0, X1[0], X1[1], X1[2]]])
X, Y, Z, U, V, W = zip(*X1_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'purple', alpha = .5,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

X2 = np.array([3, 10, 0])
X2_vec = np.array([[0, 0, 0, X2[0], X2[1], X2[2]]])
X, Y, Z, U, V, W = zip(*X2_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'purple', alpha = .5,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

################################ OLS estimation ########################################
X1 = np.array([10, 0, 0])
X2 = np.array([3, 10, 0])
X = np.vstack((X1, X2)).T
beta_hat = np.linalg.inv(X.T@X)@X.T@y

################################ OLS linear combination ########################################

X1Beta1hat = beta_hat[0]*X1
X1Beta1hat_vec = np.array([[0, 0, 0, X1Beta1hat[0], X1Beta1hat[1], X1Beta1hat[2]]])
X, Y, Z, U, V, W = zip(*X1Beta1hat_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'aqua', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

X2Beta1hat = beta_hat[1]*X2
X2Beta1hat_vec = np.array([[0, 0, 0, X2Beta1hat[0], X2Beta1hat[1], X2Beta1hat[2]]])
X, Y, Z, U, V, W = zip(*X2Beta1hat_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'aqua', alpha = .8,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

##################################Orthogonalizing X2 ##################################
Z_orth = np.array([0, 10, 0])
Z_orth_vec = np.array([[0, 0, 0, Z_orth[0], Z_orth[1], Z_orth[2]]])
X, Y, Z, U, V, W = zip(*Z_orth_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'Darkorange', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

######################################Projecting y onto Z##########################################

projZorth_y = (np.dot(y, Z_orth)/(np.dot(Z_orth, Z_orth)))*Z_orth
projZorth_y_vec = np.array([[0, 0, 0, projZorth_y[0], projZorth_y[1], projZorth_y[2]]])
X, Y, Z, U, V, W = zip(*projZorth_y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'aqua', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

######################################Projecting y onto X1 ##########################################

projX1_y = (np.dot(y, X1)/(np.dot(X1, X1)))*X1
projX1_y_vec = np.array([[0, 0, 0, projX1_y[0], projX1_y[1], projX1_y[2]]])
X, Y, Z, U, V, W = zip(*projX1_y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'LightSalmon', alpha = .7,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 4)

###################################### M1y ##########################################
M1y=y-projX1_y
M1y_vec = np.array([[0, 0, 0, M1y[0], M1y[1], M1y[2]]])
X, Y, Z, U, V, W = zip(*M1y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'black', alpha = .5,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 4)
##################################### Dashed Lines #############################################

point1 = [y[0], y[1], y[2]]
point2 = [yhat[0], yhat[1], yhat[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.5, ls = '--')

## Z_orth to X ##
point1 = [Z_orth[0], Z_orth[1], Z_orth[2]]
point2 = [X2[0], X2[1], X2[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'k', lw = 1.5,alpha =0.8, ls = '--')

## Z_alpha2 to yhat ##
point1 = [yhat[0], yhat[1], yhat[2]]
point2 = [projZorth_y[0], projZorth_y[1], projZorth_y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'k', lw = 1.5,alpha =0.8, ls = '--')

## X1beta1 to yhat ##
point1 = [yhat[0], yhat[1], yhat[2]]
point2 = [X1Beta1hat[0], X1Beta1hat[1], X1Beta1hat[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'k', lw = 1.5,alpha =0.8, ls = '--')

## X1alpha1 to yhat ##
point1 = [yhat[0], yhat[1], yhat[2]]
point2 = [projX1_y[0], projX1_y[1], projX1_y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'k', lw = 1.5,alpha =0.8, ls = '--')

## M1y to y ##
point1 = [y[0], y[1], y[2]]
point2 = [M1y[0], M1y[1], M1y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'k', lw = 1.5,alpha =0.8, ls = '--')

## M1y to Zalpha2 ##
point1 = [projZorth_y[0], projZorth_y[1], projZorth_y[2]]
point2 = [M1y[0], M1y[1], M1y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'k', lw = 1.5,alpha =0.8, ls = '--')

## y to Zalpha2 ##
point1 = [projZorth_y[0], projZorth_y[1], projZorth_y[2]]
point2 = [y[0], y[1], y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

###########################################################################

ax.text(x = y[0], y = y[1], z = y[2], s = '$y$', size = 16)
ax.text(9, 9, 0, '$Col\ X$', size = 16)
ax.text(x = y[0], y = y[1], z = 0, s = r'$\hat{y}=X\hat{\beta}$', size = 16)
ax.text(x = X1[0], y = X1[1], z = X1[2], s = r'$X_1$', size = 16)
ax.text(x = X2[0], y = X2[1], z = X2[2], s = r'$X_2$', size = 16)
ax.text(x = X1Beta1hat[0], y = X1Beta1hat[1], z = X1Beta1hat[2], s = r'$X_1\hat{\beta}_1$', size = 16)
ax.text(x = X2Beta1hat[0], y = X2Beta1hat[1], z = X2Beta1hat[2], s = r'$X_2\hat{\beta}_2$', size = 16)
ax.text(x = Z_orth[0], y = Z_orth[1], z = Z_orth[2], s = r'$Z=M_1X_2$', size = 16)
ax.text(x = projZorth_y[0], y = projZorth_y[1], z = projZorth_y[2], s = r'$Z\hat{\alpha}_2$', size = 16)
ax.text(x = projX1_y[0], y = projX1_y[1], z = projX1_y[2], s = r'$X_1\hat{\alpha}_1$', size = 16)
ax.text(x = M1y[0], y = M1y[1], z = M1y[2], s = r'$M_1y$', size = 16)
ax.text(x = (y[0]+yhat[0])/2, y = (y[1]+yhat[1])/2, z = (y[2]+yhat[2])/2, s = r'$\hat{u}$', size = 16, color = 'r')
ax.text(x = (y[0]+projZorth_y[0])/2, y = (y[1]+projZorth_y[1])/2, z = (y[2]+projZorth_y[2])/2, s = r'$\hat{v}$', size = 16, color = 'r')

for i in ['x', 'y', 'z']:
    exec('ax.set_'+i+'lim3d(0, 10)')

ax.set_title('Geometry of the Frisch-Waugh-Lovell Theorem', fontsize = 17)
ax.set_xlabel('X-axis'),ax.set_ylabel('Y-axis'),ax.set_zlabel('Z-axis')
plt.show()
```

Verbally, $\text{FWL}$ theorems are

1. The OLS estimates of $\beta_2$ from
\begin{align}
\boldsymbol{y} &= \boldsymbol{X}_1\alpha_1+\boldsymbol{M}_1\boldsymbol{X}_2\beta_2 + \boldsymbol{u}\\
\boldsymbol{y} &= \boldsymbol{M}_1\boldsymbol{X}_2\beta_2 +\boldsymbol{v}
\end{align}
are numerically identical.
2. The residuals from models
\begin{align}
\boldsymbol{M}_1\boldsymbol{y}&=\boldsymbol{M}_1\boldsymbol{X}_2\beta_2+\boldsymbol{u}\\
\boldsymbol{y}&=  \boldsymbol{X}_1\beta_1 + \boldsymbol{X}_2\beta_2 +\boldsymbol{u}
\end{align}
are numerically identical.

#  Goodness of Fit, $\mathbf{R}^2$

The coefficient of determination $R^2$ is defined as

$$
R^2 = \frac{ESS}{TSS} = \frac{\|\boldsymbol{P_X}\boldsymbol{y}\|^2}{\|\boldsymbol{y}\|^2}=\cos^2{\vartheta}
$$

where angle $\vartheta$ is between the vector $\|\boldsymbol{y}\|$ and $\|\boldsymbol{P_Xy}\|$. Reproduce a OLS simulation

```python
const = np.ones(100)
const = const[np.newaxis, :]
X_inde = np.random.randn(3, 100)
X = np.concatenate((const.T, X_inde.T), axis=1)

beta_array = np.array([2, 3, 4, 5])
beta_array = beta_array[np.newaxis, :].T
```

```python
u = np.random.randn(100)
u = u[np.newaxis, :].T
y = X@beta_array + u
```

```python
beta_hat = np.linalg.inv(X.T@X)@X.T@y

TSS = np.linalg.norm(y)
ESS = np.linalg.norm(X@beta_hat)
RSS = np.linalg.norm(y-X@beta_hat)
print('TSS : {:.4f}'.format(TSS**2))
print('ESS : {:.4f}'.format(ESS**2))
print('RSS : {:.4f}'.format(RSS**2))
print('ESS + RSS : {:.4f}'.format(ESS**2+RSS**2))
```

    TSS : 5096.1204
    ESS : 5014.5395
    RSS : 81.5809
    ESS + RSS : 5096.1204

```python
print('R-sqr : {:.4f}'.format(ESS/TSS))
```

    R-sqr : 0.9920

Compare with ```statsmodels``` result, the discrepancy is due to ```statsmodels``` using centered $\text{TSS}$.

```python
ols_obj = sm.OLS(y, X)
ols_obj_fit = ols_obj.fit()
ols_obj_fit.rsquared
```

    0.9828323691840988

Here's a $\text{2D}$ graphic demonstration. The larger of angle, the less of $\text{ESS}$, i.e. ${R}^2$ is smaller.

```python
fig, ax = plt.subplots(figsize = (7, 7))
ax.arrow(0, 0, 20, 0, color = 'red', width = .08,
         length_includes_head = True,
         head_width = .3, # default: 3*width
         head_length = .6,
         overhang = .4)
ax.arrow(0, 0, 20, 20, color = 'red', width = .08,
         length_includes_head = True,
         head_width = .3, # default: 3*width
         head_length = .6,
         overhang = .4)

point1 = [20, 0]
point2 = [20, 20]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], c = 'k', lw = 1.5,alpha =0.8, ls = '--')

ax.text(1.3, .2, r'$\theta$', size = 18)
ax.text(6, 7, '$TSS=\|\|y\|\|^2$', size = 18, rotation = 45)
ax.text(6, .2, r'$ESS=\|\|P_Xy\|\|^2$', size = 18)
ax.text(20, 4, r'$RSS=\|\|M_Xy\|\|^2$', size = 18, rotation = 90)

ax.text(3,15,r'$\cos^2{\vartheta}= \frac{ESS}{TSS}=\mathbf{R}^2$', size = 21)
ax.axis([0, 21, -.2, 21])
plt.show()
```

The equality holds as long as $\text{ESS}$ and $\text{RSS}$ forms a right angle, hence any other estimation methods i.e. a non-OLS $\tilde{\beta}$'s $R^2$ no longer has this pure geometric interpretation.

The $R^2 = \frac{ESS}{TSS}$ is **uncentered $R^2$** since we are using uncertered $\text{TSS}$, sometimes it is denoted as $R^2_u$, because $R^2_u$ is not _invariant_ to change of $\vartheta$.

To see this, suppose the data matrix $\boldsymbol{X} = [\boldsymbol{\iota}\quad \boldsymbol{x}]$, where $\boldsymbol{\iota}$ is the constant term, if we add $\alpha \boldsymbol{\iota}$ onto $\boldsymbol{y}$ then decompose $\boldsymbol{y}+\alpha \boldsymbol{\iota}$

$$
\boldsymbol{y}+\alpha \boldsymbol{\iota} = \boldsymbol{P_X}(\boldsymbol{y}+\alpha \boldsymbol{\iota})+\boldsymbol{M_X}(\boldsymbol{y}+\alpha \boldsymbol{\iota})= \boldsymbol{P_X}\boldsymbol{y}+\alpha \boldsymbol{\iota}+\boldsymbol{M_X}\boldsymbol{y}
$$

The $R^2_u$ is

$$
R^2_u = \frac{\|\boldsymbol{P_X}\boldsymbol{y}+\alpha \boldsymbol{\iota}\|^2}{\|\boldsymbol{y}+\alpha \boldsymbol{\iota}\|^2}
$$

Note that if $\alpha\rightarrow \infty$ , $R^2_u$ converges to $1$.

However, if we project $\boldsymbol{y}$ off $\iota$, this won't be the case anymore, because $\alpha$ doesn't exist in this case.

$$
R^2_c = \frac{\|\boldsymbol{P_X}\boldsymbol{M_\iota}\boldsymbol{y}\|^2}{\|\boldsymbol{M_\iota}\boldsymbol{y}\|^2}
$$
where $R^2_c$ means **centered** $R^2$.

##  Visualizing $\mathbf{R}^2_u$ and $\mathbf{R}^2_c$

Suppose $\iota = (1,1,1)^T$, $\boldsymbol{x} = (1,2,5)^T$ and $\boldsymbol{y} = (2,1,10)^T$. We want to show that by adding $\alpha\boldsymbol{\iota}$ onto $\boldsymbol{y}$, the angle for calculating $R_u^2$ will change, in this case it becomes smaller, i.e. $R^2_u$ increases.

```python
x = np.array([1, 2, 5])
y = np.array([2, 1, 10])
iota = np.array([1, 1, 1])

s = np.linspace(-5, 10, 10)
t = np.linspace(-5, 10, 10)
S, T = np.meshgrid(s, t)
X = S + x[0]*T
Y = S + x[1]*T
Z = S + x[2]*T

fig, ax = plt.subplots(figsize = (8, 8))
ax = fig.add_subplot(projection='3d')
ax.plot_surface(X, Y, Z, alpha = .3)

y_vec = np.array([[0, 0, 0, y[0], y[1], y[2]]])
X, Y, Z, U, V, W = zip(*y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'red', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

x_vec = np.array([[0, 0, 0, x[0], x[1], x[2]]])
X, Y, Z, U, V, W = zip(*x_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'blue', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

iota_vec = np.array([[0, 0, 0, iota[0], iota[1], iota[2]]])
X, Y, Z, U, V, W = zip(*iota_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'blue', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

X = np.vstack((iota, x)).T
Px_y = X@np.linalg.inv(X.T@X)@X.T@y
Px_y_vec = np.array([[0, 0, 0, Px_y[0], Px_y[1], Px_y[2]]])
X, Y, Z, U, V, W = zip(*Px_y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'aqua', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)
##################################################################

alpha = 5
alpha_iota = np.array([alpha, alpha, alpha])
alpha_iota_vec = np.array([[0, 0, 0, alpha_iota[0], alpha_iota[1], alpha_iota[2]]])
X, Y, Z, U, V, W = zip(*alpha_iota_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'blue', alpha = .5,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

yhat_plus_ai = Px_y + alpha_iota
yhat_plus_ai_vec = np.array([[0, 0, 0, yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2]]])
X, Y, Z, U, V, W = zip(*yhat_plus_ai_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'Darkorange', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

y_plus_ai = y + alpha_iota
y_plus_ai_vec = np.array([[0, 0, 0, y_plus_ai[0], y_plus_ai[1], y_plus_ai[2]]])
X, Y, Z, U, V, W = zip(*y_plus_ai_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'Green', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

##################################################################
## y to yhat ##
point1 = [Px_y[0], Px_y[1], Px_y[2]]
point2 = [y[0], y[1], y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## yhat+(alpha*iota) to yhat ##
point1 = [yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2]]
point2 = [Px_y[0], Px_y[1], Px_y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## yhat+(alpha*iota) to alpha*iota ##
point1 = [yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2]]
point2 = [alpha_iota[0], alpha_iota[1], alpha_iota[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## y to y+(alpha*iota) ##
point1 = [y[0], y[1], y[2]]
point2 = [y_plus_ai[0], y_plus_ai[1], y_plus_ai[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## alpha*iota to y+(alpha*iota) ##
point1 = [alpha_iota[0], alpha_iota[1], alpha_iota[2]]
point2 = [y_plus_ai[0], y_plus_ai[1], y_plus_ai[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

#####################################################################
ax.text(alpha_iota[0], alpha_iota[1],alpha_iota[2], r'$\alpha \iota$', size = 18)
ax.text(iota[0], iota[1],iota[2], r'$\iota$', size = 18)
ax.text(x[0], x[1],x[2], r'$x$', size = 18)
ax.text(y[0], y[1],y[2], r'$y$', size = 18)
ax.text(Px_y[0], Px_y[1], Px_y[2], r'$\hat{y}$', size = 18)
ax.text(yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2], r'$\hat{y}+\alpha \iota$', size = 18)
ax.text(y_plus_ai[0], y_plus_ai[1], y_plus_ai[2], r'$y+\alpha \iota$', size = 18)

for i in ['x', 'y', 'z']:
    exec('ax.set_'+i+'lim3d(0, 15)')

ax.set_title(r'Geometric Mechanism of $\mathbf{R}^2_u$ and $\mathbf{R}^2_c$', fontsize = 17)
ax.set_xlabel('X-axis'),ax.set_ylabel('Y-axis'),ax.set_zlabel('Z-axis')

ax.view_init(0, -35)
```

We can see from the graph above, the initial angle for $R^2_u$ is formed by vector $\boldsymbol{y}$ and $\boldsymbol{\hat{y}}$, after adding $\alpha\boldsymbol{\iota}$, the new angle is formed by $\boldsymbol{y}+\alpha\boldsymbol{\iota}$ and $\boldsymbol{\hat{y}}+\alpha\boldsymbol{\iota}$. Let's evaluate the $\cos^2 {\vartheta}$ with numerical values in the graph.

Recall that we have a formula

$$
\mathbf{u}\cdot \mathbf{v} =\|\mathbf{u}\|\|\mathbf{v}\|\cos{\vartheta}
$$
Rearrange then take power of $2$
$$
\cos^2{\vartheta} = \bigg(\frac{\mathbf{u}\cdot \mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}\bigg)^2=R^2_u
$$

```python
cos_theta = np.inner(y, Px_y)/(np.linalg.norm(y)*np.linalg.norm(Px_y))
print('The coefficient of determinant before addition is {0:.3f}.'.format(cos_theta**2))
```

    The coefficient of determinant before addition is 0.947.

```python
cos_theta = np.inner(y_plus_ai, yhat_plus_ai)/(np.linalg.norm(y_plus_ai)*np.linalg.norm(yhat_plus_ai))
print('The coefficient of determinant after addition is {0:.3f}.'.format(cos_theta**2))
```

    The coefficient of determinant after addition is 0.982.

The $R^2_u$ before and after the addition shows that adding constants onto $\boldsymbol{y}$ can change the value of $\cos^2{\vartheta}$.

However, in practice you don't need to go into details of their difference.

But now we need to show how to solve this problem by using $R_c^2$.

```python
x = np.array([1, 2, 5])
y = np.array([2, 1, 10])
iota = np.array([1, 1, 1])

s = np.linspace(-8, 10, 10)
t = np.linspace(-8, 10, 10)
S, T = np.meshgrid(s, t)
X = S + x[0]*T
Y = S + x[1]*T
Z = S + x[2]*T

fig, ax = plt.subplots(figsize = (8, 8))
ax = fig.add_subplot(projection='3d')
ax.plot_surface(X, Y, Z, alpha = .3)

y_vec = np.array([[0, 0, 0, y[0], y[1], y[2]]])
X, Y, Z, U, V, W = zip(*y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'red', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

X = np.vstack((iota, x)).T
Px_y = X@np.linalg.inv(X.T@X)@X.T@y
Px_y_vec = np.array([[0, 0, 0, Px_y[0], Px_y[1], Px_y[2]]])
X, Y, Z, U, V, W = zip(*Px_y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'aqua', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)
##################################################################

alpha = 5
alpha_iota = np.array([alpha, alpha, alpha])
alpha_iota_vec = np.array([[0, 0, 0, alpha_iota[0], alpha_iota[1], alpha_iota[2]]])
X, Y, Z, U, V, W = zip(*alpha_iota_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'blue', alpha = .5,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

yhat_plus_ai = Px_y + alpha_iota
yhat_plus_ai_vec = np.array([[0, 0, 0, yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2]]])
X, Y, Z, U, V, W = zip(*yhat_plus_ai_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'Darkorange', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

y_plus_ai = y + alpha_iota
y_plus_ai_vec = np.array([[0, 0, 0, y_plus_ai[0], y_plus_ai[1], y_plus_ai[2]]])
X, Y, Z, U, V, W = zip(*y_plus_ai_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'Green', alpha = 1,arrow_length_ratio = .04, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

iota = iota[:,np.newaxis]
M_iota_y = (np.eye(3,3) - iota@np.linalg.inv(iota.T@iota)@iota.T)@y
M_iota_y_vec = np.array([[0, 0, 0, M_iota_y[0], M_iota_y[1], M_iota_y[2]]])
X, Y, Z, U, V, W = zip(*M_iota_y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'SteelBlue', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)

x = np.array([1, 2, 5])
iota = np.array([1, 1, 1])
X = np.vstack((iota, x)).T
Px_M_iota_y = X@np.linalg.inv(X.T@X)@X.T@M_iota_y
Px_M_iota_y_vec = np.array([[0, 0, 0, Px_M_iota_y[0], Px_M_iota_y[1], Px_M_iota_y[2]]])
X, Y, Z, U, V, W = zip(*Px_M_iota_y_vec)
ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = 'Teal', alpha = 1,arrow_length_ratio = .08, pivot = 'tail',
          linestyles = 'solid',linewidths = 3)
##################################################################
## y to yhat ##
point1 = [Px_y[0], Px_y[1], Px_y[2]]
point2 = [y[0], y[1], y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## yhat+(alpha*iota) to yhat ##
point1 = [yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2]]
point2 = [Px_y[0], Px_y[1], Px_y[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## yhat+(alpha*iota) to alpha*iota ##
point1 = [yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2]]
point2 = [alpha_iota[0], alpha_iota[1], alpha_iota[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## y to y+(alpha*iota) ##
point1 = [y[0], y[1], y[2]]
point2 = [y_plus_ai[0], y_plus_ai[1], y_plus_ai[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

## alpha*iota to y+(alpha*iota) ##
point1 = [alpha_iota[0], alpha_iota[1], alpha_iota[2]]
point2 = [y_plus_ai[0], y_plus_ai[1], y_plus_ai[2]]
line1 = np.array([point1, point2])
ax.plot(line1[:,0], line1[:,1], line1[:, 2], c = 'b', lw = 1.5,alpha =0.8, ls = '--')

#####################################################################
ax.text(alpha_iota[0], alpha_iota[1],alpha_iota[2], r'$\alpha \iota$', size = 18)
ax.text(y[0], y[1],y[2], r'$y$', size = 18)
ax.text(Px_y[0], Px_y[1], Px_y[2], r'$\hat{y}$', size = 18)
ax.text(yhat_plus_ai[0], yhat_plus_ai[1], yhat_plus_ai[2], r'$\hat{y}+\alpha \iota$', size = 18)
ax.text(y_plus_ai[0], y_plus_ai[1], y_plus_ai[2], r'$y+\alpha \iota$', size = 18)
ax.text(M_iota_y[0], M_iota_y[1], M_iota_y[2], r'$M_\iota y$', size = 18)
ax.text(Px_M_iota_y[0], Px_M_iota_y[1], Px_M_iota_y[2], r'$P_X M_\iota y$', size = 18)

for i in ['x', 'y', 'z']:
    exec('ax.set_'+i+'lim3d(0, 15)')

ax.set_title(r'Geometric Mechanism of $\mathbf{R}^2_u$ and $\mathbf{R}^2_c$', fontsize = 17)
ax.set_xlabel('X-axis'),ax.set_ylabel('Y-axis'),ax.set_zlabel('Z-axis')

ax.view_init(-64, -48)
```

Once we have project $\boldsymbol{y}$ off $\boldsymbol{\iota}$, the angle $\vartheta$ will be preserved, no matter what contants being added onto $\boldsymbol{y}$, we can also show this algebreically

$$
R^2_c = \frac{\|\boldsymbol{P_X}\boldsymbol{M_\iota}(\boldsymbol{y}+\alpha\iota)\|^2}{\|\boldsymbol{M_\iota}(\boldsymbol{y}+\alpha\iota)\|^2}= \frac{\|\boldsymbol{P_X}\boldsymbol{M_\iota}\boldsymbol{y}\|^2}{\|\boldsymbol{M_\iota}\boldsymbol{y}\|^2}
$$

Nevertheless, the $R^2_c$ does not make sense when there is no constant term.

Again, when the estimates are other than OLS, say $\tilde{\beta}$, Pythagoras' Theorem will not hold, sometimes $R^2_c$ might be even higher than $1$ or lower than $0$.

##  Adjusted $\mathbf{R}^2$

In general, $R^2$ is a misleading indicator, it is easy to lose the insight of it. And one problem the beginners facing constantly is that adding explanatory variables can reward you by gaining higher $R_u^2$ or $R_c^2$.

It is easy to imagine the geometry why this is the case. Adding one variable means the $\text{Span}(X)$ is also adding one dimension, the new $\text{Span}(X^+)$ will be orthogonally closer to $\boldsymbol{y}$, i.e. larger $\text{ESS}$ and smaller $\text{RSS}$. To neutralize this phenomenon, we invented $\bar{R}^2$, the adjusted $R^2$.

In OLS, $\bar{R}^2$ can be negative, which signals a very poor fit.

The formula of $\bar{R}^{2}$ is

$$\bar{R}^{2} \equiv 1-\frac{\frac{1}{n-k} \sum_{t=1}^{n} \hat{u}_{t}^{2}}{\frac{1}{n-1} \sum_{t=1}^{n}\left(y_{t}-\bar{y}\right)^{2}}=1-\frac{(n-1) \boldsymbol{y}^{\top} \boldsymbol{M}_{\boldsymbol{X}} \boldsymbol{y}}{(n-k) \boldsymbol{y}^{\top} \boldsymbol{M}_{\iota} \boldsymbol{y}}$$

The numerator and denominator of $R_c^2$ were replaced by its unbiased estimator, there is no need to derive it, accept it as a fact.

##  Influential Observations

From OLS formula, we have one insight, that each element of vector $\boldsymbol{\hat{\beta}}$ is actually a weighted average of $\boldsymbol{y}$
$$
\boldsymbol{\hat{\beta}} = (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}
$$

In that sense, some observations have higher influence of $\boldsymbol{\hat{\beta}}$ than others. Here are two plots, the main difference is that the right-side plot has one outlier (blue dot). Hence the regression lines are different due to influence of blue dot outlier.

```python
n = 30
const = np.ones(n)
const = const[np.newaxis, :]
x = np.arange(1, n+1)
x = x[np.newaxis, :]
X = np.concatenate((const.T, x.T), axis=1)

u = np.random.randn(n)*20
u = u[np.newaxis, :].T

beta = np.array([2, 3])
beta = beta[np.newaxis, :].T
```

```python
y = X@beta + u
ols_obj_1_fit = sm.OLS(y, X).fit()
```

```python
y = pd.DataFrame(y)
X = pd.DataFrame(X)
```

```python
y_influ = pd.concat([y, pd.DataFrame([1])], axis = 0)
X_influ = pd.concat([X, pd.DataFrame([[1, 35]])], axis = 0)
ols_obj_2_fit = sm.OLS(y_influ, X_influ).fit()
```

```python
fig, ax = plt.subplots(figsize=(18, 8), nrows=1, ncols=2)
ax[0].scatter(X[1], y)
ax[0].plot(X[1], ols_obj_1_fit.fittedvalues, color = 'k', lw = 3)
ax[0].set_xlim([0, 36])
ax[0].set_title('Without Outlier')

ax[1].scatter(X_influ[1], y_influ)
ax[1].plot(X_influ[1], ols_obj_2_fit.fittedvalues, color = 'k', lw = 3)
ax[1].scatter(X_influ[1].iloc[30], y_influ.iloc[30], color = 'blue', s= 100)
ax[1].set_xlim([0, 36])
ax[1].set_title('With Outlier')
plt.show()
```

One common way of handling outliers is to include its corresponding dummy variable, in this example we can create a dummy variable $e_t$ that has $1$ at the $t$-th observation which is an outlier and rest of elements are $0$s.
$$
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\alpha{e_t} + \boldsymbol{u}
$$

The reason that it holds can be seen from applying $\text{FWL}$ theorem, i.e.
$$
\boldsymbol{M_t y} = \boldsymbol{M_t X}\boldsymbol{\beta} + \boldsymbol{v}
$$
where $\boldsymbol{M_t} = \boldsymbol{M}_{e_t}=\boldsymbol{I}- e_t(e_t^Te_t)^{-1}e_t^T$. With this expression, we can further conclude that $\boldsymbol{M}_{t} \boldsymbol{y}$ is the $\boldsymbol{y}$ with $t$-th element $y_t$ as $0$.

$$
\boldsymbol{M}_{t} \boldsymbol{y}=\boldsymbol{y}-\boldsymbol{e}_{t} \boldsymbol{e}_{t}^{\top} \boldsymbol{y}=\boldsymbol{y}-y_{t} \boldsymbol{e}_{t}
$$

```python
dummy = np.zeros(30)
dummy[-1] = 1
```

```python
X[2]=dummy
```

```python
X.tail()
```

      0
      1
      2

      25
      1.0
      26.0
      0.0

      26
      1.0
      27.0
      0.0

      27
      1.0
      28.0
      0.0

      28
      1.0
      29.0
      0.0

      29
      1.0
      30.0
      1.0

```python
ols_obj_3_fit = sm.OLS(y, X).fit()
print(ols_obj_3_fit.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      0   R-squared:                       0.687
    Model:                            OLS   Adj. R-squared:                  0.664
    Method:                 Least Squares   F-statistic:                     29.63
    Date:                Fri, 28 Jan 2022   Prob (F-statistic):           1.55e-07
    Time:                        16:12:01   Log-Likelihood:                -129.84
    No. Observations:                  30   AIC:                             265.7
    Df Residuals:                      27   BIC:                             269.9
    Df Model:                           2
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    0              1.6699      7.369      0.227      0.822     -13.451      16.791
    1              3.1925      0.429      7.441      0.000       2.312       4.073
    2             -9.0867     20.689     -0.439      0.664     -51.536      33.363
    ==============================================================================
    Omnibus:                        1.567   Durbin-Watson:                   2.196
    Prob(Omnibus):                  0.457   Jarque-Bera (JB):                0.992
    Skew:                          -0.445   Prob(JB):                        0.609
    Kurtosis:                       3.006   Cond. No.                         104.
    ==============================================================================

    Notes:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#  The Diagonal of $\boldsymbol{P_X}$

We denote the $t^{th}$ diagonal element of $\boldsymbol{P_X}$ as $h_t$.

$$
h_t = \boldsymbol{e}_t^T\boldsymbol{P_X}\boldsymbol{e}_t =(\boldsymbol{P_X}\boldsymbol{e}_1)^T(\boldsymbol{P_X}\boldsymbol{e}_1)= \|\boldsymbol{P_X}\boldsymbol{e}_t\|^2
$$

where $\boldsymbol{e}_t$ means the $t^{th}$ element is $1$ and rest are $0$'s.

From last equation, we can tell that $0 \leq h_t$, also because $\|\boldsymbol{P_X}\boldsymbol{e}_t\|<\|\boldsymbol{e}_t\|$ and  $\|\boldsymbol{e}_t\|=1$.So we pin down the range of $h_t$:

$$0 \leq h_t < 1$$

Chapter 1 - Geometry of Ordinary Least Squares.ipynb


> **Student Names and IDs**:
>
> - Pablo Ortiz, 0686443
> - Jongwan Park, 0848815

# Homework 5

## Homework Submission Workflow

When you submit your work, follow the instructions on the [submission workflow page](https://www.cs.duke.edu/courses/fall18/compsci371d/homework/workflow.html) for full credit, but see the changes mentioned below.

**Important: Failure to do any of the following will result in lost points:**

- Submit **one** PDF file and **one** notebook per group

- Enter **all the group members** through the Gradescope GUI when you submit your PDF files. It is **not** enough to list group members in your documents

- Match each **answer** (not question!) with the appropriate page in Gradescope

- Avoid large blank spaces in your PDF file

**Important changes to homework preparation workflow:** _This assignment is different from the others in that you are required to run it on the Google Colaboratory, a cloud service that Google makes available for reseach in machine learning. This is necessary because some of the problems require you to train a deep network on hardware that is faster than what is typically available on a standard laptop or desktop, including a high-end GPU. Even if you do have a high-end GPU, please run your notebook on the Colaboratory, so we can grade your work consistently._

_**To work on this assignment, go to the [Colaboratory](https://colab.research.google.com) and upload the template notebook for this assignment through the `File` menu at the top of the Colaboratory page. Then work on the assignment, making sure to pay attention to instructions in Part 4 where you are asked to change the runtime type.**_

_**When you are done, download the notebook (after making sure that all the outputs from running the code show up properly), and proceed as usual to turn that notebook into a PDF file for submission.**_

#### Programming Notes

+ The Colaboratory is a cloud service. If a notebook sits idle for a long time, it automatically disconnects from its execution kernel, and you need to rerun all the cells.
+ Some of the cells in the Part on neural networks are to be run with different runtime types, as explained later. Because of this, you will not be able to just restart the notebook and run all its cells with a single command. Instead, you need to run the cells one at a time, changing runtime type as instructed. Make sure you do this once you are done with the assignment, making sure that the output from your code matches the text where you describe that output.
+ Depending on circumstances, changing the runtime type may erase some or all of the notebook state. This will require you to rerun the cells that generate state.
+ Training depends on random initialization of the network parameters. Because of this, your results may vary relative to the sample solution, even if your code is no different. Results may also vary from run to run.

## Part 1: Exam-Style Questions, Set 1

The small neural net in the figure below uses the ReLU as the nonlinearity at the output of each neuron. The values specified in the hollow circles are biases, and the values along the edges are gains. Weigths number 1, 2, 3 refer to the first neuron, 4, 5,6 to the second, 7, 8, 9 to the third.

### Problem 1.1

Are all the layers in the net above fully connected?

### Answer

Yes

### Problem 1.2

What is the output $y$ from the net above when the input is as follows?

$$
x_1 = 0 \;\;\; \text{and}\;\;\; x_2 = 3
$$

### Answer

$$
\begin{align}
y &= w_7 ReLU(w_1 x_1 + w_2 x_2 + w_3) + w_8 ReLU(w_4 x_1 + w_5 x_2 + w_6) + w_9\\
&= -1*ReLU(1*0 + -2*3 + 0) + 1*ReLU(2*0 + 1*3 + 0) + 1 = 0 + 3 + 1 \\
&= 4.
\end{align}
$$

### Problem 1.3

What is the gradient $\mathbf{g}$ of the output $y$ of the network above with respect to the weight vector

$$
\mathbf{w} = [w_1,\ w_2,\ w_3,\ w_4,\ w_5,\ w_6,\ w_7,\ w_8,\ w_9]^T
$$

when the input has the values given in the previous problem? Just give the result if you are confident of your answer.

### Answer

Define:

$$
\begin{align*}
z_1 &= w_1 x_1 + w_2 x_2 + w_3 \\
&= 3w_2 + w_3, \\
z_2 &= w_4 x_1 + w_5 x_2 + w_6 \\
&= 3w_5 + w_6.
\end{align*}
$$

Then,

$$
\begin{align*}
\frac{\partial y}{\partial \mathbf{w}} &=
\left[\begin{array}@
\frac{\partial y}{\partial w_1}\;\;
\frac{\partial y}{\partial w_2}\;\;
\frac{\partial y}{\partial w_3}\;\;
\frac{\partial y}{\partial w_4}\;\;
\frac{\partial y}{\partial w_5}\;\;
\frac{\partial y}{\partial w_6}\;\;
\frac{\partial y}{\partial w_7}\;\;
\frac{\partial y}{\partial w_8}\;\;
\frac{\partial y}{\partial w_9}
\end{array}\right]\\
&= \left[\begin{array}@
0\;\;
3 w_7 H(z_1)\;\;
w_7 H(z_1)\;\;
0\;\;
3 w_8 H(z_2)\;\;
w_8 H(z_2)\;\;
ReLU(z_1)\;\;
ReLU(z_2)\;\;
1
\end{array}\right],\;
\end{align*}
$$

where $H(\cdot)$ is the Heaviside unit step function with

$$
H(x) =
\begin{cases}
    1,& \text{if } x>0\\
    0,              & \text{otherwise}
\end{cases}.
$$

## Part 2: Exam-Style Questions, Set 2

Let $\mathbf{p} = f(\mathbf{x})$ be the output of the network's soft-max layer of some neural network classifier with $K$ layers when the network's input is $\mathbf{x}$. The classifier's output is then

$$
\hat{y} = \arg\max \mathbf{p}\;.
$$

If $y_n$ is the true label corresponding to training input $\mathbf{x}_n$, the loss is $\ell_n = \ell(y_n, f(\mathbf{x}_n))$ for some appropriate loss function $\ell(y, \mathbf{p})$.

We saw in class that if $\mathbf{x}^{(k)}$ is the output from layer $k$ and $\mathbf{w}^{(k)}$ is a vector with all the parameters in layer $k$, then back-propagation computes the partial derivatives by the following recursion,  where $\mathbf{x}^{(0)} = \mathbf{x}$ is the input to the network and $\mathbf{x}^{(K)} = \mathbf{p}$:

\begin{eqnarray*}
\frac{\partial \ell_n}{\partial \mathbf{w}^{(k)}} &=& \frac{\partial \ell_n}{\partial \mathbf{x}^{(k)}} \frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{w}^{(k)}}
\;\;\;\text{for}\;\;\; k = K,\ldots, 1 \\
\frac{\partial \ell_n}{\partial \mathbf{x}^{(k-1)}} &=& \frac{\partial \ell_n}{\partial \mathbf{x}^{(k)}} \frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{x}^{(k-1)}}
\;\;\;\text{for}\;\;\; k = K,\ldots, 2\\
\frac{\partial \ell_n}{\partial \mathbf{x}^{(K)}} &=& \frac{\partial \ell}{\partial \mathbf{p}}
\end{eqnarray*}

The derivatives above are computed for the $n$-th training sample $(\mathbf{x}_n, y_n)$ and for the values of $\mathbf{w}^{(k)}$ that are current at any given point during training.

### Problem 2.1

Suppose that the network has only fully-connected layers (with ReLU nonlinearities) before the soft-max. Refer in detail to the equations given above to explain clearly why training would not work if the parameter vector $\mathbf{w} = [\mathbf{w}^{(1)},\ldots, \mathbf{w}^{(K)}]^T$ is initialized with zeros for training.

### Answer

Because this network is connected via ReLU activated nonlinearities:

$$\mathbf{x}^{(k)} = ReLU(\mathbf{w}^{(k)}_{1}*\mathbf{x}^{(k-1)}+\mathbf{w}^{(k)}_{2})\;\text{for}\;\; k = K-1,\ldots, 2$$

From this, we compute the value of $$\mathbf{x}^{(k)} = ReLU(0*\mathbf{x}^{(k-1)} + 0)=0\;\text{for}\;\; k = K-1,\ldots, 2$$

As the equation in the cell above demonstrates, it is necessary to have the term

$\frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{x}^{(k-1)}}
\;\;\;\text{defined for}\;\;\; k = K,\ldots, 2$

When trying to compute such derivative:
$$\mathbf{x}^{(k)} = ReLU(\mathbf{w}^{(k)}_{1}*\mathbf{x}^{(k-1)}+\mathbf{w}^{(k)}_{2})$$
By the chain rule:
$$\frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{x}^{(k-1)}} = \frac{\partial ReLU(\mathbf{w}^{(k)}_{1}*\mathbf{x}^{(k-1)}+\mathbf{w}^{(k)}_{2})}{\partial {x}^{(k-1)}}=\mathbf{w}^{(k)}_{1}*\frac{\partial ReLU(\mathbf{w}^{(k)}_{1}*\mathbf{x}^{(k-1)}+\mathbf{w}^{(k)}_{2})}{\partial \mathbf{w}^{(k)}_{1}*\mathbf{x}^{(k-1)}+\mathbf{w}^{(k)}_{2}} $$
For simplicity, let us define
$$\mathbf{z}^{(k)}=\mathbf{w}^{(k)}_{1}*\mathbf{x}^{(k-1)}+\mathbf{w}^{(k)}_{2}$$
With this, we re-write the equation into:
$$\frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{x}^{(k-1)}} =\mathbf{w}^{(k)}_{1}*\frac{\partial ReLU(\mathbf{z}^{(k)})}{\partial \mathbf{z}^{(k)}} $$

At this point, we start to see problems with the training: $\mathbf{z}^{(k)}=0$, and the derivative of the ReLU at 0 is not defined as the function is not differentiable at this point, meaning that the backpropagation calculations can not be computed at this point.
A workaround at this point could be worked out however, by choosing the derivative to be either 1 or 0... Let's proceed to see if doing this would fix the issue.. In either case,
$$\frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{x}^{(k-1)}} =\mathbf{w}^{(k)}_{1}*\frac{\partial ReLU(\mathbf{z}^{(k)})}{\partial \mathbf{z}^{(k)}}=0 $$
since $$\mathbf{w}^{(k)}_{1}=0$$
We evaluate the expression:
\begin{eqnarray*}
\frac{\partial \ell_n}{\partial \mathbf{x}^{(k-1)}} &=& \frac{\partial \ell_n}{\partial \mathbf{x}^{(k)}} \frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{x}^{(k-1)}}
\;\;\;\text{for}\;\;\; k = K,\ldots, 2
\end{eqnarray*}

$$\frac{\partial \ell_n}{\partial \mathbf{x}^{(k-1)}} = 0$$

We proceed with a similar analysis for the first equation from two cells above:
$$\frac{\partial \ell_n}{\partial \mathbf{w}^{(k)}} = \frac{\partial \ell_n}{\partial \mathbf{x}^{(k)}} \frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{w}^{(k)}}
\;\;\;\text{for}\;\;\; k = K,\ldots, 1 $$
where, from the cell above we can conclude that:
$$\frac{\partial \ell_n}{\partial \mathbf{w}^{(k)}} = 0*\frac{\partial \mathbf{x}^{(k)}}{\partial \mathbf{w}^{(k)}}=0
\;\;\;\text{for}\;\;\; k = K-1,\ldots, 1 $$
The only case where there can be a non-zero gradient is for k = K

We work to evaluate $$\frac{\partial \mathbf{x}^{(K)}}{\partial \mathbf{w}^{(K)}}$$
through the equation
$$\mathbf{x}^{(K)} = softMax(\mathbf{w}^{(K)}_{1}*\mathbf{x}^{(K-1)}+\mathbf{w}^{(K)}_{2})$$
we obtain
$$\frac{\partial \mathbf{x}^{(K)}}{\partial \mathbf{w}^{(K)}_1}=\mathbf{x}^{(K-1)}*\frac{\partial softMax(\mathbf{z}^{(K)})}{\partial \mathbf{z}^{(K)}}$$

$$\frac{\partial \mathbf{x}^{(K)}}{\partial \mathbf{w}^{(K)}_2}=\frac{\partial softMax(\mathbf{z}^{(K)})}{\partial \mathbf{z}^{(K)}}$$
After that, since
$$\mathbf{x}^{(K-1)} = ReLU(\mathbf{w}^{(K-1)}_1*\mathbf{x}^{(K-2)} + \mathbf{w}^{(K-1)}_2)=0$$
$$\frac{\partial \mathbf{x}^{(K)}}{\partial \mathbf{w}^{(K)}_{1}}=0*\frac{\partial softMax(\mathbf{z}^{(K)})}{\partial \mathbf{z}^{(K)}}=0$$
and
$$\frac{\partial \ell_n}{\partial \mathbf{w}^{(K)}_{1}} =\frac{\partial \ell_n}{\partial \mathbf{x}^{(K)}}*0=0$$

This shows that the gain of the last layer has a gradient of 0 on the loss function, just like all of the weights in the previous layers.

After the above analysis, the only possible parameter that can have a non-zero gradient is the bias in the very last layer: $\mathbf{w}^{(K)}_{2}$.
$$\frac{\partial \ell_n}{\partial \mathbf{w}^{(K)}_{2}}=\frac{\partial \ell_n}{\partial \mathbf{x}^{(K)}} \frac{\partial softMax(\mathbf{z}^{(K)})}{\partial \mathbf{z}^{(K)}} = \frac{\partial \ell}{\partial \mathbf{p}} \frac{\partial softMax(\mathbf{z}^{(K)})}{\partial \mathbf{z}^{(K)}}$$
This expression is not necessarily 0, as a matter of fact, it's almost certainly not 0 (depends on the definition of the loss function and/or whether the ground truth values are just zeros).

Since we have a parameter with a non-zero gradient, the algorithm can optimize it. But even after the algorithm tweaks this parameter, the analysis in this cell and the previous one will still hold on the rest of parameters, meaning this bias (or biases, depending on the number of neurons in the last layer) will be the only parameter that will be changed in the optimization--rendering the neural network useless for all other weights.

Reminder: The algorithm won't change parameters with a loss gradient of zero--it treats them as local minima. Remember that the magnitude of variation in a parameter from iteration to iteration is set by the derivative of the loss function on it--which is why none of our weights will change (other than $\mathbf{w}^{(K)}_{2}$).

### Problem 2.2

A neural net classifier with only fully-connected layers (with ReLU nonlinearities) and a soft-max layer at its output has parameter vector $\mathbf{w}$, and the network implements the function $f(\mathbf{x}, \mathbf{w})$ for any network input $\mathbf{x}$. Is $\mathbf{w} = \mathbf{0}$ a stationary point for the function $\phi(\mathbf{w}) = f(\mathbf{x}, \mathbf{w})$ when $\mathbf{x}$ is fixed? Justify your answer.

### Answer

This answer depends on whether the neural network has bias weights or not.

- If the network does not include biases:
    Yes, this is a stationary point because the gradient of the loss function from every single weight in the system is 0 (see answer from previous problem).

- If the network includes biases in its weights:
    No, as shown in the previous problem's answer, the bias from the very last layer will have a non-zero gradient and therefore this will not be a stationary point and it will optimize for that weight.

### Problem 2.3

Stochastic gradient descent with momentum is used to train a certain neural network with $m$ parameters. Just before iteration $t$ of training is performed, the parameter vector has value $\mathbf{w}_t$, and the velocity (or step) is $\mathbf{v}_t = \mathbf{a}$, where $\mathbf{a}$ is some nonzero vector in $\mathbb{R}^m$ (refer to the class notes for notation). The momentum coefficient is kept constant at $\mu = 0.9$ throughout training. If the risk function has a saddle point at $\mathbf{w}_t$, what is the step $\mathbf{v}_{t+1}$ at iteration $t$?

### Answer

Since the risk function is a saddle point, $\nabla L_{T}(\mathbf{w}_{t})=0$
$$\mathbf{v}_{t+1} = \mu_{t}\mathbf{v}_{t}-\alpha \nabla L_{T}(\mathbf{w}_{t})=0.9\mathbf{a}-\alpha*0=0.9\mathbf{a}$$

### Problem 2.4

A friend of yours argues that in the situation described in the previous problem, the steps after iteration $t$ decay exponentially. Her argument is based on the fact that the risk is at a saddle point at $\mathbf{w}_t$, and the momentum coefficient is constant, so that $\mathbf{v}_{t+\tau} = \mu^{\tau}\mathbf{a}$, an exponential decay. Explain why your friend's argument is wrong.

### Answer

Her argument is wrong because after iteration t, we will have overshot the saddle point and at the next iteration we will no longer be in a saddle point, which is the basis of her argument.

### Problem 2.5

In the situation described in Problem 2.3, will the training algorithm always eventually converge back towards $\mathbf{w}_t$? Explain your answer briefly and clearly.

### Answer

No. If the momentum variable is very large, the energy of the step in iteration t might be so strong that it moves away from the valley that the local saddle is located--moving to a different local minimum.
Additionally, the saddle is not necessarily a local minimum, in which case overshooting it could simply mean that the algorithm entered a gradient that is already stepping away from that point $\mathbf{w}_{t}$

## Part 3: Exam-Style Questions, Set 3

The following problems take you through the computation of the set of all least-squares solutions to the following linear system:

\begin{eqnarray*}
3x + 4y &=& 2\\
3x + 4y &=& 3
\end{eqnarray*}

and the solutions to a related optimization problem.
All the answers to the questions in this problem are numerical and exact. They refer only to the data given in the problem, and no more general answers are required. You may leave your answers in the form of fractions, with expressions like the following:

$$
\frac{\sqrt{3}}{2} \left[\begin{array}{c} 2\\-5\end{array}\right]\;,
$$

but please simplify as much as possible.

_As usual, it is easiest to answer these questions using software (and perhaps guess the exact values from the approximate ones output by your code). However, this would rob you of the opportunity to understand this material and to practice for the exam. **In any event, no answers will be accepted to problems in this part that embed software in your submission.**_

### Problem 3.1

What are $A$ and $\mathbf{b}$ if we write the system in this problem in the following form?
$$
A \mathbf{x} = \mathbf{b}
$$

### Answer

$$
A = \left[\begin{array}
& 3 & 4 \\
3 & 4
\end{array}\right],\\
b = \left[\begin{array}
& 2 \\
3
\end{array}\right].\\
$$

### Problem 3.2

What is the rank of $A$?

### Answer

$$
A = \left[\begin{array}
& 3 & 4 \\
3 & 4
\end{array}\right] \Rightarrow \left[\begin{array}
& 3 & 4 \\
0 & 0
\end{array}\right]
$$

$$
\therefore \; rank(A) = 1.
$$

### Problem 3.3

Give a _unit_ column vector $\mathbf{r}$ that spans the row space of $A$.

### Answer

Since

$$
Row(A) = Col(A^T) = Col\left\{{\left[\begin{array}
& 3 & 3 \\
4 & 4
\end{array}\right]}\right\},\\
$$

we have

$$
Row(A) = Span \left\{ \mathbf{r} \right\},\;\;\; \mathbf{r} = \left[\begin{array}{c} 3/5 \\ 4/5
\end{array}\right].
$$

### Problem 3.4

Give a _unit_ column vector $\mathbf{n}$ that spans the null space of $A$.

### Answer

From

$$
A\mathbf{x} = \mathbf{0},
$$

we get

$$
3x_1 + 4x_2 = 0\Leftrightarrow x_1 = -\frac{4}{3}x_2, \\
\mathbf{x} = x_2\left[\begin{array}{c}
-4/3 \\
1
\end{array}\right].
$$

Therefore,

$$
Nul(A) = Span \left\{ \mathbf{n} \right\},\;\;\; \mathbf{n} = \left[\begin{array}{c} -4/5 \\ 3/5
\end{array}\right].
$$

### Problem 3.5

Write the matrix $V$ in the SVD $A = U\Sigma V^T$ of $A$.

### Answer

Taking SVD of $A\in\mathbb{R}^{2\times2}$ yields:

$$
\begin{align}
A &= U\Sigma V^T \\
&=
[\mathbf{u}_1\;\mathbf{u}_2]
\left[\begin{array}{cc} \sigma_1 & 0 \\ 0 & \sigma_2=0 \end{array}\right]
\left[\begin{array}{c} \mathbf{v}_1^T \\ \mathbf{v}_2^T \end{array}\right].
\end{align}
$$

Since $\mathbf{v}_1$ spans $Row(A)$ and $\mathbf{v}_2$ spans $Nul(A)$,

$$
V = \left[\begin{array}{cc} 3/5 & -4/5 \\ 4/5 & 3/5 \end{array}\right].
$$

### Problem 3.6

Compute the matrices $U$ and $\Sigma$ in the SVD of $A$. [Hint: compute $U\Sigma$ first.]

### Answer

Since $\mathbf{u}_1$ spans $Col(A)$, $\mathbf{u}_1 = \left[\begin{array}{c} 1/\sqrt{2} \\ 1/\sqrt{2} \end{array}\right]$.

Since $\mathbf{u}_2$ spans $LeftNul(A)$ ($\Leftrightarrow Nul(A^T)$), $\mathbf{u}_2 = \left[\begin{array}{c} -1/\sqrt{2} \\ 1/\sqrt{2} \end{array}\right]$.

Therefore, we get

$$
U = \left[\begin{array}{cc} 1/\sqrt{2} & -1/\sqrt{2} \\ 1/\sqrt{2} & 1/\sqrt{2} \end{array}\right].
$$

Compute $U\Sigma$:

$$
\begin{align}
U\Sigma &= AV = \left[\begin{array}{cc} 5 & 0 \\ 5 & 0 \end{array}\right] \\
&= \left[\begin{array}{cc} \sigma_1/\sqrt{2} & 0 \\ \sigma_1/\sqrt{2} & 0 \end{array}\right].
\end{align}
$$

Therefore,

$$
\Sigma = \left[\begin{array}{cc} 5\sqrt{2} & 0 \\ 0 & 0 \end{array}\right].
$$

### Problem 3.7

Compute the pseudo-inverse $A^{\dagger}$ of $A$.

### Answer

The pseudo-inverse of $A$ can be computed as:

$$
\begin{align}
A^{\dagger} &= V\Sigma^{\dagger}U^T \\
&= \left[\begin{array}{cc} 3/5 & -4/5 \\ 4/5 & 3/5 \end{array}\right]
\left[\begin{array}{cc} 5\sqrt{2} & 0 \\ 0 & 0 \end{array}\right]
\left[\begin{array}{cc} 1/\sqrt{2} & -1/\sqrt{2} \\ 1/\sqrt{2} & 1/\sqrt{2} \end{array}\right] \\
&=\left[\begin{array}{cc} 3/50 & 3/50 \\ 4/50 & 4/50 \end{array}\right].
\end{align}
$$

### Problem 3.8

Find the minimum-norm least-squares solution $\mathbf{x}^*$ of the system $A\mathbf{x} = \mathbf{b}$.

### Answer

The minimum-norm least-squares solution $\mathbf{x}^*$ can be computed as:

$$
\begin{align}
\mathbf{x}^* &= A^{\dagger}\mathbf{b} \\
&= \left[\begin{array}{cc} 3/50 & 3/50 \\ 4/50 & 4/50 \end{array}\right]
\left[\begin{array}{c} 2 \\ 3 \end{array}\right]\\
&= \left[\begin{array}{c} 3/10 \\ 4/10 \end{array}\right]\\
\end{align}.
$$

### Problem 3.9

Give an expression for the set $S$ of all least-squares solutions of the system $A\mathbf{x} = \mathbf{b}$.

### Answer

We can re-write the least-squares problem as:

$$
\min \|A\mathbf{x}-\mathbf{b}\| = \min\|\Sigma \mathbf{y}-\mathbf{c}\|, \; \text{where}\; \mathbf{y}=V^T\mathbf{x}, \mathbf{c}=U^T\mathbf{b}.
$$

Let's look closely at $\Sigma \mathbf{y}-\mathbf{c}$:

$$
\Sigma \mathbf{y}-\mathbf{c} = \left[\begin{array}{cc} \sigma_1 & 0 \\ 0 & 0 \end{array}\right]
\left[\begin{array}{c} y_1 \\ y_2 \end{array}\right] -
\left[\begin{array}{c} c_1 \\ c_2 \end{array}\right].
$$

Although $y_1$ should be chosen as $c_1/\sigma_1$ to minimize the overall norm, since $y_2$ does not affect the norm at all, it can be chosen however.

Let's compute $\mathbf{c}$:

$$
\begin{align}
\left[\begin{array}{c} c_1 \\ c_2 \end{array}\right] &=
\left[\begin{array}{cc} 1/\sqrt{2} & -1/\sqrt{2} \\ 1/\sqrt{2} & 1/\sqrt{2} \end{array}\right]
\left[\begin{array}{c} 2 \\ 3 \end{array}\right] \\
&= \left[\begin{array}{c} 5/\sqrt{2} \\ 1/\sqrt{2} \end{array}\right].
\end{align}
$$

Find expression for $\mathbf{y}$:

$$
\left[\begin{array}{c} y_1 \\ y_2 \end{array}\right] =
\left[\begin{array}{cc} 3/5 & 4/5 \\ -4/5 & 3/5 \end{array}\right]
\left[\begin{array}{c} x_1 \\ x_2 \end{array}\right] \\
$$

$$
\begin{align}
y_1 &= \frac{3}{5}x_1 + \frac{4}{5}x_2 \\
&= \frac{c_1}{\sigma_1} = \frac{1}{2} \\
y_2 &= -\frac{4}{5}x_1 + \frac{3}{5}x_2 \\
&= k,\;\;\forall k\in\mathbb{R}.
\end{align}
$$

From above, we can find an expression for $\mathbf{x}$:

$$
\begin{align}
x_1 &= \frac{3}{10} - \frac{4}{5}k \\
x_2 &= \frac{2}{5} + \frac{3}{5}k
\end{align}
$$

$$
\therefore \hat{\mathbf{x}} = \left[\begin{array}{c} 3/10 \\ 4/10 \end{array}\right] + k \left[\begin{array}{c} -4/5 \\ 3/5 \end{array}\right], \;\;\forall k\in\mathbb{R}.
$$

Note that when $k=0$, we get the minimum-norm least-squares solution $\mathbf{x}^* =\left[\begin{array}{c} 3/10 \\ 4/10 \end{array}\right]$.

### Problem 3.10

Find all the solutions to

$$
 \hat{\mathbf{x}} = \arg\min_{\|\mathbf{x}\| = 1} \|A\mathbf{x}\|\;.
$$

### Answer

The unit-norm least-squares solution to the homogeneous linear system system $A\mathbf{x}=0$ corresponds to the last column $\mathbf{v}_2$ of right singular vector matrix $V$.

Therefore,

$$
\hat{\mathbf{x}} = \left[\begin{array}{c} -4/5 \\ 3/5 \end{array}\right],\;
\left[\begin{array}{c} 4/5 \\ -3/5 \end{array}\right],\;
\left[\begin{array}{c} 4/5 \\ 3/5 \end{array}\right],\;\text{and}\;
\left[\begin{array}{c} -4/5 \\ -3/5 \end{array}\right].
$$

Multiple solutions are possible due to sign ambiguity of SVD.

## Part 4: Neural Networks

The code in this part is somewhat modified from the [Keras documentation](https://keras.io/examples/cifar10_cnn/). It downloads the CIFAR-10 dataset, a set of 60000 labeled images grouped in 10 categories, which it splits into training, validation, and test sets. It then defines a function `network` that returns a simple convolutional neural network (the `model`), and a function `train` that trains the model for a single epoch by default, checking performance on the validation set.The function `train` also saves the trained model in a file in the cloud and evaluates the model on the test data. Finally, it returns a history of training and validation accuracies achieved after each epoch of training. The function `train` uses SGD as the default optimizer.

_**Important:**_ Make sure you select Python 3 through the `Runtime->Change runtime type` menu at the top of the notebook. Also set the hardware acceleration to `None` in that same menu. We will turn on GPU acceleration later on. TPU acceleration is not always available, so we won't use it.

```python
from __future__ import print_function
import keras
from keras.datasets import cifar10
from sklearn.model_selection import train_test_split

num_classes = 10

# The data, split between train, validation, and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
(x_train, x_validate, y_train, y_validate) = train_test_split(x_train, y_train,
                                                             test_size=0.2)
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'training samples')
print(x_validate.shape[0], 'validation samples')
print(x_test.shape[0], 'test samples')

# Convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_validate = keras.utils.to_categorical(y_validate, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

x_train = x_train.astype('float32')
x_validate = x_validate.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_validate /= 255
x_test /= 255
```

    Using TensorFlow backend.

    Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
    170500096/170498071 [==============================] - 96s 1us/step
    x_train shape: (40000, 32, 32, 3)
    40000 training samples
    10000 validation samples
    10000 test samples

```python
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D

activation_function = 'relu'

def network(activation_function='relu'):
  model = Sequential()
  model.add(Conv2D(32, (3, 3), padding='same',
                  input_shape=x_train.shape[1:]))
  model.add(Activation(activation_function))
  model.add(Conv2D(32, (3, 3)))
  model.add(Activation(activation_function))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.25))

  model.add(Conv2D(64, (3, 3), padding='same'))
  model.add(Activation(activation_function))
  model.add(Conv2D(64, (3, 3)))
  model.add(Activation(activation_function))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.25))

  model.add(Flatten())
  model.add(Dense(512))
  model.add(Activation(activation_function))
  model.add(Dropout(0.5))
  model.add(Dense(num_classes))
  model.add(Activation('softmax'))

  return model

model = network()
```

    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.

```python
from keras.preprocessing.image import ImageDataGenerator
import os
from math import ceil

def train(model, epochs=1,
          opt = keras.optimizers.SGD(lr=0.01, momentum=0.7, decay=0.001),
          verbose=2):

  batch_size = 32
  save_dir = os.path.join(os.getcwd(), 'saved_models')
  model_name = 'keras_cifar10_trained_model.h5'

  # Configure the model for training
  model.compile(loss='categorical_crossentropy',
                optimizer=opt,
                metrics=['accuracy'])

  history = model.fit(x_train, y_train,
                      batch_size=batch_size,
                      epochs=epochs,
                      validation_data=(x_validate, y_validate),
                      shuffle=False,
                      verbose=verbose)

  # Save model and weights
  if not os.path.isdir(save_dir):
    os.makedirs(save_dir)
  model_path = os.path.join(save_dir, model_name)
  model.save(model_path)

  # Score trained model.
  scores = model.evaluate(x_test, y_test, verbose=1)
  print('Test loss:', scores[0])
  print('Test accuracy:', scores[1])
  return [history.epoch, history.history['acc'], history.history['val_acc']]
```

### Problem 4.1

Using Stochastic Gradient Descent (SGD) with the default parameters in `train`, train the model for one epoch _with no hardware acceleration_.

Show your call to `train` and the outputs it generates. Is validation accuracy a reasonably good estimate of test accuracy?

#### Programming Notes

+ Hardware acceleration is turned off through the `Runtime->Change runtime type` menu at the top of the notebook, and selecting `None` for hardware acceleration.

+ Depending on circumstances, after you change the runtime type, some or all of the notebook state may be lost. This will require you to rerun some of the cells above.

+ Tensorflow may generate warning messages that depend on how the Colaboratory interface is implemented. These messages are typically harmless.

### Answer

```python
model = network()
[epoc_out, acc, val_acc] = train(model)
```

    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.
    Train on 40000 samples, validate on 10000 samples
    Epoch 1/1
     - 213s - loss: 1.9953 - acc: 0.2680 - val_loss: 1.6989 - val_acc: 0.3925
    10000/10000 [==============================] - 15s 2ms/step
    Test loss: 1.6941104961395264
    Test accuracy: 0.3906

```python
epoc_out
```

    [0]

```python
acc
```

    [0.268025]

```python
val_acc
```

    [0.3925]

Yes. Validation accuracy is a reasonably good estimate of test accuracy in this case. You can see how the 0.3983 for test accuracy is close to the 0.3894 accuracy for validation. This assumption that validation accuracy isa . good estimate of test accuracy usually holds true--asusming that they both come from similar data.
However, after a lot of messing with the nueral network's architecture in hopes of increasing validation accuracy, this might be less true.

### Problem 4.2

Repeat the previous experiment _after turning on GPU acceleration_ from the `Runtime->Change runtime type` menu.

Are the accuracy values the same as before? Explain why or why not. What is the approximate ratio of running times of CPU (no acceleration) versus GPU training?

### Answer

```python
from __future__ import print_function
import keras
from keras.datasets import cifar10
from sklearn.model_selection import train_test_split

num_classes = 10

# The data, split between train, validation, and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
(x_train, x_validate, y_train, y_validate) = train_test_split(x_train, y_train,
                                                             test_size=0.2)
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'training samples')
print(x_validate.shape[0], 'validation samples')
print(x_test.shape[0], 'test samples')

# Convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_validate = keras.utils.to_categorical(y_validate, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

x_train = x_train.astype('float32')
x_validate = x_validate.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_validate /= 255
x_test /= 255
```

    Using TensorFlow backend.

    Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
    170500096/170498071 [==============================] - 44s 0us/step
    x_train shape: (40000, 32, 32, 3)
    40000 training samples
    10000 validation samples
    10000 test samples

```python
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D

activation_function = 'relu'

def network(activation_function='relu'):
  model = Sequential()
  model.add(Conv2D(32, (3, 3), padding='same',
                  input_shape=x_train.shape[1:]))
  model.add(Activation(activation_function))
  model.add(Conv2D(32, (3, 3)))
  model.add(Activation(activation_function))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.25))

  model.add(Conv2D(64, (3, 3), padding='same'))
  model.add(Activation(activation_function))
  model.add(Conv2D(64, (3, 3)))
  model.add(Activation(activation_function))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.25))

  model.add(Flatten())
  model.add(Dense(512))
  model.add(Activation(activation_function))
  model.add(Dropout(0.5))
  model.add(Dense(num_classes))
  model.add(Activation('softmax'))

  return model

model = network()
```

    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.

```python
from keras.preprocessing.image import ImageDataGenerator
import os
from math import ceil

def train(model, epochs=1,
          opt = keras.optimizers.SGD(lr=0.01, momentum=0.7, decay=0.001),
          verbose=2):

  batch_size = 32
  save_dir = os.path.join(os.getcwd(), 'saved_models')
  model_name = 'keras_cifar10_trained_model.h5'

  # Configure the model for training
  model.compile(loss='categorical_crossentropy',
                optimizer=opt,
                metrics=['accuracy'])

  history = model.fit(x_train, y_train,
                      batch_size=batch_size,
                      epochs=epochs,
                      validation_data=(x_validate, y_validate),
                      shuffle=False,
                      verbose=verbose)

  # Save model and weights
  if not os.path.isdir(save_dir):
    os.makedirs(save_dir)
  model_path = os.path.join(save_dir, model_name)
  model.save(model_path)

  # Score trained model.
  scores = model.evaluate(x_test, y_test, verbose=1)
  print('Test loss:', scores[0])
  print('Test accuracy:', scores[1])
  return [history.epoch, history.history['acc'], history.history['val_acc']]
```

```python
# model = network() ## model has already been initialized, so no need to do it again.. This line is here so the grader can see that we didn't forget to initialize
[epoc_out, acc, val_acc] = train(model)
```

    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.
    Train on 40000 samples, validate on 10000 samples
    Epoch 1/1
     - 17s - loss: 1.9964 - acc: 0.2652 - val_loss: 1.6874 - val_acc: 0.3915
    10000/10000 [==============================] - 1s 119us/step
    Test loss: 1.6719089736938477
    Test accuracy: 0.3972

```python
acc
```

    [0.265175]

```python
val_acc
```

    [0.3915]

```python
gpu_cpu_ratio = 15 / 1.6874
print('The approximate ratio of running times of GPU to CPU is: ' + str(gpu_cpu_ratio))
```

    The approximate ratio of running times of GPU to CPU is: 8.889415669076685

The accuracies are different in each case. The reason for this is that the initialization of the parameters in the network is randomized each time. This means the optimization for each run started at a different configuration and therefore shouldn't be expected to end up at the exact location after one epoch

### Problem 4.3

We keep GPU acceleration turned on from now on.

Repeat the experiment above with the ADAM optimizer with the default parameters. This optimizer selects the descent step size adaptively. The ADAM optimizer is invoked by using parameter `opt = keras.optimizers.Adam()` in `train`.

Compare accuracies and running times with those achieved in the previous experiment.

### Answer

```python
model = network()
[epoc_out, acc, val_acc] = train(model, opt = keras.optimizers.Adam())
```

    Train on 40000 samples, validate on 10000 samples
    Epoch 1/1
     - 15s - loss: 1.5826 - acc: 0.4234 - val_loss: 1.1923 - val_acc: 0.5689
    10000/10000 [==============================] - 1s 124us/step
    Test loss: 1.1892164852142333
    Test accuracy: 0.5711

The accuracy is different as it was in our experiment with the SGD. In the ADAM case, the accuracy is larger (0.5711>0.3972), which suggests it's a better optimizer for this particular network.

The running time is not really very different (15s vs 17s, and 1s per epoch vs 1s per epoch) in both cases.

### Problem 4.4

We use the ADAM optimizer with default parameters from now on.

Repeat the previous experiment with 30 epochs of training instead of 1 (`epochs=30`). This time, store the value returned by `train`, as you will need it for plotting.

When done, plot both training accuracy and validation accuracy as functions of epoch number on the same diagram. Label the axes and add a legend to specify which plot is which.

Do you think that the classifier would perform much better if you were to train longer? Explain briefly.

#### Programming Notes

+ Look at the definition of `train` to figure out what the output from that function contains.
+ Set the value of the `verbose` parameter in the call to `train` to 0 to suppress output, which would be too long to include in your PDF file. You can estimate from your previous experiments how long the code will take to run. Alternatively, set `verbose` to 2 in early test runs, but then set it to 0 in your final run.

### Answer

```python
model = network()
[epoc_out, acc, val_acc] = train(model, opt = keras.optimizers.Adam(), epochs = 30, verbose = 0)
```

    10000/10000 [==============================] - 1s 125us/step
    Test loss: 0.7619545970916748
    Test accuracy: 0.7689

```python
# Plots for training and testing process: loss and accuracy
from matplotlib import pyplot as plt
import numpy as np
plt.figure(0)
plt.plot(acc,'r')
plt.plot(val_acc,'g')
plt.xticks(np.arange(0, 31, 2.0))
plt.rcParams['figure.figsize'] = (8, 6)
plt.xlabel("Num of Epochs")
plt.ylabel("Accuracy")
plt.title("Training Accuracy vs Validation Accuracy")
plt.legend(['train','validation'])

plt.show()
```

I do not think this would be much better if you trained it longer. As evidenced in the graph, the accuracy measures begin to plateau--especially the validation accuracy. This is suggestive of over-fitting, which means training more will not help. (But even the train accuracy plateaus.. so even if there was no overfitting, it still wouldn't make much sense to keep training)

### Problem 4.5

Suggest at least three different ways to improve the performance of the classifier defined in this Part. For each way, explain why that would help. This is an open-ended question, and answers may vary. Do _not_ implement your suggestions, and do _not_ refer to techniques we have not covered in class (such as batch normalization or other techniques you may have heard of).

If you suggest more than three ways, we will grade you for the best ones. However, we _will_ deduct points for patently wrong statements in any of your suggestions.

### Answer

1. Adding more convolutional layers in the architecture: the more layers there are, the more parameters the system can optimize for, which will make the model more flexible. The only drawbacks might be training time and overfitting--however, since our dataset is so large, we are less exposed to overfitting.

2. Change the allocation of validation and training samples. By increasing the number of training images you might be able to improve the accuracy. The drawback will be that you have a less trust-worthy validation to determine how generalizable the results are.

3. Increase the number of neurons in each convolutional layer. You can do that by increasing the number of filters in each layer. This will make the model more powerful and flexible to increase the accuracy.

hw5/Google Colab download (LOOK HERE FOR TURN IN FILES)/homework05.ipynb