# DATA 442/642 - Advanced Machine Learning Midterm
## [Yunting Chiu](https://www.linkedin.com/in/yuntingchiu/)

# Exercise 1 (10 points)
Show that the $\ell 1$ norm is a convex function (as all norms), yet it is not strictly convex. In contrast, show that the squared Euclidean norm is a strictly convex function.

## Answer
A function $f$ is **convex** if
$$
f(\lambda x + (1 − \lambda )y) \leq \lambda f(x) + (1 − \lambda )f(y)
$$
for all x, y $\in$ dom f and all $\lambda \in$ [0, 1].

If the inequality holds strictly (i.e. $<$ rather than $\leq$) for all $\lambda \in$ (0, 1) and x not equal to y, then we say that $f$ is **strictly convex**.\
The norm of a vector is a generalization of the concept of length. The $l_1$ norm of a vector $x \in R^n$ is defined as:
$$
||x||_1 := \sum_{i=1}^{n}|x_i|
$$
According to Minkowski's Inequality, we have the following inequality for $l_1$ norm:
$$
\sum_{i}^{l}(|\lambda x_i + (1- \lambda)y_i|) \leq \lambda \sum_{i}^{l}|X_i| + (1 - \lambda) \sum_{i}^{l}|y_i|
$$
which is not strictly convex. All $x_i$ and all $y_i$ are non-negative then there is no reason for strict inequality.

By constract, the squared Euclidean norm is a strictly convex function because of the follwing inequality
$$
\sum_{i}(|\lambda x_i + (1-\lambda)y_i|)^2 < \lambda \sum_{i}|X_i|^2+ (1 - \lambda) \sum_{i}|y_i|^2
$$
which gives
$$
\lambda ^2 \sum_{i}x^2_i + (1-\lambda)^2\sum_{i}y^2_i + 2\lambda (1-\lambda)x_iy_i  < \lambda \sum_{i}x^2_i + (1-\lambda)^2\sum_{i}y^2_i
$$
then
$$
2 \lambda(1- \lambda) x_iy_i < \lambda(1-\lambda)(\sum_{i}x^2_i + \sum_{i}y^2_i)
$$
So we have
$$
\sum_{i}(x_i-y_i) > 0, for x \neq y
$$
Which is strictly convex.

# Exercise 2 (10 points)
Let the observations resulting from an experiment be $x_n, n = 1, 2,..., N$. Assume that they are independent and that they originate from a Gaussian PDF with mean $\mu$ and standard deviation $\sigma$. Both, the mean and the variance, are unknown. Prove that the maximum likelihood (ML) estimates of these quantities are given

$$
\hat{\mu}ML = \frac{1}{N}\sum_{n=1}^{N}X_n,\space\hat{\sigma}^2ML =  \frac{1}{N}\sum_{n=1}^{N}X_n(X_n - \hat{\mu}ML)^2
$$
## Answer
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution. The log-likelihood function is given by
$$
L(\mu, \sigma^2) = - \frac{N}{2}ln(2 \pi) - \frac{N}{2}ln(\sigma^2) - \frac{1}{2\sigma^2}\sum_{n=1}^{N}(x_n - \mu)^2
$$
Why we start with log-likehood? Because we can simply the notation. Taking the derivative with respect to $\mu, \sigma^2$, and equating it to zero we can get the following equations. We have two unknowns, so we need to derivate two equations.
$$
 \frac{\partial L}{\partial \mu} = \frac{1}{\sigma^2}\sum_{n=1}^{N}(x_n - \mu) = \frac{1}{\sigma^2}(\sum_{n=1}^{N}x_n \sum_{n=1}^{N} \mu)=0
$$
$$
 \frac{\partial L}{\partial \sigma^2} = -\frac{N}{2}\frac{1}{\sigma^2} + \frac{1}{2\sigma^4}\cdot \sum_{n=1}^{N}(x_n - \mu)^2 = 0
$$
We know the $\sigma^4$ is $\sigma^2 \cdot \sigma^2$, so we can slove it in the above equation.

# Exercise 3 (15 points)
For the regression model where the noise vector $\eta= [ \, \eta_1,..., \eta_N	] \,^T$ comprises samples from zero mean Gaussian random variable, with covariance matrix $\Sigma_n$, show that the Fisher information matrix is given by

$$
I(\theta) = X^T\Sigma^{-1}_{n}X,
$$
where $X$ is the input matrix.

## Answer
A Gaussian distribution is defined as
$$
P(y, \theta) = \frac{1}{2 \pi^\frac{N}{2}\cdot \sqrt{det(\Sigma_{n}})} \cdot{exp(-\frac{1}{2}(y - x \cdot \theta)^T \cdot \Sigma_{n}(y - x \cdot \theta)}
$$
We take natural log of the equation and get
$$
ln (p(y, \theta)) = -\frac{1}{2}(y - x \cdot \theta)^T \cdot \Sigma_{n}^{-1}(y - x \cdot \theta) + C
$$
where $C$ is a constant.

We take the first derivative from the above equation:
$$
\frac{\partial ln (p(y, \theta))}{\partial \theta} = -\frac{1}{\theta^2}(x - \mu)
$$
Then, we take the second derivative
$$
\frac{\partial^2 ln (p(y, \theta))}{\partial \theta^2} = - x^T \cdot \Sigma_{n}^{-1} \cdot x
$$

Therefore, the Fisher information matrix is defined as follows:
$$
I(\theta) = - E(\frac{\sigma^2\ell_n P (y, \theta)}{\sigma \theta^2}) = x^T \cdot \Sigma_{n}^{-1} \cdot x
$$

# Exercise 4 (20 points)
Consider the regression problem described in one of our labs. Read the same audio file, then add white Gaussian noise at a 15 dB level and randomly “hit” 10% of the data samples with outliers (set the outlier values to 80% of the maximum value of the data samples).

Recall from Lab 6 of the kernal ridge regression

```python
from sklearn.svm import SVR
from datetime import datetime
import numpy as np
import math
import soundfile as sf
import matplotlib.pyplot as plt
import sys
import os
path = "/content/drive/MyDrive/American_University/2021_Fall/DATA-642-001_Advanced Machine Learning/GitHub/Labs/06/BladeRunner.wav"
```

```python
# This is a helping funvtion that contains different types of kernels
def kappa(x, y, kernel_type, kernel_params):
  value = None
  if kernel_type == 'gaus':
    sigma = kernel_params[0]
    N = len(x)
    norm = sum((x-y)**2)
    value = np.exp(-norm/(sigma**2))
  elif kernel_type == 'gaus_c':
    sigma = kernel_params[0]
    N = len(x)
    exponent = sum( (x-y.conj())**2 )
    # value = 2*(math.exp( -np.real(exponent)/(sigma**2) ))
    value = 2*np.real(np.exp(-exponent/(sigma**2)))
  elif kernel_type == 'linear':
    value = 0
    N = len(x)
    for i in range(0, N):
      value = value + x[i]*y[i].conj()
  elif kernel_type == 'poly':
    d = kernel_params[0]
    value = (1 + x*y.transpose())**d
    # value = ( (1 + x*y.transpose())/( math.sqrt(np.real(x*x.transpose())* np.real(y*y.transpose()) ) ) )**d;
  elif kernel_type == 'poly_c':
    d = kernel_params[0]
    value = 2*np.real((1 + x*y.transpose())**d)
  # value = 2*np.real( ( (1 + x*y.transpose())/( math.sqrt(np.real(x*x.transpose()) * np.real(y*y.transpose()) ) ) )**d )
  return value
```

```python
def kernel_regression_l2_l2_unbiased(X, y, params, kernel_type, kernel_params):
  lmbda = params[0]
  # build kernel matrix
  [d, N] = X.shape
  # for n=1:N
  #   for m=1:N
  #     K(n,m) = kappa(X(:,n), X(:,m), kernel_type, kernel_params);
  #   end;
  # end;
  if kernel_type == 'gaus':
    par = kernel_params
    norms = np.zeros(shape=(N, N))
    for i in range(0, N):
      T = X - X[:, i] # bsxfun(@minus,X,X(:,i))
      norms[i, :] = np.sum(T**2, axis=0)
    K = np.exp(-norms/(par**2))
  elif kernel_type == 'gaus_c':
    par = kernel_params
    norms = np.zeros(shape=(N, N))
    for i in range(0, N):
      T = X - X[:, i].conj() # bsxfun(@minus,X,conj(X(:,i)))
      norms[i, :] = np.sum(T**2, axis=0)
    K = 2*np.real(np.exp(-norms/(par**2)))
  else:
    K = np.zeros(shape=(N, N))
    for i in range(0, N):
      for j in range(0, N):
        K[i, j] = kappa(X[:, i], X[:, j], kernel_type, kernel_params)

  I = np.eye(N)
  A = lmbda*I+K
  c = y

  # Solve A*x=c
  sol = np.linalg.solve(A, c)
  return sol
```

```python
# -----------------------------------------------------------------
# Kernel Ridge Regression
# on an audio sequence corrupted by
# Gaussian noise and outliers
# You need a .wav file to run the experiment...
# Python3 required packages: numpy, soundfile, matplotlib
# -----------------------------------------------------------------
def kernelridge_11_19():
  np.random.seed(0)
  # --------------------------------------------------------------------
  # Reading wav file. x corresponds to time instances (is., x_i in [0,1])
  # fs is the sampling frequency
  # Replace the name "BladeRunner.wav" with the name of the file
  # you intend to use.
  # --------------------------------------------------------------------
  N = 100
  samples = 1000
  indices = range(0, samples,int(samples/N))
  start = 100000
  [data, fs] = sf.read(path)
  sound = np.array(data[start:(start+samples+1), :], dtype=np.float32)
  y = np.reshape(sound[indices, 0], newshape=(len(indices), 1))
  Ts = 1/fs # sampling period
  x = np.array(range(0, samples)).conj().transpose()*Ts # time instances of sampling
  x = x[indices]
  x = np.reshape(x, newshape=(x.shape[0], 1))
  #print(x)
  # -------------------------------------------------------
  # Add white Gaussian noise
  snr = 15 # dB
  y = py_awgn(y, snr)
  # add outliers
  O = 0.8*np.max(np.abs(y))
  percent = 0.1
  M = int(math.floor(percent*N))
  out_ind = np.random.choice(N, M, replace=False)
  outs = np.sign(np.random.randn(M, 1))*O
  y[out_ind] = y[out_ind] + outs
  M = len(y)
  # ----------Code for unbiased L2 Kernel Ridge Regression (KRR-L2)-----------
  C = 0.0001
  kernel_type = 'gaus'
  kernel_params = 0.004
  sol = kernel_regression_l2_l2_unbiased(x.conj().transpose(), y, [C], kernel_type, kernel_params)
  a0 = sol[0:N]
  # Generate regressor
  t = np.array([range(0, samples)]).conj().transpose()*Ts # Here we generate all 1000 points that will be used for prediction
  #t = np.array([range(0, samples+1)]).conj().transpose()*Ts
  M2 = len(t)
  print(M2)
  z0 = np.zeros(shape=(M2, 1))
  for k in range(0, M2):
    z0[k] = 0
    for l in range(0, N):
      z0[k] = z0[k] + a0[l]*kappa(x[l], t[k], kernel_type,[kernel_params])
  # ------------------end unbiased KRR-L2-------------------------------------
  # For unbiased KRR-L2
  plt.figure(1)
  # plot(x,y);
  plt.xlabel('time in sec')
  plt.ylabel('amplitude')
  plt.plot(t, z0, 'r', lw=1)
  plt.plot(x, y, '.', markeredgecolor=[0.3, 0.3, 0.3], markersize=5)
  title = 'unbiased KRR-L2 C= %s' % str(C)
  plt.title(title)
  plt.show()

```

```python
def py_awgn(input_signal, snr_dB, rate=1.0):
  """ Addditive White Gaussian Noise (AWGN) Channel.
  Parameters
  __________
  input_signal : 1D ndarray of floats
  Input signal to the channel.
  snr_dB : float
  Output SNR required in dB.
  rate : float
  Rate of the a FEC code used if any, otherwise 1.
  Returns
  _______
  output_signal : 1D ndarray of floats
  Output signal from the channel with the specified SNR.
  """
  avg_energy = np.sum(np.dot(input_signal.conj().T, input_signal)) / input_signal.shape[0]
  snr_linear = 10 ** (snr_dB / 10.0)
  noise_variance = avg_energy / (2 * rate * snr_linear)
  if input_signal.dtype is np.complex:
    noise = np.array([np.sqrt(noise_variance) * np.random.randn(input_signal.shape[0]) * (1 + 1j)], ndmin=2)
  else:
    noise = np.array([np.sqrt(2 * noise_variance) * np.random.randn(input_signal.shape[0])], ndmin=2)
  output_signal = input_signal + noise.conj().T
  return output_signal

if __name__ == '__main__':
  kernelridge_11_19()
```

Construct the support vector regression model

```python
N = 100
samples = 1000
indices = range(0, samples,int(samples/N))
start = 100000
[data, fs] = sf.read(path)
sound = np.array(data[start:(start+samples+1), :], dtype=np.float32)
y = np.reshape(sound[indices, 0], newshape=(len(indices), 1))
Ts = 1/fs # sampling period
x = np.array(range(0, samples)).conj().transpose()*Ts # time instances of sampling
x = x[indices]
x = np.reshape(x, newshape=(x.shape[0], 1))
#print(x.shape)

#-------------------------------------------------------
# Add white Gaussian noise
snr = 15 # dB
y = py_awgn(y, snr)

# add outliers
O = 0.8*np.max(np.abs(y)) # set the outlier values to 80%
percent = 0.1
M = int(math.floor(percent*N))
out_ind = np.random.choice(N, M, replace=False)
outs = np.sign(np.random.randn(M, 1))*O
y[out_ind] = y[out_ind] + outs
y = y.ravel()
M = len(y)
# print(M)
#print(x.shape)
#print(y.shape)
#-------------------------------------------------------
# support vector regression
#C=1
#epsilon=0.003
#kernel_type='gaus'
#kernel_params=0.004

#-------------------------------------------------------

def svm_regression(sigma, epsilon, C):
  # fit regression model
  # C - float, default=1.0: Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive. The penalty is a squared l2 penalty.
  # epsilon - float, default=0.1 Epsilon in the epsilon-SVR model. It specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value.
  svr_rbf = SVR(kernel='rbf', C=C, gamma = 1/(2*(sigma**2)), epsilon = epsilon)
  svr_lin = SVR(kernel='linear', C=C, gamma = 1/(2*(sigma**2)), epsilon = epsilon)
  svr_poly = SVR(kernel='poly', C=C, gamma = 1/(2*(sigma**2)), epsilon = epsilon, degree=2)
  y_rbf = svr_rbf.fit(x, y).predict(x)
  y_lin = svr_lin.fit(x, y).predict(x)
  y_poly = svr_poly.fit(x, y).predict(x)

  ###############################################################################
  # look at the results
  plt.scatter(x, y, c='k', label='data')
  #plt.hold('on')
  plt.plot(x, y_rbf, c='g', label='RBF kernel')
  plt.plot(x, y_lin, c='r', label='Linear kernel')
  plt.plot(x, y_poly, c='b', label='Polynomial kernel')
  plt.xlabel('time in sec')
  plt.ylabel('amplitude')
  plt.title('Support Vector Regression')
  plt.legend()
  plt.show()
  plt.tight_layout()

```

(a) Find the reconstructed data samples obtained by the support vector regression. Employ the Gaussian kernel with $\sigma$ = 0.004 and set $\epsilon$ = 0.003 and $C$ = 1. Plot the fitted curve of the reconstructed samples together with the data used for training.

```python
print(svm_regression(sigma = 0.004, epsilon = 0.003, C = 1))
```

(b) Repeat step (a) using $C$ = 0.05, 0.1, 0.5, 5, 10, 100.

```python
print(svm_regression(sigma = 0.004, epsilon = 0.003, C = 0.05))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.003, C = 0.1))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.003, C = 0.5))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.003, C = 5))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.003, C = 10))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.003, C = 100))
```

(c) Repeat step (a) using $\epsilon$ = 0.0005, 0.001, 0.01, 0.05, 0.1.

```python
print(svm_regression(sigma = 0.004, epsilon = 0.0005, C = 1))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.001, C = 1))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.01, C = 1))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.05, C = 1))
```

```python
print(svm_regression(sigma = 0.004, epsilon = 0.1, C = 1))
```

(d) Repeat step (a) using $\sigma$ = 0.001, 0.002, 0.01, 0.05, 0.1.

```python
print(svm_regression(sigma = 0.001, epsilon = 0.003, C = 1))
```

```python
print(svm_regression(sigma = 0.002, epsilon = 0.003, C = 1))
```

```python
print(svm_regression(sigma = 0.01, epsilon = 0.003, C = 1))
```

```python
print(svm_regression(sigma = 0.05, epsilon = 0.003, C = 1))
```

```python
print(svm_regression(sigma = 0.1, epsilon = 0.003, C = 1))
```

(e) Comment on the results.

We experiment with various Gaussian kernal, epsilon, and regularization parameters. After that, we conclude that the polynomial RBF kernel fits the dataset better. When we change the value of C, the model does not change vastly, and the model doesn't perform well in the linear and polynomial kernel. When we increase the value of espilon above 0.01, the predicted lines of SVR models almost touch the 0 in the y-axis, meaning that using a greater epsilon in SVR models with this dataset is not a good idea. Also, if the value of Gaussian kernal above 0.05, the fitted curve would not performs well.

# Exercise 5 (15 points)
Show, using Lagrange multipliers, that the $\ell_2$ minimizer in equation (9.18) from the textbook accepts the closed form solution

$$
\hat{\theta} = X^T(XX^T)^{−1}y
$$

Now, show that for the system $y = X\theta$ with $X \in R^{n×l}$ and $n > l$ the least squares solution is given by:

$$
\hat{\theta} = (X^TX)^{−1}X^Ty
$$

## Answer - $\ell>n$
In regression, we want to select those components of $\theta$ that have the most say in the formation of the output variable. The solution here is given by $\theta$. The $\ell_2$ minimizer tries to minimize

$$
min ||\theta||^{2}_{2}
$$

Such that

$$
x^{T}_{n}\theta = y_n, n = 1, 2, ..., N \to y = x \cdot \theta
$$

where $N<\ell$ (features < observations) is the assumption. Because it penalizes very large coeffients. The corresponding Lagrangian can be calculated as follows:

$$
L(\theta) = \theta^{T}\theta +  \sum_{n=1}^{N}\lambda_n(y_n-X^{T}_n\theta)
$$

Where $\lambda$ is the parameter that controls the weight of the constant.
We take the first order of the above equation and set that to zero. This will give us the minimum:

$$
L'(\theta) = 0 \to \theta = X^T\lambda
$$

where$\lambda = [ \, \lambda_1, \lambda_2, ..., \lambda_N	] \,^T$, and $X$ being the input matrix

$$
X = \begin{bmatrix}
    x_1^T \\
    \vdots  \\
    x_N^T
\end{bmatrix}
$$

We plug the solution into the set of constraints, which can be written
as $ y = X\theta$, now we get:

$$
X\theta = y = XX^T\lambda \to \lambda = (XX^T)^{-1} y
$$

So in ridge regression we have a closed form solution. Therefore, the final result is

$$
\theta = X^T(XX^T)^{-1}y
$$

## Answer - $n>\ell$

We know that
$$
y = X \cdot \theta
$$
based on the above, we can get,
$$
||y - x \cdot \theta ||^2_2 = (y - x \cdot \theta)^T \cdot (y - x \cdot \theta)^T
$$
then,
$$
= y^T\cdot y-y^T\cdot x\cdot\theta-\theta^T\cdot x^T\cdot y+\theta^T\cdot x^T\cdot x\cdot\theta
$$
then,
$$
(-y^T\cdot x)^T - (x^T\cdot y) + 2 \cdot x^T \cdot x \cdot \theta = 0
$$
So we can get,
$$
\theta = X^T(XX^T)^{-1}y
$$

# Exercise 6 (10 points)
Show that the null space of a full rank $N × l$ matrix $X$ is a subspace of imensionality $l − N$ , for $N < l$.

## Answer
Because $N \cdot \ell$ is a full rank matrix where $N < \ell$. We also know `rank(x) = dim(rank(x)) = N`. So we have `dim(null(x)) + dim(rank(x)) = l`. Therefore, `dim(null(x)) = l-n`, where `dim` is a dimension of a matrix `x`.

# Exercise 7
Generate in Python a sparse vector $\theta \in R^l, l = 100$, with its first five components taking random values drawn from a normal distribution with mean zero and variance one and the rest being equal to zero. Build, also, a sensing matrix $X$ with $N = 30$ rows having samples normally distributed, with mean zero and variance $1/N$, in order to get 30 observations based on the linear regression model $y = X\theta$. Then perform the following tasks.

```python
import numpy as np
import matplotlib.pyplot as plt
import math
import random

np.random.seed(1234)
N = 30 # 30 rows
k = 5 # first five components taking random values drawn from a normal distribution with mean zero and variance one and the rest being equal to zero
l = 100 # 100 columns

theta = np.zeros((l, 1))
theta[0:k] = np.random.standard_normal(size = (k, 1)) # draw samples from a standard Normal distribution (mean=0, stdev=1)
# print(theta)
X = np.dot(np.random.standard_normal(size = (N, l)),(1/math.sqrt(N)))
y = np.dot(X, theta)
print("X size is {}".format(X.shape))
print("y size is {}".format(y.shape))
#print(theta)
```

    X size is (30, 100)
    y size is (30, 1)

(a) Use a LASSO implementation to reconstruct $\theta$ from $y$ and $X$.

Lasso regression performs L1 regularization, which adds a penalty equal to the absolute value of the magnitude of coefficients. This type of regularization can result in sparse models with few coefficients; Some coefficients can become zero and eliminated from the model. Larger penalties result in coefficient values closer to zero, which is the ideal for producing simpler models.

L1 tends to shink coefficients to zero. As we can see below, the cofficients are almost zero.

```python
# https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html
from sklearn.linear_model import Lasso
from sklearn.linear_model import LinearRegression

lasso_reconst = Lasso(fit_intercept=True, normalize=True, max_iter=1e6)
lasso_reconst.fit(X, y)
error_lasso = np.linalg.norm(np.reshape(np.array(lasso_reconst.coef_), newshape=theta.shape)-theta)
print(lasso_reconst.coef_, lasso_reconst.intercept_)
print(error_lasso)
```

    [ 0. -0.  0. -0. -0. -0. -0.  0.  0. -0. -0. -0.  0.  0.  0.  0. -0. -0.
     -0. -0. -0.  0.  0.  0. -0. -0.  0.  0. -0.  0.  0.  0. -0. -0. -0.  0.
      0.  0.  0.  0. -0.  0.  0.  0.  0. -0.  0.  0. -0.  0.  0. -0.  0. -0.
      0.  0. -0. -0.  0. -0.  0.  0. -0. -0. -0.  0.  0.  0. -0. -0. -0.  0.
     -0.  0.  0.  0. -0.  0. -0.  0.  0.  0.  0. -0. -0.  0.  0.  0. -0. -0.
      0.  0.  0. -0. -0. -0. -0.  0. -0. -0.] [0.03136091]
    2.0761316931119906

When we compare the coefficients in the linear model, they are non-zero.

```python
linear_reconst = LinearRegression()
linear_reconst.fit(X, y)
#error_lin = np.linalg.norm(theta - linear_reconst.coef_)
print(linear_reconst.coef_)
#print(error_lin)
```

    [[ 0.1504216  -0.24852887  0.2851724  -0.08915576 -0.10327576 -0.12618162
      -0.04602372  0.05473597  0.07021556 -0.05015795 -0.0767972  -0.00251141
       0.08548682  0.19020808  0.06857647  0.03895134 -0.09698147 -0.10210427
      -0.22613006 -0.04752304 -0.0878581   0.07235256  0.04932862 -0.00278751
      -0.11543169 -0.00846009  0.12466218 -0.01432902 -0.04694153  0.09194229
       0.02842385  0.05432163 -0.02859287 -0.10724279 -0.06568882  0.12864594
       0.01044759  0.16077251  0.10993205  0.12337903 -0.04621973  0.08477134
       0.04870411  0.11392832  0.11209879 -0.02476117 -0.02252337 -0.00092738
      -0.11909358  0.07232582 -0.01891333 -0.14706331 -0.03034476 -0.08839947
       0.0574367   0.08681011 -0.02308903 -0.02174072  0.13930187 -0.0745217
       0.02068142  0.0311157  -0.03109188 -0.16113841 -0.01404451  0.07568557
       0.04640705  0.06853807 -0.09811168 -0.19712532 -0.00386257  0.07060244
      -0.11120965 -0.01141535 -0.02502231  0.12010114 -0.05180121  0.2208544
      -0.10120215  0.02234681  0.04724423 -0.07606814  0.07811075 -0.00621995
      -0.01792998 -0.00034544  0.06737222  0.01680416 -0.09043721 -0.03464077
       0.02525259 -0.00389582  0.01578966 -0.03366412 -0.01596172 -0.105041
      -0.19233346  0.01089446  0.05992144  0.02080798]]

(b) Repeat the experiment 500 times, with different realizations of $X$, in order to compute the probability of correct reconstruction (assume the reconstruction is exact when $||y =X\theta|| < 10^{−8}$).

The probability is 0 if we computer $||y = X\theta|| < 10^{-8}$ with the experiment 500 times.

```python
rep = 500 # repeat the experiment 500 times
error = np.zeros((rep, 1))
print(error.shape)

for i in range(0, rep):
  X = np.random.randn(N,l)*(1/math.sqrt(N))
  #X = np.dot(np.random.standard_normal(size = (N, l)),(1/math.sqrt(N))) # random values
  y = np.dot(X, theta)
  model = Lasso(alpha=0.0001, fit_intercept=False, normalize=False, max_iter=1e6).fit(X, y)
  values = model.coef_
  errorX = np.linalg.norm(np.reshape(np.array(values), newshape = theta.shape)-theta)
  #print(errorX)
  if errorX < 10**(-8):
    error[i] = 1
  else:
    error[i] = 0

probRandn = np.sum(error)/rep
print('Random Sensing Mtx: '+ str(probRandn))
```

    (500, 1)
    Random Sensing Mtx: 0.0

(c) Repeat the same experiment (500 times) with matrices of the form

$$
\begin{equation}
            \nonumber X(i, j) = \left\{
            \begin{array}{l l}
              +  \sqrt{\frac{\sqrt{p}}{N}}, & \quad\text{with probability}  \frac{1}{2\sqrt{p}}\\
              0, & \quad\text{with probability}  1- \frac{1}{\sqrt{p}}\\
              - \sqrt{\frac{\sqrt{p}}{N}}, &\quad\text{with probability}  \frac{1}{2\sqrt{p}}
            \end{array} \right.
            \end{equation}
$$

for p equal to 1, 9, 25, 36, 64 (make sure that each row and each column of $X$ has at least a nonzero component). Give an explanation why the probability of reconstruction falls as $p$ increases (observe that both the sensing matrix and the unknown vector are sparse).

## Explanation
The range of error terms are going to wider if p increased. With the larger error value, the estimated results are more likely to be bias, so the probability of reconstruction falls as p increases.

```python
p = [1, 9, 25, 36, 64]
Error = np.zeros((rep, 1))
eachMinErr = 9999
eachMaxErr = 0

# construct sparse sensing matrices
for pval in p:
  for i in range(rep):
    OK = False
    while not OK:
      kk = np.zeros((N*l, 1))
      P = np.random.permutation(N*l)
      #numofzeros = round(N*l*(1-1/math.sqrt(pval)))
      numofzeros = int(np.around(N*l*(1-1/math.sqrt(pval))))
      P[(N*l-numofzeros+1):] = 0
      #kk[P] = np.multiply(np.matmul(math.sqrt((math.sqrt(pval)/N)), np.ones((len(P),1))), np.sign(np.random.randn(len(P),1)))
      kk[P] = np.sqrt((math.sqrt(pval)/N))*np.multiply(np.ones(shape=(len(P), 1)), np.sign(np.random.randn(len(P), 1)))
      X = np.zeros((N, l))
      X[:(l*N)] = np.reshape(kk, newshape = (N, l))
      #OK = np.linalg.matrix_rank(X) == N # Return matrix rank of array using SVD method, also check if there is a full rank
      # Check if it is full rank
      if np.linalg.matrix_rank(X) == N:
        OK = True
      else:
        OK = False
    y = np.dot(X, theta)
    L1_model = Lasso(alpha=0.0001, fit_intercept=False, normalize=False, max_iter=1e6).fit(X, y)
    values = L1_model.coef_
    #errorX = np.linalg.norm(L1_model.coef_-theta)
    errorX = np.linalg.norm(np.reshape(np.array(values),newshape=theta.shape) - theta)
    #print(errorX)
    if errorX < 10 ** (-8):
      Error[i] = 1
    else:
      Error[i] = 0
    eachMinErr = min(eachMinErr, errorX)
    eachMaxErr = max(eachMaxErr, errorX)
  print("min err: {}, max err:{}".format(eachMinErr, eachMaxErr))
  probSparse = np.sum(Error)/rep
  #print(pval)
  print('Sparse Sensing Mtx, p = '+str(pval)+', the probability is '+str(probSparse))
```

    min err: 0.0039041880894823524, max err:1.0101686575393911
    Sparse Sensing Mtx, p = 1, the probability is 0.0
    min err: 0.0039041880894823524, max err:1.5537180405710938
    Sparse Sensing Mtx, p = 9, the probability is 0.0
    min err: 0.003875605385666494, max err:1.5537180405710938
    Sparse Sensing Mtx, p = 25, the probability is 0.0
    min err: 0.0038363421335191514, max err:1.6001763493778818
    Sparse Sensing Mtx, p = 36, the probability is 0.0
    min err: 0.0034991983600240503, max err:1.7485271633545891
    Sparse Sensing Mtx, p = 64, the probability is 0.0

# Testing Zone

```python
"""
kk = np.zeros((N*l, 1))
X = np.zeros((N, l))
print(kk.shape)
print(X.shape)
print(X[:l*N].shape)
a = np.multiply(np.dot(math.sqrt((math.sqrt(9)/N)), np.ones((len(P),1))) , np.sign(np.random.rand(len(P),1)))
print(a.shape)
"""
```

    '\nkk = np.zeros((N*l, 1))\nX = np.zeros((N, l))\nprint(kk.shape)\nprint(X.shape)\nprint(X[:l*N].shape)\na = np.multiply(np.dot(math.sqrt((math.sqrt(9)/N)), np.ones((len(P),1))) , np.sign(np.random.rand(len(P),1)))\nprint(a.shape)\n'

```python
"""
P = np.random.permutation(N*l)
print(P.shape)
"""
```

    '\nP = np.random.permutation(N*l)\nprint(P.shape)\n'

# Output

```python
# should access the Google Drive files before running the chunk
%%capture
!sudo apt-get install texlive-xetex texlive-fonts-recommended texlive-plain-generic
!jupyter nbconvert --to html "/content/drive/MyDrive/American_University/2021_Fall/DATA-642-001_Advanced Machine Learning/GitHub/Midterm/submit/midterm_Yunting.ipynb"
```

# References:
- https://www.statisticshowto.com/lasso-regression/
- https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html
- https://ai.stanford.edu/~gwthomas/notes/convexity.pdf
- https://en.wikipedia.org/wiki/Convex_function
- https://math.stackexchange.com/questions/1356842/is-l-2-norm-a-strictly-convex-function
- https://cims.nyu.edu/~cfgranda/pages/OBDA_spring16/material/convex_optimization.pdf
- https://math.stackexchange.com/questions/80139/why-is-the-l-p-norm-strictly-convex-for-1p-infty
- https://ai.stanford.edu/~gwthomas/notes/convexity.pdf
- https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html


# Binary quadratic problems

We discuss a simple solver for unconstrained binary quadratic problems, that is optimization problems of the form

\begin{equation}
\begin{array}{ll}
\textrm{minimize}  &   x^TQx+P^Tx+R \\
\textrm{subject to}&   x\in\{0,1\}^n,
\end{array}
\end{equation}

where $Q$ is symmetric. This framework can encode many hard problems, such as max-cut or maximum clique. Note that the continuous version with $x\in[0,1]^n$ need not be convex.

We present a simple SDP-based branch-and-bound solver. This implementation is not intended to compete with specialized tools such as [BiqCrunch](http://lipn.univ-paris13.fr/BiqCrunch/) or other sophisticated algorithms for this problem. The aim is rather to show that one can achieve pretty decent results with under 100 lines of quickly prototyped Python code using the MOSEK Fusion API.

For complete source code and examples see the folder at [GitHub](https://github.com/MOSEK/Tutorials/tree/master/Fusion/BinaryQuadratic-SDP).

# Algorithm

We use the standard semidefinite relaxation of the problem, namely:

\begin{equation}
\begin{array}{ll}
\textrm{minimize}  &   \sum_{ij}Q_{ij}X_{ij} + \sum_i P_ix_i + R \\
\textrm{subject to}&   \left[\begin{array}{cc}X & x\\ x^T & 1\end{array}\right]\succeq 0,\\
                   &   \mathrm{diag}(X) = x.
\end{array}
\end{equation}

The value of the relaxation is a lower bound for the original problem as we see by setting $X=xx^T$ for $x\in\{0,1\}^n$.

The algorithm maintains a lower bound ``lb``, upper bound ``ub`` (objective value of the best integer solution found) and a queue of active nodes. We follow these rules:

* In each iteration we pick the node with smallest objective value of the relaxation.
* Each time a relaxation is solved we round the solution to nearest integers and try to use the result to improve the upper bound.
* We branch on the variable whose value in the relaxation is closest to $\frac12$. Both child nodes, obtained by fixing that variable to $0$ or $1$, are smaller instances of the same problem with easy to compute $Q',P',R'$.

The main node-processing loop follows below.

```python
# The node-processing loop
def solveBB(self):
    while self.active:
        # Pop the node with minimum objective from the queue
        obj, _, Q, P, R, curr, idxmap, pivot = heapq.heappop(self.active)

        self.lb = obj

        # Optimality is proved
        if self.lb >= self.ub - self.gaptol:
            self.active = []
            return

        self.stats()

        if len(P) <= 1: continue

        pidx = idxmap[pivot]

        # Construct and solve relaxations of two child nodes
        # where the pivot variable is assigned 0 or 1
        for val in [0, 1]:
            idxmap0     = np.delete(idxmap, pivot)
            curr0       = np.copy(curr)
            curr0[pidx] = val

            QT = np.delete(Q, pivot, axis=0)
            Q0 = np.delete(QT, pivot, axis=1)
            QZ = QT[:,pivot]
            PT = np.delete(P, pivot)
            P0 = PT + 2*val*QZ
            R0 = val*Q[pivot,pivot] + val*P[pivot] + R

            obj0, pivot0 = self.solveRelaxation(Q0, P0, R0, curr0, idxmap0)

            heapq.heappush(self.active, (obj0, self.relSolved, Q0, P0, R0, curr0, idxmap0, pivot0))
```

The methods which solve the relaxation, find the pivot (branching) index and update the best solution are straightforward. We demonstrate only the fragment which defines the semidefinite relaxation in Fusion API:

```python
# SDP relaxation
# min QX + Px + R
# st  Z = [X, x; x^T, 1] >> 0
def fusionSDP(Q, P, R):
    n = len(P)
    M = Model("fusionSDP")
    Z = M.variable("Z", Domain.inPSDCone(n+1))
    X = Z.slice([0,0], [n,n])
    x = Z.slice([0,n], [n,n+1])
    M.constraint(Expr.sub(X.diag(), x), Domain.equalsTo(0.))
    M.constraint(Z.index(n,n), Domain.equalsTo(1.))

    M.objective(ObjectiveSense.Minimize, Expr.add([Expr.constTerm(R), Expr.dot(P,x), Expr.dot(Q,X)]))

    return M, x
```

The solver is straightforward to use. It suffices to write:

```
solver = BB(Q, P, R)
solver.solve()
solver.summary()
```

```
REL_SOLVED ACTIVE_NDS       OBJ_BOUND        BEST_OBJ
         1          0  -10000000000.0  -34.1829220672
         1          0  -40.9578610377  -34.1829220672
         2          0  -40.9578610377  -34.5354934947
         3          1  -40.0967961795  -34.5354934947
         5          2  -40.0967961795  -37.6612921792
         5          2  -39.9359433001  -37.6612921792
         7          3  -38.9245605048  -37.6612921792
         9          4  -38.4153688863  -37.6612921792
        11          5  -38.1908232037  -37.6612921792
        13          6  -38.1309765476  -37.6612921792
        15          7  -37.9917627647  -37.6612921792
        17          8  -37.8758057302  -37.6612921792
        19          9  -37.6906063328  -37.6612921792
        21         10  -37.6747901176  -37.6612921792
        23         11  -37.6613714044  -37.6612921792
        25          0  -37.6612921492  -37.6612921792
val = -37.6612921792
sol = [0 0 1 0 1 1 0 1 0 1 0 1 1 1 1 0 0 1 0 1 1 0 1 0 1 0 0 0 0 0]
relaxations   = 25
intpntIter    = 243
optimizerTime = 0.194850206375
```

# Example 1. Random data

We consider a random problem

\begin{equation}
\mathrm{minimize}_{x\in\{0,1\}^n}\ x^TQx
\end{equation}

where $Q_{i,j}\sim \mathrm{Normal}(0,1)$ for $i\geq j$. We generate instances with $30\leq n\leq 100$.

Horizontal scale: $n$. Each dot corresponds to an instance. Left: number of relaxations solved. Right: total time spent in the MOSEK optimizer (sec.). Log scale.

# Example 2: BiqMac

We next consider problems of the same form from the library [BiqMac](http://biqmac.aau.at/biqmaclib.html) of binary quadratic problems. We take all instances with $n\leq 100$. Since all the $Q$ matrices in BiqMac are integral, we can use a stronger termination criterion:

```
if np.ceil(self.lb) >= self.ub:
```

Horizontal scale: $n$. Each dot corresponds to an instance. Left: number of relaxations solved. Right: total time spent in the MOSEK optimizer (sec.). Log scale.

# Example 3: Binary least squares

Binary least squares is the problem

\begin{equation}
\mathrm{minimize}_{x\in\{0,1\}^n}\ \|Ax-b\|_2^2 \quad (=x^T(A^TA)x-(2A^Tb)^Tx+b^Tb)
\end{equation}

where $A\in\mathbb{R}^{m\times n}, b\in \mathbb{R}^m$. This is a mixed-integer SOCP, so we can compare our solver with the solutions obtained directly with MOSEK. As suggested in [Park, Boyd 2017](https://web.stanford.edu/~boyd/papers/pdf/int_least_squares.pdf) we generate reasonably hard random data by taking

\begin{equation}
\begin{array}{l}
A\in\mathbb{R}^{m\times n},\quad A_{i,j}\sim \mathrm{Normal}(0,1)\\
b=Ac,\ c\in\mathbb{R}^n,\ c_i\sim\mathrm{Uniform}(0,1)
\end{array}
\end{equation}

For this test we fix $n=50$ and vary $40\leq m\leq 150$.

Horizontal scale: $m$. Each dot corresponds to an instance. Blue: this algorithm. Red: mixed-integer SOCP in MOSEK. Left: number of relaxations solved. Right: total time spent in the MOSEK optimizer (single-thread, sec.). Log scale.

# Small executable example

```python
import branchbound
from branchbound import BB
import numpy as np

n=25
Q1 = np.random.normal(0.0, 1.0, (n,n))
solver = BB((Q1+Q1.transpose())/2, np.random.uniform(-1.0, 3.0, n), 0.0)
solver.solve()
solver.summary()
```

    REL_SOLVED ACTIVE_NDS       OBJ_BOUND        BEST_OBJ
             1          0  -10000000000.0 -33.82841986228613
             1          0 -38.964075740162144 -33.82841986228613
             2          0 -38.964075740162144 -35.385939851022236
             3          1 -38.1921615228864 -35.385939851022236
             5          2 -37.90655091386468 -35.385939851022236
             7          3 -37.4440427985024 -35.385939851022236
             9          4 -37.35261510608118 -35.385939851022236
            11          5 -37.08333365956387 -35.385939851022236
            12          5 -37.08333365956387 -35.682369297628
            13          6 -36.86248188395808 -35.682369297628
            15          7 -36.83311033169883 -35.682369297628
            17          8 -36.66241508003347 -35.682369297628
            19          9 -36.6218827352734 -35.682369297628
            21         10 -36.511788608469644 -35.682369297628
            23         11 -36.341141651093864 -35.682369297628
            25         12 -36.25532343851607 -35.682369297628
            27         13 -36.045055821776245 -35.682369297628
            29         14 -36.002354367603616 -35.682369297628
            31         15 -35.802361179707304 -35.682369297628
            33         16 -35.789284694829355 -35.682369297628
            35         17 -35.74261587243334 -35.682369297628
            37         18 -35.70278561398737 -35.682369297628
            39         19 -35.69573463951267 -35.682369297628
            41          0 -35.68236925395102 -35.682369297628
    val = -35.682369297628
    sol = [0 1 1 1 1 1 1 1 0 0 1 1 0 0 1 1 1 1 0 0 0 1 1 1 0]
    relaxations   = 41
    intpntIter    = 414
    optimizerTime = 0.33767127990722656

This work is licensed under a Creative Commons Attribution 4.0 International License. The **MOSEK** logo and name are trademarks of Mosek ApS. The code is provided as-is. Compatibility with future release of **MOSEK** or the `Fusion API` are not guaranteed. For more information contact our [support](mailto:support@mosek.com).


# COMP 135 day 04: Linear Regression with Polynomial Features and Hyperparameter Selection on a fixed validation set

# Objectives

* Learn how to compute polynomial features
* Try out selecting the polynomial degree on a fixed validation set

* Learn how to use sklearn's built-in Polynomial feature transformer
* Learn how to use sklearn pipelines to compose useful elementary transformations and predictors

# Outline

* [Part 1: Linear Regression on Sine Wave dataset](#part1)
* [Part 2: Trying Polynomial Features + Linear Regression by hand](#part2)
* [Part 3: Selecting degree on Fixed Validation Set](#part3)
* [Part 4: Polynomial Feature transformer](#part4)
* [Part 5: Pipelines](#part5)

```python

```

# Takeaways

* Hyperparameter selection is important to avoid overfitting (and underfitting)
* We cannot use the training set alone to select hyperparameters
* We should use a separate validation set to compute the error we use to select hyperparameters (or do cross-validation)
* Sklearn has many built-in features (Pipelines, feature transformers, etc) that make your life easier if you know how to use them well
* * They all follow a standard interface (aka "API")

```python

```

```python
import numpy as np

# Import the pandas (data management library)
import pandas as pd
```

```python
# import plotting libraries
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', font_scale=1.25, style='whitegrid')
```

```python
import sklearn.linear_model
import sklearn.pipeline
```

# Prepare a simple 'sine-wave' dataset

```python
SEED = 12345
prng = np.random.RandomState(SEED)
```

```python
def true_prediction_function(x):
    return np.sin(2.1 * x)
```

```python
N = 9        # num training examples (to estimate weight parameters)

x_N = np.linspace(-3, 3, N) + 0.09 * prng.randn(N)
y_N = true_prediction_function(x_N) + 0.06 * prng.randn(N)
```

```python
L = 15      # num validation examples (to select hyperparameters)

x_va_L = np.linspace(-3.1, 3.1, L)
y_va_L = true_prediction_function(x_va_L) + 0.06 * prng.randn(L)
```

```python
M = 500      # num testing examples (to show "true" generalization)

x_te_M = np.linspace(-3.5, 3.5, M)
y_te_M = true_prediction_function(x_te_M) + 0.06 * prng.randn(M)
```

```python
plt.plot(x_te_M, y_te_M, 'r.', label='test set')
plt.plot(x_N, y_N, 'bs', markersize=10, label='training set')
plt.plot(x_va_L, y_va_L, 'cd', markersize=10, label='validation set')

plt.legend(loc='upper right');
plt.ylim([-1.5, 1.5]);
```

# Part 1: Linear Regression

First, let's try to fit plain old linear regression to this dataset.

```python
lin_regr = sklearn.linear_model.LinearRegression()
```

### Exercise 1: Can you fit a linear regression and see what its predictions look like?

```python
lin_regr.fit(x_N.reshape(-1, 1), y_N) # TODO fix me. Read error message below. Why does this break?
```

    LinearRegression()

### Exercise 1b: Can you make predictions using the model above on the heldout test set?

```python
yhat_te_M = lin_regr.predict(x_te_M.reshape(-1, 1)) # TODO something like "lin_regr.predict(x_te_M)""
```

```python
plt.plot(x_te_M, yhat_te_M, 'b.-', label='predictions from LR')
plt.plot(x_N, y_N, 'bs', markersize=10, label='training set')

plt.legend(bbox_to_anchor=(1.0, 0.5))
plt.ylim([-1.5, 1.5]);
plt.xlabel('x');
plt.ylabel('y');
```

# Part 2: Polynomial Features

Given our scalar feature $x_i \in \mathbb{R}$, we want to TRANSFORM it to a $F$-degree polynomial feature vector (of size $F+1$)

$$
\begin{align}
\phi_0(x_i) &= [1]
\\
\phi_1(x_i) &= [1 ~~ x_i]
\\
\phi_2(x_i) &= [1 ~~ x_i ~~ x_i^2]
\\
\phi_3(x_i) &= [1 ~~ x_i ~~ x_i^2 ~~ x_i^3] \\
\vdots \\
\phi_F(x_i) &= [1 ~~ x_i ~~ x_i^2 ~~ x_i^3 ~~ \ldots x_i^F]
\end{align}
$$

And then use these transformed features to do linear regression.

That is, we'll learn weights for each of the entries of the vector produced by $\phi_F$

### Exercise 2a: FIX this function so it returns a polynomial feature expansion like above

```python
def make_poly_features(x_N1, degree=0):
    ''' Transform some 1-dim feature array into polynomial features

    Args
    ----
    x_N1 : 2D array, shape (N, 1)
        Input features of a 1d regression problem
    degree : int
        Indicates degree of polynomial

    Returns
    -------
    phi_NG : 2D array, shape (N, G)
        where G = degree + 1
    '''
    assert x_N1.ndim == 2
    N, _ = x_N1.shape

    if degree == 0:
        phi_NG = np.ones((N, 1))
    elif degree >= 1:
        # stack together horizontally a column of all ones and the x array
        phi_NG = np.hstack(
            [np.ones((N, 1))]
            + [(x_N1.copy())**d for d in range(degree)]  # TODO WHAT TO DO HERE??

            )
    return phi_NG
```

Create an array to try things out as input

```python
my_x_N1 = np.asarray([[0.], [1], [2], [3]])
print(my_x_N1)
```

    [[0.]
     [1.]
     [2.]
     [3.]]

```python
make_poly_features(my_x_N1, degree = 0)
```

    array([[1.],
           [1.],
           [1.],
           [1.]])

```python
make_poly_features(my_x_N1, degree = 1)
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.],
           [1., 1.]])

```python
make_poly_features(my_x_N1, degree = 2)
```

    array([[1., 1., 0.],
           [1., 1., 1.],
           [1., 1., 2.],
           [1., 1., 3.]])

```python
make_poly_features(my_x_N1, degree = 4)
```

    array([[ 1.,  1.,  0.,  0.,  0.],
           [ 1.,  1.,  1.,  1.,  1.],
           [ 1.,  1.,  2.,  4.,  8.],
           [ 1.,  1.,  3.,  9., 27.]])

# Part 3: Hyperparameter Selection for Linear Regression with Poly. Features

```python
x_N1 = np.reshape(x_N, (N, 1))
x_va_L1 = np.reshape(x_va_L, (L, 1))
x_te_M1 = np.reshape(x_te_M, (M, 1))
```

### Exercise 3a: Edit code below to properly do the polynomial feature transformation (use your function above)

```python
fig_h, ax_grid = plt.subplots(nrows=2, ncols=4, sharex=True, sharey=True, figsize=(20,6))

for ii, degree in enumerate([1, 2, 3, 4, 5, 6, 7, 8]):
    cur_ax = ax_grid.reshape((8,))[ii]

    # Please Carefully Think through why we set "fit_intercept=False" here:
    # This is not because we don't want bias (i.e. set it to zero).
    # The reason is the bias(intercept) to be fitted is included as degree 0 of the polynomial.
    lin_regr = sklearn.linear_model.LinearRegression(fit_intercept=False)

    phi_NG = x_N1       # TODO fix to transform x with your function "make_poly_features" above!
    phi_va_LG = x_va_L1 # TODO fix to transform validation features
    phi_te_MG = x_te_M1 # TODO fix to transform test features
    phi_NG = make_poly_features(phi_NG, degree = ii)
    phi_va_LG = make_poly_features(phi_va_LG, degree = ii)
    phi_te_MG = make_poly_features(phi_te_MG, degree = ii)

    # Train the model
    lin_regr.fit(phi_NG, y_N)

    # Make predictions
    yhat_N = lin_regr.predict(phi_NG)
    yhat_va_L = lin_regr.predict(phi_va_LG)
    yhat_te_M = lin_regr.predict(phi_te_MG)

    # Evaluate the ERROR on training set and test set
    tr_err = sklearn.metrics.mean_squared_error(y_N, yhat_N)
    va_err = sklearn.metrics.mean_squared_error(y_va_L, yhat_va_L)
    te_err = sklearn.metrics.mean_squared_error(y_te_M, yhat_te_M)

    print("degree %d | train error %10.3f | valid error %10.3f | test error %10.3f" % (
        degree, tr_err, va_err, te_err))

    # Plot the predictions on current axis
    cur_ax.plot(x_te_M1, yhat_te_M, 'b-');
    cur_ax.plot(x_N, y_N, 'bs');

    cur_ax.set_title('degree %d' % degree)
```

### Exercise 3b: Which degree offers the lowest error on the validation dataset?

```python
# TODO fill
```

### Exercise 3c: Which degree offers the lowest error on the test dataset?

WARNING: if you really used this to pick your degree you could not later use the test set to fairly report your model's ability to generalize. Once you use the test set to train or select, you can't reuse it.

```python
# TODO fill
```

# Part 4: Using sklearn's built in Polynomial feature transform

You can read up here:

<https://scikit-learn.org/stable/modules/preprocessing.html#generating-polynomial-features>

Or dive in!

```python
from sklearn.preprocessing import PolynomialFeatures

my_x_N1 = np.arange(3).reshape(3, 1)
print("Original Features my_x_N1")
print(my_x_N1)
```

```python
poly = PolynomialFeatures(2)
phi_NG = poly.fit_transform(my_x_N1) # call 'fit_transform' to perform the transformation

print("Transformed Features phi_NG using 2-degree polynomial")
print(phi_NG)
```

```python
poly = PolynomialFeatures(5)
phi_NG = poly.fit_transform(my_x_N1)

print("Transformed Features phi_NG using 5-degree polynomial")
print(phi_NG)
```

Now try with more than one feature (e.g. each x vector is 2-dim.)

```python
my_x_N2 = np.asarray([[0, 3], [1, 4], [2, 5]])
print("Original Features my_x_N2")
print(my_x_N2)
```

```python
poly = PolynomialFeatures(1)
phi_NG = poly.fit_transform(my_x_N2)

print("Transformed Features phi_NG using 1-degree polynomial")
print(phi_NG)
```

```python
poly = PolynomialFeatures(2)
phi_NG = poly.fit_transform(my_x_N2)

print("Transformed Features phi_NG using 2-degree polynomial")
print(phi_NG)
```

Conveniently, we can ask the feature transformer how it came up with these features

```python
poly.get_feature_names(['a', 'b'])
```

Now try 3-rd degree

```python
poly = PolynomialFeatures(3)
phi_NG = poly.fit_transform(my_x_N2)

print("Transformed Features phi_NG using 3-degree polynomial")
print(phi_NG)
```

```python
poly.get_feature_names(['a', 'b'])
```

# Part 5: Using sklearn's built in Pipeline system

Often, we want to compose together some feature transformations and some predictor (like Linear Regression)

Sklearn lets us do that easily with a Pipeline

Read up here:

<https://scikit-learn.org/stable/modules/compose.html#pipeline>

The key advantages of pipelines here are

**Convenience**

A pipeline is one object that handles everything

* every time we call fit, we first transform the raw features "x" into the transformed ones "phi", and then we do linear regression fitting using those transformed features
* every time we call predict, we first transform the raw features "x" into the transformed ones "phi", and then use linear regression to predict

Otherwise, we'd have to call `make_poly_features' so many times.

**Correctness**

Pipelines help us avoid having data leaks (e.g. have test data impact the transformation we learn). More about this later in the course.

To construct a pipeline, we just provide a list of steps.

Each STEP is just a tuple:
* first entry is the string name of the step (can be anything you want)
* second entry is the constructed sklearn estimator that will implement that step

```python
pipeline = sklearn.pipeline.Pipeline([
    ("step1", sklearn.preprocessing.PolynomialFeatures()),
    ("step2", sklearn.linear_model.LinearRegression()),
    ])
```

```python
pipeline
```

Now, we can use a pipeline like so:

```python
fig_h, ax_grid = plt.subplots(nrows=3, ncols=4, sharex=True, sharey=True, figsize=(20,9))

for ii, degree in enumerate([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]):

    cur_ax = ax_grid.reshape((12,))[ii]

    pipeline = sklearn.pipeline.Pipeline([
        ("step1", sklearn.preprocessing.PolynomialFeatures(degree)), # create custom Poly featurizer
        ("step2", sklearn.linear_model.LinearRegression()),
        ])

    # Train the model
    pipeline.fit(x_N1, y_N)                # Can use raw x here, unlike above

    # Make predictions
    yhat_N = pipeline.predict(x_N1)        # Can use raw x here too!
    yhat_va_L = pipeline.predict(x_va_L1)
    yhat_te_M = pipeline.predict(x_te_M1)

    # Evaluate the ERROR on training set and test set
    tr_err = sklearn.metrics.mean_squared_error(y_N, yhat_N)
    va_err = sklearn.metrics.mean_squared_error(y_va_L, yhat_va_L)
    te_err = sklearn.metrics.mean_squared_error(y_te_M, yhat_te_M)

    print("degree %3d | train error % 10.3f | valid error %10.3f | test error % 10.3f" % (
        degree, tr_err, va_err, te_err))

    # Plot the predictions
    cur_ax.plot(x_te_M1, y_te_M, 'r.');
    cur_ax.plot(x_te_M1, yhat_te_M, 'b-');
    cur_ax.plot(x_N, y_N, 'bs');
    cur_ax.set_title('degree %d' % degree);
    cur_ax.set_ylim([-2, 2]);
```

### Exercise 5a: Which degree offers would you select based on the validation set?

### Discussion 5b: What do you notice about the error INSIDE the bounds of observed training data versus OUTSIDE the bounds?

```python

```


# MNIST handwritten digits classification with naive Bayes

In this notebook, we'll use [naive Bayes classifiers](https://scikit-learn.org/stable/modules/naive_bayes.html) to classify MNIST digits using scikit-learn (version 0.20 or later required).

First, the needed imports.

```python
%matplotlib inline

from pml_utils import get_mnist, show_failures

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import datasets, __version__
from sklearn.naive_bayes import GaussianNB, BernoulliNB
from sklearn.metrics import accuracy_score

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

from distutils.version import LooseVersion as LV
assert(LV(__version__) >= LV("0.20")), "Version >= 0.20 of sklearn is required."
```

Then we load the MNIST data. First time we need to download the data, which can take a while.

```python
X_train, y_train, X_test, y_test = get_mnist('MNIST')

print('MNIST data loaded: train:',len(X_train),'test:',len(X_test))
print('X_train:', X_train.shape)
print('y_train:', y_train.shape)
print('X_test', X_test.shape)
print('y_test', y_test.shape)
```

The training data (`X_train`) is a matrix of size (60000, 784), i.e. it consists of 60000 digits expressed as 784 sized vectors (28x28 images flattened to 1D). `y_train` is a 60000-dimensional vector containing the correct classes ("0", "1", ..., "9") for each training digit.

Let's take a closer look. Here are the first 10 training digits:

```python
pltsize=1
plt.figure(figsize=(10*pltsize, pltsize))

for i in range(10):
    plt.subplot(1,10,i+1)
    plt.axis('off')
    plt.imshow(X_train[i,:].reshape(28, 28), cmap="gray")
    plt.title('Class: '+y_train[i])
```

## Naive Bayes classifiers

Naive Bayes classifiers are a family of simple classifiers based on applying Bayes' theorem. The classifiers are called "naive" as we make a strong assumption that the features are conditionally independent given the value of the class variable. While this assumption is not usually true, a naive Bayes classifier may in practice work reasonably well. Naive Bayes classifiers are also simple and fast compared to many more sophisticated methods.

The classification rule for naive Bayes is
\begin{equation}
\hat{y} = \arg\max_yP(y)\prod_{i=1}^nP(x_i|y)
\end{equation}
where $P(y)$ is the prior probability of class $y$ and $P(x_i|y)$ is the class-conditional likelihood of feature $i$.

## Gaussian naive Bayes

In Gaussian naive Bayes, the likelihood of the features is assumed to be Gaussian
\begin{equation}
P(x_i|y) = \mathcal{N}(x_i\,|\,\mu_{iy},\sigma_{iy}^2)
\end{equation}
where $\mu_{iy}$ and $\sigma_{iy}^2$ are the mean and variance, respectively, of feature $i$ in objects of class $y$.

```python
mu = 192.
sigma = 32.
x = np.arange(255.)
plt.plot(x, 1/(sigma * np.sqrt(2 * np.pi)) * np.exp( - (x - mu)**2 / (2 * sigma**2)),
         lw=3)
plt.xticks([0,127,255])
plt.title('Gaussian distribution with $\mu={}$ and $\sigma={}$'.format(mu, sigma));
```

The prior probabilities $P(y)$ are learned from training data by default.

### Learning

Training a naive Bayes classifier is fast:

```python
%%time

clf_gnb = GaussianNB()
clf_gnb.fit(X_train, y_train)
```

We can take a look at the mean and variance of features for each class.

```python
pltsize=1
plt.figure(figsize=(10*pltsize, pltsize))
plt.suptitle('Mean of each feature', y=1.3)

for i in range(10):
    plt.subplot(1,10,i+1)
    plt.axis('off')
    plt.imshow(clf_gnb.theta_[i,:].reshape(28, 28), cmap="gray")
    plt.title('Class: '+str(i))

plt.figure(figsize=(10*pltsize, pltsize))
plt.suptitle('Variance of each feature', y=1.1)
for i in range(10):
    plt.subplot(1,10,i+1)
    plt.axis('off')
    plt.imshow(clf_gnb.sigma_[i,:].reshape(28, 28), cmap="gray")
```

### Inference

Evaluating a naive Bayes classifier is also fast:

```python
%%time

predictions = clf_gnb.predict(X_test)
```

The accuracy of the classifier:

```python
print('Predicted', len(predictions), 'digits with accuracy:', accuracy_score(y_test, predictions))
```

We can also inspect the results in more detail. Let's use the `show_failures()` helper function to show the wrongly classified test digits.

```python
show_failures(predictions, y_test, X_test)
```

```python
show_failures(predictions, y_test, X_test, trueclass='5')
```

We can observe that the classifier makes rather easy mistakes.

## Bernoulli naive Bayes

Gaussian naive Bayes assumes that the features are normally distributed, which is not a good assumption for the MNIST digits. Let's therefore use a second approach and model each feature as a binary variable ("black" or "white"). A suitable distribution in this case is the Bernoulli
\begin{equation}
P(x_i|y) = \mathrm{Ber}(x_i\,|\,\theta_{iy})
\end{equation}
where $\theta_{iy}$ is the probability that feature $i$ is "white" in objects of class $y$.

```python
theta = 0.4
plt.bar([0, 255], [1-theta, theta], width=16.)
plt.xticks([0,127,255])
plt.title('Bernoulli distribution with $\\theta={}$'.format(theta));
```

### Learning

Bernoulli naive Bayes assumes binary data, so we'll binarize the digits with a threshold in the middle.

```python
%%time

clf_bnb = BernoulliNB(binarize=128.)
clf_bnb.fit(X_train, y_train)
```

We can take a look at the probabilities of features for each class.

```python
pltsize=1
plt.figure(figsize=(10*pltsize, pltsize))
plt.suptitle('Probability of each feature', y=1.3)

for i in range(10):
    plt.subplot(1,10,i+1)
    plt.axis('off')
    plt.imshow(np.exp(clf_bnb.feature_log_prob_[i,:]).reshape(28, 28), cmap="gray")
    plt.title('Class: '+str(i))
```

### Inference

```python
%%time

predictions = clf_bnb.predict(X_test)
```

```python
print('Predicted', len(predictions), 'digits with accuracy:', accuracy_score(y_test, predictions))
```

```python
show_failures(predictions, y_test, X_test)
```

```python
show_failures(predictions, y_test, X_test, trueclass='5')
```

```python

```


# Optimising functions

Now for something a bit different.
PyTorch is a tensor processing library and whilst it has a focus on neural networks, it can also be used for more standard funciton optimisation.
In this example we will use torchbearer to minimise a simple function.

**Note**: The easiest way to use this tutorial is as a colab notebook, which allows you to dive in with no setup. We recommend you enable a free GPU with

> **Runtime**   →   **Change runtime type**   →   **Hardware Accelerator: GPU**

## Install Torchbearer

First we install torchbearer if needed.

```python
try:
    import torchbearer
except:
    !pip install -q torchbearer
    import torchbearer

print(torchbearer.__version__)
```

    0.3.2

## A Simple Function

For this task, we are going to try to optimise the following:

\begin{equation}
min_{\textbf{x}} (\textbf{x}[0] - 5)^2 + \textbf{x}[1]^2 + (\textbf{x}[2] - 1)^2\; ,
\end{equation}
which has a minimum at $\textbf{x}=[5, 0, 1]$.

### The Model

First we will need to create something that looks very similar to a neural network model - but with the purpose of minimising our function.
We store the current estimates for the minimum as parameters in the model (so PyTorch optimisers can find and optimise them) and we return the function value in the forward method.

```python
import torch
import torch.nn as nn

ESTIMATE = torchbearer.state_key('est')

class Net(nn.Module):
    def __init__(self, x):
        super().__init__()
        self.pars = nn.Parameter(x)

    def f(self):
        """
        function to be minimised:
        f(x) = (x[0]-5)^2 + x[1]^2 + (x[2]-1)^2
        Solution:
        x = [5,0,1]
        """
        out = torch.zeros_like(self.pars)
        out[0] = self.pars[0]-5
        out[1] = self.pars[1]
        out[2] = self.pars[2]-1
        return torch.sum(out**2)

    def forward(self, _, state):
        state[ESTIMATE] = self.pars.detach().unsqueeze(1)
        return self.f()
```

### The Loss

For function minimisation we have an analogue to neural network losses - we minimise the value of the function under the current estimates of the minimum.
Note that as we are using a base loss, torchbearer passes this the network output and the "label" (which is of no use here).

```python
def loss(y_pred, y_true):
    return y_pred
```

### Optimising

We need two more things before we can start optimising with torchbearer.
We need our initial guess - which we've set to [2.0, 1.0, 10.0] and we need to tell torchbearer how "long" an epoch is - I.e. how many optimisation steps we want for each epoch.
For our simple function, we can complete the optimisation in a single epoch, but for more complex optimisations we might want to take multiple epochs and include tensorboard logging and perhaps learning rate annealing to find a final solution.

```python
p = torch.tensor([2.0, 1.0, 10.0])
training_steps = 5000
```

The learning rate chosen for this example is very low and we could get convergence much faster with a larger rate, however this allows us to view convergence in real time.
We define the model and optimiser in the standard way.

```python
model = Net(p)
optim = torch.optim.SGD(model.parameters(), lr=0.001)
```

Finally we start the optimising on the GPU and print the final minimum estimate.

```python
device = 'cuda' if torch.cuda.is_available() else 'cpu'
trial = torchbearer.Trial(model, optim, loss, [torchbearer.metrics.running_mean(ESTIMATE, dim=1), 'loss'])
trial.for_train_steps(training_steps).to(device)
trial.run()
print(list(model.parameters())[0].data)
```

    HBox(children=(IntProgress(value=0, description='0/1(t)', max=5000), HTML(value='')))

    tensor([4.9999e+00, 4.4948e-05, 1.0004e+00], device='cuda:0')

## Visualising Himmelblau's Function

We'll now have a go at a more complex example, which we also visualise, with multiple optima; [Himmelblau's function](https://en.wikipedia.org/wiki/Himmelblau%27s_function). This is defined as:

\begin{equation}
f(x, y) = (x^2 + y - 11)^2 + (x + y^2 - 7)^2\; ,
\end{equation}
and has minima at
\begin{equation}
f(3, 2) = f(-2.805118, 3.131312) = f(-3.779310, -3.283186) = f(3.584428, -1.848126) = 0\; .
\end{equation}

Let's first define the function

```python
himm = lambda x, y: (x ** 2 + y - 11) ** 2 + (x + y ** 2 - 7) ** 2
```

and plot its surface

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.colors import LogNorm

xmin, xmax, xstep = -5, 5, .2
ymin, ymax, ystep = -5, 5, .2
x, y = np.meshgrid(np.arange(xmin, xmax + xstep, xstep), np.arange(ymin, ymax + ystep, ystep))
z = himm(x, y)

fig = plt.figure(figsize=(8, 5))
ax = plt.axes(projection='3d', elev=50, azim=-50)
ax.plot_surface(x, y, z, norm=LogNorm(), rstride=1, cstride=1,
                edgecolor='none', alpha=.8, cmap=plt.cm.jet)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.set_zlabel('$z$')

ax.set_xlim((xmin, xmax))
ax.set_ylim((ymin, ymax))

plt.show()
```

That looks about right, now we'd like to view the optimisation path of a few different optimisers.

### The Model

Since we already have the function definition, we can define our model simply:

```python
import torch
import torch.nn as nn

ESTIMATE = torchbearer.state_key('est')

class Net(nn.Module):
    def __init__(self, x):
        super().__init__()
        self.pars = nn.Parameter(x)

    def forward(self, _, state):
        state[ESTIMATE] = self.pars.detach()
        return himm(self.pars[0], self.pars[1])
```

### Visualising

We now wish to create a visualisation. There are a few ways to do this, we'll use a couple of callback functions, combined in a [`CallbackList`](https://torchbearer.readthedocs.io/en/latest/code/callbacks.html#torchbearer.callbacks.callbacks.CallbackList).

```python
import torchbearer
from torchbearer import callbacks

def make_callback(ax, color, label):
    xs = []
    ys = []

    @callbacks.on_step_training
    def log_estimate(state):
        est = state[torchbearer.METRICS][ESTIMATE].cpu().numpy()
        xs.append(est[0])
        ys.append(est[1])

    @callbacks.on_end
    def plot_history(state):
        ax.plot(xs, ys, color=color, label=label, linewidth=2)

    return callbacks.CallbackList([log_estimate, plot_history])
```

This callback will allow us to create an axis (2d this time) and subsequently plot the optimisation path on it for a series of optimisers.

### Optimising

Let's see how it looks for a few different optimisers from a range of starting points

```python
import numpy as np
import torchbearer
device = 'cuda' if torch.cuda.is_available() else 'cpu'

xmin, xmax, xstep = -5, 5, .2
ymin, ymax, ystep = -5, 5, .2
x, y = np.meshgrid(np.arange(xmin, xmax + xstep, xstep), np.arange(ymin, ymax + ystep, ystep))
z = himm(x, y)

fig, ax = plt.subplots(figsize=(8, 8))
ax.contourf(x, y, z, levels=np.logspace(0, 5, 35), norm=LogNorm(), cmap=plt.cm.gray)

def make_plots(p):
    def run(optimiser, color, label):
        model = Net(p)
        optim = optimiser(model.parameters())
        training_steps = 50
        trial = torchbearer.Trial(model, optim, loss, [ESTIMATE], callbacks=[make_callback(ax, color, label)], verbose=-1)
        trial.for_train_steps(training_steps).to(device)
        trial.run()

    cmap = plt.cm.viridis

    run(lambda param: torch.optim.SGD(param, lr=0.01), cmap(0), 'SGD')
    run(lambda param: torch.optim.Adam(param, lr=0.1), cmap(0.25), 'Adam')
    run(lambda param: torch.optim.Adam(param, lr=0.1, amsgrad=True), cmap(0.5), 'AMSGrad')
    run(lambda param: torch.optim.RMSprop(param, lr=0.1), cmap(0.75), 'RMSProp')
    run(lambda param: torch.optim.Adamax(param, lr=0.1), cmap(1.0), 'Adamax')

make_plots(torch.tensor([0.0, 0.0]))
plt.legend()

for x in np.arange(-1.0, 1.0, 0.5):
    for y in np.arange(-2.0, 1.0, 0.5):
        make_plots(torch.tensor([x, y]))

ax.set_xlabel('$x$')
ax.set_ylabel('$y$')

ax.set_xlim((xmin, xmax))
ax.set_ylim((ymin, ymax))

fig.savefig('optimisers.png', bbox_inches='tight')
```

```python

```


# Classical Iterative Eigensolvers

## Reduction to Hessenberg or Tridiagonal Form

### A Bad Idea

To compute the Schur Factorization $A=QTQ^*$ we would like to apply a unitary similarity transformation to $A$ to introduce zeros below the diagonal.

One might attempt using Householder reflectors $Q_1^*$ on the left of $A$:

$$
\begin{bmatrix}\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\end{bmatrix} \quad \stackrel{Q_1^*\cdot}{\longrightarrow} \quad \begin{bmatrix} \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\0 & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\0 & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\0 & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\0 & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\end{bmatrix}
$$

Unfortunately, one must also multiply on the right by $Q_1$:

$$
\begin{bmatrix} \times & \times & \times & \times &\times\\0 &  \times & \times & \times & \times\\0 &  \times & \times & \times & \times\\0 &  \times & \times & \times & \times\\0 &  \times & \times & \times & \times\end{bmatrix}\quad \stackrel{Q_1^*\cdot}{\longrightarrow} \quad \begin{bmatrix}\boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\\boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\\boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\\boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\\boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\end{bmatrix}
$$

Giving us a matrix no closer to being triangular.

We have already mentioned that this approach was doomed to fail by Galois' Solvability Theorem.

### A Good Idea

Instead we settle for slightly less: the _Hessenberg Form_.

$$
\begin{bmatrix}\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\\\times & \times & \times & \times & \times\end{bmatrix} \quad \stackrel{Q_1^*\cdot}{\longrightarrow} \quad \begin{bmatrix} {\times} & {\times} & {\times} & {\times} &{\times}\\\boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\0 & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\0 & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\0 & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\end{bmatrix}\quad \stackrel{\cdot Q_1}{\longrightarrow} \quad \begin{bmatrix} {\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ {\times} & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\end{bmatrix}
$$

$$
\begin{bmatrix} {\times} & {\times} & {\times} & {\times} &{\times}\\ {\times} & {\times} & {\times} & {\times} &{\times}\\ & {\times} & {\times} & {\times} &{\times}\\ & {\times} & {\times} & {\times} &{\times}\\ & {\times} & {\times} & {\times} &{\times}\end{bmatrix} \quad \stackrel{Q_2^*\cdot}{\longrightarrow} \quad \begin{bmatrix} {\times} & {\times} & {\times} & {\times} &{\times}\\ {\times} & {\times} & {\times} & {\times} &{\times}\\ & \boldsymbol{\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ & 0 & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ & 0 & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\end{bmatrix}\quad \stackrel{\cdot Q_2}{\longrightarrow} \quad \begin{bmatrix} {\times} & {\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ {\times} & {\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ & {\times} & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ &  & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\\ &  & \boldsymbol{\times} & \boldsymbol{\times} &\boldsymbol{\times}\end{bmatrix}
$$

Repeating this $m-2$ times, we have a product in Hessenberg form:

$$Q_{m-2}^*\cdots Q_1^* A Q_1\cdots Q_{m-2} = H \sim \begin{bmatrix} {\times} & {\times} & {\times} & {\times} &{\times}\\ {\times} & {\times} & {\times} & {\times} &{\times}\\ & {\times} & {\times} & {\times} &{\times}\\ &  & {\times} & {\times} &{\times}\\ &  &  & {\times} &{\times}\end{bmatrix}$$

```julia
m = 6
A = 10*rand(m,m)
round(A)
```

    6×6 Array{Float64,2}:
     6.0  9.0  6.0   9.0  4.0  5.0
     7.0  8.0  5.0  10.0  2.0  4.0
     4.0  6.0  4.0   5.0  1.0  4.0
     5.0  4.0  6.0   2.0  7.0  0.0
     4.0  7.0  7.0   1.0  9.0  8.0
     3.0  5.0  0.0   6.0  2.0  0.0

```julia
for k = 1:m-2
    x = A[k+1:m,k]
    v = [x[1]+sign(x[1])*norm(x); x[2:end]]
    v = v/norm(v)
    A[k+1:m, k:m] = A[k+1:m, k:m]-2v*(v'*A[k+1:m, k:m])
    A[1:m, k+1:m] = A[1:m,k+1:m]-2*(A[1:m,k+1:m]*v)*v'
end
round(A)
```

    6×6 Array{Float64,2}:
       6.0  -15.0  2.0  -1.0  -0.0   1.0
     -11.0   23.0  0.0   1.0   0.0  -2.0
       0.0    6.0  4.0  -4.0   4.0  -1.0
       0.0   -0.0  8.0  -3.0  -2.0  -1.0
       0.0    0.0  0.0   3.0  -1.0   3.0
       0.0    0.0  0.0   0.0   1.0  -1.0

```julia
F = hessfact(A) # Returns a dictionary with both Q and H
round(F[:H])
```

    6×6 Array{Float64,2}:
      6.0  15.0  -2.0   1.0   0.0   1.0
     11.0  23.0   0.0   1.0   0.0   2.0
      0.0   6.0   4.0  -4.0   4.0   1.0
      0.0   0.0   8.0  -3.0  -2.0   1.0
      0.0   0.0   0.0   3.0  -1.0  -3.0
      0.0   0.0   0.0   0.0  -1.0  -1.0

Here the matrix:
$$Q= \prod_{k=1}^{m-2} Q_k$$

is never actually formed explicitly.

Instead, the vectors $\mathbf{v}_k$ are saved, and can be used, if necessary, to reconstitute $Q$.

### Operation Count

Rule of thumb: unitary operations cost 4 times as many flops.

The work in the above algorithm is dominated by the final two updates of submatrices of $A$.

The first loop applies a Householder reflector on the left of the matrix.  The $k$-th such reflector operators on the last $m-k$ rows.   Since at the time the reflector is applied, these rows have zeros in the first $k-1$ columns, arithmetic need only be done to the remaining $m-k+1$ entries.

The second loop applies a Householder reflector on the right of the matrix.  At the $k$-th step the reflector operators on the last $m-k$ rows.   However, there are no zero entries this time, for a total of $m(m-k)$ for each $k$.

Summing and considering only the dominant term, we find:
$$
\frac{4}{3} m^3  + 2 m^3 = \frac{10}{3} m^3.
$$

> ** THEOREM. ** The work for Hessenberg reduction using the above algorithm is: $$\sim \frac{10}{3} m^3 \text{ flops}.$$

### The Hermitian Case: Reduction to Tridiagonal Form

If $A=A^*$, the algorithm reduces $A$ to _Tridiagonal form_:

$$Q_{m-2}^*\cdots Q_1^* A Q_1\cdots Q_{m-2} = H \sim \begin{bmatrix} {\times} & {\times} &  & & \\ {\times} & {\times} & {\times} &  &\\ & {\times} & {\times} & {\times} &\\ &  & {\times} & {\times} &{\times}\\ &  &  & {\times} &{\times}\end{bmatrix}$$

Since zeros are now introduced in rows and columns, there are additional arithmetic savings.  In particular, we sum to find:
$$
\frac{4}{3} m^3 + \frac{4}{3}m^3 = \frac{8}{3}m^3.
$$

However, this assumes only the sparsity of the upper corner, we can use the symmetry property, i.e. the matrix is Hermitian, which saves us half the work.

> ** THEOREM. ** The work for Hessenberg reduction using the above algorithm on a Hermitian matrix is: $$\sim \frac{4}{3} m^3 \text{ flops}.$$

### Stability

Like the Householder algorithm for QR factorization, this algorithm is backwards stable.  The argument is similar to the QR factorization via Householder orthogonal triangularization.

Let $\tilde H$ be the computed Hessenberg matrix, and let $\tilde Q$ be, as before, the exactly unitary matrix constructed via the computed reflection vectors $\tilde v_k$.

> ** THEOREM. ** Let the Hessenberg reduction $A=QHQ^*$ of a matrix $A\in\mathbb{C}^{m\times m}$ be computed by the above algorithm on a computer satisfying our axioms for $\epsilon_{\text{machine}}$, and let $\tilde Q$ and $\tilde H$ be defined as indicated above.  Then we have: $$\tilde Q\tilde H\tilde Q^* = A + \delta A, \quad \frac{\|\delta A\|}{\|A\|} = O(\epsilon_{\text{machine}})$$ for some $\delta A\in\mathbb{C}^{m\times m}$.

## Rayleigh Quotient, Inverse Iteration

### Restriction to Real Symmetric Matrices

For the moment, we restrict to the case that $A=A^\top$ is real symmetric, and that $\|\cdot \| = \|\cdot \|_2$ is the $2$-norm.

We let: $$A=A^\top \in \mathbb{R}^{m\times m}, \quad \mathbf{x} \in \mathbb{R}^m, \quad \mathbf{x}^* = \mathbf{x}^\top, \quad \|\mathbf{x}\|=\sqrt{\mathbf{x}^\top \mathbf{x}}.$$

### Rayleigh Quotient

Given $\mathbf{x}$, which scalar $\alpha$ behaves most like an eigenvalue for $\mathbf{x}$, in the sense that: $$\|A\mathbf{x}-\alpha\mathbf{x}\|_2$$ is minimized?

This is an $m\times 1$ least squares problem of the form:
$$
\mathbf{x}\alpha \approx A\mathbf{x}.
$$
Solving this via normal equations, we obtain $\alpha = \frac{\mathbf{x}^\top A \mathbf{x}}{\mathbf{x}^\top \mathbf{x}}.$

We define the _Rayleigh quotient_ of a vector $\mathbf{x}\in\mathbb{R}^m$ to be:
$$
r(\mathbf{x} ) = \frac{\mathbf{x}^\top A \mathbf{x}}{\mathbf{x}^\top \mathbf{x}}.
$$

Notice that if $\mathbf{x}$ is an eigenvector, then $r(x)$ is its eigenvector.

Consider this as a function: $$r: \mathbb{R}^m \to \mathbb{R}.$$

We are interested in the local behavior of $r$ when $\mathbf{x}$ is close to an eigenvector.   We compute the partial derivatives.

\begin{align}
\frac{\partial r}{\partial x_j} &= \frac{\frac{\partial}{\partial x_j} \left(\mathbf{x}^\top A \mathbf{x}\right)}{\mathbf{x}^\top\mathbf{x}} -\frac{\left(\mathbf{x}^\top A \mathbf{x}\right) \frac{\partial}{\partial x_j} \left(\mathbf{x}^\top \mathbf{x}\right)}{\left(\mathbf{x}^\top \mathbf{x}\right)^2}\\ &= \frac{2\left(A\mathbf{x}\right)_j}{\mathbf{x}^\top\mathbf{x}} - \frac{\left(\mathbf{x}^\top A \mathbf{x}\right) 2 x_j}{\left(\mathbf{x}^\top\mathbf{x}\right)^2}\\
&= \frac{2}{\mathbf{x}^\top\mathbf{x}} \left(A\mathbf{x} - r(\mathbf{x})\mathbf{x}\right)_j.
\end{align}

Compiling these parital into an $m$-vector, i.e. $\nabla r$, we see:

$$\nabla r = \frac{2}{\mathbf{x}^\top \mathbf{x}} \left(A\mathbf{x} - r(\mathbf{x})\mathbf{x}\right).$$

> ** THEOREM. ** $\nabla r (\mathbf{x}) = \mathbf{0}$ with $\mathbf{x}\ne \mathbf{0}$ if and only if $\mathbf{x}$ is an eigenvector with eigenvalue $r(\mathbf{x})$.

Thus the eigenvectors are precisely the critical points of the Rayleigh quotient.  In fact, since $r$ is scale independent, we may restrict the domain of $r$ to the unit $n$-sphere.

```julia
using PyPlot
n = 100
u = linspace(0,2*π,n);
v = linspace(0,π,n);

x = cos(u) * sin(v)';
y = sin(u) * sin(v)';
z = ones(n) * cos(v)';

# The rstride and cstride arguments default to 10
surf(x,y,z, rstride=4, cstride=4)
```

Because of this we see that if $\mathbf{q}$ is an eigenvector of $A$, then since $\nabla r(\mathbf{q})=\mathbf{0}$, and $r$ is smooth, we see that:

$$
r(\mathbf{x})-r(\mathbf{q}) = O\left(\|\mathbf{x}-\mathbf{q}\|^2\right),\quad \text{as } \mathbf{x}\to \mathbf{q}.
$$

Thus the Rayleigh quotient is a _quadratically accurate_ estimate of an eigenvalue!

```julia
A = [2 1 1;
     1 3 1;
     1 1 4]
(λ,X) = eig(A);
@show λ;
```

    λ = [1.32487,2.46081,5.21432]

```julia
X
```

    3×3 Array{Float64,2}:
      0.88765   -0.233192  -0.397113
     -0.427132  -0.739239  -0.520657
     -0.172148   0.631781  -0.755789

The max of  $r(\mathbf{v})$ is the third eigenvector of $A$ as computed above, and close to $(0.4,0.5,0.75)$.

```julia
x = X[:,3];
v = [0.4;0.5;0.75];
evec_err = norm(-x-v)
```

    0.02164672504409101

```julia
eval_est = v'*A*v/(v'*v)
eval_err = norm(eval_est-λ[3])
```

    0.0009521341230360747

### Power Iteration

Now that we can turn an eigenvector estimate into an (even better, in the hermitian case) eigenvalue estimate, we turn to finding such eigenvectors. Clearly, if $A=XDX^{-1}$ and $k$ is a positive integer, then  $A^k=XD^kX^{-1}$. That is,  $A$ and $A^k$ share the same eigenvectors, but the eigenvalues of  $A^k$ are raised to the  $k$-th power.

Note that if there is an eigenvalue  $\lambda_1$  such that  $|\lambda_1|>|\lambda_j|$ for all  $j>1$, the ratio

$$\left|\frac{\lambda_j}{\lambda_1}\right|^k \to 0, \quad \text{as } k\to \infty,$$

and in that case $A^k \mathbf{v}$ is dominated by the eigenvector $\mathbf{q}_1$ that belongs with $\lambda_1$, for practically any vector $\mathbf{v}$. This leads us to the **power iteration**.

```julia
v = rand(3);
evec_err = [];
for k = 1:200
    v = v/norm(v);
    evec_err = [evec_err;min(norm(v-x),norm(v+x))];
    v = A*v;
end
using PyPlot
semilogy(evec_err,".-");
```

Since $A$ is symmetric, we should expect around 12 accurate digits in an eigenvalue estimate derived from a Rayleigh quotient with our final $\mathbf{v}$.

```julia
abs( v'*A*v/(v'*v) - λ[3] )
```

    1×1 Array{Float64,2}:
     3.90799e-14

### Inverse Iteration

If $\lambda_J$ is an eigenvalue of $A$, then  $(\lambda_J-\mu)^{-1}$ is an eigenvalue of $(A-\mu I)^{-1}$, with the _same_ eigenvector.

If $\mu$ is closer to $\lambda_J$ than to any other $\lambda_j$, a power iteration on $(A-\mu I)^{-1}$ should converge very quickly — because $(\lambda_J-\mu)^{-1}$ will be much larger than any other eigenvalue.

This amounts to repeatedly solving a linear system with the matrix $(A-\mu I)$.

```julia
for μ = [4 4.5 5.1]
    LU = lufact(A-μ*eye(3));
    v = rand(3,1);
    for k = 1:20
        v = v/norm(v);
        evec_err[k] = min(norm(v-x),norm(v+x));
        v = LU\v;
    end
    semilogy(evec_err,".-")
end
```

> ** THEOREM. ** Suppose $\lambda_J$ is the closest eigenvalue to $\mu$ and $\lambda_K$ is the second closest, that is $|\mu - \lambda_J|<|\mu-\lambda_K|\leq |\mu-\lambda_j|$ for each $j\ne J$.  Furthermore, suppose $\mathbf{q}_J^\top \mathbf{v}^{(0)} \ne 0$.  Then the iterates of the above algorithm satisfy:
$$\|\mathbf{v}^{(k)} - (\pm \mathbf{q}_J)\| = O \left(\left|\frac{\mu-\lambda_J}{\mu-\lambda_K}\right|^k\right), \quad \|\lambda^{(k)} - \lambda_J | = O\left(\left|\frac{\mu-\lambda_J}{\mu-\lambda_K}\right|^{2k}\right)$$ as $k\to \infty$, where the $\pm$ sign has the same meaning as before.

### Rayleigh Quotient Iteration

The coup de grâce: a better shift means a better eigenvector estimate, which means a better eigenvalue estimate, which is an even better shift, etc. This feedback loop causes cubic convergence in the hermitian case, which is rare — and fast! It's so fast that we pretty well need extended precision to see it.

```julia
A = map(x->convert(BigFloat,x),A);
v = map(x->convert(BigFloat,x),ones(3));
μ = [];
for k = 1:5
    v = v/norm(v);
    μ = [μ; dot(v,A*v)];
    v = (A-μ[k]*eye(3))\v;
end
println("error in eigenvalue estimates: ");
[@printf("%.60f\n",μ[i]-μ[end]) for i=1:5];
```

    error in eigenvalue estimates:
    -0.214319743377535187415497700848580488907919637219499434331382
    -0.001204989279174531677792782815793603662017997875237139249415
    -0.000000000193503356367044172259691163055391791912161044219824
    -0.000000000000000000000000000000862716756381603567636210700617
    0.000000000000000000000000000000000000000000000000000000000000

```julia

```


```python
import matplotlib.pyplot as plt
import numpy as np
from ipywidgets import interactive
import sympy as sp
import math
import sys
```

```python
def plotter(x,f,xlim,ytop,x_1,y_1,t,h,r):
  plt.figure(figsize=(10,10),facecolor='white')
  plt.plot(x,f,'k',x_1,y_1,'ro')
  plt.grid()
  plt.legend(["Trajectory","Max height"])
  plt.ylabel("Height [m]")
  plt.xlabel("Range [m]")
  plt.ylim(top=ytop,bottom=0)
  plt.xlim(0,xlim)
  plt.title("Time of flight={}s, Range={}m, Maximum height={}m".format(t,r,h))
  plt.xticks(np.arange(0,xlim,step=1.0))
  plt.yticks(np.arange(0,ytop,step=1.0))
  plt.show()
```

```python
def trajectory(theta,velo,height,acceleration):

  theta=theta*np.pi/180
  u=velo
  h=height
  accel=acceleration

  d= sp.Symbol('d')
  k=d*np.tan(theta)-accel*d**2/(2*u**2*(np.cos(theta))**2)+h
  m=sp.solve(k)
  m[0]=float(m[0])
  m[1]=float(m[1])

  xlim=0
  if len(m)==2:
    if m[1]>1:
       xlim=np.ceil(m[1])
    else:
       y=round(m[1],3)
       xlim=y+0.01*y
  else:
    xlim=np.ceil(m[0])

  x=np.linspace(0,1.1*xlim,100)

  f=lambda theta,u,h,accel:x*np.tan(theta)-accel*x**2/(2*u**2*(np.cos(theta))**2)+h
  t=(u*np.sin(theta)+np.sqrt((u*np.sin(theta))**2+2*accel*h))/accel

  x_1=np.tan(theta)*(2*u**2*(np.cos(theta))**2)/(2*accel)

  if x_1<0:
    x_1=0

  if theta>=0:
      h_max=(u*np.sin(theta))**2/(2*accel)+h
  else:
      h_max=h+0.0
  y_1=h_max

  time=round(t,3)

  if len(m)==2:
     range=round(m[1],3)
  else:
     range=round(m[0],3)
  height=round(h_max,3)
  ylim=h_max+0.1*h_max
  plotter(x,f(theta,u,h,accel),xlim,ylim,x_1,y_1,time,height,range)
```

```python
iplot=interactive(trajectory,
                    theta=(-90,90,1),
                    velo=(1,25,1),
                    height=(0,50,1),
                    acceleration=(1,25,1))
iplot
```


(rotational_motion)=

# Rotational Motion

## Angular Momentum Theorem

From Newton's second law:

\\[\vec{F}=ma=m\frac{d\vec{v}}{dt}=\frac{d(m\vec{v})}{dt}\\]

The product \\(m\vec{v}\\) is called **linear momentum**, written as \\(\vec{p}\\).

Taking the cross product of both sides with the position vector \\(\vec{r}\\):

\\[\vec{r}\times\vec{F} = \vec{r}\times(\frac{d\vec{p}}{dt})\\]

The product \\(\vec{r}\times\vec{F}\\) is called the **torque**, denoted by \\(\vec{\tau}\\). From the rules of cross product, the magnitude of \\(\vec{\tau}\\) is given by:

\\[|\tau|=rF_\theta\\]

where \\(F_\theta\\) is the **tangential component** of \\(\vec{F}\\).

The direction of \\(\vec{\tau}\\) is perpendicular to the plane formed by \\(r\\) and \\(F\\).

Therefore:

\\[\vec{\tau}=\vec{r}\times\frac{d\vec{p}}{dt}\\]

From the product rule of differentiation:

\\[\frac{d(\vec{r}\times\vec{p})}{dt}=\vec{r}\times\frac{d\vec{p}}{dt}+\frac{d\vec{r}}{dt}\times\vec{p}\\]

Since \\(\frac{d\vec{r}}{dt}=\vec{v}\\) and \\(m\vec{v}=\vec{p}\\), RHS in the equation above becomes:

\\[\vec{v}\times(m\vec{v})=m(\vec{v}\times\vec{v})=0\\]

Thus:

\\[\vec{\tau}=\frac{d(\vec{r}\times\vec{p})}{dt}\\]

The product \\(\vec{r}\times\vec{p}\\) is defined as the angular momentum \\(\vec{L}\\), and this is the **angular momentum theorem**:

\\[\vec{\tau}=\frac{d\vec{L}}{dt}\\]

## Moment of inertia

The moment of inertia of a body \\(I\\) composed of N discrete sub-bodies is defined as:

\\[I=\sum_{i=1}^{N}m_ir_i^2\\]

Where \\(m_i\\) is the mass of each sub-bodies, and \\(r_i\\) is the distance of those sub-bodies from the axis of rotation.

With this definition, the angular momentum equation reduces to:

\\[\vec{\tau}=I\frac{d^2\theta}{dt^2}\\]

where \\(\frac{d^2\theta}{dt^2}\\) represents angular acceleration.

For a continuous body, its moment of inertia \\(I\\) is defined as:

\\[I=\int r^2dm=\int\rho(r)r^2dV\\]

## Tutorial Problem 4.5

A cable is wrapped several times around a uniform, solid circular cylinder that can rotate about its axis. The cylinder has radius \\(R\\), and mass \\(m\\). The cable is pulled with a force of magnitude \\(F\\). Assuming that the cable unwinds without stretching or slipping, what will be the angular acceleration of the cylinder?

Ignore the weight of the cable.

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import animation, rc
from IPython.display import HTML
from sympy import Symbol, integrate

F = Symbol('F') # force
M = Symbol('M') # mass of cylinder
R = Symbol('R') # distance from the centre of the cylinder to where the force is applied
t = Symbol('t') # time

# x component of N = -F
# y component of N = Mg
# only F contributes to net torque
# sub into angular momentum equation
# for cylinder, I=MR**2/2

angular_acceleration = -(2*F)/(M*R)

angular_velocity = integrate(angular_acceleration, t)

angle = integrate(angular_velocity, t)

print("angular acceleration = ", angular_acceleration)

print("angular velocity = ", angular_velocity)

print("angle = ", angle)
```

    angular acceleration =  -2*F/(M*R)
    angular velocity =  -2*F*t/(M*R)
    angle =  -F*t**2/(M*R)

```python
# substitute arbitrary numbers
F = 10 # N
M = 10 # kg
R = 1 # m

t = np.linspace(0, 10, 500) # create list of time from 0 to 10 seconds

a_a = -(2*F)/(M*R) # angular acceleration

a_v = -2 * F * t / (M * R) # angular velocity

a = -(F * t**2) / (M * R) # angle

print("Angular acceleration = %.2f rad/s2" % (a_a))

X = R * np.cos(a) # x coordinate

Y = R * np.sin(a) # y coordinate

length = R * (-a / (2 * np.pi)) # length of string

l = np.zeros((len(t), len(t))) # store data in matrix to make animation
for i in range(len(t)):
    for j in range(i+1):
        l[i][j] = length[j]

# plot angular velocity over time

fig = plt.figure(figsize=(6,4))
plt.plot(t, a_v, 'k')
plt.xlabel('time (s)')
plt.ylabel('angular velocity (rad/s)')
plt.title('Angular velocity over time', fontsize=14)
plt.grid(True)
plt.show()
```

```python
nframes = len(t)

# Plot background axes
fig, ax = plt.subplots(figsize=(10,2))

# plot lines
line1, = ax.plot([], [], 'ro', lw=2)
line2, = ax.plot([], [], 'k', lw=0.5)
line3, = ax.plot([], [], 'k', lw=2)

# customise axis
ax.set_xlim(-2,18)
ax.set_ylim(-2,2)
ax.set_title('Motion of cylinder and string')

lines = [line1, line2, line3]

# Plot background for each frame
def init():
    for line in lines:
        line.set_data([], [])
    return lines

# Set what data to plot in each frame
def animate(i):

    x1 = X[i]
    y1 = Y[i]
    lines[0].set_data(x1, y1)

    x2 = X
    y2 = Y
    lines[1].set_data(x2, y2)

    x3 = l[i]
    y3 = 1
    lines[2].set_data(x3, y3)

    return lines

# Call the animator
anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=nframes, interval=10, blit=True)
```

```python
HTML(anim.to_html5_video())
```

### References

Course notes from Lecture 4 of the module ESE 95011 Mechanics


## Expectation Maximization

Expectation Maximization (EM) is a powerful technique for creating maximum likelihood estimators when the variables are difficult to separate. Here, we set up a Gaussian mixture experiment with two Gaussians and derive the corresponding estimators of their means using EM.

### Experiment: Measuring from Unseen Groups

Let's investigate the following experiment:

Suppose we have a population with two distinct groups of individuals with different heights. If we randomly pick an individual from the population, assume we don't know which group the individual is from. So, we measure that individual's height and choose another individual. The goal is to estimate the mean heights of the two distinct groups when we have an unlabeled distribution of heights sampled from both groups.

Group **a** is normally distributed as

$$ \mathcal{N}_a(x) =\mathcal{N}(x; \mu_a,\sigma) $$

and likewise for group **b**

$$ \mathcal{N}_b(x) =\mathcal{N}(x; \mu_b,\sigma) $$

Note that we fix the standard deviation $\sigma$ to be the same for both groups, but the means ($\mu_a,\mu_b$) are different. The problem is to estimate the means given that you can't directly know which group you are picking from.

Then we can write the joint density for this experiment as the following:

$$ f_{\mu_a,\mu_b}(x,z)=  \frac{1}{2} \mathcal{N}_a(x) ^z \mathcal{N}_b(x) ^{1-z} $$

where $z=1$ if we pick from group **a** and $z=0$ for group **b**. Note that the $1/2$ comes from the 50/50 chance of picking either group.  Unfortunately, since we do not measure the $z$ variable, we have to integrate it out of our density function to account for this handicap. Thus,

$$ f_{\mu_a,\mu_b}(x)=  \frac{1}{2}  \mathcal{N}_a(x)+\frac{1}{2}  \mathcal{N}_b(x)$$

Now, since $n$ trials are independent, we can write out the likelihood:

$$ \mathcal{L}(\mu_a,\mu_b|\mathbf{x})= \prod_{i=1}^n f_{\mu_a,\mu_b}(x_i)$$

This is basically notation. We have just substituted everything into $ f_{\mu_a,\mu_b}(x)$ under the independent-trials assumption. Recal that the independent trials assumptions means that the joint probability is just the product of the individual probabilities. The idea of *maximum likelihood* is to maximize this as the function of $\mu_a$ and $\mu_b$ after plugging in all of the $x_i$ data.  The problem is we don't know which group we are measuring at each trial so this is trickier than just estimating the parameters for each group separately.

### Simulating the Experiment

We need the following code to setup the experiment of randomly a group and then picking an individual from that group.

```
from __future__ import division
from numpy import array, linspace, random
from scipy.stats import bernoulli, norm
from matplotlib import cm
from matplotlib.pylab import figure, subplots
#random.seed(101) # set random seed for reproducibility
mua_true=4 # we are trying to estimate this from the data
mub_true=7 # we are trying to estimate this from the data
fa=norm(mua_true,1) # distribution for group A
fb=norm(mub_true,1) # distribution for group B
fz=bernoulli(0.25) # each group equally likely

def sample(n=10):
    'simulate picking from each group n times'
    tmp=fz.rvs(n) # choose n of the coins, A or B
    return tmp*(fb.rvs(n))+(1-tmp)*fa.rvs(n) # flip it n times

xs = sample(1000) # generate some samples
```

Here's a quick look at the density functions of each group and a histogram of the samples

```
f,ax = subplots()
x = linspace(mua_true-2,mub_true+2,100)
ax.plot(x,fa.pdf(x),label='group A')
ax.plot(x,fb.pdf(x),label='group B')
ax.hist(xs,bins=50,normed=1,label='Samples');
ax.legend(loc=0);
```

Just from looking at this plot, we suspect that we will have to reconcile the samples in the overlap region since these could have come from either group. This is where the *Expectation Maximization* algorithm enters.

## Expectation maximization

The key idea of expectation maximization is that we can somehow pretend we know the unobservable $z$ value and the proceed with the usual maximum likelihood estimation process.

The idea behind expectation-maximization is that we want to use a maximum likelihood estimate (this is the *maximization* part of the algorithm) after computing the expectation over the missing variable (in this case, $z$).

The following code uses `sympy` to setup the functions symbolically and convert them to `numpy` functions that we can quickly evaluate. Because it's easier and more stable to evaluate, we will work with the `log` of the likelihood function. It is useful to keep track of the *incomplete log-likelihood* ($\log\mathcal{L}$) since it can be proved that it is monotone increasing and good way to identify coding errors. Recall that this was the likelihood in the case where we integrated out the $z$ variable to reconcile as its absence.

```
import sympy
from sympy.abc import x, z
from sympy import stats

mu_a,mu_b = sympy.symbols('mu_a,mu_b')
na=stats.Normal( 'x', mu_a,1)
nb=stats.Normal( 'x', mu_b,1)

L=(stats.density(na)(x)+stats.density(nb)(x))/2 # incomplete likelihood function
```

Next, we need to compute the expectation step. To avoid notational overload, we will just use $\Theta$ to denote the $\mu_b$ and $\mu_a$ parameters and the data $x_i$. This means that the density function of $z$ and $\Theta$ can be written as the following:

$$ \mathbb{P}(z,\Theta) = \frac{1}{2} \mathcal{N}_a(\Theta) ^ z \mathcal{N}_b(\Theta) ^ {(1-z)} $$

For the expectation part we have to compute $\mathbb{E}(z|\Theta)$ but since $z\in \lbrace 0,1 \rbrace$, this simplifies easily

$$ \mathbb{E}(z|\Theta) = 1 \cdot \mathbb{P}(z=1|\Theta) + 0 \cdot \mathbb{P}(z=0|\Theta) =  \mathbb{P}(z=1|\Theta)  $$

Now, the only thing left is to find $  \mathbb{P}(z=1|\Theta) $ which we can do using Bayes rule:

$$  \mathbb{P}(z=1|\Theta)  = \frac{ \mathbb{P}(\Theta|z=1)\mathbb{P}(z=1)}{\mathbb{P}(\Theta)} $$

The term in the denominator comes from summing (integrating) out the $z$ items in the full joint density $ \mathbb{P}(z,\Theta) $

$$ \mathbb{P}(\Theta) = (\mathcal{N}_a(\Theta) + \mathcal{N}_b(\Theta))\frac{1}{2} $$

and since $\mathbb{P}(z=1)=1/2$, we finally obtain

$$  \mathbb{E}(z|\Theta) =\mathbb{P}(z=1|\Theta)  = \frac{\mathcal{N}_a(\Theta)}{\mathcal{N}_a(\Theta) + \mathcal{N}_b(\Theta)} $$

and which is coded below.

```
def ez(x,mu_a,mu_b): # expected value of hidden variable
  return norm(mu_a).pdf(x) / ( norm(mu_a).pdf(x) + norm(mu_b).pdf(x) )
```

Now, given we we have this estimate for $z_i$,  $\hat{z}_i=\mathbb{E(z|\Theta_i)}$, we can go back and compute the log likelihood estimate of

$$ J= \log\prod_{i=1}^n \mathbb{P}(\hat{z}_i,\Theta_i) = \sum_{i=1}^n \hat{z}_i\log \mathcal{N}_a(\Theta_i) +(1-\hat{z}_i)\log \mathcal{N}_b(\Theta_i) +\log(1/2)  $$

by maximizing it using basic calculus. The trick is to remember that $\hat{z}_i$ is *fixed*, so we only have to maximize the $\log$ parts. This leads to

$$ \hat{\mu_a} = \frac{\sum_{i=1}^n \hat{z}_i x_i}{\sum_{i=1}^n  \hat{z}_i } $$

and for $\mu_b$

$$ \hat{\mu_b} = \frac{\sum_{i=1}^n (1-\hat{z}_i) x_i}{\sum_{i=1}^n  1-\hat{z}_i } $$

Now, we finally have the *maximization* step ( above ) and the *expectation* step ($\hat{z}_i$) from earlier. We're ready to simulate the algorithm and plot its performance!

```
def run():
    out, lout = [], []
    Lf=sympy.lambdify((x,mu_a,mu_b), sympy.log(abs(L)),'numpy') # convert to numpy function from sympy
    mu_a_n=random.random() * 10 # initial guess
    mu_b_n=random.random() * 10 # initial guess
    for i in range(20): # iterations of expectation and maximization
        tau=ez(xs,mu_a_n,mu_b_n)                 # expected value of z-variable
        lout.append( sum(Lf(xs,mu_a_n,mu_b_n)) ) # save incomplete likelihood value (should be monotone)
        out.append((mu_a_n, mu_b_n))             # save of (pa,pb) steps
        mu_a_n=( sum(tau*xs) / sum(tau) )        # new maximum likelihood estimate of pa
        mu_b_n=( sum((1-tau) * xs) / sum(1-tau) )
    return out, lout

out, lout = run()

fig=figure()
fig.set_figwidth(12)
ax=fig.add_subplot(121)
ax.plot(array(out),'o-')
ax.legend(('mu_a','mu_b'),loc=0)
ax.hlines([mua_true,mub_true],0,len(out),['r','g'])
ax.set_xlabel('iteration',fontsize=18)
ax.set_ylabel('$\mu_a,\mu_b$ values',fontsize=24)
ax=fig.add_subplot(122)
ax.plot(array(lout),'o-')
ax.set_xlabel('iteration',fontsize=18)
ax.set_title('Incomplete likelihood',fontsize=16)
```

The figure on the left shows the estimates for both $\mu_a$ and $\mu_b$ for each iteration and the figure on the right shows the corresponding incomplete likelihood function. The horizontal lines on the left-figure show the true values we are trying to estimate. Notice the EM algorithm converges very quickly, but because each group is equally likely to be chosen, the algorithm cannot distinguish one from the other. The code below constructs a error surface to see this effect. The incomplete likelihood function is monotone which tells us that we have not made a coding error. We're omitting the proof of this monotonicity.

```
out, lout = run()
mua_step=linspace(0,10,30)
mub_step=linspace(0,10,20)
z=Lf(xs,mua_step[:,None],mub_step[:,None,None]).sum(axis=2) # numpy broadcasting
fig=figure(figsize=(8,5))
ax=fig.add_subplot(111)
p=ax.contourf(mua_step,mub_step,z,30,cmap=cm.gray)
xa,xb=zip(*out) # unpack the container from the previous block
ax.plot(xa,xb,'ro')                                    # points per iteration in red
ax.plot(mua_true,mub_true,'bs')                        # true values in blue
ax.plot(xa[0],xb[0],'gx',ms=15.,mew=2.)                # starting point in green
ax.text(xa[0],xb[0],'   start',color='g',fontsize=11.)
ax.set_xlabel('$\mu_a$',fontsize=24)
ax.set_ylabel('$\mu_b$',fontsize=24)
ax.set_title('Incomplete Likelihood',fontsize=18)
fig.colorbar(p);
```

The figure shows the incomplete likelihood function that the algorithm is exploring. Note that the algorithm can get to the maximizer but since the surface has symmetric maxima, it has no way to pick between them and ultimately just picks the one that is closest to the starting point. This is because each group is equally likely to be chosen. I urge you to download this notebook and try different initial points and see where the maximizer winds up.

## Summary

Expectation maximization is a powerful algorithm that is especially useful when it is difficult to de-couple the variables involved in a standard maximum likelihood estimation. Note that convergence to the "correct" maxima is not guaranteed, as we observed here. This is even more pronounced when there are more parameters to estimate. There is a nice [applet](http://www.cs.cmu.edu/~alad/em/) you can use to investigate this effect and a much more detailed mathematical derivation [here](http://crow.ee.washington.edu/people/bulyko/papers/em.pdf).

As usual, the IPython notebook corresponding to this post can be found [here](https://github.com/unpingco/Python-for-Signal-Processing/blob/master/Expectation_Maximization.ipynb). I urge you to try these calculations on your own. Try changing the sample size and making the choice between the two groups no longer equal to 1/2 (equally likely).

Note you will need at least `sympy` version 0.7.2 to run this notebook.

Comments appreciated!