# Demo - The Tau method

**Mikael Mortensen** (email: `mikaem@math.uio.no`), Department of Mathematics, University of Oslo.

Date: **June 15, 2021**

**Summary.** Shenfun has primarily been developed for the spectral Galerkin method.
However, there are other methods out there that make use of global basis
functions and variational principles. One such method, which has a lot in
common with the spectral Galerkin method, is the Tau method. The
principle difference between a Tau method and a spectral Galerkin method
is in the choice of basis functions. The spectral Galerkin method is
usually defined through function spaces where the boundary conditions
of the problem are already built in. The tau-method, on the other hand,
usually considers
only the orthogonal basis, like pure Chebyshev or Legendre polynomials,
and derives differentiation matrices for these bases. The boundary conditions
are then usually fixed through manipulation of a couple of rows of the
differentiation matrix. In this demo we will show how the original
tau-method can be easily implemented using [shenfun](https://github.com/spectralDNS/shenfun).

## The tau method for Poisson's equation in 1D

Poisson's equation is given on a domain $[-1, 1]$ as

$$
\begin{equation}
\nabla^2 u(x) = f(x) \quad \text{for }\, x \in [-1, 1], \label{eq:poisson} \tag{1}
\end{equation}
$$

$$
\begin{equation}
u(-1)=a, u(1)=b, \notag
\label{_auto1} \tag{2}
\end{equation}
$$

where $u(x)$ is the solution, $f(x)$ is a function and $a, b$ are two possibly
non-zero constants. To solve Eq. ([1](#eq:poisson)) with the tau method we choose either
Legendre of Chebyshev basis functions $\phi_k$, and then look for
solutions

$$
\begin{equation}
u(x) = \sum_{k=0}^{N-1} \hat{u}_k \phi_k(x), \label{eq:u} \tag{3}
\end{equation}
$$

where $N$ is the size of the discretized problem and
$\hat{\mathbf{u}} = \{\hat{u}_k\}_{k=0}^{N-1}$ are the unknown expansion
coefficients. For this function to satisfy the given boundary conditions, it is necessary
that

$$
\begin{equation}
u(-1) = \sum_{k=0}^{N-1} \hat{u}_k \phi_k(-1) = \sum_{k=0}^{N-1}\hat{u}_{k}(-1)^k = a,
\label{eq:dirichleta} \tag{4}
\end{equation}
$$

$$
\begin{equation}
u(+1) = \sum_{k=0}^{N-1} \hat{u}_k \phi_k(+1) = \sum_{k=0}^{N-1} \hat{u}_{k} = b,
\label{eq:dirichletb} \tag{5}
\end{equation}
$$

where we have use that $\phi_k(1) = 1$ and $\phi_k(-1)=(-1)^k$ for $k=0,1, \ldots, N-1$,
for either Chebyshev or Legendre polynomials $\phi_k$. This gives two equations
for the $N$ unknowns in  $\{\hat{u}_k\}_{k=0}^{N-1}$.
We will now use variational principles, like in the Galerkin method, in order to
derive equations that can be used to close the remaining $N-2$ unknowns.
To this end we multiply Poisson's equation by a
test function $v$, a weight $w$, and integrate  over the domain

$$
\begin{equation}
\int_{-1}^1 \nabla^2 u \, v \, w\, dx = \int_{-1}^1 f \, v\, w\, dx. \label{eq:varform} \tag{6}
\end{equation}
$$

The weight function depends on the choice of basis functions. For Chebyshev
it will be $1/\sqrt{1-x^2}$, whereas it is unity for Legendre.

Finally, we insert the trial function $u=\sum_{j=0}^{N-1}\hat{u}_j \phi_j$ and
the test function $v=\phi_k$, to get

$$
\begin{equation}
\int_{-1}^1 \nabla^2 \sum_{j=0}^{N-1} \phi_j \,  \, \phi_k \, w\, dx \hat{u}_j = \int_{-1}^1 f \, \phi_k\, w\, dx. \label{eq:varform2} \tag{7}
\end{equation}
$$

This problem can be reformulated as a linear algebra problem,

$$
\begin{equation}
a_{kj} \hat{u}_j = \tilde{f}_k,
\label{_auto2} \tag{8}
\end{equation}
$$

$$
\begin{equation}
A \hat{\mathbf{{u}}} = \tilde{\mathbf{{f}}}.
\label{_auto3} \tag{9}
\end{equation}
$$

However, the matrix $A\in \mathbb{R}^{N \times N}$ is singular because it
contains two zero rows. These two rows are used to implement the two boundary
conditions. Setting $a_{N-2, j}=1$ and
$a_{N-1, j}= (-1)^j$ for $j=0, 1, \ldots, N-1$, and also fixing
the right hand sides $\tilde{f}_{N-2}=a$ and $\tilde{f}_{N-1}=b$, the
two boundary conditions will be satisfied.

## Implementation

### Preamble

We will solve Poisson's equation using the [shenfun](https://github.com/spectralDNS/shenfun) Python module. The first thing needed
is then to import some of this module's functionality
plus some other helper modules, like [Numpy](https://numpy.org) and [Sympy](https://sympy.org), and the [scipy.sparse](https://docs.scipy.org/doc/scipy/reference/sparse.html)
for handeling sparse matrices:

```
from shenfun import inner, div, grad, TestFunction, TrialFunction, Function, \
    project, Dx, Array, FunctionSpace, dx
import numpy as np
import scipy.sparse as scp
from sympy import symbols, cos, sin, exp, lambdify
```

We use `Sympy` for a manufactured solution and `Numpy` for testing.
The exact manufactured solution $u_e(x)$ and the right hand side
$f_e(x)$ are created using `Sympy` as follows

```
x = symbols("x")
ue = sin(4*np.pi*x)
fe = ue.diff(x, 2)
```

Note that we compute the right hand side function `fe` that corresponds to
the manufactured solution `ue`.

### Discretization

We create a basis with a given number of basis functions,

```
N = 32
T = FunctionSpace(N, 'Chebyshev')
#T = FunctionSpace(N, 'Legendre')
```

Note that we can either choose a Legendre or a Chebyshev basis. The
remaining code will work either way.

### Variational formulation

The variational problem ([6](#eq:varform)) can be assembled using `shenfun`'s
[TrialFunction](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.TrialFunction), [TestFunction](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.TestFunction) and [inner()](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.inner.inner) functions.

```
u = TrialFunction(T)
v = TestFunction(T)
# Assemble differentiation matrix
A = inner(v, div(grad(u)))
# Assemble right hand side
fj = Array(T, buffer=fe)
f_hat = Function(T)
f_hat = inner(v, fj, output_array=f_hat)
```

Note that the `sympy` function `fe` is be used to initialize the [Array](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Array)
`fj`, because an Array
is required as input to the [inner()](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.inner.inner) function. An
[Array](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Array) contains the solution evaluated on the
quadrature mesh. A [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function) represents a global
expansion, like Eq. ([3](#eq:u)), and its values are the
expansion coefficients $\{\hat{u}_{k}\}_{k=0}^{N-1}$.

### Fix boundary conditions

We fix two rows of the differentiation matrix in order to satisfy
Eqs. ([4](#eq:dirichleta)) and ([5](#eq:dirichletb)).

```
A = A.diags('lil')
A[-2] = (-1)**np.arange(N)
A[-1] = np.ones(N)
A = A.tocsc()
f_hat[-2] = ue.subs(x, T.domain[0])
f_hat[-1] = ue.subs(x, T.domain[1])
```

Note that the last two lines uses evaluation of the sympy function
`ue` at the borders of the domain. Implemented like this it is
easy to change to a nonstandard domain size.
The sparsity pattern of the matrix A is now modified
with the typical tau-lines that we can visualize using [plotly](https://plotly.com/)

```
import plotly.express as px
z = np.where(abs(A.toarray()) > 1e-6, 0, 1).astype(bool)
fig = px.imshow(z, binary_string=True)
fig.show()
```

### Solve linear equations

Finally, solve the linear equation system and transform the solution from the spectral
$\{\hat{u}_k\}_{k=0}^{N-1}$ vector to the real space $\{u(x_j)\}_{j=0}^{N-1}$
and then check how the solution corresponds with the exact solution $u_e$.
To this end we compute the $L_2$-errornorm using the `shenfun` function
[dx()](https://shenfun.readthedocs.io/en/latest/shenfun.utilities.html#shenfun.utilities.dx)

```
u_hat = Function(T)
u_hat[:] = scp.linalg.spsolve(A, f_hat)
uj = u_hat.backward()
ua = Array(T, buffer=ue)

print("Error=%2.16e" %(np.sqrt(dx((uj-ua)**2))))
```

### Convergence test

To do a convergence test we will now create a function `main`, that takes the
number of quadrature points as parameter, and prints out
the error.

```
def main(N, family='Chebyshev'):
    T = FunctionSpace(N, family=family)
    u = TrialFunction(T)
    v = TestFunction(T)

    # Get f on quad points
    fj = Array(T, buffer=fe)

    # Compute right hand side of Poisson's equation
    f_hat = Function(T)
    f_hat = inner(v, fj, output_array=f_hat)

    # Get left hand side of Poisson's equation
    A = inner(v, div(grad(u)))
    A = A.diags('lil')
    A[-2] = (-1)**np.arange(N)
    A[-1] = np.ones(N)
    A = A.tocsc()
    f_hat[-2] = ue.subs(x, T.domain[0])
    f_hat[-1] = ue.subs(x, T.domain[1])

    u_hat = Function(T)
    u_hat[:] = scp.linalg.spsolve(A, f_hat)
    uj = u_hat.backward()

    # Compare with analytical solution
    ua = Array(T, buffer=ue)
    l2_error = np.linalg.norm(uj-ua)
    return l2_error
```

For example, we find the error of a Chebyshev discretization
using 12 quadrature points as

```
main(12, 'Chebyshev')
```

To get the convergence we call `main` for a list
of $N=[12, 16, \ldots, 48]$, and collect the errornorms in
arrays to be plotted. The error can be plotted using
[matplotlib](https://matplotlib.org).

```
%matplotlib inline

import matplotlib.pyplot as plt

N = range(12, 50, 4)
error = {}
for basis in ('legendre', 'chebyshev'):
    error[basis] = []
    for i in range(len(N)):
        errN = main(N[i], basis)
        error[basis].append(errN)

plt.figure(figsize=(6, 4))
for basis, col in zip(('legendre', 'chebyshev'), ('r', 'b')):
    plt.semilogy(N, error[basis], col, linewidth=2)
plt.title('Convergence of Tau Poisson solvers 1D')
plt.xlabel('N')
plt.ylabel('Error norm')
plt.legend(('Legendre', 'Chebyshev'))
plt.show()
```

The spectral convergence is evident and we can see that
after $N=40$ roundoff errors dominate as the errornorm trails off around $10^{-14}$.

## Complete solver

A complete solver, that can use either Legendre or Chebyshev bases, chosen as a
command-line argument, can also be found [here](https://github.com/spectralDNS/shenfun/blob/master/demo/poisson1D_tau.py).


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Superposition of Random Signals

The superposition of two random signals

\begin{equation}
y[k] = x[k] + n[k]
\end{equation}

is a frequently applied operation in statistical signal processing. For instance, to model a measurement procedure or communication channel as superposition of the desired signal with noise. We assume that the statistical properties of the real-valued signals $x[k]$ and $n[k]$ are known. We are interested in the statistical properties of $y[k]$, as well as the joint statistical properties between the signals and their superposition $y[k]$. It is assumed for the following that $x[k]$ and $n[k]$ are drawn from wide-sense stationary (WSS) real-valued random processes.

### Cumulative Distribution and Probability Density Function

The cumulative distribution function (CDF) $P_y(\theta)$ of $y[k]$ is given by rewriting it in terms of the joint probability density function (PDF) $p_{xn}(\theta_x, \theta_n)$

\begin{equation}
P_y(\theta) = \Pr \{ y[k] \leq \theta \} = \Pr \{ (x[k] + n[k]) \leq \theta \} =
\int\limits_{-\infty}^{\infty} \int\limits_{-\infty}^{\theta - \theta_n} p_{xn}(\theta_x, \theta_n) \; \mathrm{d}\theta_x\,\mathrm{d}\theta_n
\end{equation}

Its PDF is computed by introducing above result into the [definition](distributions.ipynb#Univariate-Probability-Density-Function) of the PDF

\begin{equation}
p_y(\theta) = \frac{\mathrm{d} P_y(\theta)}{\mathrm{d}\theta} = \int\limits_{-\infty}^{\infty} p_{xn}(\theta - \theta_n, \theta_n) \; \mathrm{d}\theta_n
\end{equation}

since the inner integral on the right hand side of $P_y(\theta)$ can be interpreted as the inverse operation to the derivation with respect to $\theta$.

An important special case is that $x[k]$ and $n[k]$ are uncorrelated. Under this assumption, the joint PDF $p_{xn}(\theta_x, \theta_n)$ can be written as $p_{xn}(\theta_x, \theta_n) = p_x(\theta_x) \cdot p_n(\theta_n)$. Introducing this into above result yields

\begin{align}
p_y(\theta) &= \int\limits_{-\infty}^{\infty} p_x(\theta - \theta_n) \cdot p_n(\theta_n) \; \mathrm{d}\theta_n \\
&= p_x(\theta) * p_n(\theta)
\end{align}

Hence, the PDF of the superposition of two uncorrelated signals is given by the convolution of the PDFs of both signals.

#### Example - PDF of a superposition of two uncorrelated signals

The following example estimates the PDF of a superposition of two uncorrelated signals drawn from random processes generating samples according to the [uniformly distributed](important_distributions.ipynb#Uniform-Distribution) white noise model with $a=0$ and $b=1$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

K = 100000  # length of random signals

# generate random signals
np.random.seed(2)
x = np.random.uniform(size=K)
n = np.random.uniform(size=K)
y = x + n

# plot estimated pdf
plt.figure(figsize = (10, 6))
plt.hist(y, 100, density=True)
plt.title('Estimated PDF')
plt.xlabel(r'$\theta$')
plt.ylabel(r'$\hat{p}_y(\theta)$')
plt.grid()
```

**Exercise**

* Check the result of the numerical simulation by calculating the theoretical PDF of $y[k]$
* What PDF will results if you sum up an infinite number of independent uniformly distributed random signals?

Solution: The PDF of both signals $x[k]$ and $n[k]$ is given as $p_x(\theta) = p_n(\theta) = \text{rect}(\theta - \frac{1}{2})$, according to the assumptions stated above. The PDF of $y[k] = x[k] + n[k]$ is consequently given by $p_y(\theta) = \text{rect}(\theta - \frac{1}{2}) * \text{rect}(\theta - \frac{1}{2})$. The convolution of two rectangular signals results in a triangular signal. Hence

\begin{equation}
p_y(\theta) =
\begin{cases}
1 - |\theta -1| & \text{for } 0 < \theta < 2 \\
0 & \text{otherwise}
\end{cases}
\end{equation}

The [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) states that a superposition of independent random variables tends towards a normal distribution for an increasing number of superpositions. Numerical simulation for a high bus finite number of independent uniformly distributed random signals yields

```python
N = 1000  # number of independent random signals

# generate random signals
np.random.seed(2)
y = np.sum(np.random.uniform(size=(N, K)), axis=0)

# plot estimated pdf
plt.figure(figsize = (10, 6))
plt.hist(y, 100, density=True)
plt.title('Estimated PDF')
plt.xlabel(r'$\theta$')
plt.ylabel(r'$\hat{p}_y(\theta)$')
plt.grid()
```

### Linear Mean

The linear mean $\mu_y$ of the superposition is derived by introducing $y[k] = x[k] + n[k]$ into the [definition of the linear mean](ensemble_averages.ipynb#Linear-mean) and exploiting the [linearity of the expectation operator](ensemble_averages.ipynb#Properties) as

\begin{equation}
\mu_y[k] = E \{ x[k] + n[k] \} = \mu_x[k] + \mu_n[k]
\end{equation}

The linear mean of the superposition of two random signals is the superposition of its linear means.

### Auto-Correlation Function and Power Spectral Density

The ACF is computed in the same manner as the linear mean by inserting the superposition into the [definition](correlation_functions.ipynb#Auto-Correlation-Function) of the ACF and rearranging terms

\begin{align}
\varphi_{yy}[\kappa] &= E\{ y[k] \cdot y[k-\kappa] \} \\
&= E\{ (x[k] + n[k]) \cdot (x[k-\kappa] + n[k-\kappa]) \} \\
&= \varphi_{xx}[\kappa] + \varphi_{xn}[\kappa] + \varphi_{nx}[\kappa] + \varphi_{nn}[\kappa]
\end{align}

The ACF of the superposition of two random signals is given as the superposition of all auto- and cross-correlation functions (CCFs) of the two random signals. The power spectral density (PSD) is derived by performing a discrete-time Fourier transform (DTFT) of the ACF

\begin{equation}
\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{xn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{nx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

This can be simplified further by exploiting the symmetry property of the CCFs $\varphi_{xn}[\kappa] = \varphi_{nx}[-\kappa]$ and the DTFT for real-valued signals as

\begin{equation}
\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + 2\,\Re \{ \Phi_{xn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \} + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

where $\Re \{ \cdot \}$ denotes the real part of its argument.

### Cross-Correlation Function and Cross Power Spectral Density

The CCF $\varphi_{ny}[\kappa]$ between the random signal $n[k]$ and the superposition $y[k]$ is derived again by introducing the superposition into the [definition of the CCF](correlation_functions.ipynb#Cross-Correlation-Function)

\begin{equation}
\varphi_{ny}[\kappa] = E\{ n[k] \cdot (x[k-\kappa] + n[k-\kappa]) \} = \varphi_{nx}[\kappa] + \varphi_{nn}[\kappa]
\end{equation}

It is given as the superposition of the CCF between the two random signals and the ACF of $n[k]$. The cross PSD is derived by applying the DTFT to $\varphi_{ny}[\kappa]$

\begin{equation}
\Phi_{ny}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{nx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

The CCF $\varphi_{xy}[\kappa]$ and cross PSD $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ can be derived by exchanging the signals $n[k]$ and $x[k]$

\begin{equation}
\varphi_{xy}[\kappa] = E\{ x[k] \cdot (x[k-\kappa] + n[k-\kappa]) \} = \varphi_{xx}[\kappa] + \varphi_{xn}[\kappa]
\end{equation}

and

\begin{equation}
\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{xn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

### Additive White Gaussian Noise

In order to model the effect of distortions, it is often assumed that a random signal $x[k]$ is distorted by additive normal distributed white noise $n[k]$ resulting in the observed signal $y[k] = x[k] + n[k]$. It is furthermore assumed that the noise $n[k]$ is uncorrelated to the signal $x[k]$. This model is known as [additive white Gaussian noise](https://en.wikipedia.org/wiki/Additive_white_Gaussian_noise) (AWGN) model.

For zero-mean random processes and from the properties of the AWGN model, it follows that $\varphi_{xn}[\kappa] = \varphi_{nx}[\kappa] = 0$ and $\varphi_{nn}[\kappa] = N_0 \cdot \delta[\kappa]$. Introducing this into above results for additive random signals yields the following relations for the AWGN model

\begin{align}
\varphi_{yy}[\kappa] &= \varphi_{xx}[\kappa] + N_0 \cdot \delta[\kappa] \\
\varphi_{ny}[\kappa] &= N_0 \cdot \delta[\kappa] \\
\varphi_{xy}[\kappa] &= \varphi_{xx}[\kappa]
\end{align}

The PSDs are given as the DTFTs of the ACF/CCFs. The AWGN model is frequently assumed in communications as well as in the measurement of physical quantities to cope for background, sensor and amplifier noise.

### Example - Estimating the Period of a Noisy Periodic Signal

For the following numerical example, the disturbance of a harmonic signal $x[k] = \cos(\Omega_0 k)$ by unit variance AWGN $n[k]$ is considered. The observed signal is given as

\begin{equation}
y[k] = x[k] + n[k]
\end{equation}

The ACF $\varphi_{yy}[\kappa]$ of the compound signal, as well as the CCF between noise and signal $\varphi_{ny}[\kappa]$ is computed and plotted.

```python
N = 1024  # length of compound signals
M = 25  # period of cosine signal
K = 2*M  # maximum lag for ACF/CCF

# generate signals
x = np.cos(2*np.pi/M * np.arange(N))
np.random.seed(2)
n = np.random.normal(size=N)
# superposition of signals
y = x + n

# compute and truncate ACF of superposition
acf = 1/N * np.correlate(y, y, mode='full')
acf = acf[(len(y)-1)-K:len(y)+K]
# compute and truncate CCF of superposition and noise
ccf = 1/N * np.correlate(n, y, mode='full')
ccf = ccf[(len(y)-1)-K:len(y)+K]

# plot results
kappa = np.arange(-K, K+1)

plt.figure(figsize=(10, 10))

plt.subplot(311)
plt.stem(y)
plt.title('Signal')
plt.xlabel(r'$k$')
plt.ylabel(r'$x[k]$')
plt.axis([0, K, -3, 3])

plt.subplot(312)
plt.stem(kappa, acf)
plt.title('ACF of superposition')
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$\varphi_{yy}[\kappa]$')
plt.axis([-K, K, -.75, 1.1*np.max(acf)])
plt.grid()

plt.subplot(313)
plt.stem(kappa, ccf)
plt.title('CCF between noise and superposition')
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$\varphi_{ny}[\kappa]$')
plt.axis([-K, K, -.2, 1.1])
plt.grid()

plt.tight_layout()
```

**Exercise**

* Derive the theoretic result for the ACF $\varphi_{xx}[\kappa]$
* Based on this, can you explain the shape of the ACF?
* Estimate the periodicity/frequency of the cosine signal from the ACF
* What conclusions can you draw from the CCF between noise and signal?

Solution: The ACF of the deterministic signal $x[k] = \cos(\Omega_0 k)$ is computed via temporal averaging. Since $x[k]$ is periodic, the ACF $\varphi_{xx}[\kappa]$ is also periodic. It is hence sufficient to perform the temporal averaging over one period $N = \frac{2 \pi}{\Omega_0}$ with $N \in \mathbb{Z}$

\begin{align}
\varphi_{xx}[\kappa] &= \frac{1}{N} \sum_{k=0}^{N-1} x[k] \cdot x[k-\kappa] \\
&= \frac{1}{N} \sum_{k=0}^{N-1} \cos(\Omega_0 k) \cdot \cos(\Omega_0 (k-\kappa))
\end{align}

Applying the [product-to-sum identity](https://en.wikipedia.org/wiki/List_of_trigonometric_identities#Product-to-sum_and_sum-to-product_identities) of the cosine function and rearranging terms yields

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{2} \cos(\Omega_0 \kappa) + \frac{1}{2N} \sum_{k=0}^{N-1} \cos(2 \Omega_0 k - \Omega_0 \kappa)
\end{equation}

The remaining sum over the phase shifted cosine $\cos(2 \Omega_0 k - \Omega_0 \kappa)$ is zero since the summation is carried out over two full periods. The ACF of a cosine signal is then given as

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{2} \cos(\Omega_0 \kappa)
\end{equation}

Introducing this into the relations for the AWGN model, the ACF of the superposition is

\begin{equation}
\varphi_{yy}[\kappa] = \varphi_{xx}[\kappa] + N_0 \delta[\kappa]  = \frac{1}{2} \cos(\Omega_0 \kappa) + N_0 \delta[\kappa]
\end{equation}

Taking the statistical uncertainty due to a finite number of samples into account, this analytic result coincides well with the ACF of the numerical evaluation shown above.

The ACF of a periodic signal is also periodic. The ACF $\varphi_{yy}[\kappa]$ is composed from a superposition of the ACF $\varphi_{xx}[\kappa]$ of the periodic cosine signal and an aperiodic contribution from the additive noise. Hence, the period of the cosine can be estimated from the period of the ACF.

It can be concluded from the CCF that the additive noise is not correlated with the cosine signal.

### Example - Denoising by Repeated Averaging

Let's assume that we can have access to multiple observations of a noisy deterministic signal $x[k]$. The $i$-th observation $y_i[k]$ is then given by the following AWGN model

\begin{equation}
y_i[k] = x[k] + n_i[k]
\end{equation}

where $n_i[k]$ denotes the $i$-th sample function of a zero-mean white noise process with PSD $N_0$. It is assumed that the sample functions of the noise process are mutually independent. An estimate $\hat{x}[k]$ of the deterministic signal $x[k]$ is given by averaging over $N$ observations

\begin{equation}
\hat{x}[k] = \frac{1}{N} \sum_{i=0}^{N-1} (x[k] + n_i[k]) = x[k] + \underbrace{\frac{1}{N} \sum_{i=0}^{N-1} n_i[k]}_{n[k]}
\end{equation}

The estimate consists of a superposition of the true signal $x[k]$ and noise $n[k]$.

The [signal-to-noise ratio](https://en.wikipedia.org/wiki/Signal-to-noise_ratio) (SNR) of the estimate is defined as the power of the true signal $x[k]$ divided by the power of the additive noise. Following this definition, the average SNR is given as

\begin{equation}
\mathrm{SNR} = 10 \cdot \log_{10} \left( \frac{P_x}{\sigma_n^2} \right) \quad \text{ in dB}
\end{equation}

where $P_x$ denotes the average power of the signal $x[k]$ and $\sigma_n^2$ the variance of the noise $n[k]$. The SNR of the estimate $\hat{x}[k]$ is computed in the following to quantify the gain in SNR achieved by averaging. The average power $P_x$ of the signal is assumed to be known, the power of the noise $\sigma_x^2$ can be computed from its ACF. Repeated application of above findings for the AWGN model yields

\begin{equation}
\sigma_n^2 = \frac{1}{N^2} \, ( N \cdot N_0 ) = \frac{N_0}{N}
\end{equation}

since the scaling $\frac{1}{N}$ in above sum results in a $\frac{1}{N^2}$ scaling of the variance and $N$ uncorrelated instances of $n_i[k]$ with power $N_0$ are summed up. The SNR of the estimate follows then as

\begin{equation}
\mathrm{SNR} = 10 \cdot \log_{10} \left( P_x \right) - 10 \cdot \log_{10} \left( N_0 \right) + 10 \cdot \log_{10} \left( N \right) \quad \text{ in dB}
\end{equation}

By averaging over $N$ observations, an average gain of $10 \cdot \log_{10} \left( N \right)$ dB in terms of SNR can be achieved.

The following numerical example illustrates the denoising of a cosine signal superimposed by unit variance white Gaussian noise. One particular observation $y_i[k]$, the estimate $\hat{x}[k]$ by averaging over $N$ observations and the resulting SNR is shown.

```python
K = 1024  # number of samples
Nmax = 50  # number of observations

# generate signals
x = np.cos(2*np.pi/50 * np.arange(N))
np.random.seed(2)
n = np.random.normal(size=(Nmax, K))
# AWGN model
y = np.tile(x, (Nmax,1)) + n
# repeated averaging up to Nmax
xhat = np.zeros_like(y)
for i in range(Nmax):
    xhat[i, :] = 1/(i+1) * np.sum(y[:i+1, :], axis=0)
# compute SNR for all averages
Px = np.var(x)
Pn = np.var(xhat - x, axis=1)
SNR = 10*np.log10(Px/Pn)

# plot results
plt.figure(figsize=(10, 10))

plt.subplot(311)
plt.stem(y[0, :100])
plt.title('One observation')
plt.xlabel(r'$k$')
plt.ylabel(r'$y_0[k]$')
plt.ylim([-3, 3])

plt.subplot(312)
plt.stem(xhat[Nmax-1, :100])
plt.title('Average over {:2.0f} observations'.format(Nmax))
plt.xlabel(r'$k$')
plt.ylabel(r'$\hat{x}[k]$')
plt.ylim([-3, 3])

plt.subplot(313)
plt.plot(range(1, Nmax+1), 10*np.log10(Px * range(1, Nmax+1)), '--', label='theory')
plt.plot(range(1, Nmax+1), SNR, label='simulated')
plt.title('SNR')
plt.xlabel('number of averages $N$')
plt.ylabel('SNR in dB')
plt.grid()
plt.legend()

plt.tight_layout()
```

**Excercise**

* How does the SNR change if the instances $n_i[k]$ of the additive noise are mutually correlated?

Solution: The gain in terms of SNR will be lower since the mutually correlated AWGNs do not average out in the estimate $\hat{x}[k]$. For instance, the averaging will have no effect if the samples of the additive noise are equal for all observed signals $y_i[k] = x[k] + n_0[k]$.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


```python
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym

```

QUESTÃO 1

```python
x = sym.symbols('x')
F = x**3 - 7.5*x**2 + 18*x - 10
Fdiff = sym.expand(sym.diff(F, x))
roots = sym.solve(Fdiff, x)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ 2.0, \quad 3.0\right ]$$

    QUESTÃO 2

```python
# The local minimum of $f(x)=x^4-3x^3+2$ is at x=9/4 - f(x)=x^3 - 7.5x^2 + 18x - 10

cur_x = 6               # The algorithm starts at x=6
gamma = 0.01            # step size multiplier
precision = 0.00001
step_size = 1           # initial step size
max_iters = 10000       # maximum number of iterations
iters = 0               # iteration counter

f  = lambda x: x**3 - 7.5*x**2 + 18*x - 10  # lambda function for f(x) #Aqui você coloca sua função f
df = lambda x: 3*x**2 - 15*x + 18    # lambda function for the gradient of f(x) #Aqui você coloca sua derivada (faz regra do tombo mesmo)

while (step_size > precision) & (iters < max_iters):
    prev_x = cur_x
    cur_x -= gamma*df(prev_x)
    step_size = abs(cur_x - prev_x)
    iters+=1

print('True local minimum at {} with function value {}.'.format(9/4, f(9/4)))
print('Local minimum by gradient descent at {} with function value {}.'.format(cur_x, f(cur_x)))
```

    True local minimum at 2.25 with function value 3.921875.
    Local minimum by gradient descent at 3.000323195755751 with function value 3.5000001567170003.

```python

```


# Camera Calibration

## Coordinate System

This camera calibration follows the coordinate system below.

x: World coordinate, longitudinal. Forward direction is positive.

y: World coordinate, lateral. Leftward direction is positive.

z: World coordinate, vertical. Upward direction is positive.

u: Image coordinate, lateral. Rightward direction is positive.

v: Image coordinate, vertical. Downward direction is positive.

## Distortion Model

This camera calibration takes only barrel distortion parameters $\left( \kappa_1, \kappa_2, \kappa_3, \kappa_4, \kappa_5, \kappa_6 \right)$ into account.

Let $\left( u, v \right)$ be a point in undistorted image. Then the undistorted point is normalized as follows.

$$ \bar{u} = \frac{u - cu}{f_u} $$

$$ \bar{v} = \frac{v - cv}{f_v} $$

$$ r^2 = \bar{u}^2 + \bar{v}^2 $$

The ratio of distortion with respect to radial radius from the image center $\left( cu, cv \right)$ is computed.

$$ ratio = \frac{1.0 + \kappa_1 r^2 + \kappa_2 r^4 \kappa_3 r^6} {1.0 + \kappa_4 r^2 + \kappa_5 r^4 \kappa_6 r^6} $$

The corresponding point in distorted image $\left( u_d, v_d \right)$ is computed as follows.

$$ u_d = f_u \cdot ratio \cdot \bar{u} + cu $$

$$ u_v = f_v \cdot ratio \cdot \bar{v} + cv $$

## Homography Matrix

The homography matrix is defined as follows.

$$ H = K \cdot P \cdot \begin{bmatrix} R & R\cdot t \end{bmatrix}$$

where

$$ K = \begin{bmatrix} f_u & 0 & cu \\ 0 & f_v & cv \\ 0 & 0 & 1 \end{bmatrix} $$

$$ P = \begin{bmatrix} 0 & -1 & 0 \\ 0 & 0 & -1 \\ 1 & 0 & 0 \end{bmatrix} $$

$$ R = R_z R_y R_x $$

$$ t = \begin{bmatrix} t_x \\ t_y \\ t_z \end{bmatrix} $$

## Initial value

The homography matrix is 3x4 matrix in which intrinsic camera parameters and extrinsic camera parameters cannot be linearly separated. To solve these parameters, initial values of the parameters are computed with the following assumption. Then these values will be refined by non-linear optimization.

The assumptions are as follows.
- Camera image has no distortion i.e. $\left( \kappa_n = 0 \right)$.
- $\left(cu, cv \right)$ is at the axact center of the image.
- Angle of camera is small enough to apply small angle approximation.

```python
"""
Direct Linear Transform for computing initial values.
"""
import sympy as sp
from IPython.display import display

roll, pitch, yaw = sp.symbols('roll pitch yaw')
tx, ty, tz = sp.symbols('tx ty tz')
fu, fv, cu, cv = sp.symbols('fu fv cu cv')

cosr = 1.0  #sp.cos(roll)
sinr = roll #sp.sin(roll)
Rx = sp.Matrix([[1, 0,     0,  ],
                [0, cosr, -sinr],
                [0, sinr,  cosr]])

# pitch
cosp = 1.0   #sp.cos(pitch)
sinp = pitch #sp.sin(pitch)
Ry = sp.Matrix([[cosp,  0, sinp],
                [0,     1,    0],
                [-sinp, 0, cosp]])

# yaw
cosy = 1.0  #sp.cos(yaw)
siny = yaw  #sp.sin(yaw)
Rz = sp.Matrix([[cosy, -siny, 0],
                [siny,  cosy, 0],
                [0,        0, 1]])

# translation
T = sp.Matrix([[1, 0, 0, tx],
               [0, 1, 0, ty],
               [0, 0, 1, tz]])

# intrinsic camera parameter matrix
K = sp.Matrix([[fu, 0,  cu],
               [0,  fv, cv],
               [0,  0,  1,]])

# permutation matrix
P = sp.Matrix([[0, -1,  0],
               [0,  0, -1],
               [1,  0,  0]])

# R = Rz*Ry*Rx
# print('rotation matrix with small angle approximation')
# display(R)

R = sp.Matrix([[1.0, -yaw, pitch],
               [yaw, 1.0, -roll],
               [-pitch, roll, 1.0]])
# print('rotation matrix with small angle approximation 2')
# display(R)

RT = R*T

H  = K * P * RT
display(H)

#sp.latex(H)

H3 = H[:,3]
h3_dx = H3.diff(tx)
display(h3_dx)
h3_dy = H3.diff(ty)
display(h3_dy)
h3_dz = H3.diff(tz)
display(h3_dz)

```

$\displaystyle \left[\begin{matrix}1.0 cu - fu yaw & - cu yaw - 1.0 fu & cu pitch + fu roll & cu \left(pitch tz + 1.0 tx - ty yaw\right) - fu \left(- roll tz + tx yaw + 1.0 ty\right)\\1.0 cv + fv pitch & - cv yaw - fv roll & cv pitch - 1.0 fv & cv \left(pitch tz + 1.0 tx - ty yaw\right) - fv \left(- pitch tx + roll ty + 1.0 tz\right)\\1.0 & - yaw & pitch & pitch tz + 1.0 tx - ty yaw\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1.0 cu - fu yaw\\1.0 cv + fv pitch\\1.0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}- cu yaw - 1.0 fu\\- cv yaw - fv roll\\- yaw\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}cu pitch + fu roll\\cv pitch - 1.0 fv\\pitch\end{matrix}\right]$

From the approximated homography equation, initial guess of parameters can be obtained as follows.

$$ H = \begin{bmatrix} h_{00} & h_{01} & h_{02} & h_{03} \\ h_{10} & h_{11} & h_{12} & h_{13} \\ 1 & h_{21} & h_{22} & h_{23} \end{bmatrix} $$

$$ cu = 0.5 \cdot width $$

$$ cv = 0.5 \cdot height $$

$$ yaw = -h_{21} $$

$$ pitch = h_{22} $$

$$ fu = -\frac{f_{00} - cu}{yaw} $$

$$ fv = -\frac{f_{10} - cv}{pitch} $$

$$ roll = \frac{h_{02} - cu \cdot pitch}{fu} $$

Note that pitch and yaw angle must not be zeros.

Translation vector can be obtained by the simultaneous equations

$$ \begin{bmatrix} cu - f_u \cdot yaw & -cu \cdot yaw - fu & cu \cdot pitch + f_u \cdot roll \\ cv + f_v \cdot pitch & -cv \cdot yaw - f_v \cdot roll & cv \cdot pitch - f_v \\ 1.0 & -yaw & pitch \end{bmatrix} \begin{bmatrix}t_x \\ t_y \\ t_z\end{bmatrix} = \begin{bmatrix}h_{03} \\ h_{13} \\ h_{23} \end{bmatrix} $$

## Direct Linear Transform

```python
import sympy as sp
from IPython.display import display

roll, pitch, yaw = sp.symbols('roll pitch yaw')
tx, ty, tz = sp.symbols('tx ty tz')
fu, fv, cu, cv = sp.symbols('fu fv cu cv')

cosr = sp.cos(roll)
sinr = sp.sin(roll)
Rx = sp.Matrix([[1, 0,     0,  ],
                [0, cosr, -sinr],
                [0, sinr,  cosr]])

# pitch
cosp = sp.cos(pitch)
sinp = sp.sin(pitch)
Ry = sp.Matrix([[cosp,  0, sinp],
                [0,     1,    0],
                [-sinp, 0, cosp]])

# yaw
cosy = sp.cos(yaw)
siny = sp.sin(yaw)
Rz = sp.Matrix([[cosy, -siny, 0],
                [siny,  cosy, 0],
                [0,        0, 1]])

# translation
T = sp.Matrix([[1, 0, 0, tx],
               [0, 1, 0, ty],
               [0, 0, 1, tz]])

# intrinsic camera parameter matrix
K = sp.Matrix([[fu, 0,  cu],
               [0,  fv, cv],
               [0,  0,  1,]])

# permutation matrix
P = sp.Matrix([[0, -1,  0],
               [0,  0, -1],
               [1,  0,  0]])

R = Rz*Ry*Rx
display(R)

RT = R*T
display(RT)

H  = K * P * RT
display(H)
```

$\displaystyle \left[\begin{matrix}\cos{\left(pitch \right)} \cos{\left(yaw \right)} & \sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)} & \sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)}\\\sin{\left(yaw \right)} \cos{\left(pitch \right)} & \sin{\left(pitch \right)} \sin{\left(roll \right)} \sin{\left(yaw \right)} + \cos{\left(roll \right)} \cos{\left(yaw \right)} & \sin{\left(pitch \right)} \sin{\left(yaw \right)} \cos{\left(roll \right)} - \sin{\left(roll \right)} \cos{\left(yaw \right)}\\- \sin{\left(pitch \right)} & \sin{\left(roll \right)} \cos{\left(pitch \right)} & \cos{\left(pitch \right)} \cos{\left(roll \right)}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}\cos{\left(pitch \right)} \cos{\left(yaw \right)} & \sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)} & \sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)} & tx \cos{\left(pitch \right)} \cos{\left(yaw \right)} + ty \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)}\right) + tz \left(\sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)}\right)\\\sin{\left(yaw \right)} \cos{\left(pitch \right)} & \sin{\left(pitch \right)} \sin{\left(roll \right)} \sin{\left(yaw \right)} + \cos{\left(roll \right)} \cos{\left(yaw \right)} & \sin{\left(pitch \right)} \sin{\left(yaw \right)} \cos{\left(roll \right)} - \sin{\left(roll \right)} \cos{\left(yaw \right)} & tx \sin{\left(yaw \right)} \cos{\left(pitch \right)} + ty \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \sin{\left(yaw \right)} + \cos{\left(roll \right)} \cos{\left(yaw \right)}\right) + tz \left(\sin{\left(pitch \right)} \sin{\left(yaw \right)} \cos{\left(roll \right)} - \sin{\left(roll \right)} \cos{\left(yaw \right)}\right)\\- \sin{\left(pitch \right)} & \sin{\left(roll \right)} \cos{\left(pitch \right)} & \cos{\left(pitch \right)} \cos{\left(roll \right)} & - tx \sin{\left(pitch \right)} + ty \sin{\left(roll \right)} \cos{\left(pitch \right)} + tz \cos{\left(pitch \right)} \cos{\left(roll \right)}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}cu \cos{\left(pitch \right)} \cos{\left(yaw \right)} - fu \sin{\left(yaw \right)} \cos{\left(pitch \right)} & cu \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)}\right) - fu \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \sin{\left(yaw \right)} + \cos{\left(roll \right)} \cos{\left(yaw \right)}\right) & cu \left(\sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)}\right) - fu \left(\sin{\left(pitch \right)} \sin{\left(yaw \right)} \cos{\left(roll \right)} - \sin{\left(roll \right)} \cos{\left(yaw \right)}\right) & cu \left(tx \cos{\left(pitch \right)} \cos{\left(yaw \right)} + ty \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)}\right) + tz \left(\sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)}\right)\right) - fu \left(tx \sin{\left(yaw \right)} \cos{\left(pitch \right)} + ty \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \sin{\left(yaw \right)} + \cos{\left(roll \right)} \cos{\left(yaw \right)}\right) + tz \left(\sin{\left(pitch \right)} \sin{\left(yaw \right)} \cos{\left(roll \right)} - \sin{\left(roll \right)} \cos{\left(yaw \right)}\right)\right)\\cv \cos{\left(pitch \right)} \cos{\left(yaw \right)} + fv \sin{\left(pitch \right)} & cv \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)}\right) - fv \sin{\left(roll \right)} \cos{\left(pitch \right)} & cv \left(\sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)}\right) - fv \cos{\left(pitch \right)} \cos{\left(roll \right)} & cv \left(tx \cos{\left(pitch \right)} \cos{\left(yaw \right)} + ty \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)}\right) + tz \left(\sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)}\right)\right) - fv \left(- tx \sin{\left(pitch \right)} + ty \sin{\left(roll \right)} \cos{\left(pitch \right)} + tz \cos{\left(pitch \right)} \cos{\left(roll \right)}\right)\\\cos{\left(pitch \right)} \cos{\left(yaw \right)} & \sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)} & \sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)} & tx \cos{\left(pitch \right)} \cos{\left(yaw \right)} + ty \left(\sin{\left(pitch \right)} \sin{\left(roll \right)} \cos{\left(yaw \right)} - \sin{\left(yaw \right)} \cos{\left(roll \right)}\right) + tz \left(\sin{\left(pitch \right)} \cos{\left(roll \right)} \cos{\left(yaw \right)} + \sin{\left(roll \right)} \sin{\left(yaw \right)}\right)\end{matrix}\right]$

```python

```


# Exam Project spring 2020

```python
from itertools import product
import numpy as np
from scipy import optimize
from scipy import interpolate
from scipy import linalg
import sympy as sm
from scipy.stats import norm # normal distribution

%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.colors as colors
import matplotlib.cm as cmx
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
plt.style.use('seaborn')
import seaborn as sns

%load_ext autoreload
%autoreload 2
```

# Linear Regression

```python
def DGP(N):

    # a. independent variables
    x1 = np.random.normal(0,1,size=N)
    x2 = np.random.normal(0,1,size=N)

    # b. errors
    eps = np.random.normal(0,1,size=N)

    extreme = np.random.uniform(0,1,size=N)
    eps[extreme < 0.05] += np.random.normal(-5,1,size=N)[extreme < 0.05]
    eps[extreme > 0.95] += np.random.normal(5,1,size=N)[extreme > 0.95]

    # c. dependent variable
    y = 0.1 + 0.3*x1 + 0.5*x2 + eps

    return x1, x2, y
```

```python
np.random.seed(2020)
x1,x2,y = DGP(10000)
```

# Question 1:

We have that $\hat{\beta}=(X'X)^{-1}X'y$. Firstly we define the matrices in the equation. We define the functions and vector, in order to be able to estimate the beta vectors.

```python
# a: defining functions to make a column of 1s in X
ones = np.ones(len(x1))

# b: defining X vector by stacking ones_column, x1 and x2
X = np.column_stack((ones,x1,x2))

# c: defining X_prime, X_prime_X and X_prime_X_inv
X_prime = X.T
X_prime_X = X_prime@X
X_prime_X_inv = linalg.inv(X_prime_X)
```

We can now estimate the beta values:

```python
# Solving beta_hat

beta_hat = X_prime_X_inv@X_prime@y
print(f'The fitted values for beta are equal to {beta_hat}')
```

    The fitted values for beta are equal to [0.0956821  0.29294299 0.50332771]

# Question 2:

$$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_{1,i} + \hat{\beta}_2 x_{2,i}$$.

We construct a 3D plot, where the data has been plotted as scattered points. We define the $\hat{y}(x_1,x_2,\hat{\beta}_0,\hat{\beta}_1,\hat{\beta}_2)$ function, where we estimate the vector of coefficients $\mathbf{\beta} = (\beta_0,\beta_1,\beta_2)$

```python
#defining beta_hat values
beta_0_hat = 0.0956821
beta_1_hat = 0.29294299
beta_2_hat = 0.50332771

#defining y_hat as a function
def y_hat(x1,x2,beta_0_hat,beta_1_hat,beta_2_hat):
    return beta_0_hat+beta_1_hat*x1+beta_2_hat*x2

#defining y_hat as a vector
y_hat_vec = y_hat(x1,x2,beta_0_hat,beta_1_hat,beta_2_hat)

#defining x1_values and x2_values
x1_values = x1
x2_values = x2
```

```python
#Plotting figure

# a. actual plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
ax.scatter(x1_values,x2_values,y_hat_vec,s=50,color='red');

# b. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$\hat{y}$')

# c. invert xaxis
ax.invert_xaxis()

# d. show
plt.show()
```

# Question 3:

Estimate the vector of coefficients $\hat{\beta}=(\beta_0,\beta_1,\beta_2)$ using a numerical solver to solve the OLS problem shown below.

$$ \min_{\mathbf{\beta}} \sum^N_{i=1} (y_i - (\beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i}) )^2 $$

In the function below we define the function f, which returns the minimization problem stated above.

```python
# a: define the function with thw coefficents
def f(coefficients):
    beta0,beta1,beta2=coefficients
    return sum((y-(beta0+beta1*x1+beta2*x2))**2)

# b: inital guess
initial_guess=[1, 1, 1]

# c: optimizing the function f, to estimate the fitted values.
result=optimize.minimize(f,initial_guess, method = 'Nelder-Mead')
fitted_values = result.x
```

```python
fitted_values
```

    array([0.0956629 , 0.29291419, 0.50334836])

The $\mathbf{\beta} = (\beta_0,\beta_1,\beta_2)$ is estimated as $\mathbf{\beta}=(0.095, 0.29, 0.50)$, which is equivilent to the values estimated in question 1.

# Question 4:
We estimate the vector of coefficients $\mathbf{\beta} = (\beta_0,\beta_1,\beta_2)$ using **least absolute deviations (LAD)** using a numerical solver to solve the following problem :

$$  \min_{\beta} \sum^N_{i=1} |y_i - (\beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i}) | $$

We once again define the function, but now we wish to estimate the coefficents using LAD.

```python
def f_abs(coefficients):
    beta0,beta1,beta2=coefficients
    return sum(abs((y-(beta0+beta1*x1+beta2*x2))))

initial_guess=[1, 1, 1]
result_abs=optimize.minimize(f_abs,initial_guess, method = 'Nelder-Mead')
fitted_abs_values = result_abs.x
```

```python
fitted_abs_values
```

    array([0.09226645, 0.30745194, 0.51153822])

The estimates, when estimated with LAD, are $\mathbf{\beta} = (0.092, 0.30, 0.51)$

# Question 5:

To be able to estimate the coefficents, we implement Monte Carlo. We constructed two vectors, one containing the beta estimates from OLS and the other containing the LAD estimates.

```python
#Setting up Monte Carlo simultation for OLS
K = 5000
N = 50
vector_beta_ols= []
for i in range(K):
    x_vector  =  np.random.normal(0,1,1000)
    x = np.array([x_vector])
    y_vector = (beta*x) + np.random.normal(0,1,1000)
    y = np.array(y_vector)
    var_x_ols = np.dot(x,x.T)
    covar_xy_ols = np.dot(x,y.T)
    Matrix_inv_ols = np.linalg.inv(var_x_ols)
    beta_estimated_ols =  np.dot(Matrix_inv_ols,covar_xy_ols)
    vector_beta_ols.append(beta_estimated_ols)

#Plotting estimated betas from the Monte Carlo simulation in a histogram
NpArray_vector_beta_ols = np.array(vector_beta_ols)
vector_beta_ols_hist = NpArray_vector_beta_ols.ravel()
plt.hist(vector_beta_ols_hist ,  bins=50)
plt.show()
```

```python
#Setting up Monte Carlo simultation
K = 5000
N = 50
vector_beta_lad= []
for i in range(K):
    x_vector  =  np.random.normal(0,1,1000)
    x = np.array([x_vector])
    y_vector = sum(abs((y-(beta*x)))) + np.random.normal(0,1,1000)
    y = np.array(y_vector)
    var_x_lad = np.dot(x,x.T)
    covar_xy_lad = np.dot(x,y.T)
    Matrix_inv_lad = np.linalg.inv(var_x_lad)
    beta_estimated_lad =  np.dot(Matrix_inv_lad,covar_xy_lad)
    vector_beta_lad.append(beta_estimated_lad)

#Plotting estimated betas from the Monte Carlo simulation in a histogram
NpArray_vector_beta_lad = np.array(vector_beta_lad)
vector_beta_lad_hist = NpArray_vector_beta_lad.ravel()
plt.hist(vector_beta_lad_hist ,  bins=50)
plt.show()
```

The difference between the **OLS** and **LAD** is that OLS estimates the coefficients that minimise the sum of squared residuals, LAD estimates the coefficients that minimises the sum of the absolute residuals.

When comparing the two estimates, we wish to use the OLS estimates, given that the distribution of OLS is approximately more normally distributed than the distribution of the LAD estimates.
We wanted to plot the normal distribution in the histograms to show, which of the estimation methods would compute the most normally distributed estimates. However, as the density of the normal distribution does not match up with th density of plots, we were not able to illustrate it, in a helpful way.

# Durable Purchases

```python
# a. parameters
rho = 2
alpha = 0.8
beta = 0.96
chi=0.9
r = 0.04
Delta = 0.25

# b. vectors
v2_vec = np.empty(100)
m1_vec = np.linspace(1e-8,10,100)
m2_vec = np.linspace(1e-8,10,100)
d_vec = np.linspace(1e-8,5,100)
```

In the section below, we wish to define the functions we use to answer the questions below.

```python
# a: The utility function
def utility(c,d,x,chi,alpha,rho):
    return ((c**alpha*(d+chi*x)**(1-alpha))**(1-rho))/(1-rho)

# b: The extra durable consumption
def extra_durable_consumption(m,c):
    return m-c

# c: The houeseholds value of having cash on hand in period 2
def v2(c,m2,d,chi,alpha,rho):

    x = extra_durable_consumption(m2,c)
    return utility(c,d,x,chi,alpha,rho)

# d: utility weight on non durable consumption
def assets(m,d):
    return m-d
```

When defining the value function for the first period ($v_1(m_1)$), we firstly need to define a function w(a,d), which measures the expected value of $v_2(m_2,d)$, that is the household's value of having $m_2$ at the beginning of period 2. Here we take in to account, that the value of $m_2$ depends on the probability distribution of y.

```python
def w(a,r,Delta):

    # a. v2 value, if low income
    m2_low = (1+r)*a + 1-Delta
    v2_low = v2_interp([m2_low])[0]

    # b. v2 value, if medium income
    m2_medium = (1+r)*a + 1
    v2_medium = v2_interp([m2_medium])[0]

    # c. v2 value, if high income
    m2_high = (1+r)*a + 1 + Delta
    v2_high = v2_interp([m2_high])[0]

    # d. expected v2 value
    v2 = 0.333*v2_low + 0.333*v2_medium + 0.333*v2_high

    # e. total value
    return beta*v2
```

```python
# Defining the value function for period 1
def v1(m1,d,beta,r,Delta,v2_interp):

    a=assets(m1,d)

    return w(a,r,Delta)
```

```python
# Defining the solve function for period 1
def solve_period_2(rho,alpha,Delta,chi):

    # a. grids
    v2_grid= np.empty((100,100))
    c_grid = np.empty((100,100))

    # b. solve for each m2
    for i, m2 in enumerate(m2_vec):
        for j, d in enumerate(d_vec):

            # i. objective
            obj = lambda c: -v2(c,m2,d,chi,alpha,rho)

            # ii. initial value
            x0 = m2/2

            # iii. optimizer
            result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=[1e-8,m2])

            # iv. save
            v2_grid[i,j] = -result.fun
            c_grid[i,j] = result.x

    return m2_vec,d_vec,v2_grid,c_grid
```

# Question 1: plotting the functions $v_{2}(m_{2},d)$, $c^{\ast}(m_2,d)$, and $x^{\ast}(m_2,d)$

```python
# a. solve
m2_vec,d_vec,v2_grid,c_grid = solve_period_2(rho,alpha,Delta,chi)

# b. grids
m2_grid,d_grid = np.meshgrid(m2_vec,d_vec,indexing='ij')

# c. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(m2_grid,d_grid,v2_grid,cmap=cm.jet)

# d. add labels
ax.set_title("Household's value of having $m_2$")
ax.set_xlabel('$m_2$')
ax.set_ylabel('$d$')
ax.set_zlabel('$v_2$')
ax.invert_xaxis()
```

In the above we see a plot of $v_2$ against $m_2$ and d. From the plot it is evident, that the household's value of having cash-on-hand in the beginning of period 2 is zero for all values of $m_2$ and d, with the exception of the latter two being equal to zero. When this is the case the household's value of having cash-on-hand in the beginning of period 2 is negative.

```python
# a. solve
m2_vec,d_vec,v2_grid,c_grid = solve_period_2(rho,alpha,Delta,chi)

# b. grids
m2_grid,d_grid = np.meshgrid(m2_vec,d_vec,indexing='ij')

# c. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(m2_grid,d_grid,c_grid,cmap=cm.jet)

# d. add labels
ax.set_title("optimal consumption")
ax.set_xlabel('$m_2$')
ax.set_ylabel('$d$')
ax.set_zlabel('$c$')
ax.invert_xaxis()
```

The optimal choice of non-durable consumption is rising in the cash-on-hand in the beginning of period 2, for any given value of the pre-commited durable consumption.

```python
m2_vec,d_vec,v2_grid,c_grid = solve_period_2(rho,alpha,Delta,chi)
x_vec = m2_vec - c_grid
d_grid,m2_grid = np.meshgrid(d_vec,m2_vec,indexing='ij')

# creating figure
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
ax.plot_surface(m2_grid,d_grid,x_vec,cmap=cm.jet)

# adding title and labels
ax.set_title("optimal extra durable function")
ax.set_xlabel('$m_2$')
ax.set_ylabel('$d$')
ax.set_zlabel('$x*$')

# inverting the axis
ax.invert_xaxis()
```

For rising values of $m_2$ and declining values of d, the extra durable consumption is rising. It seems that the most optimal value, is when the household has rising cash on hand in the end of period 2. Their is a negative relationship between the pre-comitted durable consumption and the extra durable consumption.

# Question 2: Find and plot the functions $v_{1}(m_{1})$ and $d^{\ast}(m_1)$. Comment.

```python
v2_interp=interpolate.RegularGridInterpolator([m2_vec],v2_vec,bounds_error=False,fill_value=None)
```

```python
def solve_period_1(r,beta,Delta,v1,v2_interp):

    # a. grids
    m1_vec = np.linspace(1e-8,10,100)
    d_vec = np.linspace(1e-8,5,100)
    v1_vec = np.empty(100)

    # b. solve for each m1 in grid
    for i,m1 in enumerate(m1_vec):

        # i. objective
        obj = lambda d: -v1(m1,d,beta,r,Delta,v2_interp)

        # ii. initial guess
        x0 = 0.333*m1

        # iv. optimize
        result = optimize.minimize_scalar(obj,x0, method='bounded',bounds=[1e-8,m1])

        # v. save
        v1_vec[i] = -result.fun
        d_vec[i] = result.x

    return m1_vec,d_vec,v1_vec
```

```python
# c. solve period 1
m1_vec,d_vec,v1_vec = solve_period_1(r,beta,Delta,v1,v2_interp)

# d. plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(m1_vec,v1_vec,label='value of cash on hand in period 1')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_ylabel('$v_1$')
ax.set_xlim([0,4])
ax.set_ylim([-1,5]);

# d. plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(m1_vec,d_vec,label='pre-committed durable consumption')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_ylabel('$d$')
ax.set_xlim([0,10])
ax.set_ylim([-1,10]);
```

The above plots are supposed to show the value of cash on hand in period 1 and pre-committed durable consumption, respectively.
But as can be seen from above, the results are non-rational.

# Gradient descent

To be able to define the gradient_decent() algoritm, we implement the following ten steps. First and foremost, we choose a tolerance, a scale factor (Theta) and a small number (Delta), all of which are above zero.

```python
def gradient_descent(f,x0,epsilon=1e-6,Theta=0.1,Delta=1e-8,max_iter=10000):

    # step 2: guess on x0 and set n=1
    x_n = x0
    n = 1
    delta_1 = np.array([Delta,0])
    delta_2 = np.array([0,Delta])

    while True:
        # step 3: Computing a numerical approximation of the jacobian for f by
        jacobi = 1/Delta*np.array([f(x_n+delta_1)-f(x_n),f(x_n+delta_2)-f(x_n)])

        # step 4: stop if the maximum element in |Delta*f(xn_1)| is less than the tolerance level
        if np.max(np.absolute(jacobi)) < epsilon:
            break

        # step 5: Set theta=Theta
        theta = Theta
        while True:

        # step 6: Computing fn_theta = f(xn_1-theta*f(xn_1))
            fn_theta = f(x_n-theta*jacobi)

        # step 7: if fn_theta < f(xn_1) continue to step 9
            if fn_theta < f(x_n):
                break

        # step 8: set theta = theta/2
            theta = theta/2

        # step 9: set x_n=x_n - theta*jacobi
        x_n = x_n - theta * jacobi

        # step 10: set n=n+1 and return to step 3
        n += 1

    return x_n, n
```

```python
def rosen(x):
    return (1.0-x[0])**2+2*(x[1]-x[0]**2)**2

x0 = np.array([1.1,1.1])
try:
    x,it = gradient_descent(rosen,x0)
    print(f'minimum found at ({x[0]:.4f},{x[1]:.4f}) after {it} iterations')
    assert np.allclose(x,[1,1])
except:
    print('not implemented yet')
```

    minimum found at (1.0000,1.0000) after 306 iterations


# Printing with Sympy

See:
* http://docs.sympy.org/latest/tutorial/
* http://docs.sympy.org/0.7.0/tutorial.html

```python
import sympy as sp
```

```python
x = sp.symbols("x")
integ = sp.Integral(sp.sqrt(1/x), x)
```

## Pretty print

### ASCII

```python
sp.pprint(integ, use_unicode=False)
```

### Unicode

```python
sp.pprint(integ, use_unicode=True)
```

## LaTeX

```python
print(sp.latex(integ))
```

## MathML

```python
sp.printing.mathml(integ)
```


# 2. Hands-on experiment 2: k-means Clustering by Semidefinite Programming

Clustering is an unsupervised machine learning problem in which we try to partition a given data set into $k$ subsets based on distance between data points or similarity among them. The goal is to find $k$ centers and to assign each data point to one of the centers such that the sum of the square distances between them are minimal [1]. This problem is known to be NP-hard.

### Clustering problem
Given a set of $n$ points in a $d-$dimensional Euclidean space, denoted by
\begin{equation*}
S = \{ \mathbf{s}_i = (s_{i1}, \cdots, s_{id})^\top~\in \mathbb{R}^d ~~ i = 1, \cdots, n\}
\end{equation*}
find an assignment of the $n$ points into $k$ disjoint clusters $\mathcal{S} = (S_1, \cdots, S_k)$ whose centers are $\mathbf{c}_j(j = 1, \cdots, k)$ based on the total sum of squared Euclidean distances from each point $\mathbf{s}_i$ to its assigned cluster centroid $\mathbf{c}_i$, i.e.,
\begin{equation}
f(S,\mathcal{S}) = \sum_{j=1}^k\sum_{i=1}^{|S_j|}\|\mathbf{s}_i^{j} - \mathbf{c}_j \|^2,
\tag{1}
\end{equation}
where $|S_j|$ is the number of points in $S_j$, and $\mathbf{s}_i^{j} $ is the $i^{th}$ point in $S_j$.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.sparse.linalg import eigsh, svds, eigs
from math import sqrt
import scipy.io as sio
import random
import numpy.matlib
from helpers import *
import time
from scipy.sparse import isspmatrix
from Llyod_kmeans import *
# fix the seed
random.seed( 3 )
```

```python
# Load data
Problem = sio.loadmat('data/clustering_data.mat')
C = np.double(Problem['C']) # euclidean distance matrix
N = np.int(Problem['N']) # number of data points
k = Problem['k'] # number of clusters
opt_val = Problem['opt_val'] # optimum value
images = Problem['images'] # images for visualization after clustering
labels = Problem['labels'] # true labels
digits = Problem['digits'] # input data points
```

### Problem 2.1 Conditional gradient method for clustering fashion-mnist Data
---
#### (e) *(20 points)*

 $\triangleright$ Complete the missing lines in the function definitions of `HCGM` and `PDHG`, which implements Homotopy CGM and Vu-Condat algorithms, respectively. Run both methods $2000$ iterations to solve the $k$-means clustering problem.

$\triangleright$ Plot the convergence results of both algorithms using `plot_comp` function.

$\triangleright$ What are the final objective values? Are they below the optimal value provided to you? If yes, explain the reason.

$\triangleright$ Using the function `value_kmeans`, compute and report the $k$-means value before and after running both algorithms.

$\triangleright$ Run the function `kmeans` a few times and report the $k$-means value obtained by Llyod's algorithm. Compare it with the ones obtained by rounding the solution of convex methods `HCGM` and `PDHG`. Comment on the result.

( Hint:  Note that when $\mathcal{X}$ is as given in equation (2), $\kappa u u^\top \in \text{lmo}_{\mathcal{X}}(X)$, where $u$ is the eigenvector corresponding to the smallest eigenvalue of $X$.)

---

### Define operators
We provide 2 operators and their conjugates:
1. `A1`: Linear operator that takes the row sums
2. `At2`: Conjugate of operator A1
3. `A2`: Linear operator that takes the column sums
4. `At2`: Conjugate of operator A2

```python
A1 = lambda x: np.sum(x, axis = 1)
At1 = lambda y: np.transpose(np.matlib.repmat(y, N, 1))
A2 = lambda x: np.sum(x, axis = 0)
At2 = lambda y: (np.matlib.repmat(y, N, 1))

b = np.double(np.ones(N))
```

```python
def plot_comp(it, f1,f2, obj):
    plt.figure(figsize=(12, 8))
    plt.subplot(221)
    plt.loglog(it[0], f1[0], label='PDHG')#, 'go--', linewidth=2, markersize=12))
    plt.loglog(it[1], f1[1], label='HCGM')#, 'go--', linewidth=2, markersize=12))
    plt.xlabel('t (sec)',fontsize=15)
    plt.ylabel('$\|X1-1\|/1$',fontsize=15)
    plt.grid(True)

    plt.subplot(222)
    plt.loglog(it[0], f2[0], label='PDHG' )
    plt.loglog(it[1], f2[1], label='HCGM' )
    plt.xlabel('t (sec)',fontsize=15)
    plt.ylabel('dist$(X, \mathbb{R}^{n}_+)$',fontsize=15)
    plt.grid(True)
    plt.savefig('./results1.pdf', bbox_inches='tight')
    plt.show()

    obj_res_PDHG = np.reshape(np.abs(obj[0] - opt_val)/opt_val, (len(obj[0]),))
    obj_res_HCGM = np.reshape(np.abs(obj[1] - opt_val)/opt_val, (len(obj[1]),))
    plt.figure(figsize=(12, 8))
    plt.loglog((it[0]), (obj_res_PDHG), label='PDHG')
    plt.loglog((it[1]), (obj_res_HCGM), label='HCGM')
    plt.xlabel('t (sec)',fontsize=15)
    plt.ylabel('$(f(X) - f^*)/f^*$',fontsize=15)
    plt.title('Relative objective residual',fontsize=15)
    plt.legend(fontsize=15)
    plt.grid(True)
    plt.savefig('./results2.pdf', bbox_inches='tight')
    plt.show()
```

### Algorithm 1. Homotopy CGM

**Remark:** For simplicity, there is only one penalty parameter $\beta_k$ in the HCGM Algorithm. However, in practice, one can have different penalty parameters for different constraints. In our case, we advise you to **multiply by 1000 the term $ (x_k - \text{proj}_{\mathcal{K}}(x_k))$** in Algorithm 1, in order to obtain a better practical convergence. This basically corresponds to having different penalty parameters for different constraints.

```python
def HCGM(kappa=10, maxit=np.int(1e3), beta0=1):
    # Initialize
    X = np.zeros((N,N))
    AX1_b = 0.0

    feasibility1 = [] # norm(A1(X)-b1)/norm(b1)
    feasibility2 = [] # dist(X, \mathcal{K})
    objective    = [] # f(x)
    cur_iter    = []
    t    = []         # for time tracking

    iter_track = np.unique(np.ceil(np.power(2, np.linspace(0,20,50))))

    start = time.time()

    for iteration in range(1, maxit+1):

        # Update Step Size
        gamma = 2 / (iteration + 1)

        # Update beta
        beta_ = beta0 / np.sqrt(iteration + 1)

        # Write down the vk to use in the lmo (eigenvalue routine)
        vk = beta_ * C + At1(A1(X) - b) + At2(A2(X) - b) + 1e3 * (X - np.maximum(0, X))
        vk = 0.5*(vk + vk.T)

        # Linear minimization oracle
        q, u = eigsh(vk, k=1, tol=1e-16, which='SA')

        if q >= 0:
            u = 0
        else:
            u = sqrt(kappa)*u

        X_bar = np.outer(u,u)

        # Obtain A*Xbar - b
        AX_bar_b = A1(X_bar)-b

        # Update A*X - b
        AX1_b = (1.0-gamma)*AX1_b + gamma*(AX_bar_b)

        # Update X
        X = (1.0 - gamma) * X + gamma * X_bar

        if any(iteration == iter_track) or iteration==maxit:
            feasibility1.append(np.linalg.norm(AX1_b)/N)
            feasibility2.append(np.linalg.norm(np.minimum(X,0), ord='fro'))
            objective.append(np.sum(C.flatten()*X.flatten()))
            cur_iter.append(iteration)
            t.append(time.time()-start)
            print('{:03d} | {:.4e}| {:.4e}| {:.4e}|'.format(iteration, feasibility1[-1], feasibility2[-1],objective[-1]))

    return X, feasibility1, feasibility2, objective, cur_iter, t
```

#### Run HCGM

```python
X_HCGM, f1_HCGM, f2_HCGM, obj_HCGM, iter_HCGM, time_HCGM = HCGM(10, np.int(5000), 1)
```

    001 | 2.8157e-01| 0.0000e+00| 1.5028e+04|
    002 | 7.2775e-02| 2.8124e+00| 4.0633e+03|
    003 | 3.8797e-02| 2.8820e+00| 1.7951e+03|
    004 | 6.0178e-02| 1.6950e+00| 2.1542e+03|
    005 | 4.4115e-02| 1.8916e+00| 2.3621e+03|
    006 | 3.9321e-02| 1.3341e+00| 2.1423e+03|
    008 | 3.1495e-02| 1.3564e+00| 1.9023e+03|
    010 | 2.7986e-02| 1.0876e+00| 1.7042e+03|
    013 | 9.1450e-03| 1.1454e+00| 8.7334e+02|
    017 | 9.7534e-03| 8.7930e-01| 8.0521e+02|
    023 | 1.0672e-02| 6.6643e-01| 8.7141e+02|
    030 | 7.5734e-03| 4.7975e-01| 7.2881e+02|
    040 | 7.0140e-03| 3.3010e-01| 6.7838e+02|
    053 | 4.0746e-03| 3.2327e-01| 5.9426e+02|
    070 | 3.7775e-03| 2.4258e-01| 5.2278e+02|
    093 | 3.4184e-03| 2.2313e-01| 5.0501e+02|
    123 | 2.5475e-03| 1.8850e-01| 4.7864e+02|
    163 | 1.7390e-03| 1.3564e-01| 4.9768e+02|
    217 | 1.7626e-03| 9.9424e-02| 4.8616e+02|
    287 | 1.0533e-03| 8.3164e-02| 4.8309e+02|
    381 | 1.1059e-03| 5.7437e-02| 4.2825e+02|
    505 | 8.5846e-04| 5.0117e-02| 3.4963e+02|
    670 | 7.2544e-04| 4.3191e-02| 2.9828e+02|
    889 | 7.5439e-04| 3.4550e-02| 2.3581e+02|
    1180 | 5.4522e-04| 3.4669e-02| 1.8301e+02|
    1566 | 4.3218e-04| 2.7855e-02| 1.4671e+02|
    2078 | 3.2987e-04| 2.7625e-02| 1.0500e+02|
    2757 | 1.9604e-04| 2.3116e-02| 8.1500e+01|
    3658 | 1.4356e-04| 2.0797e-02| 6.3427e+01|
    4854 | 1.0048e-04| 1.8226e-02| 5.2964e+01|
    5000 | 1.0423e-04| 1.8475e-02| 5.1384e+01|

### Algorithm 2. Vu-Condat

**Remarks:**

- A similar observation applies tor the Vu-Condat algorithm: it is possible to use different dual step sizes $\{ \sigma_1 , \sigma_2, \ldots \}$. In our case, we advise you to **multiply the step-size for $y_3$ by $10^4$** to obtain a better practical convergence. (You can directly use the tuned stepsizes for PDHG.)

- In this part, you will need the projection operator of the nuclear norm ball. So, you can use what you implemented in part1 of the homework as the projection operator in the algorithm.

```python
def PDHG(kappa=10, maxit=np.int(1e3), beta0=1):
    # Initialize
    X = np.zeros((N,N))
    Xprev = X
    AX1_b = 0.0

    y1 = A1(X)
    y2 = A2(X)
    y3 = X

    normC = np.linalg.norm(C,'fro')

    feasibility1 = [] # norm(A1(X)-b1)/norm(b1)
    feasibility2 = [] # dist(X, \mathcal{K})
    objective    = [] # f(x)
    cur_iter    = []
    t    = []

    L = 1e2

    iter_track = np.unique(np.ceil(np.power(2, np.linspace(0,20,50))))

    start = time.time()
    # Primal and dual step sizes
    tau = 1/L # primal stepsize
    sigma = 1/(L**2*tau) # dual step size
    sigma2 = sigma*1e4   # different dual step size to use for the update of y3 for better practical performance.

    for iteration in range(1, maxit+1):

        # Primal variable update
        Xprev = X # store the previous iterate for reflection
        X = projSDP(Xprev - tau * (C + At1(y1) + At2(y2) + y3), kappa)

        # Dual variable updates
        Xhat = 2 * X - Xprev # the point at which the dual gradient is calculated
        y1 += sigma * (A1(Xhat) - b)
        y2 += sigma * (A2(Xhat) - b)
        y3 += sigma2 * (Xhat - np.maximum(0, y3 / sigma2 + Xhat))

        # Update A*X - b
        AX1_b = A1(X)-b

        if any(iteration == iter_track) or iteration==maxit:
            feasibility1.append(np.linalg.norm(AX1_b)/N)
            feasibility2.append(np.linalg.norm(np.minimum(X,0), ord='fro'))
            objective.append(np.sum(C.flatten()*X.flatten()))
            cur_iter.append(iteration)
            t.append(time.time()-start)
            print('{:03d} | {:.4e}| {:.4e}| {:.4e}|'.format(iteration, feasibility1[-1], feasibility2[-1],objective[-1]))

    return X, feasibility1, feasibility2, objective, cur_iter, t
```

#### Run Vu-Condat

```python
X_PDHG, f1_PDHG, f2_PDHG, obj_PDHG, iter_PDHG, time_PDHG = PDHG(10, np.int(1000), 1)
```

    001 | 3.1586e-02| 1.2231e+00| -1.8503e+03|
    002 | 3.1590e-02| 1.2607e+00| -1.8815e+03|
    003 | 3.1583e-02| 1.2534e+00| -1.8806e+03|
    004 | 3.1548e-02| 1.2014e+00| -1.8580e+03|
    005 | 3.1492e-02| 1.1321e+00| -1.8194e+03|
    006 | 3.1439e-02| 1.0802e+00| -1.7763e+03|
    008 | 3.1428e-02| 1.0664e+00| -1.7127e+03|
    010 | 3.1504e-02| 1.0696e+00| -1.6984e+03|
    013 | 3.1555e-02| 1.0467e+00| -1.7131e+03|
    017 | 7.3903e-03| 3.5223e-04| 5.4068e+01|
    023 | 7.8687e-03| 1.1596e-02| 2.7390e+01|
    030 | 6.4853e-03| 1.1006e-02| 2.8338e+01|
    040 | 4.0236e-03| 9.5649e-03| 4.4584e+01|
    053 | 3.0036e-03| 8.9718e-03| 5.6097e+01|
    070 | 2.5701e-03| 7.6986e-03| 5.1988e+01|
    093 | 1.6996e-03| 5.6568e-03| 5.9456e+01|
    123 | 1.1468e-03| 3.8540e-03| 5.6114e+01|
    163 | 5.0237e-04| 2.0250e-03| 5.6444e+01|
    217 | 4.8175e-04| 1.9808e-03| 5.6829e+01|
    287 | 2.2287e-04| 1.1415e-03| 5.6675e+01|
    381 | 1.6155e-04| 9.4620e-04| 5.6739e+01|
    505 | 1.2742e-04| 8.3259e-04| 5.6786e+01|
    670 | 2.9261e-05| 2.1072e-04| 5.6994e+01|
    889 | 1.8087e-06| 2.3632e-05| 5.7037e+01|
    1000 | 3.2585e-07| 1.2735e-05| 5.7037e+01|

### Visualize the results

```python
iters = (iter_PDHG, iter_HCGM)
times = (time_PDHG, time_HCGM)
feas1 = (f1_PDHG, f1_HCGM)
feas2 = (f2_PDHG, f2_HCGM)
obj   = (obj_PDHG, obj_HCGM)
plot_comp(times, feas1,feas2, obj)
```

### Rounding: Get the assignments from the result of the SDP
Getting the assignments requires going back to the $10$ dimensional space discussed before, and using the coordinates multiplied with the obtained matrix to construct a "denoised" version of the data points. This allows then to find the clusters from these $10$ dimensional data. See [3] for more details. Our implementation is the python reimplementation of their matlab code which can be found on [github](https://github.com/solevillar/kmeans_sdp).

```python
center_HCGM, assign_HCGM = sdp_rounding(X_HCGM,10, digits)
center_PDHG, assign_PDHG = sdp_rounding(X_PDHG,10, digits)
```

### k-means value: HCGM & Vu-Condat

```python
k_means_before = value_kmeans(digits, labels-1) # k_means value with true labels
k_means_after_HCGM  = value_kmeans(digits, assign_HCGM) # k_means value with assigned lables
k_means_after_PDHG  = value_kmeans(digits, assign_PDHG) # k_means value with assigned lables

print('k-means value initial: {:.4f}'.format(k_means_before))
print('k-means value for HCGM: {:.4f}'.format(k_means_after_HCGM))
print('k-means value for Vu-Condat: {:.4f}'.format(k_means_after_PDHG))
```

    k-means value initial: 150.9680
    k-means value for HCGM: 28.7269
    k-means value for Vu-Condat: 28.7269

###  k-means value: Lloyd's algorithm

Run the Lloyd's algorithm directly on the input digits.

```python
centers_Lloyd, classifications_Lloyd, k_means_Lloyd = kmeans(digits.T, 10)#k_means value with Lloyds k-means algorithm

print("k-means value for Lloyd's algorithm: {:.4f}".format(k_means_Lloyd))
```

    k-means value for Lloyd's algorithm: 168.4946

# OPTIONAL: Additional results for clustering fMNIST Data

### Misclassification rates: HCGM & Vu-Condat

```python
print('Misclassification rate for HCGM: {:.4f}'.format(misclassification_rate(assign_HCGM, labels)))
print('Misclassification rate for Vu-Condat: {:.4f}'.format(misclassification_rate(assign_PDHG, labels)))
```

    Misclassification rate for HCGM: 0.1250
    Misclassification rate for Vu-Condat: 0.1250

### Visualize samples and predicted labels

```python
classes = ['T-shirt/top','Trouser','Pullover','Dress','Coat','Sandal','Shirt','Sneaker','Bag','Ankle boot']
```

```python
def vis_samples(assignment, images, labels):
    assignment=assignment.astype(int)
    classes = ['T-shirt/top','Trouser','Pullover','Dress','Coat','Sandal','Shirt','Sneaker','Bag','Ankle boot']
    labels = labels-1
    rand_samp = np.random.randint(0,1000,25)
    plt.figure(figsize=(7,7))
    for i,samp in enumerate(rand_samp):
        plt.subplot(5,5,i+1)
        plt.imshow(1-np.reshape(images[samp],[28,28]), cmap=plt.cm.gray)
        plt.title('Pred. {0}\n Orig. {1}'.format(classes[assignment[samp].item()],classes[labels[samp].item()]))
        plt.axis('off')
    plt.tight_layout()
    plt.show()
```

```python
vis_samples(assign_HCGM, images,labels)
```

```python
vis_samples(assign_PDHG, images,labels)
```


# Semi-Monocoque Theory: corrective solutions

```python
from pint import UnitRegistry
import sympy
import networkx as nx
import numpy as np
import matplotlib.pyplot as plt
import sys
%matplotlib inline
from IPython.display import display
```

Import **Section** class, which contains all calculations

```python
from Section import Section
```

Initialization of **sympy** symbolic tool and **pint** for dimension analysis (not really implemented rn as not directly compatible with sympy)

```python
ureg = UnitRegistry()
sympy.init_printing()
```

Define **sympy** parameters used for geometric description of sections

```python
A, A0, t, t0, a, b, h, L, E, G = sympy.symbols('A A_0 t t_0 a b h L E G', positive=True)
```

We also define numerical values for each **symbol** in order to plot scaled section and perform calculations

```python
values = [(A, 150 * ureg.millimeter**2),(A0, 250  * ureg.millimeter**2),(a, 80 * ureg.millimeter), \
          (b, 20 * ureg.millimeter),(h, 200 * ureg.millimeter),(L, 2000 * ureg.millimeter), \
          (t, 1.3 *ureg.millimeter),(E, 72e3 * ureg.MPa), (G, 27e3 * ureg.MPa)]
datav = [(v[0],v[1].magnitude) for v in values]
```

# First example: Simple rectangular symmetric section

Define graph describing the section:

1) **stringers** are **nodes** with parameters:
- **x** coordinate
- **y** coordinate
- **Area**

2) **panels** are **oriented edges** with parameters:
- **thickness**
- **lenght** which is automatically calculated

```python
stringers = {1:[(a+2*b,h),A],
             2:[(b+sympy.Rational(1,2)*a,h),A],
             3:[(sympy.Integer(0),h),A],
             4:[(b,sympy.Integer(0)),A],
             5:[(b+a,sympy.Integer(0)),A]}
             #5:[(sympy.Rational(1,2)*a,h),A]}

panels = {(1,2):t,
          (2,3):t,
          (3,4):t,
          (4,5):t,
          (5,1):t}
```

Define section and perform first calculations

```python
S1 = Section(stringers, panels)
```

```python
S1.cycles
```

## Plot of **S1** section in original reference frame

Define a dictionary of coordinates used by **Networkx** to plot section as a Directed graph.
Note that arrows are actually just thicker stubs

```python
start_pos={ii: [float(S1.g.node[ii]['ip'][i].subs(datav)) for i in range(2)] for ii in S1.g.nodes() }
```

```python
plt.figure(figsize=(12,8),dpi=300)
nx.draw(S1.g,with_labels=True, arrows= True, pos=start_pos)
plt.arrow(0,0,20,0)
plt.arrow(0,0,0,20)
#plt.text(0,0, 'CG', fontsize=24)
plt.axis('equal')
plt.title("Section in starting reference Frame",fontsize=16);
```

## Plot of **S1** section in inertial reference Frame

Section is plotted wrt **center of gravity** and rotated (if necessary) so that *x* and *y* are principal axes.
**Center of Gravity** and **Shear Center** are drawn

```python
positions={ii: [float(S1.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S1.g.nodes() }
```

```python
x_ct, y_ct = S1.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S1.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);
```

## Standard Solution

```python
Tx, Ty, Nz, Mx, My, Mz, F, ry, ry, mz = sympy.symbols('T_x T_y N_z M_x M_y M_z F r_y r_x m_z')
```

```python
S1.set_loads(_Tx=0, _Ty=Ty, _Nz=0, _Mx=Mx, _My=0, _Mz=Mz)
```

## Loads

```python
S1.compute_stringer_actions()
S1.compute_panel_fluxes();
```

```python
S1.N
```

```python
S1.q
```

Compute **L** matrix: with 5 nodes we expect 2 **dofs**, one with _symmetric load_ and one with _antisymmetric load_

```python
S1.compute_L()
```

```python
S1.L.subs(datav)
```

Compute **H** matrix

```python
S1.compute_H()
```

```python
S1.H.subs(datav)
```

Compute $\tilde{K}$ and $\tilde{M}$ as:

$$\tilde{K} = L^T \cdot \left[ \frac{A}{A_0} \right] \cdot L$$
$$\tilde{M} = H^T \cdot \left[ \frac{l}{l_0}\frac{t_0}{t} \right] \cdot L$$

```python
S1.compute_KM(A,b,t)
```

```python
S1.Ktilde.subs(datav)
```

```python
S1.Mtilde.subs(datav)
```

Compute **eigenvalues** and **eigenvectors** as:

$$\left| \mathbf{I} \cdot \beta^2 - \mathbf{\tilde{K}}^{-1} \cdot \mathbf{\tilde{M}} \right| = 0$$

We substitute some numerical values to simplify the expressions

```python
sol_data = (sympy.N(S1.Ktilde.subs(datav).inv())*(sympy.N(S1.Mtilde.subs(datav)))).eigenvects()
```

**Eigenvalues** correspond to $\beta^2$

```python
β2 = [sol[0] for sol in sol_data]
β2
```

**Eigenvectors** are orthogonal as expected

```python
X = [sol[2][0] for sol in sol_data]
X
```

From $\beta_i^2$ we compute:
$$\lambda_i = \sqrt{\frac{E A_0 l_0}{G t_0} \beta_i^2}$$

substuting numerical values

```python
λ = [sympy.N(sympy.sqrt(E*A*b/(G*t)*βi).subs(datav)) for βi in β2]
λ
```

```python

```


# Matrix Factorization for Recommendations in Python

In this post, I'll detail a basic version of low-rank matrix factorization for recommendations employ it on a dataset of 1 million movie ratings (from 1 to 5) available from the [MovieLens](http://grouplens.org/datasets/movielens/) project. The MovieLens datasets were created collected by GroupLens Research at the University of Minnesota.

[Previously](https://beckernick.github.io/music_recommender/), I used item-based collaborative filtering to make music recommendations from raw artist listen-count data. I had a relatively small amount of data, and ended up making some pretty good recommendations. Collaborative filtering methods that compute distance relationships between items or users are generally thought of as "neighborhood" methods, since they center on the idea of "nearness". Unfortunately, there are two issues with taking this approach:

1. It doesn't scale particularly well to massive datasets
2. There's a theoretical concern with raw data based approaches.

I talked about the scaling issue in the previous post, but not the conceptual issue. The key concern is that ratings matrices may be overfit and noisy representations of user tastes and preferences. When we use distance based "neighborhood" approaches on raw data, we match to sparse low-level details that we assume represent the user's preference vector instead of the vector itself. It's a subtle difference, but it's important.

If I've listened to ten Red Hot Chili Peppers songs and you've listened to ten different Red Hot Chili Peppers songs, the raw user action matrix wouldn't have any overlap. We'd have nothing in common, even though it seems pretty likely we share at least some underlying preferencs.

If it sounds like using song features (such as genre) could help, you're right. But, to steal Joseph Konstan's (professor at Minnesota involved with GroupLens Research who has an awesome [Coursera course](https://www.coursera.org/specializations/recommender-systems) on Recommender Systems) example, what if we both like songs with great storytelling, regardless of the genre. So, how do we resolve this? I would need a method that can derive the tastes and preference vectors from the raw data.

Low-Rank Matrix Factorization is that kind of method.

## Matrix Factorization via Singular Value Decomposition

Matrix factorization is the breaking down of one matrix in a product of multiple matrices. It's extremely well studied in mathematics, and it's highly useful. There are many different ways to factor matrices, but singular value decomposition is particularly useful for making recommendations.

So what is singular value decomposition (SVD)? At a high level, SVD is an algorithm that decomposes a matrix $R$ into the best lower rank (i.e. smaller/simpler) approximation of the original matrix $R$. Mathematically, it decomposes R into a two unitary matrices and a diagonal matrix:

$$\begin{equation}
R = U\Sigma V^{T}
\end{equation}$$

where R is users's ratings matrix, $U$ is the user "features" matrix, $\Sigma$ is the diagonal matrix of singular values (essentially weights), and $V^{T}$ is the movie "features" matrix. $U$ and $V^{T}$ are orthogonal, and represent different things. $U$ represents how much users "like" each feature and $V^{T}$ represents how relevant each feature is to each movie.

To get the lower rank approximation, we take these matrices and keep only the top $k$ features, which we think of as the underlying tastes and preferences vectors.

## Setting Up the Ratings Data

Okay, enough with the math. Let's get to the code.

```python
import pandas as pd
import numpy as np

ratings_list = [i.strip().split("::") for i in open('/users/nickbecker/Downloads/ml-1m/ratings.dat', 'r').readlines()]
users_list = [i.strip().split("::") for i in open('/users/nickbecker/Downloads/ml-1m/users.dat', 'r').readlines()]
movies_list = [i.strip().split("::") for i in open('/users/nickbecker/Downloads/ml-1m/movies.dat', 'r').readlines()]
```

```python
ratings = np.array(ratings_list)
users = np.array(users_list)
movies = np.array(movies_list)
```

```python
ratings_df = pd.DataFrame(ratings_list, columns = ['UserID', 'MovieID', 'Rating', 'Timestamp'], dtype = int)
movies_df = pd.DataFrame(movies_list, columns = ['MovieID', 'Title', 'Genres'])
movies_df['MovieID'] = movies_df['MovieID'].apply(pd.to_numeric)
```

I'll also take a look at the movies and ratings dataframes.

```python
movies_df.head()
```

      MovieID
      Title
      Genres

      0
      1
      Toy Story (1995)
      Animation|Children's|Comedy

      1
      2
      Jumanji (1995)
      Adventure|Children's|Fantasy

      2
      3
      Grumpier Old Men (1995)
      Comedy|Romance

      3
      4
      Waiting to Exhale (1995)
      Comedy|Drama

      4
      5
      Father of the Bride Part II (1995)
      Comedy

```python
ratings_df.head()
```

      UserID
      MovieID
      Rating
      Timestamp

      0
      1
      1193
      5
      978300760

      1
      1
      661
      3
      978302109

      2
      1
      914
      3
      978301968

      3
      1
      3408
      4
      978300275

      4
      1
      2355
      5
      978824291

These look good, but I want the format of my ratings matrix to be one row per user and one column per movie. I'll `pivot` `ratings_df` to get that and call the new variable `R`.

```python
R_df = ratings_df.pivot(index = 'UserID', columns ='MovieID', values = 'Rating').fillna(0)
R_df.head()
```

      MovieID
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      ...
      3943
      3944
      3945
      3946
      3947
      3948
      3949
      3950
      3951
      3952

      UserID

      1
      5.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      2
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      3
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      4
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      5
      0.0
      0.0
      0.0
      0.0
      0.0
      2.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

5 rows × 3706 columns

The last thing I need to do is de-mean the data (normalize by each users mean) and convert it from a dataframe to a numpy array.

```python
R = R_df.as_matrix()
user_ratings_mean = np.mean(R, axis = 1)
R_demeaned = R - user_ratings_mean.reshape(-1, 1)
```

All set. With my ratings matrix properly formatted and normalized, I'm ready to do the singular value decomposition

## Singular Value Decomposition

Scipy and Numpy both have functions to do the singular value decomposition. I'm going to use the Scipy function `svds` because it let's me choose how many latent factors I want to use to approximate the original ratings matrix (instead of having to truncate it after).

```python
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k = 50)
```

Done. The function returns exactly what I detailed earlier in this post, except that the $\Sigma$ returned is just the values instead of a diagonal matrix. This is useful, but since I'm going to leverage matrix multiplication to get predictions I'll convert it to the diagonal matrix form.

```python
sigma = np.diag(sigma)
```

## Making Predictions from the Decomposed Matrices

I now have everything I need to make movie ratings predictions for every user. I can do it all at once by following the math and matrix multiply $U$, $\Sigma$, and $V^{T}$ back to get the rank $k=50$ approximation of $R$.

I also need to add the user means back to get the actual star ratings prediction.

```python
all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) + user_ratings_mean.reshape(-1, 1)
```

If I wanted to put this kind of system into production, I'd want to create a training and validation set and optimize the number of latent features ($k$) by minimizing the Root Mean Square Error. Intuitively, the Root Mean Square Error will decrease on the training set as $k$ increases (because I'm approximating the original ratings matrix with a higher rank matrix).

However, for movies, between around 20 and 100 feature "preferences" vectors have been found to be optimal for generalizing to unseen data.

I could create a training and validation set and optimize $k$ by minimizing RMSE, but since I'm just going through proof of concept I'll leave that for another post. I just want to see some movie recommendations.

## Making Movie Recommendations
Finally, it's time. With the predictions matrix for every user, I can build a function to recommend movies for any user. All I need to do is return the movies with the highest predicted rating that the specified user hasn't already rated. Though I didn't use actually use any explicit movie content features (such as genre or title), I'll merge in that information to get a more complete picture of the recommendations.

I'll also return the list of movies the user has already rated, for the sake of comparison.

```python
preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns)
preds_df.head()
```

      MovieID
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      ...
      3943
      3944
      3945
      3946
      3947
      3948
      3949
      3950
      3951
      3952

      0
      4.288861
      0.143055
      -0.195080
      -0.018843
      0.012232
      -0.176604
      -0.074120
      0.141358
      -0.059553
      -0.195950
      ...
      0.027807
      0.001640
      0.026395
      -0.022024
      -0.085415
      0.403529
      0.105579
      0.031912
      0.050450
      0.088910

      1
      0.744716
      0.169659
      0.335418
      0.000758
      0.022475
      1.353050
      0.051426
      0.071258
      0.161601
      1.567246
      ...
      -0.056502
      -0.013733
      -0.010580
      0.062576
      -0.016248
      0.155790
      -0.418737
      -0.101102
      -0.054098
      -0.140188

      2
      1.818824
      0.456136
      0.090978
      -0.043037
      -0.025694
      -0.158617
      -0.131778
      0.098977
      0.030551
      0.735470
      ...
      0.040481
      -0.005301
      0.012832
      0.029349
      0.020866
      0.121532
      0.076205
      0.012345
      0.015148
      -0.109956

      3
      0.408057
      -0.072960
      0.039642
      0.089363
      0.041950
      0.237753
      -0.049426
      0.009467
      0.045469
      -0.111370
      ...
      0.008571
      -0.005425
      -0.008500
      -0.003417
      -0.083982
      0.094512
      0.057557
      -0.026050
      0.014841
      -0.034224

      4
      1.574272
      0.021239
      -0.051300
      0.246884
      -0.032406
      1.552281
      -0.199630
      -0.014920
      -0.060498
      0.450512
      ...
      0.110151
      0.046010
      0.006934
      -0.015940
      -0.050080
      -0.052539
      0.507189
      0.033830
      0.125706
      0.199244

5 rows × 3706 columns

```python
def recommend_movies(predictions_df, userID, movies_df, original_ratings_df, num_recommendations=5):

    # Get and sort the user's predictions
    user_row_number = userID - 1 # UserID starts at 1, not 0
    sorted_user_predictions = preds_df.iloc[user_row_number].sort_values(ascending=False) # UserID starts at 1

    # Get the user's data and merge in the movie information.
    user_data = original_ratings_df[original_ratings_df.UserID == (userID)]
    user_full = (user_data.merge(movies_df, how = 'left', left_on = 'MovieID', right_on = 'MovieID').
                     sort_values(['Rating'], ascending=False)
                 )

    print 'User {0} has already rated {1} movies.'.format(userID, user_full.shape[0])
    print 'Recommending highest {0} predicted ratings movies not already rated.'.format(num_recommendations)

    # Recommend the highest predicted rating movies that the user hasn't seen yet.
    recommendations = (movies_df[~movies_df['MovieID'].isin(user_full['MovieID'])].
         merge(pd.DataFrame(sorted_user_predictions).reset_index(), how = 'left',
               left_on = 'MovieID',
               right_on = 'MovieID').
         rename(columns = {user_row_number: 'Predictions'}).
         sort_values('Predictions', ascending = False).
                       iloc[:num_recommendations, :-1]
                      )

    return user_full, recommendations
```

```python
already_rated, predictions = recommend_movies(preds_df, 837, movies_df, ratings_df, 10)
```

    User 837 has already rated 69 movies.
    Recommending highest 10 predicted ratings movies not already rated.

So, how'd I do?

```python
already_rated.head(10)
```

      UserID
      MovieID
      Rating
      Timestamp
      Title
      Genres

      36
      837
      858
      5
      975360036
      Godfather, The (1972)
      Action|Crime|Drama

      35
      837
      1387
      5
      975360036
      Jaws (1975)
      Action|Horror

      65
      837
      2028
      5
      975360089
      Saving Private Ryan (1998)
      Action|Drama|War

      63
      837
      1221
      5
      975360036
      Godfather: Part II, The (1974)
      Action|Crime|Drama

      11
      837
      913
      5
      975359921
      Maltese Falcon, The (1941)
      Film-Noir|Mystery

      20
      837
      3417
      5
      975360893
      Crimson Pirate, The (1952)
      Adventure|Comedy|Sci-Fi

      34
      837
      2186
      4
      975359955
      Strangers on a Train (1951)
      Film-Noir|Thriller

      55
      837
      2791
      4
      975360893
      Airplane! (1980)
      Comedy

      31
      837
      1188
      4
      975360920
      Strictly Ballroom (1992)
      Comedy|Romance

      28
      837
      1304
      4
      975360058
      Butch Cassidy and the Sundance Kid (1969)
      Action|Comedy|Western

```python
predictions
```

      MovieID
      Title
      Genres

      516
      527
      Schindler's List (1993)
      Drama|War

      1848
      1953
      French Connection, The (1971)
      Action|Crime|Drama|Thriller

      596
      608
      Fargo (1996)
      Crime|Drama|Thriller

      1235
      1284
      Big Sleep, The (1946)
      Film-Noir|Mystery

      2085
      2194
      Untouchables, The (1987)
      Action|Crime|Drama

      1188
      1230
      Annie Hall (1977)
      Comedy|Romance

      1198
      1242
      Glory (1989)
      Action|Drama|War

      897
      922
      Sunset Blvd. (a.k.a. Sunset Boulevard) (1950)
      Film-Noir

      1849
      1954
      Rocky (1976)
      Action|Drama

      581
      593
      Silence of the Lambs, The (1991)
      Drama|Thriller

Pretty cool! These look like pretty good recommendations. It's also good to see that, though I didn't actually use the genre of the movie as a feature, the truncated matrix factorization features "picked up" on the underlying tastes and preferences of the user. I've recommended some film-noirs, crime, drama, and war movies - all of which were genres of some of this user's top rated movies.

## Conclusion

We've seen that we can make good recommendations with raw data based collaborative filtering methods (neighborhood models) and latent features from low-rank matrix factorization methods (factorization models).

Low-dimensional matrix recommenders try to capture the underlying features driving the raw data (which we understand as tastes and preferences). From a theoretical perspective, if we want to make recommendations based on people's tastes, this seems like the better approach. This technique also scales **significantly** better to larger datasets.

However, we still likely lose some meaningful signals by using a lower-rank matrix. And though these factorization based techniques work extremely well, there's research being done on new methods. These efforts have resulted in various types probabilistic matrix factorization (which works and scales even better) and many other approaches.

One particularly cool and effective strategy is to combine factorization and neighborhood methods into one [framework](http://www.cs.rochester.edu/twiki/pub/Main/HarpSeminar/Factorization_Meets_the_Neighborhood-_a_Multifaceted_Collaborative_Filtering_Model.pdf). This research field is extremely active, and I highly recommend Joseph Konstan's Coursera course, [Introduction to Recommender Systems](https://www.coursera.org/specializations/recommender-systems), for anyone looking to get a high level overview of the field. The optional readings are influential papers in the field from the last 15-ish years, and they're really cool.

***
For those interested, the Jupyter Notebook with all the code can be found in the [Github repository](https://github.com/beckernick/matrix_factorization_recommenders) for this post.