# Multi-output Gaussian processes in GPflow

This notebook shows how to construct a multi-output GP model using GPflow, together with different interdomain inducing variables which lead to different approximation properties. GPflow provides a framework for specifying multioutput GP priors, and interdomain approximations which is
- modular, by providing a consistent interface for the user of the resulting `SVGP` model,
- extensible, by allowing new interdomain variables and kernels to be specified while reusing exising code where possible,
- efficient, by allowing the most efficient custom code path to be specified where desired.

Getting to grips with the maths and code can be a bit daunting, so to accompany the documentation there is an [in-depth review on arXiv](https://arxiv.org/abs/2003.01115), which provides a unified mathematical framework, together with a high-level description of software design choices in GPflow.

This notebook shows the various design choices that can be made, to show the reader the flexibility of the framework. This is done in the hope that an example is provided that can be easily adapted to the special case that the reader wants to implement.

A reader who just wants to use a multioutput kernel should simply choose the most efficient set of inducing variables.

To cite this framework, please reference our [arXiv paper](https://arxiv.org/abs/2003.01115).
```
@article{GPflow2020multioutput,
  author = {{van der Wilk}, Mark and Dutordoir, Vincent and John, ST and
            Artemev, Artem and Adam, Vincent and Hensman, James},
  title = {A Framework for Interdomain and Multioutput {G}aussian Processes},
  year = {2020},
  journal = {arXiv:2003.01115},
  url = {https://arxiv.org/abs/2003.01115}
}
```

\begin{equation}
\newcommand{\GP}{\mathcal{GP}}
\newcommand{\NN}{\mathcal{N}}
\newcommand{\LL}{\mathcal{L}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\valpha}{\boldsymbol\alpha}
\newcommand{\vf}{\mathbf{f}}
\newcommand{\vF}{\mathbf{F}}
\newcommand{\vg}{\mathbf{g}}
\newcommand{\vW}{\mathbf{W}}
\newcommand{\vI}{\mathbf{I}}
\newcommand{\vZ}{\mathbf{Z}}
\newcommand{\vu}{\mathbf{u}}
\newcommand{\vU}{\mathbf{U}}
\newcommand{\vX}{\mathbf{X}}
\newcommand{\vY}{\mathbf{Y}}
\newcommand{\identity}{\mathbb{I}}
\end{equation}

## Task
We will consider a regression problem for functions $f: \mathbb{R}^D \rightarrow \mathbb{R}^P$. We assume that the dataset is of the form $(X, f_1), \dots, (X, f_P)$, that is, we observe all the outputs for a particular input location (for cases where there are **not** fully observed outputs for each input, see [A simple demonstration of coregionalization](./coregionalisation.ipynb)).

Here we assume a model of the form:
\begin{equation}
f(x) = W g(x),
\end{equation}
where $g(x) \in \mathbb{R}^L$, $f(x) \in \mathbb{R}^P$ and $W \in \mathbb{R}^{P \times L}$. We assume that the outputs of $g$ are uncorrelated, and that by *mixing* them with $W$ they become correlated. In this notebook, we show how to build this model using Sparse Variational Gaussian Process (SVGP) for $g$, which scales well with the numbers of data points and outputs.

Here we have two options for $g$:
1. The output dimensions of $g$ share the same kernel.
1. Each output of $g$ has a separate kernel.

In addition, we have two further suboptions for the inducing inputs of $g$:
1. The instances of $g$ share the same inducing inputs.
1. Each output of $g$ has its own set of inducing inputs.

The notation is as follows:
- $X \in \mathbb{R}^{N \times D}$ denotes the input
- $Y \in \RR^{N \times P}$ denotes the output
- $k_{1..L}$, $L$ are kernels on $\RR^{N \times D}$
- $g_{1..L}$, $L$ are independent $\GP$s  with $g_l \sim \GP(0,k_l)$
- $f_{1..P}$, $P$ are correlated  $\GP$s  with $\vf = \vW \vg$

```python
import numpy as np
import matplotlib.pyplot as plt
import gpflow as gpf
import tensorflow as tf

from gpflow.utilities import print_summary
from gpflow.ci_utils import ci_niter

gpf.config.set_default_float(np.float64)
gpf.config.set_default_summary_fmt("notebook")
np.random.seed(0)
%matplotlib inline

MAXITER = ci_niter(2000)
```

## Generate synthetic data
We create a utility function to generate synthetic data. We assume that:

```python
N = 100  # number of points
D = 1  # number of input dimensions
M = 15  # number of inducing points
L = 2  # number of latent GPs
P = 3  # number of observations = output dimensions
```

```python
def generate_data(N=100):
    X = np.random.rand(N)[:, None] * 10 - 5  # Inputs = N x D
    G = np.hstack((0.5 * np.sin(3 * X) + X, 3.0 * np.cos(X) - X))  # G = N x L
    W = np.array([[0.5, -0.3, 1.5], [-0.4, 0.43, 0.0]])  # L x P
    F = np.matmul(G, W)  # N x P
    Y = F + np.random.randn(*F.shape) * [0.2, 0.2, 0.2]

    return X, Y
```

```python
X, Y = data = generate_data(N)
Zinit = np.linspace(-5, 5, M)[:, None]
```

We create a utility function for plotting:

```python
def plot_model(m, lower=-8.0, upper=8.0):
    pX = np.linspace(lower, upper, 100)[:, None]
    pY, pYv = m.predict_y(pX)
    if pY.ndim == 3:
        pY = pY[:, 0, :]
    plt.plot(X, Y, "x")
    plt.gca().set_prop_cycle(None)
    plt.plot(pX, pY)
    for i in range(pY.shape[1]):
        top = pY[:, i] + 2.0 * pYv[:, i] ** 0.5
        bot = pY[:, i] - 2.0 * pYv[:, i] ** 0.5
        plt.fill_between(pX[:, 0], top, bot, alpha=0.3)
    plt.xlabel("X")
    plt.ylabel("f")
    plt.title(f"ELBO: {m.elbo(data):.3}")
    plt.plot(Z, Z * 0.0, "o")
```

## Model the outputs of $f(x)$ directly
The three following examples show how to model the outputs of the model $f(x)$ directly. Mathematically, this case is equivalent to having:
\begin{equation}
f(x) = I g(x),
\end{equation}
i.e. $W = I$ and $P = L$.

### 1. Shared independent multi-output kernel (MOK) and shared independent inducing variables
Here the priors on all outputs are constrained to have the same kernel hyperparameters. We also share the inducing inputs between all outputs. The different GPs are independent both in the prior and the approximate posterior.

```python
# create multi-output kernel
kernel = gpf.kernels.SharedIndependent(
    gpf.kernels.SquaredExponential() + gpf.kernels.Linear(), output_dim=P
)
# initialization of inducing input locations (M random points from the training inputs)
Z = Zinit.copy()
# create multi-output inducing variables from Z
iv = gpf.inducing_variables.SharedIndependentInducingVariables(
    gpf.inducing_variables.InducingPoints(Z)
)
```

```python
# create SVGP model as usual and optimize
m = gpf.models.SVGP(kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, num_latent_gps=P)
print_summary(m)
```

name                                      	class    	transform       	prior  	trainable  	shape      	dtype  	value

SVGP.kernel.kernel.kernels[0].variance    	Parameter	Softplus        	       	True       	()         	float64	1.0

SVGP.kernel.kernel.kernels[0].lengthscales	Parameter	Softplus        	       	True       	()         	float64	1.0

SVGP.kernel.kernel.kernels[1].variance    	Parameter	Softplus        	       	True       	()         	float64	1.0

SVGP.likelihood.variance                  	Parameter	Softplus + Shift	       	True       	()         	float64	1.0

SVGP.inducing_variable.inducing_variable.Z	Parameter	Identity        	       	True       	(15, 1)    	float64	[[-5....

SVGP.q_mu                                 	Parameter	Identity        	       	True       	(15, 3)    	float64	[[0., 0., 0....

SVGP.q_sqrt                               	Parameter	FillTriangular  	       	True       	(3, 15, 15)	float64	[[[1., 0., 0....

```python
def optimize_model_with_scipy(model):
    optimizer = gpf.optimizers.Scipy()
    optimizer.minimize(
        model.training_loss_closure(data),
        variables=model.trainable_variables,
        method="l-bfgs-b",
        options={"disp": True, "maxiter": MAXITER},
    )

optimize_model_with_scipy(m)
```

```python
print_summary(m)
```

name                                      	class    	transform       	prior  	trainable  	shape      	dtype  	value

SVGP.kernel.kernel.kernels[0].variance    	Parameter	Softplus        	       	True       	()         	float64	0.9273340728744635

SVGP.kernel.kernel.kernels[0].lengthscales	Parameter	Softplus        	       	True       	()         	float64	0.7982225737914499

SVGP.kernel.kernel.kernels[1].variance    	Parameter	Softplus        	       	True       	()         	float64	1.213461789503938

SVGP.likelihood.variance                  	Parameter	Softplus + Shift	       	True       	()         	float64	0.03966248736927796

SVGP.inducing_variable.inducing_variable.Z	Parameter	Identity        	       	True       	(15, 1)    	float64	[[-4.83957914...

SVGP.q_mu                                 	Parameter	Identity        	       	True       	(15, 3)    	float64	[[-0.90112327, 0.69862769, -1.47416329...

SVGP.q_sqrt                               	Parameter	FillTriangular  	       	True       	(3, 15, 15)	float64	[[[1.71169882e-02, 0.00000000e+00, 0.00000000e+00...

```python
# Plot predictions and observations
plot_model(m)
```

```python
print_summary(m.kernel)
m.kernel.kernel.kernels[0].lengthscales
```

name                                            	class    	transform  	prior  	trainable  	shape  	dtype  	   value

SharedIndependent.kernel.kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	0.927334

SharedIndependent.kernel.kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.798223

SharedIndependent.kernel.kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	1.21346

    <Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.7982225737914499>

### 2. Separate independent MOK and shared independent inducing variables
Here we allow different hyperparameters for the priors of each output. We still share the inducing inputs between all outputs.

```python
# Create list of kernels for each output
kern_list = [gpf.kernels.SquaredExponential() + gpf.kernels.Linear() for _ in range(P)]
# Create multi-output kernel from kernel list
kernel = gpf.kernels.SeparateIndependent(kern_list)
# initialization of inducing input locations (M random points from the training inputs)
Z = Zinit.copy()
# create multi-output inducing variables from Z
iv = gpf.inducing_variables.SharedIndependentInducingVariables(
    gpf.inducing_variables.InducingPoints(Z)
)
```

```python
# create SVGP model as usual and optimize
m = gpf.models.SVGP(kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, num_latent_gps=P)
```

```python
optimize_model_with_scipy(m)
```

    WARNING:tensorflow:From /home/circleci/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:605: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Use fn_output_signature instead

```python
print_summary(m.kernel)
```

name                                                  	class    	transform  	prior  	trainable  	shape  	dtype  	   value

SeparateIndependent.kernels[0].kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	0.863554

SeparateIndependent.kernels[0].kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.934669

SeparateIndependent.kernels[0].kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	0.861487

SeparateIndependent.kernels[1].kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	0.750967

SeparateIndependent.kernels[1].kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.969749

SeparateIndependent.kernels[1].kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	0.529794

SeparateIndependent.kernels[2].kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	1.11168

SeparateIndependent.kernels[2].kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.749168

SeparateIndependent.kernels[2].kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	2.22002

```python
plot_model(m)
```

```python
[k.kernels[0].lengthscales for k in m.kernel.kernels]
```

    [<Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.934668596843637>,
     <Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.969748605016044>,
     <Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.7491682608215007>]

### 3. Separate independent kernel and separate independent inducing variables
Here we allow different hyperparameters for the priors of each output. We now allow different inducing inputs for each output.

```python
# Create list of kernels for each output
kern_list = [gpf.kernels.SquaredExponential() + gpf.kernels.Linear() for _ in range(P)]
# Create multi-output kernel from kernel list
kernel = gpf.kernels.SeparateIndependent(kern_list)
# initialization of inducing input locations, one set of locations per output
Zs = [Zinit.copy() for _ in range(P)]
# initialize as list inducing inducing variables
iv_list = [gpf.inducing_variables.InducingPoints(Z) for Z in Zs]
# create multi-output inducing variables from iv_list
iv = gpf.inducing_variables.SeparateIndependentInducingVariables(iv_list)
```

**NOTE:** While the inducing points are independent, there needs to be the same number of inducing points per dimension.

```python
# create SVGP model as usual and optimize
m = gpf.models.SVGP(kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, num_latent_gps=P)
```

```python
optimize_model_with_scipy(m)
```

```python
plot_model(m)
```

The following plot shows that we use different inducing *inputs* in each output dimension.

```python
for i in range(len(m.inducing_variable.inducing_variable_list)):
    q_mu_unwhitened, q_var_unwhitened = m.predict_f(m.inducing_variable.inducing_variable_list[i].Z)
    plt.plot(
        m.inducing_variable.inducing_variable_list[i].Z.numpy(),
        q_mu_unwhitened[:, i, None].numpy(),
        "o",
    )
plt.gca().set_xticks(np.linspace(-6, 6, 20), minor=True)
plt.gca().set_yticks(np.linspace(-9, 9, 20), minor=True)
plt.grid(which="minor")
```

```python
m.inducing_variable.inducing_variable_list
```

    ListWrapper([<gpflow.inducing_variables.inducing_variables.InducingPoints object at 0x7f804c125ef0>, <gpflow.inducing_variables.inducing_variables.InducingPoints object at 0x7f804c1259b0>, <gpflow.inducing_variables.inducing_variables.InducingPoints object at 0x7f804c11dbe0>])

## Model $f(x)$ by doing inference in the $g$ space
### Mixed kernel and uncorrelated inducing variables

Remember the general case: $f(x) = W g(x)$, where $g(x) \in \mathbb{R}^L$, $f(x) \in \mathbb{R}^P$ and $W \in \mathbb{R}^{P \times L}$, where $L \leq P$.
We assume that the outputs of $g$ are uncorrelated, and by *mixing* them with $W$ they become correlated.
With this setup we perform the optimal routine to calculate the conditional. Namely, calculate the conditional of the uncorrelated latent $g$ and afterwards project the mean and variance using the mixing matrix: $\mu_f = W \mu_g$ and $\Sigma_f = W\Sigma_g W^\top$

- $K_{uu} = L \times M \times M$
- $K_{uf} = L \times M \times N$

```python
# Create list of kernels for each output
kern_list = [gpf.kernels.SquaredExponential() + gpf.kernels.Linear() for _ in range(L)]
# Create multi-output kernel from kernel list
kernel = gpf.kernels.LinearCoregionalization(
    kern_list, W=np.random.randn(P, L)
)  # Notice that we initialise the mixing matrix W
# initialisation of inducing input locations (M random points from the training inputs)
Z = Zinit.copy()
# create multi-output inducing variables from Z
iv = gpf.inducing_variables.SharedIndependentInducingVariables(
    gpf.inducing_variables.InducingPoints(Z)
)
```

```python
# initialize mean of variational posterior to be of shape MxL
q_mu = np.zeros((M, L))
# initialize \sqrt(Σ) of variational posterior to be of shape LxMxM
q_sqrt = np.repeat(np.eye(M)[None, ...], L, axis=0) * 1.0

# create SVGP model as usual and optimize
m = gpf.models.SVGP(
    kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, q_mu=q_mu, q_sqrt=q_sqrt
)
```

```python
optimize_model_with_scipy(m)
```

```python
plot_model(m)
```

## Illustration of GPflow's multi-output capabilities
This section shows the inheritance structure in GPflow's multi-output framework.

### Multi-output kernels (MOK) class diagram
We include three multi-output kernels:
- `SharedIndependent`: This kernel is included mainly as an illustration of specifying a conditional using the multiple dispatch framework. The same functionality is provided by using a normal kernel and passing in multiple approximate posteriors by stacking `q_mu` and `q_sqrt`.
- `SeparateIndependent`: This kernel allows you to use different priors for each output GP.
- `LinearCoregionalization`: This kernel describes the prior of the linear model of coregionalization. As shown previously, this implementation supports various inducing point approximations.

We include several base classes. Two are noteworthy:
- `MultioutputKernel` is included to be the base class for all multi-output kernels.
- `IndepedentLatent` is the base class for all multi-output kernels which are constructed from independent latent processes. Including this kernel allows the specification of a default approximation method which, while not the most efficient, does take advantage of _some_ structure. It can be applied to _any_ kernel constructed from independent latent processes.

There is a similarity in the meaning of `SeparateIndependent` and `IndependentLatent`. Both kernels indicate that independent processes are used, and that $\mathbf{K}_{\bf uu}$ can therefore be represented as a `[L, M, M]` tensor. It could therefore be suggested that `SeparateIndependent` be the parent class of all "independent latent" kernels, instead of having a separate `IndependentLatent` class. We decided against this because:
- this would increase the complexity in specifying `conditionals()` for the otherwise simple multi-output kernels `SeparateIndependent` and `SharedIndependent`.
- we did not want to specify too much of an implementation in `IndependentLatent`, leaving implementation details to child classes. Using `SeparateIndependent` as the base class would force all child classes to be a `Combination` kernel.

### Multi-output inducing variables class diagram

#### Inducing points
The goal of this class is to provide inducing variables that can be used with _any_ kernel, even if the method ends up being slow.

The multiouput framework extends `InducingPoints` to work with multi-output kernels. Just like for single-output kernels, we want `InducingPoints` to work for all `MultioutputKernel`s. We do this by defining `InducingPoints` to take _all_ outputs for specific inducing inputs as inducing variables.

#### Fallback shared/separate independent inducing variables
The goal of these classes is to provide a reasonably efficient implementation for kernels that give exploitable independence structure in the prior of inducing variables (that is, subclasses of `IndependentLatent`), while only needing to implement `Kuu()` and `Kuf()` methods.

#### Shared/separate independent inducing variables
The goal of these classes is to provide the most efficient code path for kernels that allow exploiting independence structure in the prior of inducing variables.

For more specialized multi-output kernels (i.e. `{Shared|Separate}Independent` or `LinearCoregionalization`) we define `{Shared|Separate}IndependentInducingVariables`. These wrap (a list of) single-output inducing variables to define groups of a-priori independent inducing variables, which leads to a $\mathbf{K}_{\bf uu}$ that can be represented as a `[L, M, M]` tensor. We saw the use of these previously.

`{Shared|Separate}IndependentInducingVariables` inherit from `Fallback{Shared|Separate}IndependentInducingVariables`, so the multiple dispatch will fall back on the slower but general implementation.

### Implemented combinations
Multiple dispatch is applied to both `Kuu()`, `Kuf()`, and `conditional()`. The return values of the covariances can therefore be tailored to a specific implementation of `conditional()`. The following table lists combinations which are currently available in GPflow. Thanks to the multiple dispatch code, implementing your own outside of GPflow should require only a small amount of code!

| Inducing variable class                                      | Kernel                  | Kuu           | Kuf           | conditional                         | note                                                                                |
|----------------------------------------------|-------------------------|---------------|---------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `InducingPoints`                               | `MultioutputKernel`       | `[M, P, M, P]` | `[M, P, N, P]` | `inducing_point_conditional()`, which calls `fully_correlated_conditional()`       | Works for all kernels, but might be very inefficient. In this case `q_mu` and `q_sqrt` should have shapes of `[1, MP]` and `[1, MP, MP]`  |
| `SharedIndependentInducingVariables`           | `SharedIndependent`       | `[M, M]`         | `[M, N]`         | `shared_independent_conditional()`, which calls `base_conditional()`                    | The combination of these two classes is in a sense redundant, because we can achieve the same behavior by using the single output Kernel and InducingVariable classes. They are added for illustrative purposes. Thanks to the conditional dispatch, the most efficient code path is used. |
| `SeparateIndependentInducingVariables`         | `SharedIndependent`       | `[P, M, M]`     | `[P, M, N]`     | `separate_independent_conditional()`, which calls `base_conditional()` P times               | We loop P times over the `base_conditional()`         |
| `SeparateIndependentInducingVariable`         | `SeparateIndependent`     | `[P, M, M]`     | `[P, M, N]`     |  `separate_independent_conditional()`, which calls `base_conditional()` P times                | We loop P times over the `base_conditional()`           |
| `SharedIndependentInducingVariables`           | `SeparateIndependent`     | `[P, M, M]`     | `[P, M, N]`     | `separate_independent_conditional()`, which calls `base_conditional()` P times                | We loop P times over the `base_conditional()`        |
| `FallbackSharedIndependentInducingVariables`   | `IndependentLatent`       | `[L, M, M]`     | `[M, L, N, P]` | `fallback_independent_latent_conditional()`, which calls `independent_interdomain_conditional()` | Implementation which only requires custom `Kuu()` and `Kuf()` |
| `FallbackSeparateIndependentInducingVariable` | `IndependentLatent`       | `[L, M, M]`     | `[M, L, N, P]` | `fallback_independent_latent_conditional()`, which calls `independent_interdomain_conditional()` | Implementation which only requires custom `Kuu()` and `Kuf()`  |
| `SharedIndependentInducingVariables`           | `LinearCoregionalization` | `[L, M, M]`     | `[L, M, N]`     | `coregionalization_conditional()`, which calls `base_conditional()`                    | This is the most efficient implementation for linear coregionalization. The inducing outputs live in g-space. Here we use the output of the base conditional and project the mean and covariance with the mixing matrix W.                                                                    |
| `SeparateIndependentInducingVariables`         | `LinearCoregionalization` | `[L, M, M]`     | `[L, M, N]`     | base_conditional                    | This is the most efficient implementation for linear coregionalization. The inducing outputs live in g-space. Here we use the output of the base conditional and project the mean and covariance with the mixing matrix W.                                                                    |

## Debugging: introspect

Given all these possibilities it can be hard to determine which conditional will be called for which set of kernel and inducing variable. The following method lets you proactively introspect which implementation will be executed. This can be useful when debugging new code.

```python
def inspect_conditional(inducing_variable_type, kernel_type):
    """
    Helper function returning the exact implementation called
    by the multiple dispatch `conditional` given the type of
    kernel and inducing variable.

    :param inducing_variable_type:
        Type of the inducing variable
    :param kernel_type:
        Type of the kernel

    :return: String
        Contains the name, the file and the linenumber of the
        implementation.
    """
    import inspect
    from gpflow.conditionals import conditional

    implementation = conditional.dispatch(object, inducing_variable_type, kernel_type, object)
    info = dict(inspect.getmembers(implementation))
    return info["__code__"]

# Example:
inspect_conditional(
    gpf.inducing_variables.SharedIndependentInducingVariables, gpf.kernels.SharedIndependent
)
```

## Further Reading:
- [A simple demonstration of coregionalization](./coregionalisation.ipynb), which details other GPflow features for multi-output prediction without fully observed outputs.


# Collaborative Filtering
* MovieLens Dataset
* Recompy Library ( https://github.com/CanBul/recompy )
* Matrix Factorisation
* Model Based (FunkSVD)

## User-Item Matrix

```python
%%capture
!pip install recompy
```

```python
from recompy import load_movie_data
import pandas as pd

data = load_movie_data()
```

```python
print(data.shape)
data
```

    (100000, 3)

    array([[1.96e+02, 2.42e+02, 3.00e+00],
           [1.86e+02, 3.02e+02, 3.00e+00],
           [2.20e+01, 3.77e+02, 1.00e+00],
           ...,
           [2.76e+02, 1.09e+03, 1.00e+00],
           [1.30e+01, 2.25e+02, 2.00e+00],
           [1.20e+01, 2.03e+02, 3.00e+00]])

```python
df_ratings = pd.DataFrame(data, columns = ['userId', 'itemId', 'rating'])
```

```python
df_ratings.head()
```

      userId
      itemId
      rating

      0
      196.0
      242.0
      3.0

      1
      186.0
      302.0
      3.0

      2
      22.0
      377.0
      1.0

      3
      244.0
      51.0
      2.0

      4
      166.0
      346.0
      1.0

```python
df_ratings = df_ratings.astype(int)
```

```python
df_ratings_pivot = df_ratings.pivot(index='itemId', columns='userId', values='rating')
```

User-Item Matrix is very sparse as you can see. What can we do?
* We can do nothing, keep original matrix and just take intersections while computing similarities.
* We can fill NaN cells with global mean.
* We can fill NaN cells with user's mean.

```python
df_ratings_pivot
```

      userId
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      ...
      904
      905
      906
      907
      908
      909
      910
      911
      912
      913
      914
      915
      916
      917
      918
      919
      920
      921
      922
      923
      924
      925
      926
      927
      928
      929
      930
      931
      932
      933
      934
      935
      936
      937
      938
      939
      940
      941
      942
      943

      itemId

      1
      5.0
      4.0
      NaN
      NaN
      4.0
      4.0
      NaN
      NaN
      NaN
      4.0
      NaN
      NaN
      3.0
      NaN
      1.0
      5.0
      4.0
      5.0
      NaN
      3.0
      5.0
      NaN
      5.0
      NaN
      5.0
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      5.0
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      5.0
      NaN
      NaN
      4.0
      NaN
      NaN
      2.0
      NaN
      NaN
      4.0
      3.0
      3.0
      4.0
      NaN
      3.0
      5.0
      3.0
      5.0
      NaN
      NaN
      5.0
      NaN
      3.0
      3.0
      NaN
      4.0
      3.0
      2.0
      3.0
      4.0
      NaN
      4.0
      NaN
      NaN
      5.0
      NaN
      NaN

      2
      3.0
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      2.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      4.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      5.0

      3
      4.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      2.0
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      1.0
      NaN
      NaN
      NaN
      NaN
      NaN
      4.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      4.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      4
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      5.0
      NaN
      NaN
      4.0
      NaN
      5.0
      5.0
      NaN
      NaN
      5.0
      NaN
      3.0
      4.0
      NaN
      NaN
      5.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      4.0
      NaN
      NaN
      4.0
      NaN
      NaN
      1.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      5.0
      NaN
      NaN
      NaN
      NaN
      NaN
      2.0
      NaN
      NaN
      NaN

      5
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      1.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      2.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      5.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      4.0
      NaN
      NaN
      NaN
      NaN
      NaN
      4.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1678
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      1679
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      1680
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      1681
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      1682
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

1682 rows × 943 columns

* Null Interaction Rate

```python
df_ratings_pivot.isna().sum().sum()/(df_ratings_pivot.shape[0]*df_ratings_pivot.shape[1])
```

    0.9369533063577546

* How many movies have watched and rated by an average person?

```python
(1682 - df_ratings_pivot.isna().sum()).mean()
```

    106.04453870625663

* How many people watched and rated an average movie?

```python
(943 - df_ratings_pivot.isna().T.sum()).mean()
```

    59.45303210463734

## FuncSVD Model

* **S**ingular **V**alue **D**ecomposition :
    * Data Reduction
    * Basis PCA (Key Correlations)
* **FunkSVD :** Funk SVD decomposes a matrix (with missing values) into two components \(U\) and \(V\). The singular values are folded into these matrices. The approximation for the original matrix can be obtained by **\(R = UV'\)**.

```python
class FunkSVD():

    def __init__(self):
        # Initialize default hyperparameters
        self.set_hyperparameters()

    def set_hyperparameters(self, initialization_method='random', max_epoch=5, n_latent=10, learning_rate=0.01, regularization=0.1, early_stopping=False, init_mean=0, init_std=1):
        """Initialization method, epoch num, latent feature num, learning rate,
           regularization, early stopping condition...
        """
        self.initialization_method = initialization_method
        self.max_epoch = max_epoch
        self.n_latent = n_latent
        self.learning_rate = learning_rate
        self.regularization = regularization
        self.early_stopping = early_stopping
        self.init_mean = init_mean
        self.init_std = init_std

        self.min_train_error = np.inf
        self.min_test_error = np.inf

    # unique users, unique items, train-test split, user informed etc.
    def __set_data(self, data, test_portion):

        # get distinct users, items and user_existing_ratings, items_existing_users
        self.user_existing_ratings = {}
        self.items_rated_by_users = {}
        self.user_ids = []
        self.item_ids = []

        np.random.shuffle(data)

        # variables for train and test split
        user_dictionary = {}
        item_dictionary = {}
        self.train_data = []
        self.test_data = []

        self.train_data_user_ids = []
        self.train_data_item_ids = []
        self.test_data_user_ids = []
        self.test_data_item_ids = []

        for user, item, score in data:
            # Unique users and items

            try:
                user = int(user)
            except:
                pass
            try:
                item = int(item)
            except:
                pass

            user = str(user)
            item = str(item)
            score = float(score)

            if user not in self.user_existing_ratings:
                self.user_ids.append(user)
            if item not in self.items_rated_by_users:
                self.item_ids.append(item)

            self.items_rated_by_users.setdefault(item, []).append(user)
            self.user_existing_ratings.setdefault(user, []).append(item)

            ratio = len(self.test_data) / (len(self.train_data)+0.001)

            if self.test_split:
                # train and test set
                user_dictionary.setdefault(user, 0)
                item_dictionary.setdefault(item, 0)

                if user_dictionary[user] * test_portion >= 1 and item_dictionary[item] * test_portion >= 1 and ratio <= test_portion+0.02:

                    self.test_data.append([user, item, score])
                    if user not in self.test_data_user_ids: self.test_data_user_ids.append(user)
                    if item not in self.train_data_item_ids: self.test_data_item_ids.append(item)

                    user_dictionary[user] -= 1
                    item_dictionary[item] -= 1

                else:
                    self.train_data.append([user, item, score])
                    if user not in self.train_data_user_ids: self.train_data_user_ids.append(user)
                    if item not in self.train_data_item_ids: self.train_data_item_ids.append(item)

                    user_dictionary[user] += 1
                    item_dictionary[item] += 1
            else:
                self.train_data.append([user, item, score])
                if user not in self.train_data_user_ids: self.train_data_user_ids.append(user)
                if item not in self.train_data_item_ids: self.train_data_item_ids.append(item)

        print('Your data has {} distinct users and {} distinct items.'.format(
            len(self.user_ids), len(self.item_ids)))

        if len(self.test_data) < 1 and self.test_split:
            self.test_split = False
            self.early_stopping = False
            print("Training set doesn't have enough data for given test portion.")

        if self.test_split:

            print('Your data has been split into train and test set.')
            print('Length of training set is {}. Length of Test set is {}'.format(
                len(self.train_data), len(self.test_data)))
        else:

            print('Your data has no test set.')
            print('Length of training set is {}'.format(len(self.train_data)))

    def fit(self, data, test_split=True, test_portion=0.1, search_parameter_space=False):

        # Set train_data, test_data, user_ids etc. if search parameter is False
        # If True, this lets us search parameter space with the same train-test split
        if not search_parameter_space:

            self.test_split = test_split
            self.__set_data(data, test_portion)

        # Initialization
        print('Initializing features for Users and Items...')
        initial = initializer(self.user_ids, self.item_ids, self.initialization_method,
                              self.n_latent, self.init_mean, self.init_std)

        self.user_features, self.item_features = initial.initialize_latent_vectors()

        # Training
        print('Starting training...')
        error_counter = 0
        for epoch in range(self.max_epoch):

            # updating user and item features
            for user, item, rating in self.train_data:

                error = rating - \
                    np.dot(self.user_features[user], self.item_features[item])
                # Use temp to update each item and user feature in sync.
                temp = self.user_features[user]

                # Update user and item feature for each user, item and rating pair
                self.user_features[user] += self.learning_rate * \
                    (error * self.item_features[item] -
                     self.regularization * self.user_features[user])
                self.item_features[item] += self.learning_rate * \
                    (error * temp - self.regularization *
                     self.item_features[item])

            # Calculate errors
            error_counter += 1
            train_error = Test.rmse_error(
                self.train_data, self.user_features, self.item_features)

            # Show error to Client
            if self.test_split:
                test_error = Test.rmse_error(
                    self.test_data, self.user_features, self.item_features)
                print('Epoch Number: {}/{} Training RMSE: {:.2f} Test RMSE: {}'.format(epoch+1, self.max_epoch,
                                                                                train_error, test_error))

            else:
                print('Epoch Number: {}/{} Training RMSE: {:.2f}'.format(epoch+1, self.max_epoch,
                                                                         train_error))

            # Save best features depending on test_error
            if self.test_split and test_error < self.min_test_error:
                self.min_test_error = test_error
                self.best_user_features = copy.deepcopy(self.user_features)
                self.best_item_features = copy.deepcopy(self.item_features)

                error_counter = 0
            # Save best features if test data is False
            elif not self.test_split and train_error < self.min_train_error:
                self.min_train_error = train_error
                self.best_user_features = copy.deepcopy(self.user_features)
                self.best_item_features = copy.deepcopy(self.item_features)

            # Break if test_error didn't improve for the last n rounds and early stopping is true
            if self.early_stopping and error_counter >= self.early_stopping:

                print("Test error didn't get lower for the last {} epochs. Training is stopped.".format(
                    error_counter))
                print('Best test error is: {:.2f}. Best features are saved.'.format(
                    self.min_test_error))
                break

        print('Training has ended...')
        self.user_features = copy.deepcopy(self.best_user_features)
        self.item_features = copy.deepcopy(self.best_item_features)

    def get_recommendation_for_existing_user(self, user_id, howMany=10):
        result_list = []
        # this might be more effective using matrix multiplication
        for item in self.item_ids:
            # if user did not already rate the item
            if item not in self.user_existing_ratings[user_id]:
                prediction = np.dot(
                    self.user_features[user_id], self.item_features[item])
                bisect.insort(result_list, [prediction, item])

        return [x[1] for x in result_list[::-1][0:howMany]]

    def get_recommendation_for_new_user(self, user_ratings,
                                        similarity_measure='mean_squared_difference', howManyUsers=3, howManyItems=5):

        # Get user predictions on same movies
        user_predictions = self.__user_prediction_for_same_movies(user_ratings)
        # Find most most similar user_ids
        user_ids = Similarities.get_most_similar_users(
            user_ratings, user_predictions, similarity_measure, howManyUsers)

        result_list = []
        # get user features for users who are most similar to given new user
        for user in user_ids:
            for item, item_feature in self.item_features.items():
                # predict ratings for most similar users
                prediction = np.dot(
                    self.user_features[user], item_feature)
                bisect.insort(result_list, [prediction, item])

        # remove duplicates
        return_list = []
        for pair in result_list:
            if len(return_list) >= howManyItems:
                break
            if pair[1] in return_list:
                continue

            return_list.append(pair[1])

        return return_list

    def get_similar_products(self, item_id, howMany=10):

        result_list = []
        product_features = self.item_features[item_id]

        for item in self.item_ids:

            if item == item_id:
                continue
            # add cosine sim function from similarites
            cos_sim = Similarities.cosine_similarity(
                self.item_features[item], product_features)

            bisect.insort(result_list, [cos_sim, item])

        return [x[1] for x in result_list[::-1][0:howMany]]

    def __user_prediction_for_same_movies(self, user_ratings):
        result = {}
        for key in user_ratings:
            if key not in self.item_features:
                continue

            for user in self.user_features:
                result.setdefault(user, []).append(
                    np.dot(self.user_features[user], self.item_features[key]))

        return result
```

## Training

```python
from recompy import load_movie_data, FunkSVD

# get MovieLens data
data = load_movie_data()
# initialization of FunkSVD model
myFunk = FunkSVD()
myFunk.set_hyperparameters(max_epoch=200, n_latent=15,early_stopping=True, initialization_method='random', regularization=0.1)
# training of the model
myFunk.fit(data)
```

    Your data has 943 distinct users and 1682 distinct items.
    Your data has been split into train and test set.
    Length of training set is 89285. Length of Test set is 10715
    Initializing features for Users and Items...
    Starting training...
    Epoch Number: 1/200 Training RMSE: 0.96 Test RMSE: 0.9670916963503041
    Epoch Number: 2/200 Training RMSE: 0.93 Test RMSE: 0.9522431850824344
    Epoch Number: 3/200 Training RMSE: 0.92 Test RMSE: 0.9478483827565287
    Epoch Number: 4/200 Training RMSE: 0.91 Test RMSE: 0.9458730577563561
    Epoch Number: 5/200 Training RMSE: 0.91 Test RMSE: 0.9446572171923738
    Epoch Number: 6/200 Training RMSE: 0.91 Test RMSE: 0.9436758029458507
    Epoch Number: 7/200 Training RMSE: 0.90 Test RMSE: 0.9427027755865681
    Epoch Number: 8/200 Training RMSE: 0.90 Test RMSE: 0.9416069334010283
    Epoch Number: 9/200 Training RMSE: 0.90 Test RMSE: 0.9402940836122284
    Epoch Number: 10/200 Training RMSE: 0.89 Test RMSE: 0.9386958780769502
    Epoch Number: 11/200 Training RMSE: 0.89 Test RMSE: 0.9367785368439625
    Epoch Number: 12/200 Training RMSE: 0.88 Test RMSE: 0.9345579282754947
    Epoch Number: 13/200 Training RMSE: 0.88 Test RMSE: 0.9321064547279629
    Epoch Number: 14/200 Training RMSE: 0.87 Test RMSE: 0.9295392851989963
    Epoch Number: 15/200 Training RMSE: 0.87 Test RMSE: 0.9269811842185286
    Epoch Number: 16/200 Training RMSE: 0.86 Test RMSE: 0.9245318535704998
    Epoch Number: 17/200 Training RMSE: 0.86 Test RMSE: 0.9222488111243613
    Epoch Number: 18/200 Training RMSE: 0.85 Test RMSE: 0.9201511724475051
    Epoch Number: 19/200 Training RMSE: 0.85 Test RMSE: 0.9182339494966082
    Epoch Number: 20/200 Training RMSE: 0.84 Test RMSE: 0.9164817917728538
    Epoch Number: 21/200 Training RMSE: 0.84 Test RMSE: 0.9148775163566587
    Epoch Number: 22/200 Training RMSE: 0.83 Test RMSE: 0.9134058179016363
    Epoch Number: 23/200 Training RMSE: 0.83 Test RMSE: 0.9120541184601167
    Epoch Number: 24/200 Training RMSE: 0.83 Test RMSE: 0.9108121892141483
    Epoch Number: 25/200 Training RMSE: 0.82 Test RMSE: 0.9096714612279164
    Epoch Number: 26/200 Training RMSE: 0.82 Test RMSE: 0.9086244160926359
    Epoch Number: 27/200 Training RMSE: 0.81 Test RMSE: 0.9076641656860744
    Epoch Number: 28/200 Training RMSE: 0.81 Test RMSE: 0.9067842099911058
    Epoch Number: 29/200 Training RMSE: 0.81 Test RMSE: 0.9059783249803499
    Epoch Number: 30/200 Training RMSE: 0.80 Test RMSE: 0.9052405315860627
    Epoch Number: 31/200 Training RMSE: 0.80 Test RMSE: 0.9045651077569149
    Epoch Number: 32/200 Training RMSE: 0.80 Test RMSE: 0.9039466180494989
    Epoch Number: 33/200 Training RMSE: 0.80 Test RMSE: 0.9033799453824735
    Epoch Number: 34/200 Training RMSE: 0.79 Test RMSE: 0.9028603167892371
    Epoch Number: 35/200 Training RMSE: 0.79 Test RMSE: 0.9023833196610496
    Epoch Number: 36/200 Training RMSE: 0.79 Test RMSE: 0.9019449077592955
    Epoch Number: 37/200 Training RMSE: 0.79 Test RMSE: 0.9015413978075724
    Epoch Number: 38/200 Training RMSE: 0.78 Test RMSE: 0.9011694582104498
    Epoch Number: 39/200 Training RMSE: 0.78 Test RMSE: 0.9008260916997659
    Epoch Number: 40/200 Training RMSE: 0.78 Test RMSE: 0.9005086136857591
    Epoch Number: 41/200 Training RMSE: 0.78 Test RMSE: 0.9002146279175588
    Epoch Number: 42/200 Training RMSE: 0.78 Test RMSE: 0.8999420008161508
    Epoch Number: 43/200 Training RMSE: 0.77 Test RMSE: 0.8996888355811169
    Epoch Number: 44/200 Training RMSE: 0.77 Test RMSE: 0.899453446918486
    Epoch Number: 45/200 Training RMSE: 0.77 Test RMSE: 0.8992343370067665
    Epoch Number: 46/200 Training RMSE: 0.77 Test RMSE: 0.8990301731188818
    Epoch Number: 47/200 Training RMSE: 0.77 Test RMSE: 0.8988397671517254
    Epoch Number: 48/200 Training RMSE: 0.77 Test RMSE: 0.8986620571815404
    Epoch Number: 49/200 Training RMSE: 0.76 Test RMSE: 0.8984960910603107
    Epoch Number: 50/200 Training RMSE: 0.76 Test RMSE: 0.8983410119916525
    Epoch Number: 51/200 Training RMSE: 0.76 Test RMSE: 0.8981960459713191
    Epoch Number: 52/200 Training RMSE: 0.76 Test RMSE: 0.8980604909429976
    Epoch Number: 53/200 Training RMSE: 0.76 Test RMSE: 0.8979337075011932
    Epoch Number: 54/200 Training RMSE: 0.76 Test RMSE: 0.8978151109659706
    Epoch Number: 55/200 Training RMSE: 0.76 Test RMSE: 0.8977041646564616
    Epoch Number: 56/200 Training RMSE: 0.76 Test RMSE: 0.8976003741984914
    Epoch Number: 57/200 Training RMSE: 0.76 Test RMSE: 0.8975032827141567
    Epoch Number: 58/200 Training RMSE: 0.75 Test RMSE: 0.897412466756335
    Epoch Number: 59/200 Training RMSE: 0.75 Test RMSE: 0.8973275328669897
    Epoch Number: 60/200 Training RMSE: 0.75 Test RMSE: 0.8972481146541557
    Epoch Number: 61/200 Training RMSE: 0.75 Test RMSE: 0.8971738702978173
    Epoch Number: 62/200 Training RMSE: 0.75 Test RMSE: 0.8971044804088678
    Epoch Number: 63/200 Training RMSE: 0.75 Test RMSE: 0.8970396461780348
    Epoch Number: 64/200 Training RMSE: 0.75 Test RMSE: 0.8969790877627091
    Epoch Number: 65/200 Training RMSE: 0.75 Test RMSE: 0.8969225428690579
    Epoch Number: 66/200 Training RMSE: 0.75 Test RMSE: 0.8968697654949704
    Epoch Number: 67/200 Training RMSE: 0.75 Test RMSE: 0.8968205248059342
    Epoch Number: 68/200 Training RMSE: 0.75 Test RMSE: 0.8967746041215992
    Epoch Number: 69/200 Training RMSE: 0.75 Test RMSE: 0.896731799995145
    Epoch Number: 70/200 Training RMSE: 0.74 Test RMSE: 0.8966919213712353
    Epoch Number: 71/200 Training RMSE: 0.74 Test RMSE: 0.8966547888112006
    Epoch Number: 72/200 Training RMSE: 0.74 Test RMSE: 0.8966202337764598
    Epoch Number: 73/200 Training RMSE: 0.74 Test RMSE: 0.8965880979627466
    Epoch Number: 74/200 Training RMSE: 0.74 Test RMSE: 0.8965582326794762
    Epoch Number: 75/200 Training RMSE: 0.74 Test RMSE: 0.896530498269364
    Epoch Number: 76/200 Training RMSE: 0.74 Test RMSE: 0.8965047635644108
    Epoch Number: 77/200 Training RMSE: 0.74 Test RMSE: 0.8964809053751118
    Epoch Number: 78/200 Training RMSE: 0.74 Test RMSE: 0.8964588080100775
    Epoch Number: 79/200 Training RMSE: 0.74 Test RMSE: 0.8964383628238486
    Epoch Number: 80/200 Training RMSE: 0.74 Test RMSE: 0.8964194677909202
    Epoch Number: 81/200 Training RMSE: 0.74 Test RMSE: 0.896402027104227
    Epoch Number: 82/200 Training RMSE: 0.74 Test RMSE: 0.8963859507966574
    Epoch Number: 83/200 Training RMSE: 0.74 Test RMSE: 0.8963711543841437
    Epoch Number: 84/200 Training RMSE: 0.74 Test RMSE: 0.8963575585292598
    Epoch Number: 85/200 Training RMSE: 0.74 Test RMSE: 0.8963450887240556
    Epoch Number: 86/200 Training RMSE: 0.74 Test RMSE: 0.8963336749912736
    Epoch Number: 87/200 Training RMSE: 0.74 Test RMSE: 0.896323251602876
    Epoch Number: 88/200 Training RMSE: 0.74 Test RMSE: 0.8963137568151053
    Epoch Number: 89/200 Training RMSE: 0.74 Test RMSE: 0.8963051326191713
    Epoch Number: 90/200 Training RMSE: 0.74 Test RMSE: 0.896297324506856
    Epoch Number: 91/200 Training RMSE: 0.74 Test RMSE: 0.8962902812502018
    Epoch Number: 92/200 Training RMSE: 0.73 Test RMSE: 0.8962839546946862
    Epoch Number: 93/200 Training RMSE: 0.73 Test RMSE: 0.8962782995650908
    Epoch Number: 94/200 Training RMSE: 0.73 Test RMSE: 0.8962732732834883
    Epoch Number: 95/200 Training RMSE: 0.73 Test RMSE: 0.8962688357986784
    Epoch Number: 96/200 Training RMSE: 0.73 Test RMSE: 0.8962649494264936
    Epoch Number: 97/200 Training RMSE: 0.73 Test RMSE: 0.8962615787003615
    Epoch Number: 98/200 Training RMSE: 0.73 Test RMSE: 0.8962586902315758
    Epoch Number: 99/200 Training RMSE: 0.73 Test RMSE: 0.8962562525787305
    Epoch Number: 100/200 Training RMSE: 0.73 Test RMSE: 0.896254236125793
    Epoch Number: 101/200 Training RMSE: 0.73 Test RMSE: 0.8962526129683074
    Epoch Number: 102/200 Training RMSE: 0.73 Test RMSE: 0.8962513568072479
    Epoch Number: 103/200 Training RMSE: 0.73 Test RMSE: 0.8962504428500337
    Epoch Number: 104/200 Training RMSE: 0.73 Test RMSE: 0.8962498477182915
    Epoch Number: 105/200 Training RMSE: 0.73 Test RMSE: 0.8962495493619214
    Epoch Number: 106/200 Training RMSE: 0.73 Test RMSE: 0.8962495269790506
    Epoch Number: 107/200 Training RMSE: 0.73 Test RMSE: 0.8962497609414769
    Test error didn't get lower for the last 1 epochs. Training is stopped.
    Best test error is: 0.90. Best features are saved.
    Training has ended...

## Testing

```python
new_user = {'1':5,
            '2':4,
            '4':3}

# To find the most similar user resulting from cosine similarity. Recommend 5 items using the most similar user
myFunk.get_recommendation_for_new_user(new_user,
                                       similarity_measure = 'cosine_similarity',
                                       howManyUsers = 1, howManyItems = 5)
```

    ['1304', '1618', '906', '1254', '987']

# Content Based Filtering

* **Used Dataset :** https://www.kaggle.com/rounakbanik/the-movies-dataset/data
* **Latest MovieLens Dataset :** https://grouplens.org/datasets/movielens/latest/

**Files :**
  * _movies_metadata_ : Features belong to movies (~45k)
  * _keywords_ : Keywords extracted from plot of the movie
  * _credits_ : Cast and crew information
  * _links_ : TMDB and IMDB IDs of all movies
  * _ratings_ : User-Movie interactions

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

```python
import pandas as pd
from numpy import dot
from numpy.linalg import norm
import numpy as np

metadata = pd.read_csv('/content/drive/MyDrive/applied_ai_enes_safak/recommender_systems/MovieLens/movies_metadata.csv', low_memory=False)

metadata['overview'] = metadata['overview'].fillna('')

metadata.head(3)
```

      adult
      belongs_to_collection
      budget
      genres
      homepage
      id
      imdb_id
      original_language
      original_title
      overview
      popularity
      poster_path
      production_companies
      production_countries
      release_date
      revenue
      runtime
      spoken_languages
      status
      tagline
      title
      video
      vote_average
      vote_count

      0
      False
      {'id': 10194, 'name': 'Toy Story Collection', ...
      30000000
      [{'id': 16, 'name': 'Animation'}, {'id': 35, '...
      http://toystory.disney.com/toy-story
      862
      tt0114709
      en
      Toy Story
      Led by Woody, Andy's toys live happily in his ...
      21.946943
      /rhIRbceoE9lR4veEXuwCC2wARtG.jpg
      [{'name': 'Pixar Animation Studios', 'id': 3}]
      [{'iso_3166_1': 'US', 'name': 'United States o...
      1995-10-30
      373554033.0
      81.0
      [{'iso_639_1': 'en', 'name': 'English'}]
      Released
      NaN
      Toy Story
      False
      7.7
      5415.0

      1
      False
      NaN
      65000000
      [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...
      NaN
      8844
      tt0113497
      en
      Jumanji
      When siblings Judy and Peter discover an encha...
      17.015539
      /vzmL6fP7aPKNKPRTFnZmiUfciyV.jpg
      [{'name': 'TriStar Pictures', 'id': 559}, {'na...
      [{'iso_3166_1': 'US', 'name': 'United States o...
      1995-12-15
      262797249.0
      104.0
      [{'iso_639_1': 'en', 'name': 'English'}, {'iso...
      Released
      Roll the dice and unleash the excitement!
      Jumanji
      False
      6.9
      2413.0

      2
      False
      {'id': 119050, 'name': 'Grumpy Old Men Collect...
      0
      [{'id': 10749, 'name': 'Romance'}, {'id': 35, ...
      NaN
      15602
      tt0113228
      en
      Grumpier Old Men
      A family wedding reignites the ancient feud be...
      11.7129
      /6ksm1sjKMFLbO7UY2i6G1ju9SML.jpg
      [{'name': 'Warner Bros.', 'id': 6194}, {'name'...
      [{'iso_3166_1': 'US', 'name': 'United States o...
      1995-12-22
      0.0
      101.0
      [{'iso_639_1': 'en', 'name': 'English'}]
      Released
      Still Yelling. Still Fighting. Still Ready for...
      Grumpier Old Men
      False
      6.5
      92.0

## Scoring with Weighted Ratings

---
\begin{equation}
\text Weighted Rating (\bf WR) = \left({{\bf v} \over {\bf v} + {\bf m}} \cdot R\right) + \left({{\bf m} \over {\bf v} + {\bf m}} \cdot C\right)
\end{equation}

---

* v is the number of votes for the movie;

* m is the minimum votes required to be listed in the chart;

* R is the average rating of the movie;

* C is the mean vote across the whole report.

---

* Average rating of a movie on IMDB is around 5.6 on a scale of 10

```python
C = metadata['vote_average'].mean()
C
```

    5.618207215133889

* Only %10 of all movies rated more than 160 times

```python
m = metadata['vote_count'].quantile(0.90)
m
```

    160.0

```python
q_movies = metadata.copy().loc[metadata['vote_count'] >= m]
print(q_movies.shape)
print(metadata.shape)
```

    (4555, 24)
    (45466, 24)

```python
def weighted_rating(x, m=m, C=C):
    v = x['vote_count']
    R = x['vote_average']
    # Calculation based on the IMDB formula
    return (v/(v+m) * R) + (m/(m+v) * C)
```

```python
q_movies['score'] = q_movies.apply(weighted_rating, axis=1)
```

## Sort by Score

```python
q_movies = q_movies.sort_values('score', ascending=False)

q_movies[['title', 'vote_count', 'vote_average', 'score']].head(20).reset_index(drop=True)
```

      title
      vote_count
      vote_average
      score

      0
      The Shawshank Redemption
      8358.0
      8.5
      8.445869

      1
      The Godfather
      6024.0
      8.5
      8.425439

      2
      Dilwale Dulhania Le Jayenge
      661.0
      9.1
      8.421453

      3
      The Dark Knight
      12269.0
      8.3
      8.265477

      4
      Fight Club
      9678.0
      8.3
      8.256385

      5
      Pulp Fiction
      8670.0
      8.3
      8.251406

      6
      Schindler's List
      4436.0
      8.3
      8.206639

      7
      Whiplash
      4376.0
      8.3
      8.205404

      8
      Spirited Away
      3968.0
      8.3
      8.196055

      9
      Life Is Beautiful
      3643.0
      8.3
      8.187171

      10
      The Godfather: Part II
      3418.0
      8.3
      8.180076

      11
      One Flew Over the Cuckoo's Nest
      3001.0
      8.3
      8.164256

      12
      Forrest Gump
      8147.0
      8.2
      8.150272

      13
      The Empire Strikes Back
      5998.0
      8.2
      8.132919

      14
      Psycho
      2405.0
      8.3
      8.132715

      15
      The Intouchables
      5410.0
      8.2
      8.125837

      16
      Your Name.
      1030.0
      8.5
      8.112532

      17
      Leon: The Professional
      4293.0
      8.2
      8.107234

      18
      The Green Mile
      4166.0
      8.2
      8.104511

      19
      GoodFellas
      3211.0
      8.2
      8.077459

## **Overview** is the Content

* With **Term Frequency-Inverse Document Frequency (TF-IDF)** Analysis

```python
pd.DataFrame(metadata[['original_title','overview']].head())
```

      original_title
      overview

      0
      Toy Story
      Led by Woody, Andy's toys live happily in his ...

      1
      Jumanji
      When siblings Judy and Peter discover an encha...

      2
      Grumpier Old Men
      A family wedding reignites the ancient feud be...

      3
      Waiting to Exhale
      Cheated on, mistreated and stepped on, the wom...

      4
      Father of the Bride Part II
      Just when George Banks has recovered from his ...

```python
metadata = metadata.iloc[:20000,:] # Similarity matrix can be very large
```

```python
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english')
```

```python
metadata['overview'] = metadata['overview'].fillna('')
```

* 47487 different words in overview for 20k Movies

```python
tfidf_matrix = tfidf.fit_transform(metadata['overview'])

tfidf_matrix.shape
```

    (20000, 47487)

* **Similarity Score :** Cosine Similarity will be used.

```python
from sklearn.metrics.pairwise import linear_kernel

cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)
```

```python
cosine_sim.shape
```

    (20000, 20000)

```python
cosine_sim[0]
```

    array([1.        , 0.01575748, 0.        , ..., 0.        , 0.        ,
           0.        ])

* Construct a serie for mapping movies and IDs

```python
indices = pd.Series(metadata.index, index=metadata['title']).drop_duplicates()
```

```python
indices[:10]
```

    title
    Toy Story                      0
    Jumanji                        1
    Grumpier Old Men               2
    Waiting to Exhale              3
    Father of the Bride Part II    4
    Heat                           5
    Sabrina                        6
    Tom and Huck                   7
    Sudden Death                   8
    GoldenEye                      9
    dtype: int64

* **get_recommendations :**
  * Takes title and similarity matrix
  * Returns 10 most similar movies as recommendations

```python
def get_recommendations(title, cosine_sim=cosine_sim):

    idx = indices[title]
    sim_scores = list(enumerate(cosine_sim[idx]))
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)
    sim_scores = sim_scores[1:11]
    movie_indices = [i[0] for i in sim_scores]

    return pd.DataFrame(metadata['title'].iloc[movie_indices]).reset_index(drop=True)
```

```python
get_recommendations("Inglourious Basterds")
```

      title

      0
      The Last Metro

      1
      Black Book

      2
      The Dirty Dozen: The Fatal Mission

      3
      Goodbye, Columbus

      4
      The Diary of Anne Frank

      5
      Don't Look Now: We're Being Shot At

      6
      A Letter from Death Row

      7
      Kelly's Heroes

      8
      Defiance

      9
      Schindler's List

```python
get_recommendations('There Will Be Blood') # kesin izleyin...
```

      title

      0
      The Stars Fell on Henrietta

      1
      Dallas - War of The Ewings

      2
      Tulsa

      3
      On Deadly Ground

      4
      Written on the Wind

      5
      The Formula

      6
      Hellfighters

      7
      Simon Magus

      8
      Terror in a Texas Town

      9
      Local Hero

## Recommendation with Overview Vectors

```python
%%capture
!pip3 install spacy
!python3 -m spacy download en_core_web_sm
```

```python
import spacy
nlp = spacy.load('en_core_web_sm')
```

```python
metadata = pd.read_csv('/content/drive/MyDrive/applied_ai_enes_safak/recommender_systems/MovieLens/movies_metadata.csv', low_memory=False)

metadata['overview'] = metadata['overview'].fillna('')
```

```python
pd.DataFrame(metadata[['original_title','overview']].head())
```

      original_title
      overview

      0
      Toy Story
      Led by Woody, Andy's toys live happily in his ...

      1
      Jumanji
      When siblings Judy and Peter discover an encha...

      2
      Grumpier Old Men
      A family wedding reignites the ancient feud be...

      3
      Waiting to Exhale
      Cheated on, mistreated and stepped on, the wom...

      4
      Father of the Bride Part II
      Just when George Banks has recovered from his ...

```python
movie_docs = pd.DataFrame(metadata[['original_title','overview']])
```

```python
movie_docs.shape
```

    (45466, 2)

```python
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english')
```

```python
tfidf_matrix = tfidf.fit_transform(metadata['overview'])

tfidf_matrix.shape
```

    (45466, 75827)

```python
# spacy
from tqdm.notebook import tqdm
movie_docs_vectors = []
for idx, row in tqdm(metadata.iterrows(), total=len(metadata)):
  doc = nlp(row["overview"])
  movie_docs_vectors.append(doc.vector)
movie_docs["overview vectors"] = movie_docs_vectors
```

    HBox(children=(FloatProgress(value=0.0, max=45466.0), HTML(value='')))

```python
with open('/content/drive/MyDrive/applied_ai_enes_safak/recommender_systems/MovieLens/word_vecs.npy', 'wb') as f:
    np.save(f, np.array(movie_docs['overview vectors']), allow_pickle=True)
```

```python
with open('/content/drive/MyDrive/applied_ai_enes_safak/recommender_systems/MovieLens/word_vecs.npy', 'rb') as f:
    word_vecs = np.load(f, allow_pickle=True)
```

```python
movie_docs["overview vectors"] = word_vecs
```

```python
movie_docs.head()
```

      original_title
      overview
      overview vectors

      0
      Toy Story
      Led by Woody, Andy's toys live happily in his ...
      [0.29593316, 0.1624922, -0.0122375265, 0.31845...

      1
      Jumanji
      When siblings Judy and Peter discover an encha...
      [-0.06933068, 0.53941554, -0.22379777, -0.2023...

      2
      Grumpier Old Men
      A family wedding reignites the ancient feud be...
      [0.40886644, 0.47620142, -0.14130907, 0.581963...

      3
      Waiting to Exhale
      Cheated on, mistreated and stepped on, the wom...
      [0.12919456, 0.6737056, -0.3598231, -0.0721177...

      4
      Father of the Bride Part II
      Just when George Banks has recovered from his ...
      [0.45354685, 0.5896248, 0.1983986, -0.37084275...

```python
def get_recommendations(title, recommend_k, df):
  #cos_sim = dot(a, b)/(norm(a)*norm(b))
  vector = df[df["original_title"] == title]["overview vectors"].iloc[0]

  shape = vector.shape[0]

  list_of_vecs = df["overview vectors"].tolist()

  similarities = []
  for b in list_of_vecs:
    try:
      similarities.append(np.dot(vector, b)/(np.linalg.norm(vector)*np.linalg.norm(b)))
    except:
      pass
  indices = np.array(similarities).argsort()[0:recommend_k][::-1]

  return [df.iloc[index, 0] for index in indices]
```

```python
get_recommendations("The Departed", 5, movie_docs)
```

    ['The Long and the Short and the Tall',
     'Tower Heist',
     'Eu Não Faço a Menor Ideia do que eu Tô Fazendo Com a Minha Vida',
     'Pioneer Woman',
     'Los cronocrímenes']

```python
get_recommendations("Happy Feet", 5, movie_docs)
```

    ['Tower Heist',
     'Youngblood',
     'Eu Não Faço a Menor Ideia do que eu Tô Fazendo Com a Minha Vida',
     'Pioneer Woman',
     'Los cronocrímenes']

```python
get_recommendations("Whiplash", 5, movie_docs)
```

    ['티끌모아 로맨스',
     'Gli occhi freddi della paura',
     'Under the Boardwalk: The Monopoly Story',
     'Pioneer Woman',
     'Los cronocrímenes']

```python
get_recommendations("Kill Bill: Vol. 1", 5, movie_docs)
```

    ['Gli occhi freddi della paura',
     'Under the Boardwalk: The Monopoly Story',
     '티끌모아 로맨스',
     'Pioneer Woman',
     'Los cronocrímenes']


# Data as Geometry
Below we show how data can be expressed as geometrical shapes and points.

#### run global setup

```python
try:
    with open("../global_setup.py") as setupfile:
        exec(setupfile.read())
except FileNotFoundError:
    print('Setup already completed')
```

#### run local setup

```python
%matplotlib inline
%matplotlib notebook

from notebooks.exercises.src.data_as_geometry import widget0, widget1, widget2, \
    exercise_2_4, generate_circle_data, plot_data, plot_decision_lines, \
    plot_decision_contour, plot3d_space, ProjectionPlotter
```

***
# Notebook: Data as geometry
*Introduction to intelligent systems - the toolbox*

** Objectives **: The objectives of this notebook is to (i) refresh what scalar- and vector projection is, (ii) how projections are connected to neurons and (iii) use this knowledge to understand a simple neural network.

***

## 1. Projections
In the following set of exercises we should get confortable with the concept of vector projection. As

** Definition: The inner production** (dot product) between two vectors $\bar{x}=(x_1, x_2)$ and $\bar{w}=(w_1, w_2)$ is given by

$$ \bar{w} \cdot \bar{x} = x_1 w_1 + x_2 w_2 $$

In this case $\bar{x}$, $\bar{w}$ is 2-dimensional vectors, but the concept expands to any finite dimensional space. If $\bar{x}$, $\bar{w}$ were two n-dimensional vector, their inner product would instead be:

$$ \bar{w} \cdot \bar{x} = x_1 w_1 + ... + x_n w_n = \sum_{i=1}^n x_i w_i $$

One of the most important uses of the inner product is in the computation of projections.

**Definition: The scalar projection** of a vector $\bar{x}$ onto a vector $\bar{w}$ is the *length* of the segment AB shown in the figure below. The **vector projection** of $\bar{x}$ onto $\bar{w}$ is the vector with this length that begins at point A and points in the same direction as $\bar{w}$.

It is important to note that when we are talking about projections, we are always talking about the *orthogonal* project (the red vector in the figure).

We can compute the scalar projection $proj_{\bar{w}}(\bar{x})$ as
$$ proj_{\bar{w}}(\bar{x}) = |\bar{x}| \cdot \cos(\theta_{w-x}) $$
where the angle $\theta_{w-x}$ can be seen in the figure. However, you will now show that it can also be computed as
$$ proj_{\bar{w}}(\bar{x}) = \bar{x} \cdot \bar{w} $$

where we have assume $|\bar{w}|=1$ (unit vector).

* **Exercise 1.1**: expand the expression
$$  proj_{\bar{w}}(\bar{x}) = |\bar{x}| \cdot \cos(\theta_{w-x}) $$
by using one of the trigometric identities. Use the markdown cell below to write your answer. Hint: look at the [wiki page](https://en.wikipedia.org/wiki/List_of_trigonometric_identities) under "Angle sum and difference identities".

* **Exercise 1.2**: you should arrive at an expression like this
$$ |\bar{x}| \cdot \left(\cos(\theta_w) \cdot \cos(\theta_x) + \sin(\theta_w) \cdot \sin(\theta_x) \right) $$
this is a inner product between two vectors. Determine what the two vectors look like.

* **Exercise 1.3** the two vectors you found in the last exercise, is written in what we call polar coordinates
$$ \begin{align}
&\text{Cartesian coordinates               } &&\text{Polar coordinates} \\
&\bar{x}=(x_1, x_2) &&\bar{x}=|\bar{x}| \cdot (\cos(\theta_x), \sin(\theta_x)) \\
&\text{where } x_1 = |\bar{x}|\cos(\theta_x), x_2 = |\bar{x}|\sin(\theta_x) &&\text{where } |\bar{x}|=\sqrt{x_1^2 + x_2^2}, \theta_x = \tan\left(\frac{x_2}{x_1}\right)
   \end{align}
$$
simplify them by rewriting them in cartesian coordinates. Does it look like the expression we wanted to find?

```python
%matplotlib notebook
%matplotlib notebook
plotter = ProjectionPlotter(limits=(-3.0, 5.0), linewidth=3)
plotter.start()
```

***
*double click here to edit cell*

** Type in your answer to exerise 1.1 here**:

If $\theta_{w-x} = \theta_w - \theta_x$ then
$$  \cos(\theta_{w-x}) = \cos(\theta_w - \theta_x) = \cos(\theta_w) \cdot \cos(\theta_x) + \sin(\theta_w) \cdot \sin(\theta_x) $$

** Type in your answer to exerise 1.2 here**:

The two vectors
$$ \begin{pmatrix} |\bar{x}| \cos(\theta_x) \\ |\bar{x}| \sin(\theta_x) \end{pmatrix} , \begin{pmatrix} \cos(\theta_w) \\ \sin(\theta_w) \end{pmatrix}$$
since
$$ \begin{align}
\begin{pmatrix} |\bar{x}| \cos(\theta_x) \\ |\bar{x}| \sin(\theta_x) \end{pmatrix} \cdot \begin{pmatrix}\cos(\theta_w) \\ \sin(\theta_w) \end{pmatrix} &= |\bar{x}| \cos(\theta_x) \cos(\theta_w) + |\bar{x}| \sin(\theta_x) \sin(\theta_w) \\
&=|\bar{x}| \cdot \left(\cos(\theta_w) \cdot \cos(\theta_x) + \sin(\theta_w) \cdot \sin(\theta_x) \right)\end{align}$$

** Type in your answer to exercise 1.3 here**:
$$\begin{pmatrix} |\bar{x}| \cos(\theta_x) \\ |\bar{x}| \sin(\theta_x) \end{pmatrix} = \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} \: \text{and} \: \begin{pmatrix} \cos(\theta_w) \\ \sin(\theta_w) \end{pmatrix} = \begin{pmatrix} w_1 \\ w_2 \end{pmatrix}$$
thus the expression from last exercise reduce to
$$ \bar{w} \cdot \bar{x} = x_1 w_1 + x_2 w_2$$
***

## 2. Activation of a single neuron
The basic building blocks of a neural network, is called an neuron. It takes a number of inputs and sends out a single signal, see Figure below.

The output of a neuron we call its activation $a$. Lets assume that we have two inputs (as in the figure) and that the activation of the neuron is calculated as
$$ a = w_1 x_1 + w_2 x_2 + w_0 $$
you can think of this as a weighted sum of the inputs. The activation of the neuron can have 3 values:

$$ \begin{align}
&a > 0 \Rightarrow \text{Active} \\
&a < 0 \Rightarrow \text{Inactive} \\
&a = 0 \Rightarrow ?
\end{align}
$$

* **Exercise 2.1**: For the case $a=0$ rewrite
$$ a = w_1 x_1 + w_2 x_2 + w_0 $$
into
$$ x_2 = - \frac{w_1}{w_2} x_1 - \frac{w_0}{w_2} $$ in the cell below.

* **Exercise 2.2**: If we define $\alpha = - \frac{w_1}{w_2}$ and $\beta =  - \frac{w_0}{w_2}$ we can simplify the expression from last exercise. Can you recognize what this equation represents. What is $\alpha$, $\beta$?

* **Exercise 2.3**: The original expression for $a$,
$$ a = w_1 x_1 + w_2 x_2 + w_0 $$
can you rewrite that in terms of two vectors $\bar{x}$ and $\bar{w}$ and the inner product between them i.e $a=\bar{w} \cdot \bar{x}$?

***
*double click here to edit cell*

** Type in your answer to exerise 2.1 here**:

For a=0 we have
$$ 0 = w_1 x_1 + w_2 x_2 + w_0 $$
move term with x_2 to the left
$$ - w_2 x_2 = w_1 x_1  + w_0 $$
devide by -w_2
$$ x_2 = \frac{w_1 x_1+ w_0}{-w_2} = -\frac{w_1}{w_2} x_1 - \frac{w_0}{w_2} $$

** Type in your answer to exerise 2.2 here**:

With $\alpha = - \frac{w_1}{w_2}$ and $\beta =  - \frac{w_0}{w_2}$ the above expression is simplify to
$$ x_2 = \alpha x_1 + \beta$$
this is the expression for a streight line in 2D, where $\alpha$ is the slope and $\beta$ is the intercep with the y-axis.

** Type in your answer to exercise 2.3 here**:

Define $\bar{x}=(x_1, x_2, 1)$ and $\bar{w} = (w_1, w_2, w_0)$. Then the inner product becomes

$$ a = \bar{w} \cdot \bar{x} = w_1 x_1 + w_2 x_2 + w_0 $$

***

* **Exercise 2.4**: Use the widget below to type in some values for $\alpha, \beta$. Can you find a line that (as good as possible) seperated the blue points from the red points. What is the activation of the red, blue points? Hint: is it negative, positive or 0?.

```python
%matplotlib notebook
%matplotlib notebook
widget_0 = widget0()
exercise_2_4(widget_0)
```

Thus, the take away from this exercise is that neurons is just lines in space that is activated for some points and inactive in some other points. However, as we could see in the example, if we only use a single neuron we can only take very simple decisions. Lets therefore try out a network with more neurons on a complex dataset.

## 3. Projections in a network
We will know use the knowledge we have just gained to analyse how data is propogated through the following simple neural network:

The network is called a feed forward network, because information is only flowing from left to right. The network has two input neurons, three neurons in the hidden layer and a single output unit. In this simple case, we can interpret the network as taking a 2D point (x,y), propergate its coordinates through the network, and in the end determine if the 2D point should be colored red or blue.

## Data
We will adjust our network such that it can distingues between two classes of 2D points, a inner class of red points and outer ring of blue points.

```python
%matplotlib notebook
%matplotlib notebook
# Lets generate 500 points from two classes
X, Y = generate_circle_data(N = 500)
plot_data(X, Y)
```

* **Exercise 3.1**: Why do we need more than a single neuron to distingues between the two classes in this case?
* **Exercise 3.2**: Our hidden layer has 3 neurons, and each of these network has a number of weights attached to them. Adjust them in the widget below, such that 3 lines generated by the next code block alltogether devide the region of red points from the region of blue points. Hints: (1) what is the connection between the weights and the black 3 arrows in the plot? (2) what is the connection between the lines and the black arrows?

```python
%matplotlib notebook
%matplotlib notebook
widget_1 = widget1()
plot_decision_lines(X, Y, widget_1)
plot_decision_contour(X, Y, widget_1)
```

* **Exercise 3.3**: The three plots above shows the activation of the three neurons in the hidden layer. We can compute the activation of the hidden neuron $h_1$ as:
$$ h_1 = \tanh(w_{11}^{(1)} \cdot x + w_{21}^{(1)} \cdot y + w_{01}^{(1)}) $$
Write down the the activate of the two other hidden units. Hint: look at the figure of network

* **Exercise 3.4** In our network we use $\tanh(a)$ as activation function. This means that after computing the activation of the neuron we send the signal through the $\tanh(a)$ function. Sketch (on paper) $\tanh(\cdot)$ and discuss what happens with the output signal when $a$ is large.

* **Exercise 3.5**:The expression you just wrote down can be simplified using vector notation and inner products. Define a vector $\bar{x}$ and three weight vectors $\bar{w}_1, \bar{w}_2, \bar{w}_3$ and write down the activation for the three hidden neurons using vector notation and inner products.

***
*double click here to edit cell*

** Type in your answer to exerise 3.3 here**:

Neuron 2: $h_2 = \tanh(w_{12}^{(1)} \cdot x + w_{22}^{(1)} \cdot y + w_{02}^{(1)})$

Neuron 3: $h_3 = \tanh(w_{13}^{(1)} \cdot x + w_{23}^{(1)} \cdot y + w_{03}^{(1)})$

** Type in your answer to exerise 3.4 here**

Link to [mathworlds](http://mathworld.wolfram.com/HyperbolicTangent.html) which can plot the function.

For a > 2 or a < -2 we have that $\tanh(a)\approx \pm 1$, meaning that all activations gets restricted to the [-1,1] domain.

** Type in your answer to exerise 3.5 here**

If we define vector $\bar{x} = (x_1, x_2, 1)$ and the following weight vectors
$$\bar{w}_1 = (w_{11}^{(1)}, w_{21}^{(1)}, w_{01}^{(1)})), \bar{w}_2 = (w_{12}^{(1)}, w_{22}^{(1)}, w_{02}^{(1)})), \bar{w}_3 = (w_{13}^{(1)}, w_{23}^{(1)}, w_{03}^{(1)}))$$
then the activation of the three hidden neurons can be written as:
$$h_1 = \tanh(\bar{w}_1 \cdot \bar{x}), h_2 = \tanh(\bar{w}_2 \cdot \bar{x}), h_2 = \tanh(\bar{w}_3 \cdot \bar{x}) $$

***

* **Exercise 3.6**: Our input layer had two neurons, meaning that we could plot the activation in a 2D space. Since our hidden layer has 3 neurons, we can plot the activation of this layer in a 3D spcae. Adjust the of widget below such that the plane that is plotted, seperates the red and the blue points in the 3D space.

```python
%matplotlib notebook
%matplotlib notebook
widget_2 = widget2()
classifier = plot3d_space(X, Y, widget_1, widget_2)
```

```python
# When you are satisfied with the plot, you can
# run this cell to get the results
classifier.get_res()
```

* **Exercise 3.6**: Based on the results printed in the bottom of the cell, compute the accuracy and error rate:
$$ \text{accuracy} = \dfrac{\text{number of correctly classified points}}{\text{total number of points}} $$
$$ \text{error rate} = \dfrac{\text{number of wrongly classified points}}{\text{total number of points}} $$
what is the connection between the accuracy and error rate of an classifier?

* **Exercise 3.7**: Go back and try adjusting some of the weights. Can you optimize your classifier i.e. get a better accuracy?

***
*double click here to edit cell*

** Type in your answer to exerise 3.6 here**:
Number of correctly classified points: 475 + 500 = 975

Number of wrongly classified points: 25

Total number of points: 475 + 500 + 25 = 1000

Therefore, $\text{accuracy} = \frac{975}{1000} = 0.975 = 97,5\%$ and $\text{error rate} = \frac{25}{1000} = 0.025 = 2.5\%$.

***

## 4. Tensorflow playground (optional)
The neural network presented in this notebook is very simple, but regardless of network complexity the underlying calculations still relies on the topics presented in this notebook. Try opening [tensorflow playground](https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.30449&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false) where we can play around with number of neurons in our network, activation function ect. Try out the following:

* **Exercise 4.1** Can you recreate the network we have just investigated?

* **Exercise 4.2** What happens with the decision boundary when we change activation function from $tanh(\cdot)$ to $relu(\cdot)$?
* **Exercise 4.3** Try changing the dataset to the spiral. Can you find a network that can classify this data?

Remember to write down your conclusions!

```python

```