# Linear dimension reduction and feature extraction

## Introduction

In machine learning and statistics, dimensionality reduction or dimension reduction is the process of reducing the number of features under consideration, and can be divided into feature selection (not addressed here) and feature extraction.

Feature extraction starts from an initial set of measured data and builds derived values (features) intended to be informative and non-redundant, facilitating the subsequent learning and generalization steps, and in some cases leading to better human interpretations. Feature extraction is related to dimensionality reduction.

The input matrix $\mathbf{X}$, of dimension $N \times P$, is

$$
\begin{bmatrix}
x_{11} & \ldots     &     x_{1P}\\
       &            &      \\
\vdots & \mathbf{X} & \vdots\\
       &            &      \\
x_{N1} & \ldots     &     x_{NP}
\end{bmatrix}
$$

where the rows represent the samples and columns represent the variables.
The goal is to learn a transformation that extracts a few relevant features.

Models:

1. Linear matrix decomposition/factorisation SVD/PCA. Those models exploit the covariance $\mathbf{\Sigma_{XX}}$ between the input features.
2. Non-linear models based on manifold learning: Isomap, t-SNE. Those models

## Singular value decomposition and matrix factorization

### Matrix factorization principles

Decompose the data matrix $\mathbf{X}_{N \times P}$ into a product of a mixing matrix $\mathbf{U}_{N \times K}$ and a dictionary matrix $\mathbf{V}_{P \times K}$.

$$
\mathbf{X} = \mathbf{U} \mathbf{V}^{T},
$$

If we consider only a subset of components $K<rank(\mathbf{X}) < \min(P, N-1)$ , $\mathbf{X}$ is approximated by a matrix $\hat{\mathbf{X}}$:

$$
\mathbf{X} \approx \hat{\mathbf{X}} = \mathbf{U} \mathbf{V}^{T},
$$

Each line of $\mathbf{x_i}$ is a linear combination (mixing $\mathbf{u_i}$) of dictionary items $\mathbf{V}$.

$N$ $P$-dimensional data points lie in a space whose dimension is less than $N-1$ (2 dots lie on a line, 3 on a plane, etc.).

### Singular value decomposition (SVD) principles

Singular-value decomposition (SVD) factorises the data matrix $\mathbf{X}_{N \times P}$ into a product:

$$
\mathbf{X} = \mathbf{U}\mathbf{D}\mathbf{V}^{T},
$$

where

$$
\begin{bmatrix}
x_{11} &            & x_{1P}\\
       &            &       \\
       & \mathbf{X} &       \\
       &            &       \\
x_{N1} &            & x_{NP}
\end{bmatrix} =
\begin{bmatrix}
u_{11} &            & u_{1K}\\
       &            &       \\
       & \mathbf{U} &       \\
       &            &       \\
u_{N1} &            & u_{NK}
\end{bmatrix}
\begin{bmatrix}
d_{1}&            & 0\\
     & \mathbf{D} &\\
 0   &            & d_{K}
\end{bmatrix}
\begin{bmatrix}
v_{11}&              & v_{1P}\\
      & \mathbf{V}^T &       \\
v_{K1}&              & v_{KP}
\end{bmatrix}.
$$

$\mathbf{U}$: **right-singular**

- $\mathbf{V} = [\mathbf{v}_1,\cdots , \mathbf{v}_K]$ is a $P \times K$ orthogonal matrix.

- It is a **dictionary** of patterns to be combined (according to the mixing coefficients) to reconstruct the original samples.

- $\mathbf{V}$ perfoms the initial **rotations** (**projection**) along the $K=\min(N, P)$ **principal component directions**, also called **loadings**.

- Each $\mathbf{v}_j$ performs the linear combination of the variables that has maximum sample variance, subject to being uncorrelated with the previous $\mathbf{v}_{j-1}$.

$\mathbf{D}$: **singular values**

- $\mathbf{D}$ is a $K \times K$ diagonal matrix made of the singular values of $\mathbf{X}$ with $d_1 \geq d_2 \geq \cdots \geq d_K \geq 0$.

- $\mathbf{D}$ scale the projection along the coordinate axes by $d_1, d_2, \cdots, d_K$.

- Singular values are the square roots of the eigenvalues of $\mathbf{X}^{T}\mathbf{X}$.

$\mathbf{V}$: **left-singular vectors**

- $\mathbf{U} = [\mathbf{u}_1, \cdots , \mathbf{u}_K]$ is an $N \times K$ orthogonal matrix.

- Each row $\mathbf{v_i}$ provides the **mixing coefficients** of dictionary items to reconstruct the sample $\mathbf{x_i}$

- It may be understood as the coordinates on the new orthogonal basis (obtained after the initial rotation) called **principal components** in the PCA.

### SVD for variables transformation

$\mathbf{V}$ transforms correlated variables ($\mathbf{X}$) into a set of uncorrelated ones ($\mathbf{U}\mathbf{D}$) that better expose the various relationships among the original data items.

\begin{align}
\mathbf{X}           &= \mathbf{U}\mathbf{D}\mathbf{V}^{T},\\
\mathbf{X}\mathbf{V} &= \mathbf{U}\mathbf{D}\mathbf{V}^{T}\mathbf{V},\\
\mathbf{X}\mathbf{V} &= \mathbf{U}\mathbf{D}\mathbf{I},\\
\mathbf{X}\mathbf{V} &= \mathbf{U}\mathbf{D}
\end{align}

At the same time, SVD is a method for identifying and ordering the dimensions along which data points exhibit the most variation.

```python
import numpy as np
import scipy
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

np.random.seed(42)

# dataset
n_samples = 100
experience = np.random.normal(size=n_samples)
salary = 1500 + experience + np.random.normal(size=n_samples, scale=.5)
X = np.column_stack([experience, salary])
print(X.shape)

# PCA using SVD
X -= X.mean(axis=0)  # Centering is required
U, s, Vh = scipy.linalg.svd(X, full_matrices=False)
# U : Unitary matrix having left singular vectors as columns.
#     Of shape (n_samples,n_samples) or (n_samples,n_comps), depending on
#     full_matrices.
#
# s : The singular values, sorted in non-increasing order. Of shape (n_comps,),
#     with n_comps = min(n_samples, n_features).
#
# Vh: Unitary matrix having right singular vectors as rows.
#     Of shape (n_features, n_features) or (n_comps, n_features) depending
# on full_matrices.

plt.figure(figsize=(9, 3))

plt.subplot(131)
plt.scatter(U[:, 0], U[:, 1], s=50)
plt.axis('equal')
plt.title("U: Rotated and scaled data")

plt.subplot(132)

# Project data
PC = np.dot(X, Vh.T)
plt.scatter(PC[:, 0], PC[:, 1], s=50)
plt.axis('equal')
plt.title("XV: Rotated data")
plt.xlabel("PC1")
plt.ylabel("PC2")

plt.subplot(133)
plt.scatter(X[:, 0], X[:, 1], s=50)
for i in range(Vh.shape[0]):
    plt.arrow(x=0, y=0, dx=Vh[i, 0], dy=Vh[i, 1], head_width=0.2,
              head_length=0.2, linewidth=2, fc='r', ec='r')
    plt.text(Vh[i, 0], Vh[i, 1],'v%i' % (i+1), color="r", fontsize=15,
             horizontalalignment='right', verticalalignment='top')
plt.axis('equal')
plt.ylim(-4, 4)

plt.title("X: original data (v1, v2:PC dir.)")
plt.xlabel("experience")
plt.ylabel("salary")

plt.tight_layout()
```

## Principal components analysis (PCA)

Sources:

- C. M. Bishop *Pattern Recognition and Machine Learning*, Springer, 2006

- [Everything you did and didn't know about PCA](http://alexhwilliams.info/itsneuronalblog/2016/03/27/pca/)

- [Principal Component Analysis in 3 Simple Steps](http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html)

### Principles

- Principal components analysis is the main method used for linear dimension reduction.

- The idea of principal component analysis is to find the $K$ **principal components directions** (called the **loadings**) $\mathbf{V}_{K\times P}$ that capture the variation in the data as much as possible.

- It converts a set of $N$ $P$-dimensional observations $\mathbf{N}_{N\times P}$ of possibly correlated variables into a set of $N$ $K$-dimensional samples  $\mathbf{C}_{N\times K}$, where the $K < P$. The new variables are linearly uncorrelated. The columns of $\mathbf{C}_{N\times K}$ are called the **principal components**.

- The dimension reduction is obtained by using only $K < P$ components that exploit correlation (covariance) among the original variables.

- PCA is mathematically defined as an orthogonal linear transformation $\mathbf{V}_{K\times P}$ that transforms the data to a new coordinate system such that the greatest variance by some projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on.
$$
\mathbf{C}_{N\times K} = \mathbf{X}_{N \times P} \mathbf{V}_{P \times K}
$$

- PCA can be thought of as fitting a $P$-dimensional ellipsoid to the data, where each axis of the ellipsoid represents a principal component. If some axis of the ellipse is small, then the variance along that axis is also small, and by omitting that axis and its corresponding principal component from our representation of the dataset, we lose only a commensurately small amount of information.

- Finding the $K$ largest axes of the ellipse will permit to project the data onto a space having dimensionality $K < P$ while maximizing the variance of the projected data.

### Dataset preprocessing

#### Centering

Consider a data matrix, $\mathbf{X}$ , with column-wise zero empirical mean (the sample mean of each column has been shifted to zero), ie. $\mathbf{X}$ is replaced by $\mathbf{X} - \mathbf{1}\bar{\mathbf{x}}^T$.

#### Standardizing

Optionally, standardize the columns, i.e., scale them by their standard-deviation. Without standardization, a variable with a high variance will capture most of the effect of the PCA. The principal direction will be aligned with this variable. Standardization will, however, raise noise variables to the save level as informative variables.

The covariance matrix of centered standardized data is the correlation matrix.

### Eigendecomposition of the data covariance matrix

To begin with, consider the projection onto a one-dimensional space ($K = 1$). We can define the direction of this space using a $P$-dimensional vector $\mathbf{v}$, which for convenience (and without loss of generality) we shall choose to be a unit vector so that $\|\mathbf{v}\|_2 = 1$ (note that we are only interested in the direction defined by $\mathbf{v}$, not in the magnitude of $\mathbf{v}$ itself). PCA consists of two mains steps:

**Projection in the directions that capture the greatest variance**

Each $P$-dimensional data point $\mathbf{x}_i$ is then projected onto $\mathbf{v}$, where the coordinate (in the coordinate system of $\mathbf{v}$) is a scalar value, namely $\mathbf{x}_i^T \mathbf{v}$. I.e., we want to find the vector $\mathbf{v}$ that maximizes these coordinates along $\mathbf{v}$, which we will see corresponds to maximizing the variance of the projected data. This is equivalently expressed as

$$
\mathbf{v} = \arg \max_{\|\mathbf{v}\|=1}\frac{1}{N}\sum_i \left(\mathbf{x}_i^T \mathbf{v}\right)^2.
$$

We can write this in matrix form as

$$
\mathbf{v} = \arg \max_{\|\mathbf{v}\|=1} \frac{1}{N} \|\mathbf{X} \mathbf{v}\|^2 = \frac{1}{N} \mathbf{v}^T \mathbf{X}^T \mathbf{X} \mathbf{v} = \mathbf{v}^T\mathbf{S_{XX}}\mathbf{v},
$$

where $\mathbf{S_{XX}}$ is a biased estiamte of the covariance matrix of the data, i.e.

$$
\mathbf{S_{XX}} = \frac{1}{N} \mathbf{X}^T\mathbf{X}.
$$

We now maximize the projected variance $\mathbf{v}^T \mathbf{S_{XX}} \mathbf{v}$ with respect to $\mathbf{v}$. Clearly, this has to be a constrained maximization to prevent $\|\mathbf{v}_2\| \rightarrow \infty$. The appropriate constraint comes from the normalization condition $\|\mathbf{v}\|_2 \equiv \|\mathbf{v}\|_2^2 = \mathbf{v}^T \mathbf{v} = 1$. To enforce this constraint, we introduce a [Lagrange multiplier](https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/constrained-optimization/a/lagrange-multipliers-single-constraint) that we shall denote by $\lambda$, and then make an unconstrained maximization of

$$
\mathbf{v}^T\mathbf{S_{XX}} \mathbf{v} - \lambda (\mathbf{v}^T \mathbf{v} - 1).
$$

By setting the gradient with respect to $\mathbf{v}$ equal to zero, we see that this quantity has a stationary
point when

$$
\mathbf{S_{XX}} \mathbf{v} = \lambda \mathbf{v}.
$$

We note that $\mathbf{v}$ is an eigenvector of $\mathbf{S_{XX}}$.

If we left-multiply the above equation by $\mathbf{v}^T$ and make use of $\mathbf{v}^T \mathbf{v} = 1$, we see that the variance is given by

$$
\mathbf{v}^T \mathbf{S_{XX}} \mathbf{v} = \lambda,
$$

and so the variance will be at a maximum when $\mathbf{v}$ is equal to the eigenvector corresponding to the largest eigenvalue, $\lambda$. This eigenvector is known as the first principal component.

We can define additional principal components in an incremental fashion by choosing each new direction to be that which maximizes the projected variance amongst all possible directions that are orthogonal to those already considered. If we consider the general case of a $K$-dimensional projection space, the optimal linear projection for which the variance of the projected data is maximized is now defined by the $K$ eigenvectors, $\mathbf{v_1}, \ldots , \mathbf{v_K}$, of the data covariance matrix $\mathbf{S_{XX}}$ that corresponds to the $K$ largest eigenvalues, $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_K$.

#### Back to SVD

The sample covariance matrix of **centered data** $\mathbf{X}$ is given by

$$
\mathbf{S_{XX}} = \frac{1}{N-1}\mathbf{X}^T\mathbf{X}.
$$

We rewrite $\mathbf{X}^T\mathbf{X}$ using the SVD decomposition of $\mathbf{X}$ as

\begin{align*}
    \mathbf{X}^T\mathbf{X}
     &= (\mathbf{U}\mathbf{D}\mathbf{V}^T)^T(\mathbf{U}\mathbf{D}\mathbf{V}^T)\\
     &= \mathbf{V}\mathbf{D}^T\mathbf{U}^T\mathbf{U}\mathbf{D}\mathbf{V}^T\\
     &=\mathbf{V}\mathbf{D}^2\mathbf{V}^T\\
    \mathbf{V}^T\mathbf{X}^T\mathbf{X}\mathbf{V} &= \mathbf{D}^2\\
    \frac{1}{N-1} \mathbf{V}^T\mathbf{X}^T\mathbf{X}\mathbf{V} &= \frac{1}{N-1}\mathbf{D}^2\\
    \mathbf{V}^T\mathbf{S_{XX}}\mathbf{V} &= \frac{1}{N-1}\mathbf{D}^2
\end{align*}.

Considering only the $k^{th}$ right-singular vectors $\mathbf{v}_k$ associated to the singular value $d_k$

$$
\mathbf{v_k}^T\mathbf{S_{XX}}\mathbf{v_k} = \frac{1}{N-1}d_k^2,
$$

It turns out that if you have done the singular value decomposition then you already have the Eigenvalue decomposition for $\mathbf{X}^T\mathbf{X}$. Where
- The eigenvectors of $\mathbf{S_{XX}}$ are equivalent to the right singular vectors, $\mathbf{V}$, of $\mathbf{X}$.
- The eigenvalues, $\lambda_k$, of $\mathbf{S_{XX}}$, i.e. the variances of the components, are equal to $\frac{1}{N-1}$ times the squared singular values, $d_k$.

Moreover computing PCA with SVD do not require to form the matrix $\mathbf{X}^T\mathbf{X}$, so computing the SVD is now the standard way to calculate a principal components analysis from a data matrix, unless only a handful of components are required.

#### PCA outputs

The SVD or the eigendecomposition of the data covariance matrix provides three main quantities:

1. **Principal component directions** or **loadings** are the **eigenvectors** of $\mathbf{X}^T\mathbf{X}$. The $\mathbf{V}_{K \times P}$ or the **right-singular vectors** of an SVD of $\mathbf{X}$ are called principal component directions of $\mathbf{X}$. They are generally computed using the SVD of $\mathbf{X}$.

2. **Principal components** is the ${N\times K}$ matrix $\mathbf{C}$ which is obtained by projecting $\mathbf{X}$ onto the principal components directions, i.e.

$$
    \mathbf{C}_{N\times K} = \mathbf{X}_{N \times P} \mathbf{V}_{P \times K}.
$$

Since $\mathbf{X} = \mathbf{UDV}^T$ and $\mathbf{V}$ is orthogonal ($\mathbf{V}^T \mathbf{V} = \mathbf{I}$):

\begin{align}
    \mathbf{C}_{N\times K} &= \mathbf{UDV}^T_{N \times P} \mathbf{V}_{P \times K}\\
    \mathbf{C}_{N\times K} &= \mathbf{UD}^T_{N \times K} \mathbf{I}_{K \times K}\\
    \mathbf{C}_{N\times K} &= \mathbf{UD}^T_{N \times K}\\
\end{align}

Thus $\mathbf{c}_j = \mathbf{X}\mathbf{v}_j = \mathbf{u}_j d_j$, for $j=1, \ldots K$. Hence $\mathbf{u}_j$ is simply the projection of the row vectors of $\mathbf{X}$, i.e., the input predictor vectors, on the direction $\mathbf{v}_j$, scaled by $d_j$.

$$
\mathbf{c}_1=
\begin{bmatrix}
x_{1,1}v_{1,1}+ \ldots +x_{1,P}v_{1,P}\\
x_{2,1}v_{1,1}+ \ldots +x_{2,P}v_{1,P}\\
\vdots\\
x_{N,1}v_{1,1}+ \ldots +x_{N,P}v_{1,P}
\end{bmatrix}
$$

3. The **variance** of each component is given by the eigen values $\lambda_k, k=1, \dots K$. It can be obtained from the singular values:

\begin{align}
var(\mathbf{c}_k) =& \frac{1}{N-1}(\mathbf{X} \mathbf{v}_k)^2\\
                  =& \frac{1}{N-1}(\mathbf{u}_k d_k)^2\\
                  =& \frac{1}{N-1}d_k^2
\end{align}

### Determining the number of PCs

We must choose $K^* \in [1, \ldots,  K]$, the number of required components. This can be done by calculating the explained variance ratio of the $K^*$ first components and by choosing $K^*$ such that the **cumulative explained variance** ratio is greater than some given threshold (e.g., $\approx 90\%$). This is expressed as

$$
\mathrm{cumulative~explained~variance}(\mathbf{c}_k) = \frac{\sum_j^{K^*} var(\mathbf{c}_k)}{\sum_j^K var(\mathbf{c}_k)}.
$$

### Interpretation and visualization

**PCs**

Plot the samples projeted on first the principal components as e.g. PC1 against PC2.

**PC directions**

Exploring the loadings associated with a component provides the contribution of each original variable in the component.

Remark: The loadings (PC directions) are the coefficients of multiple regression of PC on original variables:

\begin{align}
    \mathbf{c}                                             & = \mathbf{X} \mathbf{v}\\
    \mathbf{X}^T \mathbf{c}                                & = \mathbf{X}^T \mathbf{X} \mathbf{v}\\
    (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{c} & = \mathbf{v}
\end{align}

Another way to evaluate the contribution of the original variables in each PC can be obtained by computing the correlation between the PCs and the original variables, i.e. columns of $\mathbf{X}$, denoted $\mathbf{x}_j$, for $j=1, \ldots, P$. For the $k^{th}$ PC, compute and plot the correlations with all original variables

$$
cor(\mathbf{c}_k, \mathbf{x}_j),  j=1 \ldots K, j=1 \ldots K.
$$

These quantities are sometimes called the *correlation loadings*.

```python
import numpy as np
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

np.random.seed(42)

# dataset
n_samples = 100
experience = np.random.normal(size=n_samples)
salary = 1500 + experience + np.random.normal(size=n_samples, scale=.5)
X = np.column_stack([experience, salary])

# PCA with scikit-learn
pca = PCA(n_components=2)
pca.fit(X)
print(pca.explained_variance_ratio_)

PC = pca.transform(X)

plt.subplot(121)
plt.scatter(X[:, 0], X[:, 1])
plt.xlabel("x1"); plt.ylabel("x2")

plt.subplot(122)
plt.scatter(PC[:, 0], PC[:, 1])
plt.xlabel("PC1 (var=%.2f)" % pca.explained_variance_ratio_[0])
plt.ylabel("PC2 (var=%.2f)" % pca.explained_variance_ratio_[1])
plt.axis('equal')
plt.tight_layout()
```

```python
from time import time
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import offsetbox
from sklearn import (manifold, datasets, decomposition, ensemble,
                     discriminant_analysis, random_projection, neighbors)
print(__doc__)

digits = datasets.load_digits(n_class=6)
X = digits.data
y = digits.target
n_samples, n_features = X.shape
n_neighbors = 30
```

## Eigen faces

Sources: [Scikit learn Faces decompositions](https://scikit-learn.org/stable/auto_examples/decomposition/plot_faces_decomposition.html)

Load data

```python
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_olivetti_faces
from sklearn import decomposition

n_row, n_col = 2, 3
n_components = n_row * n_col
image_shape = (64, 64)

faces, _ = fetch_olivetti_faces(return_X_y=True, shuffle=True,
                                random_state=1)
n_samples, n_features = faces.shape

# Utils function
def plot_gallery(title, images, n_col=n_col, n_row=n_row, cmap=plt.cm.gray):
    plt.figure(figsize=(2. * n_col, 2.26 * n_row))
    plt.suptitle(title, size=16)
    for i, comp in enumerate(images):
        plt.subplot(n_row, n_col, i + 1)
        vmax = max(comp.max(), -comp.min())
        plt.imshow(comp.reshape(image_shape), cmap=cmap,
                   interpolation='nearest',
                   vmin=-vmax, vmax=vmax)
        plt.xticks(())
        plt.yticks(())
    plt.subplots_adjust(0.01, 0.05, 0.99, 0.93, 0.04, 0.)
```

Preprocessing

```python
# global centering
faces_centered = faces - faces.mean(axis=0)

# local centering
faces_centered -= faces_centered.mean(axis=1).reshape(n_samples, -1)
```

First centered Olivetti faces

```python
plot_gallery("First centered Olivetti faces", faces_centered[:n_components])
```

```python
pca = decomposition.PCA(n_components=n_components)
pca.fit(faces_centered)
plot_gallery("PCA first %i loadings" % n_components, pca.components_[:n_components])
```

## Exercises

### Write a basic PCA class

Write a class `BasicPCA` with two methods:

- `fit(X)` that estimates the data mean, principal components directions $\textbf{V}$ and the explained variance of each component.

- `transform(X)` that projects the data onto the principal components.

Check that your `BasicPCA` gave similar results, compared to the results from `sklearn`.

### Apply your Basic PCA on the `iris` dataset

The data set is available at: https://github.com/duchesnay/pystatsml/raw/master/datasets/iris.csv

- Describe the data set. Should the dataset been standardized?

- Describe the structure of correlations among variables.

- Compute a PCA with the maximum number of components.

- Compute the cumulative explained variance ratio. Determine the number of components $K$ by your computed values.

- Print the $K$ principal components directions and correlations of the $K$ principal components with the original variables. Interpret the contribution of the original variables into the PC.

- Plot the samples projected into the $K$ first PCs.

- Color samples by their species.

### Run scikit-learn examples

Load the notebook or python file at the end of each examples

- [Faces dataset decompositions](https://scikit-learn.org/stable/auto_examples/decomposition/plot_faces_decomposition.html)
- [Faces recognition example using eigenfaces and SVMs](https://scikit-learn.org/stable/auto_examples/applications/plot_face_recognition.html)

```python

```


#
Transformada de Laplace

```python
from sympy import *
init_printing(use_latex='mathjax')
x, y, z = symbols('x, y, z')
```

La transformada de Laplace de una función $f$ definida para valores positivos, se realiza con la integral

$$
\mathscr{L}(f(t))=\int_0^\infty f(t)\cdot e^{-st} dt
$$

Sin embargo esta es un integral impropia y tiene serias limitaciones en las condiciones de convergencia. Lo cierto es que la transformada de Laplace se suele utilizar mecánicamente y se suelen obviar estas condiciones de convergencia.

###
Utilizando la definición, calcula la transformada de Laplace de la función $\cos(t)$.

```python
var('t')
var('s', positive = True)
Integral(cos(t)*exp(-s*t),(t,0,oo)).doit()
```

$$\frac{s}{s^{2} + 1}$$

Para calcular la transformada de Laplace utilizamos **laplace_transform**. Le debemos facilitar tres argumentos: el primero es la función, el segundo la variable de la función de partida (en general siempre se llama $t$), y la variable de la función transformada (en general siempre se llama $s$).

Este comando nos devuelve tres resultados. El primero es la transformada de Laplace en si misma. El segundo tiene que ver con el dominio de definición de la función $F(s)$ y el tercero son condiciones adicionales para asegurar la convergencia de la integral.

Como nos suele interesar únicamente la transformada, debemos acceder al primer elemento de la tupla. Ello se consigue colocando un **[0]**, pues en Python los índices empiezan en cero.

###
Calcula la transformada de Laplace de las funciones:

$$
\mathscr{L}(\sin(at)) \qquad  \mathscr{L}(\cos(at)) \qquad \mathscr{L}(e^{at})
$$

```python
laplace_transform(Heaviside(t),t,s)[0]
```

$$\frac{1}{s}$$

###
Calcula la transformada de Laplace de constantes y polinomios.

```python
laplace_transform(t**7+t**5+4*t**3,t,s)[0]
```

$$\frac{1}{s^{8}} \left(24 s^{4} + 120 s^{2} + 5040\right)$$

###
Calcular la transformada de Laplace de la función $\delta$ de Dirac y de la función escalón de Heaviside.

```python

```

###
Calcula la transformada de Laplace:

$$
\mathscr{L}\left(t^3 \cdot \exp(-5t)\cdot \cos(2t)\right)
$$

```python
laplace_transform(t**3*exp(-5*t)*cos(2*t),t,s)[0]
```

###
Calcula la transformada del seno. Deriva la función y calcula la transformada. Multiplica la función por $e^{5t}$ y vuelve a calcular la transformada.

```python

```

La transformada inversa de Laplace se calcula con **inverse_laplace_transform**. Ahora debemos poner primero la $s$ y después la $t$.

###
Calcula la transformada inversa de Laplace:

$$
\mathscr{L}^{-1}\left(\frac{1}{(s+1)^2\cdot(s+2)}\right)
$$

```python
inverse_laplace_transform(1/((s+1)**2*(s+2)),s,t)
```

$$\left(\left(t - 1\right) e^{t} + 1\right) e^{- 2 t} \theta\left(t\right)$$

```python
step()
```

notebooks/16.- Transformada de Laplace.ipynb


# Expectation Maximization

An implementation of the EM algorithm for a Gaussian mixture model (GMM) was presented [here](https://chandrusuresh.github.io/MyNotes/files/DensityEstimation/GaussianMixtureModels.html). In this chapter, a more elaborate theoretical discussion of the EM algorithm is presented with derivations.

## Theory
The goal of the EM algorithm is to find the MLE for latent variable models. Let $X$ and $Z$ denote the set of all observed and latent variables respectively. The set of all model parameters to be determined is denoted by $\theta$.

The log-likelihood function is given by:

$$ \ln{p(X|\theta)} = \ln{\frac{p(X,Z|\theta)}{p(Z|X,\theta)}} $$

Note however that this term is intractable due to the presence of the summation inside the logarithm even if the joint distribution $p(X,Z|\theta)$ belongs to the exponential family.

Denote $q(Z)$ to be any distribution over the hidden variables. The log-likelihood can be expressed as:

$$\begin{align} \ln{p(X|\theta)} &= \ln{\frac{p(X,Z|\theta)}{p(Z|X,\theta)}} \\
&= \ln{\frac{p(X,Z|\theta)}{q_z}} - \ln{\frac{p(Z|X,\theta)}{q_z}} \end{align}$$

Taking the expectation w.r.t $q(Z)$ on both sides,

$$\begin{align} \int{q_z \ln{p(X|\theta)} dZ} &= \int{q_z \ln\left\{\frac{p(X,Z|\theta)}{q_z}\right\}dZ}  - \int{q_z\ln\left\{\frac{p(Z|X,\theta)}{q_z}\right\}dZ} \\
\Rightarrow \ln{p(X|\theta)} &= \mathcal{L}\Big(q_z,\theta\Big) + KL\Big(q_z\Big|\Big|p(Z|X,\theta)\Big) \end{align}$$

Note that the `KL divergence on the right hand side is always positive`. This implies that the first term on the right hand side is a lower bound to the log-likelihood.

$$ \ln{p(X|\theta)} \ge \mathcal{L}\Big(q_z,\theta\Big) $$

with the equality holding true only when $KL(q_z\Big|\Big|p(Z|X,\theta)) = 0$ i.e., when $q_z = p(Z|X,\theta)$.

The EM algorithm is a two-stage iterative optimization technique for finding MLE. In the E step, the lower bound $\mathcal{L}\Big(q_z,\theta^{old}\Big)$ is maximized w.r.t q_z (leaving $\theta$ at this current value, hence the $^{old}$ superscript). This maximum value for the lower bound occurs when $q_z = p(Z|X,\theta^{old})$.

In the subsequent M step, the distribution $q_z$ is held fixed at the new value found in the E step, and the lower bound is maximized w.r.t $\theta$ to get a new estimate $\theta^{new}$. The algorithm then proceeds to the next iteration using $\theta^{new}$ and the process repeats till convergence.

Note that `each iteration of EM algorithm increases the log-likelihood function. In each E step, $q_z$ is set equal to the posterior distribution for
the current parameter values $\theta^{old}$, causing the lower bound to move up to the same value as the log likelihood function, with the KL divergence vanishing. In each M step, $q_Z$ is held fixed and the lower bound $\mathcal{L}(q_z, \theta)$ is maximized with respect to the parameter vector $\theta$ to give a revised value $\theta^{new}$. Because the KL divergence is nonnegative, this causes the log likelihood $\ln{p(X|\theta)$ to increase by at least as much as the lower bound does.`

## Advantages of EM algorithm
- Allows handling of missing inputs/outputs.
- Generalizes to complex models with discrete and real-values hidden variables.
- By attempting to maximize the likelihood, the EM algorithm acts as a Lyapunov function for stable learning.
- Facilitate Bayesian extensions to learning.

## References
[1]: Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.

[2]: M. P. Deisenroth, A. A. Faisal, and C. S. Ong, 2021. https://mml-book.com


## Application - Pressure Poisson Equation

The momentum equation for the velocity field $\vec{u}$ in a fluid is

\begin{equation}
\frac{\partial \vec{u}}{\partial t}+(\vec{u}\cdot\nabla)\vec{u}=-\frac{1}{\rho}\nabla p + \nu \nabla^2\vec{u}
\end{equation}

where $p$ is the pressure, $\nu$ is the fluid viscosity and $\rho$ is the fluid density. With three velocity components, plus the pressure, we have four unknowns but only three equations. For compressible fluids, we have an equation of state to complete the system. In the incompressible case, we don't have an equation of state and we need an additional constraint from somewhere else.

This is what we do: take the divergence of the momentum equation, apply the incompressibility constraint to cancel some terms, and get an equation for the pressure. It's a pretty cool trick.

Conservation of mass for an incompressible fluid requires that the divergence of $\vec{u}$ must be zero:

$$\nabla \cdot \vec{u} = 0$$

Writing out the momentum equation in $x$ and $y$ components (for two-dimensional flow), we get

$$\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y} = -\frac{1}{\rho}\frac{\partial p}{\partial x}+\nu \left(\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2} \right) $$

$$\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial x}+v\frac{\partial v}{\partial y} = -\frac{1}{\rho}\frac{\partial p}{\partial y}+\nu\left(\frac{\partial^2 v}{\partial x^2}+\frac{\partial^2 v}{\partial y^2}\right) $$

We take the divergence of the momentum equation and then apply the incompressibility constraint.  After some wrangling and cancellations, this leaves us with the pressure Poisson equation:

$$\frac{\partial^2 p}{\partial x^2}+\frac{\partial^2 p}{\partial y^2} = -\rho\left(\frac{\partial u}{\partial x}\frac{\partial u}{\partial x}+2\frac{\partial u}{\partial y}\frac{\partial v}{\partial x}+\frac{\partial v}{\partial y}\frac{\partial v}{\partial y} \right)$$

Which is an equation of the form

$$\frac{\partial ^2 p}{\partial x^2} + \frac{\partial ^2 p}{\partial y^2} = b$$

Imagine we discretize a domain using a uniform mesh of points in each spatial direction, as in the figure below:

Then the left-hand side of the Poisson equation, i.e., the Laplacian differential operator applied to $p$, is discretized using 2nd-order central differences as follows

$$\frac{p^n_{i+1, j} - 2p^n_{i,j} + p^n_{i - 1, j}}{\Delta x ^2} + \frac{p^n_{i, j+1} - 2p^n_{i,j} + p^n_{i, j-1}}{\Delta y ^2}$$

where subscripts $i,j$ denote the spatial location on a Cartesian coordinate system and superscripts $n$ denote a point in time.

We apply an appropriate finite-difference discretization to the momentum equation (forward-time, backward-space for the 1st-order terms) and also assume a uniform mesh, so $\Delta x = \Delta y$.

Using this discretized form in the Poisson equation, we will leave only the $p_{i,j}$ terms in the left-hand side, and move the other terms to the right. Then we say that we can update all the values of $p_{i,j}$ using the values at the neighboring points for both $p$ and $u, v$. This update, repeated many times, happens to converge to the solution of Poisson's equation.

## Solution procedure

### Initial velocity field

We start with a velocity field in $u$ and $v$ at some timestep $n$.

### Calculate pressure

Then, we iteratively solve the Poisson equation for pressure, as described above. Starting with an initial guess, the values $p_{i,j}$ are updated using the neighboring values of $p$, $u$ and $v$ at $(i+1,j)$ and $(i,j+1)$. The updates can be written as follows, where the $k$ superscript denotes an iteration in 'pseudo-time':

\begin{align}
p_{i,j}^{k+1} &= \frac{1}{4}\left(p_{i+1,j}^{k}+p_{i-1,j}^{k}+p_{i,j+1}^{k}+p_{i,j-1}^{k}\right) \\
&-\frac{\rho \Delta x}{16} \left( \frac{2}{\Delta t} \left(u_{i+1,j} - u_{i-1,j} + v_{i,j+1} - v_{i,j-1}\right) \right . \\
&-\frac{2}{\Delta x}\left(u_{i,j+1} - u_{i,j-1} \right) \left(v_{i+1,j} - v_{i-1,j} \right) \\
&- \left . \frac{\left(u_{i+1,j} - u_{i-1,j} \right)^2}{\Delta x}
- \frac{ \left(v_{i,j+1} - v_{i,j-1} \right)^2 }{\Delta x} \right) \\
\end{align}

In other words, we repeatedly apply the Poisson equation until the pressure reaches a quasi-steady state.

### Update the velocity

Once the pressure field reaches its quasi-steady state via the Poisson equation, we use that field for the current time step, $p^n$, to solve for the velocity components $u$ and $v$ at the next timestep, $n+1$.

The momentum equation in the $u$ direction:

\begin{align}
u_{i,j}^{n+1} = u_{i,j}^{n} &- \frac{\Delta t}{\Delta x} \left( u_{i,j}^{n}(u_{i,j}^{n}-u_{i-1,j}^{n})
+ v_{i,j}^{n} (u_{i,j}^{n}-u_{i,j-1}^{n}) + \frac{1}{2 \rho}(p_{i+1,j}^{n}-p_{i-1,j}^{n}) \right) \\
&+\frac{\nu \Delta t}{\Delta x^2}\left(u_{i+1,j}^{n} + u_{i-1,j}^{n} + u_{i,j+1}^{n} + u_{i,j-1}^{n} -4u_{i,j}^{n}\right)
\end{align}

The momentum equation in the $v$ direction:

\begin{align}
v_{i,j}^{n+1} = v_{i,j}^{n} &- \frac{\Delta t}{\Delta x} \left( u_{i,j}^{n}(v_{i,j}^{n}-v_{i-1,j}^{n})
+ v_{i,j}^{n} (v_{i,j}^{n}-v_{i,j-1}^{n}) + \frac{1}{2 \rho}(p_{i,j+1}^{n}-p_{i,j-1}^{n}) \right) \\
&+\frac{\nu \Delta t}{\Delta x^2}\left(v_{i+1,j}^{n} + v_{i-1,j}^{n} + v_{i,j+1}^{n} + v_{i,j-1}^{n} -4v_{i,j}^{n}\right)
\end{align}

Then, rinse and repeat.

## What we left out

There are various subtleties that we left out here, to get quickly to the equations we need in the code. First, there are some variations on the form of the pressure Poisson equation, depending on what terms involving $\nabla\cdot\vec{u}$ one chooses to cancel (this has caused long arguments in the literature!). Second, we say nothing about the boundary conditions, which can cause some trouble (and more arguments!). And third, we show only the simplest iterative method for solving the Poisson equation, which also happens to be the slowest to converge. This is just meant to be a pedagogical example and discussion of these subtleties would be part of a full-fledged CFD course.


# Exercise 2: Markov Chains and Markov Decision Processes (MDP)

This exercise deals with the formal handling of Markov chains and Markov decision processes.

## 1) Markov Chain: State Transition
The graph shows the last beer problem.
The nodes show the states.
The arrows define the possible transitions to other states and the numbers besides the arrows define the propability of the corresponding transition.
If you are for example in the state "Inital Beer", with 30% propability you go to have a pizza, with 60% propability you meet friends and with 10% propability you end up sleeping.

Define the state transition probability matrix $\mathcal{P}_{xx'}$ of the graph shown in the figure below!

With $p_k = \begin{bmatrix}
\text{Pr}_k \lbrace \text{Inital Beer} \rbrace \\
\text{Pr}_k \lbrace \text{Meet Friends} \rbrace \\
\text{Pr}_k \lbrace \text{Pizza} \rbrace \\
\text{Pr}_k \lbrace \text{Another Beer} \rbrace \\
\text{Pr}_k \lbrace \text{"Last Beer"}\rbrace \\
\text{Pr}_k \lbrace \text{Sleep} \rbrace \\
\end{bmatrix}^\text{T}$

## 1) Solution

The state transition probability $\mathcal{P}_{xx'} = \begin{bmatrix}
0 & 0.6 & 0.3 & 0 & 0 & 0.1 \\
0 & 0 & 0.4 & 0.4 & 0.1 & 0.1 \\
0 & 0 & 0 & 0.7 & 0 & 0.3 \\
0 & 0 & 0 & 0.6 & 0.3 & 0.1 \\
0 & 0 & 0 & 0 & 0 & 1 \\
0 & 0 & 0 & 0 & 0 & 1
\end{bmatrix}$

## 2 ) Markov Chain: Stationary State
Using $p = p \mathcal{P}$, calculate the stationary state probability.

Please note that the sum of the state propabilities equals one for any specific point in time.

## 2) Solution

\begin{align}
p &= p \mathcal{P} \\
p \left( I_\text{6} - \mathcal{P}_{xx'} \right) &= 0
\end{align}

With:

$\left( I_\text{6} - \mathcal{P}_{xx'} \right) = \begin{bmatrix}
1 & -0.6 & -0.3 & 0 & 0 & -0.1 \\
0 & 1 & -0.4 & -0.4 & -0.1 & -0.1 \\
0 & 0 & 1 & -0.7 & 0 & -0.3 \\
0 & 0 & 0 & 0.4 & -0.3 & -0.1 \\
0 & 0 & 0 & 0 & 1 & -1 \\
0 & 0 & 0 & 0 & 0 & 0
\end{bmatrix}$

Solving for $p$ yields, that $p_{1,...,5} = 0$.

Due to $\sum_n p_n = 1$: $p_6 = 1$

### What does this mean?

In the end we all go to sleep, makes sense.

## 3) Markov Reward Process: Evaluating States

In the following rewards for every state are defined.

Given the reward distribution $r_\mathcal{X}$, calculate the state-values $v_\mathcal{X}$.

The states are defined by:
$\mathcal{X} = \left\lbrace \begin{matrix}
\text{Inital Beer}\\
\text{Meet Friends}\\
\text{Pizza}\\
\text{Another Beer}\\
\text{"Last Beer"}\\
\text{Sleep}\\
\end{matrix}
\right\rbrace$

The rewards are defined by:
$r_\mathcal{X} = \begin{bmatrix}
+1\\
+1\\
+2\\
+1\\
-3\\
0\\
\end{bmatrix}$

The state-value is defined by the state-value Bellman equation: $v_\mathcal{X} = r_\mathcal{X} + \gamma \mathcal{P}_{xx'} v_\mathcal{X}$. Assume that $\gamma = 0.9$ and write a Python program to calculate $v_\mathcal{X}$. Which state is most promising? Why?

Which state is most promising when $\gamma = 0.1$?

## 3) Solution

\begin{align}
v_\mathcal{X} &= r_\mathcal{X} + \gamma \mathcal{P}_{xx'} v_\mathcal{X}\\
v_\mathcal{X} - \gamma \mathcal{P}_{xx'} v_\mathcal{X} &= r_\mathcal{X}\\
\left( I_6 - \gamma \mathcal{P}_{xx'} \right) v_\mathcal{X} &= r_\mathcal{X}\\
v_\mathcal{X} &= \left( I_6 - \gamma \mathcal{P}_{xx'} \right)^{-1} r_\mathcal{X}\\
\end{align}

For $\gamma=0.9$, the state "Initial Beer" has the highest state-value because the highest collectable reward can be expected for the paths one can take from there.

For $\gamma=0.1$, the future collectable reward has lost importance for the evaluation of the state-value, the state-value now gives a rather short-sighted estimation. Thus, the most valuable state can now be found to be the "Pizza"-state, as this state holds the highest immediate reward.

```python
import numpy as np

# define given parameters
gamma = 0.1 # discount factor

### BEGIN SOLUTION

P_xx = np.array([[0, 0.6, 0.3, 0,   0,   0.1],
                 [0, 0,   0.4, 0.4, 0.1, 0.1],
                 [0, 0,   0,   0.7, 0,   0.3],
                 [0, 0,   0,   0.6, 0.3, 0.1],
                 [0, 0,   0,   0,   0,   1],
                 [0, 0,   0,   0,   0,   1]]) # state trasition probability

r_X = np.array([1, 1, 2, 1, -3 ,0]) # rewards
r_X = r_X.reshape(-1, 1) # make column vector

v_X = np.matmul(np.linalg.inv(np.eye(6)-gamma*P_xx) , r_X)

### END SOLUTION

print(v_X)

```

    [[ 1.12751902]
     [ 1.09143404]
     [ 2.06776596]
     [ 0.96808511]
     [-3.        ]
     [ 0.        ]]

## 4) Markov Decision Process: State Transition

The graph shows an MDP.
The nodes are the states.
In every state you can choose between two actions (Lazy or Productive).
Taken actions impact the state transition probability to the next state.
If you for example have a "Hangover" and decide to be "Productive", there is a 30% chance for you to "Visit Lecture" and a 70% chance to stay in the "Hangover" state.

Define the lazy state transition probabilitiy $\mathcal{P}_{xx'}^{u=\text{Lazy}}$ and the productive state transition probability $\mathcal{P}_{xx'}^{u=\text{Productive}}$ of the graph shown in the figure below.

With $p_k = \begin{bmatrix}
\text{Pr}_k \lbrace \text{Hangover} \rbrace \\
\text{Pr}_k \lbrace \text{Sleep} \rbrace \\
\text{Pr}_k \lbrace \text{More Sleep} \rbrace \\
\text{Pr}_k \lbrace \text{Visit Lecture} \rbrace \\
\text{Pr}_k \lbrace \text{Study}\rbrace \\
\text{Pr}_k \lbrace \text{Pass the Exam} \rbrace \\
\end{bmatrix}^\text{T}$

## 4) Solution

\begin{align}
\mathcal{P}_{xx'}^{u=\text{Lazy}}&=\begin{bmatrix}
0 & 1 & 0 & 0 & 0   & 0\\
0 & 0 & 1 & 0 & 0   & 0\\
0 & 0 & 1 & 0 & 0   & 0\\
0 & 0 & 0 & 0 & 0.8 & 0.2\\
0 & 0 & 1 & 0 & 0   & 0\\
0 & 0 & 0 & 0 & 0   & 1\\
\end{bmatrix}\\
\mathcal{P}_{xx'}^{u=\text{Productive}}&=\begin{bmatrix}
0.7 & 0 & 0   & 0.3 & 0   & 0\\
0   & 0 & 0.4 & 0.6 & 0   & 0\\
0   & 0 & 0.5 & 0   & 0.5 & 0\\
0   & 0 & 0   & 0   & 1   & 0\\
0   & 0 & 0   & 0   & 0.1 & 0.9\\
0   & 0 & 0   & 0   & 0   & 1\\
\end{bmatrix}
\end{align}

## 5) Markov Decision Process: Trivial Policy Evaluation

The rewards for this problem are defined by:
$r_\mathcal{X} = r_\mathcal{X}^{u=\text{Productive}} = r_\mathcal{X}^{u=\text{Lazy}} = \begin{bmatrix}
-1\\
-1\\
-1\\
-1\\
-1\\
0\\
\end{bmatrix}$.

How can we interprete these rewards?
Evaluate both the lazy policy and the productive policy using $\gamma = 0.9$.

Bonus question: Can we evaluate the state-value of $\lbrace x=\text{More Sleep}, u=\text{Lazy}\rbrace$ for an infinite time horizon without the use of the Bellman equation?

## 5) Solution

The given rewards can be interpreted as a penalty for wasting time. The earlier you pass, the better :)

```python
import numpy as np

### BEGIN SOLUTION

P_xx_lazy = np.array([[0, 1, 0, 0, 0,   0],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 0, 0, 0.8, 0.2],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 0, 0, 0,   1]])

P_xx_productive = np.array([[0.7,  0,   0,   0.3, 0,   0],
                            [0,    0,   0.4, 0.6, 0,   0],
                            [0,    0,   0.5, 0,   0.5, 0],
                            [0,    0,   0,   0,   1,   0],
                            [0,    0,   0,   0,   0.1, 0.9],
                            [0,    0,   0,   0,   0,   1]])
gamma = 0.9
r_X = np.array([-1, -1, -1, -1, -1, 0]).reshape(-1, 1)
for P_xx in [P_xx_lazy, P_xx_productive]:
    v_X = np.matmul(np.linalg.inv(np.eye(6)-gamma*P_xx) , r_X)
    print(v_X)

### END SOLUTION
```

    [[-10. ]
     [-10. ]
     [-10. ]
     [ -8.2]
     [-10. ]
     [  0. ]]
    [[-4.15414315]
     [-3.05228771]
     [-2.71728272]
     [-1.98901099]
     [-1.0989011 ]
     [ 0.        ]]

```python
# Bonus question: Can we evaluate the state-value of {𝑥=More Sleep,𝑢=Lazy} for an infinite time horizon without the use of the Bellman equation?

### BEGIN SOLUTION
# We can evaluate the "More Sleep" state for an infinite time horizon because the reward will stay the same forever.
# Thus, we can make use of the geometric series convergence. Due to numeric inaccuracy results may deviate slightly.
import numpy as np

v_3 = -1*1/(1-0.9)
print(v_3)

### END SOLUTION
```

    -10.000000000000002

## 6) Action-Value Function Evalution

Now, the policy is defined by:
\begin{align}
\pi(u_k=\text{Productive} | x_k)&=\alpha,\\
\pi(u_k=\text{Lazy} | x_k)&=1-\alpha, \forall x_k \in \mathcal{X}
\end{align}

Calculate action-values for the problem as described using the 'fifty-fifty' policy ($\alpha = 0.5$) according to the Bellman Expectation Equation: $q_\pi(x_k, u_k) = \mathcal{R}^u_x + \gamma \sum_{x_{k+1} \in \mathcal{X}} p^u_{xx'} v_\pi(x_{k+1})$ $\forall x_k, u_k \in \mathcal{X}, \mathcal{U}$.

## 6) Solution

```python
import numpy as np

gamma = 0.9
alpha = 0.5
no_states = 6
no_actions = 2
r_X = np.array([-1, -1, -1, -1, -1, 0]).reshape(-1, 1)
q_XU = np.zeros([no_states, no_actions])
### BEGIN SOLUTION

P_xx_lazy = np.array([[0, 1, 0, 0, 0,   0],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 0, 0, 0.8, 0.2],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 0, 0, 0,   1]])

P_xx_productive = np.array([[0.7,  0,   0,   0.3, 0,   0],
                            [0,    0,   0.4, 0.6, 0,   0],
                            [0,    0,   0.5, 0,   0.5, 0],
                            [0,    0,   0,   0,   1,   0],
                            [0,    0,   0,   0,   0.1, 0.9],
                            [0,    0,   0,   0,   0,   1]])

P_xx_mean = alpha*P_xx_productive + (1-alpha)*P_xx_lazy
v_X = np.linalg.inv(np.eye(6)-gamma*P_xx_mean) @ r_X
transition_tabs = np.dstack([P_xx_lazy, P_xx_productive])
q_XU = r_X + gamma * np.squeeze(np.transpose(transition_tabs, [0, 2, 1]) @ v_X)

print(q_XU)

### END SOLUTION
```

    [[-6.07176629 -5.82932926]
     [-6.07830245 -5.1922893 ]
     [-6.07830245 -5.20681411]
     [-3.66826061 -4.33532576]
     [-6.07830245 -1.33353258]
     [ 0.          0.        ]]

## 7) Markov Decision Problem: Stochastic Policy Evalution

Plot the state-value of the states "Lecture" and "Study" for different $\alpha$. What do we see? Why?

## 7) Solution

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn-talk')

n = 6 # dimension of state space
no_of_samples = 1000

alphas = np.linspace(0, 1, no_of_samples)
v_n_alpha = np.zeros([n, no_of_samples])

### BEGIN SOLUTION
gamma = 0.9
P_xx_lazy = np.array([[0, 1, 0, 0, 0,   0],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 0, 0, 0.8, 0.2],
                      [0, 0, 1, 0, 0,   0],
                      [0, 0, 0, 0, 0,   1]])
P_xx_productive = np.array([[0.7,  0,   0,   0.3, 0,   0],
                            [0,    0,   0.4, 0.6, 0,   0],
                            [0,    0,   0.5, 0,   0.5, 0],
                            [0,    0,   0,   0,   1,   0],
                            [0,    0,   0,   0,   0.1, 0.9],
                            [0,    0,   0,   0,   0,   1]])
r_X = np.array([-1, -1, -1, -1, -1, 0]).reshape(-1, 1)

alphas = alphas.reshape(-1, 1, 1)
P_xx_mean = alphas * P_xx_productive + (1 - alphas)*P_xx_lazy
v_n_alpha = np.squeeze(np.linalg.inv(np.eye(6).reshape(1, 6, 6) - gamma * P_xx_mean) @ r_X).T

### END SOLUTION
```

```python
plt.figure(figsize=[10, 6])
states = ["Hangover", "Sleep", "More Sleep", "Visit Lecture", "Study", "Pass Exam"]
alphas = alphas.flatten()
for state, vnalp in zip(states, v_n_alpha):
    ls = '--' if state in ['Visit Lecture', 'Study'] else '-'
    plt.plot(alphas, vnalp, ls=ls, label=r"$x=${}".format(state))

plt.legend()
plt.xlabel(r"$\alpha$")
plt.ylabel(r"$v_\pi(x)$")
plt.xlim([0, 1])
plt.ylim([-10, 0])
```

If you are rather lazy (low $\alpha$), there is a good chance a studying session will end in just going to sleep. Hence, your chance (in general) to pass is lower if you tend to be lazy (due to lower $v_\pi(x)$), but you should at least visit the lecture in order to go to the exam.

If you are more productive (high $\alpha$), studying raises your chances for passing significantly.

Result: In order to be as lazy as possible, please visit the lecture :)


SymPy Cheat Sheet

```python
from sympy import symbols
x, y, z, a = symbols('x y z a')
init_printing(use_unicode=True)     # allow for pretty symbolic printing
```

###Integral of sine function:

$\int{sin(x)} dx$

```python
integrate(sin(x), (x))
```

###Differentiation of sine function:

$\frac{d}{dx}sin(x)$ = $cos(x)$

```python
diff(sin(x),x)
```

```python
factor(x**3+2*x)
```


# Contents
* Singular Value Decomposition (SVD)
* Principal Component Analysis (PCA)
* Proper Orthogonal Decomposition (POD)
* Dynamic Mode Decomposition (DMD)
* Time-delayed Embeddings

## Linear Transformations

$x = \left[\begin{array}{ccc}
1 \\
3
\end{array}\right],\; \mathbf{A}=\left[\begin{array}{ccc}
2 & 1 \\
-1 & 1
\end{array}\right] \rightarrow \,y=\mathbf{Ax}=\left[\begin{array}{ccc}
5 \\
2
\end{array}\right]$

**Rotation**
$\mathbf{A}=\left[\begin{array}{ccc}
\text{cos}\theta & \text{-sin}\theta \\
\text{sin}\theta & \text{cos}\theta
\end{array}\right]$

**Scaling**
$\mathbf{A}=\left[\begin{array}{ccc}
\alpha & 0 \\
0 & \alpha
\end{array}\right]$

**SVD Properties**

$\mathbf{A}=\mathbf{U\Sigma V^*} $

$\mathbf{U} \in \mathbb{C}^{mxm}$ is unitary

$\mathbf{V} \in \mathbb{C}^{nxn}$ is unitary

$\mathbf{\Sigma} \in \mathbb{R}^{mxn}$ is diagonal

**Computing SVD - Eigenvalur Problem**

$\begin{align}
\mathbf{A^TA} &= \mathbb{U\Sigma V^*}^T\mathbf{U\Sigma V^*}\\
& = \mathbf{V\Sigma^2V^*}
\end{align}$

$\mathbf{AA^T}= \mathbf{U\Sigma^2U^*}\;\;$ This then will become: $(AA^T)U= U\Sigma^2$.
So eigenvalue is $\Sigma^2$, eigenvector is $U$

**Question**: Are unitary matrices U, V unique? Are the singular values unique? Explain and provide several examples if they are not.

>   No, unitary matrices U, V are not unique. The singular values are unique. Given an A matrix, the function of A acting on any vector is to rotate that vector and scale it by some factor.

A = UΣV* acting on any vector means first inverse rotate it by V, scale it, then rotate by U. Suppose we rotate a vector by 20 degrees, and at the mean time scale it by factor of 3. We can first rotate it by 10 degrees, scale it by factor of 3, then rotate another 10 degrees. Or, we can first rotate it by 30 degrees, scale it by factor of 3, then rotate back 10 degrees. There are many more ways to rotate, but only one way to scale it because we only scale it once. So U, V are not unique. Singular values are unique.

Another proof of non-uniqueness of U, V:
 $A = U\Sigma V^* = U\Sigma WW^*V^* = (UW)\Sigma (W^*V^*) = (UW)\Sigma (VW)^*$
So U, V could be replaced by any other combinations with W, where W is a unitary matrix.

**Properties of SVD**

* **Theorem**: The *nonzero singular values* of $\mathbf{A}$ are the square roots of the nonzero eigenvalues of $\mathbf{A^*A}$ or $\mathbf{AA^*}$
* **Theorem**: The norm $||\mathbf{A}||_2 =\sigma_1$ (max eigenvalues) and $||\mathbf{A}||_F = \sqrt{\sigma_1^2+\sigma_2^2+ \cdot\cdot\cdot+\sigma_r^2} $
* **Theorem**: For any N so that $0\le N\le r$, we can define the partial sum

    $$\mathbf{A}_N =\sum_{j=1}^N \sigma_j \mathbf{u}_j\mathbf{v}_j^* $$
    $||\mathbf{A}-\mathbf{A_N}||_N = \sqrt{\sigma_{N+1}^2+\sigma_{N+2}^2+\cdot\cdot\cdot+\sigma_r^2} $

## PCA
subtract the mean for all variables, define the covariance matrix $X^TX$.

**Embedding**
Project each point on Principal axies - low dimension embedding

## POD

$ f(x,t)\approx \sum_{j=1}^N a_j(t)\phi_j(x) \rightarrow \; F(x,t)\approx U(x)\Sigma V(t)^T $

## DMD

DMD: Computes a set of **modes** associated with fixed **oscillation frequencies** and **decay/growth** rates for given multi dimensional time series.

Predict the next snapshot from previous one:  $v_{i+1}=Av_i $

**Algorithm**

Assume that we've taken *m* measurements from specific points in sapce for *n* time spoints, where for now assume that $n<m$. Define the data matrix as :

$X =\left[\begin{array}{ccc}
| & | & &| \\
\vec{x_1} & \vec{x_2} &...&\vec{x_{n}}\\
| & | & &| \\
\end{array}
\right] $

Define two matrices $X^*$ and $Y$, such that we can write $Y \approx AX^*$,

$X^* =\left[\begin{array}{ccc}
| & | & &| \\
\vec{x}_1 & \vec{x}_2 &...&\vec{x}_{n-1}\\
| & | & &| \\
\end{array}
\right] $

$Y =\left[\begin{array}{ccc}
| & | & &| \\
\vec{x}_2 & \vec{x}_3 &...&\vec{x}_n\\
| & | & &| \\
\end{array}
\right] $

 if *n* is small, this is relatively easy to compute - however, if *n* is large, we use SVD.

Let $X^* = U\Sigma V^T$ and $X^+=V\Sigma^{-1}U^T$

such that we can write

$YX^+=YV\Sigma^{-1}U^T=AX^*X^+=A$

Additionally, if we assume that $rank(X^*)=r\le n$, then we can use the truncated SVD such that

$U_r = U[:,:r]$, $V^T = V^T[:r,:]$, as well as $\Sigma_r= \Sigma[:r,:r]$,

Now we have a low-rank system, we can apply similarity transformation to *A* to reduce its dimensionality, without changing its spectrum. Using our spatial singular vectors $U$, we define

$\begin{align*}
\tilde{A}&=U_r^TAU_r\\
&=U_r^T(YV_r\Sigma_r^{-1}U_r^T)U_r\\
&=U_r^TYV_r\Sigma_r^{-1}
\end{align*}$

where $\tilde{A} \in \mathbb{R}^{r\times r}$. We now have a reduced-dimensional representation of our linear operator, from which we can compute the spatial modes and dynamic behavior of each mode. First, however, because of the notion of variance captured by the singular values of our original predictor matrix, we weight $\tilde{A}$ by the singular values as

$$\hat{A}=\Sigma^{1/2}A\Sigma^{1/2}$$

such that our computed spatial modes have been weighted by the amount they contribute to our measured signal. We can now compute the eigendecomposition of $\hat{A}$ as

$$\hat{A}W=W\Lambda $$

So that spatial modes becomes $\Phi = X^*(V\Sigma^{1/2})W$ .

## Time Delay Embeddings

Build delay matrix for variable $x_1$, $H = [x_1;x_1^\delta;...;x_1^{(k+1)\delta}] \;\; H =U\Sigma V^T$
Takens's Therorem: Attractor with dim *d* can be embedded in Euclidean space with *k>2d* lags.

Example: Lorentz attractor

## PCA Embedding - Truncation

d dimension -> lower (2d) dimension

## Manifold Learning

**manifold learning**: a subfiled of machine learning closely related to dimension reduction based on the assumption that one's observed data lie on a low-dimensional manifold embedded in a higher-dimensional space.

* Manifold Visualization
* Manifold approximation

## Multidimensional Scaling (MDS)

Multidimensional scaling is a **visual representation of distances or dissimilarities between sets of objects** "Objects" can be any kind of real or conceptual stimuli (e.g. colors, faces, map coordinates, etc).

* Objects that are more similar (or have shorter distances) are closer together on the graph than objects that are less similar (or have longer distances).
* MDS can also serve as a dimension reduction technique for high-dimensional data, as well as interpreting dissimilarities as distances on a graph.

**MDS Algorithm**

Compute pairwise distance between all pairs of points

$D:= \left[\begin{array}{ccc}
d_{1,1} & d_{1,2} & ... & d_{1,M}\\
d_{2,1} & d_{2,2} & ... & d_{2,M} \\
. &. & &.\\
. &. & &.\\
. &. & &.\\
d_{M,1} & d_{M,2} &... & d_{M,M}
 \end{array}\right]$

 $d_{ij} = ||x_i-x_j|| $

 Center the matrix D (double centering in rows and columns)  $\tilde{D} -> EVD(U\sqrt{\Sigma})$

**Question**: Consider $D_{n\times n}=[d_{ij}^2], $ a matrix of squared pairwise Euclidean distances between variables in the matirx $X_{n\times k}$ (rows are variables, columns are observations). Show that eigenvalues $\lambda_m$ and eigenvectors $\vec{u_m}$ of $\tilde{D}_{n\times n}$, a matrix $D$ that is centered in rows and columns, correspond to $\lambda_m=s_m^2,\vec{u_m}$ respectively, where $s_m$ are diagonal elements in $S$ and $\vec{u_m}$ are vectors in $U$, such that $X=USV^T,$ i.e. SVD of a centered matrix $X.$

This thus shows that MDS (finding representation that preserves Euclidean distances of points in $X$) and PCA of $X$ are equivalent.

> Consider the $D$ matrix, with $d_{ij}= ||x_i-x_j||^2$, where $\displaystyle X    =\left(\begin{array}{ccc}
   x_1\\
   \hline
   x_2\\
   \hline
   .\\.\\.\\\hline x_n
   \end{array}\right) $

   $d_{ij}= ||x_i-x_j||^2 = x_i^2 -2x_i\cdot x_j+x_j^2$.

   Specifically, $D := \left(\begin{array}{ccc}
   0 & ||x_1-x_2||^2 & ...\\
   ||x_2-x_1||^2 & 0 & ...\\
   ...\\...\\
   ||x_n-x_1||^2& ...&0
   \end{array}\right)$

   We want to double center the $D$ matrix, in a way that:

   Sum of rows $s_i = \displaystyle \sum_j d_{ij}^2 = \displaystyle \sum_j  x_i^2 -2x_i\cdot x_j+x_j^2 = nx_i^2-2x_i\cdot\sum_j x_j + \sum_jx_j^2$.

   Sum of all entries in $D:$ $$s= \displaystyle \sum_i s_i = n\sum_i x_i^2 -2\sum_i x_i \cdot \sum_j x_j+ \sum_i\sum_jx_j^2=2n\sum_ix_i^2-2\sum_i x_i \cdot \sum_j x_j$$

   Centering by columns such that $\displaystyle\sum_i x_i =0$, we get:

   $s_i=nx_i^2+\displaystyle\sum_jx_j^2$

   $s=\displaystyle 2n\sum_i x_i^2$

   Now we have, $$d_{ij}-\frac{s_i+s_j-\frac{1}{n}s}{n} = -2x_i\cdot x_j$$

   Divide this by -2, we have $\tilde{D}$ matrix, making the equation above to be, $RHS= x_i\cdot x_j$

   Consider the matrix $XX^T$, each entry $(XX^T)_{ij} = x_i\cdot x_j$.

   Thus the two matrices are equivalent. The results of SVD must be equivalent too. And the singular value for $A$ is indeed the square root of the singular value for $\tilde{D}$ with the same eigenvectors.

 ## ISOMAP

 **Isomap: nonlinear dimensionality reduction** estimating the intrinsic geometry of a data manifold based on a rough estimate of each data point's neighbors on the manifold.

 * **Determine the neighbors of each point**

    All points in some fixed radius

    K nearest neighbors
* **Construct a neigborhood graph**

   Each point is connected to other if it is a *K* nearest neighbor

   Edge length equal to Euclidean distance
* **Compute shortest path between two nodes**

   Dijkstra's algorithm

   Floyd-Warshall algorithm
* **Compute lower-dimensional embedding**

   Multidimensional scaling

## t-SNE: T Stochastic Neighborhood Embedding

Optimization approach.

**Crowding Problem**: It is impossibl to preserve distance in all Neighborhoods

**Possible solution**: t-distribution

If points are crowded, t-distribution looks like standard normal distribution. If they are sparse, t-distribution is more flat and smooth.

## (Temporal) Force Directed Graphs

Embedding -> Visualization

# Embedding and Clustering

* UMAP
* Clustering
* K Means
* K Nearest Neighbors (KNN)

## UMAP
**Uniform Manifold Aapproximation and Projection**

Mclnnes, Leland, John Healy, and James Melville. Umap(2018)

**Neighbor Graph**

one point - 0 simplex, two points connected - 1 simplex

The idea is that define a ball of radius r for each data point. For sparse region, the data points have larger radius. For dense region, they have smaller radius. It is complicated to measure the density of data points, we instead find the associated k nearest neighbors. This is a hyperparameter to tune.

Example: UMAP on MNIST hand-written digits, unsupervised learning.

## Clustering

### K-Means
* Unsupervised
* K - number of clusters (centers)
* Edclidean Distance
* Iterative Process

* An iterative clustering algorithm
- Initialize: Pick *K* random points as cluster centers
- Alternate:
    1. Assign data points to closest cluster center
    2. Change the cluster center to the average of its assigned points
- Stop when no points' assignments change

**Properties of K-means algorithm**
* Guaranteed to converge in a finite number of iterations
* Running time per iteration
    1. Assign data points to closest cluster center

    O(KN) time
    2. Change the cluster...

    O(N)
* Symmetric
    - D(A,B) =D(B,A)
    - Otherwise, we can say A looks like B but B does not look like A
* Positivity, and self-similarity
    - D(A,B) $\ge$ 0, and D(A,B)=0 iff A=B
    - Otherwise there will be different objects that we cannot tell apart
* Triangle inequality
    - D(A,B) +D(B,C) $\ge$ D(A,C)
    - Otherwise one can say "A is like B, B is like C, but A is not like C at all"

**K-means Convergence**: K-means taks an alternating optimization approach, each step is guaranteed to decrease the objective - thus guaranteed to converge

**K-Means Initialization**
* K-means algorithm is a heuristic
    - Requires inital means
    - It does matter what you pick!

    - What can go wrong?
    - Various schemes for preventing this kind of thing: variance-based split/merge, initialization heuristics

```python
import torch
import matplotlib.pyplot as plt
```

```python

## K Means Clustering
epochs = 10

x = torch.rand(50,2)
N,D = x.shape    # N number of samples, D dimension of one sample in space

K =3 # number of clusters

c = x[:K,:].clone() # use for updating cluster centers, initialize it with first 3 samples

xx= torch.Tensor(x.view(N,1,D))  ## dimension view (N,1,D)
yy= torch.Tensor(c.view(1,K,D))  ## dimension view (1,K,D)

for i in range(epochs):

    # Alternate 1: assign each point to its nearest cluster center

    D_xy = ((xx-yy)**2).sum(-1)   ## xx-yy dimension (N,K,D), substract each sample by K cluster centers in K columns
                               ## and then square and sum to get the distance d**2
    cl = D_xy.argmin(dim=1).long().view(-1)    # for each sample, get the nearest cluster center

    # Alternate 2: update the new cluster centers by the average of summed assigned points in groups
    c.zero_()       # zeros centers
    c.scatter_add_(0, cl[:,None].repeat(1,D),x)  # scatter add function: sum samples assigned to each cluster

    # Divide by the number of samples in each cluster
    Ncl = torch.bincount(cl,minlength=K).type_as(c).view(K,1)
    c/=Ncl
    plt.figure(figsize=(3,3))
    if i%3==0:
        cs=['r','g','b']
        for j in range(K):
            mask=cl==j
            plt.scatter(x[:,0][mask],x[:,1][mask],c=cs[j])
            plt.title(f'Iterations {i}')
            plt.scatter(c[j,0],c[j,1],c=cs[j],marker='x')
        plt.show()

```

## K-Nearest Neighbors (KNN)
* Learning Algorithm
    - Store training examples
* Prediction Algorithm:
    - To classify a new example x by finding the training example ($x^i,y^i$) that is *nearest* to x
    - Guess the class $y=y^i$

Week 8_Data Organization: Decompositions, Embeddings, and Clustering.ipynb


$\newcommand{\pr}{\textrm{Pr}}$
$\newcommand{\l}{\left}$
$\newcommand{\r}{\right}$
$\newcommand\given[1][]{\:#1\vert\:}$
$\newcommand{\var}{\textrm{Var}}$
$\newcommand{\mc}{\mathcal}$
$\newcommand{\lp}{\left(}$
$\newcommand{\rp}{\right)}$
$\newcommand{\lb}{\left\{}$
$\newcommand{\rb}{\right\}}$
$\newcommand{\iid}{\textrm{i.i.d. }}$
$\newcommand{\ev}{\textrm{E}}$
$\newcommand{\odds}{\textrm{odds}}$
$\newcommand{\normal}{\textrm{normal}}$
$\newcommand{\gamma}{\textrm{gamma}}$
$\newcommand{\mode}{\textrm{mode}}$
$\newcommand{\MSE}{\textrm{MSE}}$

# 5.1 The normal model

A random variable $Y$ is said to be normally distributed with mean $\theta$ and
variance $\sigma > 0$ if the density of $Y$ is given by
$$p\lp y \given \theta, \sigma \rp = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2}\lp\frac{y-\theta}{\sigma}\rp^2} $$
for $-\infty < y < \infty$. Some important things about this distribution:

* the distribution is symmetric around $\theta$, and the mode, median, and mean are all equal to $\theta$
* about 95% of the population lies with 95% of the mean (more precisely, 1.96 stdevs)
* if $X \sim \normal\lp \mu, \tau^2\rp$, $Y \sim \normal\lp\theta,\sigma^2\rp$, and $X$ and $Y$ are independent,
than $aX + bY \sim \normal\lp a\mu + b\theta, a^2\tau^2 + b^2\sigma^2\rp$
* the R commands for working with normal distribution take the standard deviation $\sigma$, not the variance
$\sigma^2$ as input

The normal distribution is particularly important because the central limit theorem says that
under very general conditions, the sum (or mean) of a set of random variables is approximately
normally distributed. This means the normal model will be appropriate for data that results
from the additive effects of a large number of factors.

# 5.2 Inference for the mean, conditional on the variance

Suppose our model is $\lb Y_1, \cdots, Y_n \given \theta, \sigma^2\rb \sim \iid \normal\lp\theta, \sigma^2\rp$.

$\lb \sum y_i^2, \sum y_I \rb$ make up a two-dimensional (vector) sufficient statistic for the
normal model. Knowing these values
is the same as knowing the sample mean $\bar{y} = \sum y_i /n$ and the sample variance
$s^2 = \sum \lp y_i - \bar{y} \rp^2 / \lp n-1\rp$, so $\lb \bar{y}, s^2 \rb$ is also a sufficient statistic.

A class of prior distributions is conjugate for a sampling model if the resulting posterior
distribution is in the same class. If a (conditional) prior distribution $p \lp \theta \given \sigma^2 \rp$
is normal and $y_1, \cdots, y_n \sim \iid \normal \lp\theta,\sigma^2\rp$, the posterior distribution
$p\lp\theta\given y_1, \cdots, y_n, \sigma^2 \rp$ will also be normally distributed. If
$\theta \sim \normal\lp\mu_0,\tau^2_0\rp$, then the posterior variance is
$$\tau_n^2 = \frac{1}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}}$$
and the posterior mean is
$$\mu_n = \frac{\frac{1}{\tau_0^2}\mu_0 + \frac{n}{\sigma^2}\bar{y}}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}}$$

$\tau_0^2$ is the prior variance and $\sigma^2$ is the actual sampling variance of the data (how close the $y_i$'s are to $\theta$). $\tau_n^2$ is the posterior variance of our estimate of $\theta$. See next section for more.

## Combining information

The (conditional) posterior parameters $\tau_n^2$ and $\mu_n$ combine the prior parameters $\tau_0^2$
and $\mu_0$ with terms from the data.

### Posterior variance

The formula for $1/\tau_n^2$ is
$$\frac{1}{\tau_n^2} = \frac{1}{\tau_0^2} + \frac{n}{\sigma^2}$$
and so the prior inverse variance is combined with inverse of the data variance.
The inverse variance is often referred to as the **precision**. For the normal model, let

* $\tilde{\sigma}^2 = 1 / \sigma^2 = $ sampling precision (how close the $y_i$'s are to $\theta$)
* $\tilde{\tau}_0^2 = 1 / \tau_0^2 = $ prior precision
* $\tilde{\tau}_n^2 = 1 / \tau_n^2 = $ posterior precision

Precision is a quantity that works on an additive scale:
$$\tilde{\tau}_n^2 = \tilde{\tau}_0^2 + n\tilde{\sigma}^2$$
In other words, posterior information = prior information + data information.
Here, our uncertainty in our estimate of $\theta$ is the sum of our prior uncertainty
plus the sampling variability in the data. Here, we assume we know the sampling variability
but we will jointly estimate it in the next section.

If this is confusing, remember that we are only doing inference for $\theta$ here, so we will
come up with a posterior distribution for $\theta$. That posterior distribution is normal
with mean $\mu_n$ and variance $\tau_n^2$. We need these two parameters to define the posterior
normal distribution for $\theta$. In the next section, when we do joint inference
for both $\theta$ and $\sigma^2$, we will still get a mean and variance for $\theta$. We will
also get one or more parameters that define the posterior distribution of $\sigma^2$, depending
on what the posterior distibution is.

### Posterior mean

Notice that
$$\mu_n = \frac{\tilde{\tau}_0^2}{\tilde{\tau}_0^2 + n\tilde{\sigma}^2}\mu_0 +
\frac{n\tilde{\sigma}^2}{\tilde{\tau}_0^2 + n\tilde{\sigma}^2}\bar{y}$$
so the posterior mean is a weighted average of the prior mean and the sample mean.
The weight on the sample mean is $n / \sigma^2$ (ignoring the denominator), the sampling
precision of the sample mean. The weight on the prior mean is $1/\tau_0^2$, the prior
precision. If the prior mean was based On $\kappa_0$ prior observations from the same
or similar population as $Y_1, \cdots, Y_n$ then we can set $\tau_0^2 = \sigma^2 / \kappa_0$,
the variance of the mean of the proir observations. The formula for the posterior mean
then reduces to
$$\mu_n = \frac{\kappa_0}{\kappa_0 + n}\mu_u + \frac{n}{\kappa_0 + n}\bar{y}$$

### Prediction

If we want to predict a new sample $\tilde{Y}$ from the population after having observed
$Y_1=y_1, \cdots, Y_n=y_n$, we can use the fact that
$$\lb \tilde{Y} \given \theta, \sigma^2 \rb \sim \normal\lp\theta, \sigma^2\rp \iff
\tilde{Y} = \theta + \tilde{\epsilon}, \lb\tilde{\epsilon} \given \theta, \sigma^2\rb \sim
\normal\lp 0, \sigma^2\rp$$
In other words, since $\tilde{Y}$ is normally distributed, it can be represented as its
mean plus some normally distributed noise $\tilde{\epsilon}$. We can easily compute the
predictive mean and variance and find that
$$\tilde{Y} \given \sigma^2, y_1, \cdots, y_n \sim \normal\lp\mu_n,\tau_n^2 + \sigma^2\rp$$
$\tau_n^2$ is our uncertainty in our estimate for the mean and $\sigma^2$ is the sampling
variability. As our data increases, $\tau_n^2$ will decrease but the sampling variability
will always remain.

# 5.3 Joint inference for the mean and variance

We perform joint inference in a similar way as for one parameter. Starting with Bayes' rule,
\begin{align}
p\lp\theta, \sigma^2\given y_1,\cdots,y_n\rp &= \frac{p\lp y_1,\cdots,y_n \given \theta,\sigma^2\rp
p\lp\theta,\sigma^2\rp}{p\lp y_1,\cdots,y_n \rp} \\
&= \frac{p\lp y_1,\cdots,y_n \given \theta,\sigma^2\rp p\lp\theta \given \sigma^2\rp p\lp \sigma^2\rp}
{p\lp y_1,\cdots,y_n \rp}
\end{align}
We already have a conjugate prior for the mean given the variance so we just need one for
the variance $p\lp \sigma^2 \rp$. Let's consider the
particular case where $\tau_0^2 = \sigma^2 / \kappa_0$. In other words, the prior variance is
equal to the actual sampling variance scaled by the inverse of $\kappa_0$, our prior sample size.
So far we have
$$ p\lp\theta \given \sigma^2\rp p\lp \sigma^2\rp = \textrm{dnorm}\lp\theta,\mu_0,\tau_0 = \sigma / \sqrt{\kappa_0}\rp
\times p\lp \sigma^2\rp$$
We need a prior that has support on $\lp0, \infty\rp$. One such family is the gamma family, though
it turns out that the gamma family is conjugate for $1/\sigma^2$. When using such a prior
distriubtion, we say that $\sigma^2$ has an **inverse-gamma** distribution:

* precision = $1/\sigma^2 \sim \textrm{gamma}\lp a,b\rp$
* variance = $\sigma^2 \sim \textrm{inverse-gamma}\lp a,b\rp$

We will reparameterize this as
$$1/\sigma^2 \sim \gamma\lp \frac{\nu}{2}, \frac{\nu}{2}\sigma_0^2\rp$$
Under this parameterization,
* $\ev\l[\sigma^2\r] = \sigma_0^2\frac{\nu_0 / 2}{\nu_0 / 2 - 1}$
* $\mode\l[\sigma^2\r] = \sigma_0^2\frac{\nu_0 / 2}{\nu_0 / 2 + 1}$, so
$\mode\l[\sigma^2\r] < \sigma_0^2 < \ev\l[\sigma^2\r]$
* $\var\l[\sigma^2\r]$ is decreasing in $\nu_0$

## Posterior inference

In the above section, we decomposed the prior distribution
$$p\lp\theta,\sigma^2\rp = p\lp\theta\given\sigma^2\rp p\lp\sigma^2\rp$$
Now we will decompose the posterior in the same way
$$p\lp\theta,\sigma^2 \given y_1,\cdots,y_n\rp = p\lp \theta \given \sigma^2, y_1,\cdots, y_n\rp
p\lp \sigma^2 \rp$$
We've already calculated the conditional distribution of $\theta$ given $\sigma^2$ and the data
above:
$$\lb \theta \given y_1,\cdots,y_n,\sigma^2\rb \sim \normal\lp\mu_n,\sigma^2/\kappa_n\rp$$
where
$$\kappa_n = \kappa_0 + n$$
and
$$\mu_n = \frac{\lp\kappa_0 / \sigma^2\rp\mu_0 + \lp n/\sigma^2\rp\bar{y}}{\kappa_0/\sigma^2 + n/\sigma^2} =
\frac{\kappa_0\mu_0 + n\bar{y}}{\kappa_n}$$

The posterior distribution of $\sigma^2$ can be obtained by integrating over the unknown value of
$\theta$ which gives
$$\lb 1/\sigma^2\given y_1,\cdots, y_n\rb\sim\gamma\lp\nu_n/2,\nu_n\sigma^2_n/2\rp$$
where
$$\nu_n = \nu_0 + n$$
and
$$\sigma^2_n = \frac{1}{\nu_n}\l[\nu_0 \sigma^2_0 + \lp n-1 \rp s^2 + \frac{\kappa_0 n}{\kappa_n}
\lp \bar{y} - \mu_0\rp^2\r]$$
These formulae suggest an interpretation of $\nu_0$ as a prior sample size, from which a prior
sample variance of $\sigma^2_0$ has been obtained. $s^2$ is the sample variance and $\lp n-1\rp s^2$
is the sum of the squared observations from the sample mean (sum of squares). We can think of
$\nu_0\sigma^2_0$ and $\nu_n\sigma^2_n$ as the prior and posterior sum of squares, respectively
(I suppose because they have the same "(sample size) $\times$ (variance)" form as $\lp n-1 \rp s^2$). Multiplying the
expression for $\sigma_n^2$ by $\nu_n$ almost gives us "posterior sum of squares equals
prior sum of squares plus data sum of squares." However, there is a third term
$\frac{\kappa_0 n}{\kappa_n} \lp \bar{y} - \mu_0\rp^2$. This term says that a large value $\lp \bar{y} - \mu_0\rp^2$
increases the posterior probability of a large $\sigma^2$. This makes sense for our particular
joint prior distribution for $\theta$ and $\sigma^2$: if we want to think of $\mu_0$ as the sample mean
of $\kappa_0$ prior observations with variance $\sigma^2$, then $\frac{\kappa_0 n}{\kappa_n} \lp \bar{y} - \mu_0\rp^2$
is an estimate of $\sigma^2$ and so we want to use the information this term provides. We will develop
an alternative prior distribution in the following section for situations where $\mu_0$ is not the
mean of prior observations.

## Monte Carlo sampling

Often we are interested in the population mean $\theta$ and we
just want to calculate quantities like $\ev\l[y_1\cdots,y_n\r]$, $\textrm{sd}\l[y_1,\cdots,y_n\r]$,
$\pr\l[\theta_1 < \theta_2 \given y_{1,1},\cdots,y_{n_2,2}\r]$, etc. As discussed in the last chapter,
we can obtain these quantities by sampling $\theta$ from the **marginal posterior distribution** of $\theta$
given the data $p\lp\theta\given y_1,\cdots, y_n\rp$.

Why is this a marginal distribution? It's marginal because we need to marginalize out $\sigma^2$ (I think).
This interpretation would seem to agree with section 4.4. Note that this is sampling from the posterior
predictive distribution.

So far, we have the conditional distribution of $\theta$ given $\sigma^2$ and the data. We can generate samples of
$\theta$ from the joint distribution of $\theta$ and $\sigma^2$ by first sampling $\sigma^2$ from
its inverse gamma and then using the sampled $\sigma^2$ to sample $\theta$:
\begin{align}
\sigma^{2\lp 1\rp} \sim \textrm{inverse gamma}\lp\nu_0/2, \sigma^2_n\nu_n/2\rp),&
\theta^{\lp 1 \rp} \sim \normal\lp\mu_n, \sigma^{2\lp 1\rp} / \kappa_n\rp, \\
\cdots, \\
\sigma^{2\lp S\rp} \sim \textrm{inverse gamma}\lp\nu_0/2, \sigma^2_n\nu_n/2\rp),&
\theta^{\lp S \rp} \sim \normal\lp\mu_n, \sigma^{2\lp S\rp} / \kappa_n\rp, \\
\end{align}

A sequence of pairs $\lb\lp\\sigma^{2\lp 1\rp}, \theta^{\lp 1\rp}\rp, \cdots, \lp\\sigma^{2\lp S\rp}, \theta^{\lp S\rp}\rp\rb$ simulated in this are way are independent samples from the joint posterior distribution
of $p\lp \theta, \sigma^2 \given y_1,\cdots,y_n\rp$. Additionally, the sequence $\lb \theta^{\lp 1\rp}, \cdots, \theta^{\lp S\rp}\rp$ can be seen as independent samples from the marginal posterior distribution of
$p\lp \theta \given y_1,\cdots,y_n\rp$ (having marginalized $\sigma^2$ out I suppose).

It turns out that the marginal posterior distribution of
$$t\lp\theta\rp = \frac{\lp\theta - \mu_n\rp}{\sigma_n / \sqrt{\kappa_n}}$$
is $t$-distributed with $\nu_0 + n$
degrees of freedom. If $\kappa_0$ and $\nu_0$ are small, the posterior distribution
of $t\lp\theta\rp$ will be very close to the $t_{n-1}$ distribution.

### Improper priors

You may be tempted to use Bayesian approaches but try not to use prior information in order
to not appear biased. We can let $\kappa_0$ and $\nu_0$ go to zero to understand what would
happen with no prior information. As $\kappa_0, \nu_0 \rightarrow 0$,
\begin{align}
\mu_n &\rightarrow \bar{y} \\
\sigma^2_n &\rightarrow \frac{1}{n}\sum \lp y_i-\bar{y}\rp^2
\end{align}
This leads to the following posterior distributions:
\begin{align}
\lb 1/\sigma^2 \given y_1, \cdots, y_n \rb &\sim \gamma\lp \frac{n}{2}, \frac{n}{2}\frac{1}{n}\sum \lp y_i-\bar{y}\rp^2\rp \\
\lb \theta \given y_1, \cdots, y_n \rb &\sim \normal\lp\bar{y},\frac{\sigma^2}{n}\rp
\end{align}
You can show that
$$\frac{\theta - \bar{y}}{s/\sqrt{y}}\given y_1, \cdots, y_n \sim t_{n-1}$$

This can be compared to the sampling distribution of the $t$ statistic, conditional on $\theta$
but not on the data
$$\frac{\bar{Y} - \theta}{s/\sqrt{n}}\given\theta\sim t_{n-1}$$

Ths second statement says that the deviation of the estimate $\bar{Y}$ from the true population
mean $\theta$ (scaled by the denominator) is represented by a $t_{n-1}$ distribution. The first
statement says that after you sample your data, your uncertainty is still represented by $t_{n-1}$
distribution.

Since there are no prior probabilities that will lead to the $t_{n-1}$ posterior for $\theta$,
inference based on this posterior is not formally Bayesian. Somtimes taking limits like this
can lead to reasonable answers, however.

# 5.4 Bias, variance, and mean squared error

A **point estimator** of an unknown parameter $\theta$ is a function that converts data into a single
element of the parameter space $\Theta$. In the case of a normal sampling model and conjugate prior
distribution of the last section, the posterior mean estimator of $\theta$ is
$$\hat{\theta}_b \lp y_1,\cdots,y_n\rp = \ev\l[\theta\given y_1,\cdots,y_n\r] =
\frac{n}{\kappa_0 + n}\bar{y} + \frac{\kappa_0}{\kappa_0 + n}\mu_0 = w\bar{y} + \lp 1-w\rp\mu_0$$

The sampling propertires of an estimator such as $\hat{\theta}_b$ refer to its behavior under hypothetically
repeatable surveys or experiments. Let's compare the sampling properties of $\hat{\theta}_b$ to
$\hat{\theta}_e\lp y_1,\cdots,y_n\rp = \bar{y}$, the sample mean, when the true value of the population
mean is $\theta_0$:
\begin{align}
\ev\l[\theta_e\given\theta=\theta_0\r] &= \theta_0, \\
\ev\l[\theta_b\given\theta=\theta_0\r] &= w\theta_0 + \lp 1-w\rp\mu_0
\end{align}
We say that $\hat{\theta}_e$ is unbiased because its expected value equals the true population mean.
We say that $\hat{\theta}_b$ is biased since $\mu_0 \not= \theta_0$.

Bias refers to how close the center of mass of the sampling distribution is to the true value. Bias doesn't
tell us how far away an estimate from the sampling distribution might be from the true value, however. We
can look at the mean squared error (MSE) to evaluate how close an estimator $\hat{\theta}$ is likely to be
to the true value $\theta$. Letting $m=\ev\l[\hat{\theta}\given\theta_0\r]$, the MSE is
$$\MSE\l[\hat{\theta}\given\theta_0\r] = \var\l[\hat{\theta}\given\theta_0\r] +
\textrm{Bias}\l[\hat{\theta}\given\theta_0\r]$$
This means that before the data are gathered, the expected distance from the estimator to the true value
depends on how close $\theta_0$ is to the center of the distribution of $\hat{\theta}$ (bias) and how
spread out the distribution is (the variance). While the bias $\hat{\theta}_e$ is zero, it turns out
that $\var\l[\hat{\theta}_b\r] < \var\l[\hat{\theta}_e\r]$ and that the MSE of $\hat{\theta}_b$
is less than the MSE of $\hat{\theta}_e$ if
\begin{align}
\lp \mu_0 - \theta_0\rp^2 &< \frac{\sigma^2}{n}\frac{1+w}{1-w} \\
&=\sigma^2\lp\frac{1}{n}+\frac{2}{\kappa_0}\rp
\end{align}
If you know even a little bit about the population you are about to sample from, you should be able to find
values of $\theta_0$ and $\kappa_0$ such that this inequality holds. In this case, you can construct
a Bayesian estimator that will have lower average squared distance to the truth than does the sample mean.
For example, if you are pretty sure that your best prior guess $\mu_0$ is within two standard deviations
of the true population mean, then if you pick $\kappa_0 = 1$ you can be pretty sure the Bayesian
estimator has a lower MSE.

# 5.5 Prior specification based on expectations

A $p$ dimensional exponential family model is a model whose densities can be written as
$p\lp y\given\phi\rp = h\lp y\rp c\lp\phi\rp\exp\lb\phi^T \textbf{t}\lp y\rp\rb$, where
$\phi$ is the parameter to be estimated and $\textbf{t}\lp y\rp = \lb t_1\lp y\rp, \cdots, t_p\lp y\rp\rb$
is the sufficient statistic. The normal model is a two-dimensional exponential family
model with
* $\textbf{t}\lp y\rp = \lp y,y^2\rp$
* $\phi = \lp \theta/\sigma^2, -\lp 2\sigma^2\rp^{-1}\rp$
* $c\lp\phi\rp = \left|\phi_2\right|^{1/2}\exp\lb\phi_1^2/\lp 2\phi_2\rp\rb$

A conjugate prior can be written in terms of $\phi$, giving
$p\lp\phi\given n_0, \textbf{t}_0\rp \propto c\lp\phi\rp^{n_0}\exp\lp n_0, \textbf{t}^T_0\phi\rp$, where
$\textbf{t}_0 = \lp t_{01},t_{02}\rp = \lp \ev\l[Y\r], \ev\l[Y^2\r]\rp$, the prior expectations
of $Y$ and $Y^2$. We can reparameterize in terms of $\theta, \sigma^2$ and obtain an expression for
$p\lp\theta,\sigma^2\given n_0, t_0\rp$ that is proportional to a $\normal\lp t_{01}, \sigma^2/n_0\rp$
density times an $\textrm{inverse-gamma}\lp\lp n_0 + 3\rp/2, n_0\lp t_2 - t_1^2 / 2\rp\rp$ density.

How do we interpret the prior parameters $t_{01}$ and $t_{02}$? Consider the case where we have a prior
expectation $\mu_0$ for the population mean and a prior expectation $\sigma^2$ for the population variance.
Our joint distribution for $\lp\theta,\sigma^2\rp$ is then
\begin{align}
\theta \given \sigma^2 &\sim \normal\lp\mu_0, \sigma^2 / n_0\rp \\
\sigma^2 &\sim \textrm{inverse-gamma}\lp\lp n_0 + 3\rp/2, \lp n_0+ 1\rp\sigma^2_0/2\rp
\end{align}

If our prior information is weak, we might set $n_0=1$.

# 5.6 The normal model for non-normal data

People use the normal model even for non-normal data because the sampling distribution
of the sample mean is generally close to normal. In general, using the normal model
for non-normal data is reasonable if we are only interested in obtaining a posterior
distribution for the population mean. For other population quantities the normal model
can provide misleading results.

# 5.7 Discussion and further references

A characterizing feature of the normal distribution is that the mean and variance
are independent. From a subjective probability standpoint, this suggests if your
beliefs about the sample mean are independent from those about the sample variance,
then a normal model is appropriate.

Among all distribution with a given mean $\theta$ and variance $\sigma^2$, the normal
distribution is the most diffuse in terms of entropy.

```python

```

readings/A First Course in Bayesian Statistical Methods/5. The normal model.ipynb


# ML 101

## 3. SVD

```python
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import scipy.stats

import sklearn.preprocessing
```

We start off with the [Iris flower dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set). The data is multivariate, with 150 measurements of 4 features (length and width cm of both sepal and petal) on 3 distinct Iris species. Of the 150 measurements, there are 50 measurements each for _Iris setosa_, _Iris versicolor_, and _Iris virginica_.

```python
data = pd.read_csv('https://media.githubusercontent.com/media/mariolpantunes/ml101/main/datasets/iris.csv')
variety={'Setosa':0, 'Versicolor':1, 'Virginica':2}
data.variety = [variety[item] for item in data.variety]
df_iris = data[['sepal.length','sepal.width','petal.length','petal.width']]

iris = data[['variety']]
iris = iris.to_numpy()[:,0]

print('Iris dataset has {} rows and {} columns\n'.format(*df_iris.shape))

print('Here are the first 5 rows of the data:\n\n{}\n'.format(df_iris.head(5)))

print('Some simple statistics on the Iris dataset:\n\n{}\n'.format(df_iris.describe()))
```

As we are exploring the dataset, it would be nice to view the data in order to get an idea of how the 3 species might be distributed with respect to one another in terms of their features. Perhaps we are interested in finding clusters, or maybe we would like to find a way to make class predictions?

However, since the data has 4 dimensions, we would be hard-pressed to come up with a good way to graph the data in 4D that we could easily understand.

_But what if we could reduce or compress the data so that we could work in 3 dimensions or less?_

[Singular value decomposition](http://mathworld.wolfram.com/SingularValueDecomposition.html) lets us do just that.

## Singular value decomposition

Singular value decomposition factorizes an $\mathbb{R}^{m \times n}$ matrix $X$ into

* matrix $U \in \mathbb{R}^{m \times m}$ are the left-singular vectors of $X$, where columns are the set of orthonormal eigenvectors of $X \, X^{\intercal}$
* diagonal matrix $\Sigma$ with entries $\sigma \in \mathbb{R}$ that are the non-negative singular values of $X$
* matrix $V \in \mathbb{R}^{n \times n}$ are the right-singular vectors $X$, where the columns are the set of orthonormal eigenvectors of $X^{\intercal} \, X$

such that

\begin{align}
  X &= U \, \Sigma \, V^{\intercal}
\end{align}

We can use [`numpy.linalg.svd`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html) to factorize the Iris data matrix into three components $U$, $\Sigma$, and $V^{\intercal}$.

```python
U_iris, S_iris, Vt_iris = np.linalg.svd(df_iris)
```

#### $U$: left-singular vectors of $X$

The rows of the $U$ correspond to the rows of original data matrix $X$, while the columns are the set of ordered, orthornormal eigenvectors of $X \, X^{\intercal}$.

```python
print('matrix U has {} rows, {} columns\n'.format(*U_iris.shape))

print('{}'.format(pd.DataFrame(U_iris).head(5)))
```

#### $V$: right-singular vectors of $X$

[`numpy.linalg.svd`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html) actually returns $V^{\intercal}$ instead of $V$, so it is the _columns_ of $V^{\intercal}$ that correspond to the columns of original data matrix $X$. Hence, the _rows_ are the set of ordered, orthornormal eigenvectors of $X^{\intercal} \, X$.

```python
print('matrix Vt has {} rows, {} columns\n'.format(*Vt_iris.shape))

print('{}'.format(pd.DataFrame(Vt_iris).head()))
```

#### $\Sigma$: singular values of $X$

The elements $\sigma_{i}$ of diagonal matrix $\Sigma$ are the non-zero singular values of matrix $X$, which are really just the square roots of the non-zero eigenvalues of $X^{\intercal} \, X$ (and also for $X \, X^{\intercal}$). These singular values can be used to determine the amount of variance $X^{\prime}$ explains of the original data matrix $X$ when reducing the dimensions to find a lower rank approximation.

\begin{align}
   X^{\prime}_{k} &=  U_{k} \, \Sigma_{k} \, V^{\intercal}_{k} \\
                           &\approx X_{r} & \text{ where } rank(X^{\prime}) = k \lt rank(X) = r
\end{align}

The amount of variance that the reduced rank approximation $X^{\prime}_{k}$ explains of $X_{r}$ is

\begin{align}
  \text{cum. variance explained} &= \frac{\sum_{j=1}^{k} \sigma_{j}^{2}}{\sum_{i=1}^{r} \sigma_{i}^{2}}
\end{align}

**NOTE**: [`numpy.linalg.svd`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html) actually returns a $\Sigma$ that is not a diagonal _matrix_, but a _list_ of the entries on the diagonal.

```python
num_sv_iris = np.arange(1, S_iris.size+1)

cum_var_explained_iris = [np.sum(np.square(S_iris[0:n])) / np.sum(np.square(S_iris)) for n in num_sv_iris]
```

Let's have a look at the cumulative variance explained visually as a function of the number of singular values used when reducing rank to find a lower-ranked matrix $X^{\prime}$ to approximate $X$. This will inform us as to how many dimensions we should use.

```python
fig = plt.figure(figsize=(7.0,5.5))
ax = fig.add_subplot(111)

plt.plot(num_sv_iris,
         cum_var_explained_iris,
         color='#2171b5',
         label='variance explained',
         alpha=0.65,
         zorder=1000)

plt.scatter(num_sv_iris,
            sklearn.preprocessing.normalize(S_iris.reshape((1,-1))),
            color='#fc4e2a',
            label='singular values (normalized)',
            alpha=0.65,
            zorder=1000)

plt.legend(loc='center right', scatterpoints=1, fontsize=8)

ax.set_xticks(num_sv_iris)
ax.set_xlim(0.8, 4.1)
ax.set_ylim(0.0, 1.1)
ax.set_xlabel(r'Number of singular values used')
ax.set_ylabel('Variance in data explained')
ax.set_title('Iris dataset: cumulative variance explained & singular values',
             fontsize=14,
             y=1.03)

ax.set_facecolor('0.98')

plt.grid(alpha=0.8, zorder=1)
plt.tight_layout()
plt.show()
```

#### Dimension reduction

Judging from the curve representing cumulative variance explained in the figure above, we can see that

* with 1 singular value, about 96.5% of the variance of $X$ can be explained
* with 2 singular values, that number goes up to approximately 99.8%

Since graphing the Iris dataset in 1D wouldn't be all that interesting (just dots on a line segment), let's try using the first 2 singular values to represent the data on the $x$- and $y$-axes, respectively.

```python

idx_setosa = np.where(iris==0)[0]
idx_versicolor = np.where(iris==1)[0]
idx_virginica = np.where(iris==2)[0]

setosa_x = U_iris[idx_setosa, 0]
setosa_y = U_iris[idx_setosa, 1]

versicolor_x = U_iris[idx_versicolor, 0]
versicolor_y = U_iris[idx_versicolor, 1]

virginica_x = U_iris[idx_virginica, 0]
virginica_y = U_iris[idx_virginica, 1]
```

We will use different marker shapes and colors to represent the three Iris species on our 2D graph.

```python
fig = plt.figure(figsize=(7.0,5.5))
ax = fig.add_subplot(111)

plt.scatter(setosa_x,
            setosa_y,
            marker='o',
            color='#66c2a5',
            label='Iris-setosa',
            zorder=1000)

plt.scatter(versicolor_x,
            versicolor_y,
            marker='D',
            color='#fc8d62',
            label='Iris-versicolor',
            zorder=1000)

plt.scatter(virginica_x,
            virginica_y,
            marker='^',
            color='#8da0cb',
            label='Iris-virginica',
            zorder=1000)

plt.legend(loc='upper left', scatterpoints=1, fontsize=8)

ax.set_xlabel(r'singular value $\sigma_{1}$')
ax.set_ylabel(r'singular value $\sigma_{2}$')
ax.set_title('2D plot of Iris dataset',
             fontsize=14,
             y=1.03)
ax.set_facecolor('0.98')

plt.grid(alpha=0.6, zorder=1)
plt.tight_layout()
```

There!

Now that we are viewing the originally 4D data with 2 dimensions using the first 2 singular value columns of the $U$ left singular vectors matrix, we can see that there should be a very clear separation for the _Iris setosa_ class and the others. On the other hand, the demarcation between _Iris versicolor_ and _Iris virginica_ might not be as clear cut.

Nevertheless, since this 2D reduced-rank matrix representation $X^{\prime}$ explains nearly 99.8% of the variance in the original dataset, we can be pretty certain that clustering and classification should be possible.

# Helpful Resources

* [Making sense of principal component analysis, eigenvectors & eigenvalues](https://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues/140579#140579)
* [Correspondence analysis is a useful tool to uncover the relationships among categorical variables](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3718710/)

----

# Appendix A

## PCA and SVD

Principal components analysis (PCA) and singular value decomposition are closely related, and you may often hear both these terms used in the same breath.

Here is a quick mathematical treatment explaining how PCA and SVD are related.

Consider data matrix $X \in \mathbb{R}^{m \times n}$ where $m > n$, and all $x_{ij}$ are centered about the column means. With principal components analysis, we have

\begin{align}
  \text{covariance matrix } C &= \frac{1}{m} \, X^{\intercal} \, X & \text{from PCA} \\
  &= \frac{1}{m} \, V \, \Gamma \, V^{\intercal} & \text{by eigendecomposition of } X^{\intercal} \, X \\
  \\
  \text{ but } X &= U \, \Sigma V^{\intercal} & \text{from SVD} \\
  \\
  \Rightarrow C &= \frac{1}{m} \, V \, \Sigma \, U^{\intercal} \, U \, \Sigma V^{\intercal} \\
  &= \frac{1}{m} \, V \, \Sigma^{2} \, V^{\intercal} \\
\end{align}

So we see that:

1. the singular values in $\Sigma$ obtained via SVD are really just the square roots of the eigenvalues in $\Gamma$ of the covariance matrix in PCA.
1. if you mean-center your raw data matrix $X$ and then calculate SVD, you are doing the same thing as PCA.
1. the above example shows covariance of $X$ with respect to its columns ($X^{\intercal} \, X$); it also applies for covariance of $X$ with respect to rows ($X \, X^{\intercal}$).

#### Iris dataset: PCA & SVD

```python
from sklearn.decomposition import PCA

pca = PCA()
pca.fit(df_iris)

# don't forget to mean-center the data before SVD
X = df_iris - np.mean(df_iris, axis=0)
U, S, Vt = np.linalg.svd(X)
```

#### Compare the eigenvalues of $\Gamma$ derived from PCA with the singular values from $\Sigma$ derived with SVD: $\Gamma = \Sigma^{2}$?

```python
Cov_pca = pca.get_covariance()

print('eigenvalues from PCA:\n{}\n'.format(np.linalg.eigvals(Cov_pca * X.shape[0])))

print('squared singular values from SVD:\n{}'.format(np.square(S)))
```

#### Can we obtain the covariance matrix $C$ derived with PCA, but using  $\frac{1}{m} \, V \, \Sigma^{2} \, V^{\intercal}$ from SVD?

```python
print('covariance matrix C derived from PCA:\n{}\n'.format(Cov_pca))

Cov_svd = (1. / X.shape[0]) * Vt.T.dot(np.diag(np.square(S))).dot(Vt)
print('covariance matrix using S and Vt from SVD:\n{}\n'.format(Cov_svd))

allclose = np.allclose(Cov_pca, Cov_svd, atol=1e-1)
print('Are these matrices equivalent (element-wise closeness comparison)?\n{}'.format(allclose))
```


## 1. 선형 연립방정식을 활용한 예측모델

선형 연립방정식의 문제를 해결하는 것은 선형 예측모델의 가중치 벡터를 구하는 것과 같습니다.

$$
\begin{align}
\begin{matrix}
x_{11} w_1 & + \;& x_{12} w_2   &\; + \cdots + \;& x_{1N} w_N &\; = \;& y_1 \\
x_{21} w_1 & + \;& x_{22} w_2   &\; + \cdots + \;& x_{2N} w_N &\; = \;& y_2 \\
\vdots\;\;\; &   & \vdots\;\;\; &                & \vdots\;\;\; &     & \;\vdots \\
x_{N1} w_1 & + \;& x_{N2} w_2   &\; + \cdots + \;& x_{NN} w_N &\; = \;& y_N \\
\end{matrix}
\end{align}
$$

위의 식은 선형 연립방정식입니다. 이는 N개의 목표값(y)을 출력하는 함수의 집합으로도 볼 수 있습니다.

여기서 x는 목표값을 예측하는 데 활용될 데이터를 나타냅니다. 여기서 필요한 것은, x라는 데이터(input)을 받으면 합리적인 목표값(y)를 출력해내는 함수를 만들어내는 것입니다.

그리고 결국, 이 함수는 아래와 같은 행렬과 벡터의 연산으로 나타낼 수 있습니다.

$$
\begin{align}
Xw = y
\end{align}
$$

input(x, 데이터)과 output(y, 예측값)을 이어주는 관계를 나타내는 모델을 찾아야 합니다. 이를 위해 우리가 찾아야 하는 것은 최적의 $w$ 입니다.

$w$는 특징행렬(X)의 역행렬을 통해 다음과 같이 구할 수 있습니다.

$$
\begin{align}
w = X^{-1} y
\end{align}
$$

## 2. 선형 연립방정식을 활용한 예측모델 : 예제 (보스턴 집값 예측모델 생성)

Python의 패키지 중 하나인 scikit-learn 패키지는 머신러닝을 공부하다보면 적어도 한 번쯤은 들어보셨을만큼, 유명하고, 또 그 만큼 유용한 패키지입니다. scikit-learn은 다양한 데이터셋을 제공하기도 하는 데, 그 중 보스턴 집값과 다양한 feature를 묶어놓은 데이터셋을 제공합니다.

앞에서 간단히 살펴본 선형 연립방정식을 활용해, 보스턴 집값을 예측하는 모델을 만들어보려 합니다.

보스턴 집값 문제를 선형 예측모델 $Ax=\hat{b}$ 로 놓고, 가중치 벡터 $x$ 를 구해, 예측모델을 완성해보겠습니다.

문제를 간단히 하기 위해, 입력 데이터를 범죄율(CRIM), 공기 오염도(NOX), 방의 개수(RM), 오래된 정도(AGE)의 4종류로 제한해서 진행해보겠습니다.

```python
from sklearn.datasets import load_boston
boston = load_boston()
X = boston.data
y = boston.target
print(boston.DESCR)
```

    .. _boston_dataset:

    Boston house prices dataset
    ---------------------------

    **Data Set Characteristics:**

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    https://archive.ics.uci.edu/ml/machine-learning-databases/housing/

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    .. topic:: References

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.

보스턴 집값 데이터의 간략한 소개입니다. 카네기 멜론 대학교로부터 데이터가 수집되었다고 하네요.

데이터는 실수형 / 범주형(categorical) 데이터가 섞여있습니다. 데이터의 index는 506개라고 하네요. Missing value는 없다고 합니다. :)

그럼 데이터를 실제로 출력해서 살펴보겠습니다.

```python
X.shape
```

    (506, 13)

역시, 데이터는 13개의 feature를 대상으로 506개의 index를 갖고 있습니다. 저희는 이 중 4개의 feature만을 대상으로 간략히 조사해보려 하기에, 이를 따로 추출해내야 합니다.

"범죄율(CRIM) : 0, 공기 오염도(NOX) : 4, 방의 개수(RM) : 5, 오래된 정도(AGE) : 6" 이 4가지를 추출해서 특징행렬 $A$ 를 구성하겠습니다.

```python
A = X[:,[0,4,5,6]]
```

```python
# 특징행렬 A를 시각적으로 표현하기 위해 A_라는 별도의 객체를 만들어 데이터프레임으로 나타낸 것입니다.

A_ = pd.DataFrame(A,columns = ['crim','nox','rm','age'])
A_.describe()
```

      crim
      nox
      rm
      age

      count
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.613524
      0.554695
      6.284634
      68.574901

      std
      8.601545
      0.115878
      0.702617
      28.148861

      min
      0.006320
      0.385000
      3.561000
      2.900000

      25%
      0.082045
      0.449000
      5.885500
      45.025000

      50%
      0.256510
      0.538000
      6.208500
      77.500000

      75%
      3.677083
      0.624000
      6.623500
      94.075000

      max
      88.976200
      0.871000
      8.780000
      100.000000

다음은 저희의 목표값인 집값 데이터를 벡터 b로 구성하겠습니다.

```python
b = boston.target
b.shape
```

    (506,)

그럼 이제 $Ax=\hat{b}$ 의 x(가중치 벡터)를 구하기 위해, 선형 연립방정식을 풀어보겠습니다. 저희는 $A$ 와 $b$를 구했으니까요!

다만,,, 저희는 지금 sqaure matrix가 아닌 A를 보고 있습니다. 따라서, **A의 역행렬을 구할 수 없습니다.**
역행렬은 기본적으로 square matrix인 경우에만 정확히 구할 수 있기 때문이죠! (물론, approximate approach로 psuedo inverse 가 있기는 하지만, 이후에 다시 살펴보겠습니다.)

따라서, 아직 살펴보진 않았지만, 최소자승문제를 활용한 풀이를 먼저 진행하고 다시 역행렬을 활용해 가중치 벡터($x$)를 찾는 연습을 다시 해보겠습니다.

아직 살펴보진 않았지만, 최소자승문제(Least square problem)의 해결은 지금과 같이 정확한 solution(가중치 벡터 $x$)을 찾을 수 없을 때 활용하는 가장 대표적인 방법입니다. 가장 $x$와 **가까운** solution set을 찾아보자는 접근입니다. 보다 자세히는 추후에 다시 살펴보기로 하겠습니다! : )

### 2_1. Least square problem

Least square problem 해결은 Numpy에서 제공하는 메서드를 활용하겠습니다.

`lstsq()` 메서드는 Least square problem 해결을 위한 코드입니다.

```python
x, resid, rank, s = np.linalg.lstsq(A, b)
```

자 그럼, 우리가 찾고자 했던 solution(x, 가중치벡터)가 어떻게 나왔는지 살펴보겠습니다.

```python
for i in range(4):
    print("{}의 가중치 : {}".format(A_.columns[i],x[i]))
```

    crim의 가중치 : -0.1839836056341421
    nox의 가중치 : -19.396230746735444
    rm의 가중치 : 5.673593152217298
    age의 가중치 : -0.022788798368235375

음... 간단히 해석해보자면,

범죄율, 공기오염도, 연식이 높을 수록 집값은 떨어지는 반비례 관계를 갖는 다는 것을 확인할 수 있습니다. 반면, 방의 갯수는 많을 수록 집값이 높다는 것 을 볼 수 있네요!

보통의 상식과 부합하는 결과입니다! :) 아마도 Least square method가 문제를 어느정도 맞게 해결해낸 것 같습니다!

### 2_2. Solve the systems of linear equations using inverse matrix

이번에는 역행렬을 통해 solution(x, 가중치벡터)를 구해보겠습니다!
저희가 정의했던 문제는 바로 $Ax=\hat{b}$ 였었죠.

그런데, 문제가 있습니다. 역행렬은 square matrix에서만 존재합니다. 따라서, 저희는 A를 바꿔줘야 합니다. 바로, 정방행렬(square matrix)로 말이죠!

앞에서 최소자승법으로 풀때의 행렬 A는 row가 column에 비해 매우 긴 skinny matrix였죠!

```python
A.shape
```

    (506, 4)

이제 저희는 A를 다시 square matrix로 변환해주겠습니다 !

```python
A = A[:4]
```

```python
A.shape
```

    (4, 4)

그리고 b도 마찬가지로 4개의 component로 변환하겠습니다. 그래야 $(4x4) @ (4x1) = (4x1)$ 이 될테니까요!

```python
b = b[:4]
b.shape
```

    (4,)

그리고 이제 $x = A^{-1}b$ 를 풀기 위해, A의 역행렬을 구하고 x를 구해보겠습니다.

```python
A_inv = np.linalg.inv(A)
```

```python
x = A_inv@b
for i in range(4):
    print("{}의 가중치 : {}".format(A_.columns[i],x[i]))
```

    crim의 가중치 : -312.710043270391
    nox의 가중치 : -115.19394234554954
    rm의 가중치 : 14.4996465318047
    age의 가중치 : -0.1132593173503273

음.. 앞에서 506개의 데이터를 대상으로 찾아낸 가중치와 조금 다릅니다.
물론 비례/반비례 관계는 같지만... 가중치의 값이 매우 큽니다. 아무래도 506개의 데이터를 대상으로 찾은 모델보다는 다소 부정확하고, 매우 경직된(?)것이라 생각해볼 수 있을 것 같습니다!

저희는 Least square method, inverse matrix를 통해 가중치벡터를 찾아보았습니다. 머신러닝과 딥러닝은 이러한 연립방정식의 가중치를 찾는 다양한 방법들의 집합입니다. 앞으로 기회가 된다면, 이러한 다양한 방법들을 함께 본다면 좋겠네요! :)

1.Study/2. with computer/1.Math_code/5. Mathematics for Machine Learning/2. Linear Algebra_3_2. Solving Systems of Linear Equations.ipynb