```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.patches import Ellipse
from ipywidgets import interact, FloatSlider
from IPython.display import display
```

# Gaussian Mixture Models

References for today's lecture:

1. Bishop, "Pattern recognition and machine learning", **Chapter 9**
1. Murphy, "Machine Learning: A Probabilistic Perspective", *MIT Press*, 2012, **Chapter 11**

## Introduction

**Unsupervised methods:**

- Contrary to supervised (regression, classification) in this scenario there are no labels/targets
- The objective of unsupervised methods is **to find structure in the data**
- We do this to better understand the data: find hidden causes, find shared features

### Clustering

Group the data into clusters (sets) such that
- "Similar" data samples go to the same cluster
- Elements of one cluster are "different" from the elements of another cluster

Clustering requires a notion of similarity

### Latent variable models (LVM)

- Model the data using a **cause and effect** structure
    - Observed variables (effect): Data
    - Latent variables (cause): Hidden variables that explains the observed variables
- The objective is to infer the latent given the observed
- We assume properties for the latent variable, for example
    - Dimension of the latent is smaller than the observed → Dimensionality reduction
    - Latent variable is continuous and Gaussian distributed → **PCA**
    - Latent variables are not correlated (observed data might be)

## Mixture model

In this class we focus on LVM for clustering

> We assume that there are $K$ clusters (sets)

We represent each data point

$$
x_i \in \mathbb{R}^D \text{ with } i=1,\ldots, N
$$

as having a **categorical latent variable**

$$
z_i \in {1, 2, \ldots, K}
$$

This is the same as saying that $z_i$ has a categorical prior

$$
p(z_i) = \text{Cat}(\vec \pi),
$$

where $\pi_k \in [0, 1]$ are the mixing coefficients and $\sum_{k=1}^K \pi_k=1$

> **Summary:** the latent variable associates each data point with one of the $K$ clusters

Another use for mixture models: **Density estimation**

### Likelihood of the mixture model

We can write the likelihood as

$$
\begin{align}
p(x_i | \theta) &= \sum_{k=1}^K p(x_i|z_i = k, \theta) p(z_i=k| \theta) \nonumber \\
&= \sum_{k=1}^K p_k(x_i| \theta_k) \pi_k ,  \nonumber
\end{align}
$$

To continue we need to specify the family $p_k(x_i|\theta_k)$

## Gaussian mixture model (GMM)

In the GMM we further assume that each component of the mixture is a Gaussian/Normal distribution.

The we can write the likelihood as

$$
p(x_i | \theta) = \sum_{k=1}^K \pi_k \mathcal{N}(x_i|\mu_k, \Sigma_k),
$$

where each cluster has three parameters
- The **centroid/prototype** of the cluster is the mean of the clusters: $\mu_k$
- The size and orientation of the cluster, given by the covariance: $\Sigma_k$
- The contribution of the cluster to the mixture (importance): $\pi_k$

> Fitting the GMM: **Finding the best value of these parameters** for all clusters given the data

**Generative model:** Given a value for the parameters we can sample from the GMM

Let's explore the influence of these parameters on the "shape" of the mixture

```python
def create_mixture(pis, mus, Sigmas, N=5000, rseed=None):
    """
    pis (list of floats): mixing coefficients : cluster importance
    mus (list of arrays): mean: cluster centers
    Sigmas (list of arrays): covariance: cluster shape
    N (int): number of points to create
    """
    if rseed is not None:
        np.random.seed(rseed)
    points = list()
    labels = list()
    for k in range(len(pis)):
        Nc = int(pis[k]*N)
        if len(mus[k]) > 1:
            x = np.random.multivariate_normal(mus[k], Sigmas[k], Nc)
        else:
            x = mus[k] + np.sqrt(Sigmas[k])*np.random.randn(Nc)
        points.append(x)
        labels.append([k]*Nc)
    return np.concatenate(points), np.concatenate(labels)
```

```python
data, labels = create_mixture(pis=[0.4, 0.5, 0.1],
                              mus=[[-1], [2], [2.5]],
                              Sigmas=[0.1, 0.5, 0.1])

fig, ax = plt.subplots(1, figsize=(6, 3), tight_layout=True)
ax.hist(data, bins=50);
```

```python
data, labels = create_mixture(pis=[0.9, 0.1],
                              mus=[[-1, -1], [1, 1]],
                              Sigmas=[0.5*np.eye(2), [[1, -0.9],[-0.9, 1]]])

fig, ax = plt.subplots(1, 2, figsize=(6, 3), tight_layout=True)

ax[0].scatter(data[:, 0], data[:, 1], c=labels, s=10, cmap=plt.cm.coolwarm,
              linewidths=0.1, edgecolors='k', vmin=0, vmax=1);
ax[1].hist2d(x=data[:, 0], y=data[:, 1], bins=30, cmap=plt.cm.Spectral_r);
```

## Posterior probability a.k.a. picking a cluster

Let's assume that we know the parameters of the mixture

Typically we want to infer the most probable cluster for a given data point

> This is given by the posterior probability of $z$ given $x$

Using the Bayes rule and our previous assumptions (likelihood) we have

$$
\begin{align}
r_{ik} = p(z_i = k|x_i, \theta) &= \frac{ p(x_i|z_i = k, \theta) p(z_i=k| \theta)}{p(x_i)} \nonumber \\
&= \frac{ p(x_i|z_i = k, \theta) p(z_i=k| \theta)}{ \sum_{k=1}^K p(x_i|z_i = k, \theta) p(z_i=k| \theta) } \nonumber \\
&= \frac{ \mathcal{N}(x_i|\mu_k, \Sigma_k)  \pi_k}{ \sum_{k=1}^K \mathcal{N}(x_i|\mu_k, \Sigma_k) \pi_k }, \nonumber\end{align}
$$

this posterior is also called the **responsibility** $r_{ik}$ of cluster $k$ for point $i$

Note that it is a **soft-cluster assignment**

In some cases we only need a **hard-cluster assignment**, *i.e.* the single most probable cluster, which can be obtained as

$$
c_{i} = \text{arg}\max_k r_{ik} = \text{arg}\max_k \log \mathcal{N}(x_i|\mu_k, \Sigma_k) + \log \pi_k,
$$

because of the maximum operator we can omit the evidence and use the log likelihood

```python
def jointGMM(data, pi, mu, Sigma):
    # Assume 2 dimensional data
    data_centered = data - mu
    norm = pi/(2*np.pi*np.sqrt(np.linalg.det(Sigma)))
    return norm*np.exp(-0.5*np.sum(np.multiply(data_centered, np.linalg.solve(Sigma, data_centered.T).T), axis=1))

fig, ax = plt.subplots(1, 2, figsize=(6, 3), tight_layout=True)

pis = [0.6, 0.4]; mus=[[-1, -1], [1, 1]];
Sigmas=[0.5*np.eye(2), [[1, -0.3],[-0.3, 1]]]
data, labels = create_mixture(pis, mus, Sigmas, rseed=0)

posterior = np.zeros(shape=(len(data), len(pis)))
for k in range(len(pis)):
    posterior[:, k] = jointGMM(data, pis[k], mus[k], Sigmas[k])
posterior = posterior/np.sum(posterior, axis=1)[:, np.newaxis]

ax[0].scatter(data[:, 0], data[:, 1], c=posterior[:, 0], s=10,
              cmap=plt.cm.coolwarm,
              linewidths=0.1, edgecolors='k', vmin=0, vmax=1); ax[0].set_title("Soft label")
ax[1].scatter(data[:, 0], data[:, 1], c=np.argmax(posterior, axis=1), s=10,
              cmap=plt.cm.coolwarm,
              linewidths=0.1, edgecolors='k', vmin=0, vmax=1); ax[1].set_title("Hard label");
```

## How do we fit the GMM?

We want to find the optimal point estimates of the parameters of a statistical model

Can use **MLE**? The log likelihood in this case is

$$
\log L(\pi,\mu,\Sigma) = \sum_{i=1}^N \log \sum_{k=1}^K \pi_k \mathcal{N}(x_i|\mu_k, \Sigma_k)
$$

**Problem** With the exception of $K=1$, we won't be able to find analytic expressions for the parameters

> In general when we don't have complete data, i.e. there are missing or hidden variables, the posterior does not factorize

In this case we have the latent/hidden variables $z_i$, we need to marginalize it (sum over k) to compute the likelihood

Other problems of the MLE/MAP

- Non-convex, it has local optima (Murphy 11.3.2)
- Very hard to compute (NP-complete hard!)
- **Unidentifiability:** There exist $K!$ configurations that yield the exact same solution (label switch)
- **Singularities:** A component may collapse to a single point ($\Sigma \to 0$ and $\log L \to \infty$)

What can we do?

- Gradient descent with constraints
- Approximate MLE estimation: **Expectation Maximization (EM)**

## Expectation Maximization (EM)

The MLE solution can be computed if we **"fully-observe"** the data

> We know $z_i$ for each $x_i$, *i.e.* we know which cluster owns $x_i$

The **fully-observed** log likelihood (FOLL) is

$$
\log L_{\text{FO}}(\theta) = \sum_{i=1}^N \log p(z_i) p(x_i|z_i, \theta),
$$

but in practice we do not observe $z$ so we marginalize it by computing the **expected value** of the FOLL given the posterior

$$
\begin{align}
Q(\theta, \theta^{\text{old}}) &= \mathbb{E}_{p(z|x, \theta^{\text{old}})} \left[\log L_{\text{FO}}(\theta) \right]   \nonumber \\
&= \sum_{k=1}^K \sum_{i=1}^N p(k|x_i, \theta^{\text{old}})  \log p(k) p(x_i|\theta_k) \nonumber
\end{align}
$$

which is called the **auxiliary function**

After this we update our parameters by **maximizing**

$$
\theta^{\text{new}} = \text{arg}\max_\theta Q(\theta, \theta^{\text{old}}),
$$

and finally we do $\theta^{\text{old}} \leftarrow \theta^{\text{new}}$ and repeat the procedure until convergence

This iterative procedure of

1. **E-step:** Estimating the expected value of the likelihood given our current parameters
1. **M-step:** Maximizing the estimator and updating our parameters

is called **Expectation Maximization** (EM)

EM is a general algorithm:

- it can be used for other problems with non-analytical solution/non-totally observed data
- it has many variants: Incremental, Variational, Monte-Carlo (Murphy 11.4.8 and 11.4.9)

### EM updates for the GMM

Let's start by defining the auxiliary function for the GMM

$$
\begin{align}
Q(\theta, \theta^{\text{old}})
&= \sum_{k=1}^K \sum_{i=1}^N p(k|x_i, \theta^{\text{old}})  \log p(k) p(x_i|\theta_k)  \nonumber \\
&= \sum_{k=1}^K \sum_{i=1}^N r_{ik}^{\text{old}}  \left(\log \pi_k + \log \mathcal{N}(x_i|\mu_k, \Sigma_k) \right),\nonumber
\end{align}
$$

where $r_{ik}^{\text{old}}$ is the responsibility given the parameters of the previous iteration

Now we maximize $Q$ to get an estimator of the parameters

For the means

$$
\begin{align}
\frac{d Q}{d\mu_{k'}} &= \frac{d}{d\mu_{k'}} \sum_{k=1}^K \sum_{i=1}^N r_{ik}^{\text{old}}  \left(\log \pi_k + \log \mathcal{N}(x_i|\mu_k, \Sigma_k) \right) \nonumber \\
&= \sum_{i=1}^N r_{ik'}^{\text{old}}  \frac{d}{d\mu_{k'}}  \log \mathcal{N}(x_i|\mu_{k'}, \Sigma_{k'})\nonumber \\
&= \sum_{i=1}^N r_{ik'}^{\text{old}}  \frac{d}{d\mu_{k'}} \left( -\frac{1}{2} (x_i - \mu_{k'})^T \Sigma_{k'}^{-1} (x_i - \mu_{k'}) -\frac{D}{2} \log(2\pi) -\frac{1}{2} \log(|\Sigma_{k'}|) \right)\nonumber \\
&= -\frac{1}{2} \sum_{i=1}^N r_{ik'}^{\text{old}}  \frac{d}{d\mu_{k'}} (x_i - \mu_{k'})^T \Sigma_{k'}^{-1} (x_i - \mu_{k'})  \nonumber \\
&= \sum_{i=1}^N r_{ik'}^{\text{old}} \Sigma_{k'}^{-1} (x_i - \mu_{k'})  = 0 \nonumber
\end{align}
$$

and from here we can show that the new value of the cluster mean is given by

$$
\mu_{k}^{\text{new}}  = \frac{1}{\sum_{i=1}^N r_{ik}^{\text{old}}} \sum_{i=1}^N r_{ik}^{\text{old}} x_i
$$

Following the same procedure for the covariance we arrive to

$$
\begin{align}
\Sigma_k^{\text{new}} &= \frac{1}{\sum_{i=1}^N r_{ik}^{\text{old}}} \sum_{i=1}^N r_{ik}^{\text{old}} (x_i - \mu_k) (x_i - \mu_k)^T \nonumber \\
&= -\mu_k \mu_k^T  + \frac{1}{\sum_{i=1}^N r_{ik}^{\text{old}}} \sum_{i=1}^N r_{ik}^{\text{old}} x_i x_i^T   \nonumber
\end{align}
$$

Finally, we incorporate the constraint that $\sum_k \pi_k = 1$ to arrive to

$$
\pi_k^{\text{new}} = \frac{\sum_{i=1}^N r_{ik}^{\text{old}}}{N}
$$

[← Go back to the index](#index)

## Fitting a GMM using EM

The following algorithms summarizes the application of EM to the GMM problem

- **(1)** Set initial conditions
- **(2)** Compute cluster responsibilities for each data point using

$$
r_{ik}^{\text{old}} = \frac{ \mathcal{N}(x_i|\mu_k^{\text{old}}, \Sigma_k^{\text{old}})  \pi_k^{\text{old}}}{ \sum_{k=1}^K \mathcal{N}(x_i|\mu_k^{\text{old}}, \Sigma_k^{\text{old}}) \pi_k^{\text{old}} }
$$

- **(3)** Compute the new values for the parameters using

$$
\mu_{k}^{\text{new}}  = \frac{1}{\sum_{i=1}^N r_{ik}^{\text{old}}} \sum_{i=1}^N r_{ik}^{\text{old}} x_i,
$$

$$
\Sigma_k^{\text{new}} = -\mu_k \mu_k^T  + \frac{1}{\sum_{i=1}^N r_{ik}^{\text{old}}} \sum_{i=1}^N r_{ik}^{\text{old}} x_i x_i^T ,
$$

and

$$
\pi_k^{\text{new}} = \frac{\sum_{i=1}^N r_{ik}^{\text{old}}}{N}
$$

- **(4)** Update the parameters $\theta^{\text{old}} \leftarrow \theta^{\text{new}}$

- **(5)** If convergence criterion is met stop otherwise go back to **2**

### Example using "Old faithful" data

Iteration by iteration  (Bishop Fig. 9.8)

Animation (wikipedia)

## Practical details of EM

**Initialization and restarts**

The EM procedure is guaranteed to converge to a *stationary point* (local minimum/maximum or saddle point)

- One can obtain a better solution using the following heuristic:
    1. Set a **number of retries**
    1. Run GMM with a random initialization
    1. If the likelihood of this solution is larger than the previous replace best
    1. Repeat until **number of retries** is achieved
- Another option is to initialize with the solution from a simpler algorithms (*e.g.* k-means)
    - This can avoid catastrophic results
    - But it can also limit exploration

**Number of clusters**

The number of clusters $K$ is a parameter set *a priori* by the user

> How do we know what value of $K$ is good?

Option 1: The Elbow method,
- Plot log likelihood (in the validation set) as a function of $K$
- log likelihood may keep decreasing with $K$ (overfit)
- but at some point the decrease is negligible: find the **elbow** of the curve

Option 2: Use Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC), or other criterion for model selection that penalizes complexity

Option 3: [The Silhouette score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html)

**Convergence**

EM is an iterative procedure

> How many times do we repeat it?

- Check log likelihood vs iteration
- For example: stop if log likelihood increment is less than 0.1\% wrt to previous

## Special cases of GMM: The shape of the covariance

The covariance $\Sigma \in \mathbb{R}^{D\times D}$, where $D$ is the dimension of the data, defines the **shape** and **size** of the clusters

Instead of a full covariance (maximum flexibility) you could impose constraints such as
1. Spherical covariance: $\Sigma = \sigma^2 \text{I}$, $\sigma \in \mathbb{R}_{+}$
1. Diagonal covariance: $\Sigma = \vec \sigma^2 \text{I}$, $\vec \sigma \in \mathbb{R}_{+}^D$

```python
fig, ax = plt.subplots(1, 2, figsize=(6, 3), sharex=True, sharey=True, tight_layout=True)

eps = np.random.randn(2, 5000)

def update(s1, s2, rho):
    Sigma = [[s1**2, rho*s1*s2], [rho*s1*s2, s2**2]]
    data = np.dot(np.linalg.cholesky(Sigma), eps).T
    for ax_ in ax:
        ax_.cla();
    ax[0].scatter(data[:, 0], data[:, 1], s=10,
                  linewidths=0.1, edgecolors='k', vmin=0, vmax=1);
    ax[1].hist2d(x=data[:, 0], y=data[:, 1], range=[[-6, 6], [-6, 6]],
                 bins=50, cmap=plt.cm.Blues);

interact(update,
         s1=FloatSlider(min=0.5, max=2.),
         s2=FloatSlider(min=0.5, max=2.),
         rho=FloatSlider(min=-0.9, max=0.9));
```

## Gaussian mixture with [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html)

From the `sklearn.mixture` scikit learn module

```python
GaussianMixture(n_components=1, # Number of clusters
                covariance_type='full', # Shape of the covariance
                tol=0.001, # termination tolerance of EM
                reg_covar=1e-06, # small number added to the diagonal of the covariance
                max_iter=100, # Max number of EM iterations
                n_init=1, # Number of random initializations
                init_params='kmeans', # Initialization for the responsabilities (kmeans or random)
                ...
               )
```

Create data to test the model

```python
data, labels = create_mixture(pis=[0.33, 0.33, 0.34],
                              mus=[[-3, 0], [0, 0], [3 ,0]],
                              Sigmas=[[[1, 0.9],[0.9, 1]],
                                      [[1, -0.9],[-0.9, 1]],
                                      [[1, 0.9],[0.9, 1]]])

fig, ax = plt.subplots(figsize=(6, 3), tight_layout=True)
ax.scatter(data[:, 0], data[:, 1], s=10, cmap=plt.cm.coolwarm,
           linewidths=0.1, edgecolors='k');
```

Find best number of clusters using the Bayesian information criterion (BIC). A model with lowest BIC is preferred

$$
BIC = K \log N - 2 \log L
$$

The BIC grows with $k$ the number of parameters and number of samples $n$ and decreases with the maximum value of the log likelihood

Try spherical, diagonal and full covariance

```python
from sklearn.mixture import GaussianMixture

maxK = 20;
cov = 'full'
metrics = np.zeros(shape=(maxK-1, ))
for i, K in enumerate(range(2, maxK+1)):
    gmm = GaussianMixture(n_components=K, covariance_type=cov, n_init=3)
    gmm.fit(data)
    metrics[i] = gmm.bic(data)

fig, ax = plt.subplots(figsize=(6, 3))
ax.plot(range(2, maxK+1), metrics, linestyle='--', marker='o');
ax.set_ylabel('BIC');
```

Inspect results

- Cluster shapes and cluster assignments
- uncertainty in the responsibilities
- Generative results

```python
gmm = GaussianMixture(n_components=10, covariance_type=cov, n_init=3)
gmm.fit(data)
hatr = gmm.predict(data);
hatdata = gmm.sample(n_samples=10000)

fig, ax = plt.subplots(3, figsize=(6, 8), sharex=True, sharey=True, tight_layout=True)

ax[0].scatter(data[:, 0], data[:, 1], c=hatr, s=5, cmap=plt.cm.tab20,
              linewidths=0.1, edgecolors='k');
ax[0].set_title('Arg max of responsibilities')

p = gmm.predict_proba(data)
H = -np.sum(p*np.log(p), axis=1)
ax[1].scatter(data[:, 0], data[:, 1], c=H, s=10, cmap=plt.cm.Reds,
              linewidths=0.1, edgecolors='k');
ax[1].set_title('Entropy of of responsibilities')

ax[2].scatter(hatdata[0][:, 0], hatdata[0][:, 1], s=5, alpha=0.1)
ax[2].set_title('Generated data')

ax[2].scatter(gmm.means_[:, 0], gmm.means_[:, 1], s=50, c='k', marker='d')

for pos, covar, w in zip(gmm.means_, gmm.covariances_, gmm.weights_):
    if cov == 'full':
        U, s, Vt = np.linalg.svd(covar)
        angle = np.degrees(np.arctan2(U[1, 0], U[0, 0]))
        width, height = 2 * np.sqrt(s)
    elif cov == 'diag':
        angle = 0
        width, height = 2 * np.sqrt(covar)
    elif cov == 'spherical':
        angle = 0
        width, height = 2 * np.sqrt(covar), 2 * np.sqrt(covar)

    ax[2].add_patch(Ellipse(pos, 2.5*width, 2.5*height, angle, color='k', alpha=w))

```

## Appendix: Relation to K-means

The K-means algorithm is a GMM with the following constraints

- Spherical clusters: $\Sigma = \sigma^2 \text{I}$, $\sigma \in \mathbb{R}_{+}$
- All clusters have equal size: $\sigma^2 = \sigma_1^2 = \sigma_2^2 = \ldots = \sigma_K^2$
- Uniform prior for the mixture coefficients: $\pi_k = \frac{1}{K}$
- Hard labels are used instead of responsibilities

The multivariate normal for spherical clusters

$$
\mathcal{N}(x_i|\mu_k, \Sigma_k = \sigma^2 I) = \frac{1}{\sqrt{(2\pi)^D} \sigma} \exp \left(-\frac{1}{2\sigma^2} \|x_i - \mu_k\|^2 \right)
$$

The update rule for the responsibility

$$
\begin{align}
\hat r_{ik}^{\text{old}} &= \text{arg} \max_k \frac{ \exp \left(-\frac{1}{2\sigma^2} \|x_i - \mu_k\|^2 \right) }{ \sum_{k=1}^K \exp \left(-\frac{1}{2\sigma^2} \|x_i - \mu_k\|^2 \right)  } \nonumber \\
&=  \text{arg} \max_k  \log \exp \left(-\frac{1}{2\sigma^2} \|x_i - \mu_k\|^2 \right)  \nonumber \\
&=  \text{arg} \min_k  \frac{1}{2\sigma^2}  \|x_i - \mu_k\|^2   \nonumber \\
&=  \text{arg} \min_k  \|x_i - \mu_k\|^2   \nonumber
\end{align}
$$

and the means
$$
\mu_{k}^{\text{new}}  = \frac{1}{\sum_{i=1}^N \hat r_{ik}^{\text{old}}} \sum_{i=1}^N \hat r_{ik}^{\text{old}} x_i
$$

Hence the EM for K-means reduces to

1. Update "hard" responsibilities
1. Update the cluster means

## Variational mixture of Gaussians (Bishop 10.2, Murphy 21)

In EM the parameters are point estimates (similar to MLE and MAP)

But in a fully Bayesian LVM the parameters are distributions that we estimate (posterior) given some assumptions (priors)

$$
p(z, \theta |x) = \frac{p(x|z,\theta)p(z, \theta)}{\int p(x|z,\theta)p(z, \theta) dz d\theta}
$$

but computing this integral is intractable (sum over all combinations of z)

We have two choices: stochastic approximation (MCMC) or deterministic approximation

**Variational Bayes** is an example of the latter

VB approximates intractable posteriors with analytical factorized distributions

- $p(z|x)$ is the intractable posterior
- $q_\nu(z)$ is a family of "easier" factorized posterior
- We look for $\nu$ that maximizes a lower bound on the KL divergence between posteriors
- This topic is fully explored in INFO320!

### Priors for the GMM

In GMM we set priors for the weights $\pi$, means $\mu$ and covariance $\Sigma$
- For $\mu$ we use a normal prior
- For $\Sigma$ we use Wishart (covariance) or a mix between LKJ (correlation) and half normal (variance)
- For the weights a Dirichlet prior is used

**Recap on Dirichlet distribution**
- Two parameters: K (number of categories) and $\alpha_k$ concentrations
- Its support is the simplex: $x_1, x_2, \ldots, x_K$, $x_k \in [0,1]$ and $\sum_k x_k = 1$
- By setting the concentration (hyperparameter) we can choose between have equally important clusters or a few active clusters

[Picture source](http://www.sumsar.net/blog/2015/04/the-non-parametric-bootstrap-as-a-bayesian-model/)

**Example: Variational GMM**

- Using a large number of clusters $K$ but with a small concentration $\alpha_k = 0.01$
- Clusters with negligible concentration are not plotted

### Variational GMM with [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.BayesianGaussianMixture.html)

```python
BayesianGaussianMixture(...,
                        weight_concentration_prior_type='dirichlet_process',
                        weight_concentration_prior=None,
                        mean_precision_prior=None,
                        mean_prior=None,
                        degrees_of_freedom_prior=None,
                        covariance_prior=None,
                        ...
                       )
```

```python
BayesianGaussianMixture?
```

```python
from sklearn.mixture import BayesianGaussianMixture

data, labels = create_mixture(pis=[0.33, 0.33, 0.34],
                              mus=[[-3, 0], [0, 0], [3 ,0]],
                              Sigmas=[[[1, 0.9],[0.9, 1]],
                                      [[1, -0.9],[-0.9, 1]],
                                      [[1, 0.9],[0.9, 1]]])

gmm = BayesianGaussianMixture(n_components=10 ,
                              weight_concentration_prior_type='dirichlet_distribution',
                              weight_concentration_prior=10.,
                              covariance_type="full", max_iter=1000, n_init=5)
gmm.fit(data)
hatr = gmm.predict(data)
print(np.unique(hatr))
hatdata = gmm.sample(n_samples=10000)
```

```python
fig, ax = plt.subplots(3, figsize=(6, 8), sharex=True, sharey=True, tight_layout=True)

ax[0].scatter(data[:, 0], data[:, 1], c=hatr, s=5, cmap=plt.cm.tab20,
              linewidths=0.1, edgecolors='k');
ax[0].set_title('Arg max of responsibilities')

p = gmm.predict_proba(data)
H = -np.sum(p*np.log(p), axis=1)
ax[1].scatter(data[:, 0], data[:, 1], c=H, s=10, cmap=plt.cm.Reds,
              linewidths=0.1, edgecolors='k');
ax[1].set_title('Entropy of of responsibilities')

ax[2].scatter(hatdata[0][:, 0], hatdata[0][:, 1], s=5, alpha=0.1)
ax[2].set_title('Generated data')

ax[2].scatter(gmm.means_[:, 0], gmm.means_[:, 1], s=50, c='k', marker='d')

for pos, covar, w in zip(gmm.means_, gmm.covariances_, gmm.weights_):
    U, s, Vt = np.linalg.svd(covar)
    angle = np.degrees(np.arctan2(U[1, 0], U[0, 0]))
    width, height = 2 * np.sqrt(s)
    ax[2].add_patch(Ellipse(pos, 2.5*width, 2.5*height, angle, color='k', alpha=w))
```

## More material

- EM for mixture of Bernoulli distributions: Bishop 9.3.3
- EM for mixture of student-t distributions: Murphy 11.4.5
- Online EM: Murphy 11.4.8
- GMM with priors (MAP): Murphy 11.4.2.8
- [Scikit-learn article: Setting the concentration prior](https://scikit-learn.org/stable/auto_examples/mixture/plot_concentration_prior.html)
- [Variational GMM in pymc3](https://docs.pymc.io/notebooks/gaussian-mixture-model-advi.html)

```python

```


#Introduction

It would be fantastic if we can precisely predict how stock prices will change in near or far future. We would be rich, but it is almost impossible to create exact predictions.

There are so many factors impacting the movement of stock prices that are hard to find. For example the investor sentiment or confidence can cause the market to go up or down. Investors, for sure, make their decisions based on empirical evidence, stock market indicators, and macro/micro economics

The information or data could be interpreted differently depending on the risk appetite and the mood of the investor at the moment investing. While a piece of breaking news or a simple twitter from US president causes an investor to buy a stock, it causes another one to sell that same stock.

Therefore, predicting precisely the variation of stock prices is very difficult. However, we still have valuable models which can help us to understand the stock price movement up to some point.

This short tutorial will provide an example of simulating the stock prices using one of well-known stochastic processes so-called "**Geometric brownian motion (GBM)**"

# Theory Geometric brownian motion

In order to find the expected stock price of the asset, a **Geometric Brownian Motion** has been used, which expresses the change in stock price using a constant drift $\mu$ and volatility $\sigma$ as a stochastic differential equation (SDE):
\begin{equation}
	\left\lbrace
  \begin{aligned}
      \frac{dX_{t}}{X_t} &= \mu dt + \sigma dW_t\\
      X_0   &= x
  \end{aligned}
 	\right.
\end{equation}

Where:

* $\mu$ is a diagonal matrix of expected index returns.

* $D$ is a diagonal matrix with $X_t$ along the diagonal.

* $\sigma$ is a diagonal matrix of standard deviations of index returns.

By using the substitution $Z_t = log(X_t)$, and since the corresponding deterministic linear equation is an exponential function of time **Itô's formula yields**, so we have the following equation for $dZ_t$:

\begin{equation}
	\left\lbrace
  \begin{aligned}
      dZ_{t} &= (\mu - \frac{1}{2} \sigma^2) dt + \sigma dW_t\\
      Z_0   &= log(x)
  \end{aligned}
 	\right.
\end{equation}

The solution of the equation is defined as follow:

$X_t = x e^{(\mu - \frac{1}{2} \sigma^2)t + \sigma W_t} \tag{1}$

As illustrated below, the stock price follows a series of steps, where each step is a drift plus or minus a random shock (itself a function of the stock's standard deviation):

#Overiew
In this tutorial a Geometric Brownian Motion (GBM) has been used to predict the closing prices of the Tesla stock price.
Using the full historical closing prices, the predicted prices have also been compared to observed stock prices, in order to evaluate the validity of the prediction models.

#Tesla stock price - Data analysis

In order to assess the validity of the prediction models, historical closing prices of the Tesla stock has been compared to simulated prices by using basic statistical tests. A time series of the closing prices of the Tesla stock, as well as the log returns of the series can be seen in *Figure 1* and *figure 2*.

We are using Yahoo stock market data open-API to extract the closing prices.
This API offer a real-time or historical data on financial assets that are currently being traded in the markets, it usually offer prices of public stocks, ETFs, ETNs.
```
# To install yfinance before you use it.
> pip install yfinance
```

```
import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import pandas_datareader as web

#load stock prices using yahoo API
tesla = web.get_data_yahoo("TSLA")

closePrice = tesla['Adj Close']

closePrice.plot()
plt.xlabel("Date")
plt.ylabel("Adj close price")
plt.title("Figure 1 - Tesla stock price")
plt.show()

```

As can be seen by the time series of returns, the data shows signs of volatility clustering, with large volatility around the time of COVID-19 crisis.

```
import math
import numpy as np

# Compute the logarithmic returns using the adjusted closing price
logReturn = np.log(closePrice/closePrice.shift(1))

n  = np.size(logReturn)
mu   = np.mean(logReturn);
sigma       = np.std(logReturn);

logReturn.plot()
plt.xlabel("Date")
plt.ylabel("Log return")
plt.title("Figure 2 - Tesla log return")

```

**Historical volatility of Risk-Adjusted Return for Tesla**

The rolling function uses a window of **20 trading days**. Each of the days in the selected lookback period is assigned an equal weight.

As can be seen in the graph below, the volitilty is very high during the period of lockdown caused by COVID-19.

```
sigmaR = logReturn.rolling(window=20).std() * np.sqrt(252)
print(sigmaR.values)
plt.plot(sigmaR)
plt.title('Volatility as measured by standard deviation')
plt.show()
```


# The Tractable Buffer Stock Model

```python
# This cell has just a bit of initial setup. You can click the arrow to expand it.
%matplotlib inline
import matplotlib.pyplot as plt

import sys
import os
sys.path.insert(0, os.path.abspath('../lib'))

import numpy as np
import HARK
from time import clock
from copy import deepcopy
mystr = lambda number : "{:.3f}".format(number)

from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

from HARK.utilities import plotFuncs
from HARK.ConsumptionSaving.TractableBufferStockModel import TractableConsumerType
```

The [TractableBufferStock](http://www.econ2.jhu.edu/people/ccarroll/public/LectureNotes/Consumption/TractableBufferStock/) model is a (relatively) simple framework that captures all of the qualitative, and many of the quantitative features of optimal consumption in the presence of labor income uncertainty.

The key assumption behind the model's tractability is that there is only a single, stark form of uncertainty.  So long as an employed consumer remains employed, his labor income will rise at a constant rate.  But between any period and the next there is constant hazard $p$ of transitioning to the "unemployed" state. Unemployment is irreversible, like retirement or disability.  When unemployed, the consumer receives a fixed amount of income (for simplicity, zero).

\begin{eqnarray*}
\mathbb{E}_{t}[V_{t+1}^{\bullet}(M_{t+1})] & = & (1-p)V_{t+1}^{e}(M_{t+1})+p V_{t+1}^{u}(M_{t+1})
\end{eqnarray*}

A consumer with CRRA utility $U(C) = \frac{C^{1-\rho}}{1-\rho}$ solves an optimization problem that looks standard (where $P$ is Permanent income and $\Gamma$ is the income growth factor):

\begin{eqnarray*}
V_t(M_t) &=& \max_{C_t} ~ U(C_t) + \beta \mathbb{E}[V_{t+1}^{\bullet}], \\
M_{t+1} &=& R A_t + \mathbb{1}(P_{t+1}), \\
P_{t+1} &=& \Gamma_{t+1} P_t,
\end{eqnarray*}

where $\mathbb{1}$ is an indicator of whether the consumer is employed in the next period.

Under plausible parameter values the model has a target level of $m = M/P$ (market resources to permanent income) with an analytical solution that exhibits plausible relationships among all of the parameters.  (See the linked handout for details).

```python
# Define a parameter dictionary and representation of the agents for the tractable buffer stock model
TBS_dictionary =  {'UnempPrb' : .00625,    # Probability of becoming unemployed
                   'DiscFac' : 0.975,      # Intertemporal discount factor
                   'Rfree' : 1.01,         # Risk-free interest factor on assets
                   'PermGroFac' : 1.0025,  # Permanent income growth factor (uncompensated)
                   'CRRA' : 2.5}           # Coefficient of relative risk aversion
MyTBStype = TractableConsumerType(**TBS_dictionary)
```

```python
# Define a function that plots the employed consumption function and sustainable consumption function
# for given parameter values

def makeTBSplot(DiscFac,CRRA,Rfree,PermGroFac,UnempPrb,mMax,mMin,cMin,cMax,plot_emp,plot_ret,plot_mSS,show_targ):
    MyTBStype.DiscFac = DiscFac
    MyTBStype.CRRA = CRRA
    MyTBStype.PermGroFac = PermGroFac
    MyTBStype.UnempPrb = UnempPrb

    try:
        MyTBStype.solve()
    except:
        print('Those parameter values violate a condition required for solution!')

    plt.xlabel('Market resources $M_t$')
    plt.ylabel('Consumption $C_t$')
    plt.ylim([cMin,cMax])
    plt.xlim([mMin,mMax])

    m = np.linspace(mMin,mMax,num=100,endpoint=True)
    if plot_emp:
        c = MyTBStype.solution[0].cFunc(m)
        c[m==0.] = 0.
        plt.plot(m,c,'-b')

    if plot_mSS:
        plt.plot([mMin,mMax],[(MyTBStype.PermGroFacCmp/MyTBStype.Rfree + mMin*(1.0-MyTBStype.PermGroFacCmp/MyTBStype.Rfree)),(MyTBStype.PermGroFacCmp/MyTBStype.Rfree + mMax*(1.0-MyTBStype.PermGroFacCmp/MyTBStype.Rfree))],'--k')

    if plot_ret:
        c = MyTBStype.solution[0].cFunc_U(m)
        plt.plot(m,c,'-g')

    if show_targ:
        mTarg = MyTBStype.mTarg
        cTarg = MyTBStype.cTarg
        targ_label = '$m^* =$' + mystr(mTarg) + '\n$c^* =$' + mystr(cTarg)
        plt.annotate(targ_label,xy=(0.0,0.0),xytext=(0.8,0.05),textcoords='axes fraction')

    plt.show()
    return None

# Define widgets to control various aspects of the plot

# Define a slider for the discount factor
DiscFac_widget = widgets.FloatSlider(
    min=0.9,
    max=0.99,
    step=0.0002,
    value=0.95,
    continuous_update=False,
    readout_format='.4f',
    description='$\\beta$')

# Define a slider for relative risk aversion
CRRA_widget = widgets.FloatSlider(
    min=0.1,
    max=8.0,
    step=0.01,
    value=2.5,
    continuous_update=False,
    readout_format='.2f',
    description='$\\rho$')

# Define a slider for permanent income growth
PermGroFac_widget = widgets.FloatSlider(
    min=0.9,
    max=1.1,
    step=0.0002,
    value=1.0025,
    continuous_update=False,
    readout_format='.4f',
    description='$\\Gamma$')

# Define a slider for unemployment (or retirement) probability
UnempPrb_widget = widgets.FloatSlider(
    min=0.00001,
    max=0.10,
    step=0.00001,
    value=0.00625,
    continuous_update=False,
    readout_format='.5f',
    description='$\\mho$')

# Define a slider for unemployment (or retirement) probability
Rfree_widget = widgets.FloatSlider(
    min=1.0,
    max=1.1,
    step=0.0001,
    value=1.01,
    continuous_update=False,
    readout_format='.4f',
    description='$R$')

# Define a text box for the lower bound of M_t
mMin_widget = widgets.FloatText(
    value=0.0,
    step=0.1,
    description='$M$ min',
    disabled=False)

# Define a text box for the upper bound of M_t
mMax_widget = widgets.FloatText(
    value=50.0,
    step=0.1,
    description='$M$ max',
    disabled=False)

# Define a text box for the lower bound of C_t
cMin_widget = widgets.FloatText(
    value=0.0,
    step=0.1,
    description='$C$ min',
    disabled=False)

# Define a text box for the upper bound of C_t
cMax_widget = widgets.FloatText(
    value=1.5,
    step=0.1,
    description='$C$ max',
    disabled=False)

# Define a check box for whether to plot the employed consumption function
plot_emp_widget = widgets.Checkbox(
    value=True,
    description='Plot employed $C$ function',
    disabled=False)

# Define a check box for whether to plot the retired consumption function
plot_ret_widget = widgets.Checkbox(
    value=True,
    description='Plot retired $C$ function',
    disabled=False)

# Define a check box for whether to plot the sustainable consumption line
plot_mSS_widget = widgets.Checkbox(
    value=True,
    description='Plot sustainable $C$ line',
    disabled=False)

# Define a check box for whether to show the target annotation
show_targ_widget = widgets.Checkbox(
    value=True,
    description = 'Show target $(M,C)$',
    disabled = False)

```

## Target Wealth

Whether the model exhibits a "target" or "stable" level of wealth for employed consumers depends on whether the 'Growth Impatience Condition' (the GIC) holds:

\begin{equation}\label{eq:GIC}
 \left(\frac{(R \beta (1-\mho))^{1/\rho}}{\Gamma}\right)  <  1
\end{equation}

```python
# Make an interactive plot of the tractable buffer stock solution

# To make some of the widgets not appear, replace X_widget with fixed(desired_fixed_value) in the arguments below.
interact(makeTBSplot,
         DiscFac = DiscFac_widget,
#         CRRA = CRRA_widget,
         CRRA = fixed(2.5),
         Rfree = Rfree_widget,
         PermGroFac = PermGroFac_widget,
         UnempPrb = UnempPrb_widget,
         mMin = mMin_widget,
         mMax = mMax_widget,
         cMin = cMin_widget,
         cMax = cMax_widget,
         show_targ = show_targ_widget,
         plot_emp = plot_emp_widget,
         plot_ret = plot_ret_widget,
         plot_mSS = plot_mSS_widget,
        );

```

    interactive(children=(FloatSlider(value=0.95, continuous_update=False, description='$\\beta$', max=0.99, min=0…


---
# Section 3.2: Orthogonal Matrices
---

## Inner-product notation

We will use the following notation for the **inner-product** between vectors $x, y \in \mathbb{R}^n$:

$$
\langle x, y \rangle = \sum_{i=1}^n x_i y_i = x^T y = \|x\|_2 \|y\|_2 \cos\theta,
$$

where $0 \leq \theta \leq \pi$ is the **angle** between $x$ and $y$.

**Note:** $\|x\|_2 = \sqrt{\langle x, x \rangle}$.

---

## Orthogonal matrix definition

$Q \in \mathbb{R}^{n \times n}$ is **orthogonal** if the columns of $Q$ are:

1. **unit-length**:
$$
\|q_i\|_2 = 1, \qquad \forall i,
$$
2. **mutually orthogonal**:
$$
\langle q_i, q_j \rangle = 0, \qquad i \neq j
$$

This is equivalent to saying that

$$Q^T Q = I$$

which is equivalent to

$$Q^{-1} = Q^T.$$

The rows of $Q$ are also unit-length and mutually orthogonal since $QQ^T = I$.

---

## Exercise

1. Prove that the product of orthogonal matrices is orthogonal.

2. Prove that the transpose of an othogonal matrix is orthogonal.

### Part 1

Suppose that $Q_i \in \mathbb{R}^{n \times n}$ is orthogonal for $i = 1,\ldots,k$. Let

$$
Q = Q_1 Q_2 \cdots Q_k.
$$

Now we want to show that $Q$ is orthogonal. To show this, we compute

\begin{align}
Q^T Q
& = (Q_1 Q_2 \cdots Q_k)^T (Q_1 Q_2 \cdots Q_k) \\
& = (Q_k^T \cdots Q_2^T Q_1^T) (Q_1 Q_2 \cdots Q_k) \\
& = Q_k^T \cdots Q_2^T (Q_1^T Q_1) Q_2 \cdots Q_k \\
& = Q_k^T \cdots Q_2^T I Q_2 \cdots Q_k \\
& = Q_k^T \cdots Q_3^T (Q_2^T Q_2) Q_3 \cdots Q_k \\
& = Q_k^T \cdots Q_3^T Q_3 \cdots Q_k \\
& \quad \vdots \\
& = Q_k^T Q_k \\
& = I.
\end{align}

Therefore, $Q^T Q = I$, so $Q$ is orthogonal.

### Part 2

Suppose that $Q \in \mathbb{R}^{n \times n}$ is orthogonal. To show that $Q^T$ is orthogonal, we need to show that $(Q^T)^T (Q^T) = I$. So, we compute

$$
(Q^T)^T (Q^T) = Q Q^T = Q Q^{-1} = I.
$$

Therefore, $Q^T$ is also an orthogonal matrix.

---

> ## Theorem:
>
> If $Q \in \mathbb{R}^{n \times n}$ is orthogonal, then:
>
> 1. $\langle Qx, Qy \rangle = \langle x, y \rangle$
>
> 2. $\|Qx\|_2 = \|x\|_2$
>
> This theorem states that any **orthogonal transformation**, $x \mapsto Qx$, preserve angles and lengths.

---

## Exercise:

Prove the theorem.

### Part 1

Since $Q$ is an orthogonal matrix, we have

\begin{align}
\langle Qx, Qy \rangle
&= (Qx)^T (Qy) \\
&= x^T Q^T Q y \\
&= x^T I y \\
&= x^T y \\
&= \langle x, y \rangle.
\end{align}

### Part 2

Using part 1, we have

\begin{align}
\| Q x \|_2
&= \sqrt{ \langle Q x, Q x \rangle } \\
&= \sqrt{ \langle x, x \rangle } \\
&= \| x \|_2. \\
\end{align}

---

## Rotation matrices

A $2 \times 2$ rotation matrix has the form

$$
Q = \begin{bmatrix}
\cos\theta & -\sin\theta \\
\sin\theta & \cos\theta
\end{bmatrix}.
$$

We can use rotation matrices to introduce zeros into vectors.

---

## Exercise

1. Prove that $2 \times 2$ rotation matrices are orthogonal.

2. Find a rotation matrix $Q$ such that
$$
Q^T \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} y_1 \\ 0 \end{bmatrix}
$$
where $y_1 \geq 0$.

### Part 1

Let $Q$ be a $2 \times 2$ rotation matrix, as above. Then

$$
\begin{align}
Q^T Q
&= \begin{bmatrix}
\cos\theta & -\sin\theta \\
\sin\theta & \cos\theta
\end{bmatrix}^T
\begin{bmatrix}
\cos\theta & -\sin\theta \\
\sin\theta & \cos\theta
\end{bmatrix} \\
&= \begin{bmatrix}
\cos\theta & \sin\theta \\
-\sin\theta & \cos\theta
\end{bmatrix}
\begin{bmatrix}
\cos\theta & -\sin\theta \\
\sin\theta & \cos\theta
\end{bmatrix} \\
&= \begin{bmatrix}
\cos^2\theta + \sin^2\theta & -\cos\theta\sin\theta + \sin\theta\cos\theta \\
-\sin\theta\cos\theta + \cos\theta\sin\theta & \sin^2\theta + \cos^2\theta
\end{bmatrix} \\
&= \begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix} \\
&= I.
\end{align}
$$

Therefore, $Q$ is an orthogonal matrix.

### Part 2

Let

$$
Q =
\begin{bmatrix}
c & -s \\ s & c
\end{bmatrix},
$$

where $c^2 + s^2 = 1$. Then

$$
Q^T \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} y_1 \\ 0 \end{bmatrix}
$$

implies that

$$
\begin{bmatrix}
c & -s \\ s & c
\end{bmatrix}^T
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} y_1 \\ 0 \end{bmatrix}.
$$

Thus,

$$
\begin{bmatrix}
c x_1 + s x_2 \\ -s x_1 + c x_2
\end{bmatrix} = \begin{bmatrix} y_1 \\ 0 \end{bmatrix}.
$$

Also, since $Q^T$ is orthogonal,

$$
\left\| \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \right \|_2 = \left\| \begin{bmatrix} y_1 \\ 0 \end{bmatrix} \right \|_2,
$$

which implies that $y_1 = \sqrt{x_1^2 + x_2^2}$ since $y_1 \ge 0$.

If $x = 0$, then $y_1 = \sqrt{x_1^2 + x_2^2} = 0$, and we would let $c = 1$ and $s = 0$.

Now, we assume that $x \ne 0$. Thus, $y_1 > 0$.

Multiply the first equation by $s$ and the second equation by $c$. Then,

\begin{align}
c s x_1 + s^2 x_2 &= s y_1 \\
-c s x_1 + c^2 x_2 &= 0. \\
\end{align}

Summing these equations gives us

$$
(s^2 + c^2) x_2 = s y_1.
$$

Since we want $s^2 + c^2 = 1$, we have $x_2 = s y_1$, so

$$
s = \frac{x_2}{y_1}.
$$

In a similar way, we find that

$$
c = \frac{x_1}{y_1}.
$$

---

Another way to find $s$ and $c$ is to rewrite

$$
\begin{bmatrix}
c x_1 + s x_2 \\ -s x_1 + c x_2
\end{bmatrix} = \begin{bmatrix} y_1 \\ 0 \end{bmatrix}
$$

as

$$
\begin{bmatrix}
x_1 & x_2 \\ x_2 & -x_1
\end{bmatrix} \begin{bmatrix} c \\ s \end{bmatrix} = \begin{bmatrix} y_1 \\ 0 \end{bmatrix}.
$$

Then, just solve the above system for $c$ and $s$ by multiplying both sides by the inverse of the coefficient matrix.

Since

$$
\begin{bmatrix}
a & b \\ c & d
\end{bmatrix}^{-1} =
\frac{1}{ad - bc}
\begin{bmatrix}
d & -b \\ -c & a
\end{bmatrix},
$$

we have that

$$
\begin{bmatrix}
x_1 & x_2 \\ x_2 & -x_1
\end{bmatrix}^{-1} =
\frac{1}{-x_1^2 - x_2^2}
\begin{bmatrix}
-x_1 & -x_2 \\ -x_2 & x_1
\end{bmatrix} =
\frac{1}{x_1^2 + x_2^2}
\begin{bmatrix}
x_1 & x_2 \\ x_2 & -x_1
\end{bmatrix}.
$$

Therefore,

$$
\begin{bmatrix}
c \\ s
\end{bmatrix} =
\frac{1}{x_1^2 + x_2^2}
\begin{bmatrix}
x_1 & x_2 \\ x_2 & -x_1
\end{bmatrix}
\begin{bmatrix}
y_1 \\ 0
\end{bmatrix} =
\frac{1}{y_1^2}
\begin{bmatrix}
x_1 y_1 \\ x_2 y_1
\end{bmatrix} =
\begin{bmatrix}
x_1/y_1 \\ x_2/y_1
\end{bmatrix}.
$$

```julia
using LinearAlgebra
```

```julia
x = randn(2)
```

```julia
y1 = norm(x)
```

```julia
c, s = x[1]/y1, x[2]/y1
```

```julia
c, s = x/norm(x)
```

```julia
c^2 + s^2
```

```julia
Q = [c -s; s c]
```

```julia
Q'x
```

---

## $QR$-decomposition of a $2 \times 2$ matrix $A$

Suppose

$$
A = \begin{bmatrix}
a_{11} & a_{12} \\
a_{21} & a_{22}
\end{bmatrix}.
$$

Let $Q$ be the rotation matrix that introduces a zero in the first column of $A$:

$$
Q^T \begin{bmatrix} a_{11} \\ a_{21} \end{bmatrix} = \begin{bmatrix} r_{11} \\ 0 \end{bmatrix}.
$$

Let

$$
\begin{bmatrix} r_{12} \\ r_{22} \end{bmatrix} = Q^T \begin{bmatrix} a_{21} \\ a_{22} \end{bmatrix}.
$$

Let

$$
R = \begin{bmatrix}
r_{11} & r_{12} \\
0 & r_{22}
\end{bmatrix}.
$$

Then $Q^T A = R$, and since $Q$ is orthogonal,

$$
A = QR.
$$

---

## Exercise

Compute the $QR$-decomposition of

$$
A = \begin{bmatrix}
1 & 2 \\
1 & 3
\end{bmatrix}
$$

and check your answer using the `qr` function in Julia.

```julia
A = [1 1; 2 3.0]
```

```julia
function formQ(x)
    c, s = x/norm(x)
    Q = [c -s; s c]
end
```

```julia
Q = formQ(A[:,1])
```

```julia
R = Q'A
```

```julia
A - Q*R
```

```julia
F = qr(A)
```

```julia
A - F.Q*F.R
```

---

## Givens rotations

A **Givens rotation** matrix is

$$
Q =
\begin{bmatrix}
1 \\
&\ddots\\
&&1\\
&&&c&&&&-s\\
&&&&1\\
&&&&&\ddots\\
&&&&&&1\\
&&&s&&&&c\\
&&&&&&&&1\\
&&&&&&&&&\ddots\\
&&&&&&&&&&1\\
\end{bmatrix},
$$

where $c = \cos\theta$ and $s = \sin\theta$. This matrix rotates the $(x_i,x_j)$ plane by an angle of $\theta$.

These matrices can be used to introduce zeros in general $n \times n$ matrices.

---

## Exercise

Use Givens rotations to compute the $QR$-decomposition of

$$
A = \begin{bmatrix}
1 & 2 & 0 \\
0 & 1 & 3 \\
1 & 3 & 0
\end{bmatrix}.
$$

```julia
A = [ 1 2 0; 0 1 3; 1 3 0.0 ]
```

```julia
x = A[[1,3],1]
```

```julia
c, s = x/norm(x)
```

```julia
Q1 = [
    c 0 -s
    0 1 0
    s 0 c
]
```

```julia
A1 = Q1'A
```

```julia
x = A1[[2,3],2]
```

```julia
c, s = x/norm(x)
```

```julia
Q2 = [
    1 0 0
    0 c -s
    0 s c
]
```

```julia
A2 = Q2'A1
```

```julia
R = A2
```

```julia
Q = Q1*Q2
```

```julia
A - Q*R
```

```julia
qr(A)
```

```julia
UpperTriangular(R)
```

---

## Solving $Ax = b$ using $QR$

If $Ax = b$ and $A = QR$, then

$$
Q(Rx) = b.
$$

If we let $c = Rx$, then we have $Qc = b$.

Thus, we have the following algorithm for solving $Ax = b$:

1. Let $c = Q^Tb$.
2. Solve $Rx = c$ using backward substitution.

---

## Exercise

Use the $QR$-decomposition of $A$ to solve $Ax = b$.

$$
A = \begin{bmatrix}
1 & 2 \\
1 & 3
\end{bmatrix},
\qquad
b = \begin{bmatrix} 1 \\ 2 \end{bmatrix}.
$$

```julia
A = [1 2; 1 3.0]
b = [1, 2.0]

Q, R = qr(A)
```

```julia
c = Q'b
```

```julia
x = R\c
```

```julia
A*x - b
```

---

## Reflection matrices

Another way to create zeros in a matrix is by the [Householder reflection transformation](https://en.wikipedia.org/wiki/Householder_transformation):

$$
Q = I - 2uu^T, \qquad \|u\|_2 = 1.
$$

Let $L$ be the set of vectors $v$ that are orthogonal to the unit vector $u$,

$$
L = \left\{ v \in \mathbb{R}^n : u^T v = 0 \right\}.
$$

Then $L$ is a **hyperplane** containing the origin, and $Q$ reflects vectors $x$ across $L$.

---

## Properties of $Q = I - 2uu^T$

1. $Qu = -u$
2. If $v \in L$, then $Qv = v$.
3. $Q = Q^T$
4. $Q^TQ = I$
5. $Q^{-1} = Q$

---

## Exercise

Prove the above properties.

### Part 1

Since $\|u\|_2 = 1$, we have

\begin{align}
Q u
&= (I - 2 u u^T) u \\
&= u - 2 u (u^T u) \\
&= u - 2 u \|u\|_2^2 \\
&= u - 2 u \\
&= -u.
\end{align}

### Part 2

Since $v \in L$, we have that $u^T v = 0$. Thus,

\begin{align}
Q v
&= (I - 2 u u^T) v \\
&= v - 2 u (u^T v) \\
&= v - 2 u (0) \\
&= v.
\end{align}

### Part 3

We have that

\begin{align}
Q^T
&= (I - 2 u u^T)^T \\
&= I^T - 2 (u u^T)^T \\
&= I - 2 (u^T)^T u^T \\
&= I - 2 u u^T \\
&= Q. \\
\end{align}

Therefore, $Q$ is symmetric.

### Part 4

Since $Q^T = Q$, we have

\begin{align}
Q^T Q
&= Q Q \\
&= (I - 2 u u^T)(I - 2 u u^T) \\
&= I - 2 u u^T - 2 u u^T + 4 u (u^T u) u^T \\
&= I - 2 u u^T - 2 u u^T + 4 u u^T \\
&= I. \\
\end{align}

Therefore, $Q$ is orthogonal.

### Part 5

Since $QQ = I$, the matrix $Q$ is its own inverse, so $Q^{-1} = Q$.

---

## Reflecting $x$ to $y$

If $\|u\|_2 \neq 1$, then the **Householder reflector** is

$$
Q = I - \gamma uu^T, \qquad \gamma = \frac{2}{\|u\|_2^2}.
$$

If $x, y \in \mathbb{R}^n$ such that $\|x\|_2 = \|y\|_2$, then the reflector $Q$ using

$$u = x - y$$

satisfies

$$
Qx = y.
$$

---

## Exercise

Test that $Qx = y$ on random vectors $x$ and $y$.

```julia
n = 4
x = randn(n)
y = randn(n)
L = 10*rand()
x *= L/norm(x)
y *= L/norm(y)

norm(x) ≈ norm(y)
```

```julia
u = x - y
γ = 2/dot(u,u)

Q = I - γ*(u*u')
```

```julia
Qmap(v) = v - (γ*dot(u,v))*u
```

```julia
[Q*x y]
```

```julia
norm(Q*x - y)
```

```julia
[Qmap(x) y]
```

```julia
norm(Qmap(x) - y)
```

---

## Creating zeros using reflectors

We want the reflector $Q$ that reflects $x$ to $y$, where

$$
x =
\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix},
\qquad
y =
\begin{bmatrix} -\tau \\ 0 \\ \vdots \\ 0 \end{bmatrix},
\qquad
\tau = \mathrm{sign}(x_1)\|x\|_2.
$$

We define $u$ as

$$
u = \frac{x - y}{\tau + x_1} =
\begin{bmatrix} 1 \\ x_2/(\tau + x_1) \\ \vdots \\ x_n/(\tau + x_1) \end{bmatrix}.
$$

Note that we have divided by $\tau + x_1$ to ensure that $u_1 = 1$.

Since $\tau$ and $x_1$ have the same sign, the calculation $\tau + x_1$ avoids catastrophic cancellation.

Letting

$$
Q = I - \gamma uu^T,
\qquad
\gamma = \frac{2}{\|u\|_2^2},
$$

we have

$$
Qx = \begin{bmatrix} -\tau \\ 0 \\ \vdots \\ 0 \end{bmatrix}.
$$

---

## Exercise

Prove that
$$\gamma = \frac{\tau + x_1}{\tau}.$$

### Proof.

First note that

$$
(\tau + x_1) u =
\begin{bmatrix} \tau + x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}.
$$

Now, taking the norm squared of both sides, we have

$$
\|(\tau + x_1) u \|_2^2 = (\tau + x_1)^2 + x_2^2 + \cdots + x_n^2.
$$

Thus,

$$
|\tau + x_1|^2 \|u\|_2^2 = \tau^2 + 2 \tau x_1 + x_1^2 + \cdots + x_n^2.
$$

Then we have

$$
(\tau + x_1)^2 \|u\|_2^2 = \tau^2 + 2 \tau x_1 + \|x\|_2^2.
$$

Since $\tau = \mathrm{sign}(x_1) \|x\|_2$, we have that $\tau^2 = \|x\|_2^2$. Therefore,

$$
(\tau + x_1)^2 \|u\|_2^2 = \tau^2 + 2 \tau x_1 + \tau^2.
$$

Thus,

$$
(\tau + x_1)^2 \|u\|_2^2 = 2 \tau( \tau + x_1 ).
$$

Since $\tau + x_1 \ne 0$, we can divide both sides by $\tau + x_1$, and we get

$$
(\tau + x_1) \|u\|_2^2 = 2 \tau.
$$

Rearranging, we have that

$$
\frac{\tau + x_1}{\tau} = \frac{2}{\|u\|_2^2}.
$$

Since $\gamma = 2/\|u\|_2^2$, we have that

$$
\frac{\tau + x_1}{\tau} = \gamma.
$$

Q.E.D.

---

## Exercise

Test above method for generating $Q$ on a random vector $x$.

```julia
n = 5
x = randn(5)
```

```julia
τ = sign(x[1])*norm(x)
```

```julia
u = [1; x[2:end]/(τ + x[1])]
```

```julia
γ = (τ + x[1])/τ
```

```julia
2/dot(u,u)
```

```julia
Q = I - (γ*u)*u'
```

```julia
Q*x
```

```julia
Qmap(v) = v - (γ*dot(u,v))*u
```

```julia
Qmap(x)
```

---

## `house`

We can now write a function to compute the $u$ and $\gamma$ of the Householder reflector $Q = I - \gamma uu^T$:

```julia
u, γ, τ = house(x)
```

```julia
function house(x)
    u = copy(x)

    τ = norm(x)
    if τ == 0.0
        γ = 0.0
    else
        if x[1] < 0
            τ = -τ    # τ = sign(x[1])*norm(x)
        end
        γ = τ + x[1]  # γ temporarily stores τ + x[1]
        u[1] = 1.0    # u normalized to u[1] = 1
        u[2:end] /= γ # divide u[2:end] by τ + x[1]
        γ /= τ        # γ = (τ + x[1])/τ
    end

    return u, γ, τ
end
```

```julia
n = 5
x = randn(n)

u, γ, τ = house(x)

Q = I - γ*(u*u')
```

```julia
issymmetric(Q)
```

```julia
Q*Q
```

```julia
[x Q*x]
```

```julia
Qmap(x)
```

---

## `housetimes`

The way we computed $Qx$ in the above numerical example was inefficient. Note that

$$
Qx = \left(I - \gamma uu^T\right)x = x - \left[\gamma \left(u^T x\right)\right] u.
$$

```julia
housetimes(x::Vector, u, γ) = x - (γ*dot(u, x))*u
```

```julia
u, γ, τ = house(x)

housetimes(x, u, γ)
```

---

## Exercise

Count the number of flops:
1. To form $Q$ and compute $Qx$.
2. To compute $x - \left[\gamma \left(u^T x\right)\right] u$.

**Solution:**

1. $3n^2 + 2n$ flops
2. $4n + 1$ flops

### Part 1

Recall that $Q = I - \gamma u u^T$.

1. Computing $y = (-\gamma) u$ requires $n$ multiplications.
2. Computing $Q = y u^T$ requires $n^2$ multiplications.
3. Computing $Q = I + Q$ reqires $n$ additions (along the diagonal).
4. Computing $Qx$ requires $2n^2$ operations.

So, in total we have $3n^2 + 2n$ flops for forming the $n \times n$ matrix $Q$ and computing matrix-vector multiplication $Qx$.

### Part 2

To compute $x - \left[\gamma \left(u^T x\right)\right] u$, we do the following.

1. Computing $\delta = u^T x$ requires $n$ multiplications and $n$ additions.
2. Computing $\mu = \gamma \delta$ requires one multiplication.
3. Computing $y = \mu u$ requires $n$ multiplications.
4. Computing $z = x - y$ requires $n$ subtractions.

So, in total we have $4n + 1$ flops for computing $x - \left[\gamma \left(u^T x\right)\right] u$.

---

In the algorithm for computing the $QR$ decomposition of a matrix $A$, we will need to compute $QB$ where $B$ is a matrix.

$$
QB = B - (\gamma u) \left(u^TB\right)
$$

```julia
housetimes(B::Matrix, u, γ) = B - (γ*u)*(u'*B)
```

```julia
methods(housetimes)
```

```julia
n = 5
B = rand(n,n)
```

```julia
u, γ, τ = house(B[:,1])

housetimes(B, u, γ)
```

---

## Exercise

Use Householder reflectors to numerically compute the $QR$-decomposition of

$$
A = \begin{bmatrix}
1 & 2 & 0 \\
0 & 1 & 3 \\
1 & 3 & 0
\end{bmatrix}.
$$

Check your answer using the `qr` function in Julia.

```julia
A = [1 2 0; 0 1 3; 1 3 0.0]
R = copy(A)
```

```julia
u, γ, τ = house(R[:,1])

R[1,1] = -τ
R[2:3,1] .= 0
R[:,2:3] = housetimes(R[:,2:3], u, γ)
R
```

```julia
u, γ, τ = house(R[2:3,2])

R[2,2] = -τ
R[3,2] = 0
R[2:3,3] = housetimes(R[2:3,3], u, γ)
R
```

```julia
qr(A)
```

---

## The $QR$ Decomposition Algorithm

$$A_0 = A$$

$$
A_1 = Q_1A =
\left[\begin{array}{c|c}
-\tau_1 & a_1^T \\ \hline
0 & \hat{A}_1
\end{array}\right],
\qquad
Q_1 = I_n - \gamma_1 u_1 u_1^T
$$

$$
A_2 = Q_2Q_1A =
\left[\begin{array}{c|c}
-\tau_1 & a_1^T \\ \hline
0 &
\begin{array}{c|c}
-\tau_2 & a_2^T \\ \hline
0 & \hat{A}_2
\end{array}
\end{array}\right],
\qquad
Q_2 =
\left[\begin{array}{c|c}
1 & \\\hline
& I_{n-1} - \gamma_2 u_2 u_2^T
\end{array}\right]
$$

$$
A_3 = Q_3Q_2Q_1A =
\left[\begin{array}{c|c}
-\tau_1 & a_1^T \\ \hline
0 &
\begin{array}{c|c}
-\tau_2 & a_2^T \\ \hline
0 &
\begin{array}{c|c}
-\tau_3 & a_3^T \\ \hline
0 & \hat{A}_3
\end{array}
\end{array}
\end{array}\right],
\qquad
Q_3 =
\left[\begin{array}{c|c}
I_2 & \\\hline
& I_{n-2} - \gamma_3 u_3 u_3^T
\end{array}\right]
$$

$$\vdots$$

$$
A_{n-1} = Q_{n-1} \cdots Q_1A =
\left[\begin{array}{c|c}
-\tau_1 & a_1^T \\ \hline
0 &
\begin{array}{c|c}
-\tau_2 & a_2^T \\ \hline
0 &
\begin{array}{c|c}
-\tau_3 & a_3^T \\ \hline
0 &
\begin{array}{c|c}
\ddots & \ddots \\ \hline
0 &
\begin{array}{c|c}
-\tau_{n-1} & a_{n-1}^T \\ \hline
0& \hat{A}_{n-1}
\end{array}
\end{array}
\end{array}
\end{array}
\end{array}\right] = R
$$

We then let $Q = Q_1Q_2 \cdots Q_{n-1}$ and obtain $A = QR$.

---

## Storing $u_i$'s and $\gamma_i$'s

Each $u_i$ is normalized so that

$$
u_i = \begin{bmatrix} 1\\*\\\vdots\\* \end{bmatrix}.
$$

Thus we do not need to store the first entry since it is always $1$.

The rest of the entries of $u_i$ can be stored where the zeros are created.

To store the $\gamma_i$'s, we create a separate vector

$$
\gamma = \begin{bmatrix} \gamma_1\\\vdots\\\gamma_{n-1} \end{bmatrix}.
$$

---

## `myqr`

```julia
struct myQRfactorization
    V::Matrix{Float64}
    γ::Vector{Float64}
end

function myqr(A::Matrix{Float64})
    m, n = size(A)

    m == n || error("This QR decomposition algorithm requires a square input matrix.")

    V = copy(A)
    γ = zeros(n-1)
    for k = 1:n-1
        u, γ[k], τ = house(V[k:n,k])  # compute the Householder reflector I - γuu'
        V[k,k] = -τ                   # diagonal entries become -τ
        V[k+1:n,k] = u[2:end]         # store u's in the strictly lower-triangular part of V
        V[k:n,k+1:n] -= (γ[k]*u)*(u'*V[k:n,k+1:n]) # housetimes
    end

    myQRfactorization(V, γ)
end
```

```julia
n = 5
A = rand(n, n)

myF = myqr(A)
```

```julia
F = qr(A)
```

```julia
UpperTriangular(myF.V)
```

---

## Flop count of $QR$ Decomposition Algorithm

In each iteration, we need to compute `housetimes`:

1. $\left(I_n - \gamma_1 u_1 u_1^T\right) A_{0}[1:n,\ 2:n]$

2. $\left(I_{n-1} - \gamma_2 u_2 u_2^T\right) A_{1}[2:n,\ 3:n]$

3. $\left(I_{n-2} - \gamma_3 u_3 u_3^T\right) A_{2}[3:n,\ 4:n]$

$\qquad\vdots$

In iteration $k$, we compute

$$
\left(I_{n-k+1} - \gamma_k u_k u_k^T\right) A_{k-1}[k:n,\ k+1:n].
$$

This operation requires approximately $4(n - k + 1)^2$ flops if done efficiently.

We also need to compute $\gamma_k$ and $u_k$ from $A_{k-1}[k:n, k]$, but this is only $O(n)$.

Therefore, the $QR$ Decomposition Algorithm requires

$$
\sum_{k=1}^{n-1} 4(n-k+1)^2 \approx \int_1^{n-1} 4(n-x+1)^2 dx = \frac{4}{3}n^3.
$$

This does not include forming $Q$ though.

---

## Forming $Q$

We can use the $u_i$'s and the $\gamma_i$'s to compute

$$QB = Q_1Q_2\cdots Q_{n-1} B$$

$$Q^T B = Q_{n-1}\cdots Q_2 Q_1 B$$

efficiently without forming $Q$.

We can form $Q$ by computing $QI_n$:

$$
\begin{align}
Q = QI_n &= Q_1 Q_2\cdots Q_{n-1} I_n\\
&=
\left[\begin{array}{c}
I_{n} - \gamma_1 u_1 u_1^T
\end{array}\right]
\left[\begin{array}{c|c}
I_1 & \\\hline
& I_{n-1} - \gamma_2 u_2 u_2^T
\end{array}\right]
\cdots
\left[\begin{array}{c|c}
I_{n-2} & \\\hline
& I_{2} - \gamma_{n-1} u_{n-1} u_{n-1}^T
\end{array}\right] I_n
\end{align}
$$

Done efficiently (from right to left), this calculation requires an additional $\frac43n^3$ flops.

---

## `Qtimes` and `formQ`

```julia
function Qtimes!(F::myQRfactorization, B::Matrix; T=false)
    n = size(F.V, 1)
    cols = T ? (1:n-1) : (n-1:-1:1)
    for k = cols
        γk = F.γ[k]
        uk = [1.0; F.V[k+1:n,k]]
        B[k:n,k:n] -= (γk*uk)*(uk'*B[k:n,k:n])
    end
    B
end
Qtimes(F::myQRfactorization, B::Matrix) = Qtimes!(F, copy(B))

QTtimes!(F::myQRfactorization, B::Matrix) = Qtimes!(F, B, T=true)
QTtimes(F::myQRfactorization, B::Matrix) = QTtimes!(F, copy(B))

formQ(F::myQRfactorization) = Qtimes(F, Matrix{Float64}(I, size(F.V)))
```

```julia
methods(formQ)
```

```julia
n = 5
A = rand(n, n)
F = myqr(A)

Q = formQ(F)
```

```julia
Q'*A
```

```julia
QTtimes(F, A)
```

---

## Flop count summary for matrix factorizations

Let $A \in \mathbb{R}^{n \times n}$.

`chol(A)`: $\frac13n^3$ flops

`lu(A)`:  $\frac23n^3$ flops

`F = qr(A)` does not form $Q$: $\frac43n^3$ flops

`Q = F.Q*Matrix(I,n,n)` forms $Q$: $\frac83n^3$ flops

---

## Flop count to solve $Ax = b$ by $QR$

**Algorithm:**

1. Compute the $QR$ Decomposition of $A$, but do not form $Q$.
2. $c = Q^Tb$
3. Use backward substitution to solve $Rx = c$

The cost of the $QR$ Decomposition is $\frac43n^3$.

The cost of computing $c = Q^Tb$ efficiently (using the $u_i$'s and the $\gamma_i$'s) is about $2n^2$ flops.

Backward substitution is $n^2$ flops.

Therefore, in total we have

$$\frac43n^3 + O(n^2)$$

flops to solve $Ax = b$ by $QR$.

---

## `x = F\b`

```julia
function Qtimes!(F::myQRfactorization, b::Vector; T=false)
    n = size(F.V, 1)
    cols = T ? (1:n-1) : (n-1:-1:1)
    for k = cols
        γk = F.γ[k]
        uk = [1.0; F.V[k+1:n,k]]
        b[k:n] -= (γk*uk)*dot(uk, b[k:n])
    end
    b
end
Qtimes(F::myQRfactorization, b::Vector) = Qtimes!(F, copy(b))

QTtimes!(F::myQRfactorization, b::Vector) = Qtimes!(F, b, T=true)
QTtimes(F::myQRfactorization, b::Vector) = QTtimes!(F, copy(b))
```

```julia
methods(Qtimes)
```

```julia
n = 5
A = rand(n, n)
b = rand(n)

F = myqr(A)

c = QTtimes(F, b)
```

```julia
R = UpperTriangular(F.V)
```

```julia
x = R\c
```

```julia
b - A*x
```

```julia
b - A*(A\b)
```

---

```julia
import Base.\

\(F::myQRfactorization, b::Vector) = UpperTriangular(F.V)\QTtimes(F, b)
```

```julia
methods(\)
```

```julia
x = F\b
```

```julia
b - A*x
```

---

> ## $QR$ Decomposition Theorem
>
> Let $A \in \mathbb{R}^{n \times n}$. Then the following hold.
>
> 1. There exists $Q, R \in \mathbb{R}^{n \times n}$ such that $Q$ is orthogonal, $R$ is upper-triangular, and $A = QR$.
>
> 2. If $A$ is **nonsingular**, then $\exists$ **unique** $Q, R \in \mathbb{R}^{n \times n}$ such that $Q$ is orthogonal, $R$ is upper-triangular with **positive diagonal entries**, and $A = QR$.

### Proof.

1. Hint: Use Householder reflectors and induction on $n$.
2. Hint: Let $D$ be diagonal with $d_{ii} = \mathrm{sign}(r_{ii})$.

### Part 1

If $n = 1$ then $A$ is a $1 \times 1$ matrix. Thus, $A = [a_{11}]$. Let $Q = [1]$ and $R = [a_{11}]$. Then $Q$ is orthogonal, $R$ is upper-triangular, and $A = QR$.

Now suppose that all $k \times k$ matrices have a $QR$ decomposition, for some positive integer $k$.

Let $n = k+1$ and $A \in \mathbb{R}^{n \times n}$. We partition $A$ as

$$
A =
\begin{bmatrix}
a_{11} & b^T \\ c & D
\end{bmatrix},
$$

where $D$ is a $k \times k$ matrix.

Let $Q_1$ be a Householder reflector for the first column of $A$. Then

$$
Q_1 A =
\begin{bmatrix}
\hat{a}_{11} & \hat{b}^T \\ 0 & \hat{D}
\end{bmatrix}.
$$

Note that $\hat{D}$ is a $k \times k$ matrix, so, by our induction hypothesis, $\hat{D}$ has a $QR$ decomposition: $\hat{D} = \hat{Q} \hat{R}$. Thus,

$$
Q_1 A =
\begin{bmatrix}
\hat{a}_{11} & \hat{b}^T \\ 0 & \hat{Q} \hat{R}
\end{bmatrix}.
$$

Let

$$
Q_2 =
\begin{bmatrix}
1 & 0 \\ 0 & \hat{Q}^T
\end{bmatrix},
$$

and note that $Q_2$ is orthogonal. Then

$$
Q_2 Q_1 A =
\begin{bmatrix}
1 & 0 \\ 0 & \hat{Q}^T
\end{bmatrix}
\begin{bmatrix}
\hat{a}_{11} & \hat{b}^T \\ 0 & \hat{Q} \hat{R}
\end{bmatrix} =
\begin{bmatrix}
\hat{a}_{11} & \hat{b}^T \\ 0 & \hat{Q}^T \hat{Q} \hat{R}
\end{bmatrix} =
\begin{bmatrix}
\hat{a}_{11} & \hat{b}^T \\ 0 & \hat{R}
\end{bmatrix}.
$$

Let $R = Q_2 Q_1 A$. Note that $R$ is upper-triangular. Let $Q = Q_1^T Q_2^T$. Note that $Q$ is orthogonal since it is the product of orthogonal matrices. Finally, we have that $A = Q_1^T Q_2^T R = Q R$.

### Part 2

First, we let $A$ be an $n \times n$ nonsingular matrix. Then, by part 1, the matrix $A$ has a $QR$ decomposition:  $A = QR$. Note that the diagonal entries of $R$ are nonzero since, otherwise, $\det(R) = 0$ and that would imply that $\det(A) = \det(Q) \det(R) = 0$, contradicting our assumption that $A$ is nonsingular.

Let $D$ be the diagonal matrix with diagonal entries $d_{ii} = \mathrm{sign}(r_{ii})$. Thus, $D$ has diagonal entries that are $\pm 1$, so $D^2 = I$. That implies that

$$
A = Q R = Q D^2 R = (Q D) (D R).
$$

Let $\hat{Q} = Q D$ and $\hat{R} = D R$. Then $\hat{Q}$ is orthogonal since $Q$ and $D$ are orthogonal. Also, $\hat{R}$ is upper-triangular with positive diagonal entries, and $A = \hat{Q} \hat{R}$.

Now suppose that $A = Q_1 R_1 = Q_2 R_2$, where $R_1$ and $R_2$ have positive diagonal entries. Then,

$$
A^T A = R_1^T Q_1^T Q_1 R_1 = R_1^T R_1
$$

and $A^T A = R_2^T R_2$ are both Cholesky decompositions of the symmetric positive definite matrix $A^T A$. But the Cholesky decomposition is unique, so $R_1 = R_2$. Then,

$$
Q_1 = A R_1^{-1} = A R_2^{-1} = Q_2.
$$

---

## Stability

Multiplication by rotators or reflectors is stable:

$$
\mathrm{fl}(QA) = Q(A + E)
$$

where $\frac{\|E\|_2}{\|A\|_2}$ is tiny.

Also,

\begin{align}
\mathrm{fl}(Q_2 Q_1 A)
&= Q_2( Q_1(A + E_1) + E_2 ) \\
&= Q_2( Q_1(A + E_1) + Q_1 Q_1^T E_2 ) \\
&= Q_2 Q_1( A + E_1 + Q_1^T E_2 ) \\
&= Q_2 Q_1( A + E ) \\
\end{align}

where $E = E_1 + Q_1^T E_2$.

Thus,

$$
\mathrm{fl}(Q_2Q_1A) = Q_2Q_1(A + E),
$$

where $\|E\|_2 = \left\|E_1 + Q_1^T E_2\right\|_2 \leq \|E_1\|_2 + \left\|Q_1^TE_2\right\|_2 = \|E_1\|_2 + \left\|E_2\right\|_2$. Therefore, $\frac{\|E\|_2}{\|A\|_2}$ is tiny.

---


Computers, Waves, Simulations

Finite-Difference Method - First Derivative

#### This exercise covers the following aspects:

* Calculation of numerical first derivative
* Comparison with analytical solution
* Dependence of error on space increment

**Note:** Alternative solution added that looks at the convergence by decreasing the spatial increment dx (May 12, 2020)

```python
# Import Libraries
import numpy as np
from math import *
import matplotlib.pyplot as plt
```

We initialize a space-dependent *sin* function

\begin{equation}
f(x)= \sin (k x)
\end{equation}

where the wavenumber k is

\begin{equation}
k = \dfrac{2 \pi}{\lambda}
\end{equation}

and $\lambda$ is wavelength.

```python
# Initial parameters
xmax = 10.0                     # physical domain (m)
nx = 200                        # number of samples
dx = xmax/(nx-1)                # grid increment dx (m)
x  = np.linspace(0,xmax,nx)     # space coordinates

# Initialization of sin function
l = 40*dx     # wavelength
k = 2*pi/l    # wavenumber
f = np.sin(k*x)
```

```python
# Plot sin function
plt.figure(figsize=(10,6))
plt.plot(x, f)
plt.title('Sin function')
plt.xlabel('x, m')
plt.ylabel('Amplitude')
plt.xlim((0, xmax))
plt.grid()
plt.show()
```

In the cell below we calculate the central finite-difference derivative of f(x) using two points

\begin{equation}
f^{\prime}(x)=\dfrac{f(x+dx)-f(x-dx)}{2dx}
\end{equation}

and  compare with the analytical derivative

\begin{equation}
f^{\prime}(x) = k \cos(k x)
\end{equation}

```python
# First derivative with two points

# Initiation of numerical and analytical derivatives
nder=np.zeros(nx)          # numerical derivative
ader=np.zeros(nx)          # analytical derivative

# Numerical derivative of the given function
for i in range (1, nx-1):
    nder[i]=(f[i+1]-f[i-1])/(2*dx)

# Analytical derivative of the given function
ader= k * np.cos(k*x)
# Exclude boundaries
ader[0]=0.
ader[nx-1]=0.

# Error (rms)
rms = np.sqrt(np.mean((nder-ader)**2))
```

```python
# Plotting
# ----------------------------------------------------------------
plt.figure(figsize=(10,6))
plt.plot (x, nder,label="Numerical Derivative, 2 points", marker='+', color="blue")
plt.plot (x, ader, label="Analytical Derivative", lw=2, ls="-",color="black")
plt.plot (x, nder-ader, label="Difference", lw=2, ls=":")
plt.title("First derivative, Err (rms) = %.6f " % (rms) )
plt.xlabel('x, m')
plt.ylabel('Amplitude')
plt.legend(loc='lower left')
plt.grid()
plt.show()
```

#### The concept of number of points per wavelength

\begin{equation}
n_\lambda = \dfrac{\lambda}{dx}
\end{equation}

How does the error of the numerical derivative change with the number of points per wavelength?

```python
# Plotting number of points per wavelength
plt.figure(figsize=(10,6))
plt.plot (x, nder,label="Numerical Derivative, 2 points", marker='+', color="blue")
plt.title("First derivative, Error = %.6f, $n_\lambda$ = %.2f " % ( rms, l/dx) )
plt.xlabel('x, m')
plt.ylabel('Amplitude')
plt.legend(loc='lower left')
plt.xlim((xmax/2-l,xmax/2+l))
plt.grid()
plt.show()
```

#### Investigate the error as a function of grid points per wavelength

```python
# Define a range of number of points per wavelength: [nmin=4,5,6 ... ,nmax=40]
# Loop over points, calculate corresponding wavelength and calculate error

# Initialize vectors
nmin=3
nmax=40
na =  np.zeros(nmax-nmin+1)    # Vector with number of points per wavelength
err = np.zeros(nmax-nmin+1)    # Vector with error

j = -1  # array index

# Loop through finite-difference derivative calculation
for n in range (nmin,nmax+1):

    j = j+1   # array index
    na[j] = n

    # Initialize sin function
    l = na[j]*dx  # wavelength
    k = 2*pi/l    # wavenumber
    f = np.sin(k*x)

    # Numerical derivative of the sin function
    for i in range (1, nx-1):
        nder[i]=(f[i+1]-f[i-1])/(2*dx)

    # Analytical derivative of the sin function
    ader= k * np.cos(k*x)
    # Exclude boundaries
    ader[0]=0.
    ader[nx-1]=0.

    # Error (rms)
    err[j] = np.sum((nder-ader)**2)/np.sum((ader**2)) * 100
```

```python
# ----------------------------------------------------------------
# Plotting error as function of number of points per wavelength
plt.figure(figsize=(10,6))
plt.plot(na,err, ls='-', color="blue")
plt.title('Error as a function of $n_\lambda$ ')
plt.xlabel('n$_\lambda$')
plt.ylabel('rms ')
plt.grid()
plt.show()
```

**Alternative Solution (as requested by several comments):**

Let us fix the wavelength lambda and decrease the spatial increment $dx$ and by that increasing the number of points per wavelength. To make it compatible with the Nyquist theorem we start the discretization close to the Nyquist wavelength $\lambda_{Ny} = 2 dx$.

```python
# Let us loop over number of points in the interval [5,100], 3 points corresponds to Nyquist wavelength

n1 =5
n2 =100
err = np.zeros(n2-n1-1)   # vector for error
dxa = np.zeros(n2-n1-1)   # vector for dx
npl = np.zeros(n2-n1-1)   # vector for number of points per lambda

ii = -1

for n in range (n1,n2-1):
    ii   = ii + 1
    x    = np.linspace(0,2*np.pi,n)
    dx   = x[1]-x[0]
    f    = np.sin(x)
    df   = np.cos(x)     # Analytical derivative

    dfn = np.zeros(n)
    # Numerical derivative of the sin function
    for i in range (1, n-1):
        dfn[i]=(f[i+1]-f[i-1])/(2*dx)

    # Calculate error in the interval in which numerical derivative was calculated
    err[ii] = np.sum((df[1:n-1]-dfn[1:n-1])**2)/np.sum((df[1:n-1]**2)) * 100

    dxa[ii] = dx         # dx
    npl[ii] = 2*np.pi/dx # number of points per wavelength

# ----------------------------------------------------------------
# Plotting error as function of dx
plt.figure(figsize=(10,6))
plt.plot(dxa,err, ls='-', color="blue")
plt.title('Error as a function of $dx$ ')
plt.xlabel('dx')
plt.ylabel('Misfit energy (%) ')
plt.grid()
plt.show()

# ----------------------------------------------------------------
# Plotting error as function of points per wavelength
plt.figure(figsize=(10,6))
plt.plot(npl,err, ls='-', color="blue")
plt.title('Error as a function of $n/\lambda$ ')
plt.xlabel('n/$\lambda$')
plt.ylabel('Misfit energy (%) ')
plt.grid()
plt.show()

```

### Conclusions

* 2-point finite-difference approximations can provide  estimates of the 1st derivative of a function
* The accuracy depends on the "number of points per wavelength", i.e., how well we sample the original function
* The more points per wavelength we use the more accurate is the derivative approximation

```python

```


```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# function that returns dy/dt
def model(y,t):
    k = 0.3
    dydt = -k * y
    return dydt

# initial condition
y0 = 5

# time points
t = np.linspace(0,20)

# solve ODE
y = odeint(model,y0,t)

# plot results
plt.plot(t,y)
plt.xlabel('time')
plt.ylabel('y(t)')
plt.show()

```

```python
from sympy.abc import x, t
```

## Phase Portraits Example 1

Matlab post An undamped pendulum with no driving force is described by

y′′+sin(y)=0
We reduce this to standard matlab form of a system of first order ODEs by letting y1=y and y2=y′1. This leads to:

y′1=y2

y′2=−sin(y1)

The phase portrait is a plot of a vector field which qualitatively shows how the solutions to these equations will go from a given starting point. here is our definition of the differential equations:

To generate the phase portrait, we need to compute the derivatives y′1 and y′2 at t=0 on a grid over the range of values for y1 and y2 we are interested in. We will plot the derivatives as a vector at each (y1, y2) which will show us the initial direction from each point. We will examine the solutions over the range -2 < y1 < 8, and -2 < y2 < 2 for y2, and create a grid of 20 x 20 points.

```python
import numpy as np
import matplotlib.pyplot as plt

def f(Y, t):
    y1, y2 = Y
    return [y2, -np.sin(y1)]

y1 = np.linspace(-2.0, 8.0, 20)
y2 = np.linspace(-2.0, 2.0, 20)

Y1, Y2 = np.meshgrid(y1, y2)

t = 0

u, v = np.zeros(Y1.shape), np.zeros(Y2.shape)

NI, NJ = Y1.shape

for i in range(NI):
    for j in range(NJ):
        x = Y1[i, j]
        y = Y2[i, j]
        yprime = f([x, y], t)
        u[i,j] = yprime[0]
        v[i,j] = yprime[1]

Q = plt.quiver(Y1, Y2, u, v, color='r')

plt.xlabel('$y_1$')
plt.ylabel('$y_2$')
plt.xlim([-2, 8])
plt.ylim([-4, 4])

from scipy.integrate import odeint

for y20 in [0, 0.5, 1, 1.5, 2, 2.5]:
    tspan = np.linspace(0, 50, 200)
    y0 = [0.0, y20]
    ys = odeint(f, y0, tspan)
    plt.plot(ys[:,0], ys[:,1], 'b-') # path
    plt.plot([ys[0,0]], [ys[0,1]], 'o') # start
    plt.plot([ys[-1,0]], [ys[-1,1]], 's') # end
```

What do these figures mean?
For starting points near the origin, and small velocities, the pendulum goes into a stable limit cycle.
For others, the trajectory appears to fly off into y1 space. Recall that y1 is an angle that has values from −π to π.
The y1 data in this case is not wrapped around to be in this range.

## Phase Portraits Example 2

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint
def plot_phase_portraits(func):
    y1 = np.linspace(-2.0, 2.0, 20)
    y2 = np.linspace(-2.0, 2.0, 20)
    Y1, Y2 = np.meshgrid(y1, y2)

    t = 0
    u, v = np.zeros(Y1.shape), np.zeros(Y2.shape)

    NI, NJ = Y1.shape
    for i in range(NI):
        for j in range(NJ):
            x = Y1[i, j]
            y = Y2[i, j]
            yprime = func([x, y], t)
            u[i,j] = yprime[0]
            v[i,j] = yprime[1]

    Q = plt.quiver(Y1, Y2, u, v, color='r')

    plt.xlabel('$y_1$')
    plt.ylabel('$y_2$')
    plt.xlim([-2, 2])
    plt.ylim([-2, 2])

    for y20 in [-2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5]:
        tspan = np.linspace(0, 50, 200)
        y0 = [0.0, y20]
        ys = odeint(f, y0, tspan)
        plt.plot(ys[:,0], ys[:,1], 'b-') # path
        plt.plot([ys[0,0]], [ys[0,1]], 'o') # start
        plt.plot([ys[-1,0]], [ys[-1,1]], 's') # end

        y0 = [y20, 0.0]
        ys = odeint(f, y0, tspan)
        plt.plot(ys[:,0], ys[:,1], 'b-') # path
        plt.plot([ys[0,0]], [ys[0,1]], 'o') # start
        plt.plot([ys[-1,0]], [ys[-1,1]], 's') # end

        y0 = [y20, y20]
        ys = odeint(f, y0, tspan)
        plt.plot(ys[:,0], ys[:,1], 'b-') # path
        plt.plot([ys[0,0]], [ys[0,1]], 'o') # start
        plt.plot([ys[-1,0]], [ys[-1,1]], 's') # end

        y0 = [-y20, y20]
        ys = odeint(f, y0, tspan)
        plt.plot(ys[:,0], ys[:,1], 'b-') # path
        plt.plot([ys[0,0]], [ys[0,1]], 'o') # start
        plt.plot([ys[-1,0]], [ys[-1,1]], 's') # end
```

```python
def f(Y, t):
    y1, y2 = Y
    return [-3*y1 + np.sqrt(2)*y2, np.sqrt(2)*y1 - 2*y2]
plot_phase_portraits(f)
```

```python
def f(Y, t):
    y1, y2 = Y
    return [y1 + y2, 4*y1 + y2]
plot_phase_portraits(f)
```

# one resource http://sam-dolan.staff.shef.ac.uk/mas212/docs/l4.pdf

```python

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2021 Lorena A. Barba

# Linear regression by gradient descent

This module of _Engineering Computations_ takes a step-by-step approach to introduce you to the essential ideas of deep learning, an algorithmic technology that is taking the world by storm.
It is at the core of the artificial intelligence boom, and we think every scientist and engineer should understand the basics, at least.

Another term for deep learning is deep neural networks.
In this module, you will learn how neural-network models are built, computationally.
The inspiration for deep learning may have been how the brain works, but in practice what we have is a method to build models, using mostly linear algebra and a little bit of calculus.
These models are not magical, or even "intelligent"—they are just about _optimization_, which every engineer knows about!

In this lesson, we take the first step of model-building: linear regression.
The very first module of the _Engineering Computations_ series discusses [linear regression with real data](http://go.gwu.edu/engcomp1lesson5), and there we found the model parameters (slope and $y$-intercept) analytically.
Let's forget about that for this lesson.
The key concept we introduce here will be _gradient descent_. Start your ride here!

## Gradient descent

This lesson is partly based on a tutorial at the 2019 SciPy Conference by Eric Ma [1]. He begins his tutorial by presenting the idea of _gradient descent_ with a simple quadratic function: the question is how do we find this function's minimum?

$$f(w) = w^2 +3w -5$$

We know from calculus that at the minimum, the derivative of the function is zero (the tangent to the function curve is horizontal), and the second derivative is positive (the curve slants _up_ on each side of the minimum).
The analytical derivative of the function above is $f^\prime(w) = 2w + 3$ and the second derivative is $f^{\prime\prime}(w)=2>0$. Thus, we make $2w+3=0$ to find the minimum.

Let's play with this function using SymPy. We'll later use NumPy, and make plots with Matplotlib, so we load all the libraries in one place.

```python
import sympy
import numpy

from matplotlib import pyplot
%matplotlib inline
```

We run this SymPy method to get beautiful typeset symbols and equations (in the Jupyter notebook, it will use [MathJax](https://en.wikipedia.org/wiki/MathJax) by default):

```python
sympy.init_printing()
```

Now we'll define the Python variable `w` to be a SymPy symbol, and create the expression `f` to match the mathematical function above, and plot it.

```python
w = sympy.Symbol('w', real=True)

f = w**2 + 3*w - 5
f
```

```python
sympy.plotting.plot(f);
```

A neat parabola. We can see from the plot that the minimum of $f(w)$ is reached somewhere between $w=-2.5$ and $w=0$. SymPy can tell us the derivative, and the value where it is zero:

```python
fprime = f.diff(w)
fprime
```

```python
sympy.solve(fprime, w)
```

That looks about right: $-3/2$ or $-1.5$.
We could have also solved this by hand, because it's a simple function.
But for more complicated functions, finding the minimum analytically could be more difficult.
Instead, we can use the iterative method of gradient descent.

The idea in gradient descent is to find the value of $w$ at the function minimum by starting with an initial guess, then iteratively taking small steps down the slope of the function, i.e., in the negative gradient direction.
To illustrate the process, we turn the symbolic expression `fprime` into a Python function that we can call, and use it in a simple loop taking small steps:

```python
fpnum = sympy.lambdify(w, fprime)
type(fpnum)
```

    function

Yep. We got a Python function with the [`sympy.lambdify()`](https://docs.sympy.org/latest/modules/utilities/lambdify.html) method, whose return value is of type `function`. Now, you can pick any starting guess, say $w=10$, and advance in a loop taking steps of size $0.01$ (a choice we make; more on this later):

```python
w = 10.0   # starting guess for the min

for i in range(1000):
    w = w - fpnum(w)*0.01 # with 0.01 the step size

print(w)
```

    -1.4999999806458753

That gave a result very close to the true value $-1.5$, and all we needed was a function for the derivative of $f(w)$. This is how you find the argument of the minimum of a function iteratively.

##### Note

> Implied in this method is that the function is differentiable, and that we can step *down* the slope, meaning its second derivative is positive, or the function is _convex_.

#### Gradient descent steps in the direction of the negative slope to approach the minimum.

## Linear regression

Suppose you have data consisting of one independent variable and one dependent variable, and when you plot the data it seems to noisily follow a trend line.
To build a model with this data, you assume the relationship is _linear_, and seek to find the line's slope and $y$-intercept (the model parameters) that best fit the data.

Though this sounds straightforward, some key ideas of machine learning are contained:

- we don't _know_ the true relationship between the variables, we _assume_ it is linear (and go for it!)
- the model we chose (linear) has some parameters (slope, intercept) that are unknown
- we will need some data (observational, experimental) of the dependent and independent variables
- we find the model parameters by fitting the "best" line to the data
- the model with its parameters can then be used to make _predictions_

Let's make some synthetic data to play with, following the example in Eric Ma's tutorial [1].

```python
# make sythetic data (from Eric's example)
x_data = numpy.linspace(-5, 5, 100)
w_true = 2
b_true = 20

y_data = w_true*x_data + b_true + numpy.random.normal(size=len(x_data))

pyplot.scatter(x_data,y_data);
```

This situation arises often. In **Module 1** of _Engineering Computations_, we used a real data set of Earth temperature over time and we fit an ominously sloped line.
We derived analytical formulas for the model coefficients and wrote our own custom functions, and we also learned that NumPy has a built-in function that will do it for us: `numpy.polyfit(x, y, 1)` will return the two parameters $w, b$ for the line

$$y = w x + b $$

Here, we will instead use gradient descent to get the parameters of the linear model.
The first step is to define a function that represents the _deviation_ of the data from the model.
For linear regression, we use the sum (or the mean) of the square _errors_: the differences between each data point and the predicted value from the linear model (also called _residuals_).

#### Each data point deviates from the linear regression: we aim to minimize the sum of squares of the residuals.

Let's review our ingredients:

1. observational data, in the form of two arrays: $x, y$
2. our linear model: $y = wx + b$
3. a function that measures the discrepancy between the data and the fitting line: $\frac{1}{N}\sum (y_i - f(x_i))^2$

The last item is called a "loss function" (also sometimes "cost function"). Our method will be to step down the slope of the loss function, to find its minimum.

As a first approach, let's again use SymPy, which can compute derivatives for us. Below, we define the loss function for a single data point, and make Python functions with its derivatives with respect to the model parameters.
We will call these functions in a sequence of steps that start at an initial guess for the parameters (we choose zero), and step in the negative gradient multiplied by a step size (we choose $0.01$).
After $1000$ steps, you see that the values of $w$ and $b$ are quite close to the true values from our synthetic data.

```python
w, b, x, y = sympy.symbols('w b x y')

loss = (w*x + b - y)**2
loss
```

```python
grad_b = sympy.lambdify([w,b,x,y], loss.diff(b), 'numpy')
grad_w = sympy.lambdify([w,b,x,y], loss.diff(w), 'numpy')
```

Be sure to read the documentation for [`sympy.lambdify()`](https://docs.sympy.org/latest/modules/utilities/lambdify.html), which explains the argument list.
Now, we step down the slope.
Note that we first compute the derivatives with respect to both parameters _at all the data points_ (thanks to NumPy array operations), and we take the average.
Then we step both parameters (starting from an initial guess of zero).

```python
w = 0
b = 0

for i in range(1000):
    descent_b = numpy.sum(grad_b(w,b,x_data,y_data))/len(x_data)
    descent_w = numpy.sum(grad_w(w,b,x_data,y_data))/len(x_data)
    w = w - descent_w*0.01 # with 0.01 the step size
    b = b - descent_b*0.01

print(w)
print(b)
```

    2.03136593978268
    20.030956250406764

```python
pyplot.scatter(x_data,y_data)
pyplot.plot(x_data, w*x_data + b, '-r');
```

It works! That line looks to be fitting the data pretty well. Now we have a "best fit" line that represents the data, and that we can use to estimate the value of the dependent variable for any value of the independent variable, even if not present in the data. That is, _to make predictions_.

##### Key idea

> "Learning" means building a model by finding the parameters that best fit the data. We do it by minimizing a loss function (a.k.a. cost function), which involves computing derivatives with respect to the parameters in the model.

Here, we used SymPy to help us out with the derivatives, but for more complex models (which may have many parameters), this could be a cumbersome approach.
Instead, we will make use of the technique of _automatic differentiation_, which evaluates the derivative of a function written in code.
You'll learn more about it in the next lesson, on **logistic regression**.

## What we've learned

- Gradient descent can find a minimum of a function.
- Linear regression starts by assuming a linear relationship between two variables.
- A model includes the assumed relationship in the data and model parameters.
- Observational data allows finding the parameters in the model (slope, intercept).
- A loss function captures the deviation between the observed and the predicted values of the dependent variable.
- We find the parameters by minimizing the loss function via gradient descent.
- SymPy computes derivatives with `sympy.diff()` and returns numeric functions with `simpy.lambdify()`.

## References

1. Eric Ma, "Deep Learning Fundamentals: Forward Model, Differentiable Loss Function & Optimization," SciPy 2019 tutorial. [video on YouTube](https://youtu.be/JPBz7-UCqRo) and [archive on GitHub](https://github.com/ericmjl/dl-workshop/releases/tag/scipy2019).

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


# Aufgabe 5a)

```
%matplotlib inline
from sympy import *
init_printing()
```

```
x = symbols("x")
f = 1/(1+3*x)-(1-2*x)/(1+x)
f
```

```
f_strich = diff(f, x)
f_strich
```

```
krel = (f_strich*x/f)
simplify(krel)
```

```
plot(krel, (x, -1/4, 2))
```

Intervall ist kompakt und Krel(x) ist stetig, daraus folgt: Min/Max werden angenommen.

```
krel_strich = simplify(diff(krel, x))
krel_strich
```

```
solve(krel_strich, x)
```

    /usr/lib/python3.3/site-packages/IPython/core/formatters.py:239: FormatterWarning: Exception in image/png formatter:
    \begin{bmatrix}- \frac{1}{2} - \sqrt{3} i / 6, & - \frac{1}{2} + \sqrt{3} i / 6\end{bmatrix}
    ^
    Unknown symbol: \begin (at char 0), (line:1, col:1)
      FormatterWarning,

$$\begin{bmatrix}- \frac{1}{2} - \frac{\sqrt{3} i}{6}, & - \frac{1}{2} + \frac{\sqrt{3} i}{6}\end{bmatrix}$$

Krel'(x) hat also nur imaginäre Nullstellen, dass heißt, die Funktion ist immer positiv oder negativ. Also ist Krel(x) entweder fallend oder steigend (fallend wie im Plot zu sehen ist). Daher werden Min/Max an den Intervallgrenzen angenommen.

```
krel.subs(x, 1/4)
```

```
krel.subs(x, -1/4)
```

Die relative Konditionszahl bewegt sich also im Bereich 1.37 - 5.33. Somit ist das Problem gut konditioniert.


```python
import epipack as epk
import epipack.plottools as epl
import numpy as np
import matplotlib.pyplot as pl

import sympy as sy

S, E, I, A, R = comp = sy.symbols("S E I A R")

latent_period = sy.symbols("t_L") #d
R0 = sy.symbols("R_0")
infectious_period = sy.symbols("t_I") #d
inf_rate = R0 / infectious_period
p = sy.symbols("p")
t = sy.symbols("t")

temporal_infection_rate = inf_rate * (sy.sin(2*sy.pi*t/5)**2+0.5)
#return inf_rate * (np.sin(2*np.pi*t/5)**2+0.5)

model = ( epk.SymbolicEpiModel(comp)
             .set_processes([
                    (S,I, temporal_infection_rate ,E, I),
                    (S,A, temporal_infection_rate ,E, I),
                    (E, (1-p)/latent_period ,A),
                    (E, p/latent_period ,I),
                    (I, 1/infectious_period ,R),
                    (A, 1/infectious_period ,R),
                 ])
             #.set_initial_conditions({
             #       'S' : 1-I0,
             #       'I' : I0,
             #    })
             )

#t = np.linspace(0,100,1000)
#result = model.integrate(t)

model.ODEs_jupyter()

```

$\displaystyle \frac{d}{d t} S = - \frac{R_{0} S \left(A + I\right) \left(\sin^{2}{\left(\frac{2 \pi t}{5} \right)} + 0.5\right)}{t_{I}}$

$\displaystyle \frac{d}{d t} E = \frac{- E t_{I} + R_{0} S t_{L} \left(A + I\right) \left(\sin^{2}{\left(\frac{2 \pi t}{5} \right)} + 0.5\right)}{t_{I} t_{L}}$

$\displaystyle \frac{d}{d t} I = \frac{E p t_{I} + t_{L} \left(A R_{0} S \left(\sin^{2}{\left(\frac{2 \pi t}{5} \right)} + 0.5\right) - I\right)}{t_{I} t_{L}}$

$\displaystyle \frac{d}{d t} A = - \frac{A t_{L} \left(R_{0} S \left(\sin^{2}{\left(\frac{2 \pi t}{5} \right)} + 0.5\right) + 1\right) + E t_{I} \left(p - 1\right)}{t_{I} t_{L}}$

$\displaystyle \frac{d}{d t} R = \frac{A + I}{t_{I}}$

```python

```


## Note for Linear Regression and Data Analysis - Zhibo (Darren) Zhang
      (Partially refer from STA302 University of Toronto, prof Mark Ebden)

Given a set of data points, we want to construct a SLR (simple linear regression) model for the data of the form $y = \beta_0 + \beta_1 x$.

Note that we assume that there is such a model that perfectly fits the data ($y=\beta_0 + \beta_1 x + e$, where $e$ is the random error term), but we never know the actual model. So we want to use some method to estimate it. To be specific, given $x$, we want to use parameters to estimate the prediction output, denoted as $\hat y$.

In this way, if we can estimate the "tool" for the model, which is the parameter $\beta_1$ and $\beta_2$, then we can use the estimator (linear function) to make predctions.

Denote the estimated parameters as $\hat \beta_0$ and $\hat \beta_1$ accordingly. The next step is to get an estimation equation for these two parameters.

$\bullet$ Given a set of training examples of size N, $\{x_1, x_2, ..., x_N\}$ and its according labels $\{y_1, y_2, ..., y_N\}$, denote an input matrix $\mathbf{X}$ to be:

$$ \mathbf{X} =
\begin{bmatrix}
    1 & x_1 \\
    1 & x_2  \\
    \vdots & \vdots \\
    1 & x_N
\end{bmatrix}
$$
where $\mathbf{X}$ is of size N $\times$ 2, denote the parameter vector $\boldsymbol{\hat \beta}$ to be:

$$\boldsymbol{\hat\beta} =
\begin{bmatrix}
    \hat\beta_0 \\
    \hat\beta_1
\end{bmatrix}
$$

$\bullet$ To denote the residual terms (the difference between the actual value and the prediction output, which can be regarded as an estimation for the random error), denote $ \mathbf{\hat e}$ and label vector $\mathbf{Y}$to be:
$$ \mathbf{\hat e} =
\begin{bmatrix}
    \hat e_1 \\
    \hat e_2\\
    \vdots \\
    \hat e_N
\end{bmatrix}, \quad \mathbf{Y} =
\begin{bmatrix}
    y_1\\
    y_2 \\
    \vdots \\
    y_N
\end{bmatrix}
$$

In this way, the equation becomes $\mathbf{Y} = \mathbf{X} \boldsymbol{\hat\beta} + \mathbf{\hat e}$

Our goal is to minimize the sum of residuals, which is the sum of residual vector, so called RSS (Residual Sum of Squares). Thus, denote an equation for RSS:
$$
\begin{equation}
\begin{split}
RSS &= (\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})'(\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})\\
      &=  (\mathbf{Y}' - (\mathbf{X} \boldsymbol{\hat\beta})')(\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})\\
      &=  (\mathbf{Y}' - \boldsymbol{\hat\beta}'\mathbf{X}')(\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})\\
      &=  \mathbf{Y}'\mathbf{Y} - \mathbf{Y}'\mathbf{X} \boldsymbol{\hat\beta} - \boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{Y} + \boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{X} \boldsymbol{\hat\beta}\\
      &=  \mathbf{Y}'\mathbf{Y} - 2\boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{Y} + \boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{X} \boldsymbol{\hat\beta}
\end{split}
\end{equation}
$$

In order to minimize the above equation, take the derivative with respect to $\boldsymbol{\hat\beta}$ and set it to zero:

$$
\frac{\partial RSS}{\partial \boldsymbol{\hat\beta}} = - 2\mathbf{X}'\mathbf{Y} + 2\mathbf{X}'\mathbf{X} \boldsymbol{\hat\beta} = 0
$$

Therefore, we get the estimation for the parameter matrix:

$$
 \boldsymbol{\hat\beta} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y}
$$

_I wil realize this process using python as the next step_

```python

```


# Motivation for Implicit Time Integration
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
import numpy as np
from numpy import *
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
from scipy.integrate import odeint
from scipy.optimize import fsolve
```

# Solve the following initial value problem using the Forward Euler method
Solve the ODE:
\begin{equation}
\frac{\text{d}y}{\text{d}t} = -ay;\quad y(0) = y_0
\end{equation}
The analytical solution is
\begin{equation}
y(t) = y_0 e^{-a t}
\end{equation}

We first define the RHS for this ODE

```python
a = 1000.0
def rhs_sharp_transient(f, t):
    return - a * f
```

we now plot it

```python
yexact = lambda a,y0,t : y0 * np.exp(-a*t)
t = linspace(0,0.1,300)
y0 = 1
plt.plot(t,yexact(1000,y0,t),label='Exact')
plt.legend()
plt.grid()
# plt.savefig('sharp-transient-exact.pdf')
```

# Using Forward Euler
Using the forward Euler method, solve the above ODE using various timestep sizes, starting with $\Delta t = 0.003$.

```python
def forward_euler(rhs, f0, tend, dt):
    ''' Computes the forward_euler method '''
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        f[n+1] = f[n] + dt * rhs(f[n], time[n])
    return time, f
```

```python
f0 = 1.0
tend = 0.1
dt = 0.0005
# dt = 2.0/a
# dt = 0.5*dt
print('dt',dt)
# a = 1000
t,ffe = forward_euler(rhs_sharp_transient,f0, tend, dt)
plt.plot(t,ffe,'r-o',label='Forward Euler, dt='+str(dt))

texact = np.linspace(0,tend,400)
plt.plot(texact,yexact(a,f0,texact),label='Exact')
plt.legend()
plt.grid()
plt.savefig('motivation for implicit dt='+str(dt)+'.pdf')
```

    dt 0.0005

# Using Backward Euler

```python
def be_residual(fnp1, rhs, fn, dt, tnp1):
    '''
    Nonlinear residual function for the backward Euler implicit time integrator
    '''
    return fnp1 - fn - dt*rhs(fnp1, tnp1)

def backward_euler(rhs, f0, tend, dt):
    '''
    Computes the backward euler method
    :param rhs: a rhs function
    '''
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        fn = f[n]
        tnp1 = time[n+1]
        fnew = fsolve(be_residual, fn, args=(rhs,fn,dt,tnp1) )# solve for residual function
        f[n+1] = fnew
    return time, f
```

```python
f0 = 1.0
tend = 0.1
dt = 0.0001

t,fbe = backward_euler(rhs_sharp_transient, f0, tend, dt)
plt.plot(t,fbe, 'r*-',label='Backward Euler')

texact = np.linspace(0,tend,400)
plt.plot(texact,yexact(a,f0,texact),label='Exact')
```

    [<matplotlib.lines.Line2D at 0x15152f3b10>]

# Using Crank-Nicolson

```python
def cn_residual(fnp1, rhs, fn, dt, tnp1, tn):
    '''
    Nonlinear residual function for the Crank-Nicolson implicit time integrator
    '''
    ??

def crank_nicolson(rhs,f0,tend,dt):
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        fn = f[n]
        tnp1 = time[n+1]
        tn = time[n]
        fnew = ??
        f[n+1] = fnew
    return time, f
```

```python
f0 = 1.0
tend = 0.1
dt = 0.0025

t,fcn = crank_nicolson(rhs_sharp_transient, f0, tend, dt)
plt.plot(t,fcn, 'r*-',label='Backward Euler')

texact = np.linspace(0,tend,400)
plt.plot(texact,yexact(a,f0,texact),label='Exact')
```

    [<matplotlib.lines.Line2D at 0x151c892cd0>]

# Using odeint

```python
time = np.linspace(0,0.1,20)
sol = odeint(rhs_sharp_transient,f0,time)
texact = np.linspace(0,tend,400)
plt.plot(texact,y(a,f0,texact),label='Exact')
plt.plot(time,sol, 'k-o', label='ODEint')
plt.legend()
```

    <matplotlib.legend.Legend at 0x152621ed50>

topics/initial-value-problems/Motivation for Implicit Methods - With Gaps.ipynb