$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\L}{\mathcal{L}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Parents}{\mathrm{Parents}}
\newcommand{\NonDesc}{\mathrm{NonDesc}}
\newcommand{\I}{\mathcal{I}}
\newcommand{\dsep}{\text{d-sep}}
\newcommand{\Cat}{\mathrm{Categorical}}
\newcommand{\Bin}{\mathrm{Binomial}}
$$

```python
from __future__ import division

# scientific
%matplotlib inline
from matplotlib import pyplot as plt;
import matplotlib as mpl;
import numpy as np;
import sklearn as skl;
import sklearn.datasets;
import sklearn.cluster;
import sklearn.mixture;

# ipython
import IPython;

# python
import os;
import random;

#####################################################

# image processing
import PIL;

# trim and scale images
def trim(im, percent=100):
    print("trim:", percent);
    bg = PIL.Image.new(im.mode, im.size, im.getpixel((0,0)))
    diff = PIL.ImageChops.difference(im, bg)
    diff = PIL.ImageChops.add(diff, diff, 2.0, -100)
    bbox = diff.getbbox()
    if bbox:
        x = im.crop(bbox)
        return x.resize(((x.size[0]*percent)//100, (x.size[1]*percent)//100), PIL.Image.ANTIALIAS);

#####################################################

# daft (rendering PGMs)
import daft;

# set to FALSE to load PGMs from static images
RENDER_PGMS = True;

# decorator for pgm rendering
def pgm_render(pgm_func):
    def render_func(path, percent=100, render=None, *args, **kwargs):
        print("render_func:", percent);
        # render
        render = render if (render is not None) else RENDER_PGMS;

        if render:
            print("rendering");
            # render
            pgm = pgm_func(*args, **kwargs);
            pgm.render();
            pgm.figure.savefig(path, dpi=300);

            # trim
            img = trim(PIL.Image.open(path), percent);
            img.save(path, 'PNG');
        else:
            print("not rendering");

        # error
        if not os.path.isfile(path):
            raise Exception("Error:  Graphical model image %s not found.  You may need to set RENDER_PGMS=True." % path);

        # display
        return IPython.display.Image(filename=path);#trim(PIL.Image.open(path), percent);

    return render_func;

######################################################
```

# EECS 445:  Machine Learning

## Lecture 17:  Clustering, Mixture Models, and EM

- Instructor:  **Jacob Abernethy**
- Date:  November 14, 2016

*Lecture Exposition:*  Benjamin Bray

# This lecture has notes!

> They can be found [here](./eecs445-f16-em-notes.pdf).
> Written by our very own Benjamin Bray!

## References

- **[MLAPP]** Murphy, Kevin. [*Machine Learning:  A Probabilistic Perspective*](https://mitpress.mit.edu/books/machine-learning-0).  2012.
- **[PRML]** Bishop, Christopher.  [*Pattern Recognition and Machine Learning*](http://research.microsoft.com/en-us/um/people/cmbishop/prml/).  2006.
- **[Koller & Friedman 2009]** Koller, Daphne and Nir Friedman.  [*Probabilistic Graphical Models*](https://mitpress.mit.edu/books/probabilistic-graphical-models).  2009.
- **[Neal & Hinton 1998]** Neal, Radford and Geoffrey Hinton.  ["A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants"](http://www.cs.toronto.edu/~fritz/absps/emk.pdf).  1998.
- **[Do 2008]** Do, Chuong B. and Serafim Batzoglou.  ["What is the Expectation Maximization Algorithm?"](http://www.nature.com/nbt/journal/v26/n8/full/nbt1406.html). 2008.

## Outline

- Clustering & Mixture Models
    - Latent Variable Models
    - Gaussian Mixture Models
- Review:  Information Theory
- Expectation Maximization
    - Evidence Lower Bound
    - Variational Approximation
- Examples
    - Coin Flip Example
    - Gaussian Mixture Models

# Clustering & Mixture Models

> Uses material from **[MLAPP]**

### Review:  Latent Variable Models

In a **Latent Variable Model**, we have
- Observed variables $X$
- Hidden variables $Z$ that influence $X$
- Parameters $\theta$ governing the overall model

In general, our goal is to
> Use what we know to make *inferences* about what we don't know.

### Review:  Latent Variable Models

The most general latent variable model for iid observations $\X$ is
- $X_n$ and $Z_n$ may be sets of variables
- Parameters $\theta$ can be divided into variables $\theta_x$ influencing $X$ and $\theta_z$ influencing $Z$

```python
@pgm_render
def pgm_latent():
    pgm = daft.PGM([4,4], origin=[-2,-1], node_unit=0.8, grid_unit=2.0);
    # nodes
    pgm.add_node(daft.Node("z", r"$Z_n$", 0.7, 1));
    pgm.add_node(daft.Node("x", r"$X_n$", 1.3, 1, observed=True));
    pgm.add_node(daft.Node("theta", r"$\theta$", 1.3, 0.3));

    # edges
    pgm.add_edge("z", "x", head_length=0.08);
    pgm.add_edge("theta", "x", head_length=0.08);
    pgm.add_edge("theta", "z", head_length=0.08);

    pgm.add_plate(daft.Plate([0.4,0.8,1.3,0.5], label=r"$\qquad\qquad\qquad\;\; N$",
    shift=-0.1))

    return pgm;
```

```python
%%capture
pgm_latent("images/pgm/latent.png")
```

### Review:  Clustering

**Goal:** Partition data $\X = \{ x_1, \dots, x_n \} \subset \R^d$ into $K$ disjoint **clusters**.
- Points within a cluster should be more similar to each other than to points in other clusters.
- Estimate **cluster centers** $\mu_k \in \R^d$ for $k=1,\dots,K$
- Estimate **cluster assignments** $z_j \in \{1,\dots,K\}$ for each point $x_j$

Usually, we fix $K$ beforehand!  Use model selection to overcome this limitation.

### Review:  K-Means Clustering

First, pick random cluster centers $\mu_k$.  Then, repeat until convergence:

**E-Step:** Assign $x_j$ to the nearest cluster center $\mu_k$,
$$
z_j = \arg\min_k || x_j - \mu_k ||^2
$$

**M-Step:** Re-estimate cluster centers by averaging over assignments:
$$
\mu_k = \frac{1}{ \# \{ j \mid z_j =k \}} \sum_{j=1}^N x_j \mathbb{I}(z_j = k)
$$

### Mixture Models

Clustering problems motivate the definition of a **mixture model**.
- Observe data from multiple **base distributions** $p_k(x|\theta_k)$ over the same space.
- Proportion of each distribution given by **mixing weights** $\pi = (\pi_1,\dots,\pi_K)$
- Don't know *which* distribution each point comes from.

$$
p(x | \theta) = \sum_{k=1}^K \pi_k p_k(x | \theta_k)
$$

### Mixture Models for Clustering

In clustering, we represent mixtures via latent **cluster indicators**,
$$
\begin{align}
z_n &\sim \Cat[\pi] & \forall\, n = 1,\dots,N \\
x_n \mid z_n &\sim p_{z_n}(\cdot \mid \theta_{z_n}) & \forall\, n = 1,\dots,N
\end{align}
$$

In this form, mixture models are clearly applicable to clustering problems.

> For a deeper understanding of mixture models, see [De Finetti's Theorem](http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture1.pdf).

### Gaussian Mixture Models

In a **Gaussian Mixture Model**, the base distributions are Gaussian.
- Cluster $k$ has mean $\mu_k$, covariance $\Sigma_k$

$$
\begin{align}
z_n &\sim \Cat[\pi] & \forall\, n = 1,\dots,N \\
x_n \mid z_n &\sim \mathcal{N}(\mu_{z_n}, \Sigma_{z_n}) & \forall\, n = 1,\dots,N
\end{align}
$$

### Gaussian Mixture Models:  Visualization

Here are the contours of each base distribution plotted together:

```python
# example stolen from scikit-learn docs
def plot_gmm():
    n_samples = 300

    # generate random sample, two components
    np.random.seed(0)

    # generate spherical data centered on (20, 20)
    shifted_gaussian = np.random.randn(n_samples, 2) + np.array([20, 20])

    # generate zero centered stretched Gaussian data
    C = np.array([[0., -0.7], [3.5, .7]])
    stretched_gaussian = np.dot(np.random.randn(n_samples, 2), C)

    # concatenate the two datasets into the final training set
    X_train = np.vstack([shifted_gaussian, stretched_gaussian])

    # fit a Gaussian Mixture Model with two components
    clf = skl.mixture.GMM(n_components=2, covariance_type='full')
    clf.fit(X_train)

    # display predicted scores by the model as a contour plot
    x = np.linspace(-20.0, 30.0)
    y = np.linspace(-20.0, 40.0)
    X, Y = np.meshgrid(x, y)
    XX = np.array([X.ravel(), Y.ravel()]).T
    Z = -clf.score_samples(XX)[0]
    Z = Z.reshape(X.shape)

    plt.figure(figsize=(10,8))
    CS = plt.contour(X, Y, Z, norm=mpl.colors.LogNorm(vmin=1.0, vmax=1000.0),
                     levels=np.logspace(0, 3, 10))
    CB = plt.colorbar(CS, shrink=0.8, extend='both')
    plt.scatter(X_train[:, 0], X_train[:, 1], .8)

    plt.title('Negative log-likelihood predicted by a GMM')
    plt.axis('tight')
```

```python
plot_gmm()
```

### Gaussian Mixture Models:  Inference & Learning

**Goal:** Since only the data points $x_n$ are observed, we wish to
1. Learn the cluster parameters $\mu_{z_n}$, $\Sigma_{z_n}$
2. Infer the cluster assignments $z_n$ for each point $x_n$

We will use the **Expectation Maximization** algorithm.
- Same general idea as K-means, but *fuzzier* (probabilistic!)
- Alternate between *guessing* latent variables and *re-estimating* parameters.

# Review of Information Theory
> See **Lecture 13** or **[MLAPP]** §2.8 for more details.

### Review:  Entropy

The **entropy** of a discrete random variable $X$ with distribution $p$ is
$$
H[X] = H[p] = E[I(p(X))] = - \sum_{x \in X} p(x) \log p(x)
$$

Entropy measures the **optimal code length**, in bits, for messages drawn from $p$.
- Expected information received when sampling from $X$.

### Review:  Cross Entropy

The **cross entropy** measures the average number of bits needed to encode messages drawn from $p$ when we use a code optimal for $q$:
$$
H(p,q) = -\sum_{x \in \X} p(x) \log q(x)
= - E_p[\log q(x)]
$$

Intuitively, $H(p,q) \geq H(p)$.  The **relative entropy** is the difference $H(p,q) - H(p)$.

### Review:  Relative Entropy

The **relative entropy** or **Kullback-Leibler divergence** of $q$ from $p$ is

$$
\begin{align}
D_{KL}(p || q)
&= \sum_{x \in X} p(x) \log \frac{p(x)}{q(x)} \\
&= H(p,q) - H(p)
\end{align}
$$

> Measures the number of *extra* bits needed to encode messages from $p$ if we use a code optimal for $q$.

# Expectation Maximization

> Uses material from **[Neal & Hinton 1998]**, **[Do 2008]**, and **[MLAPP]**

### Theoretical Perspective

Most textbooks (including **[MLAPP]** and **[PRML]**) introduce EM without much justification.
- This is confusing and unsatisfying!

Instead, we will tackle the theory head-on, using ideas from **[Neal & Hinton 1998]**.

> **Question:** How can we perform maximum likelihood estimation in latent variable models?

### Parameter Estimation with Latent Variables

> **Question:**  How can we perform MLE/MAP in latent variable models?

```python
%%capture
pgm_latent("images/pgm/latent.png")
```

### Complete Data Log-Likelihood

Suppose we observe both $X$ and $Z$.  The **complete data log-likelihood** is
$$
\ell_c(\theta)
= \log P(X,Z | \theta)
= \sum_{n=1}^N \log p(x_n, z_n | \theta)
$$

> **[MLAPP]:** This likelihood function is convex when $p(x_n,z_n | \theta)$ is an exponential family distribution.

### Observed Data Log-Likelihood

In reality, we only observe $\X=(x_1,\dots,x_n)$.  The **observed data log-likelihood** is given by
$$
\begin{align}
\ell(\theta | \X)
&= \log p(\X | \theta)
= \sum_{k=1}^N \log p(x_k | \theta) \\
&= \sum_{k=1}^N \left[ \log \sum_{z_k} p(x_k, z_k | \theta) \right]
\end{align}
$$

> **[MLAPP]:** This likelihood function is **non-convex** even assuming exponential family distributions.  Multiple modes make inference (NP) hard!

### Wrangling a Non-Convex Likelihood

How can we optimize the observed likelihood $\ell(\theta|\X)$?
- *Gradient-based methods?* Tricky to enforce constraints (e.g. positive-definiteness)

> We need a different approach...

### Wrangling a Non-Convex Likelihood

**Observation:** In most models, MLE/MAP is easy with fully observed data.  We will alternate between
- **E-Step:** Inferring the values of the latent variables.
- **M-Step:** Re-estimating the parameters, assuming we have complete data.

> Note the similarity to K-Means!

### Optimizing via Lower-Bounds

From a theoretical standpoint, we will alternate between
1. Estimating a lower bound $\L(q,\theta)$ on $\log p(\X|\theta)$
2. Maximizing this lower bound to obtain a new estimate of $\theta$

(Figure 9.14 of **[PRML]**)

### Expectation Maximization

Our general approach will be to reason about the hidden variables through a proxy distribution $q(Z)$.
- Distribution $q$ is an approximation to $P(Z | \X, \theta)$
- We use $q$ to compute a lower bound on the log-likelihood.

> The $q$ distribution will disappear later, but this perspective is useful for extending EM to new settings.

### Review:  Jensen's Inequality

For a convex function $f$,
$$
\sum_{k=1}^K \lambda_k f(x_k) \geq f\left( \sum_{k=1}^K \lambda_k x_k \right)
$$

If $\lambda_k \geq 0$ and $\sum_{k=1}^K \lambda_k = 1$.

### Evidence Lower Bound

Through Jensen's inequality, we obtain the **evidence lower bound (ELBO)** on the log-likelihood:
$$
\begin{align}
\ell(\theta | \X)
&= \log \sum_z p(\X, z | \theta) \\
&= \log \sum_z q(z) \frac{p(\X, z | \theta)}{q(z)} \\
&\geq \sum_z q(z) \log \frac{p(\X, z | \theta)}{q(z)} \equiv \L(q, \theta)
\end{align}
$$
Important: the choice of distribution $q(z)$ is arbitrary, above holds for all $q$! More on this soon

### Evidence Lower Bound

We can rewrite the lower bound $\L(q, \theta)$ as follows:
$$
\begin{align}
\ell(\theta | \X)
\geq \L(q,\theta)
&= \sum_z q(z) \log \frac{p(\X,z|\theta)}{q(z)} \\
&= \sum_q q(z) \log p(\X,z|\theta) - \sum_z q(z) \log q(z) \\
&= E_q[\log p(\X,Z|\theta)] - E_q[\log q(z)] \\
&= E_q[\log p(\X,Z|\theta)] + H[q]
\end{align}
$$

### ELBO:  Relationship to Relative Entropy

We have just shown that
$$
\ell(\theta|\X) \geq \L(q,\theta)
= E_q[\log p(\X,Z|\theta)] - E_q[\log q(z)]
$$

This very closely resembles the formula for **relative entropy**
$$
\begin{align}
D_{KL}(q || p)
&= E_q[-\log p(Z)] + E_q[\log q(Z)] \\
&= H[q,p] - H[q]
\end{align}
$$

Except that the variables $X$ are clamped to our observations $X=\X$.
- So $P(\X,Z)$ is an *unnormalized* distribution.

### ELBO:  Relationship to Relative Entropy

It is easy to show $\L(q,\theta)$ differs from a relative entropy by only a constant wrt $\theta$:
$$
\begin{align*}
D_{KL}(q || p(Z|\X,\theta))
&= H(q, p(Z|\X,\theta)) - H(q) \\
&= E_q[ -\log p(Z|\X,\theta) ] - H(q) \\
&= E_q[ -\log p(Z,\X | \theta) ] - E_q[ -\log p(\X|\theta) ] - H(q) \\
&= E_q[ -\log p(Z,\X | \theta) ] + \log p(\X|\theta) - H(q) \\
&= -\mathcal{L}(q,\theta) + \mathrm{const.}
\end{align*}
$$

### ELBO:  Tightness

This yields a second proof of the ELBO, since $D_{KL} \geq 0$,
$$
\boxed{
\log p(\X | \theta)
= D_{KL}(q || p(Z | \X, \theta)) + \mathcal{L}(q,\theta)
\geq \mathcal{L}(q,\theta)
}
$$

The quality of our bound $\L(q,\theta)$ depends heavily on the choice of proxy distribution $q(Z)$.

> **Remark:** Maximizing $\L(q, \theta)$ with respect to $q$ is equivalent to minimizing the relative entropy between $q$ and the hidden posterior $P(Z|\X,\theta)$.  Hence, the optimal $q$ is exactly the hidden posterior, for which $\log p(\X|\theta) = \L(q,\theta)$.

The bound is **tight**--we can make it *touch* the log-likelihood.

## Break time!

### Expectation Maximization

Finding a global maximum of the likelihood is difficult in the presence of latent variables.
$$
\hat\theta_{ML} = \arg\max_\theta \ell(\theta | \X) = \arg\max_\theta \log p(\X|\theta)
$$

Instead, the **Expectation Maximization** algorithm gives an iterative procedure for finding a *local* maximum of the likelihood.
- Under the assumption that the hidden posterior $P(Z|\X,\theta)$ is tractable.
- Exploits the evidence lower bound $\ell(\theta|\X) \geq \L(q,\theta)$.

### Expectation Maximization

Consider only proxy distributions of the form
$$
q_\vartheta(Z) = p(Z | \X, \vartheta)
$$

The optimal value for $\vartheta$, in the sense that $\L(\vartheta, \theta) \equiv \L(q_\vartheta, \theta)$ is maximum, depends on $\theta$.
- Similarly, the optimal $\theta$ depends on $\vartheta$.

This suggests an *iterative* scheme.

### Expectation Maximization:  E-Step

Suppose we have an estimate $\theta_t$ of the parameters.  To improve our estimate, we first compute a new lower bound on the observed log-likelihood,
$$
\vartheta_{t+1}
= \arg\max_\vartheta \L(\vartheta, \theta_t)
= \theta_t
$$

Since $q_\vartheta(Z) = p(Z|\X,\vartheta)$, we know from before that $\vartheta=\theta_t$ maximizes $\L(\vartheta, \theta)$.

### Expectation Maximization:  M-Step

Next, we estimate new parameters by optimizing over the lower bound.
$$
\begin{align}
\theta_{t+1}
&= \arg\max_\theta \L(\vartheta_{t+1}, \theta) \\
&= \arg\max_\theta E_q[\log p(\X,Z|\theta)] - E_q[\log q_\vartheta(z)] \\
&= \arg\max_\theta E_q[ \log p(\X,Z | \theta) ]
\end{align}
$$

The last line optimizes over the **expected complete data log-likelihood**.

### Expectation Maximization:  Algorithm

Expectation maximization performs **coordinate ascent** on $\L(\vartheta,\theta)$.

**E-Step:** Compute a new lower bound on the observed log-likelihood,
    $$
    \vartheta_{t+1}
    = \arg\max_\vartheta \L(\vartheta, \theta_t)
    = \theta_t
    $$

**M-Step:** Estimate new parameters by maximizing the bound.
    $$
    \theta_{t+1}
    = \arg\max_\theta \L(\vartheta_{t+1}, \theta)
    = \arg\max_\theta E_q[ \log p(\X,Z | \theta) ]
    $$

### Expectation Maximization:  Standard Formulation

Some authors define the **auxiliary function** or **Q-function**
$$
Q(\theta_t, \theta) = E_q[\log p(\X,Z|\theta)]
$$
where $q(Z) = p(Z|\X,\theta_t)$ as before.

> Some forgo the $q$ function altogether!

### Expectation Maximization:  Standard Formulation

Expectation maximization performs updates on $Q(\theta_t,\theta)$.

**E-Step:** Write down an expression for
    $$
    Q(\theta_t, \theta) = E_q[ \log p(\X,Z | \theta) ],
    $$
    where  $q(Z) = p(Z | \X, \theta_t)$

**M-Step:** Maximize the auxiliary function,
    $$
    \theta_{t+1}
    = \arg\max_\theta Q(\theta_t, \theta)
    $$

### Expectation Maximization:  Convergence

**Theorem:** *After a single iteration of Expectation Maximization, the observed data likelihood of the parameters has not decreased, that is,*
    $$
    \ell(\theta_t | \X) \leq \ell(\theta_{t+1} | \X)
    $$

*Proof:*  This result is a simple consequence of all our hard work so far!
    $$
    \begin{align}
    \ell(\theta_t | \X)
    &= \mathcal{L}(q_{\vartheta_{t+1}}, \theta_t)
        & \text{(Tightness)} \\
    &\leq \mathcal{L}(q_{\vartheta_{t+1}}, \theta_{t+1})
        & \text{(M-Step)} \\
    &\leq \ell(\theta_{t+1} | \X)
        & \text{(ELBO)}
    \end{align}
    $$

### Expectation Maximization:  Convergence

**Theorem:** *(Neal & Hinton 1998, Thm. 2)* Every local maximum of $\L(q, \theta)$ is a local maximum of $\ell(\theta | \X)$.

> Exercise! :)

### Expectation Maximization:  Interpretation

The graph from before should now make more sense:

(Figure 9.14 from **[PRML]**)

### Expectation Maximization:  Examples

We're done with the theory.  Now for some examples:
- *Today:* Coin Flip Example, Gaussian Mixture Models
- *Later:* Hidden Markov Models

We'll also generalize to new settings:
- "Hard" Expectation Maximization
- Variational Expectation Maximization

### Coin Flip:  Problem

Suppose we have two coins, each with unknown bias $\theta_A$ and $\theta_B$, and collect data in the following way:

1. Repeat for $n=1, \dots, N$:
    1. Choose a random coin $z_n$.
    2. Flip this same coin $M$ times.
    3. Record only the total number $x_n$ of heads.

### Coin Flip:  Sample Data

Below is sample data for $N=5$ and $M=10$
```
HTTTHHTHTH
HHHHTHHHHH
HTHHHHHTHH
HTHTTTHHTT
THHHTHHHTH
```

### Coin Flip:  Model

The corresponding probabilistic model is a **mixture of binomials**:
$$
\begin{align}
\theta &= (\theta_A, \theta_B)  &
    &&\text{fixed coin biases} \\
Z_n &\sim \mathrm{Uniform}\{A, B \}    & \forall\, n=1,\dots,N
    &&\text{coin indicators} \\
X_n | Z_n, \theta &\sim \Bin[\theta_{Z_n}, M] & \forall\, n=1,\dots,N
    &&\text{head count}
\end{align}
$$

### Coin Flip:  Graphical Model

```python
@pgm_render
def pgm_coinflip():
    pgm = daft.PGM([4,4], origin=[-2,-1], node_unit=0.8, grid_unit=2.0);
    # nodes
    pgm.add_node(daft.Node("z", r"$Z_n$", 0.7, 1));
    pgm.add_node(daft.Node("x", r"$X_n$", 1.3, 1, observed=True));
    pgm.add_node(daft.Node("theta", r"$\theta$", 1.3, 0.3));

    # edges
    pgm.add_edge("z", "x", head_length=0.08);
    pgm.add_edge("theta", "x", head_length=0.08);

    pgm.add_plate(daft.Plate([0.4,0.8,1.3,0.5],
                             label=r"$\qquad\qquad\qquad\;\; N$",
                             shift=-0.1))
    return pgm;
```

```python
%%capture
pgm_coinflip("images/pgm/coinflip.png")
```

### Coin Flip:  Complete Data Log-Likelihood

The **complete data log-likelihood** for a single trial $(x_n,z_n)$ is
$$
\log p(x_n, z_n | \theta) = \log p(z_n) + \log p(x_n | z_n, \theta)
$$
Here, $P(z_n)=\frac{1}{2}$.  The remaining term is
$$
\begin{align}
\log p(x_n | z_n, \theta)
&= \log \binom{M}{x_n} \theta_{z_n}^{x_n} (1-\theta_{z_n})^{M-x_n} \\
&= \log \binom{M}{x_n} + x_n \log\theta_{z_n} + (M-x_n)\log(1-\theta_{z_n})
\end{align}
$$

### Coin Flip:  Likelihood Plot

There aren't many latent variables, so we can plot $\log P(\X|\theta_A,\theta_B)$ directly!

```python
def coin_likelihood(roll, bias):
    # P(X | Z, theta)
    numHeads = roll.count("H");
    flips = len(roll);
    return pow(bias, numHeads) * pow(1-bias, flips-numHeads);

def coin_marginal_likelihood(rolls, biasA, biasB):
    # P(X | theta)
    trials = [];
    for roll in rolls:
        h = roll.count("H");
        t = roll.count("T");
        likelihoodA = coin_likelihood(roll, biasA);
        likelihoodB = coin_likelihood(roll, biasB);
        trials.append(np.log(0.5 * (likelihoodA + likelihoodB)));
    return sum(trials);

def plot_coin_likelihood(rolls, thetas=None):
    # grid
    xvals = np.linspace(0.01,0.99,100);
    yvals = np.linspace(0.01,0.99,100);
    X,Y = np.meshgrid(xvals, yvals);

    # compute likelihood
    Z = [];
    for i,r in enumerate(X):
        z = []
        for j,c in enumerate(r):
            z.append(coin_marginal_likelihood(rolls,c,Y[i][j]));
        Z.append(z);

    # plot
    plt.figure(figsize=(10,8));
    C = plt.contour(X,Y,Z,150);
    cbar = plt.colorbar(C);
    plt.title(r"Likelihood $\log p(\mathcal{X}|\theta_A,\theta_B)$", fontsize=20);
    plt.xlabel(r"$\theta_A$", fontsize=20);
    plt.ylabel(r"$\theta_B$", fontsize=20);

    # plot thetas
    if thetas is not None:
        thetas = np.array(thetas);
        plt.plot(thetas[:,0], thetas[:,1], '-k', lw=2.0);
        plt.plot(thetas[:,0], thetas[:,1], 'ok', ms=5.0);
```

```python
plot_coin_likelihood([ "HTTTHHTHTH", "HHHHTHHHHH",
         "HTHHHHHTHH", "HTHTTTHHTT", "THHHTHHHTH"]);
```

### Coin Flip:  E-Step

The **E-Step** involves writing down an expression for
$$
\begin{align}
E_q[\log p(\X, Z | \theta )]
&= E_q[\log p(\X | Z, \theta) p(Z)] \\
&= E_q[\log p(\X | Z, \theta)] + \log p(Z) \\
\end{align}
$$

The $\log p(Z)$ term is constant wrt $\theta$, so we ignore it.
> Recall $q \equiv q_{\theta_t} = p(Z | \X,\theta_t)$

### Coin Flip:  E-Step

$$
\begin{align}
E_q[\log p(\X | Z, \theta)]
&= \sum_{n=1}^N E_q \bigg[
        x_n \log \theta_{z_n} + (M-x_n) \log (1-\theta_{z_n})
    \bigg] + \text{const.}
\\
&= \sum_{n=1}^N q_\vartheta(z_n=A)
    \bigg[ x_n \log \theta_A + (M-x_n) \log \theta_A \bigg] \\
&+ \sum_{n=1}^N q_\vartheta(z_n=B)
    \bigg[ x_n \log \theta_B + (M-x_n) \log \theta_B \bigg]
 + \text{const.}
\end{align}
$$

### Coin Flip:  M-Step

Let $a_n = q(z_n = A)$ and $b_n = q(z_n = B)$.  Taking derivatives with respect to $\theta_A$ and $\theta_B$, we obtain the following update rules:
$$
\theta_A = \frac{\sum_{n=1}^N a_n x_n}{\sum_{n=1}^N a_n M}
\qquad
\theta_B = \frac{\sum_{n=1}^N b_n x_n}{\sum_{n=1}^N b_n M}
$$

> **Interpretation:** For each coin, examples are *weighted* according to the probability that they belong to that coin.  Observing $M$ flips is equivalent to observing $a_n M$ *effective* flips.

### Coin Flip:  Python Code

```python
def coin_em(rolls, theta_A=None, theta_B=None, maxiter=10):
    # Initial Guess
    theta_A = theta_A or random.random();
    theta_B = theta_B or random.random();
    thetas = [(theta_A, theta_B)];
    # Iterate
    for c in range(maxiter):
        print("#%d:\t%0.2f %0.2f" % (c, theta_A, theta_B));
        # assign a coin to each trial
        heads_A, tails_A = 0,0;
        heads_B, tails_B = 0,0;

        for trial in rolls:
            likelihood_A = coin_likelihood(trial,theta_A);
            likelihood_B = coin_likelihood(trial,theta_B);
            p_A = likelihood_A / (likelihood_A + likelihood_B);
            p_B = likelihood_B / (likelihood_A + likelihood_B);
            heads_A += p_A * trial.count("H");
            tails_A += p_A * trial.count("T");
            heads_B += p_B * trial.count("H");
            tails_B += p_B * trial.count("T");
        # recompute thetas
        theta_A = heads_A / (heads_A + tails_A);
        theta_B = heads_B / (heads_B + tails_B);

    thetas.append((theta_A,theta_B));
    return thetas, (theta_A,theta_B);
```

### Coin Flip:  Demo

```python
rolls = [ "HTTTHHTHTH", "HHHHTHHHHH", "HTHHHHHTHH",
          "HTHTTTHHTT", "THHHTHHHTH" ];
thetas, _ = coin_em(rolls, 0.1, 0.3, maxiter=6);
```

    #0:	0.10 0.30
    #1:	0.43 0.66
    #2:	0.50 0.75
    #3:	0.51 0.78
    #4:	0.52 0.79
    #5:	0.52 0.79

### Coin Flip:  Plot of Convergence

```python
plot_coin_likelihood(rolls, thetas)
```

### Gaussian Mixture Models:  Specification

Recall from earlier the **Gaussian Mixture Model**:
$$
\begin{align}
\theta &= (\vec\pi, \vec\mu, \vec\Sigma) && \text{model parameters} \\
z_n &\sim \Cat[\pi]  && \text{cluster indicators} \\
x_n | z_n, \theta &\sim \mathcal{N}(\mu_{z_n}, \Sigma_{z_n}) && \text{base distribution}
\end{align}
$$

### GMM:  Complete Likelihood

The complete data log-likelihood for a single datapoint $(x_n, z_n)$ is
$$
\begin{align}
\log p(x_n, z_n | \theta)
&= \log \prod_{k=1}^K \pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k)^{\mathbb{I}(z_n = k)} \\
&= \sum_{k=1}^K \mathbb{I}(z_n = k) \log \pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k)
\end{align}
$$

### GMM:  Hidden Posterior

The hidden posterior for a single point $(x_n, z_n)$ can be found using Bayes' rule:
$$
\begin{align}
p(z_n = k | x_n, \theta)
&= \frac{P(z_n=k | \theta) p(x_n | z_n=k, \theta)}{p(x_N | \theta)} \\
&= \frac{\pi_k \mathcal{N}(x_n | \mu_k, \Sigma_k)}{\sum_{k'=1}^K \pi_{k'} \mathcal{N}(x_n | \mu_{k'}, \Sigma_{k'})}
\end{align}
$$

### GMM:  E-Step

The **expected complete likelihood** is
$$
\begin{align}
E_q[ \log p(\X,Z|\theta)]
&= \sum_{n=1}^N \sum_{k=1}^K
        E_q\big[ \mathbb{I}(z_n = k) \big]
        \log \pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k) \\
&= \sum_{n=1}^N \sum_{k=1}^K
        r_{nk} \log \pi_k
 + \sum_{n=1}^N \sum_{k=1}^K
        r_{nk} \log \mathcal{N}(x_n \mid \mu_k, \Sigma_k)
\end{align}
$$

where $r_{nk} \equiv p(z_n = k \mid x_n,\theta_t)$ is the **responsibility** that cluster $k$ takes for datapoint $x_n$ after step $t$.

### GMM:  M-Step

During the M-Step, we optimize the lower bound with respect to $\theta=(\pi,\mu,\Sigma)$.  Verify that the correct updates are
$$
\begin{gather}
\pi_k = \frac{1}{N} \sum_{n=1}^N r_{nk} = \frac{r_k}{N}
\quad
\mu_k = \frac{\sum_{n=1}^N r_{nk} x_n}{r_k}
\\
\Sigma_k = \frac{\sum_{n=1}^N r_{nk} x_n x_n^T}{r_k} - \mu_k \mu_k^T
\end{gather}
$$

where $r_k = \sum_{n=1}^N r_{nk}$ is the *effective* sample size for cluster $k$.

### General Advice for EM

1. **Specify the model.**  Identify the observed variables, hidden variables, and parameters.  Draw a graphical model.
2. **Identify relevant probabilities.**  This includes the complete data likelihood $p(X,Z|\theta)$ and hidden posterior $p(Z|X,\theta)$.
3. **Derive the E-Step.** Compute $E_q[\log p(\X,Z|\theta)]$.
4. **Derive the M-Step.** Take derivatives and set to zero, when possible.  Otherwise, gradient-based methods can be used.


# Perturbation Analysis for Parabolic Partial Differential Equations

<h2> Cristopher Arenas</h2>
<h3> M.Sc. Student </h3>
<h3> Universidad Técnica Federico Santa Maria, Campus San Joaquín </h3>
cristopher.arenas@alumnos.usm.cl

```python
import numpy as np
import scipy as sc
from scipy.linalg import toeplitz
import matplotlib.pyplot as plt
from ipywidgets import *
from support import *

%matplotlib inline

import matplotlib as mpl
mpl.rcParams['lines.linewidth'] = 2
mpl.rcParams['font.size'] = 15
```

## Introduction

Commonly, exist some phenomena that can be represented as mathematical models, using differential equations. In some cases, simplificationes in the moment of making the model can reduce the difficulty of the representation, but in change, the model could not consider the effect of small factors that are occurring.

Parabolic Partial Differential Equations model different process in the scope of Engineering and Science. One representative Parabolic Partial Differential Equation is the following:
$$
\frac{\partial u(x,t)}{\partial t} = c\,\frac{\partial^2 u(x,t)}{\partial x^2} + b\,\frac{\partial u(x,t)}{\partial x}
$$

Tipically, some process involved in parabolic partial differential equations are related with some quantity or a compond that travels along a fluid and changes its concentration. The process that describe the movement of the compond across the fluid is called **advection** and has a speed asociated. The process that describe the spreading of the concentration of compond in the fluid is called **diffusion** and has a coefficient of diffusion associated.

Is interesting to analize what happens when the process are acting at the same time at different rythms and the implications that carry when time advances. Next, a numerical method will be explained in order to obtain a good estimation of a general equation that models advection-difusion process in differents orders of magnitude.

## Method of multiple scales

The method of multiple scales is a general method applicable in a wide range of problems. In this problems there are two processes acting at the same time, each one with its own scale.
The problem of determinate $u(x,t)$ usually has an approximated value using the approximation:

$$
u(x,t, \varepsilon) \approx u_0(x,t) + \varepsilon \, u_1(x,t)
$$

This expansion works fine with a fixed value of $t$ and $\varepsilon$ near to zero. However, there are problems if $t \geq \mathcal{O}(\varepsilon ^{-1})$, since the approximation is no longer valid (secular terms can appear). There are two processes acting simultaneously that has its own scales. So, it is necessary to introduce two time variables:

- $\tau = t$ the _fast time_
- $T = \varepsilon t$ the _slow time_

With this variables, the solution proposed by the method has the form:

$$
u(x,t,\varepsilon) \approx u(x,\tau, T, \varepsilon)
$$

In this maner, variations of time must be deteminated in function of the variations of the _fast time_ and _slow time_. In terms of algebra:

$$
\frac{\mathrm{d}}{\mathrm{d}t} = \left( \frac{ \partial }{\partial \tau} \right) + \varepsilon \left( \frac{\partial}{\partial T} \right)
$$

Finally, the approximation for $u(x,t)$ that now is asymptotic for $T = \mathcal{O}(1)$ is:

$$
u(x,t,\varepsilon) \approx u_0(x,\tau,T) + \varepsilon\, u_1(x,\tau,T)
$$

## A diffusion-advection equation

Consider the following problem that shows advection around a periodic domain $0 < \theta < 2\pi$ with some small diffusion.

\begin{align*}
\frac{\partial u(\theta,t)}{\partial t} + \frac{\partial}{\partial \theta} (\omega(\theta)\,u(\theta,t)) &= \varepsilon\frac{\partial^2u(\theta,t)}{\partial \theta^2}\\
u(\theta,0) &= F(\theta)\\
u(0,t) &= u(2\pi,t)
\end{align*}

where $\omega(\theta)$ is given over $0 \leq \theta \leq 2\pi$, periodic and positive. $F(\theta)$ is a known function and $\varepsilon$ is a tiny value.

### Application of the Method of multiple scales

 As can be seen in previous equation, there are two process acting at simultaneously: advection on the fast time scale and difusion on the slow scale. According to the method, the next expansion is used:

$$
u(\theta,t,\varepsilon) \approx u_0(\theta,\tau,T) + \varepsilon\, u_1(\theta,\tau,T)
$$

Sustituing into the equation and comparing coefficients there are derivated problems to solve.

**First problem**

$
\mathcal{O}(1):
$

$$
\frac{\partial u_0}{\partial \tau} + \frac{\partial}{\partial \theta}(\omega(\theta) \, u_0) = 0
$$

The first problem can be rewrited as:

$$
\frac{1}{\omega} \left( \frac{\partial}{\partial \tau} + \omega \frac{\partial}{\partial \theta}\right) (\omega\, u_0) = 0
$$

This is an advection problem that depends only of the fast time, no slow time are implicated. An observed moving at  a speed $\omega$ will see the quantity $\omega \, u_0$ remain constant. If the value of $\omega(\theta)$ was $\omega(\theta) = \omega$, where $\omega$ is constant, the solution is reduced to $u_0(\theta,\tau,T) = F(\theta-\omega\tau)$, where $F(\theta)$ is the initial value. However, in a general scenario, is necessary to know how much advanced is done after a time $\tau$. Consider the following initial value problem:

\begin{align*}
\frac{\mathrm{d}\theta}{\mathrm{d}t} &= \omega(\theta)\\
\theta(0) &= 0
\end{align*}

Supouse that $\Theta(t)$ is the solution of this problem and, since $\theta$ is a periodic variable, the function $\Theta(t)$ has a period $P$.

The next function solves the Initial Value Problem explained before using the Forward Euler Method. The return of this function is the function $\Theta(t)$ constructed from a cubic spline. Also, the period $P$ of the function $\Theta(t)$ was computed with the reconstructed funcion form cubic spline [2] modulo $2\pi$. A verification of the period was employed using the Bisection method [1]. In order to determinate a value $s$ such that $\Theta(s) = \hat{\theta}$, since $\Theta(0) = 0$. Let $T(s)$ be the function:

$$
T(s) = \Theta(s) - \hat{\theta}
$$

So, finding a root for $T(s)$, is possible to verify when the displacement reaches multiplies of $2\pi$ and later, obtain the period $P$.

```python
def solve_IVP(w,t_max=10,show_plot=False,delta=1):
    #using Explicit Euler Method to estimate de IVP
    t = np.linspace(0,t_max,50)
    tt = np.linspace(0,t_max,1000)
    h = t[1] - t[0]
    theta = np.zeros(t.shape)
    theta[0] = 0
    for i in range(t.shape[0]-1):
        theta[i+1] = theta[i] + h*w(theta[i],delta)

    #compute cubic splines for estimate the funtion
    spl = cubic_spline(t,theta,end="Natural")

    if show_plot:
        plt.figure(figsize=(8,5))
        #plot points of the numerical solution for theta
        plt.plot(t,theta,"ro",label=r"$\Theta(t)$")
        #plot spline function using the previous points
        plt.plot(tt,cubic_spline_eval2(tt,spl),"-g",label=r"Splines for $\Theta(t)$")

        #get the period of the function
        #th = [2*np.pi,4*np.pi,6*np.pi]
        th = [2*np.pi]
        list_s = []

        for i in th:
            f = lambda x: cubic_spline_eval(x,spl) - i
            #obtain the time s where theta(s) = 2pi, 4pi and 6pi
            s = bisect(f,0,t_max,verb=False)
            #plot a vertical line
            plt.axvline(x=s, color='k', linestyle='--')
            list_s.append(s)

        #plot the points where theta(s) = 2pi, 4pi and 6pi
        plt.plot(list_s,th,"Db")

        #plot the spline functions mod 2pi to verify the period
        plt.plot(tt,(cubic_spline_eval2(tt,spl)%(2*np.pi)),"-b",label=r"$\Theta(t)$ mod $2\pi$")
        #other configurations of the plot
        plt.grid(True)
        plt.legend(loc="best")
        plt.xlabel(r"$t$")
        plt.ylabel(r"$\Theta(t)$")
        plt.title("Period P = "+str(list_s[0])+" [s]")
        #save image in pdf
        plt.savefig("Theta.pdf")
        plt.show()

    Theta = lambda x: cubic_spline_eval(x,spl)

    return Theta
```

In order to solve the first problem, a change of variable to Lagrangian coordinates $s$ and $\tau$ will be done. Let:

$$
\theta(s,\tau) = \Theta(\tau - s)
$$

Where $s$ is the **time delay** since $\theta$ was zero. With that, the coordinate system consisting of one spatial variable $\theta$ and temporal variables $\tau$ and $T$, changes into temporal variables $\tau$, $T$ and $s$.

Also, asociated to this change of variable, exists an inverse that let to obtain $s$ knowing the time $\tau$ and the displacement $\theta$:

$$
s = S(\theta,\tau)
$$

Using the chain rule to recast the differential equation, the following equalities are valid:

\begin{align*}
\left(\frac{\partial}{\partial \tau} \right)_s &= \left(\frac{\partial \tau}{\partial \tau} \right)_s \left(\frac{\partial }{\partial \tau} \right)_\theta + \left(\frac{\partial \theta}{\partial \tau} \right)_s \left(\frac{\partial }{\partial \theta} \right)_\tau = \left( \frac{\partial}{\partial \tau}  \right)_\theta + \omega \left( \frac{\partial}{\partial \theta} \right)_{\tau}\\
\left( \frac{\partial}{\partial \theta} \right)_\tau &= \left(\frac{\partial \tau}{\partial \theta} \right)_\tau \left(\frac{\partial }{\partial \tau} \right)_s + \left(\frac{\partial s}{\partial \theta} \right)_\tau \left(\frac{\partial }{\partial s} \right)_\tau = -\frac{1}{\omega} \left( \frac{\partial}{\partial s} \right)_{\tau}
\end{align*}

Replacing into the equation, the problem is reduced to:

$$
\left(\frac{\partial}{\partial \tau} \right)_s (\omega\,u_0) = 0
$$

Which has constant solution for variable $\tau$ in this coordinate system. The value of $u_0$ can be determinated defining a function $A_0$, which depends of $s$ and $T$ and the speed $\omega$:

$$
u_0(\theta, \tau, T) = \frac{A_0(s,T)}{\omega(\theta)}
$$

This shows that an observed moving at a speed $\omega(\theta)$ will see the quantity $\omega\,u_0$ remain constant. But, this constant is allowed to modify its value in the slow scale $T$. The initial value of this _constant_ is given by the initial condition of the problem:

$$
A_0(s,0) = F(\Theta(-s))\omega(\Theta(-s))
$$

In order to deteminate $A_0$ the second problem must be aborded.

**Second Problem**

$
\mathcal{O}(\varepsilon):
$

$$
\frac{\partial u_0}{\partial T}+ \frac{\partial u_1}{\partial \tau} + \frac{\partial}{\partial \theta} (\omega(\theta) \, u_1) = \frac{\partial^2 u_0}{\partial \theta^2}
$$

Replacing the partially determined $u_0$, the problem becomes:

$$
\frac{1}{\omega} \frac{\partial A_0}{\partial T} + \frac{1}{\omega} \left(\frac{\partial}{\partial \tau}\right)_s (\omega\,u_1) = \frac{1}{\omega} \left( \frac{\partial}{\partial s} \right)_{\tau}\frac{1}{\omega} \left( \frac{\partial}{\partial s} \right)_{\tau} \frac{A_0}{\omega}
$$

with $\omega = \omega(\Theta(\tau-s))$ a $P$-periodic function of $s$ and $\tau$. Thus:

$$
\frac{\partial A_0}{\partial T} + \left(\frac{\partial}{\partial \tau}\right)_s (\omega\,u_1) = \frac{1}{\omega^2} \frac{\partial^2A_0}{\partial s^2}+ \frac{3}{2} \frac{\partial}{\partial s}\left( \frac{1}{\omega^2} \right) \frac{\partial A_0}{\partial s}+\frac{1}{2}\frac{\partial^2}{\partial s^2} \left( \frac{1}{\omega^2}\right) A_0
$$

The right side of the equation has non zero average. In order to mantain the asymptoticness of the expansion of $f$ in $T = \mathcal{O}(1)$, the quantity $\omega \,u_1$ has to be bounded as $\tau$ increases. Thus, the average with respect to $\tau$ of the right hand side must be removed by setting equal to $\displaystyle\frac{\partial A_0}{\partial T}$. The second and third terms of the right side has zero average because they are derivatives with respect to $s$ of functions of $\omega$ and so derivatives with respect to $\tau$. The next equalty must be satisfied:

$$
\frac{\partial A_0}{\partial T} = \frac{1}{P} \int_{0}^{P} \frac{1}{\omega^2(\Theta(\tau-s))} \mathrm{d}\tau \frac{\partial^2A_0}{\partial s^2}
$$

The previous equation corresponds to a Parabolic PDE with diffusion coeficient equals to the average value of $1/\omega^2$, that can be solved numerically in order to detemine $A_0$.

**Generically**, the problem can be solve using a numerical aproximation for the integral and using the value obtained in the PDE:

$$
\begin{aligned}
\frac{1}{P}\int_{0}^{P} \frac{1}{\omega^2(\Theta(\tau-s))} \mathrm{d}\tau \approx \frac{1}{P} \sum_{i=1}^{K} w_i \frac{1}{\omega^2(\Theta(\tau_i-s))} = c(s)
\end{aligned}
$$

Where $w_i$ is the weigh used in the sum of $K$ terms and depends of the numerical method used to estimate the value of the integral. Also, this value depends of temporal variable $s$. So, let $c(s)$ the value of the integral divided by the period $P$ using a numerical method.

Now, considering the PDE with the estimation of the integral as the following equation:

$$
\begin{aligned}
\frac{\partial A_0}{\partial T} &= c(s) \frac{\partial^2A_0}{\partial s^2}
\end{aligned}
$$

Using an implicit finite difference approach (that is inconditionally stable in this cases) for the variable $T$ and central difference for the variable $s$:

The equations to be solved for $j = 1, 2, \ldots N$ are:

$$
- \sigma_j\,a_{j-1,n+1} + (1+2\,\sigma_j)\,a_{j,n+1}-\sigma_j\,a_{j+1,n} = a_{j,n}
$$

where $\sigma_j = \displaystyle\frac{c(s_j)\Delta T}{\Delta s^2}$. And $s$ is a periodic variable.

The next code implements the solution of this PDE, according to the previous explanation

```python
def solve_a0(w,F,eps=1,t_max=1,points_s=50,points_T=80,delta=1,num=1):
    Theta = solve_IVP(w,25,False,delta)
    th1 = lambda x: Theta(x) - 2*np.pi
    #t1 is the period P of Theta(t)
    t1 = bisect(th1,0,25,verb=False)
    #t2 is two periods of Theta(t)
    t2 = 2*t1

    #take equidistant values in the s axis
    s = np.linspace(0,t1,points_s)
    delta_s = t1/(points_s-1)

    #tau is taked between P and 2*P for correctly evaluation of Theta(t)
    tau = np.linspace(t1,t2,points_s)
    #vector c for save the values of c_j
    c = np.zeros_like(s)

    for j in range(s.shape[0]):
        #compute the numerical quadrature of myfun with variable s
        myfun = lambda x: 1/((w(Theta(x-s[j]),delta))**2)
        #using simpsons rule, number points_s must be EVEN
        val = simpsons(myfun, points_s, t1, t2)/t1
        #saving the value of c_j
        c[j] = val

    #computing delta_T
    delta_T = (t_max)/(points_T-1)
    #compute the values of sigma_j
    sigma = (delta_T/(delta_s**2))*c

    #compute matrix
    A = np.zeros((points_s,points_s))
    for i in range(points_s):
        A[i][i] = 1+2*sigma[i]
        if i < points_s-1:
            A[i+1][i] = -sigma[i]
        if i > 0:
            A[i-1][i] = -sigma[i]
    A[-1][0] = -sigma[0]
    A[0][-1] = -sigma[-1]

    sols = np.zeros((points_s,points_T+1))
    for i in range(points_s):
        #sols[i,0] = F(Theta(t1-s[i]),num)*w(Theta(t1-s[i]),delta)
        sols[i,0] = F(Theta(-s[i]%t1),num)*w(Theta(-s[i]%t1),delta)

    for i in range(1,points_T+1):
        sols[:,i] = np.linalg.solve(A,sols[:,i-1])
    return s,sols,delta_T
```

```python
def print_simulation_a0(w,F,i,eps,delta=1,num=1):
    t_max = 1
    s,a0,d_T = solve_a0(w,F,eps,t_max,50,501,delta,num)
    plt.figure(figsize=(10,5))
    plt.plot(s,a0[:,i],"ro")
    plt.xlabel("$s$")
    plt.ylabel("$A_0(s,T)$")
    plt.title(r"$A_0(s,T)$ for $T = "+str(d_T*i)+"$")
    plt.ylim(-4,4)
    plt.grid(True)
    plt.show()
```

Finally, in order to obtain $u_0(\theta,\tau,T)$, is necessary to determinate which $s$ corresponds to a value of $\hat{\theta}$ given. The value of $u_0(\hat{\theta},\tau,T)$ is given by:

$$
u_0(\hat{\theta},\tau,T) = \frac{A_0(s,T)}{\omega(\hat{\theta})}
$$

Let $R(\alpha)$ be the function:

$$
R(\alpha) = \Theta(\alpha) - \hat{\theta}
$$

So, finding a root for $R(\alpha)$, the value of $\alpha$ such $\Theta(\alpha) = \hat{\theta}$ is found. In this case, is necesary go from lagrangian coordinate system to a traditional coordinate system space-time and $\alpha$ is interpretated as $\tau-s$ of the lagrangian coordinates. So, $s$ is creared from the original definition of lagrangian coordinates:

$$
\begin{aligned}
\Theta(\tau-s) &= \Theta(\alpha) = \hat{\theta}\\
& \Rightarrow \tau-s = \alpha\\
& \Rightarrow s = \tau - \alpha
\end{aligned}
$$

Finally, with this value of $s$ is possible to come back to traditional coordinate system to interpretate and compare results.

The next code implements the calculation of values for $u_0$ taking equdistant points in the $\theta$ axis from $0$ to $2\pi$. The bisection method [1] was used for obtain the zeros of $R(\alpha)$ when was needed.

```python
def get_u0(w,F,i,eps=1,t_max=10,points_th=50,points_T=500,delta=1,num=1):
    Theta = solve_IVP(w,25,False,delta)
    th1 = lambda x: Theta(x) - 2*np.pi
    #get period of the function
    period = bisect(th1,0,25,verb=False)

    #points to using in the s axis
    points_s = 50

    #computing A0(s,T)
    s, a0, delta_T = solve_a0(w,F,eps,t_max,points_s,points_T,delta,num)

    #obtain points between 0 and 2pi
    th = np.linspace(0,2*np.pi,points_th)

    #generate structure to save values of u0
    u_0 = np.zeros_like(th)

    #compute cubic spline for a0 in T = i*delta_T
    spl = cubic_spline(s,a0[:,i],end="Natural")
    a_0 = lambda x: cubic_spline_eval(x,spl)

    for j in range(1,th.shape[0]):
        fun = lambda x: Theta(x)-th[j]
        alpha = bisect(fun,0,2*period,verb=False)
        s2 = ((i*delta_T/eps)-alpha)%period
        u_0[j] = a_0(s2)/w(th[j],delta)
    u_0[0] = u_0[-1]
    return th,u_0,delta_T
```

```python
def print_simulation_u0(w,F,i,eps,delta=1,num=1):
    t_max = 1
    th,u0,d_T = get_u0(w,F,i,eps,t_max,50,501,delta,num)
    plt.figure(figsize=(10,5))
    plt.plot(th,u0,"gD")
    plt.xlabel(r"$\theta$")
    plt.ylabel(r"$u_0(\theta,\tau,T)$")
    plt.title(r"$u_0(\theta,\tau,T)$ for $T = "+str(d_T*i)+"$, $t="+str(d_T*i/eps)+"$")
    plt.ylim(-4,4)
    plt.grid(True)
    plt.show()
```

### Using Finite Difference Method

Next, the problem is solved numerically with finite differences with an implicit method. Consider the problem in the form:

$$
\frac{\partial u(\theta,t)}{\partial t} + \frac{\mathrm{d}\omega(\theta)}{\mathrm{d} \theta} u(\theta,t) + \frac{\partial u(\theta,t)}{\partial \theta} \omega(\theta) = \varepsilon \frac{\partial^2 u(\theta,t)}{\partial \theta^2}
$$

The formulas of the **implicit** finite differences will be **backward in time** and **central in space** for the first derivative (in order to mantain the spatial derivative with a second order aproximation). Let $u_{i,n} \approx u(\theta_i,t_n)$, where $\theta_i$ and $t_n$ are $N$ and $M$ equidistant points in the spatial and temporal axes, respectively.

$$
\begin{aligned}
\frac{\partial u(\theta,t)}{\partial t} &= \frac{u_{i,n}-u_{i,n-1}}{\Delta t} + \mathcal{O}(\Delta t)\\
\frac{\partial u(\theta,t)}{\partial \theta} &= \frac{u_{i+1,n}-u_{i-1,n}}{2\,\Delta \theta} + \mathcal{O}(\Delta \theta^2)\\
\frac{\partial^2 u(\theta,t)}{\partial \theta^2} &= \frac{u_{i-1,n}-2\,u_{i,n}+u_{i+1,n}}{\Delta \theta^2} + \mathcal{O}(\Delta \theta^2)\\
\end{aligned}
$$

with $\Delta \theta = \displaystyle\frac{2\pi}{N-1}$ and $\Delta t = \displaystyle\frac{t_{max}}{M-1}$ .

Replacing into the PDE, there are a system of equation to be solved for unkowns in different values of time where the equations are given by:

$$
\left( \frac{\omega(\theta_i)\,\Delta t}{2\Delta \theta}-\frac{\varepsilon\, \Delta t}{\Delta \theta^2} \right)\,u_{i-1,n+1}+ \left( 1+\frac{\mathrm{d}\omega(\theta_i)}{\mathrm{d} \theta}\Delta t + 2\frac{\varepsilon\, \Delta t}{\Delta \theta^2} \right)\,u_{i,n+1} + \left(- \frac{\omega(\theta_i)\,\Delta t}{2\Delta \theta}-\frac{\varepsilon\, \Delta t}{\Delta \theta^2} \right)\, u_{i+1,n+1} = u_{i,n}
$$

for $0 \leq i<N$ and $n \in [0,M]$. For a better visualization, let be:

$$
\begin{aligned}
\alpha &= \frac{\varepsilon\,\Delta t}{\Delta \theta^2}\\
\beta_i &= \frac{\omega(\theta_i)\,\Delta t}{\Delta \theta}\\
\gamma_i &= \omega'(\theta_i)\,\Delta t
\end{aligned}
$$

So, the previous equation can be written as:

$$
\left( \frac{1}{2}-\frac{\varepsilon\, \Delta t}{\Delta \theta^2} \right)\,u_{i-1,n+1}+ \left( 1+\frac{\mathrm{d}\omega(\theta_i)}{\mathrm{d} \theta}\Delta t + 2\frac{\varepsilon\, \Delta t}{\Delta \theta^2} \right)\,u_{i,n+1} + \left(- \frac{\omega(\theta_i)\,\Delta t}{2\Delta \theta}-\frac{\varepsilon\, \Delta t}{\Delta \theta^2} \right)\, u_{i+1,n+1} = u_{i,n}
$$

So, the previous equation can be written as:

$$
\left( \frac{\beta_i}{2}- \alpha \right)\,u_{i-1,n+1}+ \left( 1+ \gamma_i + 2 \alpha \right)\,u_{i,n+1} + \left(-  \frac{\beta_i}{2} - \alpha \right)\, u_{i+1,n+1} = u_{i,n}
$$

for $0 \leq i<N$ and $n \in [0,M]$.

An **explicit** finite difference can be employed using the forward difference for the temporal variable and mantaining the previous differences for the spatial variable:

$$
\begin{aligned}
\frac{\partial u(\theta,t)}{\partial t} &= \frac{u_{i,n+1}-u_{i,n}}{\Delta t} + \mathcal{O}(\Delta t)\\
\frac{\partial u(\theta,t)}{\partial \theta} &= \frac{u_{i+1,n}-u_{i-1,n}}{2\,\Delta \theta} + \mathcal{O}(\Delta \theta^2)\\
\frac{\partial^2 u(\theta,t)}{\partial \theta^2} &= \frac{u_{i-1,n}-2\,u_{i,n}+u_{i+1,n}}{\Delta \theta^2} + \mathcal{O}(\Delta \theta^2)\\
\end{aligned}
$$

So, the iterative step to be calculated will be:
$$
u_{i,n+1} = \left( \frac{\beta_i}{2}+ \alpha \right)\,u_{i-1,n}+ \left( 1- \gamma_i - 2 \alpha \right)\,u_{i,n} + \left(-  \frac{\beta_i}{2} + \alpha \right)\, u_{i+1,n}
$$

for $0 \leq i<N$ and $n \in [0,M]$.

The following code implements finite difference method for this problem.

```python
def fd_method(w,wp,F,N,M,t_max,eps,delta=1,num=1,method="Implicit"):
    Theta = np.linspace(0,2*np.pi,N)
    dt = t_max/(M-1)

    dth = Theta[1]-Theta[0]

    A = np.zeros((N-1,N-1))
    b = np.zeros((N,M))

    b[:-1,0] = F(Theta[:-1],num)
    b[-1] = b[0]

    #defining some "constants" for matrix generation
    alpha = ((eps*dt)/(dth**2))*np.ones(N)
    beta = (dt/dth)*w(Theta,delta)
    gamma = dt*wp(Theta)
    #print("eps",eps)
    #print("dt",dt)
    #print("dth**2",dth**2)
    #print("eps*dt/dth**2 = alpha",alpha)

    if method=="Implicit":
        for i in range(N-1):
            A[i,i] = 1+2*alpha[i]+gamma[i]
            A[i,i-1] = -alpha[i]-0.5*beta[i]
            if i<N-2:
                A[i,i+1] = -alpha[i]+0.5*beta[i]
            else:
                A[i,0] = -alpha[i]+0.5*beta[i]
        for j in range(M-1):
            b[:-1,j+1] = np.linalg.solve(A,b[:-1,j])
            b[-1,:] = b[0,:]
    elif method=="Explicit":
        for i in range(N-1):
            A[i,i] = 1-gamma[i]-2*alpha[i]
            A[i,i-1] = 0.5*beta[i]+alpha[i]

            if i<N-2:
                A[i,i+1] = -0.5*beta[i]+alpha[i]
            else:
                A[i,0] = -0.5*beta[i]+alpha[i]
        for j in range(M-1):
            b[:-1,j+1] = np.dot(A,b[:-1,j])
            b[-1,:] = b[0,:]

    return b
```

### Using FFT

The Fast Fourier Transform will be used to determine the spatial derivative of $u(\theta,t)$ knowing the value of $u$ in the initial time, i.e. since the value of $u(\theta,t=0)$ is known, the value of $\displaystyle\frac{\partial^n u(\theta, t=0)}{\partial \theta^n}$ can be determinated. Besides, the spatial variable is periodic with period equal to $2\pi$. Later, using a forward approach for the temporal variable, solutions of $u(\theta,t)$ will be obtained.

Consider the problem in the form:

$$
\frac{\partial u(\theta,t)}{\partial t} + \frac{\mathrm{d} \omega(\theta)}{\mathrm{d} \theta} u(\theta,t) + \frac{\partial u(\theta,t)}{\partial \theta} \omega(\theta) = \varepsilon \frac{\partial^2 u(\theta,t)}{\partial \theta^2}
$$

Let be $w^{(k)}_{j,n}$ the value obtained using FFT for the derivative of $k$ order for $u(\theta,t)$ evaluated in $\theta_j$ for the time $t_n$. Let be $v_{j,n}$ the aproximation of $u(\theta,t)$ evaluated in $\theta_j$ for the time $t_n$. Put all the previous definitions together, the problem is reduced to an iterative method that calculates the next time step using the information of the values of $u(\theta_j,t_n)$, the first derivative of $u(\theta_j,t_n)$ and the second derivative of $u(\theta_j,t_n)$ for the actual time. Consider that $\theta_i$ and $t_n$ are $N$ and $M$ equidistant points in the spatial and temporal axes.

The iterative step to be used in this case is:

$$
v_{j,n+1} = \varepsilon\,\Delta t\,w_{j,n}^{(2)} - \omega(\theta_j)\,\Delta t\,w_{j,n}^{(1)}+(1-\omega'(\theta_j)\,\Delta t)v_{j,n}
$$
for $0 \leq j<N$ and $n \in [0,M]$.

The next code implements the spectral derivative using FFT in order to obtain the derivative of $\nu$ order of a function $v$ valid for the spatial interval $[0,2\pi]$. The code was extracted from [3].

```python
def spectralDerivativeByFFT(v,nu=1):
    if not np.all(np.isreal(v)):
        raise ValueError('The input vector must be real')
    N=v.shape[0]
    K=np.fft.fftfreq(N)*N
    iK=(1j*K)**nu
    v_hat=np.fft.fft(v)
    w_hat=iK*v_hat
    if np.mod(nu,2)!=0:
        w_hat[int(N/2)]=0
    return np.real(np.fft.ifft(w_hat))
```

The next code implements the iterative step in order to solve this kind of problem.

```python
def fft_method(w,wp,F,N,M,t_max,eps,delta=1,num=1):
    Theta = np.linspace(0,2*np.pi,N)
    #T = np.linspace(0,2,M)
    #dt = T[1]-T[0]
    dt = t_max/(M-1)

    #defining a matrix to save all the values of the function in all the time steps
    v = np.zeros((N,M))
    #setting values for the initial time
    v[:,0] = F(Theta,num)

    w1 = np.zeros(N)
    w2 = np.zeros(N)

    #iterative step
    for n in range(M-1):
        #calculate the first derivative with FFT
        w1[1:] = spectralDerivativeByFFT(v[1:,n],1)
        w1[0] = w1[-1]
        #calculate the second derivative with FFT
        w2[1:] = spectralDerivativeByFFT(v[1:,n],2)
        w2[0] = w2[-1]

        #get the values of the function in the next time
        v[:,n+1] = eps*dt*w2 - dt*w(Theta,delta)*w1+(np.ones(N)-dt*wp(Theta))*v[:,n]
    return v
```

### Numerical Examples

For Numerical Examples, three different functions associated to the initial values will be used. These are:

$$
\begin{aligned}
F_1(\theta) &= 2\,\sin{(\theta)}\\
F_2(\theta) &= 2\,\exp{(-(\theta-\pi)^2)}\\
F_3(\theta) &= 2\,\sin^2{(0.5\,\theta)}\\
\end{aligned}
$$

```python
def F(theta,num=1):
    if num == 1:
        return 2*np.sin(theta)
    elif num==2:
        return 2*np.exp(-(theta-np.pi)**2)
    elif num==3:
        return 2*np.sin(0.5*theta)**2
    return theta*0
```

According to all the theoretical part explained before, two numerical examples will be tested for the diffusion and advection equation.

Again, consider the following problem that shows advection around a periodic domain $0 < \theta < 2\pi$ with some small diffusion.

\begin{align*}
\frac{\partial u(\theta,t)}{\partial t} + \frac{\partial}{\partial \theta} (\omega(\theta)\,u(\theta,t)) &= \varepsilon\frac{\partial^2u(\theta,t)}{\partial \theta^2}\\
u(\theta,0) &= F(\theta)\\
u(0,t) &= u(2\pi,t)
\end{align*}

where $\omega(\theta)$ is given over $0 \leq \theta \leq 2\pi$, periodic and positive. $F(\theta)$ is a known function and $\varepsilon$ is a tiny value.

**Numerical Example 1**
The first numerical example considers a constant value $\delta$ for the speed $\omega$. Let be:
\begin{align}
\omega(\theta) &= \delta\\
\omega'(\theta) &= 0\\
\end{align}

```python
def w(theta, delta=1):
    return delta*(theta**0)
def wp(theta):
    return theta*0
```

```python
def plot_functions(w,wp,F,delta=1,num=1):
    th = np.linspace(0,4*np.pi,200)
    thh = np.linspace(0,2*np.pi,100)

    plt.figure(figsize=(18,5))

    label_fun = "0"
    if num==1:
        label_fun=r"2\,\sin{(\theta)}"
    elif num==2:
        label_fun=r"2\,\exp(-(\theta-\pi)^2)"
    elif num==3:
        label_fun=r"2\,\sin(0.5\,\theta)^2"

    plt.subplot(121)
    plt.plot(th,w(th,delta),"-r",label=r"$w(\theta) = "+str(delta)+"$")
    plt.plot(th,wp(th),"-b",label=r"$w'(\theta) = 0$")
    plt.xlim(-1,12)
    plt.ylim(-4,4)
    plt.axvline(x=0, color='k', linestyle='--')
    plt.axvline(x=2*np.pi, color='k', linestyle='--')
    plt.grid(True)
    plt.xlabel(r"$\theta$")
    plt.legend(loc="best")

    plt.subplot(122)
    plt.plot(thh,F(thh,num),"-g",label = r"$F(\theta) = "+label_fun+"$")
    plt.xlim(-1,7)
    plt.ylim(-4,4)
    plt.axvline(x=0, color='k', linestyle='--')
    plt.axvline(x=2*np.pi, color='k', linestyle='--')
    plt.grid(True)
    plt.xlabel(r"$\theta$")
    plt.ylabel(r"$F(\theta)$")
    plt.legend(loc="best")
    plt.show()
```

```python
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
interact(plot_functions,w=fixed(w),wp=fixed(wp),F=fixed(F),delta=delta_widget,num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.plot_functions>

As can be seen previously, the function $\omega(\theta)$ is a positive and periodic function around the interval $[0,2\pi]$. The derivative of a constant function is zero. The function $F(\theta)$ is shown between the same interval.

The first approach for this example consists in determinate numerically the solution of the partial differential equation using Finite Differences (Explicit and Implicit) and Fast Fourier Transform.

```python
def simulation(w,wp,F,i,N,M,t_max,eps,fd_mode,delta,num):

    #spatial axis
    Theta = np.linspace(0,2*np.pi,N)
    #temporal axis
    T = np.linspace(0,t_max,M)

    y1 = fft_method(w,wp,F,N,M,t_max,eps,delta,num)
    y2 = np.zeros_like(y1)
    label_fd = "No mode selected for FD"
    if fd_mode == "Implicit":
        y2 = fd_method(w,wp,F,N,M,t_max,eps,delta,num)
        label_fd = "Implicit Finite Difference"
    elif fd_mode == "Explicit":
        y2 = fd_method(w,wp,F,N,M,t_max,eps,delta,num,"Explicit")
        label_fd = "Explicit Finite Difference"

    a1 = np.zeros(i+1)
    a2 = np.zeros(i+1)

    for k in range(i+1):
        a1[k] = max(abs(y1[:,k]))
        a2[k] = max(abs(y2[:,k]))

    plt.figure(figsize=(17,13))

    plt.subplot(2, 2, 1)
    plt.title("Simulation at t = "+str(T[i])+" [s]")
    plt.plot(Theta[:],y1[:,i],"-yo", label="Fast Fourier Transform")
    plt.ylim(-5,6)
    plt.grid(True)
    plt.xlabel(r"$\theta$")
    plt.ylabel(r"$u(\theta,t)$")
    plt.legend(loc="best")

    plt.subplot(2, 2, 2)
    plt.title("Simulation at t = "+str(T[i])+" [s]")
    plt.plot(Theta[:],y2[:,i],"-bo", label=label_fd)
    plt.ylim(-5,6)
    plt.grid(True)
    plt.xlabel(r"$\theta$")
    plt.ylabel(r"$u(\theta,t)$")
    plt.legend(loc="best")

    plt.subplot(2, 2, 3)
    plt.title("Max Absolute Value")
    plt.plot(T[:i+1],a1,"-yo", label="Fast Fourier Transform")
    plt.xlim(-0.1*t_max,1.1*t_max)
    plt.grid(True)
    plt.xlabel(r"$t$")
    plt.ylabel(r"$u(\theta,t)$")
    plt.legend(loc="best")

    plt.subplot(2, 2, 4)
    plt.title("Max Absolute Value")
    plt.plot(T[:i+1],a2,"-bo", label=label_fd)
    plt.xlim(-0.1*t_max,1.1*t_max)
    plt.grid(True)
    plt.xlabel(r"$t$")
    plt.ylabel(r"$u(\theta,t)$")
    plt.legend(loc="best")

    plt.show()
```

The numerical simulation can be observed on a interactive way changing some parameters:
* $i$: solution of the equation at time $t_i = i\,\Delta t$. Initially $i = 0$.
* $N$: number of points to be evaluated in the $\theta$ axis. Initially $N = 50$.
* $M$: number of intervals of time to be displayed. Initially $M = 80$.

Aditionally, various values of $\varepsilon$ will be used.

```python
i_widget = IntSlider(min=0, max=79, value=0)
N_widget = IntSlider(min=2, max=200, value=50)
M_widget = IntSlider(min=2, max=501, value=80)
eps_widget = Dropdown(options=[100.,10.,2.,1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17,0.],
                      value=0, description='eps')
delta_widget = FloatSlider(min=0.1,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
fd_widget = Dropdown(options =  ["Explicit", "Implicit"], value="Implicit", description = "FD Mode")
time_widget = IntSlider(min=1,max=10,step=1,value=1,description="Max Time")
def update_i_values(*args):
    i_widget.max = M_widget.value-1
M_widget.observe(update_i_values, 'value')

#def Wp(theta,delta=1):
#    return 0*theta

interact(simulation,
         w=fixed(w),
         wp=fixed(wp),
         F=fixed(F),
         i=i_widget,
         N=N_widget,
         M=M_widget,
         eps=eps_widget,
         t_max = time_widget,
         fd_mode=fd_widget,
         delta=delta_widget,
         num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.simulation>

The first case $\varepsilon = 0$ is simply an advection equation without diffusion. Since the speed $\omega$ is constant the solution, tipically a wave is moving at this constant rate. Is expected that the maximum absolute value of the solution observed be the same in all the time elapsed.
- The FFT has stability problems when $N$ is much greater than $M$ (apparently with a quotient $N/M$ greater to 1 and nearer to 2). Also, when the value of $M$ is tiny the simulation has stability problems more prematurely. In cases where there are no stability issues, the maximum absolute value of the solution is increasing slightly. When the value of $M$ is bigger, i.e., the time between two neighbor points is shorter, the increment of the maximun value obtained is lower that with a $M$ small. This factor maybe is associated with the numerical error of the method.
- The Implicit FD does not have problems of stability. It can be observed that the maximum absolute value of the solutions is decresing in time slightly, and also, with bigger values of $M$ the change in this decrease is lower that the same effect with a small $M$. Again, this difference between the expected behavior and the numerical results can be interpretated as numerical error asociated.
- The Explicit FD does not have problems of stability with in this case and its behaviour is better to comparing with FFT. The maximum value is increasing int time such as FFT. Also, with as greater is the value of $M$, the total increment of the maximum value in time $t=10$ is lower.

Cases when $\varepsilon > 0$ are examples of a advection-diffusion equation when the solution is disipated around the axis. When the coeficient is greater, the disipation of the values is faster. The advection process, barely can be seen. The problem can be seen as the oposite as the original problem with a slow difusion, where now the advection is the slow process compared to the diffusion and if the coeficient of difusion is a big value, then the solution is practicaly a constant solution (explained for a really fast difusion, without considering what was the initial distribution of the function).
- Again, the FFT has stability problems related to the quotient $N/M$ in the same way that the previous case. However, in this cases there is an aditional effect of the $\varepsilon$ parameter. For $\varepsilon = 1$, the problem of stability ocurrs with a $N/M$ quotient near to 0.10. It can be seen that at higher values of $\varepsilon$, the problem of stability occurs sooner, so the quotient $N/M$ has to mantain as low as possible. When the stability problem arises, the maximum absolute value is disproportionately huge, and as long as the coeficient $N/M$ increases, the increment of the maximum absolute value ocurs sooner. When there are not stability problems, the behaviour of the maximum absolute value is the expected, observing a diffusion processs in the time and decresing this maximum value faster if the diffusion coefficient is bigger.
- The Implicit FD does not have problems of stability. The behaviour of the maximun absolute value is the expected in all the cases and, as ocurrs in FFT, as bigger the diffusion coefficient is, more sooner is the time when the maximum aplitude reaches the zero value.
- The Explicit FD has problems of stability with the quotient $N/M$ near to 0.16, but the stability issues ocurrs with minor frequency that the FFT. The behaviour in this cases, when the method is stable, is the expected behaviours for a not small diffusion coefficient.

The final cases consider $0 < \varepsilon \ll 1$ and is the interest for this work. Is expected to observe advection in time and apparently no diffusion process.

- In the FFT the phenomenon of slow diffusion cannot be observed clearly, but the plot of maximum aplitude value shows some differences varying the paremeters. With $\varepsilon  = 0.1$ and $\varepsilon = 0.01$ the maximum absolute value of the solutions decrease slower as the order of magnitude of $\varepsilon$ decrease. However, with orders of magnitude for $\varepsilon$ smaller than $10^{-3}$ the phenomenon the maximum is increasing in time. As the order of magnitude is continue decreasing, the numerical error is reaching the results obtained for the first case.
- In the Implicit FD during the decrease of orders of magnitude for $\varepsilon$ it can be seen that the inclination of maximum absolute value to have lower values is the expected and for smaller values of $\varepsilon$, the decreace is smaller too. Aproximately, for order of magnitud of $10^{-7}$ it cannot be seen differences between the maximum absolute value comparing to the first case explained before.
- The Explicit FD has the same behaviour than FFT. When $\varepsilon$ has a order of magnitude lower than $10^{-2}$ the decreasing maximum value in time of the solutions changes to an increasing maximum value in time. The solutions are approaching to the first case.

In general, the Fast Fourier Transform has some problems caused by certain parameter values and is a conditionally stable method. The Explicit Finite Difference is better than FFT, but has some issues on stability too. Otherwise, the Implicit Finite Difference seems to be more trustworthy, since the behaviour expected for this case is aproximately the same that the numerical results. However, all the methods introduce some numerical error, which could be seen in the first case when $\varepsilon = 0$.

Now, the question is, ¿what is the effect of the small values for $\varepsilon$ of the equation for the numerical results? First of all, using the method of multiscales, is necessary to known how is the advance of the solutions according to the speed $\omega$.

The next code shows the numerical solution of the Initial Value Problem asociated to the advance of $\Theta$ in the time $t$.

$$
\frac{\mathrm{d}\theta}{\mathrm{d}t} = \omega(\theta)
$$

```python
def plot_Theta(w,delta):
    t_max = 25
    Theta = solve_IVP(w,t_max,False,delta)
    vTheta = np.vectorize(Theta)
    t = np.linspace(0,t_max,50)
    tt = np.linspace(0,t_max,1000)
    plt.figure(figsize=(8,5))
    #plot points of the numerical solution for theta
    plt.plot(t,vTheta(t),"ro",label=r"$\Theta(t)$")
    #plot spline function using the previous points
    plt.plot(tt,vTheta(tt),"-g",label=r"Splines for $\Theta(t)$")

    #get the period of the function
    f = lambda x: Theta(x) - 2*np.pi
    #obtain the time s where theta(s) = 2pi
    s = bisect(f,0,t_max,verb=False)
    #plot a vertical line
    plt.axvline(x=s, color='k', linestyle='--')

    #plot the points where theta(s) = 2pi, 4pi and 6pi
    plt.plot(s,2*np.pi,"Db")

    #plot the spline functions mod 2pi to verify the period
    plt.plot(tt,vTheta(tt)%(2*np.pi),"-b",label=r"$\Theta(t)$ mod $2\pi$")
    #other configurations of the plot
    plt.grid(True)
    plt.legend(loc="best")
    plt.xlabel(r"$t$")
    plt.ylabel(r"$\Theta(t)$")
    plt.title("Period P = "+str(s)+" [s]")
    #save image in pdf
    #plt.savefig("Theta.pdf")
    plt.show()
```

```python
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
interact(plot_Theta,w=fixed(w),delta=delta_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.plot_Theta>

As expected, since the speed $\omega$ in this case is $\omega(\theta) = 1$, a constant value. The position $\Theta$ in time is the function $\Theta(t) = t$, the identity function. If a different scalar value was used, the solution of the Initial Value Problem would be a multiple of the identity function by exactly this scalar value. The numerical method used in this case is appropiated since reflects the behavior of the solution. The period $P$ in this case corresponds to the value $P = 2\pi \approx 6.28312$.

The next step is to solve the Partial Differential Equation asociated to $A_0(s,T)$:

$$
\frac{\partial A_0}{\partial T}(s,T) = \frac{1}{P} \int_{0}^{P} \frac{1}{\omega^2(\Theta(\tau-s))} \mathrm{d}\tau \frac{\partial^2A_0}{\partial s^2}(s,T)
$$

The next code solves the equation, mixing a numerical method for the integral and a finite difference method for the PDE.

```python
i_widget = IntSlider(min=0, max=499, value=0)
eps_widget = Dropdown(options=[100.,10.,2.,1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17,0.],
                      value=0, description='eps')
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
interact(print_simulation_a0,w=fixed(w),F=fixed(F),i=i_widget,eps=eps_widget,delta=delta_widget,num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.print_simulation_a0>

As can be seen, the solution is the usual for a diffusion equation. The difference with traditional approaches is that the horizontal axis corresponds to a time variable that measures the delay to be applied to the function $\Theta(t)$ such that $\theta = 0$. Recalling that the $s$ that corresponds to a certain $\hat{\theta}$ is given by $s = \tau - \alpha$ with $\alpha$ such that $\Theta(\alpha) = \hat{\theta}$, for some values of $\tau$ the value of $s$ needed is negative. However, the variable is periodic on the interval $[0,P]$ and applying the modulo operator, the $s$ appears in the plot nearer of $P$ for the right side. For example, consider $T = \tau = 0$ and the displacement $\theta = \pi/2$. The value of $\alpha$ such that $\Theta(\alpha) = \pi/2$ is $\alpha = \pi/2$ and the value of $s$ in this case is $s = 0 - \pi/2 = -\pi/2$. Using the periodicity of variable $s$ the value delivered to make next calculations is $s = (-\pi/2)\%2\pi = 3\pi/2$. The expected displacement of the advection process is realized because with a fixed value of displacement $\theta$ in different moments of time, the value of $s$ obtained is different and moves around the values $[0,P]$.

The next code shows the solution $u_0(\theta,\tau,T)$ obtained with the method of multiscales and expresed in function of the slow scale: The last part is really important, because there are not a direct implication of the $\varepsilon$ parameter in this scale. With different values of $\varepsilon$, the caused effect is that the diffusion ocurrs more faster or slower in the fast scale; that is the expected effect of the coefficient of diffusion.

```python
i_widget = IntSlider(min=0, max=500, value=0)
eps_widget = Dropdown(options=[100.,10.,2.,1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17],
                      value=1, description='eps')
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
interact(print_simulation_u0,w=fixed(w),F=fixed(F),i=i_widget,eps=eps_widget,delta=delta_widget,num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.print_simulation_u0>

The final experiment for this numerical example considers the comparation of the numerical solution obtained with one pure method numerically and the solution obtained using the method of multiscales. For that, is necesary to use equivalent times for the pure methods to the results in the slow scale. This times will be changed each time that a different value of $\varepsilon$ be selected.

Consider a given $T_{max}$ in the slow time scale. For simplicity, $T_{max} = 1$ and $\varepsilon$ will take the values $1 \times 10^{\,p}$ with negative values for $p$. Also, a few values of $\varepsilon > 1$ will be taken into account, but it is not the main porpouse of this work. For example, let be $\varepsilon = 10^{-5}$. Since $T_{max}= 1$ units of time in the slow scale, the equvalent value in the fast scale with this $\varepsilon$ is $t_{max} = 1/\varepsilon = 10^{5}$ units of time. With this huge number and a reasonable low value for time step a really large number of iterations (and memory if is necessary to store all the values) is necessary to do. But, with the multiscales is possible to realize few iterations and advance "big" steps of time without recuring to a large time step or a using too much memory.

The next code shows the comparison between the method of multiscales and a traditional approach in order to determinate the numerical values of the advection-diffusion problem where the diffusion occurs very slowly.

```python
def comparison(w,wp,F,i,eps,T_max,N,M,method=1,delta=1,num=1):
    th,u0,d_T = get_u0(w,F,i,eps,T_max,N,M,delta,num)

    t_max = T_max/eps
    M2 = int(t_max/d_T + 1)

    label_method = ""
    u1 = np.zeros((N,M))
    if method == 1:
        u1 = fd_method(w,wp,F,N,M2,t_max,eps,delta,num)
        label_method = "Implicit FD"
    elif method == 2:
        u1 = fd_method(w,wp,F,N,M2,t_max,eps,delta,num,"Explicit")
        label_method = "Explicit FD"
    elif method == 3:
        u1 = fft_method(w,wp,F,N,M2,t_max,eps,delta,num)
        label_method = "FFT"

    j = int(i/eps)

    print("Points Fast Time:",M)
    print("Points Slow Time:",M2)
    print("i",i)
    print("j",j)

    plt.figure(figsize=(13,5))

    plt.subplot(121)
    plt.plot(th,u0,"rs",ms=10,label="Multiscales")
    plt.plot(th,u1[:,j],"gD",ms=4,label=label_method)
    plt.title(r"Simulation at $T = "+str(d_T*i)+"$, $t="+str(d_T*i/eps)+"$")
    plt.ylim(-4,4)
    plt.grid()
    plt.legend(loc="best")

    plt.subplot(122)
    plt.plot(th,u0,"rs",ms=10,label="Multiscales")
    plt.plot(th,u1[:,j],"gD",ms=4,label=label_method)
    plt.title(r"Simulation at $T = "+str(d_T*i)+"$, $t="+str(d_T*i/eps)+"$")
    #plt.ylim(-4,4)
    plt.grid()
    plt.legend(loc="best")

```

```python
i_widget = IntSlider(min=0, max=79, value=0)
N_widget = IntSlider(min=2, max=200, value=50)
M_widget = IntSlider(min=2, max=400, value=80)
eps_widget = Dropdown(options=[10,2,1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17],
                      value=1, description='eps')
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],value=1, description='Function F')
time_widget = FloatSlider(min=0.1,max=10,step=0.1,value=1,description="Max Time")
method_widget=Dropdown(options={"Implicit FD":1, "Explicit FD":2, "FFT":3},value=1)
def update_i_values(*args):
    i_widget.max = M_widget.value-1
M_widget.observe(update_i_values, 'value')
interact(comparison,
         w=fixed(w),
         wp=fixed(wp),
         F=fixed(F),
         i=i_widget,
         eps=eps_widget,
         T_max=time_widget,
         N=N_widget,
         M=M_widget,
         method=method_widget,
         delta=delta_widget,
         num=fun_widget)
```

It can be seen that the result are very near when there are not a big difference between scales. However, when the time starts to be too much large to the traditional computation, the method of multiscales starts to show differences in the numerical simulatons obtained.

```python

```

**Numerical Example 2**
The second numerical example consider a variable speed $\omega$ that depends of the current position $\theta$. A positive parameter $\delta$ will be empoyed in order to avoid having positions of $\theta$ without speed.
\begin{align}
\omega(\theta) &= \sin^2{\left(\frac{\theta}{2}\right)} +\delta\\
\omega'(\theta) &= \frac{1}{2}\sin{(\theta)}\\
\end{align}

```python
def w2(theta,delta=1):
    return (np.sin(0.5*theta))**2+delta

def wp2(theta):
    return 0.5*np.sin(theta)
```

```python
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
interact(plot_functions,w=fixed(w2),wp=fixed(wp2),F=fixed(F),delta=delta_widget,num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.plot_functions>

The delta parameter only desplaces the speed $\omega$ in the vertical axis. This causes that with a greater delta, the speed of the solutions will be greater too.

Now, with the same numerical methods used in previous example, some analysis of the solutions in cases where the speed is not constant will be realized. Consider the next code that uses the functions $\omega(\theta)$ and $w'(\theta)$.

```python
i_widget = IntSlider(min=0, max=79, value=0)
N_widget = IntSlider(min=2, max=200, value=50)
M_widget = IntSlider(min=2, max=400, value=80)
eps_widget = Dropdown(options=[100.,10.,2.,1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17,0.],
                      value=0, description='eps')
delta_widget = FloatSlider(min=0.1,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
fd_widget = Dropdown(options =  ["Explicit", "Implicit"], value="Implicit", description = "FD Mode")
time_widget = IntSlider(min=1,max=10,step=1,value=1,description="Max Time")
def update_i_values(*args):
    i_widget.max = M_widget.value-1
M_widget.observe(update_i_values, 'value')

interact(simulation,
         w=fixed(w2),
         wp=fixed(wp2),
         F=fixed(F),
         i=i_widget,
         N=N_widget,
         M=M_widget,
         eps=eps_widget,
         t_max = time_widget,
         fd_mode=fd_widget,
         delta=delta_widget,
         num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.simulation>

The first case with $\varepsilon = 0$, where only there is a advection process is different when the speed is not a constant value. The maximum value is increasing in time where the solutions are pushed. When time is more than 1 second the maximum value of the solutions decreases in time.
- FFT apparently has no problems of stability, but with the increment of the max time to 10 seconds the problem arises.
- Implicit FD does not have problems of stability.
- Explicit FD has the same problems of stability that FFT, but the problems starts later in this case.

The second case $\varepsilon > 0$, shows the expected diffusion with some advection that is not been well observed as the order of magnitude of $\varepsilon$ increases.
- FFT has problems of stability since the $\varepsilon$ parameter has an efect in the moment of time when this problem arises. As greater this value is, sooner is the moment of inestability.
- Explicit FD does not have problem of stability.
- Implicit FD has similar problems that FFT and also, the $\varepsilon$ parameter has an influence in the stability of the method, but FFT seems to be more inestable that Implicit FD.

The third case, and interest of this work is $0 < \varepsilon \ll 1$.
- FFT does not have stability problems with the default parameters, but with an increment of max time for the simulation great oscilations can be seen at 3 seconds. In order to observate the different phenomena it is necessary to increase as much as possible the number of points $M$ and keep $N$ low. With a considerable value of max time is necesarry to keep as low as possible the time steps. With an order of magnitude lower than $10^{-2}$ the behaviour of the solution starts to seems similar to the first case when $\varepsilon = 0$.
- Implicit FD does not have stability problems and in this cases it is possible to observe the whole view of the different functions with different parameters. It can be seen that the variable speed causes thatthe maximum values in time oscillates, but with a inclination to decrease in time. With an order of magnitude lower than $10^{-2}$ the behaviour of the solution starts to seems similar to the first case when $\varepsilon = 0$.
- Explicit FD has problems of stability but it seems to be more stable that FFT. The behaviour is in general the same that the explained en previous cases. With an order of magnitude lower than $10^{-2}$ the behaviour of the solution starts to seems similar to the first case when $\varepsilon = 0$.

In general, the methods does not let to visualize what is happening with low orders of magnitude for the diffusion coefficient (lower to $10^{-2}$). So, again, with using the multiscales of time the behaviour of this situation could be observed.

The next plot shows the displacement of a certain value in order of time. In a different form to the example with constant value for speed, in this case is expected that the dispacement in time be non-linear.

```python
delta_widget = FloatSlider(min=0.1,max=2.0, step=0.1, value=1.0)
interact(plot_Theta,w=fixed(w2),delta=delta_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.plot_Theta>

In this case, it can be seen that the role of $\delta$ parameter is important. If $\delta$ was zero, then for certain values of positions could not be movement, since $\omega(\theta) = 0$ for some $\theta$. This could causes an infinity value for the period. It can be seen that with $\delta = 0.1$ the period is equal to 19 seconds approx, and with increments of this parameter the period quickly reduce its value.

The displacement is not a constant, such as in the first numerical example, and seems to be like a curves that increases its value in time. A important thing to control with $\delta$ parameter is that if this value is too high, then the displacement is like a stright line and start to seems like the first numerical example. So is important to mantaint it bounded.

The period obtain in this case is not simple to calculate. In first numerical case, period was a multiple of $2\pi$ because, changes in $\delta$ only changes the inclination of the straight line that defines the displacement. Using the same value of delta, the period obtain in this cases is lower that the linear case. The variable speed helps to move faster that with a constant value.

Next, the function $A_0$ calculates from the diffusion equation is shown below.

```python
i_widget = IntSlider(min=0, max=499, value=0)
eps_widget = Dropdown(options=[100.,10.,2.,1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17,0.],
                      value=0, description='eps')
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
interact(print_simulation_a0,w=fixed(w2),F=fixed(F),i=i_widget,eps=eps_widget,delta=delta_widget,num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.print_simulation_a0>

It can be seen that in this case the shape of the function is a little bit different for function 1. Functions 2 and 3 appears to be similar to the linear speed case, but the amplitude in the initial value is different. In order of advances of time in slow scale it can be seen a regular behaviour that expreses the diffusion process.

Now, with that result, the function $u_0$ can be reconstructed using the multiscale. The following code shows the result of computing $u_0$ using the previous values for $A_0$.

```python
i_widget = IntSlider(min=0, max=499, value=0)
eps_widget = Dropdown(options=[100.,10.,2.,1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17],
                      value=1, description='eps')
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],
                      value=1, description='Function F')
interact(print_simulation_u0,w=fixed(w2),F=fixed(F),i=i_widget,eps=eps_widget,delta=delta_widget,num=fun_widget)
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

    <function __main__.print_simulation_u0>

It can be seen that the functions are reconstructed in a good way and shows similar behaviour that the Implicit Finite Difference (more stable). It can be seen clearly that there is two process involved, the advection that produces movement of the values around the spatial axis and diffusion that shows that the values tends to spread.

Finally, the comparison between a traditional approach with one scale and the approximations with multiscales are shown next.

```python
i_widget = IntSlider(min=0, max=79, value=0)
N_widget = IntSlider(min=2, max=200, value=50)
M_widget = IntSlider(min=2, max=400, value=80)
eps_widget = Dropdown(options=[1.,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,
                               1e-8,1e-9,1e-10,1e-11,1e-12,1e-13,1e-14,1e-15,1e-16,1e-17],
                      value=1, description='eps')
delta_widget = FloatSlider(min=0.4,max=2.0, step=0.1, value=1.0)
fun_widget = Dropdown(options=[1,2,3],value=1, description='Function F')
time_widget = FloatSlider(min=0.1,max=10,step=0.1,value=1,description="Max Time")
method_widget=Dropdown(options={"Implicit FD":1, "Explicit FD":2, "FFT":3},value=1)
def update_i_values(*args):
    i_widget.max = M_widget.value-1
M_widget.observe(update_i_values, 'value')
interact(comparison,
         w=fixed(w2),
         wp=fixed(wp2),
         F=fixed(F),
         i=i_widget,
         eps=eps_widget,
         T_max=time_widget,
         N=N_widget,
         M=M_widget,
         method=method_widget,
         delta=delta_widget,
         num=fun_widget)
```

In this case, it seems to be more differences between both approximations. The presence of a non-constant speed seems to affect the results of the approximations with high orders of magintude for $\varepsilon$. The first function $F(\theta)$ has the major differences between methods.

## Conclusions

In this work, a method that uses different scales in order to obtain an approximation asymtotic at $T = \mathcal{O}(1)$ was studied and developed with numerical examples.

It is important to consider the effect of defining the _slow scale_ $T$, since with this scale is possible to avoid secular terms that ocurs in the _fast scale_.

When a numerical method using a unique _fast scale_ was compared with the multiscale, several differences appear as time is increasing. Also, using a slow scale is less expensive that fast scale when $\varepsilon$ is too small and the effect of diffusion in time is too long.

## References
[1] Jupyter Notebooks of SC1. Claudio Torres. https://github.com/tclaudioe/Scientific-Computing/tree/master/SC1.

[2] Jupyter Notebooks of SC2. Claudio Torres. https://github.com/tclaudioe/Scientific-Computing/tree/master/SC2.

[3] Jupyter Notebooks of SC5. Claudio Torres. https://github.com/tclaudioe/Scientific-Computing/tree/master/SC5.

[4] Using observe to update values in a widget. http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html#Arguments-that-are-dependent-on-each-other

**Pendiente**
- poner datos informativos
- algunos 3D
- comparison en el mismo tiempo al cambiar el M

```python

```