# Mutual Information Estimation

$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}$

```python
import ipywidgets as widgets
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from IPython import display
from torch import nn, Tensor
from torch.nn import functional as F

%matplotlib inline
SEED = 0
```

## Problem formulations

**How to formulate the problem of mutual information estimation?**

The problem of estimating the mutual information is:

---
Given $n$ samples

$$(\R{X}_1,\R{Y}_1),\dots, (\R{X}_n,\R{Y}_n) \stackrel{iid}{\sim} P_{\R{X},\R{Y}}\in \mc{P}(\mc{X},\mc{Y})$$

i.i.d. drawn from an *unknown* probability measure $P_{\R{X},\R{Y}}$ from the space $\mc{X}\times \mc{Y}$, estimate the *mutual information (MI)*

$$
\begin{align}
I(\R{X}\wedge\R{Y}) &:= E\left[\log \frac{d P_{\R{X},\R{Y}}(\R{X},\R{Y})}{d (P_{\R{X}} \times P_{\R{Y}})(\R{X},\R{Y})} \right].
\end{align}
$$ (MI)

---

Run the following code, which uses `numpy` to
- generate i.i.d. samples from a multivariate gaussian distribution, and
- store the samples as numpy arrays assigned to `XY`.

```python
import numpy as np

# Seeded random number generator for reproducibility
XY_rng = np.random.default_rng(SEED)

# Sampling from an unknown probability measure
rho = 1 - 0.19 * XY_rng.random()
mean, cov, n = [0, 0], [[1, rho], [rho, 1]], 1000
XY = XY_rng.multivariate_normal(mean, cov, n)
plt.scatter(XY[:, 0], XY[:, 1], s=2)
```

You may find more details below:
- https://numpy.org/doc/stable/reference/random/generated/numpy.random.multivariate_normal.html
- https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.scatter.html

You can also use
- `XY_rng?` to see the doctring such of `XY_rng`, and
- `dir(XY_rng)` to see the available methods/properties `XY_rng.*`.

```python
XY_rng?
dir(XY_rng)
```

    ['__class__',
     '__delattr__',
     '__dir__',
     '__doc__',
     '__eq__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__getstate__',
     '__gt__',
     '__hash__',
     '__init__',
     '__init_subclass__',
     '__le__',
     '__lt__',
     '__ne__',
     '__new__',
     '__pyx_vtable__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__setattr__',
     '__setstate__',
     '__sizeof__',
     '__str__',
     '__subclasshook__',
     '_bit_generator',
     '_poisson_lam_max',
     'beta',
     'binomial',
     'bit_generator',
     'bytes',
     'chisquare',
     'choice',
     'dirichlet',
     'exponential',
     'f',
     'gamma',
     'geometric',
     'gumbel',
     'hypergeometric',
     'integers',
     'laplace',
     'logistic',
     'lognormal',
     'logseries',
     'multinomial',
     'multivariate_hypergeometric',
     'multivariate_normal',
     'negative_binomial',
     'noncentral_chisquare',
     'noncentral_f',
     'normal',
     'pareto',
     'permutation',
     'poisson',
     'power',
     'random',
     'rayleigh',
     'shuffle',
     'standard_cauchy',
     'standard_exponential',
     'standard_gamma',
     'standard_normal',
     'standard_t',
     'triangular',
     'uniform',
     'vonmises',
     'wald',
     'weibull',
     'zipf']

    Type:        Generator
    String form: Generator(PCG64)
    File:        /opt/conda/lib/python3.8/site-packages/numpy/random/_generator.cpython-38-x86_64-linux-gnu.so
    Docstring:
    Generator(bit_generator)

    Container for the BitGenerators.

    ``Generator`` exposes a number of methods for generating random
    numbers drawn from a variety of probability distributions. In addition to
    the distribution-specific arguments, each method takes a keyword argument
    `size` that defaults to ``None``. If `size` is ``None``, then a single
    value is generated and returned. If `size` is an integer, then a 1-D
    array filled with generated values is returned. If `size` is a tuple,
    then an array with that shape is filled and returned.

    The function :func:`numpy.random.default_rng` will instantiate
    a `Generator` with numpy's default `BitGenerator`.

    **No Compatibility Guarantee**

    ``Generator`` does not provide a version compatibility guarantee. In
    particular, as better algorithms evolve the bit stream may change.

    Parameters
    ----------
    bit_generator : BitGenerator
        BitGenerator to use as the core generator.

    Notes
    -----
    The Python stdlib module `random` contains pseudo-random number generator
    with a number of methods that are similar to the ones available in
    ``Generator``. It uses Mersenne Twister, and this bit generator can
    be accessed using ``MT19937``. ``Generator``, besides being
    NumPy-aware, has the advantage that it provides a much larger number
    of probability distributions to choose from.

    Examples
    --------
    >>> from numpy.random import Generator, PCG64
    >>> rg = Generator(PCG64())
    >>> rg.standard_normal()
    -0.203  # random

    See Also
    --------
    default_rng : Recommended constructor for `Generator`.

---

**Exercise** What is unknown about the above sampling distribution?

The density is

$$
\frac{d P_{\R{X},\R{Y}}}{dxdy} = \mc{N}_{\M{0},\left[\begin{smallmatrix}1 & \rho \\ \rho & 1\end{smallmatrix}\right]}(x,y)
$$

but $\rho$ is unknown (uniformly random over $[0.8,0.99)$).

---

To show the data samples using `pandas`:

```python
import pandas as pd

XY_df = pd.DataFrame(XY, columns=["X", "Y"])
XY_df
```

      X
      Y

      0
      -0.029492
      0.285584

      1
      0.030093
      -0.233446

      2
      -0.671256
      -0.029712

      3
      -0.744867
      -1.091092

      4
      1.379858
      1.073218

      ...
      ...
      ...

      995
      -1.740407
      -0.884540

      996
      -0.370652
      0.244189

      997
      -0.590843
      -0.864294

      998
      2.180489
      1.733067

      999
      -0.461016
      -0.254751

1000 rows × 2 columns

To plot the data using `seaborn`:

```python
def plot_samples_with_kde(df, **kwargs):
    p = sns.PairGrid(df, **kwargs)
    p.map_lower(sns.scatterplot, s=2)  # scatter plot of samples
    p.map_upper(sns.kdeplot)  # kernel density estimate for pXY
    p.map_diag(sns.kdeplot)  # kde for pX and pY
    return p

plot_samples_with_kde(XY_df)
plt.show()
```

**Exercise** Complete the following code by replacing the blanks `___` so that `XY_ref` stores the i.i.d. samples of $(\R{X}',\R{Y}')$ where $\R{X}'$ and $\R{Y}'$ are zero-mean independent gaussian random variables with unit variance.

```Python
...
cov_ref, n_ = ___, n
XY_ref = XY_ref_rng_ref.___(mean, ___, n_)
...
```

```python
XY_ref_rng = np.random.default_rng(SEED)
### BEGIN SOLUTION
cov_ref, n_ = [[1, 0], [0, 1]], n
XY_ref = XY_ref_rng.multivariate_normal(mean, cov_ref, n_)
### END SOLUTION
XY_ref_df = pd.DataFrame(XY_ref, columns=["X", "Y"])
plot_samples_with_kde(XY_ref_df)
plt.show()
```

**Can we generalize the problem further?**

Estimating MI may be viewed as a special case of the following problem:

---

Estimate the KL *divergence*

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) &:= E\left[\log \frac{d P_{\R{Z}}(\R{Z})}{d P_{\R{Z}'}(\R{Z})} \right]
\end{align}
$$ (D)

using
- a sequence $\R{Z}^n:=(\R{Z}_1,\dots, \R{Z}_n)\sim P_{\R{Z}}^n$ of i.i.d. samples from $P_{\R{Z}}$ if $P_{\R{Z}}$ is unknown, and
- another sequence ${\R{Z}'}^{n'}\sim P_{\R{Z}'}^{n'}$ of i.i.d. samples from $P_{\R{Z}'}$  if $P_{\R{Z}'}$, the *reference measure* of $P_{\R{Z}}$, is also unknown.

---

---
**Exercise**

Although $\R{X}^n$ and $\R{Y}^n$ for MI estimation should have the same length, $\R{Z}^n$ and ${\R{Z}'}^{n'}$ can have different lengths, i.e., $n \not\equiv n'$. Why?

**Solution** The dependency between $\R{Z}$ and $\R{Z}'$ does not affect the divergence.

---

Regarding the mutual information as a divergence from joint to product distributions, the problem can be further generalized to estimtate other divergences such as the $f$-divergence:

For a strictly convex function $f$ with $f(1)=0$,

$$
\begin{align}
D_f(P_{\R{Z}}\|P_{\R{Z}'}) &:= E\left[ f\left(\frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')}\right) \right].
\end{align}
$$ (f-D)

$f$-divergence in {eq}`f-D` reduces to KL divergence when $f=u \log u$:

$$
\begin{align}
E\left[ \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')} \log \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')}  \right] &= \int_{\mc{Z}} \color{gray}{d P_{\R{Z}'}(z)} \cdot \frac{d P_{\R{Z}}(z)}{\color{gray}{d P_{\R{Z}'}(z)}} \log \frac{d P_{\R{Z}}(z)}{d P_{\R{Z}'}(z)}.
\end{align}
$$

---

**Exercise**

Show that $D_f(P_{\R{Z}}\|P_{\R{Z}'})\geq 0$ with equality iff $P_{\R{Z}}=P_{\R{Z}'}$ using Jensen's inequality and the properties of $f$.

**Solution**

It is a valid divergence because, by Jensen's inequality,

$$
D_f(P_{\R{Z}}\|P_{\R{Z}'}) \geq  f\bigg( \underbrace{E\left[ \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')} \right]}_{=1}\bigg) = 0
$$

with equality iff $P_{\R{Z}}=P_{\R{Z}'}$.

---

Regarding the divergence as an expectation, it is approximated by the sample average:

$$
\begin{align}
D_f(P_{\R{Z}}\|P_{\R{Z}'}) &\approx
\frac1n \sum_{i\in [n]} f\left(\frac{d P_{\R{Z}}(\R{Z}'_i)}{d P_{\R{Z}'}(\R{Z}'_i)}\right).
\end{align}
$$ (avg-f-D)

However, this is not a valid estimate because it involves the unknown measures $P_{\R{Z}}$ and $P_{\R{Z}'}$.

One may further estimate the *density ratio*

$$
\begin{align}
z \mapsto \frac{d P_{\R{Z}}(z)}{d P_{\R{Z}'}(z)}
\end{align}
$$ (dP-ratio)

or estimate the density defined respective to some reference measure $\mu$:

$$
\begin{align}
p_{\R{Z}}&:=\frac{dP_{\R{Z}}}{d\mu} \in \mc{P}_{\mu}(\mc{Z}).
\end{align}
$$ (density)

## Neural estimation of KL divergence

Estimating MI well neither require nor imply the divergence/density to be estimated well. However,
- MI estimation is often not the end goal, but an objective to train a neural network to return the divergence/density.
- The features/representations learned by the neural network may be applicable to different downstream inference tasks.

To explain the idea of neural estimation, consider the following characterization of divergence:

---
**Proposition**

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) & =  \sup_{Q\in \mc{P}(\mc{Z})} E \left[ \log \frac{dQ(\R{Z})}{dP_{\R{Z}'}(\R{Z})} \right]
\end{align}
$$ (D1)

where the unique optimal solution is $Q=P_{\R{Z}}$.

---

{eq}`D1` is {eq}`D` but with $P_{\R{Z}}$ replaced by a parameter $Q$.

- The proposition essentially gives a tight lower bound on KL divergence.
- The unknown distribution is recovered as the optimal solution.

---

**Proof**

To prove {eq}`D1`,

$$
\begin{align*}
D(P_{\R{Z}}\|P_{\R{Z}'})  &= D(P_{\R{Z}}\|P_{\R{Z}'}) - \inf_{Q\in \mc{P}(\mc{Z})} \underbrace{D(P_{\R{Z}}\|Q)}_{\geq 0 \text{ with equality iff } Q=P_{\R{Z}}\kern-3em} \\
&= \sup_{Q\in \mc{P}(\mc{Z})}  \underbrace{D(P_{\R{Z}}\|P_{\R{Z}'})}_{=E \left[\frac{dP_{\R{Z}}(\R{Z})}{dP_{\R{Z}'}(\R{Z})}\right]} -  \underbrace{D(P_{\R{Z}}\|Q)}_{=E \left[\frac{dP_{\R{Z}}(\R{Z})}{dQ(\R{Z})}\right]}\\
&= \sup_{Q\in \mc{P}(\mc{Z})} E \left[\frac{dQ(\R{Z})}{dP_{\R{Z}'}(\R{Z})}\right]
\end{align*}
$$

---

The idea of neural estimation is to

- estimate the expectation in {eq}`D1` by the sample average

$$
\frac1n \sum_{i\in [n]} \log \underbrace{\frac{dQ(\R{Z}_i)}{dP_{\R{Z}'}(\R{Z}_i)}}_{\text{(*)}},
$$

- use a neural network to compute the density ratio (*), and train the network to maximizes the expectation, e.g., by gradient ascent on the above sample average.

Since $Q$ is arbitrary, the sample average above is a valid estimate.

**But how to compute the density ratio?**

We will first consider estimating the KL divergence $D(P_{\R{Z}}\|P_{\R{Z}'})$ when both $P_{\R{Z}}$ and $P_{\R{Z}'}$ are unknown.

If $P_{\R{Z}'}$ is unknown, we can apply a change of variable

$$
r(z) = \frac{dQ(z)}{dP_{\R{Z}'}(z)},
$$ (Q->r)

which absorbs the unknown reference into the parameter.

---
**Proposition**

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) & =  \sup_{\substack{r:\mc{Z}\to \mathbb{R}_+\\ E[r(\R{Z}')]=1}} E \left[ \log r(\R{Z}) \right]
\end{align}
$$ (D1)

where the optimal $r$ satisfies
$
r(\R{Z}) = \frac{dP_{\R{Z}}(\R{Z})}{dP_{\R{Z}'}(\R{Z})}.
$

---

---

**Exercise** Show using {eq}`Q->r` that the optimal solution satisfies the constraint stated in the supremum {eq}`D1`.

The constraint on $r$ is obtained from the constraint on $Q\in \mc{P}(\mc{Z})$, i.e., with $dQ(z)=r(z)dP_{\R{Z}'}(z)$,

$$
\begin{align*}
dQ(z) \geq 0 &\iff r(z)\geq 0\\
\int_{\mc{Z}}dQ(z)=1 &\iff E[r(\R{Z}')]=1.
\end{align*}
$$

---

The next step is to train a neural network that computes $r$. What about?

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) \approx \sup_{\substack{r:\mc{Z}\to \mathbb{R}_+\\ \frac1{n'}\sum_{i\in [n']} r(\R{Z}'_i)]=1}} \frac1n \sum_{i\in [n]} \log r(\R{Z}_i)
\end{align}
$$ (avg-D1)

**How to impose the constraint on $r$ when training a neural network?**

We can apply by a change of variable:

$$
\begin{align}
r(z)&=\frac{e^{t(z)}}{E[e^{t(\R{Z}')}]}.
\end{align}
$$ (r->t)

**Exercise** Show that $r$ defined in {eq}`r->t` satisfies the constraint in {eq}`D1` for all real-valued function $t:\mc{Z}\to \mathbb{R}$.

**Proof**

$$
\begin{align}
E\left[ \frac{e^{t(\R{Z}')}}{E[e^{t(\R{Z}')}]} \right] =  \frac{E\left[ e^{t(\R{Z}')} \right]}{E[e^{t(\R{Z}')}]} = 1.
\end{align}
$$

Substituting {eq}`r->t` into {eq}`D1` gives the well-known *Donsker-Varadhan (DV)* formula:

---
**Corollary** [{cite}`donsker1983asymptotic`]

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) =  \sup_{t: \mc{Z} \to \mathbb{R}} E[t(\R{Z})] - \log E[e^{t(\R{Z}')}]
\end{align}
$$ (DV)

where the optimal $t$ satisfies
$$
\begin{align}
t(\R{Z}) = \log \frac{dP_{\R{Z}}(\R{Z})}{dP_{\R{Z}'}(\R{Z})} + c
\end{align}
$$ (DV:sol)

almost surely for some constant $c$.

---

The divergence can be estimated as follows instead of {eq}`avg-D1`:

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) \approx \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t(\R{Z}_i) - \frac1{n'}\sum_{i\in [n']} e^{t(\R{Z}'_i)}
\end{align}
$$ (avg-DV)

In summary, the neural estimation of KL divergence is a sample average of {eq}`D` but with the unknown density ratio replaced by {eq}`r->t` trained as a neural network to maximize the estimate.

$$
D(P_{\R{Z}}\| P_{\R{Z}'}) = \underbrace{}_{\sup_{t}}\underbrace{E}_{\op{avg}} \bigg[ \log \underbrace{\frac{P_{\R{Z}}(\R{Z})}{P_{\R{Z}'}(\R{Z})}}_{\frac{e^{t(\R{Z})}}{\underbrace{E}_{\op{avg}}[e^{t(\R{Z}')}]}} \bigg].
$$

## Training a neural network

We will train a neural network with `torch` and use GPU if available:

```python
import torch

# use GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"

if device == "cuda":  # print current GPU name if available
    print("Using GPU:", torch.cuda.get_device_name(torch.cuda.current_device()))
```

    Using GPU: Tesla T4

- https://pytorch.org/docs/stable/notes/randomness.html
- https://pytorch.org/docs/stable/cuda.html?highlight=cuda#module-torch.cuda

When GPU is available, you can use GPU DASHBOARDS on the left to monitor GPU utilizations.

The following code defines a simple neural networks with 3 fully-connected (fc) hidden layers:

where

- $\M{W}_{\ell}$ and $\M{b}_{\ell}$ are the weight and bias respectively for the linear transformation $\R{W}_l a_l + b_l$ of the $l$-th layer; and
- $\sigma$ for the first 2 hidden layers is an activation function chosen to be the [*exponential linear unit (ELU)*](https://pytorch.org/docs/stable/generated/torch.nn.ELU.html).

```python
from torch import nn
from torch.nn import functional as F

class Net(nn.Module):
    def __init__(self, input_size=2, hidden_size=100, sigma=0.02):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)  # fully-connected (fc) layer
        self.fc2 = nn.Linear(hidden_size, hidden_size)  # layer 2
        self.fc3 = nn.Linear(hidden_size, 1)  # layer 3
        nn.init.normal_(self.fc1.weight, std=sigma)  #
        nn.init.constant_(self.fc1.bias, 0)
        nn.init.normal_(self.fc2.weight, std=sigma)
        nn.init.constant_(self.fc2.bias, 0)
        nn.init.normal_(self.fc3.weight, std=sigma)
        nn.init.constant_(self.fc3.bias, 0)

    def forward(self, z):
        a1 = F.elu(self.fc1(z))
        a2 = F.elu(self.fc2(a1))
        t = self.fc3(a2)
        return t

torch.manual_seed(SEED)  # seed RNG for PyTorch
net = Net()
net.to(device)
print(net)
```

    Net(
      (fc1): Linear(in_features=2, out_features=100, bias=True)
      (fc2): Linear(in_features=100, out_features=100, bias=True)
      (fc3): Linear(in_features=100, out_features=1, bias=True)
    )

- https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#define-the-network

`Net` is a subclass of `nn.Module` with the methods:

- `__init__` that creates/initializes the neural network parameters, which is often organized into layers.
- `forward` that computes an output from a given input using the neural network parameters.

To compute $t(\M{z})$ at $\M{z}=\begin{bmatrix}0\\ 0\end{bmatrix}$:

```python
from torch import Tensor

z = Tensor([[0.0, 0]]).to(device)
net(z)
```

    tensor([[0.]], device='cuda:0', grad_fn=<AddmmBackward>)

The neural network indeed computes a vectorized function. E.g., we can pass in the vector `XY` of $(\R{X},\R{Y})$ samples as input:

```python
tXY = (
    net(Tensor(XY).to(device))  # convert XY to tensor
    .cpu()  # copy back to CPU
    .detach()  # detach from current graph (no gradient calculation)
    .numpy()  # convert output back to numpy
)
```

- https://pytorch.org/docs/stable/generated/torch.no_grad.html

The output needs to be converted back to a `numpy` array on CPU so it can be plotted as follows:

```python
cmap = "RdBu"
ax = sns.scatterplot(data=XY_df, x="X", y="Y", c=tXY, cmap=cmap, s=10)

# add a color bar
norm = plt.Normalize(tXY.min(), tXY.max())
sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)
ax.figure.colorbar(sm)
ax.set_title(r"$e^{t(X,Y)}$")
plt.show()
```

The values are all very close to $0$ as we have set a small variance `sigma=0.02` to initialize the neural network parameters.

To train the neural network, we will use the minibatch *Adam* gradient descend algorithm on a randomly choice batch of the samples.

```python
import torch.optim as optim
resample_rng = np.random.default_rng(SEED) # RNG for minibatch sampling

class DV():
    def __init__(self, Z, Z_ref, batch_size=32, hidden_size=100, lr=1e-3):
        self.Z, self.Z_ref = Z, Z_ref
        self.lr, self.batch_size, setl.hidden_size = lr, batch_size, hidden_size
        self.net = Net(Z.shape[1], input_size=Z.shape[1], hidden_size=hidden_size)
        self.optimizer = optim.Adam(self.net.parameters(), hidden_size=hidden_size, lr=lr)

    def step(self):
        optimizer.zero_grad()

        # obtain a random batch of samples
        batch_Z = Tensor(resample_rng.choice(Z, size=self.batch_size)).to(device)
        batch_Z_ref = Tensor(resample_rng.choice(Z_ref, size=ref_batch_size)).to(device)

        # define loss as negative divergence lower bound from VD formula
        avg_tZ = net(batch_Z).mean()  # (a)
        avg_etZ_ref = net(batch_Z_ref).logsumexp(0) - np.log(batch_Z_ref.shape[0]) # (b) - (c)
        loss = -(avg_tZ - avg_etZ_ref)

        # gradient descent
        loss.backward()  # (d)
        optimizer.step() # (f)

    def DV_bound(self, Z, Z_ref):
        Z = Tensor(resample_rng.choice(Z, size=self.batch_size)).to(device)
        Z_ref = Tensor(resample_rng.choice(Z_ref, size=ref_batch_size)).to(device)

```

Let $\theta$ be the tuple of parameters (weights and biases) of the neural network that computes $t$:

$$
\theta := (\M{W}_l,\M{b}_l|\ell\in [3])
$$

Define the batch loss function as the negative lower bound of the VD formula in {eq}`VD`:

$$
\begin{align}
\R{L}(\theta) := - \bigg[\underbrace{\frac1{\abs{\R{B}}} \sum_{i\in \R{B}} t(\R{Z}_i)}_{\text{(a)}} - \underbrace{\log \frac1{\abs{\R{B}'}} \sum_{i\in \R{B}'} e^{t(\R{Z}'_i)}}_{ \underbrace{\log \sum_{i\in \R{B}'} e^{t(\R{Z}'_i)}}_{\text{(b)}} - \underbrace{\log \abs{\R{B}'}}_{\text{(c)}}} \bigg]
\end{align}
$$

but on the minibatches

$$\R{Z}_{\R{B}}:=(\R{Z}_i\mid i\in \R{B})\quad \text{and}\quad \R{Z}'_{\R{B}'}$$

where $\R{B}$ and $\R{B}'$ uniformly randomly and independently chosen from $[n]$ and $[n']$ respectively.

The method `step` updates the neural network parameters as

$$\theta^{(i+1)} = \underbrace{\theta^{(i)} - s_i \overbrace{\nabla L(\theta^{(i)})}^{\text{(d)}}}_{\text{(f)}} $$

1. by defining the `loss` $L$,
2. calling `loss.backward` to computes the gradient $\nabla L $ w.r.t the neural network parameters $\theta^{(i)}$, and
3. updating the parameter to $\theta^{(i+1)}$ by descending along the negative gradient with a step size `s` dynamically chosen by the Adam's `optimizer`.

```python
estimates = []
```

```python
from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter()
Z = Tensor(XY).to(device)
Z_ref = Tensor(XY_ref).to(device)
```

```python
n_epochs = 1
batch_size = ref_batch_size = 100

for i in range(n_epochs):
    idx = torch.randperm(Z.shape[0])
    for i in range(0, Z.shape[0], batch_size):
        optimizer.zero_grad()
        # obtain a random batch of samples
        batch_Z = Z[idx[i:i+batch_size]]
        batch_Z_ref = Z[idx[i:i+batch_size]]
        # define loss as negative divergence lower bound from VD formula
        avg_tZ = net(batch_Z).mean()  # (a)
        avg_etZ_ref = net(batch_Z_ref).logsumexp(0) - np.log(batch_Z_ref.shape[0]) # (b) - (c)
        loss = -(avg_tZ - avg_etZ_ref)
        # gradient descent
        loss.backward()  # (d)
        optimizer.step() # (f)
        writer.add_scalar('Loss/train', loss.item())
```

```python
%load_ext tensorboard
%tensorboard --logdir=runs
```

    The tensorboard extension is already loaded. To reload it, use:
      %reload_ext tensorboard

    Reusing TensorBoard on port 6006 (pid 37434), started 6:42:15 ago. (Use '!kill 37434' to kill it.)

```python
def step():
    batch_size = ref_batch_size = 100
    optimizer.zero_grad()
    # obtain a random batch of samples
    batch_Z = Tensor(resample_rng.choice(XY, size=batch_size)).to(device)
    batch_Z_ref = Tensor(resample_rng.choice(XY_ref, size=ref_batch_size)).to(device)
    # define loss as negative divergence lower bound from VD formula
    avg_tZ = net(batch_Z).mean()  # (a)
    avg_etZ_ref = net(batch_Z_ref).logsumexp(0) - np.log(batch_Z_ref.shape[0]) # (b) - (c)
    loss = -(avg_tZ - avg_etZ_ref)
    # gradient descent
    loss.backward()  # (d)
    optimizer.step() # (f)

def estimate():
    estimates.append(-loss.item())
```

```python
ax = sns.lineplot(x=range(len(estimates)), y=estimates)
ax.set(xlabel='n', ylabel='estimate', title=r'$D(P_{\mathsf{Z}}||P_{\mathsf{Z}})$ using VD formula')
```

```python
sns.lineplot(x=range(len(estimates)), y=estimates, estimator='mean', err_style='band')
```

```python
df_estimates = pd.DataFrame(estimates, name=')
```

### Classifier-based estimation

### With known reference

Consider the problem of estimating the KL divergence with a known reference, i.e., $P_{\R{Z}}$ is unknown but $P_{\R{Z}'}$ is known.

**But how to use a neural network to compute the density ratio?**

KL divergence is related to the entropy and cross entropy:

$$
\begin{align}
D(P_{\R{Z}}\| P_{\R{Z}'})= \overbrace{\underbrace{E\left[-\log p_{\R{Z}'}(\R{Z})\right]}_{\approx \frac1n \sum_{i\in [n]} \log p_{\R{Z}'}(\R{Z}_i)}}^{\text{cross entropy}} - \overbrace{\underbrace{H(\R{Z})}_{:= E\left[ \log \frac{1}{p_{\R{Z}}(\R{Z})}\right]}}^{\text{entropy}}
\end{align}
$$ (D->H)

Cross entropy can be approximated by the sample average above since $p_{\R{Z}'}$ is known.
In otherwise, KL divergence can be estimated using an entropy estimate.

**How to estimate the entropy?**

Substituting {eq}`D->H` into {eq}`D1` gives:

---

**Corollary**

$$
\begin{align}
H(\R{Z}) = \inf_{q\in \mc{P}_{\mu}(\mc{Z})} E\left[- \log q(\R{Z})\right]
\end{align}
$$ (H1)

where the optimal $q$ satisfies $q(\R{Z})=p_{\R{Z}}(\R{Z})$ in {eq}`density` almost surely.

---

Suppose $\R{Z}$ is discrete. We can apply the following change of variable to {eq}`H1` that

$$
q(z) = \frac{e^{t_z}}{\sum_{z\in \mc{Z}} e^{t_z}} \quad \text{for }z\in \mc{Z},
$$ (q->t_z)

called the *soft-max* layer, and optimize a neural network that takes no input but returns a vector

$$\M{t}:=\begin{bmatrix} t_z\end{bmatrix}_{z\in \mc{Z}} \in \mathbb{R}^{\mc{Z}},$$

called the *logits*.

**What if $\R{Z}$ is continuous?**

We can apply the following change of variable instead

$$
q(z) := \frac{e^{t(z)}}{ \underbrace{\int_{\mc{Z}} e^t\,d\mu}_{c:=} } \quad \text{for }z\in \op{supp}(p_{\R{Z}}):=\Set{z\in \mc{Z}\mid p_{\R{Z}}>0}.
$$ (q->t:1)

which must be in $\mc{P}_{\mu}(\mc{Z})$ for all real-valued function $t:\mc{Z}\to \mathbb{R}$.
In particular, since $e^{t(z)}>0$, $z$ has to be restricted to the support $\op{supp}(p_{\R{Z}})$ of $p_{\R{Z}}$.

The neural network can take input $z$ and returns output $t(z)$.

**But how to compute the normalization factor $c$ in {eq}`q->t:1`?**

We can apply importance sampling.

Consider any distribution $P_{\R{Z}''}\in \mc{P}(\mc{Z})$ such that

$$P_{\R{Z}''}\succeq P_{\R{Z}},$$

which means that any event $\mc{Z}$ is probable w.r.t. $P_{\R{Z}}$ is also probable w.r.t. $P_{\R{Z}''}$, or equivalently,

$$
\op{supp}(p_{\R{Z}''})\supseteq \op{supp}(p_{\R{Z}}).
$$  (Porder)

The desired integral can then be turned into an expectation:

$$
\begin{align}
\int_{\mc{Z}} e^t\,d\mu &= \int_{\mc{Z}} \frac{e^t}{p_{\R{Z}''}} p_{\R{Z}''}\,d\mu \\
& = E\left[ \frac{e^{t(\R{Z}'')}}{p_{\R{Z}''}(\R{Z}'')} \right].
\end{align}
$$ (int->avg)

---
**Proposition**

For any $P_{\R{Z}''}\succeq P_{\R{Z}}$ {eq}`Porder`,

$$
\begin{align}
H(\R{Z}) &=  - \sup_{t: \mc{Z} \to \mathbb{R}} E[t(\R{Z})] + \log E\left[ \frac{e^{t(\R{Z}'')}}{p_{\R{Z}''}(\R{Z}'')} \right]
\end{align}
$$ (H2)

where the optimal $t$ satisfies $t(\R{Z}) = \log p_{\R{Z}}(\R{Z}) + c$
almost surely for some constant $c\in \mathbb{R}$.

---

In summary, applying {eq}`H2` to {eq}`D->H` gives the following divergence estimate:

---
**Corollary**

For any $P_{\R{Z}''}\succeq P_{\R{Z}}$ {eq}`Porder`,

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) &= \underbrace{E\left[-\log p_{\R{Z}'}(\R{Z})\right]}_{\approx \frac{1}{n} \sum_{i\in [n]} -\log p_{\R{Z}'}(\R{Z}_i) } +  \overbrace{\sup_{t: \mc{Z} \to \mathbb{R}} \underbrace{E[t(\R{Z})]}_{\approx \frac1{n} \sum_{i\in [n]} t(\R{Z}_i)} - \log \underbrace{E\left[ \frac{e^{t(\R{Z}'')}}{p_{\R{Z}''}(\R{Z}'')} \right]}_{\color{blue}{\frac1{n''} \sum_{i\in [n'']} \frac{e^{t(\R{Z}''_i)}}{p_{\R{Z}''}(\R{Z}''_i)}}}}^{H(\R{Z})=}
\end{align}
$$ (D2)

for i.i.d. samples $\R{Z}^n$ and ${\R{Z}''}^{n''}$ of $\R{Z}$ and $\R{Z}''$ respectively. The optimal $t$ satisfies $t(\R{Z}) = \log p_{\R{Z}}(\R{Z}) + c$
almost surely for some constant $c\in \mathbb{R}$.

---

Since the divergence does not depend on $P_{\R{Z}''}$, we can further optimize it to reduce the variance of the final sample average highlighted in blue.

---

**Exercise** Show that the optimal choice of $P_{\R{Z}''}$ that minimizes the variance of

$$\frac1{n''} \sum_{i\in [n'']} \frac{e^{t(\R{Z}''_i)}}{p_{\R{Z}''}(\R{Z}''_i)}$$

has $p_{\R{Z}''}=e^t/c$ for some constant $c$.

**Proof**

With $p_{\R{Z}''}=e^t/c$,

$$
\begin{align}
\frac1{n''} \sum_{i\in [n'']} \frac{e^{t(\R{Z}''_i)}}{\underbrace{p_{\R{Z}''}(\R{Z}''_i)}_{=e^{t(\R{Z}'')}/c}} &= \frac1{n''} \sum_{i\in [n'']} c = c
\end{align}
$$

which has $0$ variance since $c$ is a constant. Hence, the variance is minimized as desired.

---

In practice, such a choice is impractical because:

- The constant $c$ used in defining the optimal $p_{\R{Z}''}$ is precisely what needs to be estimated.
- Even if $c$ is known, it is difficult to sample according to the optimal according to the optimal density $e^{t}/c$. E.g., the inverse sampling method requires the cdf inverse, which is intractable for a complex neural network computing $t(z)$.

Altogether, we can apply the following change of variable assuming the non-degenerate case that {eq}`Porder` holds:

$$
q(z) := \frac{e^{t(z)}p_{\R{Z}''}(z)}{ E\left[e^{t(\R{Z}'')}\right]} \quad \text{for }z\in \op{supp}(p_{\R{Z}}).
$$ (q->t)

---
**Proposition**

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) & =  \sup_{Q\in \mc{P}(\mc{Z})} E \left[ \log \frac{dQ(\R{Z})}{dP_{\R{Z}'}(\R{Z})} \right]
\end{align}
$$ (D1)

where the optimal solution is $Q=P_{\R{Z}}$.

---

---

**Exercise** The first equality in {eq}`int->avg` requires the mild assumption {eq}`Porder`. Show that the assumption holds iff $D(P_{\R{Z}}\| P_{\R{Z}''})<\infty$.

**Solution** By definitions {eq}`D` and {eq}`density`,

$$
D(P_{\R{Z}}\|P_{\R{Z}''}) = E\left[ \frac{p_{\R{Z}}(\R{Z})}{p_{\R{Z}''}(\R{Z})} \right]
$$

which is bounded iff $\frac{p_{\R{Z}}(\R{Z})}{p_{\R{Z}''}(\R{Z})}$ is bounded almost surely, i.e., $p_{\R{Z}''}(\R{Z})>0$ almost surely. This happens iff {eq}`Porder` holds.

---

---

**Exercise** How should the neural network compute $q$ if $\R{Z}=(\R{Z}_{\text{d}},\R{Z}_{\text{c}})$ has both a continuous component $\R{Z}_{\text{c}}$ and a discrete component $\R{Z}_{\text{d}}$?

**Solution** The neural network can take the continuous component $z_{\text{c}}$
as input and return the vector $\M{t}(z_{\text{c}}) = \begin{bmatrix} t_{z_{\text{d}}}(z_{\text{c}}) \end{bmatrix}_{z_{\text{d}}}$.

---

---

**Exercise** Indeed, {eq}`q->t_z` is a special case of {eq}`q->t` with a suitable choice of $P_{\R{Z}''}$. How?

**Solution** {eq}`q->t_z` can be obtained from {eq}`q->t` by choosing $P_{\R{Z}''} = \op{Uniform}_{\mc{Z}}$, i.e., the uniform distribution of $\mc{Z}$:

$$
\begin{align*}
\frac{e^{t(z)}\overbrace{p_{\R{Z}''}(z)}^{=1/k}}{ \underbrace{E\left[e^{t(\R{Z}'')}\right]}_{=\frac1k \sum_{i\in [k]} e^{t(z_i)}}} \quad \text{where } k=\abs{Z}.
\end{align*}
$$

---

**How good is the sample average estimate for the normalization factor?**

## MI estimation via KL divergence

**How to estimate MI via KL divergence?**

One way is to obtain MI {eq}`MI` from KL divergence {eq}`D` as follows:

$$
\begin{align*}
I(\R{X}\wedge \R{Y}) = D(\underbrace{P_{\R{X},\R{Y}}}_{P_{\R{Z}}}\| \underbrace{P_{\R{X}}\times P_{\R{Y}}}_{P_{\R{Z}'}}).
\end{align*}
$$

Since both $P_{\R{Z}}$ and $P_{\R{Z}'}$ are unknown, we can apply {eq}`avg-DV` to estimate the divergence.

$$
\begin{align}
I(\R{X}\wedge \R{Y}) \approx \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t(\R{X}_i,\R{Y}_i) - \frac1{n'}\sum_{i\in [n']} e^{t(\R{X}'_i,\R{Y}'_i)}
\end{align}
$$ (MINE)

where $P_{\R{X}',\R{Y}'}:=P_{\R{X}}\times P_{\R{Y}}$. This is the *Mutual Information Neural Estimation (MINE)* proposed by {cite}`belghazi2018mine`.

**But how to obtain the reference samples ${\R{Z}'}^{n'}$, i.e., ${\R{X}'}^{n'}$ and ${\R{Y}'}^{n'}$?**

We can approximate the i.i.d. sampling of $P_{\R{X}}\times P_{\R{Y}}$ using samples from $P_{\R{X},\R{Y}}$ by a re-sampling trick:

$$
\begin{align}
P_{\R{Z}'^{n'}} &\approx P_{((\R{X}_{\R{J}_i},\R{Y}_{\R{K}_i})\mid i \in [n'])}
\end{align}
$$ (resample)

where $\R{J}_i$ and $\R{K}_i$ for $i\in [n']$ are independent and uniformly random indices

$$
P_{\R{J},\R{K}} = \op{Uniform}_{[n]\times [n]}
$$

and $[n]:=\Set{1,\dots,n}$.

MINE {cite}`belghazi2018mine` uses the following implementation that samples $(\R{J},\R{K})$ but without replacement. You can change $n'$ using the slider for `n_`.

```python
def resample(XY, size, replace=False):
    index = rng.choice(range(XY.shape[0]), size=size, replace=replace)
    return XY[index]

@widgets.interact
def plot_resampled_data_without_replacement(n_=(2, n)):
    XY_ = np.block([resample(XY[:, [0]], n_), resample(XY[:, [1]], n_)])
    resampled_data = pd.DataFrame(XY_, columns=["X'", "Y'"])
    p_ = plot_samples_with_kde(resampled_data)
    plt.show()
```

    interactive(children=(IntSlider(value=51, description='n_', min=2), Output()), _dom_classes=('widget-interact'…

**Exercise** To allow $n>n'$, we need to sample the index with replacement. Complete the following code and observe what happens when $n \gg n'$

```python
@widgets.interact
def plot_resampled_data_with_replacement(
    n_=widgets.IntSlider(20 * n, 2, 50 * n, continuous_update=False)
):
    ### BEGIN SOLUTION
    XY_ = np.block(
        [resample(XY[:, [0]], n_, replace=True), resample(XY[:, [1]], n_, replace=True)]
    )
    ### END SOLUTION
    resampled_data = pd.DataFrame(XY_, columns=["X'", "Y'"])
    p_ = plot_samples_with_kde(resampled_data)
    plt.show()
```

    interactive(children=(IntSlider(value=2000, continuous_update=False, description='n_', max=5000, min=2), Outpu…

---

**Exercise** Explain whether the resampling trick gives i.i.d. samples $(\R{X}_{\R{J}_i},\R{Y}_{\R{K}_i})$ for the cases with replacement and without replacement respectively?

**Solution** The samples are identically distributed. However, they are not independent except in the trivial case $n=1$ or $n'=1$, regardless of whether the sample is with replacement or not. Consider $n=1$ and $n'=2$ as an example.

---

**Is it possible to generate i.i.d. samples for ${\R{Z}'}^{n'}$?**

Consider another formula for MI:

---
**Proposition**

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'}) - D(P_{\R{X}}\|P_{\R{X}'}) - D(P_{\R{Y}}\|P_{\R{Y}'})
\end{align}
$$ (MI-E)

for $P_{\R{X}'}\times P_{\R{Y}'}$, called the (product) reference distribution, satisfying {eq}`Porder`
$P_{\R{X}'}\times P_{\R{Y}'} \succeq P_{\R{X},\R{Y}}$ to ensure that the divergences are finite.

---

---

**Proof**

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= H(\R{X}) + H(\R{Y}) - H(\R{X},\R{Y})\\
&= E\left[-\log dP_{\R{X}'}(\R{X})\right] - D(P_{\R{X}}\|P_{\R{X}'})\\
&\quad+E\left[-\log dP_{\R{Y}'}(\R{Y})\right] - D(P_{\R{Y}}\|P_{\R{Y}'})\\
&\quad-E\left[-\log d(P_{\R{X}'}\times P_{\R{Y}'})(\R{X},\R{Y})\right] + D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'})\\
&= D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'}) - D(P_{\R{X}}\|P_{\R{X}'}) - D(P_{\R{Y}}\|P_{\R{Y}'})
\end{align}
$$

---

*Mutual Information Neural Entropic Estimation (MI-NEE)* {cite}`chan2019neural` uses {eq}`MI-E` to estimate MI by estimating the three divergences.

Applying {eq}`avg-DV` to each divergence:

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &\approx \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t_{\R{X},\R{Y}}(\R{X}_i,\R{Y}_i) - \frac1{n'}\sum_{i\in [n']} e^{t_{\R{X},\R{Y}}(\R{X}'_i,\R{Y}'_i)}\\
&\quad - \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t_{\R{X}}(\R{X}_i) - \frac1{n'}\sum_{i\in [n']} e^{t_{\R{X}}(\R{X}'_i)} \\
&\quad - \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t_{\R{Y}}(\R{Y}_i) - \frac1{n'}\sum_{i\in [n']} e^{t_{\R{Y}}(\R{Y}'_i)}
\end{align}
$$ (MINE)

$P_{\R{X}'}$ and $P_{\R{Y}'}$ are known distributions and so arbitrarily many i.i.d. samples can be drawn from them directly without using the resampling trick {eq}`resample`.

Furthermore, the the dis

Indeed, one need only a generator for the reference samples:

$$
\begin{alig  n
$$

---

**Corollary**

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= \inf_{\substack{P_{\R{X}'}\in \mc{P}(\mc{X})\\ P_{\R{Y}'}\in \mc{P}(\mc{Y})}} D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'}).
\end{align}
$$ (MI-ub)

where the optimal solution is $P_{\R{X}'}\times P_{\R{Y}'}=P_{\R{X}}\times P_{\R{Y}}$, the product of marginal distributions of $\R{X}$ and $\R{Y}$.

---

The corollary follows from the proposition because

removing from the {eq}`(MI-E)` the two divergences $$D(P_{\R{X}}\|P_{\R{X}'}), D(P_{\R{Y}}\|P_{\R{Y}'})\geq 0$$ in {eq}`

and $I(\R{X}\wedge\R{Y})$ does not depend on the reference distribution.

```python

```

We can also write mutual information in terms of the entropy:

Another way is to write the mutual information as a conditional divergence:

$$
\begin{align*}
I(\R{X}\wedge \R{Y}) &= E\left[\log \frac{P_{\R{Y}|\R{X}}(\R{Y}|\R{X})}{P_{\R{Y}}(\R{Y})} \right]\\
&= E\left[D(P_{\R{Y}|\R{X}}(\cdot|\R{X})\|P_{\R{Y}})\right]
=: E\left[D(P_{\R{Y}|\R{X}}\|P_{\R{Y}}|P_{\R{X}})\right]
\end{align*}
$$

Note that $g$

The *mutual information neural estimation (MINE)* in {cite}`belghazi2018mine` uses the Donsker-Varadhan formula for the KL divergence:

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= D(P_{\R{X},\R{Y}}\|P_{\R{X}}\times P_{\R{Y}}) \\
&=  \sup_{t: \mc{X}\times \mc{Y} \to \mathbb{R}} E[t(\R{X},\R{Y})] - \log E[e^{t(\R{X}', \R{Y}')}]\\
&\approx \sup_{\theta} \frac1n \sum_{i\in [n]} g_{\theta}(\R{X}_i,\R{Y}_i)] - \log \frac1{n'} \sum_{i\in [n']}e^{g_{\theta}(\R{X}_{\R{J}_i}, \R{Y}_{\R{K}_i})}.
\end{align}
$$ (MINE)

where $P_{\R{X}',\R{Y}'}:=P_{\R{X}}\times P_{\R{Y}}$ and $g_{\theta}$ is a neural network with parameter $\theta$.

Consider another change of variable:

$$
\begin{align}
q(z) &= \frac{e^{t(z)}}{1+e^{t(z)}}, & \text{or equivalently}\\
t(z) &= \log \frac{q(z)}{1-q(z)}
\end{align}
$$

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) =  \sup_{q: \mc{Z} \to (0,1)} E\left[\log \frac{q(\R{Z})}{1-q(\R{Z})}\right] - \log E\left[\frac{q(\R{Z}')}{1-q(\R{Z}')}\right]
\end{align}
$$

The mutual information between two random variables is defined as
$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= D(P_{\R{X},\R{Y}}\|P_{\R{X}},P_{\R{Y}})\\
&= D(P_{\R{Y}|\R{X}}\|P_{\R{Y}}|P_{\R{X}})
\end{align}
$$

```python

```

To estimate mutual information using a neural network,

**Proposition**

For two random variables $\R{Z}$ and $\R{Z}'$, the Donsker Varadhan formula for divergence is

$$
\begin{align}
D(P_{\R{Z}} \| P_{\R{Z}'}) &= \sup_{g\in \mathcal{G}} E[t(Z)] - \log E[e^{t(Z')}]
\end{align}
$$ (VD)

where $\mathcal{G}$ is the set of function $t:\mc{Z}\to \mathbb{R}$. The optimal solution satisfies

$$
\begin{align}
g(\R{Z}') \stackrel{a.s.}{=} \log \frac{d P_{\R{Z}}}{ d P_{\R{Z}'}} + c
\end{align}
$$ (VD_sol)

for some constant $c$. The random variables can be discrete, continuous, or a mixture of both.

A simple proof follows from the positivity of divergence.

$$
\begin{align}
H(\R{Z}) &= \inf_{Q \in \mc{P}(\mc{Z})} E\left[ -\log q(\R{Z}) \right]\\
&= \inf_{t:\mc{Z}\to \mathbb{R}} E\left[ -\log \frac{e^{t(\R{Z})}}{E[e^{t(\R{Z}')}/p_{\R{Z}'}(\R{Z}')]} \right]\\
\end{align}
$$
where the last expressing is obtained by rewriting
$q(\cdot):=\frac{e^{t(\cdot)}}{c}$ using the exponent $t$ with the normalization factor $c$ given by
$$
\begin{align}
E[e^{t(\R{Z}')}/P_{\R{Z}'(\R{Z}')}] &= \int_{\mc{Z}} \frac{e^{t(\R{Z}')}}{p_{\R{Z}'}(\R{Z}')} p_{\R{Z}'}(\R{Z}') dz\\
&=  \int_{\mc{Z}} e^{t(\R{Z}')}dz.
\end{align}
$$

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) &= E\left[-\log p_{\R{Z}'}(\R{Z})\right] - H(\R{Z})\\
&=  \sup_{t:\mc{Z}\to \mathbb{R}} E\left[ -\log \frac{e^{t(\R{Z})}/p_{\R{Z}'}(\R{Z}')}{E[e^{t(\R{Z}')}/p_{\R{Z}'}(\R{Z}')]} \right]\\
&= \sup_{t:\mc{Z}\to \mathbb{R}} E\left[ -\log \frac{e^{f(\R{Z})}}{E[e^{f(\R{Z}')}]} \right]
\end{align}
$$
where the last equality is obtained with $f:=t-\log p_{\R{Z}'}$.

The optimal solution satisfies

$$
t(\R{Z}) \stackrel{a.s.}{=} \log p_{\R{Z}}(\R{Z}) + \log c
$$
for some constant $c$.

The optimal solut

```python

```


```python
import numpy as np
import numpy.linalg as nl
import numpy.random as nr
import sympy as sy
import IPython.display as disp

sy.init_printing()

```

# 역행렬
Inverse matrix

## 2x2

다음 비디오는 역행열 찾는 가우스 조단법을 소개한다.

Following video introduces Gauss Jordan method finding the inverse matrix. (36:23 ~ 42:20)

[](https://www.youtube.com/watch?v=FX4C-JpTFgY&list=PL221E2BBF13BECF6C&index=9&start=2183&end=2540)

아래 2x2 행렬을 생각해 보자.

Let's think about the 2x2 matrix.

```python
A22 = np.array([
    [1, 3],
    [2, 7]
])

```

```python
A22

```

오른쪽에 같은 크기의 단위행렬을 붙여 보자.

Let's augment an identity matrix of the same size.

```python
I22 = np.identity(2)

```

```python
I22

```

```python
AX22 = np.hstack([A22, I22])

```

```python
AX22

```

이제 왼쪽 2x2 부분을 단위행렬로 만들어 보자.

Let's make the left 2x2 part an identity matrix.

첫 행에 2를 곱한 후 2행에서 빼 보자.

Let's multipy 2 to the first row and then subtract from the second row.

```python
AX22[1, :] -= 2 * AX22[0, :]

```

```python
AX22

```

이번에는 2번째 행에 3을 곱해서 첫 행에서 빼 보자.

Now let's multipy 3 to the second row and subtract from the first row.

```python
AX22[0, :] -= 3 * AX22[1, :]

```

```python
AX22

```

위 `AX22` 행렬에서 오른쪽 두 행을 따로 떼어 보자.

Let's separate the right two columns of the `AX22` matrix above.

```python
A22_inv = AX22[:, 2:]

```

```python
A22_inv

```

A 행렬과 곱해보자.

Let's multipy with the A matrix.

```python
A22 @ A22_inv

```

## `numpy`

`numpy.ndarray` 의 `.I` 속성을 이용할 수도 있다.

We can use the `.I` property of `numpy.ndarray`.

```python
mat_A22_inv = np.matrix(A22).I

```

```python
mat_A22_inv

```

```python
A22 @ mat_A22_inv

```

또한 `numpy.linalg.inv()` 함수도 있다.

Also, `numpy.linalg.inv()` function is available.

```python
A22_inv = nl.inv(A22)

```

```python
A22_inv

```

```python
A22_inv @ A22

```

## 3x3

다음 비디오는 역행열 찾는 가우스 조단법을 소개한다.

Following video introduces Gauss Jordan method finding the inverse matrix.

[](https://www.youtube.com/watch?v=obts_JDS6_Q)

아래 행렬을 생각해 보자.

Let's think about the following matrix.

```python
A33_list = [
    [1, 0, 1],
    [0, 2, 1],
    [1, 1, 1],
]

```

```python
A33 = np.array(A33_list)

```

```python
A33

```

오른쪽에 같은 크기의 단위행렬을 붙여 보자.

Let's augment an identity matrix of the same size.

```python
I33 = np.identity(A33.shape[0])

```

```python
I33

```

```python
AX33 = np.hstack([A33, I33])

```

```python
AX33

```

이제 왼쪽 부분을 단위행렬로 만들어 보자.

Let's make the left part an identity matrix.

첫 행을 3행에서 빼 보자.

Let's subtract the first row from the third row.

```python
AX33[2, :] -= AX33[0, :]

```

```python
AX33

```

이번에는 2번째 행과 3번째 행을 바꾸자.

Now let's swap the second and the third rows. ([ref](https://stackoverflow.com/a/54069951))

```python
AX33[[1, 2]] = AX33[[2, 1]]

```

```python
AX33

```

두번째 행에 2를 곱해서 3행에서 빼 보자.

Let's multiply 2 to the second row and subtract from the third row.

```python
AX33[2, :] -= 2 * AX33[1, :]

```

```python
AX33

```

첫번째 행에서 3번째 행을 빼 보자.

Let's subtract the third row from the first row.

```python
AX33[0, :] -= AX33[2, :]

```

```python
AX33

```

위 `AX` 행렬에서 오른쪽 두 행을 따로 떼어 보자.

Let's separate the right two columns of the `AX` matrix above.

```python
A33_inv = AX33[:, 3:]

```

```python
A33_inv

```

A 행렬과 곱해보자.

Let's multipy with the A matrix.

```python
A33 @ A33_inv

```

## `numpy`

`numpy.ndarray` 의 `.I` 속성을 이용할 수도 있다.

We can use the `.I` property of `numpy.ndarray`.

```python
mat_A33_inv = np.matrix(A33).I

```

```python
mat_A33_inv

```

```python
A33 @ mat_A33_inv

```

## 표준 기능으로 구현한 가우스 조단법
Gauss Jordan method in Standard library

다음 셀은 가우스 조단법을 표준기능 만으로 구현한다.

Following cell implements the Gauss Jordan method with standard library only.

```python
import typing

Scalar = typing.Union[int, float]
Row = typing.Union[typing.List[Scalar], typing.Tuple[Scalar]]
Matrix = typing.Union[typing.List[Row], typing.Tuple[Row]]

def get_zero(n:int) -> Matrix:
    return [
        [0] * n for i in range(n)
    ]

def get_identity(n:int) -> Matrix:
    result = get_zero(n)
    for i in range(n):
        result[i][i] = 1

    return result

def augment_mats(A:Matrix, B:Matrix):
    assert len(A) == len(B)
    return [row_A + row_B for row_A, row_B in zip(A, B)]

def gauss_jordan(A:Matrix) -> Matrix:
    AX = augment_mats(A, get_identity(len(A)))

    # pivot loop
    for p in range(len(AX)):

        one_over_pivot = 1.0 / AX[p][p]

        # normalize a row with one_over_pivot
        for j in range(len(AX[p])):
            AX[p][j] *= one_over_pivot

        # row loop
        for i in range(len(AX)):
            if i != p:
                # row operation
                multiplier = - AX[i][p]

                # column loop
                for j in range(0, len(AX[p])):
                    AX[i][j] += multiplier * AX[p][j]

    return [row[len(A):] for row in AX]

```

위 행렬의 예로 확인해 보자.

Let's check with the matrix above.

```python
mat_A33_inv_GJ = gauss_jordan(A33_list)

```

```python
import pprint
pprint.pprint(mat_A33_inv_GJ, width=40)

```

```python
np.array(mat_A33_inv_GJ) @ A33

```

## 4x4

아래 행렬을 생각해 보자.

Let's think about the following matrix.

```python
A44_list = [
    [1, 0, 2, 0],
    [1, 1, 0, 0],
    [1, 2, 0, 1],
    [1, 1, 1, 1],
]

```

```python
A44 = np.array(A44_list)

```

```python
A44

```

다음 셀은 넘파이 다차원 배열 `numpy.ndarray` 을 위해 구현한 가우스 조르단 소거법 함수를 불러들인다.

Following cell imports an implementation of the Gauss Jordan Elimination for a `numpy.ndarray`.

```python
import gauss_jordan

```

위 행렬에 적용해 보자.

Let's apply to the matrix above.

```python
A44_inv_array = gauss_jordan.inv(A44)

```

```python
A44_inv_array

```

```python
A44_inv_array @ A44

```

```python
import numpy.testing as nt

nt.assert_array_almost_equal(A44_inv_array @ A44, np.array(get_identity(len(A44_list))))

```

## 연습 문제
Exercise

위 가우스 조단법에서 메모리를 더 절약하는 방안을 제안해 보시오

Regarding the Gauss Jordan implementation above, propose how we can save more memory.

## 참고문헌
References

* Gilbert Strang. 18.06 Linear Algebra. Spring 2010. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA.
* Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong, Mathematics For Machine Learning, Cambridge University Press, 2020, ISBN 978-1108455145.

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```


$\newcommand{\xv}{\mathbf{x}}
 \newcommand{\tv}{\mathbf{t}}
 \newcommand{\wv}{\mathbf{w}}
 \newcommand{\Chi}{\mathcal{X}}
 \newcommand{\R}{\rm I\!R}
 \newcommand{\sign}{\text{sign}}
 \newcommand{\Tm}{\mathbf{T}}
 \newcommand{\Xm}{\mathbf{X}}
 \newcommand{\Im}{\mathbf{I}}
$
### ITCS6155

# Linear Model

**Supervised Learning**:

$$ f: \mathcal{X} \rightarrow y $$

Supervised learning can be formulated as above. When we want to predict tomorrow's temparature, for instance, we might look for the data that we can use as an input $\mathcal{X}$ such as humidity, history of temparature changes, air pressure, and vapor pressure along with the output (today's temparature) $y$. Once we recorded or found the data, we can build a table as follows.

humidity (%) | last year temparature (ºF) | yesterday's air pressure (inHG) | vapor pressure (inHG) | **Today's Temp** (ºF)
---|---|---|---|---
23 | 72 | 30.12 | 0.79 | 76
15 | 82 | 29.32 | 0.68 | 81
 | | ... | |

Here, we note that the output is *today's* not *tomorrow's* temparature.
As we discussed in the first lecture, machine learning model *learns* from data or experiences.
This learning is called as *"training"* and the data used for training is called *training samples*.
In this example, the table is training samples that we will feed in to our models.
To maintain the right relation between input data and output prediction, however, the humidity and pressure values from yesterday are paired with the output, today's temparature.

Once you have data to play with, now you can apply learning algorithms ($f$) to find parameters.
This model with the learned parameters is *hypothesis*, and will be your model for prediction.
From today's measurements, applying the hypothesis model can simply generate the prediction output.
When the training is successful, it is more likely to have a good estimation.

## Linear Model

Linear model can be defined as a Euclidean dot product between two vectors:

$$
\begin{align}
  f(\xv; \wv) &= \wv^\top \xv = \sum_0^D w_i x_i \\
     &= w_0 x_0 + w_1 x_1 + \cdots + w_D x_D
\end{align}
$$

where $\wv$ is a weight vector and $\xv$ is an input vector.
When it is a one dimensional vector, it represents a straight line, so it is called *linear*.

Assume that we have $N$ data observations, $\xv_i$ and target outputs, $t_i$, for $i = 1, \cdots, N$.
The simpliest model that we can think of is constant model, $f(\xv) = c$, where $c$ is any scalar.
In this case, we have the zero weights in the linear model.

The linearity in the parameter $\xv$ make the optimization based on deravatives to solvable analytically.
The model limits the complexity so its representation is also limited. However, it can prevent possible overfitting with the simple modeling, especially when you have sparsely sampled data.

### Dot Product

The dot product, also known as inner product or scalar product, computes the product of pair of elements in each vector and the summation of the products. It is geometrically interpreted as a cosine between two vectors as well.
Thus, it can be written as

$$
    \wv^\top \xv = \| \wv \| \| \xv \| \cos(\theta),
$$

where $\theta$ is the angle between two vectors.
Thus, when the vectors are unit vectors, the dot product is simply consine of the angle between two vectors.

<center>(from wikipedia)</center>

### Regression

The target output $t$ is s real number ($y, t \in \R$) as we discussed in last lecture. Thus, training $f(\xv; \wv)$ generates $y$ values close to the target outputs. We will discuss more about linear regression in next section.

### Classification

Classification has target outputs as discrete values. In case of binary classification, you have two values as target (ie. $t \in \{ -1, 1 \}$). As we know that $y$ can be any real value, we can cap the model to generate discrete values as below:

$$ y = \sign ( f(\xv; \wv) ).$$

### Advantages of Linear Model

- Simple
- Stable
- Avoid Overfitting
- Scalable

# Practice
**Finish this excercise and submit on Canvas.**

Q: Write a python code that create two vectors, $\wv$ and $\xv$ as follows:

$
\xv = \begin{bmatrix}
         4.0 \\
         2.3 \\
         1.2 \\
         5.8
        \end{bmatrix},
\wv = \begin{bmatrix}
         0.8 \\
         0.1 \\
         0.53 \\
         0.33
        \end{bmatrix}
$

```python
import numpy as np

x = np.array([4.0, 2.3, 1.2, 5.8])
w = np.array([0.8, 0.1, 0.53, 0.33])
```

Q: Write a function *linear_model(x, w)* that returns the result of dot product.

```python
def linear_model(x, w):
    return np.dot(x, w)
```

Q: Pass the $\wv$ and $\xv$ and print the output of the linear model.

```python
dot_val = linear_model(x, w)

dot_val
```

    5.9799999999999995

# Linear Regression

In this note, we solve regression problems using the linear model as follows.
For instance, we have example data as follows.
The goal we want to acheive in this problem is to find a best fit on all the data.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
X = np.linspace(0,10, 101)
T = 2 * X  + 4+ np.random.rand(101) * 5
```

```python
def data_scatter(k=101):
    plt.plot(T[:k], '.')
    plt.xticks(range(0, 101, 20)[:k], range(0, 11, 2)[:k])
```

```python
data_scatter()
```

When we use a linear model, there can be multiple options. One of them, or the simplest solution, can be the avergage value.

```python
mean = np.mean(T)

data_scatter()
plt.plot([0, 100],[mean, mean], 'r-')
print(mean)
```

When the data is linear or when we need simple solution, the linear model can suggest better solutions.
For instance, one dimensional affine model can be written as

$$
f(x; a, b) = a x + b.
$$

Unifying the weight symbol with $w$,
$$
f(x; \wv) =  w_1 x + w_0.
$$

Considering multiple inputs for $x$, we can extend the input $x$ to input vector $\xv$ with dummy input $x_0 = 1$:

$$
\begin{align}
f(\xv; \wv) &=  w_D x_D + \cdots + w_1 x_1 + w_0 \\
            &= \sum_{i=0}^{D} w_i x_i \quad\text{where } x_0 = 1\\
            &= \wv^\top \xv.
\end{align}
$$

#### Error (Cost) Function

When we define $\wv \in \R^D$, the choice of $D$ real numbers can result in the best result.
Here, the word "*best*" can be vague. Thus, we need to define what *best* means.
The sum of square error function defines as follows:

$$
E(\wv) = \sum_{i=1}^N \Big( f(\xv_i; \wv_i) - t_i \Big)^2
$$

This error function says that we want to minimize the sum of Euclidean distances between the target values and the model outputs. Here the square term leads the distances are not biased a few samples since sacrificing others with large errors will disturb the objective.

## Least Squares

The parameter that gives best fit will be

$$
\wv^* = \arg\min_\wv \sum_{i=1}^{N} \Big( f(\xv_i; \wv) - t_i \Big)^2
$$

Since the error funciton is quadratic, the problem can be analytically solved by simply setting derivative with respect to $\wv$ to zero.
For this, let us prepare data in matrix.
The target values are collected in matrix $\tv$, and the input samples are in matrix $\Xm$.

$$
\begin{align}
\tv &= [t_1, t_2, \cdots, t_N]^\top \\
\\
\wv &= [w_0, w_1, \cdots, w_D]^\top \\
\\
\Xm &= \begin{bmatrix}
    x_{10} & x_{11} & x_{12} & \dots  & x_{1D} \\
    x_{20} & x_{21} & x_{22} & \dots  & x_{2D} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    x_{N0} & x_{N1} & x_{N2} & \dots  & x_{ND}
\end{bmatrix}
\end{align}
$$
where the first column is one's, $\xv_{*0} = [1, 1, \dots, 1]^\top$.

With this matrix, $f(\xv; \wv)$ can be written in matrix form as:
$$
f(\xv; \wv) = \Xm \wv.
$$
Thus, the error function can be
$$
\begin{align}
E(\wv) &= \sum_{i=1}^N \Big(f(\xv_i; \wv_i) - t_i \Big)^2 \\
            \\
            &= (\Xm \wv - \tv)^\top (\Xm \wv - \tv) \\
            \\
            &= \wv^\top \Xm^\top \Xm \wv - 2 \tv^\top \Xm \wv + \tv^\top \tv
\end{align}
$$
because $\tv^\top \Xm \wv$ is a scalar, so symmetric.

Now, let us do the derivative.
$$
\begin{align}
\frac{\partial E(\wv)}{\partial \wv} &= \frac{\partial (\Xm \wv - \tv)^\top (\Xm \wv - \tv)}{\partial \wv} \\
        \\
        &= \frac{\partial (\wv^\top \Xm^\top \Xm \wv - 2 \tv^\top \Xm \wv + \tv^\top \tv )}{\partial \wv} \\
        \\
        &= \frac{\partial (\wv^\top \Xm^\top \Xm \wv)}{\partial \wv} - 2 \Xm^\top \tv \\
        \\
        &= \Xm^\top \Xm \wv + (\Xm^\top \Xm)^\top \wv - 2 \Xm^\top \tv  \\
        \\
        &= 2 \Xm^\top \Xm \wv - 2 \Xm^\top \tv
\end{align}
$$

Setting this to zero,
$$
\begin{align}
 2 \Xm^\top \Xm \wv - 2 \Xm^\top \tv  &= 0\\
\\
\Xm^\top \Xm \wv &= \Xm^\top \tv\\
\\
\wv &= \big(\Xm^\top \Xm\big)^{-1} \Xm^\top \tv
\end{align}
$$

# Practice

Implement the least squares model and apply to the simulated data X and T.
Consider using **np.linalg.inv** (to inverse a matrix), **np.linalg.solve** (to solve system of linear scalar equations), **np.linalg.lstsq** (least-squares solution to a linear matrix equation).
After getting the parameter w, plot the approximation line.

```python
import numpy as np

# First creast X1 by adding 1's column to X
N = X.shape[0]
X1 = np.c_[np.ones((N, 1)), X]

# the following line is also valid to insert a column containing 1s
#X1 = np.hstack((np.ones((N, 1)), X.reshape((X.shape[0], -1))))

# print(X1)
# print(X1.transpose())

# Next, using inverse, solve, lstsq function to get w*
w = np.linalg.inv(X1.transpose().dot(X1)).dot(X1.transpose()).dot(T)

print(w)
```

    [6.44338295 2.05984345]

```python
# plot the linear line (w)

y = X1.dot(w)

data_scatter()
plt.plot(y)
```

## Least Mean Squares (LMS)

Previously we observed that the least squares use all the available data for training or finding the best fit.
This can be often computationally costly, especially with large data sets. When data is sufficiently large, we can consider *sequential* or *online* learning.
During the online learning process, we introduce a data point one by one, and update the parameters. Using the updated parameters, it makes a new estimation and repeat these steps.

For this, we start with an initial guess $\wv$ and changes it as it reads more data until it converges.
When $k$ represents the steps for the repetition,

$$
\wv^{(k+1)} = \wv^{(k)} - \alpha \nabla E_k
$$

where $E_k$ is the error for the $k$'th sample and $\alpha$ is a learning rate.
This is called *stochastic gradient descent* or *sequential gradient descent*.

For the $k$'th sample $\xv_k$, the gradient for the sum-of-squares error is

$$
\begin{align}
\nabla E_k = \frac{\partial E}{\partial \wv^{(k)}} &= \frac{\partial }{\partial \wv^{(k)}}\Big( f(\xv_k; \wv^{(k)}) - t_k \Big)^2 \\
        &= 2 \Big( f(\xv_k; \wv^{(k)}) - t_k \Big) \frac{\partial }{\partial \wv^{(k)}} \Big( f(\xv_k; \wv^{(k)}) - t_k \Big) \\
        &= 2 \Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \frac{\partial }{\partial \wv^{(k)}} \Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \\
        &= 2\Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \xv_k.
\end{align}
$$
This gives the following update rule for each sample:

$$
\wv^{(k+1)} = \wv^{(k)} - \alpha \Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \xv_k.
$$

# Pratice

Implement the LMS for the simulated sample X, and show the plot

```python
print(X1.shape)
print(w.shape)
print(w.transpose().shape)
```

    (101, 2)
    (2,)
    (2,)

```python
import IPython.display as ipd  # for display and clear_output

# initial weights with random values
w = np.random.rand(X1.shape[1])

# learning rate
alpha = 0.01

fig = plt.figure()

# sequential learning
for k in range(N):
    # online update of weights
    w = w - (alpha * ((X1[k:k+1, 0:2].dot(w.transpose()) - T[k:k+1]).dot(X1[k:k+1, 0:2])))

    plt.clf()
    data_scatter(k+1)

    # Plot the current model's estimation in a line
    tmpY = X1 @ w
    plt.plot(tmpY)
    plt.ylim([0,30])

    ipd.clear_output(wait=True)
    ipd.display(fig)
ipd.clear_output(wait=True)
```


# May 24 final review

Suggestions from students about how to study: https://PollEv.com/discourses/gFCtSJE1KXATAXls02RMz/respond

## Logistics

- exam begins Monday May 31 at 8am
- Due Friday June 4 at 5p for 1ec point
- automatic extension to Sun June 6 at 11:59p
- graded on effort (1pt) and accuracy (1pt)
- make sure you show your work!!!
- DEI survey + final course evaluation -- replace worst homework grade

## Today

- study tips (polleverywhere)
- second half review
--- state space control
--- frequency analysis
--- frequency control

## cruise control

problem from: https://ctms.engin.umich.edu/CTMS/index.php?example=CruiseControl&section=SystemModeling#:~:text=The%20purpose%20of%20the%20cruise,according%20to%20a%20control%20law.

$$m\dot{v}+bv=u$$

$$y=v$$

$m=1000$ kg (vehicle mass)

$b = 50$ Ns/m (dampening coefficient)

## week 6: state space control

```python
# numpy = numerical Python, implements arrays (/ matrices)
import numpy as np
# limit number of decimal places printed for floating-point numbers
np.set_printoptions(precision=3)

# scipy = scientific Python, implements operations on arrays / matrices
import scipy as sp
# linalg = linear algebra, implements eigenvalues, matrix inverse, etc
from scipy import linalg as la
# optimize = optimization, root finding, etc
from scipy import optimize as op

# produce matlab-style plots
import matplotlib as mpl
# increase font size on plots
mpl.rc('font',**{'size':18})
# use LaTeX to render symbols
mpl.rc('text',usetex=False)
# animation
from matplotlib import animation as ani
# Matlab-style plotting
import matplotlib.pyplot as plt

# symbolic computation, i.e. computer algebra (like Mathematica, Wolfram Alpha)
import sympy as sym

```

```python
# test whether this is a Colaboratory or Jupyter notebook
try:
  import google.colab
  COLAB = True
  print('Colaboratory Notebook')
except:
  COLAB = False
  print('Jupyter Notebook')
# Colab notebook
if COLAB:
  # pip = Python package manager; "!" means "run at system level"
  !pip install control
  !pip install JSAnimation

  # render SymPy equations nicely in Colaboratory Notebook
  def colab_latex_printer(exp,**options):
    from google.colab.output._publish import javascript
    url = "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=default"
    javascript(url=url)
    return sym.printing.latex(exp,**options)

  sym.init_printing(use_latex="mathjax",latex_printer=colab_latex_printer)

# Jupyter notebook
else:
  init_printing(use_latex='mathjax')

# Python's Control Systems Toolbox
import control as ctrl

# SciPy module that implements many of the routines in ctrl
from scipy import signal as sig

# render animations in the notebook
from JSAnimation.IPython_display import display_animation
```

    Colaboratory Notebook
    Requirement already satisfied: control in /usr/local/lib/python3.7/dist-packages (0.9.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from control) (1.4.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from control) (3.2.2)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from control) (1.19.5)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control) (2.8.1)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib->control) (1.15.0)
    Requirement already satisfied: JSAnimation in /usr/local/lib/python3.7/dist-packages (0.1)

```python
def Jacobian(g,y,d=1e-4):
  """
  approximate derivative via finite-central-differences

  input:
    g - function - g : R^n -> R^m
    y - n array
    (optional)
    d - scalar - finite differences displacement parameter

  output:
    Dg(y) - m x n - approximation of Jacobian of g at y
  """
  # given $g:\mathbb{R}^n\rightarrow\mathbb{R}^m$:
  # $$D_y g(y)e_j \approx \frac{1}{2\delta}(g(y+\delta e_j) - g(y - \delta e_j)),\ \delta\ll 1$$
  e = np.identity(len(y))
  Dyg = []
  for j in range(len(y)):
      Dyg.append((.5/d)*(g(y+d*e[j]) - g(y-d*e[j])))
  return np.array(Dyg).T

def numerical_simulation(f,t,x,t0=0.,dt=1e-4,ut=None,ux=None,utx=None,return_u=False):
  """
  simulate x' = f(x,u)

  input:
    f : R x X x U --> X - vector field
      X - state space (must be vector space)
      U - control input set
    t - scalar - final simulation time
    x - initial condition; element of X

    (optional:)
    t0 - scalar - initial simulation time
    dt - scalar - stepsize parameter
    return_u - bool - whether to return u_

    (only one of:)
    ut : R --> U
    ux : X --> U
    utx : R x X --> U

  output:
    t_ - N array - time trajectory
    x_ - N x X array - state trajectory
    (if return_u:)
    u_ - N x U array - state trajectory
  """
  t_,x_,u_ = [t0],[x],[]

  inputs = sum([1 if u is not None else 0 for u in [ut,ux,utx]])
  assert inputs <= 1, "more than one of ut,ux,utx defined"

  if inputs == 0:
    assert not return_u, "no input supplied"
  else:
    if ut is not None:
      u = lambda t,x : ut(t)
    elif ux is not None:
      u = lambda t,x : ux(x)
    elif utx is not None:
      u = lambda t,x : utx(t,x)

  while t_[-1]+dt < t:
    if inputs == 0:
      _t,_x = t_[-1],x_[-1]
      dx = f(t_[-1],x_[-1]) * dt
    else:
      _t,_x,_u = t_[-1],x_[-1],u(t_[-1],x_[-1])
      dx = f(_t,_x,_u) * dt
      u_.append( _u )

    x_.append( _x + dx )
    t_.append( _t + dt )

  if return_u:
    return np.asarray(t_),np.asarray(x_),np.asarray(u_)
  else:
    return np.asarray(t_),np.asarray(x_)
```

### 6a. state feedback

- pole placement / eigenvalue assignment -- how to pick?
- determine shape of $K$
- determine values of $K$ symbolically and numerically

- $\lambda=\sigma\pm j\omega,a(s)=s^2+2j\sigma\omega+(\omega^2+\sigma^2)$ (pole placement)
- $\lambda=-\zeta\omega\pm \omega\sqrt{\zeta^2-1},a(s)=s^2+2\zeta\omega s+\omega^2$ (canonical second order system)

```python
# define parameters
m = 1000. #kg
b = 50. # Ns/m dampening coeff

# define A, B, C matrices
A = np.reshape(np.array([-b/m]),(1,1))
B = np.reshape(np.array([1/m]),(1,1))
C = np.reshape(np.array([1]),(1,1))
D = np.reshape(np.array([0]),(1,1))

p = -1.
K = ctrl.place(A,B,[p])
K
```

    array([[950.]])

```python
def f_LTI(t,x,u):
  # x is a 1-D array, i.e. a "row", so we left-multiply the transpose of (A-BK)
  return np.dot(x,(A-np.dot(B,K)).T)

# simulation duration, timestep
T = 8.
dt = 1e-2

# non-zero initial state
v0 = 1.
x0 = np.array([v0])

# zero input
def u(t):
  ut = 0
  return ut
# this could equivalently be written in one line:
# u = lambda t : 0.

# numerical simulation returns:
# t_ - size N array
# x_ - size N x 2 array (since the state is 2-dimensional)
t_,x_ = numerical_simulation(f_LTI,T,x0,ut=u,dt=dt)

# the first column contains theta(t), the second column contains dtheta(t)
# (I append an underscore "_" to remind myself these are time series)
theta_ = x_[:,0]

# we'll use the Matplotlib library for plots
# (as the name suggests, it has a similar interface to MATLAB)
import pylab as plt

# plot theta
plt.figure()
plt.subplot(2,1,1)
plt.plot(t_,theta_,label='(Linear)')
plt.plot(t_[0],theta_[0],'gs',ms=10)
plt.plot(t_[-1],theta_[-1],'ro',ms=10)
plt.ylabel(r'$v(t)$')

# plot dtheta
u = -K*x_ # also -K@x or -np.multiply(K,x)
plt.subplot(2,1,2)
plt.plot(t_,u,label='(Linear)')
plt.plot(t_[0],u[0],'gs',ms=10)
plt.plot(t_[-1],u[-1],'ro',ms=10)
plt.ylabel(r'$u(t)$')
plt.xlabel(r'time $t$')

plt.tight_layout();

# # phase portrait
# plt.figure()
# plt.subplot(1,1,1)
# plt.plot(theta_,dtheta_,label='(Linear)')
# plt.plot(theta_[0],dtheta_[0],'gs',ms=10)
# plt.plot(theta_[-1],dtheta_[-1],'ro',ms=10)
# plt.ylabel(r'$\dot{\theta}(t)$')
# plt.xlabel(r'${\theta}(t)$')

# plt.tight_layout();
```

### 6b. state estimation

- pole placement / eigenvalue assignment -- how to pick?
- determine shape of $L$
- determine values of $L$ symbolically and numerically

```python
zeta = 5*p
L = -np.array(ctrl.place(A.T,C.T,zeta).T)
L
```

    array([[-4.95]])

```python
# vector field
def f_error(t,e,u):
  # x is a 1-D array, i.e. a "row", so we left-multiply the transpose of (A-LC)
  return np.dot(e,(A+np.dot(L,C)).T)

# non-zero initial state
error0 = 10.
e0 = np.array([error0])

# zero input
def u(t):
  ut = 0
  return ut
# this could equivalently be written in one line:
# u = lambda t : np.sin(t)

# numerical simulation returns:
# t_ - size N array
# e_ - size N x 2 array (since the state is 2-dimensional)
t_,e_ = numerical_simulation(f_error,T,e0,ut=u,dt=dt)

# the first column contains theta(t), the second column contains dtheta(t)
# (I append an underscore "_" to remind myself these are time series)
error_ = e_[:,0]

# we'll use the Matplotlib library for plots
# (as the name suggests, it has a similar interface to MATLAB)
import pylab as plt

# plot theta
plt.figure()
plt.subplot(2,1,1)
plt.plot(t_,error_,label='(Linear)')
plt.plot(t_[0],error_[0],'gs',ms=10)
plt.plot(t_[-1],error_[-1],'ro',ms=10)
plt.ylabel(r'$e(t)$')

# # plot dtheta
# plt.subplot(2,1,2)
# plt.plot(t_,derror_,label='(Linear)')
# plt.plot(t_[0],derror_[0],'gs',ms=10)
# plt.plot(t_[-1],derror_[-1],'ro',ms=10)
# plt.ylabel(r'$\dot{e}(t)$')
# plt.xlabel(r'time $t$')

plt.tight_layout();

# # phase portrait
# plt.figure()
# plt.subplot(1,1,1)
# plt.plot(error_,derror_,label='(Linear)')
# plt.plot(error_[0],derror_[0],'gs',ms=10)
# plt.plot(error_[-1],derror_[-1],'ro',ms=10)
# plt.ylabel(r'$\dot{e}(t)$')
# plt.xlabel(r'$e(t)$');

# plt.tight_layout();
```

### 6c. stabilizing controller

- why do we do this?

```python
# closed-loop vector field
def f_cl(t,z,u):

  # extract controlled system and observer system state from full system state
  x,x_hat = z[0],z[1] # equivalent to z[0:2],z[2:-1]

  # compute control input using observer state
  u = -np.dot(x_hat,K.T)

  # output from controlled system and observer system
  y = np.dot(x,C.T) + np.dot(u,D.T)
  y_hat = np.dot(x_hat,C.T) + np.dot(u,D.T)

  # dynamics of controlled system and observer system
  dx = f_LTI(t,x,u)
  dx_hat = np.dot(x_hat,A.T) - np.dot(u,B.T) - np.dot((y-y_hat),L.T)

  # combine dynamics of controlled system and observer system
  dz = np.hstack((np.squeeze(dx),np.squeeze(dx_hat)))

  return dz

v0 = 3.
v0_hat = 10.

# non-zero initial state
z0 = np.array([v0, v0_hat])

# zero input
def u(t):
  ut = 0
  return ut
# this could equivalently be written in one line:
# u = lambda t : np.sin(t)

# numerical simulation returns:
# t_ - size N array
# z_ - size N x 2 array (since the state is 2-dimensional)
t_,z_ = numerical_simulation(f_cl,T,z0,ut=u,dt=dt)

# the first column contains theta(t), the second column contains dtheta(t)
# (I append an underscore "_" to remind myself these are time series)
x_ = z_[:,0]
x_hat_ = z_[:,1]
# theta_,dtheta_, = x_.T
# theta_hat_,dtheta_hat_, = x_hat_.T

# we'll use the Matplotlib library for plots
# (as the name suggests, it has a similar interface to MATLAB)
import pylab as plt
plt.figure()

# plot theta, theta_hat
plt.subplot(2,1,1)
plt.plot(t_,x_,'--',lw=4,label=r'$v(t)$')
plt.plot(t_,x_hat_,label=r'$\widehat{v(t)}$')
plt.plot(t_[0],x_[0],'gs',ms=10)
plt.plot(t_[-1],x_[-1],'ro',ms=10)
plt.plot(t_[0],x_hat_[0],'gs',ms=10)
plt.plot(t_[-1],x_hat_[-1],'ro',ms=10)
plt.ylabel(r'$v(t)$')
plt.legend();

# # plot dtheta, dtheta_hat
# plt.subplot(2,1,2)
# plt.plot(t_,dtheta_,'--',lw=4,label=r'$\dot{\theta}$')
# plt.plot(t_,dtheta_hat_,label=r'$\dot{\widehat{\theta}}$')
# plt.plot(t_[0],dtheta_[0],'gs',ms=10)
# plt.plot(t_[-1],dtheta_[-1],'ro',ms=10)
# plt.plot(t_[0],dtheta_hat_[0],'gs',ms=10)
# plt.plot(t_[-1],dtheta_hat_[-1],'ro',ms=10)
# plt.ylabel(r'$\dot{\theta}(t)$')
# plt.xlabel(r'time $t$')

# plt.figure()
# plt.subplot(1,1,1)
# plt.plot(theta_,dtheta_,'--',lw=4,label=r'$x$')
# plt.plot(theta_hat_,dtheta_hat_,label=r'$\widehat{x}$')
# plt.plot(theta_[0],dtheta_[0],'gs',ms=10)
# plt.plot(theta_[-1],dtheta_[-1],'ro',ms=10)
# plt.plot(theta_hat_[0],dtheta_hat_[0],'gs',ms=10)
# plt.plot(theta_hat_[-1],dtheta_hat_[-1],'ro',ms=10)
# plt.ylabel(r'$\dot{\theta}(t)$')
# plt.xlabel(r'${\theta}(t)$')
# plt.legend();
```

### 6c. stabilizing controller (nonlinear)

- why do we do this?

```python
def h_NL(t,x,u):
  return x

def f_NL(t,x,u):
  return np.array([(u-b*x)/m])

# vector field
def f_cl(t,z,u):

  # extract controlled system and observer system state from full system state
  x,x_hat = z[0],z[1]

  # compute control input using observer state
  u = -np.dot(x_hat,K.T)

  # output from controlled system and observer system
  y = h_NL(t,x,u)
  y_hat = np.dot(x_hat,C.T) + np.dot(u,D.T)

  # dynamics of controlled system and observer system
  dx = f_NL(t,x,u)
  dx_hat = np.dot(x_hat,A.T) + np.dot(u,B.T) - np.dot((y-y_hat),L.T)

  # combine dynamics of controlled system and observer system
  dz = np.hstack((np.squeeze(dx),np.squeeze(dx_hat)))

  return dz

# NOTE:  nonlinear system's initial state needs to be VERY close to x_e
v0 = .001
v0_hat = 0.

# non-zero initial state
z0 = np.array([v0,v0_hat])

# zero input
def u(t):
  ut = 0
  return ut
# this could equivalently be written in one line:
# u = lambda t : np.sin(t)

# numerical simulation returns:
# t_ - size N array
# z_ - size N x 2 array (since the state is 2-dimensional)
t_,z_ = numerical_simulation(f_cl,T,z0,ut=u,dt=dt)

# the first column contains theta(t), the second column contains dtheta(t)
# (I append an underscore "_" to remind myself these are time series)
x_ = z_[:,0]
x_hat_ = z_[:,1]
# theta_,dtheta_, = x_.T
# theta_hat_,dtheta_hat_, = x_hat_.T

# we'll use the Matplotlib library for plots
# (as the name suggests, it has a similar interface to MATLAB)
import pylab as plt
plt.figure()

# plot theta, theta_hat
plt.subplot(2,1,1)
plt.plot(t_,x_,'--',lw=4,label=r'$v(t)$')
plt.plot(t_,x_hat_,label=r'$\widehat{v(t)}$')
plt.ylabel(r'$v(t)$')
plt.legend()

# # plot dtheta, dtheta_hat
# plt.subplot(2,1,2)
# plt.plot(t_,dtheta_,'--',lw=4,label=r'$\dot{\theta}$')
# plt.plot(t_,dtheta_hat_,label=r'$\dot{\widehat{\theta}}$')
# plt.ylabel(r'$\dot{\theta}(t)$')
# plt.xlabel(r'time $t$')

# plt.figure()
# plt.subplot(1,1,1)
# plt.plot(theta_,dtheta_,'--',lw=4,label=r'$x$')
# plt.plot(theta_hat_,dtheta_hat_,label=r'$\widehat{x}$')
# plt.ylabel(r'$\dot{\theta}(t)$')
# plt.xlabel(r'${\theta}(t)$')
# plt.legend();
```

## week 7: frequency analysis

### 7a. state space to transfer function

$$G_{yu}(s)=C(sI-A)^{-1}B+D$$

- algebraically or computationally

```python
A = sym.Matrix([[-b/m]])
B = sym.Matrix([[1/m]])
C = sym.Matrix([[1]]).T
sym.var("s")
Gyu = C*(s*sym.eye(1)-A).inv()*B
Gyu = Gyu[0].simplify().cancel()
Gyu
```

$\displaystyle \frac{0.001}{1.0 s + 0.05}$

### 7b. bode plots

```python
Gyu_ = sym.lambdify(s,Gyu)

omega = np.logspace(-1e1,1e1,1000)
xlim = omega[[0,-1]]

abs_L = np.abs(Gyu_(1.j*omega))
angle_L = np.unwrap(np.angle(Gyu_(1.j*omega)))*180./np.pi

# critical = (angle_L < -180).nonzero()[0][0]

plt.figure(); axs = []
ax = plt.subplot(2,1,1); ax.grid('on')
ax.loglog(omega,abs_L,'b-')
ax.loglog(xlim,[1.,1.],'k-')
# ax.loglog(omega[critical],abs_L[critical],'ro')
ax.set_ylabel(r'$|G_{yu}(j\omega)|$')
ax.set_xticks([])
ax.set_xlim(xlim)
axs.append(ax)

ax = plt.subplot(2,1,2); ax.grid('on')
ax.semilogx(omega,angle_L,'b-')
ax.semilogx(xlim,[-180.,-180.],'k-')
# ax.semilogx(omega[critical],angle_L[critical],'ro')
ax.set_xlabel(r'frequency $\omega$')
ax.set_ylabel(r'$\angle G_{yu}(j\omega)$ (deg)')
ax.set_yticks([-90,-180,-270])
ax.set_xlim(xlim)
axs.append(ax)

for ax in axs:
  for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] +
               ax.get_xticklabels() + ax.get_yticklabels()):
      item.set_fontsize(16)
```

### 7c effect of disturbances

can't simultaneously reject disturbances on input and output at the same time
- sensitivity $G_{yw}=S(j\omega)=\frac{1}{1+PC}$, e.g. sensor noise
- and complementary sensitivity $-G_{uv} = T(j\omega)=\frac{PC}{1+PC}$, e.g. wind pushing on car, hill

can't reshape $S$ to whatever you desire - Bode integral formula / principle
- any range of frequencies where controller attenuates disturbance is compensated by a range where controller amplifies disturbance

## week 8: frequency control

### 8a. Nyquist stability criterion

if $L = PC$ has no poles in the right-half plane, then $\frac{L}{1+L}$ is stable $\leftrightarrow$ $\Omega$ does not encircle $-1\in\mathbb{C}$

Nyquist plot: $\Omega=\{L(j\omega)\in\mathbb{C}: -\infty<\omega<\infty\}$

```python
P = .001 / (s+.05)
C = 250.
PC = sym.lambdify(s,P*C)

PC = sym.lambdify(s,2*(s-2)/((s+1)*(s+2)*(s+3)))

omega = np.linspace(0.,1e1,1000)

Omega = PC(1.j*omega)

plt.figure()
ax = plt.subplot(1,1,1); ax.grid('on')
ax.plot(Omega.real,Omega.imag,'b-',label=r'$L(j\omega), \omega > 0$')
ax.plot(Omega.real,-Omega.imag,'b--',label=r'$L(j\omega), \omega < 0$')
ax.plot(-1.,0.,'ro',label=r'critical point $-1\in\mathbb{C}$')
# ax.legend(fontsize=14,frameon=True,shadow=True)
ax.set_xlabel(r'$\operatorname{Re}\ L(j\omega)$')
ax.set_ylabel(r'$\operatorname{Im}\ L(j\omega)$');

for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] +
             ax.get_xticklabels() + ax.get_yticklabels()):
    item.set_fontsize(16)
```

### 8b. gain, phase, and stability margins

* *stability margin* $s_m$ = distance from $\Omega$ to $-1\in\mathbb{C}$;
* *gain margin* $g_m$ = smallest increase in gain that causes $\Omega$ to encircle $-1$ (computed as the inverse of distance from $\Omega$ to $-1\in\mathbb{C}$);
* *phase margin* $\varphi_m$ = smallest change in phase that causes $\Omega$ to encircle $-1$ (computed as the distance from $\Omega$ to $-1\in\mathbb{C}$ restricted to rotation of $\Omega$).

```python
omega = np.logspace(-1e1,1e1,1000)
xlim = omega[[0,-1]]

abs_L = np.abs(PC(1.j*omega))
angle_L = np.unwrap(np.angle(PC(1.j*omega)))*180./np.pi

# critical = (angle_L < -180).nonzero()[0][0]

plt.figure(); axs = []
ax = plt.subplot(2,1,1); ax.grid('on')
ax.loglog(omega,abs_L,'b-')
ax.loglog(xlim,[1.,1.],'k-')
# ax.loglog(omega[critical],abs_L[critical],'ro')
ax.set_ylabel(r'$|G_{yu}(j\omega)|$')
ax.set_xticks([])
ax.set_xlim(xlim)
axs.append(ax)

ax = plt.subplot(2,1,2); ax.grid('on')
ax.semilogx(omega,angle_L,'b-')
ax.semilogx(xlim,[-180.,-180.],'k-')
# ax.semilogx(omega[critical],angle_L[critical],'ro')
ax.set_xlabel(r'frequency $\omega$')
ax.set_ylabel(r'$\angle G_{yu}(j\omega)$ (deg)')
ax.set_yticks([-90,-180,-270])
ax.set_xlim(xlim)
axs.append(ax)

for ax in axs:
  for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] +
               ax.get_xticklabels() + ax.get_yticklabels()):
      item.set_fontsize(16)
```

```python

omega = np.linspace(0.75,1e1,1000)
Omega = PC(1.j*omega)

abs_L = np.abs(Omega)
angle_L = np.unwrap(np.angle(Omega))*180./np.pi

circle = np.exp(1.j*np.linspace(0.,2*np.pi))

# index (i.e. entry) of Omega array corresponding to stability margin
# g_m_idx = (angle_L < -180).nonzero()[0][0]
phi_m_idx = np.argmin(np.min(np.abs(Omega - circle[:,np.newaxis]),axis=0))
s_m_idx = np.argmin(np.abs(Omega - (-1.)))

# numerical value of stability margin
# g_m = -1./Omega[g_m_idx].real
phi_m = np.angle(Omega[phi_m_idx])*180./np.pi
phi_m_arc = np.exp(1.j*np.linspace(-180,phi_m)*np.pi/180.)
s_m = np.abs(Omega[s_m_idx] - (-1.))

axs = []

plt.figure(figsize=(6,6))
ax = plt.subplot(1,1,1); ax.grid('on'); ax.axis('equal')
# Omega, i.e. graph of L(j omega)
ax.plot(Omega.real,Omega.imag,'b-',label=r'$L(j\omega), \omega > 0$')
ax.plot(Omega.real,-Omega.imag,'b--',label=r'$L(j\omega), \omega < 0$')
# unit circle
ax.plot(circle.real,circle.imag,'k--')
# stability margins
# ax.plot(-1./g_m,0.,'ro',label=r'$-1/g_m$')
ax.plot(phi_m_arc.real,phi_m_arc.imag,'r--',label=r'$\varphi_m$')
ax.plot(Omega[phi_m_idx].real,Omega[phi_m_idx].imag,'ro')
ax.plot([-1.,Omega[s_m_idx].real],[0.,Omega[s_m_idx].imag],'r.-',ms=10,label=r'$s_m$')
# critical point (-1. + 0.j)
ax.plot(-1.,0.,'ko',label=r'critical point $-1\in\mathbb{C}$')
# legend and labels
ax.legend(fontsize=14,frameon=True,shadow=True)
ax.set_xlabel(r'$\operatorname{Re}\ L(j\omega)$')
ax.set_ylabel(r'$\operatorname{Im}\ L(j\omega)$');
axs.append(ax)

for ax in axs:
  for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] +
               ax.get_xticklabels() + ax.get_yticklabels()):
      item.set_fontsize(16)
```

### 8c. root locus

general idea: for a causal plant $P$ with proportional feedback controller $C(s) = K$, the closed-loop system $\frac{PC}{1+PC}$ will approximately behave like the poles of $P(s)$ at small $K$s and more like the zeros of $P(s)$ at large $K$s.

- this will not be on the exam


```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
import math

import plot_ball as p1
%matplotlib inline
```

## Andrew Malfavon Exercise 5.9

Plot a formula for the trajectory of a ball

$$y(t) = v_ot - \frac{1}{2} gt^2$$

```python
p1.plot_here()
```


```
# This script generates the equations of motion for a double pendulum where the
# bob rotates about the pendulum rod. It can be shown to be chaotic when
# simulated.

# import sympy and the mechanics module
import sympy as sym
import sympy.physics.mechanics as me
```

```
# declare the constants #
# gravity
gravity = sym.symbols('g')

# center of mass length, mass and  moment of inertia of the slender rod
lA, mA, IAxx = sym.symbols('lA mA IAxx')

# center of mass length, mass and moment of inertia of the plate
lB, mB, IBxx, IByy, IBzz = sym.symbols('lB mB IBxx IByy IBzz')
```

```
## kinematics ##

# declare the coordinates and speeds and their derivatives #
# theta : angle of the rod
# phi : angle of the plate relative to the rod
# omega : angular speed of the rod
# alpha : angular speed of the plate
theta, phi, omega, alpha = me.dynamicsymbols('theta phi omega alpha')
thetad, phid, omegad, alphad = me.dynamicsymbols('theta phi omega alpha', 1)
```

```
# reference frames #
# create a Newtonian reference frame
N = me.ReferenceFrame('N')
# create a reference for the rod, A, and the plate, B
A = me.ReferenceFrame('A')
B = me.ReferenceFrame('B')
```

```
# orientations #
# the rod rotates with respect to the Newtonian reference frame about the 2 axis
A.orient(N, 'Axis', [theta, N.y])

# the plate rotates about the rod's primay axis
B.orient(A, 'Axis', [phi, A.z])
```

```
# positions #
# origin of the Newtonian reference frame
No = me.Point('No')
# create a point for the mass centers of the two bodies
Ao = me.Point('Ao')
Bo = me.Point('Bo')
# define the positions of the mass centers relative to the Newtonian origin
Ao.set_pos(No, lA * A.z)
Bo.set_pos(No, lB * A.z)

# angular velocities and accelerations #
A.set_ang_vel(N, omega * N.y)
B.set_ang_vel(A, alpha * A.z)

# take the derivative of the angular velocities to get angular accelerations
A.set_ang_acc(N, A.ang_vel_in(N).dt(N))
B.set_ang_acc(N, B.ang_vel_in(N).dt(N))

# linear velocities and accelerations #
No.set_vel(N, 0) # the newtonian origin is fixed
Ao.set_vel(N, omega * lA * A.x)
Ao.a2pt_theory(No, N, A)
Bo.set_vel(N, omega * lB * A.x)
Bo.a2pt_theory(No, N, A)

# kinematical differential equations #
kinDiffs = [omega - thetad, alpha - phid]
```

```
## kinetics ##

# rigid bodies #
# create the empty rod object
rod = me.RigidBody('rod', Ao, A, mA, (me.inertia(A, IAxx, IAxx, 0.0), Ao))
# create the empty plate object
plate = me.RigidBody('plate', Bo, B, mB, (me.inertia(B, IBxx, IByy, IBzz), Bo))
```

```
# forces #
# add the gravitional force to each body
rodGravity = (Ao, N.z * gravity * mA)
plateGravity = (Bo, N.z * gravity * mB)
```

```
## equations of motion with Kane's method ##

# make a list of the bodies and forces
bodyList = [rod, plate]
forceList = [rodGravity, plateGravity]

# create a Kane object with respect to the Newtonian reference frame
kane = me.KanesMethod(N, q_ind=[theta, phi], u_ind=[omega, alpha],
        kd_eqs=kinDiffs)

# calculate Kane's equations
fr, frstar = kane.kanes_equations(forceList, bodyList)
zero = fr + frstar

# solve Kane's equations for the derivatives of the speeds
eom = sym.solvers.solve(zero, omegad, alphad)

# add the kinematical differential equations to get the equations of motion
eom.update(kane.kindiffdict())

# print the results
me.mpprint(eom)
```

    ⎧                   2
    ⎪    (IBxx - IByy)⋅ω (t)⋅sin(2⋅φ)      -IBxx⋅α⋅ω⋅sin(2⋅φ) + IByy⋅α⋅ω⋅sin(2⋅φ)
    ⎨α̇: ────────────────────────────, ω̇: ───────────────────────────────────────
    ⎪               2⋅IBzz                                          2
    ⎩                                                IAxx + IBxx⋅sin (φ) + IByy⋅co

                                                   ⎫
    - g⋅lA⋅mA⋅sin(θ) - g⋅lB⋅mB⋅sin(θ)              ⎪
    ─────────────────────────────────, φ̇: α, θ̇: ω⎬
     2        2        2                           ⎪
    s (φ) + lA ⋅mA + lB ⋅mB                        ⎭

```
me.KanesMethod?
```

```
me.mpprint(sym.solvers.solve(zero, omegad, alphad))
```

    ⎧                   2
    ⎪    (IBxx - IByy)⋅ω (t)⋅sin(2⋅φ)      -IBxx⋅α⋅ω⋅sin(2⋅φ) + IByy⋅α⋅ω⋅sin(2⋅φ)
    ⎨α̇: ────────────────────────────, ω̇: ───────────────────────────────────────
    ⎪               2⋅IBzz                                          2
    ⎩                                                IAxx + IBxx⋅sin (φ) + IByy⋅co

                                     ⎫
    - g⋅lA⋅mA⋅sin(θ) - g⋅lB⋅mB⋅sin(θ)⎪
    ─────────────────────────────────⎬
     2        2        2             ⎪
    s (φ) + lA ⋅mA + lB ⋅mB          ⎭

```
from sympy import init_printing
init_printing()
```

```
sym.solvers.solve(zero, omegad, alphad)
```

```
eom
```

```

```


## Quantum Mechanics: Variational energy on 4 levels of particle in the box

In Levine's *Quantum Chemistry, 7th Ed*, page 24 notes that the exact solutions to the one-dimensional particle in a box are the following:

$$ \psi_n = \biggl(\frac{2}{l}\biggr)^{\frac{1}{2}} \sin\biggl(\frac{n \pi x}{l}\biggr) $$

$$ E_n = \frac{n^2 h^2}{8 m l^2} $$

$$ n = 1, 2, 3, \dots $$

with the Hamiltomian for the system as:

$$ \hat{H} = -\frac{\hbar^2}{2 m} \frac{d^2}{dx^2} $$

On page 212, there is an example of using the linear variational principle and trial wavefunctions to approximate the first 4 energy levels. This post works through that example using Python and SymPy.

The basis set of functions are:

$$ f_1 = x (l - x) $$
$$ f_2 = x^2 (l-x)^2 $$
$$ f_3 = x(l -x)\biggl(\frac{1}{2}l - x\biggr) $$
$$ f_4 = x^2 (l-x)^2\biggl(\frac{1}{2}l - x\biggr) $$

which are combined into trial functions of:

$$ \phi_1 = c_{1}^{(1)} f_1 + c_{2}^{(1)} f_2 $$
$$ \phi_2 = c_{3}^{(2)} f_3 + c_{4}^{(2)} f_4 $$
$$ \phi_3 = c_{1}^{(3)} f_1 + c_{2}^{(3)} f_2 $$
$$ \phi_4 = c_{3}^{(4)} f_3 + c_{4}^{(4)} f_4 $$

Because f1 and f2 are even, whereas f3 and f4 are odd, many integrals will vanish which simplifies the secular determinants that need to be solved.

To start, import the needed parts of SymPy and define the symbols that we will use with SymPy in its operations.

```python
from sympy import symbols, diff, integrate
from sympy.matrices import Matrix
from sympy.solvers import solve
from math import pi

x, l, h_bar, m, W, h, k, c_1_2 = symbols('x l h_bar m W h k c_1_2')
```

## Part 1: Solve the integrals

A full derivation for the secular determinants are in Levine, but the bottom line is that we need to solve the following two determinants to find \\(W_i\\):

$$
    \begin{vmatrix}
    H_{11}-S_{11}W & H_{12}-S_{12}W \\
    H_{21}-S_{21}W & H_{22}-S_{22}W
    \end{vmatrix}
    = 0
$$

$$
    \begin{vmatrix}
    H_{33}-S_{33}W & H_{34}-S_{34}W \\
    H_{43}-S_{43}W & H_{44}-S_{44}W
    \end{vmatrix}
    = 0
$$

### Evaluate H integrals

$$ H_{ij} = \int_{0}^{l} f_i^{\star} \hat{H} f_j dx $$

```python
H11 = (-h_bar**2 / (2*m)) * integrate(x*(l-x) * diff(x*(l-x), x, 2), (x, 0, l))
H11
```

$\displaystyle \frac{h_{bar}^{2} l^{3}}{6 m}$

```python
H12 = (-h_bar**2 / (2*m)) * integrate(x*(l-x) * diff(x**2 * (l-x)**2, x, 2), (x, 0, l))
H12
```

$\displaystyle \frac{h_{bar}^{2} l^{5}}{30 m}$

```python
H21 = (-h_bar**2 / (2*m)) * integrate(x**2 * (l-x)**2 * diff(x*(l-x), x, 2), (x, 0, l))
H21
```

$\displaystyle \frac{h_{bar}^{2} l^{5}}{30 m}$

```python
H22 = (-h_bar**2 / (2*m)) * integrate(x**2 * (l-x)**2 * diff(x**2 * (l-x)**2, x, 2), (x, 0, l))
H22
```

$\displaystyle \frac{h_{bar}^{2} l^{7}}{105 m}$

```python
H33 = (-h_bar**2 / (2*m)) * integrate(x*(l-x)*(0.5*l-x) * diff(x*(l-x)*(0.5*l-x), x, 2), (x, 0, l))
H33
```

$\displaystyle \frac{0.0249999999999999 h_{bar}^{2} l^{5}}{m}$

```python
H34 = (-h_bar**2 / (2*m)) * integrate(x*(l-x)*(0.5*l-x) * diff(x**2 * (l-x)**2 * (0.5*l-x), x, 2), (x, 0, l))
H34
```

$\displaystyle \frac{0.003571428571429 h_{bar}^{2} l^{7}}{m}$

```python
H43 = (-h_bar**2 / (2*m)) * integrate(x**2 * (l-x)**2 * (0.5*l-x) * diff(x*(l-x) * (0.5*l-x), x, 2), (x, 0, l))
H43
```

$\displaystyle \frac{0.003571428571429 h_{bar}^{2} l^{7}}{m}$

```python
H43 = (-h_bar**2 / (2*m)) * integrate(x**2 * (l-x)**2 * (0.5*l-x) * diff(x * (l-x) * (0.5*l-x), x, 2), (x, 0, l))
H43
```

$\displaystyle \frac{0.003571428571429 h_{bar}^{2} l^{7}}{m}$

```python
H44 = (-h_bar**2 / (2*m)) * integrate(x**2 * (l-x)**2 * (0.5*l-x) * diff(x**2 * (l-x)**2 * (0.5*l-x), x, 2), (x, 0, l))
H44
```

$\displaystyle \frac{0.000793650793649903 h_{bar}^{2} l^{9}}{m}$

## Evaluate S integrals

$$ S_{ij} = \int_{0}^{l} f_i^{\star}f_j dx $$

Since all our functions are real, we don't need to worry about the complex conjugate. These makes our integrals to find even simpler:

$$ S_{ij} = S_{ji} = \int_{0}^{l} f_if_j dx $$

```python
S11 = integrate((x *(l-x))**2, (x, 0, l))
S11
```

$\displaystyle \frac{l^{5}}{30}$

```python
S12 = integrate((x * (l-x)) * (x**2 * (l-x)**2), (x, 0, l))
S21 = S12
S12
```

$\displaystyle \frac{l^{7}}{140}$

```python
S22 = integrate((x**2 * (l-x)**2)**2, (x, 0, l))
S22
```

$\displaystyle \frac{l^{9}}{630}$

```python
S33 = integrate((x*(l-x)*(0.5*l-x))**2, (x, 0, l))
S33
```

$\displaystyle 0.0011904761904763 l^{7}$

```python
S34 = integrate((x*(l-x)*(0.5*l-x)) * (x**2*(l-x)**2*(0.5*l-x)), (x, 0, l))
S43 = S34
S34
```

$\displaystyle 0.000198412698412753 l^{9}$

```python
S44 = integrate((x**2*(l-x)**2*(0.5*l-x))**2, (x, 0, l))
S44
```

$\displaystyle 3.60750360750006 \cdot 10^{-5} l^{11}$

### Matrices and Determinants

Now we can plug our \\(H_{ij}\\) and \\(S_{ij}\\) into matrices to find deterimants and solve for \\(W_i\\)

```python
M1 = Matrix([[H11-S11*W, H12-S12*W], [H21 - S21*W, H22-S22*W]])
M1
```

$\displaystyle \left[\begin{matrix}- \frac{W l^{5}}{30} + \frac{h_{bar}^{2} l^{3}}{6 m} & - \frac{W l^{7}}{140} + \frac{h_{bar}^{2} l^{5}}{30 m}\\- \frac{W l^{7}}{140} + \frac{h_{bar}^{2} l^{5}}{30 m} & - \frac{W l^{9}}{630} + \frac{h_{bar}^{2} l^{7}}{105 m}\end{matrix}\right]$

```python
M2 = Matrix([[H33-S33*W, H34-S34*W], [H43-S43*W, H44-S44*W]])
M2
```

$\displaystyle \left[\begin{matrix}- 0.0011904761904763 W l^{7} + \frac{0.0249999999999999 h_{bar}^{2} l^{5}}{m} & - 0.000198412698412753 W l^{9} + \frac{0.003571428571429 h_{bar}^{2} l^{7}}{m}\\- 0.000198412698412753 W l^{9} + \frac{0.003571428571429 h_{bar}^{2} l^{7}}{m} & - 3.60750360750006 \cdot 10^{-5} W l^{11} + \frac{0.000793650793649903 h_{bar}^{2} l^{9}}{m}\end{matrix}\right]$

```python
M1.det()
```

$\displaystyle \frac{W^{2} l^{14} m^{2} - 56 W h_{bar}^{2} l^{12} m + 252 h_{bar}^{4} l^{10}}{529200 m^{2}}$

```python
M2.det()
```

$\displaystyle \frac{1.0 \left(3.57887262643157 \cdot 10^{-9} W^{2} l^{18} m^{2} - 4.29464715176573 \cdot 10^{-7} W h_{bar}^{2} l^{16} m + 7.08616780042809 \cdot 10^{-6} h_{bar}^{4} l^{14}\right)}{m^{2}}$

### Solve the determinants

M1, the first determinant, is solved for the f1 and f2 functions in the trial functions for phi1 and phi3. Therefore, these trial functions mean its two roots are W1 and W3. Similarly, M2 is for trial function phi2 and phi4, so its two roots are W2 and W4.

```python
W1, W3 = solve(M1.det(), W)
W2, W4 = solve(M2.det(), W)
```

```python
W1
```

$\displaystyle \frac{2 h_{bar}^{2} \left(14 - \sqrt{133}\right)}{l^{2} m}$

```python
W2
```

$\displaystyle \frac{19.7507764050006 h_{bar}^{2}}{l^{2} m}$

```python
W3
```

$\displaystyle \frac{2 h_{bar}^{2} \left(\sqrt{133} + 14\right)}{l^{2} m}$

```python
W4
```

$\displaystyle \frac{100.249223596336 h_{bar}^{2}}{l^{2} m}$

## Exact versus variational solutions

Let's compare the exact one-dimensional particle in a box with the variational energies. Here, I depart from the book to find numeric solutions by setting \\(m=1\\) and \\(l=1\\).

First, I make a couple of functions to calculate floating-point values from the exact particle in box energy and simplifying the symbolic values of W  calculated above.

```python
def pib_E(n, m_pib=1, l_pib=1):
    return n**2 * (6.626e-34)**2 / 8. / m_pib / l_pib**2

def eval_W(symbol):
    return symbol.subs({h_bar: 1.055e-34, m: 1., l: 1.}).evalf()
```

```python
W1_evalf = eval_W(W1)
W2_evalf = eval_W(W2)
W3_evalf = eval_W(W3)
W4_evalf = eval_W(W4)
```

Then, I compare the actual energies of the particle in a box with the values from the variational method. The actual values are always lower than the variational values, as one expects for the variational method. Thus, the variational values range from good to poor fit.

```python
(pib_E(1) / W1_evalf - 1.) * 100
```

$\displaystyle -0.0847413438686662$

```python
(pib_E(2) / W2_evalf - 1.) * 100
```

$\displaystyle -0.141790270538589$

```python
(pib_E(3) / W3_evalf - 1.) * 100
```

$\displaystyle -13.098741620058$

```python
(pib_E(4) / W4_evalf - 1.) * 100
```

$\displaystyle -21.305039507864$

# WTF