# Fun with continued fractions and Taylor approximations

A Taylor series of a function is kind of like a decimal representation of a real number, where instead of base 10 we use "base x" to represent functions of x.

What happens if we replace "decimal representation" with "[continued fraction](https://en.wikipedia.org/wiki/Continued_fraction) representation"?

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Say we want to approximate y = log(1 + x) near x = 0. Here's a graph of this function:

```python
x = np.linspace(-0.5, 0.5)
y = np.log1p(x)

plt.plot(x, y)
plt.axhline(0, color='black')
plt.axvline(0, color='black')
```

Our first order approximation is y = x. This matches the slope and intercept at x = 0:

```python
plt.plot(x, y)
plt.plot(x, x)
plt.axhline(0, color='black')
plt.axvline(0, color='black')
```

Suppose now we want more terms in the Taylor approximation. We could take derivatives and plug it into the general formula but here's a more heuristic approach (which we will modify later).

If $ y \approx x $, then $ y/x \approx 1 $. Define $z_1 = y/x$ and plot it as a function of x. We find that we can get a better approximation for $z_1$ by matching *its* slope at x = 0:

```python
plt.plot(x, y/x)
plt.plot(x, 1 - x/2)
plt.axvline(0, color='black')
```

Working backward this gives us a second order Taylor approximation.

$$ \begin{align}
y/x &= z_1 \approx 1 - x/2 \\
y &= xz_1 \\
y &\approx x(1 - x/2) \\
&\approx x - x^2/2
\end{align} $$

```python
plt.plot(x, y)
plt.plot(x, x - x*x/2)
plt.axhline(0, color='black')
plt.axvline(0, color='black')
```

We can keep going to get the third term.

$$ \begin{align}
y &\approx x + x^2(-1/2) \\
y &= x + x^2 z_2 \\
y/x &= 1 + xz_2 \\
y/x - 1 &= xz_2 \\
(y/x - 1)/x &= z_2
\end{align} $$

so we plot (y/x - 1)/x and once again match its slope

```python
plt.plot(x, (y/x-1)/x)
plt.plot(x, -1/2 + x/3)
plt.axvline(0, color='black')
```

and plug it back in to get the third order Taylor expansion

$$ \begin{align}
y &= x + x^2 z_2 \\
y &\approx x + x^2 (-1/2 + x/3) \\
&= x - x^2/2 + x^3/3
\end{align} $$

```python
plt.plot(x, y)
plt.plot(x, x - x*x/2 + x*x*x/3)
plt.axhline(0, color='black')
plt.axvline(0, color='black')
```

And we can keep going like this to get more and more terms in the Taylor series.

To summarize what we've been doing so far:

* Approximate a function as constant + linear in x
* Subtract out the constant and divide what's left by x, to get something approximately constant
* Replace "constant" with "constant + linear" and repeat

## Enter continued fractions

Now to this basic procedure we're going to make one change:

* Approximate a function as constant + linear in x
* Subtract out the constant and **divide x by what's left**, to get something approximately constant
* Replace "constant" with "constant + linear" and repeat

This is inspired by continued fractions as it's analogous to how you would generate a continued fraction expansion for a real number.

Let's see what happens when we apply the modified procedure to our function from before. In the first step we have $y \approx x$. Last time we went from this to $y = xz_1$, now let's do $y = x/z_1$ and plot $z_1 = x/y$:

```python
plt.plot(x, x/y)
plt.plot(x, 1 + x/2)
plt.axvline(0, color='black')
```

We can approximate $z_1$ better with 1 + x/2. So

$$ \begin{align}
y &= x/z_1 \\
y &\approx x/(1 + x/2)
\end{align} $$

And that's our second order continued fraction approximation!

```python
plt.plot(x, y)
plt.plot(x, x/(1 + x/2))
plt.axhline(0, color='black')
plt.axvline(0, color='black')
```

Continuing,

$$ \begin{align}
y &\approx x/(1 + x/2) \\
y &= x/(1 + x/z_2) \\
x/y &= 1 + x/z_2 \\
x/y - 1 &= x/z_2 \\
x/(x/y - 1) &= z_2
\end{align} $$

so we plot and approximate x/(x/y - 1)

```python
plt.plot(x, x/(x/y - 1))
plt.plot(x, 2 + x/3)
plt.axvline(0, color='black')
```

and plug it back in to get the third order continued fraction approximation

$$ \begin{align}
y &= x/(1 + x/z_2) \\
y &\approx x/(1 + x/(2 + x/3))
\end{align} $$

```python
plt.plot(x, y)
plt.plot(x, x/(1 + x/(2 + x/3)))
plt.axhline(0, color='black')
plt.axvline(0, color='black')
```

It seems to work pretty well! I don't have a proof of convergence or anything, but for this example, it seems to produce better approximations than the same order Taylor series.

Here's a bigger plot of the second order Taylor and continued fraction approximations, together:

```python
plt.figure(figsize=(15, 10))
plt.plot(x, x - x*x/2, color='#ff7f00', label="2nd order Taylor approximation")
plt.plot(x, x/(1 + x/2), color='#00cc00', label="2nd order continued fraction approximation")
plt.plot(x, y, label="y = ln(1 + x)")
plt.axhline(0, color='black')
plt.axvline(0, color='black')
plt.legend()
```

And the relative error of the second order and third order approximations. In both cases, continued fraction (green) beats Taylor expansion (orange).

```python
x = np.linspace(-0.5, 0.5, 200)
y = np.log1p(x)

plt.figure(figsize=(15, 10))
plt.plot(x, np.abs((x - x*x/2)/y-1), color='#ff7f00', label="2nd order Taylor approximation")
plt.plot(x, np.abs((x/(1 + x/2))/y-1), color='#00cc00', label="2nd order continued fraction approximation")
plt.ylabel('relative error')
plt.yscale('log')
plt.axhline(0, color='black')
plt.axvline(0, color='black')
plt.legend()

plt.figure(figsize=(15, 10))
plt.plot(x, np.abs((x - x*x/2 + x*x*x/3)/y-1), color='#ff7f00', label="3rd order Taylor approximation")
plt.plot(x, np.abs((x/(1 + x/(2 + x/3)))/y-1), color='#00cc00', label="3rd order continued fraction approximation")
plt.ylabel('relative error')
plt.yscale('log')
plt.axhline(0, color='black')
plt.axvline(0, color='black')
plt.legend()
```

We can do this for other functions too! Here's y = e^x

```python
x = np.linspace(-1, 1)
y = np.exp(x)
plt.figure(figsize=(15, 10))
plt.plot(x, y, label="y = e^x")
plt.plot(x, 1 + x/(1 - x/2), label="2nd order continued fraction approximation")
plt.plot(x, 1 + x/(1 - x/(2 + x/3)), label="3rd order continued fraction approximation")
plt.axvline(0, color='black')
plt.legend()
```

And y = sin(x)

```python
x = np.linspace(-2, 2)
y = np.sin(x)
plt.figure(figsize=(15, 10))
plt.plot(x, y, label="y = sin(x)")
plt.plot(x, x, label="1st order Taylor (or continued fraction) approximation")
plt.plot(x, x/(1 + x*x/6), label="3rd order continued fraction approximation")
plt.axvline(0, color='black')
plt.legend()
```

In the case of y = sin(x), this seems to be worse than the same order Taylor approximation. Is there a theory for when this method does well?

I searched for information about this and I came across [Padé approximant](https://en.wikipedia.org/wiki/Pad%C3%A9_approximant). Does the procedure I've described compute a Padé approximant?

2018-06-28 continued rational function approximations.ipynb


## The vanilla RBM.

   * weights $W_{ji}$ means the weight between the j-th hidden unit and the i-th visible unit.
   * $W_{0i}$ is "bias" into the i-th visible unit.
   * $W_{j0}$ is "bias" into the j-th hidden unit.
The joint probability under the RBM factorisation is:
$$ P^\star(h,v) = \prod_i \prod_j e^{h_j W_{ji} v_i} \;\; \times \;\; \prod_{i^\prime} e^{W_{0i^\prime} v_{i^\prime}} \;\; \times \;\; \prod_{j^\prime} e^{W_{j^\prime 0} h_{j^\prime}} $$
and its logarithm is
$$ \log P^\star(h,v) = \sum_i  \sum_j h_j W_{ji} v_i \;\; + \;\; \sum_i W_{0i} v_i \;\; + \;\; \sum_j W_{j0} h_j $$

## Gibbs in vanilla RBM
To sample from this distribution we can figure out the Gibbs update step.

The probability $p(h_j=1|v)$ that the j-th hidden unit generates a 1 can be written as $ \sigma(\psi_j) $ with $\sigma(x)=1/(1+e^{-x})$ and $\psi_j = \log P^*(h,v | h_j =1 ) - \log P^*(h,v | h_j = 0)$.
And from the $\log P^\star(h,v)$ given above, this is easily seen to be
$ \psi_j(v) = \sum_i W_{ji} v_i + W_{j0}$

Similarly, for the visible units the probability $p(v_i=1|h)$ that the i-th visible generates a 1 can be written as
$\sigma(\phi_i(h)) $ with $\phi_i(h) = \log P^*(h,v | v_i =1 ) - \log P^*(h,v | v_i = 0)$.
And similarly this can be seen to be
$ \phi_i(h) = \sum_j W_{ji} h_j + W_{0i}$ for the i-th visible unit.

For convenience we will sometimes write $\sigma(\phi_i(h))$ as just $\sigma_i(h)$.

## Gibbs in an equivalent network

Note this is equivalent to a 3 layer network in which the top 2 layers form an RBM and the lower one is a sigmoid belief network.
Sampling from the latter model involves drawing Bernoulli variables repeatedly (eg. via alternating Gibbs sampling) from the RBM until equilibrium and then sampling $v$ from the visibles given the hiddens $h$, ie. "ancestral sampling" in the belief net, but from an $h$ generated by an RBM.

We know (from above) how to generate the $h$ sample: Gibbs sampling from the RBM will do it. Then, the conditional probability of $v$ under a sigmoid belief net is (by definition) $p(v_i=1|h) = \sigma_i(h)$.
Thus _Gibbs sampling from a simple RBM ending in a sample for $v$ is the same as sampling $h$ from the same RBM and then using a sigmoid belief net for the last step_.

However, there's another way to draw such samples. Write (product rule) $\log P^\star(h,v) = \log P^\star(h) + \log P(v|h)$. We have the second term already:
$$ \log P(v|h) = \sum_i v_i \log \sigma_i(h) + (1-v_i) \log (1 - \sigma_i(h)$$

To find $P^\star(h)$ we need to marginalise that joint over all $\mathbf{v}$ configurations:
$$\begin{align} P^\star(h) &= \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \exp \bigg[  \log P^{\star}(h,v) \bigg] \\
&= \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \exp \bigg[  \sum_i  \sum_j h_j W_{ji} v_i \;\; + \;\; \sum_i W_{0i} v_i \;\; + \;\; \sum_j W_{j0} h_j \bigg] \\
&= \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \exp \bigg[  \sum_i v_i \phi_i(h)  \;\; + \;\; \sum_j W_{j0} h_j \bigg] \\
\text{where } \phi_i(h) &= \sum_j W_{ji} h_j + W_{0i} \\
&= \exp\left[ \sum_j h_j  W_{j0} \right] \;\; \times \sum_{v_1=0}^1 \cdots \sum_{v_n=0}^1 \prod_i \exp\bigg[ v_i \phi_i(h) \bigg] \\
&= \exp\left[\sum_j h_j  W_{j0}\right] \;\; \times \prod_i \bigg( 1 + e^{\phi_i(h) } \bigg) \\
\text{and so}
\log P^\star(h) &= \sum_j h_j  W_{j0} \;\; +  \sum_i \log \bigg( 1 + e^{\phi_i(h) } \bigg)
\\
&= \sum_j h_j  W_{j0} \;\; + \; \sum_i \phi_i(h) \;  - \; \sum_i \log \sigma_i(h)
\end{align} $$

So far we've figured out $\log P^\star(h)$ for the RBM that is the "top layer".

Therefore another way to write $\log P^\star(h,v)$ is
$$
\log P^\star(h,v) = \underbrace{\sum_j h_j  W_{j0} \;\; + \; \sum_i \phi_i(h) \;  - \; \sum_i \log \sigma_i(h)}_{\log P^\star(h)} \;\;+\;\; \underbrace{\sum_i v_i \log \sigma_i(h) + (1-v_i) \log (1 - \sigma_i(h))}_{\log P(v \mid h)}
$$
By collecting terms and simplifying one can readily those that this matches the earlier form.

# a model of two causes

Now we'd like to change this slightly, so that 2 RBMs that are independent are used to model two causes, which are then combined at the last moment via a sigmoid belief net to form $v$.
Suppose that at a given moment the 2nd RBM is in a state which contributes an extra activation $\epsilon_i$ respectively to each of the visible units. We're interested in how this will affect the Gibbs updates to the hidden units $h$ in the first RBM.

Note: $\epsilon_i$ is in general going to be a weighted sum of inputs from the second RBM's hidden layer, _plus a new bias_ arising from the second RBM. Although the two biases both going into the visible units seems (and might be) redundant, in the generative model it seems sensible that visible activations under the two "causes" would have different background rates if taken separately (ie. different biases). So for now I think we should leave them in, but maybe hope to eliminate / merge if possible in future, if it helps anything...

Before going on to the Gibbs Sampler version in which visible units are clampled, consider "ancestral" sampling from the model: each RBM independently does alternating Gibbs Sampling for a (longish) period, and then both combine to generate a sample $v$ vector, by adding both their weighted sums ($\phi$) _and adding in both their visible biases too_ . That's a much more efficient way to do the "sleep" phase than doing what follows (which is mandatory for the "wake" phase samples however).

The Gibbs update step is given by $p_j = \sigma(\phi_j) $ with
$ \psi_j = \log P^*(h,v ; h_j =1 ) - \log P^*(h,v ; h_j = 0)$.
However this time we don't have exact correspondence with an RBM because only the final step involves $\epsilon$, not the reverberations in the RBM above it that generates $h$. So it's not enough to consider just the RBM alone, with it's joint being just the product of factors in the first line of math above. We need to incorporate the last step explicitly, with its slight difference in the form of $\epsilon$. We know the joint decomposes into this:
$$ \log P^* (h,v) = \log P^*(h) + \log P(v|h)$$
where the first term is the vanilla RBM probability but the second is the final layer's probability, now given by
$$ \log P(v|h,\epsilon) = \sum_i v_i \log \sigma (\phi_i(h) + \epsilon_i) + (1-v_i) \log (1 - \sigma(\phi_i(h) + \epsilon_i)$$

To carry out Gibbs sampling in the hidden layer of this architecture we need to calculate $\psi_j = \log P^*(h,v ; h_j =1 ) - \log P^*(h,v ; h_j = 0)$. Using the fact that $\phi_i(h ; h_j=1) = \phi_i(h ; h_j=0) + W_{ji}$, and abbreviating $\phi_i(h ; h_j=0)$ to $\phi_i^0$, we obtain

$$\psi_j = \sum_i v_i \log \left( \frac{1+ e^{-\phi_i^0 - \epsilon_i}}{1+e^{-\phi_i^0 - W_{ji} -\epsilon_i}} \frac{1+ e^{\phi_i^0 + W_{ji} + \epsilon}}{1+e^{\phi_i^0 + \epsilon_i}}\right) \;\;+ \;\;\sum_i \log \left(\frac{1+e^{\phi_i^0 + W_{ji}}}{1+ e^{\phi_i^0}}
\frac{1+e^{\phi_i^0 + \epsilon_i}}{1+ e^{\phi_i^0 + W_{ji} + \epsilon_i}} \right)$$

Now $\phi = \log \frac{1+e^{\phi}}{1+e^{-\phi}}$ (Marcus' magic identity), which is$ = \log \frac{\sigma(\phi)}{\sigma(-\phi)}$.
So the first term simplifies to
$ \sum_i v_i W_{ji}$, which is the same as that in a "vanilla RBM".

The second term can also be simplified, using the identity $\log(1-\sigma(\phi)) = \phi - \log(1+e^\phi)$.

This leads to the following Gibbs Sampler probability of the j-th hidden unit being 1: $p_j = \sigma(\psi_j)$ with

$$\psi_j = \sum_i (W_{ji} v_i + C_{ji}) $$
where
$$C_{ji} \; = \;\log \bigg[ \frac{\sigma (\phi_i^0)}{\sigma (\phi_i^0 + W_{ji})} . \frac{\sigma (\phi_i^0 + W_{ji} + \epsilon_i) }{\sigma (\phi_i^0 + \epsilon_i)} \bigg] $$

Note that $\sum_i C_{ji}$ can be thought of as correction to vanilla RBM Gibbs "input" to the hidden node. Weirdly, $v$ plays no role in the $C$ term!

Written another way this is
$$C_{ji} = \log \sigma(\phi_i^0)  \; +\log \sigma (\phi_i^0 + W_{ji} + \epsilon_i) \;- \log \sigma (\phi_i^0 + W_{ji})  \;- \log \sigma ( \phi_i^0  + \epsilon _i) $$

It is clear that adding the single $\epsilon$ has introduced a dependency between the whole of $h$, which is a worry.

1) What happens if $\epsilon$ feeds into $u$ as well?
2) How coupled is it really?
3) Can we just hack it?
    a) Conditionals on the size of $\phi$, $\epsilon$
    b) Truncated Markov chain
4) Linear RBM
5) Is any hidden node coupling bad?
6) How to interpret as Blind Source Separation

## What does the correction to standard RBM $\psi$ look like?

Thought: the "correction" is all hinge functions and should have an approximation as "if../else.." piecewise linear regimes, and this ought to have an intuitive hand-wave type explanation that makes sense.

So let's plot the correction contours on axes $\epsilon_i$ versus $\phi_i$, for (say) a positive $w_{ij}$.
We'll need two plots for the two cases $h_i=0,1$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import numpy.random as rng
np.set_printoptions(precision = 2)
plt.rcParams['figure.figsize'] = (10.0, 8.0)
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

def sigmoid(x):
    return 1.0/(1.0 + np.exp(-x))

def calc_psi_correction(phi, eps, w,h):
    # note here the Phi is the full weighted sum into the visible node. We're explicitly taking care of the hidden activation too.
    correction = np.log(sigmoid(phi+w+eps-h*w)) + np.log( sigmoid(phi-h*w)) - np.log( sigmoid(phi+w-h*w)) - np.log( sigmoid(phi+eps-h*w))
    return correction

def calc_psi0_correction(phi0, eps, w):
    # note here the phi0 is what the weighted sum into the visible node WOULD be IF h=0.
    correction = np.log(sigmoid(phi0+w+eps)) + np.log( sigmoid(phi0)) - np.log( sigmoid(phi0+w)) - np.log( sigmoid(phi0+eps))
    return correction

reach = 40
phi, eps = np.mgrid[-reach:reach:100j, -reach:reach:100j]
wgt = 3.0
levels = np.arange(-abs(wgt)-0.5, abs(wgt)+0.5, abs(wgt)/10.)
cmap = cm.RdYlGn

plt.subplot(121)
psi_correction_h0 = calc_psi_correction(phi, eps, wgt, 0)
C = plt.contourf(phi, eps, psi_correction_h0, levels, origin='lower', cmap=cm.get_cmap(cmap, len(levels)-1))
#plt.colorbar()
#plt.title('$h_j=0$')
plt.xlabel('$\phi_i$ (left network)')
plt.ylabel('$\phi_i$ (right network)')
plt.plot([0,0], [-reach, reach],'-k', [-reach, reach], [0,0], '-k', [-reach, reach], [reach,-reach], '-k')
plt.axis([-reach, reach, -reach, reach])
plt.axis('off')
plt.axis('equal')
plt.xticks([-reach/2,reach/2])
plt.yticks([-reach/2,reach/2])

plt.subplot(122)
psi_correction_h0 = calc_psi_correction(phi, eps, wgt, 1)
#plt.title('$h_j=1$')
C = plt.contourf(phi, eps, psi_correction_h0, levels, origin='lower', cmap=cm.get_cmap(cmap, len(levels)-1))
plt.plot([0,0], [-reach, reach],'-k', [-reach, reach], [0,0], '-k', [-reach, reach], [reach,-reach], '-k')
plt.axis('off')
plt.axis('equal')
plt.xticks([-reach/2,reach/2])
plt.yticks([-reach/2,reach/2])

plt.savefig('correction.png', dpi=200)
```

```python
def calc_APPROX_psi_correction(phi, eps, w, h):
    # note here the Phi is the full weighted sum into the visible node. We're explicitly taking care of the hidden activation too.
    correction = np.log(sigmoid(phi+w+eps-h*w)) + np.log( sigmoid(phi-h*w)) - np.log( sigmoid(phi+w-h*w)) - np.log( sigmoid(phi+eps-h*w))
    return correction

plt.subplot(121)
psi_correction_h0 = calc_APPROX_psi_correction(phi, eps, wgt, 0)
C = plt.contourf(phi, eps, psi_correction_h0, levels, origin='lower', cmap=cm.get_cmap(cmap, len(levels)-1))
#plt.colorbar()
#plt.title('$h_j=0$')
plt.xlabel('$\phi_i$ (left network)')
plt.ylabel('$\phi_i$ (right network)')
plt.plot([0,0], [-reach, reach],'-k', [-reach, reach], [0,0], '-k', [-reach, reach], [reach,-reach], '-k')
plt.axis([-reach, reach, -reach, reach])
plt.axis('off')
plt.axis('equal')
plt.xticks([-reach/2,reach/2])
plt.yticks([-reach/2,reach/2])

plt.subplot(122)
psi_correction_h0 = calc_psi_correction(phi, eps, wgt, 1)
#plt.title('$h_j=1$')
C = plt.contourf(phi, eps, psi_correction_h0, levels, origin='lower', cmap=cm.get_cmap(cmap, len(levels)-1))
plt.plot([0,0], [-reach, reach],'-k', [-reach, reach], [0,0], '-k', [-reach, reach], [reach,-reach], '-k')
plt.axis('off')
plt.axis('equal')
plt.xticks([-reach/2,reach/2])
plt.yticks([-reach/2,reach/2])

plt.savefig('correction.png', dpi=200)
```

# Approximations (trying to find a simple one)
Within its "wedges" the correction is pretty much flat, so an approximation is just to compute true/false on the appropriate condition. The red section is where v is ON under full input from both nets, but would be OFF if it were the first alone. It's fairly closely approximated by $\sigma(\phi+\epsilon) (1-\sigma(\phi))$. And under this condition we SUBTRACT the weight from $\psi$. Similarly the green section is where it's very likely that v is OFF under full input, but would be ON if it were the first alone. Close approx to this is $(1-\sigma(\phi+\epsilon)) \sigma(\phi)$. And in this case we'd ADD in the weight to $\psi$.

Putting the green and red bits together, maybe a good approximation to the correction is just going to be something like
$$ C_{ji} =  -W_{ji} \bigg[\sigma_{\phi+\epsilon} (1-\sigma_{\phi})  - (1-\sigma_{\phi+\epsilon}) \sigma_{\phi} \bigg] $$
where hopefully the notation is obvious. But it gets better: that's actually just
$$ C_{ji} =  W_{ji} \bigg[\sigma_{\phi}- \sigma_{\phi+\epsilon} \bigg]$$

And a nice feature of this is that it's just a multiplication by $W$, just like the vanilla part was, which means our correction can be thought of as equivalently a correction to $v_i$, the activity of the visible unit.

## A better approximation
The true correction is composed of the sum of two terms of form $\log(\sigma(\phi)/\sigma(\phi+W))$.

The first term is exactly $\log(\sigma(\phi)/\sigma(\phi+W))$, which goes from being $-W$ at $\phi=-\infty$ to  $0$ at $\phi=+\infty$. The second term is $\log(\sigma(\phi+\epsilon+W)/\sigma(\phi+\epsilon))$, which goes from being $W$ at $\phi=-\infty$ to  $0$ at $\phi=+\infty$.

Each of these is "roughly sigmoid", so my approximation is going to use a sigmoid in its place.
Focussing on the 1st term, the best sigmoid has it's "switching point" at $\phi = -W/2$.
It should also have a "gain" of $\alpha = (4/W)*(2\sigma(W/2)-1)$ if you want its slope at the switching point to match that of the true correction.

The 2nd term has switching point at $\phi^A = -\phi^B - W/2$, and so the whole approximation is
$$
\tilde{C} = W \bigg[ \sigma(\phi^{A0} + W/2) \;\; - \;\; \sigma(\phi^{A0} + W/2 + \phi^B) \bigg]
$$

And again, the fact that it's got a multiplication by $W$ means our correction can be thought of as equivalently a correction to $v_i$, the activity of the visible unit. So, filling in all the tedious super and subscripts...

The Gibbs update for this hopefully improved approximation is:
$$
\begin{align}
\psi_j &= \sum_i W_{ji} \bigg( v_i - \sigma_i^{AB} \;\;+\;\; \sigma_i^A \bigg) \\
\text{where}\;\; \sigma_i^{AB} &= \sigma(\alpha \times (\phi_i^{A0} + W_{ji}/2 +\phi_i^B)) & \text{ie. both nets}
\\
\sigma_i^A &= \sigma(\alpha \times (\phi_i^{A0} + W_{ji}/2) & \text{ie. one net}
\end{align}
$$
This seems very intuitive to me - it's almost literally "explaining away". If it works, that's a really nice story to tell.

"Ideally" by some measure the sigmoids here should also use the suggested "gain" of $\alpha$. But that's probably not completely crucial...?

```python
# testing the improved approximation
W = -10.0
alpha = 4*(2*sigmoid(W/2)-1)/W # this is the scaler that would match the slope at the 'midpoint'
phi = np.linspace(-30, 30, 1001)
plt.plot(phi, np.log(sigmoid(phi)/sigmoid(phi+W)),'k')
plt.plot(phi, -W*sigmoid(-alpha*(phi+W/2)),'b')
print(alpha)
```

## ELEPHANT IN THE ROOM

what a pain it is that, in order to update $h_j^A$, we need to find the input to each visible unit _in the case that $h_j^A = 0$. That really sucks. It'd be way way cooler if we could somehow just use the existing input to the visible unit, and perhaps account for the fact that $h_j^A \neq 0$ somehow..._

THIS IS A BIG FAT TODO.

## this looks like a pig, but perhaps can be found locally...

Building in the effect of the ACTUAL hidden unit activity $h_j^A$, we have that
$$
\begin{align}
\psi_j &= \psi_j^\text{RBM} \;\; + \;\; \sum_i W^A_{ji} \bigg[ \sigma(\phi^A_i - \hat{h}^A_j W^A_{ji}) - \sigma(\phi^{AB}_i - \hat{h}^A_j W^A_{ji}) \bigg]
\end{align}
$$
where
   * $\phi^{A}_i $ is the actual input to visible unit $v_i$ arising from the $A$ net alone
   * $\phi^{AB}_i = \phi^{A}_i + \phi^{B}_i $ is the total input from both nets
   * $\hat{h}^A_j = h^A_j - 1/2$

The above was ignoring the "optimal gain" mentioned earlier.
Let's denote the optimal gain as
$\omega = 8\,(\sigma(W/2)-0.5)/W$:

$$
\begin{align}\psi_j &= \psi_j^\text{RBM} \;\; + \;\; \sum_i  W^A_{ji} \bigg[ \sigma\bigg(\phi^A_i \omega^A_{ji} \, - \, \hat{h}^A_j \omega^A_{ji} W^A_{ji}\bigg) \; - \; \sigma\bigg(\phi^{AB}_i \omega^A_{ji} \, - \, \hat{h}^A_j \omega^A_{ji} W^A_{ji} \bigg) \bigg]
\end{align}
$$

```python
# testing
phi = np.linspace(-3, 3, 1001)
plt.plot(phi, sigmoid(phi),'k')
plt.plot(phi, phi/4.+.5,'r')
#plt.axis('equal')
```

```python
# Now to put the "eps" term in too, to see the entire approximation
eps = 10.0
truth = np.log(sigmoid(phi)/sigmoid(phi+W)) - np.log(sigmoid(phi+eps)/sigmoid(phi+W+eps))
approx = -W*(sigmoid(-alpha*(phi+W/2)) - sigmoid(-alpha*(phi+eps+W/2))   )
plt.plot(phi, truth,'k', phi, approx,'b')
```

# The learning algorithm
Stochastic ascent of the log likelihood.

SWITCHING NOTATION HERE IS A PAIN, BUT IT BECOMES MORE STRAIGHT-UP TO DITCH EPSILON AND INSTEAD PUT A/B SUPERSCIPTS ON PHI AND H, TO DENOTE THE TWO NETWORKS. NEED TO MAKE A DECISION ON WHICH IS BETTER AND USE IT EVERYWHERE I GUESS.

Handy shorthands:
   * $\phi_i^A = \sum_j h_j^A W_{ji}$, and similarly for $B$
   * $\phi_i^\text{AB} =  \phi_i^A + \phi_i^B$

We have that
$$
\log P^\star(h^A, h^B, v) = \log P^\star(h^A) + \log P^\star(h^B) + \log P^\star(v \mid h^A, h^B)
$$

The first term is:
$ \log P^\star(h^A) = \sum_i \log (1 + e^{\phi^A_i}) = -\sum_i \log (\sigma(\phi^A_i))$

Second is the same...

Third is
$\sum_i v_i\log \sigma(\phi_i^\text{AB})  \; + \; (1-v_i) \log \sigma(- \phi_i^\text{AB}) $

So now we differentiate it w.r.t. some particular weight $W_{ml}^A$ like this:

First term:
$\frac{\partial}{\partial W_{ml}^A} \log P^\star(h^A)  = \sigma(\phi^A_l) h_m^A$ is the first term, which is just the average of the usual RBM Hebbian change.

The second term: is zero.

Third term:

$\frac{\partial}{\partial W_{ml}^A} \log P^\star(v \mid h^A, h^B)   = (v_l-\sigma(\phi_l^\text{AB})) h_m^A$
which is "just the perceptron rool".

## so the learning algorithm is...

### WAKE PHASE

use our Gibbs chain to get samples from $h$ for each $v \in \mathcal{D}$, and do

$\Delta W_{ji}^A \;\;\; \propto \;\;\; \underbrace{\sigma(\phi^A_i) h_j^A}_\text{mean field Hebbian} \; + \; \underbrace{(v_i - \sigma(\phi_i^\text{AB})) h_j^A}_\text{Perceptron learning rule}$

and similarly for the other RBM. Note that the $\phi$'s are not the same in the two terms: the first only sums input from the A side, but the second sums from both hidden layers.

Another way to say the same thing would be

$\Delta W_{ji}^A \;\;\; \propto \;\;\; \big[ v_i \, + \, \sigma(\phi^A_i) \, - \,\sigma(\phi_i^\text{AB}) \big] h_j^A$

which is like a Hebbian update but with a modified visible activation, in effect.

_Q: What do we make of the resemblance between this and the approximation, below?_

### SLEEP PHASE
use "regular" Gibbs sampling _in each model separately_ to sample independently from the two RBMs, and do

$\Delta W_{ji}^A \;\;\; \propto \;\;\; - \sigma( \phi^{A}_i) \; h_j^A$

and similarly for the other RBM.

### notice the free phase is free
In the FREE PHASE the expected $\psi$ is simply:
$$\bar{\psi}_j = \sum_i W_{ji} \sigma_{\phi}$$
Does that mean that the free phase in this model corresponds to what it would be if the two RBMs were not even connected?
Looks like it (although this is just an approximation, isn't it?).
BUT OF COURSE IT DOES! In the "unrolled" version of the net, the visibles are not clamped in the free phase and hence there's no explaining away going on between the two RBMs: they're independent RBMs doing their own thing, in the free phase. It's *only the wake phase that needs any update to the vanilla Gibbs* $\psi$.

Thought: maybe even the wake phase could be implemented via samples instead of having to calculate and propogate those sigmoid floats? THINK ABOUT THIS SOME MORE.

```python
import sympy as sp
x,y,z = sp.symbols('x y z')
sp.simplify(sp.log( (1+sp.exp(-x-y)) * (1+sp.exp(x+y+z)) / (1+sp.exp(-x-y-z)) / (1+sp.exp(x+y))) )
sp.simplify(sp.log( (1+sp.exp(x)) * (1+sp.exp(x+y+z)) / (1+sp.exp(x+y)) / (1+sp.exp(x+z)) ))
```


# Fourier Transform

## Polynomials
- How to multiply two polynomials
- Brute force algorithm (multiplying all terms together): O(n * m) for polynomials with degrees n-1 and m-1

## Representation
- Coefficient representation vs. point-value representation
- Multiplying polynomials in point-value representation is much easier:
  - (fg)(x) = f(x)g(x)

# Discrete Fourier Transform
Convert from coefficient representation to point-value representation
- $\mathcal{O}(n \lg n)$ runtime
- Evaluate a polynomial of degree n - 1 at n points to find its point-value representation
- Choose these points carefully

# Definitions
Let $\mathbf{a} = [a_0, a_1, ..., a_{n-1}]$ be the sequence of coefficients of a polynomial $P$ with degree $n-1$ and $\mathbf{w} = [w_0, w_1, ..., w_{n-1}]$, $w_j \in \mathbb{C}$. Then the discrete Fourier transform of $P$ gives the set of point-values $[b_0, b_1, ..., b_{n-1}]$, where each $b_j$ is given by
\begin{align}
b_j = P(w_j) = \sum_{k=0}^{n-1} a_k w_j^k
\end{align}

Written in matrix form, we have
\begin{align}
\begin{bmatrix}
1 & w_0 & w_0^2 & ... & w_0^{n-1} \\
1 & w_1 & w_1^2 & ... & w_1^{n-1} \\
1 & w_2 & w_2^2 & ... & w_2^{n-1} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
1 & w_{n-1} & w_{n-1}^2 & ... & w_{n-1}^{n-1} \\
\end{bmatrix}
\begin{bmatrix}
a_0 \\
a_1 \\
a_2 \\
\vdots \\
a_{n-1} \\
\end{bmatrix}
 = &
\begin{bmatrix}
b_0 \\
b_1 \\
b_2 \\
\vdots \\
b_{n-1} \\
\end{bmatrix}
\end{align}

The fast Fourier transform algorithm calculates each $b_j$. We will show that this algorithm runs in $\mathcal{O} (n \log{} n)$ when $n$ is a power of 2. To do so requires a selecting a special set of points $\mathbf{w}$ called the $N^{th}$ roots of unity.

# $N^{th}$ Roots of Unity
## Definition
The $N^{th}$ roots of unity are the set of complex numbers
\begin{align}
\{ \; e^{2\pi i \frac{j}{N}} \; \mid \; j = 0, 1, \dots, N-1 \}
\end{align}
For example, the $5^{th}$ roots of unity:

## Notation
When $\mathbf{w}$ is the $n^{th}$ roots of unity, we will refer to $\mathbf{w}$ as $w_n$. The $j^{th}$ element (previously $w_j$) is indicated using the array index notation, $w_n[j]$.

## Properties
Let $w_n[j] = e^{2\pi i j/n}$. Then $w_n[j]$ is said to be an $n^{th}$ root of unity and has the following properties:
1. $w_n[j]^k = w_n[jk]$
2. $w_n[j] w_n[k] = w_n[j + k]$
3. $w_n[j]^n = 1$
4. If $n = 2^r$, then $w_n[2j] = w_{\frac{n}{2}}[j]$

*Proof of 4:*

\begin{align}
w_n[2j] & = (e^{2 \pi i (2j)/2^r}) \\
& = e^{2 \pi i j/2^{r-1}} \\
& = w_{\frac{n}{2}}[j] \\
\end{align}

# Fast Fourier Transform
We are now ready to derive a recursive algorithm for the DFT.

\begin{align}
F(a, w_n[j]) = \sum_{k=0}^{n-1} a[k] w_n[j]^k
\end{align}

If we split the righthand side into two summations, one over even indices of $a$, the other over the odd, we get

\begin{align}
\sum_{k=0}^{n-1} a[k] w_n[j]^k & = \sum_{m=0}^{\frac{n}{2}-1} a[2m] w_n[j]^{2m} + \sum_{m=0}^{\frac{n}{2}-1} a[2m+1] w_n[j]^{2m+1}
\end{align}

From properties 1. and 4., we can clearly see that the lefthand summation is $F(a_{even}, w_{\frac{n}{2}}[j])$. The righthand side requires a bit more manipulation:

\begin{align}
\sum_{m=0}^{\frac{n}{2}-1} a[2m + 1] w_n[j]^{2m+1} & = \sum_{m=0}^{\frac{n}{2}-1} a[2m + 1] w_n[j(2m+1)]\\
& = \sum_{m=0}^{\frac{n}{2}-1} a[2m + 1] w_n[j2m] w_n[j] \\
& = w_n[j] \biggl( \sum_{m=0}^{\frac{n}{2}-1} a[2m + 1] w_n[2j]^m \biggr) \\
& = w_n[j] \cdot F(a_{odd}, w_{\frac{n}{2}}[j]) \\
\end{align}

Therefore, our function $F$ can be written as

\begin{align}
F(a, w_n[j]) & = F(a_{even}, w_{\frac{n}{2}}[j]) + w_n[j] \cdot F(a_{odd}, w_{\frac{n}{2}}[j]) \\
\end{align}
for $j = (0, 1, \dots, n-1)$

This recurrence returns a single value for a particular $j$, however we can modify it to instead return the Fourier transform for all values of $j$, giving us

\begin{align}
F(a, w_n) & = F(a_{even}, w_{\frac{n}{2}}) + w_n \cdot F(a_{odd}, w_{\frac{n}{2}}) \\
\end{align}

Written in Julia, the algorithm is as follows:

```julia
function FFT(n::Integer, x̄::Array{<:Number})
    """
    Calculate the fast Fourier transform of n numbers in x̄.
    (Note that n must be a power of 2 (n = 2ᵏ))
    Returns:
        ȳ - A complex array of size n
    """
    # Julia is 1-indexed
    if n == 1
        return [x̄[1]]
    end

    evens = [x̄[Int(2i)] for i = 1:n/2]
    odds = [x̄[Int(2i-1)] for i = 1:n/2]
    # Since Julia is 1-indexed, we flip the odds and evens at the recursive step
    ū = FFT(Int(n/2), odds)
    v̄ = FFT(Int(n/2), evens)
    ȳ = zeros(Complex, n)
    for j = 1:n
        τ = exp(2π*im*(j-1)/n)
        ȳ[j] = ū[(j-1)%Int(n/2)+1] + τ * v̄[(j-1)%Int(n/2)+1]
    end
    return ȳ
end
```

    FFT (generic function with 1 method)

# Time Complexity
Let $N = n$, and $T(N) = F(\mathbf{a}, w_n[j])$,
\begin{align}
T(N) = 2 \cdot T(\frac{N}{2}) + N,
\end{align}
where the $+N$ comes from the loop after the recursive step.

This algorithm runs in $\mathcal{O}(N \lg N)$ when $N$ is a power of 2.

*Proof:*
\begin{align}
T(N) & = 2 \cdot T(\frac{N}{2}) + N \\
\end{align}
Let $N = 2^k$, and $t_k = T(2^k)$
\begin{align}
t_k & = 2t_{k-1} + 2^k \\
t_k - 2t_{k-1} & = 2^k \\
\end{align}

We can use the characteristic equation to solve this inhomogeneous system.
\begin{align}
(x - 2)^2 & = 0 \\
\end{align}
So we end up with
\begin{align}
t_k & = c_1 2^k + c_2 k 2^k; k = \lg N\\
T(N) & = c_1 N + c_2 N \lg N \\
T(N) & = \mathcal{O}(N \lg N) \\
\end{align}

# Example
Say you want to perform the following multiplication:
\begin{align}
(1 + x)(1 + x + x^2).
\end{align}
We can use the FFT algorithm to find the point-value representation of each polynomial, then multiply those two together:

```julia
n = 4
x̄₁ = [1 1 0im 2]
ȳ₁ = FFT(n, x̄₁)
display(ȳ₁)
```

    4-element Array{Complex,1}:
      4.0+0.0im
      1.0-1.0im
     -2.0+3.67394e-16im
      1.0+1.0im

Calculating the FFT of each,
\begin{align}
FFT(1 + x) & = [2, 1+i, 0, 1-i] \\
FFT(1 + x + x^2) & = [3, i, 1, -i], \\
\end{align}
which leads to the point-value representation of their product: $[6, -1 + i, 0, -1-i]$.


# Lecture 2: second-order ordinary differential equations

We now look at solving second-order ordinary differential equations using a computer algebra system.

To use SymPy, we first need to import it and call `init_printing()` to get nicely typeset equations:

```python
from sympy import *

# This initialises pretty printing
init_printing()
from IPython.display import display

# This command makes plots appear inside the browser window
%matplotlib inline
```

# Mass-spring-damper system

The differential equation that governs an unforced, single degree-of-freedom mass-spring-damper system is

$$
m \frac{d^{2}y}{dt^{2}} + \lambda \frac{dy}{dt} + ky = 0
$$

To solve this problem using SymPy, we first define the symbols $t$ (time), $m$ (mass), $\lambda$ (damper coefficient) and $k$ (spring stiffness), and the function $y$ (displacement):

```python
t, m, lmbda, k = symbols("t m lambda k")
y = Function("y")
```

Note that we mis-spell $\lambda$ as `lmbda` because `lambda` is a protected keyword in Python.

Next, we define the differential equation, and print it to the screen:

```python
eqn = Eq(m*Derivative(y(t), t, t) + lmbda*Derivative(y(t), t) + k*y(t), 0)
display(eqn)
```

Checking the order of the ODE:

```python
print("This order of the ODE is: {}".format(ode_order(eqn, y(t))))
```

    This order of the ODE is: 2

and now classifying the ODE:

```python
print("Properties of the ODE are: {}".format(classify_ode(eqn)))
```

    Properties of the ODE are: ('nth_linear_constant_coeff_homogeneous', '2nd_power_series_ordinary')

we see as expected that the equation is linear, constant coefficient, homogeneous and second order.

The `dsolve` function solves the differential equation:

```python
y = dsolve(eqn, y(t))
display(y)
```

The solution looks very complicated because we have not specified values for the constants $m$, $\lambda$ and $k$. The nature of the solution depends heavily on the relative values of the coefficients, as we will see later. We have four constants because the most general case the solution is complex, with two complex constants having four real coefficients.

Note that the solution is make up of expoential functions and sinusoidal functions. This is typical of second-order ODEs.

# Second order, constant coefficient equation

We'll now solve

$$
\frac{d^{2}y}{dx^{2}} + 2 \frac{dy}{dx} - 3 y = 0
$$

The solution for this problem will appear simpler because we have concrete values for the coefficients.

Entering the differential equation:

```python
y = Function("y")
x = symbols("x")
eqn = Eq(Derivative(y(x), x, x) + 2*Derivative(y(x), x) - 3*y(x), 0)
display(eqn)
```

Solving this equation,

```python
y1 = dsolve(eqn)
display(y1)
```

which is the general solution. As expected for a second-order equation, there are two constants.

Note that the general solution is of the form

$$
y = C_{1} e^{\lambda_{1} x} + C_{2} e^{\lambda_{2} x}
$$

The constants $\lambda_{1}$ and $\lambda_{2}$ are roots of the \emph{characteristic} equation

$$
\lambda^{2} + 2\lambda - 3 = 0
$$

This quadratic equation is trivial to solve, but for completeness we'll look at how to solve it using SymPy. We first define the quadratic equation:

```python
eqn = Eq(lmbda**2 + 2*lmbda -3, 0)
display(eqn)
```

and then compute the roots:

```python
solve(eqn)
```

which as expected are the two exponents in the solution to the differential equattion.


# Numerical Recipes Workshop 6
For the week of 28 October to 1 November, 2019.

This notebook will cover some boundary value problem solving and minimization.

```python
from matplotlib import pyplot as plt
%matplotlib inline
import numpy as np
```

```python
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['font.size'] = 14
```

## Solving Boundary Value Problems

SciPy's [solve_bvp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_bvp.html) function will solve a boundary value problem given a system of ordinary differential equations (ODEs). We will consider an example from the function's documentation.

### The Bratu Problem
This is defined as:

$
\begin{align}
\large
\frac{d^{2}y}{dx^{2}} + e^{y} = 0
\end{align}
$

$
\begin{align}
\large
y(0) = y(1) = 0
\end{align}
$

The setup is similar to that of [solve_ivp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp) for initial value problems. We need to define a function that returns the derivative of the system of equations. As with the parabolic motion example, we can decompse the nth order ODE into a system of n 1st order ODEs.

$
\begin{align}
\large
\frac{dy}{dx} = v(x)
\end{align}
$

$
\begin{align}
\large
\frac{dv}{dx} = -e^{y(x)}
\end{align}
$

The difference here is that both $y$ and $v$ will be arrays instead of single values.

```python
def derivatives(x, y):
    """
    y0 = f(x)  => dy0/dx = y1
    y1 = df/dx => dy1/dx = -e^y0
    """
    return np.vstack((y[1], -np.exp(y[0])))
```

It may be helpful to look up the documentation on `np.vstack` and to stick a print statement into the `derivatives` to understand what the arguments look like.

We also need to create a function to return the relevant boundary conditions. Similar to the events functions for `solve_ivp`, the `solve_bvp` function will look for solutions where the boundary conditions function returns 0s. For a systen of n ODEs, the boundary conditions function must n values, but they can be related to any one of the equations and be for either the left or right side.

The arguments of the boundary conditions function are `ya` and `yb`, the values of the system of equations on the left side ($x = a$) and the right side ($x = b$). For the Bratu problem, the relevant boundary conditions are $y(a) = 0$ and $y(b) = 0$, where $a = 0$ and $b = 1$.

```python
def bc(ya, yb):
    """
    ya is [f(x), df/dx] on the left (x = 0)
    yb is [f(x), df/dx] on the right (x = 1)

    For this problem, we want y(0) = 0 and y(1) = 0
    """
    return np.array([ya[0], # f(x) on left side
                     yb[0]]) # f(x) on right side

```

Now define the initial x-space over which to solve the problem.

```python
x = np.linspace(0, 1, 5)
```

Create an initial guess for the solution. The array must have the shape ($N_{equations}$, $N_x$).

```python
# start with all zeroes
y = np.zeros((2, x.size))
```

Now solve the BVP. The `verbose` keyword gives some additional output. Remove or set it to 0 to get rid of the output.

```python
from scipy.integrate import solve_bvp

sol = solve_bvp(derivatives, bc, x, y, verbose = 2)
```

       Iteration    Max residual  Max BC residual  Total nodes    Nodes added
           1          1.05e-04       0.00e+00           5              0
    Solved in 1 iterations, number of nodes 5.
    Maximum relative residual: 1.05e-04
    Maximum boundary residual: 0.00e+00

### What `solve_bvp` returns

Printing the return value of `solve_bvp` (in this case `sol`) shows a complicate object with multiple components.

```python
print(sol)
```

           message: 'The algorithm converged to the desired accuracy.'
             niter: 1
                 p: None
     rms_residuals: array([9.86500717e-05, 1.05360602e-04, 1.05360602e-04, 9.86500717e-05])
               sol: <scipy.interpolate.interpolate.PPoly object at 0x15229e09b0>
            status: 0
           success: True
                 x: array([0.  , 0.25, 0.5 , 0.75, 1.  ])
                 y: array([[ 0.00000000e+00,  1.04784145e-01,  1.40534773e-01,
             1.04784145e-01,  0.00000000e+00],
           [ 5.49349275e-01,  2.84320977e-01, -1.02436237e-17,
            -2.84320977e-01, -5.49349275e-01]])
                yp: array([[ 5.49349275e-01,  2.84320977e-01, -1.02436237e-17,
            -2.84320977e-01, -5.49349275e-01],
           [-1.00000000e+00, -1.11047088e+00, -1.15088910e+00,
            -1.11047088e+00, -1.00000000e+00]])

The most important thing returned is the `sol` attribute of the return value (in this case `sol.sol`). This is a function that will return the value of the solution for given $x$ values.

Note, a similar object is returned for `solve_ivp` when the `dense_output=True` keyword is given.

```python
x_sol = np.linspace(0, 1, 100)
y_sol = sol.sol(x_sol)
# sol.sol return values for the whole system [f(x), df/dx]
plt.plot(x_sol, y_sol[0])
```

The Bratu problem has two solutions. The second can be found by changing the initial guess slightly.

```python
# start with all zeroes
y = np.zeros((2, x.size))
# small change in initial guess
y[0] = 3
```

Now, solve the BVP again, compare the output and plot the two solutions.

```python
x = np.linspace(0, 1, 10)
# start with all zeroes
y = np.zeros((2, x.size))
from scipy.integrate import solve_bvp

sol = solve_bvp(derivatives, bc, x, y,)
```

```python
x_sol = np.linspace(0, 1, 100)
y_sol = sol.sol(x_sol)
# sol.sol return values for the whole system [f(x), df/dx]
plt.plot(x_sol, y_sol[0])
```

### Heat Exchanger Problem
(from the old bvp_solver tutorial)

Consider a device where heat is exchanged between two fluids. The hot fluid (with temperature $T_1$) enters from the left, moving to the right. The cold fluid (with temperature $T_2$) enters from the right, moving to the left. They exchange heat through a metal plate in between them. The cold fluid has twice the specific heat of the hot fluid. This system can be expressed as

$
\begin{align}
\large
q = U * (T_1 - T_2)
\end{align}
$

$
\begin{align}
\large
\frac{dT_1}{dx} = -q
\end{align}
$

$
\begin{align}
\large
\frac{dT_2}{dx} = \frac{-q}{2}
\end{align}
$

where $U$ is a coefficient of heat transfer.

The temperatures of the hot and cold fluids when they enter the device are known.

$
\begin{align}
\large
T_1(x = 0) = 200
\end{align}
$

$
\begin{align}
\large
T_2(x = L) = 50
\end{align}
$

Relevant constants are given below.

```python
T1_0 = 200
T2_L = 50
L = 5
U = 1
```

Implement the derivatives and bounary conditions functions that describe this system.

```python
def derivatives(x , y):
    """
    y[0] is T_1
    y[1] is T_2
    """
    q = U * (y[0] - y[1])
    return np.vstack((- q, - q / 2))

def bc(ya, yb):
    """
    ya is [T1, T2] on the left
    yb is [T1, T2] on the right
    """
    return np.array([ya[0]-200, yb[0]-50])

```

```python
derivatives(x, y).shape
```

    (2, 10)

Set an initial guess for the solution. Try $T_1(x) = T_1(0)$ and $T_2(x) = T_2(L)$.

```python
x = np.linspace(0, L, 10)
y = np.empty([2, x.size])
y[0] = T1_0
y[1] = T2_L
```

Now find the solution and plot $T_1(x)$ and $T_2(x)$.

```python
sol = solve_bvp(derivatives, bc, x, y,)
x_sol = np.linspace(0, L, 100)
y_sol = sol.sol(x_sol)
plt.plot(x_sol, y_sol[0])
plt.plot(x_sol, y_sol[1])
```

## Minimization

The [scipy.optimize](https://docs.scipy.org/doc/scipy/reference/optimize.html) module provides a number of options for finding function minima and maxima. The most relevant to our purposes is [minimize_scalar](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize_scalar.html#scipy.optimize.minimize_scalar).

Study the documentation of `minimize_scalar` and use it to find all local minima of the function below in the range [-10, 10]. Plot $f(x)$ to gain some intuition. Is there a more informative way to plot $f(x)$?

```python
def f(x):
    return (x - 4*np.euler_gamma) * 0.5 * x * (x + np.e)**2 - 5*np.sin(3*x) + 11
```

```python
from scipy.optimize import minimize_scalar
```

```python
x = np.arange(-10, 10, 0.1)
y = f(x)
#plt.ylim(0,100)
plt.semilogy(x,y)
#plt.plot(x,y)
```

```python
res = minimize_scalar(f, method = 'Brent', bracket = (-10, 0, 10))
sol_1 = res.x
res = minimize_scalar(f, method = 'Brent', bracket = (-5, -3, -2.5))
sol_2 = res.x
res = minimize_scalar(f, method = 'Brent', bracket = (0, 1, 2.5))
sol_3 = res.x
sol = np.array([sol_1, sol_2, sol_3])
print(sol)
```

    [-1.65627449 -3.3207098   0.73763619]

```python

```

```python

```


# Solving Ordinary Differential Equations using Numerical Methods

### The following shows different numerical methods in solving ODEs

## Euler Method (1st Order)

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def Euler(f, x0, y, x, deltax):

    X = []
    Y = []
    X.append(x0)
    Y.append(y)

    while x0 < x:
        deltax = min(deltax, x - x0)
        y = y + deltax*f(x,y)
        x0 = x0 + deltax
        X.append(x0)
        Y.append(y)

    return np.array(X), np.array(Y)
```

### Exponential Decay

Let $N_u$ be the number of $U^{235}$ nuclei present in the sample at a time $t$:
\begin{equation}
\frac{dN_U}{dt} = - \frac{N_U}{\tau}
\end{equation}
Initial condition: $N_U = N_{U,0}$ at $t = 0$ with an exact solution $N_U(t) = N_{U,0} e^{-t/\tau}$

```python
N = lambda t,y:  -y*tau

tau = 1
N0 = 100
t0, tf = 0, 5

deltat1 = Euler(N, x0=t0, y=N0, x=tf, deltax=0.5)
deltat2 = Euler(N, x0=t0, y=N0, x=tf, deltax=0.1)
deltat3 = Euler(N, x0=t0, y=N0, x=tf, deltax=0.05)

t1, N1 = deltat1[0], deltat1[1]
t2, N2 = deltat2[0], deltat2[1]
t3, N3 = deltat3[0], deltat3[1]

t_exact = np.linspace(0,5)
N_exact = N0*np.exp(-t_exact/tau)

plt.plot(t1,N1, 'b.', label = r'$\Delta t = 0.5$')
plt.plot(t2,N2, 'bx', label = r'$\Delta t = 0.1$')
plt.plot(t3,N3, 'b+', label = r'$\Delta t = 0.05$')
plt.plot(t_exact, N_exact, 'r', label = 'Exact')
plt.legend()
plt.xlabel('time')
plt.ylabel(r'$N_u$')
plt.title('Number of nuclei of $U^{235}$')
plt.show()
```

### Linear Motion with Air Drag

It is often the case that the frictional force on an object will increase as the object moves faster. An example is the motion of a falling parachutist; the role of the parachute is to produce a frictional force due to air drag, which is larger than would normally be the case without the parachute. Assume that the velocity of the parachutist is described by the equation:
\begin{equation}
\frac{dv}{dt} = g - Cv
\end{equation}
where $g$ and $C$ are constants. The constant $g$ is the acceleration due to gravity while $C$ is a drag coefficient. Provide a way of estimating $v$ using the Euler method.

Using Euler method, we could estimate the $v$ as:
\begin{equation}
v_{n+1} = v_n + (g - Cv_n)\Delta t
\end{equation}
where $\Delta t$ is small increments of time $t$

Let $C=1$, $g=9.8$, $v_0=100$, at $t=0$ to $t=5$ and $\Delta t=0.05$

```python
C = 1
g = 9.8
v0 = 100
t0, tf = 0, 5
deltat = 0.05

v = lambda t, y: g - C*y

v_deltat = Euler(v, x0=t0, y=v0, x=tf, deltax=deltat)
t, v = v_deltat[0], v_deltat[1]

vexact = np.exp(-C*t)*(v0*C-g+g*np.exp(C*t))/C

plt.plot(t, v, 'b.', label = r'$\Delta t = 0.05$')
plt.plot(t, vexact, 'r', label = 'Exact')
plt.legend()
plt.xlabel('time')
plt.ylabel('$v$')
plt.title('Velocity of the parachutist through time')
plt.show()
```

## Euler Method (2nd Order)

```python
def Euler2nd(f, t0, tf, y0, v0, dt, Cromer= 'False'):

    t = np.arange(t0, tf+dt, dt)
    n = len(t)

    y = np.zeros(n)
    v = np.zeros(n)

    y[0] = y0
    v[0] = v0

    for i in range(n-1):
        if Cromer == 'False':
            v[i+1] = v[i] + f(y[i],v[i],t)*dt
            y[i+1] = y[i] + v[i]*dt
        else:
            v[i+1] = v[i] + f(y[i],v[i],t[i])*dt
            y[i+1] = y[i] + v[i+1]*dt

    return t, y
```

### Simple Harmonic Motion

\begin{equation}
\frac{d^2y}{dt^2} = -ky
\end{equation}
where, $y_0 = 0.2$, $v_0 = 0$, $k = 1$, and $\Delta t = 0.01$

```python
y0 = 0.2
v0 = 0
k = 1
dt = 0.01
tf = 20.

f = lambda y,v,t: -k*y

sol = Euler2nd(f, t0=0, tf=tf, y0=y0, v0=v0, dt=dt, Cromer= 'False')
t, y = sol[0], sol[1]

yexact = y0*np.cos(t*np.sqrt(k))

plt.plot(t, y, 'b.', label = 'Euler')
plt.plot(t, yexact, 'r', label = 'Exact')
plt.title('Simple Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

## Euler-Cromer (2nd Order)

### Simple Harmonic Motion

\begin{equation}
\frac{d^2y}{dt^2} = -ky
\end{equation}
where, $y_0 = 0.2$, $v_0 = 0$, $k = 1$, and $\Delta t = 0.01$

```python
y0 = 0.2
v0 = 0
k = 1
dt = 0.01
tf = 20.

f = lambda y,v,t: -k*y

sol = Euler2nd(f, t0=0, tf=tf, y0=y0, v0=v0, dt=dt, Cromer= 'True')
t, y = sol[0], sol[1]

yexact = y0*np.cos(t*np.sqrt(k))

plt.plot(t, y, 'b.', label = 'Euler-Cromer')
plt.plot(t, yexact, 'r', label = 'Exact')
plt.title('Simple Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

### Idealized Projectile Motion

Consider a projectile such as a shell shot from a cannon. The equations of motion of the cannon shell are obtained from Newton’s 2nd law:

\begin{equation}
\frac{d^2x}{dt^2} = 0, \frac{d^2y}{dt^2} = -g
\end{equation}

Express each of these 2nd-order ODEs as 1st-order ODEs and write down the equations for the Euler method that will approximate the motion of the cannon shell.

\begin{equation}
\frac{dx}{dt} = v, \frac{dy}{dt} = -gt + v
\end{equation}

\begin{equation}
x = x_0 + vt, y = y_0 + vt - \frac{gt^2}{2}
\end{equation}

\begin{equation}
v_{i+1} = v_i \\
x_{i+1} = x_i + v_{i+1}\Delta t
\end{equation}

\begin{equation}
v_{i+1} = v_i - g\Delta t \\
y_{i+1} = y_i + v_{i+1}\Delta t
\end{equation}

Let $g=9.8$, $v_{0x}=0.1$, $v_{0y}=10$, at $t=0$ to $t=2$ and $\Delta t=0.01$

```python
g = 9.8
v0x, v0y = 0.1, 10
x0, y0 = 0, 0
t0, tf = 0, 2
dt = 0.01

fx = lambda y,v,t: 0
fy = lambda y,v,t: -g

solx = Euler2nd(f=fx, t0=t0, tf=tf, y0=x0, v0=v0x, dt=dt, Cromer= 'True')
soly = Euler2nd(f=fy, t0=t0, tf=tf, y0=y0, v0=v0y, dt=dt, Cromer= 'True')
x, y = solx[1], soly[1]
t = solx[0]

xexact = v0x*t
yexact = y0 + v0y*t - 0.5*g*(t**2.)

plt.plot(x, y, 'b.', label = 'Euler-Cromer')
plt.plot(xexact, yexact, 'r', label = 'Exact')
plt.title('Idealized Projectile Motion')
plt.legend()
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

### Damped Harmonic Oscillator

The motion of a damped harmonic oscillator can be described using Newton’s 2nd Law:

\begin{equation}
m\frac{d^2x}{dt^2} + C\frac{dx}{dt} + kx = 0
\end{equation}

where $m$ is the mass of the oscillator, $C$ is the drag coefficient that damps the motion, and $k$ is a constant that determines the restoring force. Provide a numerical scheme that will approximate the motion of the oscillator.

Let $m=1$, $C=1$, $k=10$, $v_0=0$, $x_0=1$, at $t=0$ to $t=20$ and $\Delta t=0.01$

```python
m = 1
C = 1
k = 10
v0 = 0
x0 = 1
t0, tf = 0, 20
dt = 0.01

f = lambda y,v,t: -((C/m)*v + (k/m)*y)

sol = Euler2nd(f, t0=t0, tf=tf, y0=x0, v0=v0, dt=dt, Cromer= 'True')
t, x = sol[0], sol[1]

xexact = (1./39.)*np.exp(-t/2.)*(39*np.cos(0.5*t*np.sqrt(39)) + \
         np.sqrt(39)*np.sin(0.5*t*np.sqrt(39)))

plt.plot(t, x, 'b.', label = 'Euler-Cromer')
plt.plot(t, xexact, 'r', label = 'Exact')
plt.title('Damped Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('x')
plt.show()
```

### Pendulum

Consider a pendulum of length $L$ and mass $m$ which is acted upon by a frictional force and an external driving force. Let the driving force be a sinusoidally varying force with constant amplitude $F_D$ and driving frequency $\Omega_D$. Assume that the frictional force is proportional to the pendulum’s velocity with a constant damping factor $q$. (1) Write down the differential equation that describes the motion of the damped, driven pendulum. From the differential equation, write down the equations that approximate the solution using (2) Euler-Cromer, (3) 2nd-order Runge-Kutta, and (4) Verlet algorithms.

\begin{equation}
\frac{d^2y}{dt^2} + q\frac{dy}{dt} + \frac{g}{l}\sin{y} = F_D\cos{\Omega_D}
\end{equation}

\begin{equation}
v_{i+1} = v_i + (- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt})\Delta t \\
y_{i+1} = y_i + (v_i + (- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt})\Delta t)\Delta t
\end{equation}

Let $L=1$, $Fd=0.5$, $\Omega_D=10$, $g=9.8$, $q=0.1$, $v_0=0$, $y_0=10$, at $t=0$ to $t=20$ and $\Delta t=0.005$

```python
L = 1
Fd = 0.5
omegaD = 10
g = 9.8
q = 0.1
v0, y0 = 0, 10
t0, tf = 0, 20
dt = 0.005

f = lambda y,v,t: -q*v-(g/L)*np.sin(y)+Fd*np.cos(omegaD*t)

solEu = Euler2nd(f=f, t0=t0, tf=tf, y0=y0, v0=v0, dt=dt, Cromer= 'True')
t, y = solEu[0], solEu[1]

plt.plot(t, y, 'b.', label = 'Euler-Cromer')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

## 2nd - Order Runge Kutta

```python
def RK2(f, t0, tf, y0, v0, dt):

    t = np.arange(t0, tf+dt, dt)
    n = len(t)

    y = np.zeros(n)
    v = np.zeros(n)

    y[0] = y0
    v[0] = v0

    for i in range(n-1):
        v[i+1] = v[i] + f(y[i],v[i],t[i])*dt
        y[i+1] = y[i] + v[i+1]*dt

    return t, v, y
```

### Linear Motion with Air Drag

Use RK2 to approximate the motion of the falling parachutist:

\begin{equation}
\frac{dv}{dt} = g - Cv
\end{equation}

The constants are the same with the Linear Motion problem stated previously under the Euler Method section

```python
C = 1
g = 9.8
y0, v0 = 0, 100
t0, tf = 0, 5
dt = 0.05

f = lambda y,v,t: g-C*(v + (dt/2.)*(g-C*v))

sol = RK2(f, t0, tf, y0, v0, dt)
t, v = sol[0], sol[1]

vexact = np.exp(-C*t)*(v0*C-g+g*np.exp(C*t))/C

plt.plot(t, v, 'b.', label = 'RK2')
plt.plot(t, vexact, 'r', label = 'Exact')
plt.title('Linear Motion with Air Drag')
plt.legend()
plt.xlabel('time')
plt.ylabel('v')
plt.show()
```

### Damped Harmonic Oscillator

Use the RK2 method to estimate the position of the damped harmonic oscillator at any time t.

\begin{equation}
m\frac{d^2x}{dt^2} + C\frac{dx}{dt} + kx = 0
\end{equation}

The constants used are the same with the problem on the Euler-Cromer section

```python
m = 1
C = 1
k = 10
v0 = 0
x0 = 1
t0, tf = 0, 20
dt = 0.01

f = lambda y,v,t: -((C/m)*(v - (dt/2.)*((C/m)*v + (k/m)*y)) + (k/m)*y)

sol = RK2(f=f, t0=t0, tf=tf, y0=x0, v0=v0, dt=dt)
t, x = sol[0], sol[2]

xexact = (1./39.)*np.exp(-t/2.)*(39*np.cos(0.5*t*np.sqrt(39)) + \
         np.sqrt(39)*np.sin(0.5*t*np.sqrt(39)))

plt.plot(t, x, 'b.', label = 'RK2')
plt.plot(t, xexact, 'r', label = 'Exact')
plt.title('Damped Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('x')
plt.show()
```

### Pendulum

Consider a pendulum of length $L$ and mass $m$ which is acted upon by a frictional force and an external driving force. Let the driving force be a sinusoidally varying force with constant amplitude $F_D$ and driving frequency $\Omega_D$. Assume that the frictional force is proportional to the pendulum’s velocity with a constant damping factor $q$. (1) Write down the differential equation that describes the motion of the damped, driven pendulum. From the differential equation, write down the equations that approximate the solution using (2) Euler-Cromer, (3) 2nd-order Runge-Kutta, and (4) Verlet algorithms.

\begin{equation}
\frac{d^2y}{dt^2} + q\frac{dy}{dt} + \frac{g}{l}\sin{y} = F_D\cos{\Omega_D}
\end{equation}

\begin{equation}
v_{i+1} = v_i + (- q(v_i + \frac{\Delta t}{2}(- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt}) ) - \frac{g}{l}\sin{y} + F_D\cos{\Omega_D(t + \frac{dt}{2}}))\Delta t \\
y_{i+1} = y_i + (v_i + (- q(v_i + \frac{\Delta t}{2}(- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt}) ) - \frac{g}{l}\sin{y} + F_D\cos{\Omega_D(t + \frac{dt}{2}}))\Delta t)\Delta t
\end{equation}

The constants used are the same as that of the Pendulum problem under the Euler-Cromer section

```python
L = 1
Fd = 0.5
omegaD = 10
g = 9.8
q = 0.1
v0, y0 = 0, 10
t0, tf = 0, 20
dt = 0.005

f = lambda y,v,t: -q*(v + (dt/2.)*(-q*v - (g/L)*np.sin(y) + Fd*np.cos(omegaD*t))) - \
                 (g/L)*np.sin(y) + Fd*np.cos(omegaD*(t+dt/2.))

solRK = RK2(f=f, t0=t0, tf=tf, y0=y0, v0=v0, dt=dt)
t, y = solRK[0], solRK[2]

plt.plot(t, y, 'b.', label = 'RK2')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

## Verlet Method

### Pendulum

Consider a pendulum of length $L$ and mass $m$ which is acted upon by a frictional force and an external driving force. Let the driving force be a sinusoidally varying force with constant amplitude $F_D$ and driving frequency $\Omega_D$. Assume that the frictional force is proportional to the pendulum’s velocity with a constant damping factor $q$. (1) Write down the differential equation that describes the motion of the damped, driven pendulum. From the differential equation, write down the equations that approximate the solution using (2) Euler-Cromer, (3) 2nd-order Runge-Kutta, and (4) Verlet algorithms.

\begin{equation}
\frac{d^2y}{dt^2} + q\frac{dy}{dt} + \frac{g}{l}\sin{y} = F_D\cos{\Omega_D}
\end{equation}

\begin{equation}
y_{i+1} = (2y_i - y_{i-1} + \frac{q}{2}y_{i-1}\Delta t + (-\frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt})\Delta t^2)(1 - \frac{q}{2}\Delta t)
\end{equation}

The constants used are the same with the Pendulum problem under the Euler Cromer section

```python
def Verlet(f1, f2, f3, t0, tf, y0, v0, dt):

    t = np.arange(t0, tf+dt, dt)
    n = len(t)

    y = np.zeros(n)
    v = np.zeros(n)

    v1 = v0 + f1*dt
    y1 = y0 + v0*dt

    y[0], y[1] = y0, y1
    v[0], v[1] = v0, v1

    for i in range(1,n-1):
        y[i+1] = (2.*y[i] - y[i-1] + f3*y[i-1]*dt + (f2(y[i],t[i]))*(dt**2))*(1. - f3*dt)

    return t, y
```

```python
L = 1
Fd = 0.5
omegaD = 10
g = 9.8
q = 0.1
v0, y0 = 0, 10
t0, tf = 0, 20
dt = 0.005

f1 = (-q*v0-(g/L)*np.sin(y0)+Fd*np.cos(omegaD*t0))
f2 = lambda y, t: (-(g/L)*np.sin(y) + Fd*np.cos(omegaD*t))
f3 = q/2.

solVer = Verlet(f1, f2, f3, t0, tf, y0, v0, dt)
t, y = solVer[0], solVer[1]

plt.plot(t, y, 'b.', label = 'Verlet')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

```python
t1, y1 = solEu[0], solEu[1]
t2, y2 = solRK[0], solRK[2]
t3, y3 = solVer[0], solVer[1]

plt.plot(t1, y1, 'b', label = 'Euler-Cromer')
plt.plot(t2, y2, 'r--', label = 'RK2')
plt.plot(t3, y3, 'g:', label = 'Verlet')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```


# Contitional probability

Let us assume that we have two random variables $X$ and $Y$.  Both are generated by rolling fair six-sided dice, so the possible outcomes are:

\begin{align}
Pr(X=1) & = \frac{1}{6} & Pr(Y=1) & = \frac{1}{6} \\
Pr(X=2) & = \frac{1}{6} & Pr(Y=2) & = \frac{1}{6} \\
Pr(X=3) & = \frac{1}{6} & Pr(Y=3) & = \frac{1}{6} \\
Pr(X=4) & = \frac{1}{6} & Pr(Y=4) & = \frac{1}{6} \\
Pr(X=5) & = \frac{1}{6} & Pr(Y=5) & = \frac{1}{6} \\
Pr(X=6) & = \frac{1}{6} & Pr(Y=6) & = \frac{1}{6}
\end{align}

Further, we assume that $X$ and $Y$ are independent.

So, if one defines notation $Pr(X=x \cap Y=y)$ to mean the probability that $X=x$ and $Y=y$
then $X$ and $Y$ being independent means that $Pr(X=x \cap Y=y) = Pr(X=x)Pr(Y=y)$.

For example, $Pr(X=1 \cap Y=2) = Pr(X=1)Pr(Y=2) = \frac{1}{6} \frac{1}{6} = \frac{1}{36}$.

With this notation in mind, we define the conditional probality of $X=x$ given $Y=y$, denoted by $Pr(X=x | Y=y)$, as:

$$
Pr(X=x | Y=y) = \frac{Pr(X=x \cap Y=y)}{Pr(Y=y)}
$$

If $X$ and $Y$ are independent then the conditional probability is

$$
Pr(X=x | Y=y) = \frac{Pr(X=x \cap Y=y)}{Pr(Y=y)} = \frac{Pr(X=x) Pr(Y=y)}{Pr(Y=y)} = Pr(X=x).
$$

In other words, knowing $Y=y$ does not provide any information about $Pr(X=x)$.

Note, since

$$
Pr(X=x | Y=y) = \frac{Pr(X=x \cap Y=y)}{Pr(Y=y)}
$$

we also have that

$$
Pr(X=x | Y=y)Pr(Y=y) = Pr(X=x \cap Y=y).
$$

So, we have the following identity

$$
Pr(X=x | Y=y)Pr(Y=y) = Pr(X=x \cap Y=y) = Pr(Y=y \cap X=x) = Pr(Y=y | X=x)Pr(X=x).
$$

# Example

Things get more interesting when you have dependent random variables so let us define the random variables $Z$ by setting $Z=X+Y$.  All of the possible values for $Z$ as a function of $Y$ and $Z$ can be seen in the following table.

x\Y  |  1 | 2 | 3 | 4 | 5 | 6
-----|:---|:--|:--|:--|:--|:--
1    |  2 | 3 | 4 | 5 | 6 | 7
2    |  3 | 4 | 5 | 6 | 7 | 8
3    |  4 | 5 | 6 | 7 | 8 | 9
4    |  5 | 6 | 7 | 8 | 9 | 10
5    |  6 | 7 | 8 | 9 | 10| 11
6    |  7 | 8 | 9 | 10| 11| 12

Let us compute the

$$
Pr(Z=5 \cap X=2).
$$

Note there are two ways to proceed since

$$
Pr(Z=5 \cap X=2) = Pr(Z=5 | X=2) Pr(X=2) = Pr(X=2 | Z=5) Pr(Z=5)
$$

Doing it the first way we have that

\begin{align}
Pr(Z=5 \cap X=2) &= Pr(Z=5 | X=2) Pr(X=2) \\
                 &= Pr(Y=3) Pr(X=2) \\
                 &= \frac{1}{6} \frac{1}{6} \\
                 &= \frac{1}{36}
\end{align}

We use the face that $Pr(Z=5 | X=2) = Pr(Y=3)$ since, given that $X=2$, $Z=5$ if, and only if, $Y=3$.

Doing it the second way we have that

\begin{align}
Pr(X=2 \cap Z=5) &= Pr(X=2 | Z=5) Pr(Z=5) \\
                 &= \frac{1}{4} \frac{4}{36} \\
                 &= \frac{1}{36}
\end{align}

We use the face that $Pr(Z=5)$ in 4 of the 36 entries in the table above giving us that
$Pr(Z=5) = \frac{4}{36}$.  We also know that the four places where  $Z=5$ occur when

\begin{align}
X=1 ,\;& Y = 4 \\
X=2 ,\;& Y = 3 \\
X=3 ,\;& Y = 2 \\
X=4 ,\;& Y = 1
\end{align}

and $X=2$ in exactly one of those places, so $Pr(X=2 | Z=5) = \frac{1}{4}$.

The two ways match!

```python

```

lectures/05 Basic Statistics, Probability, and Linear Algebra/ConditionalProbabilty.ipynb


# 2.9 Model Selection and the Bias-Variance Tradeoff

All the models described have a *smoothing* or *complexity* parameter that has to be determined:

- the multiplier of the penalty term;

- the width of the kernel

- or the number of basis functions

We cannont use RSS on the training data to determine these parameters, since we would always pick those that gave interpolating fits and have zero residuals.

The kNN regression fit $\hat{f_k}(x_0)$ illustrates the competing forces that effect the predictive ability of such approximations. Suppose the data arise from a model $Y = f(X) + \varepsilon$ with $E(\varepsilon)=0$ and $Var(\varepsilon) = \sigma^2$. We assume that the values of $x_i$ in the sample are fixed. The EPE at $x_0$:

$$
\begin{align}
EPE_k(x_0) &= E[(Y - \hat{f_k}(x_0))^2|X=x_0]\\
&=\sigma^2 + [Bias^2(\hat{f_k}(x_0)) + Var_\tau(\hat{f_k}(x_0))]\\
&=\sigma^2 + \left[f(x_0) - \frac{1}{k}\sum_{l=1}^k{f(x_{(l)})} \right]^2 + \frac{\sigma^2}{k}
\end{align}
$$

The subscripts in parentheses ($l$)  indicates the sequence of nearest neighbors to $x_0$. There are three terms in this expression:

1. $\sigma^2$ is the *irreducible* error - is beyond our control, even if we know the true $f(x_0)$.

2. The bias term and the expected value of the estimate - $[E_\tau(\hat{f_k}(x_0))-f(x_0)]^2$ - where the expected averages the randomness in the training data. This term increases with $k$ if the function is smooth.

3. The variance term and it decreases as the inverse of k. The expected value of the variance is:

$$
\begin{align}
Var_\tau(\hat{f_k}(x_0)) &= E_\tau\left[\hat{f_k}(x_0) - E_\tau(\hat{f_k}(x_0))\right]^2\\
&= E_\tau\left[\frac{1}{k}\sum_{l=1}^k (f(x_{(l)}) + \varepsilon_l) - \frac{1}{k}\sum_{l=1}^k{f(x_{(l)})}\right]^2\\
&= E_\tau\left[\frac{1}{k}\sum_{l=1}^k \varepsilon_l\right]^2\\
&= \frac{1}{k^2}E_\tau\left[\sum_{l=1}^k \varepsilon_l\right]^2\\
&= \frac{1}{k^2}E_\tau\left[\sum_{l=1}^k \varepsilon_l^2\right]\\
&= \frac{\sigma^2}{k}
\end{align}
$$

As the *model complexity* of our procedure is increased, the variance tends to increase and the squared bias tends to decrease. The opposite behavior occurs as the model complexity is decreased.


# Frequency folding
## The relationship between the Fourier transform of a function and the Fourier transform of its sampled function
Consider the function $f(t)$. The function is sampled with the sampling period $h$ (correspoding to sampling frequency $\omega_s = \frac{2\pi}{h}$) to obtain the sequence $f(kh)$.
If $F(\omega)$ is the Fourier transform of the function $f(t)$ and $F_s(\omega)$ is the Fourier transform of the sampled signal $f(kh)$, then
\begin{equation}
F_s(\omega) = \frac{1}{h} \sum_{k=-\infty}^{\infty} F(\omega + k\omega_s)
\end{equation}

From this we see that the spectrum of a sampled signal is periodic, since the function $F_s(\omega)$ is periodic with period $\omega_s$, i.e.
\begin{equation}
F_s(\omega + m\omega_s) = \frac{1}{h} \sum_{k=-\infty}^{\infty} F(\omega + k\omega_s + m\omega_s) = \frac{1}{h} \sum_{k=-\infty}^{\infty} F(\omega + (k+m)\omega_s) = F_s(\omega)
\end{equation}

We also see that the power of the signal $f(kh)$ at a frequency $\omega_1$, $0 \le \omega \le \omega_N$ contains contributions from all frequencies of the original signal at the frequencies
\begin{equation}
\omega = \omega_1 + k\omega_s, \; k=-\infty, \ldots, 0, \ldots, \infty
\end{equation}
and
\begin{equation}
\omega = -\omega_1 + k\omega_s, \; k=-\infty, \ldots, 0, \ldots, \infty
\end{equation}
We say that $\omega_1$ is the alias of all these frequencies. The lowest such alias frequency is the frequency
\begin{equation}
\omega = -\omega_1 + \omega_s,
\end{equation}
which can be written
\begin{equation}
\omega = |\omega_1 - \omega_s| = | \omega_1 + \omega_N - \omega_N - \omega_s | = | (\omega_a + \omega_N) - \omega_s - \omega_N| = | (\omega_1 + \omega_N)\, \mathrm{mod}\, \omega_s - \omega_N|.
\end{equation}

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt

def spectrum(x,h):
    """ Computes spectrum using np.fft.fft  Returns frequency in rad/s from -\omega_N to \omega_N"""
    wN = np.pi/h # The Nyquist frequency
    N = len(x)
    X = np.fft.fft(x) # Computes the Fourier transform
    Xpos = X[:N/2] # Positive part of the spectrum
    Xneg = X[N/2:] # Negative part. Obs: for frequencies wN up to ws
    wpos = np.linspace(0, wN, N/2) # Positive frequencies, goes from 0 to wN

    W = np.hstack((-wpos[::-1], wpos))
    XX = np.hstack((Xneg, Xpos))

    return (XX, W)

# Assume too slow sampling of signal consisting of two high-frequency sinusoids
ws = 16 # Sampling frequency in rad/s
wN = ws/2
h = np.pi/wN

w1 = 10 # rad/s
w2 = w1+1*ws # rad/s

w1Alias = np.abs( (w1+wN) % ws - wN )
w2Alias = np.abs( (w1+wN) % ws - wN )

M = 4000 # Number of samples in the over-sampled ("continuous") signal
t = np.linspace(0, 60*2*np.pi/w1, M) # 60 periods of the slowest sinusoid
y = np.sin(w1*t) + np.sin(w2*t) # Continuous time (sort of) signal

N = 400 # Number of samples to take
ts = np.arange(N)*h
ys = np.sin(w1*ts) + np.sin(w2*ts) # Sampled signal

(Y,W) = spectrum(y,t[1]-t[0]) # get spectrum (from FFT) of the "continuous" signal
(Ys, Ws) = spectrum(ys, h) # Spectrum of discrete signal

plt.figure(figsize=(10,7))
plt.subplot(2,1,1)
plt.plot(W, np.real(Y))
plt.plot(Ws, np.real(Ys))
plt.plot([wN, wN], [-50, 150], 'k--')
plt.plot([-wN, -wN], [-50, 150], 'k--')
plt.xlim((-1.2*w2, 1.2*w2))
plt.xticks((-w2, -w1, -wN, -w1Alias, w1Alias, wN, w1, w2))
plt.ylim((-500, 500))
plt.ylabel('Real part')
plt.subplot(2,1,2)
plt.plot(W, np.imag(Y))
plt.plot(Ws, np.imag(Ys))
plt.plot([wN, wN], [-1500, 1500], 'k--')
plt.plot([-wN, -wN], [-1500, 1500], 'k--')
plt.xlim((-1.2*w2, 1.2*w2))
plt.xticks((-w2, -w1, -wN, -w1Alias, w1Alias, wN, w1, w2))

plt.ylim((-400, 400))
#plt.xticks((-10, -5, -1, 0, 1, 5, 10))
plt.ylabel('Imaginary part')
plt.xlabel(r'$\omega$  [rad/s]')

plt.legend(('Continuous', 'Sampled', 'Nyquist frequency'), loc=1, borderaxespad=0.)

```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x7f102ccdf510>

```python
plt.figure(figsize=(10,5))
plt.plot(t,y, color=(0.7,0.7,1))
plt.stem(ts[:20], ys[:20],linefmt='r--', markerfmt='ro', basefmt = 'r-')
plt.xlim((0,7.8))
plt.xlabel('t  [s]')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x7f102e1b38d0>

```python

```


# Truncation Error Analysis via sympy

Copyright (C) 2020 Andreas Kloeckner

<details>
<summary>MIT License</summary>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
</details>

```python
import sympy as s
s.init_printing()
```

Establish some variables that we'll need:

```python
u = s.Function("u")
a, x, t, h_x, h_t = s.symbols("a, x, t, h_x, h_t")
xi_1, xi_2, tau = s.symbols("xi1, xi2, tau")
```

`taylor` is a utility function that spits out a taylor expansion for $f(x+h)$, optionally including a remainder term, with all variables under our control.

```python
def taylor(f, x, h, n, remainder_variable=None):
    result = sum(f.diff(x, i)*h**i/s.factorial(i) for i in range(n))
    if remainder_variable:
        result += f.diff(x, n).subs(x, remainder_variable)*h**n/s.factorial(n)
    return result
```

- Try it out by expanding $u(x+h_x,t)$
- Vary the order
- Expand $u(x,t+h_t)$ instead

```python
#clear
taylor(u(x,t), x, h_x, 3, xi)
```

Assign the PDE we're solving to `pde`:

```python
#clear
pde = u(x, t).diff(t) + a * u(x, t).diff(x)
pde
```

Write out the scheme we're analyzing, in this case ETCS:

```python
etcs = (
    (u(x, t+h_t) - u(x, t))/h_t
    +
    a*(u(x+h_x, t) - u(x-h_x, t))/(2*h_x))
etcs
```

Follow this general pattern:
```
etcs
.subs(u(x, t+h_t), taylor(u(x,t), t, h_t, 2, tau))
```
to arrive at the truncation error.

⚠ Make sure to keep the two $x$ remainder terms separate.

```python
#clear
etcs_taylor = (
    etcs
    .subs(u(x, t+h_t), taylor(u(x,t), t, h_t, 2, tau))
    .subs(u(x+h_x, t), taylor(u(x,t), x, h_x, 3, xi_1))
    .subs(u(x-h_x, t), taylor(u(x,t), x, -h_x, 3, xi_2))
)
sp.simplify(etcs_taylor - pde)
```

```python

```

demos/fd-tdep/Truncation Error Analysis via sympy.ipynb


# Lesson 01 - From Machine Learning to Deep Learning

## Training Logistic classifier
- Is a Linear Classifier

```
W X + b = Y
```
- W, X are matrices
- X is input
- W (Weights) and b (bias) are found by training the model
- Y vector
    - contains output prediction score for each possible output
    - also known as Logits for Logistic Regression
    - turned into probabilities (by using SoftMax function)

```python
scores = [3.0, 1.0, 0.2]
```

```python
import numpy as np

def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)
```

```python
import matplotlib.pyplot as plt

x = np.arange(-2.0, 6.0, 0.1)
scores = np.vstack([x, np.ones_like(x), 0.2 * np.ones_like(x)])
scores.shape
```

    (3, 80)

```python
plt.plot(x, softmax(scores).T, linewidth = 2)
plt.legend(['x', '1', '0.2'])
plt.show()
```

If we increase the magnitude of scores then classifier becomes more confident but if we decrease the size of the outputs then the classifier becomes less confident

## One Hot encoding
- Each label would be represented by a vector which is as long as the number of labels and has value 1.0 for the label and 0 for other labels
- Works not well when we have tens of thousands of labels or more. We have lots of zeros at that time
- We can find how well we are doing by simply comparing 2 vectors

Putting all pieces together we have the following Multinomial Logistic Classification

```
D(S(W * X + b), L)
```

## Minimizing Cross Entropy
- Now how do we find the W and b so that our classifier works i.e. D(A, a) is less but D(A, b) is high
- We can try and minimize the loss when summed over all training data
\begin{align}
L = (1/ N) *  \sum_{i} D(S(WX_{i} + b), L_{i})
\end{align}

- We need to calculate the derivative w.r.t to parameters and follow derivative to solution. This is called gradient descent

## Numerical stability
- You have to worry abut calculating values that are too big or too small whenever doing numerical computation

```python
a = 10 ** 9
for i in range(10 ** 6):
    a += 10 ** (-6)

a - 10 ** 9
```

    0.95367431640625

- For numerical stability the values involved in our Loss function L should not get too big or too small.
- A good principle is
    - 0 mean
    - equal variance

- For images it is simple to normalize it by
\begin{align}
(R/G/B - 128) / 128
\end{align}

- The weights and bias should also be good enough for gradient descent to proceed
- Many many but a simple way is to take from gaussian distribution with mean 0 and small variance. Higher variance would mean that the our model is opinioninated in the beginning which we don't want. Our model should be uncertain in the beginning and learn from data

- Validation sets are needed because as we iterate we are exposing the test set to classifier through our decisions
- Using 30000 examples as validation set is good. But if that is a lot of data then we could possibly use cross validation

- Logistic Regression's biggest problem is that it is very difficult to scale
    - Depends on all of data
    - Usually computing the gradient uses 3 times the compute of Loss function
    - In gradient descent we iterate
    - So instead of using all of data we use a small random sample and iterate more. This is called Stochastic Gradient Descent (SGD)
    - SGD is important because it scales well with big data and big models

## Helping SGD
- Momentum
    - Instead of using the current direction we use the knowledge of the general direction in which we are going to decide the current direction in which we want to go
    -
- Learning Rate average
    - Learing rate should be decreased over time


# Alice's Rose

## Dependencies

The `import` statement is special... it imports programmer's wisdom!
The common usage is to acquire access to python packages.

```python
import sympy
import math
```

## A Romantic Introduction to Matematical Optimization and to Python

Note: this story was originally told in the book [Optimization](https://press.princeton.edu/books/hardcover/9780691102870/optimization) and the protagonist there is called Samantha, but we stick to the alphabetical order...

As the story goes, Alice receives a beautiful rose. She has nothing but a lemonade glass to hold the rose and becomes very distressed when the ensemble falls down.

Adding a bit of water helps! Not only that helps the rose, but it also helps the stability: glass, with some water, and rose stands!

Alice thinks: if a bit of water helps, the let us fill the glass!

However, it tilts and falls, as in the beginning, just much more wet.

Alice has a problem to solve: what is the _optimal_ level of water for her rose on a lemonade glass?

She learns from [Archimedes]( https://en.wikipedia.org/wiki/Archimedes) how to compute the _center of gravity_ of the glass with water, which has height
$h = \frac{m_w}{m_w+m_g} h_w + \frac{m_g}{m_w+m_g} h_g$ with:

* $m_w$ the mass of water
* $m_g$ the mass of glass
* $h_w$ the height of the center of gravity of the water in the glass
* $h_g$ the height of the center of gravity of the glass without water

Since Alice's glass is $20$ cm tall, $4$ cm wide and weighs $100$ gram, Alice may fill the glass with water up to height $x$ cm, provided that $0 \leq x \leq 20$ since the water must fit in the glass.

The volume of water is $\pi r^2 x$ with $r$ the radius of the base, i.e. $r=2$.
The volume is therefore $4\pi x$ cubic centimetres.

Since the density of water can be [taken](https://en.wikipedia.org/wiki/Gram_per_cubic_centimetre) as being $1$ gram per cubic centimeter we have:

* $m_w = 4\pi x$
* $m_g = 100$
* $h_w = \frac{x}{2}$
* $h_g = \frac{20}{2} = 10$

And from here we finally obtain the following formula for the height of the center of gravity of the glass with water:

$$
h = \frac{4\pi x}{4\pi x + 100} \frac{x}{2} + \frac{100}{4\pi x + 100} 10 = \frac{4\pi x^2 + 2000}{8\pi x + 200}
$$

Alice's problem is therefore:

$$
\begin{array}{rl}
\min    & \frac{4\pi x^2 + 2000}{8\pi x + 200}  \\
s.t.    & x \geq 0   \\
        & x \leq 20  \\
\end{array}
$$

## Analytical solution

Alice learns from [Fermat]( https://en.wikipedia.org/wiki/Pierre_de_Fermat) that for a function to reach its highest and lowest points inside its domain the derivative must vanish.

This is a good moment to play with symbolic mathematics in python, we will use [sympy](https://www.sympy.org/en/index.html).

### With $\pi$ as a number

```python
# x is a symbol and pi is a number
x = sympy.Symbol('x')
pi = math.pi

# h is a function of x, and hprime its derivative
h = (4*pi*x**2 + 2000)/(8*pi*x+200)
hprime = sympy.diff(h, x)

# sol is(are) the value(s) of x that solve hprime(x) == 0
sol = sympy.solveset(hprime, x)
sol
```

$\displaystyle \left\{-22.8735335189926, -7.95774715459477, 6.95803920980307\right\}$

Above we see that the equation $h^\prime(x) = 0$ has two solutions: one negative and one positive.
Obviously, only the positive may be feasible for Alice.
And, since its value is between $0$ and $20$, it is indeed feasible.

You may recall that the sign of the second derivative tells you whether the root of the first derivative is a *maximum*, a *minimum* or a *saddle point*.

```python
opt = max(sol)
sympy.diff(hprime, x).subs(x,opt).evalf()
```

$\displaystyle 0.0670430626699561$

Since $h^{\prime\prime}(\text{opt}) > 0$ it is indeed a (local) **minimum**.

### With $\pi$ as a symbol

```python
# now pi is a symbol, just like x
pi = sympy.Symbol('pi')

# we redefine h using the same right-hand-side code as before,
# but now with x and pi as symbols
h = (4*pi*x**2 + 2000)/(8*pi*x + 200)

# to have the drivative on the symbol pi we need it from the new version of h
hprime = sympy.diff(h, x)

solution = sympy.solveset(sympy.diff(h, x), x )
solution
```

$\displaystyle \left\{\frac{- 5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}, \frac{5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}\right\}$

### From a symbolic $\pi$ to a numeric $\pi$

```python
s = max(solution.subs(pi, math.pi).evalf())
print(s)
```

    6.95803920980307

### A picture says more than thousand words

```python
import matplotlib.pyplot as plt
import numpy as np

def plot_alice(h, s, start, stop, width=18, height=8):

    plt.rcParams["figure.figsize"] = (18,8)

    x = sympy.Symbol('x')
    f = sympy.lambdify(x, h.subs( pi, math.pi))

    x = np.linspace(start=start,stop=stop,num=100)
    y = f(x)

    plt.plot(x,y,label='$'+sympy.latex(h)+'$',linewidth=3)
    plt.plot(s,f(s), 'ro', label='optimum', markersize=12)

    plt.legend()
    plt.show()
```

```python
plot_alice( h, s, 0, 20 )
```

## What if we only care about the numerical solution?

### Introducing `pyomo`

This is the moment to meet:
 * mathematical models expressed in `python`, using `pyomo`,
 * powerful numerical optimization algorithms and how to use them.

We will see that `pyomo` completely separates modeling from solving, which allows us to switch solver without recoding!

### Notebook dependencies requiring installation on `colab`

Note that [this notebook](https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.02-Running-Pyomo-on-Google-Colab.ipynb) explains how to run `Pyomo` on Google Colab.
For a complete overview please check the [cookbook](https://jckantor.github.io/ND-Pyomo-Cookbook/).

```python
import shutil
if not shutil.which('pyomo'):
    !pip install -q pyomo
    assert(shutil.which('pyomo'))
```

```python
import pyomo.environ as pyo

alice = pyo.ConcreteModel('Alice')
alice.h = pyo.Var(bounds=(0,20))

@alice.Objective(sense=pyo.minimize)
def cog(m):
    return (4*math.pi*alice.h**2 + 2000)/(8*math.pi*alice.h + 200)

alice.pprint()
```

    1 Var Declarations
        h : Size=1, Index=None
            Key  : Lower : Value : Upper : Fixed : Stale : Domain
            None :     0 :  None :    20 : False :  True :  Reals

    1 Objective Declarations
        cog : Size=1, Index=None, Active=True
            Key  : Active : Sense    : Expression
            None :   True : minimize : (12.566370614359172*h**2 + 2000)/(25.132741228718345*h + 200)

    2 Declarations: h cog

We will use `ipopt`. We refer again to [this notebook](https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.02-Running-Pyomo-on-Google-Colab.ipynb) explains how to run `Pyomo` **and how to install solvers** on Google Colab. For a complete overview please check the [cookbook](https://jckantor.github.io/ND-Pyomo-Cookbook/).

```python
import sys
if 'google.colab' in sys.modules:
    !wget -N -q 'https://ampl.com/dl/open/ipopt/ipopt-linux64.zip'
    !unzip -o -q ipopt-linux64
```

```python
results = pyo.SolverFactory('ipopt').solve(alice)
print(results.solver.status, results.solver.termination_condition )
alice.display()
```

    ok optimal
    Model Alice

      Variables:
        h : Size=1, Index=None
            Key  : Lower : Value            : Upper : Fixed : Stale : Domain
            None :     0 : 6.95803921230998 :    20 : False : False :  Reals

      Objectives:
        cog : Size=1, Index=None, Active=True
            Key  : Active : Value
            None :   True : 6.95803920980307

      Constraints:
        None

## Conclusion

This notebook shows how to solve Alice's problem: finding the most stable amount of water in a vase.

The notebook shows how to solve the problem analytically with `sympy`, how to use `matplotlib` to visualize the function and the optimum. And how to model Alice's problem on `pyomo` and solve it with `ipopt` both at [neos](https://neos-server.org/neos/solvers/index.html) and "locally" at your own Colab session.

## Last remarks

This notebook deferred installation of the packages needed to the moment that we needed them. This was deliberate, but subsequent notebooks will normally list all dependencies on their top part, which we often call the _preamble_. Furthermore, the `colab' dependencies will be streamlined in future notebooks.

```python

```


# Solving Linear Systems: Direct Methods

This notebook by Xiaozhou Li is licensed under a Creative Commons Attribution 4.0 International License.
All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

```python
%matplotlib inline
import numpy as np
from scipy.linalg import lu
```

## Gaussian Elimination

The simplest version of Gaussian Elimination only involves two operations as follows:
* Add or substrct a muliple of one equation from another.
* Multiply an equation by a nonzero constant.

### Solving a linear system by Gaussian elimination
The general form of a linear system for $n$ equations in $n$ unknown can be wriiten as
\begin{equation}
\begin{pmatrix}{a_{11}} & {a_{12}} & {\cdots} & {a_{1 n}} \\ {a_{21}} & {a_{22}} & {\cdots} & {a_{2 n}} \\ {\vdots} & {\vdots} & {} & {\vdots} \\ {a_{n 1}} & {a_{n 2}} & {\cdots} & {a_{n n}}\end{pmatrix}\begin{pmatrix}{x_{1}} \\ {x_{2}} \\ {\vdots} \\ {x_{n}}\end{pmatrix}=\begin{pmatrix}{b_{1}} \\ {b_{2}} \\ {\vdots} \\ {b_{n}}\end{pmatrix}
\end{equation}

To solve this linear system by Gaussian elimination
1. Using the allowed row operations to eliminate the system to an upper triangular system
\begin{equation}
\begin{pmatrix}{a_{11}} & {a_{12}} & {\cdots} & {a_{1 n}} & {b_{1}} \\ {} & {a_{22}^{(1)}} & {\cdots} & {a_{2 n}^{(1)}} & {b_{2}^{(1)}} \\ {} & {} & {\ddots} & {\vdots} & {\vdots} \\ {} & {} & {} & {a_{n n}^{(n-1)}} & {b_{n}^{(n-1)}}\end{pmatrix}
\end{equation}
2. Using the back substituion (backsolving) to solve the upper triangular system,
\begin{align}
x_{n}=& \frac{b_{n}^{(n-1)}}{a_{n-1}^{(n-1)}}  \\ x_{k}=& \frac{b_{k}^{(k-1)}-\sum_{j=k+1}^{n} a_{k j}^{(k-1)} x_{j} }{a_{k k}^{(k-1)}}, \quad k=n-1, \ldots, 1.
\end{align}
Here, assuming $a^{(k-1)}_{kk} \neq 0,\,\, k = 1,\ldots,n$

**Example**

Using Gaussian elimination to solve the following linear system
\begin{align}
 10^{-20}x_1 + x_2 & = 1 \\
 x_1 + 2x_2 & = 4
\end{align}
The argumented matrix reads
\begin{pmatrix}
10^{-20} & 1 & 1 \\
1 & 2 & 4
\end{pmatrix}

1. Elimination: $\text{Row}_2 = \text{Row}_2 - 10^{20}\times\text{Row}_1$

```python
A = np.array([[1e-20, 1, 1], [1, 2, 4]])
A[1] = A[1] - 10**20*A[0]
print (A[1])
```

    [ 0.e+00 -1.e+20 -1.e+20]

2. The echelon form of the argumented matrix reads
\begin{pmatrix}
10^{-20} & 1 & 1 \\
0 & -10^{20} & -10^{20}
\end{pmatrix}
3. Using the back substituion, the solution is
$$ x_2 = 1,\qquad x_1 = 0.$$

## The $PA = LU$ factorization
### Partial pivoting
At the start of classical Gaussian elimination of $n$ equations in $n$ unknowns, the first step is to use the diagonal element $a_{11}$ as a pivot to eliminate the first column.  The partial pivoting protocol consists of comparing numbers before carrying out each elimination step. The largest entry of the first column is located, and its row is swapped with the pivot row, in this case the top row.

**Example**

Solve the following linear system
\begin{align}
 10^{-20}x_1 + x_2 & = 1 \\
 x_1 + 2x_2 & = 4
\end{align}

The argumented matrix reads
\begin{pmatrix}
10^{-20} & 1 & 1 \\
1 & 2 & 4
\end{pmatrix}

1. Interchange $\text{Row}_1$ with $\text{Row}_2$
\begin{pmatrix}
1 & 2 & 4 \\
10^{-20} & 1 & 1
\end{pmatrix}
2. Elimination: $\text{Row}_2 = \text{Row}_2 - 10^{-20}\times\text{Row}_1$

```python
A = np.array([[1, 2, 4], [1e-20, 1, 1]])
A[1] = A[1] - 10**(-20)*A[0]
print (A[1])
```

    [0. 1. 1.]

3. The echelon form of the argumented matrix reads
\begin{pmatrix}
1 & 2 & 4 \\
0 & 1 & 1
\end{pmatrix}
4. Using the back substituion, the solution is
$$ x_2 = 1,\qquad x_1 = 2.$$

* Solving this equation by the build-in linear solver in numpy:

```python
A = np.array([[1e-20, 1], [1, 2]])
b = np.array([1, 4])

x = np.linalg.solve(A, b)
print (x)
```

    [2. 1.]

### Implementation
#### The $LU$ factorization

```python
def LU_factor(A):
    A = A.astype(np.float)
    print (A)
    n = len(A)
    for j in range(0,n-1):
        if A[j,j] != 0:
            for i in range(j+1,n):
                lam = A[i,j]/A[j,j]
                A[i,j+1:n] = A[i,j+1:n] - lam*A[j,j+1:n]
                A[i,j] = lam
    return A
```

**Example**
Find the LU factorization of matrix
$$
A=\begin{pmatrix}{2} & {4} & {4} & {2} \\ {3} & {3} & {12} & {6} \\ {2} & {4} & {-1} & {2} \\ {4} & {2} & {1} & {1}\end{pmatrix}
$$
and
$$
A=\begin{pmatrix}{1} & {1} & {3} & {4} \\ {2} & {4} & {1} & {3} \\ {3} & {2} & {1} & {1} \\ {2} & {1} & {2} & {1}\end{pmatrix}
$$

```python
A = np.array([[2, 4, 4, 2], [3, 3, 12, 6], [2, 4, -1, 2], [4, 2, 1, 1]])
print (LU_factor(A))
```

    [[ 2.  4.  4.  2.]
     [ 3.  3. 12.  6.]
     [ 2.  4. -1.  2.]
     [ 4.  2.  1.  1.]]
    [[ 2.   4.   4.   2. ]
     [ 1.5 -3.   6.   3. ]
     [ 1.  -0.  -5.   0. ]
     [ 2.   2.   3.8 -9. ]]

```python
A = np.array([[1, 1, 3, 4], [2, 4, 1, 3], [3, 2, 1, 1], [2, 1, 2, 1]])
print (LU_factor(A))
```

    [[1. 1. 3. 4.]
     [2. 4. 1. 3.]
     [3. 2. 1. 1.]
     [2. 1. 2. 1.]]
    [[  1.           1.           3.           4.        ]
     [  2.           2.          -5.          -5.        ]
     [  3.          -0.5        -10.5        -13.5       ]
     [  2.          -0.5          0.61904762  -1.14285714]]

#### The $PA = LU$ factorization

All of the techniques described so far are implemented in Python (Scipy). The most sophisticated form of Gaussian elimination we have discussed is the $PA=LU$ (or $A = PLU$) factorization. Scipy package has the command accepts a square coefficient matrix $A$ and returns $P$ , $L$, and $U$, such that $A = PLU$.

```python
from scipy.linalg import lu
A = np.array([[2, 1, 5], [4, 4, -4], [1, 3, 1]])
P, L, U = lu(A)
print (P)
print (L)
print (U)
```

    [[0. 0. 1.]
     [1. 0. 0.]
     [0. 1. 0.]]
    [[ 1.    0.    0.  ]
     [ 0.25  1.    0.  ]
     [ 0.5  -0.5   1.  ]]
    [[ 4.  4. -4.]
     [ 0.  2.  2.]
     [ 0.  0.  8.]]

```python
A = np.array([[2, 4, 4, 2], [3, 3, 12, 6], [2, 4, -1, 2], [4, 2, 1, 1]])
P, L, U = lu(A)
print (P)
print (L)
print (U)
```

    [[0. 0. 0. 1.]
     [0. 0. 1. 0.]
     [0. 1. 0. 0.]
     [1. 0. 0. 0.]]
    [[1.         0.         0.         0.        ]
     [0.5        1.         0.         0.        ]
     [0.75       0.5        1.         0.        ]
     [0.5        1.         0.41666667 1.        ]]
    [[ 4.     2.     1.     1.   ]
     [ 0.     3.    -1.5    1.5  ]
     [ 0.     0.    12.     4.5  ]
     [ 0.     0.     0.    -1.875]]

```python
print (np.dot(P, np.dot(L, U)) - A)
```

    [[0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]]

## Erros, Conditions Number

**Example** (Hilbert Matrix)

The $n\times n$ Hilbert Matrix $H_n$ is defined as follows:
$$
H_n = \begin{pmatrix}{1} & {1 / 2} & {1 / 3} & {\cdots} & {1 / n} \\ {1 / 2} & {1 / 3} & {1 / 4} & {\cdots} & {1 /(n+1)} \\ {1 / 3} & {1 / 4} & {1 / 5} & {\cdots} & {1 /(n+2)} \\ {\cdots} & {\cdots} & {\cdots} & {\cdots} & {\cdots} \\ {1 / n} & {1 /(n+1)} & {1 /(n+2)} & {\cdots} & {1 /(2 n-1)}\end{pmatrix}
$$

Solving the linear system
$$ H_n x = b$$
with
$$ b = H_n \cdot\begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix} $$
for $n = 5, 10, 20$
* The exact solution is $x = \begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix} $

```python
def hil(n):
    A = np.empty([n, n])
    for i in range(n):
        for j in range(n):
            A[i,j] = 1/(i+j+1)
    return A

def hil_example(n):
    A = hil(n)
    x_exact = np.zeros(n) + 1.
    b = np.dot(A, x_exact)
    x = np.linalg.solve(A, b)

    print ("The Hilbert Example")
    print ("Exact Solution: ", x_exact)
    print ("Numerical Solution: ", x)
    print ("Max Norm Error: ", np.max(x_exact - x))
```

```python
n = 30
#A = hil(n)
#print (A, '\n')

hil_example(n)
```

    The Hilbert Example
    Exact Solution:  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
     1. 1. 1. 1. 1. 1.]
    Numerical Solution:  [  0.99999999   1.00000549   0.9995934    1.01063208   0.86492543
       1.95654753  -2.98980522  10.66985353 -10.64549422   1.40496892
      15.91216488  -9.67086839  -3.19672645   5.14709635  -5.92419006
      16.63359268   3.83136369 -13.36494379  -3.48351104  -2.37205148
      14.37850702  22.52283623 -37.62581185  20.90624367  -8.37575394
       1.62476071  10.16821532  -4.99742976   1.20141018   1.41386896]
    Max Norm Error:  38.62581184885538

```python
#n = 5
A = hil(n)
print (np.linalg.cond(A, np.inf))
```

    6.598338740631285e+18

**Example**

Solve the following linear system
\begin{align}
 (1 + 10^{-20})x_1 + x_2 & = 2 + 10^{-20} \\
 x_1 + x_2 & = 2
\end{align}

```python
A = np.array([[1e-20, 1], [1, 1]])
b = np.array([2+1e-20, 2])

x = np.linalg.solve(A, b)
print (x)
```

    [0. 2.]

```python

```