# Example: Using Loman to price Interest Rate Swaps

In this example, we'll look at calibrating interest rate curves to market prices, and then using them to price portfolios of swaps.

## Curve Classes

Because the focus will be on using Loman, we'll adopt deliberately simplifying assumptions; time is a float, day count and business day conventions are ignored, quarters are exactly 0.25 years long and so on.

Our interest rate curves can be used for two things: discounting and projecting rates. To do this, we define a continuously-compounded forward rate $r(t)$, so that the discount rate from a payment at time $t$ to a time $s$ is
\begin{equation} df(s,t) = \exp\left[-\int_s^t r(\tau) d\tau\right] \end{equation}
and zero/FRA rates are defined by
\begin{equation} df(s,t) = \frac{1}{1+\text{FRA}(s,t)(t-s)}. \end{equation}

Our BaseIRCurve class leaves the definition of $r$ blank, but otherwise fleshes out the methods we'll need, including methods to PV a set of cashflows, and also to plot the continuously-compounded forward rate, 3M FRA rates, and spot swap rates (once we define swap_rate, further below).

```python
%matplotlib inline
import numpy as np
from scipy.integrate import quad
import matplotlib.pyplot as plt

class BaseIRCurve(object):
    def r_quad(self, s, t):
        b = np.broadcast(s,t)
        out = np.empty(b.shape)
        out.flat = [quad(self.r, s0, t0)[0] for s0, t0 in b]
        return out
    def r(self, t):
        pass
    def df(self, s, t):
        return np.exp(-self.r_quad(s, t))
    def fra(self, s, t):
        return (1 / self.df(s,t) - 1.) / (t - s)
    def pv(self, amts, ts, s=0):
        dfs = self.df(s, ts)
        return np.sum(amts * dfs)
    def plot(self):
        ts = np.linspace(0.0, 29.75, 360)
        plt.plot(ts, self.r(ts), label='r')
        ts = np.linspace(0.0, 29.75, 120)
        plt.plot(ts, self.fra(ts, ts+0.25), label='3M fwd')
        plt.plot(ts, np.vectorize(swap_rate)(0, ts, 0.25, self, self), label='Swap Rate')
        plt.legend()
```

The FlatIRCurve class fills out the definitions of r and r_quad. $r(t)$ is piece-wise flat, and hence its integral is piecewise linear. Together with the methods defined by BaseIRCurve we will now have a basic but functional interest rate curve.

```python
class FlatIRCurve(BaseIRCurve):
    def __init__(self, ts, rates):
        self.ts = ts
        self.rates = rates
        self.ts0 = np.zeros(len(self.ts) + 1)
        self.ts0[1:] = self.ts
        self.rquads = np.zeros_like(self.ts0)
        self.rquads[1:] = np.cumsum(np.diff(self.ts0) * self.rates)
    def r(self, t):
        idx = np.minimum(np.searchsorted(self.ts, t, 'right'), len(self.ts)-1)
        return self.rates[idx]
    def r_quad(self, s, t):
        return np.interp(t, self.ts0, self.rquads) - np.interp(s, self.ts0, self.rquads)
```

Finally, we'll need to set up some functions to calculate quarterly FRAs and discount them to calculate PVs of the fixed and floating legs of swaps, as well as the swap rate.

```python
def sched(a, b, p):
    n = int((b-a)/p)
    if b - n * p > a + p / 2.:
        n = n + 1
    ts = np.linspace(b - n * p, b, n+1)
    ts[0] = a
    return ts

def swap_leg_pvs(a, b, p, projection_curve, discount_curve):
    ts = sched(a, b, p)
    pers = np.diff(ts)
    fixed_pv = discount_curve.pv(pers, ts[1:])
    fras = projection_curve.fra(ts[:-1], ts[1:])
    float_pv = discount_curve.pv(pers * fras, ts[1:])
    return fixed_pv, float_pv

def swap_rate(a, b, p, projection_curve, discount_curve):
    fixed_pv, float_pv = swap_leg_pvs(a, b, p, projection_curve, discount_curve)
    return float_pv / fixed_pv
```

## Calibrating a LIBOR curve

Up until 2008, it was standard to use the LIBOR curve for both projection of LIBOR swap cashflows *and* for discounting those cashflows. We set up a Loman computation with inputs **usd_libor_ts**, a set of swap maturities, and **usd_libor_c_rate** a set of continuous compounding rates for each period. From those, we create a curve object **usd_libor_curve**, and then calculate swap rates using that curve for both projection and discounting, in **usd_libor_swap_rates**.

```python
import loman
comp = loman.Computation()
comp.add_node('usd_libor_ts', value=np.array([1., 2., 3., 5., 7., 10., 15., 20., 30.]))
comp.add_node('usd_libor_c_rates', value=0.03*np.ones(9))
comp.add_node('usd_libor_curve', lambda usd_libor_ts, usd_libor_c_rates: FlatIRCurve(usd_libor_ts, usd_libor_c_rates))
comp.add_node('usd_libor_swap_rates',
              lambda usd_libor_curve, usd_libor_ts:
              np.vectorize(swap_rate)(0, usd_libor_ts, 0.25, usd_libor_curve, usd_libor_curve))
comp.draw_graphviz()
```

If we calculate all the nodes in Loman, we can plot the curve. Because we set all the continuously compounding rates to 3%, it's not a very interesting curve yet.

```python
comp.compute_all()
comp.value('usd_libor_curve').plot()
```

Now, to help calibrate our curve, we add a node that is market swap rates (**usd_libor_mkt_swap_rates**), and calculate the difference between the rates our curve is producing, and the market rates we are trying to fit to, in **usd_libor_fitting_error**.

```python
comp.add_node('usd_libor_mkt_swap_rates', value=np.array([0.01364, 0.01593, 0.01776, 0.02023, 0.02181, 0.02343, 0.02499, 0.02566, 0.02593]))
comp.add_node('usd_libor_fitting_error',
              lambda usd_libor_swap_rates, usd_libor_mkt_swap_rates:
                  usd_libor_swap_rates - usd_libor_mkt_swap_rates)
```

To calibrate our curve, we use a minimizer from scipy. Our objective function is to insert a trial set of continuously-compounded rates into the input **usd_libor_c_rates**, calculate the error vector **usd_libor_fitting_error** using Loman, and then return the sum of the squares (scaled appropriately for the solver). Our initial guess is taken from the current set of rates in **usd_libor_c_rates**.

```python
from scipy.optimize import minimize

def error(xs):
    comp.insert('usd_libor_c_rates', xs)
    comp.compute('usd_libor_fitting_error')
    return 10000. * np.sum(comp.value('usd_libor_fitting_error')**2)
res = minimize(error, comp.value('usd_libor_c_rates'))
res.success, res.nfev
```

    (True, 275)

The solver indicates that it rans successfully with 275 evaluations, so we insert its solution set of input rates into the computation (solvers aren't required that their last evaluation be the solution value), re-compute everything, and plot the resulting curve. We also show the market swap rates so we can see our calibration was successful.

```python
comp.insert('usd_libor_c_rates', res.x)
comp.compute_all()
comp.value('usd_libor_curve').plot()
plt.scatter(comp.value('usd_libor_ts'), comp.value('usd_libor_mkt_swap_rates'), label='Mkt Swap Rates')
plt.legend()
```

## Valuing a Portfolio of Interest Rate Swaps

Now that we have a valid interest rate curve, we can use it to value interest rate swaps. For now, our swaps will all be valued with using the same USD LIBOR curve.

We define a swap as a collection of named parameters, create a function to value them, and add a node **portfolio** to our computation, which is a set of swaps

```python
from collections import namedtuple
Swap = namedtuple('Swap', ['notional', 'start', 'end', 'rate', 'freq'])

def swap_pv(swap, projection_curve, discount_curve):
    fixed_pv, float_pv = swap_leg_pvs(swap.start, swap.end, swap.freq, projection_curve, discount_curve)
    return swap.notional * (float_pv - swap.rate * fixed_pv)

comp.add_node('portfolio', value=[
    Swap(10000000, 5, 10, 0.025, 0.25),
    Swap(-5000000, 2.5, 12.5, 0.02, 0.25)
])
```

To value our portfolio, we simply apply our valuation function to each position in our portfolio, and Loman gives us an array back, with the value of each position

```python
comp.add_node('portfolio_val',
              lambda portfolio, usd_libor_curve:
              [swap_pv(swap, usd_libor_curve, usd_libor_curve) for swap in portfolio])
comp.draw_graphviz()
```

```python
comp.compute_all()
comp.value('portfolio_val')
```

    [86395.045323357786, -274086.54029392399]

## Dual Bootstrap Curves

Since 2008, the spread between LIBOR and OIS has been material, and it is standard practice to calibrate curves to LIBOR swaps and LIBOR-OIS basis swaps, say. To start, we'll need some inputs, a curve, and the market rates.

```python
comp.add_node('usd_ois_ts', value=np.array([1., 2., 3., 5., 7., 10., 15., 20., 30.]))
comp.add_node('usd_ois_c_rates', value=0.03*np.ones(9))
comp.add_node('usd_ois_curve', lambda usd_ois_ts, usd_ois_c_rates: FlatIRCurve(usd_ois_ts, usd_ois_c_rates))
comp.add_node('usd_libor_ois_mkt_spreads', value=np.array([24.2, 26, 27.2, 29, 30.9, 33.4, 36.4, 38, 39.8])/10000.)
```

Now we need a function to calculate LIBOR-OIS spreads from our two curves using
\begin{equation} PV_\text{LIBOR float leg} = PV_\text{OIS float leg} + s \times PV_\text{OIS 1bp fixed leg} \end{equation}

```python
def swap_spread(a, b, p, projection_curve1, projection_curve2, discount_curve):
    fixed_pv1, float_pv1 = swap_leg_pvs(a, b, p, projection_curve1, discount_curve)
    fixed_pv2, float_pv2 = swap_leg_pvs(a, b, p, projection_curve2, discount_curve)
    return (float_pv1 - float_pv2) / fixed_pv2
```

We use that function to calculate LIBOR-OIS spreads using our two curves, as well as another fitting error vector that we will use in calibration.

```python
comp.add_node('usd_libor_ois_spreads',
              lambda usd_libor_curve, usd_ois_curve, usd_ois_ts:
              np.vectorize(swap_spread)(0, usd_ois_ts, 0.25, usd_libor_curve, usd_ois_curve, usd_ois_curve))

comp.add_node('usd_libor_ois_fitting_error',
              lambda usd_libor_ois_spreads, usd_libor_ois_mkt_spreads:
                  usd_libor_ois_spreads - usd_libor_ois_mkt_spreads)
```

We should also update our LIBOR swap rate calculation to use our OIS curve for discounting:

```python
comp.add_node('usd_libor_swap_rates',
              lambda usd_libor_curve, usd_ois_curve, usd_libor_ts:
              np.vectorize(swap_rate)(0, usd_libor_ts, 0.25, usd_libor_curve, usd_ois_curve))
```

Finally, for our calibration, it would be more convenient if we could just insert one vector, containing both LIBOR and OIS inputs, and have those feed through into **usd_libor_c_rates** and **usd_ois_c_rates**. Thanks to Loman, we can do this by giving them a common parent node. If we later want to insert values directly into **usd_libor_c_rates** and **usd_ois_c_rates**, then we can do that too, even though they are calculation nodes. We also define a node  **usd_fitting_error** to collect the fitting error vectors.

```python
comp.add_node('usd_c_rates', value=0.03*np.ones(18))
comp.add_node('usd_libor_c_rates', lambda usd_c_rates: usd_c_rates[0:9])
comp.add_node('usd_ois_c_rates', lambda usd_c_rates: usd_c_rates[9:18])
comp.add_node('usd_fitting_error',
              lambda usd_libor_fitting_error, usd_libor_ois_fitting_error:
              np.concatenate([usd_libor_fitting_error, usd_libor_ois_fitting_error]))
comp.draw_graphviz(graph_attr={'size': '10,8'})
```

Complicated.

But it's ok. Loman can take care of this.

We can just apply the same method as before. Having the solver insert sets of inputs, and iterate until it has minimized the error. This time, the set of inputs that it is adjusting flow into the construction of two separate curves, **usd_ois_curve**, and **usd_libor_curve**.

```python
def error(xs):
    comp.insert('usd_c_rates', xs)
    comp.compute('usd_fitting_error')
    return 10000. * np.sum(comp.value('usd_fitting_error')**2)
res = minimize(error, comp.value('usd_c_rates'))
res.success, res.nfev
```

    (True, 780)

```python
comp.insert('usd_c_rates', res.x)
comp.compute_all()
```

Finally, we can plot LIBOR-OIS spreads from our dual boot-strapped curve, as well as LIBOR and OIS swap rates, to check the calibration.

```python
ts = np.linspace(0.0, 29.75, 360)
spreads = np.vectorize(swap_spread)(0.0, ts, 0.25, comp.value('usd_libor_curve'), comp.value('usd_ois_curve'), comp.value('usd_ois_curve'))
plt.plot(ts, spreads, label='Model LIBOR-OIS Spreads')
plt.scatter(comp.value('usd_ois_ts'), comp.value('usd_libor_ois_mkt_spreads'), label='Market LIBOR-OIS Spreads')
plt.legend()
```

```python
comp.value('usd_libor_curve').plot()
plt.scatter(comp.value('usd_libor_ts'), comp.value('usd_libor_mkt_swap_rates'), label='Mkt Swap Rates')
plt.legend()
```

```python
comp.value('usd_ois_curve').plot()
```

## Revisiting Portfolio Valuation

Now that we have two curves, and our swaps could potentially be projected and discounted with different curves, it's appropriate to introduce a new node, **curveset** and have the valuation of our portfolio depend on that curveset, rather than pointing directly at specific curves. When we create **curveset**, we can also put in a check that our fit was good.

We update the design of our Swap instrument, and recreate the portfolio with the additional information.

By design, Loman lets us redefine the function that calculates portfolio_val and have it depend on **curveset** rather than **usd_libor_curve**, and the Swap instruments themselves direct curves to take from **curveset** to perform valuation.

```python
Swap = namedtuple('Swap', ['notional', 'start', 'end', 'rate', 'freq', 'projection_curve', 'discount_curve'])

def create_curveset(usd_libor_curve, usd_ois_curve, usd_fitting_error):
    if np.max(np.abs(usd_fitting_error) > 0.00001):
        raise Exception("Fitting error > 0.1bps")
    return {'USD-LIBOR-3M': usd_libor_curve, 'USD-OIS': usd_ois_curve}

comp.add_node('curveset', create_curveset)

comp.insert('portfolio', [
    Swap(10000000, 5, 10, 0.025, 0.25, 'USD-LIBOR-3M', 'USD-OIS'),
    Swap(-5000000, 2.5, 12.5, 0.02, 0.25, 'USD-LIBOR-3M', 'USD-OIS')
])

comp.add_node('portfolio_val',
              lambda portfolio, curveset:
              [swap_pv(swap, curveset[swap.projection_curve], curveset[swap.discount_curve]) for swap in portfolio])
comp.draw_graphviz(graph_attr={'size': '10,8'})
```

Finally, we can see that our valuation is close but materially different to the previous LIBOR-only curve's result (86,395 and -274,086), exactly as we expect.

```python
comp.compute('portfolio_val')
comp.value('portfolio_val')
```

    [85859.407115714959, -278864.22970371263]

# Risks

There are a few ways to calculate risk for a book of interest rate derivatives:

  * Spot risks directly. We can directly perturb the inputs **usd_libor_mkt_swap_rates** or **usd_libor_ois_mkt_spreads** one element at a time, recalibrating, and repricing the portfolio each time.
  * Spot risks, using Jacobians. We can perturb the inputs **usd_libor_c_rates** and **usd_ois_c_rates**, using the changes in **usd_libor_swap_rates** and **usd_libor_ois_spreads** to calculate a Jacobian. Applying the inverse of this Jacobian matrix on the matrix of portfolio valuation changes gives us the sensitivities to market instruments as above, without the need to recalibrate.
  * Forward risk. We can directly perturb the curve using $r'(t; a,b) = r(t) + \delta I_{a \le t < b}$ to create a new curve where the continuously-compounded rate is elevated in the region $[a,b)$. This will cause the forward swap rate to change by some amount, and by scaling $\delta$, we can make that amount 1bp (or we can scale the PV changes to the same effect), so we can get the risk for our portfolio to a set of forwards, potentially with much higher granularity than our initial set of calibration instruments. Additionally this method is more numerically stable when using non-local interpolations for curve construction.

For this exposition, we will go with the latter method, and to keep things moving along, we will only produce risks to the LIBOR curve.

To accommodate this change, we'll create a new node called **usd_libor_curve_perturbed** which will be fed into **curveset** for valuation, and will itself be created from **usd_libor_curve** and a control input called **usd_libor_curve_perturbation**.

First things first, we need to define how to create a perturbed curve, using the recipe above. Note that once we've defined r and r_quad, everything else will flow through

```python
class SumCurve(BaseIRCurve):
    def __init__(self, *curves):
        self.curves = curves
    def r_quad(self, s, t):
        return sum(curve.r_quad(s, t) for curve in self.curves)
    def r(self, t):
        return sum(curve.r(t) for curve in self.curves)

def create_perturbed_curve(curve, start, end, amount):
    pert_curv = FlatIRCurve(np.array([start, end, np.max(curve.ts)]), np.array([0., amount, 0.]))
    return SumCurve(curve, pert_curv)
```

To test this is working, we can see what happens when we bump the continuously-compounded rate by 5% between years 5 and 10 for example.

```python
create_perturbed_curve(comp.value('usd_libor_curve'), 5., 10., 0.05).plot()
```

That looks as we expect, so we can go ahead and add our new nodes:

```python
comp.add_node('usd_libor_curve_perturbation', value=(0, 1, 0.))
comp.add_node('usd_libor_curve_perturbed',
              lambda usd_libor_curve, usd_libor_curve_perturbation:
              create_perturbed_curve(usd_libor_curve, *usd_libor_curve_perturbation))
def create_curveset(usd_libor_curve_perturbed, usd_ois_curve, usd_fitting_error):
    if np.max(np.abs(usd_fitting_error) > 0.00001):
        raise Exception("Fitting error > 0.1bps")
    return {'USD-LIBOR-3M': usd_libor_curve_perturbed, 'USD-OIS': usd_ois_curve}

comp.add_node('curveset', create_curveset)

comp.draw_graphviz(graph_attr={'size': '10,8'})
```

We can iterate over each year, and see the sensitivity of the portfolio to a change of 1bp in the forward swap rate for that year:

```python
comp.compute('portfolio_val')
comp.insert('usd_libor_curve_perturbation', value=(0, 1, 0.))
base_value = np.array(comp.value('portfolio_val'))
pert_values = np.empty((30, base_value.shape[0]))
delta_value = np.empty((30, base_value.shape[0]))
ts = np.arange(30)
for i in ts:
    comp.insert('usd_libor_curve_perturbation', value=(i, i+1, 0.0001))
    comp.compute('portfolio_val')
    pert_values[i] = comp.value('portfolio_val')
    swap_rate_base = swap_rate(i, i+1, 0.25, comp.value('usd_libor_curve'), comp.value('usd_ois_curve'))
    swap_rate_pert = swap_rate(i, i+1, 0.25, comp.value('usd_libor_curve_perturbed'), comp.value('usd_ois_curve'))
    swap_rate_delta = swap_rate_pert - swap_rate_base
    delta_value[i] = 0.0001 / swap_rate_delta * (pert_values[i] - base_value)

plt.bar(ts, delta_value.sum(axis=1), np.diff(np.concatenate([[0],ts])))
```

## Conclusion

This concludes our interest example. We have shown that Loman makes it easy to simultaneously calibrate interest rate curves to market quotes, to value portfolios of interest rate derivatives, and to produce bucketed sensitivities.

The portfolio valuation also naturally extends to cover other interest rate derivatives.

The calibration approach - an external routine driving Loman to produce calibrated inputs for the computation - easily and naturally extends to calibrating to a global set of interest rate markets, with curveset drawing from many separate independent and interdependent calibrations. It can also extend to other markets, such as FX derivatives, listed options, credit derivatives.

Loman's ability to serialize computations allows the calibration to happen once, centrally, and be broadcast to user desktops for firm-wide consistent live valuation.

In short, we have only scratched the surface of what is possible with Loman with this example, and we look forward to further exploration in the future.

examples/Example - Using Loman to Price Interest Rate Swaps.ipynb


# 微積分の計算について　　　N0.3　不定積分の内容－2  置換積分

###  学籍番号[＿＿＿＿＿＿＿＿＿]クラス[＿＿＿＿＿]　クラス番号[＿＿＿＿＿] 名前[＿＿＿＿＿＿＿＿＿＿＿＿＿＿＿]

##### 積分の手法　（１）置換積分法

$$  t= g(x) または　x=u(t) $$

$$ \int f(x)dx = \int f(u)\frac{du}{dt}dt $$

```python
from sympy import *
x, n , y, a = symbols('x n y a')
init_printing()
m ='3-2'
i =0
```

 ### 例題ー１

```python
expr = (x+1)**2
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python
expr = sqrt(x+1)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python
expr = cos(2*x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
expr = cos(x)**3
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python
itg.doit()
```

```python
expr = x/(x+1)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python
expr = 1/(x-1)**2
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python
expr = x/sqrt(x**2+a**2)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python
expr = sin(x)**2 * cos(x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
itg.doit()
```

```python

```

```python

```


# Diffusion in heterogeneous media

Diffusion in heterogeneous media normally implies a non-constant
diffusion coefficient $\alpha = \alpha (x)$.
A 1D diffusion model with such a variable diffusion coefficient reads

$$
\begin{equation}
\frac{\partial u}{\partial t} =
\frac{\partial}{\partial x}\left( \alpha (x) \frac{\partial u}{\partial x}
\right) + f(x,t), \quad x\in (0,L),\ t\in (0,T],
\label{diffu:pde2} \tag{1}
\end{equation}
$$

$$
\begin{equation}
u(x,0) = I(x), \quad  x\in [0,L],
\label{diffu:pde2:ic:u} \tag{2}
\end{equation}
$$

$$
\begin{equation}
u(0,t)  = U_0, \quad  t>0,
\label{diffu:pde2:bc:0} \tag{3}
\end{equation}
$$

$$
\begin{equation}
u(L,t)  = U_L, \quad  t>0.
\label{diffu:pde2:bc:L} \tag{4}
\end{equation}
$$

A short form of the diffusion equation with variable coefficients is
$u_t = (\alpha u_x)_x + f$.

## Discretization

We can discretize ([1](#diffu:pde2)) by a $\theta$-rule in time
and centered differences in space:

$$
\lbrack D_t u\rbrack^{n+\frac{1}{2}}_i = \theta\lbrack D_x(\overline{\dfc}^x
D_x u) + f\rbrack^{n+1}_i +
(1-\theta)\lbrack D_x(\overline{\dfc}^x
D_x u) + f\rbrack^{n}_i\thinspace .
$$

Written out, this becomes

$$
\begin{align*}
\frac{u^{n+1}_i-u^{n}_i}{\Delta t} &=
\theta\frac{1}{\Delta x^2}
(\dfc_{i+\frac{1}{2}}(u^{n+1}_{i+1} - u^{n+1}_{i})
- \dfc_{i-\frac{1}{2}}(u^{n+1}_i - u^{n+1}_{i-1})) +\\
&\quad (1-\theta)\frac{1}{\Delta x^2}
(\dfc_{i+\frac{1}{2}}(u^{n}_{i+1} - u^{n}_{i})
- \dfc_{i-\frac{1}{2}}(u^{n}_i - u^{n}_{i-1})) +\\
&\quad \theta f_i^{n+1} + (1-\theta)f_i^{n},
\end{align*}
$$

where, e.g., an arithmetic mean can to be used for $\dfc_{i+\frac{1}{2}}$:

$$
\dfc_{i+\frac{1}{2}} = \frac{1}{2}(\dfc_i + \dfc_{i+1})\thinspace .
$$

## Implementation

Suitable code for solving the discrete equations is very similar to
what we created for a constant $\dfc$.
Since the Fourier number has no meaning for varying
$\dfc$, we introduce a related parameter $D=\Delta t /\Delta x^2$.

```python
def solver_theta(I, a, L, Nx, D, T, theta=0.5, u_L=1, u_R=0,
                 user_action=None):
    x = linspace(0, L, Nx+1)   # mesh points in space
    dx = x[1] - x[0]
    dt = D*dx**2
    Nt = int(round(T/float(dt)))
    t = linspace(0, T, Nt+1)   # mesh points in time

    u   = zeros(Nx+1)   # solution array at t[n+1]
    u_n = zeros(Nx+1)   # solution at t[n]

    Dl = 0.5*D*theta
    Dr = 0.5*D*(1-theta)

    # Representation of sparse matrix and right-hand side
    diagonal = zeros(Nx+1)
    lower    = zeros(Nx)
    upper    = zeros(Nx)
    b        = zeros(Nx+1)

    # Precompute sparse matrix (scipy format)
    diagonal[1:-1] = 1 + Dl*(a[2:] + 2*a[1:-1] + a[:-2])
    lower[:-1] = -Dl*(a[1:-1] + a[:-2])
    upper[1:]  = -Dl*(a[2:] + a[1:-1])
    # Insert boundary conditions
    diagonal[0] = 1
    upper[0] = 0
    diagonal[Nx] = 1
    lower[-1] = 0

    A = scipy.sparse.diags(
        diagonals=[diagonal, lower, upper],
        offsets=[0, -1, 1],
        shape=(Nx+1, Nx+1),
        format='csr')

    # Set initial condition
    for i in range(0,Nx+1):
        u_n[i] = I(x[i])

    if user_action is not None:
        user_action(u_n, x, t, 0)

    # Time loop
    for n in range(0, Nt):
        b[1:-1] = u_n[1:-1] + Dr*(
            (a[2:] + a[1:-1])*(u_n[2:] - u_n[1:-1]) -
            (a[1:-1] + a[0:-2])*(u_n[1:-1] - u_n[:-2]))
        # Boundary conditions
        b[0]  = u_L(t[n+1])
        b[-1] = u_R(t[n+1])
        # Solve
        u[:] = scipy.sparse.linalg.spsolve(A, b)

        if user_action is not None:
            user_action(u, x, t, n+1)

        # Switch variables before next step
        u_n, u = u, u_n
```

The code is found in the file [`diffu1D_vc.py`](${src_diffu}/diffu1D_vc.py).

## Stationary solution

As $t\rightarrow\infty$, the solution of the
problem ([1](#diffu:pde2))-([4](#diffu:pde2:bc:L))
will approach
a stationary limit where $\partial u/\partial t=0$. The governing
equation is then

$$
\begin{equation}
\frac{d}{dx}\left(\alpha\frac{du}{dx}\right) =0,
\label{diffu:fd2:pde:st} \tag{5}
\end{equation}
$$

with boundary conditions $u(0)=U_0$ and $u(L)=U_L$.
mathcal{I}_t is possible to obtain an exact solution of ([5](#diffu:fd2:pde:st))
for any $\alpha$. Integrating twice and applying the boundary conditions
to determine the integration constants gives

$$
\begin{equation}
u(x) = U_0 + (U_L-U_0)\frac{\int_0^x (\alpha(\xi))^{-1}d\xi}{\int_0^L (\alpha(\xi))^{-1}d\xi}
\thinspace .
\label{diffu:fd2:pde:st:sol} \tag{6}
\end{equation}
$$

## Piecewise constant medium

Consider a medium built of $M$ layers. The layer boundaries
are denoted $b_0, \ldots, b_M$,
where $b_0=0$ and $b_M=L$.
If the layers potentially have different material properties, but
these properties are constant within each layer, we can express $\alpha$ as a
*piecewise constant function* according to

$$
\begin{equation}
\alpha (x) = \left\lbrace\begin{array}{ll}
\alpha_0,& b_0 \leq x < b_1,\\
\vdots &\\
\alpha_i,& b_i \leq x < b_{i+1},\\
\vdots &\\
\alpha_{M-1},& b_{M-1} \leq x \leq b_M.
\end{array}\right.
\end{equation}
\label{diffu:fd2:pde:st:pc:alpha} \tag{7}
$$

The exact solution ([6](#diffu:fd2:pde:st:sol)) in case of such a
piecewise constant $\alpha$ function is easy to derive. Assume that
$x$ is in the $m$-th layer: $x\in [b_m, b_{m+1}]$. In the integral
$\int_0^x (a(\xi))^{-1}d\xi$ we must integrate through the first
$m-1$ layers and then add the contribution from the remaining part
$x-b_m$ into the $m$-th layer:

$$
\begin{equation}
u(x) = U_0 + (U_L-U_0)
\frac{\sum_{j=0}^{m-1} (b_{j+1}-b_j)/\alpha(b_j) + (x-b_m)/\alpha(b_m)}{\sum_{j=0}^{M-1} (b_{j+1}-b_j)/\alpha(b_j)}
\label{diffu:fd2:pde:st:sol:pc} \tag{8}
\end{equation}
$$

**Remark.**
mathcal{I}_t may sound strange to have a discontinuous $\alpha$ in a differential
equation where one is to differentiate, but a discontinuous $\alpha$
is compensated by a discontinuous $u_x$ such that $\alpha u_x$ is
continuous and therefore can be differentiated as $(\alpha u_x)_x$.

## Implementation of diffusion in a piecewise constant medium

Programming with piecewise function definitions quickly becomes
cumbersome as the most naive approach is to test for which interval
$x$ lies, and then start evaluating a formula like
([8](#diffu:fd2:pde:st:sol:pc)). In Python, vectorized expressions may
help to speed up the computations.
The convenience classes `PiecewiseConstant` and
`IntegratedPiecewiseConstant` in the [`Heaviside`](${src_diffu}/Heaviside.py)
module were made to simplify programming with
functions like ([7](#diffu:fd2:pde:st:pc:alpha)) and expressions like
([8](#diffu:fd2:pde:st:sol:pc)). These utilities not only represent
piecewise constant functions, but also *smoothed* versions of them
where the discontinuities can be smoothed out in a controlled fashion.

The `PiecewiseConstant` class is created by sending in the domain as a
2-tuple or 2-list and a `data` object describing the boundaries
$b_0,\ldots,b_M$ and the corresponding function values
$\alpha_0,\ldots,\alpha_{M-1}$. More precisely, `data` is a nested
list, where `data[i][0]` holds $b_i$ and `data[i][1]` holds the
corresponding value $\alpha_i$, for $i=0,\ldots,M-1$. Given $b_i$ and
$\alpha_i$ in arrays `b` and `a`, it is easy to fill out the nested
list `data`.

In our application, we want to represent $\alpha$ and $1/\alpha$
as piecewise constant functions, in addition to the $u(x)$ function
which involves the integrals of $1/\alpha$. A class creating the
functions we need and a method for evaluating $u$, can take the
form

```python
class SerialLayers:
    """
    b: coordinates of boundaries of layers, b[0] is left boundary
    and b[-1] is right boundary of the domain [0,L].
    a: values of the functions in each layer (len(a) = len(b)-1).
    U_0: u(x) value at left boundary x=0=b[0].
    U_L: u(x) value at right boundary x=L=b[0].
    """

    def __init__(self, a, b, U_0, U_L, eps=0):
        self.a, self.b = np.asarray(a), np.asarray(b)
        self.eps = eps  # smoothing parameter for smoothed a
        self.U_0, self.U_L = U_0, U_L

        a_data = [[bi, ai] for bi, ai in zip(self.b, self.a)]
        domain = [b[0], b[-1]]
        self.a_func = PiecewiseConstant(domain, a_data, eps)

        # inv_a = 1/a is needed in formulas
        inv_a_data = [[bi, 1./ai] for bi, ai in zip(self.b, self.a)]
        self.inv_a_func = \
             PiecewiseConstant(domain, inv_a_data, eps)
        self.integral_of_inv_a_func = \
             IntegratedPiecewiseConstant(domain, inv_a_data, eps)
        # Denominator in the exact formula is constant
        self.inv_a_0L = self.integral_of_inv_a_func(b[-1])

    def __call__(self, x):
        solution = self.U_0 + (self.U_L-self.U_0)*\
                   self.integral_of_inv_a_func(x)/self.inv_a_0L
        return solution
```

A visualization method is also convenient to have. Below we plot $u(x)$
along with $\alpha (x)$ (which works well as long as $\max \alpha(x)$
is of the same size as $\max u = \max(U_0,U_L)$).

```python
%matplotlib inline

class SerialLayers:
    ...

    def plot(self):
        x, y_a = self.a_func.plot()
        x = np.asarray(x); y_a = np.asarray(y_a)
        y_u = self.u_exact(x)
        import matplotlib.pyplot as plt
        plt.figure()
        plt.plot(x, y_u, 'b')
        plt.hold('on')  # Matlab style
        plt.plot(x, y_a, 'r')
        ymin = -0.1
        ymax = 1.2*max(y_u.max(), y_a.max())
        plt.axis([x[0], x[-1], ymin, ymax])
        plt.legend(['solution $u$', 'coefficient $a$'], loc='upper left')
        if self.eps > 0:
            plt.title('Smoothing eps: %s' % self.eps)
        plt.savefig('tmp.pdf')
        plt.savefig('tmp.png')
        plt.show()
```

[Figure](#diffu:fd2:pde:st:sol:pc:fig1) shows the case where

```python
b = [0, 0.25, 0.5, 1]   # material boundaries
a = [0.2, 0.4, 4]       # material values
U_0 = 0.5;  U_L = 5     # boundary conditions
```

Solution of the stationary diffusion equation corresponding to a piecewise constant diffusion coefficient.

By adding the `eps` parameter to the constructor of the `SerialLayers`
class, we can experiment with smoothed versions of $\alpha$ and see
the (small) impact on $u$. [Figure](#diffu:fd2:pde:st:sol:pc:fig2)
shows the result.

Solution of the stationary diffusion equation corresponding to a smoothed piecewise constant diffusion coefficient.

## Axi-symmetric diffusion

Suppose we have a diffusion process taking place in a straight tube
with radius $R$. We assume axi-symmetry such that $u$ is just a
function of $r$ and $t$, with $r$ being the radial distance from the center
axis of the tube to a point. With such axi-symmetry it is
advantageous to introduce *cylindrical coordinates* $r$, $\theta$, and
$z$, where $z$ is in the direction of the tube and $(r,\theta)$ are
polar coordinates in a cross section. Axi-symmetry means that all
quantities are independent of $\theta$. From the relations $x=\cos\theta$,
$y=\sin\theta$, and $z=z$, between Cartesian and cylindrical coordinates,
one can (with some effort) derive the diffusion equation in cylindrical
coordinates, which with axi-symmetry takes the form

$$
\frac{\partial u}{\partial t} = \frac{1}{r}\frac{\partial}{\partial r}
\left(r\dfc(r,z)\frac{\partial u}{\partial r}\right) + \frac{\partial}{\partial z}
\left(\alpha(r,z)\frac{\partial u}{\partial z}\right) + f(r,z,t)\thinspace .
$$

Let us assume that $u$ does not change along the tube axis so it
suffices to compute variations in a cross section. Then $\partial u/\partial
z = 0$ and we have a 1D diffusion equation in the radial coordinate
$r$ and time $t$. In particular, we shall address the initial-boundary
value problem

$$
\begin{equation}
\frac{\partial u}{\partial t} = \frac{1}{r}\frac{\partial}{\partial r}
\left(r\dfc(r)\frac{\partial u}{\partial r}\right) + f(t),  r\in (0,R),\ t\in (0,T],
\label{diffu:fd2:radial:PDE} \tag{9}
\end{equation}
$$

$$
\begin{equation}
\frac{\partial u}{\partial r}(0,t) = 0,  t\in (0,T],
\label{diffu:fd2:radial:symmr0} \tag{10}
\end{equation}
$$

$$
\begin{equation}
u(R,t) = 0,  t\in (0,T],
\label{diffu:fd2:radial:uR} \tag{11}
\end{equation}
$$

$$
\begin{equation}
u(r,0) = I(r),  r\in [0,R].
\label{diffu:fd2:radial:initial} \tag{12}
\end{equation}
$$

The condition ([10](#diffu:fd2:radial:symmr0)) is a necessary symmetry condition
at $r=0$, while ([11](#diffu:fd2:radial:uR)) could be any Dirichlet
or Neumann condition (or Robin condition in case of cooling or heating).

The finite difference approximation will need the discretized version
of the PDE for $r=0$ (just as we use the PDE at the boundary when
implementing Neumann conditions). However, discretizing the PDE at
$r=0$ poses a problem because of the $1/r$ factor. We therefore need
to work out the PDE for discretization at $r=0$ with care.
Let us, for the case of constant $\dfc$, expand the spatial derivative term to

$$
\alpha\frac{\partial^2 u}{\partial r^2} + \alpha\frac{1}{r}\frac{\partial u}{\partial r}\thinspace .
$$

The last term faces a difficulty at $r=0$, since it becomes a $0/0$ expression
caused by the symmetry condition at $r=0$.
However, L'Hosptial's rule can be used:

$$
\lim_{r\rightarrow 0} \frac{1}{r}\frac{\partial u}{\partial r}
= \frac{\partial^2 u}{\partial r^2}\thinspace .
$$

The PDE at $r=0$ therefore becomes

$$
\begin{equation}
\frac{\partial u}{\partial t} = 2\dfc\frac{\partial^2 u}{\partial r^2}
+ f(t)\thinspace .
\label{diffu:fd2:radial:eq_PDEr0:aconst} \tag{13}
\end{equation}
$$

For a variable coefficient $\dfc(r)$ the expanded spatial derivative term reads

$$
\dfc(r)\frac{\partial^2 u}{\partial r^2} +
\frac{1}{r}(\dfc(r) + r\dfc'(r))\frac{\partial u}{\partial r}\thinspace .
$$

We are interested in this expression for $r=0$. A necessary condition
for $u$ to be axi-symmetric is that all input data, including $\alpha$,
must also be axi-symmetric, implying that $\alpha'(0)=0$ (the second
term vanishes anyway because of $r=0$). The limit of interest is

$$
\lim_{r\rightarrow 0}
\frac{1}{r}\dfc(r)\frac{\partial u}{\partial r} =
\dfc(0)\frac{\partial^2 u}{\partial r^2}\thinspace .
$$

The PDE at $r=0$ now looks like

$$
\begin{equation}
\frac{\partial u}{\partial t} = 2\dfc(0)
\frac{\partial^2 u}{\partial r^2}
+ f(t),
\label{diffu:fd2:radial:eq_PDEr0:avar} \tag{14}
\end{equation}
$$

so there is no essential difference between the constant coefficient
and variable coefficient cases.

The second-order derivative in ([13](#diffu:fd2:radial:eq_PDEr0:aconst))
and ([14](#diffu:fd2:radial:eq_PDEr0:avar))
is discretized in the usual way.

$$
2\dfc\frac{\partial^2}{\partial r^2}u(r_0,t_n) \approx
[2\dfc D_rD_r u]^n_0 =
2\dfc \frac{u^{n}_{1} - 2u^{n}_0 + u^n_{-1}}{\Delta r^2}\thinspace .
$$

The fictitious value $u^n_{-1}$ can be eliminated using the discrete
symmetry condition

$$
[D_{2r} u =0]^n_0 \quad\Rightarrow\quad u^n_{-1} = u^n_1,
$$

which then gives the modified approximation to the term with the second-order derivative
of $u$ in $r$ at $r=0$:

$$
\begin{equation}
4\dfc \frac{u^{n}_{1} - u^{n}_0}{\Delta r^2}\thinspace .
\label{_auto1} \tag{15}
\end{equation}
$$

The discretization of the term with the second-order derivative in $r$ at any
internal mesh point is straightforward:

$$
\begin{align*}
\left[\frac{1}{r}\frac{\partial}{\partial r}
\left(r\dfc\frac{\partial u}{\partial r}\right)\right]_i^n
& \approx [r^{-1} D_r (r \dfc D_r u)]_i^n\\
&= \frac{1}{r_i}\frac{1}{\Delta r^2}\left(
r_{i+\frac{1}{2}}\dfc_{i+\frac{1}{2}}(u_{i+1}^n - u_i^n) - r_{i-\frac{1}{2}}\dfc_{i-\frac{1}{2}}(u_{i}^n - u_{i-1}^n)\right)\thinspace .
\end{align*}
$$

To complete the discretization, we need a scheme in time, but that can
be done as before and does not interfere with the discretization in space.

## Spherically-symmetric diffusion

### Discretization in spherical coordinates

Let us now pose the problem from the section [Axi-symmetric diffusion](#diffu:fd2:radial)
in spherical coordinates, where $u$ only depends on the radial coordinate
$r$ and time $t$. That is, we have spherical symmetry.
For simplicity we restrict the diffusion coefficient $\dfc$ to be
a constant. The PDE reads

$$
\begin{equation}
\frac{\partial u}{\partial t} = \frac{\dfc}{r^\gamma}\frac{\partial}{\partial r}
\left(r^\gamma\frac{\partial u}{\partial r}\right) + f(t),
\label{_auto2} \tag{16}
\end{equation}
$$

for $r\in (0,R)$ and $t\in (0,T]$. The parameter $\gamma$ is 2 for
spherically-symmetric problems and 1 for axi-symmetric problems.
The boundary and initial conditions
have the same mathematical form as
in ([9](#diffu:fd2:radial:PDE))-([12](#diffu:fd2:radial:initial)).

Since the PDE in spherical coordinates has the same form as the PDE
in the section [Axi-symmetric diffusion](#diffu:fd2:radial), just with the $\gamma$ parameter
being different, we can use the same discretization approach.
At the origin $r=0$ we get problems with the term

$$
\frac{\gamma}{r}\frac{\partial u}{\partial t},
$$

but L'Hosptial's rule shows that this term equals $\gamma\partial^2 u/
\partial r^2$, and the PDE at $r=0$ becomes

$$
\begin{equation}
\frac{\partial u}{\partial t} = (\gamma+1)\dfc\frac{\partial^2 u}{\partial r^2}
+ f(t)\thinspace .
\label{_auto3} \tag{17}
\end{equation}
$$

The associated discrete form is then

$$
\begin{equation}
[D_t u = \frac{1}{2} (\gamma+1)\dfc D_rD_r \overline{u}^t + \overline{f}^t]^{n+\frac{1}{2}}_i,
\label{_auto4} \tag{18}
\end{equation}
$$

for a Crank-Nicolson scheme.

### Discretization in Cartesian coordinates

The spherically-symmetric spatial derivative can be transformed to
the Cartesian counterpart by introducing

$$
v(r,t) = ru(r,t)\thinspace .
$$

Inserting $u=v/r$ in

$$
\frac{1}{r^2}\frac{\partial}{\partial r}
\left(\dfc(r)r^2\frac{\partial u}{\partial r}\right),
$$

yields

$$
r\left(\frac{d \dfc}{dr}\frac{\partial v}{\partial r} +
\dfc\frac{\partial^2 v}{\partial r^2}\right) - \frac{d \dfc}{dr}v
\thinspace .
$$

The two terms in the parenthesis can be combined to

$$
r\frac{\partial}{\partial r}\left( \dfc\frac{\partial v}{\partial r}\right)\thinspace .
$$

The PDE for $v$ takes the form

$$
\begin{equation}
\frac{\partial v}{\partial t} = \frac{\partial}{\partial r}\left( \dfc
\frac{\partial v}{\partial r}\right) - \frac{1}{r}\frac{d\dfc}{dr}v + rf(r,t),
\quad r\in (0,R),\ t\in (0,T]\thinspace .
\label{_auto5} \tag{19}
\end{equation}
$$

For $\alpha$ constant we immediately realize that we can reuse a
solver in Cartesian coordinates to compute $v$. With variable $\alpha$,
a "reaction" term $v/r$ needs to be added to the Cartesian solver.
The boundary condition $\partial u/\partial r=0$ at $r=0$, implied
by symmetry, forces $v(0,t)=0$, because

$$
\frac{\partial u}{\partial r} = \frac{1}{r^2}\left(
r\frac{\partial v}{\partial r} - v\right) = 0,\quad r=0\thinspace .
$$


```python
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    try:
        tmp = DX(str(float(obj)))
        obj = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(obj)):
            tmp = DX(str(float(obj[i])))
            obj[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return obj

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""
# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm, deg) = ( m/1000, m/100, pi/180)
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton

half = S(1)/2

# ---

n0, v0, t, r0 = var("n0, v0, t, r0")
w0 = n0 * 2*pi
phi = w0 * t
cp , sp = cos(phi), sin(phi)

R = r0 + v0 * t
Rp = diff(R, t)

sub_list=[
    ( n0, 1 /s   / 10 ),
    ( v0, 2 *m/s / 10 ),
    (  t, 4 *s        ),
    ( r0, 2 *m   / 10 ),
    ]

vx = Rp * cp - R * w0 *sp
vy = Rp * sp + R * w0 *cp

v = Matrix([vx, vy])
# v for t = 4 s:
v4 = v.subs(sub_list)
vabs = v4.norm()
pprint("\n|v (4s)| / (m/s):")
tmp = vabs
tmp /= m/s
tmp = iso_round(tmp,0.001)
pprint(tmp)

# |v (4s)| / (m/s):
# 0.659

```


```julia

using JuliaAcademyData; activate("Foundations of machine learning")
```

      Updating registry at `~/.julia/registries/General`
      Updating git-repo `https://github.com/JuliaRegistries/General.git`

# Modeling data 2

## Building a model

Recall that in notebook 3, we saw that we could use a mathematical function to classify an image as an apple or a banana, based on the average amount of green in an image:

A common function for performing this kind of **classification** is the sigmoid that we saw in the last notebook, and that we will now extend by adding two **parameters**, $w$ and $b$:

$$\sigma(x; w, b) := \frac{1}{1 + \exp(-wx + b)}$$

$$ x = \mathrm{data} $$

\begin{align}
\sigma(x;w,b) &\approx 0 \implies \mathrm{apple} \\
\sigma(x;w,b) &\approx 1 \implies \mathrm{banana}
\end{align}

In our mathematical notation above, the `;` in the function differentiates between the **data** and the **parameters**. `x` is the data and is determined from the image. The parameters, `w` and `b`, are numbers which we choose to make our function match the results it should be modeling.

Note that in the code below, we don't distinguish between data and parameters - both are just inputs to our function, σ!

```julia
using Images, Statistics

apple = load(datapath("data/10_100.jpg"))
banana = load(datapath("data/104_100.jpg"))

apple_green_amount = mean(Float64.(green.(apple)))
banana_green_amount = mean(Float64.(green.(banana)))

println("Average green for apple = $apple_green_amount")
println("Average green for banana = $banana_green_amount")
```

    Average green for apple = 0.33811764705882347
    Average green for banana = 0.8806921568627452

```julia
σ(x, w, b) = 1 / (1 + exp(-w * x + b))
```

    σ (generic function with 1 method)

What we want is that when we give σ as input the average green for the apple, roughly `x = 0.3385`, it should return as output something close to 0, meaning "apple". And when we give σ the input `x = 0.8808`, it should output something close to 1, meaning "banana".

By changing the parameters of the function, we can change the shape of the function, and hence make it represent, or **fit**, the data better!

## Data fitting by varying parameters

We can understand how our choice of `w` and `b` affects our model by seeing how our values for `w` and `b` change the plot of the $\sigma$ function.

```julia
using Plots; gr()   # GR works better for interactive manipulations
```

    ┌ Info: Recompiling stale cache file /Users/ameilij/.julia/compiled/v1.0/Plots/ld3vC.ji for Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
    └ @ Base loading.jl:1190

    Plots.GRBackend()

Run the code in the next cell. You should see two "sliders" appear, one for `w` and one for `b`.

**Game**:
Change w and b around until the blue curve, labeled "model", which is the graph of the `\sigma` function, passes through *both* of the data points at the same time.

```julia
w = 10.0 # try manipulating w between -10 and 30
b = 10.0 # try manipulating b between 0 and 20

plot(x -> σ(x, w, b), xlim=(-0,1), ylim=(-0.1,1.1), label="model", legend=:topleft, lw=3)

scatter!([apple_green_amount],  [0.0], label="apple", ms=5)   # marker size = 5
scatter!([banana_green_amount], [1.0], label="banana", ms=5)
```

Notice that the two parameters do two very different things. The **weight**, `w`, determines *how fast* the transition between 0 and 1 occurs. It encodes how trustworthy we think our data  actually is, and in what range we should be putting points between 0 and 1 and thus calling them "unsure". The **bias**, `b`, encodes *where* on the $x$-axis the switch should take place. It can be seen as shifting the function left-right. We'll come to understand these *parameters* more in notebook 6.

Here are some parameter choices that work well:

```julia
w = 25.58; b = 15.6

plot(x -> σ(x, w, b), xlim=(0,1), ylim=(-0.1,1.1), label="model", legend=:topleft, lw=3)

scatter!([apple_green_amount], [0.0], label="apple")
scatter!([banana_green_amount],[1.0], label="banana")
```

(Note that in this problem there are many combinations of `w` and `b` that fit the data well.)

Once we have a model, we have a computational representation for how to choose between "apple" and "banana". So let's pull in some new images and see what our model says about them!

```julia
apple2 = load(datapath("data/107_100.jpg"))
```

```julia
green_amount = mean(Float64.(green.(apple2)))
@show green_amount

scatter!([green_amount], [0.0], label="new apple")
```

    green_amount = 0.4687290196078432

Our model successfully says that our new image is an apple! Pat yourself on the back: you've actually just trained your first neural network!

#### Exercise 1

Load the image of a banana in `data/8_100.jpg` as `mybanana`. Edit the code below to calculate the amount of green in `mybanana` and to overlay data for this image with the existing model and data points.

```julia
# load new banana
mybanana = load(datapath("data/8_100.jpg"))
mybanana_green_amount = mean(Float64.(green.(mybanana)))

w = 25.58; b = 15.6

plot(x -> σ(x, w, b), xlim=(0,1), ylim=(-0.1,1.1), label="model", legend=:topleft, lw=3)

scatter!([apple_green_amount], [0.0], label="apple")
scatter!([banana_green_amount],[1.0], label="banana")
scatter!([mybanana_green_amount], [1.0], label="mybanana")
```

#### Solution

To get the desired overlay, the code we need is

```julia
mybanana = load(datapath("data/8_100.jpg"))
mybanana_green_amount = mean(Float64.(green.(banana)))
scatter!([mybanana_green_amount], [1.0], label="my banana")
```

## Closing remarks: bigger models, more data, more accuracy

That last apple should start making you think: not all apples are red; some are yellow. "Redness" is one attribute of being an apple, but isn't the whole thing. What we need to do is incorporate more ideas into our model by allowing more inputs. However, more inputs would mean more parameters to play with. Also, we would like to have the computer start "learning" on its own, instead of modifying the parameters ourselves until we think it "looks right". How do we take the next step?

The first thing to think about is, if you wanted to incorporate more data into the model, how would you change the sigmoid function? Play around with some ideas. But also, start thinking about how you chose parameters. What process did you do to finally end up at good parameters? These two problems (working with models with more data and automatically choosing parameters) are the last remaining step to understanding deep learning.


```python
import pyprob
from pyprob import Model
from pyprob.distributions import Normal, Uniform

import torch
import numpy as np
import math
import matplotlib.pyplot as plt
%matplotlib inline
fig = plt.figure();
```

    <Figure size 432x288 with 0 Axes>

# Defining the model
First, we define the model as a probabilistic program inheriting from `pyprob.Model`. Models inherit from `torch.nn.Module` and can be potentially trained with gradient-based optimization (not covered in this example).

The `forward` function can have any number and type of arguments as needed.

```python
class GaussianUnknownMean(Model):
    def __init__(self):
        super().__init__(name='Gaussian with unknown mean (Marsaglia)') # give the model a name
        self.prior_mean = 1
        self.prior_std = math.sqrt(5)
        self.likelihood_std = math.sqrt(2)

    def marsaglia(self, mean, stddev):
        uniform = Uniform(-1, 1)
        s = 1
        while float(s) >= 1:
            x = pyprob.sample(uniform)
            y = pyprob.sample(uniform)
            s = x*x + y*y
        return mean + stddev * (x * torch.sqrt(-2 * torch.log(s) / s))

    def forward(self): # Needed to specifcy how the generative model is run forward
        # sample the (latent) mean variable to be inferred:
        mu = self.marsaglia(self.prior_mean, self.prior_std)

        # define the likelihood
        likelihood = Normal(mu, self.likelihood_std)

        # Lets add two observed variables
        # -> the 'name' argument is used later to assignment values:
        pyprob.observe(likelihood, name='obs0') # NOTE: observe -> denotes observable variables
        pyprob.observe(likelihood, name='obs1')

        # return the latent quantity of interest
        return mu

model = GaussianUnknownMean()
```

# Finding the correct posterior analytically
Altough this model seems to implement complicated distributions, the likelihood is the same as Gaussian unknown mean. Therefore, posterior is defined in the same way as normal [Gaussian unknown mean](./gaussian_unknown_mean.ipynb) distribution.

Assuming that the prior and likelihood are $p(x) = \mathcal{N}(\mu_0, \sigma_0)$ and $p(y|x) = \mathcal{N}(x, \sigma)$ respectively and, $y_1, y_2, \ldots y_n$ are the observed values, the posterior would be $p(x|y) = \mathcal{N}(\mu_p, \sigma_p)$ where,
$$
\begin{align}
\sigma_{p}^{2} & = \frac{1}{\frac{n}{\sigma^2} + \frac{1}{\sigma_{0}^{2}}} \\
\mu_p & = \sigma_{p}^{2} \left( \frac{\mu_0}{\sigma_{0}^{2}} + \frac{n\overline{y}}{\sigma^2} \right)
\end{align}
$$
The following class implements computing this posterior distribution. We also implement some helper functions and variables for plotting the correct posterior and prior.

```python
def plot_function(min_val, max_val, func, *args, **kwargs):
        x = np.linspace(min_val,max_val,int((max_val-min_val)*50))
        plt.plot(x, np.vectorize(func)(x), *args, **kwargs)

def get_dist_pdf(dist):
    return lambda x: math.exp(dist.log_prob(x))

class CorrectDistributions:
    def __init__(self, model):
        self.prior_mean = model.prior_mean
        self.prior_std = model.prior_std
        self.likelihood_std = model.likelihood_std
        self.prior_dist = Normal(self.prior_mean, self.prior_std)

    @property
    def observed_list(self):
        return self.__observed_list

    @observed_list.setter
    def observed_list(self, new_observed_list):
        self.__observed_list = new_observed_list
        self.construct_correct_posterior()

    def construct_correct_posterior(self):
        n = len(self.observed_list)
        posterior_var = 1/(n/self.likelihood_std**2 + 1/self.prior_std**2)
        posterior_mu = posterior_var * (self.prior_mean/self.prior_std**2 + n*np.mean(self.observed_list)/self.likelihood_std**2)
        self.posterior_dist = Normal(posterior_mu, math.sqrt(posterior_var))

    def prior_pdf(self, model, x):
        p = Normal(model.prior_mean,model.prior_stdd)
        return math.exp(p.log_prob(x))

    def plot_posterior(self, min_val, max_val):
        if not hasattr(self, 'posterior_dist'):
            raise AttributeError('observed values are not set yet, and posterior is not defined.')
        plot_function(min_val, max_val, get_dist_pdf(self.posterior_dist), label='correct posterior', color='orange')

    def plot_prior(self, min_val, max_val):
        plot_function(min_val, max_val, get_dist_pdf(self.prior_dist), label='prior', color='green')

correct_dists = CorrectDistributions(model)
```

# Prior distribution
We inspect the prior distribution to see if it behaves in the way we intended. First we construct an `Empirical` distribution with forward samples from the model.

Note: Extra arguments passed to `prior_distribution` will be forwarded to model's `forward` function.

```python
prior = model.prior_distribution(num_traces=1000)
```

    Time spent  | Time remain.| Progress             | Trace     | Traces/sec
    0d:00:00:00 | 0d:00:00:00 | #################### | 1000/1000 | 1,515.82

We can plot a historgram of these samples that are held by the `Empirical` distribution.

```python
prior.plot_histogram(show=False, alpha=0.75, label='emprical prior')
correct_dists.plot_prior(min(prior.values_numpy()),max(prior.values_numpy()))
plt.legend();
```

# Posterior inference with importance sampling
For a given set of observations, we can get samples from the posterior distribution.

```python
correct_dists.observed_list = [8, 9] # Observations
# sample from posterior (5000 samples)
posterior = model.posterior_distribution(
                                         num_traces=5000, # the number of samples estimating the posterior
                                         inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING, # specify which inference engine to use
                                         observe={'obs0': correct_dists.observed_list[0],
                                                  'obs1': correct_dists.observed_list[1]} # assign values to the observed values
                                         )
```

    Time spent  | Time remain.| Progress             | Trace     | Traces/sec
    0d:00:00:02 | 0d:00:00:00 | #################### | 5000/5000 | 1,783.46

Regular importance sampling uses proposals from the prior distribution. We can see this by plotting the histogram of the posterior distribution without using the importance weights. As expected, this is the same with the prior distribution.

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, alpha=0.75, label='empirical proposal')
correct_dists.plot_prior(min(posterior_unweighted.values_numpy()),
                         max(posterior_unweighted.values_numpy()))
correct_dists.plot_posterior(min(posterior_unweighted.values_numpy()),
                             max(posterior_unweighted.values_numpy()))
plt.legend();
```

When we do use the weights, we end up with the correct posterior distribution. The following shows the sampled posterior with the correct posterior (orange curve).

```python
posterior.plot_histogram(show=False, alpha=0.75, bins=50, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior_unweighted.values_numpy()))
plt.legend();
```

In practice, it is advised to use methods of the `Empirical` posterior distribution instead of dealing with the weights directly, which ensures that the weights are used in the correct way.

For instance, we can get samples from the posterior, compute its mean and standard deviation, and evaluate expectations of a function under the distribution:

```python
print(posterior.sample())
```

    tensor(6.3480)

```python
print(posterior.mean)
```

    tensor(7.3206)

```python
print(posterior.stddev)
```

    tensor(0.8764)

```python
print(posterior.expectation(lambda x: torch.sin(x)))
```

    tensor(0.5998)

# Inference compilation
Inference compilation is a technique where a deep neural network is used for parameterizing the proposal distribution in importance sampling (https://arxiv.org/abs/1610.09900). This neural network, which we call inference network, is automatically generated and trained with data sampled from the model.

We can learn an inference network for our model.

```python
model.learn_inference_network(num_traces=100000,
                              observe_embeddings={'obs0' : {'dim' : 32},
                                                  'obs1': {'dim' : 32}},
                              inference_network=pyprob.InferenceNetwork.LSTM)
```

    Creating new inference network...
    Observable obs0: observe embedding not specified, using the default FEEDFORWARD.
    Observable obs0: embedding depth not specified, using the default 2.
    Observable obs1: observe embedding not specified, using the default FEEDFORWARD.
    Observable obs1: embedding depth not specified, using the default 2.
    Observe embedding dimension: 64
    Train. time | Epoch| Trace     | Init. loss| Min. loss | Curr. loss| T.since min | Traces/sec
    New layers, address: 34__forward__marsaglia__x__Uniform__1, distribution: Uniform
    New layers, address: 44__forward__marsaglia__y__Uniform__1, distribution: Uniform
    New layers, address: 34__forward__marsaglia__x__Uniform__2, distribution: Uniform
    New layers, address: 44__forward__marsaglia__y__Uniform__2, distribution: Uniform
    New layers, address: 34__forward__marsaglia__x__Uniform__3, distribution: Uniform
    New layers, address: 44__forward__marsaglia__y__Uniform__3, distribution: Uniform
    New layers, address: 34__forward__marsaglia__x__Uniform__4, distribution: Uniform
    New layers, address: 44__forward__marsaglia__y__Uniform__4, distribution: Uniform
    New layers, address: 34__forward__marsaglia__x__Uniform__5, distribution: Uniform
    New layers, address: 44__forward__marsaglia__y__Uniform__5, distribution: Uniform
    Total addresses: 10, distribution types: 1, parameters: 2,968,878
    New layers, address: 34__forward__marsaglia__x__Uniform__6, distribution: Uniform00:00:04 | 270.0
    New layers, address: 44__forward__marsaglia__y__Uniform__6, distribution: Uniform
    Total addresses: 12, distribution types: 1, parameters: 3,263,388
    New layers, address: 34__forward__marsaglia__x__Uniform__7, distribution: Uniform.53e+00 | 0d:00:00:00 | 270.0
    New layers, address: 44__forward__marsaglia__y__Uniform__7, distribution: Uniform
    Total addresses: 14, distribution types: 1, parameters: 3,557,898
    New layers, address: 34__forward__marsaglia__x__Uniform__8, distribution: Uniform00:00:07 | 128.0
    New layers, address: 44__forward__marsaglia__y__Uniform__8, distribution: Uniform
    New layers, address: 34__forward__marsaglia__x__Uniform__9, distribution: Uniform
    New layers, address: 44__forward__marsaglia__y__Uniform__9, distribution: Uniform
    Total addresses: 18, distribution types: 1, parameters: 4,146,918
    0d:00:08:47 | 1    | 100,032   | +1.73e+00 | +8.64e-01 | +1.42e+00 | 0d:00:05:43 | 155.0

We now construct the posterior distribution using samples from inference compilation, using the trained inference network.

A much smaller number of samples are enough (500 vs. 5000) because the inference network provides good proposals based on the given observations. We can see that the proposal distribution given by the inference network is doing a job much better than the prior, by plotting the posterior samples without the importance weights, for a selection of observations.

```python
# sample from posterior (500 samples)
posterior = model.posterior_distribution(
                                         num_traces=500, # the number of samples estimating the posterior
                                         inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING_WITH_INFERENCE_NETWORK, # specify which inference engine to use
                                         observe={'obs0': correct_dists.observed_list[0],
                                                  'obs1': correct_dists.observed_list[1]} # assign values to the observed values
                                         )
```

    Time spent  | Time remain.| Progress             | Trace   | Traces/sec
    0d:00:00:08 | 0d:00:00:00 | #################### | 500/500 | 60.31

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, bins=50, alpha=0.75, label='empirical proposal')

correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior.values_numpy()))
plt.legend();
```

```python
posterior.plot_histogram(show=False, bins=50, alpha=0.75, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior.values_numpy()))
plt.legend();
```

Inference compilation performs amortized inferece which means, the same trained network provides proposal distributions for any observed values.

We can try performing inference using the same trained network with different observed values.

```python
correct_dists.observed_list = [12, 10] # New observations

posterior = model.posterior_distribution(
                                         num_traces=500,
                                         inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING_WITH_INFERENCE_NETWORK, # specify which inference engine to use
                                         observe={'obs0': correct_dists.observed_list[0],
                                                  'obs1': correct_dists.observed_list[1]}
                                         )
```

    Time spent  | Time remain.| Progress             | Trace   | Traces/sec
    0d:00:00:07 | 0d:00:00:00 | #################### | 500/500 | 62.97

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, bins=50, alpha=0.75, label='empirical proposal')

correct_dists.plot_posterior(min(posterior.values_numpy()),
                       max(posterior.values_numpy()))
plt.legend();
```

```python
posterior.plot_histogram(show=False, bins=50, alpha=0.75, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                       max(posterior.values_numpy()))
plt.legend();
```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, I. Hawke. Partly based on content by David Ketcheson, also under CC-BY.

# Full phugoid model

This is the third IPython Notebook of the series on the _phugoid model of glider flight_, our first learning module of the course ["Practical Numerical Methods with Python"](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about). In the first notebook, we described the physics of the trajectories known as phugoids obtained from an exchange of potential and kinetic energy in an idealized motion with no drag. We gave you a neat little code to play with and plot various phugoid curves.

In the second notebook, we looked at the equation representing small perturbations on the straight-line phugoid, resulting in simple harmonic motion. This is a second-order ordinary differential equation, and we solved it numerically using **Euler's method**: the simplest numerical method of all. We learned about convergence and calculated the error of the numerical solution, comparing with an analytical solution. That is a good foundation!

Now, let's go back to the dynamical model, and take away the idealization of no-drag. Let's remind ourselves of the forces affecting an aircraft, considering now that it may be accelerating, with an instantaneous upward trajectory. We use the designation $\theta$ for the angle, and consider it positive upwards.

####Figure 1. Forces with a positive trajectory angle.

In Figure 1, $L$ is the lift, $W$ is the weight, $D$ is the drag, and $\theta$ the positive angle of the trajectory, instantaneously.

In lesson 1, we wrote the force balance in the directions perpendicular and parallel to the trajectory for a glider in _equilibrium_. What if the forces are _not_ in balance? Well, there will be acceleration terms in the equations of motion, and we would have in that case:

\begin{align}
m \frac{dv}{dt} & = - W \sin\theta - D \\
m v \, \frac{d\theta}{dt} & = - W \cos\theta + L
\end{align}

We can use a few little tricks to make these equations more pleasing. First, use primes to denote the time derivatives and divide through by the weight:

\begin{align}
 \frac{v'}{g} & = - \sin\theta - D/W \\
\frac{v}{g} \, \theta' & = - \cos\theta + L/W
\end{align}

Recall, from our first lesson, that the ratio of lift to weight is known from the trim conditions—$L/W=v^2/v_t^2$— and also from the definitions of lift and drag,

$$\begin{eqnarray}
L &=& C_L S \times \frac{1}{2} \rho v^2 \\
D &=& C_D S \times \frac{1}{2} \rho v^2
\end{eqnarray}$$

we see that $L/D=C_L/C_D$. The system of equations can be re-written:

\begin{align}
 v' & = - g\, \sin\theta - \frac{C_D}{C_L} \frac{g}{v_t^2} v^2 \\
 \theta' & = - \frac{g}{v}\,\cos\theta + \frac{g}{v_t^2}\, v
\end{align}

It is very interesting that the first equation has the factor $C_D/C_L$, which is the inverse of a measure of the aerodynamic efficiency of the aircraft. It turns out, this is the term that contributes damping to the phugoid model: if drag is zero, there is no damping. Drag is never zero in real life, but as engineers design more aerodynam-ically efficient aircraft, they make the phugoid mode more weakly damped. At altitude, this is nothing but a slight bother, but vertical oscillations are unsafe during final approach to land, so this is something to watch out for!

## The initial value problem

If we want to visualize the flight trajectories predicted by this model, we are going to need to integrate the spatial coordinates, which depend on both the forward velocity and the trajectory angle. The position of the glider on a vertical plane will be designated by coordinates $(x, y)$ with respect to an inertial frame of reference, and are obtained from:

\begin{align}
x'(t) & = v \cos(\theta) \\
y'(t) & = v \sin(\theta).
\end{align}

Augmenting our original two differential equations by the two equations above, we have a system of four first-order differential equations to solve. We will use a time-stepping approach, like in the previous lesson. To do so, we do need *initial values* for every unknown:

$$
v(0) = v_0 \quad \text{and} \quad \theta(0) = \theta_0\\
x(0) = x_0 \quad \text{and} \quad y(0) = y_0
$$

## Solve with Euler's method

We know how to apply Euler's method from the previous lesson. We replace each of the time derivatives by an approximation of the form:

$$v'(t) \approx \frac{v^{n+1} - v^n}{\Delta t},$$

where we are now using a superscript $n$ to indicate the $n$-th value in the time iterations. The first differential equation, for example, gives:

$$\frac{v^{n+1} - v^n}{\Delta t} = - g\, \sin\theta^n - \frac{C_D}{C_L} \frac{g}{v_t^2} (v^n)^2$$

Alright, we know where this is going. At each time iteration $t^n$, we want to evaluate all the known data of our system to obtain the state at $t^{n+1}$—the next time step. We say that we are _stepping in time_ or _time marching_.

The full system of equations discretized with Euler's method is:

\begin{align}
v^{n+1} & = v^n + \Delta t \left(- g\, \sin\theta^n - \frac{C_D}{C_L} \frac{g}{v_t^2} (v^n)^2 \right) \\
\theta^{n+1} & = \theta^n + \Delta t \left(- \frac{g}{v^n}\,\cos\theta^n + \frac{g}{v_t^2}\, v^n \right) \\
x^{n+1} & = x^n + \Delta t \, v^n \cos\theta^n \\
y^{n+1} & = y^n + \Delta t \, v^n \sin\theta^n.
\end{align}

As we've learned before, the system of differential equations can also be written as a vector equation:

$$u'(t) = f(u)$$

where

\begin{align}
u & = \begin{pmatrix} v \\ \theta \\ x \\ y \end{pmatrix} & f(u) & = \begin{pmatrix} - g\, \sin\theta - \frac{C_D}{C_L} \frac{g}{v_t^2} v^2 \\ - \frac{g}{v}\,\cos\theta + \frac{g}{v_t^2}\, v \\ v\cos\theta \\ v\sin\theta \end{pmatrix}.
\end{align}

It's a bit tricky to code the solution using a NumPy array holding all your independent variables. But if you do, a function for the Euler step can be written that takes any number of simultaneous equations. It simply steps in time using the same line of code:

```Python
def euler_step(u, f, dt):
    return u + dt * f(u)
```

This function can take a NumPy array `u` with any number of components. All we need to do is create an appropriate function `f(u)` describing our system of differential equations. Notice how we are passing a _function_ as part of the arguments list to `euler_step()`. Neat!

### And solve!

As always, we start by loading the modules and libraries that we need for this problem. We'll need a few transcendental functions, including the $\log$ for a convergence study later on. And remember: the line `%matplotlib inline` is a magic function that tells Matplotlib to give us the plots in the notebook (the default behavior of Matplotlib is to open a pop-up window). In addition, we are importing the module `rcParams` to define notebook-wide plotting parameters: font family and size. Here we go!

```
from math import sin, cos, log, ceil
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

Next, we need to set things up to start our numerical solution: the parameter
values and the _initial values_. You know what the acceleration of gravity is: 9.8 m/s$^2$, but what are good values for $C_D/C_L$, the inverse of the aerodynamic efficiency? Some possible values are given on a table in the Wikipedia entry for [lift-to-drag ratio](http://en.wikipedia.org/wiki/Lift-to-drag_ratio): a modern sailplane can have $L/D$ of 40 to 60, depending on span (and, in case you're interested, a flying squirrel has $L/D$ close to 2).

For the _trim velocity_, the speed range for typical sailplanes is between 65 and 280 km/hr, according to Wikipedia (it must be right!). Let's convert that to meters per second: 18 to 78 m/s. We'll pick a value somewhere in the middle of this range.

Here's a possible set of parameters for the simulation, but be sure to come back and change some of these, and see what happens!

```
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 30.0   # trim velocity in m s^{-1}
C_D = 1/40.  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = v_t     # start at the trim velocity (or add a delta)
theta0 = 0.0 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 1000.0  # initial altitude
```

We'll define a function `f()` to match the right-hand side of Equation (15), the full differential system in vector form. This function assumes that we have available the parameters defined above. If you re-execute the cell above with different parameter values, you can just run the solution without re-executing the function definition.

```
def f(u):
    """Returns the right-hand side of the phugoid system of equations.

    Parameters
    ----------
    u : array of float
        array containing the solution at time n.

    Returns
    -------
    dudt : array of float
        array containing the RHS given u.
    """

    v = u[0]
    theta = u[1]
    x = u[2]
    y = u[3]
    return numpy.array([-g*sin(theta) - C_D/C_L*g/v_t**2*v**2,
                      -g*cos(theta)/v + g/v_t**2*v,
                      v*cos(theta),
                      v*sin(theta)])
```

Compare the code defining function `f(u)` with the differential equations, and convince yourself that it's right!

\begin{align}
u & = \begin{pmatrix} v \\ \theta \\ x \\ y \end{pmatrix} & f(u) & = \begin{pmatrix} - g\, \sin\theta - \frac{C_D}{C_L} \frac{g}{v_t^2} v^2 \\ - \frac{g}{v}\,\cos\theta + \frac{g}{v_t^2}\, v \\ v\cos\theta \\ v\sin\theta \end{pmatrix} \nonumber
\end{align}

Now, Euler's method is implemented in a simple function `euler_step()`:

```
def euler_step(u, f, dt):
    """Returns the solution at the next time-step using Euler's method.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        approximate solution at the next time step.
    """

    return u + dt * f(u)
```

After definiting a final time for the solution, and the time step $\Delta t$, we can construct the grid in time using the NumPy function [`linspace()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linspace.html). Make sure you study the decisions we made here to build the time grid: why do we add 1 to the definition of `N`, for example?

Look at the code below, and make sure you understand the following aspects of it.

* The NumPy array `u` contains the solution at every time-step,  consisting of the velocity, angle and location of the glider.
* The first element of the array `u` is set to contain the initial conditions.
* In the `for`-loop, the function `euler_step()` is called to get the solution at time-step $n+1$.

```
T = 100.0                          # final time
dt = 0.1                           # time increment
N = int(T/dt) + 1                  # number of time-steps
t = numpy.linspace(0.0, T, N)      # time discretization

# initialize the array containing the solution for each time-step
u = numpy.empty((N, 4))
u[0] = numpy.array([v0, theta0, x0, y0])# fill 1st element with initial values

# time loop - Euler method
for n in range(N-1):

    u[n+1] = euler_step(u[n], f, dt)
```

## Plot the trajectory

In order to plot the path of the glider, we need the location (`x`, `y`) with respect to time. That information is already contained in our NumPy array containing the solution; we just need to pluck it out.

Make sure you understand the indices to `u`, below, and the use of the colon notation. If any of it is confusing, read the Python documentation on [Indexing](http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html).

```
# get the glider's position with respect to the time
x = u[:,2]
y = u[:,3]
```

Time to plot the path of the glider and get the distance travelled!

```
# visualization of the path
pyplot.figure(figsize=(8,6))
pyplot.grid(True)
pyplot.xlabel(r'x', fontsize=18)
pyplot.ylabel(r'y', fontsize=18)
pyplot.title('Glider trajectory, flight time = %.2f' % T, fontsize=18)
pyplot.plot(x,y, 'k-', lw=2);
```

## Grid convergence

Let's study the convergence of Euler's method for the phugoid model. In the previous lesson, when we studied the straight-line phugoid under a small perturbation, we looked at convergence by comparing the numerical solution with the exact solution. Unfortunately, most problems don't have an exact solution (that's why we compute in the first place!). But here's a neat thing: we can use numerical solutions computed on different grids to study the convergence of the method, even without an analytical solution.

We need to be careful, though, and make sure that the fine-grid solution is resolving all of the features in the mathematical model. How can we know this? We'll have a look at that in a bit. Let's see how this works first.

You need a sequence of numerical solutions of the same problem, each with a different number of time grid points.

Let's create a NumPy array called `dt_values` that contains the  time-increment of each grid to be solved on. For each element `dt_values[i]`, we will compute the solution `u_values[i]` of the glider model using Euler's method. If we want to use five different values of $\Delta t$, we'll have five solutions: we put them in an array ... but each one is also an array! We'll have an array of arrays. How meta is that?

We have one more trick up our sleeve: `enumerate()`. To get all the numerical solutions—each with its value of $\Delta t$—done in one fell swoop, we will loop over the elements of the array `dt_values`. Within the loop, we need to access both `dt_values[i]` and the index `i`. It turns out, `enumerate()` is a built-in Python function that will give us consecutive `index, value` pairs just like we need.

Read the code below carefully, and remember: you can get a help pane on any function by entering a question mark followed by the function name. For example, add a new code cell below and type:  `?numpy.empty_like`.

```
dt_values = numpy.array([0.1, 0.05, 0.01, 0.005, 0.001])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt) + 1    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):

        u[n+1] = euler_step(u[n], f, dt)   ### call euler_step() ###

    # store the value of u related to one grid
    u_values[i] = u
```

In [Lesson 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_02_Phugoid_Oscillation.ipynb), we compared our numerical result to an analytical solution, but now we will instead compare numerical results from different grids.

For each solution, we'll compute the difference relative to the finest grid. You will be tempted to call this an _"error"_, but be careful: the solution at the finest grid is _not the exact_ solution, it is just a reference value that we can use to estimate grid convergence.

To calculate the difference between one solution `u_current` and the solution at the finest grid, `u_finest`, we'll use the $L_1$-norm, but any norm will do.

There is a small problem with this, though.  The coarsest grid, where $\Delta t = 0.1$, has 1001 grid points, while the finest grid, with $\Delta t = 0.001$ has 100001 grid points.  How do we know which grid points correspond to the same location in two numerical solutions, in order to compare them?

If we had time grids of 10 and 100 steps, respectively, this would be relatively simple to calculate.  Each element in our 10-step grid would span ten elements in our 100-step grid.

Calculating the _ratio_ of the two grid sizes will tell us how many elements in our fine-grid will span over one element in our coarser grid.

Recall that we can _slice_ a NumPy array and grab a subset of values from it.  The syntax for that is

```Python
my_array[3:8]
```

An additional slicing trick that we can take advantage of is the "slice step size."  We add an additional `:` to the slice range and then specify how many steps to take between elements. For example, this code

```Python
my_array[3:8:2]
```

will return the values of `my_array[3]`, `my_array[5]` and `my_array[7]`

With that, we can write a function to obtain the differences between coarser and finest grids. Here we go ...

```
def get_diffgrid(u_current, u_fine, dt):
    """Returns the difference between one grid and the fine one using L-1 norm.

    Parameters
    ----------
    u_current : array of float
        solution on the current grid.
    u_finest : array of float
        solution on the fine grid.
    dt : float
        time-increment on the current grid.

    Returns
    -------
    diffgrid : float
        difference computed in the L-1 norm.
    """

    N_current = len(u_current[:,0])
    N_fine = len(u_fine[:,0])

    grid_size_ratio = ceil(N_fine/float(N_current))

    diffgrid = dt * numpy.sum( numpy.abs(\
            u_current[:,2]- u_fine[::grid_size_ratio,2]))

    return diffgrid
```

Now that the function has been defined, let's compute the grid differences for each solution, relative to the fine-grid solution. Call the function `get_diffgrid()` with two solutions, one of which is always the one at the finest grid. Here's a neat Python trick: you can use negative indexing in Python!  If you have an array called `my_array` you access the _first_ element with

`my_array[0]`

But you can also access the _last_ element with

`my_array[-1]`

and the next to last element with

`my_array[-2]`

and so on.

```
# compute difference between one grid solution and the finest one
diffgrid = numpy.empty_like(dt_values)

for i, dt in enumerate(dt_values):
    print('dt = {}'.format(dt))

    ### call the function get_diffgrid() ###
    diffgrid[i] = get_diffgrid(u_values[i], u_values[-1], dt)
```

    dt = 0.1
    dt = 0.05
    dt = 0.01
    dt = 0.005
    dt = 0.001

Time to create a plot of the results! We'll create a *log-log* plot with the Matplotlib function [`loglog()`](http://matplotlib.org/api/pyplot_api.html?highlight=loglog#matplotlib.pyplot.loglog). Remember to skip the difference of the finest-grid solution with itself, which is zero.

```
# log-log plot of the grid differences
pyplot.figure(figsize=(6,6))
pyplot.grid(True)
pyplot.xlabel('$\Delta t$', fontsize=18)
pyplot.ylabel('$L_1$-norm of the grid differences', fontsize=18)
pyplot.axis('equal')
pyplot.loglog(dt_values[:-1], diffgrid[:-1], color='k', ls='-', lw=2, marker='o');
```

### Order of convergence

The order of convergence is the rate at which the numerical solution approaches the exact one as the mesh is refined. Considering that we're not comparing with an exact solution, we use 3 grid resolutions that are refined at a constant ratio $r$ to find the *observed order of convergence* ($p$), which is given by:

$$
\begin{equation}
p = \frac{\log \left(\frac{f_3-f_2}{f_2-f_1} \right) }{\log(r)}
\end{equation}
$$

where $f_1$ is the finest mesh solution, and $f_3$ the coarsest.

```
r = 2
h = 0.001

dt_values2 = numpy.array([h, r*h, r**2*h])

u_values2 = numpy.empty_like(dt_values2, dtype=numpy.ndarray)

diffgrid2 = numpy.empty(2)

for i, dt in enumerate(dt_values2):

    N = int(T/dt)    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):

        u[n+1] = euler_step(u[n], f, dt)         ### call euler_step() ###

    # store the value of u related to one grid
    u_values2[i] = u

#calculate f2 - f1
diffgrid2[0] = get_diffgrid(u_values2[1], u_values2[0], dt_values2[1])

#calculate f3 - f2
diffgrid2[1] = get_diffgrid(u_values2[2], u_values2[1], dt_values2[2])

# calculate the order of convergence
p = (log(diffgrid2[1]) - log(diffgrid2[0])) / log(r)

print('The order of convergence is p = {:.3f}'.format(p));
```

    The order of convergence is p = 1.014

See how the observed order of convergence is close to 1? This means that the rate at which the grid differences decrease match the mesh-refinement ratio. We say that Euler's method is of *first order*, and this result is a consequence of that.

# Paper airplane challenge

Suppose you wanted to participate in a paper-airplane competition, and you want to use what you know about the phugoid model to improve your chances. For a given value of $L/D$ that you can obtain in your design, you want to know what is the best initial velocity and launch angle to fly the longest distance from a given height.

Using the phugoid model, write a new code to analyze the flight of a paper airplane, with the following conditions:

*  Assume $L/D$ of 5.0 (a value close to measurements in Feng et al. 2009)
*  For the trim velocity, let's take an average value of 4.9 m/s.
*  Find a combination of launch angle and velocity that gives the best distance.
*  Think about how you will know when the flight needs to stop ... this will influence how you organize the code.
*  How can you check if your answer is realistic?

## References

* Feng, N. B. et al. _"On the aerodynamics of paper airplanes"_, AIAA paper 2009-3958, 27th AIAA Applied Aerodynamics Conference, San Antonio, TX. [PDF](http://www.ntu.edu.sg/home/schluter/pdf/aiaa-2009-3958.pdf)

* Simanca, S. R. and Sutherland, S. _"Mathematical problem-solving with computers,"_ 2002 course notes, Stony Brook University, chapter 3: [The Art of Phugoid](https://www.math.sunysb.edu/~scott/Book331/Art_Phugoid.html). (Note that there is an error in the figure: sine and cosine are switched.)

---

######The cell below loads the style of the notebook.

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>