```python
from resources.workspace import *
from IPython.display import display
from scipy.integrate import odeint
import copy

%matplotlib inline
```

# Instability, stability and chaos in dynamical systems

Chaos is commonly understood by the butterfly effect; "A buttefly that flaps its wings in Brazil can "cause" a hurricane in Texas". As opposed to the opinions of Descartes/Newton/Laplace, chaos effectively means that even in a deterministic (non-stochastic) universe, we can only predict "so far" into the future.  We will introduce two very typical "toy" models that exhibit these features.

## Examples of models chaotic systems

### The "Lorenz-95" model

The [Lorenz 95/ 96 system](http://eaps4.mit.edu/research/Lorenz/Predicability_a_Problem_2006.pdf) is a one dimensional model, designed to simulate atmospheric convection.  Each variable $x^j$  can be considered some atmospheric quantity in one of $m$ sectors along a single lattitude.  The differential equation for $x^j$  reads,

$$
\frac{{\rm d} x^j}{{\rm d} t} \triangleq -x^{j-1} x^{j-2} + x^{j-1}x^{j+1} - x^j + F,
$$

where all indices $j$ are taken modulo $m$.

There are **no accurate physics** represented in this model.  Rather, the model only seeks to capture qualitative features of the atmosphere, in that:

 there is external forcing, determined by a parameter $F$;

 there is internal dissipation, simulated by linear terms;

 there is advection, simulated by quadratic terms.

The number of sectors $m$ is assumed to be at least $m=4$ but typically it is taken that $m=40$. See the following link for
[further description](resources/DA_intro.pdf#page=23).

**Exc 4.2**: The system above has an easy to find fixed point, i.e., a point $\mathbf{x}_0$ such that

$$ \frac{{\rm d}}{{\rm d} t} \mathbf{x}_0 \equiv 0$$

Can you identify one?

```python
# Example solution

# show_answer('fixed_point')
```

A fixed point is **stable** if for any perturbations sufficiently small, a trajectory evolved from this perturbation must be attracted to the fixed point.  That means, all nearby solutions will settle to a solution that doesn't change in time.  A classification of various fixed point dynamics is illustrated in the figure below.

**By Freesodas (Gimp) [GFDL or CC BY-SA 4.0], via Wikimedia Commons**

In the Lorenz-95 model, different values for $F$ will produce different behaviors in the model.  For some values of $F$, the fixed point from **Exc 4.2** is stable.  For some values of $F$, perturbations won't be drawn to the fixed point, but will settle to another kind of **steady behavior**.  For some values of $F$, small perturbations will behave wildy, with growth and decay that is difficult to predict.

**Exc 4.4**: Run the code below to interactively plot the behavior of the Lorenz-95 sytem.  The figure on the left hand side below plots a time lapse of the values of $x^j$ on the $y$-axis, while the $x$-axis varies each sector $j$, modulo $m=10$.  The time variable is given by "T" below.

The figure on the right hand side below plots the time series of the total engergy of the system, defined by

$$\begin{align}
\mathbf{E}(\mathbf{x}) &= \frac{1}{2} \sum_{j=1}^m \left(\mathbf{x}^j\right)^2.
\end{align}$$

Note that for $T>30$, we only plot the time series in the interval $[T-30, T]$.

For each value of $F$, we initialize the model with a small perturbation of size "eps=0.5" to the fixed point found in **Exc 4.2**.  Answer the following questions:

 For what values of $F$ does it look like the fixed point is stable?  How is this reflected in the energy in the system?

 For what values of $F$ does it look like the system settles to periodic motion? How is this reflected in the energy in the system?

 For what values of $F$ does the evolution become "chaotic"? How is this reflected in the energy in the system?

 The classical choice for $F$ is $F=8$.  What kind of behavior is exhibited?

```python
# For all i, any n: s(x,n) := x[i+n], circularly.
def s(x,n):
    return np.roll(x,-n)

def animate_lorenz_95(F=0.8,T=0):
    # Initial conditions: perturbations
    eps=.5
    m=10
    x0 = ones(m)
    x0 = x0 * F
    x0[0] += eps

    def dxdt(x,t):
        return (s(x,1)-s(x,-2))*s(x,-1) - x + F

    tt = linspace(0, T, int(T/.1) + 1)
    xx = odeint(lambda x,t: dxdt(x,t), x0, tt)
    energy =  .5 * np.sum(xx*2, axis=1)
    xx = np.concatenate([xx,
                         np.reshape(xx[:,0], [len(xx[:,0]), 1])],axis=1)

    # Plot multiple
    fig = plt.figure(figsize=(16,6))
    ax1 = fig.add_axes([.08, .095,  .4, .89])
    ax2 = fig.add_axes([.525, .095, .4, .89])

    Lag = 4
    colors = plt.cm.cubehelix(0.1+0.6*linspace(0,1,Lag))
    for k in range(Lag,0,-1):
        ax1.plot(xx[max(0,len(xx)-k)],c=colors[Lag-k])

    ax1.set_ylim(-10,20)
    ax1.set_xlabel(r'Sector $j$', size=30)
    ax1.set_xticks(range(0,12,2))
    ax1.set_ylabel(r'$x^j$', size=30)
    ax1.tick_params(
        labelsize=20)
    ax2.plot(tt[-300:], energy[-300:])
    ax2.set_xlabel('Time T',size=30)
    ax2.yaxis.set_label_position("right")
    ax2.set_ylabel('Total Energy',size=30, rotation=270)
    ax2.yaxis.set_label_coords(1.175,.5)
    ax2.tick_params(
        axis='x',
        labelsize=20)
    ax2.tick_params(
        axis='y',
        labelsize=20,
        right=True,
        labelright=True,
        left=False,
        labelleft=False
    )
    tics = [np.round(i,decimals=1) for i in linspace(np.min(energy[-300:]) - 1, np.max(energy[-300:]) + 1, 5)]
    ax2.set_yticks(tics)
    ax2.set_yticklabels([str(i).zfill(2) for i in tics])

    plt.show()

interact(animate_lorenz_95,T=(0.2,60.2,0.1),F=(0,12,.2));
```

### The Lorenz (1963) system

The [Lorenz-63 system](https://journals.ametsoc.org/doi/abs/10.1175/1520-0469%281963%29020%3C0130%3ADNF%3E2.0.CO%3B2), commonly known as the "butterfly attractor", is a simplified mathematical model for atmospheric convection respresenting real physics.  The Lorenz equations are derived from the Oberbeck-Boussinesq approximation to the equations describing fluid circulation in a shallow layer of fluid, heated uniformly from below and cooled uniformly from above - this describes Rayleigh-Bénard convection.

The Lorenz-63 system is given by the 3 coupled ordinary differential equations (ODE):

$$\begin{aligned}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{aligned}$$

where

$$\dot{\ast} \triangleq \frac{{\rm d} }{{\rm d} t}$$

As a test case for DA, the state vector is $\mathbf{x} = (x,y,z)$, and the parameters are typically set to

```python
SIGMA = 10.0
BETA  = 8/3
RHO   = 28.0
```

The equations relate the properties of a two-dimensional fluid layer uniformly warmed from below and cooled from above. In particular, the equations describe the rate of change of three quantities with respect to time: x is proportional to the rate of convection, y to the horizontal temperature variation, and z to the vertical temperature variation.

The dynamics can be written as follows

```python
def dxdt(xyz, t0, sigma=SIGMA, beta=BETA, rho=RHO):
    """Compute the time-derivative of the Lorenz-63 system."""
    x, y, z = xyz
    return array([
        sigma * (y - x),
        x * (rho - z) - y,
        x * y - beta * z
    ])
```

#### Numerical computation of trajectories

Below is code to numerically integrate the differential equations and plot the solutions. This function has arguments that control the parameters of the differential equation $(\sigma,\beta,\rho)$.  In the following we will study how small perturbations of a "control" trajectory change over time.

Additional parameters in the code inlcude:

 "N", defining the number of perturbations;

 "eps", defining the size of perturbations;

 "T", defining the length of the forward evolution.  **Note**: we will only plot the trajectory along times $[T-10, T]$ for $T>10$

**Exc 4.6**: Use the code below to investigate sensititivy to initial conditions.  Answer the following questions:

 For small pertubations, eps=0.01, how long does it take to see the nearby initial conditions lose track of the control trajectory?

 Does it appear that there are parameter configurations where there are **stable** fixed points?

 Does it appear that there are parameter configurations where there are **stable** periodic behaviors?  **Hint**: what pattern emerges with $\rho = 350$?  What about $\rho=100.5$?

When all trajectories are drawn to a limiting behavior, we call this behavior **globally stable**.  Do the periodic solutions appear to be globally or locally stable?  Try multiple values of epsilon, and consider what happens when there is more than one periodic behavior.

```python
def animate_lorenz(sigma=SIGMA, beta=BETA, rho=RHO, N=2, eps=0.01, T=0.1):

    # Initial conditions: perturbations around some "proto" state
    seed(1)
    x0_proto = array([-6.1, 1.2, 32.5])
    x0 = x0_proto + eps*randn((N, 3))

    # Compute trajectories
    tt = linspace(0, T, int(T/.01)+1)               # Time instances for trajectory
    d2 = lambda x,t: dxdt(x,t, sigma,beta,rho)      # Define dxdt(x,t) with fixed params.
    xx = array([odeint(d2, x0i, tt) for x0i in x0]) # Integrate

    # PLOTTING
    ax = plt.figure(figsize=(16,8)).add_subplot(111, projection='3d')

    colors = plt.cm.jet(linspace(0,1,N))
    for i in range(N):
        # plot each ensemble member, but only the last 1000 steps of the trajectory
        ax.plot(*(xx[i,-1000:,:].T),'-'  ,c=colors[i])
        ax.scatter3D(*xx[i,-1,:],s=40,c=colors[i])

    ax.axis('off')
    plt.show()

w = interactive(animate_lorenz, sigma=(0.,50), rho=(0.,350, .5), beta=(0.,5),
                N=(1,10), eps=(0.01,10.01, .1),T=(0.1,100))

w
```

In the standard configuration of the Lorenz-63 model we see that small perturbations quickly diverge from control trajectories.  But even with perfect knowledge of the state of the atmosphere, our numerical approximations of its evolution would quickly degrade the forecast skill to zero.  As a proof of concept, suppose the "true" atmosphere is equal to the Lorenz-63 system, evolved via the simple forward Euler method, with a time step of

```python
h_true = 0.0001
```

Suppose we are given the exact state of the atmosphere, defined as

```python
xyz_exact = array([-6.1, 1.2, 32.5])
```

But we must use the Order 4.0 Runge-Kutta scheme,

```python
def l63_rk4_step(xyz, h):
    """ calculate the evolution of Lorenz-63 one step forward via RK-4"""

    k_xyz_1 = dxdt(xyz, h, sigma=SIGMA, beta=BETA, rho=RHO)
    k_xyz_2 = dxdt(xyz + k_xyz_1 * (h / 2.0), h, sigma=SIGMA, beta=BETA, rho=RHO)
    k_xyz_3 = dxdt(xyz + k_xyz_2 * (h / 2.0), h, sigma=SIGMA, beta=BETA, rho=RHO)
    k_xyz_4 = dxdt(xyz + k_xyz_3 * h, h, sigma=SIGMA, beta=BETA, rho=RHO)

    xyz_step = xyz + (h / 6.0) * (k_xyz_1 + 2 * k_xyz_2 + 2 * k_xyz_3 + k_xyz_4)

    return xyz_step
```

with a time step of

```python
h_approximate = 0.1
```

to derive an approximate forecast, due to computational constraints.  Note that the discretization error of both the forward Euler (with time step h_true=0.0001) and the Order 4.0 Runge-Kutta (with time step h_approximate=0.1) is on the order $\mathcal{O}\left(10^{-4}\right)$.

**Exc 4.8**: Use the function "dxdt" defined above to code the forward Euler method.  Fill in the missing line in the code below.

```python
def l63_forward_euler_step(xyz, h):
    """x_step is the one-step-forward state, derived from the initial condition xyz"""

    ### Fill in missing line here ###

    return xyz_step
```

```python
## Example solution

# show_answer('forward_euler')
```

**Exc 4.10**: Verify that your solution to Exc 4.6 works, using the GUI slider below.  The following code will generate the two paths from the **same** initial condition:

the "true" atmosphere, generated by the forward Euler scheme and time step of 0.0001;

the approximate atmospher, generated by the Runge-Kutta scheme with a time step of 0.1;

where the discretization error of each scheme is on the order of $\mathcal{O}\left(10^{-4}\right)$.

```python
def animate_approximation_divergence(T=0.1):

    ## Compute trajectories
    xyz_approx_step = copy.copy(xyz_exact)
    xyz_true_step = copy.copy(xyz_exact)

    # define the number of integration steps
    true_steps = int(T / h_true)
    approx_steps = int(T / h_approximate)

    # define storage for the approximate trajectory
    xyz_approx = zeros([approx_steps + 1, 3])

    # define storage for the true trajectory, but where we only store the same time steps as the approximate one
    xyz_true = zeros([approx_steps + 1, 3])

    for i in range(approx_steps + 1):
        # store the value for each the true and approximate trajectory
        xyz_true[i, :] = xyz_true_step
        xyz_approx[i, :] = xyz_approx_step

        # forward propagate the approximate trajectory only at increments of 0.1
        xyz_approx_step = l63_rk4_step(xyz_approx_step, h_approximate)

        for j in range(1000):
            # forward propagate the true trajectory at every step of 0.0001
            xyz_true_step = l63_forward_euler_step(xyz_true_step, h_true)

    # PLOTTING
    ax = plt.figure(figsize=(10,5)).add_subplot(111, projection='3d')
    xx = np.dstack([xyz_true, xyz_approx])

    colors = plt.cm.jet(linspace(0,1,2))
    for i in range(2):
        # plot each of the trajectories, integrated with separate rules
        ax.plot(*(xx[-4:,:,i].T),'-'  ,c=colors[i])
        ax.scatter3D(*xx[-1,:,i], s=40, c=colors[i])

    ax.set_xlim((-15, 15))
    ax.set_ylim((-25, 25))
    ax.set_zlim((15, 40))
    ax.view_init(30, 120)
    ax.axis('off')
    plt.show()

w = interactive(animate_approximation_divergence, T=(0.4,4))
w
```

**Exc 4.12**: What do you notice about this plot?  What does this say about small simulation errors in chaotic systems?

### Next: [Dynamics of ensembles and perturbations](T2 - Dynamics of ensembles and perturbations.ipynb)

tutorials/DA and the Dynamics of Ensemble Based Forecasting/T1 - Instability, stability and chaos in dynamical systems.ipynb


# 17 PDEs: Waves

(See *Computational Physics* Ch 21 and *Computational Modeling* Ch 6.5.)

## Background: waves on a string

Assume a 1D string of length $L$ with mass density per unit length $\rho$ along the $x$ direction. It is held under constant tension $T$ (force per unit length). Ignore frictional forces and the tension is so high that we can ignore sagging due to gravity.

### 1D wave equation
The string is displaced in the $y$ direction from its rest position, i.e., the displacement $y(x, t)$ is a function of space $x$ and time $t$.

For small relative displacements $y(x, t)/L \ll 1$ and therefore small slopes $\partial y/\partial x$ we can describe $y(x, t)$ with a *linear* equation of motion:

Newton's second law applied to short elements of the string with length $\Delta x$ and mass $\Delta m = \rho \Delta x$: the left hand side contains the *restoring force* that opposes the displacement, the right hand side is the acceleration of the string element:

\begin{align}
\sum F_{y}(x) &= \Delta m\, a(x, t)\\
T \sin\theta(x+\Delta x) - T \sin\theta(x) &= \rho \Delta x \frac{\partial^2 y(x, t)}{\partial t^2}
\end{align}

The angle $\theta$ measures by how much the string is bent away from the resting configuration.

Because we assume small relative displacements, the angles are small ($\theta \ll 1$) and we can make the small angle approximation

$$
\sin\theta \approx \tan\theta = \frac{\partial y}{\partial x}
$$

and hence

\begin{align}
T \left.\frac{\partial y}{\partial x}\right|_{x+\Delta x} - T \left.\frac{\partial y}{\partial x}\right|_{x}  &= \rho \Delta x \frac{\partial^2 y(x, t)}{\partial t^2}\\
\frac{T \left.\frac{\partial y}{\partial x}\right|_{x+\Delta x} - T \left.\frac{\partial y}{\partial x}\right|_{x}}{\Delta x} &= \rho \frac{\partial^2 y}{\partial t^2}
\end{align}

or in the limit $\Delta x \rightarrow 0$ a linear hyperbolic PDE results:

\begin{gather}
\frac{\partial^2 y(x, t)}{\partial x^2} = \frac{1}{c^2} \frac{\partial^2 y(x, t)}{\partial t^2}, \quad c = \sqrt{\frac{T}{\rho}}
\end{gather}

where $c$ has the dimension of a velocity. This is the (linear) **wave equation**.

### General solution: waves

General solutions are propagating waves:

If $f(x)$ is a solution at $t=0$ then

$$
y_{\mp}(x, t) = f(x \mp ct)
$$

are also solutions at later $t > 0$.

Because of linearity, any linear combination is also a solution, so the most general solution contains both right and left propagating waves

$$
y(x, t) = A f(x - ct) + B g(x + ct)
$$

(If $f$ and/or $g$ are present depends on the initial conditions.)

In three dimensions the wave equation is

$$
\boldsymbol{\nabla}^2 y(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2 y(\mathbf{x}, t)}{\partial t^2} = 0\
$$

### Boundary and initial conditions

* The boundary conditions could be that the ends are fixed

  $$y(0, t) = y(L, t) = 0$$

* The *initial condition* is a shape for the string, e.g., a Gaussian at the center

  $$
  y(x, t=0) = g(x) = y_0 \frac{1}{\sqrt{2\pi\sigma}} \exp\left[-\frac{(x - x_0)^2}{2\sigma^2}\right]
  $$

  at time 0.
* Because the wave equation is *second order in time* we need a second initial condition, for instance, the string is released from rest:

  $$
  \frac{\partial y(x, t=0)}{\partial t} = 0
  $$

  (The derivative, i.e., the initial displacement velocity is provided.)

### Analytical solution
Solve (as always) with *separation of variables*.

$$
y(x, t) = X(x) T(t)
$$

and this yields the general solution (with boundary conditions of fixed string ends and initial condition of zero velocity) as a superposition of normal modes

$$
y(x, t) = \sum_{n=0}^{+\infty} B_n \sin k_n x\, \cos\omega_n t,
\quad \omega_n = ck_n,\ k_n = n \frac{2\pi}{L} = n k_0.
$$

(The angular frequency $\omega$ and the wave vector $k$ are determined from the boundary conditions.)

The coefficients $B_n$ are obtained from the initial shape:

$$
y(x, t=0) = \sum_{n=0}^{+\infty} B_n \sin n k_0 x = g(x)
$$

In principle one can use the fact that $\int_0^L dx \sin m k_0 x \, \sin n k_0 x = \pi \delta_{mn}$ (orthogonality) to calculate the coefficients:

\begin{align}
\int_0^L dx \sin m k_0 x \sum_{n=0}^{+\infty} B_n \sin n k_0 x &= \int_0^L dx \sin(m k_0 x) \, g(x)\\
\pi \sum_{n=0}^{+\infty} B_n \delta_{mn} &= \dots \\
B_m &= \pi^{-1} \dots
\end{align}

(but the analytical solution is ugly and I cannot be bothered to put it down here.)

## Numerical solution

1. discretize wave equation
2. time stepping: leap frog algorithm (iterate)

Use the central difference approximation for the second order derivatives:

\begin{align}
\frac{\partial^2 y}{\partial t^2} &\approx \frac{y(x, t+\Delta t) + y(x, t-\Delta t) - 2y(x, t)}{\Delta t ^2} = \frac{y_{i, j+1} + y_{i, j-1} - 2y_{i,j}}{\Delta t^2}\\
\frac{\partial^2 y}{\partial x^2} &\approx \frac{y(x+\Delta x, t) + y(x-\Delta x, t) - 2y(x, t)}{\Delta x ^2} = \frac{y_{i+1, j} + y_{i-1, j} - 2y_{i,j}}{\Delta x^2}
\end{align}

and substitute into the wave equation to yield the *discretized* wave equation:

$$
\frac{y_{i+1, j} + y_{i-1, j} - 2y_{i,j}}{\Delta x^2} = \frac{1}{c^2} \frac{y_{i, j+1} + y_{i, j-1} - 2y_{i,j}}{\Delta t^2}
$$

Re-arrange so that the future terms $j+1$ can be calculated from the present $j$ and past $j-1$ terms:

$$
y_{i,j+1} = 2(1 - \beta^2)y_{i,j} - y_{i, j-1} + \beta^2 (y_{i+1,j} + y_{i-1,j}), \quad
\beta := \frac{c}{\Delta x/\Delta t}
$$

This is the time stepping algorithm for the wave equation.

## Numerical implementation

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

plt.style.use('ggplot')
```

### Student version

```python
L = 0.5  # m
Nx = 50
Nt = 100

Dx = L/Nx
Dt = 1e-4  # s

rho = 1.5e-2   # kg/m
tension = 150  # N

c = np.sqrt(tension/rho)

# TODO: calculate beta
beta = c*Dt/Dx
beta2 = beta**2

print("c = {0} m/s".format(c))
print("Dx = {0} m,  Dt = {1} s, Dx/Dt = {2} m/s".format(Dx, Dt, Dx/Dt))
print("beta = {}".format(beta))

X = np.linspace(0, L, Nx+1)  # need N+1!

def gaussian(x, y0=0.05, x0=L/2, sigma=0.1*L):
    return y0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2/(2*sigma**2))

# displacements at j-1, j, j+1
y0 = np.zeros_like(X)
y1 = np.zeros_like(y0)
y2 = np.zeros_like(y0)

# save array
y_t = np.zeros((Nt+1, Nx+1))

# boundary conditions
y0[0] = y0[-1] = y1[0] = y1[-1] = 0
y2[:] = y0

# initial conditions: velocity 0, i.e. no difference between y0 and y1
y0[1:-1] = y1[1:-1] = gaussian(X)[1:-1]

# save initial
t_index = 0
y_t[t_index, :] = y0
t_index += 1
y_t[t_index, :] = y1

for jt in range(2, Nt):
    y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
    y0[:], y1[:] = y1, y2

    t_index += 1
    y_t[t_index, :] = y2
    print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

### Fancy version
Package as a function and can use `step` to only save every `step` time steps.

```python
def wave(L=0.5, Nx=50, Dt=1e-4, Nt=100, step=1,
        rho=1.5e-2, tension=150.):

    Dx = L/Nx

    #rho = 1.5e-2   # kg/m
    #tension = 150  # N

    c = np.sqrt(tension/rho)

    beta = c*Dt/Dx
    beta2 = beta**2

    print("c = {0} m/s".format(c))
    print("Dx = {0} m,  Dt = {1} s, Dx/Dt = {2} m/s".format(Dx, Dt, Dx/Dt))
    print("beta = {}".format(beta))

    X = np.linspace(0, L, Nx+1)  # need N+1!

    def gaussian(x, y0=0.05, x0=L/2, sigma=0.1*L):
        return y0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2/(2*sigma**2))

    # displacements at j-1, j, j+1
    y0 = np.zeros_like(X)
    y1 = np.zeros_like(y0)
    y2 = np.zeros_like(y0)

    # save array
    y_t = np.zeros((int(np.ceil(Nt/step)) + 1, Nx+1))

    # boundary conditions
    y0[0] = y0[-1] = y1[0] = y1[-1] = 0
    y2[:] = y0

    # initial conditions: velocity 0, i.e. no difference between y0 and y1
    y0[1:-1] = y1[1:-1] = gaussian(X)[1:-1]

    # save initial
    t_index = 0
    y_t[t_index, :] = y0
    if step == 1:
        t_index += 1
        y_t[t_index, :] = y1

    for jt in range(2, Nt):
        y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
        y0[:], y1[:] = y1, y2

        if jt % step == 0 or jt == Nt-1:
            t_index += 1
            y_t[t_index, :] = y2
            print("Iteration {0:5d}".format(jt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

    return y_t, X, Dx, Dt, step

```

```python
y_t, X, Dx, Dt, step = wave()
```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

### 1D plot
Plot the output in the save array `y_t`. Vary the time steps that you look at with `y_t[start:end]`.

We indicate time by color changing.

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis(i) for i in np.linspace(0, 1, len(y_t))])
ax.plot(X, y_t[:40].T, alpha=0.5);
```

### 1D Animation
For 1D animation to work in a Jupyter notebook, use

```python
%matplotlib widget
```

If no animations are visible, restart kernel and execute the `%matplotlib notebook` cell as the very first one in the notebook.

We use `matplotlib.animation` to look at movies of our solution:

```python
import matplotlib.animation as animation
```

Generate one full period:

```python
y_t, X, Dx, Dt, step = wave(Nt=100)
```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

The `update_wave()` function simply re-draws our image for every `frame`.

```python
y_limits = 1.05*y_t.min(), 1.05*y_t.max()

fig1 = plt.figure(figsize=(5,5))
ax = fig1.add_subplot(111)
ax.set_aspect(1)

def update_wave(frame, data):
    global ax, Dt, y_limits
    ax.clear()
    ax.set_xlabel("x (m)")
    ax.set_ylabel("y (m)")
    ax.plot(X, data[frame])
    ax.set_ylim(y_limits)
    ax.text(0.1, 0.9, "t = {0:3.1f} ms".format(frame*Dt*1e3), transform=ax.transAxes)

wave_anim = animation.FuncAnimation(fig1, update_wave, frames=len(y_t), fargs=(y_t,),
                                    interval=30, blit=True, repeat_delay=100)

```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

If you have ffmpeg installed then you can export to a MP4 movie file:

```python
wave_anim.save("string.mp4", fps=30, dpi=300)
```

The whole video can be incoroporated into an html page (uses the HTML5 `<video>` tag).

```python
 with open("string.html", "w") as html:
    html.write(wave_anim.to_html5_video())
```

```python

```

### 3D plot
(Uses functions from previous lessons.)

```python
def plot_y(y_t, Dt, Dx, step=1):
    T, X = np.meshgrid(range(y_t.shape[0]), range(y_t.shape[1]))
    Y = y_t.T[X, T]    # intepret index 0 as "t" and index 1 as "x", but plot x along axis 1 and t along axis 2
    fig = plt.figure()
    ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X*Dx, T*Dt*step, Y)
    ax.set_ylabel(r"time $t$ (s)")
    ax.set_xlabel(r"position $x$ (m)")
    ax.set_zlabel(r"displacement $y$ (m)")
    fig.tight_layout()
    return ax

def plot_surf(y_t, Dt, Dx, step=1, filename=None, offset=-1, zlabel=r'displacement',
             elevation=40, azimuth=-20, cmap=plt.cm.coolwarm):
    """Plot y_t as a 3D plot with contour plot underneath.

    Arguments
    ---------
    y_t : 2D array
          displacement y(t, x)
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    """

    t = np.arange(y_t.shape[0], dtype=int)
    x = np.arange(y_t.shape[1], dtype=int)
    T, X = np.meshgrid(t, x)
    Y = y_t.T[X, T]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    surf = ax.plot_surface(X*Dx, T*Dt*step, Y, cmap=cmap, rstride=1, cstride=1, alpha=1)
    cset = ax.contourf(X*Dx, T*Dt*step, Y, 20, zdir='z', offset=offset+Y.min(), cmap=cmap)

    ax.set_xlabel('x')
    ax.set_ylabel('t')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Y.min(), Y.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

```python
plot_y(y_t, Dt, Dx)
```

```python
plot_surf(y_t, Dt, Dx, offset=-0.04, cmap=plt.cm.coolwarm)
```

## von Neumann stability analysis: Courant condition

Assume that the solutions of the discretized equation can be written as normal modes

$$
y_{m,j} = \xi(k)^j e^{ikm\Delta x}, \quad t=j\Delta t,\ x=m\Delta x
$$

The time stepping algorith is stable if

$$
|\xi(k)| < 1
$$

Insert normal modes into the discretized equation

$$
y_{i,j+1} = 2(1 - \beta^2)y_{i,j} - y_{i, j-1} + \beta^2 (y_{i+1,j} + y_{i-1,j}), \quad
\beta := \frac{c}{\Delta x/\Delta t}
$$

and simplify (use $1-\cos x = 2\sin^2\frac{x}{2}$):

$$
\xi^2 - 2(1-2\beta^2 s^2)\xi + 1 = 0, \quad s=\sin(k\Delta x/2)
$$

The characteristic equation has roots

$$
\xi_{\pm} = 1 - 2\beta^2 s^2 \pm \sqrt{(1-2\beta^2 s^2)^2 - 1}.
$$

It has one root for

$$
\left|1-2\beta^2 s^2\right| = 1,
$$

i.e., for

$$
\beta s = 1
$$

We have two real roots for

$$
\left|1-2\beta^2 s^2\right| < 1 \\
\beta s > 1
$$

but one of the roots is always $|\xi| > 1$ and hence these solutions will diverge and not be stable.

For

$$
\left|1-2\beta^2 s^2\right| ≥ 1 \\
\beta s ≤ 1
$$

the roots will be *complex conjugates of each other*

$$
\xi_\pm = 1 - 2\beta^2s^2 \pm i\sqrt{1-(1-2\beta^2s^2)^2}
$$

and the *magnitude*

$$
|\xi_{\pm}|^2 =  (1 - 2\beta^2s^2)^2 - (1-(1-2\beta^2s^2)^2) = 1
$$

is unity: Thus the solutions will not grow and will be *stable* for

$$
\beta s ≤ 1\\
\frac{c}{\frac{\Delta x}{\Delta t}} \sin\frac{k \Delta x}{2} ≤ 1
$$

Assuming the "worst case" for the $\sin$ factor (namely, 1), the **condition for stability** is

$$
c ≤ \frac{\Delta x}{\Delta t}
$$

or

$$
\beta ≤ 1.
$$

This is also known as the **Courant condition**. When written as

$$
\Delta t ≤ \frac{\Delta x}{c}
$$

it means that the time step $\Delta t$ (for a given $\Delta x$) must be *smaller than the time that the  wave takes to travel one grid step*.

## Simplified simulation
The above example used real numbers for a cello string. Below is bare-bones where we just set $\beta$.

```python
L = 10.0
Nx = 100
Nt = 200
step = 1

Dx = L/Nx

beta2 = 1.0

X = np.linspace(0, L, Nx+1)  # need N+1!

def gaussian(x):
    return np.exp(-(x-5)**2)

# displacements at j-1, j, j+1
y0 = np.zeros_like(X)
y1 = np.zeros_like(y0)
y2 = np.zeros_like(y0)

# save array
y_t = np.zeros((int(np.ceil(Nt/step)) + 1, Nx+1))

# boundary conditions
y0[0] = y0[-1] = y1[0] = y1[-1] = 0
y2[:] = y0

# initial conditions: velocity 0, i.e. no difference between y0 and y1
y0[:] = y1[:] = gaussian(X)

# save initial
t_index = 0
y_t[t_index, :] = y0
if step == 1:
    t_index += 1
    y_t[t_index, :] = y1

for jt in range(2, Nt):
    y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
    y0[:], y1[:] = y1, y2

    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        y_t[t_index, :] = y2
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

```

    Completed   199 iterations: t=0.0199 s

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis_r(i) for i in np.linspace(0, 1, len(y_t))])
ax.plot(X, y_t.T);
```

```python

```


# DYNAMICAL SYSTEMS

# Preliminaries

```python
try:
    import controlSBML as ctl
except:
    !pip -q install controlSBML
    import controlSBML as ctl
from controlSBML.util import makeSimulationTimes
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import sympy as sy
import tellurium as te
print("controlSBML version: " + ctl.__version__)
```

    controlSBML version: 0.2.4

# Sequential Pathways

```python
LINEAR_MDL= """
S1 -> S2; k1*S1
S2 -> S3; k2*S2

S1 = 10
S2 = 0
S3 = 0
k1 = 2
k2 = 1
"""
LINEAR_RR = te.loada(LINEAR_MDL)
LINEAR_RR.plot(LINEAR_RR.simulate())
```

How do we explain this behavior?

For the system $\dot{{\bf x}} (t) = {\bf A} {\bf x}(t)$, the solution is
${\bf x}(t) = \sum_{n=1}^N c_n {\bf u}_n e^{\lambda_n t}$, where the $c_n$ are computed from ${\bf x})0)$.

```python
# A is the Jacobian
A = LINEAR_RR.getFullJacobian()
ctl.mat2DF(A)
```

      S1
      S2
      S3

      S1
      -2.0
      0.0
      0.0

      S2
      2.0
      -1.0
      0.0

      S3
      0.0
      1.0
      0.0

```python
eigenvalues, eigenvectors = np.linalg.eig(A)
ctl.mat2DF(eigenvalues, row_names=A.rownames)
```

      0

      S1
      0.0

      S2
      -1.0

      S3
      -2.0

```python
ctl.mat2DF(eigenvectors, column_names=["0", "-1", "-2"],
           row_names=["S1", "S2", "S3"])
```

      0
      -1
      -2

      S1
      0.0
      0.000000
      0.408248

      S2
      0.0
      0.707107
      -0.816497

      S3
      1.0
      -0.707107
      0.408248

To find the $c_n$, we solve ${\bf U} {\bf c} = {\bf x}(0)$.
Or, ${\bf U}^{-1} {\bf x}(0) = {\bf c}$.

```python
# Note that the eigenvectors are linearly independent. So, the above inverse exists, and
# we get a unique solution for the linear system.
np.linalg.matrix_rank(eigenvectors)
```

    3

```python
np.matmul(np.linalg.inv(eigenvectors), np.array([10, 0, 0]))
```

    [10.         28.28427125 24.49489743]

```python
def calcX(A, time, x0):
    """
    Calculate the time solution of a linear system with initial values.

    Parameters
    ----------
    A: np.ndarray (N X N)
    time: float
    x0: np.ndarray (N X 1)

    Returns
    -------
    nd.array (N X 1)
    """
    eigenvalues, eigenvectors = np.linalg.eig(A)
    eigenvectors_inv = np.linalg.inv(eigenvectors)
    c_vec = np.matmul(eigenvectors_inv, x0)
    c_mat = np.diag(c_vec)
    result1 = np.matmul(eigenvectors, c_mat)
    exp_vec = np.array([np.e**(time*v) for  v in eigenvalues])
    result = np.matmul(result1, exp_vec)
    return result

# TESTS
x0 = np.array([10, 0, 0])
x = calcX(A, 0, x0)
assert(all([np.isclose(x[n], x0[n], 0) for n in range(len(x))]))
assert(x[0] == x0[0])
print("OK!")
```

    OK!

```python
def getXEigens(A, time, x0):
    """
    Calculates the contributions of the different eigenvectors.

    Parameters
    ----------
    A: np.ndarray (N X N)
    time: float
    x0: np.ndarray (N X 1)

    Returns
    -------
    pd.DataFrame
        Column: eigenvalue
        Row: species
    """
    N = np.shape(A)[0]
    eigenvalues, eigenvectors = np.linalg.eig(A)
    eigenvectors_inv = np.linalg.inv(eigenvectors)
    c_vec = np.matmul(eigenvectors_inv, x0)
    weighted_eigens = [c_vec[n]*eigenvectors[:, n] for n in range(N)]
    exp_vec =[np.e**(time*v) for  v in eigenvalues]
    weighted_eigens = [w*e for w, e in zip(weighted_eigens, exp_vec)]
    df = pd.DataFrame(weighted_eigens).transpose()
    if "rownames" in dir(A):
        if len(A.rownames) > 0:
            rownames = A.rownames
        else:
            rownames = None
    else:
        rownames = None
    df.index = rownames
    df.columns = [round(v, 4) for v in eigenvalues]
    df["total"] = df.sum(axis=1)
    return df

# TESTS
x0 = np.array([10, 0, 0])
df = getXEigens(A, 1, x0)
assert(isinstance(df, pd.DataFrame))
assert(len(df) > 0)
print("OK!")
```

    OK!

```python
dfss = [[] for n in range(4)]  # Container for the data frames
fuzz = 0.2
# Collect the weighted eigenvalues over time
x0 = np.array([10, 0, 0])
times = makeSimulationTimes(end_time=5)
for time in times:
    df = getXEigens(A, time, x0)
    columns = list(df.columns)
    for idx, col in enumerate(columns):
        col_df = pd.DataFrame(df[col]).transpose()
        dfss[idx].append(col_df)
df_dct = {}
for dfs in dfss:
    df = pd.concat(dfs, axis=0)
    df = df.applymap(lambda v: v + np.random.rand()*fuzz)
    lams = list(df.index)
    if isinstance(lams[0], str):
        lam = lams[0]
    else:
        lam = str(int(lams[0]))
    df.index = times
    df_dct[lam] = ctl.Timeseries(df)
```

```python
tss = list(df_dct.values())
ctl.plotManyTS(*tss, ncol=3, figsize=(15, 5), names=["l=0", "l=-1", "l=-2", "total"])
```

```python
# Create 4 Timeseries, one for each eigenvector and one for the total.
# Then create 3 Timeseries, one for each species with a column for each eigenvalue
# Plot over time.
for time in [0, 0.2, 0.5, 1, 2, 4]:
    ctl.ppMat(getXEigens(A, time, x0))
```

         0.0  -1.0  -2.0         total
    S1   0.0   0.0  10.0  1.000000e+01
    S2   0.0  20.0 -20.0  0.000000e+00
    S3  10.0 -20.0  10.0  1.776357e-15
         0.0       -1.0       -2.0     total
    S1   0.0   0.000000   6.703200  6.703200
    S2   0.0  16.374615 -13.406401  2.968214
    S3  10.0 -16.374615   6.703200  0.328585
         0.0       -1.0      -2.0     total
    S1   0.0   0.000000  3.678794  3.678794
    S2   0.0  12.130613 -7.357589  4.773024
    S3  10.0 -12.130613  3.678794  1.548181
         0.0      -1.0      -2.0     total
    S1   0.0  0.000000  1.353353  1.353353
    S2   0.0  7.357589 -2.706706  4.650883
    S3  10.0 -7.357589  1.353353  3.995764
         0.0      -1.0      -2.0     total
    S1   0.0  0.000000  0.183156  0.183156
    S2   0.0  2.706706 -0.366313  2.340393
    S3  10.0 -2.706706  0.183156  7.476451
         0.0      -1.0      -2.0     total
    S1   0.0  0.000000  0.003355  0.003355
    S2   0.0  0.366313 -0.006709  0.359604
    S3  10.0 -0.366313  0.003355  9.637042

```python
# Compare with Tellurium simulation
times = np.linspace(0, 5, 51)
times
result = np.array([calcX(A, t, x0) for t in times])
plt.plot(times, result)
plt.title("Calculated")
LINEAR_RR.reset()
LINEAR_RR.plot(LINEAR_RR.simulate(), title="Simulated")
```

# Feedback

```python
FEEDBACK_MDL= """
S1 -> S2; k1*S1
S2 -> S3; k2*S2
S3 -> S1; k3*S3

S1 = 10
S2 = 0
S3 = 0
k1 = 2
k2 = 1
"""
LINEAR_RR = te.loada(LINEAR_MDL)
LINEAR_RR.plot(LINEAR_RR.simulate())
```

# Accuracy of Linear Approximations


# Bayes' Theorem

## Motivating Examples - Limitations of Machine Learning and (frequentist) Statistics

(1) Consider a model for detection of fraud in financial transactions. We trained and validated such a model using (labelled) data from the past. The model is deployed in production and predicts a given transaction to be fraud. Knowing our model's True and False Positive Rates, and having an estimate of the frequency of fraud, what is the _probability_ that this transaction is actually fraudulent?

(2) Consider we observe a few thousand heart rate measurements from a person's last running activity. What is the _probability_ (interval) that the observed mean and variance in this sample represent this person's true distribution of exercise intensity? How does this inference change if we have prior knowledge (e.g. population data)?

ML and stats tools lack common ways to:

- Express uncertainty about inference of parameters,
- Express uncertainty about predictions,
- Use **and make explicit** (subjective) prior knowledge for inference

## The basics: Probability Mass/Density Functions

A **random variable** $X$ can be:

### Discrete:

$X \in \{Red, Black\}$, with **probability mass function (PMF)**

$$
p(x) =
\begin{cases}
0.8,&x = Red\\
0.2,&x = Black\\
0,&otherwise
\end{cases}
$$
</div

```python
def pmf(x):
    return {
        'Red': 0.8,
        'Black': 0.2
    }.get(x, 0.0)
```

### Continuous:

$X \in [30, 230]$, with probability **density** function **(PDF)**, assuming a normal distribution with:

* mean $\bbox[1pt,border:2px solid red]{\mu} = 120$,
* variance $\bbox[1pt,border:2px solid blue]{\sigma^2} = 400$

$$p(x) = \frac{1}{\sqrt{2\pi\bbox[1pt,border:2px solid blue]{400}}}e^{-\frac{(x - \bbox[1pt,border:2px solid red]{120})^2}{2*\bbox[1pt,border:2px solid blue]{400}}}$$

Be aware there are [many more](https://en.wikipedia.org/wiki/List_of_probability_distributions) types of probability distributions.

```python
# No need to implement these PDFs yourselves, see scipy.stats
import numpy as np
from scipy.stats import norm

heart_rate_mean = 120
heart_rate_std = 20

norm.pdf(130, loc=heart_rate_mean, scale=heart_rate_std)
```

```python
# For reuse of the same distribution params, a distribution can be _frozen_:
norm_hr = norm(loc=heart_rate_mean, scale=heart_rate_std)
norm_hr.pdf(130)
```

- What does this value mean?
- Is it a probability?

- What is the probability $Pr(X = 120)$, where $X$ is _exactly_ 120?

---

```python
import plotly.graph_objs as go

x=np.linspace(30, 230, 100)

go.FigureWidget(
    data=[
        go.Scatter(x=x, y=norm_hr.pdf(x), mode='lines', line={'shape': 'spline', 'width': 4}, showlegend=False),
        go.Scatter(x=[130, 155], y=norm_hr.pdf([130, 155]), mode='markers', marker={'size': 8}, showlegend=False)
    ],
    layout={
        'width': 800,
        'title': 'p(X), for μ=120, σ=20',
        'xaxis': {'title': 'X'},
        'yaxis': {'title': 'p(X)'}
    }
)
```

A PDF gives the **relative likelihood** of $X$ having a given value:

```python
norm_hr.pdf(130) / norm_hr.pdf(155)
```

meaning that with $\mu=120$ and $\sigma=20$, a heart rate of 130 is around 4 times more likely than a heart rate of 155.

### Probabilities from PDF's

You can derive probabilities from PDFs by integration:

$$Pr(100 \le X \le 120) = \int_{100}^{120}p(x)$$

And for a valid PDF:

$$ \int_{-\infty}^{\infty}p(x) = 1$$

The integral of a PDF is called a **Cumulative Distribution Function (CDF)**.

## Adding 1 Dimension: Joint and Conditional Probability

The joint probability density of 2 random variables $A$ and $B$ is given as

$$
p(A, B) = p(A|B)p(B)
$$

where $p(A|B)$ represents the density of $A$, **given** $B$, or **conditioned on** $B$.

How should $p(A, B)$, or $p(A|B)$ be interpreted?

Let's use maximum heart rates as an example. A person's maximum heart rate usually decreases with age. A commonly mentioned formula to estimate maximum heart rate is 220 - age, (see [wikipedia](https://en.wikipedia.org/wiki/Heart_rate#Haskell_&_Fox)).

```python
def to_max_heart_rate(age):
    return 220 - age
```

This method is a gross oversimplification and shouldn't be used in practice, but serves well for this example.

```python
ages = np.linspace(15, 85, 100)  # simulate 100 users of some fitness app, ages "uniformly" distributed between 15 and 85
max_heart_rate_means = to_max_heart_rate(ages)
```

```python
y = np.linspace(130, 250, 100)
max_heart_rate_densities = np.array([norm.pdf(y, loc=age_hr, scale=10) for age_hr in max_heart_rate_means])
```

```python
go.FigureWidget(
    data=[
        go.Surface(x=ages, y=y, z=max_heart_rate_densities)
    ],
    layout=go.Layout(
        width=600,
        height=600,
        xaxis=dict(title='age'),
        yaxis=dict(title='hr max')
    )
)
```

$p(H,A)$ represents the relative likelihood that age ($A$) and heart rate ($H$) have some values **simultaneously**. For discrete random variables, the joint PMF is a 2-d lookup table.

$p(H|A=a)$ represents the relative likelihood of a heart rate for a fixed age (imagine a slice cutting through the sphere above, going through x=a). This results in a single-variable PDF.

Similar to 1-d PDFs, probabilities can be obtained by (double) integration.

If $A$ and $B$ are **independent**, $p(A|B) = p(A)$. Most often, this is _not_ the case, so don't interpret $p(A,B)$ as being as simple as $p(A) \times p(B)$

## Since Variable Order Doesn't Matter...

$$
\begin{align}
p(A,B) &= p(B,A)\Leftrightarrow\\
p(A|B)p(B) &= p(B|A)p(A)\Leftrightarrow
\end{align}
$$

$$
p(A|B) = \frac{p(B|A)p(A)}{p(B)}
$$

a.k.a. **Bayes' Theorem**, or **Bayes' Rule**.

## That Fraud Detection Model

Let's try to plug our initial question about our fraud detection model into this formula. There are 2 (discrete) random variables involved:

- Transaction Fraud ($F \in \{Fraud, OK\}$)
- Model Alert ($A \in \{Alert, OK\}$)

Having trained and validated our model using historical data, we obtained a confusion matrix that looks as follows:

| predicted\true| fraud | ok     |
|---------------|-------|--------|
| predict_fraud | 0.95  | 0.0001 |
| predict_ok    | 0.05  | 0.9999 |

Further, we know from past experience, that roughly 1 in a million transactions are fraudulent, i.e. $p(F=Fraud) = 0.000001$

We are interested in the probability $p(F=Fraud|A=Alert)$

According to Bayes' Theorem, this is equal to:

$$
\frac{\bbox[1pt,border:2px solid red]{p(A=Alert|F=Fraud)}\times\bbox[1pt,border:2px solid yellow]{p(F=Fraud)}}{\bbox[1pt,border:2px solid blue]{p(A=Alert)}}
$$

with

$$
\begin{align}
p(A=Alert) & = \sum_{f \in F}p(A=Alert|f).p(f)\\
& = 0.95\times0.000001 + 0.0001\times0.999999\\
& \approx 0.0001 \\
\end{align}
$$

which leads to

$$
p(F=Fraud|A=Alert) = \frac{\bbox[1pt,border:2px solid red]{0.95}\times\bbox[1pt,border:2px solid yellow]{0.000001}}{\bbox[1pt,border:2px solid blue]{0.0001}} \approx 0.01
$$

```python
def p_fraud_given_alert(p_fraud, true_positive_rate, false_positive_rate):
    return true_positive_rate * p_fraud / (true_positive_rate * p_fraud + false_positive_rate * (1 - p_fraud))

p_fraud_given_alert(0.001, 0.95, 0.0001)
```

```python
p_fraud_given_alert(0.0001, 0.95, 0.0001)
```

```python
p_fraud_given_alert(0.00001, 0.95, 0.0001)
```

To conclude, the first application of Bayes' Rule is for cases where (for discrete events) we can directly measure some conditional probability $p(B|A)$, and prior probabilities $p(A)$ and $p(B)$, but our probability of interest $p(A|B)$ is less straightforward.

## (Social) Sience, Null Hypothesis Significance Tests, and that P Value

- What's the meaning of the P value in a significance test? (I dare you!)
- Given some P value, can we infer anything about the **probability** of the Null Hypothesis (or Alternative Hypothesis) being true?
- Can we express statements about P values in terms of $p(\text{Hypothesis}\mid\text{Data})$, or $p(\text{Data}\mid\text{Hypothesis})$


In this module, we prepare the time series model for the uncertain inflow energy $X_t$.

Whole model description:
https://www2.isye.gatech.edu/people/faculty/Alex_Shapiro/Rep-Dec2011.pdf

```python
import pandas
import numpy
# read historical data for the four reserviors
hist = [
    pandas.read_csv(
        "./data/hist_{}.csv".format(i),
        sep=";",
        index_col=0,
    )
    for i in range(4)
]
# concatenate the data to fill missing value yearly
hist_all = pandas.concat(hist, axis=1, ignore_index=True)
hist_all.fillna(hist_all.mean(), inplace=True)
hist_all_log = numpy.log(hist_all)
# split the data back to individual reserviors
hist = [hist_all.iloc[:,12*i:12*(i+1)] for i in range(4)]
```

Let $Y_t = log(X_t)$. We hope to fit into the following model:
\begin{equation}
    Y_t - \mu_t = \phi_t(Y_{t-1}-\mu_{t-1})+ \epsilon_t
\end{equation}
Where:
1. $\mu_t=\mu_{t+12}$ be the monthly averages of $Y_t$
2. $\epsilon_t$ independently follows multivariate normal distribution

The above model is not linear w.r.t $X_t$. Plug in $Y_t = log(X_t)$ and linearize the model (first order taylor expansion approximation), we get the following model

\begin{equation}
    X_t = e^{\epsilon_t} [\tilde{\mu}_t + \gamma_t \frac{\tilde{\mu}_t}{\tilde{\mu}_{t-1}} (X_{t-1}-\tilde{\mu}_{t-1})]
\end{equation}
Where:
1. $\gamma_t = \gamma_{t+12}$
2. $\tilde{\mu}_t = e^{\mu_t},\tilde{\mu}_{t+12}=\tilde{\mu}_t$
2. $\epsilon_t$ independently follows multivariate normal distribution $N(0,\Sigma_t)$, $\Sigma_{t+12}=\Sigma_{t}$

This model is called auto-regressive process with multiplicative error. We will fit into this model.

Denote $R_t = \frac{X_t - \tilde{\mu}_t}{\tilde{\mu}_t}$ and set $\epsilon_t$ to 0, the above model can be written as
\begin{equation}
    R_t = \gamma_t R_{t-1}
\end{equation}
$\Sigma_t$ is modelled as sample covariance matrix for
\begin{equation}
    log(\frac{X_t}{(1-\gamma_t)\tilde{\mu}_t +\gamma_t \frac{\tilde{\mu}_t}{\tilde{\mu}_{t-1}} X_{t-1}})
\end{equation}

Compute $\tilde{\mu}_t$

```python
hist_log = [numpy.log(hist[i]) for i in range(4)]
mu = [hist_log[i].mean().values for i in range(4)]
exp_mu = numpy.exp(mu).T
```

Perform least square fit to the $R_t$ sequence on a monthly basis. Beware to keep temporal order (December data should be lagged properly).

```python
gamma = numpy.empty([12,4])
for i in range(4):
    for t in range(12):
        if t == 0:
            X = hist[i].iloc[:-1,11]
            Y = hist[i].iloc[1:,0]
        else:
            X = hist[i].iloc[:,t-1]
            Y = hist[i].iloc[:,t]
        X = X/exp_mu.iloc[t-1,i] - 1
        Y = Y/exp_mu.iloc[t,i] - 1
        gamma[t][i] = numpy.dot(X,Y)/numpy.dot(X,X)
```

Compute the sample cov

```python
sigma = [None] * 12
for t in range(12):
    if t == 0:
        X = pandas.DataFrame([hist[i].iloc[:-1,11] for i in range(4)]).T
        Y = pandas.DataFrame([hist[i].iloc[1:,0] for i in range(4)]).T
    else:
        X = pandas.DataFrame([hist[i].iloc[:,t-1] for i in range(4)]).T
        Y = pandas.DataFrame([hist[i].iloc[:,t] for i in range(4)]).T
    X.reset_index(drop=True,inplace=True)
    Y.reset_index(drop=True,inplace=True)
    X.columns = [0,1,2,3]
    Y.columns = [0,1,2,3]
    matrix = numpy.log(
        Y
        / (
            (1-gamma.iloc[t]) * exp_mu.iloc[t]
            + gamma.iloc[t] * exp_mu.iloc[t]/exp_mu.iloc[t-1] * X
        )
    )
    sigma[t] = numpy.cov(matrix.T,ddof=1)
```

Now we can construct a sample path generator for the above time series model

```python
def generator(random_state, size):
    inflow = numpy.empty([size,13,4])
    inflow[:,0,:] = [[41248.715300,7386.860854,10124.561460,6123.808537]]
    for t in range(12):
        noise = numpy.exp(random_state.multivariate_normal(
            mean=[0]*4, cov=sigma[t],size=size))
        inflow[:,t+1,:] = noise * (
            (1-gamma[t]) * exp_mu[t]
            + gamma[t] * exp_mu[t]/exp_mu[t-1] * inflow[:,t,:]
        )
    return inflow[:,1:,:]
```

Use our sample path generator to simulate inflow data.

```python
import matplotlib.pyplot as plt
from msppy.utils.plot import fan_plot
simulation = [[] for _ in range(4)]
a = generator(numpy.random.RandomState(0),100)
```

Use fan plots to compare simulated data and historical data. The black dash line represents the average value. Plots on the left column show the simulated data while plots on the right column show the historical data. It is clear that the shapes are similar, except that the simulated data for the second reservior have a bit more tails than the real data. This behavior is also recognized in the paper:

http://www.optimization-online.org/DB_FILE/2017/09/6201.pdf

Overall, the above time series model fits well to the data.

For more discussion of validating the above model, have a look at the following paper:
https://www2.isye.gatech.edu/people/faculty/Alex_Shapiro/Rep-Dec2011.pdf

```python
fig = plt.figure(figsize=(10,10))
ax = [None] * 4
ax[0] = plt.subplot(421)
ax[1] = plt.subplot(423)
ax[2] = plt.subplot(425)
ax[3] = plt.subplot(427)
for i in range(4):
    fan_plot(a[:,:,i], ax[i])
bx = [None] * 4
bx[0] = plt.subplot(422, sharey=ax[0])
bx[1] = plt.subplot(424, sharey=ax[1])
bx[2] = plt.subplot(426, sharey=ax[2])
bx[3] = plt.subplot(428, sharey=ax[3])
for i in range(4):
    fan_plot(hist[i].values, bx[i])
```

Output coefficients of the generator for later use.

```python
pandas.DataFrame(gamma).to_csv("./data/gamma.csv")
for t in range(12):
    pandas.DataFrame(sigma[t]).to_csv("./data/sigma_{}.csv".format(t))
pandas.DataFrame(exp_mu).to_csv("./data/exp_mu.csv")
```


```python
import sympy as sp
import pandas as pd
from operator import mul
from functools import reduce
```

```python
def print_derivatives(nodes, *args):
    index = nodes
    cols = ['f' + i * '\'' for i in range(len(derivatives_per_node[0]))]
    return pd.DataFrame(derivatives_per_node, index=index, columns=cols)
```

```python
def calculate_div_diffs(nodes, *args, max_order=None):
    derivatives_per_node = list(zip(*args))
    max_order_per_node = [len([j for j in i if j != None]) for i in derivatives_per_node]
    if max_order == None:
        max_order = sum(max_order_per_node)

    nodes_with_order = [nodes[i] for i in range(len(nodes)) for j in range(max_order_per_node[i])]
    values_with_order = [args[0][i] for i in range(len(nodes)) for j in range(max_order_per_node[i])]
    node_to_pos = {n: p for p, n in enumerate(nodes)}
    nwo_to_pos = {i: node_to_pos[nodes_with_order[i]] for i in range(len(nodes_with_order))}

    diffs = [values_with_order]
    for order in range(max_order - 1):
        current_diffs = []
        for i in range(1, len(nodes_with_order) - order):
            if nodes_with_order[i + order] == nodes_with_order[i - 1]:
                current_diffs.append(
                    (derivatives_per_node[nwo_to_pos[i]][order + 1]) /
                    (reduce(mul, [1] + [j for j in range(1, order + 1)]))
                )
            else:
                p = diffs[-1][i] - diffs[-1][i-1]
                current_diffs.append((diffs[-1][i] - diffs[-1][i-1]) / (nodes_with_order[i + order] - nodes_with_order[i-1]))
        if len(current_diffs) == 0:
            break
        diffs.append(current_diffs)
    return diffs
```

```python
def print_diffs(diffs):
    rows = 2 * len(diffs[0]) - 1
    for r in range(rows):
        row = '\t' if r % 2 == 1 else ''
        for i in range(r % 2, min(rows - r, min(r + 1, len(diffs))), 2):
            row += f'{float(diffs[i][(r - i) // 2]):.3f}\t\t'
        print(row[:-2])
```

```python
nodes = [-1, 1, 2, 3]
fvals = [7, -11, -5, 43]
dfvals = [None, 5, 1, None]
ddfvals = [None, -22, None, None]
```

```python
derivatives_per_node = list(zip(fvals, dfvals, ddfvals))
max_order_per_node = [len([j for j in i if j != None]) for i in derivatives_per_node]
nodes_with_order = [nodes[i] for i in range(len(nodes)) for j in range(max_order_per_node[i])]
```

```python
print_derivatives(nodes, fvals, dfvals, ddfvals)
```

      f
      f'
      f''

      -1
      7
      NaN
      NaN

      1
      -11
      5.0
      -22.0

      2
      -5
      1.0
      NaN

      3
      43
      NaN
      NaN

```python
dds = calculate_div_diffs(nodes, fvals, dfvals, ddfvals)
```

```python
print_diffs(dds)
```

    7.000
    	-9.000
    -11.000		7.000
    	5.000		-14.500
    -11.000		-22.000		12.500
    	5.000		23.000		-13.833
    -11.000		1.000		-29.000		9.083
    	6.000		-6.000		22.500
    -5.000		-5.000		16.000
    	1.000		26.000
    -5.000		47.000
    	48.000
    43.000

```python
x = sp.symbols('x')
polynom = 0
mult = 1
for i in range(len(dds)):
    polynom += dds[i][0] * mult
    mult *= x - nodes_with_order[i]
```

```python
sp.expand(polynom)
```

$\displaystyle 9.08333333333333 x^{6} - 68.3333333333333 x^{5} + 176.833333333333 x^{4} - 149.333333333333 x^{3} - 87.9166666666667 x^{2} + 208.666666666667 x - 100.0$

```python
polynom_test = []
dp = polynom
for i in range(len(derivatives_per_node[0])):
    f_i = []
    for j in range(len(nodes)):
        if derivatives_per_node[j][i] != None:
            f_i.append(dp.evalf(subs={x: nodes[i]}))
        else:
            f_i.append(None)
    dp = sp.diff(dp)
    polynom_test.append(f_i)
```

```python
print_derivatives(nodes, *polynom_test)
```

      f
      f'
      f''

      -1
      7
      NaN
      NaN

      1
      -11
      5.0
      -22.0

      2
      -5
      1.0
      NaN

      3
      43
      NaN
      NaN


# Juliaによる数値計算

## 浮動小数点の演算

- **浮動小数点**:
    - 小数をコンピュータ上で表現するための数
    - 固定長の仮数部と指数部をビット2進数により表現する
    - 2進数で表現するため、ほとんどの小数は無限小数となり、有限のリソースで計算を行うコンピュータ上では誤差が生じる

簡単な例として、0.001 を 1000 個加算してみると以下のようになる

```julia
a = 0

for i = 1:1000
    a += 0.001
end

println(a)
```

    1.0000000000000007

数学的には 0.001 を 1000 個加算すれば 1 になるはずだが、上記のように $7.0\times10^{-16}$ の誤差が生じてしまっている

これが2進数表現で小数を表現する場合に生じてしまう誤差であり、**計算機イプシロン** と呼ばれる

Julia において、計算機イプシロンは `eps` 関数を用いて取得することが出来る

```julia
# Float64型浮動小数点における計算機イプシロン
eps(Float64)
```

    2.220446049250313e-16

上記のように浮動小数点では誤差が生じるが、その誤差は非常に小さく、実用上は近似計算してしまって問題ないことがほとんどである

しかしながら、このような内部表現による誤差があることは、常に意識しておかなければ予想外の問題が発生しうるため注意が必要である

ここではまず、条件分岐に関する注意点を示す

浮動小数点に関する条件式では数値誤差を十分に考慮しなければならず、特に数値比較で `==` のような完全一致比較を行うのは悪手であることが多い

```julia
s = 0
# s を 0.0 から 0.1 ずつ加算していく
for i = 1:20
    # s が 1.0 になったら終了
    ## => 浮動小数点の内部表現により s が正確に 1.0 になることはないため、この条件式は永遠に true にならない
    if s == 1.0
        break
    end
    println(s)
    s += 0.1
end
```

    0
    0.1
    0.2
    0.30000000000000004
    0.4
    0.5
    0.6
    0.7
    0.7999999999999999
    0.8999999999999999
    0.9999999999999999
    1.0999999999999999
    1.2
    1.3
    1.4000000000000001
    1.5000000000000002
    1.6000000000000003
    1.7000000000000004
    1.8000000000000005
    1.9000000000000006

上記のように、浮動小数点の比較に `==` や `!=` のような完全一致演算子を用いるのは意図と異なる結果になることが多い

こういった場合には、十分に小さな正の数を用意して、誤差がその数値以内になるかどうかをチェックするという手法がよく用いられる

例えば、Juliaにおいては概ね17桁程度の精度の浮動小数点を扱う事ができるため、$10^{-10}$ のような十分に小さな値を設定して数値比較を行えば、意図通りの結果が得られるはずである

```julia
# 十分に小さな正の値
## ϵ: \epsilon <Tab>
ϵ = 1e-10

s = 0

# s を 0.0 から 0.1 ずつ加算していく
for i = 1:20
    # s が 1.0（付近）になったら終了したい
    ## => |s-1.0| が ϵ より小さくなったら終了
    if abs(s - 1.0) < ϵ
        break
    end
    println(s)
    s += 0.1
end
```

    0
    0.1
    0.2
    0.30000000000000004
    0.4
    0.5
    0.6
    0.7
    0.7999999999999999
    0.8999999999999999

上記では $\epsilon$ を $10^{-10}$ と設定したが、この値をどのようにして決めるのが良いのかは、解く問題によるため一概には言えない

理論的に値が決められることもあれば、実験的にうまくいく値を採用することもある

### 演算による桁落ち

次に2次方程式の解を求める関数を考えてみる

2次方程式 $ax^2 + bx + c = 0 \quad (a\neq0)$ の解は

$$
x = \frac{-b\pm\sqrt{b^2 - 4ac}}{2a}
$$

で与えられる

これを計算する関数を定義してみる

ここでは話を簡単にするため、解が実数になる場合、つまり $b^2-4ac \geq 0$ の場合のみを考えることにする

```julia
# 2次方程式の解を求める関数
qeq(a, b, c) = begin
    d = sqrt(b^2 - 4*a*c)
    # 2つの解をTupleで返す
    ((-b + d) / 2a, (-b - d) / 2a)
end

"""
test: x^2 + 5x + 6 = 0
    a = 1
    b = 5
    c = 6
"""
# answer: (x + 2)(x + 3) = 0 より
## x = -2, -3 になるはず
qeq(1, 5, 6)
```

    (-2.0, -3.0)

上記の通り、理論値通り正確に計算できた

続いて、2次方程式 $x^2 + 1.000000001x + 0.000000001 = 0$ について考えてみる

```julia
"""
test: x^2 + 1.000000001x + 0.000000001 = 0
    a = 1
    b = 1.000000001
    c = 0.000000001
"""
# answer: (x + 1)(x + 0.000000001) = 0 より
## x = -1, -0.000000001 になるはず
qeq(1, 1.000000001, 0.000000001)
```

    (-1.0000000272292198e-9, -1.0)

上記の通り、理論値と異なる値が算出された

これは浮動小数点の **桁落ち** と呼ばれる現象により、誤差が生じているためである

今回の場合、分子の $-b + \sqrt{b^2 - 4ac}$ が 0 に非常に近いため、桁落ちが生じている

実際に $\sqrt{b^2 - 4ac}$ を計算してみると以下のようになる

```julia
sqrt(1.000000001^2 - 4 * 1 * 0.000000001)
```

    0.999999999

上記の結果と $-b = -1.000000001$ を加算すると、小数点第8位までは消えてしまう

つまり、もともとは約17桁まで信用できた数字が、9桁分までの数字が消えてしまうことにより、信用できる桁数が残り8桁になってしまうということである

これが桁落ちと呼ばれる現象である

```julia
-1.000000001 + sqrt(1.000000001^2 - 4 * 1 * 0.000000001)
```

    -2.0000000544584395e-9

このように、絶対値の近い減算処理では有効桁数を大きく失う可能性があるため危険である

#### 桁落ちの回避方法
桁落ちを回避するためには、絶対値が近い数値同士の減算処理が発生しそうな計算を回避できるように、数式を同値な式で置き換えれば良い

上記2次方程式の場合、$\sqrt{}$ の計算結果は必ず正になるはずであるため、絶対値の近い数値同士の減算処理が発生する可能性は $b$ の符号によって分けて考える必要がある

$b\geq0$ のときは、分子の $\pm$ が $+$ のときに、近い数値同士の減算処理が発生しうる

逆に $b\gt0$ のときは、分子の $\pm$ が $-$ のときに、近い数値同士の減算処理が発生しうる

これらをまとめて表現するために符号関数 $\rm{sign}(x)$ という関数を導入する

$$
\rm{sign}(x) = \left\{
\begin{align}
-1\quad&(x\lt0) \\
0\quad&(x=0) \\
1\quad&(x\gt0) \\
\end{align}
\right.
$$

符号関数を用いて、直接計算しても問題ない（絶対値の近い減算処理が**発生しない**）式を考えると、以下のように表される

$$
\frac{-b-\rm{sgin}(b)\sqrt{b^2 - 4ac}}{2a}
$$

次に直接計算すると問題のある（絶対値の近い減算処理が発生しうる）方の解を計算する方法を考える

ここで、2次方程式 $ax^2 + bx + c = 0$ の解を $\alpha, \beta$ と置くと、次の式が成り立つことが知られている

$$
\alpha\beta = \frac{c}{a}
$$

このことは、以下のように実際に計算してみるとすぐに分かる

$$
\begin{align}
\alpha\beta &= \frac{-b + \sqrt{b^2 - 4ac}}{2a}\times\frac{-b - \sqrt{b^2 - 4ac}}{2a} \\
&= \frac{(-b)^2 - (\sqrt{b^2 - 4ac})^2}{4a^2} \\
&= \frac{b^2 - (b^2 - 4ac)}{4a^2} \\
&= \frac{c}{a}
\end{align}
$$

上記を利用することで、直接計算に問題のある方の解 $\beta$ を、直接計算しても問題ない方の解 $\alpha$ で表すことが出来るようになる

つまり、

$$
\begin{align}
\alpha &= \frac{-b - \rm{sign}(b)\sqrt{b^2 - 4ac}}{2a} \\
\beta &= \frac{c}{a\alpha}
\end{align}
$$

という形で計算すれば桁落ちを回避できるはずである

```julia
# 2次方程式の解を求める関数（桁落ち回避版）
qeq(a, b, c) = begin
    # Julia には sign 関数は事前定義されているためそれを利用
    alpha = (-b - sign(b) * sqrt(b^2 - 4*a*c)) / 2a
    beta = c / (a * alpha)
    (alpha, beta)
end

# test: x^2 + 1.000000001x + 0.000000001 = 0
qeq(1, 1.000000001, 0.000000001)
```

    (-1.0, -1.0e-9)

上記の通り、前回よりも正確な値を計算することが出来た

このように、誤差を比較的小さく抑えられる方法を **数値的に安定な解法** と呼ぶ

### 数値範囲の考慮
次に、ソフトプラス関数と呼ばれる次のような関数を考える

$$
\rm{softplus}(x) = \ln(1+e^x)
$$

この関数は $x$ が十分に大きい場合、$\rm{softplus}(x) \approx x$ で近似される

実際 $e^x$ は $x$ が大きくなると急激に大きくなるため、$1 + e^x \approx e^x$ と近似できるようになり、$\rm{softplus}(x) \approx \ln e^x = x$ となる

まずは、この関数を定義そのままで記述してみる

```julia
softplus(x) = log(1 + exp(x))

# test
softplus(-1) |> println
softplus(0) |> println
softplus(1000) |> println
```

    0.31326168751822286
    0.6931471805599453
    Inf

上記のように $x=-1$ と $x=0$ のときは順調に計算出来ていたが、$x=1000$ のときは `Inf`（無限大）となってしまった

これは、Julia で扱える浮動小数点の範囲を超えて大きな値になったことを意味する

しかしながら、前述の考察によると $\rm{softplus}(1000) \approx 1000$ になるはずであるため、浮動小数点型で十分に計算できる範囲のように思われる

それでも `Inf` になってしまうのは、`exp(1000)` の計算時点で浮動小数点型の範囲を超えてしまうためである

```julia
exp(1000)
```

    Inf

ここでの問題は、数学的に考えれば計算結果はさほど大きくない（Julia の浮動小数点型として十分に取り扱い可能）にも関わらず、計算過程で `Inf` が発生してその後の計算も全て `Inf` になってしまうという点である

この問題を克服するには、元の式を同値な式に変形し、計算過程で `Inf` が発生しないようにすることである

試しに以下のように式変形してみる

$$
\begin{align}
\ln(1 + e^x) &= \ln\{e^x(x^{-x} + 1)\} \\
&= \ln{e^x} + \ln(e^{-x} + 1) \\
&= x + \ln(e^{-x} + 1)
\end{align}
$$

上記のように変形すれば、$x$ が十分に大きくなると $e^{-x}\right0$ となるため `Inf` が発生する問題は回避できる

しかし、今度は $x$ が十分に小さくなる場合に $e^{-x}\right\inf$ となってしまい、この場合は元の式の方が良かったことになる

したがって、どこかの $x$ を境目にして2つの式を分けるのが良さそうである

ここでは $x=0$ を境目にして、以下のように式変形してみる

$$
\begin{align}
\rm{softplus}(x) &= \left\{
\begin{array}{cc}
\ln(1 + e^x) & (x\lt0) \\
x + \ln(1 + e^{-x}) & (x\geq0)
\end{array}
\right. \\
&= \max(0,x) + \ln(1 + e^{-|x|})
\end{align}
$$

ここで $\max(a,b)$ とは $a$, $b$ の内の大きい方を返す関数である

上式に従って、`softplus` 関数を修正してみる

```julia
softplus(x) = max(0, x) + log(1 + exp(-abs(x)))

# test
softplus(-1) |> println
softplus(0) |> println
softplus(1000) |> println
softplus(-1000) |> println
```

    0.31326168751822286
    0.6931471805599453
    1000.0
    0.0

上記の通り、今度は $x=1000$ の場合も $x=-1000$ の場合も問題なく計算できている

以上、コンピュータによる数値計算で気をつけるべき点を、例を挙げながら見てきたが、ポイントをまとめると以下のようになる

- 原則として、浮動小数点型の比較には `==` や `!=` のような完全一致演算子を用いない
    - 十分な小さな正の数を設定して、「比較対象との差がその数値以内になる」という条件式を用いる方法がよく採用される
- 数値が近いものの減算処理は、桁落ち現象により有効桁数が失われるため注意
    - 数値が近いものの減算処理が入らないように、同値な式に変形して計算する
- 計算過程で `Inf` や `-Inf` が生じてしまう計算に注意
    - 浮動小数点型で扱える範囲の数値しか現れないように式変形して計算する

```julia

```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Probability Distributions
***

The probability density function for a Gaussian distribution is given by

\begin{equation}\label{eq:gaussian_pdf}
    p(x) = \frac{1}{\sigma \sqrt{2 \pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right),
\end{equation}

where $\mu$ is the mean and $\sigma$ is the standard deviation. This probability density function may be calculated using the SciPy implementation ***scipy.stats.norm.pdf(x, loc, scale)***, where the keyword ***loc*** is the mean and keyword ***scale*** is the standard deviation.

Other probability functions may be found from the routines in `scipy.stats` as shown in this example.
***

Set the location, scale and shape factor for the distribution

```python
loc = 2.0

scale = 1.0

shape_factor = 10.0
```

Set the type of distribution (Gaussian, Weibull, Rayleigh, Rice or Chi-Squared)

```python
distribution_type = 'Gaussian'
```

Calculate the distributions from the `scipy` routines

```python
from scipy.stats import norm, weibull_min, rayleigh, rice, chi2

from numpy import linspace

if distribution_type == 'Gaussian':

    x = linspace(norm.ppf(0.001, loc, scale),  norm.ppf(0.999, loc, scale), 200)

    y = norm.pdf(x, loc, scale)

elif distribution_type == 'Weibull':

    x = linspace(weibull_min.ppf(0.001, shape_factor), weibull_min.ppf(0.999, shape_factor), 200)

    y = weibull_min.pdf(x, shape_factor, loc, scale)

elif distribution_type == 'Rayleigh':

    x = linspace(rayleigh.ppf(0.001, loc, scale), rayleigh.ppf(0.999, loc, scale), 200)

    y = rayleigh.pdf(x, loc, scale)

elif distribution_type == 'Rice':

    x = linspace(rice.ppf(0.001, shape_factor), rice.ppf(0.999, shape_factor), 200)

    y = rice.pdf(x, shape_factor, loc, scale)

elif distribution_type == 'Chi Squared':

    x = linspace(chi2.ppf(0.001, shape_factor), chi2.ppf(0.999, shape_factor), 200)

    y = chi2.pdf(x, shape_factor, loc, scale)
```

Display the resulting probability distribution using the `matplotlib` routines

```python
from matplotlib import pyplot as plt

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(x, y, '')

# Set the plot title and labels

plt.title('Probability Density Function', size=14)

plt.xlabel('x', size=12)

plt.ylabel('Probability p(x)', size=12)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)
```

```python

```


Linear Regression

Brief derivation of OLS meethod of Linear Regression.
There are many alternate derivations
Assume a system of linear equations or assume a set of points with the goal of
drawing a line through them.

For a set of input points$x_i, y_i$ define a least squares distance measure:

$error = (y_i-x_i)^2$

Take the derivative to minimize this error and set it to 0 to calculate the coefficients:
$$

```python
import numpy as np

pos_examples1 = np.array([[0.871429,0.624585,1],[ -0.020000,-0.923588,1],
                         [0.362857,-0.318937,1],[0.888571,-0.870432,1]])

neg_examples1 = np.array([[-0.80857,0.83721,1],[0.35714,0.85050,1],
                         [-0.75143,-0.73090,1],[-0.30000,0.12625,1]
                        ])

```

The simplest neuron is a linear neuron used for performing linear separation.

We define the operation of the neuron as multiplying the input with a weight: $w_i$.
Define the loas function as the expected output minus the actual squared. $loss = \frac{1}{2}\sum\limits_{i=1}^n(t_n-y_n)^2$.
We are going to take the sum of all the errors for all the sample points and update the weights. This is batch training.
If we process 1 sample at at time this is online training. To find the values of weights we use gradient descent.

We want to minimize loss as a function of the weights. Using the chain rule we can expand the partial deriviative WRT w
as follows:

$\frac{\partial loss}{\partial w_i} = \frac{\partial y_i}{\partial w_i} \cdot \frac{\partial loss}{\partial y_i}$

$\frac{\partial loss}{\partial w_i} = \frac{1}{2} \cdot (2)(t_i-y_i)(-1) = -(t_i-y_i)$

$y_i = \sum\limits_{i=1}^N w_ix_i$ so $\frac{\partial y_i}{\partial w_i} = \sum\limits_{i=1}^N x_i$

$w_{i+1} = w_i+\delta \cdot w_i$

$\delta w_i = \epsilon \cdot \frac{\partial loss}{\partial w_i} $
where $\epsilon$ is the learning rate

$\delta w_i = \epsilon \cdot \sum\limits_{i=1}^N x_i(t_i-y_i)$

Set the initial weights to a random value with variance=1 and compute the delta w for all the training samples and
keep iterating until the weight values converge.

```python
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from sklearn import datasets

iris = datasets.load_iris()
print (type(iris))
print( "iris.feature_names:",iris.feature_names)
print ("iris.data:",iris.data)
print("iris.target_names:", iris.target_names)
print (iris.target)

#pick first and last columns to do regression on
x = np.array([x[3] for x in iris.data])
y = np.array([y[0] for y in iris.data])

print("x:",x)
print("y:",y)

print("x shape:",x.shape)
print("y shape:",y.shape)

global_step = tf.Variable(0, trainable=False,name="global_step")

#model
x_ = tf.placeholder(tf.float32, shape=[None,1])
y_ = tf.placeholder(tf.float32, shape=[None,1])

m = tf.Variable(tf.zeros(shape=[1,1]))
b = tf.Variable(tf.zeros(shape=[1,1]))

output = tf.add(tf.matmul(x_,m),b)

loss = tf.reduce_mean(tf.square(y_ - output))

print("global_step:", global_step)
learning_rate = tf.train.exponential_decay(
    learning_rate=0.1, global_step=global_step,
    decay_steps=100, decay_rate=0.001,name="learning_rate")
print("learning_rate:", learning_rate)

sess = tf.Session()
init = tf.global_variables_initializer()

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
rand_index = np.random.choice(len(x), size=len(x))
#print("rand_index:", rand_index)
rand_x = np.transpose([x[rand_index]])
rand_y = np.transpose([y[rand_index]])
x_transpose = np.transpose(x)
y_transpose = np.transpose(y)
#print("rand_x:",rand_x)
#print("rand_y:",rand_y)
sess.run(init)
sess.run(optimizer, feed_dict={x_: rand_x, y_: rand_y})
graph_loss = []
for epochs in range(100):
    sess.run(optimizer, feed_dict={x_: rand_x, y_: rand_y})
    epoch_loss = sess.run(loss,feed_dict={x_: rand_x, y_: rand_y} )
    graph_loss.append(epoch_loss)
    print("epochs:",epochs," epoch_loss:",epoch_loss," m:",sess.run(m)," b:",sess.run(b))

x=[]
x.extend(range(100))
import matplotlib.pyplot as plt
plt.plot(x,graph_loss)
plt.show()
```

Results Discussion

The results for fixed learning rate of .1 are here:

The results for fixed learning rate decay starting at .1 are here:

Is the lower loss for the fixed learning rate mean it is a better solution?

There are 150 samples so a .239-.19=.040 percent difference means it got 7 more correct out of 150. This does not
meet our 30 minimum for statistical significance.
Note: these should end up in a visualization so consumers of APIs are clear.

Logistic Regression for classification. Simplest case for 2 classes.

Logistic Neuron
$y = \frac{1}{1+exp(-z)}$

$\frac{dy}{dz}=y(1-y)$

```python
import tensorflow as tf

iris = datasets.load_iris()
X = iris.data[:, :2]  # we only take the first two features.
Y = iris.target

print("X:",X)
print("Y:",Y)

#pick a column, mess with the data to get a 2 class problem.

```

    X: [[ 5.1  3.5]
     [ 4.9  3. ]
     [ 4.7  3.2]
     [ 4.6  3.1]
     [ 5.   3.6]
     [ 5.4  3.9]
     [ 4.6  3.4]
     [ 5.   3.4]
     [ 4.4  2.9]
     [ 4.9  3.1]
     [ 5.4  3.7]
     [ 4.8  3.4]
     [ 4.8  3. ]
     [ 4.3  3. ]
     [ 5.8  4. ]
     [ 5.7  4.4]
     [ 5.4  3.9]
     [ 5.1  3.5]
     [ 5.7  3.8]
     [ 5.1  3.8]
     [ 5.4  3.4]
     [ 5.1  3.7]
     [ 4.6  3.6]
     [ 5.1  3.3]
     [ 4.8  3.4]
     [ 5.   3. ]
     [ 5.   3.4]
     [ 5.2  3.5]
     [ 5.2  3.4]
     [ 4.7  3.2]
     [ 4.8  3.1]
     [ 5.4  3.4]
     [ 5.2  4.1]
     [ 5.5  4.2]
     [ 4.9  3.1]
     [ 5.   3.2]
     [ 5.5  3.5]
     [ 4.9  3.1]
     [ 4.4  3. ]
     [ 5.1  3.4]
     [ 5.   3.5]
     [ 4.5  2.3]
     [ 4.4  3.2]
     [ 5.   3.5]
     [ 5.1  3.8]
     [ 4.8  3. ]
     [ 5.1  3.8]
     [ 4.6  3.2]
     [ 5.3  3.7]
     [ 5.   3.3]
     [ 7.   3.2]
     [ 6.4  3.2]
     [ 6.9  3.1]
     [ 5.5  2.3]
     [ 6.5  2.8]
     [ 5.7  2.8]
     [ 6.3  3.3]
     [ 4.9  2.4]
     [ 6.6  2.9]
     [ 5.2  2.7]
     [ 5.   2. ]
     [ 5.9  3. ]
     [ 6.   2.2]
     [ 6.1  2.9]
     [ 5.6  2.9]
     [ 6.7  3.1]
     [ 5.6  3. ]
     [ 5.8  2.7]
     [ 6.2  2.2]
     [ 5.6  2.5]
     [ 5.9  3.2]
     [ 6.1  2.8]
     [ 6.3  2.5]
     [ 6.1  2.8]
     [ 6.4  2.9]
     [ 6.6  3. ]
     [ 6.8  2.8]
     [ 6.7  3. ]
     [ 6.   2.9]
     [ 5.7  2.6]
     [ 5.5  2.4]
     [ 5.5  2.4]
     [ 5.8  2.7]
     [ 6.   2.7]
     [ 5.4  3. ]
     [ 6.   3.4]
     [ 6.7  3.1]
     [ 6.3  2.3]
     [ 5.6  3. ]
     [ 5.5  2.5]
     [ 5.5  2.6]
     [ 6.1  3. ]
     [ 5.8  2.6]
     [ 5.   2.3]
     [ 5.6  2.7]
     [ 5.7  3. ]
     [ 5.7  2.9]
     [ 6.2  2.9]
     [ 5.1  2.5]
     [ 5.7  2.8]
     [ 6.3  3.3]
     [ 5.8  2.7]
     [ 7.1  3. ]
     [ 6.3  2.9]
     [ 6.5  3. ]
     [ 7.6  3. ]
     [ 4.9  2.5]
     [ 7.3  2.9]
     [ 6.7  2.5]
     [ 7.2  3.6]
     [ 6.5  3.2]
     [ 6.4  2.7]
     [ 6.8  3. ]
     [ 5.7  2.5]
     [ 5.8  2.8]
     [ 6.4  3.2]
     [ 6.5  3. ]
     [ 7.7  3.8]
     [ 7.7  2.6]
     [ 6.   2.2]
     [ 6.9  3.2]
     [ 5.6  2.8]
     [ 7.7  2.8]
     [ 6.3  2.7]
     [ 6.7  3.3]
     [ 7.2  3.2]
     [ 6.2  2.8]
     [ 6.1  3. ]
     [ 6.4  2.8]
     [ 7.2  3. ]
     [ 7.4  2.8]
     [ 7.9  3.8]
     [ 6.4  2.8]
     [ 6.3  2.8]
     [ 6.1  2.6]
     [ 7.7  3. ]
     [ 6.3  3.4]
     [ 6.4  3.1]
     [ 6.   3. ]
     [ 6.9  3.1]
     [ 6.7  3.1]
     [ 6.9  3.1]
     [ 5.8  2.7]
     [ 6.8  3.2]
     [ 6.7  3.3]
     [ 6.7  3. ]
     [ 6.3  2.5]
     [ 6.5  3. ]
     [ 6.2  3.4]
     [ 5.9  3. ]]
    Y: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2]

Softmax regression model/classification and cross entropy

For more than 2 classes we can use a combination of logistic regression models or softmax regression which is the
generalization of logistic regression to multiple classes.

We saw the logistic function used to output probability and
$\{ (x^{(1)}, y^{(1)}), \ldots, (x^{(m)}, y^{(m)}) \}$ from a dataset with input X, class Y per input and with m trained
samples. Using Ng's notation our hypothesis/logistic is:
$
\begin{align}
h_\theta(x) = \frac{1}{1+\exp(-\theta^Tx)},
\end{align}
$
where $\theta$ is our weight matrix. Hinton uses notation $y=\frac{1}{1+exp(-z)}$

For multiple classes logistc loss also called the cross entropy loss function is:

$
\begin{align}
J(\theta) = -\frac{1}{m} \left[ \sum_{i=1}^m y^{(i)} \log h_\theta(x^{(i)}) + (1-y^{(i)}) \log (1-h_\theta(x^{(i)})) \right]
\end{align}
$

which is derived from the law of total probability for multiple classes. Ecah class is reprented by $y_k$ where for 10 classes k would be 1..10

The softmax loss is:

$p(y^{(i)} = j | x^{(i)} ; \theta) = \frac{e^{\theta_j^T x^{(i)}}}{\sum_{l=1}^k e^{ \theta_l^T x^{(i)}} }$

and the derivative of the logistic loss is in Ng notation:

$
\begin{align}
\nabla_{\theta_j} J(\theta) = - \frac{1}{m} \sum_{i=1}^{m}{ \left[ x^{(i)} \left( 1\{ y^{(i)} = j\}  - p(y^{(i)} = j | x^{(i)}; \theta) \right) \right]  }
\end{align}
$

Hinton calls Cross entropy as the right cost function to use for softmax.

```python

```


# Vector Norm

```python
import numpy as np
from scipy import signal
from scipy.spatial import distance
```

```python
A = np.array([1+1j, 2+2j, 3+3j, 4+4j, 5+5j])
B = np.array([6-6j, 7-7j, 8-8j, 9-9j, 10-10j])
C = np.array([2,3,5,7,11])
Z = np.array([0,0,0,0,0])

D = np.array([A,B])
```

For every complex inner product space V(-,-), we can define a norm or length which is a function defined as
\begin{align}
  | |: V -> E
\end{align}
defined as
\begin{align}
  |V| = |\sqrt{V . V}|
\end{align}

```python
[
np.linalg.norm(A) == np.abs(np.sqrt(np.dot(A,A))),
np.linalg.norm(B) == np.abs(np.sqrt(np.dot(B,B))),
np.linalg.norm(C) == np.abs(np.sqrt(np.dot(C,C)))
]
```

    [True, True, True]

```python
[
    np.linalg.norm(A),
    np.linalg.norm(B),
    np.linalg.norm(C),
]
```

    [10.488088481701515, 25.69046515733026, 14.422205101855956]

# Vector Distance

For every complex inner product space V(-,-), we can define a distance function
\begin{align}
      d(,) : V x V -> E
\end{align}
where
\begin{align}
      d(V1,V2) : |V1 - V2| = \sqrt{V1-V2, V1-V2}
\end{align}

```python
distance.euclidean(A, B)
```

    27.748873851023216

```python
np.linalg.norm(A-B) == distance.euclidean(A, B)
```

    True

```python
np.round(  distance.euclidean(A, B),                 10) == \
np.round(  np.abs(np.sqrt(np.dot(A,A)-np.dot(B,B))), 10)
```

    True

Distance is symmetric: d(V, W) = d(W, V)

```python
distance.euclidean(A, B) == distance.euclidean(B, A)
```

    True

Distance satisfies the triangle inequality: d(U, V) ≤ d(U, W) + d(W, V)

```python
distance.euclidean(A, C), distance.euclidean(A, B) + distance.euclidean(B, C)
```

    (10.295630140987, 47.13959328068853)

```python
distance.euclidean(A, C) <= distance.euclidean(A, B) +  distance.euclidean(B, C)
```

    True

Distance is nondegenerate: d(V, W) > 0 if V ≠ W and d(V, V) = 0.

```python
distance.euclidean(Z,Z)
```

    0.0

```python
distance.euclidean(A,Z), distance.euclidean(A,Z) > 0
```

    (10.488088481701515, True)

## Orthogonal Vectors

The dot product of orthogonal vectors is zero

```python
X = np.array([1,0])
Y = np.array([0,1])
```

```python
np.dot(X,Y)
```

    0

## Kronecker Delta

δj,k is called the Kronecker delta function.

δj,k =
  1 (if i == j);
  0 (if i != j);

```python
M = np.matrix([[1,2,3],[4,5,6],[7,8,9]]); X
```

    array([1, 0])

```python
{ "shape": M.shape, "size": M.size }
```

    {'shape': (3, 3), 'size': 9}

```python
def kronecker_delta(matrix):
    output = np.copy(matrix)
    for i in range(0, matrix.shape[0]):
        for j in range(0, matrix.shape[1]):
            output[i,j] = output[i,j] if i == j else 0
    return output
```

```python
kronecker_delta(M)
```

    array([[1, 0, 0],
           [0, 5, 0],
           [0, 0, 9]])

It is equlivant to element wise multiplication by the identity matrx

```python
np.multiply(M, np.identity(3))
```

    matrix([[1., 0., 0.],
            [0., 5., 0.],
            [0., 0., 9.]])

```python
kronecker_delta(M) == np.multiply(M, np.identity(M.shape[0]))
```

    matrix([[ True,  True,  True],
            [ True,  True,  True],
            [ True,  True,  True]])

NOTE: np.kron is the Kronecker (tensor) product function, and not the Kronecker DELTA

```python
np.kron(M,M)
```

    matrix([[ 1,  2,  3,  2,  4,  6,  3,  6,  9],
            [ 4,  5,  6,  8, 10, 12, 12, 15, 18],
            [ 7,  8,  9, 14, 16, 18, 21, 24, 27],
            [ 4,  8, 12,  5, 10, 15,  6, 12, 18],
            [16, 20, 24, 20, 25, 30, 24, 30, 36],
            [28, 32, 36, 35, 40, 45, 42, 48, 54],
            [ 7, 14, 21,  8, 16, 24,  9, 18, 27],
            [28, 35, 42, 32, 40, 48, 36, 45, 54],
            [49, 56, 63, 56, 64, 72, 63, 72, 81]])


# Solving Systems of Linear Equations

### A notebook detailing codes for solving a system of linear equations using the Gaussian Elimination Method:

```python
def Gaussian(M):
    # eliminate columns
    for i in range(len(M[0])):
        for j in range(i+1, len(M)):
            r = [(jValue * (-(M[j][i] / M[i][i]))) for jValue in M[i]]
            M[j] = [sum(pair) for pair in zip(M[j], r)]
    # backsolve by substitution
    ans = []
    M.reverse() # makes it easier to backsolve
    for sol in range(len(M)):
            if sol == 0:
                ans.append(M[sol][-1] / M[sol][-2])
            else:
                inner = 0
                # substitute in all known coefficients
                for x in range(sol):
                    inner += (ans[x]*M[sol][-2-x])
                # the equation is now reduced to ax + b = c form
                # solve with (c - b) / a
                ans.append((M[sol][-1]-inner)/M[sol][-sol-2])
    ans.reverse()
    return ans
```

Given:
\begin{equation}
2x+y+z = 1 \\
6x+2y+z = -1 \\
-2+2y+z = 7
\end{equation}

```python
M = [[2, 1, 1, 1],
     [6, 2, 1, -1],
     [-2, 2, 1, 7]]
Mans = Gaussian(M)
print 'x = %i' %Mans[0]
print 'y = %i' %Mans[1]
print 'z = %i' %Mans[2]
```

    x = -1
    y = 2
    z = 1

Numerical Method Codes/Gaussian Elimination Method.ipynb


# 更加优雅的输出打印结果

sympy支持unicode,可以使用`pprint()`函数输出优雅的数学公式,通过初始化时传入参数`use_latex=True`,使用latex作为引擎美化输出,我们以上一个例子中欧拉公式左边的泰勒展开作为例子

```python
import sympy as sp
from sympy import init_printing
init_printing(use_latex=True)
from sympy import pprint
```

```python
from sympy import exp,I#e的幂
from sympy import series#泰勒展开函数
from sympy import symbols
x  = symbols("x")
tmp = series(exp(I*x), x, 0, 10)#公式,变量,
```

```python
from sympy import re,im #获取实部虚部
```

```python
tmp
```

```python
re(tmp)
```

```python
im(tmp)
```

```python
pprint(tmp)
```

               2      3    4      5     6      7      8        9
              x    ⅈ⋅x    x    ⅈ⋅x     x    ⅈ⋅x      x      ⅈ⋅x      ⎛ 10⎞
    1 + ⅈ⋅x - ── - ──── + ── + ──── - ─── - ──── + ───── + ────── + O⎝x  ⎠
              2     6     24   120    720   5040   40320   362880

## 输出latex代码

我们也可以将公式用`sp.latex(}`输出为latex代码

```python
sp.latex(tmp)
```

    '1 + i x - \\frac{x^{2}}{2} - \\frac{i x^{3}}{6} + \\frac{x^{4}}{24} + \\frac{i x^{5}}{120} - \\frac{x^{6}}{720} - \\frac{i x^{7}}{5040} + \\frac{x^{8}}{40320} + \\frac{i x^{9}}{362880} + O\\left(x^{10}\\right)'

src/数据分析篇/工具介绍/SymPy/输出/更加优雅的输出打印结果.ipynb


# Rocket Assignment

The equations of motion for a rocket in purely vertical flight are given by

$$
\begin{align}
\frac{dh}{dt} &= v\\
(m_s+m_p) \frac{dv}{dt}& = -(m_s+m_p)g + \dot{m}_pv_e - \frac{1}{2}\rho v|v|AC_D
\end{align}
$$

$h$ is the altitude of the rocket

$m_s = 50kg$ is the weight of the rocket shell

$g = 9.81 \frac{m}{s^2}$

$\rho = 1.091 \frac{kg}{m^3}$ is the average air density (assumed constant throughout flight)

$A = \pi r^2$ is the maximum cross sectional area of the rocket, where $r = 0.5 m$

$v_e = 325 \frac{m}{s}$ is the exhaust speed

$C_D = 0.15 $ is the drag coefficient

$m_{po} = 100 kg$ at time $t = 0$ is the initial weight of the rocket propellant

The mass of the remaining propellant is given by:

$$m_p = m_{po} - \int^t_0 \dot{m}_p d\tau$$

where $\dot{m}_p$ is the time-varying burn rate given by the following expression:

$$
\begin{equation}
    \dot{m}_p \left( t \right) =
    \begin{cases}
        20 & \quad \text{if} \quad t < 5 \\
        0 & \quad \text{otherwise}
    \end{cases}
\end{equation}
$$

Using Euler's method with a time-step size of $\Delta t=0.1s$, create a Python script to calculate the altitude and velocity of the rocket from launch until crash down.

```python
#essential libraries
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
plt.rcParams['font.family'] = 'serif'
plt.rcParams['font.size'] = 16
```

```python
#Euler's method(too lazy to write description D:)
def Euler_method(u, dt, f, *args):
    u_new = u + dt*f(u, *args)
    return u_new
```

```python
def weight_of_propellant(t, mp0):
    if t< 5:
        return mp0*(1-t/5)
    else:
        return 0

def rate_of_burn(t):
    if t<5:
        return 20
    else:
        return 0
```

```python
def rhs_rocket(u, weight_of_propellant, rate_of_burn, t, ms, g, ro, A, ve, CD, mp0):
    h, v = u
    mp = weight_of_propellant(t, mp0)
    dmpdtt = rate_of_burn(t)
    rhs = np.array([v,
                    -g - 0.5*ro*A*CD/(ms+mp)*v*abs(v) + ve*dmpdtt/(ms+mp)])
    return rhs
```

```python
dt = 0.1
T = 100.0
N = int(T/dt)

ms = 50.0
g = 9.81
ro = 1.091
r = 0.5
A = np.pi*r**2
ve = 325.0
CD = 0.15
mp0 = 100.0
```

```python
u = np.empty((N,2))
u[0] = [0,0]
for i in range(N-1):
    u[i+1] = Euler_method(u[i], dt, rhs_rocket, weight_of_propellant, rate_of_burn, i*dt, ms, g, ro, A, ve, CD, mp0)
```

```python
h = u[:,0]
v = u[:,1]
t = np.arange(0,T,dt)
```

```python
ground_hit = np.where(h < 0)[0][0]
print(ground_hit)
```

    371

```python
print("Time of flight = {} s".format(t[ground_hit-1]))
fig = plt.figure(figsize = ((10,10)))
plt.plot(t[:ground_hit],h[:ground_hit], linewidth = 2.0, color = 'r')
plt.ylabel('Altitude, m')
plt.xlabel('Time, s')
plt.grid()
plt.show()

fig = plt.figure(figsize = ((10,10)))
plt.plot(t[:ground_hit],v[:ground_hit], linewidth = 2.0, color = 'b')
plt.ylabel('Speed, m/s')
plt.xlabel('Time, s')
plt.grid()
plt.show()
```

## First question
1. At time $t=3.2s$, what is the mass (in kg) of rocket propellant remaining in the rocket?

```python
#print(weight_of_propellant(3.2, 100))
print("Propellant_mass(t = 3.2 s) = {:.1f}.".format(weight_of_propellant(3.2, 100)))
```

    Propellant_mass(t = 3.2 s) = 36.0.

## Second question
2. What is the maximum speed of the rocket in $\frac{m}{s}$?
    At what time does this occur (in seconds)?
    What is the altitude at this time (in meters)?

```python
max_v = np.argmax(v)
#print(v[max_v])
#print(t[max_v])
#print(h[max_v])
print("Max speed = {:.2f} m/s, corresponding time = {:.2f} s, corresponding altitude = {:.2f} m.".format(v[max_v], t[max_v], h[max_v]))
```

    Max speed = 232.11 m/s, corresponding time = 5.00 s, corresponding altitude = 523.52 m.

## Third question
3. What is the rocket's maximum altitude during flight (in meters)? At what time (in seconds) does this occur?

```python
max_h = np.argmax(h)
#print(v[max_h])
#print(t[max_h])
#print(h[max_h])
print("Max altitude = {:.2f} m, corresponding time = {:.2f} s.".format(h[max_h], t[max_h]))
```

    Max altitude = 1334.18 m, corresponding time = 15.70 s.

## Fourth question
4. At what time (in seconds) does the rocket impact the ground? What is the velocity of the rocket (in $\frac{m}{s}$) at time of impact?

```python
#print(h[ground_hit])
#print(h[ground_hit-1])
#print(v[ground_hit-1])
print("Impact time = {:.2f} s, velocity of rocket at time of impact = {:.2f} m/s.".format(t[ground_hit-1], v[ground_hit-1]))
```

    Impact time = 37.00 s, velocity of rocket at time of impact = -85.98 m/s.


# Exercises to Lecture 2: Introduction to Python 3.6
By Dr. Anders S. Christensen
`anders.christensen @ unibas.ch`

## Exercise 2.1: define a function

One of the fundamental things we learned was how to define functions.

For example, the function defined below `square(x)` calculates the square, $x^2$:

```
def square(x):
  y = 2 * x**2
  return y

print(square(2))
```

    8

### Question 2.1.1:

Similarly to the above code, make a function called. for example, `poly(x)` to calculate the following polynomial:
\begin{equation}
p\left( x \right) = 5 x^2 - 4x + 1
\end{equation}

Lastly, print the value of $p\left( 5 \right)$

```
def poly(x):

  # Fill out the rest of the function yourself!
  p = 5* x**2  - 4*x + 1

  return p

# Print the value for x=5
print(poly(5))
```

    106

## Exercise 2.2: Loop within function

The code below implements the product of all numbers up to n, i.e. the factorial function "$!$"

\begin{equation}
f(n) = n! = \prod_{i=1}^n i = 1 \cdot 2 \cdot \quad  ... \quad \cdot (n -1) \cdot n
\end{equation}

As an example, the code to calculate $5!$ is shown here:

```
n = 5

f = 1
for i in range(1,n+1):
  f =  f * i

print("Factorial", n, "is", f)
```

    Factorial 5 is 120

### Question 2.2.1

Unfortunately the above code is not very practical and re-usable, and will only work for $n=5$. Instead we would like a function named `factorial(n)`. In this Exercise, write your own function which calculates the factorial.

As output print `factorial(10)`.

```
def factorial(n):

  # Write the rest of the function yourself
  f = 1
  for i in range(1,n+1):
    f =  f * i

  return f

print(factorial(10))
```

    3628800

### Question 2.2.2:

Using the `factorial(n)` function you wrote in the previous question, print all $n!$ for all n from 1 to 20.

Hint: Use another for loop!

```
# Below, write the code to print all n! from n=1 to n=20

for n in range(1, 21):

  print(n, factorial(n))

```

    1 1
    2 2
    3 6
    4 24
    5 120
    6 720
    7 5040
    8 40320
    9 362880
    10 3628800
    11 39916800
    12 479001600
    13 6227020800
    14 87178291200
    15 1307674368000
    16 20922789888000
    17 355687428096000
    18 6402373705728000
    19 121645100408832000
    20 2432902008176640000

## Exercise 2.3: `if` / `else` statements

`if` and `else` statments can are used to make decisions based on defined criteria.

```
n = 10

if n < 10:
  print("n is less than 10")
else:
  print("n is greater than 10")

```

### Question 2.3.1:

One example of mathematical functions that contain such statements are the so-called "rectifier linear unit" (ReLU) functions which are often used in Neural Networks.

An example is given here:
\begin{equation}
f\left(x\right) =
\begin{cases}
      0 & x\leq 0 \\
      x & x \gt 0
   \end{cases}
\end{equation}

In this question, write the above ReLU function, which returns $0$ if $x \leq 0$ and returns $x$ otherwise.

Lastly, verify the correctness of your code your code using the 5 print statments in the code block below.

```
def relu(x):

  # Implement the content of the ReLU function yourself!
  if x <= 0.0:
    return 0.0
  else:
    return x

print(relu(-10))  # should return 0
print(relu(-0.1)) # should return 0
print(relu(0))    # should return 0
print(relu(0.1))  # should return 0.1
print(relu(10))   # should return 10
```

    0.0
    0.0
    0.0
    0.1
    10

## Exercise 2.4: Plotting

In the below example, the value of
 $$y= x^2$$
 is calculated for six values of $x$, and appended to a list.

Next, the points are plottet using the `plt.plot()` function from the `pyplot` library.

The function `plt.plot()` tells matplotlib to draw a lineplot that connects all the pairs of points in two lists.

```
import matplotlib.pyplot as plt

# Some x-values
x = [0.0, 1.0, 2.0, 3.0, 4.0, 5.0]

# Initialize an empty list for the y-values we wish to plot
y = []

for i in range(len(x)):

  # Calculate the square of each x in the list
  value = x[i]**2

  # Add the calculated value to the list "y"
  y.append(value)

# Print the two lists
print(x)
print(y)

# Make a line plot/figure
plt.plot(x, y)

# Actually draw the figure below
plt.show()
```

### Question 2.4.1:

Instead of plotting a line through all points, it is also possible to plot datapoints using a so-called [scatterplot](https://en.wikipedia.org/wiki/Scatter_plot).

For this behavior, you can replace the function `plt.plot()` in the above example with the function `plt.scatter()`.

In this question you are given two lists of numbers below, `a` and `b`.

Use pyplot to draw a *scatterplot* of the two lists.

```
a = [10.0, 8.0, 13.0, 9.0, 11.0, 14.0, 6.0, 4.0, 12.0, 7.0, 5.0]
b = [8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]

# Here, write the code to show a scatterplot for the lists a, b
plt.scatter(a, b)
plt.show()
```

### Question 2.4.2:

In order to give the plot a title and label the axes, insert the three functions in the code in Question 2.4.1:

*    `plt.xlabel()`
*    `plt.ylabel()`
*    `plt.title()`

Make the plot title "Python 2020", label the x-axis "Apples" and the y-axis "Oranges"

```
plt.scatter(a, b)
plt.xlabel("Apples")
plt.ylabel("Oranges")
plt.title("Python 2020")
plt.show()
```

## Exercise 2.5: More plotting (harder)

### Question 2.5.1:
In the example in Problem 2.4, the code for a line plot for $y = x^2$ is shown.

Write your own code to plot $y = \cos(x)$ from 0 to 10.

**Hints:**

*   Import the `np.cos()` function using `import numpy as np`
*   If the figure does not look smooth, how can you make the points on the x-axis closer than 1.0?

```
# First try:
# Note that there are many ways to solve this exercise!

import numpy as np

x = []
y = []

for i in range(11):
  x.append(i)
  y.append(np.cos(i))

# Print the two lists to verify that you have the right numbers
print(x)
print(y)

# Make the figure
plt.plot(x, y)

# Actually draw the figure below
plt.show()
```

```
# Smooth, with more x-values:
# Note that there are many ways to solve this exercise!

import numpy as np

x = []
y = []

for i in range(100):

  # Now there are 100 points between 0 and 10 with 0.1 spacing
  # => smoother curve
  xval = i/10.0
  x.append(xval)

  yval = np.cos(xval)
  y.append(yval)

# Print the two lists to verify that you have the right numbers
print(x)
print(y)

# Make the figure
plt.plot(x, y)

# Actually draw the figure below
plt.show()
```


```python
import sympy as sym
from sympy import Matrix
from sympy import exp
from sympy import re, im, I, E, symbols
from sympy.physics.quantum import TensorProduct
from sympy.physics.quantum.dagger import Dagger
```

```python
z = Matrix([[1, 0], [0, -1]])
y = Matrix([[0, -I], [I, 0]])
identity = Matrix([[1, 0], [0, 1]])
```

```python
gamma = symbols('gamma', real=True)
beta = symbols('beta', real=True)
```

```python
vec = Matrix([exp(-3*I*gamma), exp(-I*gamma), exp(-I*gamma), exp(-I*gamma), exp(-I*gamma), exp(-I*gamma), exp(-I*gamma), exp(-3*I*gamma)]) / sym.sqrt(2**3)
vec_c = sym.conjugate(vec.transpose())
```

```python
main = sym.sin(2*beta)**2 * TensorProduct(y, y, identity) + sym.cos(2*beta)**2 * TensorProduct(z, z, identity) + sym.cos(2*beta)*sym.sin(2*beta) * (TensorProduct(y, z, identity) + TensorProduct(z, y, identity))
main
```

$\displaystyle \left[\begin{matrix}\cos^{2}{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - \sin^{2}{\left(2 \beta \right)} & 0\\0 & \cos^{2}{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - \sin^{2}{\left(2 \beta \right)}\\i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - \cos^{2}{\left(2 \beta \right)} & 0 & \sin^{2}{\left(2 \beta \right)} & 0 & i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0\\0 & i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - \cos^{2}{\left(2 \beta \right)} & 0 & \sin^{2}{\left(2 \beta \right)} & 0 & i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)}\\i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & \sin^{2}{\left(2 \beta \right)} & 0 & - \cos^{2}{\left(2 \beta \right)} & 0 & i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0\\0 & i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & \sin^{2}{\left(2 \beta \right)} & 0 & - \cos^{2}{\left(2 \beta \right)} & 0 & i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)}\\- \sin^{2}{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & \cos^{2}{\left(2 \beta \right)} & 0\\0 & - \sin^{2}{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & - i \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)} & 0 & \cos^{2}{\left(2 \beta \right)}\end{matrix}\right]$

```python
sym.expand(vec_c * main * vec)
```

$\displaystyle \left[\begin{matrix}- \frac{e^{2 i \gamma} \sin^{2}{\left(2 \beta \right)}}{4} - \frac{i e^{2 i \gamma} \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)}}{2} + \frac{\sin^{2}{\left(2 \beta \right)}}{2} - \frac{e^{- 2 i \gamma} \sin^{2}{\left(2 \beta \right)}}{4} + \frac{i e^{- 2 i \gamma} \sin{\left(2 \beta \right)} \cos{\left(2 \beta \right)}}{2}\end{matrix}\right]$

```python

```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use("ggplot")
from scipy import signal
```

```python
from MyPlots import plot_DFT
```

## **Discrete Fourier Transform**

$X[k] = \sum_{n=0}^{N-1}x[n]e^{\frac{-j2\pi kn}{N}}$, $k = 0, 1, ..., N-1$

- $X[k]:$ É o 'spectrum'.
- $N$: É o número de amostras do sinal.
- $n:$ index discreto no tempo (tempo normalizado, T=1).
- $k:$ index discreto na frequência.
- $\omega_k = (\frac{2\pi k}{N}):$ frequência em $\frac{radianos}{segundos}$.
- $f_k = (\frac{f_s k}{N}):$ frequência em Hz ($f_s$: frequência de amostragem).

Vamos simplificar a notação utilizando a seguinte equação para as exponenciais complexas $s^*_k = e^{\frac{-j(2\pi kn)}{N}}$. Dessa forma podemos escrever a $DFT$ como um produto escalar:

$X[k] = \langle x {,} s_k\rangle = \sum_{n=0}^{N-1}x[n]s^*_k[n]$

---

**Exemplo:**
Suponha o seguinte sinal no tempo $x[n] = [1,-1,1,-1]$, com 4 amostras ($N=4$), isto é, $n=0,1,2,3$. Portante para realizar a transformada precisamos de 4 frequências ($k=0,1,2,3$). Para cada combinação amostra frequência calculamos uma exponencial complexa, da seguinte forma:

\begin{align}
    s^*_0 &= cos(\frac{2\pi 0n}{4}) - jsin(\frac{2\pi 0n}{4}) = [1,1,1,1] \\
    s^*_1 &= cos(\frac{2\pi 1n}{4}) - jsin(\frac{2\pi 1n}{4}) = [1,-j,-1,j] \\
    s^*_2 &= cos(\frac{2\pi 2n}{4}) - jsin(\frac{2\pi 2n}{4}) = [1,-1,1,-1] \\
    s^*_3 &= cos(\frac{2\pi 3n}{4}) - jsin(\frac{2\pi 3n}{4}) = [1,j,-1,-j]
\end{align}

<span style='color:purple'> <b>OBS: </span> A variável $n$ é o indice, **não** é o valor referente a $x[n]$.

Agora basta calcularmos os produtos escalares para cada frequência:

\begin{align}
\langle x {,} s^*_0\rangle &= 0 \\
\langle x {,} s^*_1\rangle &= 0 \\
\langle x {,} s^*_2\rangle &= 4 \\
\langle x {,} s^*_3\rangle &= 0
\end{align}

No índice igual a dois temos uma componente de magnitude quatro. A frequência correspondente a esse coeficiente dependerá da taxa de amostragem do sinal e pode ser encontrada com a seguinte fórmula:

$$
freq = \frac{indice*f_s}{N}
$$

---

```python
#Parêmetros iniciais
N  = 200 # Samples
f0 = 20  #Frequência Fundamental

ti = 0
tf = 1 # PRECISA ESTAR NORMALIZADO
dt = (tf-ti)/N

t = np.arange(ti,tf,dt)

# Sinal de input
# x_exp  = np.exp(1j*2*np.pi*f0 * t)
# x_sin  = np.sin(2*np.pi*f0    * t)
x_cos  = np.cos(2*np.pi*f0 * t)

#Fourier Transform
X_k = []

ns = np.arange(-N/2, N/2)
ks = np.arange(-N/2, N/2)

for k in ks:
    s = np.exp(-1j*2*np.pi*k/N * ns)
    X_k.append(sum(x_cos*s))

#Plotting figure
fig, axs = plt.subplots(1,2,figsize = (20,6))

# Mapping k to Hz
freqs = (np.array(ks)*(1/dt))/N

axs[0].stem(freqs, np.real(X_k))
axs[1].stem(freqs, np.unwrap(np.angle(X_k,deg=True),discont=180))

plt.suptitle(f"N: {round(N,2)}; fs: {round(1/dt,2)}; f0: {round(f0,2)}\t" + r"$(\frac{%.2f}{%.2f}\times%.2f)$" % (N,1/dt,f0))
plt.show()
```

## **Inverse Discrete Fourier Transform**

$x[n] = \frac{1}{N}\sum_{k=0}^{N-1}X[k]e^{\frac{j 2\pi kn}{N}}$, $n = 0,1, ... , N-1$

```python
#Input Signal
N  = 200 # Samples
f0 = 5  #Frequência Fundamental

x_cos  = np.cos(2 * np.pi * f0 /N * np.arange(N))

# X_k é do último gráfico
X_k = plot_DFT(x_cos, plot=False)

x_n = []
ns  = np.arange(-N/2,N/2)
ks  = np.arange(-N/2,N/2)

for n in ns:
    s_k = np.exp(1j * 2 * np.pi * n/N * ks)
    x_n.append(1.0/N * sum(X_k*s_k))

fig = plt.figure(figsize=(20,4))
plt.plot(ks, np.real(x_n))
plt.axis([-N/2, N/2-1,-1.1,1.1])
plt.show()
```


```python
from math import e, pi

```

```python
from sympy import *
```

```python
x = Symbol('x')
y = Symbol('y')
z = Symbol('z')

f = pow(e,x**2 + y**2)
f
```

$\displaystyle 2.71828182845905^{x^{2} + y^{2}}$

```python
def triple_integral(a,b,c,d,q,r,n,m,p):

    h = (b-a)/n
    k = (d-c)/m
    l = (r-q)/p

    print(f'h3 = {l}')
    print(f'h2 = {k}')
    print(f'h1 = {h}')

    sumPairK = 0
    sumPairH = 0
    sumPairL = 0
    sumNotPairK = 0
    sumNotPairH = 0
    sumNotPairL = 0

    for i in range(1,(p//2)+1):
        sumNotPairL += f.subs(z,q + (2*i - 1)*l)

    for i in range(1,(p//2)):
        sumPairL += f.subs(z,q + 2*i*l)

    #dz
    g = (l/3)*(f.subs(z,q) + 4*sumNotPairL + 2*sumPairL + f.subs(z,r))

    for i in range(1,(m//2)+1):
        sumNotPairK += g.subs(y, c + (2*i - 1)*k)

    for i in range(1,(m//2)):
        sumPairK += g.subs(y,c + 2*i*k)

    #dy
    u = (k/3)*(g.subs(y,c) + 4*sumNotPairK + 2*sumPairK + g.subs(y,d))

    for i in range(1,(n//2)+1):
        sumNotPairH += u.subs(x,a + (2*i - 1)*h)

    for i in range(1,(n//2)):
        sumPairH += u.subs(x,a + 2*i*h)

    #dx
    I = (h/3)*(u.subs(x,a) + 4*sumNotPairH + 2*sumPairH + u.subs(x,b))

    print(f'Respuesta: {I.evalf()}')
```

```python
triple_integral(0,1,0,1,-x*y,x*y,6,6,6)
```

    h3 = x*y/3
    h2 = 0.16666666666666666
    h1 = 0.16666666666666666
    Respuesta: 1.47759679468811

```python

```