# Calculating Times of Rise, Set, and Culmination

Suppose we want to calculate when a given celestial object rises above the horizon, sets below the horizon, or reaches the highest point above the horizon (*culminates*), as seen by an observer at a given location on the surface of the Earth.

### Azimuth and altitude

Let us consider how this observer can describe the apparent location of the celestial object, say a star, at a specific moment in time using a pair of angles.  Let the *azimuth* $A$ be the angle measured westward from due south along the horizon needed to face the star. Let the *altitude* $h$ be the angle of the star above the horizon. (If the star is below the horizon, $h < 0$.)

We define a pair of co-rotating Cartesian coordinate systems for such an observer, called *horizontal* coordinates and *local equatorial* coordinates.  In both systems we treat the entire sky as a sphere of unit radius centered on the observer.

### Horizontal coordinates

In the horizontal coordinate system we have three directions $x$, $y$, and $z$ such that
- $x$ points due south toward the horizon,
- $y$ points due west toward the horizon, and
- $z$ points straight up toward the zenith.

Thus it is possible to relate the angles $A$ and $h$ with horizontal Cartesian coordinates $x$, $y$, and $z$ by

\begin{align}
x & = \cos{h} \cos{A} \\
y & = \cos{h} \sin{A} \tag{1} \\
z & = \sin{h}
\end{align}

To understand these equations a little more intuitively, consider a star that is exactly on the horizon and due west. Then $h=0$, which means $\cos{h}=1$ and $\sin{h}=0$.  Also, $A=90^{\circ}$, so $\cos{A}=0$ and $\sin{A}=1$. This results in $x=0$, $y=1$, $z=0$, which is consistent with the above descriptions of the three coordinates. Another example is where $h=90^{\circ}$ and $A$ has any value, resulting in the vertical vector $x=0$, $y=0$, $z=1$.

### Declination and hour angle

Usually in astronomy when you see the phrase "equatorial coordinates," it refers to right ascension $\alpha$ and declination $\delta$. We need those two quantities for this discussion, but they are not the same as local equatorial coordinates.  Here I will refer to right ascension and declination as "celestial equatorial coordinates" to avoid ambiguity.

Like their celestial counterparts, local equatorial coordinates are defined with respect to the plane of the Earth's equator. But being local means they rotate along with the observer, while celestial equatorial coordinates are fixed to the starry background of the celestial sphere.  In other words, an earthbound observer experiences stationary local equatorial coordinates, but sees stars rotating around him. Meanwhile, an observer far out in space sees fixed celestial equatorial coordinates, but sees a rotating Earth.

*Declination* $\delta$ is the angle of a star north ($\delta>0$) or south ($\delta<0$) of the celestial equator. For example, the north celestial pole has $\delta=90^{\circ}$.

I will describe right ascension later.

*Hour angle* $\tau$ is the angle from the local meridian westward toward a given star. It is measured around a celestial circle having same declination as the star.  It represents the amount of time that has passed since the star last crossed the meridian, or equivalently, since the star culminated (reached its highest point above the horizon).  Hour angle can be measured in degrees (0 to 360) or in sidereal hours (0 to 24). You can divide degrees by 15 to get sidereal hours.

### Local equatorial coordinates

Specifically, local equatorial coordinates are Cartesian coordinates $\hat{x}$, $\hat{y}$, $\hat{z}$ such that

- $\hat{x}$ points to where the celestial equator intersects the meridian,
- $\hat{y}$ points due west toward the horizon, and
- $\hat{z}$ points to the north celestial pole.

Using $\delta$ as the star's declination and $\tau$ as the star's hour angle at a given time, we can write the local equatorial coordinates of the star as

\begin{align}
\hat{x} & = \cos{\delta} \cos{\tau}  \\
\hat{y} & = \cos{\delta} \sin{\tau} \tag{2} \\
\hat{z} & = \sin{\delta} \nonumber
\end{align}

Note that $\tau$ changes with the Earth's rotation but $\delta$ does not. Therefore, for a fixed star, $\hat{x}$ and $\hat{y}$ change with time but $\hat{z}$ remains constant.

### Relating the two local systems

We can write a system of equations that relates these two local rotating coordinate systems.

First note that the $y$-axis is the same as the $\hat{y}$-axis; they are both aimed at the point on the horizon due west from the observer.

The $\hat{z}$-axis, pointing toward the north celestial pole, has an angle $\phi$ above the due-north point on the horizon, where $\phi$ is the geographic latitude of the observer. (If the observer is in the southern hemisphere, then $\phi$ is negative and the north celestial pole is below the horizon.)  That northernmost point on the horizon is in the $-x$ direction, because $x$ points due south.  Similarly, the $\hat{x}$-axis is rotated by $\phi$ from the vertical $z$-axis.  In general, $x$ and $z$ each depend on both $\hat{x}$ and $\hat{z}$ through a rotation of $\phi$.

We can express horizontal coordinates in terms of local equatorial coordinates as

\begin{align}
x & = \hat{x} \sin{\phi} - \hat{z} \cos{\phi} \\
y & = \hat{y} \tag{3} \\
z & = \hat{x} \cos{\phi} + \hat{z} \sin{\phi}
\end{align}

### Relating (azimuth, altitude) to (hour angle, declination)

Substituting equations (1) and (2) into (3), we can write a new system of equations that relate azimuth, altitude, hour angle, and declination:

\begin{align}
\cos{h} \cos{A} &= \cos{\delta} \cos{\tau} \sin{\phi} - \sin{\delta} \cos{\phi} \\
\cos{h} \sin{A} &= \cos{\delta} \sin{\tau} \tag{4} \\
\sin{h} &= \cos{\delta} \cos{\tau} \cos{\phi} + \sin{\delta} \sin{\phi}
\end{align}

The third equation in (4) is most helpful for finding rise and set times because it eliminates the azimuth as a factor.

### Calculating hour angle for a given star at a given time

To calculate hour angle $\tau$ we need to use

\begin{equation}
\tau = \theta + \frac{\lambda}{15} - \alpha \tag{5}
\end{equation}

where
- $\theta$ is the current Greenwich Apparent Sidereal Time (GAST) at the moment of observation,
- $\lambda$ is the geographic longitude of the observer east of Greenwich in degrees, and
- $\alpha$ is the right ascension of the star.

Note that $\tau$, $\theta$, and $\alpha$ are all expressed in sidereal hours. We divide $\lambda$ by 15 to convert degrees to hours.

### Determining when the object is highest or lowest in the sky

The object reaches maximum altitude angle $h_1$ when its hour angle $\tau=0$ and its minimum value of $h_2$ when $\tau = 12$ hours. Let's look at the general case where we are trying to find the *event time* when the object reaches an arbitrary hour angle $\tau$:

\begin{align}
\theta + \frac{\lambda}{15} - \alpha & = \tau \\
\theta = \tau + \alpha - \frac{\lambda}{15} \tag{6}
\end{align}

A star's right ascension $\alpha$ is almost constant over time, and we assume the observer's longitude $\lambda$ is fixed.  In practice, $\alpha$ for the Sun, the Moon, or a planet is not constant. The Moon especially will move in right ascension significantly over the course of a few hours. So we will end up needing to numerically iterate to find the exact event time.

The approach I use in the Astronomy library is to determine the GAST at a start search time. This start search time represents a time after which the next event is to be found. This initial guess $\theta_{x}$ of the GAST value will be incorrect by the approximate amount

\begin{align}
\Delta \theta = \left( \tau + \alpha - \frac{\lambda}{15} \right) - \theta_x \tag{7}
\end{align}

On the first iteration, if $\Delta\theta$ is negative, it is corrected by adding 24 sidereal hours to guarantee finding a culmination time after the start search time. On subsequent iterations, the correction may be positive or negative to home in on the correct time.

At this point, it is important to note that sidereal hours are not exactly the same as clock hours.  The Earth rotates with respect to a fixed star once every 23 hours, 56 minutes, and 4.091 seconds, or 86164.091 seconds. This is compared to a mean solar day of 24 hours = 86400 seconds. This gives a ratio of $\rho = 0.99726957$ mean solar days per sidereal day.

After each iteration, we correct by adding $\rho \Delta \theta$ days of terrestrial time and calculating the right ascension $\alpha$ and declination $\delta$ of the object, and GAST, again at the new time. We plug the updated values of $\theta_x$ and $\alpha$ back into (7) to obtain a more accurate estimate. We keep iterating until $\Delta \theta$ is tolerably small (less than 0.1 seconds). This typically takes 4 or 5 iterations.

### Calculating rise and set times

Now that we can determine when an object reaches its highest and lowest altitudes in the sky for a given observer, we know that *if* the object rises or sets, it will be bounded by those two events. At locations close to one of the Earth's poles, an object may stay in the sky for weeks at a time, never setting, or it may stay below the horizon for weeks at a time, never rising.

Specifically, if the object is at its lowest at time $t_L$ and at its highest at $t_H \approx t_L + 12$ hours, then we can calculate the object's altitudes $h_L$ and $h_H$ at those respective times.  If $h_L \lt 0$ and $h_H \gt 0$, we know the object must have risen at some time $t_R$ such that $t_L \lt t_R \lt t_H$.


## Resistance Temperature Detectors (RTD)

All RTDs are based on the change in resistance due to the temperature coefficient of resistance (TCR) of the metal being used. The resistance of a conductor of length $L$ with constant cross-sectional area $S$ and conductivity $\sigma$ (Figure 3.1) is

$$ R = \frac{L}{\sigma s} $$

The conductivity of the material itself is temperature dependent and given as

\begin{align}
\sigma = \frac{\sigma_0}{1+\alpha\left[T-T_0\right]}
\end{align}

Equation (1) stands for ...

```python

```


```python
from sympy.vector import curl, divergence, gradient, CoordSys3D
```

```python
transformation  = 'cartesian'
var_names = ['x', 'y', 'z']
vec_names = ['e_' + v for v in var_names]
R = CoordSys3D('', transformation=transformation, variable_names=var_names, vector_names=vec_names)
```

```python
%run -i define_symbols_beta.py
```

    1.1.2.dev

# Coriolis term for horizontal velocity ($w \to 0 $)

Vertical component of Earth's rotation vector

```python
from sympy.abc import beta
f = (f1 + beta * R.y) * R.k
f
```

```python
vort = curl(u)
vort_r = vort.dot(R.k)
div = divergence(u_h).doit()
```

```python
corio = -f.cross(u)
```

```python
corio
```

## Coriolis term in vorticity equation

$\nabla \times (-f \hat{e}_z \times \bf{u})$ is given by

```python
curl(corio).dot(R.k).expand()
```

Divergence in cartesian coordinates is given by

```python
div.expand()
```

$u_3 = 0$ in the above equation because it is the 2D divergence --- which can be separated from the expression for $\nabla \times (-f \hat{e}_z \times \bf{u})$

```python
(- f * div).expand()  - beta * u2
```

Therefore linearized vorticity equation can be written as (probably wrong):
$$\partial_t \zeta = -\beta v - (f_0 + \beta y) \nabla . \mathbf{u}$$

## Coriolis term in divergence equation

$\nabla . (-f \hat{e}_r \times \bf{u})$ is given by

```python
divergence(corio).expand()
```

Vertical component of vorticity (in the $\hat{e}_r$ direction), $\zeta$ in spherical coordinates is given by:

```python
vort_r.doit()
```

```python
-beta * u1 + (f * vort_r).expand().doit()
```

Therefore linearized divergence equation can be written as:
$$\partial_t \nabla.\mathbf{u} = -\beta u + (f_0 +\beta y)\zeta - c^2 \nabla^2 \eta $$

## Linearized equations in vorticity-divergence formulation $(\zeta, \delta, \eta)$

$$
\begin{align}
\partial_t \zeta &= -\beta v - f\delta \\
\partial_t \delta &= -\beta u + f\zeta - c^2 \nabla^2 \eta \\
\partial_t \eta &= -\delta
\end{align}
$$

## Linearized equations in stream function-potential formulation $(\Psi, \Phi, \eta)$

Let stream function and velocity potential be defined as:

$$ \mathbf u = - \nabla \times (\Psi \hat{e}_z) + \nabla \Phi $$

which implies the velocities are given by:

$$u = -\Psi_y + \Phi_x$$
$$v = \Psi_x + \Phi_y$$

and also implies vorticity and divergence to be represented as:
$$\zeta = \nabla^2 \Psi$$
$$\delta = \nabla^2 \Phi$$

$$
\begin{align}
\partial_t \nabla^2 \Psi &= -\beta(\Psi_x + \Phi_y)  - f \nabla^2 \Phi \\
\partial_t \nabla^2 \Phi &= \beta(-\Psi_y + \Phi_x) + f\nabla^2 \Psi  - c^2 \nabla^2 \eta \\
\partial_t \eta &= -\nabla^2 \Phi
\end{align}
$$

**The eigenvalue problem and normal modes are solved in `normalmode_nb-beta.ipynb`**


```python
from epipack import SymbolicEpiModel, DeterministicEpiModel, StochasticEpiModel
import sympy as sy
%matplotlib notebook
```

```python
%matplotlib notebook
```

```python
S, I, R, eta, rho, omega = sy.symbols("S I R eta rho omega")
```

```python
SIRS = SymbolicEpiModel([S,I,R])

SIRS.set_processes([
    #### transmission process ####
    # S + I (eta)-> I + I
    (S, I, eta, I, I),

    #### transition processes ####
    # I (rho)-> R
    # R (omega)-> S
    (I, rho, R),
    (R, omega, S),

])
```

    <epipack.symbolic_epi_models.SymbolicEpiModel at 0x119a67ad0>

```python
SIRS.ODEs()
```

    [Eq(Derivative(S, t), -I*S*eta + R*omega),
     Eq(Derivative(I, t), I*(S*eta - rho)),
     Eq(Derivative(R, t), I*rho - R*omega)]

```python
SIRS.find_fixed_points()
```

$\displaystyle \left\{\left( S, \  0, \  0\right), \left( \frac{\rho}{\eta}, \  \frac{R \omega}{\rho}, \  R\right)\right\}$

```python
SIRS.jacobian()
```

$\displaystyle \left[\begin{matrix}- I \eta & - S \eta & \omega\\I \eta & S \eta - \rho & 0\\0 & \rho & - \omega\end{matrix}\right]$

```python
SIRS.get_eigenvalues_at_disease_free_state()
```

    {-omega: 1, eta - rho: 1, 0: 1}

```python
S, I, R = list("SIR")
R0 = 2.5
rho = recovery_rate = 1 # let's say 1/days
eta = infection_rate = R0 * recovery_rate
omega = 1/14 # in units of 1/days

SIRS = DeterministicEpiModel([S,I,R])

SIRS.set_processes([
    #### transmission process ####
    # S + I (eta)-> I + I
    (S, I, eta, I, I),

    #### transition processes ####
    # I (rho)-> R
    # R (omega)-> S
    (I, rho, R),
    (R, omega, S),

])

SIRS.set_initial_conditions({S:1-0.01, I:0.01})
```

    <epipack.deterministic_epi_models.DeterministicEpiModel at 0x119f62710>

```python
t = np.linspace(0,40,1000)
result = SIRS.integrate(t)
```

```python
import matplotlib.pyplot as pl
pl.figure()
for compartment, timeseries in result.items():
    pl.plot(t, timeseries, label=compartment)
pl.xlabel(r"time $t\times\rho$")
pl.ylabel("incidence")
pl.ylim([0,1])
pl.legend()
pl.gcf().tight_layout()
```

    <IPython.core.display.Javascript object>

```python
import networkx as nx

k0 = 50
eta = R0 * rho / k0
N = int(1e4)
edges = [ (e[0], e[1], 1.0) for e in nx.fast_gnp_random_graph(N,k0/(N-1)).edges() ]

SIRS = StochasticEpiModel([S,I,R],N,edge_weight_tuples=edges)

SIRS.set_link_transmission_processes([
    #### transmission process ####
    # I + S (eta)-> I + I
    (I, S, eta, I, I),
])

SIRS.set_node_transition_processes([
    #### transition processes ####
    # I (rho)-> R
    # R (omega)-> S
    (I, rho, R),
    (R, omega, S),

])

SIRS.set_random_initial_conditions({S:N-int(1e-2*N), I:int(1e-2*N)})
```

    <epipack.stochastic_epi_models.StochasticEpiModel at 0x11c174e50>

```python
t_s, result_s = SIRS.simulate(40)
```

```python
pl.figure()

for compartment, timeseries in result.items():
    #pl.plot(t, timeseries, label=compartment)
    pl.plot(t_s, result_s[compartment]/N, label=compartment+" (sim.)")
pl.xlabel(r"time $t\times\rho$")
pl.ylabel("incidence")
pl.ylim([0,1])
pl.legend()
pl.gcf().tight_layout()
pl.show()
```

    <IPython.core.display.Javascript object>


Mathematical Optimization for Engineers

Lab 12 - Direct single shooting

In this exercise, we want to solve the van der Pol oscillator optimal control problem using the direct single shooting approach.
The problem without state constraints is as follows:
$\begin{align}
\min_{u(t)} \int_{t_{0}}^{t_{f}} x_1^2(t) + x_2^{2}(t) + u(t)^2 dt\\
s.t.  \dot{x} = f(x(t),u(t))\\
x(t_{0}) = x_{0} \\
u_{lb} \leq u(t) \leq u_{ub}\\
\end{align}
$

For direct single shooting, the differential equations are solved by an integration scheme, which is needed to be set-up.

```python
import numpy as np

from scipy.integrate import odeint, solve_ivp

from scipy.optimize import fmin,fmin_bfgs, Bounds, minimize, NonlinearConstraint

from matplotlib import pyplot as plt
from matplotlib import animation
```

```python
def eval_rhs(t, x, u, p):

    # your code

    return f
```

```python
def objective_ode_integration(p, func, x0, finalTime, output_state,
                              is_path_constraint, doPlot):
    # inputs are as follows
    # p = parameter vector of control discretization
    # func: right-hand side of ODE f(t, x, u, p)
    # x0 = initial condition
    # finalTime = tN
    # doPlot = boolean variable, if true, plot solution

    N = len(p) # Number of discretization intervals
    h = finalTime /N # length of a single interval
    n_x = len(x0)  # number of states
    x0_cur = x0
    y_all = np.array([x0]).transpose()
    t_all = np.array([0])
    output = list()

    for i in range(N): # loop over discretization intervals

        # solve initial value problem
        # your code

        # update initial conditions
        # your code

        y_all = np.concatenate((y_all, sol.y), axis=1)
        t_all = np.concatenate((t_all, sol.t), axis=0)
        output.append(y[output_state, -1])

    if doPlot:
        y_all = np.array(y_all)
        plt.subplot(221)
        plt.plot(t_all, y_all[0, :])
        plt.title('x1')

        plt.subplot(222)
        plt.plot(t_all, y_all[1, :])
        plt.title('x2')

        plt.subplot(223)
        plt.plot(t_all, y_all[2, :])
        plt.title('x3')
        #
        plt.subplot(224)
        plt.plot(np.linspace(0, finalTime, N), p)
        plt.title('u')

        plt.tight_layout()
        plt.show()

    if not is_path_constraint:
        output = output[-1]
    else:
        # constraint hard coded here to allow for
        # constraint definition with arguments
        # in scipy, c(x) >= 0
        # your code here

    return output
```

```python
objective_ode_integration([1.0, 0.5, 0.2, 0.4], eval_rhs, [0, 1, 0], 5, [], [], True)
```

```python
finalTime = 5
N = 25 # Number of discretization intervals for the control
doPlot = False
p0 = 1.0*np.ones(N) # initial guess for optimization
x0 = [0, 1, 0]

bnds = Bounds(-0.3, 1.0)

# define objective
output_state = 2  # for objective
is_path_constraint = False
# call optimization with arguments
# 'eval_rhs, x0, finalTime, output_state, is_path_constraint, doPlot'
# your code here

# call ODE integration to plot trajectories with optimal solution of controls
# your code here

```

## Adding state path constraint
After solvindg the problem without state constraints, we want to include the path constraint on the state $x_1$:
$ \begin{align}
x_1(t) \geq - 0.4 \forall t \in [t_{0},t_{f}]
\end{align}$
For this task, we must add this constraint to the optimization problem. One way is to define the constraint function as constraint dictionary in Python.

```python
# in scipy, the inequality constraints are defined as
# c(x) >= 0
cons = ({'type': 'ineq',
       'fun': objective_ode_integration,
       'args': (eval_rhs, x0, finalTime, 0, True, False)
       })

output_state = 2  # for objective
is_path_constraint = False

# call optimization with arguments
# 'eval_rhs, x0, finalTime, output_state, is_path_constraint, doPlot'
# your code here

# call ODE integration to plot trajectories with optimal solution of controls
# your code here
```

## What could be made more efficient for this optimization?

```python

```


# Fundamental geometric operations for B-Splines

```python
# needed imports
import numpy as np
import matplotlib.pyplot as plt
```

Having more control on a curve, adding new control points, can be done in two different ways:

1. insert new knots

2. elevate the polynomial degree

In general, one way need to use both approaches.
For each strategy, we shall consider one algorithm.

**Remark**:
In this notebook we only use B-Splines curves, but similar functions are also available for NURBS.

## Knot insertion

Assuming an initial B-Spline curve defined by:

* its degree $p$
* knot vector $T=(t_i)_{0\leqslant i \leqslant n + p + 1}$
* control points $(\mathbf{P}_i)_{ 0 \leqslant i \leqslant n}$

We are interested in the new B-Spline curve as the result of the insertion of the knot $t$, $m$ times (with a span $j$, i.e. $t_j \leqslant t < t_{j+1}$).

After such operation, the degree is remain unchanged, while the knot vector is enriched by $t$, $m$ times. The aim is then to compute the new control points $(\mathbf{Q}_i)_{ 0 \leqslant i \leqslant n + m}$

For this purpose we use the DeBoor algorithm:

\begin{align}
   n &:= n+m
   \\
   p &:= p
   \\
   T &:= \{ t_0,.., t_j, \underbrace{t, ..., t}_{m}, t_{j+1},.., t_{n+k}\}
   \\
   \textbf{Q}_i &:= \textbf{Q}_i^{m}
\end{align}

where
\begin{align}
   \textbf{Q}_i^0 &= \textbf{P}_i
   \\
   \textbf{Q}_i^r &= \alpha_i^r \textbf{Q}_i^{r-1} + (1-\alpha_i^r) \textbf{Q}_{i-1}^{r-1}
\end{align}
with
\begin{align}
   \alpha_i^r &=
      \left\{\begin{array}{cc}
         1 &  i \leqslant j-p+r-1 \\
         \frac{t-t_i}{t_{i+p-r+1}-t_i} & j-p+r \leqslant i \leqslant j-m \\
         0 & j-m+1 \leqslant i
      \end{array}\right.
\end{align}

The following python function implements the previous algorithm.

```python
# import the knot insertion function
from bsplines_utilities import insert_knot_bspline_curve
```

From now on, we shall also use the following function to plot curves and their control points:

```python
from bsplines_utilities import plot_curve
```

**Examples**

```python
def example_1(times=1):
    knots  = [0., 0., 0., 0., 1., 1., 1., 1.]
    degree = 3
    n      = len(knots) - degree - 1

    P = np.zeros((n, 2))
    P[:, 0] = [1., 1., 0., -1.]
    P[:, 1] = [0., 1., 1.,  0.]
    plot_curve(knots, degree, P, with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{P}_{' + str(i) + '}$')

    t = 0.5
    knots, degree, P = insert_knot_bspline_curve(knots, degree, P, t, times=times)
    plot_curve(knots, degree, P, color='r', label='Q', with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    n = len(knots) - degree - 1
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{Q}_{' + str(i) + '}$')

    # plot endpoints
    for i in [0, n-1]:
        x = P[i,0] ; y = P[i,1]
        if i == 0:
            Pi = '\mathbf{P}_{' + str(i) + '}'
        elif i == n-1:
            Pi = '\mathbf{P}_{' + str(i-1) + '}'

        Qi = '\mathbf{Q}_{' + str(i) + '}'
        text = '${Pi} = {Qi}$'.format(Pi=Pi, Qi=Qi)
        plt.text(x+0.05,y+0.05,text)
```

```python
example_1(times=1)
```

```python
example_1(times=2)
```

```python
example_1(times=3)
```

```python
def example_2a(nt = 3):
    knots  = [0., 0., 0., 0., 1., 1., 1., 1.]
    degree = 3
    n      = len(knots) - degree - 1

    P = np.zeros((n, 2))
    P[:, 0] = [1., 1., 0., -1.]
    P[:, 1] = [0., 1., 1.,  0.]
    plot_curve(knots, degree, P, with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{P}_{' + str(i) + '}$')

    ts = np.linspace(0., 1., nt+2)[1:-1]
    for t in ts:
        knots, degree, P = insert_knot_bspline_curve(knots, degree, P, t, times=1)

    plot_curve(knots, degree, P, color='r', label='Q', with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    n = len(knots) - degree - 1
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{Q}_{' + str(i) + '}$')

    # plot endpoints
    for i in [0, n-1]:
        x = P[i,0] ; y = P[i,1]
        if i == 0:
            Pi = '\mathbf{P}_{' + str(i) + '}'
        elif i == n-1:
            Pi = '\mathbf{P}_{' + str(i-1) + '}'

        Qi = '\mathbf{Q}_{' + str(i) + '}'
        text = '${Pi} = {Qi}$'.format(Pi=Pi, Qi=Qi)
        plt.text(x+0.05,y+0.05,text)
```

```python
# curve after inserting [0.25 0.5  0.75]
example_2a(nt=3)
```

```python
# curve after inserting [0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9]
example_2a(nt=9)
```

## Order elevation

There are also many algorithms for degree elevation of a B-Spline curve. In the sequel, we will be using the one developped by Huand, Hu and Martin (2005).

The previous algorithm is implemented by the following function

```python
# import the degree elevation function
from bsplines_utilities import elevate_degree_bspline_curve
```

**Example 1.**

```python
def example_1(m=1):
    knots  = [0., 0., 0., 0., 0.5, 1., 1., 1., 1.]
    degree = 3
    n      = len(knots) - degree - 1

    P = np.zeros((n, 2))
    P[:, 0] = [1., 1., .5, -.5, -1.]
    P[:, 1] = [0., .5, 1.,  .5,  0.]

    plot_curve(knots, degree, P, with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{P}_{' + str(i) + '}$')

    knots, degree, P = elevate_degree_bspline_curve(knots, degree, P, m=1)
    plot_curve(knots, degree, P, color='r', label='Q', with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    n = len(knots) - degree - 1
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{Q}_{' + str(i) + '}$')

    # plot endpoints
    for i in [0, n-1]:
        x = P[i,0] ; y = P[i,1]
        if i == 0:
            Pi = '\mathbf{P}_{' + str(i) + '}'
        elif i == n-1:
            Pi = '\mathbf{P}_{' + str(i-1) + '}'

        Qi = '\mathbf{Q}_{' + str(i) + '}'
        text = '${Pi} = {Qi}$'.format(Pi=Pi, Qi=Qi)
        plt.text(x+0.05,y+0.05,text)

example_1()
```

**Example 2.**

```python
def example_2(m=1):
    knots  = [0., 0., 0., 0., 0.5, 0.5, 1., 1., 1., 1.]
    degree = 3
    n      = len(knots) - degree - 1
    n_initial = n

    P = np.zeros((n, 2))
    P[:, 0] = [260., 100., 260., 420., 580., 420.]
    P[:, 1] = [100., 260., 420., 420., 260., 100.]
    plot_curve(knots, degree, P, with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{P}_{' + str(i) + '}$')

    knots, degree, P = elevate_degree_bspline_curve(knots, degree, P, m=m)
    plot_curve(knots, degree, P, color='r', label='Q', with_ctrl_pts=False)

    # we don't plot the first and last points
    # they will be plotted later
    n = len(knots) - degree - 1
    for i in range(1, n-1):
        x,y = P[i,:]
        plt.text(x+0.05,y+0.05,'$\mathbf{Q}_{' + str(i) + '}$')

    # plot endpoints
    for i in [0, n-1]:
        x = P[i,0] ; y = P[i,1]
        if i == 0:
            Pi = '\mathbf{P}_{' + str(i) + '}'
        elif i == n-1:
            Pi = '\mathbf{P}_{' + str(n_initial-1) + '}'

        Qi = '\mathbf{Q}_{' + str(i) + '}'
        text = '${Pi} = {Qi}$'.format(Pi=Pi, Qi=Qi)
        plt.text(x+0.05,y+0.05,text)
```

```python
example_2(m=1)
```

```python
example_2(m=2)
```

```python
example_2(m=4)
```

```python
example_2(m=8)
```

## Translation

Since B-Spline curves are invariant under affine transformations, whan translating a curve with a given displacement, the new curve has the same knot vector while the control points are obtained by applying the same translation to the initial ones.

The following function implements this utility

```python
# import the translation of a B-Spline curve
from bsplines_utilities import translate_bspline_curve
```

```python
def example_1():
    knots  = [0., 0., 0., 0., 1., 1., 1., 1.]
    degree = 3
    n      = len(knots) - degree - 1

    P = np.zeros((n, 2))
    P[:, 0] = [1., 1., 0., -1.]
    P[:, 1] = [0., 1., 1.,  0.]
    plot_curve(knots, degree, P)

    knots, P = translate_bspline_curve(knots, P, [1., 1.5])
    plot_curve(knots, degree, P, color='r', label='Q')

example_1()
```

## Rotation

Rotating a B-Spline curve with an angle $\theta$ is done by applying it on the control points, as for the translation algorithm.

The following function implements this utility.

```python
# import the rotation of a B-Spline curve
from bsplines_utilities import rotate_bspline_curve
```

```python
def example_1():
    knots  = [0., 0., 0., 0., 1., 1., 1., 1.]
    degree = 3
    n      = len(knots) - degree - 1

    P = np.zeros((n, 2))
    P[:, 0] = [1., 1., 0., -1.]
    P[:, 1] = [0., 1., 1.,  0.]
    plot_curve(knots, degree, P)

    knots, P = rotate_bspline_curve(knots, P, np.pi/3., center=[-1.,0.])
    plot_curve(knots, degree, P, color='r', label='Q')

example_1()
```

## Homothetic transformation

The following function implements a hmothetic transformation for B-Spline curves.

```python
# import the homothetic transformation of a B-Spline curve
from bsplines_utilities import homothetic_bspline_curve
```

```python
def example_1():
    knots  = [0., 0., 0., 0., 1., 1., 1., 1.]
    degree = 3
    n      = len(knots) - degree - 1

    P = np.zeros((n, 2))
    P[:, 0] = [1., 1., 0., -1.]
    P[:, 1] = [0., 1., 1.,  0.]
    plot_curve(knots, degree, P)

    knots, P = homothetic_bspline_curve(knots, P, 0.5, center=[0.,0.])
    plot_curve(knots, degree, P, color='r', label='Q')

example_1()
```

```python
# css style
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


# Section 7.1 $\quad$ Eigenvalues and Eigenvectors

Let $L: V\to V$ be a linear transformation. $L$ maps a vector $\mathbf{v}$ in $V$ to another vector $L(\mathbf{v})$ in $V$.

Let $L: V\to V$ be a linear transformation. The number $\lambda$ is called

Note that a  linear transformation can be expressed in terms of a matrix:

>**Definition** Let $A$ be an $n\times n$ matrix.

### Example 1

Find the eigenvalues and the associated eigenvectors of the matrix $A$, where
\begin{equation*}
  A = \left[
        \begin{array}{cc}
          1 & 1 \\
          -2 & 4 \\
        \end{array}
      \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[1, 1], [-2, 4]]);

A.eigenvects()
```

    [(2, 1, [Matrix([
       [1],
       [1]])]), (3, 1, [Matrix([
       [1/2],
       [  1]])])]

>**Definition** Let $A = [a_{ij}]$ be an $n\times n$ matrix.

>**Theorem** Let $A $ be an $n\times n$ matrix.

### Example 2

Let
\begin{equation*}
  A =
  \left[
    \begin{array}{ccc}
      1 & 2 & -1 \\
      1 & 0 & 1 \\
      4 & -4 & 5 \\
    \end{array}
  \right]
\end{equation*}
Find all eigenvalues and associated eigenvectors of $A$.

```python
from sympy import *

A = Matrix([[1, 2, -1], [1, 0, 1], [4, -4, 5]]);

A.eigenvects()
```

    [(1, 1, [Matrix([
       [-1/2],
       [ 1/2],
       [   1]])]), (2, 1, [Matrix([
       [-1/2],
       [ 1/4],
       [   1]])]), (3, 1, [Matrix([
       [-1/4],
       [ 1/4],
       [   1]])])]


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, I. Hawke. Partly based on content by David Ketcheson, also under CC-BY.

# Full phugoid model

This is the third IPython Notebook of the series on the _phugoid model of glider flight_, our first learning module of the course ["Practical Numerical Methods with Python"](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about). In the first notebook, we described the physics of the trajectories known as phugoids obtained from an exchange of potential and kinetic energy in an idealized motion with no drag. We gave you a neat little code to play with and plot various phugoid curves.

In the second notebook, we looked at the equation representing small perturbations on the straight-line phugoid, resulting in simple harmonic motion. This is a second-order ordinary differential equation, and we solved it numerically using **Euler's method**: the simplest numerical method of all. We learned about convergence and calculated the error of the numerical solution, comparing with an analytical solution. That is a good foundation!

Now, let's go back to the dynamical model, and take away the idealization of no-drag. Let's remind ourselves of the forces affecting an aircraft, considering now that it may be accelerating, with an instantaneous upward trajectory. We use the designation $\theta$ for the angle, and consider it positive upwards.

####Figure 1. Forces with a positive trajectory angle.

In Figure 1, $L$ is the lift, $W$ is the weight, $D$ is the drag, and $\theta$ the positive angle of the trajectory, instantaneously.

In lesson 1, we wrote the force balance in the directions perpendicular and parallel to the trajectory for a glider in _equilibrium_. What if the forces are _not_ in balance? Well, there will be acceleration terms in the equations of motion, and we would have in that case:

\begin{align}
m \frac{dv}{dt} & = - W \sin\theta - D \\
m v \, \frac{d\theta}{dt} & = - W \cos\theta + L
\end{align}

We can use a few little tricks to make these equations more pleasing. First, use primes to denote the time derivatives and divide through by the weight:

\begin{align}
 \frac{v'}{g} & = - \sin\theta - D/W \\
\frac{v}{g} \, \theta' & = - \cos\theta + L/W
\end{align}

Recall, from our first lesson, that the ratio of lift to weight is known from the trim conditions—$L/W=v^2/v_t^2$— and also from the definitions of lift and drag,

$$\begin{eqnarray}
L &=& C_L S \times \frac{1}{2} \rho v^2 \\
D &=& C_D S \times \frac{1}{2} \rho v^2
\end{eqnarray}$$

we see that $L/D=C_L/C_D$. The system of equations can be re-written:

\begin{align}
 v' & = - g\, \sin\theta - \frac{C_D}{C_L} \frac{g}{v_t^2} v^2 \\
 \theta' & = - \frac{g}{v}\,\cos\theta + \frac{g}{v_t^2}\, v
\end{align}

It is very interesting that the first equation has the factor $C_D/C_L$, which is the inverse of a measure of the aerodynamic efficiency of the aircraft. It turns out, this is the term that contributes damping to the phugoid model: if drag is zero, there is no damping. Drag is never zero in real life, but as engineers design more aerodynam-ically efficient aircraft, they make the phugoid mode more weakly damped. At altitude, this is nothing but a slight bother, but vertical oscillations are unsafe during final approach to land, so this is something to watch out for!

## The initial value problem

If we want to visualize the flight trajectories predicted by this model, we are going to need to integrate the spatial coordinates, which depend on both the forward velocity and the trajectory angle. The position of the glider on a vertical plane will be designated by coordinates $(x, y)$ with respect to an inertial frame of reference, and are obtained from:

\begin{align}
x'(t) & = v \cos(\theta) \\
y'(t) & = v \sin(\theta).
\end{align}

Augmenting our original two differential equations by the two equations above, we have a system of four first-order differential equations to solve. We will use a time-stepping approach, like in the previous lesson. To do so, we do need *initial values* for every unknown:

$$
v(0) = v_0 \quad \text{and} \quad \theta(0) = \theta_0\\
x(0) = x_0 \quad \text{and} \quad y(0) = y_0
$$

## Solve with Euler's method

We know how to apply Euler's method from the previous lesson. We replace each of the time derivatives by an approximation of the form:

$$v'(t) \approx \frac{v^{n+1} - v^n}{\Delta t},$$

where we are now using a superscript $n$ to indicate the $n$-th value in the time iterations. The first differential equation, for example, gives:

$$\frac{v^{n+1} - v^n}{\Delta t} = - g\, \sin\theta^n - \frac{C_D}{C_L} \frac{g}{v_t^2} (v^n)^2$$

Alright, we know where this is going. At each time iteration $t^n$, we want to evaluate all the known data of our system to obtain the state at $t^{n+1}$—the next time step. We say that we are _stepping in time_ or _time marching_.

The full system of equations discretized with Euler's method is:

\begin{align}
v^{n+1} & = v^n + \Delta t \left(- g\, \sin\theta^n - \frac{C_D}{C_L} \frac{g}{v_t^2} (v^n)^2 \right) \\
\theta^{n+1} & = \theta^n + \Delta t \left(- \frac{g}{v^n}\,\cos\theta^n + \frac{g}{v_t^2}\, v^n \right) \\
x^{n+1} & = x^n + \Delta t \, v^n \cos\theta^n \\
y^{n+1} & = y^n + \Delta t \, v^n \sin\theta^n.
\end{align}

As we've learned before, the system of differential equations can also be written as a vector equation:

$$u'(t) = f(u)$$

where

\begin{align}
u & = \begin{pmatrix} v \\ \theta \\ x \\ y \end{pmatrix} & f(u) & = \begin{pmatrix} - g\, \sin\theta - \frac{C_D}{C_L} \frac{g}{v_t^2} v^2 \\ - \frac{g}{v}\,\cos\theta + \frac{g}{v_t^2}\, v \\ v\cos\theta \\ v\sin\theta \end{pmatrix}.
\end{align}

It's a bit tricky to code the solution using a NumPy array holding all your independent variables. But if you do, a function for the Euler step can be written that takes any number of simultaneous equations. It simply steps in time using the same line of code:

```Python
def euler_step(u, f, dt):
    return u + dt * f(u)
```

This function can take a NumPy array `u` with any number of components. All we need to do is create an appropriate function `f(u)` describing our system of differential equations. Notice how we are passing a _function_ as part of the arguments list to `euler_step()`. Neat!

### And solve!

As always, we start by loading the modules and libraries that we need for this problem. We'll need a few transcendental functions, including the $\log$ for a convergence study later on. And remember: the line `%matplotlib inline` is a magic function that tells Matplotlib to give us the plots in the notebook (the default behavior of Matplotlib is to open a pop-up window). In addition, we are importing the module `rcParams` to define notebook-wide plotting parameters: font family and size. Here we go!

```
from math import sin, cos, log, ceil
import numpy
import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

Next, we need to set things up to start our numerical solution: the parameter
values and the _initial values_. You know what the acceleration of gravity is: 9.8 m/s$^2$, but what are good values for $C_D/C_L$, the inverse of the aerodynamic efficiency? Some possible values are given on a table in the Wikipedia entry for [lift-to-drag ratio](http://en.wikipedia.org/wiki/Lift-to-drag_ratio): a modern sailplane can have $L/D$ of 40 to 60, depending on span (and, in case you're interested, a flying squirrel has $L/D$ close to 2).

For the _trim velocity_, the speed range for typical sailplanes is between 65 and 280 km/hr, according to Wikipedia (it must be right!). Let's convert that to meters per second: 18 to 78 m/s. We'll pick a value somewhere in the middle of this range.

Here's a possible set of parameters for the simulation, but be sure to come back and change some of these, and see what happens!

```
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 30.0   # trim velocity in m s^{-1}
C_D = 1/40.  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = v_t     # start at the trim velocity (or add a delta)
theta0 = 0.0 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 1000.0  # initial altitude
```

We'll define a function `f()` to match the right-hand side of Equation (15), the full differential system in vector form. This function assumes that we have available the parameters defined above. If you re-execute the cell above with different parameter values, you can just run the solution without re-executing the function definition.

```
def f(u):
    """Returns the right-hand side of the phugoid system of equations.

    Parameters
    ----------
    u : array of float
        array containing the solution at time n.

    Returns
    -------
    dudt : array of float
        array containing the RHS given u.
    """

    v = u[0]
    theta = u[1]
    x = u[2]
    y = u[3]
    return numpy.array([-g*sin(theta) - C_D/C_L*g/v_t**2*v**2,
                      -g*cos(theta)/v + g/v_t**2*v,
                      v*cos(theta),
                      v*sin(theta)])
```

Compare the code defining function `f(u)` with the differential equations, and convince yourself that it's right!

\begin{align}
u & = \begin{pmatrix} v \\ \theta \\ x \\ y \end{pmatrix} & f(u) & = \begin{pmatrix} - g\, \sin\theta - \frac{C_D}{C_L} \frac{g}{v_t^2} v^2 \\ - \frac{g}{v}\,\cos\theta + \frac{g}{v_t^2}\, v \\ v\cos\theta \\ v\sin\theta \end{pmatrix} \nonumber
\end{align}

Now, Euler's method is implemented in a simple function `euler_step()`:

```
def euler_step(u, f, dt):
    """Returns the solution at the next time-step using Euler's method.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        approximate solution at the next time step.
    """

    return u + dt * f(u)
```

After definiting a final time for the solution, and the time step $\Delta t$, we can construct the grid in time using the NumPy function [`linspace()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linspace.html). Make sure you study the decisions we made here to build the time grid: why do we add 1 to the definition of `N`, for example?

Look at the code below, and make sure you understand the following aspects of it.

* The NumPy array `u` contains the solution at every time-step,  consisting of the velocity, angle and location of the glider.
* The first element of the array `u` is set to contain the initial conditions.
* In the `for`-loop, the function `euler_step()` is called to get the solution at time-step $n+1$.

```
T = 100.0                          # final time
dt = 0.1                           # time increment
N = int(T/dt) + 1                  # number of time-steps
t = numpy.linspace(0.0, T, N)      # time discretization

# initialize the array containing the solution for each time-step
u = numpy.empty((N, 4))
u[0] = numpy.array([v0, theta0, x0, y0])# fill 1st element with initial values

# time loop - Euler method
for n in range(N-1):

    u[n+1] = euler_step(u[n], f, dt)
```

## Plot the trajectory

In order to plot the path of the glider, we need the location (`x`, `y`) with respect to time. That information is already contained in our NumPy array containing the solution; we just need to pluck it out.

Make sure you understand the indices to `u`, below, and the use of the colon notation. If any of it is confusing, read the Python documentation on [Indexing](http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html).

```
# get the glider's position with respect to the time
x = u[:,2]
y = u[:,3]
```

Time to plot the path of the glider and get the distance travelled!

```
# visualization of the path
plt.figure(figsize=(8,6))
plt.grid(True)
plt.xlabel(r'x', fontsize=18)
plt.ylabel(r'y', fontsize=18)
plt.title('Glider trajectory, flight time = %.2f' % T, fontsize=18)
plt.plot(x,y, 'k-', lw=2);
```

## Grid convergence

Let's study the convergence of Euler's method for the phugoid model. In the previous lesson, when we studied the straight-line phugoid under a small perturbation, we looked at convergence by comparing the numerical solution with the exact solution. Unfortunately, most problems don't have an exact solution (that's why we compute in the first place!). But here's a neat thing: we can use numerical solutions computed on different grids to study the convergence of the method, even without an analytical solution.

We need to be careful, though, and make sure that the fine-grid solution is resolving all of the features in the mathematical model. How can we know this? We'll have a look at that in a bit. Let's see how this works first.

You need a sequence of numerical solutions of the same problem, each with a different number of time grid points.

Let's create a NumPy array called `dt_values` that contains the  time-increment of each grid to be solved on. For each element `dt_values[i]`, we will compute the solution `u_values[i]` of the glider model using Euler's method. If we want to use five different values of $\Delta t$, we'll have five solutions: we put them in an array ... but each one is also an array! We'll have an array of arrays. How meta is that?

We have one more trick up our sleeve: `enumerate()`. To get all the numerical solutions—each with its value of $\Delta t$—done in one fell swoop, we will loop over the elements of the array `dt_values`. Within the loop, we need to access both `dt_values[i]` and the index `i`. It turns out, `enumerate()` is a built-in Python function that will give us consecutive `index, value` pairs just like we need.

Read the code below carefully, and remember: you can get a help pane on any function by entering a question mark followed by the function name. For example, add a new code cell below and type:  `?numpy.empty_like`.

```
dt_values = numpy.array([0.1, 0.05, 0.01, 0.005, 0.001])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt) + 1    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):

        u[n+1] = euler_step(u[n], f, dt)   ### call euler_step() ###

    # store the value of u related to one grid
    u_values[i] = u
```

In [Lesson 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_02_Phugoid_Oscillation.ipynb), we compared our numerical result to an analytical solution, but now we will instead compare numerical results from different grids.

For each solution, we'll compute the difference relative to the finest grid. You will be tempted to call this an _"error"_, but be careful: the solution at the finest grid is _not the exact_ solution, it is just a reference value that we can use to estimate grid convergence.

To calculate the difference between one solution `u_current` and the solution at the finest grid, `u_finest`, we'll use the $L_1$-norm, but any norm will do.

There is a small problem with this, though.  The coarsest grid, where $\Delta t = 0.1$, has 1001 grid points, while the finest grid, with $\Delta t = 0.001$ has 100001 grid points.  How do we know which grid points correspond to the same location in two numerical solutions, in order to compare them?

If we had time grids of 10 and 100 steps, respectively, this would be relatively simple to calculate.  Each element in our 10-step grid would span ten elements in our 100-step grid.

Calculating the _ratio_ of the two grid sizes will tell us how many elements in our fine-grid will span over one element in our coarser grid.

Recall that we can _slice_ a NumPy array and grab a subset of values from it.  The syntax for that is

```Python
my_array[3:8]
```

An additional slicing trick that we can take advantage of is the "slice step size."  We add an additional `:` to the slice range and then specify how many steps to take between elements. For example, this code

```Python
my_array[3:8:2]
```

will return the values of `my_array[3]`, `my_array[5]` and `my_array[7]`

With that, we can write a function to obtain the differences between coarser and finest grids. Here we go ...

```
def get_diffgrid(u_current, u_fine, dt):
    """Returns the difference between one grid and the fine one using L-1 norm.

    Parameters
    ----------
    u_current : array of float
        solution on the current grid.
    u_finest : array of float
        solution on the fine grid.
    dt : float
        time-increment on the current grid.

    Returns
    -------
    diffgrid : float
        difference computed in the L-1 norm.
    """

    N_current = len(u_current[:,0])
    N_fine = len(u_fine[:,0])

    grid_size_ratio = ceil(N_fine/float(N_current))

    diffgrid = dt * numpy.sum( numpy.abs(\
            u_current[:,2]- u_fine[::grid_size_ratio,2]))

    return diffgrid
```

Now that the function has been defined, let's compute the grid differences for each solution, relative to the fine-grid solution. Call the function `get_diffgrid()` with two solutions, one of which is always the one at the finest grid. Here's a neat Python trick: you can use negative indexing in Python!  If you have an array called `my_array` you access the _first_ element with

`my_array[0]`

But you can also access the _last_ element with

`my_array[-1]`

and the next to last element with

`my_array[-2]`

and so on.

```
# compute difference between one grid solution and the finest one
diffgrid = numpy.empty_like(dt_values)

for i, dt in enumerate(dt_values):
    print('dt = {}'.format(dt))

    ### call the function get_diffgrid() ###
    diffgrid[i] = get_diffgrid(u_values[i], u_values[-1], dt)
```

    dt = 0.1
    dt = 0.05
    dt = 0.01
    dt = 0.005
    dt = 0.001

Time to create a plot of the results! We'll create a *log-log* plot with the Matplotlib function [`loglog()`](http://matplotlib.org/api/pyplot_api.html?highlight=loglog#matplotlib.pyplot.loglog). Remember to skip the difference of the finest-grid solution with itself, which is zero.

```
# log-log plot of the grid differences
plt.figure(figsize=(6,6))
plt.grid(True)
plt.xlabel('$\Delta t$', fontsize=18)
plt.ylabel('$L_1$-norm of the grid differences', fontsize=18)
plt.axis('equal')
plt.loglog(dt_values[:-1], diffgrid[:-1], color='k', ls='-', lw=2, marker='o');
```

### Order of convergence

The order of convergence is the rate at which the numerical solution approaches the exact one as the mesh is refined. Considering that we're not comparing with an exact solution, we use 3 grid resolutions that are refined at a constant ratio $r$ to find the *observed order of convergence* ($p$), which is given by:

$$
\begin{equation}
p = \frac{\log \left(\frac{f_3-f_2}{f_2-f_1} \right) }{\log(r)}
\end{equation}
$$

where $f_1$ is the finest mesh solution, and $f_3$ the coarsest.

```
r = 2
h = 0.001

dt_values2 = numpy.array([h, r*h, r**2*h])

u_values2 = numpy.empty_like(dt_values2, dtype=numpy.ndarray)

diffgrid2 = numpy.empty(2)

for i, dt in enumerate(dt_values2):

    N = int(T/dt)    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):

        u[n+1] = euler_step(u[n], f, dt)         ### call euler_step() ###

    # store the value of u related to one grid
    u_values2[i] = u

#calculate f2 - f1
diffgrid2[0] = get_diffgrid(u_values2[1], u_values2[0], dt_values2[1])

#calculate f3 - f2
diffgrid2[1] = get_diffgrid(u_values2[2], u_values2[1], dt_values2[2])

# calculate the order of convergence
p = (log(diffgrid2[1]) - log(diffgrid2[0])) / log(r)

print('The order of convergence is p = {:.3f}'.format(p));
```

    The order of convergence is p = 1.014

See how the observed order of convergence is close to 1? This means that the rate at which the grid differences decrease match the mesh-refinement ratio. We say that Euler's method is of *first order*, and this result is a consequence of that.

# Paper airplane challenge

Suppose you wanted to participate in a paper-airplane competition, and you want to use what you know about the phugoid model to improve your chances. For a given value of $L/D$ that you can obtain in your design, you want to know what is the best initial velocity and launch angle to fly the longest distance from a given height.

Using the phugoid model, write a new code to analyze the flight of a paper airplane, with the following conditions:

*  Assume $L/D$ of 5.0 (a value close to measurements in Feng et al. 2009)
*  For the trim velocity, let's take an average value of 4.9 m/s.
*  Find a combination of launch angle and velocity that gives the best distance.
*  Think about how you will know when the flight needs to stop ... this will influence how you organize the code.
*  How can you check if your answer is realistic?

```
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 4.9  # trim velocity in m s^{-1}
C_D = 1/5.  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = v_t     # start at the trim velocity (or add a delta)
theta0 = 0.25 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 10.0  # initial altitude
```

```
T = 20.0                          # final time
dt = 0.1                           # time increment
N = int(T/dt) + 1                  # number of time-steps
t = numpy.linspace(0.0, T, N)      # time discretization

# initialize the array containing the solution for each time-step
u = numpy.empty((N, 4))
u[0] = numpy.array([v0, theta0, x0, y0])# fill 1st element with initial values

# time loop - Euler method
for n in range(N-1):
    u[n+1] = euler_step(u[n], f, dt)
```

```
# get the glider's position with respect to the time
x = u[:,2]
y = u[:,3]
```

```
# visualization of the path
plt.figure(figsize=(8,6))
plt.grid(True)
plt.xlabel(r'x', fontsize=18)
plt.ylabel(r'y', fontsize=18)
plt.title('Glider trajectory, flight time = %.2f' % T, fontsize=18)
plt.plot(x,y, 'k-', lw=2);
```

```
beginTheta = 0.2
endTheta = 0.3
numThetas = 100
thetas = numpy.linspace(beginTheta, endTheta, numThetas)

x0 = 0.
y0 = 10.0

beginV0 = 4.85
endV0 = 4.95
numVelocities = 100
velocities = numpy.linspace(beginV0, endV0, numVelocities)

dt = 0.1

# shape is rows, columns
distances = numpy.zeros(shape=(numThetas, numVelocities))

maxDistance = 0.
bestV = 0.
bestTheta = 0.

for thetaIndex in range(numThetas - 1):
    for velocityIndex in range(numVelocities - 1):
        v0 = velocities[velocityIndex]
        theta0 = thetas[thetaIndex]
        u0 = numpy.array([v0, theta0, x0, y0])# fill 1st element with initial values
        u = u0
        time = 0.
        distance = 0.
        try:
            while time < T and u[3] > 0.:
                u = euler_step(u, f, dt)
                time += dt
            distance = u[2]
        except:
            pass
        # Filter out inf distances
        if distance > maxDistance and distance < 10000:
            maxDistance = distance
            bestV = v0
            bestTheta = theta0
        distances[thetaIndex, velocityIndex] = distance

print(maxDistance)
print(bestV)
print(bestTheta)
```

    49.3723130565
    4.86515151515
    0.2

```
from mpl_toolkits.mplot3d.axes3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import numpy as np
%matplotlib inline

fig = plt.figure()
ax = fig.gca(projection='3d')
#X = thetas
#Y = velocities
#Z = distances
X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X**2 + Y**2)
Z = np.sin(R)
# surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.coolwarm, linewidth=0, antialiased=False)
surf = ax.plot_surface(X, Y, Z)
ax.set_zlim(-1.01, 1.01)

#ax.zaxis.set_major_locator(LinearLocator(10))
#ax.zaxis.set_major_formatter(FormatterStrFormatter('%.02f'))
#fig.colorbar(surf, shrink=0.5, aspect=5)

plt.show()
```

## References

* Feng, N. B. et al. _"On the aerodynamics of paper airplanes"_, AIAA paper 2009-3958, 27th AIAA Applied Aerodynamics Conference, San Antonio, TX. [PDF](http://www.ntu.edu.sg/home/schluter/pdf/aiaa-2009-3958.pdf)

* Simanca, S. R. and Sutherland, S. _"Mathematical problem-solving with computers,"_ 2002 course notes, Stony Brook University, chapter 3: [The Art of Phugoid](https://www.math.sunysb.edu/~scott/Book331/Art_Phugoid.html). (Note that there is an error in the figure: sine and cosine are switched.)

---

######The cell below loads the style of the notebook.

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```


# Chapter 1: Generative Models

## Generative Versus Discriminative Modeling

- A _discriminative model_ estimates $p(y|\mathbf{x})$, i.e. the probability of a label $y$ given an observation $\mathbf{x}$.

- A _generative model_ estimates $p(\mathbf{x})$, i.e. the probability of observing observation $\mathbf{x}$. If the dataset is labeled, then a generative model estimates the distribution $p(y|\mathbf{x})$.

## The Generative Modeling Framework

- We have a dataset of observations $\mathbf{X}$.

- We assume the observations have been generated according to some unknown distribution, $p_\text{data}$.

- A generative model $p_\text{model}$ tries to mimic $p_\text{data}$ so that we can generate new observations.

We are generally impressed with $p_\text{model}$ if:

1. It can generate observations that appear to have been drawn from $p_\text{data}$.

2. It can generate examples suitably different from the observations it has seen, i.e. it does not just copy previously seen observations.

## Probabilistic Generative Models

### Sample Space

A _sample space_ is the complete set of all values that $\mathbf{x}$ can take.

### Probability Density Function

A _probability density function_ (or _density function_), $p(\mathbf{x})$, is a function from the sample space, $S$, to the interval [0, 1] such that

$$ \int_{\mathbf{x}\,\in\,S} p(\mathbf{x})\,d\mathbf{x} = 1 $$

if $S$ is continuous or

$$ \sum\limits_{\mathbf{x}\,\in\,S} p(\mathbf{x}) = 1 $$

if $S$ is discrete.

### Parametric Modeling

A _parametric model_, $p_\theta(\mathbf{x})$, is a family of density functions that can be described using a finite number of parameters represented by the vector, $\theta$.

### Likelihood

The _likelihood_, $\mathcal{L} (\theta | \mathbf{x})$, of a parameter set $\theta$ is a function that gives you the likelihood of $\theta$ given some observed point $\mathbf{x}$ is defined to be the value of a density function parameterized by theta. It is defined as

$$ \mathcal{L}(\theta|\mathbf{x}) = p_\theta(\mathbf{x}). $$

Given a dataset, $\mathbf{X}$, the likelihood of the distribution parameterized by $\theta$ is given by

$$ \mathcal{L}(\theta|\mathbf{X}) = \prod\limits_{\mathbf{x}\,\in\,\mathbf{X}} p_\theta (\mathbf{x}). $$

It is more common to use the _log-likelihood_ function instead, which is less computationally intense:

$$ \ell(\theta|\mathbf{X}) = \prod\limits_{\mathbf{x}\,\in\,\mathbf{X}} \log p_\theta(\mathbf{x}). $$

### Maximum Likelihood Estimation

The _maximum likelihood estimation_ (MLE) is defined by the set of parameters, $\hat{\theta}$, which maximize the likelihood function, i.e.

$$ \hat{\theta} = \underset{\theta}{\text{argmax}} \, \mathcal{L}(\theta|\mathbf{X}). $$

## Probabalistic Generative Models

### Multinomial Distribution

The _multinomial distribution_ assigns each combination of features a probablity given by the number of times that combination appears in the original dataset. The MLE, $\hat{\theta}_j$ is given by

$$ \hat{\theta}_j = \frac{n_j}{N}. $$

The one clear downside of this distribution is that it cannot be used to generate any new samples. This can be remedied is to add a _pseudocount_ of 1 to each possible combination. This would make the MLE

$$ \hat{\theta}_j = \frac{n_j + 1}{N + d} $$

where $d$ is the number of total possible combinations of features. This technique is known as _additive smoothing_. This technique ensures that all combinations have some probability of being observed, but it also means that any unobserved combination has an equal probability of being drawn.

### Naive Bayes

The _Naive Bayes_ assumption assumes that for all features of the dataset, $x_j$ and $x_k$, we have that

$$ p\left(x_j | x_k\right) = p\left(x_j\right) $$

i.e. that any two features' values are independent of each other. Using this assumption and the chain rule of probability, we can compute the probability of an observation, $\mathbf{x}$

$$ \begin{align} p(\mathbf{x}) & = p\left(x_1,...,x_K\right) \\ & = p\left(x_2,...,x_K|x_1\right)p\left(x_1\right) \\ & = p\left(x_3,...,x_K|x_1,x_2\right) p\left(x_2|x_1\right) p\left(x_1\right) \\ & = \prod\limits_{k=1}^K p\left(x_k | x_1,...,x_{k-1}\right) \end {align} $$

where $K$ is the number of features. Using the Naive Bayes' assumption, we can simplify this expression to

$$ p(\mathbf{x}) = \prod\limits_{k=1}^K p\left(x_k\right) $$

This is the Naive Bayes' model. The problem of finding the best model is to finding the best parameters

$$ \theta_{kl} = p\left(x_k = l\right). $$

The MLE $\hat{\theta}_{kl}$ is given by

$$ \hat{\theta}_{kl} = \frac{n_{kl}}{N} $$

where $n_{kl}$ is the number of times $x_k$ has the value $l$ in the dataset of observations.


# Support Vector Machines (SVM)

Le __[Support Vector Machines](https://it.wikipedia.org/wiki/Macchine_a_vettori_di_supporto)__ sono un **algoritmo di machine learning supervisionato in grado sia di fare classificazione binaria e di fare regressione, risulta anche essere in grado di essere robusto agli** __[outliers](https://en.wikipedia.org/wiki/Outlier)__ **ovvero dati che sono molto diversi dalle altre osservazioni precedenti per possibili errori o condizioni diversi**.

Il funzionamento di queste macchine su basa su un principio piuttosto semplice, quando noi effettuiamo una classificazione binaria quello che definiamo è un sottospazio detto __[iperpiano](https://it.wikipedia.org/wiki/Iperpiano)__ attraverso cui riusciamo a definire due regioni dello spazio a cui noi possiamo associare una delle due classi, se per esempio siamo in uno spazio 2d con dei dati quello che possiamo fare è associare una linea in grado di dirci che se i punti sono a sinistra o a destra di esso allora abbiamo una classe determinabile dai punti usati per definire l'iperpiano attraverso il training.

Ora però abbiamo un problema, **qualora fosse possibile definire un'iperpiano(*hyperplane in inglese*) quello che potremmo trovarci è che ne esiste più di uno, allora come fare a scegliere il migliore?**

## Maximum Margin Optimization
In genere l'iperpiano migliore è quello che riesce a massimizzare la distanza dall'iperpiano ai punti della classi, detto anche margine in matematica, abbiamo quindi un problema di __[massimizzazione margine dell'iperpiano](https://www.sciencedirect.com/topics/computer-science/maximum-margin-hyperplane)__, in genere per risolvere questo problema si usano i **vettori di supporto(support vectors) ovvero i vettori ottenuti dalla distanza dell'iperpiano ai punti più vicini ad esso**.

Tradotto in fotmule matematiche noi abbiamo:

\begin{equation}
\Large w^{T}x + b = 0 \quad Hyperplane\\
\Large \frac{1}{\Vert w \Vert} \quad Margin (\Vert w \Vert \;is\; the\; norm\; of\; w)\\
\Large |w^{T}x + b| = 1 \quad classification
\end{equation}

Ora utilizzando la condizione di massimizzare il margine dell'iperpiano contando la capacità di classificare otteniamo il seguente problema:

\begin{equation}
\Large Maximize \quad \frac{1}{\Vert w \Vert} \, subject \, to \, \min_{n = 1,2,..,N} |w^{T}x + b| = 1
\end{equation}

che è possibile tradurre in:

\begin{equation}
\Large Minimize \quad \frac{w^{T}w}{2} \, subject \, to \, y_n(w^{T}x_n + b) \geqslant 1 \quad with \; n = 1,2,..,N \, and \, y_n \in [-1,1]
\end{equation}

Usando le condizioni di __[KKT](https://it.wikipedia.org/wiki/Condizioni_di_Karush-Kuhn-Tucker)__ e imponendo la minimizzazione si ottiene la seguente formula di massimizzazione:

\begin{equation}
\Large Maximize \; \mathcal{L}(\alpha) = \sum_{n=1}^{N} \alpha_n -\frac{1}{2} \sum_{n=1}^{N} \sum_{m=1}^{N} y_n y_m \alpha_n \alpha_m x^{T}_{n} x_{m}\\
\Large subject\, to \ , \alpha_n \geqslant 0 \,  for \, n = 1,..N \, and \,  \sum_{n=1}^{N} \alpha_n y_n = 0 \\
\Large If \; y_n(w^{T}x_n + b) = 1 \; then \; x_n \; is \; a \; support \; vector
\end{equation}

quindi il termine $\alpha$  sarebbe un termine che influisce sulla creazione del margine ed è definibile dal training sui dati.

## Il problema degli outlier

Abbiamo però il problema che fino ad ora abbiamo ipotizzato che non ci fossero outlier e che quindi non ci fosser dati "strani", **se però per esempio ipotizziamo che un dato è stato classificato male, ovvero è un outlier, ma poichè presenta tutte le caratteristiche dell'altra classe sappiamo che è stato classificato male, dobbiamo allora fare in modo che l'algoritmo ne tenga conto al fine di poter generalizzare meglio in seguito**, in tal caso quello che possiamo fare è introdurre un ***termine di violazione del margine tale che il nostro margine diventi "soft" ovvero permetta la sua violazione, in tal modo avremo la possibilità di permettere ad alcuni punti di essere outliers".

Dal punto di vista matematico quello che succede è:

\begin{equation}
\Large Minimize \quad \frac{w^{T}w}{2} + C \sum_{i=1}^{N} \xi_n \quad subject \; to \; y_n(w^{T}x_n + b) \geqslant 1 - \xi_n, \; \xi_n \geqslant 0
\end{equation}

dove il termine $\xi$ rappresenta la violazione del dato nella violazione del margine, ora dal punto di vista della loss function essa diventa usando sempre __[KKT](https://it.wikipedia.org/wiki/Condizioni_di_Karush-Kuhn-Tucker)__:

\begin{equation}
\Large Maximize \; \mathcal{L}(\alpha) = \sum_{n=1}^{N} \alpha_n -\frac{1}{2} \sum_{n=1}^{N} \sum_{m=1}^{N} y_n y_m \alpha_n \alpha_m x^{T}_{n} x_{m}\\
\Large subject\, to \ , 0 \geqslant \alpha_n \geqslant C \,  for \, n = 1,..N \, and \,  \sum_{n=1}^{N} \alpha_n y_n = 0 \\
\end{equation}

Notate bene che in questa formula $\alpha$ non deve essere solo positiva come nel caso precedente, ma deve essere superiormente limitata da $C$ che ora è diventato il nostro termine di regoralizzazione sulla quale abbiamo imposto quanto pesa la nostra violazione dei dati.

In __[scikit](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC)__ il parametro $C$ controlla effettivamente se vogliamo pesare molto le misclassification oppure no, in genere se imponente **un $C$ non nullo e non troppo grande dite all'algoritmo che permettete la possibilità di misclassification e state definendo un "soft margin"**, mentre se imponete **un $C$ tendente ad infinite state dicendo di non fare misclassification e quindi non accettate violazioni del margine detto "hard margin"**, a qui un __[esempio di scikit](https://scikit-learn.org/stable/auto_examples/svm/plot_svm_margin.html#sphx-glr-auto-examples-svm-plot-svm-margin-py)__.

## Dati non lineari e il Kernel

Ora il problema è che non sempre i dati sono classificabili in maniera lineare, in tal caso è necessario effettuare una trasformazione non lineare che possiamo definire come $z = \Phi(x)$, ora se noi applicassimo la formula di minimizzazione precedente avremmo:

\begin{equation}
\Large Maximize \; \mathcal{L}(\alpha) = \sum_{n=1}^{N} \alpha_n -\frac{1}{2} \sum_{n=1}^{N} \sum_{m=1}^{N} y_n y_m \alpha_n \alpha_m z^{T}_{n} z_{m}\\
\Large subject\, to \ , 0 \geqslant \alpha_n \geqslant C \,  for \, n = 1,..N \, and \,  \sum_{n=1}^{N} \alpha_n y_n = 0 \\
\end{equation}

Ora abbiamo che il kernel è definito come(ho usato z vettorizzato):

\begin{equation}
\Large Kernel \quad K(x,x') = z^{T}z'
\end{equation}

Ora se volessimo ad esempio una trasformazione polinomiale di grado 2 ad uno spazio 2D avremmo:

\begin{equation}
\Large x = (x_1, x_2) \quad \rightarrow \quad z = \Phi(x) = (1, x_1, x_2, x_{1}^{2}, x_{2}^{2}, x_1 x_2)\\
\Large K(x,x') = z^{T}z' = 1 + x_1 x'_{1} + x_2 x'_{2} + x^{2}_{1} x'^{2}_{1} + x^{2}_{2} x'^{2}_{2} + x_1 x'_{1} x_2 x'_{2}
\end{equation}

Ora se potete notare calcolare in questa maniera è molto dispendioso visto che devo trasformare e poi applicare un prodotto scalare che porta numerosi termini, un trucco per ridurre il calcolo computazionale è usare il __[kernel trick](https://towardsdatascience.com/the-kernel-trick-c98cdbcaeb3f)__, per vederlo potete notare che:

\begin{equation}
\Large K(x,x') = (1 + x^{T}x')^2 = 1 + 2x_1 x'_{1} + 2x_2 x'_{2} + x^{2}_{1} x'^{2}_{1} + x^{2}_{2} x'^{2}_{2} + 2x_1 x'_{1} x_2 x'_{2}\\
\end{equation}

Questa formula aparte dei coefficienti che non ci interessano poichè non influiscono sulla definizione del nuovo spazio, il prodotto scalare precedente è esattamente uguale al prodotto scalare delle trasformazioni dello spazio lineare come definito primo, in poche parole il kernel trick ci dice che:

\begin{equation}
    K(x,x') = < \Phi(x), \Phi(x')> = \Phi(<x,x'>)
\end{equation}

**Il simbolo $<,>$ inidica il prodotto scalare, grazie a questo a questo è possibile usare la __[radial basis function](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.kernels.RBF.html?highlight=rbf#sklearn.gaussian_process.kernels.RBF)__ attraverso cui possiamo fare una classificazione lineare in uno spazio infinito dimensionale in tempo finito! Not Bad!**.

## Il Support Vector Regression (SVR)

Il __[Support Vector Regression](https://www.educba.com/support-vector-regression/)__ è la SVM applicata nel contesto di regressione, **la differenza fondamentale con la classificazione è che in questo caso voi cercate di mantenere i punti della regressione all'interno dei margini della nostra SVM**, questo ci dice che l'algoritmo considererà outliers i punti al di fuori del margine, pe averne una visione completa guardate questi __[esempi scikit](https://scikit-learn.org/stable/auto_examples/svm/plot_svm_regression.html#sphx-glr-auto-examples-svm-plot-svm-regression-py)__.

    La trattazione che ho cercato di fare è quanto più completa possibile, ma so che potrei non essere stato molto chiaro, pertanto vi rimando a questo link in italiano che lo spiega in maniera più semplice e meno matematica.

Qui sotto metto invece un link youtube video in cui potete visualzzare i concetti appena visti, sono 3 video, ma non posso linkare una playlist cercate comunque su youtube.

```python
from IPython.display import YouTubeVideo

#put link yotube video, only final part
YouTubeVideo("efR1C6CvhmE")
```

```python
import pandas as pd
import matplotlib.pyplot as plt
import time
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.metrics import plot_confusion_matrix, classification_report
from sklearn.svm import SVC, SVR

#regression data
boston = load_boston()
X_boston, y_boston = boston.data, boston.target

#classification data
diabetes = pd.read_csv('../data/diabetes2.csv')
X_diabetes, y_diabetes = diabetes.drop('Outcome', axis = 1).values, diabetes.Outcome.values
feature_diabetes = diabetes.columns.values[:-1]
target_names = ["Not Diabetes", "Diabetes"]

#divide the data in training and testing
X_boston_train, X_boston_test, y_boston_train, y_boston_test = train_test_split(
    X_boston, y_boston, random_state=0, test_size = 0.2)

X_diabetes_train, X_diabetes_test, y_diabetes_train, y_diabetes_test = train_test_split(
    X_diabetes, y_diabetes, random_state=0, test_size = 0.2)

print("SVM FOR REGRESSION TESTED ON BOSTON DATASET")

print('R^2 score training and testing using SVR with C = 1 kernel:')
svr = SVR(kernel = "poly", degree=3)
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- polynomial of degree 3, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")
svr = SVR(kernel = "linear")
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- linear, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")
svr = SVR(kernel = "rbf")
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- Radial basis function, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")
svr = SVR(kernel = "sigmoid")
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- Sigmoid, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")

print('-'*80)

print('R^2 score training and testing using SVR with C = 1000 kernel:')
svr = SVR(C = 1e3, kernel = "poly", degree=3)
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- polynomial of degree 3, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")
svr = SVR(C = 1e3, kernel = "linear")
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- linear, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")
svr = SVR(C = 1e3,kernel = "rbf")
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- Radial basis function, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")
svr = SVR(C = 1e3, kernel = "sigmoid")
start = time.time()
svr.fit(X_boston_train, y_boston_train)
end = time.time()
print(f"- Sigmoid, training score:{svr.score(X_boston_train, y_boston_train)},"
      f" testing score :  {svr.score(X_boston_test, y_boston_test)}, time taken :{end-start}s")

```

    SVM FOR REGRESSION TESTED ON BOSTON DATASET
    R^2 score training and testing using SVR with C = 1 kernel:
    - polynomial of degree 3, training score:0.24644868913152274, testing score :  0.053244563027002534, time taken :0.01099848747253418s
    - linear, training score:0.7484998446645601, testing score :  0.5159537820682599, time taken :1.0620331764221191s
    - Radial basis function, training score:0.24935461402924874, testing score :  0.03623285217626426, time taken :0.009000062942504883s
    - Sigmoid, training score:0.06599931272060566, testing score :  -0.0072226293343120584, time taken :0.011972188949584961s
    --------------------------------------------------------------------------------
    R^2 score training and testing using SVR with C = 1000 kernel:
    - polynomial of degree 3, training score:0.6972777900096314, testing score :  0.4208946625258052, time taken :0.0780022144317627s
    - linear, training score:-20.15355480442324, testing score :  -18.408675490837567, time taken :33.40475797653198s
    - Radial basis function, training score:0.72281432475884, testing score :  0.43222121960176785, time taken :0.018999814987182617s
    - Sigmoid, training score:-135246.221578731, testing score :  -157703.380591555, time taken :0.010997772216796875s

```python
#let's use now svm for classification
svcs = [SVC(C = 1, kernel = 'poly', degree = 3), SVC(C = 1, kernel = 'rbf'),
        SVC(C = 1, kernel = 'linear'), SVC(C = 1, kernel = 'sigmoid')]

#lets train the models
for svc in svcs:
        start = time.time()
        svc.fit(X_diabetes_train, y_diabetes_train)
        print(f"Time taken to train {str(svc)}: {time.time() - start}s \t")

#lets make a confusion matrix for every single one
#confusion matrixes plots
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(16,10))

for svc, ax in zip(svcs, axes.flatten()):
    plot_confusion_matrix(svc,
                          X_diabetes_test,
                          y_diabetes_test,
                          ax=ax,
                          display_labels=target_names,
                          colorbar= False)
    ax.title.set_text(str(svc))

plt.tight_layout()
plt.show()

#lets use a classification report
for svc in svcs:
    print(f"Classification report of {str(svc)}\n")
    print(classification_report(y_diabetes_test, svc.predict(X_diabetes_test),
                                  target_names=target_names))

#let's use now svm for classification
svcs = [SVC(C = 1e3, kernel = 'poly', degree = 3), SVC(C = 1e3, kernel = 'rbf'),
        SVC(C = 1e3, kernel = 'linear'), SVC(C = 1e3, kernel = 'sigmoid')]

#lets train the models
for svc in svcs:
        start = time.time()
        svc.fit(X_diabetes_train, y_diabetes_train)
        print(f"Time taken to train {str(svc)}: {time.time() - start}s \t")

#lets make a confusion matrix for every single one
#confusion matrixes plots
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(16,10))

for svc, ax in zip(svcs, axes.flatten()):
    plot_confusion_matrix(svc,
                          X_diabetes_test,
                          y_diabetes_test,
                          ax=ax,
                          display_labels=target_names,
                          colorbar= False)
    ax.title.set_text(str(svc))

plt.tight_layout()
plt.show()

#lets use a classification report
for svc in svcs:
    print(f"Classification report of {str(svc)}\n")
    print(classification_report(y_diabetes_test, svc.predict(X_diabetes_test),
                                  target_names=target_names))
```

Qualora voleste provare altri algoritmi andate alla __[guida scikit](https://scikit-learn.org/stable/modules/svm.html)__ che offre un enorme varietà e esempi a tal proposito.

***

COMPLIMENTI AVETE FINITO LA LEZIONE DI SVM!


# Jury's stability criterion

```python
import numpy as np
from sympy import *
import itertools
init_printing()
%matplotlib inline
```

```python
def jury(a):
    n = len(a)
    tab = []
    for i in range(n):
        line1 = a[:n-i]
        line2 = line1[::-1]
        tab.append(line1)
        tab.append(line2)
        alpha = line1[-1]/line2[-1]
        a[:n-i] = line1-alpha*line2
    for line in tab:
        print line

def juryrec(a,tab):
    n = len(a)
    if n==1:
        tab.append(a)
    else:
        line1 = a
        line2 = line1[::-1]
        tab.append(line1)
        tab.append(line2)
        alpha = line1[-1]/line2[-1]
        aa = [el1 - alpha*el2 for (el1,el2) in itertools.izip(line1,line2)]
        juryrec(aa[:-1],tab)

```

```python
aa1 = np.array([1,-2,1.4,-0.1])
aa2 = np.array([4,-4,-7,2])
a2 = [Integer(4),Integer(-4),Integer(-7),Integer(2)]

tab=[]
juryrec(a2, tab)
print "Coefficients"
print aa2
for line in tab:
    print line
np.roots(aa2)
tab=[]
juryrec(aa1, tab)
print "Coefficients"
print aa1
for line in tab:
    print line
print "Roots"
print np.roots(aa1)
print "Magnitudes"
print np.abs(np.roots(aa1))
```

    Coefficients
    [ 4 -4 -7  2]
    [4, -4, -7, 2]
    [2, -7, -4, 4]
    [3, -1/2, -5]
    [-5, -1/2, 3]
    [-16/3, -4/3]
    [-4/3, -16/3]
    [-5]
    Coefficients
    [ 1.  -2.   1.4 -0.1]
    [ 1.  -2.   1.4 -0.1]
    [-0.1  1.4 -2.   1. ]
    [0.98999999999999999, -1.8600000000000001, 1.2]
    [1.2, -1.8600000000000001, 0.98999999999999999]
    [-0.46454545454545459, 0.39454545454545475]
    [0.39454545454545475, -0.46454545454545459]
    [-0.12945205479452027]
    Roots
    [ 0.95986894+0.56970991j  0.95986894-0.56970991j  0.08026212+0.j        ]
    Magnitudes
    [ 1.11620687  1.11620687  0.08026212]

```python
-0.4645+0.3945**2/0.4645
```

    -0.12945102260495156

## Proportional control of second order system (DC-motor with zero-cancellation)
The transfer function is given by
\begin{equation}
H(z) = \frac{1}{z^2 - 1.5z + 0.5} = \frac{B(z)}{A(z)}.
\end{equation}
The closed loop pulse-transfer function becomes
\begin{equation}
H_c(z) = \frac{K H(z)}{KH(z) + 1} = \frac{K B(z)}{A(z) + KB(z)}.
\end{equation}
The characteristic equation of the closed loop system is thus
\begin{equation}
z^2  -1.5z + 0.5+K
\end{equation}

```python
K = symbols('K')
a = [Integer(1), -1.5, 0.5+K]
tab=[]
juryrec(a,tab)
for line in tab:
    print line
crit1 = simplify(tab[2][0])
crit2 = simplify(tab[4][0])
print crit1
print solve(crit1, K)
print crit2
print solve(crit2, K)
plot(crit2, xlim=(-1.0, 5.0), ylim=(-1,1))
plot(crit1, xlim=(-1.0, 5.0), ylim=(-1,1))
```

## Proportional control of the normalized DC-motor
Zero-order hold sampling of the DC motor with transfer function $G(s)=\frac{1}{s(s+1)}$ gives the discrete time system
\begin{equation} H(z) = \frac{\big(h-1+e^{-h}\big)z + \big(1-e^{-h}-he^{-h}\big)}{z^2 -\big(1+e^{-h}\big)z + e^{-h}} \end{equation}

Let $h=\ln 2 \approx 0.693$. This gives the pulse-transfer function
\begin{equation} H(z) = \frac{B(z)}{A(z)} = \frac{0.19z + 0.15}{z^2 - 1.5z + 0.5} \end{equation}

Proportional control gives the closed loop system
\begin{equation}
H_c(z) = \frac{K H(z)}{KH(z) + 1} = \frac{K B(z)}{A(z) + KB(z)}.
\end{equation}
The characteristic equation of the closed loop system is
\begin{equation}
z^2 + (-1.5+0.19K)z + 0.5+0.15K
\end{equation}

```python
z,h = symbols('z,h')
eh = exp(-h)
H = ( (h-1+eh)*z + 1-eh-h*eh ) / (z**2 -(1+eh)*z + eh )
print H
H1 = H.subs(h,log(2))
print H1
H2 = N(H1)
print H2
```

    (-h*exp(-h) + z*(h - 1 + exp(-h)) + 1 - exp(-h))/(z**2 - z*(1 + exp(-h)) + exp(-h))
    (z*(-1/2 + log(2)) - log(2)/2 + 1/2)/(z**2 - 3*z/2 + 1/2)
    (0.193147180559945*z + 0.153426409720027)/(z**2 - 1.5*z + 0.5)

### Apply Jury's criterion

```python
K = symbols('K')
a = [Integer(1), -1.5+0.19*K, 0.5+0.15*K]
tab=[]
juryrec(a,tab)
for line in tab:
    print line

```

    [1, 0.19*K - 1.5, 0.15*K + 0.5]
    [0.15*K + 0.5, 0.19*K - 1.5, 1]
    [-(0.15*K + 0.5)**2 + 1, 0.19*K - (0.15*K + 0.5)*(0.19*K - 1.5) - 1.5]
    [0.19*K - (0.15*K + 0.5)*(0.19*K - 1.5) - 1.5, -(0.15*K + 0.5)**2 + 1]
    [-(0.15*K + 0.5)**2 + 1 - (0.19*K - (0.15*K + 0.5)*(0.19*K - 1.5) - 1.5)**2/(-(0.15*K + 0.5)**2 + 1)]

```python
print tab[2][0]
print solve(tab[2][0], K)
print solve(tab[4][0], K)
print N(tab[4][0].subs(K,3))
print N(tab[2][0].subs(K,3))
```

    -(0.15*K + 0.5)**2 + 1
    [-10.0000000000000, 3.33333333333333]
    [0.0, 75.0000000000000]
    0.0753230769230770
    0.0975000000000000

```python
from sympy.solvers.inequalities import solve_rational_inequalities
solve_rational_inequalities([[ ( (Poly(tab[2][0], K),Poly(tab[2][0], K)), '>=') ]])

```

    (-oo, -10) U (-10, 10/3) U (10/3, oo)

## Proportional control of the double integrator
\begin{equation}
H(z) = \frac{B(z)}{A(z)} = \frac{h^2(z+1)}{2(z-1)^2} = \frac{h^2z + h^2}{2z^2 -4z + 2}
\end{equation}

Proportional control gives closed-loop system
\begin{equation}
H_c(z) = \frac{KH(z)}{1+KH(z)} = \frac{KB(z)}{A(z) + KB(z)}
\end{equation}
with denominator polynomial
\begin{equation}
A(z) + KB(z) = 2z^2 +(-4+Kh^2)z + 2+Kh^2
\end{equation}

```python
a = [Integer(2), -4+h**2*K, 2+h**2*K]
tab= []
juryrec(a,tab)
for line in tab:
    print line

```

    [2, K*h**2 - 4, K*h**2 + 2]
    [K*h**2 + 2, K*h**2 - 4, 2]
    [-(K*h**2/2 + 1)*(K*h**2 + 2) + 2, K*h**2 - (K*h**2/2 + 1)*(K*h**2 - 4) - 4]
    [K*h**2 - (K*h**2/2 + 1)*(K*h**2 - 4) - 4, -(K*h**2/2 + 1)*(K*h**2 + 2) + 2]
    [-(K*h**2/2 + 1)*(K*h**2 + 2) + 2 - (K*h**2 - (K*h**2/2 + 1)*(K*h**2 - 4) - 4)**2/(-(K*h**2/2 + 1)*(K*h**2 + 2) + 2)]

```python
line
```

    [-(K*h**2/2 + 1)*(K*h**2 + 2) + 2 - (K*h**2 - (K*h**2/2 + 1)*(K*h**2 - 4) - 4)**2/(-(K*h**2/2 + 1)*(K*h**2 + 2) + 2)]

```python
e3 = line[0].subs(h,1)
e3
```

    -(K/2 + 1)*(K + 2) + 2 - (K - (K/2 + 1)*(K - 4) - 4)**2/(-(K/2 + 1)*(K + 2) + 2)

```python
solve(e3,K)
```

    []

```python
e1 = tab[2][0].subs(h,1)
```

```python
solve(e1,K)
```

    [-4, 0]

```python
e1
```

    -(K/2 + 1)*(K + 2) + 2

## PD control of the double integrator
The pulse-transfer function for the system obtained by zero-order hold sampling of $G(s)=\frac{1}{s^2}$:
\begin{equation}
H(z) = \frac{B(z)}{A(z)} = \frac{h^2/2(z+1)}{(z-1)^2} = \frac{h^2}{2}\frac{z + 1}{z^2 -2z + 1}
\end{equation}

PD control (simple form) can be implemented using the filter
\begin{equation}
u(kh) = k_1 e(kh) + k_2 e(kh-h)
\end{equation}
with z-transform
\begin{equation}
R(z)U(z) = zU(z) = zk_1 E(z) + k_2 E(z) = S(z) E(z).
\end{equation}

This gives the closed loop system

\begin{equation}
H_c(z) = \frac{\frac{S(z)}{R(z)} \frac{B(z)}{A(z)} } {1 + \frac{S(z)}{R(z)} \frac{B(z)}{A(z)}} = \frac{S(z)B(z)}{R(z)A(z) + S(z)B(z)}
\end{equation}
with denominator polynomial
\begin{equation}
R(z)A(z) + S(z)B(z) = z(z^2 -2z + 1) + h^2/2(k_1z + k_2)(z+1) = z^3 -2z^2 + z + h^2/2\big(k_1z^2 + (k_1+k_2)z + k_2\big) = z^3 + (h^2/2k_1 - 2)z^2 + (1 + h^2/2(k_1+k_2)z
\end{equation}

```python
2/np.log(2)
```

    2.8853900817779268

```python
np.arccos(0.75)
```

    0.72273424781341566

```python
0.72/np.log(2)
```

    1.0387404294400537

```python
aa1 = np.array([9,-11.81,3.15])
tab=[]
juryrec(aa1, tab)
print "Coefficients"
print aa1
for line in tab:
    print line
np.roots(aa1)

```

    Coefficients
    [  9.   -11.81   3.15]
    [  9.   -11.81   3.15]
    [  3.15 -11.81   9.  ]
    [7.8975, -7.6765000000000008]
    [-7.6765000000000008, 7.8975]
    [0.43581563786008104]

    array([ 0.93980424,  0.37241798])

```python
tab[2][1]/tab[2][0]
```

    -0.97201646090534988

```python

```

```python

```


# Autodiff: Calculus  from another angle

We know differentiation

Then, how to differentiate using computer?

##  1. Numerical differentiation
    Finite Difference (FVM, FEM..)

i.e.

\begin{equation}
    f(x) = x^2 + 2x - 1
\end{equation}

Then $\dfrac{df}{dx}$?
we know the analytic solution is
\begin{equation}
    f'(x) = 2x + 2
\end{equation}

```julia
using Plots
gr()
#plotly()
n = 100
@. f(x) = x^2 + 2x - 1
@. fp(x) = 2x + 2
x = range(-5, 5, length=n+1)
u = f(x)
up = fp(x[2:n])
dudx = [(u[i + 1] -  u[i - 1]) / (2*(x[2] - x[1])) for i=2:n]

plot(x[2:n], dudx, linewidth=8,  alpha=0.5, label="FDM")
plot!(x[2:n], up, linewidth=2,  alpha=1, label="Analytic")
```

* Pros : Simple implmenetation
* Cons : Not accurate if high order or few grid points

## 2. Symbolic Differentiation

Mathematica way

```julia
#import Pkg; Pkg.add("SymPy")
using SymPy
```

```julia

@vars x
diff(cos(x), x)
```

```julia
diff( x^2 + 2x - 1, x)
```

* Pros : Accurate
* Cons : Very slow or even not applicable to complex function

## 3. Auto differentiation

Let's start with $\sqrt{x}$

## Babylonian sqrt

1. Make an initial guess. $x_0$
2. Improve the guess. $x1 = (x0 + S / x0) / 2$
3. Iterate until convergence. $x_{n+1} = (x_n + S / x_n) /2 $

More algorithmic way

 > Repeat $ t \leftarrow  (t+x/t) / 2 $ until $t$ converges to $\sqrt{x}$.

 Next Each iteration has one add and two divides. For illustration purposes, 10 iterations suffice.

```julia
function Babylonian(x; N = 10)
    t = (1+x)/2
    for i = 2:N; t=(t + x/t)/2  end
    t
end
```

    Babylonian (generic function with 1 method)

Check that it works:

```julia
α = π
Babylonian(α), √α
```

    (1.7724538509055159, 1.7724538509055159)

```julia
x=2; Babylonian(x),√x  # Type \sqrt+<tab> to get the symbol
```

    (1.414213562373095, 1.4142135623730951)

```julia
using Plots
gr()
```

    Plots.GRBackend()

```julia
## Warning first plots load packages, takes time
i = 0:.01:49

plot([x->Babylonian(x,N=i) for i=1:5],i,label=["Iteration $j" for i=1:1,j=1:5])

plot!(sqrt,i,c="black",label="sqrt",
      title = "Those Babylonians really knew how to √")
```

## ...and now the derivative, almost by magic

we have a derivatve of `sqrt(x)`

\begin{equation}
(\sqrt{x})' = (x^{\frac{1}{2}})' = \dfrac{1}{2 \sqrt{x}}
\end{equation}

Introduce Dual number,  invented by the famous algebraist Clifford in 1873, as `D`.

```julia
struct D <: Number  # D is a function-derivative pair
    # D = (Number (func), Number (derivative))
    f::Tuple{Float64,Float64}
end
```

* Sum Rule
\begin{equation}
    (x+y)' = x' + y'
\end{equation}

* Quotient Rule
\begin{equation}
(x/y)' = (yx'-xy') / y^2
\end{equation}

```julia
import Base: +, /, convert, promote_rule
+(x::D, y::D) = D(x.f .+ y.f)
/(x::D, y::D) = D((x.f[1]/y.f[1], (y.f[1]*x.f[2] - x.f[1]*y.f[2])/y.f[1]^2))
convert(::Type{D}, x::Real) = D((x,zero(x)))
promote_rule(::Type{D}, ::Type{<:Number}) = D
```

    promote_rule (generic function with 160 methods)

The same algorithm with no rewrite at all computes properly
the derivative as the check shows.

```julia
x=49; Babylonian(D((x,1))), (√x,.5/√x)
```

    (D((7.0, 0.07142857142857142)), (7.0, 0.07142857142857142))

```julia
x=π; Babylonian(D((x,1))), (√x,.5/√x)
```

    (D((1.7724538509055159, 0.28209479177387814)), (1.7724538509055159, 0.28209479177387814))

## It just works!

## Symbolically

We haven't yet explained how it works, but it may be of some value to understand that the below is mathematically
equivalent, though not what the computation is doing.

Notice in the below that Babylonian works on SymPy symbols.

Note: Python and Julia are good friends.  It's not a competition!  Watch how nicely we can use the same code now with SymPy.

```julia
#Pkg.add("SymPy")
using SymPy
```

```julia
x = symbols("x")
display("Iterations as a function of x")
for k = 1:5
 display( simplify(Babylonian(x,N=k)))
end

display("Derivatives as a function of x")
for k = 1:5
 display(simplify(diff(simplify(Babylonian(x,N=k)),x)))
end
```

## How autodiff is getting the answer

1. Create a basic rule (known rule) for Dual (Function & Derivate)
2. Let code works

Let us by hand take the "derivative" of the Babylonian iteration wi
th respect to x. Specifically t′=dt/dx.  This is the old fashioned way of a human rewriting code.

```julia
function dBabylonian(x; N = 10)
    t = (1+x)/2
    t′ = 1/2
    for i = 1:N;
        t = (t+x/t)/2;
        t′= (t′+(t-x*t′)/t^2)/2;
    end
    t′

end
```

    dBabylonian (generic function with 1 method)

See this rewritten code gets the right answer.  So the trick is for the computer system to do it for you, and without any loss of speed or convenience.

```julia
x = π; dBabylonian(x), .5/√x
```

    (0.2820947917738782, 0.28209479177387814)

What just happened?  Answer: We created an iteration by hand for t′ given our iteration for t. Then we ran the iteration alongside the iteration for t.

```julia
Babylonian(D((x,1)))
```

    D((1.7724538509055159, 0.28209479177387814))

How did this work?  It created the same derivative iteration that we did by hand, using very general rules that are set once and need not be written by hand.

## Dual Number Notation

Instead of D(a,b) we can write a + b ϵ, where ϵ satisfies ϵ^2=0.

The four rules are

$ (a+b\epsilon) \pm (c+d\epsilon) = (a+c) \pm (b+d)\epsilon$

$ (a+b\epsilon) * (c+d\epsilon) = (ac) + (bc+ad)\epsilon$

$ (a+b\epsilon) / (c+d\epsilon) = (a/c) + (bc-ad)/d^2 \epsilon $

```julia
Base.show(io::IO,x::D) = print(io,x.f[1]," + ",x.f[2]," ϵ")
```

```julia
# Add the last two rules
import Base: -,*
-(x::D, y::D) = D(x.f .- y.f)
*(x::D, y::D) = D((x.f[1]*y.f[1], (x.f[2]*y.f[1] + x.f[1]*y.f[2])))
```

    * (generic function with 394 methods)

```julia
D((1,0))
```

    1.0 + 0.0 ϵ

```julia
D((0,1))^2
```

    0.0 + 0.0 ϵ

```julia
D((2,1)) ^2
```

    4.0 + 4.0 ϵ

```julia
ϵ * ϵ
```

    0.0 + 0.0 ϵ

```julia
ϵ^2
```

    0.0 + 0.0 ϵ

```julia
1/(1+ϵ)  # Exact power series:  1-ϵ+ϵ²-ϵ³-...
```

    1.0 + -1.0 ϵ

```julia
(1+ϵ)^5 ## Note this just works (we didn't train powers)!!
```

    1.0 + 5.0 ϵ

## Generalization to arbitrary roots

```julia
function nthroot(x, n=2; t=1, N = 10)
    for i = 1:N;   t += (x/t^(n-1)-t)/n; end
    t
end
```

    nthroot (generic function with 2 methods)

```julia
nthroot(2,3), ∛2 # take a cube root
```

    (1.2599210498948732, 1.2599210498948732)

```julia
nthroot(2+ϵ,3)
```

    1.2599210498948732 + 0.20998684164914552 ϵ

```julia
nthroot(7,12), 7^(1/12)
```

    (1.1760474285795146, 1.1760474285795146)

```julia
x = 2.0
nthroot( x+ϵ,3), ∛x, 1/x^(2/3)/3
```

    (1.2599210498948732 + 0.20998684164914552 ϵ, 1.2599210498948732, 0.20998684164914552)

## Forward Diff
Now that you understand it, you can use the official package

```julia
using ForwardDiff
```

```julia
ForwardDiff.derivative(sqrt, 2)
```

    0.35355339059327373

```julia
ForwardDiff.derivative(Babylonian, 2)
```

    0.35355339059327373

# How Frameworks compute this

Let function $f$ as
\begin{equation}
f(x_1, x_2) = \cos{(x_1)} + x_1 \exp{(x_2)}
\end{equation}

## Derivative Expression of $f$

\begin{align}
w_1 &= x_1 \\
w_2 &= x_2 \\
w_3 &= \exp{(w_2)} \\
w_4 &= w_1 w_3 \\
w_5 &= \cos{(w_1)} \\
w_6 &= w_4 + w_5 \\
f(x_1, x_2) &= w_6
\end{align}

## Computation Graph for function

\begin{align}
w_1' &= \textrm{seed} \in {0, 1} \\
w_2' &= \textrm{seed} \in {0, 1} \\
w_3' &= \exp{(w_2)} w_2' \\
w_4' &= w_1' w_3 + w_1 w_3' \\
w_5' &= -\sin{(w_1)} w_1' \\
w_6' &= w_4' + w_5' \\
f'(x_1, x_2) &= w_6'
\end{align}

## Computational Graph for derivatives

## LLVM Ecosystem

## Tensorflow Compiler Ecosystem

# Conclusion

1. There is some different method to implement computational derivatives
2. Autodifferentiation is not only for machine learning, it could be used various way such as optimal control.
3. Knowing lower level is not manadatory, but this helps to understand whole architecture and create better code
4. Switching langauge is doesn't matter, but only applied for modern language.

##  Reference

* https://github.com/JuliaComputing/JuliaBoxTutorials/blob/master/introductory-tutorials/broader-topics-and-ecosystem/intro-to-ml/20.%20Automatic%20Differentiation%20in%2010%20Minutes.ipynb
* https://www.cs.usask.ca/~spiteri/CMPT898/notes/ad.pdf
* https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec10.pdf