```
from sympy import *
init_printing(use_latex=True)
from IPython.display import display
from numpy import array
from scipy.linalg import eigh
from IPython.display import HTML
HTML(open("00_custom.css", "r").read())
```

<style>
@font-face {
    font-family: 'Charis SIL';
    src: url('fonts/CharisSILEur-R.woff') format('woff');
}
@font-face {
    font-family: 'Architects Daughter';
    font-style: normal;
    src: url(https://fonts.gstatic.com/s/architectsdaughter/v6/RXTgOOQ9AAtaVOHxx0IUBM3t7GjCYufj5TXV5VnA2p8.woff2) format('woff2');
    unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215, U+E0FF, U+EFFD, U+F000;
}
.prompt{display: None;}
div.cell{margin: auto;width:900px;}

div #notebook { /* centre the content */
    background: #fff9f0;
    margin: auto;
    padding: 0em;
    padding-top: 1em;
}
div #notebook_container {width: 960px!important;}
#notebook li { /* More space between bullet points */
    margin-top:0.2em;
}

div.text_cell_render{
    font-family: "Charis SIL", Cambria,  serif;
    line-height: 155%;
    font-size: 130%;
}
.CodeMirror{
    font-family: Consolas, monospace;
    font-size: 140%;
    background-color:#fffcf6!important;
}

.output_area {
    font-family: Consolas,monospace;font-size: 120%;
    background-color:#fffcf6!important;
    margin-top:0.8em;
}

div.output_latex{overflow:hidden}

h1 {
    font-family: 'Architects Daughter', serif;
    font-size:   48pt!important;
    text-align:  center;
    text-shadow: 4px 4px 4px #aaa;
    padding-bottom: 48pt;
}

.warning{color: rgb( 240, 20, 20 )}
</style>

# Rayleigh Quotient

We are going to use a little symbolic algebra, hence we'll need the following base symbols:

```
x1, x2, x3, k, L, m = symbols('x1, x2, x3, k, L, m')
```

The single degrees of freedom have to be collected in a vector,

```
x = Matrix((x1,x2,x3))
x
```

We want to compute the (double of the) strain energy, to do so we need  the relative rotations, that we can compute if we know in advance the rotations of each individual bar...

```
phi1, phi2, phi3 = x1/L, (x2-x1)/L, (x3-x2)/L
phi21, phi32 = phi2-phi1, phi3-phi2
W2 = x2**2*k + phi21**2*k*L*L + phi32**2*k*L*L
print latex((W2/k).expand())
W2.expand()
```

By comparison with the previuous expression, we have the stiffness matrix

```
K = k*Matrix(((5,-4,1),(-4,6,-2),(1,-2,1)))
K
```

To compute the (double of the) kinetic energy, we have to compute the rotatory inertia and the displacements of the centers of the bars, as we have already computed the bars' rotations,

```
J = m*L*L/12
xg1, xg2, xg3 = x1/2, (x1+x2)/2, (x2+x3)/2
T2 = m*(xg1**2+xg2**2+xg3**2) + J*(phi1**2+phi2**2+phi3**2)
print latex((T2/m).expand())
T2.expand()
```

Again, by comparison with the kinetic energy we can derive the mass matrix,

```
M = m*Matrix(((4,1,0),(1,4,1),(0,1,2)))/6
M
```

Having computed, as requested, the structural matrices we are ready for the Rayleigh estimates, but first we define a trial vector and an auxiliary matrix, that we'll use in the following to simplify our lengthy expressions

```
x0 = Matrix((1,2,4))
D = K.inv()*M
```

Here are the expression for the energies, normalized with respect to $Z_0^2/2$

```
w2 = 1
V0 = (x0.T*K*x0)[0] *     w2**0
T0 = (x0.T*M*x0)[0] *     w2**1
V1 = (x0.T*M*D*x0)[0] *   w2**2
T1 = (x0.T*M*D*D*x0)[0] * w2**3
V0, T0, V1, T1
```

The Rayleigh estimates are as follows, fractions first and then floating point numbers

```
display(V0/T0, T0/V1, V1/T1)
map(N,(V0/T0, T0/V1, V1/T1))
```

Just to be sure, a check with the results of a library function...

```
for x in eigh(
    array(((5,-4,1),(-4,6,-2),(1,-2,1)))/1.,
    array(((4,1,0),(1,4,1),(0,1,2)))/6., eigvals=(0,0)): print x
```

    [ 0.40540716]
    [[ 0.2843138 ]
     [ 0.60585859]
     [ 1.11964382]]


```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import scipy.special
import scipy.ndimage
import scipy.optimize
import sklearn.datasets

from chmp.ds import mpl_set
```

```python
# helper for gradient checking
def approximate_gradient(x, func, eps=1e-5):
    res = np.zeros(x.size)

    for i in range(x.size):
        d = np.zeros(x.size)
        d[i] = eps

        res[i] = (func(x + d) - func(x - d)) / (2 * eps)

    return res
```

# Kernel Methods

## Dual Representations

$$
\begin{align}
    J
    &= \frac{1}{2} \sum_n \left( w^T \phi(x_n) - t_n \right)^2 + \frac{\lambda}{2} w^T w
\\
    &= \frac{1}{2} \left( \Phi w - t  \right)^T \left( \Phi w - t \right) + \frac{\lambda}{2} w^T w
\end{align}
$$

with $\Phi_{ni} = \phi_i(x_n)$. Further define $w = \Phi^T a$. That this is always possible can be shown, by differentiating $J$ with respect to $w$. At optimality one finds:

$$
\begin{align}
    0 &= \left( w^T \Phi^T - t^T \right) \Phi + \lambda w \\
    w &= -\frac{1}{\lambda} \left( w^T \Phi^T - t^T \right) \Phi \\
    w &= -\frac{1}{\lambda} \Phi^T \left( \Phi w - t \right) \\
    w &= \Phi^T a
\end{align}
$$

Inserting $w = \Phi^T a$ into the objective function results in:

$$
\begin{align}
    J
    &=
        \frac{1}{2} \left( \Phi \Phi^T a - t  \right)^T \left( \Phi \Phi^T a - t \right) +
        \frac{\lambda}{2} a^T \Phi \Phi^T a
\\
    &=
        \frac{1}{2} a^T \Phi \Phi^T \Phi \Phi^T  a -
        a^T \Phi \Phi^T t +
        \frac{1}{2} t^T t +
        \frac{\lambda}{2} a^T \Phi \Phi^T a
\\
    &=
        \frac{1}{2} a^T K K  a - a^T K t + \frac{1}{2} t^T t + \frac{\lambda}{2} a^T K a
\end{align}
$$

with $K = \Phi \Phi^T$. Searching for the optimal $a$ yields

$$
\begin{align}
    0 &= K K  a - K t + \lambda K a \\
    0 &= K ((K + \lambda) a - t) \\
    a &= (K + \lambda)^{-1} t
\end{align}
$$

Note, that $K$ has the form:

$$
    K_{mn} = \sum_i \phi_i(x_m) \phi_i(x_n) = k(x_m, x_n).
$$

Now, the number size of the problem depends only on the number of training examples, not the number of features. This allows to introduce abitrary many features, even infinitely many via kernels.

The predicted values are given by

$$
\begin{align}
    \hat{t} &= w^T \phi(x) \\
        &= a^T \Phi \phi(x) \\
        &= \sum_{ni} a_n \phi_i(x_n) \phi_i(x) \\
        &= \sum_n a_n k(x_n, x)
\end{align}
$$

```python
x, y = sklearn.datasets.make_regression(n_features=10)

K = x @ x.T
a = np.linalg.pinv(K + 1e-2) @ y
w_opt = x.T @ a

print(K.shape, a.shape, w_opt.shape)
print('mean squared error:', np.mean((y - x @ w_opt) ** 2.0))
```

    (100, 100) (100,) (10,)
    mean squared error: 2.701454169929355e-26

## Constructing Kernels

**TODO: summarize examples mentioned**

## Radial Basis Function networks

```python
np.random.seed(13)
x = np.random.uniform(low=0, high=2 * np.pi, size=50)
x = np.sort(x)
y = np.random.normal(np.cos(x), scale=0.1)

u = np.linspace(0, 2 * np.pi, 100)

# build the kernel
w = np.exp(-0.5 * (u[:, None] - x[None, :]) ** 2 / (0.25))
w = w / w.sum(axis=1, keepdims=True)
```

```python
plt.figure(figsize=(8, 3))
plt.subplot(1, 2, 1)
plt.plot(x, y, '.')
plt.plot(u, w @ y)

plt.subplot(1, 2, 2)
plt.pcolor(x, u, w)
mpl_set(colorbar=True, xlabel='Data position', ylabel='Interpolated position', title='Weight')
```

## Gaussian processes

### Regression

Basic idea: postulate covraiance structure between data points via:

$$
    p(t|\{x\}) = \mathcal{N}(t|0, C)
$$

with $C_{ij} = \beta^{-1} \delta_{ij} + k(x_i, x_j)$ with the inverse observation noise variance $\beta$.

For prediction, use the same structure for the joint $p(t_{n+1}, \{t\}|x_{n+1}, \{x\}) = \mathcal{N}(t| 0, C) $ and compute the conditional:

$$
\begin{align}
    p(t_{n+1}|\{t\}, \{x\}) &= \mathcal{N}(t_{n+1}|m_{n+1}, \sigma_{n+1}), \\
    m_{n+1} &= k^T C_N^{-1} t, \\
    \sigma_{n+1} &= c - k^T C_N^{-1} k,
\end{align}
$$

where $C_N, k, c$ form the full covariance matrix as in

$$
C = \begin{pmatrix}
    C_N & k^T \\
    k & c
\end{pmatrix}.
$$

```python
k = np.exp(-1 * abs(x[:, None] - x[None, :]))
```

```python
s = np.real(scipy.linalg.sqrtm(k))
```

```python
plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
for _ in range(5):
    plt.plot(x, s @ np.random.normal(size=s.shape[0]))

plt.subplot(1, 2, 2)
plt.pcolor(x, x, k)
```

```python
np.random.seed(13)
x = np.random.uniform(low=0, high=2 * np.pi, size=50)
x = np.sort(x)
y = np.random.normal(np.cos(x), scale=0.1)
```

```python
xi = np.linspace(0, 2 * np.pi, 100)
```

```python
alpha = 0.25
beta = 10.
```

```python
c_N = (1.0 / beta) * np.eye(x.size) + np.exp(-alpha * abs(x[:, None] - x[None, :]) ** 2.0)
k = np.exp(-alpha * abs(x[:, None] - xi[None, :]) ** 2.0)
c = (1.0 / beta) + np.diag(np.exp(-alpha * abs(xi[:, None] - xi[None, :]) ** 2.0))
```

```python
m = k.T @ np.linalg.pinv(c_N) @ y
s = c - np.diag(k.T @ np.linalg.pinv(c_N) @ k)
```

```python
plt.plot(xi, m)
plt.fill_between(xi, m - np.sqrt(s), m + np.sqrt(s), alpha=0.2)
plt.plot(x, y, 'o')
```

The hyperparameters (here, $\alpha, \beta$) can for example be optimized by maximum likelihood. The log likelihood of the data is given by

$$
    \mathcal{L} = \log p(\{t\}| \{x\}) = -\frac{1}{2} |C_N| - \frac{1}{2} t^T C_N^{-1} t - \frac{N}{2} \log 2 \pi
$$

By using different lengthscales for different dimensions of $x$, automatic relevance determination can be implemented.

```python
def loglikelihood(params):
    alpha, beta = params
    alpha = np.log(1 + np.exp(alpha))
    beta = np.log(1 + np.exp(beta))
    c_N = (1.0 / beta) * np.eye(x.size) + np.exp(-alpha * abs(x[:, None] - x[None, :]) ** 2.0)
    return (y.T @ np.linalg.pinv(c_N) @ y + np.linalg.det(c_N)) / x.size

def fit_gp_params():
    res = scipy.optimize.minimize(loglikelihood, [0, 0])
    if not res.success:
        raise RuntimeError(res.message)
    return np.log(1 + np.exp(res.x))

```

```python
alpha, beta = fit_gp_params()
print(alpha, beta)
```

    0.5649704149113319 1.4485272009115577

### Classification

Define probability of target in terms of logits $a$:

$$
    p(t|a) = \sigma(a)^t (1 - \sigma)^{(1 - t)}
$$

Assume the logits are distributed according to a Gaussian process:

$$
\begin{align}
    a &\sim \mathcal{GP} \\
    p(a) &= \mathcal{N}(a|0, C)
\end{align}
$$

Divide the covariance matrix into:

$$
    C_{m,n} = k(x_m, x_n) + \nu \delta_{mn}
$$

Now introduce a fully observed set $\{x, t\}$ and make a prediction for a new point $x_{N + 1}$

The new logits are distributed according to:

$$
    p(a_{N+1}|\{x, t\}) = \int \mathrm{d}a_N\; p(a_{N+1}|a_N) p(a_N|\{x, t\})
$$

Further:

$$
\begin{align}
    \log p(a_N|\{x, t\})
    &= \log p(\{t\}|a_N) + \log p(a_N|\{x\}) + \mathrm{const}
\\
    &=
        -\frac{1}{2} a_N^T C_N^{-1} a_N +
        \sum_n t_n \log \sigma(a_n) +
        \sum_n (1 - t_n) \log (1 - \sigma(a_n))
\end{align}
$$

This density is intractable to compute. Use Laplace approximation:

$$
\begin{align}
    \frac{\partial}{\partial a_{Ni}} \log p(a_N|\{x, t\})
    &= - \sum_j C_{Nij}^{-1} a_j + t_i (1 - \sigma(a_i)) - (1 - t_i) \sigma(a_i)
\\
    &= - \sum_j C_{Nij}^{-1} a_j + t_i - \sigma(a_i)
\\
    \frac{\partial^2}{\partial a_{Ni} \partial a_{Nj}} \log p(a_N|\{x, t\})
    &= - C_{Nij}^{-1} - \sigma(a_i) (1 - \sigma(a_i)) \delta_{ij}
\end{align}
$$

Given the Laplace approximation $p(a_{N+1}|\{x, t\})$ can be evaluated.

```python
# TODO: implement
```

```python

```


```
import os
import sys
from matplotlib import pyplot as plt
import numpy as np
from datadrivenpdes.core import equations
from datadrivenpdes.core import grids
import datadrivenpdes as pde
import tensorflow as tf
tf.enable_eager_execution()
import xarray as xr
```

# First example: Advection diffusion

In this example we'll see how to integrate in time a pre-defined equation. Here we deal with the Advection-Diffusion equation, which describes the time evolution of the concentration $c(x,y,t)$ when it is advected by the velocity field $\vec v(x,y)=(v_x(x,y), v_y(x,y)$ and also undergoes diffusion. The equation reads
$$\frac{\partial c}{\partial t}+\vec{v}\cdot\vec{\nabla}c= D \nabla^2 c$$
where $D$ is the diffusion coefficient. The equation is implemented in various forms in the folder `advection/equations`. Here we choose the Finite Volume formulation.

```
equation = pde.advection.equations.FiniteVolumeAdvectionDiffusion(diffusion_coefficient=0.01)
grid = grids.Grid.from_period(size=256, length=2*np.pi)
```

Note that we also chose a grid to solve the equation on. The $x$ and $y$ coordinates can be obtained by

```
x, y = grid.get_mesh()
```

To integrate in time we need an initial state. Equations instances have a `random_state` method that generates a state. The distribution of these initial conditions, when sampled from different seeds, will define the training set for later. Let's sample one random initial state and plot it:

```
initial_state = equation.random_state(grid, seed=7109179)
fig, axs = plt.subplots(1,2, figsize=(8,4))
axs[0].pcolor(grid.get_mesh()[1],
              grid.get_mesh()[0],
              initial_state['concentration'])
axs[0].set_title('initial concentration')

axs[1].streamplot(grid.get_mesh()[1],
                  grid.get_mesh()[0],
                  initial_state['y_velocity'],initial_state['x_velocity'],
                  density=2)
axs[1].set_title('velocity field');
```

The state of an equation is a `dict` object that contains all relevant fields needed for integrating in time. For advection diffusion these are `concentration`, `x_velocity`, and  `y_velocity`:

```
print(initial_state.keys())
```

    dict_keys(['concentration', 'x_velocity', 'y_velocity'])

To perform the actual integration we need to choose a method with which to estimate the spatial derivatives of the concentration $c$. The object which estimates the derivatives is called a `model` and there are various models defined in `models.py`. Here we will use a finite difference estimation. Lastly, we need to choose a timestep, which we can ask the equation instance to supply.

```
time_step = equation.get_time_step(grid)
times = time_step*np.arange(400)
results = pde.core.integrate.integrate_times(
    model=pde.core.models.FiniteDifferenceModel(equation,grid),
    state=initial_state,
    times=times, axis=0)
```

The result is a `dict` object. The `concentration` member of the dict is a tensor whose first axis corresponds to the times at which the solution was evaluated. Here we save the result as an `xarray.DataArray`, which makes it easy to plot.

```
conc=xr.DataArray(results['concentration'].numpy(),
                  dims=['time', 'x','y'],
                  coords={'time':times, 'x': x[:,0], 'y': y[0]}
                 )
conc[::99].plot(col='time', robust=True, aspect=1)
```

# Defining a new equation

In this section we learn how to define a new equation. We will look at coupled reaction diffusion equations, aka the **Turing Equation**. They decscribe the evolution of two fields, $A$ and $B$, according to:
$$\begin{align}
\frac{\partial A}{\partial t} &= D_A\nabla^2 A + R_A(A,B)+S\\
\frac{\partial B}{\partial t} &= D_B\nabla^2 B + R_B(A,B)
\end{align}$$
$D_{A,B}$ are the diffusion constants of $A$ and $B$, $R_{A,B}$ are nonlinear reaction terms and $S$ is some constant source term. For example, we'll take
$$\begin{align}
R_A&=A(1-A^2)-\alpha B &
R_B&=\beta(A-B)
\end{align}$$
where $\alpha$ and $\beta$ are model parameters. For simplicity, we'll implelment the equation in one spatial dimension.

## Equation Keys
Because the computational framework is [fully differentiable](https://en.wikipedia.org/wiki/Differentiable_programming), defining an equation requires specifiying in advance what are the quantities that are used in calcualting time derivatives. These are called **keys** and are stored in the `equation` attribute `key_definitions`. In our case, to calculate the time evolution we need $A, B, \partial_{xx}A, \partial_{xx}B $ and $S$.

The auxilliary function `states.StateDefinition` defines these keys. Its input arguments are:
* `name` - The base name of the field. For example, the field $\partial_{xx} A$ is derived from the base field `A`.
* `tensor_indices` - In 2D and above, specify whether a field is a component of a tensor (like $v_x$ and $v_y$ in the advection example).
* `derivative_orders` - Specifies whether a key is a spatial derivative of a different key.
* `offset` - Specifies whether a field is evaluated off the center point of a grid (useful for staggered grids, e.g. finite volume schemes)

For example, in our case the `key_definitions` for $A$ and $\partial_{xx}A$ are

```python
key_definitions = {
    'A': states.StateDefinition(name='A',
                  tensor_indices=(),          # Not used in one dimenional equations
                  derivative_orders=(0,0,0),  # A is not a derivative of anything else
                  offset=(0,0)),              # A is evaluated on the centerpoints of the grid

    'A_xx': states.StateDefinition(name='A',       # A_xx is is derived from A
                     tensor_indices=(),
                     derivative_orders=(2, 0, 0),  # Two derivatives on the x axis
                     offset=(0, 0)),
  }
```
There are two types of keys: those that evolve in time, in our case $A$ and $B$, and constant ones, in our case $S$ (and in the Advection Diffusion example - the velocity field $v$). When defining the equation we need to set the attributes `evolving_keys` and `constant_keys`, which are both python `set`s.

The different keys of an `Equation` instance can be inspected with
```python
equation.all_keys        # in our case: {'A', 'A_xx', 'B', 'B_xx', 'Source'}
equation.base_keys       # in our case: {'A', 'B', 'Source'}
equation.evolving_keys   # in our case: {'A', 'B'}
equation.constant_keys   # in our case: {'Source'}
```

## Defining the equation

Here is a full definition of the equation:

```
from datadrivenpdes.core import equations
from datadrivenpdes.core import grids
from datadrivenpdes.core import polynomials
from datadrivenpdes.core import states
import scipy as sp

def smooth_random_field(N, amp=0.1, np_random_state=None):
    """
    generates a random field of shape (N,1) and smoothes it a bit
    """
    if np_random_state is None:
      np_random_state = np.random.RandomState()
    noise=np_random_state.randn(N)
    kernel=np.exp(-np.linspace(-6,6,N)**2)
    return amp*sp.ndimage.convolve(noise, kernel, mode='wrap')[:,np.newaxis]

class TuringEquation(equations.Equation):

    DISCRETIZATION_NAME = 'finite_difference'
    METHOD = polynomials.Method.FINITE_DIFFERENCE
    MONOTONIC = False
    CONTINUOUS_EQUATION_NAME = 'Turing'

    key_definitions = {
        'A': states.StateDefinition(name='A',
                      tensor_indices=(),
                      derivative_orders=(0,0,0),
                      offset=(0,0)),
        'A_xx': states.StateDefinition(name='A',
                         tensor_indices=(),
                         derivative_orders=(2, 0, 0),
                         offset=(0, 0)),
        'B': states.StateDefinition(name='B',
                      tensor_indices=(),
                      derivative_orders=(0, 0, 0),
                      offset=(0, 0)),
        'B_xx': states.StateDefinition(name='B',
                         tensor_indices=(),
                         derivative_orders=(2, 0, 0),
                         offset=(0, 0)),
        'Source' : states.StateDefinition(name='Source',
                         tensor_indices=(),
                         derivative_orders=(0, 0, 0),
                         offset=(0, 0)),
    }
    evolving_keys = {'A', 'B'}
    constant_keys = {'Source'}

    def __init__(self, alpha, beta, D_A, D_B, timestep=1e-4):
        self.alpha = alpha
        self.beta = beta
        self.D_A = D_A
        self.D_B = D_B
        self._timestep = timestep
        super().__init__()

    def time_derivative(
          self, grid, A, A_xx, B, B_xx, Source):
        """See base class."""
        rA = self.reaction_A(A, B)
        rB = self.reaction_B(A, B)
        diff_A = self.D_A * A_xx
        diff_B = self.D_B * B_xx
        return {'A': rA + diff_A + Source,
                'B': rB + diff_B,}

    def reaction_A(self, A, B):
        return A - (A ** 3) - B + self.alpha

    def reaction_B(self, A, B):
        return (A - B) * self.beta

    def get_time_step(self, grid):
        return self._timestep

    def random_state(self, grid, seed=None, dtype=tf.float32):
        if seed is None:
            R = np.random.RandomState()
        else:
            R = np.random.RandomState(seed=seed)

        state = {
          'A': smooth_random_field(N=grid.size_x, np_random_state=R),
          'B': smooth_random_field(N=grid.size_x, np_random_state=R),
          'Source': smooth_random_field(N=grid.size_x, np_random_state=R),
        }

        state = {k: tf.cast(v, dtype) for k, v in state.items()}
        return state
```

Now we can generate a random state and evolve it in time

```
eq = TuringEquation(alpha=-0.0001, beta=10, D_A=1, D_B=30)
NX=100
NY=1 # 1D can be obtained by haveing a y dimension of size 1
LX=200
grid = grids.Grid(NX, NY, step=LX/NX)
x, y=grid.get_mesh()

initial_state = eq.random_state(grid=grid, seed=12345)
times = eq._timestep*np.arange(0, 1000, 20)
model = pde.core.models.FiniteDifferenceModel(eq,grid)
res = pde.core.integrate.integrate_times(
    model=model,
    state=initial_state,
    times=times, axis=0)
```

```
fig, axs=plt.subplots(1,2, figsize=(10,5), sharey=True)
for ax, k in zip(axs, ['A','B']):
    ax.pcolormesh(x.flat, times, res[k].numpy()[...,0], cmap='RdBu')
    ax.set_title(k)
    ax.set_xlabel('x')
axs[0].set_ylabel('time')
fig.tight_layout()
```


```python
!pip install pandas
import sympy as sym
import numpy as np
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt

sym.init_printing()
```

    Requirement already satisfied: pandas in c:\users\usuario\.conda\envs\sys\lib\site-packages (1.1.2)
    Requirement already satisfied: pytz>=2017.2 in c:\users\usuario\.conda\envs\sys\lib\site-packages (from pandas) (2020.1)
    Requirement already satisfied: numpy>=1.15.4 in c:\users\usuario\.conda\envs\sys\lib\site-packages (from pandas) (1.19.1)
    Requirement already satisfied: python-dateutil>=2.7.3 in c:\users\usuario\.conda\envs\sys\lib\site-packages (from pandas) (2.8.1)
    Requirement already satisfied: six>=1.5 in c:\users\usuario\.conda\envs\sys\lib\site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)

## Correlación

La correlación entre las señales $f(t)$ y $g(t)$ es una operación que indica cuán parecidas son las dos señales entre sí.

\begin{equation}
(f \; \circ \; g)(\tau) = h(\tau) = \int_{-\infty}^{\infty} f(t) \cdot g(t + \tau) \; dt
\end{equation}

Observe que la correlación y la convolución tienen estructura similares.

\begin{equation}
f(t) * g(t) = \int_{-\infty}^{\infty} f(\tau) \cdot g(t - \tau) \; d\tau
\end{equation}

## Señales periódicas

La señal $y(t)$ es periódica si cumple con la condición $y(t+nT)=y(t)$ para todo $n$ entero. En este caso, $T$ es el periodo de la señal.

La señal seno es la oscilación más pura que se puede expresar matemáticamente. Esta señal surge al considerar la proyección de un movimiento circular uniforme.

## Serie de Fourier

Si se combinan apropiadamente un conjunto de oscilaciones puras, como combinaciones lineales de señales desplazadas y escaladas en tiempo y amplitud, podría recrearse cualquiér señal periódica. Esta idea da lugar a las series de Fourier.

\begin{equation}
y(t) = \sum_{n=0}^{\infty} C_n \cdot cos(n \omega_0 t - \phi_n)
\end{equation}

La señal $y(t)$ es igual a una combinación de infinitas señales coseno, cada una con una amplitud $C_n$, una frecuencia $n \omega_0$ y un desfase $\phi_n$.

También puede expresarse como:

\begin{equation}
y(t) = \sum_{n=0}^{\infty} A_n \cdot cos(n \omega_0 t) + B_n \cdot sin(n \omega_0 t)
\end{equation}

La serie queda definida si se encuentran los valores apropiados de $A_n$ y $B_n$ para todos los valores de $n$.

Observe que:
- $A_n$ debe ser más grande si $y(t)$ se "parece" más a un cos.
- $B_n$ debe ser más grande si $y(t)$ se "parece" más a un sin.

\begin{equation}
y(t) = \sum_{n=0}^{\infty} A_n \cdot cos(n \omega_0 t) + B_n \cdot sin(n \omega_0 t)
\end{equation}

\begin{equation}
(f \; \circ \; g)(\tau) = \int_{-\infty}^{\infty} f(t) \cdot g(t + \tau) \; dt
\end{equation}

\begin{equation}
(y \; \circ \; sin_n)(\tau) = \int_{-\infty}^{\infty} y(t) \cdot sin(n \omega_0(t + \tau)) \; dt
\end{equation}

Considerando:
- $\tau=0$ para no incluir desfases.
- la señal $y(t)$ es periódica con periodo $T$.

\begin{equation}
(y \; \circ \; sin_n)(0) = \frac{1}{T} \int_{0}^{T} y(t) \cdot sin(n \omega_0 t) \; dt
\end{equation}

Esta expresión puede interpretarse como el parecido de una señal $y(t)$ a la señal $sin$ con crecuencia $n \omega_0$ promediado a lo largo de un periodo sin desfase del seno.

Retomando la idea inicial

\begin{equation}
y(t) = \sum_{n=0}^{\infty} A_n \cdot cos(n \omega_0 t) + B_n \cdot sin(n \omega_0 t)
\end{equation}

donde
\begin{equation}
A_n = \frac{1}{T} \int_{0}^{T} y(t) \cdot cos(n \omega_0 t) \; dt
\end{equation}

\begin{equation}
B_n = \frac{1}{T} \int_{0}^{T} y(t) \cdot sin(n \omega_0 t) \; dt
\end{equation}

Se recomienda al estudiante que encuentre la relación entre las Series anteriores y la siguiente alternativa para representar la Series de Fourier.

\begin{equation}
y(t) = \sum_{n=-\infty}^{\infty} C_n \cdot e^{j n \omega_0 t}
\end{equation}

donde

\begin{equation}
C_n = \frac{1}{T} \int_{0}^{T} y(t) \cdot e^{j n \omega_0 t} \; dt
\end{equation}

Los valores $C_n$ son el espectro de la señal periódica $y(t)$ y son una representación en el dominio de la frecuencia.

**Ejemplo # 1**

La señal $y(t) = sin(2 \pi t)$ es en sí misma una oscilación pura de periodo $T=1$.

```python
# Se define y como el seno de t
t = sym.symbols('t', real=True)
#T = sym.symbols('T', real=True)
T = 1
nw = sym.symbols('n', real=True)
delta = sym.DiracDelta(nw)
w0 = 2 * sym.pi / T

y = 1*sym.sin(w0*t) + 0.5
# y = sym.sin(w0*t)
# y = (t-0.5)*(t-0.5)
y

```

Aunque la sumatoria de las series de Fourier incluye infinitos términos, solamente se tomaran las primeras 3 componentes.

```python
n_max = 5

y_ser = 0
C = 0
ns = range(-n_max,n_max+1)
espectro = pd.DataFrame(index = ns,
                        columns= ['C','C_np','C_real','C_imag','C_mag','C_ang'])

for n in espectro.index:
    C_n = (1/T)*sym.integrate(y*sym.exp(-1j*n*w0*t), (t,0,T)).evalf()
    C = C + C_n*delta.subs(nw,nw-n)
    y_ser = y_ser + C_n*sym.exp(1j*n*w0*t)
    espectro['C'][n]=C_n
    C_r = float(sym.re(C_n))
    C_i = float(sym.im(C_n))
    espectro['C_real'][n] = C_r
    espectro['C_imag'][n] = C_i
    espectro['C_np'][n] = complex(C_r + 1j*C_i)
    espectro['C_mag'][n] = np.absolute(espectro['C_np'][n])
    espectro['C_ang'][n] = np.angle(espectro['C_np'][n])
espectro
```

      C
      C_np
      C_real
      C_imag
      C_mag
      C_ang

      -5
      -1.44265290902902e-129 + 3.60663227257255e-130*I
      (-1.4426529090290212e-129+3.606632272572553e-1...
      -1.44265e-129
      3.60663e-130
      1.48705e-129
      2.89661

      -4
      5.77061163611608e-129 + 1.15412232722322e-128*I
      (5.770611636116085e-129+1.154122327223217e-128j)
      5.77061e-129
      1.15412e-128
      1.29035e-128
      1.10715

      -3
      -4.61648930889287e-128 - 4.61648930889287e-128*I
      (-4.616489308892868e-128-4.616489308892868e-128j)
      -4.61649e-128
      -4.61649e-128
      6.5287e-128
      -2.35619

      -2
      9.23297861778574e-128 + 9.23297861778574e-128*I
      (9.232978617785736e-128+9.232978617785736e-128j)
      9.23298e-128
      9.23298e-128
      1.30574e-127
      0.785398

      -1
      1.05879118406788e-22 + 0.5*I
      (1.0587911840678754e-22+0.5j)
      1.05879e-22
      0.5
      0.5
      1.5708

      0
      0.500000000000000
      (0.5+0j)
      0.5
      0
      0.5
      0

      1
      3.58856254064475e-26 - 0.5*I
      (3.5885625406447527e-26-0.5j)
      3.58856e-26
      -0.5
      0.5
      -1.5708

      2
      9.23297861778574e-128 - 9.23297861778574e-128*I
      (9.232978617785736e-128-9.232978617785736e-128j)
      9.23298e-128
      -9.23298e-128
      1.30574e-127
      -0.785398

      3
      -4.61648930889287e-128 + 4.61648930889287e-128*I
      (-4.616489308892868e-128+4.616489308892868e-128j)
      -4.61649e-128
      4.61649e-128
      6.5287e-128
      2.35619

      4
      5.77061163611608e-129 - 1.15412232722322e-128*I
      (5.770611636116085e-129-1.154122327223217e-128j)
      5.77061e-129
      -1.15412e-128
      1.29035e-128
      -1.10715

      5
      -1.44265290902902e-129 - 3.60663227257255e-130*I
      (-1.4426529090290212e-129-3.606632272572553e-1...
      -1.44265e-129
      -3.60663e-130
      1.48705e-129
      -2.89661

La señal reconstruida con un **n_max** componentes

```python
y_ser
```

```python
plt.rcParams['figure.figsize'] = 7, 2
g1 = sym.plot(y, (t,0,1), ylabel=r'Amp',show=False,line_color='blue',legend=True, label = 'y(t) original')
g2 = sym.plot(sym.re(y_ser), (t,-1,2), ylabel=r'Amp',show=False,line_color='red',legend=True, label = 'y(t) reconstruida')
g1.extend(g2)
g1.show()
```

```python
C
```

```python
plt.rcParams['figure.figsize'] = 7, 2
plt.stem(espectro.index,espectro['C_mag'])

```

    <StemContainer object of 3 artists>

**Ejercicio para entregar 02-Octubre-2020**

Use las siguientes funciones para definir un periodo de una señal periódica con periodo $T=1$:

\begin{equation}
y_1(t) = \begin{cases}
-1 &  0 \leq t < 0.5 \\
1 &  0.5 \leq t < 1
\end{cases}
\end{equation}

\begin{equation}
y_2(t) = t
\end{equation}

\begin{equation}
y_3(t) = 3 sin(2 \pi t)
\end{equation}

Varíe la cantidad de componentes que reconstruyen cada función y analice la reconstrucción obtenida y los valores de $C_n$

```python

```


# Content
* Libraries
* Introduction to Problem
    * Loading Dataset
    * Visualizing Raw Dataset
    * Preprocessing
    * Visualizing Proprocessed Dataset
* Logistic Regression with numpy
    * Forward Propagation
    * Backward Propagation
    * Complete Propagation
    * Combining All Together
    * Training
    * Prediction
    * Evaluation
* Logistic Regression with tensorflow
    * Just Forward Propagation
    * Training and Evaluation
* Logistic Regression with keras
    * Just Layer Description
    * Training
    * Evaluation

# Libraries

```python
%matplotlib inline
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import seaborn as sb
import pandas as pd
from sklearn import datasets
from sklearn.model_selection import train_test_split
from utils import plot_prediction
```

# Introduction to Problem
Given a set of inputs X as features of flowers, we want to assign them to one of two possible categories 0 or 1 that means what type of flower they are.
For solving this problem we use logistic regression because it models the probability that each input belongs to a particular category.

## Loading Dataset

How many features this dataset have?

What is the label for this problem?

```python
iris = pd.read_csv('./data/Iris.csv')
iris.sample(5)
```

      Id
      SepalLengthCm
      SepalWidthCm
      PetalLengthCm
      PetalWidthCm
      Species

      122
      123
      7.7
      2.8
      6.7
      2.0
      Iris-virginica

      23
      24
      5.1
      3.3
      1.7
      0.5
      Iris-setosa

      77
      78
      6.7
      3.0
      5.0
      1.7
      Iris-versicolor

      31
      32
      5.4
      3.4
      1.5
      0.4
      Iris-setosa

      89
      90
      5.5
      2.5
      4.0
      1.3
      Iris-versicolor

```python
print("Number of Flowers: {}".format(iris.shape[0]))
```

    Number of Flowers: 150

## Visualizing Dataset

```python
sepalPlt = sb.FacetGrid(iris, hue="Species", size=6).map(plt.scatter, "SepalLengthCm", "SepalWidthCm")
plt.legend(loc='upper left')
```

## Preprocessing

In this step we will simplify the problem into a binary classification problem with just 2 features.

```python
X = iris.iloc[:, 1:3]
Y = (iris['Species'] != 'Iris-setosa') * 1
```

We devide data to train and test sets:

```python
X_train, X_test, y_train, y_test = train_test_split(X, Y)
```

```python
plt.figure(figsize=(10, 6))
plt.scatter(X_train[y_train == 0].iloc[:, 0], X_train[y_train == 0].iloc[:, 1], color='b', label='Iris-setosa train')
plt.scatter(X_train[y_train == 1].iloc[:, 0], X_train[y_train == 1].iloc[:, 1], color='r', label='Others train')
plt.scatter(X_test[y_test == 0].iloc[:, 0], X_test[y_test == 0].iloc[:, 1], color='b', label='Iris-setosa test', marker='+', s=150)
plt.scatter(X_test[y_test == 1].iloc[:, 0], X_test[y_test == 1].iloc[:, 1], color='r', label='Others test', marker='+', s=150)
plt.xlabel('SepalLengthCm')
plt.ylabel('SepalWidthCm')
plt.legend()
```

Questions

## Logistic Regression
  Logistic Regression is a Machine Learning classification algorithm that is used to predict the probability of a categorical dependent variable. In logistic regression, the dependent variable is a binary variable that contains data coded as 1 (yes, success, etc.) or 0 (no, failure, etc.). In other words, the logistic regression model predicts P(Y=1) as a function of X.

\begin{align}
Z = w_1x_1+w_2x_2+\dots+w_nx_n + b
\end{align}

## Forward Propagation

First we will implement linear multiplication:

$$W=\begin{bmatrix}w_1, & w_2, & \dots & w_n \end{bmatrix}$$
$$X=\begin{bmatrix}x_1, & x_2, & \dots & x_n \end{bmatrix}$$

\begin{align}
Z = W^TX + b
\end{align}

```python
def linear_mult(X, w, b):
    return np.dot(w.T, X) + b
```

Now we implement a function could generate W and b for us:

```python
def initialize_with_zeros(dim):
    """
    This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.

    Argument:
    dim -- size of the w vector we want (or number of parameters in this case)

    Returns:
    w -- initialized vector of shape (dim, 1)
    b -- initialized scalar (corresponds to the bias)
    """

    w = np.zeros((dim,1))
    b = 0

    assert(w.shape == (dim, 1))
    assert(isinstance(b, float) or isinstance(b, int))

    return w, b
```

Next we will implement sigmoid function to map calculated value to a probablity:

\begin{align}
A = \sigma(Z) = \frac{1}{1 + e^{-Z}}
\end{align}

```python
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
```

Now we implement the cost function, cost function represent the difference between our predictions and actual labels(y is the actual label and a is our predicted label):

\begin{align}
J = -\frac{1}{m}\sum_{i=1}^{m}y^{(i)}\log(A^{(i)})+(1-y^{(i)})\log(1-A^{(i)})
\end{align}

```python
def cost_function(y, a):
    return -np.mean(y*np.log(a) + (1-y)*np.log(1-a))
```

Now we implement the whole forward propagation which will calculate cost and the predicted value for the each data point:

```python
def forward_propagate(w, b, X, Y):

    m = X.shape[1]

    Z = linear_mult(X, w, b)
    A = sigmoid(Z)
    cost = cost_function(Y, A)
    cost = np.squeeze(cost)

    assert(cost.shape == ())

    back_require = {
        'A': A
    }

    return back_require, cost
```

## Backward Propagation

Now we calculate W and b derivative as follow:

$$ \frac{\partial J}{\partial w} = \frac{1}{m}X(A-Y)^T\tag{7}$$
$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^m (a^{(i)}-y^{(i)})\tag{8}$$

```python
def backward_propagate(w, b, X, Y, back_require):

    m = X.shape[1]

    A = back_require['A']

    dw = (1/m) * np.dot(X,(A-Y).T)
    db = (1/m) * np.sum(A - Y)

    assert(dw.shape == w.shape)
    assert(db.dtype == float)

    grads = {"dw": dw,
             "db": db}

    return grads
```

# Complete Propagation

```python
def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b

    Tips:
    - Write your code step by step for the propagation. np.log(), np.dot()
    """

    # FORWARD PROPAGATION
    back_require, cost = forward_propagate(w, b, X, Y)

    # BACKWARD PROPAGATION
    grads = backward_propagate(w, b, X, Y, back_require)

    return grads, cost
```

# Combining All Together

Now we combine all our implemented functions together to create an optimizer which can find a linear function to devide the zero labeled data from one labeled data points by optimizng W and b as follow:
$$W=W−\alpha{dw}$$
$$b=b−\alpha{db}$$
$\alpha$ is the learning rate

```python
# GRADED FUNCTION: optimize

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    """
    This function optimizes w and b by running a gradient descent algorithm

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps

    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.

    Tips:
    You basically need to write down two steps and iterate through them:
        1) Calculate the cost and the gradient for the current parameters. Use propagate().
        2) Update the parameters using gradient descent rule for w and b.
    """

    costs = []

    for i in range(num_iterations):

        grads, cost = propagate(w,b,X,Y)

        dw = grads["dw"]
        db = grads["db"]

        w -= learning_rate*dw
        b -= learning_rate*db

        # Record the costs
        costs.append(cost)

        # Print the cost every 100 training iterations
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))

    params = {"w": w,
              "b": b}

    grads = {"dw": dw,
             "db": db}

    return params, grads, costs
```

## Training

```python
%%time
X_train_t, y_train_t = np.array(X_train.T), np.array(y_train.T)
w, b = initialize_with_zeros(2)
params, grads, costs = optimize(w, b, X_train_t, y_train_t, num_iterations= 800, learning_rate = 0.1, print_cost = False)
```

    Wall time: 50.9 ms

```python
plt.plot(range(len(costs)),costs)
plt.xlabel('Iterations')
plt.ylabel('Cost(loss) value')
```

# Prediction

```python
def predict(w, b, X):
    '''
    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)

    Returns:
    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
    '''

    m = X.shape[1]
    Y_prediction = np.zeros((1,m))

    Z = linear_mult(X, w, b)
    A = sigmoid(Z)

    for i in range(m):
        Y_prediction[0][i] = 1 if A[0][i] > .5 else 0

    assert(Y_prediction.shape == (1, m))

    return Y_prediction
```

# Evaluation

```python
preds = predict(params['w'], params['b'], X_train_t)
print('Accuracy on training set: %{}'.format((preds[0] == y_train).mean()*100))
```

    Accuracy on training set: %99.10714285714286

```python
preds = predict(params['w'], params['b'], X_train_t)
print('Accuracy on training set: %{}'.format((preds[0] == y_train).mean()*100))
preds = predict(params['w'], params['b'], np.array(X_test.T))
print('Accuracy on test set: %{}'.format((preds[0] == y_test).mean()*100))
```

    Accuracy on training set: %99.10714285714286
    Accuracy on test set: %100.0

```python
plot_prediction(X_train, y_train, X_test, y_test, predict, params)
```

# Logistic Regression with tensorflow

## Just Forward Propagation

```python
y_test = np.array(y_test).astype(np.float32).reshape(-1,1)
y_train = np.array(y_train).astype(np.float32).reshape(-1,1)

graph = tf.Graph()
with graph.as_default():
    with tf.device("/cpu:0"):

        # Input data.
        # Load the training, validation and test data into constants that are
        # attached to the graph.
        X_data = tf.placeholder(tf.float32, shape=(None, 2))
        y_data = tf.placeholder(tf.float32 , shape=(None, 1))

        # Variables.
        # These are the parameters that we are going to be training. The weight
        # matrix will be initialized using random values following a (truncated)
        # normal distribution. The biases get initialized to zero.
        weights = tf.Variable(tf.truncated_normal([2, 1]))
        biases = tf.Variable(tf.zeros([1]))

        # Training computation.
        # We multiply the inputs with the weight matrix, and add biases. We compute
        # the softmax and cross-entropy (it's one operation in TensorFlow, because
        # it's very common, and it can be optimized). We take the average of this
        # cross-entropy across all training examples: that's our loss.
        logits = tf.matmul(X_data, weights) + biases
        loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y_data, logits=logits))

        # Optimizer.
        # We are going to find the minimum of this loss using gradient descent.
        optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

        # Predictions for the training, validation, and test data.
        # These are not part of training, but merely here so that we can report
        # accuracy figures as we train.
        prediction = tf.nn.sigmoid(logits)
```

## Training and Evaluation

```python
%%time
num_steps = 800

with tf.Session(graph=graph) as session:
    # This is a one-time operation which ensures the parameters get initialized as
    # we described in the graph: random weights for the matrix, zeros for the
    # biases.
    tf.global_variables_initializer().run()
    print('Initialized')
    for step in range(num_steps):
        # Run the computations. We tell .run() that we want to run the optimizer,
        # and get the loss value and the training predictions returned as numpy
        # arrays.
        d, l, predictions = session.run([optimizer, loss, prediction],
                                        feed_dict={X_data: X_train.astype(np.float32), y_data: y_train})
        if (step % 100 == 0):
            print('Loss at step %d: %f' % (step, l))
            print('Training accuracy: %.1f%%' % accuracy(predictions, y_train))
            # Calling .eval() on valid_prediction is basically like calling run(), but
            # just to get that one numpy array. Note that it recomputes all its graph
            # dependencies.
    print('Test accuracy: %.1f%%' % accuracy(
    prediction.eval(feed_dict={X_data: X_test.astype(np.float32)}), y_test))
    plot_prediction(X_train, y_train, X_test, y_test, prediction, params, fm_type='tensorflow') #TODO
```

# Keras

## Just Layer Description

```python
model = tf.keras.models.Sequential([
  tf.keras.layers.Dense(1, activation=tf.nn.sigmoid, input_shape=(2,))
])
model.compile(optimizer='sgd',
              loss='binary_crossentropy',
              metrics=['accuracy'])
```

## Training

```python
%%time
history = model.fit(X_train, y_train, epochs=800, verbose=0)
```

    Wall time: 7.25 s

```python
plt.plot(range(len(history.history['acc'])),history.history['acc'])
plt.xlabel('Iterations')
plt.ylabel('Accuracy value')
```

## Evaluation

```python
print('Test accuracy: %.1f%%' % accuracy(
                            model.predict(X_train.astype(np.float32)), y_train))
print('Test accuracy: %.1f%%' % accuracy(
                            model.predict(X_test.astype(np.float32)), y_test))
```

    Test accuracy: 99.1%
    Test accuracy: 100.0%

```python
plot_prediction(X_train, y_train, X_test, y_test, model.predict, params, fm_type='keras')
```

# Further Reading

* Tensorflow Playground: https://playground.tensorflow.org/
* First Week of Neural Networks Course: https://www.coursera.org/learn/neural-networks-deep-learning


# Projectile Motion
## 1. Introduction
*This notebook is meant to satisfy Physics 20-A1.3s. Special thanks to Ms. Catherine Reichelt for her help on developing this notebook.*
#### *Objective: Mathematically analyze the movement of a launched object to determine flight path, speed, and time.*
Learning projectile motion will enable __you__ with the skills to understand how launched objects fly through the air and how you can design your own. It is critical for sports, military, and any scenario where you need an airborne object to reach a destination. What examples can you think of?

Projectile motion explores the physics behind anything airborne that is subjected only to gravity. From throwing a dart to shooting a basketball to medieval cannons used during sieges, all of these objects are ruled by the same basic principles.

>
>
>
 A clip from the game Besiege (http://www.indiedb.com/games/besiege/images/besiege-gif)

### Who created projectile motion analysis?

>
>
>
 Painting of Galileo (https://www.gettyimages.ca/detail/news-photo/galileo-galilei-and-his-telescope-engraving-1864-news-photo/526510364)

Galileo, one of the great fathers of astronomy and modern science, created projectile motion analysis over 400 years ago. This gave him the tools to improve the accuracy and effectiveness of military cannons in the 17th century. The reason we study it today is because his method has stood the test of time and continues to be a fundamental foundation to areas like aerodynamics, sports performance, and military design.

For the purposes of understanding the fundamentals, the effects of air resistance will be ignored. While it makes our lives simpler, it is wrong to assume that projectiles only experience the force of gravity - air resistance can play a significant role in many cases. Air resistance is responsible for lower projectile speeds and distances but it is also critical to helping airplanes slow down, keeping race cars grounded at high speeds (which is why they have spoilers), and causing crazy phenomena like the Magnus effect shown below.

>
>
>
 Basketball toss demonstrating the Magnus effect (https://giphy.com/gifs/basketball-thrown-dam-DnalOTyvY4eBi)

However, in order to understand these complex cases, we must first strip away the layer of friction and look at how objects travel solely under the force gravity.

***

## 2. Theory & Practice

Neglecting air resistance, all projectiles in the air will only be under the force of gravity. This means objects travelling in 2 dimensions will constantly be pulled down to earth and forced into a curved trajectory known as a parabola. Take a look at the gif below. What do you notice about the velocity vectors in the horizontal and vertical directions? Which one changes? Which one stays the same?

>
>
>
 Projectile motion animation (http://gbhsweb.glenbrook225.org/gbs/science/phys/mmedia/vectors/nhlp.html)

Galileo realized that projectile motion could be broken into two components: __horizontal (x)__ and __vertical (y)__ that can be analyzed separately. These two dimensions are __independent__, the only variable they share is time *__t__*. This means that changes in the vertical distance, speed, and acceleration will not affect the horizontal components and vice versa.

---

### Vertical Component
Due to the force of gravity, the object will accelerate towards the center of the Earth. This means that its initial velocity will be different from its final velocity.
There are 5 variables involved in the vertical component: acceleration, initial velocity, final velocity, *altitude* (distance travelled vertically), and time. We will assume $ \vec a $ is constant for projectile motion and equivalent to $ 9.81  m/s^2$.

\begin{equation}
\vec a , \vec v_i , \vec v_f , \vec d_y , t
\end{equation}

There are 5 kinematic equations for uniform acceleration where each one contains 4 of the 5 variables listed above. Given 3 of the variables in a problem, the other 2 can each be found by picking an appropriate equation. *Note: If the object is initially launched horizontally, then $\vec v_i $ = 0 *

\begin{equation}
\vec a_{ave} = \frac{\vec v_f - \vec v_i}{t}  \\
\vec d = \vec v_i t + \frac12\ \vec a t^2  \\
\vec d = \vec v_f t - \frac12\ \vec a t^2  \\
\vec d = \frac{\vec v_f + \vec v_i}{2} t  \\
v_f^2 = v_i^2 + 2 a d  \\
\end{equation}

  __Question 1__

  You're out with some friends swimming at a popular cliff jumping spot and they can't stop arguing about how tall the cliff *really* is (for bragging rights). You decide to end this once and for all and pull out a stopwatch. You climb up the cliff, release a rock in your hand from rest, and time its descent. You do this a couple more times and get an average descent time of 1.62 seconds. Your friends stare at you in bewilderment as you rattle off the height after punching some numbers into your phone's calculator. How do you do it?

Try solving this question on your own, it's the best way to develop your skills. Once you've given it a go, click the dropdown arrow to reveal the solution.

  __Solution__

  1) Draw a picture of the scenario. This will give you a better grasp of the problem.
  2) Define your sign convention. Assign positive and negative directions to y.

  3) Identify the variables you know and the variable(s) you're trying to find. Pick a formula that best fits the problem scenario and solve for the unknown.

    The rock is released from rest so $ \vec v_i = 0 $. Based on the information given and our sign convention, we can state the following:

\begin{equation}
\vec v_i = 0 \\
\vec a = +9.81 \ m/s \\
t = 1.62\ s \\
\vec d_y = \ ? \\
\vec v_f = \ ? \\
\end{equation}

We have 3 known variables and 2 unknown variables in the problem. However, in this case we're looking for $ \vec d_y $ so we don't care what $ \vec v_f $ is. If we take a look at the 5 vertical component equations, we see that the following equation has the 3 variables we know and the one we're looking for:

\begin{equation}
\vec d_y = \vec v_i t + \frac12\ \vec a t^2  \\
\end{equation}

We don't want one with $ \vec v_f $ because we don't know its value. Using this equation we can solve for $ \vec d_y $:

\begin{equation}
\vec d_y = (0)(2.62\ s) + \frac12\ (+9.81\ m/s^2) (1.62\ s)^2  \\
\vec d_y = +12.8727\ m \\
\vec d_y = +12.9\ m \\
\end{equation}

Using the time it took to fell, we can conclude that the cliff is 12.9 meters tall.

---

### Horizontal Component
There is no force acting horizontally. Therefore there is no acceleration so it is **uniform motion** ($ \vec v_i = \vec v_f = \vec v_x $).

Horizontal uniform motion is governed by 3 variables: velocity, *range* (distance travelled horizontally), and time.

\begin{equation}
\vec v_x , \vec d_x , t
\end{equation}

These 3 variables are related by the uniform motion equation:

\begin{equation}
\vec v_x = \frac {\vec d_x}{t}
\end{equation}

Let's do some practice to solidify the concepts.

  __Question 2__

  You're back at the same pond as Question 1 except this time you flick the rock horizontally off the top of the cliff at 30.0 km/h. The rock makes it across the pond and still takes 1.62 seconds to hit the water. How long is the pond?

  __Solution__

  1) Draw a picture of the scenario. This will give you a better grasp of the problem.
  2) Define your sign convention. Assign positive and negative directions to x.

  3) Identify the variables you know and the variable(s) you're trying to find. Pick a formula that best fits the problem scenario and solve for the unknown.

    The horizontal component is easier than the vertical component because there's only one equation with 3 variables. List the 2 variables you know and the one you're trying to find:

\begin{equation}
\vec v_x = +30.0\ km/h =\ +8.3333\ m/s \\
t = 1.62\ s \\
\vec d_x = \ ? \\
\end{equation}

Rearrange the uniform motion equation for the unknown and solve:

\begin{equation}
\vec v_x = \frac {\vec d_x}{t} \\
\vec d_x = \vec v_x t = (+8.3333\ m/s)(1.62\ s) \\
\vec d_x = +13.5\ m \\
\end{equation}

The pond is 13.5 meters long.

__Question 3__

Evil Knievel, the daredevil stunt driver, is performing his next trick. He speeds horizontally off of a 50.0 m high cliff on a motorcycle. How fast must he leave the cliff-top if he needs to soar over the 90.0 m river at the base of the cliff?

__Solution__

1) Draw a picture of the scenario. This will give you a better grasp of the problem.
2) Define your sign convention. Assign positive and negative directions to both x and y.

3) Set up a table and identify the variables you know and the variable(s) you're trying to find. Pick a vertical component formula that best fits the problem scenario and solve for the unknown.

Based on the information given and our sign convention, we can fill our data table with the following:

\begin{array}{cc}
x &y \\ \hline
\vec d_x = +90.0\ m &\vec d_y = -50.0\ m \\
\vec v_x =\ ? &\vec a = -9.81\ m/s^2 \\
\  &\vec v_i = 0 \\
\end{array}
$$ t =\ ? $$

Note that $ \vec v_i = 0 $ because the projectile is launched horizontally, and $ t $ is common to both x & y.

To find $ \vec v_x $, we need $ \vec d_x $ and $ t $. We know $ \vec d_x $ but we're missing $ t $ so we'll need to use the vertical data to solve for $ t $. Because we know $ \vec d_y, \vec a, $ and $ \vec v_i $, and we're looking for $ t $, we'll use the following equation because we can solve for $ t $ using the variables we know:

\begin{equation}
\vec d = \vec v_i t + \frac12\ \vec a t^2  \\
\end{equation}

Because $ \vec v_i = 0 $, the first term goes to 0 and the resulting equation can be rearranged to solve for $ t $:

\begin{equation}
\vec d = \frac12\ \vec a t^2  \\
t = \sqrt{\frac{2 \vec d_y}{\vec a}} \\
\end{equation}

Plug in values to find $ t $:

\begin{equation}
t = \sqrt{\frac{2 (-50.0\ m)}{(-9.81\ m/s^2)}} = 3.1928\ s
\end{equation}

Now that $ t $ is known we can solve for  $ \vec v_x $ using the uniform motion equation:

\begin{equation}
\vec v_x = \frac {\vec d_x}{t} = \frac {(+90.0\ m)}{(3.1928\ s)} = +28.2\ m/s \ \ (+101.5\ km/h)
\end{equation}

__Question 4__

Galileo predicted that an object launched horizontally and an object dropped vertically off the same ledge will reach the ground at the same time. Will they? Why or why not?

__Solution__

Yes, they will. Horizontal and vertical motion are independent for a projectile so the horizontal movement of the launched object does not affect its vertical freefall.

 (Drawing courtesy of Ms. Reichelt)

---

### Projectiles Fired at an Angle
What happens when your projectile is launched at an angle? What are the initial velocities?

In this case, the initial vertical velocity is no longer 0. It will have some value $ \vec v_i $ that will gradually decrease to 0 at the top of its trajectory and then increase in the downward direction as it returns back to ground. Let's take a football punt for example.

According to [Angelo Armenti's The Physics of Sports](https://www.livestrong.com/article/397904-maximum-speed-of-a-football/), top-level football kickers can send footballs flying at 70 mph (31 m/s)! If the player kicked it at an angle of 50°, the trajectory would look something like this:

The horizontal velocity $ \vec v_x $ will remain the same throughout the flight (uniform motion) while $ \vec v_y $ will decrease to the top of its trajectory and then increase downwards. __Note: If the initial launch height and final landing height are the same ($ \vec d_y $ = 0), then the projectile will land with the same initial speed and angle!__ The initial horizontal and vertical velocities can be solved with some simple trigonometry:

\begin{array}{cc}
x &y \\
cos(50^{\circ}) = \frac{adj}{hyp} = \frac{\vec v_x}{31\ m/s} &sin(50^{\circ}) = \frac{opp}{hyp} = \frac{\vec v_y}{31\ m/s} \\
\vec v_x = (31\ m/s)\ cos(50^{\circ}) &\vec v_y = (31\ m/s)\ sin(50^{\circ}) \\
\vec v_x = +19.9264\ m/s &\vec v_y = +23.7474\ m/s \\
\vec v_x = +20\ m/s &\vec v_y = +24\ m/s \\
\end{array}

    __Question 5__

    A football player kicks a ball across a flat field at 31.0 m/s and 50.0° from the ground. Find:
    a) The maximum height reached
    b) The flight time (time before it hits the ground)
    c) The range (how far away it hits the ground)
    d) The velocity vector at its maximum height
    e) The acceleration vector at its maximum height
    f) The velocity of the football when it hits the ground

    Again, try solving these questions on your own first, it's the best way to develop your skills. Once you've given it a go, click the dropdown arrow to reveal the solution.

    __Solution__

    __a)__

        To find the max height, let's only look at the first half of the flight path. That way, we know $ \vec v_f = 0 $ because projectiles have no vertical velocity at the top of their flight path.

        We'll use the same sign convention from Question 3. Because we're using the same values from the previous scenario, we can set up our table with the following information:

        \begin{array}{cc}
          x &y \\ \hline
          \vec v_x =\ +19.9264\ m/s &\vec v_i = +23.7474\ m/s \\
          \  &\vec v_f = 0 \\
          \  &\vec a = -9.81\ m/s^2 \\
          \  &\vec d_y =\ ? \\
        \end{array}
        $$ t =\ ? $$

        Looking at our table, we have 3 knowns and 1 unknown in the vertical column. That means we can solve for $ \vec d_y $. Looking at our vertical component equations, the one that contains our 3 known variables and $ \vec d_y $ is:

        \begin{equation}
        v_f^2 = v_i^2 + 2 a d
        \end{equation}

        $ \vec v_f = 0 $ so we can rearrange and solve for $ \vec d_y $:

        \begin{equation}
        \vec d_y = -\frac{\vec v_i^2}{2 \vec a} = -\frac{(+23.7474\ m/s)^2}{2\ (-9.81\ m/s^2)} \\
        \vec d_y = +28.7431\ m \\
        \vec d_y = +28.7\ m \\
        \end{equation}

    __b)__

        To find the flight time, we could either find the time it takes to reach the top of its trajectory and double that, or we could look at the full flight path and find the time to landing. In this case we will choose the latter.

        Because our flight path changed, some of our variable have too. In particular, $ \vec d_y = 0 $ which also means that $ \vec v_f = -\vec v_i $. Let's make a new table:

        \begin{array}{cc}
        x &y \\ \hline
        \vec v_x =\ +19.9264\ m/s &\vec v_i = +23.7474\ m/s \\
        \  &\vec v_f = -23.7474\ m/s \\
        \  &\vec a = -9.81\ m/s^2 \\
        \  &\vec d_y =\ 0 \\
        \end{array}
        $$ t =\ ? $$

        Now, we're looking for $ t $ but we don't have enough horizontal data to solve with uniform motion. However, we know 4 out of the 5 vertical variables which means we have lots of options for the vertical equation. Any of the first 4 equations will do but the second and third will require solving a quadratic equation. To make life easy, we'll use the first one:

        \begin{equation}
        \vec a_{ave} = \frac{\vec v_f - \vec v_i}{t}
        \end{equation}

        Rearrange and solve for $ t $ (Remember to keep your sign convention. If you don't you could end up with 0 here!):

        \begin{equation}
        t = \frac{\vec v_f - \vec v_i}{\vec a_{ave}} = \frac{(-23.7474\ m/s) - (+23.7474\ m/s)}{(-9.81\ m/s^2)} \\
        t = 4.8415\ s \\
        t = 4.84\ s
        \end{equation}

    __c)__

        Now that we have the flight time and $ \vec v_x $, we can use the uniform motion equation to solve for range:

        \begin{equation}
        \vec v_x = \frac {\vec d_x}{t} \\
        \vec d_x = \vec v_x \ t = (+19.9264\ m/s)\ (4.8415\ s) = +96.4730\ m \\
        \vec d_x = +96.5\ m
        \end{equation}

    __d)__

        At max height, the vertical velocity $ \vec v_y = 0 $ so the only velocity is horizontal.

    __e)__

        The only acceleration is the acceleration due to gravity which is a constant pointing downwards (see part d for image).

    __f)__

        Because $ \vec d_y = 0 $ for the full flight path, $ \vec v_f = -\vec v_i = -23.7474\ m/s $. Horizontal velocity is constant so:

        \begin{equation}
        \vec v_f = 31.0\ m/s\ \  at -50.0^{\circ}
        \end{equation}

 __----- Continue on only after attempting Question 5. -----__

```python
"""
If this block of code has suddenly popped up, don't worry! You've found the code
used to create the Projectile Trajectory graph shown below. This is normally hidden but feel
free to explore it and see how it works.

If you want to hide it again, just click on this code block and press 'Ctrl' and 'Enter'
simultaneously on your keyboard.
"""

# Import required packages
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from ipywidgets import interact
import ipywidgets as widgets
from IPython.display import HTML

# Set values for equations
g = 9.81
t = np.linspace(0,10,100)

# Define equations that plot will display
def d_x(t, theta, v_i):
    return v_i*np.cos(np.deg2rad(theta))*t

def d_y(t, theta, v_i):
    return v_i*np.sin(np.deg2rad(theta))*t - 0.5*g*t**2

# Define options for plot
def f(theta,v_i):
    plt.plot(d_x(t, theta, v_i),d_y(t, theta, v_i))
    plt.ylim(0,50)
    plt.xlim(0,100)
    plt.xlabel("Range (m)", fontsize=16)
    plt.ylabel("Altitude (m)", fontsize=16)
    plt.margins(0)
    plt.grid()
    plt.title("Projectile Trajectory", fontsize=20)

hide_me = ''
HTML('''
<form action="javascript:code_toggle()"><input style="opacity:0" type="submit" value="Click here to toggle on/off the raw code."></form>''')
```

<form action="javascript:code_toggle()"><input style="opacity:0" type="submit" value="Click here to toggle on/off the raw code."></form>

The scenario in Question 5 can be modelled by a graph using Python. The following code block creates an interactive graph where you can modify the initial velocity $ \vec v_i $ and launch angle $ \theta $ to see how the projectile's trajectory changes.

Modify `theta` and `v_i` to recreate the scenario in Question 5 and use the graph to verify your answers for Parts (a) and (c) line up.

```python
interact(f,theta = (0,90,5), v_i = (0,31,1))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    <function __main__.f(theta, v_i)>

Pretty cool, huh? If you want to learn how to make graphs using Python, Part (b) of Question 6 will give you a step-by-step breakdown on how to make simple static graphs.

---

### Determining the Optimum Launch Angle

    __Question 6__

    Picture this: The International Olympic Committee runs a worldwide survey to see what new global event people would like to see. After some fierce debate and tallying the votes, __shot-cannon__ is chosen! Shot-cannon is a fairly straightforward sport: each country develops their own medieval cannon that competes in a series of challenges. While each challenge tests a different aspect of the design, the main event is the range competition to see which country's cannon can fire a lead ball the farthest across the field. Canada has chosen __you__ to man the cannon for the main event! With the cannon already designed and providing a fixed initial velocity $ \vec v_i $, your responsibility is to pick the optimal angle $ \theta $ to fire the cannon to achieve maximum distance. If you assume no air resistance and a flat field, what angle should you pick?

    Before reading on, take a moment to picture the scenario in your head and, using your intuition, take a guess at what angle you think would provide the maximum range. Can you think of a way to prove this?

    Initially, this problem can seem daunting in its magnitude, but if we break it into two chunks it becomes more manageable:
      a) Develop an equation for the range $ \vec d_x $ as a function of $ \vec v_i $ and $\theta $. That is, come up with an equation in the form of $ \vec d_x = f(\ \vec v_i,\ \theta)$.
      b) Determine the optimal angle from the equation you've developed.

    Part (a) can be solved using the skills you've just learnt so give it a go and hit the arrow to check your answer.

    Part (b) can be solved with coding and graphing which will be covered in the next section __Python Basics__.

    __Solution__

    __a)__

    With a picture and sign convention drawn, let's mark down the values we know in a table. While we don't know the angle $\theta$, and $ \vec v_i $ is an unknown constant, we can still write down $ \vec v_x $ and $ \vec v_{i_y} $ in terms of $\theta$ and $ \vec v_i $ because we ultimately want an equation with these terms:

    \begin{array}{cc}
    x &y \\
    cos(\theta) = \frac{adj}{hyp} = \frac{\vec v_x}{\vec v_i} &sin(\theta) = \frac{opp}{hyp} = \frac{\vec v_y}{\vec v_i} \\
    \vec v_x = \vec v_i cos(\theta) &\vec v_y = \vec v_i sin(\theta) \\
    \vec v_x = v_i cos(\theta) &\vec v_y = v_i sin(\theta) \\
    \end{array}

    The vector arrow on $ v_i $ is dropped because we know it's positive in both directions based on our sign convention. Also, we assume a flat field so $ \vec d_y = 0 $ which means that $ \vec v_{y_f} = -\vec v_{y_i} $ Therefore, our data table becomes:

    \begin{array}{cc}
      x &y \\ \hline
      \vec v_x =v_i cos(\theta) &\vec v_i = v_i sin(\theta) \\
      \vec d_x =\ ?  &\vec v_f = -v_i sin(\theta) \\
      \  &\vec a = -g \\
      \  &\vec d_y =\ 0 \\
    \end{array}
    $$ t =\ ? $$

    Acceleration is written as $ -g = -9.81 m/s^2 $ for simplicity. Note that this is a constant written as a letter, __not a variable.__

    In order to come up with an equation for $ \vec d_x $, we first need an equation for $ t $ in terms of the given variables $ \vec v_i $ and $\theta$. Considering we know or have expressions for 4 of the vertical values, we will choose the first equation which makes it easy to solve for t:

    \begin{equation}
    \vec a_{ave} = \frac{\vec v_f - \vec v_i}{t}
    \end{equation}

    Rearrange and solve for $ t $:

    \begin{equation}
    t = \frac{\vec v_f - \vec v_i}{\vec a_{ave}} = \frac{(-v_i sin(\theta)) - (v_i sin(\theta))}{(-g)} \\
    t = \frac{2 v_i sin(\theta)}{g} \\
    \end{equation}

    This expression can be applied to the horizontal component to come up with an equation for $ \vec d_x $:

    \begin{equation}
    \vec v_x = \frac{\vec d_x}{t} \\
    \vec d_x = \vec v_x t \\
    \vec d_x = (v_i cos(\theta)) (\frac{2 v_i sin(\theta)}{g}) \\
    \vec d_x = \frac{2}{g} v_i^2 sin(\theta) cos(\theta) \\
    \end{equation}

    Knowing that $ g $ and $ v_i $ are constants, this means that $ \vec d_x $ is only a function of the variable $\theta$.

 __----- Continue on only after attempting Question 6 Part (a) and checking with the solution. The next section will cover Part (b). -----__

### Python Basics

From this point on, there are two ways to solve for $\theta$: you can use calculus to solve for it analytically, or we can get creative and use coding with graphing to find the answer!

Python provides us with some great tools to graph this function easily. If you have some knowledge of Python you can skip the explanations and just run the code cells. If not, we're going to take a moment to understand what the code you're about to see does. First up, let's go over the first bit of code.

__Imports__
At the beginning of most Python programs you're more than likely to see a few (or many) `import` statements. The purpose of these is to bring in other pieces of code, that either you or someone else have written, in order to keep your current program more manageable. For example, the set of import statements we're going to use to plot our functions look like this (click on the code block below and press `Ctrl` and `Enter` on your keyboard simultaneously to run the cell).

```python
%matplotlib inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
```

Starting from the top down, let's go over what each piece of this code does.

1. **%matplotlib inline:** This is very specific to the Jupyter notebook we're using. This tells matplotlib (next bullet) to output graphs in the cell directly below the one in which the code is executed. For more information about Jupyter "magic commands" feel free to read [this document](http://ipython.readthedocs.io/en/stable/interactive/magics.html).

2. **import matplotlib**: This command tells Python to import the `matplotlib` package. This package contains the python functions that we'll be using for plotting.

3. **import numpy as np**: This imports the python package `numpy` or "numerical python" and assigns the name to it within our code to `np` so we don't have to type out `numpy` every time we need a function. We'll be using this package for mathematical functions like sine and the square root.

4. **import matplotlib.pyplot as plt**: This imports the graphing subroutines from the `matplotlib` packages and assigns them the name `plt` so we don't have to type as much when we want to produce a graph.

__Plotting__
After required modules have been imported into our script, we're ready to get started graphing. First up, we need to define a number of points which to plot. Our computer doesn't understand that the variable $ \theta $ is fully continuous, so we have to give it a discrete set of points to plot our function with. We can do that in python using numpy as follows (click the next cell and press control and enter at the same time on your keyboard).

```python
theta = np.linspace(0,90,5)
print("theta = ", theta)
```

    theta =  [  0.   22.5  45.   67.5  90. ]

This creates a **list** of numbers called `theta` which consists of 5 numbers evenly spaced in the domain $[0,90]$ that we'll use to plot our function. The `print` function is a standard python function that simply displays our variables to the screen as either numbers or characters. In order to plot our functions, we type the following. Note that we've increased the number of points in `theta` to create a smoother plot. Feel free to change the number `100` to something smaller and observe how the plot changes. We also assigned a value to `v_i` and `g` so that the computer can plot the function numerically. Likewise, play around with these numbers and see how your range changes.

```python
theta = np.linspace(0,90,100)
v_i = 30.0
g = 9.81
plt.plot(theta, 2 / g * (v_i**2) * np.sin(np.deg2rad(theta)) * np.cos(np.deg2rad(theta)))
plt.show()
```

There's a fair bit going on in that last line of code that should be noted. First, by using `plt.plot` we're calling a function from `matplotlib.pyplot` (that we called `plt`) called `plot`. This function, unsurprisingly, is used to tell Python what to plot. We then pass this function a number of arguments (AKA inputs). The first argument `theta` is the list of numbers we generated earlier. The second argument `2 / g * (v_i**2) * np.sin(np.deg2rad(theta)) * np.cos(np.deg2rad(theta))` is the mathematical function we're going to plot. Here `theta` is the variable we're going to plot, and also the list of points that we generated earlier. The `**` is the Python way of saying "to the power of". Because `sin` and `cos` functions take radians as an input and not degrees, we use `np.deg2rad` which is a function that converts degrees to radians.

So, what we're really saying here is "plot $ \frac{2}{g}v_i^2 \sin(\theta)\cos(\theta)  $ for 100 $\theta$ values between 0 and 90".

Now, that graph is missing a lot of important things like axis labels and a legend. We can add those like this:

```python
plt.plot(theta, 2 / g * (v_i**2) * np.sin(np.deg2rad(theta)) * np.cos(np.deg2rad(theta)))
plt.xlabel(r"$ \theta (deg) $", fontsize=16)
plt.ylabel(r"$ \vec d_x (m) $", fontsize=16)
plt.margins(0)
plt.grid()
plt.title(r"Range as a function of launch angle $ \theta $")
plt.show()
```

where the `plt.` calls are still calling functions from `matplotlib.pyplot` however this time we're creating x axes labels with `plt.xlabel` and y axes labels with `plt.ylabel`. `plt.margins(0)` removes any unnecessary blank space around the graph and `plt.grid()` adds a nice grid to visually locate points on the graph easier. Finally, `plt.title` adds a title bar to our graph. The dollar signs and "r" characters are just there to make the lettering look nice. Play around with the code and see if you can change the title bar, label font sizes, and margins.

Based on the plot above, we see that the maximum range lines up with a launch angle of 45° . Was this your initial guess?

With no air resistance, 45° is the optimal angle because it provides the best compromise between horizontal speed and height. If you shoot it below 45°, you'll get a faster horizontal velocity but the ball will also hit the ground quicker because there's less flight time. If you shoot it above 45°, you'll get more flight time but a slower horizontal velocity. 45° is the sweet spot between extremes.

Because the graphical solution takes the shape of a parabola, this also demonstrates an important symmetry in the launch angles. Launch angles that are equidistant from the maximum of 45° will have the same range. That is, 30° and 60° have the same range, 15° and 75° have the same range, etc. The wonders of physics!

---

## 3. Conclusion & Extension

This notebook has demonstrated the basics of projectile motion that can be used to determine flight paths, speeds, and times. Reasoning for why projectile motion is foundational to physics is explained, followed by a breakdown describing the horizontal and vertical components of projectile motion. The two components were brought together in an angled launch example with an interactive graph and the final question utilized Python programming to solve a basic calculus problem. Applying the skills taught in this notebook to your practice examples will give you a strong grasp of projectile motion analysis and enable __you__ to begin designing your own basic launchers!

### Practice: Projectile Game
For a great interactive visualization of projectile motion, check out the PhET link below!

Click to Run


```python
import sys

import numpy as np
import matplotlib.pyplot as plt

sys.path.append('../src')
import utils
```

# Iterative Methods for Optimization

A general optimization problem has the form

\begin{align}
\min_x & \quad f_0(x) \\
\text{subject to} & \quad f_i(x) \leq b_i, \quad i = 1, \dots, m,
\end{align}

with components
* $x \in \mathbf{R}^n$ - optimization variable,
* $f_0: \mathbf{R}^n \to \mathbf{R}$ - objective function,
* $f_i: \mathbf{R}^n \to \mathbf{R}$ - constraint functions, and
* $b_i \in \mathbf{R}$: constraint bounds.

Very few optimization problems have a closed-form solution, so most problems are solved using iterative methods.
This notebook introduces gradient descent, a fundamental iterative method for solving unconstrained optimization problems of the form

$$\min_x f(x)$$

with $x \in \mathbf{R}^n$ and $f: \mathbf{R}^n \to \mathbf{R}$, where $f$ is differentiable. Our examples use [convex functions](https://en.wikipedia.org/wiki/Convex_function), but the intuition gained from the exercises can also be helpful for nonconvex functions.

Iterative methods start with an initial guess $x^0$.
Each iteration, a new value $x^{k+1}$ is generated based on the previous value $x^k$, often using local information such as the gradient $\nabla f(x^k)$ or Hessian $\nabla^2 f(x^k)$.
The algorithm terminates when a stopping condition is satisfied, which can be specified in terms of the size of the gradient, $\| \nabla f(x^k) \| < \epsilon$, the difference between successive iterates, $\| x^{k+1} - x^k \| < \epsilon$ or $\| f(x^{k+1}) - f(x^k) \| < \epsilon$, or a maximum number of iterations, $k > k_{\max}$.

# Gradient Descent

[Gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) is built upon the fact that the gradient of a function always points in the direction of maximum ascent, so the negative gradient points in the direction of maximum descent.
Therefore, starting at the point $x^k$ and taking a step in the direction of the negative gradient, we should arrive at a point $x^{k+1}$ that has a smaller function value, i.e., $f(x^{k+1}) < f(x^k)$.
After multiple iterations, we hope to approximate the optimal solution $x^* = \text{argmin}_x f(x)$.

Here's the algorithm for gradient descent with initial guess $x^0$, step size $\alpha$, and stopping condition based on the size of the gradient:

> **input** $x^0$, $\alpha$, $\epsilon$
> **set** $k = 0$
> **while** $\| \nabla f(x^k) \| \geq \epsilon$ **do**
>     $x^{k+1} \leftarrow x^k - \alpha \nabla f(x^k)$
>     $k \leftarrow k + 1$
> **end while**

##### How do we choose the initial guess?

Depending upon the problem, the initial guess can influence the value of the solution, the number of iterations needed to converge to a solution, or whether or not the algorithm converges at all.
Determining a "good" initial guess is a difficult problem itself, especially when $x$ is high-dimensional, so heuristics are often useful:
* $x^0$ = a vector of ones or zeros
* $x^0$ = a random vector
* $x^0$ = the solution to a similar, but easier, problem

##### How do we choose the step size?

The step size influences the rate of convergence.
If the step size is too small, it may take many more iterations to converge.
However, if the step size is too large, we may overshoot the optimal point.
If you are really unlucky, you could choose a step size that results in the iterates cycling through the same points forever.

In some cases, we can guarantee convergence based on the step size. For example, if a function is convex,

$$f( \lambda x + (1 - \lambda) y ) \leq \lambda f(x) + (1 - \lambda) f(y) \quad \text{for all} \quad x, y \in \mathbf{R}^n \quad \text{and} \quad 0 \leq \lambda \leq 1,$$

and continuously differentiable with a $\beta$-Lipschitz gradient for some $\beta > 0$,

$$\| \nabla f(x) - \nabla f(y) \| \leq \beta \| x - y\| \quad \text{for all} \quad x, y \in \mathbf{R}^n,$$

then after $k$ iterations with a step size $\alpha \leq 1/\beta$, we will have

$$f(x^k) - f(x^*) \leq \frac{\| x^0 - x^* \|^2}{2\alpha k}.$$

This means that gradient descent will converge at a rate of $\mathcal{O}(1/k)$.

As an example, consider the linear least-squares problem, where $f(x) = \frac{1}{2}\|Ax - b\|^2$.
The objective is convex and differentiable, with $\nabla f(x) = A^T(Ax - b)$.
To find the Lipschitz constant for the gradient, note that

$$\| \nabla f(x) - \nabla f(y) \| = \| A^TA(x - y) \| \leq \| A^TA \| \| x - y\|,$$

which follows from the definition of the induced matrix norm

$$\|A\| = \max_{x \neq 0} \frac{\|Ax\|}{\|x\|}.$$

Therefore, using a step size $\alpha \leq 1/\beta$, where $\beta = \|A^TA\|$, we are guaranteed to converge at a linear rate.

Other approaches for adaptively choosing step sizes include:
* [line search methods](https://en.wikipedia.org/wiki/Line_search): optimizing the step size along a given descent direction
* [trust region methods](https://en.wikipedia.org/wiki/Trust_region): optimizing the descent direction for a given step size

##### How do we choose the stopping condition?

For an unconstrained problem where $f$ is convex and differentiable, a point $x$ is optimal if and only if $\nabla f(x) = 0$, so it's natural to terminate gradient descent once the gradient becomes "small enough", i.e., $\| \nabla f(x) \| < \epsilon$.
For constrained problems, stopping conditions can similarly be derived from the [KKT conditions](https://en.wikipedia.org/wiki/Karush-Kuhn-Tucker_conditions), which are necessary, and in convex cases sufficient, conditions for optimality.

Another option is to stop iterating once the difference between successive iterates becomes small, i.e., $\|x^{k+1} - x^k\| < \epsilon$ or $\|f(x^{k+1}) - f(x^k)\| < \epsilon$. In the case of gradient descent, where the difference between iterates is influenced by the size of the gradient, these stopping conditions turn out to be similar to stopping once the gradient becomes small.

When it comes to choosing $\epsilon$, we need to balance convergence with the needs of the problem. On the one hand, if $\epsilon$ is too large, the algorithm will terminate before converging to a good solution. On the other hand, setting $\epsilon$ too small will cause the algorithm to take an unnecessarily long time. For example, if our data was recorded up to three decimal points, there would be no need to iterate until $\|x^{k+1} - x^k\| <$ 1e-6 when $\epsilon =$ 1e-3 should be sufficient. In addition to the precision of our data, another consideration for choosing $\epsilon$ is the signal-to-noise ratio in the data.

## Exercise 1: Step Size

In the example below, experiment with the step size to answer the following questions:
* What happens if the step size is too small? Try 0.001.
* What happens if the step size is too big? Try 1.001.
* Can you find a step size that converges in one step?
* Can you find a step size that results in a cycle?
* What is the Lipschitz constant of the gradient? What happens if you set the step size to $1/\beta$ or $1/\beta \pm 0.1$?

```python
step = 0.1 # modify this variable
func = lambda x: x**2
grad = lambda x: 2*x
results = utils.gradient_descent(func, grad, step=step, x0=5)
utils.plot_1d(func, results)
```

## Exercise 2: Initial Guess

In the example below, experiment with the initial guess to answer the following questions:
* Trying different values of x0, can you find a relationship between the shape of the function, the initial guess, and the number of iterations?
* Trying different values of $c \in (-2, 2)$, can you find a relationship between the shape of the function, the initial guess, and the number of iterations?

##### Condition Number

For $c \in (-2, 2)$, the function $f(x,y) = x^2 + cxy + y^2$ describes an ellipsoid with a constant Hessian $\nabla^2 f(x, y)$.
The [condition number](https://en.wikipedia.org/wiki/Condition_number#Matrices) of the matrix $\nabla^2 f(x, y)$, the ratio of its largest eigenvalue to its smallest eigenvalue, gives a measure of the ellipsoid's eccentricity:
* A condition number near one means that the ellipsoid is nearly spherical, and different initial conditions along a sublevel set will converge in a similar number of iterations. To see this, try values of $c$ near zero and initial guesses $x^0 = (\pm 1, \pm 1)$.
* A large condition number indicates that the sublevel sets are much wider in some directions than in others, so different initial conditions along a sublevel set could take much longer to converge than others. To see this, try values of $c$ near 2 and initial guesses $x^0 = (\pm 1, \pm 1)$.

```python
x0 = [0, 1]; c = 1 # modify these variables
func = lambda x: x[0]**2 + c*x[0]*x[1] + x[1]**2
grad = lambda x: np.array([2*x[0] + c*x[1], 2*x[1] + c*x[0]])
hess = [[2, c], [c, 2]]
print(f'Condition number of the Hessian: {np.linalg.cond(hess):.2f}')
results = utils.gradient_descent(func, grad, x0=x0, step=0.1)
utils.plot_2d(func, results)
```

# Stochastic Gradient Descent

For many machine learning problems with large data sets, computing the gradient can be computationally expensive, so an approximation of the gradient is used instead.
This approach is called [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent).

Consider an unconstrained optimization problem of the form

$$\min_x \frac{1}{m} \sum_{i=1}^m f_i(x)$$

with $x \in \mathbf{R}^n$ and $f_i: \mathbf{R}^n \to \mathbf{R}$, where each $f_i$ is differentiable. In many cases, each function $f_i$ corresponds to the prediction error for a single training example.
Rather than computing the full gradient $\nabla f(x) = \frac{1}{m} \sum_{i=1}^m f_i(x)$ every iteration, we can approximate it using a sample of the constituent functions.
While true stochastic gradient descent only samples one training example per iteration, it's typically more effective to use multiple examples, called a mini-batch.

Here's the algorithm for stochastic gradient descent with initial guess $x^0$, step size $\alpha^0$ (often called the [learning rate](https://en.wikipedia.org/wiki/Learning_rate) in the machine learning community), and mini-batch size $b$:

> **input** $x^0$, $\alpha^0$, $b$
> **set** $k = 0$
> **while** not converged **do**
>     Sample $b$ training examples $\mathcal{B} \subset [1, m]$
>     $x^{k+1} \leftarrow x^k - \frac{\alpha^k}{b} \sum_{i \in \mathcal{B}} \nabla f_i(x^k)$
>     $\alpha^{k+1} \leftarrow \frac{\alpha^k}{1 + dk}$
>     $k \leftarrow k + 1$
> **end while**

##### Why does the learning rate change each iteration?

One problem with approximating the gradient is that you get a noisy gradient.
Depending on the training examples used, the approximation can vary in both size and direction from the true gradient.
(For this reason, it no longer makes sense to use the size of the gradient as our stopping condition.)
One way to reduce this variance is to use a larger mini-batch, but using too large a batch size reduces the computational efficiency gained by approximating the gradient in the first place.

The same problems we had before with the step size for gradient descent are now magnified by the noisy gradient: too large a learning rate can cause the problem to diverge, while too small a learning rate can can make it slow to converge.
A common approach to this problem is to use an adaptive learning rate, defined by a learning rate schedule.
The main idea of this approach is to start with a larger learning rate, then slowly decrease it as the model nears a solution.
While this doesn't do anything to decrease the variance within the gradient estimates, it prevents the algorithm from oscillating around a solution once it's found one. Here we implement a learning rate schedule based on the previous iterate, the iterate number, and a decay parameter.

## Exercise 3: Mini-batch Size

The following two examples solve the linear least-squares problem

$$\min_x \frac{1}{m} \sum_{i=1}^m (a_i^T x - y_i)^2,$$

where $a_i \in \mathbf{R}^{784}$, $x \in \mathbf{R}^{784}$, $y_i \in \mathbf{R}$, and $m = 1000$.
Here each $a_i$ is a vector of data features, each $y_i$ is a data value, and $x$ is a vector of model parameters.
For illustrative purposes, we plot both the function value at each iteration, but it's worth noting that in practice, these values may not be available.
For large data sets, computing the function value can also be expensive, so estimates based on a subset of the observations or a test set can also be used, with the full function value being computed only periodically.

In the example below, experiment with the mini-batch size and maximum number of iterations to answer the following questions:
* Let max_iter = 1500. Using batch sizes 1 (true stochastic gradient descent), 10, 100, and 1000 (gradient descent), fill out the table below. Can you find a relationship between the batch size, minimum function value, and total computation time?

Batch Size | Function Value | Iterations | Total Time
-|-|-|-
1 |  |  |
10 |  |  |
100 |  |  |
1000 |  |  |

* Let max_iter = 7500. Using batch sizes 1 (true stochastic gradient descent), 10, 100, and 1000 (gradient descent), fill out the table below. Can you find a relationship between the batch size, minimum function value, number of iterations, and total computation time?

Batch Size | Function Value | Iterations | Total Time
-|-|-|-
1 |  |  |
10 |  |  |
100 |  |  |
1000 |  |  |

* Which of the batch sizes give a good compromise between the minimum function value and the total computation time?

```python
A = np.load('../data/train_images.npy')
y = np.load('../data/train_labels.npy')
```

```python
batch_size = 1; max_iter = 1500 # modify these variables
results = utils.stochastic_descent(A, y, batch_size=batch_size, max_iter=max_iter, learning_rate=1e-2)
utils.plot_sgd(results)
```

## Exercise 4: Decay Rate

In the example below, experiment with the decay parameter to answer the following questions:
* Using decay parameters None (constant learning rate), 1e-7, 5e-7, and 1e-6, fill out the table below. Can you find a relationship between the decay parameter, minimum function value, and total computation time?

Decay Parameter | Function Value | Iterations | Total Time
-|-|-|-
None |  |  |
1e-7 |  |  |
5e-7 |  |  |
1e-6 |  |  |

* Which of the decay parameters give a good compromise between the minimum function value and the total computation time?

```python
decay = None # modify this variable
results = utils.stochastic_descent(A, y, decay=decay, batch_size=1, max_iter=7500, learning_rate=1e-2)
utils.plot_sgd(results)
```

# Further Reading

* Kelley, C. T. (1999). *Iterative methods for optimization*. Society for Industrial and Applied Mathematics.
* Boyd, S., Boyd, S. P., & Vandenberghe, L. (2004). *Convex optimization*. Cambridge University Press.
* Nocedal, J., & Wright, S. (2006). *Numerical optimization.* Springer Science & Business Media.
* Beck, A. (2014). *Introduction to nonlinear optimization: Theory, algorithms, and applications with MATLAB*. Society for Industrial and Applied Mathematics.