# Gaussian Process Regression
**COMP9418-17s2, W09 Tutorial**

*School of Computer Science and Engineering, UNSW Sydney*

- **Instructor**: Edwin V. Bonilla
- **Teaching Assistant**: Louis Tiao

We will be using Gaussian process (GP) models for regression. A Gaussian process defines a prior distribution over functions, which we use to obtain a posterior distribution over functions after having observed some data. As such, the predictions from a GP model take the form of a full predictive distribution.

In this week's lab, we will focus gaining a practical understanding of these concepts - how to sample from a prior over functions, how to obtain and visualize the full predictive distribution, how the characteristics of a GP model changes as we vary some underlying hyperparameters, and so on. Finally, we will apply a GP regression model to a real-world dataset, and select a covariance function and its parameters to obtain automatic relevance determination (ARD), which can help us discard the irrelevant features. For a more complete treatment of Gaussian processes for regression, please refer to [GPML](http://www.gaussianprocess.org/gpml/) $\S$2.2 (Rasmussen & Williams, 2006) or [MLaPP](https://www.cs.ubc.ca/~murphyk/MLbook/) $\S$15.2 (Murphy, 2012)

This lab makes extensive use of [GPFlow](https://github.com/GPflow/GPflow), with portions derived directly from their tutorial [GP Regression with GPflow](http://gpflow.readthedocs.io/en/latest/notebooks/regression.html).

### Preamble

The following Python packages are required for this exercise:

- `numpy`
- `pandas`
- `scikit-learn`
- `tensorflow`
- `GPflow`
- `matplotlib`

Most of these may be installed with `pip`:

    pip install numpy pandas scikit-learn matplotlib tensorflow

#### Tensorflow Installation

The recommended way to install TensorFlow is in a `virtualenv` with `pip`. If your computer satisfies the [requirements](https://www.tensorflow.org/install/install_linux#nvidia_requirements_to_run_tensorflow_with_gpu_support) to run TensorFlow with GPU support, you should definitely take advantage of it. You can install TensorFlow with GPU support

    pip install tensorflow-gpu

You can also take advantage of specialized CPU instruction sets for numerical computing by compiling TensorFlow from sources. This is outside the scope of this lab and we refer you to the [install guide](https://www.tensorflow.org/install/install_sources) for this.

See the [full install guide](https://www.tensorflow.org/install/) for comprehensive instructions on installing TensorFlow for different environments with different options.

#### GPflow Installation

GPflow has not been released on PyPI. You can either download the package from the [official Github repo](https://github.com/GPflow/GPflow) and run `python setup.py install`, or install directly from Github via `pip` (recommended):

    pip install git+https://github.com/GPflow/GPflow.git

See the [install guide](http://gpflow.readthedocs.io/en/latest/intro.html#install) for further information.

Here we import the required modules and set up our notebook environment

```python
%matplotlib notebook
```

```python
from __future__ import division
from __future__ import print_function

# Required modules
import numpy as np
import pandas as pd
import tensorflow as tf
import gpflow

import matplotlib.pyplot as plt

from sklearn.datasets import load_boston
from sklearn.preprocessing import normalize
from sklearn.utils import shuffle
from IPython.display import Markdown
```

```python
plt.style.use('seaborn-notebook')
np.set_printoptions(precision=3,
                    edgeitems=5,
                    suppress=False)
```

#### Constant Definitions

```python
n_train = 12 # nbr. training points in synthetic dataset
n_query = 100 # nbr. query points
seed = 23 # set random seed for reproducibility
rng = np.random.RandomState(seed)
```

### Synthetic Dataset

First let's create a small synthetic dataset with `n_train` datapoints.

```python
X = rng.rand(n_train, 1)
Y = np.sin(12.*X) + .66*np.cos(25.*X) + rng.randn(n_train, 1)*.1 + 3.
```

```python
fig, ax = plt.subplots(figsize=(6, 4))

ax.scatter(X, Y, marker='x', color='k')

ax.set_xlabel('$x$')
ax.set_ylabel('$y$')

plt.show()
```

## Model specification

A Gaussian process model is specified in terms of a mean function $m(\mathbf{x})$ and kernel or covariance function $\kappa(\mathbf{x}, \mathbf{x}')$. The Gaussian process prior on regression functions is denoted by

$$
f(\mathbf{x}) \sim GP(m(\mathbf{x}), \kappa(\mathbf{x}, \mathbf{x}'))
$$

It is common to simply specify $m(x)=0$ since the GP is flexible enough to also model the mean arbitrarily well, so let's go ahead and specify a covariance function.

### Kernels (covariance functions)

GPFlow offers a number of commonly-used kernels, as well as an interface to define your own. See the tutorial [Using kernels in GPflow](http://gpflow.readthedocs.io/en/latest/notebooks/kernels.html) for more information. Let's take a look at the Radial Basis Function (RBF, also known as squared-exponential) kernel.

```python
# RBF kernel with input_dim=2 and lengthscale=1.
k = gpflow.kernels.RBF(1, lengthscales=1.)
```

For any finite set of $N$ points $\mathbf{X}$, the GP prior above defines a joint Gaussian

$$
p(\mathbf{f} \mid \mathbf{X}) = \mathcal{N}(\mathbf{f} \mid \boldsymbol{\mu}, \mathbf{K})
$$

where $K_{ij} = \kappa(\mathbf{x}_i, \mathbf{x}_j)$ and $\boldsymbol{\mu} = (m(\mathbf{x}_1, \dotsc, \mathbf{x}_N))$.

Let's define a set of `n_query` points.

```python
Xq = np.linspace(-5., 5., n_query).reshape(-1, 1)
```

With one input fixed at 0, we can see how the kernel changes as the other input varies and moves away from 0. The method `compute_K` takes 2 arrays as arguments and gives the kernel computed pairwise between the points in the arrays.

```python
fig, ax = plt.subplots(figsize=(5, 4.5))

ax.plot(Xq, k.compute_K(Xq, np.zeros(1).reshape(-1, 1)))

ax.set_xlabel('$x$')
ax.set_ylabel('$k(x, 0)$')
ax.set_title('RBF Kernel ($\ell={{{[0]:.2f}}}$)'
             .format(k.lengthscales.value))

plt.show()
```

Now let's call `compute_K_symm` on $\mathbf{X}$, which gives the covariance matrix $\mathbf{K}$ described above. We can then sample functions $\mathbf{f}$ from the joint Gaussian with covariance $\mathbf{K}$. This is what it means to sample functions from a GP prior. Note that $\mathbf{f}$ is a collection of function values evaluated at the finite set of points in $\mathbf{X}$.

### Exercise

Implement the function `prior_samples`, which takes a 2d array of points, a GPFlow kernel and integer argument `n_samples`, and returns `n_samples` samples of function $\mathbf{f}$ from the GP prior with zero mean function $m(\mathbf{x})=0$ and the specified kernel.

Note that while the set of `n_query` finite points `Xq` we're working with is 1-dimensional, we defined it to be a 2d array of shape `(n_query, 1)`. This is because GPFlow only accepts 2d arrays.

```python
### EXERCISE ###
def prior_samples(x, kernel, n_samples=3):

    # mean vector
    mu = np.squeeze(np.zeros_like(x))
    # covariance matrix
    K = kernel.compute_K_symm(x)

    # TODO: return n_samples from a joint Gaussian distribution
    return ...
```

```python
### SOLUTION ###
def prior_samples(x, kernel, n_samples=3):

    mu = np.squeeze(np.zeros_like(x))
    K = kernel.compute_K_symm(x)

    return rng.multivariate_normal(mu, K, n_samples).T
```

Now we can visualize draws from the GP prior.

```python
fig, ax = plt.subplots(figsize=(6, 4))

ax.plot(Xq, prior_samples(Xq, k))

ax.set_xlabel('$x$')
ax.set_ylabel('$f(x)$')
ax.set_title('Draws of $f(x)$ from GP prior')

plt.show()
```

#### Effects of kernel parameters

The kernel variance parameter $\sigma_f$ (`k.variance`) controls the vertical scale of the function, while the length scale $\ell$ (`k.lengthscales`) is the horizontal scale over which the function changes. The sampled functions look smoother as we increase the length scale. This makes sense as the effective distance between query points is inversely proportional to the length scale. Increasing the length scale causes the function values to become less correlated.

```python
fig, axes = plt.subplots(nrows=3, ncols=3, figsize=(7, 7),
                         subplot_kw=dict(xticks=[], yticks=[]))
fig.tight_layout()

for i, ax_row in enumerate(axes):

    variance = .05 * (i + 1)

    for j, ax in enumerate(ax_row):

        len_scale = .1 + .7 * j

        k = gpflow.kernels.RBF(1, variance=variance,
                               lengthscales=len_scale)

        ax.plot(Xq, prior_samples(Xq, k))
        ax.set_ylim(-1.5, 1.5)

        if i == len(axes) - 1:
            ax.set_xlabel('$\ell={{{:.2f}}}$'.format(len_scale))

        if not j:
            ax.set_ylabel('$\sigma_f={{{:.2f}}}$'.format(variance))

plt.show()
```

### Posterior predictive distribution

Now suppose we observe the training set $(\mathbf{X}, \mathbf{y})$ we created earlier, where $y_i = f(\mathbf{x}_i) + \epsilon$ is the noisy observation of the function evaluated at $\mathbf{x}_i$ and noise $\epsilon \sim \mathcal{N}(0, \sigma_y^2)$. Given a query set $\mathbf{X}_*$, we can obtain a full posterior distribution over the function outputs $\mathbf{f}_*$. The joint distribution has the following form

$$
\begin{pmatrix}
  \mathbf{y}\\
  \mathbf{f}_*
\end{pmatrix}
\sim
\mathcal{N} \left (
  0,
  \begin{pmatrix}
    \mathbf{K}_y   & \mathbf{K}_* \\
    \mathbf{K}_*^T & \mathbf{K}_{**}
  \end{pmatrix}
\right )
$$

where

$$
\begin{align}
  \mathbf{K}_y    &= \mathbf{K} + \sigma_y^2\mathbf{I} \\
  \mathbf{K}      &= \kappa(\mathbf{X}, \mathbf{X}) \\
  \mathbf{K}_*    &= \kappa(\mathbf{X}, \mathbf{X}_*) \\
  \mathbf{K}_{**} &= \kappa(\mathbf{X}_*, \mathbf{X}_*)
\end{align}
$$

Then, using the standard method for conditioning Gaussians, the posterior predictive density is given by

$$
\begin{align}
  p(\mathbf{f}_* \mid \mathbf{X}_*, \mathbf{X}, \mathbf{y})
  &= \mathcal{N}(\mathbf{f}_*, \mu_*, \Sigma_*) \\
  \mu_* &= \mathbf{K}_*^T \mathbf{K}_y^{-1} \mathbf{y} \\
  \Sigma_* &= \mathbf{K}_{**} - \mathbf{K}_*^T \mathbf{K}_y^{-1} \mathbf{K}_*
\end{align}
$$

Let's instantiate a GP regression model, conditioned on the synthetic training set (`X`, `Y`), with a squared-exponential kernel, and a sensible initial setting of the variance and length scale hyperparameters ($\sigma_y, \sigma_f, \ell$).

```python
k = gpflow.kernels.RBF(1, lengthscales=.2)
m = gpflow.gpr.GPR(X, Y, kern=k)
m.likelihood.variance = 0.01
```

For a given query set $\mathbf{X}_*$, `m.predict_y` returns the means and variances $\mu_*, \Sigma_*$. Additionally, `m.predict_f_samples` yields samples from our posterior. We can use these to visualize the full posterior predictive distribution.

#### Exercise

Complete the indicated portions of the function `plot_posterior_predictive` below.

```python
### EXERCISE ###
def plot_posterior_predictive(model, start=0., stop=1., n_query=100,
                              n_samples=3, ax=None):

    if ax is None:
        ax = plt.gca()

    Xq = np.linspace(start, stop, n_query).reshape(-1, 1)
    # COMPLETE ME #
    # Compute the mean and variances of the predictive
    # distribution, given the query points `Xq`
    mean, var = ...

    # scatter plot of the training points
    ax.scatter(model.X.value, model.Y.value, marker='x', color='k')

    # predictive mean mu_*
    ax.plot(Xq, mean, 'b', lw=2., label='$m(x)$')

    # COMPLETE ME #
    # Plot a shaded region that represents mu_* +- 2 std(f_*)
    ax.fill_between(np.squeeze(Xq),
                    # mu_* - 2 std(f_*)
                    # mu_* + 2 std(f_*)
                    color='blue', alpha=.2)

    # samples from the posterior distribution
    ax.plot(Xq, np.squeeze(model.predict_f_samples(Xq, n_samples)).T,
            '--', alpha=.8)

    ax.legend()
```

```python
### SOLUTION ###
def plot_posterior_predictive(model, start=0., stop=1., n_query=100,
                              n_samples=3, ax=None):

    if ax is None:
        ax = plt.gca()

    Xq = np.linspace(start, stop, n_query).reshape(-1, 1)
    mean, var = model.predict_y(Xq)

    # scatter plot of the training points
    ax.scatter(model.X.value, model.Y.value, marker='x', color='k')

    # predictive mean mu_*
    ax.plot(Xq, mean, 'b', lw=2., label='$m(x)$')

    ax.fill_between(np.squeeze(Xq),
                    np.squeeze(mean - 2*np.sqrt(var)),
                    np.squeeze(mean + 2*np.sqrt(var)),
                    color='blue', alpha=.2)

    # samples from the posterior distribution
    ax.plot(Xq, np.squeeze(model.predict_f_samples(Xq, n_samples)).T,
            '--', alpha=.8)

    ax.legend()
```

This is what the posterior predictive distribution looks like under our GP model. It may seem that our model doesn't yield a great fit. The function is smooth and doesn't interpolate the observed values very well. Furthermore, there is a high degree of uncertainty at the observed points. If we believe our prior and our likelihood models then this is the right answer. However, as we shall see below, we can also estimate the model hyperparameters from the data.

```python
fig, ax = plt.subplots(figsize=(6, 4))

plot_posterior_predictive(m, ax=ax)

plt.show()
```

Similar to what we did with the prior distribution, we can visualize the effects of varying the length scale parameter $\ell$.

```python
fig, axes = plt.subplots(nrows=3, figsize=(6, 8))
#                          subplot_kw=dict(xticks=[], yticks=[]))
fig.tight_layout()

for i, ax in enumerate(axes):

    len_scale = .05 * (i + 1)

    k = gpflow.kernels.RBF(1, lengthscales=len_scale)
    m = gpflow.gpr.GPR(X, Y, kern=k)
    m.likelihood.variance = .1

    ax.set_title('$\ell={{{:.2f}}}$'.format(len_scale))
    plot_posterior_predictive(m, ax=ax)

plt.show()
```

### Hyperparameter Estimation

To find the kernel parameters that result in the best fit, it is prohibitively slow to exhaustively search over a discrete grid of values to minimize some validation loss. Instead, we resort to an empirical Bayes approach (also known as type II maximum likelihood) where we maximize the *marginal likelihood*. This is much faster and amenable to gradient-based optimization methods.

GPflow implements this in the method `m.optimize` and uses L-BFGS-B from SciPy as the underlying gradient-based optimization procedure. First, we initialize our GP model again an display its parameters:

```python
k = gpflow.kernels.RBF(1, lengthscales=.3)
m = gpflow.gpr.GPR(X, Y, kern=k)
m.likelihood.variance = 0.01
m
```

Now let us optimize these parameters and display the full predictive distribution again.

```python
m.optimize()
m
```

We see that this model fits the data much better than our initial model. It interpolates the observed values well and has the highest degree of uncertainty at points furthest away from the observed datapoints.

```python
fig, ax = plt.subplots(figsize=(6, 4))

plot_posterior_predictive(m, ax=ax)

plt.show()
```

### Other covariance functions

As we mentioned early, GPFlow implements a number of common covariance functions. Let us experiment with the following subset of them.

```python
kernels = [
    gpflow.kernels.Matern12(1),
    gpflow.kernels.Matern32(1),
    gpflow.kernels.Matern52(1),
    gpflow.kernels.RBF(1),
]
```

#### Exercise

As we did for the squared-exponential (RBF) kernel, draw and visualize 3 samples from the GP priors specified by the kernels in the list defined above.

```python
### EXERCISE ###
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(6, 6),
                         subplot_kw=dict(xticks=[], yticks=[]))
fig.tight_layout()

for ax, k in zip(axes.flat, kernels):
    ### COMPLETE ME ###
    ax.set_title(k.__class__.__name__)

plt.show()
```

```python
### SOLUTION ###
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(6, 6),
                         subplot_kw=dict(xticks=[], yticks=[]))
fig.tight_layout()

for ax, k in zip(axes.flat, kernels):

    ax.set_title(k.__class__.__name__)
    ax.plot(Xq, prior_samples(Xq, k))

plt.show()
```

#### Exercise

For each of the kernels in the list defined above, fit the hyperparameters to the training set and plot the full posterior predictive distribution as we did before for the squared-exponential kernel.

Note that the log marginal likelihood is non-convex so a sensible initialization of the hyperparameters is crucial to avoid getting stuck in a local optimum.

```python
### EXERCISE ###
fig, axes = plt.subplots(nrows=4, figsize=(6, 10))
fig.tight_layout()

for ax, k in zip(axes, kernels):

    m = gpflow.gpr.GPR(X, Y, kern=k)

    ### COMPLETE ME ###

    ax.set_title(
        ('{k} ($\ell={{{l[0]:.2f}}},'
         ' \sigma_f={{{sf[0]:.2f}}},'
         ' \sigma_y={{{sy[0]:.2f}}}$)').format(k=k.__class__.__name__,
                                               l=m.kern.lengthscales.value,
                                               sf=m.kern.variance.value,
                                               sy=m.likelihood.variance.value))

    plot_posterior_predictive(m, ax=ax)

plt.show()
```

```python
### SOLUTION ###
fig, axes = plt.subplots(nrows=4, figsize=(6, 10))
fig.tight_layout()

for ax, k in zip(axes, kernels):

    m = gpflow.gpr.GPR(X, Y, kern=k)
    m.likelihood.variance = .01
    m.kern.lengthscales = .1
    m.optimize()

    ax.set_title(
        ('{k} ($\ell={{{l[0]:.2f}}},'
         ' \sigma_f={{{sf[0]:.2f}}},'
         ' \sigma_y={{{sy[0]:.2f}}}$)').format(k=k.__class__.__name__,
                                               l=m.kern.lengthscales.value,
                                               sf=m.kern.variance.value,
                                               sy=m.likelihood.variance.value))

    plot_posterior_predictive(m, ax=ax)

plt.show()
```

### Real-world Dataset: Boston Housing Prices

Finally, we apply our GP regression model to the Boston housing prices dataset.

```python
boston = load_boston()
```

The dataset has 506 datapoints, with 13 continuous/categorical features and a single target, the median property value. You can view a full description of the dataset and its features:

```python
Markdown(boston.DESCR)
```

We can load this dataset into a Pandas DataFrame and view some of its summary statistics.

```python
boston_df = pd.DataFrame(boston.data, columns=boston.feature_names)
boston_df['MEDV'] = pd.Series(boston.target)
boston_df.describe()
```

For example, we can see how the median property value (`MEDV`) varies with the average number of rooms per dwelling (`RM`).

```python
fig, ax = plt.subplots()

boston_df.plot.scatter(x='RM', y='MEDV', alpha=.8, ax=ax)

plt.show()
```

### Automatic Relevance Determination (ARD)

Now let us fit a GP regression model to this dataset. We consider the squared-exponential covariance function as before, except now, we use the *anisotropic* variant of the kernel. That is, we consider a length scale *vector* of 13 positive values.

These hyperparameter values determine how far you need to move along a particular axis in the input space for the function values to become uncorrelated. By estimating these values we effectively implement automatic relevance determination, as the inverse of the length scale determines the relevance of the dimension. If the length scale is very large, the covariance will practically become independence of that input, and effectively remove it from the inference (GPML $\S$5.1 Rasmussen & Williams, 2006).

```python
X, y = shuffle(normalize(boston.data), # scaling individual samples to have unit norm
               np.atleast_2d(boston.target).T, # gpflow labels must be 2D
               random_state=rng)
```

In GPFlow, we can easily obtain an anisotropic kernel by specifying `ARD=True`.

```python
k = gpflow.kernels.RBF(13, ARD=True)
m = gpflow.gpr.GPR(X, y, k)
m.likelihood.variance = .1
m.optimize()
m
```

Let's create a Pandas Series to hold the length scales for ease of visualization and manipulation.

```python
s = pd.Series(m.kern.lengthscales.value, index=boston.feature_names)
s
```

We display the bar chart of the length scales corresponding to each dimension.

```python
fig, ax = plt.subplots()

s.plot.bar(ax=ax)
ax.set_ylabel('$\ell_i$')

plt.show()
```

Generating the scatter plot with respect to the feature that has the smallest length scale, we find that it is indeed highly correlated with the median property value.

```python
fig, ax = plt.subplots()

boston_df.plot.scatter(x=s.argmin(), y='MEDV', alpha=.8, ax=ax)

plt.show()
```

And vice versa for the feature with the largest length scale.

```python
fig, ax = plt.subplots()

boston_df.plot.scatter(x=s.argmax(), y='MEDV', alpha=.8, ax=ax)

plt.show()
```

```python

```


# Euler Method with Theorems Applied to Non-Linear Population Equations

The more general form of a first order Ordinary Differential Equation is:
\begin{equation}
y^{'}=f(t,y).
\end{equation}
This can be solved analytically by integrating both sides but this is not straight forward for most problems.
Numerical methods can be used to approximate the solution at discrete points.
In this notebook we will work through the Euler method for two initial value problems:

1. A non-linear sigmoidal population equation
\begin{equation}
y^{'}=0.2 y− 0.01 y^2 ,
\end{equation}
2. A non-linear sigmoidal population differential equation with a wiggle,
\begin{equation}
y^{'}=0.2 y-0.01 y^2+\sin(2\pi t).
\end{equation}

## Euler method

The simplest one step numerical method is the Euler Method named after the most prolific of mathematicians [Leonhard Euler](https://en.wikipedia.org/wiki/Leonhard_Euler) (15 April 1707 – 18 September 1783) .

The general Euler formula for to the first order differential equation
\begin{equation}
y^{'} = f(t,y),
\end{equation}

approximates the derivative at time point $t_i$,
\begin{equation}
y^{'}(t_i) \approx \frac{w_{i+1}-w_i}{t_{i+1}-t_{i}},
\end{equation}
where $w_i$ is the approximate solution of $y$ at time $t_i$.
This substitution changes the differential equation  into a __difference__ equation of the form
\begin{equation}
\frac{w_{i+1}-w_i}{t_{i+1}-t_{i}}=f(t_i,w_i).
\end{equation}
Assuming uniform stepsize $t_{i+1}-t_{i}$ is replaced by $h$, re-arranging the equation gives
\begin{equation}
w_{i+1}=w_i+hf(t_i,w_i),
\end{equation}
 This can be read as the future $w_{i+1}$ can be approximated by the present $w_i$ and the addition of the input to the system $f(t,y)$ times the time step.

```python
## Library
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
import pandas as pd

warnings.filterwarnings("ignore")
```

# Non-linear population equation

The general form of the non-linear sigmoidal population growth differential equation is:
\begin{equation}
y^{'}=\alpha y-\beta y^2,
\end{equation}
where $\alpha$ is the growth rate and $\beta$ is the death rate. The initial population at time $ a $ is
\begin{equation}
 y(a)=A,
\end{equation}
\begin{equation}
 a\leq t \leq b.
\end{equation}

## Specific non-linear population equation
Given the growth rate $$\alpha=0.2,$$ and death rate $$\beta=0.01,$$ giving the specific differential equation,
\begin{equation}
 y^{'}=0.2 y-0.01 y^2,
\end{equation}
The initial population at time $2000$ is
\begin{equation}
 y(2000)=6,
\end{equation}
we are interested in the time period
\begin{equation}
 2000\leq t \leq 2020.
\end{equation}

## Initial Condition
To get a specify solution to a first order initial value problem, an __initial condition__ is required.
For our population problem the initial population is 6 billion people:
\begin{equation}
y(2000)=6.
\end{equation}

## General Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation}
 h=\frac{b-a}{N}.
\end{equation}
## Specific Discrete Interval
Here the interval is $2000\leq t \leq 2020$ with $N=200$
\begin{equation}
h=\frac{2020-2000}{200}=0.1,
\end{equation}
this gives the 201 discrete points with stepsize h=0.1:
\begin{equation}
t_0=2000, \ t_1=0.1, \ ... t_{200}=2020,
\end{equation}
which is generalised to
\begin{equation}
 t_i=2000+i0.1, \ \ \ i=0,1,...,200.
\end{equation}
The plot below illustrates the discrete time steps from 2000 to 2002.

```python
### Setting up time
t_end=2020.0
t_start=2000.0
N=200
h=(t_end-t_start)/(N)
time=np.arange(t_start,t_end+0.01,h)
fig = plt.figure(figsize=(10,4))
plt.plot(time,0*time,'o:',color='red')
plt.title('Illustration of discrete time points for h=%s'%(h))
plt.xlim((2000,2002))
plt.plot();
```

## Numerical approximation of Population growth
The differential equation is transformed using the Euler method into a difference equation of the form
\begin{equation}
      w_{i+1}=w_{i}+h (\alpha w_i-\beta w_i\times w_i).
\end{equation}
This approximates a series of of values $w_0, \ w_1, \ ..., w_{N}$.
For the specific example of the population equation the difference equation is,
\begin{equation}
       w_{i+1}=w_{i}+h 0.1 [0.2 w_i-0.01 w_i\times w_i],
\end{equation}
where $i=0,1,2,...,199$, and $w_0=6$. From this initial condition the series is approximated.

```python
w=np.zeros(N+1)
w[0]=6
for i in range (0,N):
    w[i+1]=w[i]+h*(0.2*w[i]-0.01*w[i]*w[i])

```

The plot below shows the Euler approximation $w$ in blue squares.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time,w,'s:',color='blue',label='Euler')
plt.xlim((min(time),max(time)))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Euler solution')
plt.plot();
```

### Table
The table below shows the iteration $i$, the discrete time point t[i], and the Euler approximation w[i] of the solution $y$ at time point t[i] for the non-linear population equation.

```python
d = {'time t[i]': time[0:10],    'Euler (w_i) ':w[0:10]}
df = pd.DataFrame(data=d)
df
```

      time t[i]
      Euler (w_i)

      0
      2000.0
      6.000000

      1
      2000.1
      6.084000

      2
      2000.2
      6.168665

      3
      2000.3
      6.253986

      4
      2000.4
      6.339953

      5
      2000.5
      6.426557

      6
      2000.6
      6.513788

      7
      2000.7
      6.601634

      8
      2000.8
      6.690085

      9
      2000.9
      6.779130

## Numerical Error
With a numerical solution there are two types of error:
* local truncation error at one time step;
* global error which is the propagation of local error.

### Derivation of  Euler Local truncation error
The left hand side of a initial value problem $\frac{dy}{dt}$ is approximated by __Taylors theorem__ expand about a point $t_0$ giving:
\begin{equation}
y(t_1) = y(t_0)+(t_1-t_0)y^{'}(t_0) + \frac{(t_1-t_0)^2}{2!}y^{''}(\xi), \ \ \ \ \ \ \xi \in [t_0,t_1].
\end{equation}
Rearranging and letting $h=t_1-t_0$ the equation becomes
\begin{equation}
y^{'}(t_0)=\frac{y(t_1)-y(t_0)}{h}-\frac{h}{2}y^{''}(\xi).
\end{equation}
From this the local truncation error is
\begin{equation}
\tau \leq \frac{h}{2}M,
\end{equation}
where $y^{''}(t) \leq M $.
#### Derivation of  Euler Local truncation error for the Population Growth
As the exact solution $y$ is unknown we cannot get an exact estimate of the second derivative
\begin{equation}
y'(t)=0.2 y-0.01 y^2,
\end{equation}
differentiate with respect to $t$,
\begin{equation}
y''(t)=0.2 y'-0.01 (2yy'),
\end{equation}
subbing the original equation gives
\begin{equation}
y''(t)=0.2 (0.2 y-0.01 y^2)-0.01 \big(2y(0.2 y-0.01 y^2)\big),
\end{equation}
which expresses the second derivative as a function of the exact solution $y$, this is still a problem as the value of $y$ is unknown, to side step this issue we assume the population is between $0\le y \le 20,$ this gives
\begin{equation}
\max|y''|=M\leq 0.8,
\end{equation}
this gives a local trucation for $h=0.1$ for our non-linear equation is
\begin{equation}
\tau=\frac{h}{2}0.8=0.04.
\end{equation}

```python
M=0.8
fig = plt.figure(figsize=(10,4))
plt.plot(time[0:2],0.1*M/2*np.ones(2),'v:'
         ,color='black',label='Upper Local Truncation')
plt.xlabel('time')
plt.ylim([0,0.1])
plt.legend(loc='best')
plt.title('Local Truncation Error')
plt.plot();
```

### Global truncation error for the population equation
For the population equation specific values $L$ and $M$ can be calculated.

In this case $f(t,y)=\epsilon y$ is continuous and satisfies a Lipschitz Condition with constant
\begin{equation}
\left|\frac{\partial f(t,y)}{\partial y}\right|\leq L,
\end{equation}
\begin{equation}
 \left|\frac{\partial (0.2 y-0.01 y^2)}{\partial y}\right|\leq |0.2-0.01(2\times y)| \leq |0.2-0.01(2\times 50)|\leq 0.8,
\end{equation}

on $D=\{(t,y)|2000\leq t \leq 2015, 0 < y < 50 \}$ and that a constant $M$
exists with the property that
\begin{equation}
|y^{''}(t)|\leq M\leq 6.
\end{equation}

__Specific Theorem Global Error__

Let $y(t)$ denote the unique solution of the Initial Value Problem
\begin{equation}
y^{'}=0.2 y-0.01 y^2, \ \ \ 2000\leq t \leq 2020, \ \ \ y(0)=6,
\end{equation}
and $w_0,w_1,...,w_N$ be the approx generated by the Euler method for some
positive integer N.  Then for $i=0,1,...,N$ the error is:
\begin{equation}
|y(t_i)-w_i| \leq \frac{6 h}{2\times 0.8}|e^{0.8(t_i-2000)}-1|.
\end{equation}

# Non-linear population equation with a temporal oscilation
Given the specific population differential equation with a wiggle,
\begin{equation}
y^{'}=0.2 y-0.01 y^2+sin(2\pi t),
\end{equation}
with the initial population at time $2000$ is
\begin{equation}
y(2000)=6,
\end{equation}
\begin{equation}
2000\leq t \leq 2020.
\end{equation}

For the specific example of the population equation the difference equation is
\begin{equation}
      w_{i+1}=w_{i}+h 0.5 (0.2 w_i-0.02 w_i\times w_i+sin(2 \pi t),
\end{equation}
for $i=0,1,...,199$,
where $w_0=6$. From this initial condition the series is approximated.
The figure below shows the discrete solution.

```python
w=np.zeros(N+1)
w[0]=6
for i in range (0,N):
    w[i+1]=w[i]+h*(0.2*w[i]-0.01*w[i]*w[i]+np.sin(2*np.pi*time[i]))

fig = plt.figure(figsize=(10,4))
plt.plot(time,w,'s:',color='blue',label='Euler')
plt.xlim((min(time),max(time)))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Euler solution')
plt.plot();
```

### Table
The table below shows the iteration $i$, the discrete time point t[i], and the Euler approximation w[i] of the solution $y$ at time point t[i] for the non-linear population equation with a temporal oscilation.

```python

d = {'time t_i': time[0:10],    'Euler (w_i) ':w[0:10]}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Euler (w_i)

      0
      2000.0
      6.000000

      1
      2000.1
      6.084000

      2
      2000.2
      6.227443

      3
      2000.3
      6.408317

      4
      2000.4
      6.590522

      5
      2000.5
      6.737676

      6
      2000.6
      6.827034

      7
      2000.7
      6.858187

      8
      2000.8
      6.853211

      9
      2000.9
      6.848203

```python

```

Chapter 01 - Euler Methods/102_Euler_method_with_Theorems_nonlinear_Growth_function.ipynb


# Identifying special matrices

Test if a 4×4 matrix is singular, i.e. to determine if an inverse exists, before calculating it.
Use the method of converting a matrix to echelon form, and testing if this fails by leaving zeros that can’t be removed on the leading diagonal.

```python
% matplotlib inline

import numpy as np
import numpy.linalg as la
```

```python
# Print option; suppress small numbers and exp, precision, numbers shown in matrix
# https://docs.scipy.org/doc/numpy/reference/generated/numpy.set_printoptions.html
# np.set_printoptions()  # formatter gets reset
np.set_printoptions(suppress=True, precision=3, threshold=10)
```

```python
import sympy as sp
from sympy import Matrix as spm
sp.init_printing(use_unicode=True)
from mpmath import *
mp.dps = 3 # set decimal places
```

```python
%precision 3
print(mp)
mpf(2.00000001)
2.0000001
```

```python
class MatrixIsSingular(Exception):
    """Exception extension for errors if the matrix is singular."""
    def __init__(self, wth, msg=None):
        if msg is None:
            # Pass error message
            msg = 'Runtime error: {}'.format(wth)
        super(MatrixIsSingular, self).__init__(msg)
        self.wth = wth
```

### Manual RREF

```python
def isSingular(A):
    """Try RREF, when diagonal can't be made 1, the matrix is singular"""
    # Work with copy of mutable input
    B = np.array(A, dtype=np.float_)
    try:
        fixRowZero(B)
        fixRowOne(B)
        fixRowTwo(B)
        fixRowThree(B)
    except MatrixIsSingular:
        return True
    return False
```

```python
def fixRowZero(A):
    """"""
    if A[0,0] == 0:                 # test diagonal for non-zero
        A[0] = A[0] + A[1]          # add lower vector to populate diagonal
    if A[0,0] == 0:                 # repeat for all rows ""
        A[0] = A[0] + A[2]
    if A[0,0] == 0:
        A[0] = A[0] + A[3]
    if A[0,0] == 0:
        raise MatrixIsSingular('fixRowZero')
    A[0] = A[0] / A[0,0]            # Normalize row to set the diagonal element to one
    return A
```

```python
def fixRowOne(A):
    A[1] = A[1] - A[1,0] * A[0]     # subtract upper vector to make lower-triangle zero

    if A[1,1] == 0:                 # test diagonal for non-zero
        A[1] = A[1] + A[2]          # add lower vector to populate diagonal
        A[1] = A[1] - A[1,0] * A[0] # subtract upper vector to make lower-triangle zero

    if A[1,1] == 0:                 # again test diagonal for non-zero
        A[1] = A[1] + A[3]          # add lower vector to populate diagonal
        A[1] = A[1] - A[1,0] * A[0] # subtract upper vector to make lower-triangle zero

    if A[1,1] == 0:                 # final test diagonal for non-zero
        raise MatrixIsSingular('fixRowOne')
    A[1] = A[1] / A[1,1]            # Normalize row to set the diagonal element to one
    return A
```

```python
def fixRowTwo(A) :
    A[2] = A[2] - A[2,0] * A[0]
    A[2] = A[2] - A[2,1] * A[1]

    if A[2,2] == 0:
        A[2] = A[2] + A[3]
        A[2] = A[2] - A[2,0] * A[0]
        A[2] = A[2] - A[2,1] * A[1]

    if A[2,2] == 0:
        raise MatrixIsSingular('fixRowTwo')
    A[2] = A[2] / A[2,2]
    return A
```

```python
def fixRowThree(A) :
    A[3] = A[3] - A[3,0] * A[0]
    A[3] = A[3] - A[3,1] * A[1]
    A[3] = A[3] - A[3,2] * A[2]

    if A[3,3] == 0:
        raise MatrixIsSingular('fixRowThree')
    A[3] = A[3] / A[3,3]
    return A
```

#### Test your code

```python
A = np.array([
        [0, 7, -5, 3],
        [2, 8, 0, 4],
        [3, 12, 0, 5],
        [1, 3, 1, 3]
    ], dtype=np.float_)
isSingular(A)
spm(A)
```

```python
spm(fixRowZero(A))
## row 1 added to 0 and devided by A[0,0]
```

```python
spm(fixRowOne(A).round(2))
```

```python
spm(fixRowTwo(A).round(2))
```

```python
spm(fixRowThree(A).round(2))
```

## sympy

http://docs.sympy.org/0.7.5/tutorial/matrices.html

```python
A = np.array([
        [2, 0, 0, 0],
        [0, 3, 0, 0],
        [0, 0, 4, 4],
        [0, 0, 5, 5]
    ], dtype=np.float_)
isSingular(A)
spm(A)
```

```python
spm(A).rref()[1]         # indices of the pivot columns
spm(A).rref()[0]         # rref
spm(A).nullspace()[0]    # nullspace
spm(A).diagonalize()     #
spm(A).det()             # determinant
if spm(A).det() == 0:
    raise MatrixIsSingular('determinant is 0')
spm(A)**-1           # matrix inverse
spm(A)**-1 @ spm(A)  # A-1 @ A = I - check inverse/rref
```

### Invertible matrix - TODO nxm matrices

```python
def is_invertible(B):
    Bp = np.linalg.inv(B)
    I = np.eye(B.shape[1], B.shape[1])

    assert spm(B).det != 0, 'matrix is singular'
    np.testing.assert_array_almost_equal(B @ I, B, err_msg='not invertible')
    np.testing.assert_array_almost_equal(I, Bp @ B, err_msg='not invertible')
    np.testing.assert_array_almost_equal(I, B @ Bp, err_msg='not invertible')
    np.testing.assert_array_almost_equal(Bp @ B, B @ Bp, err_msg='not invertible')
    return True

spm(B)
is_invertible(B)
```

### Transformation of base

```python
A = np.array([
        [2, 1],
        [1, 2]
    ], dtype=np.float_)
```

```python
# New base
B = np.array([
        [3, 1],
        [1, 1]
    ], dtype=np.float_)
B
```

```python
# Rotation 45
R = np.array([
        [1, -1],
        [1, 1]
    ], dtype=np.float_) #/np.sqrt(2)
R
```

```python
# Identity
I = np.eye(2)
I
```

```python
# Back to base
Binv = np.linalg.inv(B)
```

```python
spm(A)
spm((Binv @ I @ B).round(2))
spm(Binv @ R @ B)
spm((Binv @ A @ B).round(2))
```

## Eigen experiments

Eigen vectors `x` and eigen values `lambda`:
```python
A @ x = lambda @ x
A @ x - lambda @ I @ x = 0
```

```python
%precision 3
B_eig = spm(B).eigenvects()[0][2][0]
B_eigv = spm(B).eigenvects()[0][0]
spm(B).eigenvects()
B @ B_eig
B_eigv * B_eig
(B @ B_eig - B_eigv @ spm(np.eye(B.shape[0])) @ B_eig)

l, v = la.eig(B)
l
v
B
l[0] * v[:,0], B @ v[:,0]
np.testing.assert_array_almost_equal(l[0] * v[:,0], B @ v[:,0])

```

### SVD Singular Value Decomposition
When a is a 2D array, it is factorized as u @ np.diag(s) @ vh = (u * s) @ vh,
where u and vh are 2D unitary arrays and s is a 1D array of a‘s singular values.

```python
u, s, vh = np.linalg.svd(B, full_matrices=True)
u
s
vh
u @ s @ vh
```

```python
# Transform in original
a, b, c, d = 1, 0, 2, -1
T = np.array([
        [a, b],
        [c, d]
    ], dtype=np.float_)
```

```python
# e = values, v = unit eigen matrix (normalized)
e, C = la.eig(T)
'C'
spm(C.round(2))
'C_inv'
spm(la.inv(C).round(2))
'Eigenvalues: {}'.format(e)

```

```python
%precision 3
# Change of basis matrix with Eigen vectors of T
a, b, c, d = 1, 2, 0, 1
C = np.array([
        [a, b],
        [c, d]
    ], dtype=np.float_)
'C'
spm(C)

# Inverse Eigen matrix
Cinv = la.inv(C)
'Cinv'
spm(Cinv)

# Diagonalized matrix of eigen values
D = la.inv(C) @ T @ C
'D'
spm(D)

A = C @ D @ la.inv(C)
'A'
spm(A)

An = C @ D @ D @ D @ D @ D @ la.inv(C)
'An'
spm(An)

```

```python
# Transform in original
a, b, c, d, e, f, g, h, i = 4,-5,6, 7,-8,6, 3/2,-1/2,-2
A = np.array([
        [a, b, c],
        [d, e, f],
        [g, h, i]
    ], dtype=np.float_)
vals, vecs = la.eig(A)
print(vals)
print(vecs)
print('sqrt:', 6, 1/np.sqrt(6))

# Normalized
i, j, k = 0.5,-.5,-1
v = np.array([[i, j, k]])
print('normalized: ', v/la.norm(v))
```

### Eigen vector - experiment
```python
Eigen vectors A@x = lambda@x => (A@x - lambda@I@x) = 0
e = values, v = unit eigen matrix (normalized)
```

#### Complex to real numbers

```python
A = np.array([[0,0,0,1], [1,0,0,0], [0,1,0,0], [0,0,1,0]])
vals, vecs = la.eig(A)
vals
# np.real() converts complex numbers to real numbers
np.real(vals)
np.real(vecs)
```

```python
A = np.array([[0.0,0.0,0.0,1.0], [1.0,0.0,0.0,0.0], [0.0,1.0,0.0,0.0], [0.0,0.0,1.0,0.0]])
D = np.array([[0.1,0.1,0.1,0.7], [0.7,0.1,0.1,0.1], [0.1,0.7,0.1,0.1], [0.1,0.1,0.7,0.1]])
d = 0.5

M = d * A + (d - 1) * D/4
spm(A), spm(D), spm(M.round(2))
vals, vecs = la.eig(M)

np.real(vals)
np.real(vecs)
```

```python
d = 0.9
M = d * A + (d - 1) * D/4
vals, vecs = la.eig(M)

np.real(vals)
spm(np.real(vecs).round(2))
```

```python
a, b, c, d = 3/2, -1, -1/2, 1/2
T = np.array([
        [a, b],
        [c, d]
    ], dtype=np.float_)
spm(T)
```

```python

vals, vecs = la.eig(T)
print('vals', vals)
C = vecs
spm(C)

D = la.inv(C) @ T @ C
spm(D)

D_2 = vals * np.eye(2)
spm(D_2)

A2 = C @ D @ D @ la.inv(C)
spm(A2)

spm(T @ T)
```

### Eigen stuff - sympy vs. numpy.linalg

```python
-1 + sp.sqrt(3)
-1 + np.sqrt(3)
```

```python
a, b, c, d, e, f, g, h, i = 4,-5,6, 7,-8,6, 3/2,-1/2,-2
A = np.array([
        [a, b, c],
        [d, e, f],
        [g, h, i]
    ], dtype=np.float_)
spm(A)
```

```python
spm(np.array([4,-5,6, 7,-8,6, 3/2,-1/2,-2], dtype=np.float_).reshape(-1,3))
```

```python
# sympy - algebraic('non-normalised') eigenvectors
# numpy.linalg - numeric eigenvectors
sorted(spm(A).eigenvects(), key=lambda x: -x[0])
```

### Eigen functions

```python
def eigen(X, descending=True):
    """Eigen values and -vectors in descending order"""
    from collections import defaultdict
    eigen_dict = defaultdict()
    for i, v in enumerate(spm(A).eigenvects()):
        eigen_dict[spm(X).eigenvects()[i][0]] = spm(X).eigenvects()[i][2][0]
    return sorted(eigen_dict.items(), key=lambda x: [1, -1][descending] * x[0])

def eigen_sym(X, normalised=False, descending=True):
    """Eigen values and -vectors in descending order"""
    from collections import defaultdict
    eigen_dict = defaultdict()
    for i, v in enumerate(spm(X).eigenvects()):
        norm = [1, (spm(X).eigenvects()[i][2][0].T @ spm(X).eigenvects()[i][2][0])**.5][normalised]
        eigen_dict[spm(X).eigenvects()[i][0]] = spm(X).eigenvects()[i][2][0]/norm
    return sorted(eigen_dict.items(), key=lambda x: [1, -1][descending] * x[0])

def eigen_la(X, descending=True):
    """Eigen values and -vectors in descending order"""
    vals, vecs = la.eig(X)
    idx = np.argsort([1, -1][descending] * vals)
    return [(val.round(), spm(vec)) for val, vec in zip(vals[idx], vecs.T[idx])]
```

```python
eigen_sym(A)
eigen_sym(A, False)
eigen_sym(A, True)
```

```python
eigen_la(A, True)
```

### Angle (in radius) between vectors x and z

```python
def v_len(x):
    return (x.T @ x)**.5

def radius(x, y):
    x_, y_ = v_len(x), v_len(y)
    if x_ == y_: return 0
    return np.arccos(x.T @ y / (x_ * y_))
```

```python
x = np.array([1,0,6])
y = np.array([-1,0,8])
z = x - y

radius(x, y)
```

```python

```

_math/Imperial College London - Math for ML/matrix_rref_eigen_sympy_errorclass.ipynb


# Decision Trees

```python
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
```

The first steps are identical as in exercise 3. We will import the dataset and display some records and later we will define the explanatory variables and target variable.

```python
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data'
df = pd.read_csv(url, delimiter=',', names=[
    'Class',
    'Alcohol',
    'Malic acid',
    'Ash',
    'Alcalanity of ash',
    'Magnesium',
    'Total phenols',
    'Flavanoids',
    'Nonflavanoid phenols',
    'Proanthocyanins',
    'Color intensity',
    'Hue',
    'OD280/OD315 of diluted wines',
    'Proline'])

df.head()
```

      Class
      Alcohol
      Malic acid
      Ash
      Alcalanity of ash
      Magnesium
      Total phenols
      Flavanoids
      Nonflavanoid phenols
      Proanthocyanins
      Color intensity
      Hue
      OD280/OD315 of diluted wines
      Proline

      0
      1
      14.23
      1.71
      2.43
      15.6
      127
      2.80
      3.06
      0.28
      2.29
      5.64
      1.04
      3.92
      1065

      1
      1
      13.20
      1.78
      2.14
      11.2
      100
      2.65
      2.76
      0.26
      1.28
      4.38
      1.05
      3.40
      1050

      2
      1
      13.16
      2.36
      2.67
      18.6
      101
      2.80
      3.24
      0.30
      2.81
      5.68
      1.03
      3.17
      1185

      3
      1
      14.37
      1.95
      2.50
      16.8
      113
      3.85
      3.49
      0.24
      2.18
      7.80
      0.86
      3.45
      1480

      4
      1
      13.24
      2.59
      2.87
      21.0
      118
      2.80
      2.69
      0.39
      1.82
      4.32
      1.04
      2.93
      735

```python
X = df.iloc[:, 1:].values
y = df.iloc[:, 0].values
```

Now we will create basic Decision Tree classifier with default Gini index as criterion for quality of splits. It describes how homogenous a node is. A node is pure when all its samples belong to one class having Gini impurity $G$ equal to 0. On the other side a node with samples from many different classes will have $G$ closer to 1. Formally it's defined as:

\begin{equation}
G = 1 - \sum_{k = 1}^{n}\pi_{k},
\end{equation}

where $\pi_{k}$ is prior probability of a given class. To check the performance of the model we will calculate in-sample accuracy and cross validation accuracy.

```python
from sklearn.model_selection import cross_val_score
from sklearn import tree

clf = tree.DecisionTreeClassifier(random_state=0)
clf.fit(X, y)

print('In-sample accuracy: {}'.format(clf.score(X, y)))
print('Cross validation accuracy: {}'.format(cross_val_score(clf, X, y, cv=10).mean()))
```

    In-sample accuracy: 1.0
    Cross validation accuracy: 0.8705882352941178

As expected in-sample accuracy takes very high value, while cross validation accuracy has worse result. Later we will try to improve this model such that it will generalize better by pruning the tree. But now we will visualize a tree model with a simple function that uses embedded method `plot_tree()`.

```python
def draw_tree(clf, size):
    plt.figure(figsize=size)
    tree.plot_tree(clf)
    plt.show()
```

```python
draw_tree(clf, (15, 15))
```

For each node there is a splitting condition, value of Gini impurity, total number of samples and list of values that tells us how many of the samples fall into each of the classes. In the next step we will employ minimal cost-complexity pruning to choose optimal tree and avoid overfitting. By applying this algorithm to the large tree we obtain a sequence of impurities of the subtree leaves as a function of effective complexity parameter $\alpha_{eff}$. Next we create the subtrees for each value of $\alpha$, use cross validation for model evaluation and choose the $\alpha$ which the subtree has the maximum accuracy for. Finally we get the subtree corresponding to the chosen $\alpha$ value.

```python
def get_optimal_tree(clf, X_train, y_train):
    path = clf.cost_complexity_pruning_path(X_train, y_train)
    ccp_alphas = path.ccp_alphas

    cv_scores = []
    for ccp_alpha in ccp_alphas:
        clf = tree.DecisionTreeClassifier(random_state=0, ccp_alpha=ccp_alpha)
        score = cross_val_score(clf, X_train, y_train, cv=10).mean()
        cv_scores.append(score)

    optimal_ccp = ccp_alphas[np.argmax(cv_scores)]
    optimal_clf = tree.DecisionTreeClassifier(random_state=0, ccp_alpha=optimal_ccp)
    accuracy = cv_scores[np.argmax(cv_scores)]

    return optimal_clf, accuracy
```

We can use the function that we defined just now to generate the optimal subtree, check its accuracy and visualize it.

```python
optimal_clf, accuracy = get_optimal_tree(clf, X, y)
optimal_clf.fit(X, y)

print('Accuracy: {}'.format(accuracy))
draw_tree(optimal_clf, (10, 10))
```

The accuracy of the optimal model has been slightly increased in comparison with the large tree. Moreover the model generalizes better since it's been pruned. Now we will improve our model even more by selecting the optimal number of features. To do so, a new tree model will be created which will be fit on the different number of features each time and the optimal tree with its accuracy will be obtained and stored in a list. Also a difference between the depths of the large tree and optimal tree will be tracked.

```python
accuracy_list, differences = [], []
for i in range(2, 15):
    X = df.iloc[:, 1:i]
    y = df.iloc[:, 0]

    clf = tree.DecisionTreeClassifier(random_state=0)
    clf.fit(X, y)

    optimal_clf, accuracy = get_optimal_tree(clf, X, y)
    optimal_clf.fit(X, y)

    differences.append(clf.get_depth() - optimal_clf.get_depth())
    accuracy_list.append(accuracy)
```

In the following section we will plot the results in a function of the number of features.

```python
fig = plt.figure(figsize=(15, 5))

ax1 = fig.add_subplot(121)
ax1.set_xlabel('number of features')
ax1.set_ylabel('accuracy')
plt.plot(range(1, 14), accuracy_list, color='tab:red', marker='o', markeredgecolor='black')

ax2 = fig.add_subplot(122)
ax2.set_xlabel('number of features')
ax2.set_ylabel('depths difference')
plt.plot(range(1, 14), differences, color='tab:blue', marker='o', markeredgecolor='black')

plt.show()
```

We can conclude that the optimal number of features is 10 – the model reaches almost 95% of accuracy. We can also see that for small number of features the model is underfitted, its accuracy is low and the depths difference is huge. On the other hand for greater number of features than 10 model is getting overfitted – accuracy decreases and depths difference increases.


```python
from sympy import *
from IPython.display import display
init_printing()
```

# Three-body polynomial Jastrow

Three body Jastrow factor from "Jastrow correlation factor for atoms, molecules, and solids" N.D.Drummond, M.D.Towler, R.J.Needs, PRB 70 235119(2004)

See the 'gen_three_body.py' script for code generation

```python
ri = Symbol('r_i')
rj = Symbol('r_j')
rij = Symbol('r_ij')
(ri, rj, rij)
```

```python
C = Symbol('C')  # C is 2 or 3
L = Symbol('L')
gamma = IndexedBase('gamma')
r = IndexedBase('r')
l = Symbol('l',integer=True)
m = Symbol('m',integer=True)
n = Symbol('n',integer=True)
N = Symbol('N',integer=True)
N_ee = Symbol("N_ee",integer=True)
N_en = Symbol("N_en",integer=True)
```

```python
# General form of the 3-body Jastrow
f = (ri - L)**C * (rj -L)**C * Sum(Sum(Sum(gamma[l,m,n]*ri**l *rj**m*rij**n,(l,0,N_en)),(n,0,N_en)),(m,0,N_ee))
f
```

$\displaystyle \left(- L + r_{i}\right)^{C} \left(- L + r_{j}\right)^{C} \sum_{\substack{0 \leq l \leq N_{en}\\0 \leq n \leq N_{en}\\0 \leq m \leq N_{ee}}} r_{i}^{l} r_{ij}^{n} r_{j}^{m} {\gamma}_{l,m,n}$

```python
# Concrete example for N_en = 1 and N_ee = 1
f1 = f.subs(N_en,1).subs(N_ee,1).doit()
f1
```

```python
# Concrete example for N_en = 1 and N_ee = 2
f12 = f.subs(N_en,1).subs(N_ee,2).doit()
f12
```

## Constraints

$l$ is index for electron_1 - nuclei distance variable

$m$ is index for electron_2 - nuclei distance variable

$n$ is index for electron_1 - electron_2 distance variable

The Jastrow factor should be symmetric under electron exchange (swap $l$ and $m$)

```python
# How do we use this to simplify the expressions above?
Eq(gamma[l,m,n], gamma[m,l,n])

# Brute force - loop over m and l<m and create the substitutions (next cell)
```

```python
# Concrete values of N_ee and N_en for all the following
NN_ee = 2
NN_en = 2
ftmp = f1
sym_subs = {}
display(ftmp)
for i1 in range(NN_en+1):
    for i2 in range(i1):
        for i3 in range(NN_ee+1):
            print(i1,i2,i3)
            display (gamma[i2,i1,i3], gamma[i1,i2,i3])
            #ftmp = ftmp.subs(gamma[i2,i1,i3], gamma[i1,i2,i3])
            sym_subs[gamma[i2,i1,i3]] = gamma[i1,i2,i3]
ftmp = f.subs(N_en,NN_en).subs(N_ee,NN_ee).doit().subs(sym_subs)
sym_subs
```

```python
# Three body Jastrow with symmetry constraints
ftmp_sym = simplify(expand(ftmp))
ftmp_sym
```

```python
# Find the free gamma values
{a for a in ftmp_sym.free_symbols if type(a) is Indexed}
```

### No electron-electron cusp

```python
# First derivative of electron-electron distance should be zero at r_ij = 0
ftmp_ee = diff(ftmp, rij).subs(rij,0).subs(rj,ri)
ftmp_ee
```

```python
# Remove the (r_i-L)**C part
ftmp2_ee = simplify(expand(ftmp_ee)).args[1]
ftmp2_ee
```

```python
# Collect powers of r_i
ft3 = collect(ftmp2_ee,ri)
ft3
```

```python
# Convert to polynomial to extract coefficients of r_i
pt3 = poly(ft3,ri)
pt3
```

```python
pt4 = pt3.all_coeffs()
pt4
```

```python
# To enforce results are zero for all distance, the coefficients must be zero
ee_soln = solve(pt4)
ee_soln
```

### No electron-nuclei cusp

```python
# First derivative of electron-nuclei distance should be zero at r_i = 0
ftmp_en = diff(ftmp, ri).subs(ri,0).subs(rij,rj)
ftmp_en
```

```python
simplify(expand(ftmp_en))
```

```python
# Remove the (-L)**(C-1) * (r_j -L)**C part
simplify(expand(ftmp_en)).args[2]
```

```python
# Put in constraints from e-e cusp
ftmp_en2 = ftmp_en.subs(ee_soln)
ftmp_en2
```

```python
simplify(ftmp_en2)
```

```python
ftmp_en3 = simplify(expand(ftmp_en2))
ftmp_en3
```

```python
# Remove the (-L)**(C-1) * (r_j -L)**C part
ftmp_en3 = ftmp_en3.subs(sym_subs).args[2]
ftmp_en3
```

```python

```

```python
# Powers of r_j
collect(expand(ftmp_en3),rj)
```

```python
# Convert to polynomial to extract coefficients
pe3 = poly(ftmp_en3,rj)
pe3
```

```python
pe4 = pe3.all_coeffs()
print(len(pe4))
pe4
```

```python
# Solve can be very slow as the expansion size increases
#en_soln = solve(pe4)
# Using linsolve is faster
soln_var = {a for a in ftmp_en3.free_symbols if type(a) is Indexed}
en_soln = linsolve(pe4, soln_var)

en_soln
```

```python
# Don't want the C=0,L=0 solution when using solve
#en_soln2 = en_soln[1]
#en_soln2
# If using linsolve
for tmp_soln in en_soln:
    en_soln2 = {g:v for g,v in zip(soln_var, tmp_soln)}
en_soln2
```

```python
# Final expression with all the constraints inserted
ftmp_out = ftmp.subs(sym_subs).subs(ee_soln).subs(en_soln2)
ftmp_out2 = simplify(expand(ftmp_out))
ftmp_out2
```

```python
# Find the free gamma values
{a for a in ftmp_out2.free_symbols if type(a) is Indexed}
```

```python

```

## Formula from Appendix of the paper

```python
NN_en = 2
NN_ee = 2
Nc_en = 2*NN_en + 1
Nc_ee = NN_en + NN_ee + 1
N_gamma = (NN_en + 1)*(NN_en+2)//2 * (NN_ee + 1)
print('Number of gamma values (after symmetry) : ',N_gamma)
print('Number of e-e constraints : ',Nc_ee)
print('Number of e-n constraints : ',Nc_en)
print('Number of free param = ',N_gamma - Nc_ee - Nc_en)

# Note, for N_en=1 and N_en=1, this formula doesn't match the above derivation.
# For all higher values it does match.
```

    Number of gamma values (after symmetry) :  18
    Number of e-e constraints :  5
    Number of e-n constraints :  5
    Number of free param =  8

```python
# For the electron-electron cusp
for k in range(0,2*NN_en+1):
    terms = 0
    print(k)
    for l in range(0,NN_en+1):
        for m in range(0,l):
            if l+m == k:
                #print(' ',l,m,'2*gamma[%d,%d,1]'%(l,m))
                terms += 2*gamma[l,m,1]
    # sum over l,m such that l+m == k and l>m
    # plus
    # sum over l such that 2l == k
    for l in range(0,NN_en):
        if 2*l == k:
            #print(' ',l,'gamma[%d,%d,1]'%(l,l))
            terms += 2*gamma[l,l,1]
    print('k: ',k,' terms: ',terms)
```

    0
    k:  0  terms:  2*gamma[0, 0, 1]
    1
    k:  1  terms:  2*gamma[1, 0, 1]
    2
    k:  2  terms:  2*gamma[1, 1, 1] + 2*gamma[2, 0, 1]
    3
    k:  3  terms:  2*gamma[2, 1, 1]
    4
    k:  4  terms:  0

```python
# For the electron-nuclear cusp
for kp in range(0,NN_en + NN_ee+1):
    terms = 0
    if kp <= NN_ee:
        terms = C*gamma[0,0,kp] - L*gamma[1,0,kp]
    # sum of l,n such that l + n == kp and l>=1
    for l in range(1,NN_en+1):
        for n in range(NN_ee+1):
            if l + n == kp:
                terms += C*gamma[l,0,n] - L*gamma[l,1,n]
    print('kp: ',kp,' terms: ',terms)

```

    kp:  0  terms:  C*gamma[0, 0, 0] - L*gamma[1, 0, 0]
    kp:  1  terms:  C*gamma[0, 0, 1] + C*gamma[1, 0, 0] - L*gamma[1, 0, 1] - L*gamma[1, 1, 0]
    kp:  2  terms:  C*gamma[0, 0, 2] + C*gamma[1, 0, 1] + C*gamma[2, 0, 0] - L*gamma[1, 0, 2] - L*gamma[1, 1, 1] - L*gamma[2, 1, 0]
    kp:  3  terms:  C*gamma[1, 0, 2] + C*gamma[2, 0, 1] - L*gamma[1, 1, 2] - L*gamma[2, 1, 1]
    kp:  4  terms:  C*gamma[2, 0, 2] - L*gamma[2, 1, 2]

```python

```


# Solución de ecuaciones diferenciales

Dada la siguiente ecuación diferencial:

$$
\dot{x} = -x
$$

queremos obtener la respuesta del sistema que representa, es decir, los valores que va tomando $x$.

Si analizamos esta ecuación diferencial, podremos notar que la solución de este sistema es una función $\varphi(t)$, tal que cuando la derivemos obtengamos el negativo de esta misma función, es decir:

$$
\frac{d}{dt} \varphi(t) = -\varphi(t)
$$

y despues de un poco de pensar, podemos darnos cuenta de que la función que queremos es:

$$
\varphi(t) = e^{-t}
$$

Sin embargo muchas veces no tendremos funciones tan sencillas (ciertamente no es el caso en la robótica, donde usualmente tenemos ecuaciones diferenciales no lineales de orden $n$), por lo que en esta práctica veremos algunas estrategias para obtener soluciones a esta ecuación diferencial, tanto numéricas como simbolicas.

## Método de Euler

El [método de Euler](http://es.wikipedia.org/wiki/Método_de_Euler) para obtener el comportamiento de una ecuación diferencial, se basa en la intuición básica de la derivada; digamos que tenemos una ecuación diferencial general:

$$
\frac{dy}{dx} = y' = F(x, y)
$$

en donde $F(x, y)$ puede ser cualquier función que depende de $x$ y/o de $y$, entonces podemos dividir en pedazos el comportamiento de la gráfica de tal manera que solo calculemos un pequeño pedazo cada vez, aproximando el comportamiento de la ecuación diferencial, con el de una recta, cuya pendiente será la derivada:

«Método de Euler» por Vero.delgado - Trabajo propio. Disponible bajo la licencia CC BY-SA 3.0 vía Wikimedia Commons.

Esta recta que aproxima a la ecuación diferencial, podemos recordar que tiene una estructura:

$$
y = b + mx
$$

por lo que si sustituimos en $m$ la derivada y $b$ con el valor anterior de la ecuación diferencial, obtendremos algo como:

$$
\overbrace{y_{i+1}}^{\text{nuevo valor de }y} = \overbrace{y_i}^{\text{viejo valor de }y} + \overbrace{\frac{dy}{dx}}^{\text{pendiente}} \overbrace{\Delta x}^{\text{distancia en }x}
$$

pero conocemos el valor de $\frac{dy}{dx}$, es nuestra ecuación diferencial; por lo que podemos escribir esto como:

$$
y_{i+1} = y_i + F(x_i, y_i) \Delta x
$$

Resolvamos algunas iteraciones de nuestro sistema; empecemos haciendo 10 iteraciones a lo largo de 10 segundos, con condiciones iniciales $x(0) = 1$, eso quiere decir que:

$$
\begin{align}
\Delta t &= 1 \\
x(0) &= 1 \\
\dot{x}(0) &= 1
\end{align}
$$

```python
x0 = 1
Δt = 1

# Para escribir simbolos griegos como Δ, tan solo tienes que escribir su nombre
# precedido de una diagonal (\Delta) y teclear tabulador una vez

F = lambda x : -x
x1 = x0 + F(x0)*Δt
x1
```

```python
x2 = x1 + F(x1)*Δt
x2
```

### Ejercicio

Crea codigo para una iteración mas con estos mismos parametros y despliega el resultado.

```python
x3 = # Escribe el codigo de tus calculos aqui
```

```python
from pruebas_2 import prueba_2_1
prueba_2_1(x0, x1, x2, x3, _)
```

Momento... que esta pasando? Resulta que este $\Delta t$ es demasiado grande, intentemos con 20 iteraciones:

$$
\begin{align}
\Delta t &= 0.5 \\
x(0) &= 1
\end{align}
$$

```python
x0 = 1
n = 20
Δt = 10/n

F = lambda x : -x
x1 = x0 + F(x0)*Δt
x1
```

```python
x2 = x1 + F(x1)*Δt
x2
```

```python
x3 = x2 + F(x2)*Δt
x3
```

Esto va a ser tardado, mejor digamosle a Python que es lo que tenemos que hacer, y que no nos moleste hasta que acabe, podemos usar un ciclo ```for``` y una lista para guardar todos los valores de la trayectoria:

```python
xs = [x0]
for t in range(20):
    xs.append(xs[-1] + F(xs[-1])*Δt)

xs
```

Ahora que tenemos estos valores, podemos graficar el comportamiento de este sistema, primero importamos la libreria ```matplotlib```:

```python
%matplotlib inline
from matplotlib.pyplot import plot
```

Mandamos a llamar la función ```plot```:

```python
plot(xs);
```

Sin embargo debido a que el periodo de integración que utilizamos es demasiado grande, la solución es bastante inexacta, podemos verlo al graficar contra la que sabemos es la solución de nuestro problema:

```python
from numpy import linspace, exp
```

```python
ts = linspace(0, 10, 20)
plot(xs)
plot(exp(-ts));
```

Si ahora utilizamos un numero de pedazos muy grande, podemos mejorar nuestra aproximación:

```python
xs = [x0]
n = 100
Δt = 10/n

for t in range(100):
    xs.append(xs[-1] + F(xs[-1])*Δt)

ts = linspace(0, 10, 100)
```

```python
plot(xs)
plot(exp(-ts));
```

## odeint

Este método funciona tan bien, que ya viene programado dentro de la libreria ```scipy```, por lo que solo tenemos que importar esta librería para utilizar este método.

Sin embargo debemos de tener cuidado al declarar la función $F(x, t)$. El primer argumento de la función se debe de referir al estado de la función, es decir $x$, y el segundo debe de ser la variable independiente, en nuestro caso el tiempo.

```python
from scipy.integrate import odeint
```

```python
F = lambda x, t : -x
```

```python
x0 = 1
ts = linspace(0, 10, 100)
xs = odeint(func=F, y0=x0, t=ts)
```

```python
plot(ts, xs);
```

### Ejercicio

Grafica el comportamiento de la siguiente ecuación diferencial.

$$
\dot{x}  = x^2 - 5 x + \frac{1}{2} \sin{x} - 2
$$

> Nota: Asegurate de impotar todas las librerias que puedas necesitar

```python
ts = # Escribe aqui el codigo que genera un arreglo de puntos equidistantes (linspace)
x0 = # Escribe el valor de la condicion inicial

# Importa las funciones de librerias que necesites aqui

G = lambda x, t: # Escribe aqui el codigo que describe los calculos que debe hacer la funcion

xs = # Escribe aqui el comando necesario para simular la ecuación diferencial

plot(ts, xs);
```

```python
from pruebas_2 import prueba_2_2
prueba_2_2(ts, xs)
```

## Sympy

Y por ultimo, hay veces en las que incluso podemos obtener una solución analítica de una ecuación diferencial, siempre y cuando cumpla ciertas condiciones de simplicidad.

```python
from sympy import var, Function, dsolve
from sympy.physics.mechanics import mlatex, mechanics_printing
mechanics_printing()
```

```python
var("t")
```

```python
x = Function("x")(t)
x, x.diff(t)
```

```python
solucion = dsolve(x.diff(t) + x, x)
solucion
```

### Ejercicio

Implementa el codigo necesario para obtener la solución analítica de la siguiente ecuación diferencial:

$$
\dot{x} = x^2 - 5x
$$

```python
# Declara la variable independiente de la ecuación diferencial
var("")

# Declara la variable dependiente de la ecuación diferencial
 = Function("")()

# Escribe la ecuación diferencial con el formato necesario (Ecuacion = 0)
# adentro de la función dsolve
sol = dsolve()
sol
```

```python
from pruebas_2 import prueba_2_3
prueba_2_3(sol)
```

## Solución a ecuaciones diferenciales de orden superior

Si ahora queremos obtener el comportamiento de una ecuacion diferencial de orden superior, como:

$$
\ddot{x} = -\dot{x} - x + 1
$$

Tenemos que convertirla en una ecuación diferencial de primer orden para poder resolverla numericamente, por lo que necesitaremos convertirla en una ecuación diferencial matricial, por lo que empezamos escribiendola junto con la identidad $\dot{x} = \dot{x}$ en un sistema de ecuaciones:

$$
\begin{align}
\dot{x} &= \dot{x} \\
\ddot{x} &= -\dot{x} - x + 1
\end{align}
$$

Si extraemos el operador derivada del lado izquierda, tenemos:

$$
$$
\begin{align}
\frac{d}{dt} x &= \dot{x} \\
\frac{d}{dt} \dot{x} &= -\dot{x} - x + 1
\end{align}
$$
$$

O bien, de manera matricial:

$$
\frac{d}{dt}
\begin{pmatrix}
x \\
\dot{x}
\end{pmatrix} =
\begin{pmatrix}
0 & 1 \\
-1 & -1
\end{pmatrix}
\begin{pmatrix}
x \\
\dot{x}
\end{pmatrix} +
\begin{pmatrix}
0 \\
1
\end{pmatrix}
$$

Esta ecuación ya _no_ es de segundo orden, es de hecho, de primer orden, sin embargo nuestra variable ha crecido a ser un vector de estados, por el momento le llamaremos $X$, asi pues, lo podemos escribir como:

$$
\frac{d}{dt} X = A X + B
$$

en donde:

$$
A = \begin{pmatrix}
0 & 1 \\
-1 & -1
\end{pmatrix} \quad \text{y} \quad B =
\begin{pmatrix}
0 \\
1
\end{pmatrix}
$$

y de manera similar, declarar una función para dar a ```odeint```.

```python
from numpy import matrix, array
```

```python
def F(X, t):
    A = matrix([[0, 1], [-1, -1]])
    B = matrix([[0], [1]])
    return array((A*matrix(X).T + B).T).tolist()[0]
```

```python
ts = linspace(0, 10, 100)
xs = odeint(func=F, y0=[0, 0], t=ts)
```

```python
plot(xs);
```

### Ejercicio

Implementa la solución de la siguiente ecuación diferencial, por medio de un modelo en representación de espacio de estados:

$$
\ddot{x} = -8\dot{x} - 15x + 1
$$

> Nota: Tomalo con calma y paso a paso
> * Empieza anotando la ecuación diferencial en tu cuaderno, junto a la misma identidad del ejemplo
> * Extrae la derivada del lado izquierdo, para que obtengas el _estado_ de tu sistema
> * Extrae las matrices A y B que corresponden a este sistema
> * Escribe el codigo necesario para representar estas matrices

```python
def G(X, t):
    A = # Escribe aqui el codigo para la matriz A
    B = # Escribe aqui el codigo para el vector B
    return array((A*matrix(X).T + B).T).tolist()[0]

ts = linspace(0, 10, 100)
xs = odeint(func=G, y0=[0, 0], t=ts)
```

```python
plot(xs);
```

```python
from pruebas_2 import prueba_2_4
prueba_2_4(xs)
```

## Funciones de transferencia

Sin embargo, no es la manera mas facil de obtener la solución, tambien podemos aplicar una transformada de Laplace, y aplicar las funciones de la libreria de control para simular la función de transferencia de esta ecuación; al aplicar la transformada de Laplace, obtendremos:

$$
G(s) = \frac{1}{s^2 + s + 1}
$$

```python
from control import tf, step
```

```python
F = tf([0, 0, 1], [1, 1, 1])
```

```python
xs, ts = step(F)
```

```python
plot(ts, xs);
```

### Ejercicio

Modela matematicamente la ecuación diferencial del ejercicio anterior, usando una representación de función de transferencia.

> Nota: De nuevo, no desesperes, escribe tu ecuación diferencial y aplica la transformada de Laplaca tal como te enseñaron tus abuelos hace tantos años...

```python
G = tf([], []) # Escribe los coeficientes de la función de transferencia

xs, ts = step(G)
plot(ts, xs);
```

```python
from pruebas_2 import prueba_2_5
prueba_2_5(ts, xs)
```

## Problemas

1. Modela matematicamente la suspensión de un automovil de masa $m = 1200 kg$, considera que los resortes de su suspensión tienen una constante $k = 15,000 \frac{N}{m}$ y un amortiguador con constante $c = 1,500 \frac{N s}{m}$.

2. Gráfica el comportamiento del sistema ante una fuerza $F = 1 N$.
3. Que tipo de comportamiento presenta este sistema? Estable, criticamente estable, inestable?

Practicas/.ipynb_checkpoints/Practica 2 - Solucion de ecuaciones diferenciales-checkpoint.ipynb


# Comparing Data
You'll often want to compare data in your dataset, to see if you can discern trends or relationships.

## Univariate Data
*Univariate* data is data that consist of only one variable or feature. While it may initially seem as though there's not much we can do to analyze univariate data, we've already seen that we can explore its distribution in terms of measures of central tendency and measures of variance. We've also seen how we can visualize this distribution using histograms and box plots.

Here's a reminder of how you can visualize the distribution of univariate data, using our student grade data with a few additional observations in the sample:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

plt.figure()
df['Grade'].plot( kind='box', title='Grade Distribution')
plt.figure()
df['Grade'].hist(bins=9)
plt.show()
print(df.describe())
print('median: ' + str(df['Grade'].median()))
```

## Bivariate and Multivariate Data
It can often be useful to compare *bivariate* data; in other words, compare two variables, or even more (in which case we call it *multivariate* data).

For example, our student data includes three numeric variables for each student: their salary, the number of hours they work per week, and their final school grade. Run the following code to see an enlarged sample of this data as a table:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df[['Name', 'Salary', 'Hours', 'Grade']]
```

      Name
      Salary
      Hours
      Grade

      0
      Dan
      50000
      41
      50

      1
      Joann
      54000
      40
      50

      2
      Pedro
      50000
      36
      46

      3
      Rosie
      189000
      17
      95

      4
      Ethan
      55000
      35
      50

      5
      Vicky
      40000
      39
      5

      6
      Frederic
      59000
      40
      57

      7
      Jimmie
      42000
      45
      42

      8
      Rhonda
      47000
      41
      26

      9
      Giovanni
      78000
      35
      72

      10
      Francesca
      119000
      30
      78

      11
      Rajab
      95000
      33
      60

      12
      Naiyana
      49000
      38
      40

      13
      Kian
      29000
      47
      17

      14
      Jenny
      130000
      24
      85

Let's suppose you want to compare the distributions of these variables. You might simply create a boxplot for each variable, like this:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df.plot(kind='box', title='Distribution', figsize = (10,8))
plt.show()
```

Hmm, that's not particularly useful is it?

The problem is that the data are all measured in different scales. Salaries are typically in tens of thousands, while hours and grades are in single or double digits.

### Normalizing Data
When you need to compare data in different units of measurement, you can *normalize* or *scale* the data so that the values are measured in the same proportional scale. For example, in Python you can use a MinMax scaler to normalize multiple numeric variables to a proportional value between 0 and 1 based on their minimum and maximum values. Run the following cell to do this:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.preprocessing import MinMaxScaler

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Normalize the data
scaler = MinMaxScaler()
df[['Salary', 'Hours', 'Grade']] = scaler.fit_transform(df[['Salary', 'Hours', 'Grade']])

# Plot the normalized data
df.plot(kind='box', title='Distribution', figsize = (10,8))
plt.show()
```

Now the numbers on the y axis aren't particularly meaningful, but they're on a similar scale.

### Comparing Bivariate Data with a Scatter Plot
When you need to compare two numeric values, a scatter plot can be a great way to see if there is any apparent relationship between them so that changes in the value of one variable affect the value of the other.

Let's look at a scatter plot of *Salary* and *Grade*:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Create a scatter plot of Salary vs Grade
df.plot(kind='scatter', title='Grade vs Hours', x='Grade', y='Salary')
plt.show()

```

Look closely at the scatter plot. Can you see a diagonal trend in the plotted points, rising up to the right? It looks as though the higher the student's grade is, the higher their salary is.

You can see the trend more clearly by adding a *line of best fit* (sometimes called a *trendline*) to the plot:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Create a scatter plot of Salary vs Grade
df.plot(kind='scatter', title='Grade vs Salary', x='Grade', y='Salary')

# Add a line of best fit
plt.plot(np.unique(df['Grade']), np.poly1d(np.polyfit(df['Grade'], df['Salary'], 1))(np.unique(df['Grade'])))

plt.show()
```

The line of best fit makes it clearer that there is some apparent *colinearity* between these variables (the relationship is *colinear* if one variable's value increases or decreases in line with the other).

### Correlation
The apparently colinear relationship you saw in the scatter plot can be verified by calculating a statistic that quantifies the relationship between the two variables. The statistic usually used to do this is *correlation*, though there is also a statistic named *covariance* that is sometimes used. Correlation is generally preferred because the value it produces is more easily interpreted.

A correlation value is always a number between ***-1*** and ***1***.
- A positive value indicates a positive correlation (as the value of variable *x* increases, so does the value of variable *y*).
- A negative value indicates a negative correlation (as the value of variable *x* increases, the value of variable *y* decreases).
- The closer to zero the correlation value is, the weaker the correlation between *x* and *y*.
- A correlation of exactly zero means there is no apparent relationship between the variables.

The formula to calculate correlation is:

\begin{equation}r_{x,y} = \frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})(y_{i} -\bar{y})}{\sqrt{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}(y_{i} -\bar{y})^{2}}}\end{equation}

**rx, y** is the notation for the *correlation between x and y*.

The formula is pretty complex, but fortunately Python makes it very easy to calculate the correlation by using the ***corr*** function:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Calculate the correlation between *Salary* and *Grade*
print(df['Grade'].corr(df['Salary']))
```

    0.8149286388911882

In this case, the correlation is just over 0.8; making it a reasonably high positive correlation that indicates salary increases in line with grade.

Let's see if we can find a correlation between *Grade* and *Hours*:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

r = df['Grade'].corr(df['Hours'])
print('Correlation: ' + str(r))

# Create a scatter plot of Salary vs Grade
df.plot(kind='scatter', title='Grade vs Hours', x='Grade', y='Hours')

# Add a line of best fit-
plt.plot(np.unique(df['Grade']), np.poly1d(np.polyfit(df['Grade'], df['Hours'], 1))(np.unique(df['Grade'])))
plt.show()

```

In this case, the correlation value is just under -0.8; meaning a fairly strong negative correlation in which the number of hours worked decreases as the grade increases. The line of best fit on the scatter plot corroborates this statistic.

It's important to remember that *correlation* **is not** *causation*. In other words, even though there's an apparent relationship, you can't say for sure that one variable is the cause of the other. In this example, we can say that students who achieved higher grades tend to work shorter hours; but we ***can't*** say that those who work shorter hours do so *because* they achieved a high grade!

## Least Squares Regression
In the previous examples, we drew a line on a scatter plot to show the *best fit* of the data. In many cases, your initial attempts to identify any colinearity might involve adding this kind of line by hand (or just mentally visualizing it); but as you may suspect from the use of the *numpy.**polyfit*** function in the code above, there are ways to calculate the coordinates for this line mathematically. One of the most commonly used techniques is *least squares regression*, and that's what we'll look at now.

Cast your mind back to when you were learning how to solve linear equations, and recall that the *slope-intercept* form of a linear equation lookes like this:

\begin{equation}y = mx + b\end{equation}

In this equation, *y* and *x* are the coordinate variables, *m* is the slope of the line, and *b* is the y-intercept of the line.

In the case of our scatter plot for our former-student's working hours, we already have our values for *x* (*Grade*) and *y* (*Hours*), so we just need to calculate the intercept and slope of the straight line that lies closest to those points. Then we can form a linear equation that calculates the a new *y* value on that line for each of our *x* (*Grade*) values - to avoid confusion, we'll call this new *y* value *f(x)* (because it's the output from a linear equation function based on *x*). The difference between the original *y* (*Hours*) value and the *f(x)* value is the *error* between our regression line of best fit and the actual *Hours* worked by the former student. Our goal is to calculate the slope and intercept for a line with the lowest overall error.

Specifically, we define the overall error by taking the error for each point, squaring it, and adding all the squared errors together. The line of best fit is the line that gives us the lowest value for the sum of the squared errors - hence the name *least squares regression*.

So how do we accomplish this? First we need to calculate the slope (*m*), which we do using this formula (in which *n* is the number of observations in our data sample):

\begin{equation}m = \frac{n(\sum{xy}) - (\sum{x})(\sum{y})}{n(\sum{x^{2}})-(\sum{x})^{2}}\end{equation}

After we've calculated the slope (*m*), we can use is to calculate the intercept (*b*) like this:

\begin{equation}b = \frac{\sum{y} - m(\sum{x})}{n}\end{equation}

Let's look at a simple example that compares the number of hours of nightly study each student undertook with the final grade the student achieved:

| Name     | Study | Grade |
|----------|-------|-------|
| Dan      | 1     | 50    |
| Joann    | 0.75  | 50    |
| Pedro    | 0.6   | 46    |
| Rosie    | 2     | 95    |
| Ethan    | 1     | 50    |
| Vicky    | 0.2   | 5     |
| Frederic | 1.2   | 57    |

First, let's take each *x* (Study) and *y* (Grade) pair and calculate *x2* and *xy*, because we're going to need these to work out the slope:

| Name     | Study | Grade | x2 | xy   |
|----------|-------|-------|------|------|
| Dan      | 1     | 50    | 1    | 50   |
| Joann    | 0.75  | 50    | 0.55 | 37.5 |
| Pedro    | 0.6   | 46    | 0.36 | 27.6 |
| Rosie    | 2     | 95    | 4    | 190  |
| Ethan    | 1     | 50    | 1    | 50   |
| Vicky    | 0.2   | 5     | 0.04 | 1    |
| Frederic | 1.2   | 57    | 1.44 | 68.4 |

Now we'll sum *x*, *y*, *x2*, and *xy*:

| Name     | Study | Grade | x2 | xy   |
|----------|-------|-------|------|------|
| Dan      | 1     | 50    | 1    | 50   |
| Joann    | 0.75  | 50    | 0.55 | 37.5 |
| Pedro    | 0.6   | 46    | 0.36 | 27.6 |
| Rosie    | 2     | 95    | 4    | 190  |
| Ethan    | 1     | 50    | 1    | 50   |
| Vicky    | 0.2   | 5     | 0.04 | 1    |
| Frederic | 1.2   | 57    | 1.44 | 68.4 |
| **Σ**      | **6.75**  | **353**   | **8.4025**| **424.5**  |

OK, now we're ready to calculate the slope for our *7* observations:

\begin{equation}m = \frac{(7\times 424.5) - (6.75\times353)}{(7\times8.4025)-6.75^{2}}\end{equation}

Which is:

\begin{equation}m = \frac{2971.5 - 2382.75}{58.8175-45.5625}\end{equation}

So:

\begin{equation}m = \frac{588.75}{13.255} \approx 44.4172\end{equation}

Now we can calculate *b*:

\begin{equation}b = \frac{353 - (44.4172\times6.75)}{7}\end{equation}

Which simplifies to:

\begin{equation}b = \frac{53.18389}{7} = 7.597699\end{equation}

Now we have our linear function:

\begin{equation}f(x) = 44.4172x + 7.597699\end{equation}

We can use this for each *x* (Study) value to calculate the *y* values for the regression line (*f(x)*), and we can subtract the original *y* (Grade) from these to calculate the error for each point:

| Name     | Study | Grade | *f(x)* | Error |
|----------|-------|-------|------|------ |
| Dan      | 1     | 50    |52.0149 |2.0149 |
| Joann    | 0.75  | 50    |40.9106 |-9.0894|
| Pedro    | 0.6   | 46    |34.2480 |-11.752|
| Rosie    | 2     | 95    |96.4321 |1.4321 |
| Ethan    | 1     | 50    |52.0149 |2.0149 |
| Vicky    | 0.2   | 5     |16.4811 |11.4811|
| Frederic | 1.2   | 57    |60.8983 |3.8983 |

As you can see, the *f(x)* values are mostly quite close to the actual *Grade* values, and the errors (which when we're comparing estimated values from a function with actual known values we we often call *residuals*) are generally pretty small.

Let's plot the least squares regression line with the actual values:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Study':[1,0.75,0.6,2,1,0.2,1.2],
                   'Grade':[50,50,46,95,50,5,57],
                   'fx':[52.0159,40.9106,34.2480,96.4321,52.0149,16.4811,60.8983]})

# Create a scatter plot of Study vs Grade
df.plot(kind='scatter', title='Study Time vs Grade Regression', x='Study', y='Grade', color='red')

# Plot the regression line
plt.plot(df['Study'],df['fx'])

plt.show()
```

In this case, the line fits the middle values fairly well, but is less accurate for the outlier at the low end. This is often the case, which is why statisticians and data scientists often *treat* outliers by removing them or applying a threshold value; though in this example there are too few data points to conclude that the data points are really outliers.

Let's look at a slightly larger dataset and apply the same approach to compare *Grade* and *Salary*:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Calculate least squares regression line
df['x2'] = df['Grade']**2
df['xy'] = df['Grade'] * df['Salary']
x = df['Grade'].sum()
y = df['Salary'].sum()
x2 = df['x2'].sum()
xy = df['xy'].sum()
n = df['Grade'].count()
m = ((n*xy) - (x*y))/((n*x2)-(x**2))
b = (y - (m*x))/n
df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

# Create a scatter plot of Grade vs Salary
df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

# Plot the regression line
plt.plot(df['Grade'],df['fx'])

plt.show()

# Show the original x,y values, the f(x) value, and the error
df[['Grade', 'Salary', 'fx', 'error']]
```

In this case, we used Python expressions to calculate the *slope* and *y-intercept* using the same approach and formula as before. In practice, Python provides great support for statistical operations like this; and you can use the ***linregress*** function in the *scipy.stats* package to retrieve the *slope* and *y-intercept* (as well as the *correlation*, *p-value*, and *standard error*) for a matched array of *x* and *y* values (we'll discuss *p-values* later!).

Here's the Python code to calculate the regression line variables using the ***linregress*** function:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Get the regression line slope and intercept
m, b, r, p, se = stats.linregress(df['Grade'], df['Salary'])

df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

# Create a scatter plot of Grade vs Salary
df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

# Plot the regression line
plt.plot(df['Grade'],df['fx'])

plt.show()

# Show the original x,y values, the f(x) value, and the error
df[['Grade', 'Salary', 'fx', 'error']]

```

Note that the *slope* and *y-intercept* values are the same as when we worked them out using the formula.

Similarly to the simple study hours example, the regression line doesn't fit the outliers very well. In this case, the extremes include a student who scored only 5, and a student who scored 95. Let's see what happens if we remove these students from our sample:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df = df[(df['Grade'] > 5) & (df['Grade'] < 95)]

# Get the regression line slope and intercept
m, b, r, p, se = stats.linregress(df['Grade'], df['Salary'])

df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

# Create a scatter plot of Grade vs Salary
df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

# Plot the regression line
plt.plot(df['Grade'],df['fx'])

plt.show()

# Show the original x,y values, the f(x) value, and the error
df[['Grade', 'Salary', 'fx', 'error']]

```

With the outliers removed, the line is a slightly better overall fit to the data.

One of the neat things about regression is that it gives us a formula and some constant values that we can use to estimate a *y* value for any *x* value. We just need to apply the linear function using the *slope* and *y-intercept* values we've calculated from our sample data. For example, suppose a student named Fabio graduates from our school with a final grade of **62**. We can use our linear function with the *slope* and *y-intercept* values we calculated with Python to estimate what salary he can expect to earn:

\begin{equation}f(x) = (1424.50\times62) - 7822.24 \approx 80,497 \end{equation}

```python

```

Statistics and Probability by Hiren/04-03-Comparing Data.ipynb


# 11 ODE Applications (Projectile with linear air resistance)
Let's apply our ODE solvers to some problems involving balls and projectiles. We will start with projectile motion with linear air resistances.

The `integrators.py` file from the lesson on [ODE integrators](https://py4phy.github.io/PHY432/modules/ODEs/integrators/) is used here (and named [`ode.py`](https://github.com/Py4Phy/PHY432-resources/blob/main/11_ODE_applications/ode.py)).

```python
import numpy as np
import ode
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.matplotlib.style.use('ggplot')
```

## Theory
Linear drag force

$$
\mathbf{F}_1 = -b_1 \mathbf{v}
$$

Equations of motion with force due to gravity $\mathbf{g} = -g \hat{\mathbf{e}}_y$

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= - g \hat{\mathbf{e}}_y -\frac{b_1}{m} \mathbf{v}
\end{align}

Bring into standard ODE form for

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$

as

$$
\mathbf{y} = \begin{pmatrix}
x\\
y\\
v_x\\
v_y
\end{pmatrix}, \quad
\mathbf{f} = \begin{pmatrix}
v_x\\
v_y\\
-\frac{b_1}{m} v_x\\
-g -\frac{b_1}{m} v_y
\end{pmatrix}
$$

(Based on Wang 2016, Ch 3.3.1)

## Python implementation with ODE solver

- Formulate the function `f()` for the standard ODE form (note: velocity dependence)
- Set up the integration loop:
  - only integrate until the particle hits ground, i.e. while $y ≥ 0$.
  - choose an appropriate ODE solver from `ode.py` such as RK4 (because there's no energy conservation so velocity Verlet is not as useful)

```python
def simulate(v0, h=0.01, b1=0.2, g=9.81, m=0.5):

    def f(t, y):
        # y = [x, y, vx, vy]
        return np.array([y[2], y[3], -b1/m * y[2], -g - b1/m * y[3]])

    vx, vy = v0
    t = 0
    positions = []
    y = np.array([0, 0, vx, vy], dtype=np.float64)

    while y[1] >= 0:
        positions.append([t, y[0], y[1]])  # record t, x and y
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)

def initial_v(v, theta):
    x = np.deg2rad(theta)
    return v * np.array([np.cos(x), np.sin(x)])
```

### Launch at fixed angle

```python
r = simulate(initial_v(200, 30), h=0.01, b1=1)
```

```python
plt.plot(r[:, 1], r[:, 2])
plt.xlabel(r"distance $x$ (m)")
plt.ylabel(r"height $y$ (m)");
```

### Distance depends on launch angle

Plot the trajectory for launch angles from 5º to 45º.

```python
for angle in (5, 7.5, 10, 20, 30, 45):
    r = simulate(initial_v(200, angle), h=0.01, b1=1)
    plt.plot(r[:, 1], r[:, 2], label=r"$\theta = {}^\circ$".format(angle))
plt.legend(loc="best")
plt.xlabel(r"distance $x$ (m)")
plt.ylabel(r"height $y$ (m)");
plt.savefig("launch_linear_air_resistance.svg")
```

```python

```