# Non Linear Regression Analysis

In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

### Importing required libraries

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = 2*(x) + 5.

```python
x = np.arange(-10.0, 10.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 5
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.figure(figsize=(10,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-10.0, 10.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 2*x + 5
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.figure(figsize=(10,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.figure(figsize=(10,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-10.0, 10.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)
plt.figure(figsize=(10,6))
plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)
plt.figure(figsize=(10,6))
plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-15.0, 15.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))
plt.figure(figsize=(10,6))
plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

```python
df = pd.read_csv("data/china_gdp.csv")
df.head(10)
```

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

```python
df.tail()
```

      Year
      Value

      50
      2010
      6.039659e+12

      51
      2011
      7.492432e+12

      52
      2012
      8.461623e+12

      53
      2013
      9.490603e+12

      54
      2014
      1.035483e+13

### Plotting the Dataset ###

```python
plt.figure(figsize=(10,6))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))
plt.figure(figsize=(10,6))
plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)
plt.figure(figsize=(10,6))
#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
plt.show()
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.451711, beta_2 = 0.997207

Now we plot our resulting regresssion model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.figure(figsize=(10,6))
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

# split data into train/test

```python
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]
```

# build the model using train set

```python
popt, pcov = curve_fit(sigmoid, train_x, train_y)
```

# predict using test set

```python
y_hat = sigmoid(test_x, *popt)
```

# evaluation

```python
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )
```

    Mean absolute error: 0.03
    Residual sum of squares (MSE): 0.00
    R2-score: 0.97

This notebook is based on [cognitiveclass](https://courses.cognitiveclass.ai/) Machine Learning with Python course

```python

```


# Lecture 13

## Standard Normal, Normal normalizing constant

## Proof of the Universality of $Unif(0,1)$

Let $F$ be a _continuous_, _strictly increasing_ CDF.

Then

\begin{align}
  &X = F^{-1}(U) \sim F & &\text{if } U \sim Unif(0,1) & &\text{(1)}\\
  \\
  &F(X) \sim Unif(0,1) & &\text{if } X \sim F & &\text{(2)}
\end{align}

$F$ is just a function, and hence we can treat it like an operator.

Also, like $X$, $F(X)$ is just another r.v.

### Interpreting the Universality of $Unif(0,1)$

_How to interpret this universality?_

\begin{align}
  &F(x) = P(X \le x) & &\text{does not imply that } \\
  \\
  \require{cancel} &\bcancel{F(X) = P(X \le X) = 1} \\
  \\
  \\
  &F(x) = 1 - e^{-x} & &\text{where } x \gt 0 \\
  &F(X) = 1 - e^{-X}
\end{align}

_First_ evaluate $F$ at $x$, and _then_ replace $x$ for $X$.

### Simulating random variable draws using the universality of $Unif(0,1)$

These 2 approaches to universality of the Uniform are quite useful, like when you want to simulate draws from another r.v.

Let $F(x) = 1 - e^{-x}$, where $x \gt 0$. This is the exponential random distribution $Expo(1)$.

Say we have $U \sim Unif(0,1)$.

_How can we leverage universality of $Unif(0,1)$ to simulate $X \sim F$?_

\begin{align}
  u &= 1 - e^{-x} & &\text{first we obtain }F^{-1}(x) \\
  e^{-x} &= 1 - u \\
  -x &= ln(1 - u)  \\
  x &= - ln(1-u) \\
  \\
  \Rightarrow F^{-1}(u) &= -ln(1-u) \\
  \Rightarrow F^{-1}(U) &= -ln(1-U) \sim F & &\text{by universality}
\end{align}

And so we could simulate 10 i.i.d. draws from $F(x) = 1 - e^{-x}$ by calculating $-ln(1-U)$ 10 times.

### Symmetry of  $Unif(0,1)$

Note that

\begin{align}
  1 - U \sim Unif(0,1) & &\text{symmetry of Uniform} \\
  \\
  a + b U & &\text{linear transformations are also } Unif
\end{align}

Non-linear transformations will, in general, __not__ be $Unif$.

----

## Independence of Random Variables

### General case

Say we have r.v. $X_1, X_2, \dots , X_n$.

> __Definition__ $X_1, X_2, \dots , X_n$ are independent if, for all $x_1, x_2, \dots, x_n$
>
> \\begin{align}
>   P(X_1 \le x_1, X_2 \le x_2, \dots , X_n \le x_n) &= P(X_1 \le x_1) P(X_2 \le x_2) \cdots P(X_n \le x_n)
> \\end{align}

Note that in this general case, $P(X_1 \le x_1, X_2 \le x_2, \dots , X_n \le x_n)$ is called the __joint CDF__.

### Discrete case

> __Definition__ Discrete r.v. $X_1, X_2, \dots , X_n$ are independent if, for all $x_1, x_2, \dots, x_n$
>
> \\begin{align}
>   P(X_1 = x_1, X_2 = x_2, \dots , X_n = x_n) &= P(X_1 = x_1) P(X_2 = x_2) \cdots P(X_n = x_n)
> \\end{align}

In the discrete case, $P(X_1 = x_1, X_2 = x_2, \dots , X_n = x_n)$ is called the __joint PMF__.

### Interpretation

Simply put, in both the general and discrete cases, knowing any subset of the r.v. gives us __no__ information about the rest.

This is _stronger_ than pair-wise independence.

#### Ex. A penny-matching game

Consider a penny-matching game where 2 players flip a penny. If the faces showing match (HH or TT), then one of the players wins; else the other player wins.

Let

\begin{align}
  X_1, X_2 &\sim \mathbb{Bern}\left(\frac{1}{2}\right) ~~ \text{, i.i.d} \\
  \\
  X_3 &=
    \begin{cases}
    1  & \quad \text{if } X_1 = X_2 \\
    0  & \quad \text{otherwise}\\
    \end{cases}
\end{align}

$X_1, X_2, X_3$ are _pair-wise independent_, but not independent.

- Knowing $X_1, X_2$ gives us complete knowledge of $X_3$.
- However, just knowing $X_1$ tells us nothing about $X_2$, nor can it say anything about $X_3$.
____

## Normal Distribution (Gaussian)

### Description

The most important distribution in all statistics, mostly due to the Central Limit Theorem.

The Central Limit Theorem is surprising; if you sum up a large number of i.i.d. random variables will always look like a bell-shaped curve. This is irrespective of continuous or discrete; beautiful or ugly.

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.xkcd()

x = np.linspace(-5, 5, 500)
y = np.e ** ((-x**2)/2)

_, ax = plt.subplots()

ax.plot(x, y)
ax.grid(True, which='both')
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')
plt.show()
```

### Notation

$X \sim \mathcal{N}(\mu, Var)$

### Parameters

- $\mu$ mean
- $Var$ variance

### Probability density function

For the Standard Normal $\mathcal{N}(0,1)$

\begin{align}
  f(z) &= c e^{-\frac{z^{2}}{2}}
\end{align}

where $c$ is the normalizing constant that will let $f(x)$ integrate to 1.

_But what is $c$?_

If we can figure out what the integral of the PDF is, then the _inverse_ of that will be the constant $c$ that ensures $f(z) = c e^{-\frac{z^{2}}{2}} = 1$.

Consider the indefinite integral of the PDF of $\mathcal{N}$.

\begin{align}
  \int_{-\infty}^{\infty} e^{-\frac{z^2}{2}} ~~ dz
\end{align}

There is a theorem that states that this integral, as an indefinite integral, cannot be solved in closed form. But this can be solved...

First, we start by _multiplying the integral by itself_...

\begin{align}
  \int_{-\infty}^{\infty} e^{-\frac{z^2}{2}} ~~ dz \int_{-\infty}^{\infty} e^{-\frac{z^2}{2}} ~~ dz &= \int_{-\infty}^{\infty} e^{-\frac{x^2}{2}} ~~ dx \int_{-\infty}^{\infty} e^{-\frac{y^2}{2}} ~~ dy \\
  &= \iint_{-\infty}^{\infty} e^{-\frac{(x+y)^2}{2}} ~~ dxdy \\
  &= \int_{0}^{2\pi} \int_{0}^{\infty} e^{-\frac{r^2}{2}}~ \underbrace{r}_{\text{jacobian}} ~ drd\theta & \quad \text{switch to polar} \\
  &= \int_{0}^{2\pi} \left( \int_{0}^{\infty} e^{-u} du  \right) ~ d\theta & \quad \text{let } u = \frac{r^2}{2} \text{ , } du = r \\
  &= \int_{0}^{2\pi} \left( -e^{-\infty} + e^{-0}  \right) ~ d\theta \\
  &= \int_{0}^{2\pi} 1 ~ d\theta \\
  &= 2 \pi - 0 \\
  &= 2\pi \\
  \\
  \int_{-\infty}^{\infty} e^{-\frac{z^2}{2}} ~~ dz &= \sqrt{2\pi} \\
  \\
  \therefore c &= \boxed{\frac{1}{\sqrt{2\pi}} }
\end{align}

### Expected value

For the standard normal distribution, $\mathbb{E}(Z) = 0$. Let's prove that...

\begin{align}
  \mathbb{E}(Z) &= \int_{-\infty}^{\infty} z ~ \frac{e^{-\frac{z^2}{2}}}{\sqrt{2\pi}} ~ dz \\
  &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} z ~ e^{-\frac{z^2}{2}} ~ dz \\
  &= \frac{1}{\sqrt{2\pi}} (0) & \quad \text{since this is an odd function} \\
  &= 0 & \quad \blacksquare
\end{align}

```python
x = np.linspace(-5, 5, 500)
y = x * (np.e ** ((-x**2)/2))

_, ax = plt.subplots()
ax.plot(x, y, color='black')
ax.fill_between(x, y, where=y >= 0, facecolor='blue', interpolate=True)
ax.fill_between(x, y, where=y < 0, facecolor='red', interpolate=True)
ax.axhline(y=0, color='black')
ax.axvline(x=0, color='black')
plt.title("$g(x)$ is an odd function")
plt.show()
```

Recall that if $g(x)$ is an _odd function_, then $g(x) + g(-x) = 0$.

Integrating an odd function $g(x)$ from $-a$ to $a$ is 0, since the portion _above the x-axis_ cancels out the portion _below the x-axis_.

### Variance

For the standard normal distribution, $\mathbb{Var}(Z) = 1$. Let's prove this as well...

\begin{align}
  \mathbb{Var}(Z) &= \mathbb{E}Z^2 - \mathbb{E}(Z)^2 \\
  &= \int_{-\infty}^{\infty} z^2 ~ \frac{e^{-\frac{z^2}{2}}}{\sqrt{2\pi}} ~ dz - (0)^2 & \quad \text{by LOTUS; and by calculations above} \\=
  &= 2 \int_{0}^{\infty} z^2 ~ \frac{e^{-\frac{z^2}{2}}}{\sqrt{2\pi}} ~ dz & \quad \text{this is an even function} \\
  &= \frac{2}{\sqrt{2\pi}} \int_{0}^{\infty} z^2 ~ e^{-\frac{z^2}{2}} ~ dz & \quad \text{set us up for integration by parts} \\
  &= \frac{2}{\sqrt{2\pi}} \int_{0}^{\infty} z ~ z ~ e^{-\frac{z^2}{2}} ~ dz  \\
  &= \frac{2}{\sqrt{2\pi}} \int_{0}^{\infty} u ~ du ~ dv & \quad \text{let } u = z \text{, } \quad dv = z ~ e^{-\frac{z^2}{2}} \\
  & & \quad du = dz \text{, } \quad v = -e^{-\frac{z^2}{2}} \\
  &= \frac{2}{\sqrt{2\pi}} \left( \left.(uv)\right\vert_{0}^{\infty} - \int_{0}^{\infty} v du   \right) \\
  &= \frac{2}{\sqrt{2\pi}} \left( 0 + \int_{0}^{\infty} e^{-\frac{z^2}{2}} ~ dz  \right) \\
  &= \frac{2}{\sqrt{2\pi}} \left( \frac{\sqrt{2\pi}}{2} \right) & \quad \text{from calculation of normalization constant }c \\
  &= 1 & \quad \blacksquare
\end{align}

```python
x = np.linspace(-5, 5, 500)
y = (x**2) * (np.e ** ((-x**2)/2))

_, ax = plt.subplots()
ax.plot(x, y, color='black')
ax.fill_between(x, y, where=y >= 0, facecolor='blue', interpolate=True)
ax.axhline(y=0, color='black')
ax.axvline(x=0, color='black')
plt.title("$g(x)$ is an even function")
plt.show()
```

## Standard Normal Distribution

This standard normal distribution $\mathcal{N}(0,1)$ is so important that it has its own name and notation.

$\mathcal{Z} \sim \mathcal{N}(0,1)$, where mean $\mu = 0$ and variance $\sigma^2 = 1$

### Notation

The standard normal distribution's CDF is notated by $\mathbb{\Phi}(Z)$.

### Cumulative distribution function

The cumulative distribution function of $\Phi(Z)$ is

\begin{align}
  \mathbb{\Phi}(Z) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{z} e^{-\frac{t^2}{2}} ~~ dt
\end{align}

### Property of $\Phi(-Z)$

\begin{align}
  \mathbb{\Phi}(-Z) = 1 - \mathbb{\Phi}(Z)
\end{align}

----


# Physics 256
## Simple Harmonic Oscillators

```python
import style
style._set_css_style('../include/bootstrap.css')
```

## Last Time

### [Notebook Link: 15_Baseball.ipynb](./15_Baseball.ipynb)

- motion of a pitched ball
- drag and the magnus force
- surface roughness of a projectile

## Today

- The simple harmonic pendulum

## Setting up the Notebook

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
plt.style.use('../include/notebook.mplstyle');
%config InlineBackend.figure_format = 'svg'
```

## Equation of Motion

The equation of motion for a simple linear pendulum of length $\ell$ and mass $m$ is givn by:

$$ m \frac{d \vec{v}}{d t} = \vec{F}_{\rm g} = -m g \hat{y}$$

Measuring $x$ and $y$ from the equlibrium position we have
\begin{align}
x &= \ell \sin \theta \\
y &= \ell (1-\cos\theta)
\end{align}

The kinetic and potential energy are:

\begin{align}
T &= \frac{1}{2} m \dot{r}^2 \\
&= \frac{1}{2} m (\dot{x}^2 + \dot{y}^2) \\
&= \frac{1}{2} m \ell^2 \dot{\theta}^2
\end{align}

\begin{equation}
V = m g \ell (1-\cos\theta).
\end{equation}

Thus, the Lagrangian is:
\begin{align}
\mathcal{L} &= T - V \\
&= \frac{1}{2} m \ell^2 \dot{\theta}^2 - m g \ell (1-\cos\theta)
\end{align}
and the equation of motion is given by the Euler-Lagrange formula

\begin{align}
\frac{\partial \mathcal{L}}{\partial \theta} - \frac{d}{dt} \frac{\partial \mathcal{L}}{\partial \dot{\theta}} &= 0 \\
-m g \ell \sin \theta - \frac{d}{dt} (m\ell^2 \dot{\theta}) &= 0
\end{align}

which yields the familiar equation:
\begin{equation}
\ddot{\theta} = -\frac{g}{\ell} \sin\theta .
\end{equation}

To solve this analytically, we are used to considering only small angle oscillations allowing us to replace $\sin\theta \simeq \theta$ for $\theta \ll 1$.  For $\theta(0) = \theta_0 \ll 1$ and $\dot{\theta}(0) = 0$ can be it can be integrated to give

$$ \theta(t) = \theta_0 \cos \left( \sqrt{\frac{g}{\ell}} t \right).$$

<div class="span alert alert-success">
<h2> Programming challenge </h2>
Use the Euler method to directly integrate the full equation of motion and compare with the analytical expression for $\theta_0 = \pi/12$ and $\dot{\theta}(0) =0$ for $\ell = 0.25$ m.

\begin{align}
\theta_{n+1} &= \theta_n + \omega_n \Delta t \\
\omega_{n+1} &= \omega_n - \frac{g}{\ell} \sin\theta_n \Delta t \\
\end{align}
</div>

<!--
    θ[n+1] = θ[n] + ω[n]*Δt
    ω[n+1] = ω[n] -(g/ℓ)*np.sin(θ[n])*Δt
-->

```python
from scipy.constants import pi as π
from scipy.constants import g

# constants and intitial conditions
ℓ = 0.25 # m
Δt = 0.001 # s

t = np.arange(0.0,5.0,Δt)
θ,ω = np.zeros_like(t),np.zeros_like(t)
θ[0] = π/12.0 # rad

for n in range(t.size-1):
    pass

# the small angle solution
plt.plot(t, θ[0]*np.cos(np.sqrt(g/ℓ)*t), label='Small angle solution')

# the Euler method
plt.plot(t,θ, label='Euler method')
plt.legend(loc='lower left')

plt.xlabel('Time [s]')
plt.ylabel('θ(t) [rad]')
```

## What went wrong?

The oscillations are **growing** with time!  This is our first encounter with a numerical procedure that is **unstable**.

Let's examine the total energy of the system where we can approximate $\cos\theta \simeq 1 - \theta^2/2$:

\begin{align}
E &= \frac{1}{2} m \ell^2 \omega^2 + m g \ell (1-\cos\theta) \\
&\simeq \frac{1}{2}m \ell^2 \left(\omega^2 + \frac{g}{\ell}\theta^2 \right).
\end{align}

Writing things in terms of our Euler variables:

\begin{align}
E_{n+1} &= \frac{1}{2}m\ell^2 \left[\left(\omega_n - \frac{g}{\ell}\theta_n \Delta t\right)^2 + \frac{g}{\ell}\left(\theta_n + \omega_n\Delta t\right)^2 \right] \\
&= E_{n} + \frac{1}{2}mg \ell \left(\omega_i^2 + \frac{g}{\ell} \theta_n^2\right) \Delta t^2.
\end{align}

This tells us the origin of the problem: **the energy is increasing without bound, regardless of the size of $\Delta t$**.

### Question: Why didn't we encounter this problem previously?

<!-- With the exception of constant acceleration, we always had it, we just never noticed it on the timescales we were interested in. -->

### How do we fix it?

We can consider alternative higher-order ODE solvers (as described in Appendix A of the textbook).  However, there is a very simple fix that works here:

### Euler-Cromer Method
Looking at our original discretized equations:

\begin{align}
\theta_{n+1} &= \theta_n + \omega_n \Delta t \\
\omega_{n+1} &= \omega_n - \frac{g}{\ell} \sin\theta_n \Delta t
\end{align}

we can make the simple observation that we can replace the order of evaluation and use the updated value of $\omega$ in our calculation of $\theta$.

\begin{align}
\omega_{n+1} &= \omega_n - \frac{g}{\ell} \sin\theta_n \Delta t \\
\theta_{n+1} &= \theta_n + \omega_{n+1} \Delta t
\end{align}

This leads to the energy being *approximately* conserved at each step:

\begin{equation}
E_{n+1} = E_{n} + \frac{1}{2}m g \left(\omega_n^2 - \frac{g}{\ell}\theta_n^2 \right)\Delta t^2 + \mathrm{O}(\Delta t^3).
\end{equation}

```python
from scipy.constants import pi as π
from scipy.constants import g

# constants and intitial conditions
ℓ = 0.25 # m
Δt = 0.001 # s

t = np.arange(0.0,4.0,Δt)
θ,ω = np.zeros_like(t),np.zeros_like(t)
θ[0] = π/12.0 # rad

for n in range(t.size-1):
    pass

# the small angle solution
plt.plot(t, θ[0]*np.cos(np.sqrt(g/ℓ)*t), label='Small angle solution')

# the Euler-Cromer method
plt.plot(t,θ, label='Euler Cromer method')
plt.legend(loc='lower left',frameon=True)

plt.xlabel('Time [s]')
plt.ylabel('θ(t) [rad]')
```

## There are still some noticable deviations, thoughts?

<!--Non-linear corrections. -->

## Turning on Non-Linearity

An analytical solution exists without the small-angle approximation, but it is considerably more complicated:

\begin{eqnarray}
\theta(t) &=& 2 \sin^{-1} \left\{ k\, \mathrm{sn}\left[K(k^2)-\sqrt{\frac{g}{\ell}} t; k^2\right] \right\} \newline
k &=& \sin \frac{\theta_0}{2} \newline
K(m) &=& \int_0^1 \frac{d z}{\sqrt{(1-z^2)(1-m z^2)}}
\end{eqnarray}

<!--

# the exact solution
plt.plot(t,non_linear_θ(ℓ,θ[0],t), label='Exact solution')

-->

```python
def non_linear_θ(ℓ,θₒ,t):
    '''The solution for θ for the non-linear pendulum.'''
    # use special functions
    from scipy import special
    k = np.sin(θₒ/2.0)
    K = special.ellipk(k*k)
    (sn,cn,dn,ph) = special.ellipj(K-np.sqrt(g/l)*t,k*k)
    return 2.0*np.arcsin(k*sn)
```

```python
# the small angle solution
plt.plot(t, θ[0]*np.cos(np.sqrt(g/ℓ)*t), label='Small angle solution')

# the Euler-Cromer method
plt.plot(t,θ,label='Euler Cromer method')

# the exact solution in terms of special functions
plt.plot(t,non_linear_θ(ℓ,θ[0],t), label='Exact', alpha=0.5)
plt.legend(loc='lower left',frameon=True)

plt.xlabel('Time [s]')
plt.ylabel('θ(t) [rad]')
```

```python

```


# Chapter 3 - Developing Templates

Generating SoftMax distributions from normals could get quite tedious – for any sufficiently complicated shape, the number of normals to be used could be excessive. Let's add a layer of abstraction onto all our work.

##Polygon Construction

We can put everything together from all we've talked about (shifting the distribution and generating weights from normals) to a more tangible process: generating a softmax distribution from a [polytope](http://en.wikipedia.org/wiki/Polytope). Let's motivate this with an example first.

Imagine you worked at the Pentagon as an HRI researcher. One day, while pondering the nature of language, you happened to look out your window and spot an intruder. If you called a human security officer, you might say something like, "I see an intruder in front of the Heliport facade." We can use our SoftMax classifier to translate this same sentence for a security bot to understand.

First, we'd need to divide the space in a similar way we did for the Pac-Man problem:

As opposed to our Pac-Man problem, we can't assign weights by inspection. Instead, we'll use our weights-from-normals tactic to generate our weights for each class, and our shifted bias tactic to place those weights appropriately.

### Step 1: Define Polytope
We can use a geometry library like [Shapely](http://toblerity.org/shapely/shapely.html) to define custom polytopes (in this case, a pentagon). For a quick way to get ideal pentagon vertex coordinates, you can either calculate them by hand or use [some online tools](http://www.mathsisfun.com/geometry/pentagon.html).

Let's try a pentagon with the following coordinates (starting at the corner between the South Parking Entrance and the Heliport Facade):

$$
\begin{align}
P_1 &= (P_{1x}, P_{1y}) = (-1.90,-0.93) \\
P_2 &= (-1.40,1.45) \\
P_3 &= (1.03,1.71) \\
P_4 &= (2.02,-0.51) \\
P_5 &= (0.21,-2.15) \\
\end{align}
$$

### Step 2: Get Normals and Offsets
We want to get six classes, so we'd like to specify $\frac{6(6-1)}{2} = 15$ normal vectors in order to use our transformation matrix $A$. But, we only have six unknowns, so we can reduce the size of our $A$ matrix. That is, we can use:

$$
\mathbf{N} = \begin{bmatrix}
\mathbf{n}_{0,1}^T \\
\mathbf{n}_{0,2}^T \\
\mathbf{n}_{0,3}^T \\
\mathbf{n}_{0,4}^T \\
\mathbf{n}_{0,5}^T \\
\mathbf{n}_{1,2}^T \\
\end{bmatrix}
= \begin{bmatrix}
-1 & 1 & 0 & 0 & 0 & 0 \\
-1 & 0 & 1 & 0 & 0 & 0 \\
-1 & 0 & 0 & 1 & 0 & 0 \\
-1 & 0 & 0 & 0 & 1 & 0 \\
-1 & 0 & 0 & 0 & 0 & 1 \\
0 & -1 & 1 & 0 & 0 & 0 \\
\end{bmatrix}
\begin{bmatrix}
\mathbf{w}_{0}^T \\
\mathbf{w}_{1}^T \\
\mathbf{w}_{2}^T \\
\mathbf{w}_{3}^T \\
\mathbf{w}_{4}^T \\
\mathbf{w}_{5}^T \\
\end{bmatrix}
= \mathbf{A}\mathbf{W}
$$

Where $\mathbf{n}_{0,1}$ is the boundary between the interior and the South Parking Entrance, and so on.

Except, we can be smarter about this. We only care about the *relative* weights, so why not define one class and solve for the weights of all other classes? Since we have one interior class with weights $w_0$, simply define $w_0 = \begin{bmatrix}0 & 0 \end{bmatrix}^T$ and $b_0 = 0$, leaving us with the following five equations and five unkowns:

$$
\mathbf{N} = \begin{bmatrix}
\mathbf{n}_{0,1}^T \\
\mathbf{n}_{0,2}^T \\
\mathbf{n}_{0,3}^T \\
\mathbf{n}_{0,4}^T \\
\mathbf{n}_{0,5}^T \\
\end{bmatrix}
= \begin{bmatrix}
1 & 0 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 & 1 \\
\end{bmatrix}
\begin{bmatrix}
\mathbf{w}_{1}^T \\
\mathbf{w}_{2}^T \\
\mathbf{w}_{3}^T \\
\mathbf{w}_{4}^T \\
\mathbf{w}_{5}^T \\
\end{bmatrix}
= \mathbf{A}\mathbf{W}
$$

Does it make sense that the weights we'd use correspond directly to the class boundaries of each class with some zero-weighted interior class? Yes: think of a class boundary as defined by its normal vector. Those normal vectors point exactly in the direction of greatest probability of a given class.

Thus, we have:

$$
\mathbf{n}_{0,i} = \mathbf{w}_{i} \; \forall i \in N
$$

We have the normals, but solving for the class biases will require digging deeper. We need the equation for a normal fixed to the surface of the polytope (not simply its magnitude and direction!).

In $\mathbb{R}^2$, we know that a line is uniquely defined by two points passing through it – a face's bounding vertices, for instance. This can help us find the normal vectors and offsets, giving us the weights and biases.

Recall the specification of our hyperplanes in $\mathbb{R}^2$:

\begin{align}
0 &= (\mathbf{w}_i - \mathbf{w}_j)^T\mathbf{x} + (b_i - b_j) \\
  &= (w_{i,x} - w_{j,x})x + (w_{i,y} - w_{j,y})y + (b_i - b_j) \\
  &= w_{i,x}x + w_{i,y}y + b_i
\end{align}

Where the last line assumes $j$ is the interior class with weights and a bias of 0.

Since we have two points on this line segment (and any third point from a linear combination of the first two), we can use their $x$ and $y$ values to calculate our weights:

\begin{equation}\label{eq:nullspace}
\begin{bmatrix}
x_1 & y_1 & 1 \\
x_2 & y_2 & 1 \\
x_3 & y_3 & 1 \\
\end{bmatrix}
\begin{bmatrix}
w_{i,x}\\
w_{i,y}\\
b_i
\end{bmatrix}
=\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}
\end{equation}

The non-trivial solution to $\ref{eq:nullspace}$ can be found through various decomposition techniques. We use [Singular Value Decomposition](http://en.wikipedia.org/wiki/Singular_value_decomposition).

In short, given any polygon, we can use its vertices to find the equations of the normals representing the class boundaries between the interior class and an exterior class for each face. Let's try this out and see if it works well.

Note that this part, as well as several future ones, require long swaths of code
to be fully explained. Rather than include the code in this document, you can always find it
on [our Github](https://github.com/COHRINT/cops_and_robots/blob/master/src/cops_and_robots/robo_tools/fusion/softmax.py).

```python
import numpy as np
%matplotlib inline

from cops_and_robots.robo_tools.fusion.softmax import SoftMax, make_regular_2D_poly
poly = make_regular_2D_poly(5, max_r=2, theta=np.pi/3.1)
labels = ['Interior',
          'Mall Terrace Entrance',
          'Heliport Facade',
          'South Parking Entrance',
          'Concourse Entrance',
          'River Terrace Entrance',
         ]
sm = SoftMax(poly=poly, class_labels=labels, resolution=0.1)
sm.plot(plot_poly=True, plot_normals=False)

```

**NOTE: 3D Plotting currently borked**

Well, that looks like the class boundaries are lining up just fine, but what about the probability distributions themselves? They seem a bit diffuse. If you remember from Chapter 1, we can simply multiply the weights and biases by the same value to raise the steepness of each class. Let's try that:

```python
steepness = 5
sm = SoftMax(poly=poly, class_labels=labels, resolution=0.1, steepness=5)
sm.plot(plot_poly=True, plot_normals=False)
```

As expected, our boundaries stayed the same but our probabilities are less spread out. Looking good!

However, we need to address a few assumptions. Most importantly, our interior class will not always be centered at the origin. Let's look at a shifted coordinate frame again, with the center of our polygon at $(-2,3)$:

\begin{align}
\mathbf{x}' &= \begin{bmatrix}x & y\end{bmatrix}^T + \begin{bmatrix}2 & -3\end{bmatrix}^T = \begin{bmatrix}x + 2 & y -3\end{bmatrix}^T \\
0 &= (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{x}' + (b_i - b_j) \\
&= (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{x} + (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{b} + (b_i - b_j)\\
&= \mathbf{w}_i^T \mathbf{x} + \mathbf{w}_i^T \mathbf{b} + b_i\\
&= w_{i,x}x + w_{i,y}y + \begin{bmatrix}w_{i,x} & w_{i,y}\end{bmatrix}\begin{bmatrix}-2 \\ 3\end{bmatrix}  + b_i\\
&= w_{i,x}x + w_{i,y}y  + b_i^\prime\\
\end{align}

But, remember that we need to perform the coordinate shift on the points we use as well:

$$
\begin{bmatrix}
x_1 + 2 & y_1 - 3 & 1 \\
x_2 + 2 & y_2 - 3 & 1 \\
x_3 + 2 & y_3 - 3 & 1 \\
\end{bmatrix}
\begin{bmatrix}
w_{i,x}\\
w_{i,y}\\
b_i^\prime
\end{bmatrix}
=\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}
$$

```python
poly = make_regular_2D_poly(5, max_r=2, theta=-np.pi/4, origin=(-2,3))
sm = SoftMax(poly=poly, class_labels=labels, resolution=0.1, steepness=5)
sm.plot(plot_poly=True, plot_normals=False)
```

Great! We've successfully decomposed the space around the Pentagon, so we can tell the automatic security bots where the suspect is without having to pull out a map of the Pentagon and show them directly where on the map our intruder may be. That is, we've replaced communication of specific coordinates with the communication of 'zones' formed by spatial relationships to landmarks.

However, the methodology build up to this point doesn't work for all cases. For instance: what happens if we want to use a non-convex shape to develop a SoftMax model? [Chapter 4](04_mms.ipynb) will dive into some of these pitfalls and how to get around them. For the time being, we'll simply show that, while we can't handle non-convex shapes yet, we can handle convex irregular shapes:

```python
poly = Polygon([(-1.0, 0.0),
                (-1.0, 1.0),
                (-3.5, 3.3),
                (-3.0, -2.0),
                (-2.0, -2.0),
                ])

sm = SoftMax(poly=poly, steepness=6)
sm.plot(plot_poly=True)
```

```python
from IPython.core.display import HTML

# Borrowed style from Probabilistic Programming and Bayesian Methods for Hackers
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Roboto:100,100italic,500,300,300italic,400' rel='stylesheet' type='text/css'>

<style>
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1, h2, h3, h4 {
        font-family: "Roboto", "wingdings", sans-serif;
    }
    h1{
        font-weight: 500;
    }
    h2{
        font-weight: 400;
    }
    h3{
        font-weight: 300 !important;
/*         font-style: italic; */
    }
    h4{
        font-weight: 300 !important;
        font-style: italic;
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: "HelveticaNeue-light", "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 120%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }

    .rounded-box{
    border-radius: 25px;
    border: 2px solid #8AC007;
    padding: 10px;
    padding-top: 0px;
    }
</style>


# Report Title
### Team Ford

```python
import pandas as pd
import numpy as np
from scipy.spatial import distance_matrix
import sympy as sym
import seaborn as sns
import matplotlib.pyplot as plt
```

## Abstract
Short summary about the data and what it tells you about your project.

## Data input
In this section include code that reads in the csv file

```python
data = pd.read_csv('2022_Project_distance_Matrix.csv',header='infer')
data.head()
```

      ARFL
      Argonne
      Boeing
      Delta Dental
      Ford
      Hope Village
      Kellogg's
      Neogen
      Old Nation
      Qside

      0
      7
      4
      9
      8
      3
      0
      6
      2
      5
      1

      1
      0
      1
      3
      2
      4
      8
      6
      5
      7
      9

      2
      3
      2
      1
      6
      0
      8
      5
      4
      7
      9

      3
      8
      7
      3
      9
      4
      5
      0
      6
      1
      2

      4
      0
      2
      1
      8
      7
      3
      9
      6
      4
      5

## Data Cleaning
In this section provide code for converting the raw data into clean and usable data structures (if needed)

## Data Modeling

This section that builds a model.  This doesn't have to be all that advanced. You should probably start with some simple statistical models such as an average distance matrix.

```python
dist_mat = distance_matrix(data.T,data.T,p=2)
fig, ax = plt.subplots(figsize = (15,15))
ax = sns.heatmap(dist_mat,annot=True)
```

```python
np.dot(data.T,data)
```

    array([[ 656,  567,  628,  785,  679,  465,  717,  500,  406,  447],
           [ 567,  791,  740,  848,  805,  598,  798,  550,  493,  515],
           [ 628,  740, 1024,  825,  868,  485,  890,  644,  460,  456],
           [ 785,  848,  825, 1476, 1121, 1017, 1177,  802,  841,  918],
           [ 679,  805,  868, 1121, 1432,  797, 1131,  804,  594,  679],
           [ 465,  598,  485, 1017,  797, 1283, 1106,  660,  988, 1106],
           [ 717,  798,  890, 1177, 1131, 1106, 1649,  904,  921, 1102],
           [ 500,  550,  644,  802,  804,  660,  904,  747,  568,  616],
           [ 406,  493,  460,  841,  594,  988,  921,  568,  930,  909],
           [ 447,  515,  456,  918,  679, 1106, 1102,  616,  909, 1127]],
          dtype=int64)

## Data Visualization

This section make some graphs visualizing your results.  A distance matrix and/or network graph may be cool.  Think though the best way to show what you learned.

## Conclusion

This should be similar to the abstract but with more details. What can you conclude about your project from this data?