Solution to: [Day 8: Least Square Regression Line](https://www.hackerrank.com/challenges/s10-least-square-regression-line/problem)

Table of Contents

- Table of Contents
- Notes
    - Regression Line
    - Finding the Value of b
    - Finding the Value of a
    - Sums of Squares
    - Coefficient of Determination (R-squared)
- Example
    - Data
    - Important stats
    - Compute b
    - Compute a
    - Regression
- Solution
    - Imports
    - Input
    - Mean
    - Calculate b
    - Calculate a
    - Predict
    - Scale
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Notes

## Regression Line
If our data shows a linear relationship between X and Y, then the straight line which best describes the relationship is the regression line. The regression line is given by

\begin{equation}
\large
Y = a + bX.
\end{equation}

## Finding the Value of b
The value of b can be calculated using either of the following formulae:

\begin{equation}
\large
b = \frac
{n \sum (x_{i} y_{i}) - (\sum x_{i}) (\sum y_{i})}
{n \sum x_{i} ^{2} - (\sum x_{i}) ^{2}}
\end{equation}

*OR**

\begin{equation}
\large
b = p \frac
{\sigma Y}
{\sigma X}
\end{equation}

- where p is the Pearson Correlation

## Finding the Value of a
\begin{equation}
\large
a = ȳ - b * x̄
\end{equation}

## Sums of Squares
- **Total Sum of Squares** (SST) = $\sum (y_{i} - \bar{y})^{2}$
- **Regression Sum of Squares** (SSR) = $\sum (\hat{y}_{i} - \bar{y})^{2}$
- **Error Sum of Squares** (SSE) = $\sum (\hat{y}_{i} - y)^{2}$

*Notes*:
- ŷ = predicted value
- ȳ = mean value
- yi = i-th value of y

If SSE is small, we can assume that our fit is good.

## Coefficient of Determination (R-squared)

\begin{equation}
\large
R ^{2} = \frac
{SSR}{SST} = 1 - \frac{SSE}{SST}
\end{equation}

$R^{2}$ multiplied by 100  gives the percent of variation attributed to the linear regression between Y and X.

# Example

## Data

    x = [1,2,3,4,5]
    y = [2,1,4,3,5]

## Important stats
- n = 5
- $\sum x$ = 15
- $\bar{X}$ = 3
- $\sum Y$ = 15
- $\bar{y}$ = 3
- $X^{2} = Σ(x**2) = 55$
- XY = $\sum(xy)$ = 53

## Compute b

\begin{equation}
\large
b = \frac
{n \sum (x_{i} * y_{i}) - \sum x_{i} * \sum y_{i}}
{n \sum x_{i}^{2} - (\sum x_{i})^{2}}
\end{equation}

\begin{equation}
\large
b = \frac
{5 * 53 - 15 * 15}
{5 * 55 - 15 * 2}
\end{equation}

\begin{equation}
\large
= 0.8
\end{equation}

## Compute a
\begin{equation}
\large
a = \hat{y} - b * \hat{x}
\end{equation}

\begin{equation}
\large
= 3 - 0.8 * 3 = 0.6
\end{equation}

## Regression
\begin{equation}
\large
\hat{y} = 0.6 + 0.8 * y
\end{equation}

# Solution

## Imports

```python
from typing import Tuple
```

## Input

```python
def get_input() -> Tuple[list, list]:
    """Returns tuple representing data points

    Returns:
        Tuple[list, list]: X and Y, respectively
    """
    num_inputs = 5
    x, y = [], []
    for i in range(num_inputs):
        x_val, y_val = [float(val) for val in input().split()]
        x.append(x_val)
        y.append(y_val)

    return (x, y)
```

## Mean

```python
def calc_mean(x: list) -> float:
    """Returns mean of list

    Args:
        x (list): Input list

    Returns:
        float: mean of list
    """
    return sum(x) / len(x)

```

## Calculate b

```python
def calc_b(x: list, y: list) -> float:
    """Returns b coefficient for x in simple linear regression

    Args:
        x (list): series 1
        y (list): series 2

    Returns:
        float: b coefficient for x
    """
    n = len(x)
    xy = sum([x[i] * y[i] for i in range(len(x))])
    x_sq_sum = sum([val**2 for val in x])

    return (n * xy - sum(x) * sum(y)) / ( n * x_sq_sum - sum(x) ** 2)
```

## Calculate a

```python
def calc_a(x: list, y: list, b: float) -> float:
    """Returns `a` coefficient for regression equation.

    Args:
        x (list): predictor variable
        y (list): Outcome variable
        b (float): coefficient for predictor variable

    Returns:
        float: slope intercept
    """
    return calc_mean(y) - b * calc_mean(x)
```

## Predict

```python
def predict_val(a: float, b: float, x: int = 80) -> float:
    """Predicts value for simple linear regression equation

    Args:
        a (float): Slope intercept constant
        b (float): Coefficient for x
        x (int): value of X

    Returns:
        float: predicted value of y
    """
    return a + b * x
```

## Scale

```python
def print_to_scale(num: float) -> None:
    """Prints number to 3 decimal

    Args:
        num (float): Number to print
    """
    print(f"{num :.3f}")
```

## Main

```python
def main():
    x, y = get_input()

    b = calc_b(x, y)
    a = calc_a(x, y, b)

    prediction = predict_val(a, b)
    print_to_scale(prediction)

if __name__ == "__main__":
    main()
```

    95 85
    85 95
    80 70
    70 65
    60 70
    78.288


```python
import numpy as np
import sympy as sp
import pandas as pd
import math

import interpolate_exp_cos as p1
import plot_sin_eps as p2
import integrate_exp as p3
import diff_functions as p4

import matplotlib.pyplot as plt

# Needed only in Jupyter to render properly in-notebook
%matplotlib inline
```

# Homework 5

## Chinmai Raman

### 3/18/2016

### B.1 Interpolating a discrete function

Part 1:

```python
p1.f(-.45)
```

    -0.77671500104510183

```python
p1.S_k(-1, 1, -.45, 10)
```

    -0.93597357483441579

Part 2:

Value of function f(x) = $e^(-x^2) * cos(2\pi x)$ at x = -0.45 :

### q = 2

```python
p1.f(-.45)
```

    -0.77671500104510183

```python
p1.S_k(-1, 1, -.45, 2)
```

    1.284454251472851

Error in approximation at q = 2:

```python
p1.error(-1, 1, -.45, 2)
```

    2.0611692525179528

### q = 4

```python
p1.S_k(-1, 1, -.45, 4)
```

    2.6009207047642646

Error in approximation at q = 4:

```python
p1.error(-1, 1, -.45, 4)
```

    3.3776357058093662

### q = 8

```python
p1.S_k(-1, 1, -.45, 8)
```

    -0.79999999999999982

Error in approximation at q = 8:

```python
p1.error(-1, 1, -.45, 8)
```

    0.023284998954897995

### q= 16

```python
p1.S_k(-1, 1, -.45, 16)
```

    -0.98295202860868525

Error in approximation at q = 16:

```python
p1.error(-1, 1, -.45, 16)
```

    0.20623702756358342

### B.2 Studying a function for different parameter values

$f(x) = sin\dfrac{1}{x+eps}$ in [0,1]

```python
p2.sin_graph(0.2, 10)
```

```python
p2.multigraph(0.2, 10)
```

How large n needs to be in order for the difference between the max of the function $f(x) = sin\dfrac{1}{x+eps}$ in [0,1] using n nodes and n+10 nodes to be less than 0.1:

```python
p2.choose_n(0.2)
```

    3

```python
p2.choose_n(0.1)
```

    3

```python
p2.choose_n(0.05)
```

    1

n = 200 * eps (for an epsilon of 0.2). Increasing n further does not change the plot so that it is visible on the screen.

```python
p2.multigraph(0.2, 40)
```

### B.6 Using the trapezoid method to approximate integrals

```python
p3.graph()
```

The plot is symmetric over the y-axis. Therefore the integral of the function from -$\infty$ to $\infty$ will be the same as twice the integral of the function from 0 to $\infty$

```python
p3.T(100, 10)
```

    1.7370047738874057

```python
p3.trap(100, 10) * 2
```

    1.7370047738874057

$$\int_{-L}^{L} {e}^{-x^2} dx = 2\int_{0}^{L} {e}^{-x^2} dx$$

```python
p3.table()
```

      L = 2
      L = 4
      L = 6
      L = 8
      L = 10

      n = 100
      1.729607
      1.737005
      1.737005
      1.737005
      1.737005

      n = 200
      1.746886
      1.754729
      1.754729
      1.754729
      1.754729

      n = 300
      1.752645
      1.760637
      1.760637
      1.760637
      1.760637

      n = 400
      1.755525
      1.763592
      1.763592
      1.763592
      1.763592

      n = 500
      1.757252
      1.765364
      1.765364
      1.765364
      1.765364

```python
p3.error()
```

      L = 2
      L = 4
      L = 6
      L = 8
      L = 10

      n = 100
      -0.042847
      -0.035449
      -0.035449
      -0.035449
      -0.035449

      n = 200
      -0.025568
      -0.017725
      -0.017725
      -0.017725
      -0.017725

      n = 300
      -0.019808
      -0.011816
      -0.011816
      -0.011816
      -0.011816

      n = 400
      -0.016929
      -0.008862
      -0.008862
      -0.008862
      -0.008862

      n = 500
      -0.015201
      -0.007090
      -0.007090
      -0.007090
      -0.007090

The error decreases as n increases and as L increases

### B.8 Plotting functions and their derivatives

$$f'(x)={\dfrac{1}{x+\dfrac{1}{100}}}$$

```python
p4.graph(p4.diff(p4.f, 1/1000, 1, 100))
```

```python
p4.multigraph(p4.diff(p4.f, 1/1000, 1, 100), p4.fprime)
```

$$g'(x)-10{e}^{10x}sin({e}^{10x})$$

```python
p4.graph(p4.diff(p4.g, 1/1000, 1, 100))
```

```python
p4.multigraph(p4.diff(p4.g, 1/1000, 1, 100), p4.gprime)
```

$$h'(x)=(ln x)x^x+xx^{x-1}$$

```python
p4.graph(p4.diff(p4.h, 1/1000, 1, 100))
```

```python
p4.multigraph(p4.diff(p4.h, 1/1000, 1, 100), p4.hprime)
```

For the functions f(x) and h(x), an n of 100 seems to be sufficient as to make the graphs of the analytical and discrete derivatives visibly indistinguishable. For g(x), however, it is unlikely that any value of n will make the graphs indistinguishable upon superimposition, as the slope of g(x) as x -> 1 approaches very high values.

```python

```


# Statistical Data Modeling

Some or most of you have probably taken some undergraduate- or graduate-level statistics courses. Unfortunately, the curricula for most introductory statisics courses are mostly focused on conducting **statistical hypothesis tests** as the primary means for interest: t-tests, chi-squared tests, analysis of variance, etc. Such tests seek to esimate whether groups or effects are "statistically significant", a concept that is poorly understood, and hence often misused, by most practioners. Even when interpreted correctly, statistical significance is a questionable goal for statistical inference, as it is of limited utility.

A far more powerful approach to statistical analysis involves building flexible **models** with the overarching aim of *estimating* quantities of interest. This section of the tutorial illustrates how to use Python to build statistical models of low to moderate difficulty from scratch, and use them to extract estimates and associated measures of uncertainty.

```
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Set some Pandas options
pd.set_option('display.notebook_repr_html', False)
pd.set_option('display.max_columns', 20)
pd.set_option('display.max_rows', 25)
```

Estimation
==========

An recurring statistical problem is finding estimates of the relevant parameters that correspond to the distribution that best represents our data.

In **parametric** inference, we specify *a priori* a suitable distribution, then choose the parameters that best fit the data.

* e.g. $\mu$ and $\sigma^2$ in the case of the normal distribution

```
x = np.array([ 1.00201077,  1.58251956,  0.94515919,  6.48778002,  1.47764604,
        5.18847071,  4.21988095,  2.85971522,  3.40044437,  3.74907745,
        1.18065796,  3.74748775,  3.27328568,  3.19374927,  8.0726155 ,
        0.90326139,  2.34460034,  2.14199217,  3.27446744,  3.58872357,
        1.20611533,  2.16594393,  5.56610242,  4.66479977,  2.3573932 ])
_ = plt.hist(x, bins=8)
```

### Fitting data to probability distributions

We start with the problem of finding values for the parameters that provide the best fit between the model and the data, called point estimates. First, we need to define what we mean by ‘best fit’. There are two commonly used criteria:

* **Method of moments** chooses the parameters so that the sample moments (typically the sample mean and variance) match the theoretical moments of our chosen distribution.
* **Maximum likelihood** chooses the parameters to maximize the likelihood, which measures how likely it is to observe our given sample.

### Discrete Random Variables

$$X = \{0,1\}$$

$$Y = \{\ldots,-2,-1,0,1,2,\ldots\}$$

**Probability Mass Function**:

For discrete $X$,

$$Pr(X=x) = f(x|\theta)$$

***e.g. Poisson distribution***

The Poisson distribution models unbounded counts:

$$Pr(X=x)=\frac{e^{-\lambda}\lambda^x}{x!}$$

* $X=\{0,1,2,\ldots\}$
* $\lambda > 0$

$$E(X) = \text{Var}(X) = \lambda$$

### Continuous Random Variables

$$X \in [0,1]$$

$$Y \in (-\infty, \infty)$$

**Probability Density Function**:

For continuous $X$,

$$Pr(x \le X \le x + dx) = f(x|\theta)dx \, \text{ as } \, dx \rightarrow 0$$

***e.g. normal distribution***

$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right]$$

* $X \in \mathbf{R}$
* $\mu \in \mathbf{R}$
* $\sigma>0$

$$\begin{align}E(X) &= \mu \cr
\text{Var}(X) &= \sigma^2 \end{align}$$

### Example: Nashville Precipitation

The dataset `nashville_precip.txt` contains [NOAA precipitation data for Nashville measured since 1871](http://bit.ly/nasvhville_precip_data). The gamma distribution is often a good fit to aggregated rainfall data, and will be our candidate distribution in this case.

```
precip = pd.read_table("data/nashville_precip.txt", index_col=0, na_values='NA', delim_whitespace=True)
precip.head()
```

           Jan   Feb   Mar    Apr   May   Jun   Jul   Aug   Sep   Oct   Nov   Dec
    Year
    1871  2.76  4.58  5.01   4.13  3.30  2.98  1.58  2.36  0.95  1.31  2.13  1.65
    1872  2.32  2.11  3.14   5.91  3.09  5.17  6.10  1.65  4.50  1.58  2.25  2.38
    1873  2.96  7.14  4.11   3.59  6.31  4.20  4.63  2.36  1.81  4.28  4.36  5.94
    1874  5.22  9.23  5.36  11.84  1.49  2.87  2.65  3.52  3.12  2.63  6.12  4.19
    1875  6.15  3.06  8.14   4.22  1.73  5.63  8.12  1.60  3.79  1.25  5.46  4.30

```
_ = precip.hist(sharex=True, sharey=True, grid=False)
plt.tight_layout()
```

The first step is recognixing what sort of distribution to fit our data to. A couple of observations:

1. The data are skewed, with a longer tail to the right than to the left
2. The data are positive-valued, since they are measuring rainfall
3. The data are continuous

There are a few possible choices, but one suitable alternative is the **gamma distribution**:

$$x \sim \text{Gamma}(\alpha, \beta) = \frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)}$$

The ***method of moments*** simply assigns the empirical mean and variance to their theoretical counterparts, so that we can solve for the parameters.

So, for the gamma distribution, the mean and variance are:

$$ \hat{\mu} = \bar{X} = \alpha \beta $$
$$ \hat{\sigma}^2 = S^2 = \alpha \beta^2 $$

So, if we solve for these parameters, we can use a gamma distribution to describe our data:

$$ \alpha = \frac{\bar{X}^2}{S^2}, \, \beta = \frac{S^2}{\bar{X}} $$

Let's deal with the missing value in the October data. Given what we are trying to do, it is most sensible to fill in the missing value with the average of the available values.

```
precip.fillna(value={'Oct': precip.Oct.mean()}, inplace=True)
```

            Jan    Feb   Mar    Apr    May   Jun   Jul   Aug    Sep   Oct   Nov  \
    Year
    1871   2.76   4.58  5.01   4.13   3.30  2.98  1.58  2.36   0.95  1.31  2.13
    1872   2.32   2.11  3.14   5.91   3.09  5.17  6.10  1.65   4.50  1.58  2.25
    1873   2.96   7.14  4.11   3.59   6.31  4.20  4.63  2.36   1.81  4.28  4.36
    1874   5.22   9.23  5.36  11.84   1.49  2.87  2.65  3.52   3.12  2.63  6.12
    1875   6.15   3.06  8.14   4.22   1.73  5.63  8.12  1.60   3.79  1.25  5.46
    1876   6.41   2.22  5.28   3.62   3.40  5.65  7.15  5.77   2.52  2.68  1.26
    1877   4.05   1.06  4.98   9.47   1.25  6.02  3.25  4.16   5.40  2.61  4.93
    1878   3.34   2.10  3.48   6.88   2.33  3.28  9.43  5.02   1.28  2.17  3.20
    1879   6.32   3.13  3.81   2.88   2.88  2.50  8.47  4.62   5.18  2.90  5.85
    1880   3.74  12.37  8.16   5.26   4.13  3.97  5.69  2.22   5.39  7.24  5.77
    1881   3.54   5.48  2.79   5.12   3.67  3.70  0.86  1.81   6.57  4.80  4.89
    1882  14.51   8.61  9.38   3.59   7.38  2.54  4.06  5.54   1.61  1.11  3.60
    ...     ...    ...   ...    ...    ...   ...   ...   ...    ...   ...   ...
    2000   3.52   3.75  3.34   6.23   7.66  1.74  2.25  1.95   1.90  0.26  6.39
    2001   3.21   8.54  2.73   2.42   5.54  4.47  2.77  4.07   1.79  4.61  5.09
    2002   4.93   1.99  9.40   4.31   3.98  3.76  5.64  3.13   6.29  4.48  2.91
    2003   1.59   8.47  2.30   4.69  10.73  7.08  2.87  3.88   8.70  1.80  4.17
    2004   3.60   5.77  4.81   6.69   6.90  3.39  3.19  4.24   4.55  4.90  5.21
    2005   4.42   3.84  3.90   6.93   1.03  2.70  2.39  6.89   1.44  0.02  3.29
    2006   6.57   2.69  2.90   4.14   4.95  2.19  2.64  5.20   4.00  2.98  4.05
    2007   3.32   1.84  2.26   2.75   3.30  2.37  1.47  1.38   1.99  4.95  6.20
    2008   4.76   2.53  5.56   7.20   5.54  2.21  4.32  1.67   0.88  5.03  1.75
    2009   4.59   2.85  2.92   4.13   8.45  4.53  6.03  2.14  11.08  6.49  0.67
    2010   4.13   2.77  3.52   3.48  16.43  4.96  5.86  6.99   1.17  2.49  5.41
    2011   2.31   5.54  4.59   7.51   4.38  5.04  3.46  1.78   6.20  0.93  6.15

           Dec
    Year
    1871  1.65
    1872  2.38
    1873  5.94
    1874  4.19
    1875  4.30
    1876  0.95
    1877  2.49
    1878  6.04
    1879  9.15
    1880  3.32
    1881  4.85
    1882  1.52
    ...    ...
    2000  3.44
    2001  3.32
    2002  5.81
    2003  3.19
    2004  5.93
    2005  2.46
    2006  3.41
    2007  3.83
    2008  6.72
    2009  3.99
    2010  1.87
    2011  4.25

    [141 rows x 12 columns]

Now, let's calculate the sample moments of interest, the means and variances by month:

```
precip_mean = precip.mean()
precip_mean
```

    Jan    4.523688
    Feb    4.097801
    Mar    4.977589
    Apr    4.204468
    May    4.325674
    Jun    3.873475
    Jul    3.895461
    Aug    3.367305
    Sep    3.377660
    Oct    2.610500
    Nov    3.685887
    Dec    4.176241
    dtype: float64

```
precip_var = precip.var()
precip_var
```

    Jan    6.928862
    Feb    5.516660
    Mar    5.365444
    Apr    4.117096
    May    5.306409
    Jun    5.033206
    Jul    3.777012
    Aug    3.779876
    Sep    4.940099
    Oct    2.741659
    Nov    3.679274
    Dec    5.418022
    dtype: float64

We then use these moments to estimate $\alpha$ and $\beta$ for each month:

```
alpha_mom = precip_mean ** 2 / precip_var
beta_mom = precip_var / precip_mean
```

```
alpha_mom, beta_mom
```

    (Jan    2.953407
     Feb    3.043866
     Mar    4.617770
     Apr    4.293694
     May    3.526199
     Jun    2.980965
     Jul    4.017624
     Aug    2.999766
     Sep    2.309383
     Oct    2.485616
     Nov    3.692511
     Dec    3.219070
     dtype: float64, Jan    1.531684
     Feb    1.346249
     Mar    1.077920
     Apr    0.979219
     May    1.226724
     Jun    1.299403
     Jul    0.969593
     Aug    1.122522
     Sep    1.462581
     Oct    1.050243
     Nov    0.998206
     Dec    1.297344
     dtype: float64)

We can use the `gamma.pdf` function in `scipy.stats.distributions` to plot the ditribtuions implied by the calculated alphas and betas. For example, here is January:

```
from scipy.stats.distributions import gamma

precip.Jan.hist(normed=True, bins=20)
plt.plot(np.linspace(0, 10), gamma.pdf(np.linspace(0, 10), alpha_mom[0], beta_mom[0]))
```

Looping over all months, we can create a grid of plots for the distribution of rainfall, using the gamma distribution:

```
axs = precip.hist(normed=True, figsize=(12, 8), sharex=True, sharey=True, bins=15, grid=False)

for ax in axs.ravel():

    # Get month
    m = ax.get_title()

    # Plot fitted distribution
    x = np.linspace(*ax.get_xlim())
    ax.plot(x, gamma.pdf(x, alpha_mom[m], beta_mom[m]))

    # Annotate with parameter estimates
    label = 'alpha = {0:.2f}\nbeta = {1:.2f}'.format(alpha_mom[m], beta_mom[m])
    ax.annotate(label, xy=(10, 0.2))

plt.tight_layout()
```

Maximum Likelihood
==================

**Maximum likelihood** (ML) fitting is usually more work than the method of moments, but it is preferred as the resulting estimator is known to have good theoretical properties.

There is a ton of theory regarding ML. We will restrict ourselves to the mechanics here.

Say we have some data $y = y_1,y_2,\ldots,y_n$ that is distributed according to some distribution:

$$Pr(Y_i=y_i | \theta)$$

Here, for example, is a **Poisson distribution** that describes the distribution of some discrete variables, typically *counts*:

```
y = np.random.poisson(5, size=100)
plt.hist(y, bins=12, normed=True)
plt.xlabel('y'); plt.ylabel('Pr(y)')
```

The product $\prod_{i=1}^n Pr(y_i | \theta)$ gives us a measure of how **likely** it is to observe values $y_1,\ldots,y_n$ given the parameters $\theta$. Maximum likelihood fitting consists of choosing the appropriate function $l= Pr(Y|\theta)$ to maximize for a given set of observations. We call this function the *likelihood function*, because it is a measure of how likely the observations are if the model is true.

> Given these data, how likely is this model?

In the above model, the data were drawn from a Poisson distribution with parameter $\lambda =5$.

$$L(y|\lambda=5) = \frac{e^{-5} 5^y}{y!}$$

So, for any given value of $y$, we can calculate its likelihood:

```
poisson_like = lambda x, lam: np.exp(-lam) * (lam**x) / (np.arange(x)+1).prod()

lam = 6
value = 10
poisson_like(value, lam)
```

    0.041303093412337726

```
np.sum(poisson_like(yi, lam) for yi in y)
```

    11.338402687045475

```
lam = 8
np.sum(poisson_like(yi, lam) for yi in y)
```

    7.6625623857972949

We can plot the likelihood function for any value of the parameter(s):

```
lambdas = np.linspace(0,15)
x = 5
plt.plot(lambdas, [poisson_like(x, l) for l in lambdas])
plt.xlabel('$\lambda$')
plt.ylabel('L($\lambda$|x={0})'.format(x))
```

How is the likelihood function different than the probability distribution function (PDF)? The likelihood is a function of the parameter(s) *given the data*, whereas the PDF returns the probability of data given a particular parameter value. Here is the PDF of the Poisson for $\lambda=5$.

```
lam = 5
xvals = np.arange(15)
plt.bar(xvals, [poisson_like(x, lam) for x in xvals])
plt.xlabel('x')
plt.ylabel('Pr(X|$\lambda$=5)')
```

Why are we interested in the likelihood function?

A reasonable estimate of the true, unknown value for the parameter is one which **maximizes the likelihood function**. So, inference is reduced to an optimization problem.

Going back to the rainfall data, if we are using a gamma distribution we need to maximize:

$$\begin{align}l(\alpha,\beta) &= \sum_{i=1}^n \log[\beta^{\alpha} x^{\alpha-1} e^{-x/\beta}\Gamma(\alpha)^{-1}] \cr
&= n[(\alpha-1)\overline{\log(x)} - \bar{x}\beta + \alpha\log(\beta) - \log\Gamma(\alpha)]\end{align}$$

(*Its usually easier to work in the log scale*)

where $n = 2012 − 1871 = 141$ and the bar indicates an average over all *i*. We choose $\alpha$ and $\beta$ to maximize $l(\alpha,\beta)$.

Notice $l$ is infinite if any $x$ is zero. We do not have any zeros, but we do have an NA value for one of the October data, which we dealt with above.

### Finding the MLE

To find the maximum of any function, we typically take the *derivative* with respect to the variable to be maximized, set it to zero and solve for that variable.

$$\frac{\partial l(\alpha,\beta)}{\partial \beta} = n\left(\frac{\alpha}{\beta} - \bar{x}\right) = 0$$

Which can be solved as $\beta = \alpha/\bar{x}$. However, plugging this into the derivative with respect to $\alpha$ yields:

$$\frac{\partial l(\alpha,\beta)}{\partial \alpha} = \log(\alpha) + \overline{\log(x)} - \log(\bar{x}) - \frac{\Gamma(\alpha)'}{\Gamma(\alpha)} = 0$$

This has no closed form solution. We must use ***numerical optimization***!

Numerical optimization alogarithms take an initial "guess" at the solution, and iteratively improve the guess until it gets "close enough" to the answer.

Here, we will use Newton-Raphson algorithm:

$$x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}$$

Which is available to us via SciPy:

```
from scipy.optimize import newton
```

Here is a graphical example of how Newtone-Raphson converges on a solution, using an arbitrary function:

```
# some function
func = lambda x: 3./(1 + 400*np.exp(-2*x)) - 1
xvals = np.linspace(0, 6)
plt.plot(xvals, func(xvals))
plt.text(5.3, 2.1, '$f(x)$', fontsize=16)
# zero line
plt.plot([0,6], [0,0], 'k-')
# value at step n
plt.plot([4,4], [0,func(4)], 'k:')
plt.text(4, -.2, '$x_n$', fontsize=16)
# tangent line
tanline = lambda x: -0.858 + 0.626*x
plt.plot(xvals, tanline(xvals), 'r--')
# point at step n+1
xprime = 0.858/0.626
plt.plot([xprime, xprime], [tanline(xprime), func(xprime)], 'k:')
plt.text(xprime+.1, -.2, '$x_{n+1}$', fontsize=16)
```

To apply the Newton-Raphson algorithm, we need a function that returns a vector containing the **first and second derivatives** of the function with respect to the variable of interest. In our case, this is:

```
from scipy.special import psi, polygamma

dlgamma = lambda m, log_mean, mean_log: np.log(m) - psi(m) - log_mean + mean_log
dl2gamma = lambda m, *args: 1./m - polygamma(1, m)
```

where `log_mean` and `mean_log` are $\log{\bar{x}}$ and $\overline{\log(x)}$, respectively. `psi` and `polygamma` are complex functions of the Gamma function that result when you take first and second derivatives of that function.

```
# Calculate statistics
log_mean = precip.mean().apply(np.log)
mean_log = precip.apply(np.log).mean()
```

Time to optimize!

```
# Alpha MLE for December
alpha_mle = newton(dlgamma, 2, dl2gamma, args=(log_mean[-1], mean_log[-1]))
alpha_mle
```

    3.5189679152399647

And now plug this back into the solution for beta:

$$ \beta  = \frac{\alpha}{\bar{X}} $$

```
beta_mle = alpha_mle/precip.mean()[-1]
beta_mle
```

    0.84261607548413797

We can compare the fit of the estimates derived from MLE to those from the method of moments:

```
dec = precip.Dec
dec.hist(normed=True, bins=10, grid=False)
x = np.linspace(0, dec.max())
plt.plot(x, gamma.pdf(x, alpha_mom[-1], beta_mom[-1]), 'm-')
plt.plot(x, gamma.pdf(x, alpha_mle, beta_mle), 'r--')
```

For some common distributions, SciPy includes methods for fitting via MLE:

```
from scipy.stats import gamma

gamma.fit(precip.Dec)
```

    (2.2427517753152308, 0.65494604470188622, 1.570073932063466)

This fit is not directly comparable to our estimates, however, because SciPy's `gamma.fit` method fits an odd 3-parameter version of the gamma distribution.

### Example: truncated distribution

Suppose that we observe $Y$ truncated below at $a$ (where $a$ is known). If $X$ is the distribution of our observation, then:

$$ P(X \le x) = P(Y \le x|Y \gt a) = \frac{P(a \lt Y \le x)}{P(Y \gt a)}$$

(so, $Y$ is the original variable and $X$ is the truncated variable)

Then X has the density:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

Suppose $Y \sim N(\mu, \sigma^2)$ and $x_1,\ldots,x_n$ are independent observations of $X$. We can use maximum likelihood to find $\mu$ and $\sigma$.

First, we can simulate a truncated distribution using a `while` statement to eliminate samples that are outside the support of the truncated distribution.

```
x = np.random.normal(size=10000)
a = -1
x_small = x < a
while x_small.sum():
    x[x_small] = np.random.normal(size=x_small.sum())
    x_small = x < a

_ = plt.hist(x, bins=100)
```

We can construct a log likelihood for this function using the conditional form:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

```
from scipy.stats.distributions import norm

trunc_norm = lambda theta, a, x: -(np.log(norm.pdf(x, theta[0], theta[1])) -
                                      np.log(1 - norm.cdf(a, theta[0], theta[1]))).sum()
```

For this example, we will use another optimization algorithm, the **Nelder-Mead simplex algorithm**. It has a couple of advantages:

- it does not require derivatives
- it can optimize (minimize) a vector of parameters

SciPy implements this algorithm in its `fmin` function:

```
from scipy.optimize import fmin

fmin(trunc_norm, np.array([1,2]), args=(-1, x))
```

    Optimization terminated successfully.
             Current function value: 11077.807912
             Iterations: 44
             Function evaluations: 82

    array([ 0.02244612,  0.99710875])

In general, simulating data is a terrific way of testing your model before using it with real data.

### Kernel density estimates

In some instances, we may not be interested in the parameters of a particular distribution of data, but just a smoothed representation of the data at hand. In this case, we can estimate the disribution *non-parametrically* (i.e. making no assumptions about the form of the underlying distribution) using kernel density estimation.

```
# Some random data
y = np.random.random(15) * 10
y
```

    array([ 0.62402604,  1.06204307,  7.27542973,  5.91978919,  9.95408103,
            9.51695422,  9.54608936,  4.3288413 ,  4.15275767,  2.47654605,
            3.74949496,  4.9779126 ,  0.68937206,  8.82221055,  2.87731832])

```
x = np.linspace(0, 10, 100)
# Smoothing parameter
s = 0.4
# Calculate the kernels
kernels = np.transpose([norm.pdf(x, yi, s) for yi in y])
plt.plot(x, kernels, 'k:')
plt.plot(x, kernels.sum(1))
plt.plot(y, np.zeros(len(y)), 'ro', ms=10)
```

SciPy implements a Gaussian KDE that automatically chooses an appropriate bandwidth. Let's create a bi-modal distribution of data that is not easily summarized by a parametric distribution:

```
# Create a bi-modal distribution with a mixture of Normals.
x1 = np.random.normal(0, 3, 50)
x2 = np.random.normal(4, 1, 50)

# Append by row
x = np.r_[x1, x2]
```

```
plt.hist(x, bins=8, normed=True)
```

```
from scipy.stats import kde

density = kde.gaussian_kde(x)
xgrid = np.linspace(x.min(), x.max(), 100)
plt.hist(x, bins=8, normed=True)
plt.plot(xgrid, density(xgrid), 'r-')
```

### Exercise: Cervical dystonia analysis

Recall the cervical dystonia database, which is a clinical trial of botulinum toxin type B (BotB) for patients with cervical dystonia from nine U.S. sites. The response variable is measurements on the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS), measuring severity, pain, and disability of cervical dystonia (high scores mean more impairment). One way to check the efficacy of the treatment is to compare the distribution of TWSTRS for control and treatment patients at the end of the study.

Use the method of moments or MLE to calculate the mean and variance of TWSTRS at week 16 for one of the treatments and the control group. Assume that the distribution of the `twstrs` variable is normal:

$$f(x \mid \mu, \sigma^2) = \sqrt{\frac{1}{2\pi\sigma^2}} \exp\left\{ -\frac{1}{2} \frac{(x-\mu)^2}{\sigma^2} \right\}$$

```
cdystonia = pd.read_csv("data/cdystonia.csv")
cdystonia[cdystonia.obs==6].hist(column='twstrs', by=cdystonia.treat, bins=8)
```

```
# Write your answer here
```

## Regression models

A general, primary goal of many statistical data analysis tasks is to relate the influence of one variable on another. For example, we may wish to know how different medical interventions influence the incidence or duration of disease, or perhaps a how baseball player's performance varies as a function of age.

```
x = np.array([2.2, 4.3, 5.1, 5.8, 6.4, 8.0])
y = np.array([0.4, 10.1, 14.0, 10.9, 15.4, 18.5])
plt.plot(x,y,'ro')
```

We can build a model to characterize the relationship between $X$ and $Y$, recognizing that additional factors other than $X$ (the ones we have measured or are interested in) may influence the response variable $Y$.

$y_i = f(x_i) + \epsilon_i$

where $f$ is some function, for example a linear function:

$y_i = \beta_0 + \beta_1 x_i + \epsilon_i$

and $\epsilon_i$ accounts for the difference between the observed response $y_i$ and its prediction from the model $\hat{y_i} = \beta_0 + \beta_1 x_i$. This is sometimes referred to as **process uncertainty**.

We would like to select $\beta_0, \beta_1$ so that the difference between the predictions and the observations is zero, but this is not usually possible. Instead, we choose a reasonable criterion: ***the smallest sum of the squared differences between $\hat{y}$ and $y$***.

$$R^2 = \sum_i (y_i - [\beta_0 + \beta_1 x_i])^2 = \sum_i \epsilon_i^2 $$

Squaring serves two purposes: (1) to prevent positive and negative values from cancelling each other out and (2) to strongly penalize large deviations. Whether the latter is a good thing or not depends on the goals of the analysis.

In other words, we will select the parameters that minimize the squared error of the model.

```
ss = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x) ** 2)
```

```
ss([0,1],x,y)
```

    333.35000000000002

```
b0,b1 = fmin(ss, [0,1], args=(x,y))
b0,b1
```

    Optimization terminated successfully.
             Current function value: 21.375000
             Iterations: 79
             Function evaluations: 153

    (-4.3500136038870876, 3.0000002915386412)

```
plt.plot(x, y, 'ro')
plt.plot([0,10], [b0, b0+b1*10])
```

```
plt.plot(x, y, 'ro')
plt.plot([0,10], [b0, b0+b1*10])
for xi, yi in zip(x,y):
    plt.plot([xi]*2, [yi, b0+b1*xi], 'k:')
plt.xlim(2, 9); plt.ylim(0, 20)
```

Minimizing the sum of squares is not the only criterion we can use; it is just a very popular (and successful) one. For example, we can try to minimize the sum of absolute differences:

```
sabs = lambda theta, x, y: np.sum(np.abs(y - theta[0] - theta[1]*x))
b0,b1 = fmin(sabs, [0,1], args=(x,y))
print b0,b1
plt.plot(x, y, 'ro')
plt.plot([0,10], [b0, b0+b1*10])
```

We are not restricted to a straight-line regression model; we can represent a curved relationship between our variables by introducing **polynomial** terms. For example, a cubic model:

$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \epsilon_i$

```
ss2 = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x - theta[2]*(x**2)) ** 2)
b0,b1,b2 = fmin(ss2, [1,1,-1], args=(x,y))
print b0,b1,b2
plt.plot(x, y, 'ro')
xvals = np.linspace(0, 10, 100)
plt.plot(xvals, b0 + b1*xvals + b2*(xvals**2))
```

Although polynomial model characterizes a nonlinear relationship, it is a linear problem in terms of estimation. That is, the regression model $f(y | x)$ is linear in the parameters.

For some data, it may be reasonable to consider polynomials of order>2. For example, consider the relationship between the number of home runs a baseball player hits and the number of runs batted in (RBI) they accumulate; clearly, the relationship is positive, but we may not expect a linear relationship.

```
ss3 = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x - theta[2]*(x**2)
                                  - theta[3]*(x**3)) ** 2)

bb = pd.read_csv("data/baseball.csv", index_col=0)
plt.plot(bb.hr, bb.rbi, 'r.')
b0,b1,b2,b3 = fmin(ss3, [0,1,-1,0], args=(bb.hr, bb.rbi))
xvals = np.arange(40)
plt.plot(xvals, b0 + b1*xvals + b2*(xvals**2) + b3*(xvals**3))
```

Of course, we need not fit least squares models by hand. The `statsmodels` package implements least squares models that allow for model fitting in a single line:

```
import statsmodels.api as sm

straight_line = sm.OLS(y, sm.add_constant(x)).fit()
straight_line.summary()
```

    /usr/local/lib/python2.7/site-packages/statsmodels/stats/stattools.py:72: UserWarning: omni_normtest is not valid with less than 8 observations; 6 samples were given.
      "samples were given." % int(n))

OLS Regression Results

  Dep. Variable:	            y	          R-squared:         	    0.891

  Model:	                   OLS	         Adj. R-squared:    	    0.864

  Method:	             Least Squares	    F-statistic:       	    32.67

  Date:	             Wed, 18 Jun 2014	   Prob (F-statistic):	  0.00463

  Time:	                 20:56:03	       Log-Likelihood:    	  -12.325

  No. Observations:	           6	        AIC:               	    28.65

  Df Residuals:	               4	        BIC:               	    28.23

  Df Model:	                   1

  Covariance Type:	      nonrobust

    	       coef	     std err	      t	      P>|t|	 [95.0% Conf. Int.]

  const	    -4.3500	     2.937	    -1.481	  0.213	   -12.505     3.805

  x1	        3.0000	     0.525	     5.716	  0.005	     1.543     4.457

  Omnibus:	          nan	   Durbin-Watson:     	    2.387

  Prob(Omnibus):	    nan	   Jarque-Bera (JB):  	    0.570

  Skew:	           0.359	   Prob(JB):          	    0.752

  Kurtosis:	       1.671	   Cond. No.          	     17.9

```
from statsmodels.formula.api import ols as OLS

data = pd.DataFrame(dict(x=x, y=y))
cubic_fit = OLS('y ~ x + I(x**2)', data).fit()

cubic_fit.summary()
```

OLS Regression Results

  Dep. Variable:	            y	          R-squared:         	    0.929

  Model:	                   OLS	         Adj. R-squared:    	    0.881

  Method:	             Least Squares	    F-statistic:       	    19.50

  Date:	             Wed, 18 Jun 2014	   Prob (F-statistic):	  0.0191

  Time:	                 20:56:03	       Log-Likelihood:    	  -11.056

  No. Observations:	           6	        AIC:               	    28.11

  Df Residuals:	               3	        BIC:               	    27.49

  Df Model:	                   2

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	 [95.0% Conf. Int.]

  Intercept	   -11.0748	     6.013	    -1.842	  0.163	   -30.211     8.062

  x	             6.0577	     2.482	     2.441	  0.092	    -1.840    13.955

  I(x ** 2)	    -0.3027	     0.241	    -1.257	  0.298	    -1.069     0.464

  Omnibus:	          nan	   Durbin-Watson:     	    2.711

  Prob(Omnibus):	    nan	   Jarque-Bera (JB):  	    0.655

  Skew:	          -0.809	   Prob(JB):          	    0.721

  Kurtosis:	       2.961	   Cond. No.          	     270.

### Exercise: Polynomial function

Write a function that specified a polynomial of arbitrary degree.

```
# Write your answer here
```

## Model Selection

How do we choose among competing models for a given dataset? More parameters are not necessarily better, from the standpoint of model fit. For example, fitting a 9-th order polynomial to the sample data from the above example certainly results in an overfit.

```
def calc_poly(params, data):
        x = np.c_[[data**i for i in range(len(params))]]
        return np.dot(params, x)

ssp = lambda theta, x, y: np.sum((y - calc_poly(theta, x)) ** 2)
betas = fmin(ssp, np.zeros(10), args=(x,y), maxiter=1e6)
plt.plot(x, y, 'ro')
xvals = np.linspace(0, max(x), 100)
plt.plot(xvals, calc_poly(betas, xvals))
```

One approach is to use an information-theoretic criterion to select the most appropriate model. For example **Akaike's Information Criterion (AIC)** balances the fit of the model (in terms of the likelihood) with the number of parameters required to achieve that fit. We can easily calculate AIC as:

$$AIC = n \log(\hat{\sigma}^2) + 2p$$

where $p$ is the number of parameters in the model and $\hat{\sigma}^2 = RSS/(n-p-1)$.

Notice that as the number of parameters increase, the residual sum of squares goes down, but the second term (a penalty) increases.

To apply AIC to model selection, we choose the model that has the **lowest** AIC value.

```
n = len(x)

aic = lambda rss, p, n: n * np.log(rss/(n-p-1)) + 2*p

RSS1 = ss(fmin(ss, [0,1], args=(x,y)), x, y)
RSS2 = ss2(fmin(ss2, [1,1,-1], args=(x,y)), x, y)

print aic(RSS1, 2, n), aic(RSS2, 3, n)
```

    Optimization terminated successfully.
             Current function value: 21.375000
             Iterations: 79
             Function evaluations: 153
    Optimization terminated successfully.
             Current function value: 14.001110
             Iterations: 198
             Function evaluations: 372
    15.7816583572 17.6759368019

Hence, we would select the 2-parameter (linear) model.

## Logistic Regression

Fitting a line to the relationship between two variables using the least squares approach is sensible when the variable we are trying to predict is continuous, but what about when the data are dichotomous?

- male/female
- pass/fail
- died/survived

Let's consider the problem of predicting survival in the Titanic disaster, based on our available information. For example, lets say that we want to predict survival as a function of the fare paid for the journey.

```
titanic = pd.read_excel("data/titanic.xls", "titanic")
titanic.name
```

    0                      Allen, Miss. Elisabeth Walton
    1                     Allison, Master. Hudson Trevor
    2                       Allison, Miss. Helen Loraine
    3               Allison, Mr. Hudson Joshua Creighton
    4    Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
    5                                Anderson, Mr. Harry
    6                  Andrews, Miss. Kornelia Theodosia
    7                             Andrews, Mr. Thomas Jr
    8      Appleton, Mrs. Edward Dale (Charlotte Lamson)
    9                            Artagaveytia, Mr. Ramon
    ...
    1298                  Wittevrongel, Mr. Camille
    1299                        Yasbeck, Mr. Antoni
    1300    Yasbeck, Mrs. Antoni (Selini Alexander)
    1301                       Youseff, Mr. Gerious
    1302                          Yousif, Mr. Wazli
    1303                      Yousseff, Mr. Gerious
    1304                       Zabour, Miss. Hileni
    1305                      Zabour, Miss. Thamine
    1306                  Zakarian, Mr. Mapriededer
    1307                        Zakarian, Mr. Ortin
    1308                         Zimmerman, Mr. Leo
    Name: name, Length: 1309, dtype: object

```
jitter = np.random.normal(scale=0.02, size=len(titanic))
plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
plt.yticks([0,1])
plt.ylabel("survived")
plt.xlabel("log(fare)")
```

I have added random jitter on the y-axis to help visualize the density of the points, and have plotted fare on the log scale.

Clearly, fitting a line through this data makes little sense, for several reasons. First, for most values of the predictor variable, the line would predict values that are not zero or one. Second, it would seem odd to choose least squares (or similar) as a criterion for selecting the best line.

```
x = np.log(titanic.fare[titanic.fare>0])
y = titanic.survived[titanic.fare>0]
betas_titanic = fmin(ss, [1,1], args=(x,y))
```

    Optimization terminated successfully.
             Current function value: 277.621917
             Iterations: 55
             Function evaluations: 103

```
jitter = np.random.normal(scale=0.02, size=len(titanic))
plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
plt.yticks([0,1])
plt.ylabel("survived")
plt.xlabel("log(fare)")
plt.plot([0,7], [betas_titanic[0], betas_titanic[0] + betas_titanic[1]*7.])
```

If we look at this data, we can see that for most values of `fare`, there are some individuals that survived and some that did not. However, notice that the cloud of points is denser on the "survived" (y=1) side for larger values of fare than on the "died" (y=0) side.

### Stochastic model

Rather than model the binary outcome explicitly, it makes sense instead to model the *probability* of death or survival in a **stochastic** model. Probabilities are measured on a continuous [0,1] scale, which may be more amenable for prediction using a regression line. We need to consider a different probability model for this exerciese however; let's consider the **Bernoulli** distribution as a generative model for our data:

$$f(y|p) = p^{y} (1-p)^{1-y}$$

where $y = \{0,1\}$ and $p \in [0,1]$. So, this model predicts whether $y$ is zero or one as a function of the probability $p$. Notice that when $y=1$, the $1-p$ term disappears, and when $y=0$, the $p$ term disappears.

So, the model we want to fit should look something like this:

$$p_i = \beta_0 + \beta_1 x_i + \epsilon_i$$

However, since $p$ is constrained to be between zero and one, it is easy to see where a linear (or polynomial) model might predict values outside of this range. We can modify this model sligtly by using a **link function** to transform the probability to have an unbounded range on a new scale. Specifically, we can use a **logit transformation** as our link function:

$$\text{logit}(p) = \log\left[\frac{p}{1-p}\right] = x$$

Here's a plot of $p/(1-p)$

```
logit = lambda p: np.log(p/(1.-p))
unit_interval = np.linspace(0,1)
plt.plot(unit_interval/(1-unit_interval), unit_interval)
```

And here's the logit function:

```
plt.plot(logit(unit_interval), unit_interval)
```

The inverse of the logit transformation is:

$$p = \frac{1}{1 + \exp(-x)}$$

So, now our model is:

$$\text{logit}(p_i) = \beta_0 + \beta_1 x_i + \epsilon_i$$

We can fit this model using maximum likelihood. Our likelihood, again based on the Bernoulli model is:

$$L(y|p) = \prod_{i=1}^n p_i^{y_i} (1-p_i)^{1-y_i}$$

which, on the log scale is:

$$l(y|p) = \sum_{i=1}^n y_i \log(p_i) + (1-y_i)\log(1-p_i)$$

We can easily implement this in Python, keeping in mind that `fmin` minimizes, rather than maximizes functions:

```
invlogit = lambda x: 1. / (1 + np.exp(-x))

def logistic_like(theta, x, y):
    p = invlogit(theta[0] + theta[1] * x)
    # Return negative of log-likelihood
    return -np.sum(y * np.log(p) + (1-y) * np.log(1 - p))
```

Remove null values from variables

```
x, y = titanic[titanic.fare.notnull()][['fare', 'survived']].values.T
```

... and fit the model.

```
b0,b1 = fmin(logistic_like, [0.5,0], args=(x,y))
b0, b1
```

    Optimization terminated successfully.
             Current function value: 827.015955
             Iterations: 47
             Function evaluations: 93

    (-0.88238984528338194, 0.012452067664164127)

```
jitter = np.random.normal(scale=0.01, size=len(x))
plt.plot(x, y+jitter, 'r.', alpha=0.3)
plt.yticks([0,.25,.5,.75,1])
xvals = np.linspace(0, 600)
plt.plot(xvals, invlogit(b0+b1*xvals))
```

As with our least squares model, we can easily fit logistic regression models in `statsmodels`, in this case using the `GLM` (generalized linear model) class with a binomial error distribution specified.

```
logistic = sm.GLM(y, sm.add_constant(x), family=sm.families.Binomial()).fit()
logistic.summary()
```

Generalized Linear Model Regression Results

  Dep. Variable:	          y	          No. Observations:  	    1308

  Model:	                 GLM	         Df Residuals:      	    1306

  Model Family:	       Binomial	       Df Model:          	       1

  Link Function:	        logit	        Scale:             	    1.0

  Method:	               IRLS	         Log-Likelihood:    	  -827.02

  Date:	           Wed, 18 Jun 2014	   Deviance:          	   1654.0

  Time:	               20:56:06	       Pearson chi2:      	 1.33e+03

  No. Iterations:	         6

    	       coef	     std err	      t	      P>|t|	 [95.0% Conf. Int.]

  const	    -0.8824	     0.076	   -11.684	  0.000	    -1.030    -0.734

  x1	        0.0125	     0.002	     7.762	  0.000	     0.009     0.016

### Exercise: multivariate logistic regression

Which other variables might be relevant for predicting the probability of surviving the Titanic? Generalize the model likelihood to include 2 or 3 other covariates from the dataset.

```
# Write your answer here
```

## Bootstrapping

Parametric inference can be **non-robust**:

* inaccurate if parametric assumptions are violated
* if we rely on asymptotic results, we may not achieve an acceptable level of accuracy

Parmetric inference can be **difficult**:

* derivation of sampling distribution may not be possible

An alternative is to estimate the sampling distribution of a statistic *empirically* without making assumptions about the form of the population.

We have seen this already with the kernel density estimate.

### Non-parametric Bootstrap

The bootstrap is a resampling method discovered by [Brad Efron](http://www.jstor.org/discover/10.2307/2958830?uid=3739568&uid=2&uid=4&uid=3739256&sid=21102342537691) that allows one to approximate the true sampling distribution of a dataset, and thereby obtain estimates of the mean and variance of the distribution.

Bootstrap sample:

$$S_1^* = \{x_{11}^*, x_{12}^*, \ldots, x_{1n}^*\}$$

$S_i^*$ is a sample of size $n$, **with** replacement.

In Python, we have already seen the NumPy function `permutation` that can be used in conjunction with Pandas' `take` method to generate a random sample of some data without replacement:

```
np.random.permutation(titanic.name)[:5]
```

    array([u'Meek, Mrs. Thomas (Annie Louise Rowley)',
           u'Thorneycroft, Mr. Percival', u'Williams, Mr. Leslie',
           u'Graham, Mr. George Edward', u'Petroff, Mr. Nedelio'], dtype=object)

Similarly, we can use the `random.randint` method to generate a sample *with* replacement, which we can use when bootstrapping.

```
random_ind = np.random.randint(0, len(titanic), 5)
titanic.name[random_ind]
```

    41      Brown, Mrs. James Joseph (Margaret Tobin)
    1061              Nilsson, Miss. Helmina Josefina
    937                  Klasen, Miss. Gertrud Emilia
    426                            Hale, Mr. Reginald
    831                Goodwin, Mr. Charles Frederick
    Name: name, dtype: object

We regard S as an "estimate" of population P

> population : sample :: sample : bootstrap sample

The idea is to generate replicate bootstrap samples:

$$S^* = \{S_1^*, S_2^*, \ldots, S_R^*\}$$

Compute statistic $t$ (estimate) for each bootstrap sample:

$$T_i^* = t(S^*)$$

```
n = 10
R = 1000
# Original sample (n=10)
x = np.random.normal(size=n)
# 1000 bootstrap samples of size 10
s = [x[np.random.randint(0,n,n)].mean() for i in range(R)]
_ = plt.hist(s, bins=30)
```

### Bootstrap Estimates

From our bootstrapped samples, we can extract *estimates* of the expectation and its variance:

$$\bar{T}^* = \hat{E}(T^*) = \frac{\sum_i T_i^*}{R}$$

$$\hat{\text{Var}}(T^*) = \frac{\sum_i (T_i^* - \bar{T}^*)^2}{R-1}$$

```
boot_mean = np.sum(s)/R
boot_mean
```

    0.087385394806476724

```
boot_var = ((np.array(s) - boot_mean) ** 2).sum() / (R-1)
boot_var
```

    0.10590407752057245

Since we have estimated the expectation of the bootstrapped statistics, we can estimate the **bias** of T:

$$\hat{B}^* = \bar{T}^* - T$$

```
boot_mean - np.mean(x)
```

    -0.00528084680355842

### Bootstrap error

There are two sources of error in bootstrap estimates:

1. **Sampling error** from the selection of $S$.
2. **Bootstrap error** from failing to enumerate all possible bootstrap samples.

For the sake of accuracy, it is prudent to choose at least R=1000

### Bootstrap Percentile Intervals

An attractive feature of bootstrap statistics is the ease with which you can obtain an estimate of *uncertainty* for a given statistic. We simply use the empirical quantiles of the bootstrapped statistics to obtain percentiles corresponding to a confidence interval of interest.

This employs the *ordered* bootstrap replicates:

$$T_{(1)}^*, T_{(2)}^*, \ldots, T_{(R)}^*$$

Simply extract the $100(\alpha/2)$ and $100(1-\alpha/2)$ percentiles:

$$T_{[(R+1)\alpha/2]}^* \lt \theta \lt T_{[(R+1)(1-\alpha/2)]}^*$$

```
s_sorted = np.sort(s)
s_sorted[:10]
```

    array([-0.82890714, -0.77634577, -0.76588512, -0.76230089, -0.75578488,
           -0.73850118, -0.72869116, -0.72862786, -0.72840095, -0.71831374])

```
s_sorted[-10:]
```

    array([ 0.81823418,  0.86179331,  0.92314175,  0.93496722,  0.9358216 ,
            1.02058937,  1.03085586,  1.03121927,  1.22699691,  1.3599996 ])

```
alpha = 0.05
s_sorted[[(R+1)*alpha/2, (R+1)*(1-alpha/2)]]
```

    array([-0.5684053 ,  0.68682205])

### Exercise: Cervical dystonia bootstrap estimates

Use bootstrapping to estimate the mean of one of the treatment groups, and calculate percentile intervals for the mean.

```
# Write your answer here
```

statistics/4. Statistical Data Modeling.ipynb


## Fast Bayesian estimation of SARIMAX models

### Introduction

This notebook will show how to use fast Bayesian methods to estimate SARIMAX (Seasonal AutoRegressive Integrated Moving Average with eXogenous regressors) models. These methods can also be parallelised across multiple cores.

Here, fast methods means a version of Hamiltonian Monte Carlo called the No-U-Turn Sampler (NUTS) developed by Hoffmann and Gelman: see [Hoffman, M. D., & Gelman, A. (2014). The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593-1623.](https://arxiv.org/abs/1111.4246). As they say, "the cost of HMC per independent sample from a target distribution of dimension $D$ is roughly $\mathcal{O}(D^{5/4})$, which stands in sharp contrast with the $\mathcal{O}(D^{2})$ cost of random-walk Metropolis". So for problems of larger dimension, the time-saving with HMC is significant. However it does require the gradient, or Jacobian, of the model to be provided.

This notebook will combine the Python libraries [statsmodels](https://www.statsmodels.org/stable/index.html), which does econometrics, and [PyMC3](https://docs.pymc.io/), which is for Bayesian estimation, to perform fast Bayesian estimation of a simple SARIMAX model, in this case an ARMA(1, 1) model for US CPI.

Note that, for simple models like AR(p), base PyMC3 is a quicker way to fit a model; there's an [example here](https://docs.pymc.io/notebooks/AR.html). The advantage of using statsmodels is that it gives access to methods that can solve a vast range of statespace models.

The model we'll solve is given by
$$
y_t = \phi y_{t-1} + \varepsilon_t + \theta_1 \varepsilon_{t-1}, \qquad \varepsilon_t \sim N(0, \sigma^2)
$$
with 1 auto-regressive term and 1 moving average term. In statespace form it is written as:
$$
\begin{align}
y_t & = \underbrace{\begin{bmatrix} 1 & \theta_1 \end{bmatrix}}_{Z} \underbrace{\begin{bmatrix} \alpha_{1,t} \\ \alpha_{2,t} \end{bmatrix}}_{\alpha_t} \\
    \begin{bmatrix} \alpha_{1,t+1} \\ \alpha_{2,t+1} \end{bmatrix} & = \underbrace{\begin{bmatrix}
        \phi & 0 \\
        1      & 0     \\
    \end{bmatrix}}_{T} \begin{bmatrix} \alpha_{1,t} \\ \alpha_{2,t} \end{bmatrix} +
    \underbrace{\begin{bmatrix} 1 \\ 0 \end{bmatrix}}_{R} \underbrace{\varepsilon_{t+1}}_{\eta_t} \\
\end{align}
$$

The code will follow these steps:
1. Import external dependencies
2. Download and plot the data on US CPI
3. Simple maximum likelihood estimation (MLE) as an example
4. Definitions of helper functions to provide tensors to the library doing Bayesian estimation
5. Bayesian estimation via NUTS
6. Application to US CPI series

Finally, Appendix A shows how to re-use the helper functions from step (4) to estimate a different state space model, `UnobservedComponents`, using the same Bayesian methods.

#### 1. Import external dependencies

```python
%matplotlib inline
import theano
import theano.tensor as tt
import pymc3 as pm
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import pandas as pd
from pandas_datareader.data import DataReader
from pandas.plotting import register_matplotlib_converters
plt.style.use('seaborn')
register_matplotlib_converters()
```

#### 2. Download and plot the data on US CPI

We'll get the data from FRED:

```python
cpi = DataReader('CPIAUCNS', 'fred', start='1971-01', end='2018-12')
cpi.index = pd.DatetimeIndex(cpi.index, freq='MS')

# Define the inflation series that we'll use in analysis
inf = np.log(cpi).resample('QS').mean().diff()[1:] * 400
print(inf.head())
```

```python
# Plot the series
fig, ax = plt.subplots(figsize=(9, 4), dpi=300)
ax.plot(inf.index, inf, label=r'$\Delta \log CPI$', lw=2)
ax.legend(loc='lower left')
plt.show()
```

#### 3. Fit the model with maximum likelihood

Statsmodels does all of the hardwork of this for us - creating and fitting the model takes just two lines of code. The model order parameters correspond to auto-regressive, difference, and moving average orders respectively.

```python
# Create an SARIMAX model instance - here we use it to estimate
# the parameters via MLE using the `fit` method, but we can
# also re-use it below for the Bayesian estimation
mod = sm.tsa.statespace.SARIMAX(inf, order=(1, 0, 1))

res_mle = mod.fit(disp=False)
print(res_mle.summary())
```

It's a good fit. We can also get the series of one-step ahead predictions and plot it next to the actual data, along with a confidence band.

```python
predict_mle = res_mle.get_prediction()
predict_mle_ci = predict_mle.conf_int()
lower = predict_mle_ci['lower CPIAUCNS']
upper = predict_mle_ci['upper CPIAUCNS']

# Graph
fig, ax = plt.subplots(figsize=(9,4), dpi=300)

# Plot data points
inf.plot(ax=ax, style='-', label='Observed')

# Plot predictions
predict_mle.predicted_mean.plot(ax=ax, style='r.', label='One-step-ahead forecast')
ax.fill_between(predict_mle_ci.index, lower, upper, color='r', alpha=0.1)
ax.legend(loc='lower left')
plt.show()
```

#### 4. Helper functions to provide tensors to the library doing Bayesian estimation

We're almost on to the magic but there are a few preliminaries. Feel free to skip this section if you're not interested in the technical details.

---------

##### Technical section

PyMC3 is a Bayesian estimation library ("Probabilistic Programming in Python: Bayesian Modeling and Probabilistic Machine Learning with Theano") that is a) fast and b) optimised for Bayesian machine learning, for instance [Bayesian neural networks](https://docs.pymc.io/notebooks/bayesian_neural_network_advi.html). To do all of this, it is built on top of a Theano, a library that aims to evaluate tensors very efficiently and provide symbolic differentiation (necessary for any kind of deep learning). It is the symbolic differentiation that means PyMC3 can use NUTS on any problem formulated within PyMC3.

We are not formulating a problem directly in PyMC3; we're using statsmodels to specify the statespace model and solve it with the Kalman filter. So we need to put the plumbing of statsmodels and PyMC3 together, which means wrapping the statsmodels SARIMAX model object in a Theano-flavoured wrapper before passing information to PyMC3 for estimation.

Because of this, we can't use the Theano auto-differentiation directly. Happily, statsmodels SARIMAX objects have a method to return the Jacobian evaluated at the parameter values. We'll be making use of this to provide gradients so that we can use NUTS.

###### Defining helper functions to translate models into a PyMC3 friendly form

First, we'll create the Theano wrappers. They will be in the form of 'Ops', operation objects, that 'perform' particular tasks. They are initialised with a statsmodels `model` instance.

Although this code may look somewhat opaque, it is generic for any state space model in statsmodels.

```python
class Loglike(tt.Op):

    itypes = [tt.dvector] # expects a vector of parameter values when called
    otypes = [tt.dscalar] # outputs a single scalar value (the log likelihood)

    def __init__(self, model):
        self.model = model
        self.score = Score(self.model)

    def perform(self, node, inputs, outputs):
        theta, = inputs  # contains the vector of parameters
        llf = self.model.loglike(theta)
        outputs[0][0] = np.array(llf) # output the log-likelihood

    def grad(self, inputs, g):
        # the method that calculates the gradients - it actually returns the
        # vector-Jacobian product - g[0] is a vector of parameter values
        theta, = inputs  # our parameters
        out = [g[0] * self.score(theta)]
        return out

class Score(tt.Op):
    itypes = [tt.dvector]
    otypes = [tt.dvector]

    def __init__(self, model):
        self.model = model

    def perform(self, node, inputs, outputs):
        theta, = inputs
        outputs[0][0] = self.model.score(theta)
```

##### End of technical section

---------

#### 5. Bayesian estimation with NUTS

The next step is to set the parameters for the Bayesian estimation, specify our priors, and run it.

```python
# Set sampling params
ndraws = 3000  # number of draws from the distribution
nburn = 600   # number of "burn-in points" (which will be discarded)
```

Now for the fun part! There are three parameters to estimate: $\phi$, $\theta_1$, and $\sigma$. We'll use uninformative uniform priors for the first two, and an inverse gamma for the last one. Then we'll run the inference optionally using as many computer cores as I have.

```python
# Construct an instance of the Theano wrapper defined above, which
# will allow PyMC3 to compute the likelihood and Jacobian in a way
# that it can make use of. Here we are using the same model instance
# created earlier for MLE analysis (we could also create a new model
# instance if we preferred)
loglike = Loglike(mod)

with pm.Model():
    # Priors
    arL1 = pm.Uniform('ar.L1', -0.99, 0.99)
    maL1 = pm.Uniform('ma.L1', -0.99, 0.99)
    sigma2 = pm.InverseGamma('sigma2', 2, 4)

    # convert variables to tensor vectors
    theta = tt.as_tensor_variable([arL1, maL1, sigma2])

    # use a DensityDist (use a lamdba function to "call" the Op)
    pm.DensityDist('likelihood', lambda v: loglike(v), observed={'v': theta})

    # Draw samples
    trace = pm.sample(ndraws, tune=nburn, discard_tuned_samples=True, cores=4)
```

Note that the NUTS sampler is auto-assigned because we provided gradients. PyMC3 will use Metropolis or Slicing samplers if it doesn't find that gradients are available. There are an impressive number of draws per second for a "block box" style computation! However, note that if the model can be represented directly by PyMC3 (like the AR(p) models mentioned above), then computation can be substantially faster.

Inference is complete, but are the results any good? There are a number of ways to check. The first is to look at the posterior distributions (with lines showing the MLE values):

```python
plt.tight_layout()
# Note: the syntax here for the lines argument is required for
# PyMC3 versions >= 3.7
# For version <= 3.6 you can use lines=dict(res_mle.params) instead
_ = pm.traceplot(trace,
                 lines=[(k, {}, [v]) for k, v in dict(res_mle.params).items()],
                 combined=True,
                 figsize=(12, 12))
```

The estimated posteriors clearly peak close to the parameters found by MLE. We can also see a summary of the estimated values:

```python
pm.summary(trace)
```

Here Rhat, or $\hat{R}$, is the Gelman-Rubin statistic. It tests for lack of convergence by comparing the variance between multiple chains to the variance within each chain. If convergence has been achieved, the between-chain and within-chain variances should be identical. If $\hat{R}<1.2$ for all model parameters, we can have some confidence that convergence has been reached.

Additionally, the highest posterior density interval (the gap between the two values of HPD in the table) is small for each of the variables.

#### 6. Application of Bayesian estimates of parameters

We'll now re-instigate a version of the model but using the parameters from the Bayesian estimation, and again plot the one-step-ahead forecasts.

```python
# Retrieve the posterior means
params = pm.summary(trace)['mean'].values

# Construct results using these posterior means as parameter values
res_bayes = mod.smooth(params)

predict_bayes = res_bayes.get_prediction()
predict_bayes_ci = predict_bayes.conf_int()
lower = predict_bayes_ci['lower CPIAUCNS']
upper = predict_bayes_ci['upper CPIAUCNS']

# Graph
fig, ax = plt.subplots(figsize=(9,4), dpi=300)

# Plot data points
inf.plot(ax=ax, style='-', label='Observed')

# Plot predictions
predict_bayes.predicted_mean.plot(ax=ax, style='r.', label='One-step-ahead forecast')
ax.fill_between(predict_bayes_ci.index, lower, upper, color='r', alpha=0.1)
ax.legend(loc='lower left')
plt.show()
```

### Appendix A. Application to `UnobservedComponents` models

We can reuse the `Loglike` and `Score` wrappers defined above to consider a different state space model. For example, we might want to model inflation as the combination of a random walk trend and autoregressive error term:

$$
\begin{aligned}
y_t & = \mu_t + \varepsilon_t \\
\mu_t & = \mu_{t-1} + \eta_t \\
\varepsilon_t &= \phi \varepsilon_t + \zeta_t
\end{aligned}
$$

This model can be constructed in Statsmodels with the `UnobservedComponents` class using the `rwalk` and `autoregressive` specifications. As before, we can fit the model using maximum likelihood via the `fit` method.

```python
# Construct the model instance
mod_uc = sm.tsa.UnobservedComponents(inf, 'rwalk', autoregressive=1)

# Fit the model via maximum likelihood
res_uc_mle = mod_uc.fit()
print(res_uc_mle.summary())
```

As noted earlier, the Theano wrappers (`Loglike` and `Score`) that we created above are generic, so we can re-use essentially the same code to explore the model with Bayesian methods.

```python
# Set sampling params
ndraws = 3000  # number of draws from the distribution
nburn = 600   # number of "burn-in points" (which will be discarded)
```

```python
# Here we follow the same procedure as above, but now we instantiate the
# Theano wrapper `Loglike` with the UC model instance instead of the
# SARIMAX model instance
loglike_uc = Loglike(mod_uc)

with pm.Model():
    # Priors
    sigma2level = pm.InverseGamma('sigma2.level', 1, 1)
    sigma2ar = pm.InverseGamma('sigma2.ar', 1, 1)
    arL1 = pm.Uniform('ar.L1', -0.99, 0.99)

    # convert variables to tensor vectors
    theta_uc = tt.as_tensor_variable([sigma2level, sigma2ar, arL1])

    # use a DensityDist (use a lamdba function to "call" the Op)
    pm.DensityDist('likelihood', lambda v: loglike_uc(v), observed={'v': theta_uc})

    # Draw samples
    trace_uc = pm.sample(ndraws, tune=nburn, discard_tuned_samples=True, cores=4)
```

And as before we can plot the marginal posteriors. In contrast to the SARIMAX example, here the posterior modes are somewhat different from the MLE estimates.

```python
plt.tight_layout()
# Note: the syntax here for the lines argument is required for
# PyMC3 versions >= 3.7
# For version <= 3.6 you can use lines=dict(res_mle.params) instead
_ = pm.traceplot(trace_uc,
                 lines=[(k, {}, [v]) for k, v in dict(res_uc_mle.params).items()],
                 combined=True,
                 figsize=(12, 12))
```

```python
pm.summary(trace_uc)
```

```python
# Retrieve the posterior means
params = pm.summary(trace_uc)['mean'].values

# Construct results using these posterior means as parameter values
res_uc_bayes = mod_uc.smooth(params)
```

One benefit of this model is that it gives us an estimate of the underling "level" of inflation, using the smoothed estimate of $\mu_t$, which we can access as the "level" column in the results objects' `states.smoothed` attribute. In this case, because the Bayesian posterior mean of the level's variance is larger than the MLE estimate, its estimated level is a little more volatile.

```python
# Graph
fig, ax = plt.subplots(figsize=(9,4), dpi=300)

# Plot data points
inf['CPIAUCNS'].plot(ax=ax, style='-', label='Observed data')

# Plot estimate of the level term
res_uc_mle.states.smoothed['level'].plot(ax=ax, label='Smoothed level (MLE)')
res_uc_bayes.states.smoothed['level'].plot(ax=ax, label='Smoothed level (Bayesian)')

ax.legend(loc='lower left');
```


Perceptron - Regressão Linear

Data Scientist.: Dr.Eddy Giusepe Chirinos Isidro

# O Perceptron

É uma versão simplificada de um "Neurônio" que calcula a soma ponderada de todas suas entradas e depois aplica um Função de Ativação para fornecer o Resultado.

> $\hat{y} = f(\vec{w}.\vec{x}) = f(w_0 + w_1x_1 + ... + w_mx_m)$

onde $\hat{y}$ é a saída do Perceptron, $\vec{w}$ são os pesos, $\vec{x}$ são os Inputs e $f$ é a função de Ativação. A seguir podemos ver a representação do Perceptron:

# Regressão Linear

Existem várias tarefas que podemos levar a cabo com um Perceptron. Neste script vamos ver uma delas: `Regressão Linear`. Neste tipo de tarefa, como em qualquer tarefa de regressão, queremos obter um modelo que se ajuste da melhor forma possível a um conjunto de Dados determinado. No caso de regressão Linear, este modelo será uma linea reta e o PERCEPTRON, utilizando uma função de ativação linear, $f(x) = x$, é de fato capaz de levar a cabo esta tarefa.

```python
# Carregando as nossas Bibliotecas

import matplotlib.pyplot as plt
import numpy as np

```

```python
x = np.linspace(-10, 10)
y = x

plt.plot(x, y)
plt.grid(True)
plt.xlabel('x', fontsize=14)
plt.ylabel('y = f(x) = x', fontsize=14)
plt.title('Função de Ativação Linear', fontsize=14)
plt.plot(x, np.zeros(len(x)), '--r') # Linha segmentada de vermelho
plt.show()

```

Para ilustrar o funcionamento do Perceptron para tarefas de Regressão Linear, vamos a utilizar o seguinte Dataset sintético:

```python
np.random.seed(42)

x = np.random.rand(20)
y = 2*x + (np.random.rand(20)-0.5)*0.5

plt.plot(x, y,'r.')
plt.xlabel("$x_1$", fontsize=14)
plt.ylabel("$y$", rotation=0, fontsize=20)
plt.grid(True)
plt.show()
```

Neste exemplo sabemos que os pesos de nosso Modelo são: $w_0 = 0$ e $w_1 = 2$ já que são os mesmos utilizados para gerar os Dados. No entanto, nosso objetivo será encontrar estes valores utilizando o algoritmo de `DESCENSO POR GRADIENTE`.

```python
plt.plot(x, y, "r.")
plt.plot(x, 2*x, 'green') # A reta
plt.plot(0.5, 2*0.5, 'sb') # O ponto no gráfico
plt.xlabel("$x_1$", fontsize=20)
plt.ylabel("$y$", rotation=0, fontsize=20)
plt.grid(True)
plt.show()
```

## Treinando o Perceptron

Então, nosso objetivo é encontrar os pesos de nosso modelo, $\vec{w}$, que minimizem uma `FUNÇÃO DE PERDA`. Está função nos dá uma medida de erro que faz nosso modelo ao predezir os Dados de nosso Dataset. Em tarefas de regressão, a função de perda mais utilizada é `ERRO QUADRÁTICO MÉDIO` ou Mean Square Loss (MSE):

\begin{equation}
 MSE(\hat{y}, y) = \frac{1}{N} \sum^N_{j=1}(\hat{y}^{(j)} - y^{(j)})^2
\end{equation}

onde $N$ é o número de Dados em nosso Dataset. Esta função recibe as predições de nosso modelo para os Dados do Dataset, $\hat{y}^{(j)}$, e as compara com os valores reais (ground truth), $y^{(j)}$, calculando o valor médio da diferença para cada elemento do Dataset elevado ao quadrado.

O Algoritmo `Gradiente Descendente` é implementado assim:

* Calcular a saída do modelo, $\hat{y}$

* Calcular a DERIVADA da função de perda com respeito aos parâmetros do modelo:

\begin{equation}
 \frac{\partial MSE}{\partial w} = \frac{2}{N} \frac{\partial \hat{y}}{\partial w} (\hat{y} - y)
\end{equation}

onde: $ \frac{\partial \hat{y}}{\partial w} = x$.

* Atualizar os parâmetros: $w \Longleftarrow w - \eta \frac{\partial MSE}{\partial w}$, onde $\eta$ é o `Learning rate`.

* Repetir até convergir.

Inicializaremos nosso modelo com valores aleatórios para os pesos, é iremos atualizando seus valores de maneira iterativa na direção da pendente negativa da função de Perda. O algoritmo termina quando observemos que o valor da função de perda deixa de melhorar (indicando que haveremos llegado a um mínimo local de nossa função de Perda).

```python
def gradient(w, x, y):
    # calculamos la derivada de la función de pérdida
    # con respecto a los parámteros `w`
    dldw = x*w - y
    dydw = x
    dldw = dldw*dydw
    return np.mean(2*dldw)

def cost(y, y_hat):
    # calculamos la función de pérdida
    return ((y_hat - y)**2).mean()

def solve(epochs = 29, w = 1.2, lr = 0.2):
    # iteramos un número determinado de `epochs`
    # por cada epoch, calculamos gradientes y
    # actualizamos los pesos
    weights = [(w, gradient(w, x, y), cost(x*w, y))]
    for i in range(1,epochs+1):
        dw = gradient(w, x, y)
        w = w - lr*dw
        weights.append((w, dw, cost(x*w, y)))
    return weights
```

```python

```

```python

```

```python

```


# Euler Method with Theorems Applied to Non-Linear Population Equations

The more general form of a first order Ordinary Differential Equation is:
\begin{equation}
y^{'}=f(t,y).
\end{equation}
This can be solved analytically by integrating both sides but this is not straight forward for most problems.
Numerical methods can be used to approximate the solution at discrete points.
In this notebook we will work through the Euler method for two initial value problems:

1. A non-linear sigmoidal population equation
\begin{equation}
y^{'}=0.2 y− 0.01 y^2 ,
\end{equation}
2. A non-linear sigmoidal population differential equation with a wiggle,
\begin{equation}
y^{'}=0.2 y-0.01 y^2+\sin(2\pi t).
\end{equation}

## Euler method

The simplest one step numerical method is the Euler Method named after the most prolific of mathematicians [Leonhard Euler](https://en.wikipedia.org/wiki/Leonhard_Euler) (15 April 1707 – 18 September 1783) .

The general Euler formula for to the first order differential equation
\begin{equation}
y^{'} = f(t,y),
\end{equation}

approximates the derivative at time point $t_i$,
\begin{equation}
y^{'}(t_i) \approx \frac{w_{i+1}-w_i}{t_{i+1}-t_{i}},
\end{equation}
where $w_i$ is the approximate solution of $y$ at time $t_i$.
This substitution changes the differential equation  into a __difference__ equation of the form
\begin{equation}
\frac{w_{i+1}-w_i}{t_{i+1}-t_{i}}=f(t_i,w_i).
\end{equation}
Assuming uniform stepsize $t_{i+1}-t_{i}$ is replaced by $h$, re-arranging the equation gives
\begin{equation}
w_{i+1}=w_i+hf(t_i,w_i),
\end{equation}
 This can be read as the future $w_{i+1}$ can be approximated by the present $w_i$ and the addition of the input to the system $f(t,y)$ times the time step.

```python
## Library
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
import pandas as pd

warnings.filterwarnings("ignore")
```

# Non-linear population equation

The general form of the non-linear sigmoidal population growth differential equation is:
\begin{equation}
y^{'}=\alpha y-\beta y^2,
\end{equation}
where $\alpha$ is the growth rate and $\beta$ is the death rate. The initial population at time $ a $ is
\begin{equation}
 y(a)=A,
\end{equation}
\begin{equation}
 a\leq t \leq b.
\end{equation}

## Specific non-linear population equation
Given the growth rate $$\alpha=0.2,$$ and death rate $$\beta=0.01,$$ giving the specific differential equation,
\begin{equation}
 y^{'}=0.2 y-0.01 y^2,
\end{equation}
The initial population at time $2000$ is
\begin{equation}
 y(2000)=6,
\end{equation}
we are interested in the time period
\begin{equation}
 2000\leq t \leq 2020.
\end{equation}

## Initial Condition
To get a specify solution to a first order initial value problem, an __initial condition__ is required.
For our population problem the initial population is 6 billion people:
\begin{equation}
y(2000)=6.
\end{equation}

## General Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation}
 h=\frac{b-a}{N}.
\end{equation}
## Specific Discrete Interval
Here the interval is $2000\leq t \leq 2020$ with $N=200$
\begin{equation}
h=\frac{2020-2000}{200}=0.1,
\end{equation}
this gives the 201 discrete points with stepsize h=0.1:
\begin{equation}
t_0=2000, \ t_1=0.1, \ ... t_{200}=2020,
\end{equation}
which is generalised to
\begin{equation}
 t_i=2000+i0.1, \ \ \ i=0,1,...,200.
\end{equation}
The plot below illustrates the discrete time steps from 2000 to 2002.

```python
### Setting up time
t_end=2020.0
t_start=2000.0
N=200
h=(t_end-t_start)/(N)
time=np.arange(t_start,t_end+0.01,h)
fig = plt.figure(figsize=(10,4))
plt.plot(time,0*time,'o:',color='red')
plt.title('Illustration of discrete time points for h=%s'%(h))
plt.xlim((2000,2002))
plt.plot();
```

## Numerical approximation of Population growth
The differential equation is transformed using the Euler method into a difference equation of the form
\begin{equation}
      w_{i+1}=w_{i}+h (\alpha w_i-\beta w_i\times w_i).
\end{equation}
This approximates a series of of values $w_0, \ w_1, \ ..., w_{N}$.
For the specific example of the population equation the difference equation is,
\begin{equation}
       w_{i+1}=w_{i}+h 0.1 [0.2 w_i-0.01 w_i\times w_i],
\end{equation}
where $i=0,1,2,...,199$, and $w_0=6$. From this initial condition the series is approximated.

```python
w=np.zeros(N+1)
w[0]=6
for i in range (0,N):
    w[i+1]=w[i]+h*(0.2*w[i]-0.01*w[i]*w[i])

```

The plot below shows the Euler approximation $w$ in blue squares.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time,w,'s:',color='blue',label='Euler')
plt.xlim((min(time),max(time)))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Euler solution')
plt.plot();
```

### Table
The table below shows the iteration $i$, the discrete time point t[i], and the Euler approximation w[i] of the solution $y$ at time point t[i] for the non-linear population equation.

```python
d = {'time t[i]': time[0:10],    'Euler (w_i) ':w[0:10]}
df = pd.DataFrame(data=d)
df
```

      time t[i]
      Euler (w_i)

      0
      2000.0
      6.000000

      1
      2000.1
      6.084000

      2
      2000.2
      6.168665

      3
      2000.3
      6.253986

      4
      2000.4
      6.339953

      5
      2000.5
      6.426557

      6
      2000.6
      6.513788

      7
      2000.7
      6.601634

      8
      2000.8
      6.690085

      9
      2000.9
      6.779130

## Numerical Error
With a numerical solution there are two types of error:
* local truncation error at one time step;
* global error which is the propagation of local error.

### Derivation of  Euler Local truncation error
The left hand side of a initial value problem $\frac{dy}{dt}$ is approximated by __Taylors theorem__ expand about a point $t_0$ giving:
\begin{equation}
y(t_1) = y(t_0)+(t_1-t_0)y^{'}(t_0) + \frac{(t_1-t_0)^2}{2!}y^{''}(\xi), \ \ \ \ \ \ \xi \in [t_0,t_1].
\end{equation}
Rearranging and letting $h=t_1-t_0$ the equation becomes
\begin{equation}
y^{'}(t_0)=\frac{y(t_1)-y(t_0)}{h}-\frac{h}{2}y^{''}(\xi).
\end{equation}
From this the local truncation error is
\begin{equation}
\tau \leq \frac{h}{2}M,
\end{equation}
where $y^{''}(t) \leq M $.
#### Derivation of  Euler Local truncation error for the Population Growth
As the exact solution $y$ is unknown we cannot get an exact estimate of the second derivative
\begin{equation}
y'(t)=0.2 y-0.01 y^2,
\end{equation}
differentiate with respect to $t$,
\begin{equation}
y''(t)=0.2 y'-0.01 (2yy'),
\end{equation}
subbing the original equation gives
\begin{equation}
y''(t)=0.2 (0.2 y-0.01 y^2)-0.01 \big(2y(0.2 y-0.01 y^2)\big),
\end{equation}
which expresses the second derivative as a function of the exact solution $y$, this is still a problem as the value of $y$ is unknown, to side step this issue we assume the population is between $0\le y \le 20,$ this gives
\begin{equation}
\max|y''|=M\leq 6,
\end{equation}
this gives a local trucation for $h=0.1$ for our non-linear equation is
\begin{equation}
\tau=\frac{h}{2}6=0.3.
\end{equation}

```python
M=6
fig = plt.figure(figsize=(10,4))
plt.plot(time[0:2],0.1*M/2*np.ones(2),'v:'
         ,color='black',label='Upper Local Truncation')
plt.xlabel('time')
plt.ylim([0,0.1])
plt.legend(loc='best')
plt.title('Local Truncation Error')
plt.plot();
```

### Global truncation error for the population equation
For the population equation specific values $L$ and $M$ can be calculated.

In this case $f(t,y)=\epsilon y$ is continuous and satisfies a Lipschitz Condition with constant
\begin{equation}
\left|\frac{\partial f(t,y)}{\partial y}\right|\leq L,
\end{equation}
\begin{equation}
 \left|\frac{\partial (0.2 -0.01 y^2)}{\partial y}\right|\leq |0.2(20)-0.01(2\times y)| \leq |0.2-0.01(2\times 20)|\leq 0.8,
\end{equation}

on $D=\{(t,y)|2000\leq t \leq 2020, 0 < y < 20 \}$ and that a constant $M$
exists with the property that
\begin{equation}
|y^{''}(t)|\leq M\leq 6.
\end{equation}

__Specific Theorem Global Error__

Let $y(t)$ denote the unique solution of the Initial Value Problem
\begin{equation}
y^{'}=0.2 y-0.01 y^2, \ \ \ 2000\leq t \leq 2020, \ \ \ y(0)=6,
\end{equation}
and $w_0,w_1,...,w_N$ be the approx generated by the Euler method for some
positive integer N.  Then for $i=0,1,...,N$ the error is:
\begin{equation}
|y(t_i)-w_i| \leq \frac{6 h}{2\times 0.8}|e^{0.8(t_i-2000)}-1|.
\end{equation}

# Non-linear population equation with a temporal oscilation
Given the specific population differential equation with a wiggle,
\begin{equation}
y^{'}=0.2 y-0.01 y^2+sin(2\pi t),
\end{equation}
with the initial population at time $2000$ is
\begin{equation}
y(2000)=6,
\end{equation}
\begin{equation}
2000\leq t \leq 2020.
\end{equation}

For the specific example of the population equation the difference equation is
\begin{equation}
      w_{i+1}=w_{i}+h 0.5 (0.2 w_i-0.02 w_i\times w_i+sin(2 \pi t),
\end{equation}
for $i=0,1,...,199$,
where $w_0=6$. From this initial condition the series is approximated.
The figure below shows the discrete solution.

```python
w=np.zeros(N+1)
w[0]=6
for i in range (0,N):
    w[i+1]=w[i]+h*(0.2*w[i]-0.01*w[i]*w[i]+np.sin(2*np.pi*time[i]))

fig = plt.figure(figsize=(10,4))
plt.plot(time,w,'s:',color='blue',label='Euler')
plt.xlim((min(time),max(time)))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Euler solution')
plt.plot();
```

### Table
The table below shows the iteration $i$, the discrete time point t[i], and the Euler approximation w[i] of the solution $y$ at time point t[i] for the non-linear population equation with a temporal oscilation.

```python

d = {'time t_i': time[0:10],    'Euler (w_i) ':w[0:10]}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Euler (w_i)

      0
      2000.0
      6.000000

      1
      2000.1
      6.084000

      2
      2000.2
      6.227443

      3
      2000.3
      6.408317

      4
      2000.4
      6.590522

      5
      2000.5
      6.737676

      6
      2000.6
      6.827034

      7
      2000.7
      6.858187

      8
      2000.8
      6.853211

      9
      2000.9
      6.848203

```python

```

Chapter 01 - Euler Methods/102_Euler_method_with_Theorems_nonlinear_Growth_function.ipynb


# Fitting Laplace equation

11.6 $\mu$m-sized droplet making a contact angle of 140$^{\circ}$ on the AFM cantilever.

Volume of the spherical cap is given by

\begin{equation*}
V = \frac{\pi}{3} R^3 (2 + \cos \theta) (1-\cos \theta)^2
\end{equation*}

```python
import numpy as np
import matplotlib.pyplot as plt
import os
import glob
import pandas as pd
import seaborn as sns
from scipy import interpolate

import warnings
warnings.filterwarnings('ignore')

sns.set_style("ticks")
color_b = sns.xkcd_rgb["marine"]

theta = 140
cos_t = np.cos(theta/180*np.pi) # contact angle

R_ = (11.6/2)*1e-6 # um

r_ = R_*np.sin(theta/180*np.pi)

V_ = np.pi/3*R_**3*(2+cos_t)*(1-cos_t)**2

print ("Contact size on cantilever is {:e} m".format(2*r_))
print ("Droplet volume is {:e} m^3".format(V_))

H_init = (1-cos_t)*R_
print ("Initial height is {:e} m".format(H_init))
```

    Contact size on cantilever is 7.456336e-06 m
    Droplet volume is 7.863491e-16 m^3
    Initial height is 1.024306e-05 m

Reading text file containing force spectroscopy measurements

```python
file_name = 'droplet.txt'

col_names = ["Vertical Tip Position", "Vertical Deflection", "Precision 5", "Height",
             "Height (measured & smoothed)", "Height (measured)", "Series Time", "Segment Time"]

df = pd.read_csv(file_name, comment='#', sep=' ', names = col_names)

x1 = df["Vertical Tip Position"]
x1 = x1 + 0.1*1e-6
y1 = df["Vertical Deflection"]*1e9 # nN
t = df["Series Time"]
h = df["Height (measured)"]

i_start = 0
i_end  = 500000

i_turn = np.where(h == np.amin(h))[0][0]

i_skip = 1

x1_approach = x1[i_start:i_turn:i_skip]*1e6
y1_approach = y1[i_start:i_turn:i_skip]
x1_retract = x1[i_turn:i_end:i_skip]*1e6
y1_retract = y1[i_turn:i_end:i_skip]

x_shift_approach = 0
x1_approach = x1_approach.values-x_shift_approach
y1_approach = y1_approach.values

x_shift_retract = 0
x1_retract = x1_retract-x_shift_retract
```

# Fitting force curve

## Solving non-dimensionalized Laplace equation

Following the approach of Butt et al, Curr. Opin. Colloid Interface Sci. 19, 343-354 (2014), we non-dimensionalize the variables by $a$ and $\gamma$, i.e.

\begin{align}
\hat{F} &=  F \gamma^{-1} a^{-1} \\
\hat{r} &=  r a^{-1} \\
\hat{V} &=  V a^{-3} \\
\Delta \hat{P} &=  \Delta P a \gamma^{-1},\mathrm{etc}
\end{align}

and hence, we get the non-dimensionalized Young-Laplace equation

\begin{align}
\frac{\hat{u}''}{(1+\hat{u}'^{2})^{3/2}} - \frac{1}{\hat{u}\sqrt{1 + \hat{u}'^{2}}}
&= -\Delta \hat{P} \; \mathrm{for} \; \hat{z} \in (0, \hat{h}) \\
\end{align}

subject to the boundary conditions

\begin{align}
\hat{u}(0) &= \hat{r} \\
\hat{u}(\hat{h}) &= 1 \\
\int_{0}^{h} \pi \hat{u}^{2}d\hat{z} &= \hat{V} \\
\end{align}

and

\begin{align}
F &= -\frac{2 \pi \hat{r}}{\sqrt{1 + \hat{u}'(0)^{2}}} + \pi r^{2}\Delta P.
\end{align}

Using the shooting method, we chose to frame the Young-Laplace equation as an initial value problem, with initial values

\begin{align}
\hat{u}(0) &= \hat{r} \\
\hat{u}'(0) &= -\cot \theta,
\end{align}

while requiring that

$$\mathbf{\hat{m}}(\hat{r}, \Delta \hat{P}, \theta) =
\begin{pmatrix} \hat{u}(\hat{h})- 1 \\
\int_{0}^{\hat{h}} \pi \hat{u}^{2}d\hat{z} - \hat{V}\\
\hat{F} + 2 \pi \gamma \hat{r} \sin \theta - \pi \hat{r}^{2}\Delta \hat{P}
 \end{pmatrix} = 0$$.

This converts the boundary value problem into a root-finding problem of $\mathbf{\hat{m}}=0$, as implemented below

# Results

```python
from scipy.integrate import odeint
from scipy.integrate import solve_ivp
from scipy.optimize import newton, fsolve
from scipy.integrate import simps

n_point = 100  # number of points to plot droplet profile

def ode(u, z):
    return np.array([u[1], (1+u[1]**2)**(1.5)*(1/(u[0]*np.sqrt(1+u[1]**2)) - u[2]), 0])

# Calculate the residue
def m_res(x, h_f, F_exp):

    r = x[0]
    dP = x[1]
    theta = x[2]

    z_arr = np.linspace(0, h_f, n_point)

    y_start = np.array([r, -1/np.tan(theta), dP])
    sol = odeint(ode, y_start, z_arr, atol=1e-7)
#     sol = solve_ivp(ode, z_arr, y_start, atol=1e-11)

    u_arr = sol[:,0]
    dP_arr = sol[:,2]
    u_arr_1 = u_arr[dP_arr == dP]
    z_arr_1 = z_arr[dP_arr == dP]
    V_int = simps(np.pi*u_arr_1**2, z_arr_1)

    u_h = u_arr[-1]

    F = 2*np.pi*r*np.sin(theta) - dP*np.pi*r**2
    return np.array([1-u_h, V_norm-V_int, F-F_exp])

def calculate_sol(x, hf):
    r = x[0]
    dP = x[1]
    theta = x[2]

    z_arr = np.linspace(0, hf, n_point)

    y_start = np.array([r, -1/np.tan(theta), dP])
    sol = odeint(ode, y_start, z_arr, atol=1e-7)
#     sol = solve_ivp(ode, z_arr, y_start, atol=1e-11)

    u_arr = sol[:,0]
    dP_arr = sol[:,2]
    u_arr_1 = u_arr[dP_arr == dP]
    z_arr_1 = z_arr[dP_arr == dP]
    V_int = simps(np.pi*u_arr_1**2, z_arr_1)

    F_sol = 2*np.pi*r*np.sin(theta) - dP*np.pi*r**2

    return z_arr, u_arr, dP, V_int, F_sol
```

```python
fig, ax = plt.subplots(2, 1, figsize=(8,10), gridspec_kw={'height_ratios': [1.2, 1]})

i_point = 50000

H = (H_init+x1[i_point])*1e6 # Droplet height in um
F = y1[i_point] # Force in nN
print (H_init, H)

V = V_*1e15 # Droplet volume in pl
r = r_*1e6 # Contact size in um
gamma = 630. # Surface tension in mN/m

# Non-dimensionalized parameters
V_norm = V/r**3*1e3
H_norm = H/r
F_norm = -F/(r*gamma)

# guess values (non-dimensionalized)
dP = 1.0
r0 = 0.3
t0 = 140./180*np.pi

x_guess = np.array([r0, dP, t0])

# root-finding solver, look for solution m=0
x_opt = fsolve(m_res, x_guess, args=(H_norm, F_norm))
print (m_res(x_opt, H_norm, F_norm))

z_sol, u_sol, p_sol, V_sol, F_sol = calculate_sol(x_opt, H_norm)

ax[1].plot(u_sol*r, z_sol*r, color=color_b)
ax[1].plot(-u_sol*r, z_sol*r, color=color_b)

ax[1].plot([-40, 40], [0, 0], '-k') # plotting the base
ax[1].plot([-u_sol[-1]*r, u_sol[-1]*r], [H, H], '-k', alpha=0.5, lw=5)

ax[1].set_xlim([-8, 8])
ax[1].set_ylim([-1,13])
ax[1].set_xlabel("x (um)")
ax[1].set_ylabel("h (um)")

ax[1].set_aspect('equal')

#print out the fitted results
r_fit = x_opt[0]
P_fit = x_opt[1]
t_fit = x_opt[2]

ax[1].text(-1.2, 1.5, r"$\dot{\theta}$ = " + "{}".format(np.round(t_fit/np.pi*180,1))+r"$^{\circ}$")

print ("Contact size is {} um".format(np.round(2*r_fit*r,1)))
print ("Contact angle is {} deg".format(np.round(t_fit/np.pi*180,1)))
print ("Pressure inside droplet is {} kPa".format(np.round(P_fit*gamma/r,1)))

x_shift = 0.0
ax[0].plot(x1_approach-x_shift, y1_approach, color=color_b, lw=1, ls='--')
ax[0].plot(x1_retract-x_shift, y1_retract, color_b, lw=1, ls='-')
ax[0].set_xlim([-1.5, 2])

ax[0].scatter(x1[i_point]*1e6, y1[i_point])
ax[0].set_xlabel("z (um)")
ax[0].set_ylabel("F (nN)")
```


```python
import modern_robotics as mr
import numpy as np
import sympy as sp
from sympy.physics.mechanics import dynamicsymbols, mechanics_printing
mechanics_printing()
from Utilities.symbolicFunctions import *
from Utilities.kukaKinematics import Slist, Mlist
```

# Task 3
## 3.2
### Develop and implement a solution for the analytic inverse kinematics for the Agilus robot.

```python
np.set_printoptions(precision=5)
np.set_printoptions(suppress=True)

#The zero config body frame used in the analytical solution. This is defined as the frame Me found in kukaKinematics.py.
M = np.array([[1, 0, 0, 980], #End effector
             [0, 1, 0, 0],
             [0, 0, 1, 435],
             [0, 0, 0, 1]])
print("M: \n",M)

#Thetas_gen can be modified to generate a valid desired position T_sd
thetas_gen = np.array([0,0,0,0,0,0])
T_sd = mr.FKinSpace(M,Slist,thetas_gen)

# Finding the thetas. agilus_analytical_IK is defined in symbolicFunctions.py
thetas_up, thetas_down = agilus_analytical_IK(Slist, M, T_sd)

# Resulting end effector pose and Ps for elbow up:
T_up = mr.FKinSpace(M,Slist,thetas_up)
P_up = ps_from_Tsd(T_up)

# And for elbow down:
T_down = mr.FKinSpace(M,Slist,thetas_down)
P_down = ps_from_Tsd(T_down)

print("The points P reached by both solutions: ", P_down, P_up)

```

    M:
     [[  1   0   0 980]
     [  0   1   0   0]
     [  0   0   1 435]
     [  0   0   0   1]]
    The points P reached by both solutions:  [900.   0. 435.   1.] [900.   0. 435.   1.]

## Task 3.3
### Confirm that the solution of the analytical inverse kinematics from the previous point agrees with the solution from a numerical inverse kinematics solver.

```python
#Numerical solution
thetas_num, found = mr.IKinSpace(Slist,M,T_sd,[0,0,1,0,0.5,0],0.001,0.01)
thetas_num_p = [0,0,0,0,0,0] #the post processed solution

#Post process numerical angles to be [-pi, pi]
for i in range(0,6):
    thetas_num_p[i] = thetas_num[i]%(2*np.pi)
    if thetas_num_p[i]>np.pi:
        thetas_num_p[i] = thetas_num_p[i]-2*np.pi

T_num = mr.FKinSpace(M,Slist,thetas_num_p)

print("T_sd: \n",T_sd,'\n\n', "T_up: \n",T_up, '\n\n',"T_down: \n",T_down, "\n\nT_num: \n", T_num)

print('\nGenerating thetas:', thetas_gen, '\nElbow up: ', np.round(thetas_up,3), '\nElbow down: ', np.round(thetas_down,3), '\nNumerical: ', np.round(thetas_num_p,3), found, '\n')
print("\nElbow down: \n")

#apply_joint_lim is a function that finds out if a joint angle is outside the joints limits. Defined in symbolicFunctions.py:
print("Elbow down solution viable: ", apply_joint_lim(jointLimits, thetas_down), "\n")
print("\nElbow up: \n")
print("Elbow up solution viable: ", apply_joint_lim(jointLimits, thetas_up), "\n")

```

    T_sd:
     [[  1.   0.   0. 980.]
     [  0.   1.   0.   0.]
     [  0.   0.   1. 435.]
     [  0.   0.   0.   1.]]

     T_up:
     [[  1.  -0.   0. 980.]
     [  0.   1.   0.   0.]
     [ -0.  -0.   1. 435.]
     [  0.   0.   0.   1.]]

     T_down:
     [[  1.   0.   0. 980.]
     [  0.   1.   0.   0.]
     [  0.   0.   1. 435.]
     [  0.   0.   0.   1.]]

    T_num:
     [[  1.        0.        0.      979.99945]
     [  0.        1.        0.       -0.     ]
     [ -0.        0.        1.      434.99998]
     [  0.        0.        0.        1.     ]]

    Generating thetas: [0 0 0 0 0 0]
    Elbow up:  [ 0.    -0.08   0.166 -3.142  0.086 -3.142]
    Elbow down:  [ 0. -0.  0.  0.  0.  0.]
    Numerical:  [-0.  0. -0.  0.  0. -0.] True

    Elbow down:

    Elbow down solution viable:  True

    Elbow up:

    Elbow up solution viable:  True

## Task 3.4
### Using the developed analytic inverse kinematics formulation, visualize the Agilus robot in both elbow-up and elbow-down configurations for the same end-effector pose.

```python
from Utilities.RobotClass import *
```

```python
Kuka = Robot(Mlist, ['z', '-z', 'x', 'x', '-z','x']) #Initializes kuka-robot object in zero-configuration
Kuka.draw_robot()
```

    WebVisualizer(window_uid='window_0')

```python
Kuka.transform(Slist, thetas_down_num ) # Elbow DOWN
pUp = np.array(Kuka.joints[5].coord.get_center()) #Get coordinates of {6} in elbow down
Kuka.draw_robot()

Kuka.transform(Slist, thetas_up_num ) # Elbow UP
pDown = np.array(Kuka.joints[5].coord.get_center(),dtype=float)  #Get coordinates of {6} in elbow up
Kuka.draw_robot()

print("Both elbow up and elbow down config yields the same end-effector pose: ",pUp.round(4) == pDown.round(4))
```

    WebVisualizer(window_uid='window_1')

    WebVisualizer(window_uid='window_2')

    Both elbow up and elbow down config yields the same end-effector pose:  [ True  True  True]

```python

```