# day 12: Stochastic Gradient Descent

# Objectives

* See how Stochastic Gradient Descent can be used on a simple model (1-dim. linear regression)
* See how SGD can be used on a complex model (MLPClassifier)
* Understand impact of batch size and learning rate (aka step size)

# Outline
* [Review: Loss and Gradient for 1-dim. Linear Regression](#part1)
* [Part 1: Stochastic Estimates of the Loss](#part1)
* [Part 2: Stochastic Estimates of the Gradient](#part2)
* [Part 3: Stochastic Gradient Descent Algorithm in a few lines of Python](#part3)
* [Part 4: Using sklearn code to train MLPClassifier with SGD](#part4)

# Takeaways

* Stochastic estimates of loss functions are possible when the function is *additive* over training examples

* Stochastic gradient descent is a simple algorithm that can be implemented in a few lines of Python
* * Practical issues include selecting step size and batch size $B$

* Selecting batch size trades off two things:
* * Runtime cost of computing each gradient estimate (scales with $O(B)$, so smaller is better)
* * Quality of the estimate (larger $B$ leads to less variance)

```python
import numpy as np
```

```python
import sklearn.neural_network
```

```python
# import plotting libraries
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', font_scale=1.25, style='whitegrid')
```

# Create simple dataset:   y = 1.234 * x + noise

We will *intentionally* create a toy dataset where we know that a good solution has slope near 1.234.

We'll generate N = 1000 examples.

Naturally, the best slope for the finite dataset we create won't be exactly 1.234 (because of the noise added plus the fact that our dataset size is limited).

```python
def create_dataset(N=1000, slope=1.234, noise_stddev=0.1, random_state=0):
    random_state = np.random.RandomState(int(random_state))

    # input features
    x_N = np.linspace(-2, 2, N)

    # output features
    y_N = slope * x_N + random_state.randn(N) * noise_stddev

    return x_N, y_N
```

```python
xtrain_N, ytrain_N = create_dataset(N=1000, noise_stddev=0.3)
```

```python
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5,5))
plt.plot(xtrain_N, ytrain_N, 'k.');
plt.xlabel('x');
plt.ylabel('y');
```

# Review: Gradient Descent for 1-dim. Linear Regression

## Define prediction model

Consider the *simplest* linear regression model. A single weight parameter $w \in \mathbb{R}$ representing the slope of the prediction line. No bias/intercept.

To make predictions, we just compute the weight multiplied by the input feature
$$
\hat{y}(x) = w \cdot x
$$

## Define loss function

We want to minimize the total *squared error* across all N observed data examples (input features $x_n$, output responses $y_n$)

Given a full dataset of $N$ examples, we can compute the loss as:

\begin{align}
    \min_{w \in \mathbb{R}} ~~ &\ell(w)
    \\
    \text{calc_loss}(w) = \ell(w) &= \frac{1}{N} \sum_{n=1}^N \frac{1}{2} (y_n - w x_n)^2
\end{align}

Given a random minibatch of $B$ examples, we can *estimate* the loss as:

\begin{align}
    \text{estimate_loss}(w) = &= \frac{1}{B} \sum_{n=1}^B \frac{1}{2} (y_n - w x_n)^2
\end{align}

### Exercise 1A: Complete the code below

You should make it match the math expression above.

```python
def calc_loss(w, xbatch_B, ybatch_B):
    ''' Compute loss for slope-only least-squares linear regression

    Will compute the estimate of the loss given a minibatch of x,y values

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    loss : float
        Mean of squared error loss at provided w value
    '''
    B = xbatch_B.shape[0]
    yhat_B = xbatch_B * w
    half_mean_squared_error = 0.5 * np.mean(np.square(ybatch_B - yhat_B))
    return half_mean_squared_error
```

### Define the gradient function

Given a full dataset of $N$ examples, we can compute the gradient as:

\begin{align}
\text{calc_grad}(w) = \ell'(w) &= \frac{1}{N} \frac{\partial}{\partial w} [ \sum_{n=1}^N \frac{1}{2} (y_n - w x_n)^2]
\\
&= \frac{1}{N}  \sum_{n=1}^N (y_n - w x_n) (-x_n)
\\
&= \frac{1}{N} \sum_{n=1}^N (w x_n - y_n) (x_n)
\\
&= w \left( \frac{1}{N} \sum_{n=1}^N x_n^2 \right) - \frac{1}{N} \sum_{n=1}^N y_n x_n
\end{align}

Given a random minibatch of $B$ examples, we can *estimate* the gradient using:

\begin{align}
\text{estimate_grad}(w) &= \frac{1}{B} \frac{\partial}{\partial w} [ \sum_{n=1}^B \frac{1}{2} (y_n - w x_n)^2]
\\
&= w \left( \frac{1}{B} \sum_{n=1}^B x_n^2 \right) - \frac{1}{B} \sum_{n=1}^B y_n x_n
\end{align}

Below, we've implemented the gradient calculation in code for you

```python
def calc_grad(w, xbatch_B, ybatch_B):
    ''' Compute gradient for slope-only least-squares linear regression

    Will compute a deterministic estimate of the gradient given a minibatch of x,y values

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    g : float
        Value of derivative of loss function at provided w value
    '''
    g = w * np.mean(np.square(xbatch_B)) - np.mean(xbatch_B * ybatch_B)
    return g
```

# Part 1: Stochastic estimates of the loss

### Plot whole-dataset loss evaluated at each w from -3 to 8

We should see a "bowl" shape with one *global* minima, because our optimization problem is "convex"

```python
G = 101
w_grid = np.linspace(-3, 8, G) # create array of 300 values between -3 and 8
print(w_grid)
```

    [-3.   -2.89 -2.78 -2.67 -2.56 -2.45 -2.34 -2.23 -2.12 -2.01 -1.9  -1.79
     -1.68 -1.57 -1.46 -1.35 -1.24 -1.13 -1.02 -0.91 -0.8  -0.69 -0.58 -0.47
     -0.36 -0.25 -0.14 -0.03  0.08  0.19  0.3   0.41  0.52  0.63  0.74  0.85
      0.96  1.07  1.18  1.29  1.4   1.51  1.62  1.73  1.84  1.95  2.06  2.17
      2.28  2.39  2.5   2.61  2.72  2.83  2.94  3.05  3.16  3.27  3.38  3.49
      3.6   3.71  3.82  3.93  4.04  4.15  4.26  4.37  4.48  4.59  4.7   4.81
      4.92  5.03  5.14  5.25  5.36  5.47  5.58  5.69  5.8   5.91  6.02  6.13
      6.24  6.35  6.46  6.57  6.68  6.79  6.9   7.01  7.12  7.23  7.34  7.45
      7.56  7.67  7.78  7.89  8.  ]

```python
loss_grid = np.zeros(G)
for gg in range(G):
    loss_grid[gg] = calc_loss(w_grid[gg], xtrain_N, ytrain_N)

plt.plot(w_grid, loss_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('loss(w)');
plt.ylim([0, 35]);
```

### Discussion 1a: Visually, at what value of $w$ does the loss function have a minima? Is it near where you would expect (hint: look above for the "true" slope value used to generate the data)

```python

```

### Sampling a minibatch of size B

We have provided some starter code that samples a minibatch of B examples from a training set of N examples

Uses a provided `random_state` pseudo-random number generator, which defaults to numpy's if not specified.

```python
def draw_minibatch(xtrain_N, ytrain_N, batch_size=100, random_state=np.random):
    ''' Sample a minibatch of desired size from provided training set

    Returns
    -------
    xbatch_B : 1D array, size (B,)
        x values of minibatch
    ybatch_B : 1D array, size (B,)
        y values of minibatch
    '''
    N = ytrain_N.size
    selected_row_ids = random_state.choice(np.arange(N), size=batch_size, replace=False)
    xbatch_B = xtrain_N[selected_row_ids].copy()
    ybatch_B = ytrain_N[selected_row_ids].copy()
    return xbatch_B, ybatch_B
```

### Show several minibatches of size 1

```python
draw_minibatch(xtrain_N, ytrain_N, 1)
```

    (array([-1.76376376]), array([-2.28530683]))

```python
draw_minibatch(xtrain_N, ytrain_N, 1)
```

    (array([1.42342342]), array([1.62548]))

```python
draw_minibatch(xtrain_N, ytrain_N, 1)
```

    (array([-1.35935936]), array([-1.82685918]))

### Show example minibatch of size 500

```python
xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, 500)

print("Showing first 5 entries of x array of shape %s" % str(xbatch_B.shape))
print(xbatch_B[:5])
print("Showing first 5 entries of y array of shape %s" % str(ybatch_B.shape))
print(ybatch_B[:5])
```

    Showing first 5 entries of x array of shape (500,)
    [ 1.67967968  0.41041041 -0.42642643 -0.04204204 -0.96296296]
    Showing first 5 entries of y array of shape (500,)
    [ 2.0268924   0.2231359  -1.01115247  0.01427242 -1.00518248]

### Exercise 1b: Would you expect the loss on the minibatch to be better at $w=1.0$ or at $w = -1.0$?

TODO write answer here to guess, then use code below to figure it out.

Show the loss at this random minibatch when $w = 1.0$

```python
calc_loss(1.0, xbatch_B, ybatch_B)
```

    0.08034805534465543

Show the loss at this random minibatch when w = -2.0

```python
calc_loss(-1.0, xbatch_B, ybatch_B)
```

    3.290188505364923

### Exercise 1c: Can you draw a random minibatch of size 25 and display *all* of it?

```python
xbatch_25, ybatch_25 = draw_minibatch(xtrain_N, ytrain_N, 25)

print("Showing x array")
print(xbatch_25)
print("Showing y array")
print(ybatch_25)

```

    Showing x array
    [ 1.83983984 -0.03803804  0.90690691 -0.63863864  1.54754755 -1.25125125
      1.11911912 -0.8028028   1.35935936  1.83583584  1.75575576  1.23523524
      0.02602603  1.62362362 -0.47047047 -1.04304304 -0.37037037  1.003003
     -0.50650651  1.27127127  0.8028028   1.57957958  0.68268268  1.38338338
      0.25025025]
    Showing y array
    [ 1.88661403 -0.35591952  0.83928641 -0.46752726  2.19059324 -1.52839452
      1.36085275 -0.87964189  1.77088357  2.96941553  1.9289166   1.01781127
     -0.45767424  2.55036863 -1.25722983 -1.0840427  -0.43935936  1.46951416
     -0.6998666   2.07332527  0.5571766   1.52262524  0.67871684  1.64914208
      0.36732972]

### Exercise 1d: What is the loss at your minibatch at $w=1.0$ and $w = 1.5$?

```python
calc_loss(1.0, xbatch_25, ybatch_25)
```

    0.09975696099948157

```python
calc_loss(-1.0, xbatch_25, ybatch_25)
```

    3.3594047098831754

## Plot: Compute the *stochastic estimate* of loss as a function of $w$, using batch size 50

We'll make 3 lines, for 3 separate *trials* of this procedure, so we can see how much each trial's curve might vary

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_loss_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=50, random_state=random_state)
        stoch_loss_grid[gg] = calc_loss(w_grid[gg], xbatch_B, ybatch_B)

    plt.plot(w_grid, stoch_loss_grid, '.-', label='Trial %d' % trial);
plt.xlabel('w');
plt.ylabel('loss(w)');
plt.ylim([0, 35]);
```

## Exercise 1e: Compute stochastic estimate of loss as function of w, for batch_size = 5

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_loss_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=5, random_state=random_state) # TODO use draw_minibatch
        stoch_loss_grid[gg] = calc_loss(w_grid[gg], xbatch_B, ybatch_B)             # TODO use calc_loss at the current batch

    plt.plot(w_grid, stoch_loss_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('loss(w)');
```

## Exercise 1f: Compute stochastic estimate of loss as function of w, for batch_size = 1

Repeat the above for batch_size = 1

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_loss_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=1, random_state=random_state) # TODO use draw_minibatch
        stoch_loss_grid[gg] = calc_loss(w_grid[gg], xbatch_B, ybatch_B)             # TODO use calc_loss at the current batch

    plt.plot(w_grid, stoch_loss_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('loss(w)');
```

## Discussion 1f: What can you say about the *variance* of the stochastic estimates

How does variance change as a function of batch size ?

## Part 2: Stochastic estimates of the gradient

## Sanity check: plot whole dataset gradient evaluated at each w from -3 to 8

```python
grad_grid = np.zeros(G)
for gg in range(G):
    grad_grid[gg] = calc_grad(w_grid[gg], xtrain_N, ytrain_N)

plt.plot(w_grid, grad_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('grad(w)');
plt.ylim([-7, 9]);
```

## Plot: Compute the *stochastic estimate* of grad as a function of $w$, using batch size 50

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_grad_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=50, random_state=random_state)
        stoch_grad_grid[gg] = calc_grad(w_grid[gg], xbatch_B, ybatch_B)

    plt.plot(w_grid, stoch_grad_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('grad(w)');
plt.ylim([-7, 9]);
```

## Exercise 2a: Repeat the above plot at batch_size = 5

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_grad_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=5, random_state=random_state)
        stoch_grad_grid[gg] = calc_grad(w_grid[gg], xbatch_B, ybatch_B)

    plt.plot(w_grid, stoch_grad_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('grad(w)');
plt.ylim([-7, 9]);
```

## Exercise 2c: Repeat the above plot at batch_size = 1

TODO copy code from above.

```python
for trial in range(3):
    random_state = np.random.RandomState(trial) # set seed based on the trial id

    stoch_grad_grid = np.zeros(G)
    for gg in range(G):
        xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size=1, random_state=random_state)
        stoch_grad_grid[gg] = calc_grad(w_grid[gg], xbatch_B, ybatch_B)

    plt.plot(w_grid, stoch_grad_grid, '.-', label='Trial %d' % trial);

plt.xlabel('w');
plt.ylabel('grad(w)');
plt.ylim([-7, 9]);
```

## Discussion 2d: What happens to the variance of the grad estimate as batch_size increases?

## Discussion 2d: What happens to the cost of computing the estimate as batch_size increases?

## Part 3: Stochastic Gradient Descent (GD) as an algorithm in Python

### Define minimize_via_sgd algorithm

Can you understand what each step of this algorithm does?

```python
def minimize_via_sgd(xtrain_N, ytrain_N, init_w=0.0, batch_size=10, step_size=0.001, max_iters=100, random_state=0):
    ''' Perform minimization of provided loss function via gradient descent

    Each "iteration" performs one or more gradient updates, until total training set has been "seen"

    Args
    ----
    xtrain_N : numpy array, shape (N,)
    ytrain_N : numpy array, shape (N,)
    init_w : float
    batch_size : int
    step_size : float
    max_iters : positive int

    Return
    ----
    wopt: float
        array of optimized weights that approximately gives the least error
    info_dict : dict
        Contains information about the optimization procedure useful for debugging
        Entries include:
        * trace_loss_list : list of loss values
        * trace_grad_list : list of gradient values
    '''
    N = int(ytrain_N.size)
    B = int(batch_size)

    if isinstance(random_state, int):
        random_state = np.random.RandomState(random_state)

    w = 1.0 * init_w

    # Create some lists to track progress over time (for debugging)
    trace_loss_list = []
    trace_w_list = []
    trace_grad_list = []

    step_id = 0
    for iter_id in range(max_iters):

        n_examples_seen_this_iter = 0
        while n_examples_seen_this_iter < N:
            xbatch_B, ybatch_B = draw_minibatch(xtrain_N, ytrain_N, batch_size, random_state=random_state)
            n_examples_seen_this_iter += batch_size

            loss = calc_loss(w, xbatch_B, ybatch_B)
            grad = calc_grad(w, xbatch_B, ybatch_B)
            w = w - step_size * grad
            step_id += 1

        print("  iter %5d/%d done | step %5d | w  % 13.5f | loss % 13.4f | grad % 13.4f" % (
            iter_id, max_iters, step_id, w, loss, grad))

        trace_loss_list.append(loss)
        trace_w_list.append(w)
        trace_grad_list.append(grad)

    wopt = w
    info_dict = dict(
        trace_loss_list=trace_loss_list,
        trace_w_list=trace_w_list,
        trace_grad_list=trace_grad_list)

    return wopt, info_dict
```

### Discussion 2a: Which line of the above function does the *parameter update* happen?

Remember, in math, the parameter update of gradient descent is this:
$$
w \gets w - \alpha \nabla_w \ell(w)
$$

where $\alpha > 0$ is the step size.

In words, this math says *move* the parameter $w$ from its current value a *small step* in the "downhill" direction (indicated by gradient).

TODO write down here which line above *you* think it is, then discuss with your group

```python

```

### Try it! Run SGD with step_size = 0.01 and batch_size = 200

Running the cell below will have the following effects:

1) one line will be printed for every iteration, indicating the current w value and its associated loss

2) the "optimal" value of w will be stored in the variable named `wopt` returned by this function

3) a dictionary of information useful for debugging will be stored in the `info_dict` returned by this function

```python
wopt, info_dict = minimize_via_sgd(xtrain_N, ytrain_N, step_size=0.01, batch_size=200);
```

      iter     0/100 done | step     5 | w        0.08015 | loss        1.0488 | grad       -1.7202
      iter     1/100 done | step    10 | w        0.15328 | loss        0.8576 | grad       -1.4773
      iter     2/100 done | step    15 | w        0.22259 | loss        0.7444 | grad       -1.3535
      iter     3/100 done | step    20 | w        0.28735 | loss        0.6190 | grad       -1.2232
      iter     4/100 done | step    25 | w        0.35058 | loss        0.5804 | grad       -1.2300
      iter     5/100 done | step    30 | w        0.40936 | loss        0.5670 | grad       -1.2455
      iter     6/100 done | step    35 | w        0.46134 | loss        0.4426 | grad       -1.0366
      iter     7/100 done | step    40 | w        0.51053 | loss        0.3971 | grad       -0.9791
      iter     8/100 done | step    45 | w        0.55889 | loss        0.3367 | grad       -0.8819
      iter     9/100 done | step    50 | w        0.60231 | loss        0.3295 | grad       -0.8519
      iter    10/100 done | step    55 | w        0.64268 | loss        0.2927 | grad       -0.8156
      iter    11/100 done | step    60 | w        0.68140 | loss        0.2557 | grad       -0.7506
      iter    12/100 done | step    65 | w        0.71875 | loss        0.2472 | grad       -0.7419
      iter    13/100 done | step    70 | w        0.75133 | loss        0.1835 | grad       -0.5930
      iter    14/100 done | step    75 | w        0.78277 | loss        0.1722 | grad       -0.5639
      iter    15/100 done | step    80 | w        0.81064 | loss        0.1731 | grad       -0.5796
      iter    16/100 done | step    85 | w        0.83925 | loss        0.1716 | grad       -0.6222
      iter    17/100 done | step    90 | w        0.86445 | loss        0.1464 | grad       -0.5074
      iter    18/100 done | step    95 | w        0.88863 | loss        0.1269 | grad       -0.4665
      iter    19/100 done | step   100 | w        0.91101 | loss        0.1231 | grad       -0.4666
      iter    20/100 done | step   105 | w        0.93204 | loss        0.1004 | grad       -0.4243
      iter    21/100 done | step   110 | w        0.95136 | loss        0.0874 | grad       -0.3560
      iter    22/100 done | step   115 | w        0.97035 | loss        0.0987 | grad       -0.3765
      iter    23/100 done | step   120 | w        0.98704 | loss        0.0808 | grad       -0.2952
      iter    24/100 done | step   125 | w        1.00346 | loss        0.0852 | grad       -0.3169
      iter    25/100 done | step   130 | w        1.01760 | loss        0.0784 | grad       -0.3205
      iter    26/100 done | step   135 | w        1.03255 | loss        0.0701 | grad       -0.2819
      iter    27/100 done | step   140 | w        1.04555 | loss        0.0659 | grad       -0.2295
      iter    28/100 done | step   145 | w        1.05768 | loss        0.0687 | grad       -0.2576
      iter    29/100 done | step   150 | w        1.06847 | loss        0.0614 | grad       -0.2003
      iter    30/100 done | step   155 | w        1.07939 | loss        0.0661 | grad       -0.2469
      iter    31/100 done | step   160 | w        1.08989 | loss        0.0582 | grad       -0.2022
      iter    32/100 done | step   165 | w        1.10010 | loss        0.0630 | grad       -0.2164
      iter    33/100 done | step   170 | w        1.10851 | loss        0.0565 | grad       -0.1732
      iter    34/100 done | step   175 | w        1.11634 | loss        0.0552 | grad       -0.1632
      iter    35/100 done | step   180 | w        1.12382 | loss        0.0565 | grad       -0.1650
      iter    36/100 done | step   185 | w        1.12964 | loss        0.0494 | grad       -0.1079
      iter    37/100 done | step   190 | w        1.13630 | loss        0.0470 | grad       -0.1289
      iter    38/100 done | step   195 | w        1.14222 | loss        0.0478 | grad       -0.1077
      iter    39/100 done | step   200 | w        1.14832 | loss        0.0590 | grad       -0.1447
      iter    40/100 done | step   205 | w        1.15326 | loss        0.0419 | grad       -0.1074
      iter    41/100 done | step   210 | w        1.15853 | loss        0.0460 | grad       -0.0721
      iter    42/100 done | step   215 | w        1.16395 | loss        0.0509 | grad       -0.1238
      iter    43/100 done | step   220 | w        1.16902 | loss        0.0426 | grad       -0.1063
      iter    44/100 done | step   225 | w        1.17258 | loss        0.0459 | grad       -0.0702
      iter    45/100 done | step   230 | w        1.17556 | loss        0.0466 | grad       -0.0392
      iter    46/100 done | step   235 | w        1.17890 | loss        0.0428 | grad       -0.0586
      iter    47/100 done | step   240 | w        1.18149 | loss        0.0433 | grad       -0.0558
      iter    48/100 done | step   245 | w        1.18422 | loss        0.0477 | grad       -0.0653
      iter    49/100 done | step   250 | w        1.18750 | loss        0.0428 | grad       -0.0637
      iter    50/100 done | step   255 | w        1.19068 | loss        0.0525 | grad       -0.0635
      iter    51/100 done | step   260 | w        1.19410 | loss        0.0478 | grad       -0.0637
      iter    52/100 done | step   265 | w        1.19610 | loss        0.0371 | grad       -0.0345
      iter    53/100 done | step   270 | w        1.19875 | loss        0.0444 | grad       -0.0442
      iter    54/100 done | step   275 | w        1.20074 | loss        0.0458 | grad       -0.0389
      iter    55/100 done | step   280 | w        1.20295 | loss        0.0467 | grad       -0.0212
      iter    56/100 done | step   285 | w        1.20470 | loss        0.0410 | grad       -0.0131
      iter    57/100 done | step   290 | w        1.20604 | loss        0.0386 | grad       -0.0209
      iter    58/100 done | step   295 | w        1.20676 | loss        0.0387 | grad       -0.0180
      iter    59/100 done | step   300 | w        1.20872 | loss        0.0407 | grad       -0.0393
      iter    60/100 done | step   305 | w        1.20974 | loss        0.0391 | grad       -0.0075
      iter    61/100 done | step   310 | w        1.21074 | loss        0.0448 | grad       -0.0445
      iter    62/100 done | step   315 | w        1.21162 | loss        0.0469 | grad        0.0055
      iter    63/100 done | step   320 | w        1.21239 | loss        0.0385 | grad        0.0217
      iter    64/100 done | step   325 | w        1.21317 | loss        0.0404 | grad       -0.0250
      iter    65/100 done | step   330 | w        1.21514 | loss        0.0422 | grad       -0.0427
      iter    66/100 done | step   335 | w        1.21667 | loss        0.0505 | grad       -0.0682
      iter    67/100 done | step   340 | w        1.21734 | loss        0.0441 | grad       -0.0089
      iter    68/100 done | step   345 | w        1.21843 | loss        0.0487 | grad       -0.0520
      iter    69/100 done | step   350 | w        1.21968 | loss        0.0426 | grad       -0.0217
      iter    70/100 done | step   355 | w        1.22061 | loss        0.0451 | grad       -0.0162
      iter    71/100 done | step   360 | w        1.22148 | loss        0.0494 | grad       -0.0131
      iter    72/100 done | step   365 | w        1.22243 | loss        0.0432 | grad       -0.0476
      iter    73/100 done | step   370 | w        1.22310 | loss        0.0359 | grad       -0.0035
      iter    74/100 done | step   375 | w        1.22355 | loss        0.0445 | grad       -0.0733
      iter    75/100 done | step   380 | w        1.22387 | loss        0.0475 | grad       -0.0229
      iter    76/100 done | step   385 | w        1.22515 | loss        0.0418 | grad       -0.0054
      iter    77/100 done | step   390 | w        1.22519 | loss        0.0457 | grad        0.0189
      iter    78/100 done | step   395 | w        1.22512 | loss        0.0411 | grad        0.0211
      iter    79/100 done | step   400 | w        1.22509 | loss        0.0423 | grad       -0.0424
      iter    80/100 done | step   405 | w        1.22611 | loss        0.0435 | grad       -0.0184
      iter    81/100 done | step   410 | w        1.22761 | loss        0.0454 | grad       -0.0352
      iter    82/100 done | step   415 | w        1.22761 | loss        0.0444 | grad        0.0206
      iter    83/100 done | step   420 | w        1.22788 | loss        0.0434 | grad        0.0002
      iter    84/100 done | step   425 | w        1.22852 | loss        0.0393 | grad       -0.0144
      iter    85/100 done | step   430 | w        1.22787 | loss        0.0438 | grad       -0.0081
      iter    86/100 done | step   435 | w        1.22866 | loss        0.0451 | grad       -0.0259
      iter    87/100 done | step   440 | w        1.22834 | loss        0.0410 | grad        0.0073
      iter    88/100 done | step   445 | w        1.22975 | loss        0.0406 | grad       -0.0768
      iter    89/100 done | step   450 | w        1.23034 | loss        0.0427 | grad       -0.0344
      iter    90/100 done | step   455 | w        1.23006 | loss        0.0496 | grad       -0.0083
      iter    91/100 done | step   460 | w        1.23052 | loss        0.0438 | grad       -0.0056
      iter    92/100 done | step   465 | w        1.23073 | loss        0.0441 | grad        0.0013
      iter    93/100 done | step   470 | w        1.23094 | loss        0.0471 | grad        0.0267
      iter    94/100 done | step   475 | w        1.23106 | loss        0.0392 | grad        0.0114
      iter    95/100 done | step   480 | w        1.23164 | loss        0.0459 | grad        0.0033
      iter    96/100 done | step   485 | w        1.23252 | loss        0.0481 | grad       -0.0301
      iter    97/100 done | step   490 | w        1.23304 | loss        0.0368 | grad       -0.0023
      iter    98/100 done | step   495 | w        1.23335 | loss        0.0429 | grad       -0.0010
      iter    99/100 done | step   500 | w        1.23343 | loss        0.0513 | grad       -0.0131

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(12,3))

axes[0].plot(info_dict['trace_loss_list'], '.-');
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list'], '.-');
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list'], '.-');
axes[2].set_title('w');
```

### Discussion 2b: Does it appear from the *parameter* values in trace above that the SGD procedure converged?

### Discussion 2c: Does it appear from the *loss* values in trace above that the SGD procedure converged?

### Discussion 2d: Does it appear from the *grad* values in trace above that the SGD procedure converged?

```python

```

```python

```

## Try with smaller batch_size = 10

```python
wopt, info_dict = minimize_via_sgd(xtrain_N, ytrain_N, step_size=0.01, batch_size=10);
```

      iter     0/100 done | step   100 | w        0.91625 | loss        0.1191 | grad       -0.3511
      iter     1/100 done | step   200 | w        1.14951 | loss        0.0377 | grad        0.0505
      iter     2/100 done | step   300 | w        1.20424 | loss        0.0991 | grad        0.1091
      iter     3/100 done | step   400 | w        1.23031 | loss        0.0381 | grad        0.0042
      iter     4/100 done | step   500 | w        1.23886 | loss        0.0511 | grad       -0.2389
      iter     5/100 done | step   600 | w        1.24595 | loss        0.0859 | grad       -0.1187
      iter     6/100 done | step   700 | w        1.25205 | loss        0.0334 | grad        0.1019
      iter     7/100 done | step   800 | w        1.23007 | loss        0.0334 | grad       -0.0147
      iter     8/100 done | step   900 | w        1.22628 | loss        0.0573 | grad        0.0265
      iter     9/100 done | step  1000 | w        1.23810 | loss        0.0412 | grad        0.0452
      iter    10/100 done | step  1100 | w        1.22357 | loss        0.0227 | grad        0.0010
      iter    11/100 done | step  1200 | w        1.23342 | loss        0.0515 | grad       -0.0504
      iter    12/100 done | step  1300 | w        1.24008 | loss        0.0484 | grad       -0.2045
      iter    13/100 done | step  1400 | w        1.24485 | loss        0.0738 | grad       -0.0568
      iter    14/100 done | step  1500 | w        1.23728 | loss        0.0317 | grad        0.0817
      iter    15/100 done | step  1600 | w        1.22575 | loss        0.0404 | grad        0.0695
      iter    16/100 done | step  1700 | w        1.22235 | loss        0.0293 | grad        0.1196
      iter    17/100 done | step  1800 | w        1.22595 | loss        0.0330 | grad       -0.1241
      iter    18/100 done | step  1900 | w        1.23493 | loss        0.0149 | grad        0.0567
      iter    19/100 done | step  2000 | w        1.23858 | loss        0.0361 | grad       -0.0242
      iter    20/100 done | step  2100 | w        1.23371 | loss        0.0423 | grad        0.0176
      iter    21/100 done | step  2200 | w        1.23280 | loss        0.0428 | grad       -0.0151
      iter    22/100 done | step  2300 | w        1.24703 | loss        0.0544 | grad       -0.0138
      iter    23/100 done | step  2400 | w        1.24159 | loss        0.0544 | grad        0.0364
      iter    24/100 done | step  2500 | w        1.24189 | loss        0.0278 | grad       -0.0217
      iter    25/100 done | step  2600 | w        1.23713 | loss        0.0509 | grad        0.0451
      iter    26/100 done | step  2700 | w        1.23292 | loss        0.0502 | grad        0.1502
      iter    27/100 done | step  2800 | w        1.23131 | loss        0.0266 | grad       -0.0233
      iter    28/100 done | step  2900 | w        1.23236 | loss        0.0267 | grad        0.0038
      iter    29/100 done | step  3000 | w        1.22569 | loss        0.0350 | grad       -0.0734
      iter    30/100 done | step  3100 | w        1.22976 | loss        0.0349 | grad        0.0029
      iter    31/100 done | step  3200 | w        1.23033 | loss        0.0173 | grad        0.0927
      iter    32/100 done | step  3300 | w        1.23814 | loss        0.0387 | grad        0.0252
      iter    33/100 done | step  3400 | w        1.23415 | loss        0.0182 | grad       -0.0323
      iter    34/100 done | step  3500 | w        1.23714 | loss        0.0549 | grad        0.1117
      iter    35/100 done | step  3600 | w        1.23755 | loss        0.0294 | grad       -0.0801
      iter    36/100 done | step  3700 | w        1.23231 | loss        0.0303 | grad        0.1700
      iter    37/100 done | step  3800 | w        1.23270 | loss        0.0449 | grad        0.0136
      iter    38/100 done | step  3900 | w        1.22366 | loss        0.0325 | grad        0.0768
      iter    39/100 done | step  4000 | w        1.22075 | loss        0.0216 | grad       -0.0362
      iter    40/100 done | step  4100 | w        1.23983 | loss        0.0784 | grad       -0.1227
      iter    41/100 done | step  4200 | w        1.22596 | loss        0.0632 | grad        0.1253
      iter    42/100 done | step  4300 | w        1.23038 | loss        0.0280 | grad       -0.1657
      iter    43/100 done | step  4400 | w        1.23377 | loss        0.0390 | grad       -0.1455
      iter    44/100 done | step  4500 | w        1.22033 | loss        0.0512 | grad       -0.1874
      iter    45/100 done | step  4600 | w        1.22807 | loss        0.0153 | grad        0.0763
      iter    46/100 done | step  4700 | w        1.23235 | loss        0.0493 | grad       -0.2164
      iter    47/100 done | step  4800 | w        1.22289 | loss        0.0420 | grad       -0.1002
      iter    48/100 done | step  4900 | w        1.22318 | loss        0.0355 | grad        0.0695
      iter    49/100 done | step  5000 | w        1.22875 | loss        0.0320 | grad        0.0094
      iter    50/100 done | step  5100 | w        1.21265 | loss        0.0573 | grad        0.0005
      iter    51/100 done | step  5200 | w        1.23306 | loss        0.0531 | grad       -0.1133
      iter    52/100 done | step  5300 | w        1.23758 | loss        0.0500 | grad        0.0407
      iter    53/100 done | step  5400 | w        1.23312 | loss        0.0659 | grad        0.0761
      iter    54/100 done | step  5500 | w        1.23100 | loss        0.0128 | grad       -0.0745
      iter    55/100 done | step  5600 | w        1.23669 | loss        0.0392 | grad        0.0078
      iter    56/100 done | step  5700 | w        1.24015 | loss        0.0410 | grad       -0.0476
      iter    57/100 done | step  5800 | w        1.22657 | loss        0.0248 | grad        0.0298
      iter    58/100 done | step  5900 | w        1.22985 | loss        0.0573 | grad        0.0474
      iter    59/100 done | step  6000 | w        1.22832 | loss        0.0259 | grad       -0.0250
      iter    60/100 done | step  6100 | w        1.23766 | loss        0.0662 | grad       -0.0484
      iter    61/100 done | step  6200 | w        1.23839 | loss        0.0862 | grad       -0.2737
      iter    62/100 done | step  6300 | w        1.23734 | loss        0.0323 | grad       -0.0365
      iter    63/100 done | step  6400 | w        1.23474 | loss        0.0286 | grad       -0.0304
      iter    64/100 done | step  6500 | w        1.24034 | loss        0.0578 | grad        0.1138
      iter    65/100 done | step  6600 | w        1.23524 | loss        0.0212 | grad        0.0872
      iter    66/100 done | step  6700 | w        1.22606 | loss        0.0438 | grad        0.1876
      iter    67/100 done | step  6800 | w        1.23196 | loss        0.0306 | grad       -0.0178
      iter    68/100 done | step  6900 | w        1.23354 | loss        0.0525 | grad       -0.0704
      iter    69/100 done | step  7000 | w        1.22663 | loss        0.0442 | grad       -0.0648
      iter    70/100 done | step  7100 | w        1.22977 | loss        0.0419 | grad        0.0042
      iter    71/100 done | step  7200 | w        1.23995 | loss        0.0629 | grad        0.0045
      iter    72/100 done | step  7300 | w        1.22573 | loss        0.0197 | grad        0.0462
      iter    73/100 done | step  7400 | w        1.22610 | loss        0.0434 | grad        0.1277
      iter    74/100 done | step  7500 | w        1.22869 | loss        0.0187 | grad       -0.0309
      iter    75/100 done | step  7600 | w        1.22590 | loss        0.0532 | grad        0.0942
      iter    76/100 done | step  7700 | w        1.22963 | loss        0.0493 | grad        0.1311
      iter    77/100 done | step  7800 | w        1.23985 | loss        0.0852 | grad        0.0897
      iter    78/100 done | step  7900 | w        1.23384 | loss        0.0514 | grad        0.0544
      iter    79/100 done | step  8000 | w        1.23430 | loss        0.0295 | grad       -0.1374
      iter    80/100 done | step  8100 | w        1.22765 | loss        0.0141 | grad       -0.0433
      iter    81/100 done | step  8200 | w        1.23420 | loss        0.0216 | grad       -0.0395
      iter    82/100 done | step  8300 | w        1.22563 | loss        0.0665 | grad       -0.1015
      iter    83/100 done | step  8400 | w        1.24080 | loss        0.0364 | grad        0.0033
      iter    84/100 done | step  8500 | w        1.23774 | loss        0.0632 | grad       -0.0353
      iter    85/100 done | step  8600 | w        1.23484 | loss        0.0781 | grad       -0.0488
      iter    86/100 done | step  8700 | w        1.23275 | loss        0.0117 | grad       -0.0342
      iter    87/100 done | step  8800 | w        1.23411 | loss        0.0321 | grad        0.0455
      iter    88/100 done | step  8900 | w        1.22806 | loss        0.0688 | grad       -0.0674
      iter    89/100 done | step  9000 | w        1.23307 | loss        0.0222 | grad        0.1727
      iter    90/100 done | step  9100 | w        1.23491 | loss        0.0251 | grad       -0.0312
      iter    91/100 done | step  9200 | w        1.23106 | loss        0.0519 | grad        0.0098
      iter    92/100 done | step  9300 | w        1.23303 | loss        0.0437 | grad       -0.1625
      iter    93/100 done | step  9400 | w        1.23277 | loss        0.0708 | grad       -0.2593
      iter    94/100 done | step  9500 | w        1.22882 | loss        0.0636 | grad        0.0411
      iter    95/100 done | step  9600 | w        1.24039 | loss        0.0208 | grad       -0.0708
      iter    96/100 done | step  9700 | w        1.23499 | loss        0.0839 | grad        0.0251
      iter    97/100 done | step  9800 | w        1.23123 | loss        0.0467 | grad        0.0922
      iter    98/100 done | step  9900 | w        1.23677 | loss        0.0182 | grad        0.0165
      iter    99/100 done | step 10000 | w        1.22925 | loss        0.0492 | grad        0.0620

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(12,3))

axes[0].plot(info_dict['trace_loss_list'], '.-');
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list'], '.-');
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list'], '.-');
axes[2].set_title('w');
```

### Try with even smaller batch_size of 1

```python
# TODO write code here
```

### Discussion 3b: What happens here with this smaller batch size? Is it converging?

# Part 5: SGD for MLPClassifier

Let's revisit the XOR dataset from our previous lab, and try SGD for it.

```python
def make_xor_dataset(n_per_blob=50, stddev=0.4, random_state=0):
    random_state = np.random.RandomState(random_state)
    cov_22 = np.square(stddev) * np.eye(2)
    x_00 = random_state.multivariate_normal([-1, -1], cov_22, size=n_per_blob)
    x_01 = random_state.multivariate_normal([-1, +1], cov_22, size=n_per_blob)
    x_10 = random_state.multivariate_normal([+1, -1], cov_22, size=n_per_blob)
    x_11 = random_state.multivariate_normal([+1, +1], cov_22, size=n_per_blob)

    N = n_per_blob * 4
    x_N2 = np.vstack([x_00, x_11, x_01, x_10])
    assert x_N2.shape == (N, 2)

    y_N = np.hstack([np.ones(N//2), np.zeros(N//2)]).astype(np.int32)
    assert y_N.shape == (N,)

    # Shuffle the order
    perm_ids = random_state.permutation(N)
    x_N2 = x_N2[perm_ids].copy()
    y_N = y_N[perm_ids].copy()

    return x_N2, y_N
```

```python
x_tr_N2, y_tr_N = make_xor_dataset(n_per_blob=50)
```

```python
plt.figure(figsize=(4,4))
plt.plot(x_tr_N2[y_tr_N==0,0], x_tr_N2[y_tr_N==0,1], 'rx', label='y=0', mew=2);
plt.plot(x_tr_N2[y_tr_N==1,0], x_tr_N2[y_tr_N==1,1], 'b+', label='y=1', mew=2);

plt.xlabel('x_1');
plt.ylabel('x_2');
plt.legend(bbox_to_anchor=(1.0, 0.5));
```

# Setup: Create Utility function for visualizing classifier predictions

You do NOT need to understand the details of this function. We'll just use it as is.

```python
def plot_pretty_probabilities_for_clf(
        clf,
        ax=None,
        x1_grid=(-2.1, 2.1, 50), x2_grid=(-2.1, 2.1, 50),
        x1_ticks=[-1, 0, 1], x2_ticks=[-1, 0, 1],
        do_show_colorbar=False,
        c_ticks=np.asarray([0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]),
        c_num_levels=100,
        ):
    ''' Display predicted probabilities from classifier as color contour plot

    Args
    ----
    clf : sklearn object that implements classifier API
    ax : matplotlib axis handle, or None
        If provided, will use axis handle as primary axis to plot on.
        If None, will use the current handle, or make new one as necessary.
    x1_grid : tuple-like or array-like
        If tuple of length 3, interpreted as args to np.linspace
        Otherwise, cast to array and assumed to be a 1d grid of x1 values
    x2_grid : tuple-like or array-like
        If tuple of length 3, interpreted as args to np.linspace
        Otherwise, cast to array and assumed to be a 1d grid of x2 values

    '''
    # Activate the current axis, if necessary
    if ax is None:
        cur_ax = plt.gca()
    else:
        cur_ax = ax
        plt.sca(cur_ax)

    # Create dense grid of x1 and x2 values
    # useful for visualizing probabilities
    if isinstance(x1_grid, tuple) and len(x1_grid) == 3:
        x1_grid = np.linspace(x1_grid[0], x1_grid[1], x1_grid[2])
    if isinstance(x2_grid, tuple) and len(x2_grid) == 3:
        x2_grid = np.linspace(x2_grid[0], x2_grid[1], x2_grid[2])
    x1_grid = np.asarray(x1_grid).flatten()
    x2_grid = np.asarray(x2_grid).flatten()

    c_levels = np.linspace(0.0, 1.0, c_num_levels)

    # Get regular grid of G x H points, where each point is an (x1, x2) location
    G = x1_grid.size
    H = x2_grid.size
    x1_GH, x2_GH = np.meshgrid(x1_grid, x2_grid)

    # Combine the x1 and x2 values into one array
    # Flattened into M = G x H rows
    # Each row of x_M2 is a 2D vector [x_m1, x_m2]
    x_M2 = np.hstack([x1_GH.flatten()[:,np.newaxis], x2_GH.flatten()[:,np.newaxis]])

    # Predict proba for each point in the flattened grid
    yproba1_M = clf.predict_proba(x_M2)[:,1]
    # Reshape the M probas into the GxH 2D field
    yproba1_GH = np.reshape(yproba1_M, x1_GH.shape)
    # Contour plot
    cmap = plt.cm.RdYlBu
    my_contourf_h = plt.contourf(
            x1_GH, x2_GH, yproba1_GH,
            levels=c_levels, vmin=0, vmax=1.0,
            cmap=cmap, alpha=0.5)

    # Edit the ticks observed
    if x1_ticks is not None:
        plt.xticks(x1_ticks, x1_ticks);
    if x2_ticks is not None:
        plt.yticks(x2_ticks, x2_ticks);
    if do_show_colorbar:
        left, bottom, width, height = plt.gca().get_position().bounds
        cax = plt.gcf().add_axes([left + 1.1*width, bottom, 0.03, height])
        plt.colorbar(my_contourf_h, orientation='vertical', cax=cax, ticks=c_ticks);
        plt.sca(cur_ax);
```

## Try an MLP with 2 hidden units, batch size of 25

This time, we'll use SGD as built in solver

```python
mlp_2hidden_run5 = sklearn.neural_network.MLPClassifier(
    hidden_layer_sizes=[2],
    activation='relu',
    solver='sgd',
    learning_rate_init=0.1,
    random_state=5,
    batch_size=25,
    )
```

```python
# Fit the model to training data

mlp_2hidden_run5.fit(x_tr_N2, y_tr_N)
```

    MLPClassifier(batch_size=25, hidden_layer_sizes=[2], learning_rate_init=0.1,
                  random_state=5, solver='sgd')

### Visualize the results using our utility function

```python
plt.figure(figsize=(4,4))
plot_pretty_probabilities_for_clf(mlp_2hidden_run5, do_show_colorbar=True, ax=plt.gca());

plt.plot(x_tr_N2[y_tr_N==0,0], x_tr_N2[y_tr_N==0,1], 'rx', label='y=0', mew=2);
plt.plot(x_tr_N2[y_tr_N==1,0], x_tr_N2[y_tr_N==1,1], 'b+', label='y=1', mew=2);
```

```python
### Visualize the trace of the loss

plt.plot(mlp_2hidden_run5.loss_curve_, 'k.-')
```

# Try several trials with batch_size of 25

```python
for trial in [1, 2, 3, 4, 5]:
    mlp_2hidden_batchsize25 = sklearn.neural_network.MLPClassifier(
        hidden_layer_sizes=[2],
        activation='relu',
        solver='sgd',
        learning_rate_init=0.05,
        random_state=trial,
        batch_size=25,
        )

    # Fit the model to training data
    mlp_2hidden_batchsize25.fit(x_tr_N2, y_tr_N)

    # Visualize the trace of the loss
    plt.plot(mlp_2hidden_batchsize25.loss_curve_, '.-', label='trial %d' % trial)

plt.legend();
```

### Now try several trials with a batch_size of 1

```python
for trial in [1, 2, 3, 4, 5]:
    mlp_2hidden_batchsize1 = sklearn.neural_network.MLPClassifier(
        hidden_layer_sizes=[2],
        activation='relu',
        solver='sgd',
        learning_rate_init=0.05,
        random_state=trial,
        batch_size=1,
        )

    # Fit the model to training data
    mlp_2hidden_batchsize1.fit(x_tr_N2, y_tr_N)

    # Visualize the trace of the loss
    plt.plot(mlp_2hidden_batchsize1.loss_curve_, '.-', label='trial %d' % trial)

plt.legend();
```

### Discussion 4a: What is happening here?

```python

```

```python

```


## First Assignment

#### 1) Apply the appropriate string methods to the **x** variable (as '.upper') to change it exactly to: "$Dichlorodiphenyltrichloroethane$".

```python
import numpy as np
import pandas as pd
```

```python
x = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"
```

```python
print(x[34:-1].capitalize().replace(' ', ''))
```

    Dichlorodiphenyltrichloroethan

#### 2) Assign respectively the values: 'word', 15, 3.14 and 'list' to variables A, B, C and D in a single line of code. Then, print them in that same order on a single line separated by a space, using only one print statement.

```python
A,B,C,D = 'word', 15, 3.14, 'list'
print(A,B,C,D)
```

    word 15 3.14 list

#### 3) Use the **input()** function to receive an input in the form **'68.4 1.71'**, that is, two floating point numbers in a line separated by space. Then, assign these numbers to the variables **w** and **h** respectively, which represent an individual's weight and height (hint: take a look at the '.split()' method). With this data, calculate the individual's Body Mass Index (BMI) from the following relationship:

\begin{equation}
BMI = \dfrac{weight}{height^2}
\end{equation}

```python
w,h = input("Enter your weight and height").split(' ',-1)

print(f"Your weight is {w} your height is {h} ")

BMI = float(w)/(float(h)**2)
print('Your body mass index (BMI) is {}'.format(BMI))
```

    Enter your weight and height68.4 1.71
    Your weight is 68.4 your height is 1.71
    Your body mass index (BMI) is 23.39181286549708

#### This value can also be classified according to ranges of values, following to the table below. Use conditional structures to classify and print the classification assigned to the individual.

<\center>

(source: https://healthtravelguide.com/bmi-calculator/)

```python
if BMI < 18.5:
    print('Underweight')
elif BMI >=18.5 and BMI <25.0:
    print('Normal weight')
elif BMI >=25.0 and BMI <=29.9:
    print('Pre-obesity')
elif BMI > 29.9:
    print('Obese')
else:
    print(f'BMI {BMI} not valid')
```

    Normal weight

#### 4) Receive an integer as an input and, using a loop, calculate the factorial of this number, that is, the product of all the integers from one to the number provided.

```python
number = int(input("Write an integer: "))
fac = 1
n=1
while n <= number:
    fac = fac * n
    n=n+1
print(fac)
```

    Write an integer: 5
    120

```python
#alternative
number = int(input("Write an integer: "))
fac = 1
for n in range(1,number+1):
    fac =fac * n #das gleiche wie 'fac *= n'
print(fac)
```

    Write an integer: 4
    24

#### 5) Using a while loop and the input function, read an indefinite number of integers until the number read is -1. Present the sum of all these numbers in the form of a print, excluding the -1 read at the end.

```python
n=0
summe=0
while n != -1:
    n = int(input("Write a number: "))
    summe = n + summe
summe+=1
print(summe)
```

    Write a number: 2
    Write a number: 3
    Write a number: 54
    Write a number: 6
    Write a number: 7
    Write a number: 8
    Write a number: -2
    Write a number: -1
    78

#### 6) Read the **first name** of an employee, his **amount of hours worked** and the **salary per hour** in a single line separated by commas. Next, calculate the **total salary** for this employee and show it to two decimal places.

```python
name,hours,salary_hour=input('Write your name, whole hours worked and salary per hour, separated by ","').split(',')
total_salary = int(hours) * float(salary_hour)
print(f"{name}'s total salary with {int(hours)} hours is {total_salary:.2f} Euros")
```

    Write your name, whole hours worked and salary per hour, separated by ","Luana,20,12
    Luana's total salary with 20 hours is 240.00 Euros

#### 7) Read three floating point values **A**, **B** and **C** respectively. Then calculate itens a, b, c, d and e:

```python
A,B,C = input('Enter A,B,C: ').split(',')
A=float(A)
B=float(B)
C=float(C)
```

    Enter A,B,C: 3,6,8

    a) the area of the triangle rectangle with A as the base and C as the height.

```python
area_triangle = 1/2 * A  * C
print(area_triangle)
```

    12.0

    b) the area of the circle of radius C. (pi = 3.14159)

```python
area_circle = C**2 * np.pi
print(area_circle)
```

    201.06192982974676

    c) the area of the trapezoid that has A and B for bases and C for height.

```python
area_trapezoid = (A+B)/2 * C
print(area_trapezoid)
```

    36.0

    d) the area of the square that has side B.

```python
area_square = B**2
print(area_square)
```

    36.0

    e) the area of the rectangle that has sides A and B.

```python
area_rectangle = A * B
print(area_rectangle)
```

    18.0

#### 8) Read **the values a, b and c** and calculate the **roots of the second degree equation** $ax^{2}+bx+c=0$ using [this formula](https://en.wikipedia.org/wiki/Quadratic_equation). If it is not possible to calculate the roots, display the message **“There are no real roots”**.

```python
def quad_roots(a,b,c):
    N1=0
    N2=0
    w=b**2-4*a*c
    if w>=0:
        N1=(-b+np.sqrt(w))/(2*a)
        N2=(-b-np.sqrt(w))/(2*a)
        print(f"Roots are {N1} and {N2}")
        roots=True
    else:
        print("There are no real roots.")
        roots=False
    return N1,N2,roots
a,b,c = input('Enter a,b,c: ').split(',')
a=float(a);b=float(b);c=float(c)
```

    Enter a,b,c: -2,5,1

```python
quad_roots(a,b,c)
```

    Roots are -0.18614066163450715 and 2.686140661634507

    (-0.18614066163450715, 2.686140661634507, True)

#### 9) Read four floating point numerical values corresponding to the coordinates of two geographical coordinates in the cartesian plane. Each point will come in a line with its coordinates separated by space. Then calculate and show the distance between these two points.

(obs: $d=\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2}$)

```python
point_1 = input('Enter the coordinates of point 1 x y: ').split(' ')
x1 = float(point_1[0])
y1 = float(point_1[1])
point_2 = input('Enter the coordinates of point 2 x y: ').split(' ')
x2 = float(point_2[0])
y2 = float(point_2[1])
d = np.sqrt((x2-x1)**2+(y2-y1)**2)
print(f'The distance is {d:.2f}')
```

    Enter the coordinates of point 1 x y: 2 7
    Enter the coordinates of point 2 x y: 4 9
    The distance is 2.83

#### 10) Read **two floating point numbers** on a line that represent **coordinates of a cartesian point**. With this, use **conditional structures** to determine if you are at the origin, printing the message **'origin'**; in one of the axes, printing **'x axis'** or **'y axis'**; or in one of the four quadrants, printing **'q1'**, **'q2**', **'q3'** or **'q4'**.

```python
def read_node():
    point = input('Enter the coordinates of point x y: ').split(' ')
    x = float(point[0])
    y = float(point[1])
    return x,y

def check_location(x,y):
    if x==0 and y==0:
        loc='Origin'
    elif x==0 and y!=0:
        loc='y axis'
    elif x!=0 and y==0:
        loc='x axis'
    elif x>0 and y>0:
        loc='q1'
    elif x<0 and y>0:
        loc='q2'
    elif x<0 and y<0:
        loc='q3'
    elif x>0 and y<0:
        loc='q4'
    return loc
```

```python
x,y = read_node()
loc=check_location(x,y)
print(loc)
```

    Enter the coordinates of point x y: 6 2
    q1

#### 11) Read an integer that represents a phone code for international dialing.
#### Then, inform to which country the code belongs to, considering the generated table below:
(You just need to consider the first 10 entries)

```python
import pandas as pd
df = pd.read_html('https://en.wikipedia.org/wiki/Telephone_numbers_in_Europe')[1]
df = df.iloc[:,:2]
df.head(20)
```

      Country
      Country calling code

      0
      Austria
      43

      1
      Belgium
      32

      2
      Bulgaria
      359

      3
      Croatia
      385

      4
      Cyprus
      357

      5
      Czech Republic
      420

      6
      Denmark
      45

      7
      Estonia
      372

      8
      Finland
      358

      9
      France
      33

      10
      Germany
      49

      11
      Greece
      30

      12
      Hungary
      36

      13
      Iceland
      354

      14
      Ireland
      353

      15
      Italy
      39

      16
      Latvia
      371

      17
      Liechtenstein
      423

      18
      Lithuania
      370

      19
      Luxembourg
      352

```python
code=int(input('Enter an integer: '))
country=df[df["Country calling code"]==str(code)]["Country"].item()
print(country)
```

    Enter an integer: 371
    Latvia

#### 12) Write a piece of code that reads 6 numbers in a row. Next, show the number of positive values entered. On the next line, print the average of the values to one decimal place.

```python
a = input("Write 6 numbers: ").split(',')
positive=0
summe=0
for number in a:
    if int(number)>0:
        positive=positive+1
        summe=summe+int(number)
print(f'The count of positive numbers is {positive}')
average=summe/positive
print(round(average,1))
```

    Write 6 numbers: -2,3,5,71,901,-45
    The count of positive numbers is 4
    245.0

#### 13) Read an integer **N**. Then print the **square of each of the even values**, from 1 to N, including N, if applicable, arranged one per line.

```python
N = input('Write a number: ')
for x in range(1,int(N)+1):
    if x %2==0:
        x=x**2
        print(x)
```

    Write a number: 12
    4
    16
    36
    64
    100
    144

#### 14) Using **input()**, read an integer and print its classification as **'even / odd'** and **'positive / negative'** . The two classes for the number must be printed on the same line separated by a space. In the case of zero, print only **'null'**.

```python
L = int(input('Write a number: '))
if L==0:
    print('null')
else:
    if L %2==0:
        print('even')
    else:
        print('odd')
    #------
    if L<0:
        print('negative')
    else:
        print('positive')
```

    Write a number: 302
    even
    positive

## Challenge
#### 15) Ordering problems are recurrent in the history of programming. Over time, several algorithms have been developed to fulfill this function. The simplest of these algorithms is the [**Bubble Sort**](https://en.wikipedia.org/wiki/Bubble_sort), which is based on comparisons of elements two by two in a loop of passes through the elements. Your mission, if you decide to accept it, will be to input six whole numbers ramdonly ordered. Then implement the **Bubble Sort** principle to order these six numbers **using only loops and conditionals**.
#### At the end, print the six numbers in ascending order on a single line separated by spaces.

```python
liste=input('Write six whole numbers: ').split(',')
test=1
while test>0:
    test=0
    for i in range(len(liste)-1):
        a=int(liste[i])
        b=int(liste[i+1])
        if a>b:
            liste[i]=b
            liste[i+1]=a
            test=test+1
print(liste)
```

    Write six whole numbers: 82,405,2,1,56,734
    [1, 2, 56, 82, 405, '734']

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint
```

### O Metódo de Euler Explicito ou Forward Euler

#### Expansão em Taylor de uma função y

Expansão em Série de Taylor de $y(t)$ centrada em $t_0$ é dada por
$$
y(t) = \sum_{n=0}^{\infty} \frac{y^{(n)}(t_0)}{n!}(t-t_0)^n
$$
##### Expansão de y até a primeira derivada
Seja $h = t_n - t_{n-1}$
$$
y(t_{k+1}) = y(t_k) + y'(t_k)h + \mathcal{O}(h^2)\\
$$
O metódo de Euler explicito é um método recursivo de solução de equações diferenciais ordinárias, e consiste em utilizar a aproximação por Taylor e ignorar o erro $\mathcal{O}(h^2)$, nos dando:

$$
y_{n+1} \approx u_{n+1} = u_n + f(u_n,t_n) \cdot (t_{n+1} - t_n)
$$
Com $y_n = y(t_n)$ a solução analitica no ponto $t_n$ e $u_n$ a aproximação númerica, $f(a,b)$ a derivada de $a$ em $b$

```python
## F: Função derivada da função que queremos encontrar
## t0: tempo inicial
## y0: ponto inicial
## ts: range de tempo
## p: parametros particulares de cada modelo
def f_euler(F, y0, ts, p = 0):
    ys = [y0]
    h = ts[1]-ts[0]
    for tnext in ts[1:]:
        ynext = ys[-1] + F(ys[-1],tnext,p)*h
        ys.append(ynext)
        t = tnext
    return np.array(ys)
```

### O metódo de Runge-Kutta de segunda ordem
Enquanto o metódo de Euler aproxima a solução em um ponto andando na tangente daquele ponto, o metódo de Runge-Kutta de segunda ordem aproxima o mesmo ponto andando na média entre a tangente no ponto e a tangente no ponto futuro.

Seja
$$
k_1 = f(u_n,t_n)\\
k_2 = f(u_n + hk_1,t_{n+1})
$$
Então $k_1$ é a derivada no ponto e $k_2$ é a derivada no ponto futuro, aproximando a mesma pelo metódo de Euler

O passo para Runge-Kutta será então a média entre estas duas, e ficará:
$$
y_{n+1} \approx u_{n+1} = u_n + h \frac{k_1 + k_2}{2}
$$

```python
def rk_2(F, y0, ts, p = 0):
    ys = [y0]
    t = ts[0]
    h = ts[1] - ts[0]
    for tnext in ts:
        k1 = F(ys[-1], t, p)
        k2 = F(ys[-1] + h*k1,tnext, p)
        ynext = ys[-1] + h * (k1+k2) / 2.0
        ys.append(ynext)
        t = tnext
    return np.array(ys[:-1])
```

Testando para a EDO
$$ \begin{cases} y'(t) = - y(t) + 2\sin(t^2) \\ y(0) = 1.2\end{cases} $$

```python
def F(y,t,p = 0):
    return -y + 2*np.sin(t**2)
```

```python
## Definindo o dominio
ts = np.linspace(-5,5,500)
y0 = 1.2
## Criando a lista para Runge-Kutta 2nd order
ys = rk_2(F,y0,ts)

## Criando a lista para Euler Explicito
ys2 = f_euler(F,y0,ts)

#ans = f(y,ts)

plt.plot(ts,ys,label='RK')
plt.plot(ts,ys2,label='Explicito')
plt.legend()
plt.show()
```

### O metódo de Euler converge - Um pouco de Análise

### Definições:
Seja $\frac{\mathrm{d}y}{\mathrm{d}t} = f(y,t)$

Seja $t \in \mathbb{N}$ o número de 'tempos' no dominio e $t^*$ o tempo final e $\lfloor \cdot \rfloor$ a função `floor`, que retorna a parte inteira.

Seja $h \in \mathbb{R}, h > 0$, o 'tamanho' de cada partição, ou seja $h = t_{n+1} - t_n$

Podemos então definir $n$, tal que $n$ assume valores no conjunto $\{0, \dots , \lfloor \frac{t^*}{h} \rfloor\}$

Seja $\lVert \cdot \rVert$ uma norma definida no espaço

Seja $y_n$ o valor real (analitico) da função **$y$** no ponto $t_n$, ou seja $y_n = y(t_n)$

Seja $u_n$ o valor númerico aproximado da função $y$ no ponto $t_n$ pelo método de Euler, ou seja $u_{n+1} = u_{n} + f(u_{n},t_{n})\cdot h$

Uma função $f$ é dita Lipschitz se satisfaz a condição de Lipschitz: $\exists M \in \mathbb{R}; \lVert f(x_1) - f(x_2) \rVert \leq M\cdot\lVert x_1 - x_2\rVert$

Um metódo é dito convergente se:
$$
\lim_{h\to 0^+} \max_{n=0, \dots , \lfloor \frac{t^*}{h} \rfloor} \lVert u_n - y_n \rVert = 0
$$

Ou seja, sempre que a malha for refinida, a solução númerica em um ponto se aproxima da solução analitica neste ponto

### Teorema: O metódo de Euler converge

#### Prova

Tomemos $f(y,t)$ analitica, ou seja, pode ser representada pela série de Taylor centrada em um ponto $t_0$ e é Lipschitz.

$f(y,t)$ analitica implica $y$ analitica.

Vamos definir $err_n = u_n - y_n$, nosso erro númerico, então queremos provar
$$
\lim_{h\to 0^+} \max_{n=0, \dots , \lfloor \frac{t^*}{h} \rfloor} \lVert err_n \rVert = 0
$$
Expandindo nossa solução $y$ da equação diferencial por Taylor:
$$
y_{n+1} = y_n + hf(y_n,t_n)+\mathcal{O}(h^2) \tag{1}
$$
Como $y$ é analitica, então sua derivada é contínua, logo pelo `Teorema do Valor Extremo`, dado uma vizinhança em torno de $t_n$ o termo $\mathcal{O}(h^2)$ é limitado $\forall h>0$ e $n \leq \lfloor t^*/h \rfloor$ por $M>0, M \in \mathbb{R}$, e pela propriedade arquimediana do corpo do reais $\exists c \in \mathbb{R}, c>0; c\cdot h^2 \geq M$, portanto podemos limitar $\mathcal{O}(h^2)$ por $ch^2, c>0$.

Agora vamos fazer $err_{n+1} = u_{n+1} - y_{n+1}$ usando a expansão em Taylor $y_{n+1}$ e Euler em $u_{n+1}$
$$
\begin{align}
err_{n+1} &= u_{n+1} - y_{n+1}\\
&= u_n + h(f(u_n,t_n)) - y_n - h(f(yn,tn) + \mathcal{O}(h^2)\\
&= \underbrace{u_n - y_n}_{err_n} + h\left(f(u_n,t_n) - f(y_n,t_n)\right) + \mathcal{O}(h^2)\\
&= err_n + h\left(f(u_n,t_n) - f(y_n,t_n)\right) + \mathcal{O}(h^2)\\
\end{align}
$$
Daqui podemos perceber que o erro no passo seguinte depende também do erro anterior já cometido

E segue do fato de que $\mathcal{O}(h^2)$ é limitada, com uma cota superior $ch^2$ e da desigualdade triangular
$$
\lVert err_{n+1} \rVert \leq \lVert err_n\rVert + \lVert h\left(f(u_n,t_n) - f(y_n,t_n)\right)\rVert + \lVert ch^2 \rVert
$$
E pela condição de Lipschitz
$$
\lVert f(u_n,t_n) - f(y_n,t_n) \rVert \leq \lambda\lVert u_n - y_n \rVert = \lambda\lVert err_n \rVert, \lambda > 0
$$
Então temos
$$
\lVert err_{n+1} \rVert \leq \lVert err_n\rVert + \lVert h\left(f(u_n,t_n) - f(y_n,t_n)\right)\rVert + \lVert ch^2 \rVert \leq \lVert err_n\rVert + \lambda h\lVert err_n \rVert + ch^2\\$$
$\therefore$
$$
\lVert err_{n+1} \rVert \leq  (1+h\lambda)\lVert err_n \rVert + ch^2 \tag{2}
$$

---
Agora vamos propor:
$$
\lVert err_n \rVert \leq \frac{c}{\lambda}h[(1+h\lambda)^n - 1]
$$
#### Demonstração: Indução em n
Para $n = 0$
$$
\lVert err_0 \rVert \leq \frac{c}{\lambda}h[(1+h\lambda)^0 - 1] = \frac{c}{\lambda}h[1 - 1] = 0\\
err_0 = u_0 - y_0 = 0, \text{pois é a condição inicial}
$$
Temos portanto nossa hipotese de indução, vale para $n=k$, vamos para o passo indutivo: $n = k+1$. Da equação 2, temos:
$$
\lVert err_{k+1}\rVert \leq (1+h\lambda)\lVert err_k \rVert + ch^2
$$
E pela hipotese de indução
$$
\lVert err_k \rVert\leq \frac{c}{\lambda}h[(1+h\lambda)^k - 1]
$$
Logo
$$
\lVert err_{k+1} \rVert \leq (1+h\lambda)\frac{c}{\lambda}h[(1+h\lambda)^k - 1] + ch^2
$$
Desenvolvendo o termo da direita:
$$
\begin{align}
(1+h\lambda)\frac{c}{\lambda}h[(1+h\lambda)^k - 1] + ch^2 &= \frac{c}{\lambda}h[(1+h\lambda)^{k+1} - (1+h\lambda)] +ch^2\\
&= \frac{c}{\lambda}h(1+h\lambda)^{k+1} - \frac{c}{\lambda}h(1+h\lambda) +ch^2\\
&= \frac{c}{\lambda}h(1+h\lambda)^{k+1} - \frac{c}{\lambda}h - \frac{c}{\lambda}h^2\lambda + ch^2\\
&= \frac{c}{\lambda}h(1+h\lambda)^{k+1} - \frac{c}{\lambda}h\\
&= \frac{c}{\lambda}h[(1+h\lambda)^{k+1} - 1]
\end{align}
$$
Portanto
$$
\lVert err_{k+1} \rVert \leq \frac{c}{\lambda}h[(1+h\lambda)^{k+1} - 1]
$$
E o passo indutivo vale. Logo pelo principio de indução finita temos:
$$
\lVert err_n \rVert \leq \frac{c}{\lambda}h[(1+h\lambda)^n - 1] \tag{3}
$$

---
Como $h\lambda >0$, então temos $(1+h\lambda) < e^{h\lambda}$ e portanto $(1+h\lambda)^n < e^{nh\lambda}$, e n assume valor máximo em $n = \lfloor t^*/h \rfloor $, portanto:
$$(1+h\lambda)^n < e^{\lfloor t^*/h \rfloor h\lambda} \leq e^{t^*\lambda}$$
Substituindo na inequação 3 para $err_n$, teremos:
$$
\lVert err_n \rVert \leq \frac{c}{\lambda}h[e^{t^*\lambda} - 1]
$$
Passando o limite $h\to 0$, teremos:
$$
\lim_{h\to 0}\lVert err_n \rVert \leq \frac{c}{\lambda}h[e^{t^*\lambda} - 1] = 0\\
\therefore
\lim_{h\to 0}\lVert err_n \rVert = 0
$$
Portanto o Metódo de Euler converge para toda função Lipschitz. Q.E.D.

### Visualizando o teorema

Vamos plotar a solução da equação diferencial $y' = sin(t^2) - y$ com um refinamento da malha cada vez melhor e visualizar a convergência do metódo

Plotaremos também um gráfico com a evolução do erro relativo entre a solução de malha mais fina e todas as soluções anteriores

```python
## Equação Diferencial
def F(y,t,p=0):
    return -y + 2*np.sin(t**2)

# Criação dos dominios com vários h diferentes
ts = np.array([np.linspace(-10,10,i) for i in np.arange(50,300,63)])

# Condição inicial
y0 = 1.2
# Preparação da listas para plotagem
ys_e = np.array([f_euler(F,y0,i) for i in ts ])
# Estilo das curvas
lstyle = ['--','-.',':','-']

# Plot do gráfico de solução
plt.figure(figsize=(15,7))
for i in range(len(ts)):
    plt.plot(ts[i],ys_e[i], ls = lstyle[i], label='$h = '+ str("{0:.2f}".format(20.0/len(ts[i])) +'$'))
plt.title('Visualização da convergência do Metódo de Euler')
plt.xlabel('t')
plt.ylabel('y(t)')
plt.legend()
plt.show()

## Criando os arrays de erro
hs = [0.4,0.18,0.11]
ans = [[],[],[]]
for i in range(len(ys_e[:-1])):
    n = np.floor(hs[i]/0.08)
    for j in range(len(ys_e[i])):
        try: ans[i].append(ys_e[-1][n*j])
        except: ans[i].append(ys_e[-1][-1])
for i in range(len(ans)):
    ans[i] = np.array(ans[i])

err = np.array([abs(j - i) for i,j in zip(ys_e,ans)])
plt.figure(figsize=(15,7))
for i in range(len(ts)-1):
    plt.plot(ts[i],err[i], ls = lstyle[i], label='$h = '+ str("{0:.2f}".format(20.0/len(ts[i])) +'$'))
plt.title('Visualização do Erro da solução mais convergida em relação às outras soluções')
plt.xlabel('t')
plt.ylabel('err(y)')
plt.legend()
plt.show()
```

### Gráficos de Convergência de Runge-Kutta

```python
# Preparação da listas para plotagem
ys_rk = np.array([rk_2(F,y0,i) for i in ts ])
# Estilo das curvas
lstyle = ['--','-.',':','-']

# Plot do gráfico de solução
plt.figure(figsize=(15,7))
for i in range(len(ts)):
    plt.plot(ts[i],ys_rk[i], ls = lstyle[i], label='$h = '+ str("{0:.2f}".format(20.0/len(ts[i])) +'$'))
plt.title('Visualização da convergência de Runge-Kutta')
plt.xlabel('t')
plt.ylabel('y(t)')
plt.legend()
plt.show()
```

```python
plt.plot(ts[-1],abs(ys_e[-1]-ys_rk[-1]))
```

##### Teorema: O metódo de Euler converge

###### Prova

Tomemos $f(y,t)$ analitica, ou seja, pode ser representada pela série de Taylor centrada em um ponto $t_0$ e é Lipschitz continua.

$f(y,t)$ analitica implica $y$ analitica.

Vamos definir $err_n = u_n - y_n$, nosso erro númerico, então queremos provar
$$
\lim_{h\to 0^+} \max_{n=0, \dots , \lfloor \frac{t^*}{h} \rfloor} \lVert err_n \rVert = 0
$$
Expandindo nossa solução $y$ da equação diferencial por Taylor:
$$
y_{n+1} = y_n + hf(y_n,t_n)+\mathcal{O}(h^2)
$$
Como $y$ é analitica, então sua derivada é contínua, logo pelo `Teorema do Valor Extremo`, dado uma vizinhança em torno de $t_n$ o termo $\mathcal{O}(h^2)$ é limitado $\forall h>0$ e $n \leq \lfloor t^*/h \rfloor$ por $M>0, M \in \mathbb{R}$, e pela propriedade arquimediana do corpo do reais $\exists c \in \mathbb{R}, c>0; c\cdot h^2 \geq M$, portanto podemos limitar $\mathcal{O}(h^2)$ por $ch^2, c>0$.

Agora vamos fazer $err_{n+1} = u_{n+1} - y_{n+1}$ usando a expansão em Taylor e Euler em $u_n$
$$
\begin{align}
err_{n+1} &= u_{n+1} - y_{n+1}\\
&= u_n + h(f(u_n,t_n)) - y_n - h(f(yn,tn) + \mathcal{O}(h^2)\\
&= \underbrace{u_n - y_n}_{err_n} + h\left(f(u_n,t_n) - f(y_n,t_n)\right) + \mathcal{O}(h^2)\\
&= err_n + h\left(f(u_n,t_n) - f(y_n,t_n)\right) + \mathcal{O}(h^2)\\
\end{align}
$$
Daqui podemos perceber que o erro no passo seguinte depende também do erro anterior já cometido

E segue do limite superior para $\mathcal{O}(h^2)$ e da desigualdade triangular
$$
\lVert err_{n+1} \rVert \leq \lVert err_n\rVert + \lVert h\left(f(u_n,t_n) - f(y_n,t_n)\right)\rVert + \lVert ch^2 \rVert
$$
E pela condição de Lipschitz
$$
\lVert f(u_n,t_n) - f(y_n,t_n) \rVert \leq \lambda\lVert u_n - y_n \rVert = \lambda\lVert err_n \rVert, \lambda > 0
$$
Então temos
$$
\lVert err_{n+1} \rVert \leq \lVert err_n\rVert + \lVert h\left(f(u_n,t_n) - f(y_n,t_n)\right)\rVert + \lVert ch^2 \rVert \leq \lVert err_n\rVert + \lambda h\lVert err_n \rVert + ch^2\\
\therefore
\lVert err_{n+1} \rVert \leq \lVert err_n\rVert + \lVert h\left(f(u_n,t_n) - f(y_n,t_n)\right)\rVert + \lVert ch^2 \rVert \leq  (1+h\lambda)\lVert err_n \rVert + ch^2
$$

Assumiremos (provar mais tarde)
$$
\lVert err_n \rVert \leq \frac{c}{\lambda}h[(1+h\lambda)^n - 1]
$$
Como $h\lambda >0$, então temos $(1+h\lambda) < e^{h\lambda}$ e portanto $(1+h\lambda)^n < e^{nh\lambda}$, e n assume valor máximo em $n = \lfloor t^*/h \rfloor $, portanto: $(1+h\lambda)^n < e^{\lfloor t^*/h \rfloor h\lambda} = e^{t^*\lambda}$

Substituindo na inequação anterior para $err_n$, teremos:
$$
\lVert err_n \rVert \leq \frac{c}{\lambda}h[e^{t^*\lambda} - 1]
$$
Passando o limite $h\to 0$, teremos:
$$
\lim_{h\to 0}\lVert err_n \rVert \leq \frac{c}{\lambda}h[e^{t^*\lambda} - 1] = 0\\
\therefore
\lim_{h\to 0}\lVert err_n \rVert = 0
$$
O Metódo de Euler converger. Q.E.D.

```python

```

```python

```

### O modelo presa-predador de Lotka-Volterra (em construção)

O modelo é dado pelas EDOs

$$
\begin{cases}
\frac{\mathrm{d}x}{\mathrm{d}t} = (\lambda - by)x\\
\frac{\mathrm{d}y}{\mathrm{d}t} = (-\mu + cx)y\\
\end{cases}
$$
Com $\lambda, \mu, b, c$ todos reais positivos e $x$ representando a população de presas e $y$ a população de predadores

Como já visto anteriormente, vamos tratar de forma vetorial este problema, sendo:

$$
v = \begin{bmatrix}
\frac{\mathrm{d}x}{\mathrm{d}t} \\
\frac{\mathrm{d}y}{\mathrm{d}t}
\end{bmatrix}
$$

E sendo $D$ o operador linear de derivada, teremos:

$$
Dv = \begin{bmatrix}
(\lambda - by)x\\
(-\mu + cx)y
\end{bmatrix}
$$

E então podemos encontrar a solução aplicando algum metódo númerico

```python
## Parametros:
## v: vetor dos pontos iniciais
## p[l,b,m,c]: uma lista com os parametros do modelo
### l: lambda
### b: b
### m: mu
### c: c

def model(v,t,p = 0):
    if p == 0: p = [1,1,1,1]
    return np.array([(p[0]-p[1]*v[1])*v[0],(p[3]*v[0]-p[2])*v[1]])
```

```python
# Parametros de ajusate
ts = np.linspace(0,30,500)
y0 = [2,1]
```

```python
ys_e = f_euler(model,y0,ts)

plt.figure(figsize=(10,5))
plt.plot(ts,ys_e)
plt.title('Modelo Presa-Predador - Solução com Euler')
plt.legend(['Presa', 'Predador'])
plt.xlabel('$t$')
plt.ylabel('População $y(t)$')
plt.grid(alpha = 0.5)
plt.show()
```

```python
ys_rk = rk_2(model,y0,ts)

plt.figure(figsize=(10,5))
plt.plot(ts,ys_rk)
plt.title('Modelo Presa-Predador - Solução com Runge-Kutta')
plt.legend(['Presa', 'Predador'])
plt.xlabel('$t$')
plt.ylabel('População $y(t)$')
plt.grid(alpha = 0.5)
plt.show()
```

```python
## F: Função derivada da função que queremos encontrar
## t0: tempo inicial
## y0: ponto inicial
## ts: range de tempo
## p: parametros particulares de cada modelo
def f2_euler(F, y0, ts, p = 0):
    ys = [y0]
    h = ts[1]-ts[0]
    for tnext in ts[1:]:
        ynext = ys[-1] + F(ys[-1],tnext,p)*h
        ys.append(ynext)
        t = tnext
    return np.array(ys)
```

```python
## Parametros:
## v: vetor dos pontos iniciais
## p[l,b,m,c]: uma lista com os parametros do modelo
### l: lambda
### b: b
### m: mu
### c: c

def model(v,t,p = 0):
    if p == 0: p = [1,1,1,1]
    return np.array([(p[0]-p[1]*v[1])*v[0],(p[3]*v[0]-p[2])*v[1]])
```

```python
# Parametros de ajusate
ts = np.linspace(0,30,500)
y0 = [2,1]

ys_e = f2_euler(model,y0,ts)

plt.figure(figsize=(10,5))
plt.plot(ts,ys_e)
plt.title('Modelo Presa-Predador - Solução com Euler')
plt.legend(['Presa', 'Predador'])
plt.xlabel('$t$')
plt.ylabel('População $y(t)$')
plt.grid(alpha = 0.5)
plt.show()
```

```python
def model2(y, t, p = 0):
    return -0.5*y
```

```python
ts = np.linspace(0,30,500)
plt.plot(ts,f2_euler(model2,2,ts))
```

```python
def EulerFW(F,y0,ts,p=0):
    ys=[y0]
    h=ts[1]-ts[0]
    tc=ts[0]
    for t in ts[1:]:
        yn=ys[-1]+h*F(ys[-1],ts,p)
        ys.append(yn)
        tc=t
    return ys
```

```python
ts = np.linspace(0,30,500)
plt.plot(ts,EulerFW(model2,2,ts))
```

```python
ts = np.linspace(0,30,500)
plt.plot(ts,EulerFW(model,[2,1],ts))
```

```python
ts = np.linspace(0,5,500)
plt.plot(ts,f2_euler(model,[2,1],ts))
```

```python
def model3(v,t,p=0):
    if p == 0: p = [1.5,1.2,1,1]
    return np.array([p[0]*np.log(v[1])-p[1]*v[1]+p[2]*np.log(v[0])+p[3]*v[0]])
```

```python
plt.plot(ts,rk_2(model3,[2,1],ts))
```

```python
model3([2,1],ts)
```

    array([1.49314718])

### Análise qualitativa da EDO

Não temos uma solução analítica do sistema de EDO, mas podemos encontrar uma relação entre as variaveis do problema, olhando para a taxa de variação de cada uma das populações da seguinte forma

\begin{equation}
\dfrac{\mathrm{d}y}{\mathrm{d}x} = \frac{\dfrac{\mathrm{d}y}{\mathrm{d}t}}{\dfrac{\mathrm{d}x}{\mathrm{d}t}} = \dfrac{(-\mu + cx)y}{(\lambda - by)x}
\end{equation}

E esta é uma equação separavel, podemos seguir com a solução:

\begin{equation}
\dfrac{(-\mu + cx)y}{y}\mathrm{d}y = \dfrac{(\lambda - by)x}{x}\mathrm{d}x
\end{equation}

Obtendo então

\begin{equation}
\int \bigg( \dfrac{\lambda - by}{y}\bigg) dy = \int \bigg( \dfrac{-\mu + cx}{x}\bigg) dx
\end{equation}

Resolvendo, temos a solução geral para o modelo:

\begin{equation}
\lambda\ln(|y|) - by = -\mu\ln(|x|) + cx + K
\end{equation}
Como as populações $x,y$ são sempre positivas, podemos reescrever
\begin{equation}
\lambda\ln(y) - by + \mu\ln(x) - cx = K
\end{equation}

Sendo $K \in \mathbb{R}$, constante em relação a cada solução.\\
Temos assim uma relação entre cada parametro e variavel do nosso problema

```python
ys_rk = f_euler(model,[40,20],ts,p=[3,1.3,2.7,0.5])

plt.figure(figsize=(10,5))
plt.plot(ts,ys_rk)
plt.title('Modelo Presa-Predador - Solução com Runge-Kutta')
plt.legend(['Presa', 'Predador'])
plt.xlabel('$t$')
plt.ylabel('População $y(t)$')
plt.grid(alpha = 0.5)
plt.show()
```

```python
## Parametros:
## v: vetor dos pontos iniciais
## p[l,b,m,c]: uma lista com os parametros do modelo
### l: lambda
### b: b
### m: mu
### c: c

def model4(v,t,p = 0):
    if p == 0: p = [3,1.3,2.7,0.5]
    return np.array([(p[0]-p[1]*v[1])*v[0],(p[3]*v[0]-p[2])*v[1]])
```

```python
ans = odeint(model4,[2,1],ts)
plt.plot(ts,ans)
```

```python
ans2 = rk_2(model4,[2,1],ts)
plt.plot(ts,ans2)
```

```python

```


# Taylor integration example: $\dot x=x^2$

Here, we will integrate the initial-value problem (IVP) defined by

$$
\begin{align}
\dot x &= x^2 \\
x(0)&=x_0
\end{align}
$$

Given a real number $A>0$, the restriction of the function $f(x)=x^2$ over the interval $I_A=(-A,A)$ satisfies the Lipschitz condition $|f(x_1)-f(x_2)| \leq 2A|x_1-x_2|$ for every $x_1,x_2 \in I_A$. Therefore, the Picard-Lindelöf theorem for ordinary differential equations guarantees there exists a $\delta>0$ such that a solution for this IVP exists and is unique for $t \in [0,\delta]$, for any $x_0 \in I_A$. Note that in this case, it is necessary to restrict the function to a bounded interval in order to obtain a Lipschitz condition, which in turn is necessary to fulfill the conditions of the Picard-Lindelöf theorem.

Now, for any initial condition $x_0>0$, $t_0=0$ the analytical solution for this problem is:

$$
x(t)=\frac{x_0}{1-x_0\cdot t}
$$

In particular, the analytical solution exhibits a divergence at $t=1/x_0$; i.e., the solution is guaranteed to exist only for $t \in [0,1/x_0)$. How does a numerical integrator behave near this divergence?

We will try two methods to integrate this problem:

+ Adaptive time-step, 4th-order, Runge-Kutta (`ODE.jl`)
+ Taylor method (`TaylorIntegration.jl`)
+ Adaptive time-step, Runge-Kutta-Fehlberg 7/8 method (`ODE.jl`)

As an initial condition to integrate this IVP, we choose $x_0=3$, since this number is not exactly representable in a binary floating-point format. Thus, any constant time-step numerical integrator should break down when integrating up to $t_\mathrm{max}=1/3$.

We start off by including the relevant packages:

```julia
using TaylorIntegration, ODE, PyPlot
```

The ODE:

```julia
diffeq(t, x) = x.^2
```

    diffeq (generic function with 1 method)

## 1. Adaptive time-step, 4th order, Runge Kutta method

We select $x_0=3$, $t_0=0$. Then, the singularity is at $t=1/3$.

```julia
@time tRK, xRK = ode45(diffeq, 3.0, [0.0, 0.34]); #warmup lap
@time tRK, xRK = ode45(diffeq, 3.0, [0.0, 0.34]);
```

    Warning: dt < minstep.  Stopping.
      1.235191 seconds (1.71 M allocations: 77.339 MB, 1.86% gc time)
    Warning: dt < minstep.  Stopping.
      0.000999 seconds (18.45 k allocations: 530.688 KB)

Plot $x$ vs $t$ (log-log):

```julia
title("x vs t (log-log)")
xlabel(L"\log_{10}(t)")
ylabel(L"\log_{10}(x(t))")

grid(true)
plot(log10(tRK[2:end]), log10(xRK[2:end]), ".-");
```

What is the final state of the system?

```julia
tRK[end], xRK[end]
```

    (0.33333423758781194,7.125446356124256e17)

Does the integrator get past the singularity?

```julia
tRK[end]>1/3
```

    true

The answer is yes! So the last value of the solution is meaningless:

```julia
xRK[end] #this value is meaningless
```

    7.125446356124256e17

How many steps did the RK integrator perform?

```julia
length(xRK)-1
```

    166

How does the numerical solution compare to the analytical solution? The analytical solution is:

```julia
exactsol(t, x0) = x0./(1.0-x0.*t) #analytical solution
```

    exactsol (generic function with 1 method)

The relative difference between the numerical and analytical solution, $\delta x$, is:

```julia
δxRK = (xRK-exactsol(tRK, 3.0))./exactsol(tRK, 3.0) #error relative to analytical solution
;
```

The $\delta x$ vs $t$ plot (semilog):

```julia
title("Relative error (semi-log)")
xlabel(L"\log_{10}(t)")
ylabel(L"\log_{10}(\delta x(t))")
grid(true)
plot(tRK, log10(abs(δxRK)), "o-");
```

This plot means that the error of the numerical solution grows systematically; and at the end of the integration, the error in the numerical solution is

```julia
(xRK[end-1]-exactsol(tRK[end-1], 3.0))
```

    5.433622064235371e17

## 2. Taylor method

Again, we select $x_0=3$, $t_0=0$. The order of the Taylor integration is $28$, and we set the absolute tolerance equal to $10^{-20}$; this value is used during each time-step in order to compute an adaptive step size. We set the maximum number of integration steps equal to the number of steps that the previous integrator did.

```julia
@time tT, xT = taylorinteg(diffeq, 3.0, 0.0, 0.34, 28, 1e-20, maxsteps=length(xRK)-1); #warmup lap
@time tT, xT = taylorinteg(diffeq, 3.0, 0.0, 0.34, 28, 1e-20, maxsteps=length(xRK)-1);
```

      0.172252 seconds (186.71 k allocations: 9.235 MB)
      0.002670 seconds (37.91 k allocations: 2.348 MB)

    WARNING: Maximum number of integration steps reached; exiting.
    WARNING: Maximum number of integration steps reached; exiting.

```julia
tT[end], xT[end]
```

    (0.3333333329479479,2.5948055925168757e9)

How many steps did the Taylor integrator perform?

```julia
length(xT)-1
```

    166

Below, we show the $x$ vs $t$ plot (log-log):

```julia
# axis([0, 0.35, -15, 10])
title("x vs t (log-log)")
xlabel(L"\log_{10}(t)")
ylabel(L"\log_{10}(x(t))")
grid(true)
plot(log10(tT[2:end]), log10(xT[2:end]), ".-");
```

Does the integrator get past the singularity?

```julia
tT[end] > 1/3
```

    false

The answer is no! Even if increase the value of the `maxsteps` keyword in `taylorinteg`, it doesn't get past the singularity!

Now, the relative difference between the numerical and analytical solution, $\delta x$, is:

```julia
δxT = (xT.-exactsol(tT, 3.0))./exactsol(tT, 3.0);
```

The $\delta x$ vs $t$ plot (logscale):

```julia
title("Relative error (semi-log)")
xlabel(L"t")
ylabel(L"\log_{10}(\delta x(t))")
grid(true)
plot(tT, log10(abs(δxT)), "o-");
```

We observe that, while the execution time is ~10 times longer wrt 4th-order RK, the numerical solution obtained by the Taylor integrator stays within $10^{-12}$ of the analytical solution, for a same number of steps.

Now, that happens if we use a higher order Runge Kutta method to integrate this problem?

## 3. Runge-Kutta-Fehlberg 7/8 method

Here we use the Runge-Kutta-Fehlberg 7/8 method, included in `ODE.jl`, to integrate the same problem as before.

```julia
@time t78, x78 = ode78(diffeq, 3.0, [0.0, 0.34]); #warmup lap
@time t78, x78 = ode78(diffeq, 3.0, [0.0, 0.34]);
```

    Warning: dt < minstep.  Stopping.
      0.164455 seconds (150.73 k allocations: 6.613 MB, 4.77% gc time)
    Warning: dt < minstep.  Stopping.
      0.000907 seconds (17.76 k allocations: 516.594 KB)

Plot $x$ vs $t$ (log-log):

```julia
title("x vs t (log-log)")
xlabel(L"\log_{10}(t)")
ylabel(L"\log_{10}(x(t))")
grid(true)
plot(log10(t78[2:end]), log10(x78[2:end]), ".-");
```

What is the final state of the system?

```julia
t78[end], x78[end]
```

    (0.3333336190078445,1.6711546943686948e18)

Does the integrator get past the singularity?

```julia
t78[end]>1/3
```

    true

The answer is yes! So the last value of the solution is meaningless:

```julia
x78[end] #this value is meaningless
```

    1.6711546943686948e18

How many steps did the RK integrator perform?

```julia
length(x78)-1
```

    91

The relative difference between the numerical and analytical solution, $\delta x$, is:

```julia
δx78 = (x78-exactsol(t78, 3.0))./exactsol(t78, 3.0) #error relative to analytical solution
;
```

The $\delta x$ vs $t$ plot (semilog):

```julia
title("Relative error (semi-log)")
xlabel(L"t")
ylabel(L"\log_{10}(\delta x(t))")
grid(true)
plot(t78, log10(abs(δx78)), "o-");
```

This time, the RKF 7/8 integrator is "only" twice as fast as the Taylor integrator, but the error continues to be greater than the error from the latter by several orders of magnitude.

## 4. Adaptive 4th-order RK, stringer tolerance

As a last example, we will integrate once again our problem using a 4th-order adaptive RK integrator, but imposing a stringer tolerance:

```julia
#@time sol = solve(ODEProblem(diffeq, 3.0), [0.0,0.34], alg=:Feagin14) #warmup lap
#@time sol = solve(ODEProblem(diffeq, 3.0), [0.0,0.34], alg=:Feagin14)
```

```julia
#@time sol = solve(ODEProblem(diffeq, 3.0), [0.0,0.33333], alg=:DP5); #warmup lap
#@time sol = solve(ODEProblem(diffeq, 3.0), [0.0,0.34], alg=:DP5);
```

```julia
@time tRK_, xRK_ = ode45(diffeq, 3.0, [0.0, 0.34], abstol=1e-8, reltol=1e-8 ); #warmup lap
@time tRK_, xRK_ = ode45(diffeq, 3.0, [0.0, 0.34], abstol=1e-8, reltol=1e-8 ); #warmup lap
;
```

    Warning: dt < minstep.  Stopping.
      0.027715 seconds (502.96 k allocations: 15.123 MB)
    Warning: dt < minstep.  Stopping.
      0.031589 seconds (502.82 k allocations: 15.115 MB, 14.73% gc time)

Now, the integrator takes 10 times longer to complete the integration than the Taylor method.

Does it get past the singularity?

```julia
tRK_[end] > 1/3
```

    true

Yes! So, once again, the last value reported by the integrator is completely meaningless. But, has it attained a higher precision than the Taylor method? Well, let's calculate once again the numerical error relative to the analytical solution:

```julia
δxo = (xo-exactsol(to, 3.0))./exactsol(to, 3.0);
```

And now, let's plot this relative error vs time:

```julia
title("Relative error (semi-log)")
xlabel(L"t")
ylabel(L"\log_{10}(\delta x(t))")
ylim(-20,20)
grid(true)
plot(to, log10(abs(δxo)), "o-");
```

The numerical error has actually gotten worse! `TaylorIntegration.jl` is indeed a really competitive package to integrate ODEs.

```julia

```


# Regression

## Linear Regression

### Step 1 : Setup the Environment

```python
# Import necessary Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
```

### Step 2 : Clean the data and Visually analyse the data

```python
x = 10 * np.random.random(50)
y = 2 * x - 1 + np.random.randn(50)
print("x = ",x)
print("y = ",y)
```

    x =  [4.71575962 2.79611957 6.53365325 6.89171715 1.79343983 3.40168302
     2.75966938 6.47687307 4.66321201 7.44681371 8.62711037 4.93215967
     0.55907761 6.34443264 1.39039105 6.18648423 7.48840008 1.55065097
     8.22250359 6.35686719 8.57282616 5.26966386 2.56339095 6.60315119
     3.15469759 9.32347228 9.20683701 2.55341251 5.05858655 5.0380703
     1.17126163 1.98061133 6.10325314 2.00837555 4.8957142  7.17552295
     5.41394894 2.23837019 5.32201578 0.22963054 7.16808765 9.33971103
     9.98976715 5.21045821 5.16935013 7.52269385 9.7892179  2.47902801
     1.63778159 6.15557107]
    y =  [10.03012272  4.49529193 12.74188777 12.72551805  1.43077755  6.65275082
      4.86431874 12.11007062  7.54992829 12.89914534 16.6046464  10.44832372
     -0.85667271 11.8215409   2.96789924 11.30315529 15.32275064  2.45224635
     16.28522892 10.90016452 16.27444692 10.60256859  5.33915333 12.59200911
      5.1195451  18.68928542 18.44462835  4.80234441  9.21489562  9.60413798
      1.95439708  2.08977758 10.94412696  2.84986311  8.16031285 13.31348924
     10.20114927  2.53016296  8.73657176 -1.51279597 13.24798197 18.51094194
     20.27965885 10.72682573 11.09500694 13.20130847 18.60031984  3.51043453
      4.14270882 10.35636608]

```python
x = 10 * np.random.random(50)
y = 2 * x - 1 + np.random.randn(50)
plt.figure(dpi=120)
sns.scatterplot(x,y)
plt.xlabel("X - InDependend Variable")
plt.ylabel("Y - Dependend Variable")
plt.show()
```

### Step 3 : Generate X "Feature matrix" and y "Target vector"

```python
print(x.shape)
print(x.ndim)
print(y.shape)
```

    (50,)
    1
    (50,)

```python
# Convert the x vector to X matrix
X=x[:,np.newaxis]
print(X.shape)
```

    (50, 1)

### Step 4 : Choose a class of model

##### -> Linear Regression

```python
from sklearn.linear_model import LinearRegression
model=LinearRegression()
print(model)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

### Step 5 : Fit the model using the labeled data

```python
model.fit(X,y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
print("The slope value is ",model.coef_)
print("The Intercept value is ",model.intercept_)
```

    The slope value is  [1.96255648]
    The Intercept value is  -0.9280522283510404

### Step 6 : Predict the new data using unlabeled data

```python
y_pred=model.predict(X)
print(y_pred)
```

    [ 9.1701522  11.82289582  5.97712395  1.84536583 14.12704812 12.9763063
      2.45114674  5.62634149  3.68247813  3.87402249  8.70465804  3.48268106
     16.97372721  6.34891233  6.70186044 14.64568883  3.70685779 13.52536676
     10.14846648 15.29284059 -0.13688814  1.14230571 12.55199304 -0.76546266
     16.33990974  4.22307493  5.02121503  3.37163395 12.85077063  5.59682773
      2.08999748  7.71392691  9.33150718 11.10757381 18.47969758 18.27654415
     16.02077928 12.20984107  8.87712953  3.3043861  16.72153174 13.60792001
      7.68073427 17.67166163  4.50254112 17.55047799  6.45759014  2.68415955
     -0.55791672 16.55194399]

### Step 7 : Computer the accuracy of the model

#### Visual method of accuracy

```python
plt.figure(dpi=120)
plt.scatter(x,y,c="Red",alpha=0.6,label="Actual Value")
plt.plot(x,y_pred,linewidth=5,c="green",alpha=0.4,label="Predicted Value")
plt.xlabel("X - InDependend Variable",fontsize=18)
plt.ylabel("Y - Dependend Variable",fontsize=18)
plt.legend(loc="best",fontsize=20)
plt.show()
```

## Prediction using metric method

### Mean Absolute Error
Mean of the absolute difference between the actual value and the predicted value.

\begin{equation}
\frac{1}{N}\sum_{i=1}^N |y_i-\hat{y_i}|
\end{equation}

>Note: 0 -> Best

```python
from sklearn.metrics import mean_absolute_error
mean_absolute_error(y,y_pred)
```

    0.7774688999128481

### Mean Squared Error
Mean of the square of the difference between the actual value and the predicted value.

\begin{equation}
\frac{1}{N}\sum_{i=1}^N (y_i-\hat{y_i})^2
\end{equation}

>Note: 0 -> Best

```python
from sklearn.metrics import mean_squared_error
mean_squared_error(y, y_pred)
```

    0.8885217699569701

### Root Mean Squared Error
Squareroot of the average of squared difference between the predicted and estimated value

\begin{equation}
\sqrt{\frac{1}{N}\sum_{i=1}^N (y_i-\hat{y_i})^2}
\end{equation}

>Note: 0 -> Best

```python
from sklearn.metrics import mean_squared_error
np.sqrt(mean_squared_error(y,y_pred))
```

    0.9426143272606088

### R Squared
The total sum of squares (proportional to the variance of the data):

\$ SS_{tot}=\sum_{i} (y_i-\hat{y_i})^2 $

The regression sum of squares, also called the explained sum of squares:

\$ SS_{reg}=\sum_{i} (f_i-\hat{y_i})^2 $

The most general definition of the coefficient of determination is

\$ R^2 = \frac{SS_{reg}}{SS_{tot}} $

### R Squared

- Measure of the best estimate
- Coefficient of Determination

> Note: 1 -> Best

```python
from sklearn.metrics import r2_score
r2_score(y, y_pred)
```

    0.9732624820891722

### Models inbuilt score function
Calculates the R^2 values

```python
model.score(X,y)
```

    0.9732624820891722

## Input new data

```python
xnew=np.array([3,4,7])
Xnew=xnew[:,np.newaxis]
model.predict(Xnew)
```

    array([ 4.95961722,  6.92217371, 12.80984316])

# Non-linear equations

```python
x = 10 * np.random.random(100)
y = 2 * np.sin(x) - 1 + np.random.randn(100)
print("x = ",x[:11])
print("y = ",y[:11])
plt.figure(dpi=120)
sns.scatterplot(x,y)
plt.xlabel("X - InDependend Variable")
plt.ylabel("Y - Dependend Variable")
plt.show()
```

### Predict using the normal Linear Regression

```python
X=x[:,np.newaxis]
from sklearn.linear_model import LinearRegression
model=LinearRegression()
model.fit(X,y)
xnew=np.linspace(0,10,100)
Xnew=xnew[:,np.newaxis]
y_pred=model.predict(Xnew)
```

```python
plt.figure(dpi=120,figsize=(12,7))
plt.scatter(x,y,c="Red",alpha=0.6,label="Actual Value")
plt.plot(xnew,y_pred,linewidth=5,c="green",alpha=0.4,label="Predicted Value")
plt.xlabel("X - InDependend Variable",fontsize=18)
plt.ylabel("Y - Dependend Variable",fontsize=18)
plt.legend(loc="best",fontsize=20)
plt.show()
```

### Calculate the accuracy

```python
model.score(X,y)
```

    0.001899013622799095

```python
model.score(X,y)*100
```

    0.1899013622799095

## Polynomial Regression

```python
from sklearn.preprocessing import PolynomialFeatures
```

```python
dummy=np.array([2,3,4])
poly=PolynomialFeatures(degree=4)
poly.fit_transform(dummy[:,np.newaxis])
```

    array([[  1.,   2.,   4.,   8.,  16.],
           [  1.,   3.,   9.,  27.,  81.],
           [  1.,   4.,  16.,  64., 256.]])

### Transform our actual data

```python
poly=PolynomialFeatures(degree=3)
poly_out=poly.fit_transform(X)
print(poly_out.shape)
print(poly_out[:10])
```

    (100, 4)
    [[1.00000000e+00 4.84034425e+00 2.34289324e+01 1.13404098e+02]
     [1.00000000e+00 8.24433792e+00 6.79691077e+01 5.60360292e+02]
     [1.00000000e+00 5.00138680e+00 2.50138699e+01 1.25104039e+02]
     [1.00000000e+00 2.83545135e+00 8.03978437e+00 2.27964174e+01]
     [1.00000000e+00 9.35763097e+00 8.75652573e+01 8.19403364e+02]
     [1.00000000e+00 3.61606590e+00 1.30759326e+01 4.72834340e+01]
     [1.00000000e+00 7.03069465e+00 4.94306673e+01 3.47531928e+02]
     [1.00000000e+00 2.94759886e+00 8.68833902e+00 2.56097382e+01]
     [1.00000000e+00 3.38868882e+00 1.14832119e+01 3.89130319e+01]
     [1.00000000e+00 2.90474480e-01 8.43754234e-02 2.45089072e-02]]

### Ouput of Polynomial features are fitted to normal Linear Regression

```python
model=LinearRegression()
model.fit(poly_out,y)
xnew=np.linspace(0,10,100)
Xnew=xnew[:,np.newaxis]
poly=PolynomialFeatures(degree=3)
poly_out_new=poly.fit_transform(Xnew)
y_pred=model.predict(poly_out_new)
```

```python
print("The slope value is ",model.coef_)
print("The Intercept value is ",model.intercept_)
```

    The slope value is  [ 0.         -3.04072636  0.66144071 -0.03995414]
    The Intercept value is  2.3936755999800496

```python
plt.figure(dpi=120,figsize=(15,7))
plt.scatter(x,y,c="Red",alpha=0.6,label="Actual Value")
plt.plot(xnew,y_pred,linewidth=5,c="green",alpha=0.4,label="Predicted Value")
plt.xlabel("X - InDependend Variable",fontsize=18)
plt.ylabel("Y - Dependend Variable",fontsize=18)
plt.legend(loc="best",fontsize=20)
plt.show()
```

## Accuracy score

```python
model.score(poly_out,y)*100
```

    29.76026958218042

# Automate the polynomial degree

```python
deg,acc=[],[]
for i in np.arange(2,20):
    poly=PolynomialFeatures(degree=i)
    poly_out=poly.fit_transform(X)
    model=LinearRegression()
    model.fit(poly_out,y)
    deg.append(i)
    acc.append(model.score(poly_out,y)*100)
    print("Degree :",deg[-1],"Accuracy score ->",acc[-1])
bha=pd.DataFrame({"Degree":deg,"Accuracy":acc})
```

    Degree : 2 Accuracy score -> 7.846218787887116
    Degree : 3 Accuracy score -> 29.76026958218042
    Degree : 4 Accuracy score -> 63.81832646336568
    Degree : 5 Accuracy score -> 67.1770768798496
    Degree : 6 Accuracy score -> 71.38923906135227
    Degree : 7 Accuracy score -> 71.94237240276576
    Degree : 8 Accuracy score -> 72.80588501511075
    Degree : 9 Accuracy score -> 73.10447113345143
    Degree : 10 Accuracy score -> 73.45112588076599
    Degree : 11 Accuracy score -> 73.45123595642593
    Degree : 12 Accuracy score -> 73.64777733910813
    Degree : 13 Accuracy score -> 73.73233739729207
    Degree : 14 Accuracy score -> 73.84875220899892
    Degree : 15 Accuracy score -> 73.3347529364348
    Degree : 16 Accuracy score -> 72.61308273270183
    Degree : 17 Accuracy score -> 72.44276162854572
    Degree : 18 Accuracy score -> 72.35501554325751
    Degree : 19 Accuracy score -> 71.69718613551315

```python
plt.figure(dpi=120)
sns.lineplot(data=bha,x="Degree",y="Accuracy")
plt.show()
```

```python
degr=6
poly=PolynomialFeatures(degree=degr)
poly_out=poly.fit_transform(X)
model=LinearRegression()
model.fit(poly_out,y)
xnew=np.linspace(0,10,100)
Xnew=xnew[:,np.newaxis]
poly=PolynomialFeatures(degree=degr)
poly_out_new=poly.fit_transform(Xnew)
y_pred=model.predict(poly_out_new)
print(model.score(poly_out,y)*100)
```

    71.38923906135227

```python
plt.figure(dpi=120,figsize=(15,7))
plt.scatter(x,y,c="Red",alpha=0.6,label="Actual Value")
plt.plot(xnew,y_pred,linewidth=5,c="green",alpha=0.4,label="Predicted Value")
plt.xlabel("X - InDependend Variable",fontsize=18)
plt.ylabel("Y - Dependend Variable",fontsize=18)
plt.legend(loc="best",fontsize=20)
plt.show()
```

```python
print("Model Coeffients",model.coef_)
print("Model Intercept",model.intercept_)
```

    Model Coeffients [ 0.00000000e+00  5.49235132e-01  1.35614195e+00 -1.22862598e+00
      3.12573504e-01 -3.14694607e-02  1.10639586e-03]
    Model Intercept -0.2794770369857815

# Questions ??


# Lab 4 Ordinary Differential Equations, Part 1

```python
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
import numpy as np
```

```python
%run ./ODESolvers.py
```

### Introduction: The basics

Consider the given ODE
\begin{equation}
    \begin{cases}
    y'(t) = y-\frac{1}{2}e^{\frac{t}{2}}\cdot\sin(5t)+5e^{\frac{t}{2}}\cdot\cos(5t)\\
    y(0)=0
    \end{cases}
\end{equation}
on the interval $[0,\pi]$.

```python
def rhsODEs(t, y):
    return y - 0.5*np.exp(0.5*t)*np.sin(5*t)+5*np.exp(0.5*t)*np.cos(5*t)
```

```python
# initial condition
y0 = [0]
N = 20

# Time steps
t_span = (0, np.pi)
t_eval = np.linspace(t_span[0], t_span[1], 1000)

# Solve for the ODE with R-K method
sol_ex = solve_ivp(rhsODEs, t_span, y0, method='RK45', t_eval=t_eval)
sol_fe = EulerForward(rhsODEs, t_span, y0, N)
sol_he = Heun(rhsODEs, t_span, y0, N)
t_evalRK = np.linspace(t_span[0], t_span[1], N)
sol_rk = solve_ivp(rhsODEs, t_span, y0, method='RK45', t_eval=t_evalRK)

# plot
fig, ax = plt.subplots(1, figsize=(6, 6))
ax.plot(sol_ex.t,sol_ex.y.T )
ax.plot(sol_fe[0], sol_fe[1],'-*' )
ax.plot(sol_he[0], sol_he[1],'-o' )
ax.plot(sol_rk.t,sol_rk.y.T, '-d')

ax.autoscale(enable=True, axis='both', tight=True)
ax.set_ylabel(r'$y(t)$')
ax.set_xlabel(r'$t$')
ax.legend(['Exact solution(RK45)','Euler Method', 'Heuns Method', 'Classical Runge-Kutta'])
```

### Application: Predator and prey

Simulating the interaction between predator and prey, described by the Lotka-Volterra ordinary differential equation.
\begin{equation}
    \begin{cases}
    y_1'=\alpha y_1-\beta y_1y_2\\
    y_2'=\delta y_1y_2-\gamma y_2, \, t>0\\
    y_1(0)=\hat{y_1}\\
    y_2(0)=\hat{y_2}\\
    \end{cases}
\end{equation}

```python
def PredPreyODE(t,y,alpha, beta, delta, gamma):
    return [alpha*y[0]-beta*y[0]*y[1],
            delta*y[0]*y[1]-gamma*y[1]]
```

The ODEs are solved by `scipy.integrate.solve_ivp` which is a solver for initial value problem. See https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html

```python
# Time interval for the simulation
t0 = 0
t1 = 40
t_span = (t0, t1)
t_eval = np.linspace(t_span[0], t_span[1], 10000)

# Initial conditions, e.g. number of predator and prey at time zero
pred0 = 40
prey0 = 80
y0 = [prey0, pred0]

# Set parameters alfa, beta, delta, gamma
alpha = 0.8;   # Reproduction rate of prey
beta = 0.02;  # Mortality rate of predator per prey
delta = 0.02; # Reproduction rate of predator per prey
gamma = 0.4;  # Mortality rate of predator

# Solve for Van der Pohls equation
sol = solve_ivp(lambda t,y: PredPreyODE(t, y, alpha, beta, delta, gamma),
                t_span, y0, method='RK45', t_eval=t_eval)
```

```python
# plot
fig, ax = plt.subplots(1, figsize=(6, 4.5))
# plot y1 and y2 together
ax.plot(sol.t.T,sol.y.T )
ax.set_ylabel('Number of predator and prey')
ax.set_xlabel('time')
```