# COMP 135 day09: MAP estimation for Logistic Regression

## Outline

* **Part 1: Understanding sigmoids and Logistic Regression as a model**
* **Part 2: Computing the MAP objective**
* **Part 3: Gradient descent for the MAP: Comparing 1st and 2nd order GD**

## Takeaways

* First-order methods are cheap but require many iterations
* Second-order methods are awesome, but still require careful step-size selection
* For all gradient descent methods, selecting step sizes is super important. Line search is needed!

```python
import numpy as np
import pandas as pd
import scipy.stats
```

```python
np.set_printoptions(precision=3, suppress=False)
```

```python
pd.options.display.float_format = '{:,.3g}'.format  # show 3 digits of precision
```

```python
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("notebook", font_scale=1.25)
```

```python

```

# Part 1: The Probabilistic view of logistic regression

### Task: Binary classification

Given $N$ observations of *paired* feature-outcome observations: $\{ x_n, t_n \}$.

* Each input feature $x_n$ is a scalar real: $x_n \in \mathbb{R}$
* Each output or "label" or "outcome" $t_n$ is a scalar binary value: $t_n \in \{0, 1\}$

We're also given a feature transform function $\phi$ which maps each $x_n$ to a vector in $M$-dimensional space. This function is known in advance.

We want to make good predictions of new outcomes $t_*$ given new features $x_*$.

## Feature transformation

For now, we'll assume that the "feature transform" $\phi(x_n)$ just simply passes along the features $x_n$, while adding an additional offset or "intercept" feature that is always 1. This is a *simplifying* assumption for today.

```python
def calc_features(x_N1, M=2):
    ''' Transform raw features into complete features useful for prediction

    Could do any non-linear transformations thought relevant for the problem.
    Here we'll just do an identity transform with an extra intercept feature.

    Args
    ----
    x_N1 : 2D array, shape (N, 1) = (n_examples,)

    Returns
    -------
    phi_NM : 2D array, shape (N, M) = (n_examples, n_transformed_features)
        First column will contain all ones (a bias or intercept feature)
        Second column will just include the raw features
    '''
    assert x_N1.ndim == 2
    assert x_N1.shape[1] == 1
    N = x_N1.shape[0]
    phi_NM = np.zeros((N, M))
    phi_NM[:,0] = 1
    phi_NM[:,1] = x_N1[:,0]
    return phi_NM
```

```python
x_N1 = np.linspace(-1, 1, 5)[:,np.newaxis]

# Get transformed features using our "calc_features" function
# * first column will be all 1s, an "intercept"
# * second column will be the x values
calc_features(x_N1)
```

    array([[ 1. , -1. ],
           [ 1. , -0.5],
           [ 1. ,  0. ],
           [ 1. ,  0.5],
           [ 1. ,  1. ]])

## Understanding the logistic sigmoid function

As discussed in your pre-recorded lectures, the *logistic sigmoid function* is:

\begin{align}
\sigma(r) = \frac{1}{1 + e^{-r}}
\end{align}

It maps real inputs $r \in (-\infty, +\infty)$ to the probability interval $(0, 1)$.

We call it a "sigmoid" function because it has an S-shaped curve, which you'll plot below.

This function is also sometimes called the "expit" function.

We can use an existing implementation of this function available in SciPy:

* expit: https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.expit.html

```python
from scipy.special import expit as sigmoid
```

```python
sigmoid(0)
```

    0.5

```python
sigmoid(-4)
```

    0.01798620996209156

```python
sigmoid(4)
```

    0.9820137900379085

```python
sigmoid(np.asarray([-6, -4, -2, 0, 2, 4, 6]))
```

    array([0.002, 0.018, 0.119, 0.5  , 0.881, 0.982, 0.998])

## Exercise 1a: Plot the logistic sigmoid function

We give you an array of G candidate $r$ values below.

```python
G = 101
r_G = np.linspace(-8, 8, G)
sigmoid_of_r_G = np.zeros(G) # TODO evaluate sigmoid at each r value
```

```python
plt.plot(r_G, sigmoid_of_r_G, 'k.-');
plt.xlabel('r'); plt.ylabel('$\sigma(r)$');
plt.ylim([-0.001, 1.001])
```

## Define the Likelihood

Each observation (indexed by $n$) is drawn iid from a Bernoulli as follows:

$$
t_n | w \sim \text{BernPMF}\left( t_n | \sigma(w^T \phi(x_n)) \right)
$$

where $w \in \mathbb{R}^M$ is a weight vector, the same size as our feature vector $\phi(x_n) \in \mathbb{R}^M$

The key properties here are:
* The *mean* of $t_n$ is a *non-linear activation* of a linear function of the transformed features.

## Define the  Prior

For now, we'll assume that weights come from a zero mean prior with some covariance determined by a scalar parameter $\alpha$:

$$
w \sim \mathcal{N}( 0, \alpha^{-1} I_M )
$$

A zero mean prior makes sense if we don't know if the slope should be negative or positive.

### Parameter we'll treat as a random variable: $w$

* Weights vector: $w = [w_1, w_2, \ldots w_M]^T$, so $w \in \mathbb{R}^M$

### Parameters we'll treat as fixed: $\alpha$

* Prior precision $\alpha > 0$

The larger $\alpha$ is, the more confident we are in the weight values before seeing any data.

## Create a simple toy data for analysis

Just execute the cells below to get the sense of how to generate toy data from this model

We'll manually intervene to set the weight vector to a known value. This makes it easy to tell if our learning is working later on.

```python
N = 10  # Number of examples we observe
M = 2     # Number of transformed features
```

Create the weight vector we'll use to generate our dataset. Set an intercept of 1.2 and a slope of -0.75

```python
true_w_M = np.asarray([0.1, -0.25])
```

Create a "true" alpha value which controls the prior precision

```python
true_alpha = 0.01
```

Create observed features $x$ and observed outputs $t$ manually

```python
x_N1 = np.asarray([-5, -0.8, -0.7, -0.6, -0.4, 0.5, 0.8, 0.9, 4.3, 4.1]).reshape((N, 1))
phi_NM = calc_features(x_N1)
```

```python
prng = np.random.RandomState(101) # reproducible random seed
```

```python
t_N = (prng.rand(N) < sigmoid(np.dot(phi_NM, true_w_M))).astype(np.float64)
```

## Visualize the toy dataset

```python
plt.plot(x_N1, t_N, 'k.');

ax_h = plt.gca()
ax_h.set_xlim([-8, 8]); ax_h.set_xticks([-6, -4, -2, 0, 2, 4, 6]);
ax_h.set_ylim([-.1, 1.1]);

xgrid_G1 = np.linspace(-8, 8, 100)[:,np.newaxis]
plt.plot(xgrid_G1, sigmoid(np.dot(calc_features(xgrid_G1), true_w_M)), 'c-', linewidth=3);

plt.xlabel('input: x');
plt.ylabel('output: t');
plt.title("Toy Data\n true_slope %.2f \n true intercept %.2f" % (
    true_w_M[1], true_w_M[0]));
```

## Discussion 1b: What about this observed dataset of 10 points would prefer a *negative* slope vs. a positive slope?

```python
# TODO discuss
```

# Part 2: MAP estimation : View as optimization problem

There is NO closed form for the posterior over weights $p( w | t)$.

However, we can evaluate (and thus optimize) the MAP objective, since this doesn't require knowing the full posterior.

Let's see how. Begin with the MAP optimization problem:

\begin{align}
    w^* = \arg \max_{w \in \mathbb{R}^M} ~~ p( w | t_{1:N} )
\end{align}

Rewriting using the log of the objective for tractability and simplifying via Bayes rule, we get the objective function to maximize is:

\begin{align}
\mathcal{M}(w) &= \log p( w | t_{1:N})
    \\
    &= \log p( w ) + \log p( t_{1:N} | w ) - \underbrace{\log p(t_{1:N})}_{\text{const wrt}~ w}
\end{align}

Thus, we can simply ignore the constant term, and maximize the following alternative objective:
\begin{align}
  \mathcal{M}'(w)  &= \log \text{MVNormPDF}( w | 0, \alpha^{-1} I_M ) + \sum_{n=1}^N \log \text{BernPMF}( t_n | \sigma(w^T \phi(x_n) )
\end{align}

Finally, we can *standardize* our problem by transforming so we *minimize* rather than *maximize*, just by multiplying by -1. Now the *loss* function we wish to minimize is:

\begin{align}
  \mathcal{L}(w)  &= - \log \text{MVNormPDF}( w | 0, \alpha^{-1} I_M ) - \sum_{n=1}^N \log \text{BernPMF}( t_n | \sigma(w^T \phi(x_n) )
\end{align}

Thus, we can find our optimal weights $w^*$ via:

\begin{align}
    w^* = \arg \min_{w \in \mathbb{R}^M} ~~ \mathcal{L}(w)
\end{align}

How can we compute each of these terms?

* Use `scipy.stats.multivariate_normal.logpdf` to evaluate the log prior PDF $\log \text{MVNormPDF}(\cdot)$
* For the likelihood pdf, use this formula:

$$
\sum_{n=1}^N \log \text{BernPMF}(t_n | p_n ) = \sum_{n=1}^N t_n \log p_n + (1-t_n) \log (1 - p_n)
$$

This is translated into the code below.

```python
def calc_sum_of_log_bern_pmf(t_N, p_N):
    ''' Calculate the log of the bernoulli pmf for N observations

    Args
    ----
    t_N : 1D array, shape (N,)
        Binary value (0 or 1) for each example n
    p_N : 1D array, shape (N,)
        Probability parameter of the Bernoulli for each example n

    Returns
    -------
    summed_logpmf : scalar float
        Summed log PMF over all N examples given
    '''
    # Make sure provided probabilities are not hard 0 or hard 1
    # so that the log values will not be numerically bad
    safe_p_N = np.minimum(np.maximum(p_N, 1e-100), 1 - 1e-13)
    return np.sum(np.log(safe_p_N)[t_N==1]) + np.sum(np.log(1-safe_p_N)[t_N==0])
```

## Exercise 2a: Compute the objective of our minimization problem

Translate the formula for $\mathcal{L}(w)$ above into concrete NumPy expressions

```python
def calc_loss(wguess_M, phi_NM, t_N, alpha=0.1):
    ''' Compute the MAP loss objective function.

    The loss is equal to the negative log prior plus negative log likelihood

    Args
    ----
    w_M : 1D array, shape (M,)
        Weight parameter at which we want to evaluate the loss
    phi_NM : 2D array, shape (N,M)
        Observed input features
        Each row is a feature vector for one example
    t_N : 1D array, shape (N,)
        Observed outputs
        Each row is a output scalar value for one example
    alpha : positive scalar
        Prior precision

    Returns
    -------
    loss : scalar float
        The value of the loss function at provided w value
    '''
    log_prior_pdf = 0.0 # TODO compute log prior pdf value
    log_lik_pdf = 0.0 # TODO compute log likelihood pdf value

    return -1 * log_prior_pdf + -1 * log_lik_pdf
```

## Exercise 2b: Evaluate the MAP objective (aka MAP loss function) at possible w values

```python
phi_NM
```

    array([[ 1. , -5. ],
           [ 1. , -0.8],
           [ 1. , -0.7],
           [ 1. , -0.6],
           [ 1. , -0.4],
           [ 1. ,  0.5],
           [ 1. ,  0.8],
           [ 1. ,  0.9],
           [ 1. ,  4.3],
           [ 1. ,  4.1]])

```python
t_N
```

    array([1., 1., 1., 1., 0., 0., 1., 0., 0., 1.])

```python
np.log(0.5)
```

    -0.6931471805599453

```python
# Try with all zero weights
w1_M = np.zeros(M)
calc_loss(w1_M, phi_NM, t_N, true_alpha)
```

    -0.0

```python
# Try with all weights set to 10
w2_M = 10 * np.ones(M)
calc_loss(w2_M, phi_NM, t_N, true_alpha)
```

    -0.0

```python
# Try with all weights set to TRUE values

# TODO write code using calc_loss(...)
```

## Discussion 2c: Which value of the weight vector out of the 3 tried had the "best" loss  value? Does that agree with what you expect?

Use what you know about how this toy dataset was generated (hint: we know which weights were used to make the true observations).

```python
# TODO discuss
```

## Demo: Visualizing the MAP objective as a contour plot

Step through the code below to see how we create a 2d contour plot visualization of our MAP optimization problem.

```python
# Create a 2-dim grid of possible w values

G = 51 # G possible values for intercept
w0_grid_G = np.linspace(-2, 2, G)

H = 51 # H possible values for slope
w1_grid_H = np.linspace(-2, 2, H)

w0_GH, w1_GH = np.meshgrid(w0_grid_G, w1_grid_H,)
```

```python
# Compute loss at each possible value in our grid
loss_GH = np.zeros((G, H))
for gg in range(G):
    for hh in range(H):
        cur_w_M = np.hstack([w0_GH[gg,hh], w1_GH[gg, hh]])
        loss_GH[gg, hh] = calc_loss(cur_w_M, phi_NM, t_N, true_alpha)

```

```python
# Create a pretty contour plot over the grid of w[0], w[1], loss values

levels = np.linspace(0, 40, 51) # 50 evenly spaced levels

fig_handle, ax_handle = plt.subplots(nrows=1, ncols=1, figsize=(8,8));

ax_handle.contour(w0_GH, w1_GH, loss_GH, levels=levels, linewidths=0, colors='k')
cntrf_handle = ax_handle.contourf(w0_GH, w1_GH, loss_GH, levels=levels, cmap='RdBu_r', vmin=levels[0], vmax=levels[-1]);

cbar = plt.colorbar(cntrf_handle, ax=ax_handle)
cbar.set_label('MAP loss objective (lower is better)', fontsize=16);
cbar.set_ticks(levels[::10]);
plt.xlabel('intercept   $w_1$');
plt.ylabel('slope   $w_2$');
plt.gca().set_aspect('equal', 'box');
```

## Exercise 2d: Visually interpret the plot above. By inspection, which intercept and slope values are optimal? What is the loss at this optimal point?

```python
# TODO interpret the plot and discuss with your group
```

## Exercise 2e: Numerically, search the grid of computed loss values `loss_GH` and determine the MAP value of weight vector

```python
# TODO solve this cell

# Hint: you might find it easier to flatten each array of shape (G,H) into shape (L,) where L=G*H
loss_L = loss_GH.flatten()# new shape (G*H,)
w0_L = w0_GH.flatten()  # new shape (G*H,)
w1_L = w1_GH.flatten()  # new shape (G*H,)

# TODO find values of w0 (intercept) and w1 (slope) that minimize the loss
```

# Part 3: Gradients, Hessians, and Gradient Descent

### Gradient and Hessian formulas

We saw in lecture that we can compute the gradient and Hessian as:

\begin{align}
\nabla_w \mathcal{L} &= \Phi^T ( \sigma(\Phi w) - t ) + \alpha w
\\
\nabla_w \nabla_w \mathcal{L} &= \Phi^T R(w) \Phi  + \alpha I_M
\end{align}

where $R$ is a diagonal matrix given by

$$
R = \text{diag}( \sigma(\Phi w) \sigma(- \Phi w ) )
$$

The functions below compute the gradient and Hessian. You don't need to do anything, just inspect them to gain understanding.

```python
def calc_R(w_M, phi_NM):
    s_N = np.dot(phi_NM, w_M)
    R_NN = np.diag( sigmoid(s_N) * sigmoid(-s_N) )
    return R_NN
```

```python
def calc_gradient_of_map_loss(w_M, phi_NM, t_N, alpha):
    ''' Calculate the gradient.

    Returns
    -------
    g_M : 1D array, shape (M,)
        Gradient vector evaluated at current weights w
    '''
    # Compute predicted probability of positive class
    yproba_N = sigmoid( np.dot(phi_NM, w_M) )
    return np.dot(phi_NM.T, (yproba_N - t_N)) + alpha * w_M
```

```python
def calc_hessian_of_map_loss(w_M, phi_NM, t_N, alpha):
    ''' Calculate the Hessian.

    Returns
    -------
    H_MM : 2D array, shape (M,M)
        Hessian matrix evaluated at current weights w
    '''
    R_NN = calc_R(w_M, phi_NM)
    return np.dot(phi_NM.T, np.dot(R_NN, phi_NM)) + alpha * np.eye(M)
```

## First-order gradient descent

The code below performs 1st-order GD.

While not converged, we perform the updates:

$$
w_{t+1} \gets w_t - \epsilon g( w_t )
$$

```python
max_n_steps = 100

w_M = 1.5 * np.ones(M)

step_size = 0.2 # Selected by starting at 1.0, and trying smaller values until first 5 steps made loss better

GD1_history_of_w = [w_M]
GD1_history_of_loss = [calc_loss(w_M, phi_NM, t_N, true_alpha)]

for step in range(max_n_steps):

    # Compute gradient
    g_M = calc_gradient_of_map_loss(w_M, phi_NM, t_N, true_alpha)

    # Update the weights by taking a step downhill
    w_M = w_M - step_size * g_M

    # Print out progress
    cur_loss = calc_loss(w_M, phi_NM, t_N, true_alpha)
    print("step %3d/%d  loss %11.4f | gradient_norm %9.4f | intercept %9.3f | slope %9.3f" % (
        step, max_n_steps, cur_loss, np.sum(np.abs(g_M)), w_M[0], w_M[1]))
    GD1_history_of_loss.append(cur_loss)
    GD1_history_of_w.append(w_M)

    if step % 10:
        step_size = 0.95 * step_size  # slowly decay the step size

bestw_fromGD_M = w_M
```

    step   0/100  loss     -0.0000 | gradient_norm   12.4385 | intercept     1.231 | slope    -0.719
    step   1/100  loss     -0.0000 | gradient_norm    2.8867 | intercept     1.098 | slope    -0.275
    step   2/100  loss     -0.0000 | gradient_norm    2.4277 | intercept     0.878 | slope    -0.516
    step   3/100  loss     -0.0000 | gradient_norm    1.7594 | intercept     0.815 | slope    -0.262
    step   4/100  loss     -0.0000 | gradient_norm    1.5637 | intercept     0.708 | slope    -0.423
    step   5/100  loss     -0.0000 | gradient_norm    1.0530 | intercept     0.683 | slope    -0.278
    step   6/100  loss     -0.0000 | gradient_norm    0.8692 | intercept     0.634 | slope    -0.363
    step   7/100  loss     -0.0000 | gradient_norm    0.5274 | intercept     0.620 | slope    -0.299
    step   8/100  loss     -0.0000 | gradient_norm    0.3724 | intercept     0.599 | slope    -0.330
    step   9/100  loss     -0.0000 | gradient_norm    0.2076 | intercept     0.590 | slope    -0.311
    step  10/100  loss     -0.0000 | gradient_norm    0.1210 | intercept     0.581 | slope    -0.317
    step  11/100  loss     -0.0000 | gradient_norm    0.0752 | intercept     0.575 | slope    -0.313
    step  12/100  loss     -0.0000 | gradient_norm    0.0413 | intercept     0.571 | slope    -0.313
    step  13/100  loss     -0.0000 | gradient_norm    0.0350 | intercept     0.568 | slope    -0.313
    step  14/100  loss     -0.0000 | gradient_norm    0.0247 | intercept     0.565 | slope    -0.312
    step  15/100  loss     -0.0000 | gradient_norm    0.0202 | intercept     0.563 | slope    -0.312
    step  16/100  loss     -0.0000 | gradient_norm    0.0161 | intercept     0.562 | slope    -0.312
    step  17/100  loss     -0.0000 | gradient_norm    0.0130 | intercept     0.561 | slope    -0.312
    step  18/100  loss     -0.0000 | gradient_norm    0.0107 | intercept     0.560 | slope    -0.312
    step  19/100  loss     -0.0000 | gradient_norm    0.0089 | intercept     0.560 | slope    -0.311
    step  20/100  loss     -0.0000 | gradient_norm    0.0074 | intercept     0.559 | slope    -0.311
    step  21/100  loss     -0.0000 | gradient_norm    0.0063 | intercept     0.559 | slope    -0.311
    step  22/100  loss     -0.0000 | gradient_norm    0.0053 | intercept     0.558 | slope    -0.311
    step  23/100  loss     -0.0000 | gradient_norm    0.0045 | intercept     0.558 | slope    -0.311
    step  24/100  loss     -0.0000 | gradient_norm    0.0039 | intercept     0.558 | slope    -0.311
    step  25/100  loss     -0.0000 | gradient_norm    0.0034 | intercept     0.558 | slope    -0.311
    step  26/100  loss     -0.0000 | gradient_norm    0.0030 | intercept     0.557 | slope    -0.311
    step  27/100  loss     -0.0000 | gradient_norm    0.0026 | intercept     0.557 | slope    -0.311
    step  28/100  loss     -0.0000 | gradient_norm    0.0023 | intercept     0.557 | slope    -0.311
    step  29/100  loss     -0.0000 | gradient_norm    0.0021 | intercept     0.557 | slope    -0.311
    step  30/100  loss     -0.0000 | gradient_norm    0.0019 | intercept     0.557 | slope    -0.311
    step  31/100  loss     -0.0000 | gradient_norm    0.0017 | intercept     0.557 | slope    -0.311
    step  32/100  loss     -0.0000 | gradient_norm    0.0015 | intercept     0.557 | slope    -0.311
    step  33/100  loss     -0.0000 | gradient_norm    0.0014 | intercept     0.557 | slope    -0.311
    step  34/100  loss     -0.0000 | gradient_norm    0.0013 | intercept     0.557 | slope    -0.311
    step  35/100  loss     -0.0000 | gradient_norm    0.0011 | intercept     0.557 | slope    -0.311
    step  36/100  loss     -0.0000 | gradient_norm    0.0011 | intercept     0.557 | slope    -0.311
    step  37/100  loss     -0.0000 | gradient_norm    0.0010 | intercept     0.557 | slope    -0.311
    step  38/100  loss     -0.0000 | gradient_norm    0.0009 | intercept     0.557 | slope    -0.311
    step  39/100  loss     -0.0000 | gradient_norm    0.0008 | intercept     0.557 | slope    -0.311
    step  40/100  loss     -0.0000 | gradient_norm    0.0008 | intercept     0.557 | slope    -0.311
    step  41/100  loss     -0.0000 | gradient_norm    0.0007 | intercept     0.557 | slope    -0.311
    step  42/100  loss     -0.0000 | gradient_norm    0.0007 | intercept     0.556 | slope    -0.311
    step  43/100  loss     -0.0000 | gradient_norm    0.0007 | intercept     0.556 | slope    -0.311
    step  44/100  loss     -0.0000 | gradient_norm    0.0006 | intercept     0.556 | slope    -0.311
    step  45/100  loss     -0.0000 | gradient_norm    0.0006 | intercept     0.556 | slope    -0.311
    step  46/100  loss     -0.0000 | gradient_norm    0.0006 | intercept     0.556 | slope    -0.311
    step  47/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311
    step  48/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311
    step  49/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311
    step  50/100  loss     -0.0000 | gradient_norm    0.0005 | intercept     0.556 | slope    -0.311
    step  51/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  52/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  53/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  54/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  55/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  56/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  57/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  58/100  loss     -0.0000 | gradient_norm    0.0004 | intercept     0.556 | slope    -0.311
    step  59/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  60/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  61/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  62/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  63/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  64/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  65/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  66/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  67/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  68/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  69/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  70/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  71/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  72/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  73/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  74/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  75/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  76/100  loss     -0.0000 | gradient_norm    0.0003 | intercept     0.556 | slope    -0.311
    step  77/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  78/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  79/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  80/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  81/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  82/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  83/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  84/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  85/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  86/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  87/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  88/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  89/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  90/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  91/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  92/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  93/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  94/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  95/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  96/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  97/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  98/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311
    step  99/100  loss     -0.0000 | gradient_norm    0.0002 | intercept     0.556 | slope    -0.311

## Discussion 3a: Compare the GD estimate of the best weights $w$ to those found via grid search

```python
print("Optimal weights via grid search")
bestid = np.argmin(loss_GH.flatten())
bestw_fromgridsearch_M = np.asarray([w0_GH.flatten()[bestid], w1_GH.flatten()[bestid]])
print(bestw_fromgridsearch_M)

print("Optimal weights via 1st order gradient descent")
print(w_M)
```

    Optimal weights via grid search
    [-2. -2.]
    Optimal weights via 1st order gradient descent
    [ 0.556 -0.311]

```python
 calc_loss(bestw_fromgridsearch_M, phi_NM, t_N, true_alpha)
```

    -0.0

```python
 calc_loss(bestw_fromGD_M, phi_NM, t_N, true_alpha)
```

    -0.0

```python
# TODO discuss: which is better? are they similar?
```

# Second order gradient descent

```python
max_n_steps = 100

w_M = 1.5 * np.ones(M)

step_size = 0.15 # Selected by starting at 1.0, and trying smaller values until first 5 steps made loss better

GD2_history_of_w = [w_M]
GD2_history_of_loss = [calc_loss(w_M, phi_NM, t_N, true_alpha)]

for step in range(max_n_steps):

    g_M = calc_gradient_of_map_loss(w_M, phi_NM, t_N, true_alpha)
    H_MM = calc_hessian_of_map_loss(w_M, phi_NM, t_N, true_alpha)
    w_M = w_M - step_size * np.linalg.solve(H_MM, g_M) # compute H^1 times g
    cur_loss = calc_loss(w_M, phi_NM, t_N, true_alpha)

    print("step %3d/%d  loss %11.4f | gradient_norm %9.4f | intercept %9.3f | slope %9.3f" % (
        step, max_n_steps, cur_loss, np.sum(np.abs(g_M)), w_M[0], w_M[1]))
    GD2_history_of_loss.append(cur_loss)
    GD2_history_of_w.append(w_M)

    if step % 10:
        step_size = 0.95 * step_size # slowly decay step size
```

    step   0/100  loss     -0.0000 | gradient_norm   12.4385 | intercept    -0.466 | slope    -2.940
    step   1/100  loss     -0.0000 | gradient_norm    6.3231 | intercept     0.627 | slope    -0.103
    step   2/100  loss     -0.0000 | gradient_norm    3.1766 | intercept     0.613 | slope    -0.129
    step   3/100  loss     -0.0000 | gradient_norm    2.7249 | intercept     0.603 | slope    -0.151
    step   4/100  loss     -0.0000 | gradient_norm    2.3576 | intercept     0.595 | slope    -0.170
    step   5/100  loss     -0.0000 | gradient_norm    2.0560 | intercept     0.589 | slope    -0.186
    step   6/100  loss     -0.0000 | gradient_norm    1.8062 | intercept     0.584 | slope    -0.199
    step   7/100  loss     -0.0000 | gradient_norm    1.5977 | intercept     0.581 | slope    -0.210
    step   8/100  loss     -0.0000 | gradient_norm    1.4225 | intercept     0.578 | slope    -0.220
    step   9/100  loss     -0.0000 | gradient_norm    1.2743 | intercept     0.575 | slope    -0.228
    step  10/100  loss     -0.0000 | gradient_norm    1.1480 | intercept     0.573 | slope    -0.236
    step  11/100  loss     -0.0000 | gradient_norm    1.0400 | intercept     0.571 | slope    -0.242
    step  12/100  loss     -0.0000 | gradient_norm    0.9421 | intercept     0.570 | slope    -0.248
    step  13/100  loss     -0.0000 | gradient_norm    0.8578 | intercept     0.568 | slope    -0.253
    step  14/100  loss     -0.0000 | gradient_norm    0.7848 | intercept     0.567 | slope    -0.258
    step  15/100  loss     -0.0000 | gradient_norm    0.7214 | intercept     0.566 | slope    -0.261
    step  16/100  loss     -0.0000 | gradient_norm    0.6660 | intercept     0.565 | slope    -0.265
    step  17/100  loss     -0.0000 | gradient_norm    0.6174 | intercept     0.565 | slope    -0.268
    step  18/100  loss     -0.0000 | gradient_norm    0.5746 | intercept     0.564 | slope    -0.271
    step  19/100  loss     -0.0000 | gradient_norm    0.5367 | intercept     0.564 | slope    -0.273
    step  20/100  loss     -0.0000 | gradient_norm    0.5031 | intercept     0.563 | slope    -0.275
    step  21/100  loss     -0.0000 | gradient_norm    0.4732 | intercept     0.563 | slope    -0.277
    step  22/100  loss     -0.0000 | gradient_norm    0.4451 | intercept     0.562 | slope    -0.279
    step  23/100  loss     -0.0000 | gradient_norm    0.4199 | intercept     0.562 | slope    -0.281
    step  24/100  loss     -0.0000 | gradient_norm    0.3974 | intercept     0.562 | slope    -0.282
    step  25/100  loss     -0.0000 | gradient_norm    0.3771 | intercept     0.561 | slope    -0.284
    step  26/100  loss     -0.0000 | gradient_norm    0.3588 | intercept     0.561 | slope    -0.285
    step  27/100  loss     -0.0000 | gradient_norm    0.3423 | intercept     0.561 | slope    -0.286
    step  28/100  loss     -0.0000 | gradient_norm    0.3273 | intercept     0.561 | slope    -0.287
    step  29/100  loss     -0.0000 | gradient_norm    0.3137 | intercept     0.560 | slope    -0.288
    step  30/100  loss     -0.0000 | gradient_norm    0.3013 | intercept     0.560 | slope    -0.289
    step  31/100  loss     -0.0000 | gradient_norm    0.2900 | intercept     0.560 | slope    -0.290
    step  32/100  loss     -0.0000 | gradient_norm    0.2791 | intercept     0.560 | slope    -0.290
    step  33/100  loss     -0.0000 | gradient_norm    0.2692 | intercept     0.560 | slope    -0.291
    step  34/100  loss     -0.0000 | gradient_norm    0.2600 | intercept     0.560 | slope    -0.292
    step  35/100  loss     -0.0000 | gradient_norm    0.2517 | intercept     0.560 | slope    -0.292
    step  36/100  loss     -0.0000 | gradient_norm    0.2440 | intercept     0.560 | slope    -0.293
    step  37/100  loss     -0.0000 | gradient_norm    0.2369 | intercept     0.559 | slope    -0.293
    step  38/100  loss     -0.0000 | gradient_norm    0.2304 | intercept     0.559 | slope    -0.294
    step  39/100  loss     -0.0000 | gradient_norm    0.2243 | intercept     0.559 | slope    -0.294
    step  40/100  loss     -0.0000 | gradient_norm    0.2187 | intercept     0.559 | slope    -0.295
    step  41/100  loss     -0.0000 | gradient_norm    0.2136 | intercept     0.559 | slope    -0.295
    step  42/100  loss     -0.0000 | gradient_norm    0.2085 | intercept     0.559 | slope    -0.295
    step  43/100  loss     -0.0000 | gradient_norm    0.2038 | intercept     0.559 | slope    -0.296
    step  44/100  loss     -0.0000 | gradient_norm    0.1995 | intercept     0.559 | slope    -0.296
    step  45/100  loss     -0.0000 | gradient_norm    0.1954 | intercept     0.559 | slope    -0.296
    step  46/100  loss     -0.0000 | gradient_norm    0.1917 | intercept     0.559 | slope    -0.296
    step  47/100  loss     -0.0000 | gradient_norm    0.1881 | intercept     0.559 | slope    -0.297
    step  48/100  loss     -0.0000 | gradient_norm    0.1849 | intercept     0.559 | slope    -0.297
    step  49/100  loss     -0.0000 | gradient_norm    0.1818 | intercept     0.559 | slope    -0.297
    step  50/100  loss     -0.0000 | gradient_norm    0.1790 | intercept     0.559 | slope    -0.297
    step  51/100  loss     -0.0000 | gradient_norm    0.1763 | intercept     0.559 | slope    -0.298
    step  52/100  loss     -0.0000 | gradient_norm    0.1737 | intercept     0.559 | slope    -0.298
    step  53/100  loss     -0.0000 | gradient_norm    0.1712 | intercept     0.559 | slope    -0.298
    step  54/100  loss     -0.0000 | gradient_norm    0.1689 | intercept     0.558 | slope    -0.298
    step  55/100  loss     -0.0000 | gradient_norm    0.1667 | intercept     0.558 | slope    -0.298
    step  56/100  loss     -0.0000 | gradient_norm    0.1647 | intercept     0.558 | slope    -0.298
    step  57/100  loss     -0.0000 | gradient_norm    0.1628 | intercept     0.558 | slope    -0.299
    step  58/100  loss     -0.0000 | gradient_norm    0.1610 | intercept     0.558 | slope    -0.299
    step  59/100  loss     -0.0000 | gradient_norm    0.1593 | intercept     0.558 | slope    -0.299
    step  60/100  loss     -0.0000 | gradient_norm    0.1578 | intercept     0.558 | slope    -0.299
    step  61/100  loss     -0.0000 | gradient_norm    0.1563 | intercept     0.558 | slope    -0.299
    step  62/100  loss     -0.0000 | gradient_norm    0.1548 | intercept     0.558 | slope    -0.299
    step  63/100  loss     -0.0000 | gradient_norm    0.1534 | intercept     0.558 | slope    -0.299
    step  64/100  loss     -0.0000 | gradient_norm    0.1521 | intercept     0.558 | slope    -0.299
    step  65/100  loss     -0.0000 | gradient_norm    0.1509 | intercept     0.558 | slope    -0.299
    step  66/100  loss     -0.0000 | gradient_norm    0.1498 | intercept     0.558 | slope    -0.299
    step  67/100  loss     -0.0000 | gradient_norm    0.1487 | intercept     0.558 | slope    -0.300
    step  68/100  loss     -0.0000 | gradient_norm    0.1476 | intercept     0.558 | slope    -0.300
    step  69/100  loss     -0.0000 | gradient_norm    0.1467 | intercept     0.558 | slope    -0.300
    step  70/100  loss     -0.0000 | gradient_norm    0.1458 | intercept     0.558 | slope    -0.300
    step  71/100  loss     -0.0000 | gradient_norm    0.1449 | intercept     0.558 | slope    -0.300
    step  72/100  loss     -0.0000 | gradient_norm    0.1440 | intercept     0.558 | slope    -0.300
    step  73/100  loss     -0.0000 | gradient_norm    0.1432 | intercept     0.558 | slope    -0.300
    step  74/100  loss     -0.0000 | gradient_norm    0.1425 | intercept     0.558 | slope    -0.300
    step  75/100  loss     -0.0000 | gradient_norm    0.1417 | intercept     0.558 | slope    -0.300
    step  76/100  loss     -0.0000 | gradient_norm    0.1410 | intercept     0.558 | slope    -0.300
    step  77/100  loss     -0.0000 | gradient_norm    0.1404 | intercept     0.558 | slope    -0.300
    step  78/100  loss     -0.0000 | gradient_norm    0.1398 | intercept     0.558 | slope    -0.300
    step  79/100  loss     -0.0000 | gradient_norm    0.1392 | intercept     0.558 | slope    -0.300
    step  80/100  loss     -0.0000 | gradient_norm    0.1387 | intercept     0.558 | slope    -0.300
    step  81/100  loss     -0.0000 | gradient_norm    0.1381 | intercept     0.558 | slope    -0.300
    step  82/100  loss     -0.0000 | gradient_norm    0.1376 | intercept     0.558 | slope    -0.300
    step  83/100  loss     -0.0000 | gradient_norm    0.1371 | intercept     0.558 | slope    -0.300
    step  84/100  loss     -0.0000 | gradient_norm    0.1367 | intercept     0.558 | slope    -0.300
    step  85/100  loss     -0.0000 | gradient_norm    0.1362 | intercept     0.558 | slope    -0.300
    step  86/100  loss     -0.0000 | gradient_norm    0.1358 | intercept     0.558 | slope    -0.300
    step  87/100  loss     -0.0000 | gradient_norm    0.1354 | intercept     0.558 | slope    -0.300
    step  88/100  loss     -0.0000 | gradient_norm    0.1351 | intercept     0.558 | slope    -0.301
    step  89/100  loss     -0.0000 | gradient_norm    0.1347 | intercept     0.558 | slope    -0.301
    step  90/100  loss     -0.0000 | gradient_norm    0.1344 | intercept     0.558 | slope    -0.301
    step  91/100  loss     -0.0000 | gradient_norm    0.1341 | intercept     0.558 | slope    -0.301
    step  92/100  loss     -0.0000 | gradient_norm    0.1337 | intercept     0.558 | slope    -0.301
    step  93/100  loss     -0.0000 | gradient_norm    0.1334 | intercept     0.558 | slope    -0.301
    step  94/100  loss     -0.0000 | gradient_norm    0.1332 | intercept     0.558 | slope    -0.301
    step  95/100  loss     -0.0000 | gradient_norm    0.1329 | intercept     0.558 | slope    -0.301
    step  96/100  loss     -0.0000 | gradient_norm    0.1326 | intercept     0.558 | slope    -0.301
    step  97/100  loss     -0.0000 | gradient_norm    0.1324 | intercept     0.558 | slope    -0.301
    step  98/100  loss     -0.0000 | gradient_norm    0.1322 | intercept     0.558 | slope    -0.301
    step  99/100  loss     -0.0000 | gradient_norm    0.1320 | intercept     0.558 | slope    -0.301

```python
whistory_GD1_T2 = np.vstack(GD1_history_of_w)
whistory_GD2_T2 = np.vstack(GD2_history_of_w)
```

```python
# Create a pretty contour plot over the grid of w[0], w[1], loss values

levels = np.linspace(0, 40, 51) # 50 evenly spaced levels

fig_handle, ax_handle = plt.subplots(nrows=1, ncols=1, figsize=(8,8));

ax_handle.contour(w0_GH, w1_GH, loss_GH, levels=levels, linewidths=0, colors='k')
cntrf_handle = ax_handle.contourf(w0_GH, w1_GH, loss_GH, levels=levels, cmap='RdBu_r', vmin=levels[0], vmax=levels[-1]);

cbar = plt.colorbar(cntrf_handle, ax=ax_handle)
cbar.set_label('MAP loss objective (lower is better)', fontsize=16);
cbar.set_ticks(levels[::10]);

# Show the first 10 iterates of GD
ax_handle.plot(whistory_GD1_T2[:1,0], whistory_GD1_T2[:1,1], 'kx', markersize=15, label='Initial w value')
ax_handle.plot(whistory_GD1_T2[:10,0], whistory_GD1_T2[:10,1], 'ks-', label='First 10 steps of 1st-order GD')
ax_handle.plot(whistory_GD2_T2[:10,0], whistory_GD2_T2[:10,1], 'ms-', label='First 10 steps of 2st-order GD')

plt.xlabel('intercept   $w_1$');
plt.ylabel('slope   $w_2$');
plt.gca().set_aspect('equal', 'box');
plt.ylim([-3, 2]);
plt.legend(loc='upper left');
```

## Discussion: Compare the *first step* that 1st-order GD took to the *first step* that 2nd-order GD took.

* Which is a better *direction*?
* Which ended up closer to the optimal value because of well-chosen step-length?

## Discussion: Compare the overall behavior of 1st and 2nd order GD.... do you see big qualitative differences?

* Which one makes faster progress toward the minimum?
* How are both sensitive to the step-size choice?

```python

```

```python

```


# The model

## Black Scholes model

Under the risk neutral probability measure $\mathbb{Q}$, the underlying is defined as a geometric Brownian motion solving the following SDE:

\begin{equation}
\begin{cases}
\frac{d S_t}{S_t} = r dt + \sigma d W_t \\
S_0 = x
\end{cases}
\end{equation}

From that SDE, one can explcitiely compute the first two moments of $S$:

\begin{equation}
\begin{cases}
\mathbb{E}_t^{\mathbb{Q}} (S_{t+h}) = S_t e^{r h} \\
\mathbb{E}_t^{\mathbb{Q}} (S_{t+h}^2) = S_t^2 e^{2 r h + \sigma^2 h}
\end{cases}
\end{equation}

## The trinomial tree model matching the Black-Scholes model

In the sequel, we will use a recombining tree to model the underlying moves.
We define by $u, u > 1$, some multiplicative factor of the underlying from time $t$ to time $t+h$.
In the trinomial model, the underlying at time $t+h$ can take 3 values knowing the $S_t$:

\begin{equation}
\begin{cases}
S_{t+h} = u \times S_t > S_t \text { with probability } p_u \\
S_{t+h} = d \times S_t < S_t \text { with probability } p_d \\
S_{t+h} = S_t \text { with probability } p_m = 1 - p_u - p_d \\
\end{cases}
\end{equation}

with by vertue of our recombining assumption, $d = \frac{1}{u}$.

That is, given $u$, we must find the matching probabilities $p_u, p_d, p_m$ matching the Black-Scholes model.
Thus we have 3 unknown variables. We must at least give 3 independent equations to solve the system.

The probabilities should be non negative and summing to one, which is already written for the definition of $p_m$. We are left with 2 unknown variables $p_u$ and $p_d$.
Matching the first two moments given above we have:

\begin{equation}
\begin{cases}
\mathbb{E}_t^{\mathbb{Q}} (S_{t+h}) = S_t e^{r h} = S_t \left(u p_u + (1 - p_u - p_m) + d p_d \right) \\
\mathbb{E}_t^{\mathbb{Q}} (S_{t+h}^2) = S_t^2 e^{2 r h + \sigma^2 h} = S_t^2 \left(u^2 p_u + (1 - p_u - p_m) + d^2 p_d \right)
\end{cases}
\end{equation}

So we have:

$$
\begin{pmatrix}
e^{r h} - 1 \\
e^{2rh + \sigma^2 h} - 1
\end{pmatrix}
=
\begin{pmatrix}
u - 1 & d - 1 \\
u^2 - 1 & d^2 - 1
\end{pmatrix}
\begin{pmatrix}
p_u \\
p_d
\end{pmatrix}
$$

or equivalently, solving the linear system:

$$
\begin{pmatrix}
u - 1 & d - 1 \\
u^2 - 1 & d^2 - 1
\end{pmatrix}^{-1}
\begin{pmatrix}
e^{r h} - 1 \\
e^{2rh + \sigma^2 h} - 1
\end{pmatrix}
=
\begin{pmatrix}
p_u \\
p_d
\end{pmatrix}
$$

And using the classical formula of the inverse of a 2x2 matrix, we have:
$$
\begin{pmatrix}
u - 1 & d - 1 \\
u^2 - 1 & d^2 - 1
\end{pmatrix}^{-1}
= \frac{1}{(u - 1) (d^2 - 1) - (u^2 - 1) (d - 1)}
\begin{pmatrix}
d^2 - 1 & 1 - d \\
1 - u^2 & u - 1
\end{pmatrix}
$$

and $p_m := 1 - p_u - p_d$.
Moreover, once the computation is done, one should check that all the probabilities are non negative.

```python
import numpy as np

from scipy.stats import norm
```

```python
class TrinomialBSModel(object):
    def __init__(self, S0=100., r=0.02, sigma=0.2, mat=1.):
        self.__s0 = S0
        self.__r = r
        self.__sigma = sigma
        self.__T = mat

    def __compute_probs(self):
        B = np.array([-1. + np.exp(self.__r * self.__h),
                      -1. + np.exp(2. * self.__r * self.__h + self.__sigma**2 * self.__h)])

        d = self.__down
        u = self.__up

        A = np.array([[u - 1., d - 1.],
                      [u**2 - 1., d**2 - 1.]])

        det = (u - 1.) * (d**2 - 1.) - (u**2 - 1.) * (d - 1.)
        invA = 1. / det * np.array([[d**2 - 1., 1. - d],
                                    [1. - u**2, u - 1.]])

        res = invA.dot(B)

        self.__pu = res[0]
        self.__pd = res[1]
        self.__pm = 1. - self.__pu - self.__pd

        assert 0 <= self.__pu <= 1., 'p_u should lie in [0, 1] given %s' % self.__pu
        assert 0 <= self.__pd <= 1., 'p_d should lie in [0, 1] given %s' % self.__pd
        assert 0 <= self.__pm <= 1., 'p_m should lie in [0, 1] given %s' % self.__pm

    def __check_up_value(self, up):
        if up is None:
            lbda = np.sqrt(0.5 * np.pi)
            up = np.exp(lbda * self.__sigma * np.sqrt(self.__h))

        assert up > 0., 'up should be non negative'

        down = 1. / up
        assert down < up, 'up <= 1. / up = down'

        self.__up = up
        self.__down = down

    def __gen_stock_vec(self, nb):
        vec_u = self.__up * np.ones(nb)
        np.cumprod(vec_u, out=vec_u)

        vec_d = self.__down * np.ones(nb)
        np.cumprod(vec_d, out=vec_d)

        res = np.concatenate((vec_d[::-1], [1.], vec_u))
        res *= self.__s0

        return res

    def payoff(self, stock_vec):
        raise NotImplementedError()

    def compute_current_price(self, crt_vec_stock, nxt_vec_prices):
        expectation = np.zeros(crt_vec_stock.size)
        for i in range(expectation.size):
            tmp = nxt_vec_prices[i] * self.__pd
            tmp += nxt_vec_prices[i + 1] * self.__pm
            tmp += nxt_vec_prices[i + 2] * self.__pu

            expectation[i] = tmp

        return self.__discount * expectation

    def price(self, nb_steps, up=None):
        assert nb_steps > 0, 'nb_steps shoud be > 0'

        nb_steps = int(nb_steps)

        self.__h = self.__T / nb_steps
        self.__check_up_value(up)
        self.__compute_probs()

        self.__discount = np.exp(-self.__r * self.__h)

        final_vec_stock = self.__gen_stock_vec(nb_steps)
        final_payoff = self.payoff(final_vec_stock)
        nxt_vec_prices = final_payoff

        for i in range(1, nb_steps + 1):
            vec_stock = self.__gen_stock_vec(nb_steps - i)
            nxt_vec_prices = self.compute_current_price(vec_stock, nxt_vec_prices)

        return nxt_vec_prices[0]
```

```python
class TrinomialBSCall(TrinomialBSModel):
    def __init__(self, S0=100., r=0.02, sigma=0.2, mat=1., K=100.):
        super(TrinomialBSCall, self).__init__(S0, r, sigma, mat)
        self.__K = K

    def payoff(self, s):
        return np.maximum(s - self.__K, 0.)

class TrinomialBSAmericanCall(TrinomialBSCall):
    def compute_current_price(self, crt_vec_stock, nxt_vec_prices):
        crt_payoff = self.payoff(crt_vec_stock)
        crt_prices = super(TrinomialBSAmericanCall, self).compute_current_price(crt_vec_stock, nxt_vec_prices)

        return np.maximum(crt_payoff, crt_prices)
```

```python
class TrinomialBSPut(TrinomialBSModel):
    def __init__(self, S0=100., r=0.02, sigma=0.2, mat=1., K=100.):
        super(TrinomialBSPut, self).__init__(S0, r, sigma, mat)
        self.__K = K

    def payoff(self, s):
        return np.maximum(self.__K - s, 0.)

class TrinomialBSAmericanPut(TrinomialBSPut):
    def compute_current_price(self, crt_vec_stock, nxt_vec_prices):
        crt_payoff = self.payoff(crt_vec_stock)
        crt_prices = super(TrinomialBSAmericanPut, self).compute_current_price(crt_vec_stock, nxt_vec_prices)

        return np.maximum(crt_payoff, crt_prices)
```

```python
def bs_call_price(S=100., r=0.02, sigma=0.2, t=0., T=1., K=100.):
    ttm = T - t

    if ttm < 0:
        return 0.
    elif ttm == 0.:
        return np.maximum(S - K, 0.)

    vol = sigma * np.sqrt(ttm)

    d_minus = np.log(S / K) + (r - 0.5 * sigma**2) * ttm
    d_minus /= vol

    d_plus = d_minus + vol

    res = S * norm.cdf(d_plus)
    res -= K * np.exp(-r * ttm) * norm.cdf(d_minus)

    return res
```

```python
def bs_put_price(S=100., r=0.02, sigma=0.2, t=0., T=1., K=100.):
    # Using call-put parity :)
    ttm = T - t

    if ttm < 0:
        return 0.
    elif ttm == 0.:
        return np.maximum(K - S, 0.)

    dsct_strike = K * np.exp(-r * ttm)

    cap_S = S * np.exp(r * t)

    call = bs_call_price(S, r, sigma, t, T, K)
    return call - cap_S + dsct_strike
```

```python
tree = TrinomialBSCall()

print(tree.price(1000))
print(bs_call_price())
```

    8.91559126943
    8.91603727857

```python
tree = TrinomialBSPut()

print(tree.price(1000))
print(bs_put_price())
```

    6.93545860012
    6.93590460925

```python
tree = TrinomialBSAmericanCall()

print(tree.price(1000))
```

    8.91559126943

```python
tree = TrinomialBSAmericanPut()

print(tree.price(1000))
```

    7.11070940695


#
Derivadas

```python
from sympy import *
init_printing()
x, y, z = symbols('x, y, z')
```

Para derivar funciones se pueden emplear distintos métodos y funciones. Nosotros vamos a emplear la función **diff** y el método con el mismo nombre. Si la función tiene una única variable, el uso de la función **diff** es sencillo: únicamente tenemos que escribir la función con único argumento. El método se usa de la misma forma.

Si añadimos el argumento **evaluate = False**, la derivada no se calcula, únicamente se escribe. Se puede usa **doit** para realizar la derivada.

###
Calcula la derivada de las funciones:

* $3x^2+8x$

* $4y^5-6y^3$

* $\cos(x^2)$

```python

```

```python

```

Cuando en la función tiene varias variables, como segundo argumento le debemos especificar la variable respecto a la que queremos derivar.

###
Deriva la siguiente función respecto a cada una de sus variables:

$$ \cos(3xy^2) $$

```python
diff(cos(x*y),y,3,x,2, evaluate=False)
```

Para realizar derivadas de mayor orden se pueden seguir dos métodos distintos: el primero es escribir tantas veces la variable como derivadas queramos realizar. La segunda es escribir el número de orden de derivada despúes de la letra. Esto sirve también para derivadas parciales.

###
Calcula las siguientes derivadas:

* $\displaystyle \frac{\partial^5}{\partial x^5} \sin(6x) $

* $\displaystyle \frac{\partial^5}{\partial x^2 \partial y^3} \sin(4x+6y)$

```python

```

Una de las primera aplicaciones de la derivada consiste en calcular los máximos y los mínimos, así como los puntos de inflexión de curvas. Para ello debemos realizar la primera o la segunda derivada, igualar a cero y resolver la ecuación. Los extremos y los puntos de inflexión se encuentran entre las soluciones.

###
Encuentra los posibles extremos y los puntos de inflexión de:

$$ 4x^3-2x+7$$

```python
f=4*x**3-2*x+7; f
```

```python
solve(diff(f,x,2))
```

$$\begin{bmatrix}0\end{bmatrix}$$

Otra aplicación de las derivadas son los polinomios de Taylor o en general el desarrollo en serie de potencias de una cierta función. Para ello empleamos la función **series** que hoy en día solamente es capaz de desarrollar funciones en un entorno del cero. Si no especificamos nada, Sympy calcula desarrollos alrededor del cero y de orden 6.

```python
series(exp(x+y),x, n=9)
```

###
Calcula desarrollos de distintos ordenes de las funciones:

* $\exp(x)$

* $\cos(x^2)$

* $\exp(x+y)$

```python
series(cos(x),x,0,9)
```

```python
series(cos(y))
```

```python

```


```python
from sympy import *
```

```python
# square root

sqrt(32)
```

$\displaystyle 4 \sqrt{2}$

```python
# fraction reduction

frac = Rational(5,25)
frac
```

$\displaystyle \frac{1}{5}$

```python
# algebra

x, y = symbols("x y")

# expression

expr = x**2+y**2
expr
```

$\displaystyle x^{2} + y^{2}$

```python
# expression manipulation

expr + x
```

$\displaystyle x^{2} + x + y^{2}$

```python
# expansion

expand = expand((x+1)*(x-1))
expand
```

$\displaystyle x^{2} - 1$

```python
# factorization

factor(expand)
```

$\displaystyle \left(x - 1\right) \left(x + 1\right)$

```python
# simplification

expr = (2*x**2-2)/(x+1)
expr
```

$\displaystyle \frac{2 x^{2} - 2}{x + 1}$

```python
simplify(expr)
```

$\displaystyle 2 x - 2$

```python
# function solving

expand
```

$\displaystyle x^{2} - 1$

```python
solve(expand, x)
```

    [-1, 1]

```python
# function evaluation

expand.subs(x, 3)
```

$\displaystyle 8$

```python
# trigonometry

trigsimp(cos(x)**2-sin(x)**2)
```

$\displaystyle \cos{\left(2 x \right)}$

```python
# differentiation

diff(expand)
```

$\displaystyle 2 x$

```python
# integration

integrate(expand)
```

$\displaystyle \frac{x^{3}}{3} - x$

```python
# limit at infinity

limit(
    1/x,
    x,
    oo,
)
```

$\displaystyle 0$

```python
# rewriting in terms of another function

cos(x).rewrite(sin)
```

$\displaystyle \sin{\left(x + \frac{\pi}{2} \right)}$

```python

```


```python
import numpy as np
import pandas as pd
import math as mt
import matplotlib.pyplot as plt
import sympy
from sklearn.preprocessing import MinMaxScaler
```

# Caso 1: Senoide

```python
omega_min = 670*2*np.pi/60
omega_max = 1350*2*np.pi/60
omega_opt = 955*2*np.pi/60
m = 9.8
F1 = 100
k_1 = 10e3

k_2 = omega_opt**2 * m

print('k2 = {:.2f} N/m'.format(k2))

Omega_min = omega_min * (m/k_1)**0.5

Omega_max = omega_max * (m/k_1)**0.5

Omega_eval = np.linspace(Omega_min,Omega_max,num=200)

omega_r = (k_2/k_1)**0.5

a1 = 1 + 2 * omega_r**2

a2 = omega_r**2

D0_min = Omega_min**4 - a1 * Omega_min**2 + a2

D0_max = Omega_max**4 - a1 * Omega_max**2 + a2

H11_max = (omega_r**2 - Omega_max**2)/D0_max

H21_max = omega_r**2/D0_max

H11_min = (omega_r**2 - Omega_min**2)/D0_min

H21_min = omega_r**2/D0_min

print('H11_max = {:.2f}'.format(H11_max))
print('H21_max = {:.2f}'.format(H21_max))
print('H11_min = {:.2f}'.format(H11_min))
print('H21_min = {:.2f}'.format(H21_min))
```

    k2 = 98014.44 N/m
    H11_max = 0.97
    H21_max = -0.97
    H11_min = -0.08
    H21_min = -0.15

```python
D0_eval = Omega_eval**4 - a1 * Omega_eval**2 + a2

H11_eval = (omega_r**2 - Omega_eval**2)/D0_eval

H21_eval = omega_r**2/D0_eval

fig = plt.figure(figsize=[16,9])
ax = fig.subplots(1,1)

ax.set_ylabel('$H_{j1}(\Omega)$',fontsize=16)
ax.set_xlabel('$\Omega$',fontsize=16)
ax.set_title('Gráfico da amplitude adimensional em\nfunção de $\Omega$ com absorvedor',fontsize=20)

ax.plot(Omega_eval,H11_eval,'g', linewidth=2, label = '$H11')
ax.plot(Omega_eval,H21_eval,'b', linewidth=2, label = '$H21')
ax.plot(Omega_eval,(H11_eval-H21_eval),'r', linewidth=2, label = '$H21')

ax.grid()
ax.legend()
plt.show()
fig.savefig('trabalho15_com.png', bbox_inches='tight')
```

```python
H11_eval = 1/(1-Omega_eval**2)

fig = plt.figure(figsize=[16,9])
ax = fig.subplots(1,1)

ax.set_ylabel('$H_{11}(\Omega)$',fontsize=16)
ax.set_xlabel('$\Omega$',fontsize=16)
ax.set_title('Gráfico da amplitude adimensional em\nfunção de $\Omega$ sem absorvedor',fontsize=20)

ax.plot(Omega_eval,H11_eval,'r', linewidth=2, label = '$H11')

ax.grid()
ax.legend()
plt.show()
fig.savefig('trabalho15_sem.png', bbox_inches='tight')
```

```python

```


```python
from sympy.physics.mechanics import ReferenceFrame,Point,dynamicsymbols
from sympy.physics.mechanics import Point
from sympy import latex,pprint,symbols,init_printing
from sympy.algebras.quaternion import Quaternion
import numpy as np
init_printing() # Para visualizar símbolos
```

## 2.1 Representación de la orientación de cuerpos

## 2.1.1 Matrices de rotación

```python
q1,q2=dynamicsymbols('q1,q2')
a=ReferenceFrame('A')
e=a.orientnew('E','Axis',(q1,a.y))
b=e.orientnew('B','Axis',(q2,e.z))

r1,r2,r3=symbols('r1,r2,r3')
r=r1*b.x+r2*b.y+r3*b.z
# Se define r por sus componentes en b
r
```

```python
# Recuerde que puede obtener las matrices de cosenos directores con el método dcm
# Por ejemplo ^A R^b
a.dcm(b)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & - \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} & \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\\\sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & 0\\- \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\end{matrix}\right]$

```python
# ^A R^e
a.dcm(e)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} & 0 & \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\\0 & 1 & 0\\- \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} & 0 & \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\end{matrix}\right]$

```python
# Ahora calcule el producto de las dos rotaciones
a.dcm(e)*e.dcm(b)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & - \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} & \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\\\sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & 0\\- \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\end{matrix}\right]$

```python
# Obteniendo el mismo resultado con
a.dcm(b)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & - \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} & \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\\\sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & 0\\- \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\end{matrix}\right]$

```python
# Usando el sistema simbólico puede verificar la igualdad de dos expresiones
a.dcm(e)*e.dcm(b)==a.dcm(b)
```

    True

```python
# Observe que el producto no es conmutativo
e.dcm(b)*a.dcm(e)==a.dcm(b)
```

    False

## Angulos de Euler

```python
a=ReferenceFrame('A')
phi,theta,psi=symbols('phi,theta,psi')
# Defina el marco de referencia D con rotaciones sucesivas XYZ
d=a.orientnew('D','Body',(phi,theta,psi),'XYZ')
# Esto es equivalente a tres rotaciones con marcos intermedios
e1=a.orientnew('E1','Axis',(phi,a.x))
e2=e1.orientnew('E2','Axis',(theta,e1.y))
e3=e2.orientnew('E3','Axis',(psi,e2.z))

print('rotx (^AR^E1):')
pprint(a.dcm(e1))
print('roty (^E1R^E2):')
pprint(e1.dcm(e2))
print('rotz (^E2R^E3):')
pprint(e2.dcm(e3))
```

    rotx (^AR^E1):
    ⎡1    0        0   ⎤
    ⎢                  ⎥
    ⎢0  cos(φ)  -sin(φ)⎥
    ⎢                  ⎥
    ⎣0  sin(φ)  cos(φ) ⎦
    roty (^E1R^E2):
    ⎡cos(θ)   0  sin(θ)⎤
    ⎢                  ⎥
    ⎢   0     1    0   ⎥
    ⎢                  ⎥
    ⎣-sin(θ)  0  cos(θ)⎦
    rotz (^E2R^E3):
    ⎡cos(ψ)  -sin(ψ)  0⎤
    ⎢                  ⎥
    ⎢sin(ψ)  cos(ψ)   0⎥
    ⎢                  ⎥
    ⎣  0        0     1⎦

```python
#El producto de estas rotaciones corresponde a ^AR^D
a.dcm(e1)*e1.dcm(e2)*e2.dcm(e3)
#a.dcm(e3)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\psi \right)} \cos{\left(\theta \right)} & - \sin{\left(\psi \right)} \cos{\left(\theta \right)} & \sin{\left(\theta \right)}\\\sin{\left(\phi \right)} \sin{\left(\theta \right)} \cos{\left(\psi \right)} + \sin{\left(\psi \right)} \cos{\left(\phi \right)} & - \sin{\left(\phi \right)} \sin{\left(\psi \right)} \sin{\left(\theta \right)} + \cos{\left(\phi \right)} \cos{\left(\psi \right)} & - \sin{\left(\phi \right)} \cos{\left(\theta \right)}\\\sin{\left(\phi \right)} \sin{\left(\psi \right)} - \sin{\left(\theta \right)} \cos{\left(\phi \right)} \cos{\left(\psi \right)} & \sin{\left(\phi \right)} \cos{\left(\psi \right)} + \sin{\left(\psi \right)} \sin{\left(\theta \right)} \cos{\left(\phi \right)} & \cos{\left(\phi \right)} \cos{\left(\theta \right)}\end{matrix}\right]$

## 2.1.3 Rotaciones respecto a ejes fijos

```python
a=ReferenceFrame('A')
phi,theta,psi=symbols('phi,theta,psi')
# Defina el marco de referencia D con rotaciones en ejes fijos
# En el orden psi (en ax), theta (en ay), phi (en az)
b=a.orientnew('B','Space',(psi,theta,phi),'XYZ')
a.dcm(b)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\phi \right)} \cos{\left(\theta \right)} & - \sin{\left(\phi \right)} \cos{\left(\psi \right)} + \sin{\left(\psi \right)} \sin{\left(\theta \right)} \cos{\left(\phi \right)} & \sin{\left(\phi \right)} \sin{\left(\psi \right)} + \sin{\left(\theta \right)} \cos{\left(\phi \right)} \cos{\left(\psi \right)}\\\sin{\left(\phi \right)} \cos{\left(\theta \right)} & \sin{\left(\phi \right)} \sin{\left(\psi \right)} \sin{\left(\theta \right)} + \cos{\left(\phi \right)} \cos{\left(\psi \right)} & \sin{\left(\phi \right)} \sin{\left(\theta \right)} \cos{\left(\psi \right)} - \sin{\left(\psi \right)} \cos{\left(\phi \right)}\\- \sin{\left(\theta \right)} & \sin{\left(\psi \right)} \cos{\left(\theta \right)} & \cos{\left(\psi \right)} \cos{\left(\theta \right)}\end{matrix}\right]$

## 2.1.4 Representación eje-angulo

```python
a=ReferenceFrame('A')
phi,nx,ny,nz=symbols('Phi,nx,ny,nz')
# Defina el vector n con componentes en A
n=nx*a.x+ny*a.y+nz*a.z
b=a.orientnew('B','Axis',(phi,n))
# Se utiliza el metodo subs para sustituir la norma L2 del vector como unitaria.
a.dcm(b).simplify().subs(nx**2+ny**2+nz**2,1)
```

$\displaystyle \left[\begin{matrix}nx^{2} + \left(ny^{2} + nz^{2}\right) \cos{\left(\Phi \right)} & nx ny \left(1 - \cos{\left(\Phi \right)}\right) - nz \sin{\left(\Phi \right)} & nx nz \left(1 - \cos{\left(\Phi \right)}\right) + ny \sin{\left(\Phi \right)}\\nx ny \left(1 - \cos{\left(\Phi \right)}\right) + nz \sin{\left(\Phi \right)} & ny^{2} + \left(nx^{2} + nz^{2}\right) \cos{\left(\Phi \right)} & - nx \sin{\left(\Phi \right)} + ny nz \left(1 - \cos{\left(\Phi \right)}\right)\\nx nz \left(1 - \cos{\left(\Phi \right)}\right) - ny \sin{\left(\Phi \right)} & nx \sin{\left(\Phi \right)} + ny nz \left(1 - \cos{\left(\Phi \right)}\right) & nz^{2} + \left(nx^{2} + ny^{2}\right) \cos{\left(\Phi \right)}\end{matrix}\right]$

## 2.1.5 Representación usando cuaterniones

```python
a=ReferenceFrame('A')
e0,e1,e2,e3=symbols('e0,e1,e2,e3')
b=a.orientnew('B','quaternion',(e0,e1,e2,e3))
a.dcm(b)
```

$\displaystyle \left[\begin{matrix}e_{0}^{2} + e_{1}^{2} - e_{2}^{2} - e_{3}^{2} & - 2 e_{0} e_{3} + 2 e_{1} e_{2} & 2 e_{0} e_{2} + 2 e_{1} e_{3}\\2 e_{0} e_{3} + 2 e_{1} e_{2} & e_{0}^{2} - e_{1}^{2} + e_{2}^{2} - e_{3}^{2} & - 2 e_{0} e_{1} + 2 e_{2} e_{3}\\- 2 e_{0} e_{2} + 2 e_{1} e_{3} & 2 e_{0} e_{1} + 2 e_{2} e_{3} & e_{0}^{2} - e_{1}^{2} - e_{2}^{2} + e_{3}^{2}\end{matrix}\right]$

```python
#Puede aplicar rotaciones sucesivas con cuaterniones
# usando la clase Quaternion
a1,a2,b1,b2,c1,c2,d1,d2=symbols('a1,a2,b1,b2,c1,c2,d1,d2')
# Defina los cuaterniones q1 y q2
q1=Quaternion(a1,b1,c1,d1)
q2=Quaternion(a2,b2,c2,d2)
# q3 producto de q1*q2
q3=q1*q2

# Defina el marco B con q3
b=a.orientnew('B','quaternion',q3.args)

# Alternativamente puede definir los marcos E1 y E2 con q1 y q2 respectivamente
e1=a.orientnew('E1','quaternion',q1.args)
e2=e1.orientnew('E2','quaternion',q2.args)

# Observe que el resultado es el mismo
a.dcm(e2).expand()==a.dcm(b).expand()
```

    True

## 2.2 Velocidad angular

```python
q1,q2=dynamicsymbols('q1,q2')
a=ReferenceFrame('A')
e=a.orientnew('E','Axis',(q1,a.y))
b=e.orientnew('B','Axis',(q2,-e.z))
# Puede calcular la velocidad angular de B respecto a A
# con el método ang_vel_in.
b.ang_vel_in(a).express(b)
```

## 2.3 Velocidad angular y derivadas de vectores

## 2.3.1 Derivada de un vector fijo a un marco

```python
theta,phi,psi=dynamicsymbols('theta,phi,psi')
r1,r2,r3=symbols('r1,r2,r3')
a=ReferenceFrame('A')
b=a.orientnew('B','Body',(theta,phi,psi),'XYZ')
#Defina el vector r
r=r1*b.x+r2*b.y+r3*b.z
# Puede calcular la derivada total del vector r en A
r.dt(a)
```

```python
# Para calcular velocidades relativas entre marcos de referencia
# es mas conveniente crear objectos de la clase Point.
# Primero defina O como origen estacionario desde el marco de referencia A
O=Point('O')
O.set_vel(a,0)
# Ahora defina el punto R
r=O.locatenew('R',r1*b.x+r2*b.y+r3*b.z)
# Calcule la velocidad respecto a A
r.vel(a)
```

## 2.3.2 Relación entre derivadas en marcos diferentes

```python
# El punto r podría tener una velocidad diferente a 0
# En este caso defina las variables r1,r2,r3 como símbolos dinámicos
# (i.e. r1(t),r2(t),r3(t))
r1,r2,r3=dynamicsymbols('r1,r2,r3')
theta,phi,psi=dynamicsymbols('theta,phi,psi')
a=ReferenceFrame('A')
b=a.orientnew('B','Body',(theta,phi,psi),'XYZ')
O=Point('O')
O.set_vel(a,0)
# Ahora defina el punto R
r=O.locatenew('R',r1*b.x+r2*b.y+r3*b.z)
# Calcule la velocidad respecto a A y observe que aparecen los términos
# dr_i/dt, que corresponden a la velocidad de r en el marco B.
r.vel(a)
```

```python
# Aplicando esto para el caso del modelo giro-péndulo
q1,q2=dynamicsymbols('q1,q2')
# Variables de distancia l1: longitud del péndulo, R: radio del disco
l1,R=symbols('l1,R')

#Defina marcos de referencia
a=ReferenceFrame('A')
e=a.orientnew('E','Axis',(q1,a.y))
b=e.orientnew('B','Axis',(q2,-e.z))

#Defina el punto de origen
O=Point('O')
O.set_vel(a,0)
# Ahora defina el punto Q en el centro del disco
q=O.locatenew('Q',l1*e.z)
# Defina el punto P en la periferia del disco
p=q.locatenew('P',R*b.y)

# Calcule la velocidad del vector w, definido desde Q hasta P.
w=p.pos_from(q)
w.dt(a)
```

## 2.4 Velocidad angular simple

```python
# Para el ejemplo del modelo giro-péndulo observe
# la velocidad angular de E en A y de B en E.
a=ReferenceFrame('A')
e=a.orientnew('E','Axis',(q1,a.y))
b=e.orientnew('B','Axis',(q2,-e.z))
e.ang_vel_in(a).express(e)
#b.ang_vel_in(e).express(e)
```

## 2.5 Teorema de adicion de velocidades angulares

```python
# Para el ejemplo del modelo giro-péndulo observe
# la velocidad angular de E en A y de B en E.
a=ReferenceFrame('A')
e=a.orientnew('E','Axis',(q1,a.y))
b=e.orientnew('B','Axis',(q2,-e.z))
# Observe que la suma de las velocidades angulares corresponde a la velocidad angular
# del cuerpo B en el marco A.
e.ang_vel_in(a).express(e)+b.ang_vel_in(e).express(e)==b.ang_vel_in(a).express(e)
```

    True

## 2.6 Relación entre la velocidad angular y las representaciones de orientacion

## 2.6.1 Relación entre la velocidad angular y las representaciones de orientación

```python
a=ReferenceFrame('A')
phi,theta,psi=dynamicsymbols('phi,theta,psi')
# Defina el marco de referencia B con rotaciones sucesivas XYZ
b=a.orientnew('B','Body',(phi,theta,psi),'XYZ')
b.ang_vel_in(a).simplify()
```

### 2.6.3 Velocidad angular y parámetros de Euler

```python
a=ReferenceFrame('A')
e0,e1,e2,e3=dynamicsymbols('e0,e1,e2,e3')
b=a.orientnew('B','quaternion',(e0,e1,e2,e3))
# Calcule la velocidad angular expresada en términos del cuaternión
# Aqui se obtiene un factor 2 adicional a las expresiones del libro.
# Omega representa la velocidad angular de B en el marco A.
# Y esta rotacion es producida por el doble producto q ⊗ r ⊗ q *.
# Puede pensar en esto como si el quaternion q realmente captura "media" rotación.
# Para mas detalles puede consultar el capítulo "Exponential map" en https://arxiv.org/pdf/1711.02508.pdf
b.ang_vel_in(a).simplify()

```

### 2.7 Aceleración angular

```python
# Para el ejemplo del modelo giro-péndulo observe
# la aceleración angular de E en A y de B en E.
a=ReferenceFrame('A')
q1,q2=dynamicsymbols('q1,q2')
e=a.orientnew('E','Axis',(q1,a.y))
b=e.orientnew('B','Axis',(q2,-e.z))

b.ang_acc_in(a)
```


# Visualization of data

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
```

```python
file_name = r'data/data.csv'
df = pd.read_csv(file_name, engine='python')
df[['R&D Spend', 'Administration', 'Marketing Spend', 'Profit']] /= 1000
df.head()
```

      R&D Spend
      Administration
      Marketing Spend
      State
      Profit

      0
      165.34920
      136.89780
      471.78410
      New York
      192.26183

      1
      162.59770
      151.37759
      443.89853
      California
      191.79206

      2
      153.44151
      101.14555
      407.93454
      Florida
      191.05039

      3
      144.37241
      118.67185
      383.19962
      New York
      182.90199

      4
      142.10734
      91.39177
      366.16842
      Florida
      166.18794

```python
fig = plt.figure(figsize=(6, 6))

plt.matshow(df.corr(), fignum=fig.number)
cb = plt.colorbar()
plt.title('Correlation Matrix')

plt.show()
```

```python
fig = plt.figure(figsize=(24, 6))

ax = fig.add_subplot(131)
ax.scatter(df['R&D Spend'], df['Profit'], c='r', s=20, marker='o')
ax.set_xlabel('R&D Spend')
ax.set_ylabel('Profit')

ax = fig.add_subplot(132)
ax.scatter(df['Marketing Spend'], df['Profit'], c='b', s=20, marker='o')
ax.set_xlabel('Marketing Spend')
ax.set_ylabel('Profit')

ax = fig.add_subplot(133, projection='3d')
ax.scatter(df['R&D Spend'], df['Marketing Spend'], df['Profit'], s=50, marker='o')
ax.set_xlabel('R&D Spend')
ax.set_ylabel('Marketing Spend')
ax.set_zlabel('Profit')

plt.show()
```

# Loss function
\begin{equation}
J(\theta) = \frac{1}{2}\sum_{i=1}^{n} \left(h_{\theta}(x^{i}) - y^{i} \right)^{2}
\end{equation}

```python
def loss(X, y, theta):
    y_hat = np.dot(X, theta)
    J = 0.5*sum(np.power(y-y_hat, 2))
    return J
```

# Batch gradient descent algorithm
Repeat until convergence:
\begin{equation}
\theta_{j} := \theta_{j} + \alpha \sum_{i=1}^{n}\left(y^{i} - h_{\theta}(x^{i}) \right)x_{j}^{i}
\end{equation}

```python
def batch_gradient_descent(X, y, theta, alpha, tolerance):
    old_theta = []
    iterations = []
    num = 0

    while 1:
        num += 1
        if num % 1000 == 0:
            iterations.append(num)
            old_theta.append(theta)

        y_hat = np.dot(X, theta)
        new_theta = theta + alpha*np.dot(X.T, y-y_hat)
        if (abs(new_theta-theta) < tolerance).all():
            break
        theta = new_theta

    return theta, iterations, old_theta
```

```python
init_theta = np.matrix((0, 0)).T
tolerance = 1e-5
learning_rate = 5e-6

y = np.matrix([df['Profit'].values]).T
x = np.matrix([np.ones(len(y)), df['R&D Spend'].values]).T
theta, iterations, old_theta = batch_gradient_descent(x, y, init_theta, learning_rate, tolerance)
pred = np.dot(x, theta)
print(theta)

J = np.zeros(len(old_theta))
for i, value in enumerate(old_theta):
    J[i] = loss(x, y, value)

fig = plt.figure(figsize=(16, 6))

ax = fig.add_subplot(121)
ax.scatter([x[:, 1]], [y], c='b', marker='x')
ax.scatter([x[:, 1]], [pred], c='r', marker='o')
ax.set_xlabel('R&D Spend')
ax.set_ylabel('Profit')

ax = fig.add_subplot(122)
ax.plot(iterations, J, 'bo')
ax.set_xlabel('Iterations')
ax.set_ylabel('Loss function')

plt.show()
```

```python
init_theta = np.matrix((0, 0, 0)).T
tolerance = 1e-5
learning_rate = 5e-6

y = np.matrix([df['Profit'].values]).T
x = np.matrix([np.ones(len(y)), df['R&D Spend'].values, df['Marketing Spend'].values]).T
x = StandardScaler().fit_transform(x)
theta, iterations, old_theta = batch_gradient_descent(x, y, init_theta, learning_rate, tolerance)
pred = np.dot(x, theta)
print(theta)

J = np.zeros(len(old_theta))
for i, value in enumerate(old_theta):
    J[i] = loss(x, y, value)

fig = plt.figure(figsize=(16, 6))

ax = fig.add_subplot(121, projection='3d')
ax.scatter(x[:, 1], x[:, 2], y, c='b', s=50, marker='x')
ax.scatter(x[:, 1], x[:, 2], pred, c='r', s=50, marker='o')
ax.set_xlabel('Normalized R&D Spend')
ax.set_ylabel('Normalized Marketing Spend')
ax.set_zlabel('Profit')

ax = fig.add_subplot(122)
ax.plot(iterations, J, 'bo')
ax.set_xlabel('Iterations')
ax.set_ylabel('Loss function')

plt.show()
```

Batch gradient descent algorithm can't be used to find a global minimum of the loss function. It converges to a local minimum but the predicted values $h_{\theta}(x^{i})$ are far from the real values $y^{i}$. It is necessary to choose other initial coefficients $\theta_{j}$.

# Stochastic gradient descent algorithm
For $i=1$ to $n$:
\begin{equation}
\theta_{j} := \theta_{j} + \alpha\left(y^{i}-h_{\theta}(x^{i}) \right)x_{j}^{i}
\end{equation}

```python
def stochastic_gradient_descent(X, y, theta, alpha, num):
    old_theta = []
    iterations = []

    for i in range(num):
        if i % 1000 == 0:
            iterations.append(i)
            old_theta.append(theta)

        y_hat = np.dot(X, theta)
        theta = theta - alpha*np.dot(X.T, y_hat-y)

    return theta, old_theta, iterations
```

```python
init_theta = np.matrix((100, 0)).T
learning_rate = 5e-6
num = 100000

y = np.matrix(df['Profit'].values).T
x = np.matrix([np.ones(len(y)), df['R&D Spend'].values]).T
theta, old_theta, iterations = stochastic_gradient_descent(x, y, init_theta, learning_rate, num)
pred = np.dot(x, theta)
print(theta)

J = np.zeros(len(old_theta))
for i, value in enumerate(old_theta):
    J[i] = loss(x, y, value)

fig = plt.figure(figsize=(16, 6))

ax = fig.add_subplot(121)
ax.scatter([x[:,1]], [y], c='b', marker='x')
ax.scatter([x[:,1]], [pred], c='r', marker='o')
ax.set_xlabel('R&D Spend')
ax.set_ylabel('Profit')

ax = fig.add_subplot(122)
ax.plot(iterations, J, 'bo')
ax.set_xlabel('Iterations')
ax.set_ylabel('Loss function')

plt.show()
```

```python
init_theta = np.matrix((0, 0, 0)).T
learning_rate = 5e-6
num = 100000

y = np.matrix([df['Profit'].values]).T
x = np.matrix([np.ones(len(y)), df['R&D Spend'].values, df['Marketing Spend'].values]).T
x = StandardScaler().fit_transform(x)
theta, old_theta, iterations = stochastic_gradient_descent(x, y, init_theta, learning_rate, num)
pred = np.dot(x, theta)
print(theta)

J = np.zeros(len(old_theta))
for i, value in enumerate(old_theta):
    J[i] = loss(x, y, value)

fig = plt.figure(figsize=(16, 6))

ax = fig.add_subplot(121, projection='3d')
ax.scatter(x[:, 1], x[:, 2], y, c='b', s=50, marker='x')
ax.scatter(x[:, 1], x[:, 2], pred, c='r', s=50, marker='o')
ax.set_xlabel('Normalized R&D Spend')
ax.set_ylabel('Normalized Marketing Spend')
ax.set_zlabel('Profit')

ax = fig.add_subplot(122)
ax.plot(iterations, J, 'bo')
ax.set_xlabel('Iterations')
ax.set_ylabel('Loss function')

plt.show()
```

Stochastic gradient descent algorithm can't be used to find a global minimum of the loss function neither.

# Scikit-learn Linear Regression model

```python
y = np.matrix(df['Profit'].values).T
x = np.matrix([np.ones(len(y)), df['R&D Spend'].values]).T
model = LinearRegression(fit_intercept=True)
model.fit(x[:,1], y)
print(f'Intercept: {model.intercept_}')
print(f'Coefficients: {model.coef_}')

xfit = np.linspace(0, 175, 1000)
yfit = model.predict(xfit.reshape(-1, 1))

fig = plt.figure(figsize=(8, 6))

ax = fig.add_subplot(111)
ax.scatter([x[:,1]], [y], c='b', marker='x')
ax.plot(xfit, yfit, 'r')
ax.set_xlabel('R&D Spend')
ax.set_ylabel('Profit')

plt.show()
```

```python
y = np.matrix(df['Profit'].values).T
x = np.matrix([np.ones(len(y)), df['R&D Spend'].values, df['Marketing Spend'].values]).T
x = StandardScaler().fit_transform(x)
model = LinearRegression(fit_intercept=True)
model.fit(x[:,1:], y)
print(f'Intercept: {model.intercept_}')
print(f'Coefficients: {model.coef_}')

xfit_1 = np.linspace(-2, 2, 1000)
xfit_2 = np.linspace(-2, 2, 1000)
xfit = np.column_stack((xfit_1, xfit_2))
yfit = model.predict(xfit)
xfit_1, xfit_2 = np.meshgrid(xfit_1, xfit_2)

fig = plt.figure(figsize=(10, 8))

ax = fig.add_subplot(111, projection='3d')
ax.scatter(x[:,1], x[:,2], y, c='b', marker='x')
ax.plot_surface(xfit_1, xfit_2, yfit, rstride=25, cstride=25, color='red', alpha=0.5)
ax.set_xlabel('Normalized R&D Spend')
ax.set_ylabel('Normalized Marketing Spend')
ax.set_zlabel('Profit')

plt.show()
```


```
import os, sys
sys.path.insert(0, os.path.join(os.pardir, 'src'))

def sympy_solution():
    from sympy import symbols, Rational, solve
    C1, C3, C4 = symbols('C1 C3 C4')
    s = solve([C1 - 1 - C3,
           C1 - Rational(1,2) - C3 - C4,
	   2 + 2*C3 + C4], [C1,C3,C4])
    return s

import numpy as np
import matplotlib.pyplot as plt

def plot_exact_solution():
    x = np.linspace(0, 2, 101)
    u = exact_solution(x)
    plt.plot(x, u)
    plt.xlabel('$x$'); plt.ylabel('$u$')
    ax = plt.gca(); ax.set_aspect('equal')
    plt.savefig('tmp.png');  plt.savefig('tmp.pdf')

def exact_solution(x):
    if isinstance(x, np.ndarray):
        return np.where(x < 1, -1./4*x, 0.5*x**2 - 5./4*x + 0.5)
    else:
        return -1./4*x if x < 1 else 0.5*x**2 - 5./4*x + 0.5

def sine_solution(x, N):
    from numpy import pi, sin
    s = 0
    u = [] # u[i] is the solution for N=i
    for i in range(N+1):
        if i % 4 == 0:
            cos_min_cos = -1
        elif (i-1) % 4 == 0:
            cos_min_cos = 2
        elif (i-2) % 4 == 0:
            cos_min_cos = -1
        elif (i-1) % 4 == 0:
            cos_min_cos = 0

        b_i = 2/(pi*(i+1))*cos_min_cos
        A_ii = (i+1)**2*pi**2/4
        c_i = b_i/A_ii
        s += c_i*sin((i+1)*x*pi/2)
        u.append(s.copy())
    return u

def plot_sine_solution():
    x = np.linspace(0, 2, 101)
    u = sine_solution(x, N=20)
    plt.figure()
    x = np.linspace(0, 2, 101)
    plt.plot(x, exact_solution(x), '--')
    N_values = 0, 1, 5
    for N in 0, 1, 5, 10:
        plt.plot(x, u[N])
    plt.legend(['exact'] + ['N=%d' % N for N in N_values])
    plt.savefig('tmp2.png');  plt.savefig('tmp2.pdf')

def P1_solution():
    plt.figure()
    from fe1D import mesh_uniform, u_glob
    N_e_values = [2, 4, 8]
    d = 1
    legends = []
    for N_e in N_e_values:
        vertices, cells, dof_map = mesh_uniform(
            N_e=N_e, d=d, Omega=[0,2], symbolic=False)
        h = vertices[1] - vertices[0]
        Ae = 1./h*np.array(
            [[1, -1],
             [-1, 1]])
        N = N_e + 1
        A = np.zeros((N, N))
        b = np.zeros(N)
        for e in range(N_e):
            if vertices[e] >= 1:
                be = -h/2.*np.array(
                    [1, 1])
            else:
                be = h/2.*np.array(
                    [0, 0])
            for r in range(d+1):
                for s in range(d+1):
                    A[dof_map[e][r], dof_map[e][s]] += Ae[r,s]
                b[dof_map[e][r]] += be[r]
        # Enforce boundary conditions
        A[0,:] = 0; A[0,0] = 1; b[0] = 0
        A[-1,:] = 0; A[-1,-1] = 1; b[-1] = 0
        c = np.linalg.solve(A, b)

        # Plot solution
        print(('c:', c))
        print(('vertices:', vertices))
        print(('cells:', cells))
        print(('len(cells):', len(cells)))
        print(('dof_map:', dof_map))
        xc, u, nodes = u_glob(c, vertices, cells, dof_map)
        plt.plot(xc, u)
        legends.append('$N_e=%d$' % N_e)
    plt.plot(xc, exact_solution(xc), '--')
    legends.append('exact')
    plt.legend(legends, loc='lower left')
    plt.savefig('tmp3.png'); plt.savefig('tmp3.pdf')

if __name__ == '__main__':
    print((sympy_solution()))
    plot_sine_solution()
    P1_solution()
    plt.show()

```

```

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/04_CABLE_DISCONT_LOAD.ipynb


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.0 (Python 3.6.2-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

## Quadratic

We want to construct a quadratic polynomal through that points $x_{i-1}$, $x_i$, $x_{i+1}$ that gives the correct averages,
$f_{i-1}$, $f_i$, and $f_{i-1}$ when integrated over the volume, e.g.

$$\frac{1}{\Delta x} \int_{x_{i-1/2}}^{x_{i+1/2}} f(x) dx = f_i$$

There are 3 unknowns in the quadratic and three constraints, so this is a linear system we can solve.

Define the quadratic polynomial

```python
a, b, c = symbols("a b c")
x0 = symbols("x0")
f = a*(x-x0)**2 + b*(x-x0) + c
f
```

```python
dx = symbols("\Delta")
```

### constraints

Define the 3 constraint equations---here we set them up construct $A$, $B$, and $C$ as the integrals over the 3 control volumes

```python
fm, f0, fp = symbols("f_{i-1} f_i f_{i+1}")
#xm32, xm12, xp12, xp32 = symbols("x_{i-3/2} x_{i-1/2} x_{i+1/2} x_{i+3/2}")
xm32 = x0 - Rational(3,2)*dx
xm12 = x0 - Rational(1,2)*dx
xp12 = x0 + Rational(1,2)*dx
xp32 = x0 + Rational(3,2)*dx
```

interfaces

```python
xm32, xm12, xp12, xp32
```

```python
A = simplify(integrate(f/dx, (x, xm32, xm12)))
B = simplify(integrate(f/dx, (x, xm12, xp12)))
C = simplify(integrate(f/dx, (x, xp12, xp32)))
```

The analytic forms of the integrals

```python
A, B, C
```

Our linear system is now:

$$A = f_{i-1}$$
$$B = f_i$$
$$C = f_{i+1}$$

Now find the coefficients of the polynomial

```python
coeffs = solve([A-fm, B-f0, C-fp], [a,b,c])
coeffs
```

And in pretty form, here's the polynomial

```python
f.subs(a,coeffs[a]).subs(b,coeffs[b]).subs(c,coeffs[c])
```

### Cubic

We want to construct a cubic polynomal through that points $x_{i-2}$, $x_{i-1}$, $x_i$, $x_{i+1}$ that gives the correct averages,
$f_{i-2}$, $f_{i-1}$, $f_i$, and $f_{i-1}$ when integrated over the volume of each zone

```python
a, b, c, d = symbols("a b c d")
f = a*(x-x0)**3 + b*(x-x0)**2 + c*(x-x0) + d
f
```

Now perform the integals of $f(x)$ over each zone

```python
fm2, fm, f0, fp = symbols("f_{i-2} f_{i-1} f_i f_{i+1}")
xm52 = x0 - Rational(5,2)*dx
xm32 = x0 - Rational(3,2)*dx
xm12 = x0 - Rational(1,2)*dx
xp12 = x0 + Rational(1,2)*dx
xp32 = x0 + Rational(3,2)*dx
```

interfaces

```python
xm52, xm32, xm12, xp12, xp32
```

```python
A = simplify(integrate(f/dx, (x, xm52, xm32)))
B = simplify(integrate(f/dx, (x, xm32, xm12)))
C = simplify(integrate(f/dx, (x, xm12, xp12)))
D = simplify(integrate(f/dx, (x, xp12, xp32)))
```

```python
A, B, C, D
```

```python
coeffs = solve([A-fm2, B-fm, C-f0, D-fp], [a,b,c,d], check=False)
coeffs
```

and the pretty form of the polynomial

```python
fc = f.subs(a,coeffs[a]).subs(b,coeffs[b]).subs(c,coeffs[c]).subs(d,coeffs[d])
fc
```

this interpolant is symmetric about the $i-1/2$ interface---let's see the value there

```python
fc.subs(x,x0-Rational(1,2)*dx)
```

Note that this is the interpolating polynomial used to find the interface states in PPM (Colella & Woodward 1984)

## Quartic

Now we define a quartic polynomial that gives the correct averages over 5 zones, $x_{i-2}$, $x_{i-1}$, $x_i$, $x_{i+1}$, $x_{i+2}$,
with zone averages $f_{i-2}$, $f_{i-1}$, $f_i$, $f_{i+1}$, $f_{i+2}$

```python
a, b, c, d, e = symbols("a b c d e")
x0 = symbols("x0")
f = a*(x-x0)**4 + b*(x-x0)**3 + c*(x-x0)**2 + d*(x-x0) + e
f
```

Now we perform the integrals of $f(x)$ over each zone

```python
fm2, fm, f0, fp, fp2 = symbols("f_{i-2} f_{i-1} f_i f_{i+1} f_{i+2}")
#xm32, xm12, xp12, xp32 = symbols("x_{i-3/2} x_{i-1/2} x_{i+1/2} x_{i+3/2}")
xm52 = x0 - Rational(5,2)*dx
xm32 = x0 - Rational(3,2)*dx
xm12 = x0 - Rational(1,2)*dx
xp12 = x0 + Rational(1,2)*dx
xp32 = x0 + Rational(3,2)*dx
xp52 = x0 + Rational(5,2)*dx
```

interfaces

```python
xm52, xm32, xm12, xp12, xp32, xp52
```

```python
A = simplify(integrate(f/dx, (x, xm52, xm32)))
B = simplify(integrate(f/dx, (x, xm32, xm12)))
C = simplify(integrate(f/dx, (x, xm12, xp12)))
D = simplify(integrate(f/dx, (x, xp12, xp32)))
E = simplify(integrate(f/dx, (x, xp32, xp52)))
```

The analytic form of the constraints

```python
A, B, C, D, E
```

Now find the coefficients

```python
coeffs = solve([A-fm2, B-fm, C-f0, D-fp, E-fp2], [a,b,c,d,e], check=False)
coeffs
```

and the pretty form of the polynomial

```python
f.subs(a,coeffs[a]).subs(b,coeffs[b]).subs(c,coeffs[c]).subs(d,coeffs[d]).subs(e,coeffs[e])
```


# System of Equations Example - SIR Epidemic Model
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
import numpy as np
from numpy import *
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
from scipy.integrate import odeint
```

# The SIR Model for Epidemics

The SIR model for epidemics consists of three ODEs
\begin{equation}
\frac{\text{d}S}{\text{d}t} = -\beta I S
\end{equation}
\begin{equation}
\frac{\text{d}I}{\text{d}t} = \beta I S - \gamma I
\end{equation}
\begin{equation}
\frac{\text{d}R}{\text{d}t} =  \gamma I
\end{equation}

```python
def rhs_sir(PHI, t, N, beta, gamma):
    S = PHI[0]
    I = PHI[1]
    R = PHI[2]
    dSdt = -beta/N * S * I
    dIdt = beta/N * S * I - gamma * I
    dRdt = gamma * I
    return [dSdt, dIdt, dRdt]
```

```python
N  = 3e3 # total population
I0 = 10 # 10 sick individuals
R0 = 0 # 0 people recovered
S0 = N - I0 # we need to conserve the total population
β  = 0.6
γ  = 1/20.0
tend = 100 # days

t = np.linspace(0,tend,int(tend))
# Initial conditions vector
y0 = [S0, I0, R0]
sol = odeint(rhs_sir, y0, t, args=(N, β, γ))
S = sol[:,0]
I = sol[:,1]
R = sol[:,2]
```

```python
plt.plot(t, S/N, 'b', alpha=0.65, lw=2, label='Susceptible')
plt.plot(t, I/N, 'r', alpha=0.65, lw=2, label='Infected')
plt.plot(t, R/N, 'g', alpha=0.65, lw=2, label='Recovered with immunity')
plt.xlabel('# days')
plt.ylabel('Fraction of Population')
plt.xlim(0,tend)
plt.ylim(0,1.1)
plt.legend()
plt.title('SIR Model for Pandemics')
plt.grid()
# plt.savefig('SIR model.pdf')
# plt.show()
```

# Interactive Simulation

```python
def plot_sir(β,days,tend,semilogy=False):
    N = 10 # total population
    I0 = 1 # 10 sick individuals
    R0 = 0 # 0 people recovered
    S0 = N - I0 # we need to conserve the total population
    tend = tend*30 # tend is given in months
    t = np.linspace(0,tend,int(tend))
    γ = 1.0/days
    # Initial conditions vector
    y0 = [S0, I0, R0]
    # Integrate the SIR equations over the time grid, t.
    sol = odeint(rhs_sir, y0, t, args=(N, β, γ))
    [S, I, R] = sol.T

    f = plt.plot
    if (semilogy):
        f =plt.semilogy
    f(t/30, S, 'b', alpha=0.65, lw=2, label='Susceptible')
    f(t/30, I, 'r', alpha=0.65, lw=2, label='Infected')
    f(t/30, R, 'g', alpha=0.65, lw=2, label='Recovered')
    plt.xlabel('# months since outbreak')
    plt.ylabel('# Population (1000s)')
#     plt.xlim(0,tend/30)
    plt.ylim(bottom=1e-4)
    plt.legend()
    plt.title('SIR Model for Epidemics')
    plt.grid()
    # plt.savefig('SIR model.pdf')
    # plt.show()
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
style = {'description_width': 'initial'}
β = widgets.FloatSlider(value=0.143,min=0.001,max=1,step=0.001,description='$\\beta$:',continuous_update=False)
days = widgets.FloatSlider(value=14,min=1,max=24,step=0.1, description='Recovery (days):' ,style=style,continuous_update=False)
tend = widgets.FloatSlider(value=12,min=1,max=32,step=0.5, description='Simulation Time (months):',style=style,continuous_update=False)
semilogy = widgets.Checkbox(value=False,description='semilogy plot', style=style)
ui1 = widgets.HBox([β,days])
ui2 = widgets.HBox([tend,semilogy])
out = widgets.interactive_output(plot_sir, {'β': β, 'days': days, 'tend': tend, 'semilogy':semilogy})

display(ui1,ui2, out)
```

    HBox(children=(FloatSlider(value=0.143, continuous_update=False, description='$\\beta$:', max=1.0, min=0.001, …

    HBox(children=(FloatSlider(value=12.0, continuous_update=False, description='Simulation Time (months):', max=3…

    Output()

# The SIR Model for Epidemics - with loss of immunity

The SIR model for epidemics consists of three ODEs
\begin{equation}
\frac{\text{d}S}{\text{d}t} = -\beta I S + \mu R
\end{equation}
\begin{equation}
\frac{\text{d}I}{\text{d}t} = \beta I S - \gamma I
\end{equation}
\begin{equation}
\frac{\text{d}R}{\text{d}t} =  \gamma I - \mu R
\end{equation}

```python
def rhs_sir_immunity(SIR, t, N, β, γ,μ):
    [S, I, R] = SIR #SIR is a vector of solutions
    dSdt = -β/N * S * I + μ*R
    dIdt = β/N * S * I - γ * I
    dRdt = γ * I - μ*R
    return [dSdt, dIdt, dRdt]
```

```python
N = 3e3 # total population
I0 = 10 # 10 sick individuals
R0 = 0 # 0 people recovered
S0 = N - I0 # we need to conserve the total population
β = 0.6
γ = 1/20.0
μ = 1/320.0

tend = 600 # days
t = np.linspace(0,tend,tend)
# Initial conditions vector
y0 = [S0, I0, R0]
# Integrate the SIR equations over the time grid, t.
sol = odeint(rhs_sir_immunity, y0, t, args=(N, β, γ,μ))
[S, I, R] = sol.T
```

```python
plt.plot(t, S/N, 'b', alpha=0.65, lw=2, label='Susceptible')
plt.plot(t, I/N, 'r', alpha=0.65, lw=2, label='Infected')
plt.plot(t, R/N, 'g', alpha=0.65, lw=2, label='Recovered with immunity')
plt.xlabel('# days')
plt.ylabel('Fraction of Population')
plt.xlim(0,tend)
plt.ylim(0,1.1)
plt.legend()
plt.title('SIR Model for Pandemics')
plt.grid()
# plt.savefig('SIR model.pdf')
# plt.show()
```

```python
plt.plot(S,I)
```

    [<matplotlib.lines.Line2D at 0x1518402a10>]

```python

```

topics/initial-value-problems/System of Equations Example - SIR Epidemic Model.ipynb


```python
from math import e, pi
```

```python
from sympy import *
```

```python
x = Symbol('x')
y = Symbol('y')

f = e**(x**2 + y**2)
f
```

$\displaystyle 2.71828182845905^{x^{2} + y^{2}}$

```python
def double_integral(a,b,c,d,n,m):

    h = (b-a)/n
    k = (d-c)/m

    print(f'h = {h}')
    print(f'k = {k}')

    sumPairK = 0
    sumPairH = 0
    sumNotPairK = 0
    sumNotPairH = 0

    for i in range(1,(m//2)+1):
        sumNotPairK += f.subs(y,c + (2*i - 1)*k)

    for i in range(1,(m//2)):
        sumPairK += f.subs(y,c + 2*i*k)

    g = (k/3)*(f.subs(y,c) + 4*sumNotPairK + 2*sumPairK + f.subs(y,d))

    for i in range(1,(n//2)+1):
        sumNotPairH += g.subs(x, a + (2*i - 1)*h)

    for i in range(1,(n//2)):
        sumPairH += g.subs(x,a + 2*i*h)

    I = (h/3)*(g.subs(x,a) + 4*sumNotPairH + 2*sumPairH + g.subs(x,b))

    print(f'Respuesta: {I.evalf()}')
```

```python
double_integral(0,1,0,x,6,6)
```

    h = 0.16666666666666666
    k = x/6
    Respuesta: 1.07103908086675


```python
import sympy as sym

# Computing with Dirichlet conditions: -u''=2 and sines
x, L = sym.symbols('x L')
e_Galerkin = x*(L-x) - 8*L**2*sym.pi**(-3)*sym.sin(sym.pi*x/L)
e_colloc = x*(L-x) - 2*L**2*sym.pi**(-2)*sym.sin(sym.pi*x/L)

# Verify max error for x=L/2
dedx_Galerkin = sym.diff(e_Galerkin, x)
print((dedx_Galerkin.subs(x, L/2)))
dedx_colloc = sym.diff(e_colloc, x)
print((dedx_colloc.subs(x, L/2)))

# Compute max error: x=L/2, evaluate numerical, and simplify
print(('Max error Galerkin/least.sq.:', \
      sym.simplify(e_Galerkin.subs(x, L/2).evalf(n=3))))
print(('Max error colloc.:', \
      sym.simplify(e_colloc.subs(x, L/2).evalf(n=3))))
import sys
#sys.exit(0)

# Computing with Neumann and Dirichlet conditions: -u''=2
x, C, D = sym.symbols('x C D')
i, j = sym.symbols('i j', integer=True)

integrand = (i+1)*(j+1)*(1-x)**(i+j)
A_ij = sym.integrate(integrand, (x, 0, 1))
A_ij = sym.simplify(A_ij)
print(A_ij)
psi_i = (1-x)**(i+1)
integrand = 2*psi_i - D*(i+1)*(1-x)**i
b_i = sym.integrate(integrand, (x, 0, 1)) - C*psi_i.subs(x, 0)
b_i = sym.factor(sym.simplify(b_i))
print(b_i)
print((sym.expand(2 - (2+i)*(D+C))))

# Solving model2 problem with f(x) and fe1D.py
from u_xx_f_sympy import model2, x, C, D, L
m = 2
u = model2(x**m, L, C, D)
print(u)
#u_exact = lambda x: D + C*(x-L) + (1./6)*(L**3 - x**3)
u_exact = sym.lambdify([x, C, D, L], u)

import numpy as np
from fe1D import finite_element1D_naive, mesh_uniform
# Override C, D and L with numeric values
C = 5
D = 2
L = 4

d = 1

vertices, cells, dof_map = mesh_uniform(
    N_e=2, d=d, Omega=[0,L], symbolic=False)
vertices[1] = 3
essbc = {}
essbc[dof_map[-1][-1]] = D

c, A, b, timing = finite_element1D_naive(
    vertices, cells, dof_map,
    essbc,
    ilhs=lambda e, phi, r, s, X, x, h:
    phi[1][r](X, h)*phi[1][s](X, h),
    irhs=lambda e, phi, r, X, x, h:
    x**m*phi[0][r](X),
    blhs=lambda e, phi, r, s, X, x, h: 0,
    brhs=lambda e, phi, r, X, x, h:
    -C*phi[0][r](-1) if e == 0 else 0,
    intrule='GaussLegendre',
    verbose=False,
    )

# Visualize
from fe1D import u_glob
x, u, nodes = u_glob(c, cells, vertices, dof_map)
u_e = u_exact(x, C, D, L)
print((u_exact(nodes, C, D, L) - c))  # difference at the nodes
import matplotlib.pyplot as plt
plt.plot(x, u, 'b-', x, u_e, 'r--')
plt.legend(['finite elements, d=%d' %d, 'exact'], loc='upper left')
plt.savefig('tmp.png'); plt.savefig('tmp.pdf')
plt.show()

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/SRC/40_VARFORM1D_SESSION.ipynb