```python
import numpy as np
import math
from scipy import optimize
import numba as nb
import cap_equi as func
import interactive_plots as int_plot

import scipy.stats as stats

# For plots:
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
plt.style.use('seaborn-whitegrid')
mpl.style.use('seaborn')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
```

# PS2, part 1: Welfare effects of capital injection

The exercise consists of the core setup:
* Two periods, $t=0,1$,
* Two types of agents: Consumers and entrepreneurs.
* **Consumers:**
    * Linear utility $u_c = c_0^c+c_1^c$.
    * Large endowment of consumption goods in each period: $e$.
    * Labor endowment of $e_l=1$. Inelastically supplied in a competitive labor market at the price $w_1$.
* **Entrepreneurs:**
    * Linear utility $u_e = c_0^e+c_1^e$.
    * Small endowment $n_0$ in period 0.
    * Borrow $b_1$, invest in capital $k_1$.
    * Period 1 they hire labor $l_1$ at wage $w_1$.
    * Entrepreneurs utilize capital and labor in $t=1$ with Cobb-Douglas production function
    $$ \begin{align}
    y_1 = k_1^{\alpha}l_1^{1-\alpha} \notag
    \end{align} $$
    * **Entrepreneurs are borrowing constrained:**
    $$\begin{align}
    b_1\leq \lambda \left(y_1-w_1l_1\right), && \lambda\in (0,1) \notag
    \end{align} $$
* Assume that endowment of consumption goods $(e)$ is *sufficiently* large, for the interest rate to be 1 in equilibrium. (We will show what *sufficiently* means here).

## Q1: Three parts. (i) State the entrepreneur's problem. (ii) Argue that entrepreneurs will choose to maximize profits in period 1, a linear function of $k_1$, and (3) restate entrepreneur's problem as a simple linear problem.

### (i) Entrepreneur problem:

Go through the steps:

1. For the entrepreneur write up the maximization problem. This inludes:
    * Objective function, what is maximized? Constraints on the problem are
    * The budget for $t=0$, (with "=")
    * the budget for $t=1$, (with "=")
    * the production function, (with "=")
    * the credit constraint (with "$\leq$").
    Let us refer ot this as **(Problem 1)**.

2. Next, substitute for budgets $t=0,t=1$ into the utility function.
3. Substitute for the production function in both the function from step 2., and the credit constraint. We end up with what we refer to as **problem 2**, that should look like something like this:

$$\begin{align}
    \max_{k_1,l_1,b_1} n_0+k_1^{\alpha}l_1^{1-\alpha}-k_1-w_1l_1, && s.t. b_1\leq \lambda \left(k_1^{\alpha}l_1^{1-\alpha}-w_1l_1\right). \tag{Problem 2}
\end{align}$$

### (ii) Optimal choice of labor:

Note that profits in $t=1$ is given by:

$$\begin{align}
    \pi_1 = k_1^{\alpha}l_1^{1-\alpha}-w_1l_1.
\end{align}$$

Find the first order conditions in **(Problem 2)** defined above, and show that this implies:

$$\begin{align}
    w_1 = (1-\alpha)k_1^{\alpha}l_1^{-\alpha}
\end{align}$$

and thus also

$$\begin{align}
w_1l_1^* = (1-\alpha)k_1^{\alpha}(l_1^*)^{1-\alpha}, \tag{Labor Share}
\end{align}$$

and

$$\begin{align}
    l_1^* = \dfrac{(1-\alpha)^{1/\alpha}k_1}{w_1^{1/\alpha}}. \tag{$l^*$}
\end{align}$$

Use this to write the **profit function** as:

$$\begin{align}
    \pi_1(l=l^*) &= \underbrace{\dfrac{\alpha (1-\alpha)^{(1-\alpha)/\alpha}}{w_1^{(1-\alpha)/\alpha}}}_{\equiv R(w_1)}k_1 \tag{Profit}
\end{align}$$

### (iii) Restate as linear problem.

Use the results above. Refer to this linear problem as **(Problem 3)**.

## Q2: Derive entrepreneur's demand for capital, and argue how the solution depends on $\lambda$ and $R(w_1)$.

Problem $3$ is linear in $k_1$. Thus capital demand depends on the range of the interest rate. Explain the three potential solutions:

* If $R(w_1)<1$?
* If $\lambda R(w_1)>1$?
* If $\lambda R(w_1)<1<R(w_1)$?
* If $R(w_1)=1$?

In the figures below you can see how the demand changes with parameter values of e.g. $\lambda$. (The other line in the graph is $(\alpha k^{\alpha-1})$ which also has to hold in equilibrium)

.

.

.

.

.

.

.

Hint: Capital demand in the different ranges are given by (not necessarily correct order compared to bullets above)

$$\begin{align}
     k_1^* &= \dfrac{n_0}{1-\lambda R(w_1)} \\
     k_1^* &= 0 \\
     k_1^* &= \infty \\
      k_1^*&\in\left[0,\mbox{ }\dfrac{n_0}{1-\lambda}\right] \\
\end{align}$$

```python
# Baseline parameters:
n_0base, lambda_base, alpha_base, Inf = 0.1, 0.55, 0.9, 10**3

# Exogenous grid of R (interest rate):
R_min, R_max, N_R= 0.1, 2, 100
R_grid = np.linspace(R_min, R_max, N_R)

# Upper and lower limit of sliders to operate below:
n_0_lower, n_0_upper, n_0_step = 0, 0.5, 0.05
lambda_lower, lambda_upper, lambda_step = 0, 0.9, 0.05

# Plot:
int_plot.interactive_cap(n_0base, lambda_base, alpha_base, Inf, N_R, R_grid, n_0_lower, n_0_upper, n_0_step, lambda_lower, lambda_upper, lambda_step)
```

    interactive(children=(FloatSlider(value=0.55, description='$\\lambda$', max=0.9, step=0.05), FloatSlider(value…

```python
# Grids:
n_0_lower, n_0_upper, N_n = 0, 0.5, 51
lambda_lower, lambda_upper, N_lambda = 0, 0.9, 91
n_grid = np.linspace(n_0_lower, n_0_upper,N_n, dtype=np.float)
lambda_grid, lambda_step = np.linspace(lambda_lower, lambda_upper, N_lambda, retstep=True, dtype=np.float)
# baseline:
n_0base, lambda_base, alpha = 0.1, 0.5, 0.9
# Plot:
int_plot.interactive_equicap(N_n,N_lambda,alpha,n_grid,lambda_grid,math.ceil(N_lambda/2),lambda_lower, lambda_upper,lambda_step)
```

    interactive(children=(FloatSlider(value=0.46, description='$\\lambda$', max=0.9, step=0.01), Output()), _dom_c…

## Q3: Show the cutoff $n^*$, where $n_0 \geq n^*$$ implies that the first-best level of capital can be financed, and the borrowing constraint is not binding.

You can think of the following:

* Impose $l_1=1$ in equilibrium (endowment of labor from consumers, supplied inelastically).
* Define the first-best level of capital, as the one that **maximizes net ouptut** $(y_1-k_1)$.
* In equilibrium we further use that the market for capital is competitive, implying that the rate of return on capital has to follow:

$$\begin{align}
    R(w_1) = \alpha k_1^{\alpha-1}, && \Rightarrow && k_1 = \left(\dfrac{\alpha}{R(w_1)}\right)^{1/(1-\alpha)}
\end{align}$$

Given this and the capital demand function from Q2, when is the first-best level feasible?

## Q4: Show that if n<n* there is an equilibrium where the entrepreneurs are constrained and $k_1$ becomes an increasing function of $n_0$.

It follows from Q2-Q3 that the threshold exists, and that the entrepreneur is credit-constrained in this region. Now:

* Use the relevant capital demand from Q2.
* Plug in the expression for equilibrium $R(w_1)$.
* Solve for $k$ in this expression.

**Note:** When the entrepreneur is credit-constrained, $k_1$ is not only an increasing function of $n_0$, the effect of increased equity is **more** than 1-to-1. Why is this the case?

## Q5: Introduce a lump-sum tax on consumers in period 0 that transfers wealth to entrepreneurs. Derive expected utility as a function of the lump sum tax rate.

Hints:

* Split up the consideration into the region where E is credit-constrained, and where E is not.
* You can use all of the solutions from above and simply replace $n_0$ with $n_0+\tau$ for the entrepreneur.

You do not need to arrive at exactly this expression, but show that the effect of an increase in $\tau$ has the following effect on utility of entrepreneurs:

$$\begin{align}
    \dfrac{\partial U_e}{\partial \tau} = \left\lbrace \begin{array}{ll}  1 & n_0+\tau\geq n^* \\
    \dfrac{\alpha R(w_1)(1-\lambda)}{1-\alpha \lambda R(w_1)} & n_0+\tau<n^* \end{array}\right.
\end{align}$$

## Q6: Show that the relationship between consumer utility and the lump-sum tax is non-monotone. Show that for sufficiently small $n_0$ the tax can be pareto-improving.

Entrepreneurs always benefit from the tax.

For consumers the tax includes three potential effects. Show that the effect on utility is given by:

$$\begin{align}
    \dfrac{\partial U_c}{\partial \tau} = \left\lbrace \begin{array}{ll} \color{red}{-1}, & n_0+\tau\geq n^* \\ \color{red}{-1} +\color{blue}{(1-\alpha)R(w_1)}\color{blueviolet}{\dfrac{\partial k_1}{\partial \tau}}, & n_0+\tau<n^*\end{array}\right.
\end{align}$$

Explain the three effects (here marked with different colors).

## Q7: Discuss how this justifies government's capital injection during the financial crisis. What features might the model be missing?

## Q8: How does key findings in Q6 depend on the type of borrowing constraint?

# PS2, part 2: Costly state verification and optimality of debt

The core of the model setup:

* **Utility** is linear: $U_E=c_0+c_1$.
* Entrepreneur invests at time $0$, cost $I$. Return at time $t=1$ is random, distributed according to $g(s)$:

$$\begin{align}
    Return = Rs, && s\sim g(s), && \text{Support on }S\equiv \left[s^{min}, s^{max}\right]
\end{align}$$
Also assumed that
$$\begin{align}
    \mathbb{E}[s]=1, && R>I.
\end{align}$$

* **Asymmetric information:** State $s$ is unknown at time $t=0$. At time $t=1$ the entrepreneur (E) observes the true value of $s$ costlessly. The financier (F) can choose to audit and observe it, but at the cost $\kappa>0$.
* **Assumption:** The financier chooses a *deterministic* audit rule. I.e. no *random/stochastic* audit rule. By the *revelation principle:* Only focus on contracts in which E has no incentive to lie.
* **Contracts:** Specified by the triplet

$$\begin{align}
    Contract = \left( R^E(s), \mbox{ }R^F(s), \mbox{ }S_A\right), && S_A\subset S
\end{align}$$
where $R^E(s)$ and $R^F(s)$ denotes the payoffs for E,F (net of audit cost). $S_A$ is the set of states $S$, where $F$ chooses to audit.

* **Resource constraint:**

$$\begin{align}
    R^E(s)+R^F(s)+\kappa \mathbf{1}_{s\in S_A} = Rs, && R^E(s)\geq 0.
\end{align}$$

where $\mathbf{1}_{s\in S_A}$ indicates auditing, and the inequality captures *limited liability* of the entrepreneur.

* **Participation constraint:** (for F)

$$\begin{align}
    \int_S R^F(s)g(s)\mbox{ }ds\geq I.
\end{align}$$

States that expected revenue covers investment cost.

### Q1: Write down the IC constraint for E. What does it imply for F's net payoff?

A few notes before you try this:

* **IC constraint** with a continuous set of possible outcomes: *In any given state $s$ it must not be profitable to deviate to *any* state $s'$.*
* A useful theroem/principle: **The revelation principle**. The principle roughly states that every potential equilibrium outcome, can be achieved by an incentive-compatible strategy.

**What does it imply for F's net payoff?**

* What does the (IC) imply for two states $s,s'$ that are both in the no-audit region? (Remember the IC constraint has to hold for both states)
* Now consider the case where $s\in S_A$ (alternative s' is not in audit region): What does the (IC) state in this case?
* Does F receive higher/lower revenue in the audit region than in no-audit region?

### Q2: Define a standard debt contract:

$$\begin{align}
    R^F(s) = \left\lbrace \begin{array}{ll} R\bar{s}, & s\geq \bar{s} \\ Rs-\kappa, & s\leq \bar{s} \end{array}\right., && S_A=\left[s^{min},\mbox{ }\bar{s}\right].
\end{align}$$

### Check that any level $\bar{s}$ satisfies both the resource constraint and the (IC).

### Q3: Prove that the standard contract is optimal.

This is a bit tricky. Read the hint given in the problem text, it provides some guidance.

### Q4: Assume that $s$ is uniform. Characterizes the riskiness of the optimal debt contract.

Assume that $s$ is uniform on $[1-\Delta, 1+\Delta]$. In other words the pdf (f) and cdf (F) are given by:

$$\begin{align}
f(s) = \dfrac{1}{2\Delta}, && F(s) = \dfrac{s-(1-\Delta)}{2\Delta}, && s\in [1-\Delta, 1+\Delta]
\end{align}$$

Now set up the following:

* The entrepreneur's exptected profits when (1) $s\leq \bar{s}$ the entrepreneur receives 0 and when $s\geq \bar{s}$ he receives $Rs-R\bar{s}$.
* E maximizes expected profits **given** the financier participates, i.e. has non-negative expected profits. Explain that it looks like:

$$\begin{align}
    \dfrac{1}{2\Delta} \int_{\bar{s}}^{1+\Delta} R\bar{s}\mbox{ }ds+\dfrac{1}{2\Delta}\int_{1-\Delta}^{\bar{s}}(Rs-\kappa)\mbox{ }ds\geq I
\end{align}$$

(note that you can easily integrate this out, as it is only linear in $s$).

**Characterizing E's optimal contract:**

To characterize the optimal debt contract, consider E's motive for choosing $\bar{s}$: If he did not have to consider F's constraint, what is the optimal level then?

Hint 1: The figure below illustrates how the expected profits (participation constraint, PC) varies with the level of investment costs, $I$ and the audit costs.
.

.

.

.

.

(Hint: If you can, try to show that the optimal $s^*$ for E is given by:)

$$\begin{align}
 \bar{s}^* = \left\lbrace \begin{array}{ll}1+\Delta-\dfrac{\kappa}{R}-\dfrac{1}{R}\sqrt{(R(1+\Delta)-\kappa)^2+2R*\left[(1-\Delta)\kappa-\dfrac{R}{2}(1-\Delta)^2-2\Delta I\right]}, & R(1-\Delta)<I \\ 1-\Delta, & R(1-\Delta)\geq I \end{array}\right.
\end{align}$$

```python
# Parameter values:
R = 2
I = 1.5
Delta = 0.5
kappa = 0.1
# Grid of sbar:
sbar = np.linspace(1-Delta, 1+Delta, 101)
# PC constraint:
int_plot.interactive_pc(R,I,Delta,kappa,sbar)
s_star = 1+Delta-kappa/R-(1/R)*math.sqrt((R*(1+Delta)-kappa)**2+2*R*((1-Delta)*kappa-(R/2)*(1-Delta)**2-2*Delta*I))
print(s_star)
```

    interactive(children=(FloatSlider(value=1.5, description='I', max=2.0), FloatSlider(value=0.1, description='$\…

    0.8155711229775239

### Q5: Find the threshold cost $I$, where the project is profitable. Comment on the solution.

Use the result in Q4: The choice of $s^*$ follows one of two scenarios:

* (Scenario 1) If $R(1-\Delta)\geq I$, what is the level of $s^*$? What is the maximum level of $I$ where the participation constraint (PC) holds?
* (Scenario 2) If $R(1-\Delta)<I$, how is $s^*$ determined? (Hint PC holds w. equality)

Hint 1: To show the maximum level of $I$ in scenario 2, note that the (PC) equation with equality is a *quadratic equation*. This has a solution as long as the determinant $\geq 0$.

.

.

.

.

.

Hint 2: The maximum investment level is given by:

$$\begin{align}
     I^{max} = \max \Bigg\lbrace \underbrace{R-\kappa+\dfrac{\kappa^2}{4\Delta R}}_{\text{Scenario 1}},\mbox{ }\mbox{ }\underbrace{R(1-\Delta)}_{\text{Scenario 2}}\Bigg\rbrace.
\end{align}$$

```python

```


 INF285 - Computación Científica

 ODE

  [S]cientific [C]omputing [T]eam

 Version: 1.18

## Some algorithms in this jupyter notebook have been obtained from the textbook "Lloyd N. Trefethen, Spectral Methods in MATLAB, SIAM, Philadelphia, 2000" and translated to Python.

## Table of Contents

* [Introduction](#intro)
* [Initial Value Problemsl](#IVP)
    * [Three solvers](#solver)
        * [Stability Regions](#stab)
        * [Convergence](#conver)
    * [High order, higher dimensions and dynamical systems](#high)
* [Boundary Value Problems](#BVP)
    * [Shooting Method](#MD)
    * [Finite Differences](#DD)
* [Acknowledgements](#acknowledgements)

```python
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import pyplot #
import numpy as np
from scipy.integrate import odeint
from pylab import * #
from numpy import linalg as LA
from matplotlib.legend_handler import HandlerLine2D
from scipy.linalg import toeplitz
from scipy.optimize import root
from ipywidgets import interact, RadioButtons, Checkbox
import sympy as sym
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
from mpl_toolkits.axes_grid1 import make_axes_locatable
sym.init_printing()

# Global parameter that controls the figure's size
L = 15

# Source: https://github.com/tclaudioe/Scientific-Computing/blob/master/SC5/02%20Differentiation%20matrices.ipynb
def plot_matrices_with_values(ax,M,FLAG_test=True):
    N=M.shape[0]
    cmap = plt.get_cmap('GnBu')
    im = ax.matshow(M, cmap=cmap)
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    plt.colorbar(im, cax=cax)
    if FLAG_test:
        for i in np.arange(0, N):
            for j in np.arange(0, N):
                ax.text(i, j, '{:.2f}'.format(M[i,j]), va='center', ha='center', color='r')
    return
```

# Introduction
[Back to TOC](#toc)

In this jupyter noteboon we will study several numerical methods for solving ordinary differential equations (ODEs).
In particular we will study Initial Value Problems (IVP) and Boundary Value Problems (BVP), their principal numerical methods of numerical resolution, and their stability and convergence.

# Initial Value Problems
[Back to TOC](#toc)

An IVP corresponds to an ordinary differential equation of the form:
$$ \dot{y}(t) = f(t,y(t)), \quad t\in]t_0,T]$$
subject to an initial condition,
$$ y(t_0) = y_0. $$

# Three solvers
[Back to TOC](#toc)

Here we list a few solvers, just notice that we are only hightliting their one-time-step version:

* Euler's Method:

\begin{align*}
    y_{i+1} &= y_i + h\,f(t_i,y_i)
\end{align*}

* Runge-Kutta of second order (RK2)

\begin{align*}
    k_{1} &= f(t_i,y_i) \\
    y_{i+1} &= y_i + h\,f\left(t_i + \dfrac{h}{2}, y_i + \dfrac{h}{2}\,k_1\right)
\end{align*}

* Runge-Kutta of 4th order (RK4)

\begin{align*}
    k_{1} &= h \, f(t_i,y_i) \\
    k_{2} &= h \, f(t_i + \dfrac{h}{2}, y_i + \dfrac{1}{2}k_1) \\
    k_{3} &= h \, f(t_i + \dfrac{h}{2}, y_i + \dfrac{1}{2}k_2) \\
    k_{4} &= h \, f(t_i + h, y + k_3) \\
    y_{i+1} &= y_i + \dfrac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)
\end{align*}

For all this methods $h$ is the time-step it will advance the solution.
The previous algorithms only advance one time step the solution, so it should be applied the corresponding time steps expected such that it goes from the initial time $t_0$ until the final time $T$.

```python
# Forward Euler Method
def eulerMethod_one_step(yi,ti,f,h):
    return yi+h*f(ti,yi)

# Runge-Kutta of Second order
def RK2_one_step(yi,ti,f,h):
    k1=f(ti,yi)
    return yi+h*f(ti+h/2.0,yi+h/2.0*k1)

# Runge-Kutta
def RK4_one_step(yi,ti,f,h):
    k1=f(ti,yi)
    k2=f(ti+h/2.0,yi+(h/2.0)*k1)
    k3=f(ti+h/2.0,yi+(h/2.0)*k2)
    k4=f(ti+h,yi+h*k3)
    return yi+(h/6.0)*(k1+2.0*k2+2.0*k3+k4)
```

Here we define the several steps solvers.
These are the version that will be used later on in the code.

```python
# This implemantions have been upgraded respect to the ones included in the classnotes to handle
def eulerMethod(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        y[i+1] = y[i]+f(t[i],y[i])*h
    return t, y
# Challenge: Make backwardEulerMethod work!
'''def backwardEulerMethod(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    y = np.zeros(N+1)
    y[0] = y0
    for i in np.arange(N):
        f_hat= lambda x:  x - y[i]-f(t[i+1],x)*h
        # You must select a solver, "find_root" is just a generic name. The input used are the anonymous function "f_hat" and the initial guess y_i.
        x = find_root(f_hat,y[i])
        y[i+1] = x
    return t, y'''
def RK2(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        k1 = f(t[i],y[i])
        y[i+1] = y[i]+f(t[i]+h/2,y[i]+k1*h/2)*h
    return t, y
def RK4(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        k1 = f(t[i],y[i])
        k2 = f(t[i]+h/2,y[i]+k1*h/2)
        k3 = f(t[i]+h/2,y[i]+k2*h/2)
        k4 = f(t[i]+h,y[i]+k3*h)
        y[i+1] = y[i]+(k1+2*k2+2*k3+k4)*h/6
    return t, y
```

### 1D Example

In this example we will solve the logistic equation $\dot{y}(t) = y(t)(1-y(t))$ for several inital conditions $y_0$.

```python
# Logistic Equation
def f(t,y):
    return np.array(y*(1-y))

# First Example
def change_N_and_y0_1st_Example(N=20,y0=0.1, ODEsolver=eulerMethod):
    t0 = 0
    T  = 4
    t, y = ODEsolver(t0,T,N,y0,f)
    fig = plt.figure()
    ax  = fig.gca()
    ax.axis([0, 4, -1, 2])
    # Plotting stationary states for this problem.
    t = np.linspace(t0,T,N+1)
    plt.plot(t,t*0+1,'b-',lw=5, alpha=0.5, label='Stationary state y=1')
    plt.plot(t,t*0,'r-',lw=5, alpha=0.5, label='Stationary state y=0')
    # Plotting numerical solution.
    plt.plot(t,y,'.',label=r'$y_i$')

    plt.grid(True)
    ax.legend(loc='lower left', ncol=1, fancybox=True, shadow=True, numpoints=1, bbox_to_anchor=(1,0))
    plt.show()

radio_button_ODEsolvers=RadioButtons(
    options=[('Euler\'s Method',eulerMethod),('RK2',RK2),('RK4',RK4)],
    value=eulerMethod,
    description='ODE solver:',
    disabled=False
)

interact(change_N_and_y0_1st_Example,N=(3,100,1),y0=(-0.5,2,0.1), ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(IntSlider(value=20, description='N', min=3), FloatSlider(value=0.1, description='y0', ma…

    <function __main__.change_N_and_y0_1st_Example(N=20, y0=0.1, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

Now, solve several initial conditions at the same time.

```python
def plot_log_eq(N=30,ODEsolver=eulerMethod):
    t0 = 0
    T = 4
    fig = plt.figure(figsize=(L/2,L/2))
    fig.clf()
    ax = fig.gca()
    ax.axis("equal")
    ax.grid(True)
    ax.set_title("Numerical Approximations and Direction Field")
    ax.axis([0, 4, -2, 2])

    # Plotting the stationary states
    t = np.linspace(t0,T,N+1)
    plt.plot(t,t*0+1,'b-',lw=5, alpha=0.5, label='Stationary state y=1')
    plt.plot(t,t*0,'r-',lw=5, alpha=0.5, label='Stationary state y=0')
    plt.legend(loc='best')

    # Solving over time for each method.
    for y0 in np.linspace(-0.5,5,40):
        t_times, y_output = ODEsolver(t0,T,N,y0,f)

        if y0>=1:
            plt.plot(t_times,y_output,'k-',alpha=0.2)
        else:
            plt.plot(t_times,y_output,'k-',alpha=0.8)
        plt.plot(t_times,y_output,'.')

    X,Y = np.meshgrid(np.arange(0,4,.2), np.arange(-2,2,.2))
    theta = np.arctan(f(0,Y))
    U = np.cos(theta)
    V = np.sin(theta)
    plt.quiver(X,Y,U,V,alpha=0.5)

interact(plot_log_eq,N=(5,100,1),ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(IntSlider(value=30, description='N', min=5), RadioButtons(description='ODE solver:', opt…

    <function __main__.plot_log_eq(N=30, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

## Stability Regions
[Back to TOC](#toc)

To perform a linear stability analysis, we will consider the following ODE $\dot{y} = \lambda y$, with $y(0)=1$ and $\lambda\in\mathbb{C}$, for which we know its exact solution $y(t)=\exp(\lambda\,t)$.

Thus, solving the previos ODE with the Euler method, we have,
\begin{align*}
    y_{i+1} &= y_i + \lambda h y_i \\
    y_{i+1} &= (1 + \lambda h ) y_i \\
    y_{i+1} &= (1 + \lambda h )^{i+1} y_0
\end{align*}

Given that the real part of $\lambda$ is negative and $h$ is positive, we require then that $\left|1+\lambda h \right| <1$. We denote this range of values for $h$ the stability region for the Euler Method.

Following a similar approach, we the the following constraints for RK2 and RK4:
- RK2:
$$ \left|1+\lambda h + \dfrac{(\lambda h)^2}{2!} \right| <1 $$

- RK4:
$$ \left| 1+\lambda h + \dfrac{(\lambda h)^2}{2!} + \dfrac{(\lambda h)^3}{3!} + \dfrac{(\lambda h)^4}{4!} \right| <1 $$

```python
def zplot2(z, ax=plt.gca(), lw=1.5, line_color='k', label=''):
    ax.plot(np.real(z), np.imag(z), line_color, lw=lw, label=label)
def runge_kutta_stability_regions():
    z = np.exp(1j * np.pi * np.arange(201)/100.)
    r = z-1
    d = 1-1./z;
    # Order 1
    W1, W2, W4 = [0], [0], [0]
    for zi in z[1:]:
        W1.append( W1[-1]-(1.+W1[-1]-zi) )
    for zi in z[1:]:
        W2.append( W2[-1]-(1+W2[-1]+.5*W2[-1]**2-zi**2)/(1+W2[-1]) )
    for zi in z[1:]:
        num = (1+W4[-1]+.5*W4[-1]**2+W4[-1]**3/6+W4[-1]**4/24-zi**4)
        den = (1+W4[-1]+W4[-1]**2/2+W4[-1]**3/6.)
        W4.append( W4[-1] - num/den )
    return W1, W2, W4
W1,W2,W4=runge_kutta_stability_regions()
```

```python
fig = plt.figure(figsize=(L/2,L/2))
ax=fig.gca()
zplot2(W1,ax,line_color='r',label='Euler')
zplot2(W2,ax,line_color='g',label='RK2')
zplot2(W4,ax,line_color='b',label='RK4')
ax.axis("equal")
ax.axis([-5, 2, -3.5, 3.5])
ax.grid("on")
ax.legend(loc='best')
ax.set_title("Stability Regions of some Runge-Kutta ODE solvers")
```

### So, how do we use the stability region for each method?

The key point here is that we can't define $h$ a priori since we may not know the corresponding $\lambda$, thus, in principle, we can estimate $\lambda$ or just try the solver for different $h$.

```python
def plot_log_eq_stability_region(y0=1.2, N=20, T=10, ODEsolver=eulerMethod):

    # How did w get this? Why do I need it?
    def f_1D_prime(y):
        return np.array(1-2*y)

    t0=0
    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    fig = plt.figure(figsize=(L,L/3))
    ax1 = fig.add_subplot(131)
    ax2 = fig.add_subplot(132)
    ax3 = fig.add_subplot(133)

    ax1.axis('equal')
    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.axis([0, T, -0.5, 2])
    ax1.plot(t_times,y_output,'k-')
    ax1.plot(t_times,y_output,'b.')

    # The next code is to plot in logscale unstable numerical simulations
    h = (T-t0)/N
    y_all=h*f_1D_prime(y_output)
    y_pos=y_all>0
    y_neg=np.logical_not(y_pos)
    ax3.semilogy(t_times[y_pos],y_all[y_pos],'.r',ms=20)
    ax3.semilogy(t_times[y_neg],-y_all[y_neg],'.b',ms=20)
    ax3.grid(True)
    ax3.set_title(r'$\log10(|y_i|)$')

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    k_lambdah=h*f_1D_prime(y_output)
    ax2.plot(np.real(k_lambdah),np.imag(k_lambdah),'.r',ms=20)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid(True)
    ax2.set_title('Stability Region and $k=\lambda\,h$')

interact(plot_log_eq_stability_region,y0=(-2,5,0.1),N=(2,100,1), T=(1,20,1), ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(FloatSlider(value=1.2, description='y0', max=5.0, min=-2.0), IntSlider(value=20, descrip…

    <function __main__.plot_log_eq_stability_region(y0=1.2, N=20, T=10, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

## Convergence
[Back to TOC](#toc)

This is very important! Let's study it!

To talk about the convergence, we need to have in mind the order of each method.
In particular, we will consider that $y(T)$ denotes the exact solution at the final time $T$ that need to solve the ODE, and $y_N$ be the approximation of the solution at the final time $T=t_N$.
In simple words, we can say the following:

- Euler's method is a first order method since $y(T)=y_N+\mathcal{O}(h)$. This means that if $h$ is reduce by half, then the error is reduced by half.
- RK2 is a second order method since $y(T)=y_N+\mathcal{O}(h^2)$. This means that if $h$ is reduce by half, then the error is reduced by 4.
- RK4 is a forth order method since $y(T)=y_N+\mathcal{O}(h^4)$. This means that if $h$ is reduce by half, then the error is reduced by 16.

To validate this, we will solve numerically the following problem,
\begin{align*}
    \dot{y}(t)&=-3\,y(t)+6\,t+5, \quad t\in]0,0.5]\\
    y(0)&=3,
\end{align*}
for which we know the solution $y(t)=2\exp(-3t)+2t+1$.

```python
t0 = 0
T  = 0.5
y0 = 3.0
def f(t,y):
    return np.array(-3*y+6*t+5)
y_sol = lambda t: 2.*np.exp(-3.*t)+2.*t+1
```

```python
fig = plt.figure(figsize=(L/2,L/2))
fig.clf()
ax = fig.gca()
ax.axis('equal')
ax.grid(True)
ax.set_title('Convergence Analysis')

N_list = np.logspace(1,5,5,dtype=int)
for N in N_list:

    h = (T-t0)/N
    t_times, y_output = eulerMethod(t0,T,N,y0,f)
    plt.loglog(h,abs(y_output[-1]-y_sol(t_times[-1])),'b.',ms=20,label='Euler',alpha=.5)

    t_times, y_output = RK2(t0,T,N,y0,f)
    plt.loglog(h,abs(y_output[-1]-y_sol(t_times[-1])),'rs',ms=20,label='RK2',alpha=.5)

    t_times, y_output = RK4(t0,T,N,y0,f)
    plt.loglog(h,abs(y_output[-1]-y_sol(t_times[-1])),'gd',ms=20,label='RK4',alpha=.5)
    if N==N_list[0]:
        plt.legend(numpoints=1, loc='lower right')

h_list=T/N_list

plt.loglog(h_list,h_list,'k-')
plt.text(10**(-6),10**(-3),r'$\mathcal{O}(h)$', fontsize=25, horizontalalignment='center', color='b')
plt.loglog(h_list,np.power(h_list,2.),'k-')
plt.text(10**(-7),10**(-8),r'$\mathcal{O}(h^2)$', fontsize=25, horizontalalignment='center', color='r')
plt.loglog(h_list[:-2],np.power(h_list[:-2],4.),'k-')
plt.text(10**(0),10**(-11),r'$\mathcal{O}(h^4)$', fontsize=25, horizontalalignment='center', color='g')
plt.xlabel(r'$h$')
plt.ylabel(r'Error$=|y(T)-y_{N}|$')
```

# High order, higher dimensions and dynamical systems
[Back to TOC](#toc)

Why do we study higher order problems?

Because they are so cool!

### The Van der Pol oscillator

The equation that model de Van der Pol oscillator is the following:
$$\ddot{y}-\mu\,(1-y^2)\,\dot{y} + y = 0,$$
with $y(0)=2$ and $\dot{y}(0)=0$,
where $\mu$ is a scalar parameter that regulates the non-linearity of the problem and the strength of the damping.
See https://en.wikipedia.org/wiki/Van_der_Pol_oscillator.

So, since it has second derivatives respecto to time, we need to translate it to a dynamical system with the following change of variables:
\begin{align*}
    y_1(t) &= y(t),\\
    y_2(t) &= \dot{y}(t).
\end{align*}
Thus,
\begin{align*}
    \dot{y}_1 &= \dot{y} = y_2,\\
    \dot{y}_2 &= \ddot{y} = \mu (1-y^2) \dot{y} - y = \mu (1-y_1^2) y_2 - y_1,
\end{align*}
with $y_1(0)=2$ and $y_2(0)=0$.

```python
def f_vdp(t,y,mu=10):
    y1 = y[0]
    y2 = y[1]
    return np.array([y2, mu*(1-y1**2)*y2-y1])

def f_vdp_eig_jacobian(t,y,mu=10):
    J=[[0,1],[-(2*mu*y[0]*y[1]+1), mu*(1-y[0]**2)]]
    lambs,vs= LA.eig(J)
    return lambs
```

```python
def plotting_vdp(y01=2, y02=0, N=20000, T=200, mu=10, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])

    f= lambda t,y: f_vdp(t,y,mu)

    t0=0

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    fig = plt.figure(figsize=(L,L/2))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    ax1.grid(True)
    ax1.set_title("Numerical Approximation")
    ax1.plot(t_times,y_output[:,0],'-', label=r'$y_1(t)$', alpha=0.5)
    ax1.set(xlabel='t')
    ax1.plot(t_times,y_output[:,1],'-r', label=r'$y_2(t)$', alpha=0.5)
    ax1.legend(loc='best')

    ax2.grid(True)
    ax2.set_title("Phase Portrait")
    ax2.plot(y_output[:,0],y_output[:,1],'-')
    ax2.set(xlabel='$y=y_1$', ylabel='$\dot{y}=y_2$')

interact(plotting_vdp,y01=(-3,3,0.1), y02=(-3,3,0.1),
         N=(10,10000,10),T=(10,200,10),
         mu=(1,50,0.1),ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(FloatSlider(value=2.0, description='y01', max=3.0, min=-3.0), FloatSlider(value=0.0, des…

    <function __main__.plotting_vdp(y01=2, y02=0, N=20000, T=200, mu=10, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

```python
def plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_jacobian,h,c1,c2,c3):
    fig = plt.figure(figsize=(L,L/3))
    ax1 = fig.add_subplot(131)
    ax2 = fig.add_subplot(132)
    ax3 = fig.add_subplot(133)

    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.plot(t_times,y_output[:,0],'-')
    ax1.plot(t_times,y_output[:,1],'r-')

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    for i in range(1,t_times.size):
        k_lambdah=h*f_jacobian(t_times[i],y_output[i,:])
        ax2.plot(np.real(k_lambdah[0]),np.imag(k_lambdah[0]),'.r',ms=10,alpha=.4)
        ax2.plot(np.real(k_lambdah[1]),np.imag(k_lambdah[1]),'sm',ms=10,alpha=.4)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid('on')

    ax3.grid(True)
    ax3.set_title('Phase Portrait')
    ax3.plot(y_output[:,0],y_output[:,1],'-')
    ax3.set(xlabel='$y_1$', ylabel='$y_2$')
```

### What about the stability region for dynamical systems?

In this case we need to compute the eigenvalues of the Jacobian matrix of $\mathbf{F}(\mathbf{y})$.
So the eigenvalues $\lambda_i$ multiplied by $h$ have to be inside of the stability region of the real part of $\lambda_i$ is negative, for $i\in\{1,2,\dots,\text{Dimension of Dynamical System}\}$.

```python
def plotting_vpd_as_system(y01=2, y02=0, N=1000, T=40, mu=10, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])

    f= lambda t,y: f_vdp(t,y,mu)

    t0 = 0

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_vdp_eig_jacobian,h,c1,c2,c3)

interact(plotting_vpd_as_system, y01=(-3,3,0.1), y02=(-3,-3,0.1),
         N=(10,4000,10),T=(10,200,10),
         mu=(1,50,0.1),ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(FloatSlider(value=2.0, description='y01', max=3.0, min=-3.0), FloatSlider(value=-3.0, de…

    <function __main__.plotting_vpd_as_system(y01=2, y02=0, N=1000, T=40, mu=10, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

### Lotka-Volterra model

This instance models the evolution of the average population of two species living in a closed system.
The species number 1 is considered that feeds itself with an unlimited amount of food and the species number 2 feeds itself with species number 1.
We will denote the average population of species number 1 as $y_1(t)$ and the average population of species number 2 as $y_2(t)$.

So, if we consider that there is no presence of species number 2 in the system, then species number 1 will grow exponentially, this means $\dot{y}_1=y_1$.
On the other case, if no presence of species number 1 in the system, then species number 2 will decay exponentially as follows $\dot{y}_2=-y_2$.
However, we are not interested in the extreme cases, how can we understand the dynamics when there exists these two species together interacting?
This is modeled by the Lotka-Volterra dynamical system:
\begin{align*}
    \dot{y}_1(t)&=(1-y_2(t)/\mu_2)\,y_1(t),\\
    \dot{y}_2(t)&=-(1-y_1(t)/\mu_1)\,y_2(t),\\
    y_1(0)&=400,\\
    y_2(0)&=100,
\end{align*}
where $\mu_1$ and $\mu_2$ are normalization constants.

```python
mu1=300
mu2=200
def f_predprey(t,y,mu1=mu1,mu2=mu2):
    y1 = y[0]
    y2 = y[1]
    return np.array([(1-y2/mu2)*y1, -(1-y1/mu1)*y2])

def f_predprey_jacobian(t,y,mu1=mu1,mu2=mu2):
    J=[[1-y[1]/mu2,-y[0]/mu2],[y[1]/mu1, -(1-y[0]/mu1)]]
    lambs,vs= LA.eig(J)
    return lambs
```

```python
def plotting_predprey(y01=70, y02=100, N=1000, T=30, ODEsolver=eulerMethod):
    y0=np.array([y01, y02])
    f= lambda t,y: f_predprey(t,y)
    t0 = 0
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_predprey_jacobian,h,c1,c2,c3)

interact(plotting_predprey,y01=(0,1000,10),y02=(0,1000,10),
         N=(10,1000,10),T=(10,200,10),
         ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(IntSlider(value=70, description='y01', max=1000, step=10), IntSlider(value=100, descript…

    <function __main__.plotting_predprey(y01=70, y02=100, N=1000, T=30, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

## Now solving the following equation
\begin{align*}
    \dot{y}_1(t) &= y_2(t),\\
    \dot{y}_2(t) &= -y_1(t),
\end{align*}
with $y_1(0)=1$ and $y_2(0)=0$.

What functions are these? Do you recognize them?

```python
def f_trig(t,y):
    y1 = y[0]
    y2 = y[1]
    return np.array([y2, -y1])

def f_trig_jacobian(t,y):
    return np.array([0.+1.j, 0.-1.j])

def plotting_f_trig(y01=1, y02=0, N=100, T=2*np.pi, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])
    f= lambda t,y: f_trig(t,y)
    t0 = 0
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'
    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_trig_jacobian,h,c1,c2,c3)

interact(plotting_f_trig,y01=(-2,2,0.1),y02=(-2,2,0.1),
         N=(10,1000,10),T=(1,10*np.pi,0.1),
         ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(FloatSlider(value=1.0, description='y01', max=2.0, min=-2.0), FloatSlider(value=0.0, des…

    <function __main__.plotting_f_trig(y01=1, y02=0, N=100, T=6.283185307179586, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

## Last but not least, solving:
\begin{align*}
    \dot{y}(t) &= \lambda\,y(t),\\
\end{align*}
with $y(0)=1$.

What function is this? Do you recognize it?

```python
def f_exp(t,y,lam=-1):
    return lam*y
def f_exp_jac(t,y,lam=-1):
    return lam

def plotting_f_exp(y0=1, N=100, T=10, lam=-1, ODEsolver=eulerMethod):

    f= lambda t,y: f_exp(t,y, lam)
    t0 = 0
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    # Plotting

    fig = plt.figure(figsize=(L*2/3,L/3))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.plot(t_times,y_output[:],'.b')
    ax1.plot(t_times,y_output[:],'k-',alpha=0.5)
    #ax1.axis([0, T, -2, 2])

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    for i in range(1,t_times.size):
        k_lambdah=h*f_exp_jac(t_times[i],y_output[i],lam)
        ax2.plot(np.real(k_lambdah),np.imag(k_lambdah),'.r',ms=10,alpha=.4)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid('on')

interact(plotting_f_exp,y0=(-2,2,0.1),
         N=(5,1000,10),T=(1,20,1), lam=(-2,2,0.1),
         ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(FloatSlider(value=1.0, description='y0', max=2.0, min=-2.0), IntSlider(value=100, descri…

    <function __main__.plotting_f_exp(y0=1, N=100, T=10, lam=-1, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

# Boundary Value Problems
[Back to TOC](#toc)

An instance of a BVP corresponds to a ordinary differential equation of the following form:
$$ y''(x) = f(x,y(x),y'(x)),$$
with the following boundary conditions,
\begin{align*}
    y(a) &= y_a,\\
    y(b) &= y_b.
\end{align*}

To undertand them better, let's find a numerical approximation of the following problem:
\begin{align*}
    y''(x) &= 4\,y(x),\\
    y(0)   &= y_0,\\
    y(1)   &= y_n.
\end{align*}

## Shooting Method
[Back to TOC](#toc)

This method takes advantage that we already know how to handle IVP and uses that knowledge to find a numerical approximation of an BVP.
The main steps are the followings:
- We have a BVP on the form:
    \begin{align*}
        y''(x) &= f(x,y(x),y'(x)),\\
        y(a)   &= y_a,\\
        y(b)   &= y_b.
    \end{align*}
- Let's considerer the following steps to translate the BVP to an IVP:
    1. $x \rightarrow t$, this implies that $t_0=a$ and $T=b$.
    2. \begin{align*}
            \ddot{y}(t)   &= f(t,y(t),\dot{y}(t)),\\
            y(t_0)        &= y_a,\\
            \dot{y}(t_0)  &= \alpha.
       \end{align*}
       **Warning**: We actually don't know $\alpha$ but we do need it since we treating the previous problem as a IVP. What we do have is $y(b)=y(T)=y_b$.
    3. So, since we don't have $\alpha$, we need to find it by solving the following 1D root: $F(\alpha) = y_N^{(\alpha)} - y_b$, where $y_N^{(\alpha)}$ denotes the IVP approximation at time $t=T$ considering $\alpha$ for the initial condition of $\dot{y}(t_0)$.
    4. After we find the root for $F(\alpha)$, we can say we have found a numerical approximation of $y(x)$ since we have a numerical approximation that satisfies the ODE and also the boundary condition defined on the interval $[a,b]$. This is wonderful!

Notice however that the function $F(\alpha)$ is the output of the execution of an algorithm, so we only expect it to be continuous.
For instance it will be interesting to apply the Bisection method here!
The main disadvantage of this algorithm is that we need to solve several IVPs to actually find the right one, the good news is that we are re-using previous knowledge in a new type of problems!

```python
# BVP: y''(x)= 4*y(x), y(0)=1, and y(1)=3, x \in [0,1].
# Dynamical system:
# y1=y  -> y1'=y'=y2
# y2=y' -> y2'=y''=4*y=4*y1.
# y1(0) = 1, (and we know y(1)=3)
# y1(1) = 3.
# y2(0) = alpha, we don't know it.
def f(t,y):
    return np.array([y[1],4*y[0]])
```

```python
def shooting_method_101(alpha=-1, N=50, ya=1, yb=3, ODEsolver=eulerMethod):
    t0 = 0
    T  = 1
    y0 = np.array([ya,alpha])
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    fig = plt.figure(figsize=(L/2,L/2))
    ax = fig.gca()
    plt.grid(True)
    plt.title("Numerical Approximation")
    plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'$y1_i$')
    plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'$y2_i$')

    plt.plot(0,1,'*k',ms=16,label='Left BC')
    plt.plot(1,3,'*g',ms=16,label='Right BC')
    plt.plot(0,alpha,'darkorange',ms=16,label=r'$\alpha$', marker=r'$\alpha$')

    plt.legend(loc='best')

interact(shooting_method_101, alpha=(-2,2,0.1),
         N=(5,100,1), ya=(-5,5,0.1), yb=(-5,5,0.1),
         ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(FloatSlider(value=-1.0, description='alpha', max=2.0, min=-2.0), IntSlider(value=50, des…

    <function __main__.shooting_method_101(alpha=-1, N=50, ya=1, yb=3, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>)>

### Now, let's find $\alpha$ automatically

```python
y0 = np.array([1, 1.1])
```

```python
def F(alpha, ya=1, yb=3, N=100, ODEsolver=eulerMethod):
    t0 = 0
    T  = 1
    y0 = np.zeros(2)
    y0[0] = ya
    y0[1] = alpha
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    # Notice that we don use absolute value for the error since
    # we want to look at this problem as a rootfinding problem
    # and not as a minimization, that could be another approach.
    return y_output[-1,0]-yb
```

```python
t0 = 0
T  = 1
ya = 1
yb = 3
N  = 100
ODEsolver = RK4

F_root = lambda alpha: F(alpha, ya=ya, yb=yb, N=N, ODEsolver=ODEsolver)
alpha = root(F_root, 0.).x[0]

y0 = np.array([ya,alpha])
t_times, y_output = ODEsolver(t0,T,N,y0,f)

fig = plt.figure(figsize=(L/2,L/2))
plt.grid(True)
plt.title("Numerical Approximation")
plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'$y1_i$')
plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'$y2_i$')
plt.plot(0,ya,'*k',ms=16,label='Left BC')
plt.plot(1,yb,'*g',ms=16,label='Right BC')
plt.plot(0,alpha,'darkorange',ms=16,label=r'$\alpha$', marker=r'$\alpha$')
plt.legend(loc='best')
print(alpha)
```

### Let's take a look to $F(\alpha, \dots)$, find the 'BUG'!

```python
ODEsolver = eulerMethod

several_alphas = np.linspace(-1,0,100)
Fv = np.vectorize(lambda x: F(x,ODEsolver=ODEsolver))
F_alphas = Fv(several_alphas)

plt.figure(figsize=(10,10))
plt.plot(several_alphas,F_alphas,'.b',label=r'$F(\alpha)$')
plt.plot(several_alphas,F_alphas*0,'-g',alpha=0.8,linewidth=3,label=r'$0$')
plt.grid(True)
plt.plot(alpha,F(alpha,ODEsolver=ODEsolver),'.r',ms=20,label=r'Root: $\alpha$='+str(alpha))
plt.xlabel(r'$\alpha$')
plt.ylabel(r'$F(\alpha)=y_N^{(\alpha)} - y_b$')
plt.legend(loc='best')
plt.show()
```

## Finite Differences
[Back to TOC](#toc)

This method opens a new door of algorithms for solving BVP.
The main idea is to approximate each derivative of $y(x)$ by a finite difference approximation and then solve the linear or possible non-linear system of equations associated.

Here we list a few finite difference approximation for the first and one for the second derivative:

- Forward Difference:
$$ y'(x) = \dfrac{y(x+h) - y(x)}{h} + O(h)$$

- Backward Difference:
$$ y'(x) = \dfrac{y(x) - y(x-h)}{h} + O(h)$$

- Central Difference:
$$ y'(x) = \dfrac{y(x+h) - y(x-h)}{2h} + O(h^2)$$

And for the second derivative:
\begin{align*}
y''(x) &= \dfrac{\text{Forward Difference } - \text{ Backward Difference}}{h} + O(h^2) \\
y''(x) &= \dfrac{\dfrac{y(x+h) - y(x)}{h} - \dfrac{y(x) - y(x-h)}{h}}{h} + O(h^2) \\
y''(x) &= \dfrac{y(x+h) - 2\,y(x) + y(x-h)}{h^2} + O(h^2) \\
\end{align*}

Notice that in all the previous cases we first need to define the function on a grid, it does not have to be regular, but for simplicity we will consider it regular.
Recall that the BVP will be defined on $[a,b]$, thus the regular grid will be defined as follows: $x_i=i\,h$, where $h=\dfrac{b-a}{N}$.
This also help us to define the approximation of the function over the grid, this means the following,
- Traditionally we say $y(x)$ is the function defined on $x\in[a,b]$.
- Now, we will have a grid function $y_i\approx y(x_i)$ for $i\in\{0,1,2,\dots,N\}$. In particular we would like to point out that the vector $\mathbf{y}$ will be the representation of all the scalar values $y_i$ and the vector $\mathbf{x}$ will store the grid values $x_i$. Thus in this discrete world $\mathbf{y}$ will apporximate the continuous function $y(x)$ on $[a,b]$.

On the other hand, the finite difference operator over the finite grid will be then defined as follows:
- Forward Difference: $ y'(x_i)=\dfrac{y_{i+1} - y_i}{h} + O(h) $.
- Backward Difference: $ y'(x_i) = \dfrac{y_i - y_{i-1}}{h} + O(h) $.
- Central Difference: $ y'(x_i) = \dfrac{y_{i+1} - y_{i-1}}{2h} + O(h^2)$.
- Second derivative finite difference: $y''(x_i) = \dfrac{y_{i+1} - 2\,y_i + y_{i-1}}{h^2} + O(h^2)$.

Now, let's apply them!

```python
def solve_finite_difference_eq(M=4,ya=1,yb=3):
    # Spatial discretization,
    # 'M' represents the number of intervals to be used.
    h=(1.-0.)/M

    # Building Finite Difference Discretization
    deltas=-(2.+4.*(h**2.))
    A=toeplitz(np.append(np.array([deltas, 1.]), np.zeros(M-3)))
    # Building RHS
    b=np.append(-ya, np.zeros(M-3))
    b=np.append(b,-yb)
    # Solving the linear system of equations
    y=np.linalg.solve(A, b)
    # Adding back the boundary conditions into the solution
    y=np.append([ya], y)
    y=np.append(y,[yb])
    x_FD = np.linspace(0,1,M+1) # xi, i.e. the spatial grid.
    y_FD = y                    # yi, i.e. the approximation of y(x_i).
    return x_FD, y_FD, A, b

def plot_solution_finite_difference_eq(M=4,ya=1,yb=3, FLAG_test=True):

    # Solving with Finite Difference
    x_FD, y_FD, A, b = solve_finite_difference_eq(M,ya,yb)

    # Plotting
    fig = plt.figure(figsize=(L,L/2))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    # Plotting the discrete solution
    ax1.grid(True)
    ax1.set_title("FD")
    ax1.plot(x_FD,y_FD,'.',ms=20)
    ax1.grid("on")

    # Plotting the pattern and coefficients of the tri-diagonal matrix
    plot_matrices_with_values(ax2,A,FLAG_test)

interact(plot_solution_finite_difference_eq, M=(3,25,1),
         ya=(-5,5,0.1), yb=(-5,5,0.1),
         FLAG_test=Checkbox(value=True, description='Show numerical values'))
```

    interactive(children=(IntSlider(value=4, description='M', max=25, min=3), FloatSlider(value=1.0, description='…

    <function __main__.plot_solution_finite_difference_eq(M=4, ya=1, yb=3, FLAG_test=True)>

## Comparing the Shooting Method with Finite Differences

```python
def shooting_method_vs_FD(alpha=-0.420306048726636, N=50, ya=1, yb=3, ODEsolver=eulerMethod, M=4):
    # Shooting Method
    t0 = 0
    T  = 1
    y0 = np.array([ya,alpha])
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    # Finite Differences
    x_FD, y_FD, A, b = solve_finite_difference_eq(M,ya,yb)

    fig = plt.figure(figsize=(L/2,L/2))
    ax = fig.gca()
    plt.grid(True)
    plt.title("Numerical Approximations")
    plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'SM: $y1_i$')
    plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'SM: $y2_i$')

    plt.plot(x_FD,y_FD,'b.',ms=20,label=r'FD: $y_i$')

    plt.plot(0,1,'*k',ms=16,label='Left BC')
    plt.plot(1,3,'*g',ms=16,label='Right BC')
    plt.plot(0,alpha,'r',ms=16,label=r'$\alpha$', marker=r'$\alpha$')

    plt.legend(loc='best')

interact(shooting_method_vs_FD, alpha=(-2,2,0.01),
         N=(5,100,1), ya=(-5,5,0.1), yb=(-5,5,0.1),
         ODEsolver=radio_button_ODEsolvers,
         M=(3,25,1))
```

    interactive(children=(FloatSlider(value=-0.420306048726636, description='alpha', max=2.0, min=-2.0, step=0.01)…

    <function __main__.shooting_method_vs_FD(alpha=-0.420306048726636, N=50, ya=1, yb=3, ODEsolver=<function eulerMethod at 0x7fd920c4fd30>, M=4)>

In summary,
- Both methods find a reasonable approximation.
- The Shooting Method requires to solve several IVP until it find the missing slope $\alpha$, but it uses previous knowledge.
- Finite Different requieres to solve a linear (or possible non-linear) system of equations but we requires few points to get a reasonable approximation.
- Therefore, depending on the numerical problem that needs to be solve, one or the other would be better choice!

# Acknowledgements
[Back to TOC](#toc)

-  _Material creado por profesor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _y ayudantes: Alvaro Salinas y Martín
Villanueva. DI UTFSM. Abril 2016._
- **DISCLAIMER**: El presente notebook ha sido creado para el curso **ILI286 - Computación Científica 2**, del [Departamento de Informática](http://www.inf.utfsm.cl/), [Universidad Técnica Federico Santa María](http://www.utfsm.cl/). El material ha sido creado por Claudio Torres <ctorres@inf.utfsm.cl> y Sebastian Flores <sebastian.flores@usm.cl>, y es distribuido sin restricciones. En caso de encontrar un error, por favor no dude en contactarnos.
- [Update 2015] Se ha actualizado los notebooks a Python 3 e includio el "magic" "%matplotlib inline" antes de cargar matplotlib para que los gráficos se generen en el notebook.
- [Update 2016] (Álvaro) Modificaciones mayores al formato original. Agregado contexto: Introducción, Tabla de Contenidos, Explicaciones de cada método.
- [Update 2019] (C. Torres) Small changes. Fixing issue with title of sections and identation. Adding 'interact' to the logistic equation! Adding interact to everything, work in progress. All done, Enjoy!
- _Update July 2021 - v1.16 - C.Torres_ : Updating format and translating to English. Several updates in several functions! Major update!
- _Update July 2021 - v1.17 - C.Torres_ : Minor update, removing commented code.
- _Update July 2021 - v1.18 - C.Torres_ : Removing warning and changing the name of some variables.

```python

```


## The SEAIRQ model

In the present notebook, we consider the stochastic version of the SEAIRQ model.

In the SEAIRQ model we consider, there are the following classes:

* **$S$** Susceptible
* **$E$** Exposed
* **$A$** Asymptomatic infected (this class is between $E$ and $I_a$, $I_s$, and models the time an individual is already infectious, but it is not clear yet whether the individual will remain asymptomatic)
* **$I_a$** Asymptomatic infected
* **$I_s$** Symptomatic infected
* **$R$** Recovered
* **$Q$** Quarantined

In the SEAIR model, once an individual is exposed, it undergoes the transitions

\begin{equation}
    E \rightarrow A \rightarrow I_x \rightarrow R,
\end{equation}

where $I_x$ is either $I_a$ or $I_s$.

In the SEAIRQ model, there is a finite rate from the states $S$, $E$, $A$, $I_a$, $I_s$ to a quarantined class $Q$, in which an individual does not contribute to any infection rates anymore.

# Outline of this notebook

In this notebook we

1. Run and visualise a single simulation of the stochastic SEAIR model, obtained from the SEAIRQ model by setting all rates to the quarantine state $Q$ to zero.
2. Compare the result of 1. to a SEAIRQ model, in which only symptomatic infectives can get quarantined.
3. Compare the results of 1. and 2. to a SEAIRQ model, in which one could get quarantined in the states $E$, $A$, $I_a$, $I_s$.

For all cases, we also run 200 simulations to estimate the relevance of the stochastic nature of the dynamics.

# 0. Definition of the model

```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
```

```python
%matplotlib inline
import numpy as np
import pyross
import matplotlib.pyplot as plt
#from matplotlib import rc; rc('text', usetex=True)
```

```python
M = 1                  # the SIR model has no age structure
Ni = 10000*np.ones(M)   # so there is only one age group
N = np.sum(Ni)         # and the total population is the size of this age group

#
alpha = 0              # fraction of asymptomatic infectives
beta  = 0.2            # infection rate

gE    = 1.             # progression rate from E to A
gA   = 0.2            # rate to go from A to Ia
gIa   = 0.1            # recovery rate of asymptomatic infectives
gIs   = 0.1            # recovery rate of symptomatic infectives
gE    = 0.04           # recovery rate of E
fsa   = 1              # the self-isolation parameter

E0  = np.array([0])
A0  = np.array([1])
Ia0 = np.array([0])     # the SIR model has only one kind of infective
Is0 = np.array([20])     # we take these to be symptomatic
Q0  = np.array([0])
R0  = np.array([0])     # and assume there are no recovered individuals initially
S0  = N-(Ia0+Is0+R0+E0)    # The initial susceptibles are obtained from S + E + A + Ia + Is + R = N

# there is no contact structure
def contactMatrix(t):
    return np.identity(M)

# duration of simulation and number of datapoints
Tf = 500;  Nt=Tf+1;
```

# 1. SEAIR model

```python
# For the SEAIR model, we set all rates to the Q-class to zero
tS    = 0.0            # rate S  -> Q
tE    = 0.00           # rate E  -> Q
tA    = 0.00           # rate A  -> Q
tIa   = 0.00           # rate Ia -> Q
tIs   = 0.00           # rate Is -> Q
#tIs   = 0.05           # rate Is -> Q

# instantiate model
parameters = {'alpha':alpha, 'beta':beta,
              'gE':gE,'gA':gA,
              'gIa':gIa, 'gIs':gIs, 'gE':gE, 'fsa':fsa,
             'tE':tE,'tA':tA,'tIa':tIa,'tIs':tIs}
model = pyross.stochastic.SEAIRQ(parameters, M, Ni)

# simulate model
data=model.simulate(S0, E0, A0, Ia0, Is0, Q0, contactMatrix, Tf, Nt)
```

```python
# plot the results of the simulation

S  = data['X'][:,0].flatten()
E  = data['X'][:,1].flatten()
A  = data['X'][:,2].flatten()
Ia = data['X'][:,3].flatten()
Is = data['X'][:,4].flatten()
Q  = data['X'][:,5].flatten()
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
plt.plot(t, S/N, '-', color="#348ABD", label='$S$', lw=4)

plt.plot(t, E/N, '-', color="green", label='$E$', lw=4)
plt.fill_between(t, 0, E/N, color='green', alpha=0.3)

plt.fill_between(t, 0, A/N, color='#A60628', alpha=0.3)
plt.plot(t, A/N, '-', color='#A60628', label='$A$', lw=4)

plt.fill_between(t, 0, (Ia+Is)/N, color='orange', alpha=0.1)
plt.plot(t, (Ia+Is)/N, '-', color='orange', label='$I$', ls='--',lw=4)

R=N-S-E-A-Ia-Is; plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
plt.plot(t, R/N, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.xlabel(r'time')
plt.ylabel(r'fraction of total population')
plt.autoscale(enable=True, axis='x', tight=True)
```

### Averaging over several simulations

We now run 200 simulations, and consider the total number of susceptibles at the end of the simulation

```python
# run 200 simulations, calculate mean and standard deviation

# This might take a while.
N_runs = 200
trajectories_SEAIR = np.zeros([N_runs,Nt,6],dtype=float)
for i in range(N_runs):
    print('Running simulation {0} of {1}'.format(i+1,N_runs),end='\r')
    cur_result = model.simulate(S0, E0, A0, Ia0, Is0, Q0, contactMatrix, Tf, Nt)
    trajectories_SEAIR[i] = cur_result['X']
t_SEAIR = cur_result['t']
```

    Running simulation 200 of 200

```python
S_final_SEAIR = trajectories_SEAIR[:,-1,0]/N

mean_S_final_SEAIR = np.mean( S_final_SEAIR )
std_S_final_SEAIR = np.std( S_final_SEAIR )

hist, bin_edges = np.histogram(S_final_SEAIR, bins= 10,density=True)
bin_centers = bin_edges[1:] + bin_edges[:-1]
bin_centers /= 2

fig,ax = plt.subplots(1,1,figsize=(10,6))
fig.suptitle('Number of susceptibles at final time')
ax.bar(bin_centers,hist,width=bin_centers[1]-bin_centers[0],
       color='dodgerblue',label='Simulations',
      alpha=0.7)
ax.axvline(mean_S_final_SEAIR,ls='--',lw=4,color='red',
         label= 'Mean')
ax.locator_params(axis='x', nbins=6)
ax.legend() #loc=8)
ax.set_xlabel(r'S')
ax.set_ylabel(r'$P(S)$')
plt.show(fig)
#fig.savefig('S_distribution_SEAIR.png',bbox_inches='tight',dpi=100)
plt.close(fig)

print('At final time, on average {0:3.1f}% of the population has never been infected.'.format(mean_S_final_SEAIR*100))
```

# 2. SEAIRQ model

We now set the rate at which symptomatic infectives get quarantined to $1/20$ days. All other classes stay as before, i.e. never get quarantined.

(Technically, this is similar to changing the self-isolation parameter "fsa".)

```python
# We now consider a nonzero rate Is -> Q
tS    = 0.0            # rate S  -> Q
tE    = 0.00           # rate E  -> Q
tA    = 0.00           # rate A  -> Q
tIa   = 0.00           # rate Ia -> Q
tIs   = 0.05           # rate Is -> Q

# instantiate model
parameters = {'alpha':alpha, 'beta':beta,
              'gE':gE,'gA':gA,
              'gIa':gIa, 'gIs':gIs, 'gE':gE, 'fsa':fsa,
             'tE':tE,'tA':tA,'tIa':tIa,'tIs':tIs}
model = pyross.stochastic.SEAIRQ(parameters, M, Ni)

# simulate model
data=model.simulate(S0, E0, A0, Ia0, Is0, Q0, contactMatrix, Tf, Nt)
```

```python
# plot the results of the simulation

S  = data['X'][:,0].flatten()
E  = data['X'][:,1].flatten()
A  = data['X'][:,2].flatten()
Ia = data['X'][:,3].flatten()
Is = data['X'][:,4].flatten()
Q  = data['X'][:,5].flatten()
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
plt.plot(t, S/N, '-', color="#348ABD", label='$S$', lw=4)

plt.plot(t, E/N, '-', color="green", label='$E$', lw=4)
plt.fill_between(t, 0, E/N, color='green', alpha=0.3)

plt.fill_between(t, 0, A/N, color='#A60628', alpha=0.3)
plt.plot(t, A/N, '-', color='#A60628', label='$A$', lw=4)

plt.fill_between(t, 0, (Ia+Is)/N, color='orange', alpha=0.1)
plt.plot(t, (Ia+Is)/N, '-', color='orange', label='$I$', ls='--',lw=4)

R=N-S-E-A-Ia-Is; plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
plt.plot(t, R/N, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.xlabel(r'time')
plt.ylabel(r'fraction of total population')
plt.autoscale(enable=True, axis='x', tight=True)
```

```python
# run 200 simulations, calculate mean and standard deviation

# This might take a while
N_runs = 200
trajectories_SEAIRQ1 = np.zeros([N_runs,Nt,6],dtype=float)
for i in range(N_runs):
    print('Running simulation {0} of {1}'.format(i+1,N_runs),end='\r')
    cur_result = model.simulate(S0, E0, A0, Ia0, Is0, Q0, contactMatrix, Tf, Nt)
    trajectories_SEAIRQ1[i] = cur_result['X']
t_SEAIRQ1 = cur_result['t']
```

    Running simulation 200 of 200

```python
S_final_SEAIRQ1 = trajectories_SEAIRQ1[:,-1,0]/N

mean_S_final_SEAIRQ1 = np.mean( S_final_SEAIRQ1 )
std_S_final_SEAIRQ1 = np.std( S_final_SEAIRQ1 )

hist, bin_edges = np.histogram(S_final_SEAIRQ1, bins= 10,density=True)
bin_centers = bin_edges[1:] + bin_edges[:-1]
bin_centers /= 2

fig,ax = plt.subplots(1,1,figsize=(10,6))
fig.suptitle('Number of susceptibles at final time')
ax.bar(bin_centers,hist,width=bin_centers[1]-bin_centers[0],
       color='dodgerblue',label='Simulations',
      alpha=0.7)
ax.axvline(mean_S_final_SEAIRQ1,ls='--',lw=4,color='red',
         label= 'Mean')
ax.locator_params(axis='x', nbins=6)
ax.legend() #loc=8)
ax.set_xlabel(r'S')
ax.set_ylabel(r'$P(S)$')
plt.show(fig)
#fig.savefig('S_distribution_SEAIRQ1.png',bbox_inches='tight',dpi=100)
plt.close(fig)

print('At final time, on average {0:3.1f}% of the population has never been infected.'.format(mean_S_final_SEAIRQ1*100))
```

By comparing this to the result from 1, we see that if symptomatic infectives are quarantined with a rate $1/20$ days, then the fraction of the population that never gets exposed increases by about 12%.

# 3. SEAIRQ model with more nonzero rates to quarantine class

We now set the rate at which symptomatic infectives get quarantined to $1/20$ days, and the rates at which the $E$, $A$, $I_a$, $I_s$ get quarantined to $1/50$ days.

```python
# For the SEAIR model, we set all rates to the Q-state to zero
tS    = 0.0            # rate S  -> Q
tE    = 0.02           # rate E  -> Q
tA    = 0.02           # rate A  -> Q
tIa   = 0.02           # rate Ia -> Q
tIs   = 0.05           # rate Is -> Q

# instantiate model
parameters = {'alpha':alpha, 'beta':beta,
              'gE':gE,'gA':gA,
              'gIa':gIa, 'gIs':gIs, 'gE':gE, 'fsa':fsa,
             'tE':tE,'tA':tA,'tIa':tIa,'tIs':tIs}
model = pyross.stochastic.SEAIRQ(parameters, M, Ni)

# simulate model
data=model.simulate(S0, E0, A0, Ia0, Is0, Q0, contactMatrix, Tf, Nt)
```

```python
# plot the results of the simulation

S  = data['X'][:,0].flatten()
E  = data['X'][:,1].flatten()
A  = data['X'][:,2].flatten()
Ia = data['X'][:,3].flatten()
Is = data['X'][:,4].flatten()
Q  = data['X'][:,5].flatten()
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
plt.plot(t, S/N, '-', color="#348ABD", label='$S$', lw=4)

plt.plot(t, E/N, '-', color="green", label='$E$', lw=4)
plt.fill_between(t, 0, E/N, color='green', alpha=0.3)

plt.fill_between(t, 0, A/N, color='#A60628', alpha=0.3)
plt.plot(t, A/N, '-', color='#A60628', label='$A$', lw=4)

plt.fill_between(t, 0, (Ia+Is)/N, color='orange', alpha=0.1)
plt.plot(t, (Ia+Is)/N, '-', color='orange', label='$I$', ls='--',lw=4)

R=N-S-E-A-Ia-Is; plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
plt.plot(t, R/N, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.xlabel(r'time')
plt.ylabel(r'fraction of total population')
plt.autoscale(enable=True, axis='x', tight=True)
```

We now observe qualitatively different behaviour as compared to 1 and 2. There is no clear peak in the number of exposed anymore, and the number of susceptibles decreases slowly. The processes has not reached a steady state yet, even after 500 days.

We now replot only the $E$, $A$, and $I = I_a + I_s$ classes, which are hardly visible in the above plot:

```python
# plot only the E, A, and I = Ia + Is classes

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

#plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
#plt.plot(t, S/N, '-', color="#348ABD", label='$S$', lw=4)

plt.plot(t, E/N, '-', color="green", label='$E$', lw=4)
plt.fill_between(t, 0, E/N, color='green', alpha=0.3)

plt.fill_between(t, 0, A/N, color='#A60628', alpha=0.3)
plt.plot(t, A/N, '-', color='#A60628', label='$A$', lw=4)

plt.fill_between(t, 0, (Ia+Is)/N, color='orange', alpha=0.1)
plt.plot(t, (Ia+Is)/N, '-', color='orange', label='$I$', ls='--',lw=4)

#R=N-S-E-A-Ia-Is; plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
#plt.plot(t, R/N, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.xlabel(r'time')
plt.ylabel(r'fraction of total population')
plt.autoscale(enable=True, axis='x', tight=True)
```

As can be seen, the population fraction in these classes stays very low.

We now consider 200 simulations, and look at mean and standard deviation of the classes $E$, $A$, $I_a$, $I_s$:

```python
# run 200 simulations, calculate mean and standard deviation

N_runs = 200
trajectories_SEAIRQ2 = np.zeros([N_runs,Nt,6],dtype=float)
for i in range(N_runs):
    print('Running simulation {0} of {1}'.format(i+1,N_runs),end='\r')
    cur_result = model.simulate(S0, E0, A0, Ia0, Is0, Q0, contactMatrix, Tf, Nt)
    trajectories_SEAIRQ2[i] = cur_result['X']
t_SEAIRQ2 = cur_result['t']
```

    Running simulation 200 of 200

```python
# plot means and standard deviations

mean_SEAIRQ2 = np.mean(trajectories_SEAIRQ2,axis=0)
std_SEAIRQ2 = np.std(trajectories_SEAIRQ2,axis=0)

E_SEAIRQ2 = mean_SEAIRQ2[:,1].flatten()
A_SEAIRQ2 = mean_SEAIRQ2[:,2].flatten()
# I = I_asymptomatic + I_symptomatic
I_SEAIRQ2 = np.mean(trajectories_SEAIRQ2[:,:,3] + trajectories_SEAIRQ2[:,:,4] ,axis=0).flatten()

dE_SEAIRQ2 = std_SEAIRQ2[:,1].flatten()
dA_SEAIRQ2 = std_SEAIRQ2[:,2].flatten()
dI_SEAIRQ2 =  np.std(trajectories_SEAIRQ2[:,:,3] + trajectories_SEAIRQ2[:,:,4],axis=0).flatten()

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t_SEAIRQ2,(E_SEAIRQ2 - dE_SEAIRQ2)/N,(E_SEAIRQ2 + dE_SEAIRQ2)/N,color="green", alpha=0.2)
plt.plot(t_SEAIRQ2, E_SEAIRQ2/N, '-', color="green", label=r'$\langle E\rangle$', lw=4)

plt.fill_between(t_SEAIRQ2,(A_SEAIRQ2 - dA_SEAIRQ2)/N,(A_SEAIRQ2 + dA_SEAIRQ2)/N,color="#348ABD", alpha=0.2)
plt.plot(t_SEAIRQ2, A_SEAIRQ2/N, '-', color="#348ABD", label=r'$\langle S\rangle$', lw=4)

plt.fill_between(t_SEAIRQ2,(I_SEAIRQ2 - dI_SEAIRQ2)/N,(I_SEAIRQ2 + dI_SEAIRQ2)/N,color="#A60628", alpha=0.2)
plt.plot(t_SEAIRQ2, I_SEAIRQ2/N, '-', color="#A60628", label=r'$\langle I\rangle$', lw=4)

plt.legend(fontsize=26,loc='upper right',bbox_to_anchor=(1.3,1)); plt.grid()
plt.xlabel(r'time')
plt.ylabel(r'fraction of total population')
plt.autoscale(enable=True, axis='x', tight=True)
```

While the solid lines denote the instantaneous means, the colored shaded regions denote the standard deviation of the samples from the instantaneous mean.

As can be seen, the peak now is at about 350 days, and after 500 days the average numbers in all classes are decreasing.

```python

```


# Nonlinear Decision Boundaries
Load the data in `data.csv` as a data frame.

```python
# creation of the dataset
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
# get data between -pi and pi
dat = np.linspace(-np.pi,np.pi,100)
# transform by sine and cosine
datx = 0.4 * np.sin(dat) + np.random.normal(scale = 0.03,size=len(dat))
daty = 0.4 * np.cos(dat) + np.random.normal(scale = 0.03,size=len(dat))

datx2 = 0.6 * np.sin(dat) + np.random.normal(scale = 0.03,size=len(dat))
daty2 = 0.6 * np.cos(dat) + np.random.normal(scale = 0.03,size=len(dat))

# plot and save the resulting data
#plt.plot(datx,daty,'o')
df1 = pd.DataFrame({'x':datx,'y':daty,'c':np.repeat(["A"],100)})
df1
df2 = pd.DataFrame({'x':datx2,'y':daty2,'c':np.repeat(["B"],100)})
df2

df = pd.concat([df1,df2],sort=False)
df
#df.to_csv("data.csv",index=False)
```

      x
      y
      c

      0
      0.051920
      -0.378874
      A

      1
      -0.077194
      -0.391871
      A

      2
      -0.083993
      -0.418196
      A

      3
      -0.112411
      -0.355540
      A

      4
      -0.126714
      -0.417728
      A

      ...
      ...
      ...
      ...

      95
      0.184146
      -0.572692
      B

      96
      0.095896
      -0.573366
      B

      97
      0.060812
      -0.652355
      B

      98
      0.047930
      -0.594220
      B

      99
      0.013607
      -0.539394
      B

200 rows × 3 columns

Plot the data with matplotlib as a 2D scatter plot, using different colors for the different classes.

```python
import pandas as pd
import matplotlib.pyplot as plt

plt.plot(df.loc[df.c == "A",'x'],df.loc[df.c == "A",'y'],'bo')
plt.plot(df.loc[df.c == "B",'x'],df.loc[df.c == "B",'y'],'ro')
```

Which non-linear transformation $\phi$ will lead linearly separable data in the $\phi$-space? Compute this transformation, and plot the transformed data.

```python
df["trans"] = np.arctan2(df.x,df.y)
df["trans2"] = np.sqrt(np.square(df.x)+np.square(df.y))
df
plt.plot(df.loc[df.c == "A",'trans'],df.loc[df.c == "A",'trans2'],'bo')
plt.plot(df.loc[df.c == "B",'trans'],df.loc[df.c == "B",'trans2'],'ro')
```

# Kernel Density Estimators

In the lecture, you got to know kernel density estimatiors as a non-parametric method for representing a distribution. They represent the distribution of values $x$ as follows:

\begin{equation}
p(t) = \frac{1}{n \, h} \sum_{i=1}^n \varphi\left(\frac{t-x_i}{h}\right)
\end{equation}
Here,  $\varphi$ is a windowing function, e.g. the Gauss window:
\begin{equation}
\varphi(u) = \frac{1}{\sqrt{2\pi}} e^{-u^2/2}
\end{equation}

## Exercise 1
Implement the function `kde(t,h,x)` that estimates the kernel density estimate $p(t)$, given a point $t$, a window size $h$ and an array of training samples $x$.

```python
import math
import numpy as np
import matplotlib.pyplot as plt

def kde(t,h,x):
    n = len(x)
    insum = [k1((t-xj)/h) for xj in x]
    return 1/(n*h) * sum(insum)

def k1(t):
    return 1/math.sqrt(2*math.pi) * math.exp(-1/2 * t**2)

example = np.concatenate((np.random.normal(0,1,100),np.random.normal(5,1,100)))

dens = [kde(t,0.01,example) for t in np.arange(-2,8,0.05)]

plt.plot(dens)
```

## Exercise 2

Implement the function `classify_kde(xnew,x,classes)` that performs a classification with a kernel density estimator. Specifically, this is a Bayes classifier where the likelihood is represented by a kernel density estimator.

```python
import pandas as pd
from scipy.io import arff

def classify_kde(xnew,x,classes):
    uclasses = np.unique(classes)
    def getDens(c):
        xt = x[classes == c]
        return kde(xnew,0.01,xt)
    dens = np.array([getDens(c) for c in uclasses])
    prior = np.array([sum(classes == c)/len(classes) for c in uclasses])
    return uclasses[np.argmax(dens * prior)]

data = arff.loadarff('features1.arff')
df = pd.DataFrame(data[0])
feat = df["AccX_mean"]
cl = df["class"]

p = [classify_kde(x,feat,cl) for x in feat]
np.mean(p == cl)
```

    0.96045197740113


 INF285 - Computación Científica

 Gradient Descent and Nonlinear Least-Square

  [S]cientific [C]omputing [T]eam

 Version: 1.02

## Table of Contents
* [Introduction](#intro)
* [Gradient Descent](#GradientDescent)
* [Gradient Descent in 1D](#GradientDescent1D)
* [Gradient Descent for a 2D linear least-square problem](#GD_2D_LinearLeastSquare)
* [Gradient Descent for a 2D nonlinear least-square problem](#GD_2D_NonLinearLeastSquare)
* [Further Study](#FurtherStudy)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as spla
%matplotlib inline
# https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets
from sklearn import datasets
import ipywidgets as widgets
from ipywidgets import interact, interact_manual, RadioButtons
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
M=8
```

# Introduction
[Back to TOC](#toc)

This jupyter notebook presents the algorithm of Gradient Descent applied to non-linear least-square problems.

# Gradient Descent
[Back to TOC](#toc)

The algorithm of Gradient Descent is used in Optimization, in particular, in problems when we want to minimize a function (or equivalently in maximization problem by changing the sign of the function).
This algorithm considers a function $f(\mathbf{x}):\mathbb{R}^n \rightarrow \mathbb{R}$, which has at least a local minimum near the point $\mathbf{x}_0$.
The algorithm considers that we have access to the gradient of $f(\mathbf{x})$, i.e. $\nabla f(\mathbf{x})$, which indicates the direction of fastest increase of $f(\mathbf{x})$ at the point $\mathbf{x}$, or equivalently, $-\nabla f(\mathbf{x})$ is teh direction of fastest decrease.
Thus, the algorithm is the following,
- Select an initial guess, say $\mathbf{x}_0$
- Compute the direction of fastest decrease: $\mathbf{d}_0=-\nabla f(\mathbf{x}_0)$
- Update the approximation $\mathbf{x}_1=\mathbf{x}_0+\alpha\,\mathbf{d}_0$
- Iterate until certain threshold is achieved.
where $\alpha$ is a scaling factor for the Gradient Descent step.
The coefficient $\alpha$ could also depend on on the iteration number, such that it adapts based on the iterations.

# Gradient Descent in 1D
[Back to TOC](#toc)

To primary explain the algorithm, considere the following 1D example:
$$
f(x) = (x - 2)\,\sin(2\,x) + x^2.
$$
We will first plot the function as follows:

```python
# Defining the function using a 'lambda' definition.
f = lambda x: (x - 2)*np.sin(2*x) + np.power(x,2)

# Defining the grid for plotting, the number '1000' indicates the number of points of the sample.
# Suggestion: Change it and see what happends! For instance, what about if you change to 10?
xx = np.linspace(-3,3,1000)
# Plotting the function
plt.figure(figsize=(8,8))
plt.plot(xx,f(xx),'-',label=r'$f(x)$')
plt.grid(True)
plt.xlabel('$x$')
plt.legend(loc='best')
plt.show()
```

Now, we will create an interactive use of the Gradient Descent in 1D where you could define the initial guess $x_0$, the scaling factor $\alpha$ and the iteration number.
In this numerical experiment we will the importance of the coefficient $\alpha$, and how it is related to the 'gradient' and the initial guess.

```python
def GD_1D(x0=2, alpha=1, n=0):

    # Defining the function using a 'lambda' definition and its derivative.
    f = lambda x: (x-2)*np.sin(2*x)+np.power(x,2)
    fp = lambda x: 2*x+2*(x-2)*np.cos(2*x)+np.sin(2*x)

    # Plotting the function and its derivative.
    xx = np.linspace(-3,3,1000)
    plt.figure(figsize=(14,7))

    ax = plt.subplot(1,2,1)
    plt.plot(xx,f(xx),'b-',label=r'$f(x)$')
    # Warning: The 'alpha' parameter for the plt.plot function corresponds to
    # a transparency parameter, it is not related to the alpha parameter of
    # the Gradient Descent explained before.
    plt.plot(xx,fp(xx),'r-',label=r"$f'(x)$", alpha=0.5)
    plt.grid(True)
    plt.xlabel('$x$')
    plt.title('Plot in linear scale')

    # Plotting outcome with no iterations
    plt.plot(x0,f(x0),'k.',markersize=10,label=r'$x_i$')
    plt.plot(x0,fp(x0),'m.',markersize=10,label=r"$f'(x_i)$: 'Gradient'")
    ax = plt.subplot(1,2,2)
    plt.semilogy(xx,np.abs(f(xx)),'b-',label=r"$|f(x)|$")
    plt.semilogy(xx,np.abs(fp(xx)),'r-',label=r"$|f'(x)|$", alpha=0.5)
    plt.grid(True)
    plt.xlabel('$x$')
    plt.title('Plot in logarithmic scale')
    plt.semilogy(x0,np.abs(f(x0)),'k.',markersize=10,label=r'$x_i$')
    plt.semilogy(x0,np.abs(fp(x0)),'m.',markersize=10,label=r"$|f'(x_i)|$: 'Gradient'")

    # Computing steps of Gradient Descent
    if n>0:
        xi_output=np.zeros(n+1)
        xi_output[0]=x0
        for k in range(n):
            fp_x0=fp(x0)
            x1 = x0-alpha*fp_x0
            xi_output[k+1]=x1
            x0 = x1
        ax = plt.subplot(1,2,1)
        plt.plot(xi_output,f(xi_output),'k.-',markersize=10,label=r'$x_i$')
        plt.plot(xi_output,fp(xi_output),'m.',markersize=10)
        ax = plt.subplot(1,2,2)
        plt.semilogy(xi_output,np.abs(f(xi_output)),'k.-',markersize=10,label=r'$x_i$')
        plt.semilogy(xi_output,np.abs(fp(xi_output)),'m.',markersize=10)

    # Plotting outcome
    ax = plt.subplot(1,2,1)
    plt.legend(loc='best')
    ax = plt.subplot(1,2,2)
    plt.legend(loc='best')

    plt.show()

interact(GD_1D,x0=(-3,3,0.1), alpha=(0,10,0.01), n=(0,100,1))
```

    interactive(children=(FloatSlider(value=2.0, description='x0', max=3.0, min=-3.0), FloatSlider(value=1.0, desc…

    <function __main__.GD_1D(x0=2, alpha=1, n=0)>

What conclusions could be draw?

The main conclusion that can be draw is the importance of the selection of the parameter $\alpha$ for the success of the task of finding a minimum of a function.
Also, as ussual, the initial guess $x_0$ will help us to select different local minima.

Question to think about:
- What could happen if you normalize the 'gradient'? In 1D this would be computing the following coeficients: $GN=\frac{f'(x_i)}{|f'(x_i)|}$, this will gives us the 'direction' where we should move (in 1D is just the sign of the derivative), then the coefficient $\alpha$ may control a bit more the magnitude of each step from $x_i$ to $x_{i+1}$. So, how do we undertand this? Implement it!

# Gradient Descent for a 2D linear least-square problem
[Back to TOC](#toc)

In this case we will solve the following least-square problem:
$$
\begin{equation}
    \underbrace{\begin{bmatrix}
        1 & x_1 \\
        1 & x_2 \\
        1 & x_3 \\
        \vdots & \vdots \\
        1 & x_m
    \end{bmatrix}}_{\displaystyle{A}}
    \underbrace{\begin{bmatrix}
        a\\
        b
    \end{bmatrix}}_{\mathbf{x}}
    =
    \underbrace{\begin{bmatrix}
        y_1 \\
        y_2 \\
        y_3 \\
        \vdots\\
        y_m
    \end{bmatrix}}_{\displaystyle{\mathbf{b}}}.
\end{equation}
$$
This overdetermined linear least-square problem can be translated to the following form:
$$
\begin{equation}
    E(a,b)=\left\|\mathbf{b}-A\,\mathbf{x}\right\|_2^2=\sum_{i=1}^m (y_i-a-b\,x_i)^2.
\end{equation}
$$
Now, to apply the Gradient Descent algorithm we need to compute the Gradient of $E(a,b)$ with respect to $a$ and $b$, which is the following,
$$
\begin{align*}
    \frac{\partial E}{\partial a} &= \sum_{i=1}^m -2\,(y_i-a-b\,x_i),\\
    \frac{\partial E}{\partial b} &= \sum_{i=1}^m -2\,x_i\,(y_i-a-b\,x_i).
\end{align*}
$$
Notice that in this case we don't want to cancel out the "-" (minus) sign since it will change the direction of the Gradient.
Now, we have everything to apply the Gradient Descent in 2D.
For comparison purposes, we will also include the solution obtain by the normal equations.

```python
def GD_2D_linear(a0=2, b0=2, alpha=0, n=0, m=10):

    # Building data.
    np.random.seed(0)
    xi = np.random.normal(size=m)
    yi = -2+xi+np.random.normal(loc=0, scale=0.5, size=m)

    # Defining matrix A and the right-hand-side.
    # Recall that we usually denote as b the right-hand-side but to avoid confusion with
    # the coefficient b, we will just call it RHS.
    A = np.ones((m,2))
    A[:,1] = xi
    RHS = yi

    # Defining the Gradient
    E = lambda a, b: np.sum(np.power(yi-a-b*xi,2))
    G = lambda a, b: np.array([np.sum(-2*(yi-a-b*xi)), np.sum(-2*xi*(yi-a-b*xi))],dtype=float)
    # This fucntion will help us to evaluate the Gradient on the points (X[i,j],Y[i,j])
    def E_mG_XY(AA,BB):
        Z = np.zeros_like(AA)
        U = np.zeros_like(AA)
        V = np.zeros_like(AA)
        for i in range(m):
            for j in range(m):
                Z[i,j]=E(AA[i,j],BB[i,j])
                uv = -G(AA[i,j],BB[i,j])
                U[i,j] = uv[0]
                V[i,j] = uv[1]
        return Z, U, V

    # Plotting the function and its gradient.
    # Credits:
    #  https://matplotlib.org/stable/gallery/images_contours_and_fields/plot_streamplot.html
    #  https://scipython.com/blog/visualizing-a-vector-field-with-matplotlib/
    x = np.linspace(-5,5,m)
    AA, BB = np.meshgrid(x,x)
    fig = plt.figure(figsize=(14,10))
    Z, U, V = E_mG_XY(AA,BB)
    cont = plt.contour(AA,BB,Z, 100)
    stream = plt.streamplot(AA, BB, U, V, color=Z, linewidth=2, cmap='autumn', arrowstyle='->', arrowsize=2)
    fig.colorbar(stream.lines)
    fig.colorbar(cont)
    plt.scatter(a0, b0, s=300, marker='.', c='k')
    my_grad = G(a0,b0)
    my_title = r'$\alpha=$ %.4f, $E(a,b)=$ %.4f, $\nabla E(a,b)=$ [%.4f, %.4f]' %  (alpha, E(a0,b0), my_grad[0], my_grad[1])
    plt.title(my_title)

    # Computing steps of Gradient Descent
    if n>0:
        ab_output=np.zeros((n+1,2))
        z0 = np.array([a0,b0],dtype=float)
        z0[0] = a0
        z0[1] = b0
        ab_output[0,:]=z0
        # The Gradient Descent Algorithm
        for k in range(n):
            G_E_0=G(z0[0],z0[1])
            z1 = z0-alpha*G_E_0
            ab_output[k+1,:]=z1
            z0 = z1
            plt.scatter(z1[0], z1[1], s=300, marker='.', c='k')
        plt.plot(ab_output[:,0],ab_output[:,1],'k-')
        my_grad = G(ab_output[-1,0],ab_output[-1,1])
        my_title = r'$\alpha=$ %.4f, $E(a,b)=$ %.4f, $\nabla E(a,b)=$ [%.4f, %.4f]' %  (alpha, E(ab_output[-1,0],ab_output[-1,1]), my_grad[0], my_grad[1])
        plt.title(my_title)
    plt.show()

interact(GD_2D_linear, a0=(-4,4,0.1), b0=(-4,4,0.1), alpha=(0,0.1,0.0001), n=(0,100,1), m=(10,100,10))
```

    interactive(children=(FloatSlider(value=2.0, description='a0', max=4.0, min=-4.0), FloatSlider(value=2.0, desc…

    <function __main__.GD_2D_linear(a0=2, b0=2, alpha=0, n=0, m=10)>

In the previous implementation we used the following notation:
- $n$: Number of iteration of Gradient Descent
- Black dot: Solution $[a_n,b_n]$ at $n$-th step of the Gradient Descent.
- Red-Yellow streamplot: Stream plot of the vector field generated by minus the Gradient of the error function $E(a,b)$
- Blue-Green contour: Contour plot of the error function $E(a,b)$.

Questions:
- Try: $\alpha=0.02$, $n=20$, and $m=10$. What do you observe? (keep initialization values of $a_0$ and $a_1$)
- Try: $\alpha=0.04$, $n=20$, and $m=10$. What do you observe? (keep initialization values of $a_0$ and $a_1$)
- Try: $\alpha=0.08$, $n=20$, and $m=10$. What do you observe? (keep initialization values of $a_0$ and $a_1$)
- Can we use a large value of $\alpha$?
- How is related $\alpha$ and the iteration number $n$?

# Gradient Descent for a 2D nonlinear least-square problem
[Back to TOC](#toc)

In this case, we will explore the use of the the Gradient Descent algorithm applied to a nonlinear least-square problem with an exponential fit.
Let the function to be fit be,
$$
\begin{equation}
    y(t) = a\,\exp(b\,t),
\end{equation}
$$
where the error function is defined as follows,
$$
\begin{equation}
    E(a,b)=\sum_{i=1}^m (y_i-a\,\exp(b\,t_i))^2.
\end{equation}
$$
Now, to apply the Gradient Descent algorithm we need to compute the Gradient of $E(a,b)$ with respect to $a$ and $b$, which is the following,
$$
\begin{align*}
    \frac{\partial E}{\partial a} &= \sum_{i=1}^m 2\,\exp(b\,t_i)(a\,\exp(b\,t_i)-y_i),\\
    \frac{\partial E}{\partial b} &= \sum_{i=1}^m 2\,a\,\exp(b\,t_i)\,t_i\,(a\,\exp(b\,t_i)-y_i).
\end{align*}
$$
As you may expect, this approach may create very large values for the gradient, which will be very challenging to handle them numerically.
So, an alternative approach is the following,  which we will call it "The Variant":
- Select an initial guess, say $\mathbf{x}_0$
- Compute the direction of fastest decrease: $\mathbf{d}_0=-\nabla E(\mathbf{x}_0)$
- Update the approximation $\mathbf{x}_1=\mathbf{x}_0+\alpha\,\frac{\mathbf{d}_0}{\|\mathbf{d}_0\|}$
- Iterate until certain threshold is achieved.
Thus, the only change is on the magnitud of the **direction** vector used.
In this case, it will be a unitary direction.
This brings the advantage that $\alpha$ now controls the **length** of the update.
This is useful when you want to control the increment, otherwise it may require a very fine tuning of the parameter (or in general hyperparameter tuning!).

```python
def GD_2D_nonlinear(a0=0.75, b0=0.75, alpha=0, n=0, m=10, TheVariantFlag=False):

    # Building data.
    np.random.seed(0)
    a = 1.1
    b = 0.23
    y = lambda t: a*np.exp(b*t)
    T = 10
    ti = T*(np.random.rand(m)*2-1)
    yi = y(ti)+np.random.normal(loc=0, scale=0.1, size=m)

    # Defining the Gradient
    E = lambda a, b: np.sum(np.power(yi-a*np.exp(b*ti),2))
    G = lambda a, b: np.array([np.sum(2*np.exp(b*ti)*(a*np.exp(b*ti)-yi)), np.sum(2*a*np.exp(b*ti)*ti*(a*np.exp(b*ti)-yi))],dtype=float)
    # This fucntion will help us to evaluate the Gradient on the points (X[i,j],Y[i,j])
    def E_mG_XY(AA,BB):
        Z = np.zeros_like(AA)
        U = np.zeros_like(AA)
        V = np.zeros_like(AA)
        for i in range(m):
            for j in range(m):
                Z[i,j]=E(AA[i,j],BB[i,j])
                uv = -G(AA[i,j],BB[i,j])
                U[i,j] = uv[0]
                V[i,j] = uv[1]
        return Z, U, V

    # Plotting the function and its gradient.
    # Credits:
    #  https://matplotlib.org/stable/gallery/images_contours_and_fields/plot_streamplot.html
    #  https://scipython.com/blog/visualizing-a-vector-field-with-matplotlib/
    x = np.linspace(-3,3,m)
    AA, BB = np.meshgrid(x,x)
    fig = plt.figure(figsize=(14,10))
    Z, U, V = E_mG_XY(AA,BB)
    cont = plt.contour(AA,BB,Z, 10)
    stream = plt.streamplot(AA, BB, U, V, color=Z, linewidth=2, cmap='autumn', arrowstyle='->', arrowsize=2)
    fig.colorbar(stream.lines)
    fig.colorbar(cont)
    plt.scatter(a0, b0, s=300, marker='.', c='k')
    my_grad = G(a0,b0)
    my_title = r'$\alpha=$ %.4f, $E(a,b)=$ %.4f, $\nabla E(a,b)=$ [%.4f, %.4f]' %  (alpha, E(a0,b0), my_grad[0], my_grad[1])
    plt.title(my_title)

    # Computing steps of Gradient Descent
    if n>0:
        ab_output=np.zeros((n+1,2))
        z0 = np.array([a0,b0],dtype=float)
        z0[0] = a0
        z0[1] = b0
        ab_output[0,:]=z0
        # The Gradient Descent Algorithm
        for k in range(n):
            G_E_0=G(z0[0],z0[1])
            if not TheVariantFlag:
                # Traditional GD
                z1 = z0-alpha*G_E_0
            else:
                # The Variant! Why would this be useful?
                z1 = z0-alpha*G_E_0/np.linalg.norm(G_E_0)
            ab_output[k+1,:]=z1
            z0 = z1
            plt.scatter(z1[0], z1[1], s=300, marker='.', c='k')
        plt.plot(ab_output[:,0],ab_output[:,1],'k-')
        my_grad = G(ab_output[-1,0],ab_output[-1,1])
        my_title = r'$\alpha=$ %.6f, $E(a,b)=$ %.4f, $\nabla E(a,b)=$ [%.4f, %.4f]' %  (alpha, E(ab_output[-1,0],ab_output[-1,1]), my_grad[0], my_grad[1])
        plt.title(my_title)
        print('GD found:',ab_output[-1,0],ab_output[-1,1])

    # Plotting the original data and the "transformed" solution
    # Using the same notation from classnotes:
    A = np.ones((m,2))
    A[:,1]=ti
    K_c2 =np.linalg.lstsq(A,np.log(yi), rcond=None)[0]
    c1_ls = np.exp(K_c2[0])
    c2_ls = K_c2[1]
    print('Transformed Linear LS solution:',c1_ls, c2_ls)
    plt.plot(c1_ls,c2_ls,'ms',markersize=20, label='Transformed Linear LS')
    print('Original data:',a,b)
    plt.plot(a,b,'bd',markersize=20, label='Original data')

    plt.legend(loc='lower right')
    plt.show()

radio_button_TheVariant=RadioButtons(
    options=[('Traditional GD',False),('The Variant GD',True)],
    value=False,
    description='GD type:',
    disabled=False
)

interact(GD_2D_nonlinear, a0=(-2,2,0.01), b0=(-2,2,0.01), alpha=(0,1,0.0001), n=(0,1000,1), m=(10,100,10), TheVariantFlag=radio_button_TheVariant)
```

    interactive(children=(FloatSlider(value=0.75, description='a0', max=2.0, min=-2.0, step=0.01), FloatSlider(val…

    <function __main__.GD_2D_nonlinear(a0=0.75, b0=0.75, alpha=0, n=0, m=10, TheVariantFlag=False)>

In the previous implementation we used the following notation:
- $n$: Number of iteration of Gradient Descent
- Black dot: Solution $[a_n,b_n]$ at $n$-th step of the Gradient Descent.
- Red-Yellow streamplot: Stream plot of the vector field generated by minus the Gradient of the error function $E(a,b)$
- Blue-Green contour: Contour plot of the error function $E(a,b)$.

# Further Study
[Back to TOC](#toc)

Another extension of the Gradient Descent is the so called _Stochastic Gradient Descent Method (SGD)_, very popular in Data Science, Machine Learning and Artificial Neural Networks (ANN) in general.
Here a interesting reference: [Link](https://optimization.cbe.cornell.edu/index.php?title=Stochastic_gradient_descent), another good reference is the textbook _[Linear Algebra and leraning from data](https://math.mit.edu/~gs/learningfromdata/)_ by Professor Gilbert Strang, page 359.

A simple way to undertand the SGD is as follows:
- Select an initial guess, say $\mathbf{x}_0$
- Select a sample of data $D_k$ from the dataset $D$, where $k$ indicates the number of _data points_ of the sample.
- Define the error only including the _data points_ from the sample $D_k$, and call it $E_k(\cdot)$
- Compute the direction of fastest decrease: $\mathbf{d}^{[k]}_0=-\nabla E_k(\mathbf{x}_0)$
- Update the approximation $\mathbf{x}_1=\mathbf{x}_0+\alpha\,\mathbf{d}^{[k]}_0$
- Iterate until certain threshold is achieved.

So, the key point here is that we don't use all the dataset $D$ to update the coefficients on each iteration, it clearly has the advantage that the computation is way faster but the question it arises is that, _would this affect the convergence?_ Answer: Try it numerically! In general, this approximation behaves very well when used in ANN since in ANN they don't want to _overfit_ the coefficients to the dataset.
Notice that the size of the sample $k$ could it be even $1$, this makes the computation very fast!

Before we finish, it is useful to make the connection between the terminology used here and the terminology used in ANN,
- Error function $E(\cdot)$ $\rightarrow$ Loss function $L(\cdot)=\frac{1}{m}E(\cdot)$. Notice however that the loss function $L(\cdot)$ in ANN may not have a quadratic form, for instance it could be $\frac{1}{m}\sum |y_i-a-b\,x_i|$, i.e. the sum of the absolutes values. And in general it may also consider _activator functions_ $\phi(\cdot)$ to model neurons, which modify the loss function as follows $\frac{1}{m}\sum \phi(y_i-a-b\,x_i)$.
- Coefficient $\alpha$ $\rightarrow$ It is called _learning rate_ in ANN, since it controls how fast the ANN _learns_ from samples. As we say in this jupyter notebook, it is very important for a good _training_.
- Adjusting coefficients $\rightarrow$ Training. This the step where the ANN _learn_ from _samples_. Notice that in ANN it may not be required a low error, since it may affect the _generalization capabilities_ of the ANN.
- A brief but useful explanation of Deep Learning is [here](https://math.mit.edu/%7Egs/learningfromdata/siam.pdf).

# Acknowledgements
[Back to TOC](#toc)
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) DI UTFSM. November 2021.- v1.0.
* _Update November 2021 - v1.01 - C.Torres_ : Fixing TOC.
* _Update November 2021 - v1.02 - C.Torres_ : Fixing titles size, typos and adding further study section.

```python

```

SC1v2/Bonus - 07-08 - Gradient Descent and Nonlinear Least-Square.ipynb