# The transport-length hillslope diffuser

For more Landlab tutorials, click here: https://landlab.readthedocs.io/en/latest/user_guide/tutorials.html

# WARNING
**Some components used in this tutorial are not included in the latest release of Landlab. You need to do a developer install to run this tutorial (see https://landlab.github.io, and follow the links to installation instructions, and thence to developer install).**

This Jupyter notebook illustrates running the transport-length-model hillslope diffusion component in a simple example.

# The Basics

This component uses an approach similar to the Davy and Lague (2009) equation for fluvial erosion and transport, and applies it to hillslope diffusion. The formulation and implementation were inspired by Carretier et al. (2016); see this paper and references therein for justification.

## Theory

The elevation $z$ of a point of the landscape (such as a grid node) changes according to:

\begin{equation}
 \frac{\partial z}{\partial t} = -\epsilon + D + U  \tag{1}\label{eq:1},
\end{equation}

and we define:
\begin{equation}
 D = \frac{q_s}{L}  \tag{2}\label{eq:2},
\end{equation}

where $\epsilon$ is the local erosion rate [*L/T*], $D$ the local deposition rate [*L/T*], $U$ the uplift (or subsidence) rate [*L/T*], $q_s$ the incoming sediment flux per unit width [*L$^2$/T*] and $L$ is the **transport length**.

We specify the erosion rate $\epsilon$ and the transport length $L$:

\begin{equation}
 \epsilon = \kappa S  \tag{3}\label{eq:3}
\end{equation}

\begin{equation}
 L = \frac{dx}{1-({S}/{S_c})^2}  \tag{4}\label{eq:4}
\end{equation}

where $\kappa$ [*L/T*] is an erodibility coefficient, $S$ is the local slope [*L/L*] and $S_c$ is the critical slope [*L/L*].

Thus, the elevation variation results from the difference between local rates of detachment and deposition.

The detachment rate is proportional to the local gradient. However, the deposition rate ($q_s/L$) depends on the local slope and the critical slope:
- when $S \ll S_c$, most of the sediment entering a node is deposited there, this is the pure diffusion case. In this case, the sediment flux $q_s$ does not include sediment eroded from above and is thus "local".
- when $S \approx S_c$, $L$ becomes infinity and there is no redeposition on the node, the sediments are transferred further downstream.  This behaviour corresponds to mass wasting, grains can travel a long distance before being deposited. In that case, the flux $q_s$ is "non-local" as it incorporates sediments that have both been detached  locally and transited from upslope.
- for an intermediate $S$, there is a prgogressive transition between pure creep and "balistic" transport of the material. This is consistent with experiments (Roering et al., 2001; Gabet and Mendoza, 2012).

## Contrast with the non-linear diffusion model

Previous models typically use a "non-linear" diffusion model proposed by different authors (e.g. Andrews and Hanks, 1985; Hanks, 1999; Roering et al., 1999) and supported by $^{10}$Be-derived erosion rates (e.g. Binnie et al., 2007) or experiments (Roering et al., 2001). It is usually presented in the followin form:

$ $

\begin{equation}
 \frac{\partial z}{\partial t} = \frac{\partial q_s}{\partial x}  \tag{5}\label{eq:5}
\end{equation}

$ $
\begin{equation}
 q_s = \frac{\kappa' S}{1-({S}/{S_c})^2}   \tag{6}\label{eq:6}
\end{equation}

where $\kappa'$ [*L$^2$/T*] is a diffusion coefficient.

This description is thus based on the definition of a flux of transported sediment parallel to the slope:
- when the slope is small, this flux refers to diffusion-like processes such as biogenic soil disturbance, rain splash, or diffuse runoff
- when the slope gets closer to the specified critical slope, the flux increases dramatically, simulating on average the cumulative effect of mass wasting events.

Despite these conceptual differences, equations ($\ref{eq:3}$) and ($\ref{eq:4}$) predict similar topographic evolution to the 'non-linear' diffusion equations for $\kappa' = \kappa dx$, as shown in the following example.

# Example 1:

First, we import what we'll need:

```python
import numpy as np
from matplotlib.pyplot import figure, plot, show, title, xlabel, ylabel

from landlab import RasterModelGrid
from landlab.components import FlowDirectorSteepest, TransportLengthHillslopeDiffuser
from landlab.plot import imshow_grid

# to plot figures in the notebook:
%matplotlib inline
```

Make a grid and set boundary conditions:

```python
mg = RasterModelGrid(
    (100, 100), xy_spacing=10.0
)  # raster grid with 100 rows, 100 columns and dx=10m
z = np.random.rand(10000)  # random noise for initial topography
mg.add_field("topographic__elevation", z, at="node")

mg.set_closed_boundaries_at_grid_edges(
    False, True, False, True
)  # N and S boundaries are closed, E and W are open
```

Set the initial and run conditions:

```python
total_t = 2000000.0  # total run time (yr)
dt = 1000.0  # time step (yr)
nt = int(total_t // dt)  # number of time steps
uplift_rate = 0.0001  # uplift rate (m/yr)

kappa = 0.001  # erodibility (m/yr)
Sc = 0.6  # critical slope
```

Instantiate the components:
The hillslope diffusion component must be used together with a flow router/director that provides the steepest downstream slope for each node, with a D4 method (creates the field *topographic__steepest_slope* at nodes).

```python
fdir = FlowDirectorSteepest(mg)
tl_diff = TransportLengthHillslopeDiffuser(mg, erodibility=kappa, slope_crit=Sc)
```

Run the components for 2 Myr and trace an East-West cross-section of the topography every 100 kyr:

```python
for t in range(nt):
    fdir.run_one_step()
    tl_diff.run_one_step(dt)
    z[mg.core_nodes] += uplift_rate * dt  # add the uplift

    # add some output to let us see we aren't hanging:
    if t % 100 == 0:
        print(t * dt)

        # plot east-west cross-section of topography:
        x_plot = range(0, 1000, 10)
        z_plot = z[101:201]
        figure("cross-section")
        plot(x_plot, z_plot)

figure("cross-section")
title("East-West cross section")
xlabel("x (m)")
ylabel("z (m)")
```

And plot final topography:

```python
figure("final topography")
im = imshow_grid(
    mg, "topographic__elevation", grid_units=["m", "m"], var_name="Elevation (m)"
)
```

This behaviour corresponds to the evolution observed using a classical non-linear diffusion model.

# Example 2:

In this example, we show that when the slope is steep ($S \ge S_c$), the transport-length hillsope diffusion simulates mass wasting, with long transport distances.

First, we create a grid: the western half of the grid is flat at 0 m of elevation, the eastern half is a 45-degree slope.

```python
# Create grid and topographic elevation field:
mg2 = RasterModelGrid((100, 100), xy_spacing=10.0)

z = np.zeros(mg2.number_of_nodes)
z[mg2.node_x > 500] = mg2.node_x[mg2.node_x < 490] / 10
mg2.add_field("topographic__elevation", z, at="node")

# Set boundary conditions:
mg2.set_closed_boundaries_at_grid_edges(False, True, False, True)

# Show initial topography:
im = imshow_grid(
    mg, "topographic__elevation", grid_units=["m", "m"], var_name="Elevation (m)"
)

# Plot an east-west cross-section of the initial topography:
z_plot = z[101:201]
x_plot = range(0, 1000, 10)
figure(2)
plot(x_plot, z_plot)
title("East-West cross section")
xlabel("x (m)")
ylabel("z (m)")
```

Set the run conditions:

```python
total_t = 1000000.0  # total run time (yr)
dt = 1000.0  # time step (yr)
nt = int(total_t // dt)  # number of time steps
```

Instantiate the components:

```python
fdir = FlowDirectorSteepest(mg2)
tl_diff = TransportLengthHillslopeDiffuser(mg2, erodibility=0.001, slope_crit=0.6)
```

Run for 1 Myr, plotting the cross-section regularly:

```python
for t in range(nt):
    fdir.run_one_step()
    tl_diff.run_one_step(dt)

    # add some output to let us see we aren't hanging:
    if t % 100 == 0:
        print(t * dt)
        z_plot = z[101:201]
        figure(2)
        plot(x_plot, z_plot)
```

The material is diffused from the top and along the slope and it accumulates at the bottom, where the topography flattens.

As a comparison, the following code uses linear diffusion on the same slope:

```python
# Import Linear diffuser:
from landlab.components import LinearDiffuser

# Create grid and topographic elevation field:
mg3 = RasterModelGrid((100, 100), xy_spacing=10.0)
z = np.ones(mg.number_of_nodes)
z[mg.node_x > 500] = mg.node_x[mg.node_x < 490] / 10
mg3.add_field("topographic__elevation", z, at="node")

# Set boundary conditions:
mg3.set_closed_boundaries_at_grid_edges(False, True, False, True)

# Instantiate components:
fdir = FlowDirectorSteepest(mg3)
diff = LinearDiffuser(mg3, linear_diffusivity=0.1)

# Set run conditions:
total_t = 1000000.0
dt = 1000.0
nt = int(total_t // dt)

# Run for 1 Myr, plotting east-west cross-section regularly:
for t in range(nt):
    fdir.run_one_step()
    diff.run_one_step(dt)

    # add some output to let us see we aren't hanging:
    if t % 100 == 0:
        print(t * dt)
        z_plot = z[101:201]
        figure(2)
        plot(x_plot, z_plot)
```


# Programming Exercise 5:
# Regularized Linear Regression and Bias vs Variance

## Introduction

In this exercise, you will implement regularized linear regression and use it to study models with different bias-variance properties. Before starting on the programming exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics.

All the information you need for solving this assignment is in this notebook, and all the code you will be implementing will take place within this notebook. The assignment can be promptly submitted to the coursera grader directly from this notebook (code and instructions are included below).

Before we begin with the exercises, we need to import all libraries required for this programming exercise. Throughout the course, we will be using [`numpy`](http://www.numpy.org/) for all arrays and matrix operations, [`matplotlib`](https://matplotlib.org/) for plotting, and [`scipy`](https://docs.scipy.org/doc/scipy/reference/) for scientific and numerical computation functions and tools. You can find instructions on how to install required libraries in the README file in the [github repository](https://github.com/dibgerge/ml-coursera-python-assignments).

```python
# used for manipulating directory paths
import os

# Scientific and vector computation for python
import numpy as np

# Plotting library
from matplotlib import pyplot

# Optimization module in scipy
from scipy import optimize

# will be used to load MATLAB mat datafile format
from scipy.io import loadmat

# library written for this exercise providing additional functions for assignment submission, and others
import utils

# define the submission/grader object for this exercise
grader = utils.Grader()

# tells matplotlib to embed plots within the notebook
%matplotlib inline
```

## Submission and Grading

After completing each part of the assignment, be sure to submit your solutions to the grader. The following is a breakdown of how each part of this exercise is scored.

| Section | Part                                             | Submitted Function                | Points |
| :-      |:-                                                |:-                                 | :-:    |
| 1       | [Regularized Linear Regression Cost Function](#section1)      | [`linearRegCostFunction`](#linearRegCostFunction) |  25    |
| 2       | [Regularized Linear Regression Gradient](#section2)           | [`linearRegCostFunction`](#linearRegCostFunction) |25      |
| 3       | [Learning Curve](#section3)                                   | [`learningCurve`](#func2)         | 20     |
| 4       | [Polynomial Feature Mapping](#section4)                       | [`polyFeatures`](#polyFeatures)          | 10     |
| 5       | [Cross Validation Curve](#section5)                           | [`validationCurve`](#validationCurve)       | 20     |
|         | Total Points                                     |                                   |100     |

You are allowed to submit your solutions multiple times, and we will take only the highest score into consideration.

At the end of each section in this notebook, we have a cell which contains code for submitting the solutions thus far to the grader. Execute the cell to see your score up to the current section. For all your work to be submitted properly, you must execute those cells at least once.

## 1 Regularized Linear Regression

In the first half of the exercise, you will implement regularized linear regression to predict the amount of water flowing out of a dam using the change of water level in a reservoir. In the next half, you will go through some diagnostics of debugging learning algorithms and examine the effects of bias v.s.
variance.

### 1.1 Visualizing the dataset

We will begin by visualizing the dataset containing historical records on the change in the water level, $x$, and the amount of water flowing out of the dam, $y$. This dataset is divided into three parts:

- A **training** set that your model will learn on: `X`, `y`
- A **cross validation** set for determining the regularization parameter: `Xval`, `yval`
- A **test** set for evaluating performance. These are “unseen” examples which your model did not see during training: `Xtest`, `ytest`

Run the next cell to plot the training data. In the following parts, you will implement linear regression and use that to fit a straight line to the data and plot learning curves. Following that, you will implement polynomial regression to find a better fit to the data.

```python
# Load from ex5data1.mat, where all variables will be store in a dictionary
data = loadmat(os.path.join('Data', 'ex5data1.mat'))

# Extract train, test, validation data from dictionary
# and also convert y's form 2-D matrix (MATLAB format) to a numpy vector
X, y = data['X'], data['y'][:, 0]
Xtest, ytest = data['Xtest'], data['ytest'][:, 0]
Xval, yval = data['Xval'], data['yval'][:, 0]

# m = Number of examples
m = y.size

# Plot training data
pyplot.plot(X, y, 'ro', ms=10, mec='k', mew=1)
pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)');
```

### 1.2 Regularized linear regression cost function

Recall that regularized linear regression has the following cost function:

$$ J(\theta) = \frac{1}{2m} \left( \sum_{i=1}^m \left( h_\theta\left( x^{(i)} \right) - y^{(i)} \right)^2 \right) + \frac{\lambda}{2m} \left( \sum_{j=1}^n \theta_j^2 \right)$$

where $\lambda$ is a regularization parameter which controls the degree of regularization (thus, help preventing overfitting). The regularization term puts a penalty on the overall cost J. As the magnitudes of the model parameters $\theta_j$ increase, the penalty increases as well. Note that you should not regularize
the $\theta_0$ term.

You should now complete the code in the function `linearRegCostFunction` in the next cell. Your task is to calculate the regularized linear regression cost function. If possible, try to vectorize your code and avoid writing loops.

```python
def linearRegCostFunction(X, y, theta, lambda_=0.0):
    """
    Compute cost and gradient for regularized linear regression
    with multiple variables. Computes the cost of using theta as
    the parameter for linear regression to fit the data points in X and y.

    Parameters
    ----------
    X : array_like
        The dataset. Matrix with shape (m x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    y : array_like
        The functions values at each datapoint. A vector of
        shape (m, ).

    theta : array_like
        The parameters for linear regression. A vector of shape (n+1,).

    lambda_ : float, optional
        The regularization parameter.

    Returns
    -------
    J : float
        The computed cost function.

    grad : array_like
        The value of the cost function gradient w.r.t theta.
        A vector of shape (n+1, ).

    Instructions
    ------------
    Compute the cost and gradient of regularized linear regression for
    a particular choice of theta.
    You should set J to the cost and grad to the gradient.
    """
    # Initialize some useful values
    m = y.size # number of training examples

    # You need to return the following variables correctly
    J = 0
    grad = np.zeros(theta.shape)

    # ====================== YOUR CODE HERE ======================
    h = X.dot(theta)
    J = (np.sum(np.square(h - y))) / (2*m) + (lambda_/(2*m))*np.sum(np.square(theta[1:]))

    grad = [(np.sum((h - y) * X[:, j]) / m) + (lambda_*theta[j]/m) for j in range(theta.shape[0])]
    grad[0] = np.sum((h - y) * X[:, 0]) / m
    # ============================================================
    return J, grad
```

When you are finished, the next cell will run your cost function using `theta` initialized at `[1, 1]`. You should expect to see an output of 303.993.

```python
theta = np.array([1, 1])
J, _ = linearRegCostFunction(np.concatenate([np.ones((m, 1)), X], axis=1), y, theta, 1)

print('Cost at theta = [1, 1]:\t   %f ' % J)
print('This value should be about 303.993192)\n' % J)
```

    Cost at theta = [1, 1]:	   303.993192
    This value should be about 303.993192)

After completing a part of the exercise, you can submit your solutions for grading by first adding the function you modified to the submission object, and then sending your function to Coursera for grading.

The submission script will prompt you for your login e-mail and submission token. You can obtain a submission token from the web page for the assignment. You are allowed to submit your solutions multiple times, and we will take only the highest score into consideration.

*Execute the following cell to grade your solution to the first part of this exercise.*

```python
grader[1] = linearRegCostFunction
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (gresova11@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
                               Validation Curve |  25 /  25 | Nice work!
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |   0 /  20 | Your answer is incorrect.
                                 Learning Curve |   0 /  10 | Your answer is incorrect.
                     Polynomial Feature Mapping |   0 /  20 | Your answer is incorrect.
                                      --------------------------------
                                                |  50 / 100 |

### 1.3 Regularized linear regression gradient

Correspondingly, the partial derivative of the cost function for regularized linear regression is defined as:

$$
\begin{align}
& \frac{\partial J(\theta)}{\partial \theta_0} = \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left(x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} & \qquad \text{for } j = 0 \\
& \frac{\partial J(\theta)}{\partial \theta_j} = \left( \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left( x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} \right) + \frac{\lambda}{m} \theta_j & \qquad \text{for } j \ge 1
\end{align}
$$

In the function [`linearRegCostFunction`](#linearRegCostFunction) above, add code to calculate the gradient, returning it in the variable `grad`. Do not forget to re-execute the cell containing this function to update the function's definition.

When you are finished, use the next cell to  run your gradient function using theta initialized at `[1, 1]`. You should expect to see a gradient of `[-15.30, 598.250]`.

```python
theta = np.array([1, 1])
J, grad = linearRegCostFunction(np.concatenate([np.ones((m, 1)), X], axis=1), y, theta, 1)

print('Gradient at theta = [1, 1]:  [{:.6f}, {:.6f}] '.format(*grad))
print(' (this value should be about [-15.303016, 598.250744])\n')
```

    Gradient at theta = [1, 1]:  [-15.303016, 598.250744]
     (this value should be about [-15.303016, 598.250744])

*You should now submit your solutions.*

```python
grader[2] = linearRegCostFunction
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (gresova11@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
                               Validation Curve |   0 /  25 | Your answer is incorrect.
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |   0 /  20 | Your answer is incorrect.
                                 Learning Curve |   0 /  10 | Your answer is incorrect.
                     Polynomial Feature Mapping |   0 /  20 | Your answer is incorrect.
                                      --------------------------------
                                                |  25 / 100 |

### Fitting linear regression

Once your cost function and gradient are working correctly, the next cell will run the code in `trainLinearReg` (found in the module `utils.py`) to compute the optimal values of $\theta$. This training function uses `scipy`'s optimization module to minimize the cost function.

In this part, we set regularization parameter $\lambda$ to zero. Because our current implementation of linear regression is trying to fit a 2-dimensional $\theta$, regularization will not be incredibly helpful for a $\theta$ of such low dimension. In the later parts of the exercise, you will be using polynomial regression with regularization.

Finally, the code in the next cell should also plot the best fit line, which should look like the figure below.

The best fit line tells us that the model is not a good fit to the data because the data has a non-linear pattern. While visualizing the best fit as shown is one possible way to debug your learning algorithm, it is not always easy to visualize the data and model. In the next section, you will implement a function to generate learning curves that can help you debug your learning algorithm even if it is not easy to visualize the
data.

```python
# add a columns of ones for the y-intercept
X_aug = np.concatenate([np.ones((m, 1)), X], axis=1)
theta = utils.trainLinearReg(linearRegCostFunction, X_aug, y, lambda_=0)

#  Plot fit over the data
pyplot.plot(X, y, 'ro', ms=10, mec='k', mew=1.5)
pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.plot(X, np.dot(X_aug, theta), '--', lw=2);
```

## 2 Bias-variance

An important concept in machine learning is the bias-variance tradeoff. Models with high bias are not complex enough for the data and tend to underfit, while models with high variance overfit to the training data.

In this part of the exercise, you will plot training and test errors on a learning curve to diagnose bias-variance problems.

### 2.1 Learning Curves

You will now implement code to generate the learning curves that will be useful in debugging learning algorithms. Recall that a learning curve plots training and cross validation error as a function of training set size. Your job is to fill in the function `learningCurve` in the next cell, so that it returns a vector of errors for the training set and cross validation set.

To plot the learning curve, we need a training and cross validation set error for different training set sizes. To obtain different training set sizes, you should use different subsets of the original training set `X`. Specifically, for a training set size of $i$, you should use the first $i$ examples (i.e., `X[:i, :]`
and `y[:i]`).

You can use the `trainLinearReg` function (by calling `utils.trainLinearReg(...)`) to find the $\theta$ parameters. Note that the `lambda_` is passed as a parameter to the `learningCurve` function.
After learning the $\theta$ parameters, you should compute the error on the training and cross validation sets. Recall that the training error for a dataset is defined as

$$ J_{\text{train}} = \frac{1}{2m} \left[ \sum_{i=1}^m \left(h_\theta \left( x^{(i)} \right) - y^{(i)} \right)^2 \right] $$

In particular, note that the training error does not include the regularization term. One way to compute the training error is to use your existing cost function and set $\lambda$ to 0 only when using it to compute the training error and cross validation error. When you are computing the training set error, make sure you compute it on the training subset (i.e., `X[:n,:]` and `y[:n]`) instead of the entire training set. However, for the cross validation error, you should compute it over the entire cross validation set. You should store
the computed errors in the vectors error train and error val.

```python
def learningCurve(X, y, Xval, yval, lambda_=0):
    """
    Generates the train and cross validation set errors needed to plot a learning curve
    returns the train and cross validation set errors for a learning curve.

    In this function, you will compute the train and test errors for
    dataset sizes from 1 up to m. In practice, when working with larger
    datasets, you might want to do this in larger intervals.

    Parameters
    ----------
    X : array_like
        The training dataset. Matrix with shape (m x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    y : array_like
        The functions values at each training datapoint. A vector of
        shape (m, ).

    Xval : array_like
        The validation dataset. Matrix with shape (m_val x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    yval : array_like
        The functions values at each validation datapoint. A vector of
        shape (m_val, ).

    lambda_ : float, optional
        The regularization parameter.

    Returns
    -------
    error_train : array_like
        A vector of shape m. error_train[i] contains the training error for
        i examples.
    error_val : array_like
        A vecotr of shape m. error_val[i] contains the validation error for
        i training examples.

    Instructions
    ------------
    Fill in this function to return training errors in error_train and the
    cross validation errors in error_val. i.e., error_train[i] and
    error_val[i] should give you the errors obtained after training on i examples.

    Notes
    -----
    - You should evaluate the training error on the first i training
      examples (i.e., X[:i, :] and y[:i]).

      For the cross-validation error, you should instead evaluate on
      the _entire_ cross validation set (Xval and yval).

    - If you are using your cost function (linearRegCostFunction) to compute
      the training and cross validation error, you should call the function with
      the lambda argument set to 0. Do note that you will still need to use
      lambda when running the training to obtain the theta parameters.

    Hint
    ----
    You can loop over the examples with the following:

           for i in range(1, m+1):
               # Compute train/cross validation errors using training examples
               # X[:i, :] and y[:i], storing the result in
               # error_train[i-1] and error_val[i-1]
               ....
    """
    # Number of training examples
    m = y.size

    # You need to return these values correctly
    error_train = np.zeros(m)
    error_val   = np.zeros(m)

    # ====================== YOUR CODE HERE ======================
    for i in range(1, m+1):
        theta = utils.trainLinearReg(linearRegCostFunction, X[:i, :], y[:i], lambda_=lambda_)
        error_train[i - 1], _ = linearRegCostFunction(X[:i, :], y[:i], theta, 0)
        error_val[i - 1], _ = linearRegCostFunction(Xval, yval, theta, 0)

    # =============================================================
    return error_train, error_val
```

When you are finished implementing the function `learningCurve`, executing the next cell prints the learning curves and produce a plot similar to the figure below.

In the learning curve figure, you can observe that both the train error and cross validation error are high when the number of training examples is increased. This reflects a high bias problem in the model - the linear regression model is too simple and is unable to fit our dataset well. In the next section, you will implement polynomial regression to fit a better model for this dataset.

```python
X_aug = np.concatenate([np.ones((m, 1)), X], axis=1)
Xval_aug = np.concatenate([np.ones((yval.size, 1)), Xval], axis=1)
error_train, error_val = learningCurve(X_aug, y, Xval_aug, yval, lambda_=0)

pyplot.plot(np.arange(1, m+1), error_train, np.arange(1, m+1), error_val, lw=2)
pyplot.title('Learning curve for linear regression')
pyplot.legend(['Train', 'Cross Validation'])
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 150])

print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

*You should now submit your solutions.*

```python
grader[3] = learningCurve
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (gresova11@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
                               Validation Curve |  25 /  25 | Nice work!
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |  20 /  20 | Nice work!
                                 Learning Curve |   0 /  10 | Your answer is incorrect.
                     Polynomial Feature Mapping |   0 /  20 | Your answer is incorrect.
                                      --------------------------------
                                                |  70 / 100 |

## 3 Polynomial regression

The problem with our linear model was that it was too simple for the data
and resulted in underfitting (high bias). In this part of the exercise, you will address this problem by adding more features. For polynomial regression, our hypothesis has the form:

$$
\begin{align}
h_\theta(x)  &= \theta_0 + \theta_1 \times (\text{waterLevel}) + \theta_2 \times (\text{waterLevel})^2 + \cdots + \theta_p \times (\text{waterLevel})^p \\
& = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \cdots + \theta_p x_p
\end{align}
$$

Notice that by defining $x_1 = (\text{waterLevel})$, $x_2 = (\text{waterLevel})^2$ , $\cdots$, $x_p =
(\text{waterLevel})^p$, we obtain a linear regression model where the features are the various powers of the original value (waterLevel).

Now, you will add more features using the higher powers of the existing feature $x$ in the dataset. Your task in this part is to complete the code in the function `polyFeatures` in the next cell. The function should map the original training set $X$ of size $m \times 1$ into its higher powers. Specifically, when a training set $X$ of size $m \times 1$ is passed into the function, the function should return a $m \times p$ matrix `X_poly`, where column 1 holds the original values of X, column 2 holds the values of $X^2$, column 3 holds the values of $X^3$, and so on. Note that you don’t have to account for the zero-eth power in this function.

```python
def polyFeatures(X, p):
    """
    Maps X (1D vector) into the p-th power.

    Parameters
    ----------
    X : array_like
        A data vector of size m, where m is the number of examples.

    p : int
        The polynomial power to map the features.

    Returns
    -------
    X_poly : array_like
        A matrix of shape (m x p) where p is the polynomial
        power and m is the number of examples. That is:

        X_poly[i, :] = [X[i], X[i]**2, X[i]**3 ...  X[i]**p]

    Instructions
    ------------
    Given a vector X, return a matrix X_poly where the p-th column of
    X contains the values of X to the p-th power.
    """
    # You need to return the following variables correctly.
    X_poly = np.zeros((X.shape[0], p))

    # ====================== YOUR CODE HERE ======================

    for i in range(X.shape[0]):
        X_poly[i, :] = [np.power(X[i], power) for power in range(1, p + 1)]
    # ============================================================
    return X_poly
```

Now you have a function that will map features to a higher dimension. The next cell will apply it to the training set, the test set, and the cross validation set.

```python
p = 8

# Map X onto Polynomial Features and Normalize
X_poly = polyFeatures(X, p)
X_poly, mu, sigma = utils.featureNormalize(X_poly)
X_poly = np.concatenate([np.ones((m, 1)), X_poly], axis=1)

# Map X_poly_test and normalize (using mu and sigma)
X_poly_test = polyFeatures(Xtest, p)
X_poly_test -= mu
X_poly_test /= sigma
X_poly_test = np.concatenate([np.ones((ytest.size, 1)), X_poly_test], axis=1)

# Map X_poly_val and normalize (using mu and sigma)
X_poly_val = polyFeatures(Xval, p)
X_poly_val -= mu
X_poly_val /= sigma
X_poly_val = np.concatenate([np.ones((yval.size, 1)), X_poly_val], axis=1)

print('Normalized Training Example 1:')
X_poly[0, :]
```

    Normalized Training Example 1:

    array([ 1.        , -0.36214078, -0.75508669,  0.18222588, -0.70618991,
            0.30661792, -0.59087767,  0.3445158 , -0.50848117])

*You should now submit your solutions.*

```python
grader[4] = polyFeatures
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (gresova11@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
                               Validation Curve |  25 /  25 | Nice work!
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |  20 /  20 | Nice work!
                                 Learning Curve |  10 /  10 | Nice work!
                     Polynomial Feature Mapping |   0 /  20 | Your answer is incorrect.
                                      --------------------------------
                                                |  80 / 100 |

## 3.1 Learning Polynomial Regression

After you have completed the function `polyFeatures`, we will proceed to train polynomial regression using your linear regression cost function.

Keep in mind that even though we have polynomial terms in our feature vector, we are still solving a linear regression optimization problem. The polynomial terms have simply turned into features that we can use for linear regression. We are using the same cost function and gradient that you wrote for the earlier part of this exercise.

For this part of the exercise, you will be using a polynomial of degree 8. It turns out that if we run the training directly on the projected data, will not work well as the features would be badly scaled (e.g., an example with $x = 40$ will now have a feature $x_8 = 40^8 = 6.5 \times 10^{12}$). Therefore, you will
need to use feature normalization.

Before learning the parameters $\theta$ for the polynomial regression, we first call `featureNormalize` and normalize the features of the training set, storing the mu, sigma parameters separately. We have already implemented this function for you (in `utils.py` module) and it is the same function from the first exercise.

After learning the parameters $\theta$, you should see two plots generated for polynomial regression with $\lambda = 0$, which should be similar to the ones here:

You should see that the polynomial fit is able to follow the datapoints very well, thus, obtaining a low training error. The figure on the right shows that the training error essentially stays zero for all numbers of training samples. However, the polynomial fit is very complex and even drops off at the extremes. This is an indicator that the polynomial regression model is overfitting the training data and will not generalize well.

To better understand the problems with the unregularized ($\lambda = 0$) model, you can see that the learning curve  shows the same effect where the training error is low, but the cross validation error is high. There is a gap between the training and cross validation errors, indicating a high variance problem.

```python
lambda_ = 0
theta = utils.trainLinearReg(linearRegCostFunction, X_poly, y,
                             lambda_=lambda_, maxiter=55)

# Plot training data and fit
pyplot.plot(X, y, 'ro', ms=10, mew=1.5, mec='k')

utils.plotFit(polyFeatures, np.min(X), np.max(X), mu, sigma, theta, p)

pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.title('Polynomial Regression Fit (lambda = %f)' % lambda_)
pyplot.ylim([-20, 50])

pyplot.figure()
error_train, error_val = learningCurve(X_poly, y, X_poly_val, yval, lambda_)
pyplot.plot(np.arange(1, 1+m), error_train, np.arange(1, 1+m), error_val)

pyplot.title('Polynomial Regression Learning Curve (lambda = %f)' % lambda_)
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 100])
pyplot.legend(['Train', 'Cross Validation'])

print('Polynomial Regression (lambda = %f)\n' % lambda_)
print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

One way to combat the overfitting (high-variance) problem is to add regularization to the model. In the next section, you will get to  try different $\lambda$ parameters to see how regularization can lead to a better model.

### 3.2 Optional (ungraded) exercise: Adjusting the regularization parameter

In this section, you will get to observe how the regularization parameter affects the bias-variance of regularized polynomial regression. You should now modify the lambda parameter and try $\lambda = 1, 100$. For each of these values, the script should generate a polynomial fit to the data and also a learning curve.

For $\lambda = 1$, the generated plots should look like the figure below. You should see a polynomial fit that follows the data trend well (left) and a learning curve (right) showing that both the cross validation and training error converge to a relatively low value. This shows the $\lambda = 1$ regularized polynomial regression model does not have the high-bias or high-variance problems. In effect, it achieves a good trade-off between bias and variance.

For $\lambda = 100$, you should see a polynomial fit (figure below) that does not follow the data well. In this case, there is too much regularization and the model is unable to fit the training data.

*You do not need to submit any solutions for this optional (ungraded) exercise.*

### 3.3 Selecting $\lambda$ using a cross validation set

From the previous parts of the exercise, you observed that the value of $\lambda$ can significantly affect the results of regularized polynomial regression on the training and cross validation set. In particular, a model without regularization ($\lambda = 0$) fits the training set well, but does not generalize. Conversely, a model with too much regularization ($\lambda = 100$) does not fit the training set and testing set well. A good choice of $\lambda$ (e.g., $\lambda = 1$) can provide a good fit to the data.

In this section, you will implement an automated method to select the $\lambda$ parameter. Concretely, you will use a cross validation set to evaluate how good each $\lambda$ value is. After selecting the best $\lambda$ value using the cross validation set, we can then evaluate the model on the test set to estimate
how well the model will perform on actual unseen data.

Your task is to complete the code in the function `validationCurve`. Specifically, you should should use the `utils.trainLinearReg` function to train the model using different values of $\lambda$ and compute the training error and cross validation error. You should try $\lambda$ in the following range: {0, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10}.

```python
def validationCurve(X, y, Xval, yval):
    """
    Generate the train and validation errors needed to plot a validation
    curve that we can use to select lambda_.

    Parameters
    ----------
    X : array_like
        The training dataset. Matrix with shape (m x n) where m is the
        total number of training examples, and n is the number of features
        including any polynomial features.

    y : array_like
        The functions values at each training datapoint. A vector of
        shape (m, ).

    Xval : array_like
        The validation dataset. Matrix with shape (m_val x n) where m is the
        total number of validation examples, and n is the number of features
        including any polynomial features.

    yval : array_like
        The functions values at each validation datapoint. A vector of
        shape (m_val, ).

    Returns
    -------
    lambda_vec : list
        The values of the regularization parameters which were used in
        cross validation.

    error_train : list
        The training error computed at each value for the regularization
        parameter.

    error_val : list
        The validation error computed at each value for the regularization
        parameter.

    Instructions
    ------------
    Fill in this function to return training errors in `error_train` and
    the validation errors in `error_val`. The vector `lambda_vec` contains
    the different lambda parameters to use for each calculation of the
    errors, i.e, `error_train[i]`, and `error_val[i]` should give you the
    errors obtained after training with `lambda_ = lambda_vec[i]`.

    Note
    ----
    You can loop over lambda_vec with the following:

          for i in range(len(lambda_vec))
              lambda = lambda_vec[i]
              # Compute train / val errors when training linear
              # regression with regularization parameter lambda_
              # You should store the result in error_train[i]
              # and error_val[i]
              ....
    """
    # Selected values of lambda (you should not change this)
    lambda_vec = [0, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10]

    # You need to return these variables correctly.
    error_train = np.zeros(len(lambda_vec))
    error_val = np.zeros(len(lambda_vec))

    # ====================== YOUR CODE HERE ======================

    for i in range(len(lambda_vec)):
        lambda_ = lambda_vec[i]
        theta = utils.trainLinearReg(linearRegCostFunction, X, y, lambda_=lambda_)
        error_train[i], _ = linearRegCostFunction(X, y, theta, 0)
        error_val[i], _ = linearRegCostFunction(Xval, yval, theta, 0)

    # ============================================================
    return lambda_vec, error_train, error_val
```

After you have completed the code, the next cell will run your function and plot a cross validation curve of error v.s. $\lambda$ that allows you select which $\lambda$ parameter to use. You should see a plot similar to the figure below.

In this figure, we can see that the best value of $\lambda$ is around 3. Due to randomness
in the training and validation splits of the dataset, the cross validation error can sometimes be lower than the training error.

```python
lambda_vec, error_train, error_val = validationCurve(X_poly, y, X_poly_val, yval)

pyplot.plot(lambda_vec, error_train, '-o', lambda_vec, error_val, '-o', lw=2)
pyplot.legend(['Train', 'Cross Validation'])
pyplot.xlabel('lambda')
pyplot.ylabel('Error')

print('lambda\t\tTrain Error\tValidation Error')
for i in range(len(lambda_vec)):
    print(' %f\t%f\t%f' % (lambda_vec[i], error_train[i], error_val[i]))
```

*You should now submit your solutions.*

```python
grader[5] = validationCurve
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (gresova11@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
                               Validation Curve |  25 /  25 | Nice work!
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |  20 /  20 | Nice work!
                                 Learning Curve |  10 /  10 | Nice work!
                     Polynomial Feature Mapping |  20 /  20 | Nice work!
                                      --------------------------------
                                                | 100 / 100 |

### 3.4  Optional (ungraded) exercise: Computing test set error

In the previous part of the exercise, you implemented code to compute the cross validation error for various values of the regularization parameter $\lambda$. However, to get a better indication of the model’s performance in the real world, it is important to evaluate the “final” model on a test set that was not used in any part of training (that is, it was neither used to select the $\lambda$ parameters, nor to learn the model parameters $\theta$). For this optional (ungraded) exercise, you should compute the test error using the best value of $\lambda$ you found. In our cross validation, we obtained a test error of 3.8599 for $\lambda = 3$.

*You do not need to submit any solutions for this optional (ungraded) exercise.*

```python
lambda_best = 3
theta = utils.trainLinearReg(linearRegCostFunction, X_poly, y, lambda_=lambda_best)
error_test, _ = linearRegCostFunction(X_poly_test, ytest, theta, 0)
error_test
```

    3.859888017165775

### 3.5 Optional (ungraded) exercise: Plotting learning curves with randomly selected examples

In practice, especially for small training sets, when you plot learning curves to debug your algorithms, it is often helpful to average across multiple sets of randomly selected examples to determine the training error and cross validation error.

Concretely, to determine the training error and cross validation error for $i$ examples, you should first randomly select $i$ examples from the training set and $i$ examples from the cross validation set. You will then learn the parameters $\theta$ using the randomly chosen training set and evaluate the parameters $\theta$ on the randomly chosen training set and cross validation set. The above steps should then be repeated multiple times (say 50) and the averaged error should be used to determine the training error and cross validation error for $i$ examples.

For this optional (ungraded) exercise, you should implement the above strategy for computing the learning curves. For reference, the figure below  shows the learning curve we obtained for polynomial regression with $\lambda = 0.01$. Your figure may differ slightly due to the random selection of examples.

*You do not need to submit any solutions for this optional (ungraded) exercise.*

```python
def learningCurveRandomize(X, y, Xval, yval, iterations, lambda_=0):
    """
    Generates the train and cross validation set errors needed to plot a learning curve
    returns the train and cross validation set errors for a learning curve.

    In this function, you will compute the train and test errors for
    dataset sizes from 1 up to m. In practice, when working with larger
    datasets, you might want to do this in larger intervals.

    Parameters
    ----------
    X : array_like
        The training dataset. Matrix with shape (m x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    y : array_like
        The functions values at each training datapoint. A vector of
        shape (m, ).

    Xval : array_like
        The validation dataset. Matrix with shape (m_val x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    yval : array_like
        The functions values at each validation datapoint. A vector of
        shape (m_val, ).

    lambda_ : float, optional
        The regularization parameter.

    Returns
    -------
    error_train : array_like
        A vector of shape m. error_train[i] contains the training error for
        i examples.
    error_val : array_like
        A vecotr of shape m. error_val[i] contains the validation error for
        i training examples.

    Instructions
    ------------
    Fill in this function to return training errors in error_train and the
    cross validation errors in error_val. i.e., error_train[i] and
    error_val[i] should give you the errors obtained after training on i examples.

    Notes
    -----
    - You should evaluate the training error on the first i training
      examples (i.e., X[:i, :] and y[:i]).

      For the cross-validation error, you should instead evaluate on
      the _entire_ cross validation set (Xval and yval).

    - If you are using your cost function (linearRegCostFunction) to compute
      the training and cross validation error, you should call the function with
      the lambda argument set to 0. Do note that you will still need to use
      lambda when running the training to obtain the theta parameters.

    Hint
    ----
    You can loop over the examples with the following:

           for i in range(1, m+1):
               # Compute train/cross validation errors using training examples
               # X[:i, :] and y[:i], storing the result in
               # error_train[i-1] and error_val[i-1]
               ....
    """
    # Number of training examples
    m = y.size

    # You need to return these values correctly
    error_train = np.zeros(m)
    error_val   = np.zeros(m)

    # ====================== YOUR CODE HERE ======================
    for _ in range(iterations):
        for i in range(1, m+1):
            # generate random indices and then pick both X and y with these indices
            indices = np.random.choice(X.shape[0], i, replace=False)
            X_i = X[indices, :]
            y_i = y[indices]
            indices = np.random.choice(Xval.shape[0], i, replace=False)
            Xval_i = Xval[indices, :]
            yval_i = yval[indices]

            theta = utils.trainLinearReg(linearRegCostFunction,  X_i, y_i, lambda_=lambda_)
            error, _ = linearRegCostFunction(X_i, y_i, theta, 0)
            error_train[i - 1] += error
            error, _ = linearRegCostFunction(Xval_i, yval_i, theta, 0)
            error_val[i - 1] += error

    error_train = error_train / iterations
    error_val = error_val / iterations

    # =============================================================
    return error_train, error_val
```

```python
lambda_ = 0.01

pyplot.figure()
error_train, error_val = learningCurveRandomize(X_poly, y, X_poly_val, yval, 50, lambda_)
pyplot.plot(np.arange(1, 1+m), error_train, np.arange(1, 1+m), error_val)

pyplot.title('Polynomial Regression Learning Curve (lambda = %f)' % lambda_)
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 100])
pyplot.legend(['Train', 'Cross Validation'])

print('Polynomial Regression (lambda = %f)\n' % lambda_)
print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

```python

```


This notebook contains a roadmap for learning theoretical basis of main ML modules and approaches, and some excercises for evaluating your self. Hope to enjoy and learn :)

Sources:
* [SPML course in Fall 2021](https://drive.google.com/drive/folders/1hPYtFidk0rUQ9dLuWZjTf7YfNEzvP8MS?usp=sharing)
* [AI-Med internship videos](https://drive.google.com/drive/folders/1bwxmL1xBK7WD8-GPpKQGzW4GJnamy65W?usp=sharing)
* [Andrew Ng's ML course](https://www.youtube.com/playlist?list=PLoROMvodv4rMiGQp3WXShtMGgzqpfVfbU)
* [Stanford Natural Language Processing with Deep Learning course | Winter 2021](https://www.youtube.com/playlist?list=PLoROMvodv4rOSH4v6133s9LFPRHjEmbmJ)

First of all, please see the starting four sessions of famous [Andrew Ng's ML course](https://www.youtube.com/playlist?list=PLoROMvodv4rMiGQp3WXShtMGgzqpfVfbU). Then, try to answer the following questions.

## Linear Regression

Consider the linear regression $\hat{y}=w^T x$ on $S=\{(x^{(i)}, y^{(i)})\}_{i=1}^m$ with loss function $J(w)=\sum_{i=1}^{m} (y^{(i)}-\hat{y}^{(i)})^2$.

1. Simplify $\underset{w}{argmin}$ $J(w)$ by setting derivative of $J(w)$ w.r.t. $w$ to 0.
2. When the formula of the previous part does not work? Simplify $\underset{w}{argmin}$ $J(w)+\lambda\lVert w \rVert^2$ and describe how this new formulat solves the problem.

Answer the following questions:

1. Prove that
$$\frac{d\sigma(a)}{da}=\sigma(a)(1-\sigma(a)).$$

2. In logistic regression, we have $p(C_1|x)=\sigma(w^T x)$. Compute the negative log-likelihood for dataset $\{(x^{(1)}, y^{(1)}), ..., (x^{(n)}, y^{(n)})\}$.

3. Show that by computing gradient of the previous part w.r.t. $w$, we have $\sum_{i=1}^{n}(y^{(i)}-\hat{y}^{(i)})x^{(i)}$. compare this with MSE regression gradients.

4. show that $$\log{\frac{p(C_1|x)}{p(C_0|x)}}=w_1^T x+w_1'$$. Generalize it to $k$ classes and see the Softmax formula.

5. (optional) if $$L=-\sum_i y_i\log{p_i}$$ where $p_i=p(C_i|x)$, show that $\nabla_O L(x)=y-p$, where $y$ is label one-hot vector for $x$, $p$ is the output of softmax where $p_i=p(C_i|x)$, and $$o_i=w_i^T x + w_i', 1\leq i\leq k.$$

(optional) In logistic regression for $K$ classes, the posterior probability is computed in the following way
$$\begin{equation}
  \left\{
  \begin{array}{@{}ll@{}}
    P(Y=k|X=x)=\frac{exp(w_k^T x)}{1+\sum_{l=1}^{K-1}exp(w_l^T x)}, & (k=1, ..., K-1) \\
    P(Y=K|X=x)=\frac{1}{1+\sum_{l=1}^{K-1}exp(w_l^T x)}
  \end{array}\right.
\end{equation}$$
For simplicity, consider $w_K=0$.

1. How many parameters should be estimated. What are them?

2. Simplify the following log-likelihood for $n$ training samples $\{(x_1, y_1), ..., (x_n, y_n)\}$
$$L(w_1, ..., w_{K-1})=\sum_{i=1}^{n}\ln{P(Y=y_i|X=x_i)}$$

3. Compute and simplify the gradient of $L$ w.r.t. each of $w_k$s.

4. Consider the following objective function. Compute the gradient of $f$ w.r.t. each of $w_k$s.

$$f(w_1, ..., w_{K-1})=L(w_1, ..., w_{K-1})-\frac{\lambda}{2}\sum_{l=1}^{K-1}\lVert w_l\rVert_2^2$$

## Backpropagation

The source for this topic is:
* SPML course in Fall 2021
    * session 3 from min 100
    * session 4
    * session 5 upto min 45

### Questions

Consider the following network:
$$z_1=W_1 x^{(i)}+b_1$$
$$a_1=ReLU(z_1)$$
$$z_2=W_1 x'^{(i)}+b_1$$
$$a_2=ReLU(z_2)$$
$$a=a_1-a_2$$
$$z_3=W_2 a+b_2$$
$$\hat{y}^{(i)}=\sigma(z_3)$$
$$L^{(i)}=y^{(i)}\log{\hat{y}}^{(i)}+(1-y^{(i)})\log{(1-\hat{y}^{(i)})}$$
$$J=-\frac{1}{m}\sum_{i=1}^{m}L^{(i)}$$
where inputs are $x^{(i)}\in\mathbb{R}^{d_x\times1}, x'^{(i)}\in\mathbb{R}^{d_x\times 1}$ and the output is $\hat{y}\in(0,1)$ (the label is $y^{(i)}\in\{0, 1\}$). also $a\in\mathbb{R}^{d_a\times 1}$. Compute the following:

1. $\frac{\partial J}{\partial z_3}$

2. $\frac{\partial z_3}{\partial a}$

3. $\frac{\partial a}{\partial z_1}, and \frac{\partial a}{\partial z_2}$

4. $\frac{\partial z_2}{\partial W_1}, and \frac{\partial z_1}{\partial W_1}$

5. $\frac{\partial J}{\partial W_1}$

6. Write down the formula for updating all weights based on gradient descent.

## CNNs

CNNs are maybe the most important module in image processing. They have some sort of inductive bias for extracting local features.

The source for this topic is:
* SPML course in Fall 2021
    * session 5 from min 45
    * session 6

### Questions

Answer the following questions:
1. Describe the *Sparsity of Connections* property of CNNs.
2. Describe the *Parameter Sharing* property of CNNs.
3. Consider an input of shape $63\times 63\times 16$. If *stride=2* and *padding=0*, compute the shape of the output if we have 32 $7\times 7$ kernels.
4. Name three advantages of using CNN over MLP.
5. Consider a CNN network, which is trained on ImageNet. Is the output probability of the network uniform overall classes if the input is a white picture? Why?

Complete the following table. padding and stride are equal to 1, unless explicitly states.

* CONVx-N is a N filter convolution layer with height and width equal to x.
* POOL-N is a MAX Pooling of the shape $N\times N$, with *Stride=$N$*, and *Padding=$0$*.
* FC-N is a fully-connected layer with N neurons.

| Layer | Output Shape | # of Parameters |
| :- | -: | :-: |
| Input | 128$\times$128$\times$3 | 0 |
| CONV-9-32 |  |  |
| POOL-2 |  |  |
| CONV-5-64 |  |  |
| POOL-2 |  |  |
| CONV-5-64 |  |  |
| POOL-2 |  |  |
| FC-3 |  |  |

What is the number of parameters for replacing the fourth layer (CONV-5-64) with a fully-connected layer? What's your conclusion?

If we use a GPU with 12GB RAM for running the following network, what is the maximum number of pictures we could have in a batch? (you should find the memory bottleneck)

Input: 256 x 256

\[64\] Conv 3 x 3, s=1, p=1

\[64\] Conv 3 x 3, s=1, p=1

Pool 2 x 2, s=2, p=0

\[128\] Conv 3 x 3, s=1, p=1

\[128\] Conv 3 x 3, s=1, p=1

Pool 2 x 2, s=2, p=0

\[256\] Conv 3 x 3, s=1, p=1

\[256\] Conv 3 x 3, s=1, p=1

Pool 2 x 2, s=2, p=0

\[512\] Conv 3 x 3, s=1, p=1

\[512\] Conv 3 x 3, s=1, p=1

Pool 2 x 2, s=2, p=0

\[512\] Conv 3 x 3, s=1, p=1

\[512\] Conv 3 x 3, s=1, p=1
Pool 2 x 2, s=2, p=0

Flatten

FC (4096)

FC (4096)

FC (2)

Determine the receptive field of the neuron $(i, j)$ in the last convolution layer.

## RNNs

* Stanford Natural Language Processing with Deep Learning course | Winter 2021
    * sessions 5, 6, and 7
* AI-Med internship videos
    * sessions 6 and 7

Build a neural network that gets two sequence of binary number and the output is a binary sequence that is a
sum of the two (the length of the sequence is not fixed). For example:

| Time | input 1 | input 2 | output |
| --- | --- | --- | --- |
| 1 | 0 | 1 | 1 |
| 2 | 1 | 1 | 0 |
| 3 | 1 | 1 | 1 |
| 4 | 0 | 0 | 1 |

At time 1 the network gets the least significant bit and at time 4 it gets
the most significant bit. Evaluate you NN on large sequences.

## Generative Models

Generative models are deep models that generate fake data (image, text, video, etc.). We want that this fake-generated data seem real to humans and even other intelligent systems.
The source for this topic is:
* AI-Med internship videos
    * session 8


 INF285 - Computación Científica

 ODE

  [S]cientific [C]omputing [T]eam

 Version: 1.18

## Some algorithms in this jupyter notebook have been obtained from the textbook "Lloyd N. Trefethen, Spectral Methods in MATLAB, SIAM, Philadelphia, 2000" and translated to Python.

## Table of Contents

* [Introduction](#intro)
* [Initial Value Problemsl](#IVP)
    * [Three solvers](#solver)
        * [Stability Regions](#stab)
        * [Convergence](#conver)
    * [High order, higher dimensions and dynamical systems](#high)
* [Boundary Value Problems](#BVP)
    * [Shooting Method](#MD)
    * [Finite Differences](#DD)
* [Acknowledgements](#acknowledgements)

```python
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import pyplot #
import numpy as np
from scipy.integrate import odeint
from pylab import * #
from numpy import linalg as LA
from matplotlib.legend_handler import HandlerLine2D
from scipy.linalg import toeplitz
from scipy.optimize import root
from ipywidgets import interact, RadioButtons, Checkbox
import sympy as sym
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
from mpl_toolkits.axes_grid1 import make_axes_locatable
sym.init_printing()

# Global parameter that controls the figure's size
L = 15

# Source: https://github.com/tclaudioe/Scientific-Computing/blob/master/SC5/02%20Differentiation%20matrices.ipynb
def plot_matrices_with_values(ax,M,FLAG_test=True):
    N=M.shape[0]
    cmap = plt.get_cmap('GnBu')
    im = ax.matshow(M, cmap=cmap)
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    plt.colorbar(im, cax=cax)
    if FLAG_test:
        for i in np.arange(0, N):
            for j in np.arange(0, N):
                ax.text(i, j, '{:.2f}'.format(M[i,j]), va='center', ha='center', color='r')
    return
```

# Introduction
[Back to TOC](#toc)

In this jupyter noteboon we will study several numerical methods for solving ordinary differential equations (ODEs).
In particular we will study Initial Value Problems (IVP) and Boundary Value Problems (BVP), their principal numerical methods of numerical resolution, and their stability and convergence.

# Initial Value Problems
[Back to TOC](#toc)

An IVP corresponds to an ordinary differential equation of the form:
$$ \dot{y}(t) = f(t,y(t)), \quad t\in]t_0,T]$$
subject to an initial condition,
$$ y(t_0) = y_0. $$

# Three solvers
[Back to TOC](#toc)

Here we list a few solvers, just notice that we are only hightliting their one-time-step version:

* Euler's Method:

\begin{align*}
    y_{i+1} &= y_i + h\,f(t_i,y_i)
\end{align*}

* Runge-Kutta of second order (RK2)

\begin{align*}
    k_{1} &= f(t_i,y_i) \\
    y_{i+1} &= y_i + h\,f\left(t_i + \dfrac{h}{2}, y_i + \dfrac{h}{2}\,k_1\right)
\end{align*}

* Runge-Kutta of 4th order (RK4)

\begin{align*}
    k_{1} &= h \, f(t_i,y_i) \\
    k_{2} &= h \, f(t_i + \dfrac{h}{2}, y_i + \dfrac{1}{2}k_1) \\
    k_{3} &= h \, f(t_i + \dfrac{h}{2}, y_i + \dfrac{1}{2}k_2) \\
    k_{4} &= h \, f(t_i + h, y + k_3) \\
    y_{i+1} &= y_i + \dfrac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)
\end{align*}

For all this methods $h$ is the time-step it will advance the solution.
The previous algorithms only advance one time step the solution, so it should be applied the corresponding time steps expected such that it goes from the initial time $t_0$ until the final time $T$.

```python
# Forward Euler Method
def eulerMethod_one_step(yi,ti,f,h):
    return yi+h*f(ti,yi)

# Runge-Kutta of Second order
def RK2_one_step(yi,ti,f,h):
    k1=f(ti,yi)
    return yi+h*f(ti+h/2.0,yi+h/2.0*k1)

# Runge-Kutta
def RK4_one_step(yi,ti,f,h):
    k1=f(ti,yi)
    k2=f(ti+h/2.0,yi+(h/2.0)*k1)
    k3=f(ti+h/2.0,yi+(h/2.0)*k2)
    k4=f(ti+h,yi+h*k3)
    return yi+(h/6.0)*(k1+2.0*k2+2.0*k3+k4)
```

Here we define the several steps solvers.
These are the version that will be used later on in the code.

```python
# This implemantions have been upgraded respect to the ones included in the classnotes to handle
def eulerMethod(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        y[i+1] = y[i]+f(t[i],y[i])*h
    return t, y
# Challenge: Make backwardEulerMethod work!
'''def backwardEulerMethod(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    y = np.zeros(N+1)
    y[0] = y0
    for i in np.arange(N):
        f_hat= lambda x:  x - y[i]-f(t[i+1],x)*h
        # You must select a solver, "find_root" is just a generic name. The input used are the anonymous function "f_hat" and the initial guess y_i.
        x = find_root(f_hat,y[i])
        y[i+1] = x
    return t, y'''
def RK2(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        k1 = f(t[i],y[i])
        y[i+1] = y[i]+f(t[i]+h/2,y[i]+k1*h/2)*h
    return t, y
def RK4(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        k1 = f(t[i],y[i])
        k2 = f(t[i]+h/2,y[i]+k1*h/2)
        k3 = f(t[i]+h/2,y[i]+k2*h/2)
        k4 = f(t[i]+h,y[i]+k3*h)
        y[i+1] = y[i]+(k1+2*k2+2*k3+k4)*h/6
    return t, y
```

### 1D Example

In this example we will solve the logistic equation $\dot{y}(t) = y(t)(1-y(t))$ for several inital conditions $y_0$.

```python
# Logistic Equation
def f(t,y):
    return np.array(y*(1-y))

# First Example
def change_N_and_y0_1st_Example(N=20,y0=0.1, ODEsolver=eulerMethod):
    t0 = 0
    T  = 4
    t, y = ODEsolver(t0,T,N,y0,f)
    fig = plt.figure()
    ax  = fig.gca()
    ax.axis([0, 4, -1, 2])
    # Plotting stationary states for this problem.
    t = np.linspace(t0,T,N+1)
    plt.plot(t,t*0+1,'b-',lw=5, alpha=0.5, label='Stationary state y=1')
    plt.plot(t,t*0,'r-',lw=5, alpha=0.5, label='Stationary state y=0')
    # Plotting numerical solution.
    plt.plot(t,y,'.',label=r'$y_i$')

    plt.grid(True)
    ax.legend(loc='lower left', ncol=1, fancybox=True, shadow=True, numpoints=1, bbox_to_anchor=(1,0))
    plt.show()

radio_button_ODEsolvers=RadioButtons(
    options=[('Euler\'s Method',eulerMethod),('RK2',RK2),('RK4',RK4)],
    value=eulerMethod,
    description='ODE solver:',
    disabled=False
)

interact(change_N_and_y0_1st_Example,N=(3,100,1),y0=(-0.5,2,0.1), ODEsolver=radio_button_ODEsolvers)
```

    interactive(children=(IntSlider(value=20, description='N', min=3), FloatSlider(value=0.1, description='y0', ma…

    <function __main__.change_N_and_y0_1st_Example(N=20, y0=0.1, ODEsolver=<function eulerMethod at 0x7f8f0988a0d0>)>

Now, solve several initial conditions at the same time.

```python
def plot_log_eq(N=30,ODEsolver=eulerMethod):
    t0 = 0
    T = 4
    fig = plt.figure(figsize=(L/2,L/2))
    fig.clf()
    ax = fig.gca()
    ax.axis("equal")
    ax.grid(True)
    ax.set_title("Numerical Approximations and Direction Field")
    ax.axis([0, 4, -2, 2])

    # Plotting the stationary states
    t = np.linspace(t0,T,N+1)
    plt.plot(t,t*0+1,'b-',lw=5, alpha=0.5, label='Stationary state y=1')
    plt.plot(t,t*0,'r-',lw=5, alpha=0.5, label='Stationary state y=0')
    plt.legend(loc='best')

    # Solving over time for each method.
    for y0 in np.linspace(-0.5,5,40):
        t_times, y_output = ODEsolver(t0,T,N,y0,f)

        if y0>=1:
            plt.plot(t_times,y_output,'k-',alpha=0.2)
        else:
            plt.plot(t_times,y_output,'k-',alpha=0.8)
        plt.plot(t_times,y_output,'.')

    X,Y = np.meshgrid(np.arange(0,4,.2), np.arange(-2,2,.2))
    theta = np.arctan(f(0,Y))
    U = np.cos(theta)
    V = np.sin(theta)
    plt.quiver(X,Y,U,V,alpha=0.5)

interact(plot_log_eq,N=(5,100,1),ODEsolver=radio_button_ODEsolvers)
```

## Stability Regions
[Back to TOC](#toc)

To perform a linear stability analysis, we will consider the following ODE $\dot{y} = \lambda y$, with $y(0)=1$ and $\lambda\in\mathbb{C}$, for which we know its exact solution $y(t)=\exp(\lambda\,t)$.

Thus, solving the previos ODE with the Euler method, we have,
\begin{align*}
    y_{i+1} &= y_i + \lambda h y_i \\
    y_{i+1} &= (1 + \lambda h ) y_i \\
    y_{i+1} &= (1 + \lambda h )^{i+1} y_0
\end{align*}

Given that the real part of $\lambda$ is negative and $h$ is positive, we require then that $\left|1+\lambda h \right| <1$. We denote this range of values for $h$ the stability region for the Euler Method.

Following a similar approach, we the the following constraints for RK2 and RK4:
- RK2:
$$ \left|1+\lambda h + \dfrac{(\lambda h)^2}{2!} \right| <1 $$

- RK4:
$$ \left| 1+\lambda h + \dfrac{(\lambda h)^2}{2!} + \dfrac{(\lambda h)^3}{3!} + \dfrac{(\lambda h)^4}{4!} \right| <1 $$

```python
def zplot2(z, ax=plt.gca(), lw=1.5, line_color='k', label=''):
    ax.plot(np.real(z), np.imag(z), line_color, lw=lw, label=label)
def runge_kutta_stability_regions():
    z = np.exp(1j * np.pi * np.arange(201)/100.)
    r = z-1
    d = 1-1./z;
    # Order 1
    W1, W2, W4 = [0], [0], [0]
    for zi in z[1:]:
        W1.append( W1[-1]-(1.+W1[-1]-zi) )
    for zi in z[1:]:
        W2.append( W2[-1]-(1+W2[-1]+.5*W2[-1]**2-zi**2)/(1+W2[-1]) )
    for zi in z[1:]:
        num = (1+W4[-1]+.5*W4[-1]**2+W4[-1]**3/6+W4[-1]**4/24-zi**4)
        den = (1+W4[-1]+W4[-1]**2/2+W4[-1]**3/6.)
        W4.append( W4[-1] - num/den )
    return W1, W2, W4
W1,W2,W4=runge_kutta_stability_regions()
```

```python
fig = plt.figure(figsize=(L/2,L/2))
ax=fig.gca()
zplot2(W1,ax,line_color='r',label='Euler')
zplot2(W2,ax,line_color='g',label='RK2')
zplot2(W4,ax,line_color='b',label='RK4')
ax.axis("equal")
ax.axis([-5, 2, -3.5, 3.5])
ax.grid("on")
ax.legend(loc='best')
ax.set_title("Stability Regions of some Runge-Kutta ODE solvers")
```

### So, how do we use the stability region for each method?

The key point here is that we can't define $h$ a priori since we may not know the corresponding $\lambda$, thus, in principle, we can estimate $\lambda$ or just try the solver for different $h$.

```python
def plot_log_eq_stability_region(y0=1.2, N=20, T=10, ODEsolver=eulerMethod):

    # How did w get this? Why do I need it?
    def f_1D_prime(y):
        return np.array(1-2*y)

    t0=0
    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    fig = plt.figure(figsize=(L,L/3))
    ax1 = fig.add_subplot(131)
    ax2 = fig.add_subplot(132)
    ax3 = fig.add_subplot(133)

    ax1.axis('equal')
    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.axis([0, T, -0.5, 2])
    ax1.plot(t_times,y_output,'k-')
    ax1.plot(t_times,y_output,'b.')

    # The next code is to plot in logscale unstable numerical simulations
    h = (T-t0)/N
    y_all=h*f_1D_prime(y_output)
    y_pos=y_all>0
    y_neg=np.logical_not(y_pos)
    ax3.semilogy(t_times[y_pos],y_all[y_pos],'.r',ms=20)
    ax3.semilogy(t_times[y_neg],-y_all[y_neg],'.b',ms=20)
    ax3.grid(True)
    ax3.set_title(r'$\log10(|y_i|)$')

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    k_lambdah=h*f_1D_prime(y_output)
    ax2.plot(np.real(k_lambdah),np.imag(k_lambdah),'.r',ms=20)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid(True)
    ax2.set_title('Stability Region and $k=\lambda\,h$')

interact(plot_log_eq_stability_region,y0=(-2,5,0.1),N=(2,100,1), T=(1,20,1), ODEsolver=radio_button_ODEsolvers)
```

## Convergence
[Back to TOC](#toc)

This is very important! Let's study it!

To talk about the convergence, we need to have in mind the order of each method.
In particular, we will consider that $y(T)$ denotes the exact solution at the final time $T$ that need to solve the ODE, and $y_N$ be the approximation of the solution at the final time $T=t_N$.
In simple words, we can say the following:

- Euler's method is a first order method since $y(T)=y_N+\mathcal{O}(h)$. This means that if $h$ is reduce by half, then the error is reduced by half.
- RK2 is a second order method since $y(T)=y_N+\mathcal{O}(h^2)$. This means that if $h$ is reduce by half, then the error is reduced by 4.
- RK4 is a forth order method since $y(T)=y_N+\mathcal{O}(h^4)$. This means that if $h$ is reduce by half, then the error is reduced by 16.

To validate this, we will solve numerically the following problem,
\begin{align*}
    \dot{y}(t)&=-3\,y(t)+6\,t+5, \quad t\in]0,0.5]\\
    y(0)&=3,
\end{align*}
for which we know the solution $y(t)=2\exp(-3t)+2t+1$.

```python
t0 = 0
T  = 0.5
y0 = 3.0
def f(t,y):
    return np.array(-3*y+6*t+5)
y_sol = lambda t: 2.*np.exp(-3.*t)+2.*t+1
```

```python
fig = plt.figure(figsize=(L/2,L/2))
fig.clf()
ax = fig.gca()
ax.axis('equal')
ax.grid(True)
ax.set_title('Convergence Analysis')

N_list = np.logspace(1,5,5,dtype=int)
for N in N_list:

    h = (T-t0)/N
    t_times, y_output = eulerMethod(t0,T,N,y0,f)
    plt.loglog(h,abs(y_output[-1]-y_sol(t_times[-1])),'b.',ms=20,label='Euler',alpha=.5)

    t_times, y_output = RK2(t0,T,N,y0,f)
    plt.loglog(h,abs(y_output[-1]-y_sol(t_times[-1])),'rs',ms=20,label='RK2',alpha=.5)

    t_times, y_output = RK4(t0,T,N,y0,f)
    plt.loglog(h,abs(y_output[-1]-y_sol(t_times[-1])),'gd',ms=20,label='RK4',alpha=.5)
    if N==N_list[0]:
        plt.legend(numpoints=1, loc='lower right')

h_list=T/N_list

plt.loglog(h_list,h_list,'k-')
plt.text(10**(-6),10**(-3),r'$\mathcal{O}(h)$', fontsize=25, horizontalalignment='center', color='b')
plt.loglog(h_list,np.power(h_list,2.),'k-')
plt.text(10**(-7),10**(-8),r'$\mathcal{O}(h^2)$', fontsize=25, horizontalalignment='center', color='r')
plt.loglog(h_list[:-2],np.power(h_list[:-2],4.),'k-')
plt.text(10**(0),10**(-11),r'$\mathcal{O}(h^4)$', fontsize=25, horizontalalignment='center', color='g')
plt.xlabel(r'$h$')
plt.ylabel(r'Error$=|y(T)-y_{N}|$')
```

# High order, higher dimensions and dynamical systems
[Back to TOC](#toc)

Why do we study higher order problems?

Because they are so cool!

### The Van der Pol oscillator

The equation that model de Van der Pol oscillator is the following:
$$\ddot{y}-\mu\,(1-y^2)\,\dot{y} + y = 0,$$
with $y(0)=2$ and $\dot{y}(0)=0$,
where $\mu$ is a scalar parameter that regulates the non-linearity of the problem and the strength of the damping.
See https://en.wikipedia.org/wiki/Van_der_Pol_oscillator.

So, since it has second derivatives respecto to time, we need to translate it to a dynamical system with the following change of variables:
\begin{align*}
    y_1(t) &= y(t),\\
    y_2(t) &= \dot{y}(t).
\end{align*}
Thus,
\begin{align*}
    \dot{y}_1 &= \dot{y} = y_2,\\
    \dot{y}_2 &= \ddot{y} = \mu (1-y^2) \dot{y} - y = \mu (1-y_1^2) y_2 - y_1,
\end{align*}
with $y_1(0)=2$ and $y_2(0)=0$.

```python
def f_vdp(t,y,mu=10):
    y1 = y[0]
    y2 = y[1]
    return np.array([y2, mu*(1-y1**2)*y2-y1])

def f_vdp_eig_jacobian(t,y,mu=10):
    J=[[0,1],[-(2*mu*y[0]*y[1]+1), mu*(1-y[0]**2)]]
    lambs,vs= LA.eig(J)
    return lambs
```

```python
def plotting_vdp(y01=2, y02=0, N=20000, T=200, mu=10, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])

    f= lambda t,y: f_vdp(t,y,mu)

    t0=0

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    fig = plt.figure(figsize=(L,L/2))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    ax1.grid(True)
    ax1.set_title("Numerical Approximation")
    ax1.plot(t_times,y_output[:,0],'-', label=r'$y_1(t)$', alpha=0.5)
    ax1.set(xlabel='t')
    ax1.plot(t_times,y_output[:,1],'-r', label=r'$y_2(t)$', alpha=0.5)
    ax1.legend(loc='best')

    ax2.grid(True)
    ax2.set_title("Phase Portrait")
    ax2.plot(y_output[:,0],y_output[:,1],'-')
    ax2.set(xlabel='$y=y_1$', ylabel='$\dot{y}=y_2$')

interact(plotting_vdp,y01=(-3,3,0.1), y02=(-3,3,0.1),
         N=(10,10000,10),T=(10,200,10),
         mu=(1,50,0.1),ODEsolver=radio_button_ODEsolvers)
```

```python
def plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_jacobian,h,c1,c2,c3):
    fig = plt.figure(figsize=(L,L/3))
    ax1 = fig.add_subplot(131)
    ax2 = fig.add_subplot(132)
    ax3 = fig.add_subplot(133)

    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.plot(t_times,y_output[:,0],'-')
    ax1.plot(t_times,y_output[:,1],'r-')

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    for i in range(1,t_times.size):
        k_lambdah=h*f_jacobian(t_times[i],y_output[i,:])
        ax2.plot(np.real(k_lambdah[0]),np.imag(k_lambdah[0]),'.r',ms=10,alpha=.4)
        ax2.plot(np.real(k_lambdah[1]),np.imag(k_lambdah[1]),'sm',ms=10,alpha=.4)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid('on')

    ax3.grid(True)
    ax3.set_title('Phase Portrait')
    ax3.plot(y_output[:,0],y_output[:,1],'-')
    ax3.set(xlabel='$y_1$', ylabel='$y_2$')
```

### What about the stability region for dynamical systems?

In this case we need to compute the eigenvalues of the Jacobian matrix of $\mathbf{F}(\mathbf{y})$.
So the eigenvalues $\lambda_i$ multiplied by $h$ have to be inside of the stability region of the real part of $\lambda_i$ is negative, for $i\in\{1,2,\dots,\text{Dimension of Dynamical System}\}$.

```python
def plotting_vpd_as_system(y01=2, y02=0, N=1000, T=40, mu=10, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])

    f= lambda t,y: f_vdp(t,y,mu)

    t0 = 0

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_vdp_eig_jacobian,h,c1,c2,c3)

interact(plotting_vpd_as_system, y01=(-3,3,0.1), y02=(-3,-3,0.1),
         N=(10,4000,10),T=(10,200,10),
         mu=(1,50,0.1),ODEsolver=radio_button_ODEsolvers)
```

### Lotka-Volterra model

This instance models the evolution of the average population of two species living in a closed system.
The species number 1 is considered that feeds itself with an unlimited amount of food and the species number 2 feeds itself with species number 1.
We will denote the average population of species number 1 as $y_1(t)$ and the average population of species number 2 as $y_2(t)$.

So, if we consider that there is no presence of species number 2 in the system, then species number 1 will grow exponentially, this means $\dot{y}_1=y_1$.
On the other case, if no presence of species number 1 in the system, then species number 2 will decay exponentially as follows $\dot{y}_2=-y_2$.
However, we are not interested in the extreme cases, how can we understand the dynamics when there exists these two species together interacting?
This is modeled by the Lotka-Volterra dynamical system:
\begin{align*}
    \dot{y}_1(t)&=(1-y_2(t)/\mu_2)\,y_1(t),\\
    \dot{y}_2(t)&=-(1-y_1(t)/\mu_1)\,y_2(t),\\
    y_1(0)&=400,\\
    y_2(0)&=100,
\end{align*}
where $\mu_1$ and $\mu_2$ are normalization constants.

```python
mu1=300
mu2=200
def f_predprey(t,y,mu1=mu1,mu2=mu2):
    y1 = y[0]
    y2 = y[1]
    return np.array([(1-y2/mu2)*y1, -(1-y1/mu1)*y2])

def f_predprey_jacobian(t,y,mu1=mu1,mu2=mu2):
    J=[[1-y[1]/mu2,-y[0]/mu2],[y[1]/mu1, -(1-y[0]/mu1)]]
    lambs,vs= LA.eig(J)
    return lambs
```

```python
def plotting_predprey(y01=70, y02=100, N=1000, T=30, ODEsolver=eulerMethod):
    y0=np.array([y01, y02])
    f= lambda t,y: f_predprey(t,y)
    t0 = 0
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_predprey_jacobian,h,c1,c2,c3)

interact(plotting_predprey,y01=(0,1000,10),y02=(0,1000,10),
         N=(10,1000,10),T=(10,200,10),
         ODEsolver=radio_button_ODEsolvers)
```

## Now solving the following equation
\begin{align*}
    \dot{y}_1(t) &= y_2(t),\\
    \dot{y}_2(t) &= -y_1(t),
\end{align*}
with $y_1(0)=1$ and $y_2(0)=0$.

What functions are these? Do you recognize them?

```python
def f_trig(t,y):
    y1 = y[0]
    y2 = y[1]
    return np.array([y2, -y1])

def f_trig_jacobian(t,y):
    return np.array([0.+1.j, 0.-1.j])

def plotting_f_trig(y01=1, y02=0, N=100, T=2*np.pi, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])
    f= lambda t,y: f_trig(t,y)
    t0 = 0
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'
    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_trig_jacobian,h,c1,c2,c3)

interact(plotting_f_trig,y01=(-2,2,0.1),y02=(-2,2,0.1),
         N=(10,1000,10),T=(1,10*np.pi,0.1),
         ODEsolver=radio_button_ODEsolvers)
```

## Last but not least, solving:
\begin{align*}
    \dot{y}(t) &= \lambda\,y(t),\\
\end{align*}
with $y(0)=1$.

What function is this? Do you recognize it?

```python
def f_exp(t,y,lam=-1):
    return lam*y
def f_exp_jac(t,y,lam=-1):
    return lam

def plotting_f_exp(y0=1, N=100, T=10, lam=-1, ODEsolver=eulerMethod):

    f= lambda t,y: f_exp(t,y, lam)
    t0 = 0
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    # Plotting

    fig = plt.figure(figsize=(L*2/3,L/3))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.plot(t_times,y_output[:],'.b')
    ax1.plot(t_times,y_output[:],'k-',alpha=0.5)
    #ax1.axis([0, T, -2, 2])

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    for i in range(1,t_times.size):
        k_lambdah=h*f_exp_jac(t_times[i],y_output[i],lam)
        ax2.plot(np.real(k_lambdah),np.imag(k_lambdah),'.r',ms=10,alpha=.4)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid('on')

interact(plotting_f_exp,y0=(-2,2,0.1),
         N=(5,1000,10),T=(1,20,1), lam=(-2,2,0.1),
         ODEsolver=radio_button_ODEsolvers)
```

# Boundary Value Problems
[Back to TOC](#toc)

An instance of a BVP corresponds to a ordinary differential equation of the following form:
$$ y''(x) = f(x,y(x),y'(x)),$$
with the following boundary conditions,
\begin{align*}
    y(a) &= y_a,\\
    y(b) &= y_b.
\end{align*}

To undertand them better, let's find a numerical approximation of the following problem:
\begin{align*}
    y''(x) &= 4\,y(x),\\
    y(0)   &= y_0,\\
    y(1)   &= y_n.
\end{align*}

## Shooting Method
[Back to TOC](#toc)

This method takes advantage that we already know how to handle IVP and uses that knowledge to find a numerical approximation of an BVP.
The main steps are the followings:
- We have a BVP on the form:
    \begin{align*}
        y''(x) &= f(x,y(x),y'(x)),\\
        y(a)   &= y_a,\\
        y(b)   &= y_b.
    \end{align*}
- Let's considerer the following steps to translate the BVP to an IVP:
    1. $x \rightarrow t$, this implies that $t_0=a$ and $T=b$.
    2. \begin{align*}
            \ddot{y}(t)   &= f(t,y(t),\dot{y}(t)),\\
            y(t_0)        &= y_a,\\
            \dot{y}(t_0)  &= \alpha.
       \end{align*}
       **Warning**: We actually don't know $\alpha$ but we do need it since we treating the previous problem as a IVP. What we do have is $y(b)=y(T)=y_b$.
    3. So, since we don't have $\alpha$, we need to find it by solving the following 1D root: $F(\alpha) = y_N^{(\alpha)} - y_b$, where $y_N^{(\alpha)}$ denotes the IVP approximation at time $t=T$ considering $\alpha$ for the initial condition of $\dot{y}(t_0)$.
    4. After we find the root for $F(\alpha)$, we can say we have found a numerical approximation of $y(x)$ since we have a numerical approximation that satisfies the ODE and also the boundary condition defined on the interval $[a,b]$. This is wonderful!

Notice however that the function $F(\alpha)$ is the output of the execution of an algorithm, so we only expect it to be continuous.
For instance it will be interesting to apply the Bisection method here!
The main disadvantage of this algorithm is that we need to solve several IVPs to actually find the right one, the good news is that we are re-using previous knowledge in a new type of problems!

```python
# BVP: y''(x)= 4*y(x), y(0)=1, and y(1)=3, x \in [0,1].
# Dynamical system:
# y1=y  -> y1'=y'=y2
# y2=y' -> y2'=y''=4*y=4*y1.
# y1(0) = 1, (and we know y(1)=3)
# y1(1) = 3.
# y2(0) = alpha, we don't know it.
def f(t,y):
    return np.array([y[1],4*y[0]])
```

```python
def shooting_method_101(alpha=-1, N=50, ya=1, yb=3, ODEsolver=eulerMethod):
    t0 = 0
    T  = 1
    y0 = np.array([ya,alpha])
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    fig = plt.figure(figsize=(L/2,L/2))
    ax = fig.gca()
    plt.grid(True)
    plt.title("Numerical Approximation")
    plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'$y1_i$')
    plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'$y2_i$')

    plt.plot(0,1,'*k',ms=16,label='Left BC')
    plt.plot(1,3,'*g',ms=16,label='Right BC')
    plt.plot(0,alpha,'darkorange',ms=16,label=r'$\alpha$', marker=r'$\alpha$')

    plt.legend(loc='best')

interact(shooting_method_101, alpha=(-2,2,0.1),
         N=(5,100,1), ya=(-5,5,0.1), yb=(-5,5,0.1),
         ODEsolver=radio_button_ODEsolvers)
```

### Now, let's find $\alpha$ automatically

```python
y0 = np.array([1, 1.1])
```

```python
def F(alpha, ya=1, yb=3, N=100, ODEsolver=eulerMethod):
    t0 = 0
    T  = 1
    y0 = np.zeros(2)
    y0[0] = ya
    y0[1] = alpha
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    # Notice that we don use absolute value for the error since
    # we want to look at this problem as a rootfinding problem
    # and not as a minimization, that could be another approach.
    return y_output[-1,0]-yb
```

```python
t0 = 0
T  = 1
ya = 1
yb = 3
N  = 100
ODEsolver = RK4

F_root = lambda alpha: F(alpha, ya=ya, yb=yb, N=N, ODEsolver=ODEsolver)
alpha = root(F_root, 0.).x[0]

y0 = np.array([ya,alpha])
t_times, y_output = ODEsolver(t0,T,N,y0,f)

fig = plt.figure(figsize=(L/2,L/2))
plt.grid(True)
plt.title("Numerical Approximation")
plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'$y1_i$')
plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'$y2_i$')
plt.plot(0,ya,'*k',ms=16,label='Left BC')
plt.plot(1,yb,'*g',ms=16,label='Right BC')
plt.plot(0,alpha,'darkorange',ms=16,label=r'$\alpha$', marker=r'$\alpha$')
plt.legend(loc='best')
print(alpha)
```

### Let's take a look to $F(\alpha, \dots)$, find the 'BUG'!

```python
ODEsolver = eulerMethod

several_alphas = np.linspace(-1,0,100)
Fv = np.vectorize(lambda x: F(x,ODEsolver=ODEsolver))
F_alphas = Fv(several_alphas)

plt.figure(figsize=(10,10))
plt.plot(several_alphas,F_alphas,'.b',label=r'$F(\alpha)$')
plt.plot(several_alphas,F_alphas*0,'-g',alpha=0.8,linewidth=3,label=r'$0$')
plt.grid(True)
plt.plot(alpha,F(alpha,ODEsolver=ODEsolver),'.r',ms=20,label=r'Root: $\alpha$='+str(alpha))
plt.xlabel(r'$\alpha$')
plt.ylabel(r'$F(\alpha)=y_N^{(\alpha)} - y_b$')
plt.legend(loc='best')
plt.show()
```

## Finite Differences
[Back to TOC](#toc)

This method opens a new door of algorithms for solving BVP.
The main idea is to approximate each derivative of $y(x)$ by a finite difference approximation and then solve the linear or possible non-linear system of equations associated.

Here we list a few finite difference approximation for the first and one for the second derivative:

- Forward Difference:
$$ y'(x) = \dfrac{y(x+h) - y(x)}{h} + O(h)$$

- Backward Difference:
$$ y'(x) = \dfrac{y(x) - y(x-h)}{h} + O(h)$$

- Central Difference:
$$ y'(x) = \dfrac{y(x+h) - y(x-h)}{2h} + O(h^2)$$

And for the second derivative:
\begin{align*}
y''(x) &= \dfrac{\text{Forward Difference } - \text{ Backward Difference}}{h} + O(h^2) \\
y''(x) &= \dfrac{\dfrac{y(x+h) - y(x)}{h} - \dfrac{y(x) - y(x-h)}{h}}{h} + O(h^2) \\
y''(x) &= \dfrac{y(x+h) - 2\,y(x) + y(x-h)}{h^2} + O(h^2) \\
\end{align*}

Notice that in all the previous cases we first need to define the function on a grid, it does not have to be regular, but for simplicity we will consider it regular.
Recall that the BVP will be defined on $[a,b]$, thus the regular grid will be defined as follows: $x_i=i\,h$, where $h=\dfrac{b-a}{N}$.
This also help us to define the approximation of the function over the grid, this means the following,
- Traditionally we say $y(x)$ is the function defined on $x\in[a,b]$.
- Now, we will have a grid function $y_i\approx y(x_i)$ for $i\in\{0,1,2,\dots,N\}$. In particular we would like to point out that the vector $\mathbf{y}$ will be the representation of all the scalar values $y_i$ and the vector $\mathbf{x}$ will store the grid values $x_i$. Thus in this discrete world $\mathbf{y}$ will apporximate the continuous function $y(x)$ on $[a,b]$.

On the other hand, the finite difference operator over the finite grid will be then defined as follows:
- Forward Difference: $ y'(x_i)=\dfrac{y_{i+1} - y_i}{h} + O(h) $.
- Backward Difference: $ y'(x_i) = \dfrac{y_i - y_{i-1}}{h} + O(h) $.
- Central Difference: $ y'(x_i) = \dfrac{y_{i+1} - y_{i-1}}{2h} + O(h^2)$.
- Second derivative finite difference: $y''(x_i) = \dfrac{y_{i+1} - 2\,y_i + y_{i-1}}{h^2} + O(h^2)$.

Now, let's apply them!

```python
def solve_finite_difference_eq(M=4,ya=1,yb=3):
    # Spatial discretization,
    # 'M' represents the number of intervals to be used.
    h=(1.-0.)/M

    # Building Finite Difference Discretization
    deltas=-(2.+4.*(h**2.))
    A=toeplitz(np.append(np.array([deltas, 1.]), np.zeros(M-3)))
    # Building RHS
    b=np.append(-ya, np.zeros(M-3))
    b=np.append(b,-yb)
    # Solving the linear system of equations
    y=np.linalg.solve(A, b)
    # Adding back the boundary conditions into the solution
    y=np.append([ya], y)
    y=np.append(y,[yb])
    x_FD = np.linspace(0,1,M+1) # xi, i.e. the spatial grid.
    y_FD = y                    # yi, i.e. the approximation of y(x_i).
    return x_FD, y_FD, A, b

def plot_solution_finite_difference_eq(M=4,ya=1,yb=3, FLAG_test=True):

    # Solving with Finite Difference
    x_FD, y_FD, A, b = solve_finite_difference_eq(M,ya,yb)

    # Plotting
    fig = plt.figure(figsize=(L,L/2))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    # Plotting the discrete solution
    ax1.grid(True)
    ax1.set_title("FD")
    ax1.plot(x_FD,y_FD,'.',ms=20)
    ax1.grid("on")

    # Plotting the pattern and coefficients of the tri-diagonal matrix
    plot_matrices_with_values(ax2,A,FLAG_test)

interact(plot_solution_finite_difference_eq, M=(3,25,1),
         ya=(-5,5,0.1), yb=(-5,5,0.1),
         FLAG_test=Checkbox(value=True, description='Show numerical values'))
```

## Comparing the Shooting Method with Finite Differences

```python
def shooting_method_vs_FD(alpha=-0.420306048726636, N=50, ya=1, yb=3, ODEsolver=eulerMethod, M=4):
    # Shooting Method
    t0 = 0
    T  = 1
    y0 = np.array([ya,alpha])
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    # Finite Differences
    x_FD, y_FD, A, b = solve_finite_difference_eq(M,ya,yb)

    fig = plt.figure(figsize=(L/2,L/2))
    ax = fig.gca()
    plt.grid(True)
    plt.title("Numerical Approximations")
    plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'SM: $y1_i$')
    plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'SM: $y2_i$')

    plt.plot(x_FD,y_FD,'b.',ms=20,label=r'FD: $y_i$')

    plt.plot(0,1,'*k',ms=16,label='Left BC')
    plt.plot(1,3,'*g',ms=16,label='Right BC')
    plt.plot(0,alpha,'r',ms=16,label=r'$\alpha$', marker=r'$\alpha$')

    plt.legend(loc='best')

interact(shooting_method_vs_FD, alpha=(-2,2,0.01),
         N=(5,100,1), ya=(-5,5,0.1), yb=(-5,5,0.1),
         ODEsolver=radio_button_ODEsolvers,
         M=(3,25,1))
```

In summary,
- Both methods find a reasonable approximation.
- The Shooting Method requires to solve several IVP until it find the missing slope $\alpha$, but it uses previous knowledge.
- Finite Different requieres to solve a linear (or possible non-linear) system of equations but we requires few points to get a reasonable approximation.
- Therefore, depending on the numerical problem that needs to be solve, one or the other would be better choice!

# Acknowledgements
[Back to TOC](#toc)

-  _Material creado por profesor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _y ayudantes: Alvaro Salinas y Martín
Villanueva. DI UTFSM. Abril 2016._
- **DISCLAIMER**: El presente notebook ha sido creado para el curso **ILI286 - Computación Científica 2**, del [Departamento de Informática](http://www.inf.utfsm.cl/), [Universidad Técnica Federico Santa María](http://www.utfsm.cl/). El material ha sido creado por Claudio Torres <ctorres@inf.utfsm.cl> y Sebastian Flores <sebastian.flores@usm.cl>, y es distribuido sin restricciones. En caso de encontrar un error, por favor no dude en contactarnos.
- [Update 2015] Se ha actualizado los notebooks a Python 3 e includio el "magic" "%matplotlib inline" antes de cargar matplotlib para que los gráficos se generen en el notebook.
- [Update 2016] (Álvaro) Modificaciones mayores al formato original. Agregado contexto: Introducción, Tabla de Contenidos, Explicaciones de cada método.
- [Update 2019] (C. Torres) Small changes. Fixing issue with title of sections and identation. Adding 'interact' to the logistic equation! Adding interact to everything, work in progress. All done, Enjoy!
- _Update July 2021 - v1.16 - C.Torres_ : Updating format and translating to English. Several updates in several functions! Major update!
- _Update July 2021 - v1.17 - C.Torres_ : Minor update, removing commented code.
- _Update July 2021 - v1.18 - C.Torres_ : Removing warning and changing the name of some variables.

```python

```