###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c) 2019 Daniel Koehn, based on (c)2018 L.A. Barba, G.F. Forsyth [CFD Python](https://github.com/barbagroup/CFDPython#cfd-python), (c)2014 L.A. Barba, I. Hawke, B. Knaepen [Practical Numerical Methods with Python](https://github.com/numerical-mooc/numerical-mooc#practical-numerical-methods-with-python), also under CC-BY.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# 2D Time-Dependent Thermal Convection in the Earth Mantle

Finally, we can assemble our 2D time-dependent thermal convection code and compute the evolution of the temperature field for different initial conditions.

## Thermal Convection in the Earth Mantle

Before writing the convection code, lets shortly summarize the governing equations from the [introduction notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/08_Convection_2D/01_2D_Mantle_Convection_intro.ipynb).

The full coupled set of scaled PDE's describe the coupling of incompressible 2D Navier Stokes flow with an advection-diffusion equation for temperature, and can be written in dimensionless form as,

$$
   \frac{1}{\mathrm{Pr}}\left[ \frac{\partial \omega}{\partial t} + \vec{v}\cdot\nabla\omega\right] = \nabla^2\omega + \mathrm{Ra}\frac{\partial T}{\partial x}
$$
$$
    \nabla^2 \psi = -\omega
$$
$$
    \frac{\partial T}{\partial t} + \vec{v}\cdot\nabla T = \nabla^2 T
$$

where

$$
    \vec{v}=(u,0,w) = \nabla\times\psi\hat{j}=(-\frac{\partial\psi}{\partial z}, 0,  \frac{\partial\psi}{\partial x})
$$

is the fluid velocity field (which in this form is exactly incompressible with $\nabla\cdot\vec{v}=0$).  $\psi$ is the "Streamfunction"  whose contours are tangent to the fluid trajectories at all times.  The vorticity,

$$
    \omega = \bf{\vec{\omega} \cdot \hat{j}} = (\nabla\times\vec{v}) \cdot \hat{j}
$$

measures the local rate of rotation of the fluid, and is driven by horizontal variations in temperature (actually density). $\hat{j}=(0,1,0)^T$ denotes the unit vector in y-direction.

Now, if we are dealing with mantle convection, the Prandtl number Pr becomes very large, actually it can be assumed to be inifinite. Therefore, the governing equations simplify to

\begin{equation}
\begin{split}
\nabla^2\omega &=  - \mathrm{Ra}\frac{\partial T}{\partial x}\\
\nabla^2 \psi &= -\omega\\
\frac{\partial T}{\partial t} &= \nabla^2 T - \vec{v}\cdot\nabla T
\end{split}
\tag{1}
\end{equation}

The differential equations (1) consist of different subproblems. For a given temperature field $T$, we have to solve the coupled **Poisson equations**

\begin{equation}
\begin{split}
\nabla^2\omega &=  - \mathrm{Ra}\frac{\partial T}{\partial x}\\
\nabla^2 \psi &= -\omega
\end{split}
\tag{2}
\end{equation}

in order to estimate the streamfunction $\psi$ and vorticity $\omega$. We already discussed the solution of a similar problem using the steepest descent method when trying to find [stationary solutions for the 2D thermal mantle convection problem](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/07_BVP_2D/05_BVP_2D_Thermal_Mantle_Convection.ipynb).

Therefore, the remaining problem is the 2D heat advection-diffusion problem from the conservation of energy equation

\begin{equation}
\frac{\partial T}{\partial t} = \nabla^2 T - \vec{v}\cdot\nabla T \notag
\end{equation}

or by inserting the velocity vector $\vec{v}=(u,0,w) =(-\frac{\partial\psi}{\partial z}, 0,  \frac{\partial\psi}{\partial x})$ using the streamfunction

\begin{equation}
\frac{\partial T}{\partial t} = \nabla^2 T + \frac{\partial\psi}{\partial z} \frac{\partial T}{\partial x} - \frac{\partial\psi}{\partial x} \frac{\partial T}{\partial z} \tag{3}
\end{equation}

Using finite-differences, problem (3) can be further decomposed into two well known subproblems using **operator splitting**:

- The 2D linear advection problem:

\begin{equation}
\frac{\tilde{T}^{n+1}-T^{n}}{\Delta t} - \frac{\partial\psi}{\partial z} \frac{\partial T}{\partial x} + \frac{\partial\psi}{\partial x} \frac{\partial T}{\partial z} \tag{4}
\end{equation}

- The 2D heat conduction problem:

\begin{equation}
\frac{\partial \tilde{T}}{\partial t} = \nabla^2 \tilde{T} \tag{5}
\end{equation}

## How to solve the 2D time-dependent mantle convection problem

In summary, we can solve the 2D mantle convection problem using the following approach. Starting from an initial condition $T_0, \psi_0, \omega_0$, we compute at each time step ...

1. For a given temperature field $T$, solve the Poisson equations (2) to estimate the streamfunction $\psi$ and vorticity $\omega$, as described in the [2D mantle convection BVP notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/07_BVP_2D/05_BVP_2D_Thermal_Mantle_Convection.ipynb)

2. Solve the 2D linear advection-diffusion problem eqs. (4) + (5)

We solved the 2D advection-diffusion problem described in step 2. by operator splitting and a combination of Marker-in-Cell and FTCS FD method in [this notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/08_Convection_2D/03_2D_Linear_Advection_Diffusion.ipynb).

Time for some coding. As usual, we start by importing some basic Python libraries and set up the plotting environment.

```python
# Import libraries
import numpy
from matplotlib import pyplot
%matplotlib inline

# import JIT from Numba
from numba import jit

# import Numpy Linear Algebra package
from numpy import linalg as LA

# import SciPy Interpolation package
from scipy import interpolate
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

## 1. Solution of Poisson equations

We start with step 1: For a given temperature field $T$, solve the Poisson equations (2) to estimate the streamfunction $\psi$ and vorticity $\omega$, as described in the [2D mantle convection BVP notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/07_BVP_2D/05_BVP_2D_Thermal_Mantle_Convection.ipynb)

For the steepest descent method, we need a function to compute the action of the Laplace operator $\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial z^2}$ on an abritary field $u$. In `Alapl` the second derivatives are approximated by the three-point central operator:

```python
# Compute Laplace operator applied to scalar field u
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def Alapl(u,nx,nz,dx2,dz2):

    A = numpy.zeros_like(u)

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            # Compute Laplace operator for scalar field u
            d2udx2 = (u[j, i+1] + u[j, i-1] - 2 * u[j, i]) / dx2
            d2udz2 = (u[j+1, i] + u[j-1, i] - 2 * u[j, i]) / dz2

            A[j, i] = d2udx2 + d2udz2

    return A
```

Next, we have to define the RHS $b$ for the vorticity equation `bw` ...

```python
# Compute b for vorticity equation
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def bw(T,nx,nz,dx,Ra):

    b = numpy.zeros_like(T)

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            # Compute b for vorticity equation
            dTdx = - Ra * (T[j,i] - T[j,i-1]) / dx

            b[j, i] = dTdx

    return b
```

Next, we can solve eqs. (2) using the iterative numerical method described above with the steepest descent method in the function `psi_w_bvp` in order to compute the streamfunction $\psi(x,z,t)$ and vorticity $\omega(x,z,t)$ fields.

```python
# 2D thermal mantle convection code with JIT optimization
# -------------------------------------------------------
def psi_w_bvp(T, psi, w, niter, dx, dz, Ra, tol):

    """
    Computes and returns the stationary streamfunction and vorticity
    fields based on the current temperature field for the 2D thermal
    mantle convection problem using the steepest descent method.

    Parameters
    ----------
    T : numpy.ndarray
        The current temperature distribution as a 2D array of floats.
    psi : numpy.ndarray
        The current streamfunction as a 2D array of floats.
    w : numpy.ndarray
        The current vorticity as a 2D array of floats.
    niter : integer
        Maximum number of iterations using the steepest descent method.
    dx : float
        Grid spacing in the x direction.
    dz : float
        Grid spacing in the z direction.
    Ra : float
        Rayleigh number.
    tol : float
        tolerance for abortion criterion.

    Returns
    -------
    psi : numpy.ndarray
        The streamfunction as a 2D array of floats.
    w : numpy.ndarray
        The vorticity as a 2D array of floats.
    """

    # Define some constants.
    dx2 = dx**2
    dz2 = dz**2

    r = numpy.zeros_like(psi)    # initial residuals
    Ar = numpy.zeros_like(psi)   # store mat-vec multiplication Ar

    # Estimate number of grid points in x- and y-direction
    nz, nx = T.shape

    # Iterate solution using the steepest descent method
    for n in range(niter):

        # 1. Compute new vorticity from previous temperature
        # --------------------------------------------------
        # store old T
        wk = w.copy()

        # Compute the residual.
        r = bw(T,nx,nz,dx,Ra) - Alapl(w,nx,nz,dx2,dz2)

        # Compute the Laplacian of the residual.
        Ar = Alapl(r,nx,nz,dx2,dz2)

        # Compute the step size.
        alpha = numpy.sum(r * r) / (numpy.sum(r * Ar))

        # Update the vorticity solution.
        w = wk + alpha * r

        # 2. Compute new streamfunction from new vorticity
        # ------------------------------------------------
        # store old T
        psik = psi.copy()

        # Compute the residual.
        r = - w - Alapl(psi,nx,nz,dx2,dz2)

        # Compute the Laplacian of the residual.
        Ar = Alapl(r,nx,nz,dx2,dz2)

        # Compute the step size.
        alpha = numpy.sum(r * r) / (numpy.sum(r * Ar))

        # Update the streamfunction solution.
        psi = psik + alpha * r

        # Compute relative L2 norm of streamfunction
        Epsi = LA.norm(psi-psik, 2) / LA.norm(psi, 2)

        # check if tolerance criterion is satisfied
        if(Epsi <= tol):
            break

    return w, psi
```

##  2. Solution of 2D advection-diffusion problem

In step 2, we have to solve the advection-diffusion problem. The required Python code can be copied from the [previous notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/08_Convection_2D/03_2D_Linear_Advection_Diffusion.ipynb). We write a function `update_diff_2D` to update the 2D diffusion equation (5) using the FTCS method.

```python
# 2D Heat Diffusion Update
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def update_diff_2D(T, sigma_x, sigma_z, nx, nz):

    # store old temperature field
    Tn = T.copy()

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            T[j, i] = (Tn[j, i] +
                        sigma_x * (Tn[j, i+1] - 2.0 * Tn[j, i] + Tn[j, i-1]) +
                        sigma_z * (Tn[j+1, i] - 2.0 * Tn[j, i] + Tn[j-1, i]))

    return T
```

For the advection part, we have to compute the velocity field components from the streamfunction in order to use the Marker-in-Cell method. To do this, we write the small function `psi2vel`.

```python
# Convert stream function to velocity field
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def psi2vel(Vx,Vz,psi,nx,nz,dx,dz):

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            # velocities Vx = -dpsi/dz, Vz = dpsi/dx
            Vx[j,i] = -(psi[j,i] - psi[j-1,i]) / dz
            Vz[j,i] = (psi[j,i] - psi[j,i-1]) / dx

    return Vx, Vz
```

Now comes the most important part, we write the `mantle_convection` code, where the solution of the Poisson and advection-diffusion problems are assembled. During the solution of the mantle convection problem, we write the temperature field as `TIFF` images into the folder `image_out`. This requires to set the plotting routine to interactive mode, which is NOT possible in JupyterLab. Therefore, the code will NOT run in JupyterLab. The `TIFF` images can later be compiled to a time-lapse movie.

```python
# 2D thermal mantle convection code with JIT optimization
# -------------------------------------------------------
def mantle_convection(T0, psi0, w0, niter, nt, dx, dz, Lx, Lz, dt, Ra, tol):

    """
    Computes and returns the temperature field, streamfunction and vorticity
    fields for the 2D thermal mantle convection problem using the steepest
    descent method for the Poisson problem and the Marker-in-Cell/FTCS FD method
    to solve the advection-diffusion problem.

    Parameters
    ----------
    T0 : numpy.ndarray
        The initial temperature distribution as a 2D array of floats.
    psi0 : numpy.ndarray
        The initial streamfunction as a 2D array of floats.
    w0 : numpy.ndarray
        The initial vorticity as a 2D array of floats.
    niter : integer
        Maximum number of iterations using the steepest descent method.
    dx : float
        Grid spacing in the x direction.
    dz : float
        Grid spacing in the z direction.
    Ra : float
        Rayleigh number.
    Lx : float
        Extension of the model in x-direction.
    Lz : float
        Extension of the model in z-direction.
    dt : float
        Time step.
    Ra : float
        Rayleigh number.
    tol : float
        tolerance for abortion criterion.

    Returns
    -------
    T : numpy.ndarray
        The temperature distribution as a 2D array of floats.
    psi : numpy.ndarray
        The streamfunction as a 2D array of floats.
    w : numpy.ndarray
        The vorticity as a 2D array of floats.
    """

    # Copy fields
    T = T0.copy()
    psi = psi0.copy()
    w = w0.copy()

    # Initialize velocity fields
    Vx = numpy.zeros_like(psi)
    Vz = numpy.zeros_like(psi)

    # Estimate number of grid points in x- and y-direction
    nz, nx = T.shape

    # Define the locations along a gridline.
    x = numpy.linspace(0, Lx, num=nx)
    z = numpy.linspace(0, Lz, num=nz)

    # Define the initial temperature distribution as Gaussian function
    X, Z = numpy.meshgrid(x,z) # coordinates X,Z required to define T0

    # Define some constants.
    dx2 = dx**2
    dz2 = dz**2

    sigma_x = dt / dx2
    sigma_z = dt / dz2

    # Define number of markers and initial marker positions
    nx_mark = 4 * nx  # number of markers in x-direction
    nz_mark = 4 * nz  # number of markers in z-direction

    # Plot the initial temperature distribution and streamlines
    fig = pyplot.figure(figsize=(13., 5.))
    pyplot.tight_layout()
    extent = [numpy.min(x), numpy.max(x),numpy.min(z), numpy.max(z)]
    cmap = 'jet'
    im = pyplot.imshow(numpy.flipud(T), extent=extent, interpolation='spline36', cmap=cmap)

    pyplot.xlabel('x [ ]')
    pyplot.ylabel('z [ ]')
    cbar = pyplot.colorbar(im)
    pyplot.gca().invert_yaxis()
    cbar.set_label('Temperature [ ]')

    # activate interactive plot
    pyplot.ion()
    pyplot.show(block=False)

    # write temperature field snapshots every nsnap time steps to image file
    nsnap = 1

    # Loop over timesteps
    for n in range(nt):

        # 1. Compute phi and w based on current temperature field
        # -------------------------------------------------------
        w, psi = psi_w_bvp(T, psi, w, niter, dx, dz, Ra, tol)

        # 2. Compute T from Advection-Diffusion equation using Marker-in-Cell & FTCS method
        # ---------------------------------------------------------------------------------

        # Compute velocity fields from stream function
        Vx, Vz = psi2vel(Vx,Vz,psi,nx,nz,dx,dz)

        # initial marker positions
        x_mark = numpy.linspace(0, Lx, num=nx_mark)
        z_mark = numpy.linspace(0, Lz, num=nz_mark)
        X_mark, Z_mark = numpy.meshgrid(x_mark,z_mark)

        # Interpolate velocities from grid to marker position at timestep n
        f = interpolate.interp2d(x, z, Vx, kind='linear')
        vx_mark_n = f(x_mark, z_mark)

        f = interpolate.interp2d(x, z, Vz, kind='linear')
        vz_mark_n = f(x_mark, z_mark)

        # Interpolate temperature from grid to marker position at timestep n
        f = interpolate.interp2d(x, z, T, kind='cubic')
        T_mark = f(x_mark, z_mark)

        # Save current marker positions
        X0 = X_mark
        Z0 = Z_mark

        # Update marker position
        X_mark = X_mark + vx_mark_n * dt
        Z_mark = Z_mark + vz_mark_n * dt

        # Interpolate velocities from grid to marker position at timestep n+1
        vx_mark_n1 = interpolate.griddata((X.flatten(), Z.flatten()), Vx.flatten(), (X_mark, Z_mark), method='linear')
        vz_mark_n1 = interpolate.griddata((X.flatten(), Z.flatten()), Vz.flatten(), (X_mark, Z_mark), method='linear')

        # Replace Nan values
        mask = numpy.where(numpy.isnan(vx_mark_n1))
        vx_mark_n1[mask] = 0

        mask = numpy.where(numpy.isnan(vz_mark_n1))
        vz_mark_n1[mask] = 0

        # Update marker position with midpoint velocity
        X_mark = X0 + dt * (vx_mark_n + vx_mark_n1) / 2.
        Z_mark = Z0 + dt * (vz_mark_n + vz_mark_n1) / 2.

        # Interpolate temperature field from marker to grid positions
        T = interpolate.griddata((X_mark.flatten(), Z_mark.flatten()), T_mark.flatten(), (X, Z), method='cubic')

        # Replace Nan-values by old temperature field
        mask = numpy.where(numpy.isnan(T))
        T[mask] = T0[mask]

        # Update diffusion equation
        T = update_diff_2D(T, sigma_x, sigma_z, nx, nz)

        # Apply Neumann boundary condition (dTdx=0) at left and right boundary
        T[:,0] = T[:,1]
        T[:,-1] = T[:,-2]

        # update temperature/velocity fields and time
        if (n % nsnap) == 0:
            im.set_data(numpy.flipud(T))
            fig.canvas.draw()
            name_snap = "image_out/Conv_2D_" + "%0.*f" %(0,numpy.fix(n+1000)) + ".tiff"
            pyplot.savefig(name_snap, format='tiff', bbox_inches='tight', dpi=125)

    return T, w, psi
```

## First 2D Mantle Convection Modelling Run

After writing the `mantle_convection` code and all the required functions attached to it, we can define and run our first 2D mantle convection model.

Let's assume that the dimensionless width of the earth mantle is $ L_x = 2$ in x-direction and it has a thickness of $L_z = 1$. The model is discretized with $nx=204$ gridpoints in x-direction and $ny=102$ gridpoints in z-direction. Regarding the thermal properties, assume a dimensionless temperature of $\text{Tbot} = 1$ at the bottom of the mantle and a temperature $\text{Tsurf} = 0$ at the top boundary.

Furthermore, we have to make an initial guess for the temperature field $T_0$ in the mantle. Let 's assume it consists of two parts:

$$
T_0 = T_{lin} - T_{2D}
$$

a 1D linear temperature gradient

$$
T_{lin} = \frac{(Tbot-Tsurf)}{Lz} * Z + Tsurf
$$

and a 2D temperature variation in x-direction:

$$
T_{2D} = p\; cos\biggl(\frac{2 k \pi x}{L_x}\biggr)
$$

Does the 2D temperature distribution look familiar? You have already encountered it, when dealing with the [Lorenz equations](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/03_Lorenz_equations/01_LorenzEquations_intro.ipynb). It was one of the modes of the highly truncated Fourier series of the temperature field.

Regarding the parameters $p$ and $k$, lets start with $p=0.01$ and $k=1$ from the [manual of the mantle convection code ASPECT](http://www.math.clemson.edu/~heister/manual.pdf). For the initial streamfunction and vorticity, we assume $\psi_0(x,z)=\omega_0(x,z)=0$. Furthermore, Dirichlet boundary conditions for $psi$ and $\omega$ are set to

\begin{equation}
\begin{split}
\psi(0,z) &= \psi(L_x,z) = \psi(x,0) = \psi(x,L_z) = 0\\
\omega(0,z) &= \omega(L_x,z) = \omega(x,0) = \omega(x,L_z) = 0\\
\end{split} \notag
\end{equation}

For the temperature distribution $T$ Dirichlet boundary conditions are assumed on the top and bottom boundary:
\begin{equation}
\begin{split}
T(x,0) &= \text{Tsurf} = 0\\
T(x,L_z) &= \text{Tbot} = 1\\
\end{split}\notag
\end{equation}

Neumann boundary conditions are set at the left and right boundary

\begin{equation}
\begin{split}
&\frac{\partial T}{\partial x}(0,z) = 0\\
&\frac{\partial T}{\partial x}(L_x,z) = 0\\
\end{split}\notag
\end{equation}

```python
# Definition of modelling parameters
# ----------------------------------
Lx = 2.0     # width of the mantle in the x direction []
Lz = 1.0     # thickness of the mantle in the z direction []
nx = 204     # number of points in the x direction
nz = 102     # number of points in the z direction
dx = Lx / (nx - 1)  # grid spacing in the x direction []
dz = Lz / (nz - 1)  # grid spacing in the z direction []

# Define the locations along a gridline.
x = numpy.linspace(0.0, Lx, num=nx)
z = numpy.linspace(0.0, Lz, num=nz)

# Define initial temperature guess
Tbot = 1.     # temperature at the bottom boundary []
Tsurf = 0.    # temperature at the top boundary []

X, Z = numpy.meshgrid(x,z) # coordinates X,Z required to define T0

# Assume 1D linear increasing temperature profile with 2D perturbation
Tlin = (Tbot-Tsurf)/Lz * Z + Tsurf # linear temperature increase with depth

# 2D temperature perturbation in x-direction with parameters from
# the Spiegelmann script
p = 0.01
k = 1.  # boring initial condition
#k = 1.42   # more interesting initial condition
T2D = p * numpy.cos(k * 2. * numpy.pi * X / Lx)

# Initial temperature distribution
T0 = Tlin - T2D

# Apply Dirichlet boundary condition for the temperature
# at the top and bottom boundary
T0[0,:] = Tsurf
T0[-1,:] = Tbot

# Define initial streamfunction guess
psi0 = numpy.zeros((nz,nx))

# Define initial vorticity guess
w0 = numpy.zeros((nz,nx))

# Define maximum number of iterations niter
# and tolerance tol for iterative Poisson solver
niter = 10000
tol = 1e-4

# Compute time step dt using the CFL-criterion
sigma = 0.25
dt = sigma * min(dx, dz)**2  # time-step size
```

After setting almost all modelling parmeters in the cell above, it might be a good idea to compute the **Rayleigh number $Ra$** of the earth mantle:

$$
Ra = \frac{\rho g \alpha \Delta T d^3}{\eta \kappa}
$$

```python
rho = 4000.   # density of the earth mantle [kg/m^3]
g = 9.81      # gravity accelaration [m/s^2]
alpha = 3e-5  # thermal expansion coefficient [K^-1]
DeltaT = 3000 # temperature difference between earth surface and core-mantel boundary [K]
d = 2900. * 1000 # thickness of the earth mantle [m]
eta = 1e22    # shear viscosity [Pa s]
kappa = 1e-6  # thermal diffusivity [m^2/s]

# Rayleigh number of earth mantle
Ra = rho * g * alpha * DeltaT * d**3 /(eta * kappa)

print("Ra = ", Ra)
```

    Ra =  8613219.24

A Rayleigh number `Ra=8.6e6` is well above the critical Rayleigh number. It would require an even finer spatial model discretization than defined above. Therefore, we will use a Rayleigh number of `Ra=1e6`, which will already show most of the features also present when modelling with the higher Rayleigh number.

After setting all modelling parameters, we can run the `mantle_convection` modelling code to compute the numerical solution after `nt=293` time steps.

##### Warning

When executing the mantle convection code cell below, expect a computation time of roughly an hour. On my desktop gaming PC (Intel Core i7-6700K CPU @ 4 GHz, 16 GB Ram) the runtime for `nt=293` was about 37 mins.

```python
%matplotlib notebook
# Define Rayleigh number []
Ra = 1e6

# number of time steps to compute
nt = 293

# Compute temperature, streamfunction and temperature distribution
T, w, psi = mantle_convection(T0, psi0, w0, niter, nt, dx, dz, Lx, Lz, dt, Ra, tol)
```

    <IPython.core.display.Javascript object>

After assembling the snapshots from the `image_out/` directory, we get the following movie ...

```python
from IPython.display import YouTubeVideo
YouTubeVideo('uicDT4nIq-Q')
```

Notice, how fast a stable convection pattern is formed from the symmetric initial temperature distribution, despite the large Rayleigh number. In the convection cell on the left side we have a counter-clockwise convection current, while a clockwise convection occurs on the right side. The advection pattern in these cells looks quite similar to the Gaussian temperature anomaly in the velocity field used in the Lorenz approximation, as modelled in [this notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/08_Convection_2D/02_2D_Linear_Advection.ipynb). Beside the advection, the diffusion part also "averages" the temperature in the convection cells. After `nt=239` timesteps an average temperature of `T=0.5` is reached.

The temperature only changes in thin boundary layers on the top and bottom of the mantle model. From the heated bottom boundary hot material rises to the surface in a very thin plume at the center of the model. Cold material sinks from the surface to the core-mantle boundary near the left and right model boundary.

What happens, if we impose different initial conditions, by setting `k=1.42` in order to break the symmetry of the 2D initial temperature field? The animation below is based on this initial condition and covers `nt=1717` time steps with a total runtime of 3h 36min on my desktop PC.

```python
YouTubeVideo('9gqsNUD0Mdw')
```

Compared to the first modelling run with the symmetric initial condition, no stable convection pattern can be established. Instead a time-dependent, complex convection pattern occurs, where hot plumes are rising from the lower boundary layer and cold plumes are descending from the top boundary layer. Hot and cold plumes are interacting with each other. It is a quite complex solution and fascinating to watch.

## What we learned:

* How to write a complex 2D thermal convection by splitting the governing equations into subproblems, solve and verify them separately and finally assemble them again in order to solve the complex problem

* Even at a Rayleigh number of Ra = 1e6, we can get a quite stable convection pattern, assuming a symmetric 2D temperature distribution as initial condition

* On the other hand, assuming a non-symmetric 2D temperature distribution leads to complex time-dependent convection patterns.


# ReLU Layers

We can write a ReLU layer $z = \max(Wx+b, 0)$ as the
convex optimization problem
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \|z-\tilde Wx - b\|_2^2 \\[.2cm]
\mbox{subject to} & z \geq 0, \\
& \tilde W = W,
\end{array}
\label{eq:prob}
\end{equation}
with variables $z$ and $\tilde W$,
and parameters $W$, $b$, and $x$.
(Note that we have added an extra variable $\tilde W$ so
that the problem is DPP.)

We can embed this problem into a TensorFlow Keras `Layer` and use it
as a layer in a sequential neural network.
We note that this example is purely illustrative;
one can implement a ReLU layer much more efficiently
by directly performing the matrix multiplication, vector addition,
and then taking the positive part.

```python
from cvxpylayers.tensorflow import CvxpyLayer
import cvxpy as cp
import tensorflow as tf
from tensorflow.keras import layers

tf.keras.backend.set_floatx('float64')
```

```python
class ReluLayer(layers.Layer):
    def __init__(self, input_dim, output_dim):
        super(ReluLayer, self).__init__()
        self.W = tf.Variable(1e-3 * tf.random.normal((output_dim, input_dim), dtype=tf.float64))
        self.b = tf.Variable(1e-3 * tf.random.normal((output_dim,),  dtype=tf.float64))
        z = cp.Variable(output_dim)
        Wtilde = cp.Variable((output_dim, input_dim))
        W = cp.Parameter((output_dim, input_dim))
        b = cp.Parameter(output_dim)
        x = cp.Parameter(input_dim)
        problem = cp.Problem(cp.Minimize(
            cp.sum_squares(z - Wtilde @ x - b)), [z >= 0, Wtilde == W])
        self.cvxpy_layer = CvxpyLayer(problem, [W, b, x], [z])

    def call(self, x):
        if tf.rank(x) == 2:
            # when x is batched, repeat W and b
            batch_size = x.shape[0]
            return self.cvxpy_layer(
                tf.stack([self.W for _ in tf.range(batch_size)]),
                tf.stack([self.b for _ in tf.range(batch_size)]), x)[0]
        else:
            return self.layer(self.W, self.b, x)[0]
```

We generate synthetic data and create a network of two `ReluLayer`s followed by a linear layer.

```python
tf.random.set_seed(0)
model = tf.keras.Sequential([
    ReluLayer(20, 20),
    ReluLayer(20, 20),
    tf.keras.layers.Dense(1, input_shape=(20,), dtype=tf.float64)
])
X = tf.random.normal((300, 20), dtype=tf.float64)
Y = tf.random.normal((300, 1), dtype=tf.float64)
```

Now we can optimize the parameters inside the network using, for example, the ADAM optimizer.
The code below solves 15000 convex optimization problems and calls backward 15000 times.

```python
opt = tf.keras.optimizers.Adam(1e-2)
for _ in range(25):
    with tf.GradientTape() as tape:
        loss = (1 / X.shape[0]) * tf.math.reduce_sum((Y - model(X))**2)
        print(loss)
    grads = tape.gradient(loss, model.variables)
    opt.apply_gradients(zip(grads, model.variables))
```

    tf.Tensor(1.1182059444179566, shape=(), dtype=float64)
    tf.Tensor(1.1135983945033263, shape=(), dtype=float64)
    tf.Tensor(1.1057307058385368, shape=(), dtype=float64)
    tf.Tensor(1.096769654269908, shape=(), dtype=float64)
    tf.Tensor(1.084459023932493, shape=(), dtype=float64)
    tf.Tensor(1.0690279563783676, shape=(), dtype=float64)
    tf.Tensor(1.0512619022795688, shape=(), dtype=float64)
    tf.Tensor(1.0322302881090637, shape=(), dtype=float64)
    tf.Tensor(1.0125819721396856, shape=(), dtype=float64)
    tf.Tensor(0.9913663215521406, shape=(), dtype=float64)
    tf.Tensor(0.9653676807090706, shape=(), dtype=float64)
    tf.Tensor(0.9346387377331861, shape=(), dtype=float64)
    tf.Tensor(0.901583417900799, shape=(), dtype=float64)
    tf.Tensor(0.8696330587760338, shape=(), dtype=float64)
    tf.Tensor(0.8358727832552605, shape=(), dtype=float64)
    tf.Tensor(0.8012632687553815, shape=(), dtype=float64)
    tf.Tensor(0.7656949960892767, shape=(), dtype=float64)
    tf.Tensor(0.732059435220954, shape=(), dtype=float64)
    tf.Tensor(0.7007173654028725, shape=(), dtype=float64)
    tf.Tensor(0.6717739637641655, shape=(), dtype=float64)
    tf.Tensor(0.6479549948680946, shape=(), dtype=float64)
    tf.Tensor(0.6260994111911655, shape=(), dtype=float64)
    tf.Tensor(0.6061758074104243, shape=(), dtype=float64)
    tf.Tensor(0.5910289888009993, shape=(), dtype=float64)
    tf.Tensor(0.5749367021103264, shape=(), dtype=float64)


# TALENT Course 11
## Learning from Data: Bayesian Methods and Machine Learning
### York, UK, June 10-28, 2019
$% Some LaTeX definitions we'll use.
\newcommand{\pr}{\textrm{p}}
$

## Model selection (I)

### Bayesian evidence:
Please see the full version of the lecture notes here in [html](pub/model_selection-bs.html) and [pdf](pub/model_selection-minted.pdf) formats. The notes contain a somewhat adapted version of Ch. 4.1 in Sivia's book: "The story of Dr A and Prof B", which is extremely well written. It also contains a summary of Laplace's method for approximating evidence factors.

### Import of modules

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

from scipy import optimize

# Not really needed, but nicer plots
import seaborn as sns
sns.set()
sns.set_context("talk")
```

## What order polynomial?

Throughout the rest of this section, we will use data that was generated from a "true model" where x and y satisfy the following:

$$
y_i = x_i \sin(x_i)+\epsilon_i,
$$

where $0 \leq x_i \leq 3$ and the noise is drawn from a normal distribution $\epsilon_i \sim \mathcal{N}(0, \sigma_0)$. The values for 20 regularly spaced points with $\sigma_0=0.1$ are shown below.

```python
#------------------------------------------------------------
# Define our functional form
def true_func(x):
    return np.sin(x) * x

def func(x, dy):
    return np.random.normal(true_func(x), dy)

#------------------------------------------------------------
# select the (noisy) data
# np.random.seed(0)
num_data = 20
x_min = 0; x_max = 3.
x = np.linspace(x_min, x_max, num_data+2)[1:-1]
sig0 = 0.1  # try 0.5 or higher or 0.01
y = func(x, sig0)
```

```python
fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(1,1,1)
ax.errorbar(x, y, sig0, fmt='o');
ax.plot(x, true_func(x), color='red')
ax.set(xlabel='x',ylabel='y')
fig.tight_layout()
```

Assume that we have a multiple of models, with varying degree of sophistication. In this example we use polynomials of different orders to represent models of increasing complexity and with an increasing number of model parameters.

> Our task is to find which model finds the most support in the given data.

It is clear that a more complicated model with more free parameters should be able to fit the data much more closely. But what is the evidence in the data for such a complicated model? Finding the answer to this question is a task for a Bayesian, and the problem is generally known as *Model selection*.

Below, we will use an approximate way of computing the Bayesian evidence, namely the Laplace method. In some cases one can also use conjugate priors to simplify the computation of the evidence factor. Or one can use certain sampling methods to compute the evidence numerically. The highlight will be the comparison of different models using the evidences to extract odds-ratios.

### The Model

See previous lecture notes, in particular [Parameter estimation III](../bayesian-parameter-estimation/Lecture_Th1a_rjf.pdf) for some more details.

In general, we're fitting a $M$-degree polynomial to data,

$$
y_M(x) = \sum_{i=0}^M \theta_i x^i
$$

where we use $\theta$ to denote our parameter vector of length $M$.

Assuming all the points are independent, we can find the full log likelihood by adding the individual likelihoods together:
$$
\begin{align}
\log p(D\mid\theta, I) &= -\frac{1}{2}\sum_{i=1}^N\left(\log(2\pi\sigma_0^2) + \frac{\left[ y_i - y_M(x_i;\theta)\right]^2}{\sigma_0^2}\right) \\
&= \text{constant} - \sum_{i=1}^N \frac{\left[ y_i - y_M(x_i;\theta)\right]^2}{2 \sigma_0^2}
\end{align}
$$

We often define the residuals
$$
R_i = \left[ y_i - y_M(x_i;\theta) \right]/\sigma_0,
$$
so that the relevant chi-square sum reads $- \sum_{i=1}^N R_i^2 / 2$.

```python
def residuals(theta, x=x, y=y, sigma0=sig0):
    dy = y - np.polyval(theta,x)
    return dy / sigma0

# Standard likelihood with Gaussian errors as specified
# uniform prior for theta
def log_likelihood(theta):
    return -0.5 * np.sum(residuals(theta)**2)
```

### Max likelihood fits

We can maximize the likelihood to find $\theta$ within a frequentist paradigm. Let us start with a linear fit:

```python
degree = 1
theta_hat = np.polyfit(x, y, degree)
x_fit = np.linspace(0, x_max, 1000)
y_fit = np.polyval(theta_hat, x_fit)
```

Rather than just plotting this fit, we will compare several different models in the figure below.

```python
def fit_degree_n(degree, ax):
    """Fit a polynomial of order 'degree', return the chi-squared, and plot in axes 'ax'."""
    theta_hat = np.polyfit(x, y, degree)
    x_fit = np.linspace(x_min, x_max, 1000)
    y_fit = np.polyval(theta_hat, x_fit)

    ax.errorbar(x, y, sig0, fmt='o');
    ax.plot(x_fit, true_func(x_fit), color='red', alpha=.5)

    ax.text(0.03, 0.96, f"d = {degree}", transform=plt.gca().transAxes,
            ha='left', va='top',
            bbox=dict(ec='k', fc='w', pad=10))

    ax.plot(x_fit, y_fit, '-k')
    ax.set_xlabel('$x$')
    ax.set_ylabel('$y$');

    return -2. * log_likelihood(theta_hat)  # chi_squared

#------------------------------------------------------------
# First figure: plot points with a linear fit
nrows=3; ncols=3;
fig = plt.figure(figsize=(6*ncols, 6*nrows))

num_plots = nrows * ncols
degrees = np.zeros(num_plots, dtype=int)
chi_sqs_dof = np.zeros(num_plots)
print('degree  chi^2/dof')
for i in range(num_plots):
    ax = fig.add_subplot(nrows, ncols, i+1)
    degrees[i] = i
    dof = len(x) - (degrees[i])
    chi_sqs_dof[i] = fit_degree_n(i, ax) / dof
    print(f'  {degrees[i]:d}      {chi_sqs_dof[i]:.3f}')
fig.tight_layout()
```

### Questions

* Change the degree of the polynomial that is used for the fit. Plot the fits next to each other in a multi-panel figure (e.g., make it 3 by 3).
* Compute the chi-squared value per degree of freedom (the ingredients are already included in the script) and plot that as a function of the degree of the polynomial. Is it decreasing, or is there a peak?
* For which degree polynomials would you say that you're underfitting the data?
* For which degree polynomials would you say that you're overfitting the data?

## Bayesian evidence

Let us try the Bayesian approach and actually compute the evidence for these different models. We will use the Laplace method for computing the norm of the posterior distribution (i.e. approximating it as a single Gaussian).

We use simple uniform priors for the model parameters:
$$
p(\theta_i|I) = \left\{
\begin{array}{ll}
\frac{1}{\theta_\mathrm{max} - \theta_\mathrm{min}} & \text{for } \theta_\mathrm{min} \leq \theta_i \leq \theta_\mathrm{max}, \\
0 & \text{otherwise},
\end{array}
\right.
$$
which means that the posterior will be
$$
p(\theta | D, I) = \frac{1}{(\theta_\mathrm{max} - \theta_\mathrm{min})^K} \frac{1}{\sqrt{(2\pi)\sigma_0^2}^N} \exp\left( -\chi^2 / 2\right),
$$
within the allowed prior region for the $K$ parameters and zero elsewhere.

Assuming that the peak of the Gaussian is located at $\theta^*$, well inside the prior region; we can easily approximate the integral
$$
Z_p = \int d^K \theta p(\theta | D, I),
$$
using Laplace's method (see lecture notes here in [html](pub/model_selection-bs.html) and [pdf](pub/model_selection-minted.pdf) formats). With this particular choice of prior, and again under the assumption that the cut at the edges does not change the integral over the multidimensional integral, we get
$$
Z_p \approx \frac{1}{(\theta_\mathrm{max} - \theta_\mathrm{min})^K} \exp\left( -\chi^2(\theta^*) / 2\right) \frac{\sqrt{(2\pi)^K}}{\sqrt{\det(\Sigma^{-1})}},
$$
where $\Sigma^{-1}_{ij} = \partial^2\chi^2/\partial \theta_i \partial \theta_j$ (i.e. the Hessian) evaluated at the maximum $\theta^*$. Note that we removed the constant factor $\sqrt{(2\pi)\sigma_0}^N$ since it will be the same for all models.

Note that for this linear regression problem we can get all these quantities ($\theta^*$, $\Sigma$) via linear algebra. See, e.g., Dick's [lecture notes](https://github.com/NuclearTalent/Bayes2019/blob/master/topics/bayesian-parameter-estimation/Lecture_Th1a_rjf.pdf) or Hogg's nice paper: [Data analysis recipes: Fitting a model to data](https://arxiv.org/abs/1008.4686). Below, we will use `numpy.polyfit` to extract the relevant quantities.

```python
# We use a uniform prior for all parameters in [theta_min, theta_max]
theta_max = 10
theta_min = -10
prior_range = theta_max - theta_min
```

```python
degree_max = 8
evidence = np.zeros(degree_max+1)
print("Degree  P*          Best fit parameters: ")
for ideg,deg in enumerate(range(degree_max+1)):
    theta_hat, Cov = np.polyfit(x, y, deg, cov='unscaled')
    if not (np.all(theta_hat < theta_max) and np.all(theta_hat > theta_min)):
        print("Outside of prior range")
    P_star = np.exp(log_likelihood(theta_hat))
    H=np.linalg.inv(Cov)
    evidence[ideg] = P_star * np.sqrt((2*np.pi)**deg / np.linalg.det(H)) / prior_range**deg
    print (f'   {deg}    {P_star:.2e}  ',('{:5.2f} '*len(theta_hat)).format(*theta_hat))
```

    Degree  P*          Best fit parameters:
       0    2.37e-175    1.10
       1    9.63e-82    0.56  0.25
       2    3.97e-17   -0.64  2.48 -0.75
       3    2.70e-03   -0.41  1.22  0.20 -0.11
       4    2.75e-03   -0.01 -0.32  1.04  0.32 -0.13
       5    3.32e-03    0.06 -0.50  0.98 -0.49  1.05 -0.23
       6    4.52e-03   -0.12  1.15 -4.25  7.18 -5.46  2.78 -0.41
       7    4.58e-03   -0.04  0.29 -0.57 -0.57  2.95 -2.93  2.08 -0.35
    Outside of prior range
       8    5.20e-03    0.18 -2.15 10.75 -28.11 40.99 -33.10 14.10 -1.75 -0.05

```python
d = np.arange(0, degree_max+1)

fig,ax = plt.subplots(figsize=(8, 6))
ax.plot(d,evidence,'o-')

ax.set_xlabel('polynomial degree')
ax.set_ylabel('evidence');
```

### Questions

* Can you see the transition from underfit to overfit in this figure?
* What would you say is the degree of polynomial that is supported by the data?

### Questions

* What happens when you change the number of the generated data?
* What happens when you change the range of the generated data?
* What happens when you change the error of the generated data?

We see sensitivity to the prior in the peak of the evidence with the degree of the polynomial in our model.  What about the scale-invariant prior advocated by [Minka](https://tminka.github.io/papers/minka-linear.pdf)?  Another possibility advanced by Trotta (Bayes in the sky) is a multi-normal Gaussian distribution with zero mean and Fisher information matrix as covariance. These are worth trying.

#### Odds-ratios
Quoting the well-known paper by Trotta:
[Bayes in the sky: Bayesian inference and model selection in cosmology](https://arxiv.org/abs/0803.4089) we can quantify an empirical scale for evaluating the strength of evidence when comparing two models:

Here, the ratio of the evidences of model $M_0$ and $M_1$ is given by,
\begin{equation}
 \label{eq:Bayes_factor}
 B_{01} = \frac{p(\mathrm{data} | M_0)}{p(\mathrm{data} | M_1)} \; ,
\end{equation}
which is also called _Bayes factor_. That means $|\ln B_{01}| \equiv |\ln p(\mathrm{data} | M_0) - \ln p(\mathrm{data} | M_1)|$ is the relevant quantity for estimating the strength of evidence of the two models (see first and last column of the table).

### Questions

* Create a table of odds-ratios to select between pairs of the different-order polynomial models, given that the ratio of prior probabilities for the different models is unity.

```python
# Odds ratio table
print('Calculating the logarithm of the Bayes ratio of degree n to n+1')
print('  n   n+1   log(Bn,n+1)')
for ideg,deg in enumerate(range(degree_max)):
    odds_ratio = evidence[ideg] / evidence[ideg+1]
    print(f'  {deg}   {deg+1}      {np.abs(np.log(odds_ratio)):.2f}')

```

    Calculating the logarithm of the Bayes ratio of degree n to n+1
      n   n+1   log(Bn,n+1)
      0   1      212.16
      1   2      145.71
      2   3      29.11
      3   4      2.37
      4   5      1.84
      5   6      1.34
      6   7      1.23
      7   8      0.70

## Extra

```python

```

### Cross validation
This section will introduce the frequentist tool of cross-validation. This approach is used extensively within machine-learning as a way to handle overfitting and underfitting, bias and variance.

```python
# Select the cross-validation points
ncross=5
index_cv = np.random.choice(range(len(x)), ncross, replace=False)

x_cv=x[index_cv]
y_cv=y[index_cv]
```

```python
# The training data is then
x_train = np.delete(x,index_cv)
y_train = np.delete(y,index_cv)
```

```python
# Plot training and CV errors as a function of polynomial degree d
degree_max = 13
d = np.arange(0, degree_max+1)
training_err = np.zeros(d.shape)
crossval_err = np.zeros(d.shape)

fig,ax = plt.subplots(figsize=(8, 6))
for i in range(len(d)):
    p = np.polyfit(x_train, y_train, d[i])
    training_err[i] = np.sqrt(np.sum((np.polyval(p, x_train) - y_train) ** 2)
                              / len(y_train))
    crossval_err[i] = np.sqrt(np.sum((np.polyval(p, x_cv) - y_cv) ** 2)
                              / len(y_cv))

ax.plot(d, crossval_err, '--k', label='cross-validation')
ax.plot(d, training_err, '-k', label='training')
ax.plot(d, sig0 * np.ones(d.shape), ':k')

ax.set_xlim(0, degree_max)
# You might need to change the y-scale if you make modifications to the training data
ax.set_ylim(0, 0.8)

ax.set_xlabel('polynomial degree')
ax.set_ylabel('rms error')
ax.legend(loc='best');

```

### Questions

* Can you see the transition from underfit to overfit in this figure?
* What would you say is the degree of polynomial that is supported by the data?
* Try changing the size of the cross-validation and training sets. Does the conclusions become more/less clear?
* Does the results change between different runs with the same number of CV samples? If so, why?
* K-fold cross validation is a popular variant of CV. It addresses some issues with the sensitivity to the actual choice of which data is used for training and validation.  What do you think that it means, and what is the possible drawback if you have a computational expensive model?
* Leave-one-out is another variant. For linear regression problems, this type of cross-validation can actually be performed without having to do multiple fits. What do you think that it means?
* It is common to divide the data into a training set, a cross-validation set, and a test set. What do you think is the purpose of having three different sets?


        Lista de Exercícios 4a

         Guilherme Esdras

---

        Imports

```python
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import pandas as pd
from scipy import optimize

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('seaborn-poster')

import sympy as sp
sp.init_printing()
sp.var('x, y')
```

---

    Exercicio 1: Implemente o método da Bisseção descrito pelo algoritmo 1.1 no texto. Para testar o algoritmo, use os exemplos do material textual.

```python
# Implementando função do Método da Bisseção

# Versão 1 (antes da aula de quinta-feira)

def bissecao_old(f, a, b, n_its, tol, verbose=False, r_tb=False):
    '''
    Método da Bisseção v1
        [by guilherme esdras]

    Parâmetros:
    ----------
        f (function) :
            A função na qual busca-se encontrar uma solução aproximada para f(x) = 0.
        a (int), b (int):
            Limites inferior e superior, respectivamente, do intervalo.
        n_its (int):
            Número máximo de iterações.
        tol (float):
            Tolerância mínima ou Erro aceitável (critério de parada).
        verbose (boolean):
            True = exibe as tabelas e prints;
            False (padrão) = não exibe.
        r_tb (boolean):
            True = retorna uma tabela de dados para uso com Pandas;
            False (padrão) = não retorna.

    Retornos:
    -------
        xi (int):
            O ponto médio do intervalo computado na i-ésima iteração.
        i (int):
            O número máximo de iterações que foi preciso para encontrar a solução.

    '''

    # Inicializa as variaveis de entrada
    i = 0
    ai = a
    bi = b

    ''' Opcional... '''
    if verbose or r_tb: tb = []

    # Verifica a existencia da raiz,
    # primeiro testando:
    if np.sign(f(a)) * np.sign(f(b)) != -1:
        if verbose: print('Não existe raiz no intervalo. Encerrando!')
        return

    # Caso não contrário...
    else:
        # Enquanto a tolerância for aceitável e o número de iterações não tiver sido atingido...
        while ( ((bi - ai) / 2.0) > tol ) and ( i < n_its ):

            ''' Opcional... '''
            if verbose or r_tb: tb.append([]); tb[i].append(ai); tb[i].append(bi);

            # calcula o ponto médio do intervalo...
            xi = (ai + bi) / 2.0

            ''' Opcional... '''
            if verbose or r_tb: tb[i].append(xi); tb[i].append(f(xi));

            # verifica se este ponto médio é a raiz...
            if f(xi) == 0:
                ''' Opcional... '''
                if verbose: print('A raiz é o valor atual de x!')

                return ( xi, i )

            # caso contrário, verificar em que subintervalo está a raiz...
            else:

                # se o primeiro subintervalo tem a raiz...
                if np.sign(f(ai)) * np.sign(f(xi)) == -1:
                    # atualiza o limite superior:
                    bi = xi

                # caso contrário (se o segundo subintervalo tem a raiz)...
                else:
                    # atualiza o limite inferior:
                    ai = xi

                # atualiza o iterador, incrementado-o
                i += 1

        ''' Opcional... '''
        if verbose:
            msg = f'Raiz encontrada: {xi} | Em {i} iterações'
            print(pd.DataFrame(tb, columns=["a", "b", "x", "f(x)"]))
            print('-'*len(msg))
            print(msg)
            print('-'*len(msg))
        ''' ----------- '''

        # ao fim, retorna o valor exato ou aproximado da raiz
        if not r_tb: return ( xi, i )
        else: return ( xi, i, tb )
```

```python
def bissecao_do_professor(f, a, b, tol=1e-10):
    i  = 0;
    erro, x_ant = 1, a
    while(erro > tol):
        inf = np.sign(f(a))
        sup = np.sign(f(b))
        if inf*sup != -1:
            print("Não há raiz nesse intervalo")
            return
        else:
            x = (a + b)/2.
            if f(x) == 0:
                print("A raíz é", x)
                return
            elif inf*np.sign(f(x)) == -1:
                b = x
            else: a = x
        erro = np.abs((x - x_ant)/np.abs(x))
        x_ant = x
    return x
```

```python
# Implementando função do Método da Bisseção

def bissecao(f, a, b, n_its=9999, tol=1e-10, verbose=False, r_tb=False, p_err=False):
    '''
    Método da Bisseção
        [by guilherme esdras]

    Parâmetros:
    ----------
        f (function):
            A função na qual busca-se encontrar uma solução aproximada para f(x) = 0.
        a (int), b (int):
            Limites inferior e superior, respectivamente, do intervalo.
        n_its (int) [Opcional | Default: 9999]:
            Número máximo de iterações.
        tol (float) [Opcional | Default: 1e-10]:
            Tolerância mínima ou Erro aceitável (critério de parada). Se nenhum valor for passado o padrão será 10^-10.
        verbose (boolean):
            True:
                Exibe as tabelas e prints;
            False (default):
                Não exibe.
        r_tb (boolean):
            True:
                Retorna uma tabela de dados para uso com Pandas;
            False (default):
                Não retorna.
        p_err (boolean):
            True:
                Imprime o erro em cada iteração;
            False (default):
                Não imprime.

    Retornos:
    -------
        xi (int):
            O ponto médio do intervalo computado na i-ésima iteração.
        i (int):
            O número máximo de iterações que foi preciso para encontrar a solução.

    '''

    # Inicializa as variaveis de entrada
    i = 0
    ai = a
    bi = b

    ''' Opcional... '''
    if verbose or r_tb: tb = []

    # Verifica a existencia da raiz,
    # primeiro testando:
    if np.sign(f(a)) * np.sign(f(b)) != -1:
        ''' Opcional... '''
        if verbose: print('Não existe raiz no intervalo. Encerrando!')
        return

    # Caso não contrário...
    else:
        erro, x_ant = 1, a
        # Enquanto a tolerância for aceitável e o número de iterações não tiver sido atingido...
        while ( erro > tol ) and ( i < n_its ):
            ''' Opcional... '''
            if verbose or r_tb: tb.append([]); tb[i].append(ai); tb[i].append(bi);

            # calcula o ponto médio do intervalo...
            x = ai + 0.5 * (bi - ai) # (ai + bi) / 2.0

            ''' Opcional... '''
            if verbose or r_tb: tb[i].append(x); tb[i].append(f(x));

            # verifica se este ponto médio é a raiz...
            if f(x) == 0:
                ''' Opcional... '''
                if verbose: print('A raiz é o valor atual de x!')
                return ( x, i )
            # caso contrário, verificar em que subintervalo está a raiz...
            else:
                # se o primeiro subintervalo tem a raiz...
                if np.sign(f(ai)) * np.sign(f(x)) == -1:
                    # atualiza o limite superior:
                    bi = x
                # caso contrário (se o segundo subintervalo tem a raiz)...
                else:
                    # atualiza o limite inferior:
                    ai = x
                # atualiza o iterador, incrementado-o
                i += 1

            # calcula o erro
            erro = np.abs((x - x_ant) / np.abs(x))
            x_ant = x
            if p_err: print(f'Erro {i}: {erro:.5f}')

        ''' Opcional... '''
        if verbose:
            msg = f'Raiz encontrada: {x} | Em {i} iterações'
            print('-'*len(msg))
            print(pd.DataFrame(tb, columns=["a", "b", "x", "f(x)"]))
            print('-'*len(msg))
            print(msg)
            print('-'*len(msg))
        ''' ----------- '''

        # ao fim, retorna o valor exato ou aproximado da raiz
        if not r_tb: return ( x, i )
        else: return ( x, i, tb )
```

---

**Realizando Teste 01**

```python
f = lambda x: ( x**5+2*x**3-5*x-2 )
sp.var('x')
f_sp = sp.Lambda(x, x**5+2*x**3-5*x-2)
f_sp
```

```python
x = np.linspace(-5, 5, 100)
plt.plot(x, f(x))
plt.grid()
```

```python
bissecao(f, -2, 2, verbose=True)
```

```python
bissecao_do_professor(f, -2, 2)
```

---

**Realizando Teste 02**

```python
f = lambda x: ( x * np.exp(x) - np.sin(8 * x) - 1)

sp.var('x')
f_sp = sp.Lambda(x, x * sp.exp(x) - sp.sin(8 * x) - 1)
f_sp
```

```python
x, i, tb = bissecao(f, -0.5, 1, r_tb=True)

print('\nRaiz {} encontrada em {} iterações.\nComo mostrado na tabela:\n'.format(x, i))

pd.DataFrame(tb, columns=["a", "b", "x", "f(x)"])
```

    Raiz 0.43458593652030686 encontrada em 36 iterações.
    Como mostrado na tabela:

      a
      b
      x
      f(x)

      0
      -0.500000
      1.000000
      0.250000
      -1.588291e+00

      1
      0.250000
      1.000000
      0.625000
      1.126578e+00

      2
      0.250000
      0.625000
      0.437500
      2.839648e-02

      3
      0.250000
      0.437500
      0.343750
      -8.968958e-01

      4
      0.343750
      0.437500
      0.390625
      -4.392856e-01

      5
      0.390625
      0.437500
      0.414062
      -2.034669e-01

      6
      0.414062
      0.437500
      0.425781
      -8.664151e-02

      7
      0.425781
      0.437500
      0.431641
      -2.885010e-02

      8
      0.431641
      0.437500
      0.434570
      -1.526564e-04

      9
      0.434570
      0.437500
      0.436035
      1.414120e-02

      10
      0.434570
      0.436035
      0.435303
      6.999002e-03

      11
      0.434570
      0.435303
      0.434937
      3.424343e-03

      12
      0.434570
      0.434937
      0.434753
      1.636134e-03

      13
      0.434570
      0.434753
      0.434662
      7.418116e-04

      14
      0.434570
      0.434662
      0.434616
      2.945957e-04

      15
      0.434570
      0.434616
      0.434593
      7.097419e-05

      16
      0.434570
      0.434593
      0.434582
      -4.083998e-05

      17
      0.434582
      0.434593
      0.434587
      1.506739e-05

      18
      0.434582
      0.434587
      0.434585
      -1.288622e-05

      19
      0.434585
      0.434587
      0.434586
      1.090601e-06

      20
      0.434585
      0.434586
      0.434585
      -5.897806e-06

      21
      0.434585
      0.434586
      0.434586
      -2.403602e-06

      22
      0.434586
      0.434586
      0.434586
      -6.564999e-07

      23
      0.434586
      0.434586
      0.434586
      2.170507e-07

      24
      0.434586
      0.434586
      0.434586
      -2.197246e-07

      25
      0.434586
      0.434586
      0.434586
      -1.336925e-09

      26
      0.434586
      0.434586
      0.434586
      1.078569e-07

      27
      0.434586
      0.434586
      0.434586
      5.325999e-08

      28
      0.434586
      0.434586
      0.434586
      2.596153e-08

      29
      0.434586
      0.434586
      0.434586
      1.231230e-08

      30
      0.434586
      0.434586
      0.434586
      5.487689e-09

      31
      0.434586
      0.434586
      0.434586
      2.075382e-09

      32
      0.434586
      0.434586
      0.434586
      3.692286e-10

      33
      0.434586
      0.434586
      0.434586
      -4.838481e-10

      34
      0.434586
      0.434586
      0.434586
      -5.730971e-11

      35
      0.434586
      0.434586
      0.434586
      1.559597e-10

---

    Exercicio 2 - Determine as raízes reais de $f(x) = −0.5x^2 + 2.5x + 4.5$

```python
# Exibindo
f = lambda x: ( ((-0.5)*(x**2)) + (2.5*x) + (4.5) )
sp.var('x')
f_sp = sp.Lambda(x, ((-0.5)*(x**2)) + (2.5*x) + (4.5))
f_sp
```

```python
a = (-0.5)
b = 2.5
c = 4.5

coefs = np.array([a, b, c])
coefs
```

    array([-0.5,  2.5,  4.5])

    (a) Graficamente

```python
x = np.linspace(-6, 10, 500)

y = f(x)

raiz1 = optimize.root(f, -2)
raiz2 = optimize.root(f, 7)
raizes = np.array([raiz1.x, raiz2.x])

fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111)

plt.vlines(x=raiz1.x, ymin=f(-6), ymax=0, colors='gray', ls=':', lw=2)
plt.vlines(x=raiz2.x, ymin=f(-6), ymax=0, colors='gray', ls=':', lw=2)

ax.axhline(0, color='b')

ax.plot(x, f(x), 'r', label="$f(x)$")
ax.plot(raizes, f(raizes), 'kv', label="$Raizes$")

ax.legend(loc='best')
ax.set_xlabel('$x$')
ax.set_ylabel('$f(x)$')

plt.show()
```

    (b) Usando a fórmula quadrática

**Forma 1: "na raça" com Bhaskara**

```python
def formula_quadratica(coefs, verbose=False):

    a = coefs[0]
    b = coefs[1]
    c = coefs[2]

    if a == 0:
        return

    bhask = (b**2) - 4*a*c
    delta = np.sqrt(np.abs(bhask))

    if bhask > 0:
        if verbose: print(" raizes reais e diferentes ")
        x1 = (-b + delta) / (2 * a)
        x2 = (-b - delta) / (2 * a)
        return x1, x2

    elif bhask == 0:
        if verbose: print(" raizes reais e iguais ")
        x = ( -b / (2 * a) )
        return x

    else:
        if verbose: print(" raizes complexas "); print(-b / (2 * a), " + i", delta); print(-b / (2 * a), " - i", delta);
        x = -b / (2 * a)
        return x

    return
```

```python
formula_quadratica(coefs, True)
```

**Forma 2: Usando Numpy**

```python
coefs = np.array([a, b, c])
x1, x2 = np.roots(coefs)
x2, x1
```

**Forma 3: Usando SymPy (apenas simbólico)**

```python
sp.var('x')
sp.solve(((-0.5)*(x**2)) + (2.5*x) + (4.5), x)
```

    (c) Usando três iterações do método da bisseção para determinar a maior raiz. Use as aproximações iniciais $x_l = 5$ e $x_u = 10$. Calcule o erro relativo obtido entre cada iteração, e o erro entre os valores verdadeiros encontrados no item b e o valor de cada iteração.

```python
bissecao(f, 5, 10, 3, verbose=True, p_err=True)
```

---

    Exercicio 3: Localize a primeira raiz não-trivial de $sin \ x = x^3$, onde x está em radianos. Use uma técnica gráfica e a bisseção com o intervalo inicial de 0,5 a 1. Faça os cálculos até que o erro seja inferior a 2%.

```python
f = lambda x: np.sin(x) - x**3
```

```python
# gráfico
x = np.linspace(-np.pi, np.pi, 500)
y = f(x)

raiz1 = optimize.root(f, -2)
raiz2 = optimize.root(f, 7)
raizes = np.array([raiz1.x, raiz2.x])

fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111)

ax.axhline(0, color='b')

plt.vlines(x=raiz1.x, ymin=f(3), ymax=0, colors='gray', ls=':', lw=2)
plt.vlines(x=raiz2.x, ymin=f(3), ymax=0, colors='gray', ls=':', lw=2)

ax.plot(x, f(x), 'r', label="$f(x)$")
ax.plot(raizes, f(raizes), 'kv', label="$Raizes$")

ax.legend(loc='best')
ax.set_xlabel('$x$')
ax.set_ylabel('$f(x)$')

plt.show()
```

```python
# bisseção
bissecao(f, 0.5, 1, tol=0.02, verbose=True)
```

---

    Exercicio 4: Dada $f(x) = −2x^6 − 1.5x^4 + 10x + 20$, encontre o máximo dessa função ($f'(x) = 0$) usando o método da bisseção, considerando o intervalo $[0, 1]$ e um erro limite de 5%.

```python
f = lambda x: ( (-2*x**6) - (1.5*x**4) + (10*x) + 20 )
sp.var('x')
f_sp = sp.Lambda(x, (-2*x**6) - (1.5*x**4) + (10*x) + 20)
f_sp
```

```python
# derivada
sp.diff(f_sp(x), x)
```

```python
f_ = lambda x: ( (-12*x**5) - (6*x**3) + 10 )
```

```python
x, i = bissecao(f_, 0, 1, tol=0.05)

f'O máximo é {x} encontrado em {i} iterações'
```

    'O máximo é 0.84375 encontrado em 5 iterações'

---

    Exercicio 5

```python
def regula_falsi(f, xl, xu, tol=1e-10):

    if (f(xl) * f(xu) >= 0):
        return -1

    i = 0
    x = xl
    erro, x_ant = 1, x

    while erro > tol:

        x = xu - ( ( f(xu)*(xl-xu) ) / (f(xl)-f(xu)) )

        if f(x) * f(xl) < 0:
            xu = x

        else:
            xl = x

        erro = np.abs((x - x_ant) / np.abs(x))
        x_ant = x
        i += 1

    return ( x, i )
```

```python
f = lambda x: ( x**5+2*x**3-5*x-2 )
```

```python
# Comparativo de métodos
print(f'Método da Bisseção: \t {bissecao(f, -2, 2)}')
print(f'Método Regula Falsi: \t {regula_falsi(f, -2, 2)}')
```

    Método da Bisseção: 	 (1.319641167181544, 35)
    Método Regula Falsi: 	 (1.3196411670137347, 51)

Como podemos notar, o método **Regula Falsi** conseguiu ainda mais casas decimais, utilizando mais iterações e chegando a um valor ainda mais preciso que o método da **Bisseção**.

Class 04/Lista 4a - Guilherme Esdras.ipynb


# Problem Set 1: Bayesian Decision Theory

## Introduction
The MNIST database (Modified National Institute of Standards and Technology Database) is a large collection of handwritten digits which happens to be a state of the art data set to train and test machine learning models using image processing.It consist of 60000 Training samples and 10000 Testing Samples. Each sample image consist of normalized $28X28$ pixels grayscale image stored as 784 dimensional vector for each sample.This is a way of dimension reduction done by the MNIST to use it for model developments.

### Preparing the datasets
For loading the given MNIST datasets we will be using the mnist library to fetch the data.The MNIST() function will be used to fetch the datasets from the directory './data'.

```python
import numpy as np
import cv2
from mnist.loader import MNIST
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

m = MNIST('./data')
```

Now we load the different datasets as the X-Training and Y-Training set which has 60000 samples which will be used to train the classification model and then the X-Test and Y-test data set which has 10000 samples will be used to test the model. After seperating the datasets , the data sets will be converted to ND array and as float values which will be benificial for the computations.

```python
xtrain,ytrain = m.load_training()
xtest,ytest = m.load_testing()
xtrain = np.asarray(xtrain).astype(np.float32)
ytrain = np.asarray(ytrain).astype(np.float32)
xtest = np.asarray(xtest).astype(np.float32)
ytest = np.asarray(ytest).astype(np.float32)

```

### Part 1 : Finding the Mean and Standard Deviation of the datasets
First we have to divide the given images which is in form of vector of dimension 784 back to matrix image of dimension $28X28$ and save it in a seperate list consisting of 60000 matrices.After that group them seperately from 0 to 9 using the y-training sets.

```python
xmat_list = []   #list that holds 60000 matrices
for i in range(0,ytrain.shape[0]):
    x = np.array(xtrain[i])
    matrix = np.reshape(x,(28,28))
    xmat_list.append(matrix)
l0=[]
l1=[]
l2=[]
l3=[]
l4=[]
l5=[]
l6=[]
l7=[]
l8=[]
l9=[]
cat_list = [l0,l1,l2,l3,l4,l5,l6,l7,l8,l9]
#it is the list of 10 which holds its respective category matrices within it as lists

for i in range(0,60000):
    yv = int(ytrain[i])
    cat_list[yv].append(xmat_list[i])
```

The the 60000 matrices has been seperated to its respective categories. Now the mean and standard deviation can be computed element wise over all the categories to obtain a Mean Image matrix and Standard Deviation Image Matrix. We will define two functions to find the matrix mean and Standard Deviation.

\begin{equation}
Mean
\\
\mu = \frac{\sum x_i}{N}
\end{equation}
\
\begin{equation}
Standard Deviation
\\
\sigma = \sqrt{\frac{\sum(x_i - \mu)^2}{N}}
\end{equation}

```python
def matrix_mean(L):
    sum = np.zeros((28,28))
    for i in range(0,len(L)):
        sum = sum + L[i]
    mean = sum/len(L)
    return mean
def matrix_sd(L,mean):
    n = len(L)
    sumd = np.zeros((28,28))
    for i in range(0,n):
        sumd = sumd + (L[i]-mean)**2
    sd = sumd/n
    sd = np.sqrt(sd)
    return sd

```

Each list of matrices present in the 10 categories will be passed through both these functions to compute the Mean and Standard Deviation matrices. Then the matrix image cab be ploted.

```python

mean_list = []
sd_list = []
for i in range(0,10):
    mean = matrix_mean(cat_list[i])
    sd = matrix_sd(cat_list[i],mean)
    mean_list.append(mean)
    sd_list.append(sd)

for i in range(0,10):
    fig = plt.figure()
    a = fig.add_subplot(1, 2, 1)
    imgplot = plt.imshow(mean_list[i],cmap='gray')
    a.set_title('Mean')
    a = fig.add_subplot(1, 2, 2)
    imgplot = plt.imshow(sd_list[i],cmap='gray')
    a.set_title('Standard Deviation')
```

### Part 2 : Classifying the Testing data set
We will be using Linear Discriminant Analysis (LDA) to build our classifier. The LDA is a classifier with linear decision boundary which is accuired by fitting the class conditional probablities and bayesian decision theory to it.We will fit in a decision boundary from the given training data and following it find the zero-one classification loss. After that we will use the Baye's Rull to predict the values.
\begin{equation}
Bayes Rule
\\
p(A|B) = \frac{p(B|A)P(A)}{P(B)}
\\
P(B) = \sum P(B|A)P(A)
\end{equation}

```python

```

    (784,)

```python

true = 0
error = 0
for i in range(0,ytest.shape[0]):
    error = error + ((ytest[i]-ypred[i])**2)**0.5
    if(ytest[i] == ypred[i]):
        true = true+1

accuracy = true/ypred.shape[0]
Error = error/ypred.shape[0]

print('Accuracy - ',accuracy*100,'%')
print('Error - ',Error*100,'%')

```

    Accuracy -  87.3 %
    Error -  45.39 %

```python

```