###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c) 2019 Daniel Koehn, based on (c)2018 L.A. Barba, G.F. Forsyth [CFD Python](https://github.com/barbagroup/CFDPython#cfd-python), (c)2014 L.A. Barba, I. Hawke, B. Knaepen [Practical Numerical Methods with Python](https://github.com/numerical-mooc/numerical-mooc#practical-numerical-methods-with-python), also under CC-BY.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# 2D Boundary Value Problem Example: Thermal Convection in the Earth Mantle

Let's take a look at a more complex boundary value problem - stationary solutions for 2D thermal mantle convection.

## Thermal Convection in the Earth Mantle

We already derived the governing equations for thermal convection in [lecture 3](https://danielkoehnsite.files.wordpress.com/2019/11/lect_03_simple_convection_lorenz_eqs-1.pdf) , when investigating the complex dynamics of the [Lorenz equations](https://github.com/daniel-koehn/Differential-equations-earth-system/tree/master/03_Lorenz_equations#lecture-3-simplified-convection-problem-the-lorenz-equations). Let's shortly revisit the basic results:

The full coupled set of scaled PDE's describe the coupling of incompressible 2D Navier Stokes flow with an advection-diffusion equation for temperature, and can be written in dimensionless form as,

$$
   \frac{1}{\mathrm{Pr}}\left[ \frac{\partial \omega}{\partial t} + \vec{v}\cdot\nabla\omega\right] = \nabla^2\omega + \mathrm{Ra}\frac{\partial T}{\partial x}
$$
$$
    \nabla^2 \psi = -\omega
$$
$$
    \frac{\partial T}{\partial t} + \vec{v}\cdot\nabla T = \nabla^2 T
$$

where

$$
    \vec{v}=(u,0,w) = \nabla\times\psi\hat{j}=(\frac{\partial\psi}{\partial z}, 0,  -\frac{\partial\psi}{\partial x})
$$

is the fluid velocity field (which in this form is exactly incompressible with $\nabla\cdot\vec{v}=0$).  $\psi$ is the "Streamfunction"  whose contours are tangent to the fluid trajectories at all times.  The vorticity,

$$
    \omega = \bf{\vec{\omega} \cdot \hat{j}} = (\nabla\times\vec{v}) \cdot \hat{j}
$$

measures the local rate of rotation of the fluid, and is driven by horizontal variations in temperature (actually density). $\hat{j}=(0,1,0)^T$ denotes the unit vector in y-direction.

Now, if we are dealing with mantle convection, the Prandtl number Pr becomes very large, actually it can be assumed to be inifinite. Therefore, the governing equations simplify to

\begin{equation}
\begin{split}
\nabla^2\omega &=  - \mathrm{Ra}\frac{\partial T}{\partial x} \\
\nabla^2 \psi &= -\omega \\
\nabla^2 T &= \frac{\partial T}{\partial t} + \vec{v}\cdot\nabla T
\end{split} \tag{1}
\end{equation}

Next, we want to focus on finding **stationary solutions** or **fixpoints** of the PDEs describing 2D thermal mantle convection. This problem should already be familiar for you, because we extensively discussed fixpoints, when calculating [stationary solutions of the Lorenz equations](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/03_Lorenz_equations/02_Stationary_solutions_of_DE.ipynb). Fixpoints are calculated by setting all time derivatives in eqs. (1) to zero, leading to:

\begin{equation}
\begin{split}
\nabla^2\omega &=  - \mathrm{Ra}\frac{\partial T}{\partial x} \\
\nabla^2 \psi &= -\omega \\
\nabla^2 T &= \vec{v}\cdot\nabla T
\end{split} \tag{2}
\end{equation}

By inserting the definition of the velocity vector $\vec{v}=(u,0,w) =(\frac{\partial\psi}{\partial z}, 0,  -\frac{\partial\psi}{\partial x})$ using the streamfunction in eqs. (2), we get

\begin{equation}
\begin{split}
\nabla^2\omega &=  - \mathrm{Ra}\frac{\partial T}{\partial x} \\
\nabla^2 \psi &= -\omega \\
\nabla^2 T &= \frac{\partial\psi}{\partial z} \frac{\partial T}{\partial x} - \frac{\partial\psi}{\partial x} \frac{\partial T}{\partial z}
\end{split} \tag{3}
\end{equation}

Now, take a closer look at eqs. (3). Do you notice the three Poisson equations with Laplace operators on the LHS $Ap$ and some more complex terms on the RHS $b$. So each subproblem can be written as matrix equation:

$$
Ap = b
$$

The difference between problem (3) and the Poisson problems discussed in the previous notebooks, is that the Poisson equations are coupled.

### Numerical Solution of the 2D thermal mantle convection boundary value problem

Time to solve eqs. (3) using the [steepest descent method](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/07_BVP_2D/03_Iterative_Solvers.ipynb). Starting from initial fields $(\omega_0,\psi_0,T_0)$, the iterative algorithm can be summarized at each iteration step $n$ as

1. Update the vorticity $\omega^{(n+1)}$ from the previous temperature field $T^{(n)}$ by solving the Poisson equation

$$
\nabla^2\omega^{(n+1)} =  - \mathrm{Ra}\frac{\partial T^{(n)}}{\partial x}
$$

2. Update the streamfunction $\psi^{(n+1)}$ based on the new vorticity $\omega^{(n+1)}$ by solving the Poisson equation

$$
\nabla^2 \psi^{(n+1)} = -\omega^{(n+1)}
$$

3. Update the temperature field $T^{(n+1)}$ from the new streamfunction $\psi^{(n+1)}$ by solving the Poisson equation

$$
\nabla^2 T^{(n+1)} = \frac{\partial\psi^{(n+1)}}{\partial z} \frac{\partial T^{(n)}}{\partial x} - \frac{\partial\psi^{(n+1)}}{\partial x} \frac{\partial T^{(n)}}{\partial z}
$$

4. Test for convergence:

\begin{equation}
\begin{split}
E_T^{(n+1)} &= \frac{||T^{(n+1)}-T^{(n)}||}{||T^{(n+1)}||}\\
E_{\psi}^{(n+1)} &= \frac{||\psi^{(n+1)}-\psi^{(n)}||}{||\psi^{(n+1)}||}\\
\end{split} \notag
\end{equation}

5. If $E_T^{(n+1)} \le \epsilon_{T}$ and $E_\psi^{(n+1)} \le \epsilon_{\psi}$ are satisfied, stop the iteration, otherwise continue step 1.

It's coding time. As usual, we start by importing some basic Python libraries and set up the plotting environment.

```python
# Import libraries
import numpy
from matplotlib import pyplot
%matplotlib inline

# import JIT from Numba
from numba import jit

# import Numpy Linear Algebra package
from numpy import linalg as LA
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

For the steepest descent method, we need a function to compute the action of the Laplace operator $\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial z^2}$ on an abritary field $u$. In `Alapl` the second derivatives are apprximated by the three-point central operator:

```python
# Compute Laplace operator applied to scalar field u
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def Alapl(u,nx,nz,dx2,dz2):

    A = numpy.zeros_like(u)

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            # Compute Laplace operator for scalar field u
            d2udx2 = (u[j, i+1] + u[j, i-1] - 2 * u[j, i]) / dx2
            d2udz2 = (u[j+1, i] + u[j-1, i] - 2 * u[j, i]) / dz2

            A[j, i] = d2udx2 + d2udz2

    return A
```

Next, we have to define the RHS $b$ for the vorticity equation `bw` ...

```python
# Compute b for vorticity equation
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def bw(T,nx,nz,dx,Ra):

    b = numpy.zeros_like(T)

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            # Compute b for vorticity equation
            dTdx = - Ra * (T[j,i+1] - T[j,i]) / dx

            b[j, i] = dTdx

    return b
```

... and the temperature equation `bT`

```python
# Compute b for temperature equation
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def bT(T,psi,nx,nz,dx,dz):

    b = numpy.zeros_like(T)

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            # Compute b for temperature equation
            dTdx = (T[j,i+1] - T[j,i]) / dx
            dTdz = (T[j+1,i] - T[j,i]) / dz

            dpsidx = (psi[j,i+1] - psi[j,i]) / dx
            dpsidz = (psi[j+1,i] - psi[j,i]) / dz

            b[j, i] = dpsidz * dTdx - dpsidx * dTdz

    return b
```

Next, we can solve the governing eqs. (3) using the iterative numerical method described above with the steepest descent method in the function `mantle_convection_bvp`

```python
# 2D thermal mantle convection code with JIT optimization
# -------------------------------------------------------
def mantle_convection_bvp(T0, psi0, w0, niter, dx, dz, Ra, tol):
    """
    Computes and returns the stationary temperature, streamfunction
    and vorticity fields for the 2D thermal mantle convection problem
    using the steepest descent method.

    Parameters
    ----------
    T0 : numpy.ndarray
        The initial temperature distribution as a 2D array of floats.
    psi0 : numpy.ndarray
        The initial streamfunction as a 2D array of floats.
    w0 : numpy.ndarray
        The initial vorticity as a 2D array of floats.
    niter : integer
        Maximum number of iterations using the steepest descent method.
    dx : float
        Grid spacing in the x direction.
    dz : float
        Grid spacing in the z direction.
    Ra : float
        Rayleigh number.
    tol : float
        tolerance for abortion criterion.

    Returns
    -------
    T : numpy.ndarray
        The temperature distribution as a 2D array of floats.
    psi : numpy.ndarray
        The streamfunction as a 2D array of floats.
    w : numpy.ndarray
        The vorticity as a 2D array of floats.
    """

    # Define some constants.
    dx2 = dx**2
    dz2 = dz**2

    # Copy fields
    T = T0.copy()
    psi = psi0.copy()
    w = w0.copy()

    r = numpy.zeros_like(psi)    # initial residuals
    Ar = numpy.zeros_like(psi)   # store mat-vec multiplication Ar

    # Estimate number of grid points in x- and y-direction
    nz, nx = T.shape

    # Stabilization factor eps
    eps = 0.

    # Iterate solution using the steepest descent method
    for n in range(niter):

        # 1. Compute new vorticity from previous temperature
        # --------------------------------------------------
        # store old T
        wk = w.copy()

        # Compute the residual.
        r = bw(T,nx,nz,dx,Ra) - Alapl(w,nx,nz,dx2,dz2)

        # Compute the Laplacian of the residual.
        Ar = Alapl(r,nx,nz,dx2,dz2)

        # Compute the step size.
        alpha = numpy.sum(r * r) / (numpy.sum(r * Ar) + eps)

        # Update the vorticity solution.
        w = wk + alpha * r

        # 2. Compute new streamfunction from new vorticity
        # ------------------------------------------------
        # store old T
        psik = psi.copy()

        # Compute the residual.
        r = - w - Alapl(psi,nx,nz,dx2,dz2)

        # Compute the Laplacian of the residual.
        Ar = Alapl(r,nx,nz,dx2,dz2)

        # Compute the step size.
        alpha = numpy.sum(r * r) / (numpy.sum(r * Ar) + eps)

        # Update the streamfunction solution.
        psi = psik + alpha * r

        # 3. Compute new temperature from new streamfunction
        # --------------------------------------------------
        # store old T
        Tk = T.copy()

        # Compute the residual.
        r = bT(T,psi,nx,nz,dx,dz) - Alapl(T,nx,nz,dx2,dz2)

        # Compute the Laplacian of the residual.
        Ar = Alapl(r,nx,nz,dx2,dz2)

        # Compute the step size.
        alpha = numpy.sum(r * r) / (numpy.sum(r * Ar) + eps)

        # Update the temperature solution.
        T = Tk + alpha * r

        # Apply Neumann boundary condition (dTdx=0) at left and right boundary
        T[:,0] = T[:,1]
        T[:,-1] = T[:,-2]

        # Compute relative L2 norm of temperature and streamfunction
        ET = LA.norm(T-Tk, 2) / LA.norm(T, 2)
        Epsi = LA.norm(psi-psik, 2) / LA.norm(psi, 2)

        # check if tolerance criterion is satisfied
        if(max(ET,Epsi) <= tol):
            print("ET = ", ET)
            print("Final Epsi = ", Epsi)
            print("n = ", n)
            break

        if(n==niter-1):
            print("Final ET = ", ET)
            print("Final Epsi = ", Epsi)

    return T, w, psi
```

Let's assume that the dimensionless width of the lithosphere is $ L_x = 1$ in x-direction and it has a thickness of $L_z = 1$. The model is discretized with $nx=101$ gridpoints in x-direction and $ny=101$ gridpoints in z-direction. Regarding the thermal properties, assume a dimensionless temperature of $\text{Tbot} = 1$ at the bottom of the mantle and a temperature $\text{Tsurf} = 0$ at the top boundary.

Furthermore, we have to make an initial guess for the temperature field $T_0$ in the mantle. Let 's assume it consists of two parts:

$$
T_0 = T_{lin} - T_{2D}
$$

a 1D linear temperature gradient

$$
T_{lin} = \frac{(Tbot-Tsurf)}{Lz} * Z + Tsurf
$$

and a 2D temperature variation in x-direction:

$$
T_{2D} = p\; cos\biggl(\frac{k \pi x}{L_x}\biggr) sin(\pi z)
$$

Does the 2D temperature distribution look familiar? You have already encountered it, when dealing with the [Lorenz equations](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/03_Lorenz_equations/01_LorenzEquations_intro.ipynb). It was one of the modes of the highly truncated Fourier series of the temperature field.

##### Exercise 1

Why can we not just use the simple linear increasing temperature gradient $T_{lin}$ from top to bottom as initial guess for the temperature field? Hint: take a closer look at the algorithm described in the section "Numerical Solution of the 2D thermal mantle convection boundary value problem" above.

Regarding the parameters $p$ and $k$, I used $p=0.01$ and $k=1$ from the [manual of the mantle convection code ASPECT](http://www.math.clemson.edu/~heister/manual.pdf). For the initial streamfunction and vorticity, we assume $\psi_0(x,z)=\omega_0(x,z)=0$. Furthermore, Dirichlet boundary condtions for $psi$ and $\omega$ are set to

\begin{equation}
\begin{split}
\psi(0,z) &= \psi(L_x,z) = \psi(x,0) = \psi(x,L_z) = 0\\
\omega(0,z) &= \omega(L_x,z) = \omega(x,0) = \omega(x,L_z) = 0\\
\end{split} \notag
\end{equation}

For the temperature distribution $T$ Dirichlet boundary condtions are assumed on the top and bottom boundary:
\begin{equation}
\begin{split}
T(x,0) &= \text{Tsurf} = 0\\
T(x,L_z) &= \text{Tbot} = 1\\
\end{split}\notag
\end{equation}

Neumann boundary conditions are set at the left and right boundary

\begin{equation}
\begin{split}
&\frac{\partial T}{\partial x}(0,z) = 0\\
&\frac{\partial T}{\partial x}(L_x,z) = 0\\
\end{split}\notag
\end{equation}

```python
# Definition of modelling parameters
# ----------------------------------
Lx = 1.0     # width of the mantle in the x direction []
Lz = 1.0     # thickness of the mantle in the z direction []
nx = 101     # number of points in the x direction
nz = 101     # number of points in the z direction
dx = Lx / (nx - 1)  # grid spacing in the x direction []
dz = Lz / (nz - 1)  # grid spacing in the z direction []

# Define the locations along a gridline.
x = numpy.linspace(0.0, Lx, num=nx)
z = numpy.linspace(0.0, Lz, num=nz)

# Define initial temperature guess
Tbot = 1.     # temperature at the bottom boundary []
Tsurf = 0.    # temperature at the top boundary []

X, Z = numpy.meshgrid(x,z) # coordinates X,Z required to define T0

# Assume 1D linear increasing temperature profile with 2D perturbation
Tlin = (Tbot-Tsurf)/Lz * Z + Tsurf # linear temperature increase with depth

# 2D temperature perturbation in x-direction with parameters from
# the ASPECT mantle convection manual
p = 0.01
k = 1.
T2D = p * numpy.cos(k * numpy.pi * X / Lx) * numpy.sin(numpy.pi * Z)

# Initial temperature distribution
T0 = Tlin - T2D

# Apply Dirichlet boundary condition for the temperature
# at the top and bottom boundary
T0[0,:] = Tsurf
T0[-1,:] = Tbot

# Define initial streamfunction guess
psi0 = numpy.zeros((nz,nx))

# Define initial vorticity guess
w0 = numpy.zeros((nz,nx))
```

After setting all modelling parameters, we can run the `mantle_convection_bvp` modelling code to compute the numerical solution after `niter=10000` iterations for a Rayleigh number `Ra=1e5` above the critical Rayleigh number:

```python
# Define maximum number of iterations niter and tolerance tol
niter = 10000
tol = 1e-4

# Define Rayleigh number []
Ra = 1e5

# Compute temperature, streamfunction and temperature distribution
T, w, psi = mantle_convection_bvp(T0, psi0, w0, niter, dx, dz, Ra, tol)
```

    Final ET =  0.0006510623946767165
    Final Epsi =  0.001647934084594864

In order to visualize the temperature and velocity fields, we first compute the velocities from the streamfunction ...

```python
# Compute velocity field from stream function with JIT-optimization
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def stream2uw(psi,nx,nz,dx,dz):

    # initialize u and w velocity components
    u = numpy.zeros_like(psi)
    w = numpy.zeros_like(psi)

    # loop over spatial grid
    for i in range(1,nx-1):
        for j in range(1,nz-1):

            # compute velocity components u, w from streamfunction
            u[j, i] = (psi[j+1, i] - psi[j, i]) / dz
            w[j, i] = -(psi[j, i+1] - psi[j, i]) / dx

    return u, w
```

```python
u, w = stream2uw(psi,nx,nz,dx,dz)
```

... and use the `quiver` function to plot the velocity field and `contourf` to plot the temperature

```python
# Filled contour of the temperature distribution
# and quiver plot of the velocity field
pyplot.figure(figsize=(7.5, 6.0))
pyplot.xlabel('x []')
pyplot.ylabel('z []')
levels = numpy.linspace(numpy.min(T), numpy.max(T), num=201)
contf = pyplot.contourf(x, z, T, levels=levels, cmap='jet')
cbar = pyplot.colorbar(contf)
cbar.set_label('Temperature []')

# define new spatial sample interval for quiver plot
pyplot.streamplot(X, Z, u, w, linewidth=1, color='k')
pyplot.axis('scaled', adjustable='box');
pyplot.gca().invert_yaxis()
pyplot.xlim(0,Lx)
```

The resulting temperature and velocity field for `Ra=1e5` after 10000 iterations of the steepest descent method shows a convection roll rotating in counter-clockwise direction. Hot material is rising at the right boundary, cools at the top and descends at the left boundary, to be heated again at the bottom.

##### Exercise 2

Compare the numerical solution of eqs. (3) above with the stationary solutions, you computed for the Lorenz problem in [this notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/03_Lorenz_equations/02_Stationary_solutions_of_DE.ipynb) for a Rayleighnumber 28x larger than the critical Rayleighnumber. You might have noticed, that the convergence criterion of the `mantle_convection_bvp` modelling code was not satisfied after `niter=10000` iterations. Run the code again with `niter=20000`. Visualize the results, compare them again with the stationary solutions of the Lorenz equations. Make an assumption, why the abortion criterion was still not satisfied after 20000 iterations. Hint: remember the dynamical evolution of the Lorenz equations for different Rayleighnumbers, you computed in [this notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/03_Lorenz_equations/03_LorenzEquations_fdsolve.ipynb).

```python
# Define maximum number of iterations niter and tolerance tol
niter = 20000
tol = 1e-4

# Define Rayleigh number []
Ra = 1e5

# Compute temperature, streamfunction and temperature distribution
T, w, psi = mantle_convection_bvp(T0, psi0, w0, niter, dx, dz, Ra, tol)
```

In order to visualize the temperature and velocity fields, we first compute the velocities from the streamfunction ...

```python
u, w = stream2uw(psi,nx,nz,dx,dz)
```

... and use the `quiver` function to plot the velocity field and `contourf` to plot the temperature

```python
# Filled contour of the temperature distribution
# and quiver plot of the velocity field
pyplot.figure(figsize=(7.5, 6.0))
pyplot.xlabel('x []')
pyplot.ylabel('z []')
levels = numpy.linspace(numpy.min(T), numpy.max(T), num=201)
contf = pyplot.contourf(x, z, T, levels=levels, cmap='jet')
cbar = pyplot.colorbar(contf)
cbar.set_label('Temperature []')

# define new spatial sample interval for quiver plot
pyplot.streamplot(X, Z, u, w, linewidth=1, color='k')
pyplot.axis('scaled', adjustable='box');
pyplot.gca().invert_yaxis()
pyplot.xlim(0,Lx)
```

##### Exercise 3

What happens if you set the Rayleighnumber below the critical Rayleigh number, let 's say to `Ra = 1e1`? Again, describe the modelling result and compare with the stationary solution of the Lorenz equation for sub-critical Rayleigh numbers.

```python
# Define maximum number of iterations niter and tolerance tol
niter = 20000
tol = 1e-4

# Define Rayleigh number []
Ra = 1e1

# Compute temperature, streamfunction and temperature distribution
T, w, psi = mantle_convection_bvp(T0, psi0, w0, niter, dx, dz, Ra, tol)
```

In order to visualize the temperature and velocity fields, we first compute the velocities from the streamfunction ...

```python
u, w = stream2uw(psi,nx,nz,dx,dz)
```

... and use the `quiver` function to plot the velocity field and `contourf` to plot the temperature

```python
# Filled contour of the temperature distribution
# and quiver plot of the velocity field
pyplot.figure(figsize=(7.5, 6.0))
pyplot.xlabel('x []')
pyplot.ylabel('z []')
levels = numpy.linspace(numpy.min(T), numpy.max(T), num=201)
contf = pyplot.contourf(x, z, T, levels=levels, cmap='jet')
cbar = pyplot.colorbar(contf)
cbar.set_label('Temperature []')

# define new spatial sample interval for quiver plot
pyplot.streamplot(X, Z, u, w, linewidth=1, color='k')
pyplot.axis('scaled', adjustable='box');
pyplot.gca().invert_yaxis()
pyplot.xlim(0,Lx)
```

## What we learned:

* How to compute and visualize steady state solutions for the 2D thermal convection problem using the steepest descent method.


---
layout: page
title: Regressão Linear
nav_order: 15
---

[](https://colab.research.google.com/github/icd-ufmg/icd-ufmg.github.io/blob/master/_lessons/15-linear.ipynb)

# Regressão Linear Simples

{: .no_toc .mb-2 }

Entendimento de relação entre dados.
{: .fs-6 .fw-300 }

{: .no_toc .text-delta }
Resultados Esperados

1. Revisitar os mínimos quadrados (ALC)
1. Entender a regressão linear do ponto de vista estatístico (diferente de ALC)
1. Entender o coeficiente de pearson e o R2.
1. Executar regressões em python

---
**Sumário**
1. TOC
{:toc}
---

```python
# -*- coding: utf8

from scipy import stats as ss

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
```

```python
plt.style.use('seaborn-colorblind')
plt.rcParams['figure.figsize']  = (16, 10)
plt.rcParams['axes.labelsize']  = 20
plt.rcParams['axes.titlesize']  = 20
plt.rcParams['legend.fontsize'] = 20
plt.rcParams['xtick.labelsize'] = 20
plt.rcParams['ytick.labelsize'] = 20
plt.rcParams['lines.linewidth'] = 4
```

```python
plt.ion()
```

```python
def despine(ax=None):
    if ax is None:
        ax = plt.gca()
    # Hide the right and top spines
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)

    # Only show ticks on the left and bottom spines
    ax.yaxis.set_ticks_position('left')
    ax.xaxis.set_ticks_position('bottom')
```

## Introdução

Antes de iniciar o nosso estudo da regressão, vamos pensar em um modelo que tenta capturar o valor médio no eixo y. Para uma base de dados unidimensional (apenas Y), a média ($\bar{y}$) é um bom estimador. Isto é, imagine que você está em uma sala de aula e entra um novo aluno na sala. Posso prever alguma característica deste aluno (por exemplo, o seu rendimento escolar) usando a média dos outros alunos em sala de aula.

Agora, como podemos proceder quando temos duas variáveis X e Y? Em uma aula anterior, usamos a função de correlação para medir a força da relação linear entre duas variáveis. Para a maioria das aplicações, saber que esse relacionamento linear existe não é suficiente. Queremos ser capazes de entender a natureza do relacionamento. É aqui que vamos usar a regressão linear simples.

Vamos olhar para uma bas bases dados que ajudaram a criar a regressão linear.

## Poor Person's KNN  (Vizinhos Próximos Implementado de Forma Simples)

Vamos iniciar com a base de dados original analisada por [Galton](https://en.wikipedia.org/wiki/Francis_Galton). Tal base captura altura de um filho, após alguns anos, com base na altura média dos pais. O problema sendo estudado era prever qual vai ser a altura futura de uma criança usando apenas a idade dos pais como entrada. Como temos um pai e uma mãe, usamos a média das alturas entre os dois. Então temos que:

1. X é a altura média dos pais
1. Y é a altura da criança após alguns anos (o mesmo número de anos para toda criança)

```python
df = pd.read_csv('https://media.githubusercontent.com/media/icd-ufmg/material/master/aulas/16-RegressaoLinear/galton.csv')
df.head()
```

      family
      father
      mother
      midparentHeight
      children
      childNum
      gender
      childHeight

      0
      1
      78.5
      67.0
      75.43
      4
      1
      male
      73.2

      1
      1
      78.5
      67.0
      75.43
      4
      2
      female
      69.2

      2
      1
      78.5
      67.0
      75.43
      4
      3
      female
      69.0

      3
      1
      78.5
      67.0
      75.43
      4
      4
      female
      69.0

      4
      2
      75.5
      66.5
      73.66
      4
      1
      male
      73.5

Abaixo temos a dispersão dos dados

```python
x = df['midparentHeight'].values
y = df['childHeight'].values

plt.scatter(x, y, alpha=0.8, edgecolors='k', s=80)
plt.xlabel('Altura média dos pais')
plt.ylabel('Altura dos filhos')
plt.title('Regressão linear simples')
despine()
```

Uma forma de capturar a correlação é fazer uma regressão de vizinhos próximos (nearest neighbors). Para tal, precisamos agrupar, no eixo-x, pontos próximos um dos outros. É possível fazer o mesmo nos eixos x e y usando uma distância euclideana. Por simplicidade vamos agrupar apenas em x. No caso particular destes dados onde os valores são bem espalhados no eixo x, podemos simplesmente arredondar os números

```python
copy = df[['midparentHeight', 'childHeight']].copy()
copy['midparentHeight'] = copy['midparentHeight'].round()
```

Agrupar por x e tirar a média de y.

```python
model = copy.groupby('midparentHeight').mean()
model.head()
```

      childHeight

      midparentHeight

      64.0
      63.000000

      65.0
      64.229630

      66.0
      64.350000

      67.0
      65.535632

      68.0
      66.240458

Observando o modelo.

```python
x_nn = model.index
y_nn = model.values
plt.scatter(x, y, alpha=0.8, edgecolors='k', s=80)
plt.plot(x_nn, y_nn, color='magenta')
plt.xlabel('Altura média dos pais')
plt.ylabel('Altura dos filhos')
plt.title('Regressão linear simples')
```

Previsão: Para um novo ponto, arredondar seu peso. Ver valor de y_nn

```python
idade_media_pais = 71
model.loc[71.0]
```

    childHeight    67.740594
    Name: 71.0, dtype: float64

Observe que o modelo acima é muito perto de uma reta. Este é um dos pontos e partidas para entender a correlação linear. Precisamos de alguma forma:

1. Capturar a média de valores próximos um dos outros em x
1. Capturar a média de valores próximos um dos outros em y

Resolvemos o (1) problema nos dados acima com um round. No caso (2) com um group-by e mean. Observe como o modelo é quase o mesmo quando z-normalizamos os dados.

```python
znormed = df[['midparentHeight', 'childHeight']].copy()
znormed = (znormed - znormed.mean()) / znormed.std(ddof=1)
znormed.head()
```

      midparentHeight
      childHeight

      0
      3.452802
      1.803190

      1
      3.452802
      0.685637

      2
      3.452802
      0.629760

      3
      3.452802
      0.629760

      4
      2.470762
      1.887006

```python
x = znormed['midparentHeight'].copy()
y = znormed['childHeight'].copy()
```

```python
znormed['midparentHeight'] = znormed['midparentHeight'].round()
model = znormed.groupby('midparentHeight').mean()
```

```python
x_nn = model.index
y_nn = model.values
plt.scatter(x, y, alpha=0.8, edgecolors='k', s=80)
plt.plot(x_nn, y_nn, color='magenta')
plt.xlabel('Altura média dos pais')
plt.ylabel('Altura dos filhos')
plt.title('Regressão linear simples')
despine()
```

# Regressão Linear Simples

Nos slides da aula e no material do [](inferentialthinking.org), leia a discussão sobre como uma regressão linear tenta capturar a mesma ideia do nosso KNN simples acima. De forma simples, queremos uma reta possível de explicar tando a dispersão em x quanto em y. Tal reta deve se aproximar da média dos dois valores quando vistos em janelas (estilo o NN acima). Em particular, você supõe que existem constantes $\alpha$ (alfa) e $\beta$ (beta) tais que:

$$\hat{y}_i = \beta x_i + \alpha + \epsilon_i$$

Idealmente, vamos conseguir capturar o valor médio de $y_i$ para grupos de valores próximos em $x_i$. $\epsilon_i$ é um termo de erro (esperançosamente pequeno) que representa o fato de que existem outros fatores não explicados por este modelo simples.

Como escolhemos alfa e beta? Bem, qualquer escolha de alfa e beta nos dá uma saída prevista para cada entrada $x_i$. Como sabemos a saída real $y_i$, podemos calcular o erro de cada par. Computa o valor $\epsilon_i$.

```python
def error(alpha, beta, x, y):
    model = beta * x + alpha
    return y - model
```

Observe o erro médio para um primeiro modelo

```python
x = df['midparentHeight'].values
y = df['childHeight'].values

alpha = 1
beta = 0.5
e = error(alpha, beta, x, y)
e.mean()
```

    31.142544967880085

```python
x = df['midparentHeight'].values
y = df['childHeight'].values

y_model = beta * x + alpha

plt.scatter(x, y, alpha=0.8, edgecolors='k', s=80)
plt.plot(x, y_model, color='magenta')
plt.xlabel('Altura média dos pais')
plt.ylabel('Altura dos filhos')
plt.title('Regressão linear simples')
```

Agora para um segundo.

```python
alpha = 22
beta = 0.5
e = error(alpha, beta, x, y)
e.mean()
```

    10.142544967880088

```python
x = df['midparentHeight'].values
y = df['childHeight'].values

y_model = beta * x + alpha
plt.scatter(x, y, alpha=0.8, edgecolors='k', s=80)
plt.plot(x, y_model, color='magenta')
plt.xlabel('Altura média dos pais')
plt.ylabel('Altura dos filhos')
plt.title('Regressão linear simples')
```

O segundo modelo é bem melhor. Porém os dois são péssimos. Para facilitar a nossa vida, vamos fazer uso da soma dos erros ao quadrado. Isto vai nos ajudar nas derivadas.

Somamos os erros quadrados:

```python
def sum_of_squared_errors(alpha, beta, x, y):
    return (error(alpha, beta, x, y) ** 2).sum()
```

A *solução de mínimos quadrados (least squares solution)*  é escolher o alfa e o beta que tornam o `sum_of_squared_errors` o menor possível. Para chegar em tais valores, podemos ver o erro como uma função de $\alpha$, $\beta$. Vamos iniciar com o caso normalizado. Aqui $\alpha=0$.

### Dados Normalizados

Lembre-se que na z-normalização, os dados de X e Y ficam com média 0 e desvio 1. Em particular: $x'_i = \frac{x_i - \bar{x}}{s_x}$ e $y'_i = \frac{y_i - \bar{y}}{s_y}$. Note como a média dos dados tanto em X quanto em Y é zero:

$$\bar{x'} = n^{-1} \sum_i \frac{x_i - \bar{x}}{s_x}$$

$$\bar{x'} = n^{-1} s^{-1}_x (\sum_i x_i - \bar{x} \sum_i 1)$$

$$\bar{x'} = n^{-1} s^{-1}_x (\sum_i x_i - \bar{x} n)$$

$$\bar{x'} = n^{-1} s^{-1}_x (\sum_i x_i - x_i)$$

$$\bar{x'} = n^{-1} s^{-1}_x (0) = 0$$

Além do mais, o desvio padrão é 1. Provando:

$$s_{x'} = n^{-1} \sum_i (\frac{x_i - \bar{x}}{s_x} - 0)^2$$
$$s_{x'} = s^{-1}_x n^{-1} \sum_i (x_i - \bar{x})^2$$
$$s_{x'} = s^{-1}_x s_x = 1$$

Como a média dos pontos é (0, 0), a nossa melhor reta vai passar pela origem. Ou seja, $\alpha=0$. Além do mais,  cada ponto dos seus dados é uma reta entre (0, 0) e o ponto $(x'_i, y'_i)$. Neste caso, resolver o problema abaixo para chegar na equação da reta:

$$L(\beta) = \sum_i (y'_i - \hat{y}'_i)^2$$

$$L(\beta) = \sum_i (y'_i - \beta x'_i)^2$$

$$L(\beta) = \sum_i y'^2_i - 2 \beta x'_i y'_i + \beta^2 x'^2_i$$

Derivando a função em $\beta$:

$$\frac{d}{d\beta} L(\Theta) = -\sum_i 2 x'_i y'_i + 2 \beta x'^2_i$$

Fazendo $\frac{d}{d\beta}=0$:

$$\beta = \frac{\sum_i x'_i y'_i}{x'^2_i}$$

Nos slides discutimos como tal solução captura é uma função da covariância dos dados.

### Dados Originais

Agora vamos resolver sem normalizar os dados. Vamos definir $\Theta = [\alpha, \beta]$, isto é, um vetor com alfa e beta.

$$L(\Theta) = \sum_i (y_i - \hat{y}_i)^2$$

$$L(\Theta) = \sum_i (y_i - \beta x_i + \alpha)^2$$

Resolvendo o quadrado dentro do somatório temos:

$$L(\Theta) = \sum_i (y_i^2 + \alpha^2 + x_i^2\beta^2 - 2 \beta \alpha x_i - 2 \beta x_i y_i + 2\alpha y_i)$$

Derivando em relação a $\alpha$:

$$\frac{d}{d\alpha} L(\Theta)= \sum_i (2\alpha - 2 \beta x_i + 2\beta y_i)$$

Derivando em relação a $\beta$:

$$\frac{d}{d\beta} L(\Theta) = \sum_i (2x_i^2\beta - 2 \alpha x_i + 2 x_i y_i)$$

Para resolver otimizar esta função precisamos setar as duas derivadas para zero.

$$\frac{d}{d\alpha} L(\Theta) = 0$$

$$\frac{d}{d\beta} L(\Theta) = 0$$

Após isto, podemos resolver os valores ótimos de $\alpha$ e $\beta$. Note que esta é uma otimização um pouco chata de resolve.

\begin{align}
 \alpha & = \bar{y} - \beta\,\bar{x}, \\[5pt]
  \beta &= \frac{ \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^n (x_i - \bar{x})^2 } \\[6pt]
            &= \frac{ \operatorname{Cov}(x, y) }{ \operatorname{Var}(x) } \\[5pt]
            &= r_{xy} \frac{s_y}{s_x}. \\[6pt]
\end{align}

```python
def least_squares_fit(x, y):
    correlationxy = ss.pearsonr(x, y)[0]
    beta = correlationxy * np.std(y, ddof=1) / np.std(x, ddof=1)
    alpha = np.mean(y) - beta * np.mean(x)
    return alpha, beta
```

Sem passar pela matemática exata, vamos pensar em por que isso pode ser uma solução razoável. A escolha do alfa simplesmente diz que quando vemos o valor médio da variável independente $x$, predizemos o valor médio da variável dependente $y$.

A escolha de beta significa que quando o valor de entrada aumenta pelo desvio padrão de $x$, a previsão de $y$ aumenta pela correlação entre $x$ e $y$ multiplicada pelo desvio padrão de $y$ (`correlationxy * np.std(y)`). No caso em que $x$ e $y$ estão perfeitamente correlacionados, um aumento de um desvio padrão em $x$ resulta em um aumento de um desvio padrão de $y$ na predição. Quando eles são perfeitamente anti-correlacionados, o aumento em $x$ resulta em uma *diminuição* no valor da previsão. E quando a correlação é *zero*, o beta é *zero*, o que significa que as alterações em $x$ não afetam a previsão.

Para um pouco mais sobre a intuição dessa fórmula, veja este [vídeo](https://www.khanacademy.org/math/ap-statistics/bivariate-data-ap/least-squares-regression/v/calculating-the-equation-of-a-regression-line).

```python
x = df['midparentHeight'].values
y = df['childHeight'].values
alpha, beta = least_squares_fit(x, y)
y_model = beta * x + alpha
plt.scatter(x, y, alpha=0.8, edgecolors='k', s=80)
plt.plot(x, y_model, color='magenta')
plt.xlabel('Altura média dos pais')
plt.ylabel('Altura dos filhos')
plt.title('Regressão linear simples')
```

```python
alpha, beta
```

    (22.636240549589736, 0.6373608969694791)

Isso fornece valores de alfa = 22,63 e beta = 0,63. Portanto, podemos dizer que crianças vão ser aproximadamente 22 polegadas maiores do que a 63\% da média da altura dos seus pais.

É claro que precisamos de uma maneira melhor de descobrir o quão bem nós ajustamos os dados do que simplesmente encarar o gráfico da regressão. Uma medida comum é o coeficiente de determinação (ou R-quadrado, ou $R^2$, ou R-dois), que mede a fração da variação total na variável dependente (o $y$) que é capturada pelo modelo:

```python
def total_sum_of_squares(y):
    return ((y - y.mean()) ** 2).sum()

def r_squared(alpha, beta, x, y):
    return 1.0 - (sum_of_squared_errors(alpha, beta, x, y) /
                  total_sum_of_squares(y))

```

```python
r_squared(alpha, beta, x, y)
```

    0.10300883578346631

Agora, escolhemos o alfa e o beta que minimizaram a soma dos quadrados dos erros de previsão. Um modelo linear que poderíamos ter escolhido é "sempre prever a média de $y$" (correspondendo a `alpha = np.mean(y)` e `beta = 0`), cuja soma dos erros quadrados é exatamente igual a sua soma total de quadrados. Isso significa um $R^2$ de zero, que indica um modelo que (obviamente, neste caso) não funciona melhor do que apenas prever a média.

Claramente, o modelo de mínimos quadrados deve ser pelo menos tão bom quanto esse, o que significa que a soma dos erros quadrados é no máximo a soma total de quadrados, o que significa que o $R^2$ deve ser pelo menos zero. E a soma dos erros quadrados deve ser pelo menos $0$, o que significa que o $R^2$ pode ser no máximo $1$.

Quanto maior for o $R^2$, melhor o nosso modelo se ajusta aos dados. Aqui nós calculamos um $R^2$ de $0.10$, o que nos diz que o nosso modelo está fraco ajustar os dados, e que claramente existem outros fatores em jogo.

Observe como o modelo para, visualmente, bem pior ao plotar em outra escala. Existe muita variabilidade no eixo-y para ser capturada.

```python
x = df['midparentHeight'].values
y = df['childHeight'].values
alpha, beta = least_squares_fit(x, y)
y_model = beta * x + alpha
plt.scatter(x, y, alpha=0.8, edgecolors='k', s=80)
plt.plot(x, y_model, color='magenta')
plt.xlabel('Altura média dos pais')
plt.ylabel('Altura dos filhos')
plt.title('Regressão linear simples')
plt.xlim((55, 80))
plt.ylim((55, 80))
```

Por fim, a raiz de $R^2$ é a correlação.

```python
r_squared(alpha, beta, x, y) ** 0.5
```

    0.32094989606395935

```python
ss.pearsonr(x, y)[0]
```

    0.3209498960639593

Formas de interpretar o $R^2$.

1. Quão melhor é o meu modelo de um que retorna apenas a média de y.
1. Quanto da variancia de y é explicada por x. Apenas 10% acima.

Para entender o segundo ponto, o `total_sum_of_squares` é proporcional a variância. Na verdade é $s_y$ * (n-1). Ou seja, sem tirar a média dos desvios quadrados. Sendo $\hat{y}_i$ a nossa previsão.

$$R^2 = \sum_i \frac{(y_i - \bar{y})^2 - (y_i - \hat{y}_i)^2}{(y_i - \bar{y})^2}$$

$(y_i - \bar{y})^2$ define nossa variância. Quanto dela está em $(y_i - \hat{y}_i)^2$. Esta é outra forma de pensar no $R^2$.

## Bootstrap na Regressão

Lembre-se que nossa regressão foi realizada em uma amostra. Portanto, qualque estatística que temos vêm com erros. Os coeficientes da nossa regressão: $\alpha$ e $\beta$ têm erros. Além da correlação $r$ e do $R^2$. Como saber a variablidade de tais estimativas? Bootstrap. O código abaixo faz um Bootstrap de: $\alpha$, $\beta$ e $R^2$.

```python
def bootstrap_regression(df, n=10000, size=None):
    if size is None:
        size = len(df)

    alphas = np.zeros(n)
    betas = np.zeros(n)
    rs = np.zeros(n)

    for i in range(n):
        sample = df.sample(size, replace=True)
        x = sample['midparentHeight'].values
        y = sample['childHeight'].values
        alpha, beta = least_squares_fit(x, y)

        alphas[i] = alpha
        betas[i] = beta
        rs[i] = ss.pearsonr(x, y)[0] ** 2
    return alphas, betas, rs
```

```python
alphas, betas, rs = bootstrap_regression(df)
```

Plotando para alfa

```python
plt.hist(alphas, edgecolor='k')
lower = np.percentile(alphas, 2.5)
upper = np.percentile(alphas, 97.5)
despine()
print(alphas.mean())
print(lower, upper)
```

Beta

```python
plt.hist(betas, edgecolor='k')
lower = np.percentile(betas, 2.5)
upper = np.percentile(betas, 95)
despine()
print(betas.mean())
print(lower, upper)
```

R2

```python
plt.hist(rs, edgecolor='k')
lower = np.percentile(rs, 2.5)
upper = np.percentile(rs, 97.5)
despine()
print(rs.mean())
print(lower, upper)
```

## Statsmodels

Observe como entender tal ideia é interessante. Quanto usamos bibliotecas mais avançadas, tipo o statsmodels, os resultados descrevem: ICs para cada parâmetro e valores-p. No notebook anterior computamos um valor-p da correlação com permutação. Podemos fazer o mesmo para alfa e beta. Classicamente, existem estatísticas para tais valores como a $F$ measure que permitem ics e valores-p teóricos.

```python
import statsmodels.api as sm

stocks = {'Year': [2017,2017,2017,2017,2017,2017,2017,2017,2017,2017,2017,2017,2016,2016,2016,2016,2016,2016,2016,2016,2016,2016,2016,2016],
          'Month': [12, 11,10,9,8,7,6,5,4,3,2,1,12,11,10,9,8,7,6,5,4,3,2,1],
          'Interest_Rate': [2.75,2.5,2.5,2.5,2.5,2.5,2.5,2.25,2.25,2.25,2,2,2,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75,1.75],
          'Unemployment_Rate': [5.3,5.3,5.3,5.3,5.4,5.6,5.5,5.5,5.5,5.6,5.7,5.9,6,5.9,5.8,6.1,6.2,6.1,6.1,6.1,5.9,6.2,6.2,6.1],
          'Stock_Index_Price': [1464,1394,1357,1293,1256,1254,1234,1195,1159,1167,1130,1075,1047,965,943,958,971,949,884,866,876,822,704,719]
         }

df = pd.DataFrame(stocks, columns=['Year','Month', 'Interest_Rate', 'Unemployment_Rate', 'Stock_Index_Price'])
```

Oberver o IC da constante (alfa) abaixo. Note que a mesma não é siginificativa. Como o IC abrange números negativos e positivos, não podemos descartar um zero como sendo a média.

```python
X = df['Interest_Rate']
Y = df['Stock_Index_Price']

X = sm.add_constant(X) # adding a constant

model = sm.OLS(Y, X).fit()
predictions = model.predict(X)

print_model = model.summary()
print(print_model)
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:      Stock_Index_Price   R-squared:                       0.876
    Model:                            OLS   Adj. R-squared:                  0.870
    Method:                 Least Squares   F-statistic:                     155.0
    Date:                Wed, 03 Feb 2021   Prob (F-statistic):           1.95e-11
    Time:                        18:11:50   Log-Likelihood:                -136.94
    No. Observations:                  24   AIC:                             277.9
    Df Residuals:                      22   BIC:                             280.2
    Df Model:                           1
    Covariance Type:            nonrobust
    =================================================================================
                        coef    std err          t      P>|t|      [0.025      0.975]
    ---------------------------------------------------------------------------------
    const           -99.4643     95.210     -1.045      0.308    -296.918      97.990
    Interest_Rate   564.2039     45.317     12.450      0.000     470.221     658.186
    ==============================================================================
    Omnibus:                        5.361   Durbin-Watson:                   0.514
    Prob(Omnibus):                  0.069   Jarque-Bera (JB):                3.647
    Skew:                          -0.927   Prob(JB):                        0.161
    Kurtosis:                       3.459   Cond. No.                         15.8
    ==============================================================================

    Notes:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.


# Systems

Numerous dynamic systems are implemented in Underworld. They may be found within the `systems` submodule. For specific information on the different system classes, the user is directed to the API reference available at the Underworld documentation site:

[http://underworld2.readthedocs.io/](http://underworld2.readthedocs.io/)

We will consider here the basic workflow for creating and configuring an Underworld numerical system. The general process is as follows:

1. Create your mesh.
1. Create any required field(s) on the mesh (such as a temperature field).
1. Create any required boundary condition objects.
1. Create function objects to define any required physical quantities.
1. Create system.
1. Create solver and solve system.

Note that for systems that are solved many times (for perhaps time-stepping), you will generally only create the system (and solver) once, and then solve it numerous times.

## Boundary Conditions

Boundary conditions form part of your known information for the system you are creating. In Underworld, these *knowns* are either of the Dirichlet (fixed value) or Neumann (fixed gradient) type, and are applied on a per node **and** per degree of freedom (DOF) basis. Note also that the entire domain boundary must be piecemeal either Dirichlet **or** Neumann type, but no section of boundary can be both Dirichlet and Neumann. Sections of the boundary that do not have any BCs explicitly set by the user will implicitly be of Neumann type (with zero gradient).

To give this discussion a mathematical grounding, let's consider the heat equation. The strong form of the boundary value problem is

$$
\begin{align}
q_i =& - \alpha \, u_{,i}  & \\
q_{i,i} =& \: f  & \text{ in }  \Omega \\
u =& \: g & \text{ on }  \Gamma_g \\
-q_i n_i =& \: h & \text{ on }  \Gamma_h \\
\end{align}
$$

where, $\alpha$ is the diffusivity, $u$ is the temperature,
$f$ is a source term, $g$ is the Dirichlet condition, and
$h$ is a Neumann condition. The problem boundary, $\Gamma$,
admits the decomposition $\Gamma=\Gamma_g\cup\Gamma_h$ where
$\emptyset=\Gamma_g\cap\Gamma_h$.

### The Dirichlet Condition

The Dirichlet condition part of the problem specification requires that our solution temperature $u$ is equal to the user specified function $g$ across the user specified boundary $\Gamma_g$. In Underworld, the Dirichlet boundary $\Gamma_g$ is specified through the use of `IndexSet` objects. These objects simply contain lists of mesh nodes (or rather the node indices) which the user wishes to flag as boundary nodes. A number of commonly used `IndexSet` objects are provided directly through the mesh object's `specialSets` dictionary:

```python
import underworld as uw
mesh = uw.mesh.FeMesh_Cartesian()
mesh.specialSets.keys()
```

    ['MaxI_VertexSet',
     'Top_VertexSet',
     'Left_VertexSet',
     'MinI_VertexSet',
     'AllWalls_VertexSet',
     'Bottom_VertexSet',
     'Right_VertexSet',
     'MinJ_VertexSet',
     'MaxJ_VertexSet',
     'Empty']

`IndexSet` objects are discussed further in the mesh section of this user guide. Importantly, note that the `specialSets` objects are simply provided for convenience, and the user is indeed free to create any set they wish. As an example, we can easily reproduce a set provided by the `specialSets` dictionary by explicitly adding the required indices:

```python
import underworld as uw
import numpy as np
mesh = uw.mesh.FeMesh_Cartesian()

# first option, use provided convenience set.
leftwall = mesh.specialSets['Left_VertexSet']

# second option, create empty set and add required indices.
leftwall_handcrafted = mesh.specialSets['Empty']
for index,coord in enumerate(mesh.data):
    # grab only left wall indices
    if coord[0]<0.01:
        leftwall_handcrafted.add(index)

# confirm sets are identical
if len(leftwall) != len(leftwall_handcrafted):
    raise RuntimeError("Sets should have identical size!")
if not np.allclose(leftwall,leftwall_handcrafted):
    raise RuntimeError("Sets should contain identical values!")
```

For the Dirichlet condition, the user specified function $g$ which provides the required values along the Dirichlet boundary is implicitly set by directly modifying the solution `MeshVariable` at the required nodes. Note that you can set or modify your required $g$ function at **any** stage before the solve operation.

#### Scalar Systems

Let's put all these pieces together to generate a basic heat equation solution. For this system, we'll set the bottom boundary to have a temperature of $u=373$, and the top boundary to have a temperature $u=273$. We won't set anything for the side walls, but note that this will implicitly yield a zero temperature gradient (ie heat flux) Neumann condition.

```python
import underworld as uw
import glucifer

# 1.  Create a mesh.
mesh = uw.mesh.FeMesh_Cartesian(maxCoord=(2.,1.))

# 2.  Create the required field(s).
tempField = mesh.add_variable(1)

# 3.  Create required boundary condition object.
# 3.a First create the required IndexSet objects to specify the required boundary.
#     Note shorthand addition operation.
botSet = mesh.specialSets['Bottom_VertexSet']
topSet = mesh.specialSets[   'Top_VertexSet']
totSet = botSet + topSet
# 3.b Now we can create the Dirichlet condition object.
condition = uw.conditions.DirichletCondition(tempField,totSet)
# 3.c Let's set the g function now by directly modifying the temperatureField object.
#     Note that this operation can be done at any stage!
#     Note also the shorthand we use to set the required values.
tempField.data[botSet] = 373.
tempField.data[topSet] = 273.

# 4.  Let's go ahead and set a simple diffusivity.
alpha = 1.

# 5.  Create the required system.
heatSys = uw.systems.SteadyStateHeat(temperatureField=tempField,
                                     fn_diffusivity=alpha,
                                     conditions=condition)

# 6.  Create solver and solve.
solver = uw.systems.Solver(heatSys)
solver.solve()

# vis.
fig = glucifer.Figure(figsize=(800,400))
fig.append(glucifer.objects.Surface(mesh, tempField, colours="cubelaw2"))
fig.show()
```

Note in the above visualisation that we have satisfied our model requirements:

1. User specified Dirichlet condition is satisfied $(u=373 \text{ on } \Gamma_{\text{bottom}},\,\, u=273 \text{ on } \Gamma_{\text{top}})$.
1. Inspection suggests side boundaries have zero normal temperature gradient, but the example in the Neumann section below explicitly sets the zero flux Neumann condition and yields identical results.

As mentioned, the user is free to flag any mesh nodes as knowns of the system, including nodes internal to the domain. They are also free to change their $g$ specification (ie, the values set on the `MeshVariable` boundaries) at any point, however if they wish to change the boundary specification ($\Gamma_g$), it is current necessary to create a new system object. Note also that while the user may freely set values on any node of the `MeshVariable` object, for instantaneous systems only values set on BC-flagged nodes will be retained after the `solve()` operation, with the solver then delegated to determine all other nodal values. We will demonstrate these behaviours in the following example:

```python
import underworld as uw
import glucifer
res=32
mesh = uw.mesh.FeMesh_Cartesian(elementRes=(res,res),minCoord=(-0.5,-0.5), maxCoord=(0.5,0.5))
tempField = mesh.add_variable(1)

# create bc sets. first, allwalls.
allWallSet = mesh.specialSets['AllWalls_VertexSet']
# now grab central patch
centralSet = mesh.specialSets['Empty']
for index,coord in enumerate(mesh.data):
    if max(coord*coord)<0.03:
        centralSet.add(index)
condition = uw.conditions.DirichletCondition(tempField,centralSet+allWallSet)
tempField.data[allWallSet] = 0.
tempField.data[centralSet] = 1.

# for fun, let's also set a random non-bc node to a random value.
# the value will be replaced at solve time by the solver.
tempField.data[res+2] = -999999.

heatSys = uw.systems.SteadyStateHeat(temperatureField=tempField,
                                     fn_diffusivity=1.,
                                     conditions=condition)
solver = uw.systems.Solver(heatSys)
solver.solve()

# grab a copy of solved data for later use, and then let's
# flip the config and re-solve immediately
tempField_orig = tempField.copy(deepcopy=True)
tempField.data[allWallSet] = 1.
tempField.data[centralSet] = 0.
solver.solve()

# vis
fig1 = glucifer.Figure(figsize=(400,400), title="original")
fig1.append(glucifer.objects.Surface(mesh, tempField_orig, colours="ocean", colourBar=False, onMesh=False))
fig2 = glucifer.Figure(figsize=(400,400), title="flipped")
fig2.append(glucifer.objects.Surface(mesh, tempField     , colours="ocean", colourBar=False, onMesh=False))

glucifer.Figure.show_grid([fig1,fig2])
```

#### Vector Systems
So far we have only considered scalar systems having only a single degree of freedom per mesh node. For the Stokes system however we solve for a velocity unknown having 2 or 3 DOFs per mesh node (for 2D and 3D simulations respectively). While the general process is unchanged, you will now be required to specify boundary conditions on a per-DOF basis. The way this is handled in Underworld is to have the user provide an `IndexSet` object for each DOF. So for the previous examples, only a single `IndexSet` was required for the single DOF of the temperature field, while for a velocity field we require 2 (or 3) `IndexSet` objects, one for the $V_x$ DOF, and another for the $V_y$ DOF (and a further for the $V_z$ DOF in 3D).

##### No-Slip BCs
Let's consider the basic *no-slip* configuration first:

```python
import underworld as uw
import glucifer

# 1.  Create a mesh.
mesh = uw.mesh.FeMesh_Cartesian(elementRes=(32,32))

# 2.  Create the required field(s).
velField = mesh.add_variable(2)
preField = mesh.subMesh.add_variable(1)

# 3.  Create required boundary condition object.
allWalls = mesh.specialSets['AllWalls_VertexSet']
#     Note that we now pass in a tuple of IndexSet objects, representing
#     the nodes to flag for (V_x, V_y). For no-slip, we flag both DOFs for
#     each node, so therefore the index sets must be identical.
#     Note also that we generally do not set any conditions on the pressure
#     unknowns.
condition = uw.conditions.DirichletCondition(velField, (allWalls,allWalls) )
#     Init the velocity field to zero everywhere.  Therefore g=(0,0).
velField.data[:] = (0.,0.)

# 4.  Let's create a simple function for the viscosity & bodyforce.
viscosity = 1.
coord = uw.function.coord()
bodyforce = (0.,-coord[0])

# 5.  Create the required system.
stokesSys = uw.systems.Stokes(velocityField=velField,
                              pressureField=preField,
                              fn_viscosity=viscosity,
                              fn_bodyforce=bodyforce,
                              conditions=condition )

# 6.  Create solver and solve
solver = uw.systems.Solver(stokesSys)
solver.solve()

# Grab copy of solution.
noslipvel = velField.copy(deepcopy=True)

# As before, we can modify g at any point.
# Create a mobile lid config:
topWall = mesh.specialSets['Top_VertexSet']
velField.data[topWall] = (1.,0.)
# also drop body force
stokesSys.fn_bodyforce = (0.,0.)
solver.solve()

# vis
fignoslip = glucifer.Figure(figsize=(400,400), title="no slip")
fignoslip.append(glucifer.objects.VectorArrows(mesh, noslipvel))
fignoslip.append(glucifer.objects.Surface(mesh, uw.function.math.dot(noslipvel,noslipvel), colours="isorainbow"))
figtraction = glucifer.Figure(figsize=(400,400), title="mobile lid")
figtraction.append(glucifer.objects.VectorArrows(mesh, velField))
figtraction.append(glucifer.objects.Surface(mesh, uw.function.math.dot(velField,velField), colours="isorainbow"))
glucifer.Figure.show_grid([fignoslip,figtraction])
```

##### Free-Slip BCs

The setup necessary to achieve a *free-slip* configuration is more complex. In this scenario, we wish to treat velocity components normal to the boundaries as knowns of the system, tethering this component to zero if we wish to prevent in/outflows. The tangential component should be left free to allow slip. Therefore, for a basic free-slip configuration on all walls of the standard cartesian geometry, we require:

1. Left/right wall nodes: $V_x = 0$, $V_y$ free.
1. Top/bottom wall nodes: $V_x$ free, $V_y = 0$.

This translates into flagging all the $V_x$ nodes on the left/right wall, and all the $V_y$ nodes on the top/bottom walls:

```python
import underworld as uw
import glucifer

# 1.  Create a mesh.
mesh = uw.mesh.FeMesh_Cartesian(elementRes=(32,32))

# 2.  Create the required field(s).
velField = mesh.add_variable(2)
preField = mesh.subMesh.add_variable(1)

# 3.  Create required boundary condition object.
leftrightWalls = mesh.specialSets['Left_VertexSet'] + mesh.specialSets['Right_VertexSet']
topbottWalls   = mesh.specialSets[ 'Top_VertexSet'] + mesh.specialSets['Bottom_VertexSet']
condition = uw.conditions.DirichletCondition(velField, (leftrightWalls,topbottWalls) )
#     Init the velocity field to zero everywhere.
velField.data[:] = (0.,0.)

# 4.  Let's create a simple function for the viscosity & bodyforce.
viscosity = 1.
coord = uw.function.coord()
bodyforce = (0.,-coord[0])

# 5.  Create the required system.
stokesSys = uw.systems.Stokes(velocityField=velField,
                              pressureField=preField,
                              fn_viscosity=viscosity,
                              fn_bodyforce=bodyforce,
                              conditions=condition )

# 6.  Create solver and solve
solver = uw.systems.Solver(stokesSys)
solver.solve()

# Grab copy of solution.
noslipvel = velField.copy(deepcopy=True)

# As before, we can modify g at any point.
# Create inflow/outflow condition
for index in leftrightWalls:
    if mesh.data[index][1] < 0.3:
        velField.data[index] = (1.,0.)
# ramp up body force
stokesSys.fn_bodyforce = 100.*stokesSys.fn_bodyforce
solver.solve()

# vis
figfreeslip = glucifer.Figure(figsize=(400,400), title="free slip")
figfreeslip.append(glucifer.objects.VectorArrows(mesh, noslipvel))
figfreeslip.append(glucifer.objects.Surface(mesh, uw.function.math.dot(noslipvel,noslipvel), colours="gebco"))
figinflow = glucifer.Figure(figsize=(400,400), title="free slip with in/out flow")
figinflow.append(glucifer.objects.VectorArrows(mesh, velField))
figinflow.append(glucifer.objects.Surface(mesh, uw.function.math.dot(velField,velField), colours="gebco"))
glucifer.Figure.show_grid([figfreeslip,figinflow])
```

##### Inflow/Outflow

Note in the second example above that while you are free to create any inflow/outflow condition, unless you have otherwise specified, the system will be configured for incompessibility, and therefore any inflow must exactly balance outflow. For complex geometries, ensuring this balance is not always possible given the discrete nature of the models, and you might instead consider allowing some part of your domain to be compressible via the `fn_one_on_lambda` `Stokes` constructor parameter (refer to associated API reference for concise details).

If you encounter solver difficulties while using inflow/outflow, it is often the case that you are violating volume conservation, and are therefore trying to find a solution that cannot exist.

### The Neumann Condition

The general process for specifying a Neumann condition is almost identical to that for the Dirichlet condition. As before, you will specify the required boundary using `IndexSet` objects. However, as you are no longer fixing a value for the unknown itself (temperature, velocity, etc), but instead a derived quantity (heat flux, stress) you are now required to provide a `Function` object to specify the necessary $h$ function. Remember that `MeshVariable` objects are also `Function` objects, so you are free to create another `MeshVariable` to define $h$ if that makes most sense. Refer to the Function section of this guide for further details.

```python
import underworld as uw
import glucifer

# 1.  Create a mesh.
mesh = uw.mesh.FeMesh_Cartesian(elementRes=(32,32))

# 2.  Create the required field(s).
tempField = mesh.add_variable(1)

# 3.a  Create the Dirichlet Condition
botSet = mesh.specialSets['Bottom_VertexSet']
topSet = mesh.specialSets[   'Top_VertexSet']
totSet = botSet + topSet
cond_dirichlet = uw.conditions.DirichletCondition(tempField,totSet)
tempField.data[botSet] = 373.
tempField.data[topSet] = 273.
# 3.b  Create the Neumann condition
sidewalls = mesh.specialSets['Left_VertexSet'] + mesh.specialSets['Right_VertexSet']
#      Create this constant value function explicitly so we can
#      modify it later.
flux = uw.function.Function.convert(0.)
cond_neumann = uw.conditions.NeumannCondition(tempField,sidewalls,flux)

# 4.  Diffusivity
alpha = 1.

# 5.  Create the required system.
heatSys = uw.systems.SteadyStateHeat(temperatureField=tempField,
                                     fn_diffusivity=alpha,
                                     conditions=[cond_dirichlet,cond_neumann])

# 6.  Create solver and solve.
solver = uw.systems.Solver(heatSys)
solver.solve()

# grab copy
nofluxTemp = tempField.copy(deepcopy=True)

# now set flux and re-solve
flux.value = -100.
solver.solve()

# vis.
fig_noflux = glucifer.Figure(figsize=(400,400), title="No Lateral Flux")
fig_noflux.append(glucifer.objects.Surface(mesh, nofluxTemp, colours="cubelaw2"))
fig_flux = glucifer.Figure(figsize=(400,400), title="Lateral Flux")
fig_flux.append(glucifer.objects.Surface(mesh, tempField, colours="cubelaw2"))
glucifer.Figure.show_grid([fig_noflux,fig_flux])

```

### Initial Conditions

For instantaneous systems (`SteadyStateHeat`, `SteadyStateDarcyFlow`, `Stokes`) you will not generally be required to set initial conditions, although care must be taken for non-linear systems.

For the `AdvectionDiffusion` system you will be required to configure your initial variable field as necessary. See the Mesh section of the user guide for examples on initialising `MeshVariable` objects. Naturally if your model utilises particles swarms, they will also need to be configured (see the Swarm section of the user guide).

## Defining Physical Quantities

You will create an Underworld `Function` object to define your physical quantities. Refer to the Function section of the user guide for details.

## Non-Linearity

Underworld currently provides a basic Picard based iterative scheme for solving non-linear problems. Note, depending on the form of your non-linearity, a reasonable initial field configuration may be required to avoid solver convergence difficulty or division by zero type issues.

If your model has a non-linearity, the solver will flag you to this at solve time, and you must explicitly set the `solve()` parameter `nonLinearIterate` to `True` (Underworld will iterate for a solution) or `False` (Underworld will not iterate, but will perform a single solve). The `False` option might be used if you would like to explicitly handle the non-linearity, or perform some analysis or modifications after each iteration. An alternative option to enable per-iteration interaction is offered by the creating a callback function (passed in via the `callback_post_solve` paramater to the `solve()` method).

```python
import underworld as uw
import glucifer

# 1.  Create a mesh.
mesh = uw.mesh.FeMesh_Cartesian(elementRes=(32,32))

# 2.  Create the required field(s).
tempField = mesh.add_variable(1)

# 3.a  Create the Dirichlet Condition
botSet = mesh.specialSets['Bottom_VertexSet']
topSet = mesh.specialSets[   'Top_VertexSet']
totSet = botSet + topSet
cond_dirichlet = uw.conditions.DirichletCondition(tempField,totSet)
tempField.data[botSet] = 1.
tempField.data[topSet] = 0.

# 4.  Diffusivity
alpha = uw.function.math.exp(tempField)

# 5.  Create the required system.
heatSys = uw.systems.SteadyStateHeat(temperatureField=tempField,
                                     fn_diffusivity=alpha,
                                     conditions=[cond_dirichlet])

# 6.  Create solver and solve.
solver = uw.systems.Solver(heatSys)
# THIS WILL NOT WORK!
# solver.solve()
# THIS WILL
solver.solve(nonLinearIterate=True)

# Use a callback
rms_temp = uw.utils.Integral(tempField*tempField,mesh)
import numpy as np
def mycallback():
    print("Average temp is {}".format(np.sqrt(rms_temp.evaluate())))
# reset temp field first for demo
tempField.data[:] = 0.
tempField.data[botSet] = 1.
solver.solve(nonLinearIterate=True, callback_post_solve=mycallback)
```

    Average temp is [ 0.58463673]
    Average temp is [ 0.64936446]
    Average temp is [ 0.6476043]
    Average temp is [ 0.6476043]

```python

```


```python
import numpy as np
import pandas as pd
```

# Descrevendo os arquivos CSV

## TermosUnicos_<nome da coleção>-Amostra.csv

  Na primeira versão do experimento, calculamos o tamanho da amostra necessário em função da média de caracteres de cada documento dentro das coleções (arquivos <nome da coleção>-Amostra). No entanto, como os radicalizadores são executados sobre palavras e não sobre caracteres, alteramos a versão 2 para o calcular a amostragem em função da média de termos únicos por documento.

  Como os arquivos possuem a mesma estrutura, utilizaremos os Acórdãos do Segundo Grau para facilitar a explicação.

```python
amostra = pd.read_csv('../csv/TermosUnicos_SegundoGrauAcordaos-Amostra.csv')
amostra
```

      Media
      Sigma
      N
      e
      TamanhoDaAmostra

      0
      638.44896
      322.149428
      181994
      16.107471
      1524

Calculamos o tamanho da amostra utilizando a função:

\begin{equation}
n = \frac{z^2 . \sigma^2 . N}{e^2 . (N - 1) + z^2 . \sigma^2}
\end{equation}

Onde $z = 1,96$ (nível de confiança de 95%), $\sigma$ é o desvio-padrão da quantidade de termos únicos na coleção,
    N é o número de documentos que a coleção possui e erro amostral $e = 0,05\sigma$.

## <nome da coleção>-Sumario.csv

Esse arquivo registra o tamanho da amostra processada (Processados), o número de termos únicos da amostra (TUC) e o algoritmo de radicalização utilizado (Stemmer).

```python
sumario = pd.read_csv('../csv/SegundoGrauAcordaos-Sumario.csv')
sumario
```

      Processados
      TUC
      Stemmer

      0
      1524
      42141
      NoStem

      1
      1524
      23019
      Porter

      2
      1524
      20220
      RSLP

      3
      1524
      36976
      RSLPS

      4
      1524
      32007
      UniNE

## <nome da coleção>-Termos.csv

Registra o número de vezes que cada termo apareceu na amostra depois que aplicamos o radicalizador.

```python
termos = pd.read_csv('../csv/SegundoGrauAcordaos-Termos.csv', encoding='Latin-1')
termos.head()
```

      Termo
      Quantidade
      Stemmer

      0
      desditosa
      1
      NoStem

      1
      guarde
      1
      NoStem

      2
      titularizar
      4
      NoStem

      3
      espelhada
      4
      NoStem

      4
      3640
      1
      NoStem

## <nome da coleção>.csv

Para cada documento com id "Chave", registramos a quantidade de termos únicos após o uso da radicalização.

```python
colecao = pd.read_csv('../csv/SegundoGrauAcordaos.csv')
colecao.head()
```

      Chave
      TUD
      Stemmer

      0
      201100200560443632531
      576
      NoStem

      1
      201100200560443632531
      517
      Porter

      2
      201100200560443632531
      480
      RSLP

      3
      201100200560443632531
      540
      RSLPS

      4
      201100200560443632531
      535
      UniNE

## <nome da coleção>_<nome do algoritmo>_<nome da métrica>.csv

Registra o valor da métrica obtido para cada uma das consultas disparadas contra o engine de busca.

```python
metrica = pd.read_csv('../csv/asg_nostem_map.csv', header=None, index_col=0)
metrica.head()
```

      1

      0

      1
      0.947681

      2
      0.500000

      3
      1.000000

      4
      0.557677

      5
      1.000000

**Vale ressaltar que não setamos a semente nas fases de randomização dos experimentos, portanto, os arquivos gerados a cada execução podem apresentar valores diferentes dos aqui expostos. Contudo, estatisticamente, é esperado que haja ratificação das hipóteses descritas na dissertação.**

notebooks/Descrevendo os arquivos CSV.ipynb


Before we start, let's make sure we got access to a GPU:

Edit->Notebook Setting->Hardware Accelerator

```
from tensorflow.python.client import device_lib
device_lib.list_local_devices()
```

# ML Coding example #2: Atomization Energies with PyTorch (Feed-Forward Neural Network)

The QM7 dataset contains XYZ structures for 7101 small molecules, with up to 7 atoms of type CNO, saturated with H.

Some of them look like this:

Attempt to map all organic molecules with up to 7 CNO-atoms.

For each molecule you will be given the atomization energy calculated using a QM method (PBE0/def2-TZVP).

*   Instead of the raw 7101 XYZ files, we will use the Bag-of-Bonds features for each molecule, along with the atomization energies.

References:

1.   Rupp et al. (2012) Phys Rev Lett https://doi.org/10.1103/PhysRevLett.108.058301
2.   Hansen et al. (2015) J Phys Chem Lett https://doi.org/10.1021/acs.jpclett.5b00831

```
!wget -O bob.npy https://www.dropbox.com/s/vyiwza2uy4jkczg/bob.npy
!wget -O hof.npy https://www.dropbox.com/s/zy717f8mwxaegff/hof.npy
```

Load the data into numpy arrays and plot the distribution of energies:

```
import numpy as np

Xall = np.load("bob.npy")
Yall = np.load("hof.npy")

import matplotlib.pyplot as plt
plt.hist(Yall)
plt.xlabel("Atomization Energy [kcal/mol]")
plt.ylabel("Counts")
plt.show()
```

The values are very large and negative, which can be a problem for neural networks. To avoid this, we can normalize the data or scale all the target values by a factor.

Scale the values by the mean of the dataset and divide into Training and Test sets:

```
n_train = 4000
n_valid = 1000
n_test  = 2000

Yall = np.load("hof.npy")
y_mean = np.mean(Yall)
print(y_mean)
Yall /= y_mean

Xtrain = Xall[:n_train]
Xvalid = Xall[n_train:n_valid+n_train]
Xtest  = Xall[-n_test:]

Ytrain = Yall[:n_train]
Yvalid = Yall[n_train:n_valid+n_train]
Ytest  = Yall[-n_test:]
```

Plotting again!

```
plt.hist(Yall, label="All")
plt.hist(Ytrain, label="Training")
plt.hist(Ytest, label="Test")
plt.hist(Yvalid, label="Validation")

plt.xlabel("Target value")
plt.ylabel("Counts")
plt.legend()
plt.show()
```

Wondering what our representations look like? Print a row in `Xtrain`!

```
print(Xtrain[1])
print(len(Xtrain[1]))
```

## Defining out Architecture:
Four layers (including input and output), so this qualifies for the buzzword "Deep Learning":

Fully-connected neural network with two hidden layer in PyTorch:

```
import torch
import torch.nn as nn

class NeuralNet(nn.Module):

  def __init__(self, input_size, hidden1_size, hidden2_size):

    super(NeuralNet, self).__init__()

    # Define the operations in the layers
    # Here "Linear", i.e: y = a*X.T + b
    self.fc1 = nn.Linear(input_size,hidden1_size)
    self.fc2 = nn.Linear(hidden1_size,hidden2_size)
    self.fc3 = nn.Linear(hidden2_size, 1)

  def forward(self, x):

    # Define the activation functions,
    # i.e. the "forward" passes
    x = torch.sigmoid(self.fc1(x))
    x = torch.sigmoid(self.fc2(x))
    x = self.fc3(x)

    return x

```

Sigmoid functions again:\begin{equation}
\sigma\left(x\right) = \frac{1}{1 + \exp\left(-x\right)}
\end{equation}

Linear connections between the layers:
\begin{equation}
y = b + \sum_i x_i  w_i
\end{equation}

Since we are using a GPU, all the training and test data must be moved to the GPU:

```
dtype = torch.float
device = torch.device("cuda:0")

x = torch.from_numpy(Xtrain).to(device, dtype)
y = torch.from_numpy(Ytrain.reshape((n_train,1))).to(device, dtype)

xv = torch.from_numpy(Xvalid).to(device, dtype)
yv = torch.from_numpy(Yvalid.reshape((n_valid,1))).to(device, dtype)

xs = torch.from_numpy(Xtest).to(device, dtype)
ys = torch.from_numpy(Ytest.reshape((n_test,1))).to(device, dtype)
```

## Optimizing the Neural Network weights:

First, lets define the neural network, and the sizes of the hidden layers. Secondly, we have to send it to the GPU.

```
input_size = 465

# 30 & 5 are good values
# 64 & 50 also worth a try
hidden1_size = 30
hidden2_size = 5

# Define our model
model = NeuralNet(input_size, hidden1_size, hidden2_size)

# Send it to the GPU
model = model.to(device)
```

Check parameters:

```
for param in model.parameters():
  print(param.shape)
  print(param)
```

Next, we have to select a loss function. For $y = sin(x)$ we minimized the squared error:

\begin{equation}
L_2\left(\mathbf{w}\right) = \sum_{i=1}^{N} \left(y_i^\mathrm{true} -  y_i^\mathrm{predicted} \right)^{2}
\end{equation}
From experience, the L1-loss function (mean absolute error) works better for this experiment:
\begin{equation}
L_1\left(\mathbf{w}\right) = \frac{1}{N}\sum_{i=1}^{N} | y_i^\mathrm{true} -  y_i^\mathrm{predicted} |
\end{equation}

Lastly, we also need to define the optimizer. In this case we choose the "Adam" optimizer, which is a variant of gradient-descent optimization.
*  Kingma & Ba (2014) *arXiv*, https://arxiv.org/abs/1412.6980

```
# L2 loss function (Mean Squared Error)
# loss_fn = nn.MSELoss()

# L1 loss (Mean Absoulte Error)
loss_fn = nn.L1Loss()

# Optimizer, lr = learning rate
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
```

Now, we are write the optimization procedure

1.   Make prediction
2.   Calculate the loss
3.   Calculate the gradient of the loss function
4.   Take a step in the direction suggested by the gradient.

For example 30000 epoch (about 30-60 sec on Google Colab):

```
for epoch in range(30000):

  # Make a prediction with the model
  y_predicted = model.forward(x)

  # Calculate the Loss-function
  loss = loss_fn(y_predicted, y)

  # Set gradient to zero
  optimizer.zero_grad()

  # Take the gradient of the loss function
  loss.backward()

  # Take a step with the optimizer
  optimizer.step()

  # Print additional aoutput
  if (epoch % 1000 == 0):
    mse =(y - y_predicted).pow(2).sum()
    rmsd = torch.sqrt(mse/n_train) * abs(y_mean)
    mae = (y - y_predicted).abs().sum() * abs(y_mean) / n_train

    yvs = model.forward(xv)
    rmsd_v = torch.sqrt((yvs - yv).pow(2).sum()/n_valid) * abs(y_mean)
    mae_v = (yvs - yv).abs().sum() * abs(y_mean) / n_valid

    mae = mae.to("cpu").detach().numpy()
    rmsd = rmsd.to("cpu").detach().numpy()
    mae_v = mae_v.to("cpu").detach().numpy()
    rmsd_v = rmsd_v.to("cpu").detach().numpy()

    print("EPOCH %7i    MAE(train) = %7.2f   RMSE(train) = %7.2f   MAE(valid) = %7.2f   RMSE(valid) = %7.2f    [kcal/mol]" %
        (epoch, mae, rmsd, mae_v, rmsd_v))

```

### Calcluate test errors:
With the optimized neural network, we can now predict the error on the test set.

Because everything is stored on the GPU, we also have to copy back to the CPU and convert to Numpy array for convenience:

```
# Calculate test, copy back to CPU and convert to numpy
y_pred = model(xs).to("cpu").detach().numpy().flatten() * y_mean

# Copy true test values back to CPU and convert to numpy
y_true = ys.to("cpu").detach().numpy().flatten() * y_mean

print(y_pred)
print(y_true)

mae = np.mean(np.abs(y_pred - y_true))
rmse = np.sqrt(np.mean(np.square(y_pred - y_true)))

print("MAE:", mae)
print("RMSE:", rmse)
```

Lastly, lets plot a correlation plot between the true and predicted atomization energies:

```

import matplotlib.pyplot as plt

plt.scatter(y_true,y_pred)
plt.xlabel("True Atomization Energy [kcal/mol]")
plt.ylabel("Predicted Atomization Energy [kcal/mol]")
plt.show()
```

### Note on hyper parameters and optimization

In order to have the best performing neural network, we could have changed a lot of things:

*   Number of hidden layers
*   Hidden layer sizes
*   Type of loss function
*   Optimizer + setting
*   Regularization of parameters

If you are wondering, how we could have added regularization to the loss function, here is an example for example L2 regularization:

```
l2reg = 1e-4

for param in model.parameters():
    loss += torch.norm(param) * l2reg
```


```python
import pandas as pd
tour = pd.read_csv('../data/raw/sample_submission.csv')['Path'].tolist()
cities = pd.read_csv('../data/raw/cities.csv')

# Define a function for length of a path:
import numpy as np
from sympy import primerange
primes = list(primerange(0, len(cities)))
def score_tour(tour):
    # length of any given tour with primes calculation
    df = cities.reindex(tour + [0]).reset_index()
    df['prime'] = df.CityId.isin(primes).astype(int)
    df['dist'] = np.hypot(df.X - df.X.shift(-1), df.Y - df.Y.shift(-1))
    df['penalty'] = df['dist'][9::10] * (1 - df['prime'][9::10]) * 0.1
    return df.dist.sum() + df.penalty.sum()

# Let's take a look at our tour
print("Tour path (0-5):",tour[0:5])
# And the flipped tour looks like:
tourflip = tour[::-1]
print("Flipped tour path (0-5):", tourflip[0:5])
# The scores of our tours are:
print("Score of original tour:", score_tour(tour))
print("Score of flipped tour:", score_tour(tourflip))

# If the flipped tour is quicker, change our tour:
if score_tour(tourflip) < score_tour(tour):
    print("The total improvement was:", abs(score_tour(tourflip) - score_tour(tour)))
    tour = tourflip
    print("The better of the original/flipped tour is:", tour[0:5])

# pd.DataFrame({'Path': list(tour)}).to_csv('submission.csv', index=False)
```

    Tour path (0-5): [0, 1, 2, 3, 4]
    Flipped tour path (0-5): [0, 197768, 197767, 197766, 197765]
    Score of original tour: 446884407.521
    Score of flipped tour: 447882971.603