###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2019 Daniel Koehn based on (c)2014 L.A. Barba, G.F. Forsyth, C. Cooper [CFDPython](https://github.com/barbagroup/CFDPython), (c)2013 L.A. Barba, also under CC-BY license.

```python
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, 'r').read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# 1D Linear Advection Problem: Numerical Diffusion

In the last lesson, we could compute accurate solutions for the 1D linear advection equation using the Forward-Time-Backward-Space FD method (FTBS method), if we use an apropriate spatial grid spacing $\Delta x$ and satisfy the CFL-conditon in order to avoid an instability of the numerical solution.

However, beside the stability of the FD solution another problem arises when advecting the solution over larger distances. Let 's take a closer look at the problem ...

## FTBS Method

Before we begin, I shortly review the numerical method, we used so far to solve the *one-dimensional linear advection equation*

$$
\begin{equation}
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0 \tag{1}
\end{equation}
$$

We implemented **Euler's method** by using a forward-difference scheme for the time derivative. For the spatial derivatives a backward-difference scheme was choosen, resulting in the forward in time & backward in space **FTBS Method**, reading in
discrete form as:

$$
\begin{equation}
\frac{u_i^{n+1}-u_i^n}{\Delta t} + c \frac{u_i^n - u_{i-1}^n}{\Delta x} = 0 \notag
\end{equation}
$$

where $n$ and $n+1$ are two consecutive steps in time, while $i-1$ and $i$ are two neighboring points of the discretized $x$ coordinate. With given initial conditions, the only unknown in this discretization is $u_i^{n+1}$.  We solve for this unknown to get an equation that lets us step in time, as follows:

$$
\begin{equation}
u_i^{n+1} = u_i^n - c \frac{\Delta t}{\Delta x}(u_i^n-u_{i-1}^n) \notag
\end{equation}
$$

To ensure the stability of the numerical solution, we have to choose a time step $\Delta t$ satisfying the Courant-Friedrichs-Lewy (CFL) criterion:

\begin{equation}
\Delta t \le \frac{\sigma \Delta x}{c} \notag
\end{equation}

depending on the spatial gridpoint distance $\Delta x$, advection velocity $c$ and a factor $\sigma$.

Below, you find the same code, we used in the previous notebook, I only extended the spatial computational domain from $L=2$ to $L=15$ and added the analytical solution

$$
\begin{equation}
u(x,t)=u_0(x-ct) \notag
\end{equation}
$$

with the initial condition $u(x,0)=u_0(x)$ to verify the accuracy of the numerical solution.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

```python
def linear_advection_FTBS(nx, Tmax, L=15.0, c=1.0, sigma=0.5):
    """
    Solves the 1D linear advection equation
    with constant speed c in the domain [0, L]
    and plots the solution (along with the initial conditions).
    Here, the time-step size is calculated based on a CFL constraint.

    Parameters
    ----------
    nx : integer
        Number of grid points to discretize the domain.
    Tmax : float
        Maximum integration time
    L : float, optional
        Length of the domain; default: 2.0.
    c : float, optional
        Advection speed; default: 1.0.
    sigma : float, optional
        CFL constraint; default: 0.5.
    dt : float, optional
        Time-step size; default: 0.025.
    nt : integer, optional
        Number of time steps to compute; default: 20.
    """

    # Discretize spatial grid.
    dx = L / (nx - 1)
    x = numpy.linspace(0.0, L, num=nx)

    # Compute the time-step size based on the CFL constraint.
    dt = sigma * dx / c

    # Compute number of time steps
    nt = (int)(Tmax/dt)

    # Set initial conditions.
    u0 = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
    u0[mask] = 2.0

    # Define analytical solution
    ua = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5 + c*Tmax, x <= 1.0 + c*Tmax))
    ua[mask] = 2.0

    # Integrate the solution in time.
    u = u0.copy()
    for n in range(1, nt):
        un = u.copy()
        for i in range(1, nx):
            # Solving the advection problem using the FTBS method
            u[i] = un[i] - c * dt / dx * (un[i] - un[i - 1])

    # Plot the solution along with the initial conditions and analytical solution
    pyplot.figure(figsize=(4.0, 4.0))
    pyplot.xlabel('x')
    pyplot.ylabel('u')
    pyplot.grid()
    pyplot.plot(x, u0, label='Initial',
                color='C0', linestyle='--', linewidth=2)
    pyplot.plot(x, u, label='nt = {}'.format(nt),
                color='C1', linestyle='-', linewidth=2)
    pyplot.plot(x, ua, label='Analytical',
                color='C2', linestyle='--', linewidth=2)
    pyplot.legend(loc="lower left")
    pyplot.xlim(0.0, L)
    pyplot.ylim(0.0, 2.5);
```

Let's examine the results of the linear advection problem using $nx=2000$ spatial gridpoints and a maximum propagation time $T_{max}=0.5$:

```python
# Define maximum time Tmax
Tmax = 0.5
linear_advection_FTBS(2000,Tmax)
```

As you can see, the numerical solution coincides very well with the analytical solution. What happens if we increase the maximum modelling time from $T_{max}=0.5$ to $T_{max}=13$?

```python
# Define maximum time Tmax
Tmax = 13.
linear_advection_FTBS(2000,Tmax)
```

Unfortunately, for $T_{max}=13$ the numerical solution is not accurate anymore. The amplitude of the initial rectangle function is decreased and the function in general significantly smoothed.

As we will see in the next class, this solution is similar to the one we can achieve for the diffusion problem. However, in case of the advection problem, this behaviour is purely artifical and denoted as **Numerical Diffusion**. In the following exercises, you will implement and test different alternative FD methods in order to mitigate the effect of numerical diffusion.

## FTCS Method

The first idea might be that the numerical diffusion might be related to the low accuray of the backward-difference scheme for the space derivative. As we have seen in the first lesson of this lecture the central-difference scheme for the space derivative

$$
\begin{equation}
\frac{\partial u}{\partial x} \approx \frac{u(x_{i+1})-u(x_{i-1})}{2\Delta x} \quad\text{Central}\notag
\end{equation}
$$

is more accurate than the backward-difference scheme. So let's implement an Euler method using a forward-difference scheme for the time derivative and choose the central-difference scheme for the space derivative, called forward in time & central in space **FTCS Method**. Our discrete equation is then:

$$
\begin{equation}
\frac{u_i^{n+1}-u_i^n}{\Delta t} + c \frac{u_{i+1}^n - u_{i-1}^n}{2\Delta x} = 0 \notag
\end{equation}
$$

where $n$ and $n+1$ are two consecutive steps in time, while $i-1$ and $i$ are two neighboring points of the discretized $x$ coordinate. With given initial conditions, the only unknown in this discretization is $u_i^{n+1}$.  We solve for this unknown to get an equation that lets us step in time, as follows:

$$
\begin{equation}
u_i^{n+1} = u_i^n - c \frac{\Delta t}{2\Delta x}(u_{i+1}^n-u_{i-1}^n) \tag{2}
\end{equation}
$$

##### Exercise 1

Replace the FTBS method in the code below by the FTCS method eq.(2). Run the FTCS-code for
$T_{max} = 13$, describe and compare the results with the one from the FTBS code.

```python
def linear_advection_FTCS(nx, Tmax, L=15.0, c=1.0, sigma=0.5):
    """
    Solves the 1D linear advection equation
    with constant speed c in the domain [0, L]
    and plots the solution (along with the initial conditions).
    Here, the time-step size is calculated based on a CFL constraint.

    Parameters
    ----------
    nx : integer
        Number of grid points to discretize the domain.
    Tmax : float
        Maximum integration time
    L : float, optional
        Length of the domain; default: 2.0.
    c : float, optional
        Advection speed; default: 1.0.
    sigma : float, optional
        CFL constraint; default: 0.5.
    """

    # Discretize spatial grid.
    dx = L / (nx - 1)
    x = numpy.linspace(0.0, L, num=nx)

    # Compute the time-step size based on the CFL constraint.
    dt = sigma * dx / c

    # Compute number of time steps
    nt = (int)(Tmax/dt)

    # Set initial conditions.
    u0 = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
    u0[mask] = 2.0

    # Define analytical solution
    ua = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5 + c*Tmax, x <= 1.0 + c*Tmax))
    ua[mask] = 2.0

    # Integrate the solution in time.
    u = u0.copy()
    for n in range(1, nt):
        un = u.copy()
        for i in range(1, nx-1):

            # REPLACE THE SPATIAL BACKWARD-FD OPERATOR FROM THE FTBS-SCHEME ...
            #u[i] = un[i] - c * dt / dx * (un[i] - un[i - 1])

            # ... BY THE FTCS-SCHEME
            u[i] =

    # Plot the solution along with the initial conditions.
    pyplot.figure(figsize=(4.0, 4.0))
    pyplot.xlabel('x')
    pyplot.ylabel('u')
    pyplot.grid()
    pyplot.plot(x, u0, label='Initial',
                color='C0', linestyle='--', linewidth=2)
    pyplot.plot(x, u, label='nt = {}'.format(nt),
                color='C1', linestyle='-', linewidth=2)
    pyplot.plot(x, ua, label='Analytical',
                color='C2', linestyle='--', linewidth=2)
    pyplot.legend(loc="lower left")
    pyplot.xlim(0.0, L)
    pyplot.ylim(0.0, 2.5);
```

```python
# Define maximum time Tmax
Tmax = 13.
linear_advection_FTCS(2000,Tmax)
```

## Lax-Friedrichs Method

As you have seen, the **FTCS Method** seem to be quite unstable, even when you decrease the factor $\sigma$. Actually, it is unconditionally unstable, meaning that you can not estimate a time step $\Delta t$ to estimate any accurate solution, as can be proven by the [von Neumann Stability Analysis](https://de.wikipedia.org/wiki/Von-Neumann-Stabilit%C3%A4tsanalyse). While the spatial central FD operator is more accurate than the spatial backward operator, it introduces an artifical **negative diffusion**, which amplifies amplitudes instead of damping them in any physics based diffusion problem. The negative diffusion subsequently leads to the instability of the FTCS code.

So, is there a possibility to stabilize the FTCS method? Some day, two guys [Peter Lax](https://en.wikipedia.org/wiki/Peter_Lax) and [Kurt Friedrichs](https://en.wikipedia.org/wiki/Kurt_Otto_Friedrichs) thought: well let's replace the $u_i^n$ in the time derivative of the FTCS method:

$$
\begin{equation}
\frac{u_i^{n+1}-u_i^n}{\Delta t} + c \frac{u_{i+1}^n - u_{i-1}^n}{2\Delta x} = 0 \notag
\end{equation}
$$

by the spatial average of the neighbouring grid points

$u_i^n = (u_{i+1}^n+u_{i-1}^n)/2$

leading to

$$
\begin{equation}
\frac{u_i^{n+1}-(u_{i+1}^n+u_{i-1}^n)/2}{\Delta t} + c \frac{u_{i+1}^n - u_{i-1}^n}{2\Delta x} = 0 \notag
\end{equation}
$$

where $n$ and $n+1$ are two consecutive steps in time, while $i-1$ and $i$ are two neighboring points of the discretized $x$ coordinate. With given initial conditions, the only unknown in this discretization is $u_i^{n+1}$.  We solve for this unknown to get an equation that lets us step in time, as follows:

$$
\begin{equation}
u_i^{n+1} = \frac{(u_{i+1}^n+u_{i-1}^n)}{2} - c \frac{\Delta t}{2\Delta x}(u_{i+1}^n-u_{i-1}^n) \tag{3}
\end{equation}
$$

##### Exercise 2

Replace the FTBS method in the code below by the Lax-Friedrichs method eq. (3). Run the Lax-Friedrichs code for $T_{max} = 13$. Describe and compare the results with the one from the FTCS code.

```python
def linear_advection_Lax(nx, Tmax, L=15.0, c=1.0, sigma=0.5):
    """
    Solves the 1D linear advection equation
    with constant speed c in the domain [0, L]
    and plots the solution (along with the initial conditions).
    Here, the time-step size is calculated based on a CFL constraint.

    Parameters
    ----------
    nx : integer
        Number of grid points to discretize the domain.
    Tmax : float
        Maximum integration time
    L : float, optional
        Length of the domain; default: 2.0.
    c : float, optional
        Advection speed; default: 1.0.
    sigma : float, optional
        CFL constraint; default: 0.5.
    """

    # Discretize spatial grid.
    dx = L / (nx - 1)
    x = numpy.linspace(0.0, L, num=nx)

    # Compute the time-step size based on the CFL constraint.
    dt = sigma * dx / c

    # Compute number of time steps
    nt = (int)(Tmax/dt)

    # Set initial conditions.
    u0 = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
    u0[mask] = 2.0

    # Define analytical solution
    ua = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5 + c*Tmax, x <= 1.0 + c*Tmax))
    ua[mask] = 2.0

    # Integrate the solution in time.
    u = u0.copy()
    for n in range(1, nt):
        un = u.copy()
        for i in range(1, nx-1):
            # REPLACE THE SPATIAL BACKWARD-FD OPERATOR FROM THE FTBS-SCHEME ...
            #u[i] = un[i] - c * dt / dx * (un[i] - un[i - 1])

            # ... BY THE LAX-FRIEDRICHS-SCHEME
            u[i] =

    # Plot the solution along with the initial conditions.
    pyplot.figure(figsize=(4.0, 4.0))
    pyplot.xlabel('x')
    pyplot.ylabel('u')
    pyplot.grid()
    pyplot.plot(x, u0, label='Initial',
                color='C0', linestyle='--', linewidth=2)
    pyplot.plot(x, u, label='nt = {}'.format(nt),
                color='C1', linestyle='-', linewidth=2)
    pyplot.plot(x, ua, label='Analytical',
                color='C2', linestyle='--', linewidth=2)
    pyplot.legend(loc="lower left")
    pyplot.xlim(0.0, L)
    pyplot.ylim(0.0, 2.5);
```

```python
# Define maximum time Tmax
Tmax = 13.
linear_advection_Lax(2000,Tmax)
```

## Marker-in-Cell Method

Wow, the numerical diffusion in the Lax-Friedrichs approach seems to be even worse than in the original FTBS scheme. An accurate solution of the "simple" 1D advection seem to be a quite challenging task. We could implement other FD-based approaches but not achieve a really significant improvement of solution accuracy. Let's think a little bit outside of the common FD-schemes.

So what we want is a scheme that doesn't blow up, has only small numerical diffusion and is not limited by the Courant criterium. The simplest manner in doing this is to use a marker-based or tracer-based advection scheme.

The basic idea of a marker-based advection method is that you add some markers in your code (typically many more than the number of spatial numerical gridpoints). A marker is a point, which has coordinates $x_{mark}$ and a given property $u_{marks}$.

First, you define your initial property $u_0$ at each spatial grid point $x$ of the spatial finite-difference grid.

Marker-based advection, than requires the following steps during each time step:

1. Interpolate the property $u$ from the nodal gridpoints $x$ to the markers located at $x_{mark}$. In Python, this can be achieved by the function `numpy.interp`

```python
u_mark = numpy.interp(x_mark, x, u)
```

2. Compute the new marker location with

\begin{equation}
x_{mark}^{n+1}(p) = x_{mark}^n(p) + c \Delta t \tag{4}
\end{equation}

where $c$ denotes the advection velocity, $x_{mark}^n(p)$ the location of marker $p$ at time step $n$. This equation might look quite familiar because we used it in the [Exercise: How to sail without wind](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/02_finite_difference_intro/3_fd_ODE_example_sailing_wo_wind.ipynb) to calculate the streamline for a given velocity field.

3. Interpolate the property $u_{mark}$ from the new marker position $x_{mark}^{n+1}$ to the gridpoint positions $x$ using the function `numpy.interp` again

```python
u = numpy.interp(x, x_mark, u_mark)
```

4. Go back to step 1 until you reached time $T_{max}$

##### Exercise 3

Finalize the implementation of the Marker-in-Cell method in the code below. Run the resulting code for $T_{max} = 13$, describe and compare the results with the solutions from all previous codes.

```python
def linear_advection_marker_in_cell(nx, Tmax, dt, L=15.0, c=1.0):
    """
    Solves the 1D linear advection equation
    with constant speed c in the domain [0, L]
    and plots the solution (along with the initial conditions).
    Here, the time-step size is calculated based on a CFL constraint.

    Parameters
    ----------
    nx : integer
        Number of grid points to discretize the domain.
    Tmax : float
        Maximum integration time
    dt : float
        Time step
    L : float, optional
        Length of the domain; default: 2.0.
    c : float, optional
        Advection speed; default: 1.0.
    """

    # Discretize spatial grid.
    dx = L / (nx - 1)
    x = numpy.linspace(0.0, L, num=nx)

    # Define number of markers and initial marker positions
    nx_mark = 5 * nx  # number of markers
    x_mark = numpy.linspace(0.0, L, num=nx_mark) # initial marker positions

    # Compute number of time steps
    nt = (int)(Tmax/dt)

    # Set initial conditions.
    u0 = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
    u0[mask] = 2.0

    # Define analytical solution
    ua = numpy.ones(nx)
    mask = numpy.where(numpy.logical_and(x >= 0.5 + c*Tmax, x <= 1.0 + c*Tmax))
    ua[mask] = 2.0

    # Integrate the solution in time.
    u = u0.copy()

    # IMPLEMENT MARKER-IN-CELL METHOD HERE!
    for n in range(1, nt+1):

        # INTERPOLATE u FROM GRIDPOINTS TO MARKERS u_mark
        u_mark =

        # MOVE MARKERS ALONG X-DIRECTION WITH VELOCITY c USING EQ. (4)
        x_mark =

        # INTERPOLATE u_mark TO u
        u =

    # Plot the solution along with the initial conditions.
    pyplot.figure(figsize=(4.0, 4.0))
    pyplot.xlabel('x')
    pyplot.ylabel('u')
    pyplot.grid()
    pyplot.plot(x, u0, label='Initial',
                color='C0', linestyle='--', linewidth=2)
    pyplot.plot(x_mark, u_mark, label='nt = {}'.format(nt),
                color='C1', linestyle='-', linewidth=2)
    pyplot.plot(x, ua, label='Analytical',
                color='C2', linestyle='--', linewidth=2)
    pyplot.legend(loc="lower left")
    pyplot.xlim(0.0, L)
    pyplot.ylim(0.0, 2.5);
```

```python
# Define maximum time Tmax
Tmax = 13.
dt = 1e-1
linear_advection_marker_in_cell(2000,Tmax,dt)
```

##### Bonus Exercise

Note, that over time the markers are leaving the model domain $x\in(0,15)$ at $x=15$, which can become an issue when applying the Marker-in-Cell method to more complex problems

- Copy the cell with your `linear_advection_marker_in_cell` code below this cell

- Rename the code from `linear_advection_marker_in_cell` to `linear_advection_marker_in_cell_periodic`

- Implement **periodic boundary conditions**, by assuming that your mesh is periodic, which means that if $x_{mark} > L$, you set $x_{mark} = x_{mark} - L$ (assuming that your mesh extends from $0$ to $L$)

- In order to do this, you have to check which markers left the right boundary, after moving the markers using eq. (4), and shift the new marker coordinates $x_{mark}$ by $-L$

- A problem will arise in the following step during the interpolation of $u_{mark}$ from the markers to the values $u$ on the grid

```python
u = numpy.interp(x, x_mark, u_mark)
```

because the `interp` function requires the x-coordinates of the markers in ascending order. Try to solve this problem.

```python
# Define maximum time Tmax and time step dt
Tmax = 13.
dt = 1e-1
linear_advection_marker_in_cell_periodic(2000,Tmax,dt)
```

## What we learned:

- While the 1D linear advection problem looks quite simple, most FD-based approaches suffer from grid diffusion (FTBS method, Lax-Friedrichs method). Other methods like the FTCS method are unconditionally unstable.

- The Marker-in-Cell approach is the method of choice to model advection problems without numerical diffusion. Furthermore, it is quite easy to implement in existing FD modelling codes.


```python
%matplotlib inline
import seaborn as sns
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
from scipy import stats
```

# Regression Diagnostics

Whenever you undertake regression analysis of any kind you should run diagnostic tests that check the shape of your data and the fit of your model to that data.

## Not common in predictive modeling

You won't see many Kaggle competitions running these tests because they aren't as important for predictive modeling. There are less important (and sometimes completely ignored) in predictive modeling because the end-all be-all of predictive modeling is how accurate your model's predctions are on an "out of sample" dataset. This is why we split our dataset into two random halves and and then fit our model parameters using one half, and test the accuracy of our model's predictions using the other half. (It doesn't have to be 50-50 necessarily, but just an example.)

## Necessary for inferential regression modeling

However, if you ever need to run regression analysis for the purposes of inferentital modeling --because you intend to interpret and be informed by variable coefficients --these tests are of utmost importance. Each of these tests exists to test a certain assumption that we're making about the shape of our data or our model's fit to it. If one or multipile of these assumptions are violated, then doubt is cast on the reliability of our variable coefficients.

# Estimating Parameters

You'll remember that OLS and Gradient-Descent based methods of linear regression modeling both seek to **estimate** parameters that "minimize the sum of the squared error." Because we have been more focused on predictive modeling we haven't talked as much about what it means for a parameter to be an "estimate."

An estimated regression coefficient represents the **mean** change in our response variable (y) given a one unit change in our response given a one unit change in the predictor. But because it is an estimate, there is a certain confidence interval around our prediction of our coefficient. The confidence interval is vital to our interpretation of regression coefficients.

## A Parameter Estimation Example

Suppose I was fitting a regression model and calculated its coefficients and substituted them into the equation:

\begin{align}
\hat{y} = .42+ 2.05x
\end{align}

We've well established in past lectures that $\hat{\beta}_1$ reprents the slope of our regression line, but we haven't talked about how this is just an **estimate** for the slope of our regression line, and as an estimate has an associated confidence interval.

Lets say that we calculated the 95% confidence interval for $\hat{\beta}_1$ and it came out to be $(1.9 ,  2.2)$. This means that we can only be 95% confident that the average effect of x on y is within this range. Up to this point we have just taken the reported coefficient as gospel, but a lot of conditions need to be satisfied in order for us to trust regression coefficients. We'll talk about a few of them today.

```python
# We can create scatterplots that show the confidence interval!

heights = np.array([50,52,53,54,58,60,62,64,66,67, 68,70,72,74,76,55,50,45,65])
weights = np.array([25,50,55,75,80,85,50,65,85,55,45,45,50,75,95,65,50,40,45])

sns.regplot(heights, weights, color='blue').set_title('Height by Weight');
```

## Standard Error of a Coefficient

While we can calculate a 95% confidence interval for any estimated parameter, we usually won't refer to the potential spread of parameter estimates by its confidence interval. We'll usually refer to how wide or how narrow the spread is by referring to what's called the "Standard Error."

The Standard Error (SE) of a coefficient estimate is the estimated standard deviation of the error in measuring it. So the coefficient itself is the **estimated mean effect** of x on y. and the Standard Error is the **estimated standard deviation** of our coefficient. We use standard errors to calculate the confidence interval.

## Standard Error of the Regression

The standard error of a coefficient is different from the standard error of the regression. The standard error of the regression as a whole is the average distance that points fall from the regression line.

\begin{align}
SE_{est} = \sqrt{\frac{\sum(y_i-\hat{y})^2}{N}}
\end{align}

Does the numerator of that equation look familiar to you? I hope it does by now.

Standard Error of the regression as a whole is the average distance that datapoints fall from the regression line.

## Precision vs Accuracy

### Accuracy
A regression coefficient that is "Accurate" is centered around its "true" value. The problem here is that we don't know what the true value actually is, so when we say that a coefficient is more accurate we mean that we suspect that it better represents ground truth.

The more observations we have, the more precise our estimates will be.

### Precision
A regression coefficient that is "Precise" has a small standard error. It has a tighter confidence interval as well.

# Gauss Markov Assumptions

There are five Gauss Markov assumptions (also called conditions) that are required for OLS to be BLUE (the "Best Linear Unbiased Estimator").

**0) Well Defined:** $X^{T}X$ is invertible (No perfect multicollinearity), $|X| \neq 0$

**1) Linearity:** the parameters we are estimating using the OLS method must be themselves linear.

**2) Random:** our data must have been randomly sampled from the population.

**3) Non-Collinearity:** the regressors (x vars) being calculated aren’t perfectly (or highly) correlated with each other.

**4) Exogeneity:** the regressors (x vars) aren’t correlated with the error term.

- Omitted Variables Bias (Ice Cream Sales and Burglaries)
- Instrumental Variables: A regression of education on earnings would be biased both education and ability are both influenced by  influenced by natural ability. We use an additional "Instrumental Variable" that is correlated with of schooling and earnings but isn't correlated with ability in order to estimate the effect of years of schooling on earnings. (Month of birth - Angrist and Kreuger)

**5) Homoskedasticity:** no matter what the values of our regressors might be, the error of the variance is constant.

[Statistics How To - Gauss Markov Assumptions](https://www.statisticshowto.datasciencecentral.com/gauss-markov-theorem-assumptions/)

# Enough Terminology Zoo, Lets Do Stuff!

# Finding Standard Errors of Coefficients

Scikit-Learn is built to be a machine learning library, and machine learning typically prioritizes making accurate predictions over interpreting model parameters. Due to this, there aren't any easy ways to calculate standard errors of our coefficients using Sklearn. We'll need to use a different library called **statsmodels**.

### Preliminary steps

```python
# Read in dataset
df = pd.read_csv("https://raw.githubusercontent.com/ryanleeallred/datasets/master/kc_house_data.csv")
df.columns.tolist()
```

```python
# Most homes weren't renovated
df['yr_renovated'].value_counts().head()
```

```python
# Drop columns that I don't care about
df = df.drop(columns=['id','date','zipcode','lat','long','yr_renovated'])
```

```python
# Plot scatterplots
target = 'price'
features = df.columns.drop(target)
for feature in features:
    sns.scatterplot(x=feature, y=target, data=df, alpha=0.1)
    plt.show()
```

```python
# Prepare X and y
target = 'price'
features = ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
            'floors', 'waterfront', 'view', 'condition', 'grade',
            'sqft_above', 'sqft_basement', 'yr_built',
            'sqft_living15', 'sqft_lot15']

X = df[features]
y = df[target]
```

```python
# Use Statsmodels to run a regression
model = sm.OLS(y, sm.add_constant(X))
results = model.fit()
print(results.summary())
```

### Interpretation of P-Value

"The p-value for each term tests the null hypothesis that the coefficient is equal to zero (no effect). A low p-value (< 0.05) indicates that you can reject the null hypothesis. In other words, a predictor that has a low p-value is likely to be a meaningful addition to your model because changes in the predictor's value are related to changes in the response variable." [Minitab Blog](http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-interpret-regression-analysis-results-p-values-and-coefficients)

## Remove Outliers

```python
print(df.shape)
df = df[(np.abs(stats.zscore(df)) < 3).all(axis=1)]
print(df.shape)
```

```python
# Re-run regression without outliers.
X = df[features]
y = df[target]
model = sm.OLS(y, sm.add_constant(X))
results = model.fit()
print(results.summary())
```

## Log-Linear Regression

```python
df['ln_price'] = np.log(df['price'])
df = df.drop(columns='price')

target = 'ln_price'
features = df.columns.drop(target)
for feature in features:
    sns.scatterplot(x=feature, y=target, data=df, alpha=0.1)
    plt.show()
```

```python
# Log-Linear Regression
X = df[features]
y = df[target]
model = sm.OLS(y, sm.add_constant(X))
results = model.fit()
print(results.summary())
```

[King County](https://www.google.com/maps/place/King+County,+WA/@47.4269284,-122.9244266,8z/data=!3m1!4b1!4m5!3m4!1s0x54905c8c832d7837:0xe280ab6b8b64e03e!8m2!3d47.5480339!4d-121.9836029)

# Collinearity/Multicollinearity

When two variables are close to being a linear combination of each other we call this **collinearity** or having high levels of collinearity. If there are three of more variables all with significant levels of collinearity we call this "multicollinearity" but people basically use the two terms interchangeably.

## Perfect Multicollinearity
Variables variables are **perfectly** collinear when the vectors that represent them are linearly dependent. This means that if plotted against each other in a scatter plot all of the points would fall on the same line. We mentioned briefly that perfect multicollinearity breaks OLS because it makes it so that the X matrix is not invertible.

Perfect multicollinearity usually is caused by careless feature engineering usually through transforming the units of a variable and then keeping both variables in the regression. It can also be created through the one-hot-encoding of binary categorical variables.

## Why is Collinearity Bad?

High levels of Collinearity in a dataset is bad because it increases standard errors and therefore makes estimates of our coefficients less precise. Very high levels of collinearity (nearing perfect multicollinearity can cause standard errors to grow drastically.)

### Example of two collinear features:

```python
sns.scatterplot(x='sqft_basement', y='sqft_living', data=df, alpha=0.1);
```

## Testing for high levels of collinearity

We test for high levels of collinearity by calculating the dataset's **Variance Inflation Factor** or VIF. From Wikipedia:

> "In statistics, the variance inflation factor (VIF) is the ratio of variance in a model with multiple terms, divided by the variance of a model with one term alone. It quantifies the severity of multicollinearity in an ordinary least squares regression analysis. It provides an index that measures how much the variance (the square of the estimate's standard deviation) of an estimated regression coefficient is increased because of collinearity." [VIF Wikipedia](https://en.wikipedia.org/wiki/Variance_inflation_factor)

As a rule of thumb any variable that has a VIF > 10 needs to be dealt with (probably dropped from your model). If you see a VIF greater than 10 it is likely that two x variables are highly correlated. Remember that we can use the correlation matrix to check levels of correlation between our independent variables.

(Ignore the variance inflation factor for the constant. It should be high, even infinite.)

https://www.statsmodels.org/stable/generated/statsmodels.stats.outliers_influence.variance_inflation_factor.html

```python
from statsmodels.stats.outliers_influence import variance_inflation_factor

X = sm.add_constant(X)
vif = [variance_inflation_factor(X.values, i) for i in range(len(X.columns))]
pd.Series(vif, X.columns)
```

### Exclude collinear features and refit model

```python
target = 'ln_price'
features = ['bedrooms',
            'bathrooms',
            'sqft_living',
            'sqft_lot',
            'floors',
            'waterfront',
            'view',
            'condition',
            'grade',
            'yr_built',
            'sqft_living15',
            'sqft_lot15']

y = df[target]
X = df[features]
X = sm.add_constant(X)
vif = [variance_inflation_factor(X.values, i) for i in range(len(X.columns))]
pd.Series(vif, X.columns)
```

```python
model = sm.OLS(y, X)
results = model.fit()
print(results.summary())
```

# Homoskedasticity and Heteroskedasticity

What a big complicated words. Also, some poeple spell them "homoscedasticity" and "heteroscedasticity" but that just feels wrong to me somehow.

## Homoskedasticity

Homoskedasticity means that along our entire domain (x axis) the residuals are about the same distance from our regression line (on average).

## Heteroskedasticity.

Our data points exhibit heteroskedasticity when they don't exhibit homoskedasticity. This is much easier to explain by just showing a picture.

Looking at scatterplots of our data are there any places where we might be worried about heteroskedasticity?

```python
target = 'ln_price'
features = df.columns.drop(target)
for feature in features:
    sns.lmplot(x=feature, y=target, data=df, scatter_kws=dict(alpha=0.1))
    plt.show()
```

## Which variables might potentially be offenders?

## Addressing Heteroskedasticity

If heteroskedasticity exists in our dataset it will damage our standard errors and make our estimates less precise. You have to remember that any challenges that damages the reliability of standard errors also damages the reliability of confidence intervals and hypothesis tests. Therefore, these challenges that damage standard errors also damage a whole host of statistical tools that we would normally like to rely on.

Dealing with heteroskedasticity is pretty straightforward, we simply employ what are called "robust standard errors" I won't go into depth on how they this works here, but robust standard errors essentially correct heteroskedasticity in our data while the side effects are minimal. Due to this if you are suspicious of heteroskedasticity in your dataset and you intend to interpret the coefficients of your model. You should run the regression using robust standard errors the majority of the time. Lets see how much our regression output changes when we use robust standard errors.

```python
# Let's run our regression again using Robust Standard Errors
# cov_type='HC3' parameter to .fit() function

# Log-Linear Regression

X = df[['bedrooms', 'bathrooms', 'sqft_living',
       'sqft_lot', 'floors', 'waterfront', 'view', 'condition', 'grade',
       'sqft_above', 'sqft_basement', 'yr_built',
       'sqft_living15', 'sqft_lot15']]

y = df['ln_price']

model = sm.OLS(y, X)
results = model.fit(cov_type='HC3')
print(results.summary())
```

# Function Form Misspecification

Say we wanted to fit a polynomial log-linear model to this data. How might we identify (besides visually) potential candidates for polynomial functions? First off, what does the eyeball test point out might be potential candidates for polynomial forms? Here come scatter plots again.

```python
target = 'ln_price'
features = df.columns.drop(target)
for feature in features:
    sns.lmplot(x=feature, y=target, data=df, scatter_kws=dict(alpha=0.1))
    plt.show()
```

I think sqft_living and sqft_above at a minimum are potential candidates for polynomial terms. I want to remind you what an underfit linear regression looks like:

This shows that the residuals of an underfit curved functional form will oscilate from negative residuals, to positive and then back to negative.

We might expect the residual plot to look something like this:

Truly, any bowing in our residuals is cause for concern. Lets plot the actual distribution of the residual graphs and see if our residuals match our eyeball test.

# Residual Plots

Plotting our residuals to see their distribution is an extremely useful model diagnostic technique. Lets get familiar with it.

The Seaborn library coming through like a champ, yet again.

```python
for feature in features:
    sns.residplot(X[feature], y, lowess=True, line_kws=dict(color='r'))
    plt.show()
```

From our residual plots, I think we can suspect that sqft_lot sqft_lot15 and yr_built all might be candidates for polynomial forms. Lets generate some squared terms and then re-plot the residuals graphs and see if we get any improvement.

```python
df['sqft_lot_squared'] = df['sqft_lot']**2
df['sqft_lot15_squared'] = df['sqft_lot15']**2
```

Lets also create a few features from our eyeball test and we'll see which ones seem to be more statistically significant.

```python
df['sqft_living_squared'] = df['sqft_living']**2
```

Lets add these to our regression and run it again to see if it has any considerable impact on coefficients.

```python
# log-polynomial? linear regression model with robust standard errors

# to use Robust Standard Errors pass:
# cov_type='HC3' parameter to .fit() function

X = df[['bedrooms', 'bathrooms', 'sqft_living', 'sqft_living_squared',
       'sqft_lot', 'sqft_lot_squared', 'floors', 'waterfront', 'view', 'condition', 'grade',
       'sqft_basement', 'sqft_living15',
       'sqft_lot15', 'sqft_lot15_squared']]

y = df['ln_price']

model = sm.OLS(y, X)
results = model.fit(cov_type='HC3')
print(results.summary())
```

```python
X = df[['bedrooms', 'bathrooms', 'sqft_living', 'sqft_living_squared',
       'sqft_lot', 'sqft_lot_squared', 'floors', 'waterfront', 'view', 'condition', 'grade',
       'sqft_basement', 'sqft_living15',
       'sqft_lot15']]

y = df['ln_price']

model = sm.OLS(y, X)
results = model.fit(cov_type='HC3')
print(results.summary())
```


# Lead-Acid Models

We compare a standard porous-electrode model for lead-acid batteries with two asymptotic reductions. For a more in-depth introduction to PyBaMM models, see the [SPM notebook](./SPM.ipynb). Further details on the models can be found in [[1]](#ref)

[1] Sulzer, V., Chapman, S. J., Please, C. P., Howey, D. A., & Monroe, C. W. (2019). “Faster Lead-Acid Battery Simulations from Porous Electrode Theory:  II. Asymptotic Analysis”. Journal of The Electrochemical Society, 166(12), A2372-A2382.

```python
%pip install pybamm -q    # install PyBaMM if it is not installed
import pybamm
import numpy as np
import os
import matplotlib.pyplot as plt
os.chdir(pybamm.__path__[0]+'/..')
```

### "Full" model

#### Electrolyte Concentration

$$
    \frac{\partial }{\partial t}\left(\epsilon c\right) = -\frac{\partial N}{\partial x} + sj, \\
    N = -\frac{\epsilon^b  D(c)}{\mathcal{C}_\text{e}}  \frac{\partial c}{\partial x}\\
    N\big|_{x=0}= N\big|_{x=1}=0, \\
    c\big|_{t=0} = 1
$$

#### Porosity

$$
    \frac{\partial \epsilon}{\partial t} = -\beta^\text{surf}j, \\
    \epsilon\big|_{t=0} = \epsilon^0
$$

#### Electrolyte Current

$$
    \frac{\partial i_{\text{e}}}{\partial x} = j, \\
    \mathcal{C}_\text{e} i_{\text{e}} = \epsilon_k^b \kappa(c)
        \left( \chi \frac{\partial}{\partial x}\log(c)
        - \frac{\partial\phi_{\text{e}}}{\partial x}\right)\\
    i_{\text{e}}\big|_{x=0}= i_{\text{e}}\big|_{x=1}=0,
$$

#### Electrode Current

$$
    \frac{\partial i_{\text{s}}}{\partial x} = -j,\\
    i_{\text{s}} = -\sigma\frac{\partial\phi_{\text{s}}}{\partial x},\\
    \phi_{\text{s}}\big|_{x=0}
        = i_{\text{s}}\big|_{x=l_\text{n}}
        = i_{\text{s}}\big|_{x=1-l_\text{p}} = 0, \\
    i_{\text{s}}\big|_{x=1}=\mathcal{I},
$$

#### interfacial current density

$$
j = \begin{cases}
    2j_0(c) \sinh\left(\eta\right), \quad &0 < x < l_\text{n} \\
    0, \quad &l_\text{n} < x < 1-l_\text{p} \\
    2j_0(c) \sinh\left(\eta\right), \quad &1-l_\text{p} < x < 1
\end{cases}
\\
\eta = \phi_{\text{s}} - \phi_\text{e} - U(c),
$$

This model is implemented in PyBaMM as the `Full` model

```python
full = pybamm.lead_acid.Full()
```

### "Leading-order" model

$$
    \frac{\mathrm{d} }{\mathrm{d} t}\left(\epsilon c\right) = (s_\text{n} - s_\text{p})\mathrm{I}, \\
    \frac{\mathrm{d} \epsilon}{\mathrm{d} t} = -\beta^\text{surf}j, \\
    j = \begin{cases}
        \mathrm{I}/l_\text{n}, \quad &0 < x < l_\text{n} \\
        0, \quad &l_\text{n} < x < 1-l_\text{p} \\
        -\mathrm{I}/l_\text{p}, \quad &1-l_\text{p} < x < 1
    \end{cases} \\
    \phi_\text{e} = -U_\text{n}(c) + \sinh^{-1}\left(\frac{\mathrm{I}}{2l_\text{n}j_{0\text{n}}(c)}\right) \\
    V = -\phi_\text{e} + U_\text{p}(c) - \sinh^{-1}\left(\frac{\mathrm{I}}{2l_\text{p}j_{0\text{p}}(c)}\right) \\
$$

This model is implemented in PyBaMM as `LOQS` (leading-order quasi-static)

```python
loqs = pybamm.lead_acid.LOQS()
```

### "Composite" model

#### Electrolyte Concentration

$$
    \frac{\partial }{\partial t}\left(\epsilon c\right) = -\frac{\partial N}{\partial x} + sj, \\
    N = -\frac{\epsilon^b  D(c)}{\mathcal{C}_\text{e}}  \frac{\partial c}{\partial x}\\
    N\big|_{x=0}= N\big|_{x=1}=0, \\
    c\big|_{t=0} = 1
$$

#### Porosity

$$
    \frac{\partial \epsilon}{\partial t} = -\beta^\text{surf}j, \\
    \epsilon\big|_{t=0} = \epsilon^0
$$

#### Interfacial current density

$$
j = \begin{cases}
        \mathrm{I}/l_\text{n}, \quad &0 < x < l_\text{n} \\
        0, \quad &l_\text{n} < x < 1-l_\text{p} \\
        -\mathrm{I}/l_\text{p}, \quad &1-l_\text{p} < x < 1
    \end{cases} \\
$$

#### Voltage

$$
V = U_{\text{eq}} + \eta_r + \eta_c + \Delta\Phi_{\text{Elec}} + \Delta\Phi_{\text{Solid}}
$$
where
\begin{align}
    & U_{\text{eq}} = U_p(\bar{c}_p) - U_n(\bar{c}_n), \\
     &\eta_{r} = -\sinh^{-1}\left(\frac{\mathcal{I}}{2\bar{j}_{0n} l_n}\right)
     -\sinh^{-1}\left(\frac{\mathcal{I}}{2\bar{j}_{0p} l_p}\right), \\
     &\eta_c =  \mathcal{C}_e\chi\left(\log(\bar{c}_{p}) - \log(\bar{c}_{n})\right), \\
    &\Delta \Phi_{\text{Elec}}= -\frac{\mathcal{C}_e \mathcal{I}}{\kappa(\bar{c})}\left(\frac{l_n}{3\epsilon_n^b} + \frac{l_s}{\epsilon_{s}^b} + \frac{l_p}{3\epsilon_p^b} \right),
     \label{eqn:SPMe:electrolyte_ohmic_losses} \\
     &\Delta \Phi_{\text{Solid}} =  -\frac{\mathcal{I}}{3}\left(\frac{l_p}{\sigma_p} + \frac{l_n}{\sigma_n} \right),
\end{align}
with
\begin{equation}
    \bar{\cdot}_{\text{n}} =  \frac{1}{l_n}\int_0^{l_n} \cdot_{\text{n}} \, \text{d}x, \quad
    \bar{\cdot}_{\text{p}} =  \frac{1}{l_p}\int_{1-l_p}^{1} \cdot_{\text{p}} \, \text{d}x,
\end{equation}

This model is implemented in PyBaMM as `Composite`

```python
composite = pybamm.lead_acid.Composite()
```

## Solving the models

We load process parameters for each model, using the same set of (default) parameters for all. In anticipation of changing the current later, we make current an input parameter

```python
# load models
models = [loqs, composite, full]

# process parameters
param = models[0].default_parameter_values
param["Current function [A]"] = "[input]"
for model in models:
    param.process_model(model)
```

Then, we discretise the models, using the default settings

```python
for model in models:
    # load and process default geometry
    geometry = model.default_geometry
    param.process_geometry(geometry)

    # discretise using default settings
    mesh = pybamm.Mesh(geometry, model.default_submesh_types, model.default_var_pts)
    disc = pybamm.Discretisation(mesh, model.default_spatial_methods)
    disc.process_model(model)
```

Finally, we solve each model using CasADi's solver and a current of 1A

```python
timer = pybamm.Timer()
solutions = {}
t_eval = np.linspace(0, 3600 * 17, 100) # time in seconds
solver = pybamm.CasadiSolver()
for model in models:
    start = timer.time()
    solution = solver.solve(model, t_eval, inputs={"Current function [A]": 1})
    end = timer.time()
    print("Solved the {} in {:.3f} seconds".format(model.name, end-start))
    solutions[model] = solution
```

    Solved the LOQS model in 0.239 seconds
    Solved the Composite model in 4.143 seconds
    Solved the Full model in 1.681 seconds

## Results

To plot the results, the variables are extracted from the solutions dictionary. For example, we can compare the voltages:

```python
for model in models:
    time = solutions[model]["Time [h]"].entries
    voltage = solutions[model]["Terminal voltage [V]"].entries
    plt.plot(time, voltage, lw=2, label=model.name)
plt.xlabel("Time [h]", fontsize=15)
plt.ylabel("Terminal voltage [V]", fontsize=15)
plt.legend(fontsize=15)
plt.show()
```

Alternatively, using `QuickPlot`, we can compare the values of some variables

```python
solution_values = [solutions[model] for model in models]
quick_plot = pybamm.QuickPlot(solution_values)
quick_plot.dynamic_plot();
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=17.0, step=0.17), Output()), _dom_classes=('…

If we update the current, setting it to be 20 A, we observe a greater discrepancy between the full model and the reduced-order models.

```python
t_eval = np.linspace(0, 3600, 100)
for model in models:
    solutions[model] = solver.solve(model, t_eval, inputs={"Current function [A]": 20})

# Plot
solution_values = [solutions[model] for model in models]
quick_plot = pybamm.QuickPlot(solution_values)
quick_plot.dynamic_plot();
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=3563.636363636364, step=35.63636363636364), …

```python

```


```python
import matplotlib.pyplot as plt
#%matplotlib inline

import pandas as pd
import numpy as np
from math import ceil
```

```python
import io
import requests
Content = requests.get("https://raw.githubusercontent.com/flmath-dirty/matrixes_in_erlang/master/results.cvs").content
LoadedTable= pd.read_csv(io.StringIO(Content.decode('utf-8')),skipinitialspace=True,
                  names=['Test type', 'Width', 'Height','Matrix representation','No. of calls', 'Execution time'])
```

# Introduction

## Test scenarios descriptions

The LoadedTable contains statistics gathered from running the script that generates a matrix (Width x Height) and then
runs with the [fprof](http://erlang.org/doc/man/fprof.html) following tests for each representation of matrix and sizes:

The tests categories:

 - one_rows_sums - sum of values from each row in the matrix

 - one_cols_sums - sum of values from each row in the matrix

 - get_value - gets each value from the matrix once separetly

 - set_value - sets each value from the matrix once separetly

Matrixes are represented as follow:
 - [sofs](http://erlang.org/doc/man/sofs.html): Set of the sets module is a way of representing relations (in strictly mathematical sense). In this case the matrix is represented as a relation (a function) F where domain is constructed from coordinates which is related to the matrix entry value ({x,y} F Value).

 To get values in specific row we define equivalence relation aRb iff thera are {x,y1} F a {x,y2} F b.
 This relation divides antidomain in abbstract classes. If we sum elements belonging to the abstract class with element {x,_} we will get sum of the elements from this specific row. We use the same logic to receive abstrac classes for columns.

 - [digraph](http://erlang.org/doc/man/digraph.html): We have a directed graph where each value is stored in vertex with the coordinate {x,y}. Aside of that each column and row has dedicated vertex with edges from corresponding coordinate vertexes directed at it to speed up rows and columns sums calculations.

 - [array](http://erlang.org/doc/man/array.html): The matrix is represented as a tuple {Width, Height, Array}, where the array has size Width * Height, all operations are done with simply calculating positions of the values in the array.

 - big tuple: The matrix is represented as a tuple {Width, Height, BigTuple} where the BigTuple is tuple used in the same way as the Array from previous point (note: arrays indexing starts from 0).

 - [map](http://erlang.org/doc/man/maps.html): A quite fresh data structure in Erlang, we map coordinates {x,y} to values.

 - List of the lists: Each of sublists of the list contains elements representing  values in the row. Since it visually resembles a matrix, it is quite often abused by coders. Heavy usage of [lists](http://erlang.org/doc/man/lists.html) module.

 I have run tests on the square matrixes of sizes 50x50, 100x100, 1000x1000. For some of the 1000x1000 tests fprof logs exceeded the dedicated 2TB disk size. For those the values are missing.

```python
LoadedTable.head()
```

      Test type
      Width
      Height
      Matrix representation
      No. of calls
      Execution time

      0
      one_rows_sums
      50
      50
      matrix_as_sofs
      20452
      48.280

      1
      one_rows_sums
      50
      50
      matrix_as_digraph
      20531
      59.981

      2
      one_rows_sums
      50
      50
      matrix_as_array
      18483
      36.792

      3
      one_rows_sums
      50
      50
      matrix_as_map
      13684
      42.790

      4
      one_rows_sums
      50
      50
      matrix_as_big_tuple
      5962
      12.147

## Preliminary processing

```python
from scipy import stats
from functools import reduce
AggregatedTable = (LoadedTable.groupby(['Test type','Matrix representation'])['Execution time']
                   .apply(list).reset_index(name='Execution Times List'))
```

Create aggregation table with execution time values for different size of matrixes stored in one list.

```python
AggregatedTable.head()
```

      Test type
      Matrix representation
      Execution Times List

      0
      get_value
      matrix_as_array
      [59.462, 232.601, 28018.33]

      1
      get_value
      matrix_as_big_tuple
      [25.176, 121.824, 12527.693]

      2
      get_value
      matrix_as_digraph
      [42.854, 211.419, 19704.339]

      3
      get_value
      matrix_as_list_of_lists
      [280.91, 2073.251]

      4
      get_value
      matrix_as_map
      [36.883, 159.307, 15976.373]

Each list should have 3 values, if one is missing it means it exceeded testing capability of my equipment, and can be considered infinite. Also add columns with minimum values, maximum values and harmonic averages (since standard average would be too susceptible to outliers here).

```python
AggregatedTable['Harmonic Average'] = AggregatedTable['Execution Times List'].apply(
    lambda x: stats.hmean(x) if (len(x) == 3) else np.inf)
```

```python
AggregatedTable['Max'] = AggregatedTable['Execution Times List'].apply(
    lambda x: max(x) if (len(x) == 3) else np.inf)
```

```python
AggregatedTable['Min'] = AggregatedTable['Execution Times List'].apply(min)
```

```python
AggregatedTable.head()
```

      Test type
      Matrix representation
      Execution Times List
      Harmonic Average
      Max
      Min

      0
      get_value
      matrix_as_array
      [59.462, 232.601, 28018.33]
      141.828134
      2.801833e+04
      59.462

      1
      get_value
      matrix_as_big_tuple
      [25.176, 121.824, 12527.693]
      62.488602
      1.252769e+04
      25.176

      2
      get_value
      matrix_as_digraph
      [42.854, 211.419, 19704.339]
      106.701803
      1.970434e+04
      42.854

      3
      get_value
      matrix_as_list_of_lists
      [280.91, 2073.251]
      inf
      inf
      280.910

      4
      get_value
      matrix_as_map
      [36.883, 159.307, 15976.373]
      89.679282
      1.597637e+04
      36.883

# Comparing operations

## Retrieving values

 Lets compare execution times for retrieving values.

```python
GetValue = (
    AggregatedTable
    .loc[AggregatedTable['Test type']=='get_value']
    .loc[:,['Matrix representation','Harmonic Average','Max','Min']]
    )

GetValue = GetValue.set_index('Matrix representation')
```

```python
from jupyter_extensions.matplotlib.sawed_bar_plot import sawed_bar_plot
```

```python
sawed_bar_plot(GetValue)
```

It occurs that lists of the list and set of sets implementation is extremely inefficient at retrieving values.

The list of the lists sounds obvious since to get element in the middle we need to traverse to middle list representing row and then again in the row traverse to the middle element which gives O(n^2) access time.

The sofs occurs to be a trap, if you will read documentation you will see elegant (from mathematician point of view) relation algebra. But if you go inside the source code you will find that module extensively uses lists:fold which translates implementation to the list of lists scenario in the best case. We can probaly make the first conclusion and an advice here: Don't abuse sofs, remember map is actually relation too (precisely function, but noone can forbid you from returning lists or other containers).

Other implementations look resonable. The are some suprises, like the better perfomance of big tuples than arrays.

Let see set_value functions perfomances.

## Storing values

```python
SetValue = (
    AggregatedTable
    .loc[AggregatedTable['Test type']=='set_value']
    .loc[:,['Matrix representation','Harmonic Average','Max','Min']]
    )
SetValue = SetValue.set_index('Matrix representation')
```

```python
sawed_bar_plot(SetValue)
```

The lists of lists and softs are outclassed even for small matrixes like 10x10.

As we suspected the big_tuple is much worse at setting values since it is immutable and setting one value means copying whole tuple. We can notice that the array even if it is more efficient has some values above the rift, lets change a little perspective:

```python
sawed_bar_plot(SetValue,50000)
```

Lets zoom a little bit to see small values.

```python
sawed_bar_plot(SetValue,500)
```

The array is much better than tuple, but both are magnitude worse than directed graph and matrix. It is now worth mentioning that digraph are implemented as an ets table which suggest a fixed overhead at first, but good efficiency for big tables. Still implementation with maps (build in ERTS) is faster even for 1000x1000 matrixes. We will investigate execution times increse further in the document.

Since digraph represented as an ets database is quite efficient is worth to ask and keep in mind a question: if we can craft matrix representation with ets table that will be more efficient than map. I will come back to this question at the end of this document.

The need for this small research occured during implemetation of a statistics gathering module. It was important to be able to calculate sums of specific counters. Usually in that case the best way is to make interface to some c or python application which support fast calculating. But this increases the complexity of the product, which should be usually avoided. Let check a performance of columns summation.

## Summing columns

```python
OneColsSums = (
    AggregatedTable
    .loc[AggregatedTable['Test type']=='one_cols_sums']
    .loc[:,['Matrix representation','Harmonic Average','Max','Min']]
    )

OneColsSums = OneColsSums.set_index('Matrix representation')
```

```python
sawed_bar_plot(OneColsSums,200)
```

It is very confusing result, both array and tuple calculates the needed n indexes (in very C style) and sum extracted values (should be o(n^2), n calculations times n columns to calculate), still array is slower for big numbers than map which has to try to find value with hash (o(log(n)*n^2)), suggesting some inefficiency in implementation.

## Summing rows

Lets look at summing rows:

```python
OneRowsSums = (
    AggregatedTable
    .loc[AggregatedTable['Test type']=='one_rows_sums']
    .loc[:,['Matrix representation','Harmonic Average','Max','Min']]
    )

OneRowsSums = OneRowsSums.set_index('Matrix representation')
```

```python
sawed_bar_plot(OneRowsSums)
```

Results are similar to the previous, with exception of the list of the list, which has obvious advantage since the dominating operation is summation of the list implemented as a linked list. Anyway since setting a value is extremly expensive, this implementation is still not really good.

# Matrix size comparisons

## Small matrixes

### Lets remove the sofs and the lists of lists from further considerations.

The next thing we should consider is which implementation is the best for small matrixes.

```python
AggregatedTableRestricted = AggregatedTable[
            (AggregatedTable['Matrix representation']!='matrix_as_list_of_lists') &
            (AggregatedTable['Matrix representation']!='matrix_as_sofs')]
AggregatedTableRestricted.head()
```

      Test type
      Matrix representation
      Execution Times List
      Harmonic Average
      Max
      Min

      0
      get_value
      matrix_as_array
      [59.462, 232.601, 28018.33]
      141.828134
      28018.330
      59.462

      1
      get_value
      matrix_as_big_tuple
      [25.176, 121.824, 12527.693]
      62.488602
      12527.693
      25.176

      2
      get_value
      matrix_as_digraph
      [42.854, 211.419, 19704.339]
      106.701803
      19704.339
      42.854

      4
      get_value
      matrix_as_map
      [36.883, 159.307, 15976.373]
      89.679282
      15976.373
      36.883

      6
      one_cols_sums
      matrix_as_array
      [37.541, 176.126, 19173.0]
      92.685699
      19173.000
      37.541

```python
Minimums = (
    AggregatedTableRestricted
    .loc[:,['Test type','Matrix representation','Min']]
    )
#Minimums.groupby(['Test type','Matrix representation'])
Minimums = Minimums.set_index(['Test type','Matrix representation'])
Minimums.head()
```

      Min

      Test type
      Matrix representation

      get_value
      matrix_as_array
      59.462

      matrix_as_big_tuple
      25.176

      matrix_as_digraph
      42.854

      matrix_as_map
      36.883

      one_cols_sums
      matrix_as_array
      37.541

```python
TestTypes = Minimums.index.get_level_values('Test type').unique().tolist()
#MatrixRepresentations = Minimums.index.get_level_values('Matrix representation').unique().tolist()
Plot = Minimums.unstack().plot(kind='bar',figsize=(12,9))
Plot.set_xticklabels(TestTypes,rotation="horizontal")
handles, labels=Plot.get_legend_handles_labels()
labels = [Label[6:-1] for Label in labels]
Plot.legend(handles,labels)
plt.show()
```

It occurs that for small values a big tuple is quite a good implementation, unlsess we are going to use our matrix for setting heavy tasks, in that case it is better to use map implementation.

Now we can see how those implementation efficiency looks for relatively(*) big 1000 x 1000 matrixes.

(*) If you really want something big you need to interface other programming language.

## Big matrixes

```python
Maximums = (
    AggregatedTableRestricted
    .loc[:,['Test type','Matrix representation','Max']]
    )
Maximums = Maximums.set_index(['Test type','Matrix representation'])
Maximums.head()
```

      Max

      Test type
      Matrix representation

      get_value
      matrix_as_array
      28018.330

      matrix_as_big_tuple
      12527.693

      matrix_as_digraph
      19704.339

      matrix_as_map
      15976.373

      one_cols_sums
      matrix_as_array
      19173.000

```python
TestTypes = Maximums.index.get_level_values('Test type').unique().tolist()
Plot = Maximums.unstack().plot(kind='bar',figsize=(12,9))
Plot.set_xticklabels(TestTypes,rotation="horizontal")
handles, labels = Plot.get_legend_handles_labels()
labels = [Label[6:-1] for Label in labels]
Plot.legend(handles,labels)
plt.show()
```

It looks that set_value operations on the big tuple are overwhelmingly slow, to the level that they obfuscating our plot.
Lets remove them from the result and compare operations that left.

```python
 Maximums.loc['set_value','matrix_as_big_tuple']=np.nan
```

```python
TestTypes = Maximums.index.get_level_values('Test type').unique().tolist()
Plot = Maximums.unstack().plot(kind='bar',figsize=(12,9))
Plot.set_xticklabels(TestTypes,rotation="horizontal")
handles, labels=Plot.get_legend_handles_labels()
labels = [Label[6:-1] for Label in labels]
Plot.legend(handles,labels)
plt.show()
```

So the big tuple with calculated index values occurs not to scale great. It is not suprising, since tuple is immutable, every time you set a new value you need to copy all of them, what makes it beyond redemption when it comes to storing operations.

Overall the map implementation is overall winner here, and it is overall recommendated implementation. But before we congratulate ourselves, we can notice that digraph implementation even if it is slower, maybe is only slower because overhead it implements over ets tables, and with futher increase of matrixes sizes it actually become better than maps.

## Growth acceleration estimation

To get a little more insight, that would help us estimate if this scenario happens we should find a way to compare how quickly the time consumption functions grow.

Lets try to figure out an acceleration of the resource consumption increase.

For each index ([Test type, Matrix representation]) we are getting a function F that mapping the matrixes sizes:
 - 50 x 50 = 2500 cells = 25h(hecto)
 - 100 x 100 = 10k (kilo) = 100h
 - 1000 x 1000 = 1m (mega)
into the amount of resources consumed by particular test on that matrix.

See the picture below:

We want to get a hint how fast the particular function growth will accelerate.

We will start from visualising scale of distances we are dealing here with:

 - 25hTo10k = 10k - 25h = 3 x  25h
 - 10kTo1m  = 1m - 10k  = 99 x 10k = 99 x 4 x 25h = 396 x 25h

For picture above we want to find out the increase of the increase or bind to it the angle R. In our simplified calculus we can consider finding an increase as finding derivate. Since one of interpretations of derivate is actually tangens, what we can look for if we want to comparable estimates of R we can use tan(R) (since tanges is monotonic in $(\frac{-\pi}2, \frac\pi2)$).

But before we will try to find R values, we need to notice that the increase of the increase to the projected increase ratio is depended on value of the 25h, so does the R.

We want to remove this sensitivity to initial conditions, to do that we scale all the mappings, so the value of the 25h is the same for each of them.

```python
pd.set_option('mode.chained_assignment',None)
AggregatedTableRestricted['Scaled Execution Times List'] = AggregatedTableRestricted['Execution Times List'].apply(
    lambda X: [(z * 100)/X[0] for z in X])
pd.set_option('mode.chained_assignment','warn')
```

```python
AggregatedTableRestricted.head()
```

      Test type
      Matrix representation
      Execution Times List
      Harmonic Average
      Max
      Min
      Scaled Execution Times List

      0
      get_value
      matrix_as_array
      [59.462, 232.601, 28018.33]
      141.828134
      28018.330
      59.462
      [100.0, 391.175877031, 47119.7235209]

      1
      get_value
      matrix_as_big_tuple
      [25.176, 121.824, 12527.693]
      62.488602
      12527.693
      25.176
      [100.0, 483.889418494, 49760.4583731]

      2
      get_value
      matrix_as_digraph
      [42.854, 211.419, 19704.339]
      106.701803
      19704.339
      42.854
      [100.0, 493.347178793, 45980.1628786]

      4
      get_value
      matrix_as_map
      [36.883, 159.307, 15976.373]
      89.679282
      15976.373
      36.883
      [100.0, 431.925277228, 43316.3598406]

      6
      one_cols_sums
      matrix_as_array
      [37.541, 176.126, 19173.0]
      92.685699
      19173.000
      37.541
      [100.0, 469.156389015, 51072.1611039]

Now by definition:

tan(B) = the projected increase/(396 x 25h)

tan(R+B) = 10kTo1mValueIncrease/(396 x 25h)

Since denominator is the same we don't need to consider it here. Lets calculate those increases.

```python
pd.set_option('mode.chained_assignment',None)
AggregatedTableRestricted['ProjectedIncrease'] = AggregatedTableRestricted['Scaled Execution Times List'].apply(
    lambda x: 396*(x[1]-x[0]))
AggregatedTableRestricted['ActualIncrease'] = AggregatedTableRestricted['Scaled Execution Times List'].apply(
    lambda x: x[2]-x[1])
pd.set_option('mode.chained_assignment','warn')
```

```python
AggregatedTableRestricted.head()
```

      Test type
      Matrix representation
      Execution Times List
      Harmonic Average
      Max
      Min
      Scaled Execution Times List
      ProjectedIncrease
      ActualIncrease

      0
      get_value
      matrix_as_array
      [59.462, 232.601, 28018.33]
      141.828134
      28018.330
      59.462
      [100.0, 391.175877031, 47119.7235209]
      115305.647304
      46728.547644

      1
      get_value
      matrix_as_big_tuple
      [25.176, 121.824, 12527.693]
      62.488602
      12527.693
      25.176
      [100.0, 483.889418494, 49760.4583731]
      152020.209724
      49276.568955

      2
      get_value
      matrix_as_digraph
      [42.854, 211.419, 19704.339]
      106.701803
      19704.339
      42.854
      [100.0, 493.347178793, 45980.1628786]
      155765.482802
      45486.815700

      4
      get_value
      matrix_as_map
      [36.883, 159.307, 15976.373]
      89.679282
      15976.373
      36.883
      [100.0, 431.925277228, 43316.3598406]
      131442.409782
      42884.434563

      6
      one_cols_sums
      matrix_as_array
      [37.541, 176.126, 19173.0]
      92.685699
      19173.000
      37.541
      [100.0, 469.156389015, 51072.1611039]
      146185.930050
      50603.004715

We have tan(R+B) and tan(B), but we want to calculate tan(R).

Fortunately there are identities that can help us with that.

```python
import sympy as sp
R, B = sp.symbols('R B')
tanRplusB = sp.expand_trig(sp.tan(R + B))
tanRplusB
```

    (tan(B) + tan(R))/(-tan(B)*tan(R) + 1)

We want to experess tan(R).

SymPy trigonometric expansion/simplification functions would give us quite complicated answers (including atan function). But we just want to treat tan(R) as rational function of the other tanges, so we make substitution.
 - tan(R+B) = C
 - tan(B) = B
 - tan(R) = R

```python
Ratio= tanRplusB.subs(sp.tan(R),R).subs(sp.tan(B),B)
C = sp.symbols('C')
RatioSolution = sp.solve(sp.Eq(Ratio , C),R)
RatioSolution
```

    [(-B + C)/(B*C + 1)]

```python
FinalSolution = RatioSolution[0].subs(B,sp.tan(B)).subs(C,sp.tan(R+B))
FinalSolution
```

    (-tan(B) + tan(B + R))/(tan(B)*tan(B + R) + 1)

Double check that result is actually valid:

```python
sp.simplify(FinalSolution-sp.tan(R)) == 0
```

    True

Lets apply the result equation on our data:

```python
pd.set_option('mode.chained_assignment',None)
AggregatedTableRestricted['TanB'] = AggregatedTableRestricted['ProjectedIncrease'].apply(
    lambda x: x/396)
AggregatedTableRestricted['TanAplusB'] = AggregatedTableRestricted['ActualIncrease'].apply(
    lambda x: x/396)
AggregatedTableRestricted['TanA'] = (
    (AggregatedTableRestricted['TanAplusB']-AggregatedTableRestricted['TanB'])/((AggregatedTableRestricted['TanAplusB']*AggregatedTableRestricted['TanB'])+1))
pd.set_option('mode.chained_assignment','warn')
```

```python
Diffs = (
    AggregatedTableRestricted
    .loc[:,['Test type','Matrix representation','TanA']]
    )
#Minimums.groupby(['Test type','Matrix representation'])
Diffs = Diffs.set_index(['Test type','Matrix representation'])

```

```python
TestTypes = Diffs.index.get_level_values('Test type').unique().tolist()
Plot = Diffs.unstack().plot(kind='bar',figsize=(12,9))
Plot.set_xticklabels(TestTypes,rotation="horizontal")
handles, labels=Plot.get_legend_handles_labels()
labels = [Label[7:-1] for Label in labels]
Plot.legend(handles,labels)
plt.show()
```

Bars in this plot represent growth acceleration, which is related to the second derivative of the function.

# Conclusion

It looks that maps implementation is actually the best choice. It is also hearth-warming that majority of operations execution times growth is slower than a linear one.

What left for another articles are implementations with ets tables with different keys formats, which I will compare to our map representation and also map with different formats of keys (key as list, binary ets. instead of tuple).
I'm also considering implementing matrix multiplication instead columns and rows sums (those operations are comparable, if it comes to complexity).

After that I have plans to more sophisticated implementations using Erlang main advantage, possibility to store values in small processes.


# Quantum Integer Programming (QuIP) 47-779. Fall 2020, CMU
## Quiz 2

### Problem statement
#### Integer linear program
Solve the following problem
$$
\min_{\mathbf{x}} 2𝑥_0+4𝑥_1+4𝑥_2+4𝑥_3+4𝑥_4+4𝑥_5+5𝑥_6+4𝑥_7+5𝑥_8+6𝑥_9+5𝑥_{10} \\
s.t. \begin{bmatrix}
1 & 0 & 0 & 1 & 1 & 1 & 0 & 1 & 1 & 1 & 1\\
0 & 1 & 0 & 1 & 0 & 1 & 1 & 0 & 1 & 1 & 1\\
0 & 0 & 1 & 0 & 1 & 0 & 1 & 1 & 1 & 1 & 1
\end{bmatrix}\mathbf{x}=
\begin{bmatrix}
1\\
1\\
1
\end{bmatrix} \\
\mathbf{x} \in \{0,1 \}^{11}
$$
Equivalently written as

$$
\min_{\mathbf{x}} \mathbf{c}^\top \mathbf{x}\\
s.t. \mathbf{A}\mathbf{x}=\mathbf{b} \\
\mathbf{x} \in \{0,1 \}^{11}
$$

```python
# Import Matplotlib to generate plots
import matplotlib.pyplot as plt
# Import numpy and scipy for certain numerical calculations below
import numpy as np
from scipy.special import gamma
import math
```

```python
from sympy import *
import networkx as nx
```

## BPT Method

```python
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, z = symbols('x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 z')
eqs = [2*x0+4*x1+4*x2+4*x3+4*x4+4*x5+5*x6+4*x7+5*x8+6*x9+5*x10-z,
       x0+x3+x4+x5+x7+x8+x9+x10-1,
       x1+x3+x5+x6+x8+x9+x10-1,
       x2+x4+x6+x7+x8+x9+x10-1]
eqs = eqs + [x*(x-1) for x in [x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10]]

result = groebner(eqs, x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, z, order = 'lex')

list(result)
```

    [40*x0 - 7*z**4 + 202*z**3 - 2137*z**2 + 9822*z - 16560,
     120*x1 - 120*x4 - 120*x7 - z**4 + 26*z**3 - 251*z**2 + 1066*z - 1680,
     120*x2 + 120*x4 + 120*x7 + 9*z**4 - 254*z**3 + 2619*z**2 - 11734*z + 19320,
     12*x3 + 12*x4 + 12*x5 + 12*x7 + z**4 - 28*z**3 + 287*z**2 - 1280*z + 2100,
     x4**2 - x4,
     x4*x5,
     x4*x7,
     x10*x4,
     x4*z - 8*x4,
     x5**2 - x5,
     x5*x7,
     x10*x5,
     x5*z - 8*x5,
     6*x6 - z**4 + 29*z**3 - 308*z**2 + 1420*z - 2400,
     x7**2 - x7,
     x10*x7,
     x7*z - 8*x7,
     30*x10 + 30*x8 - z**4 + 31*z**3 - 356*z**2 + 1796*z - 3360,
     8*x9 + z**4 - 30*z**3 + 331*z**2 - 1590*z + 2800,
     x10**2 - x10,
     x10*z - 5*x10,
     z**5 - 36*z**4 + 511*z**3 - 3576*z**2 + 12340*z - 16800]

```python
zs = solve(result[-1],z)
zstar = min(zs)
print("zs:",zs)
print("zstar:",zstar)
```

    zs: [5, 6, 7, 8, 10]
    zstar: 5

## CT Method

```python
z1,z2,z3,w0,w1,w2,w3,w4,w5,w6,w7,w8,w9,w10 = symbols('z1 z2 z3 w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10')
eqs = [z1-w0,z2-w1,z3-w2,z1*z2-w3,z1*z3-w4,z1*z2-w5,z2*z3-w6,z1*z3-w7,z1*z2*z3-w8,z1*z2*z3-w9,z1*z2*z3-w10]
result = groebner(eqs,z1,z2,z3,w0,w1,w2,w3,w4,w5,w6,w7,w8,w9,w10, order='lex')
result = groebner(eqs,z1,z2,z3,w10,w9,w8,w7,w6,w5,w4,w3,w2,w1,w0, order='lex')

list(result)
```

    [-w0 + z1,
     -w1 + z2,
     -w2 + z3,
     -w0*w1*w2 + w10,
     -w0*w1*w2 + w9,
     -w0*w1*w2 + w8,
     -w0*w2 + w7,
     -w1*w2 + w6,
     -w0*w1 + w5,
     -w0*w2 + w4,
     -w0*w1 + w3]

```python
r = z1**1*z2**1*z3**1
r = r.subs({(z1,w0),(z2,w1),(z3,w2)})
print(r)
r1 = r.subs({(w0*w1*w2, w10)})
print(r1)
r2 = r.subs({(w0*w1*w2, w9)})
print(r2)
r3 = r.subs({(w0*w1*w2, w8)})
print(r3)
r4 = r.subs({(w0*w2, w7)})
print(r4)
r5 = r.subs({(w1*w2, w6)})
print(r5)
r6 = r.subs({(w0*w1, w5)})
print(r6)
r7 = r.subs({(w0*w2, w4)})
print(r7)
r8 = r.subs({(w0*w1, w3)})
print(r8)

```

    w0*w1*w2
    w10
    w9
    w8
    w1*w7
    w0*w6
    w2*w5
    w1*w4
    w2*w3

This means that
- $x_{0}=1，x_{1}=1，x_{2}=1$ is a feasible solution whose objective function is 10;

- $x_{10}=1$ is a feasible solution whose objective function is 5;

- $x_{9}=1$ is a feasible solution whose objective function is 5;

- $x_{8}=1$ is a feasible solution whose objective function is 5;

- $x_{1}=1，x_{7}=1$ is a feasible solution whose objective function is 8;

- $x_{0}=1，x_{6}=1$ is a feasible solution whose objective function is 7;

- $x_{2}=1，x_{5}=1$ is a feasible solution whose objective function is 8;

- $x_{1}=1，x_{4}=1$ is a feasible solution whose objective function is 8;

- $x_{2}=1，x_{3}=1$ is a feasible solution whose objective function is 8.

In conclusion, the best solution is 5.

```python

```


# Surfinpy

#### Tutorial 3 - Pressure

In the previous tutorials we went through the process of generating a simple phase diagram for bulk phases and introducing temperature dependence for gaseous species.  This useful however, sometimes it can be more beneficial to convert the chemical potenials (eVs) to partial presure (bar).

Chemical potential can be converted to pressure values using

\begin{align}
P & = \frac{\mu_O}{k_B T} ,
\end{align}

where P is the pressure, $\mu$ is the chemical potential of oxygen, $k_B$ is the Boltzmnann constant and T is the temperature.

```python
import matplotlib.pyplot as plt

from surfinpy import bulk_mu_vs_mu as bmvm
from surfinpy import utils as ut
from surfinpy import data

colors = ['#5B9BD5', '#4472C4', '#A5A5A5', '#772C24', '#ED7D31', '#FFC000', '#70AD47']
```

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 1, energy = -92.0, funits = 10)
```

Additionally, SurfinPy has the functionality to allow you to choose which colours are used for each phase.  Specify within the DataSet class color.

```python
Bulk = data.DataSet(cation = 10, x = 0, y = 0, energy = -92.0, color=colors[0], label = "Bulk")
D = data.DataSet(cation = 10, x = 10, y = 0, energy = -310.0, color=colors[1], label = "D")
B = data.DataSet(cation = 10, x = 0, y = 10, energy = -228.0, color=colors[2], label = "B")
F = data.DataSet(cation = 10, x = 8, y = 10, energy = -398.0, color=colors[3], label = "F")
A = data.DataSet(cation = 10, x = 5, y = 20, energy = -470.0, color=colors[4], label = "A")
C = data.DataSet(cation = 10, x = 10, y = 30, energy = -706.0, color=colors[5], label = "C")
E = data.DataSet(cation = 10, x = 10, y = 50, energy = -972.0, color=colors[6], label = "E")

```

```python
data = [Bulk, A, B, C,  D, E, F]
```

```python
x_energy=-20.53412969
y_energy=-12.83725889
```

```python
CO2_exp = ut.fit_nist("ref_files/CO2.txt")[298]
Water_exp = ut.fit_nist("ref_files/H2O.txt")[298]

CO2_corrected = x_energy + CO2_exp
Water_corrected = y_energy + Water_exp

deltaX = {'Range': [ -1, 0.6],  'Label': 'CO_2'}
deltaY = {'Range': [ -1, 0.6], 'Label': 'H_2O'}
```

```python
system = bmvm.calculate(data, bulk, deltaX, deltaY, x_energy=CO2_corrected, y_energy=Water_corrected)

ax = system.plot_phase(figsize=(6, 4.5))

plt.show()
```

To convert chemical potential to pressure use the plot_pressure command and the temperature at which the pressure is calculated.  For this example we have used 298 K.

```python
import surfinpy
ax = system.plot_pressure(temperature=298, figsize=(5, 4), cbar_title="Bulk Phase System")
plt.show()
```

```python

```


# Get Unique Lambdas from H (plus other restrictions)

The only objective of this notebooks it to get a set of unique lambdas from H plus some extra assumptions.
Numerical methods would be fine, though riskier (since a random start would give different lambdas, bringing unstability to the maximization GMM procedure)

+ $Z$ = firm data
+ $\theta$ = deep parameters
+ $H = g(Z'\theta)$, $Eβ = m(Z'\theta)$

Thus, for a given $\theta$ try (GMM try), we have a fixed $(H, Eβ)$. This is our starting point and we should get $\lambda$s from then (hopefully just one set!!)

```python
from scipy.stats import entropy
from scipy import optimize
import numpy as np
import sympy as sp
sp.init_printing()
```

```python
p1, p2, p3 = sp.symbols('p1 p2 p3')
h_sp = -(p1*sp.log(p1) + p2*sp.log(p2) + (1 - p1 - p2)*sp.log(1 - p1 - p2))

sp.simplify(sp.diff(h_sp, p2))
```

```python
b1, b2, b3 = sp.symbols('b1 b2 b3')
eb_sp = b1 *p1 + b2*p2 + b3*(1-p1-p2)
sp.simplify(sp.diff(eb_sp, p2))
```

    b2 - b3

```python
m = np.array([[2, 3, 4], [2, 1, 2]])
m
```

    array([[2, 3, 4],
           [2, 1, 2]])

```python
m.shape
```

    (2, 3)

```python
Eβ = 1.2
βs = [0.7, 1.1, 1.5] # Corresponding to each lambda
H = 0.95
#θ = 0.1

def my_entropy(p):
    return -np.sum(p * np.log(p))

def x_to_lambdas(x):
    return [x[0], x[1], 1 - x[0] - x[1]]

# Set of lambdas that solve two equations
def fun(x):
    lambdas = x_to_lambdas(x)
    return [my_entropy(lambdas) - H,
         np.dot(βs, lambdas) - Eβ]

#sol = optimize.root(fun, [0.2, 0.1]) --> [0.311, 0.12, 0.56]
sol = optimize.root(fun, [0.1, 0.4])# --> [0.111, 0.52, 0.36]
print(sol.message)

lambdas_sol = x_to_lambdas(sol.x)
print(lambdas_sol)
print("Values: ", H, Eβ)
entropy(lambdas_sol), np.dot(βs, lambdas_sol)

```

    The solution converged.
    [0.11150149216228898, 0.5269970156754217, 0.36150149216228933]
    Values:  0.95 1.2

    (0.9500000000000002, 1.2000000000000002)

```python
#With a jacobian

def jac(x):
    dh_dx = np.array([-np.log(x[0]), -np.log(x[1])]) + np.log(1-x[0]-x[1])

    deb_dx = np.array([βs[0], βs[1]]) -  βs[2]

    return np.array([dh_dx, deb_dx])

sol = optimize.root(fun, [0.8, 0.1], jac=jac)# --> [0.111, 0.52, 0.36]
print(sol.message)

lambdas_sol = x_to_lambdas(sol.x)
print(lambdas_sol)
print("Values: ", H, Eβ)
entropy(lambdas_sol), np.dot(βs, lambdas_sol)
```

```python
# Set of lambdas that solve just the H equation

def fun(x):
    lambdas = x_to_lambdas(x)
    return [entropy(lambdas) - H, 0.]

sol = optimize.root(fun, [0.1, 0.05])
lambdas_sol = x_to_lambdas(sol.x)
print(sol.message)
print(lambdas_sol)
print("True H: ", H, " . Obtained: ", my_entropy(lambdas_sol))

```

    The solution converged.
    [0.4493011313143122, 0.10072317846500357, 0.4499756902206842]
    True H:  0.95  . Obtained:  0.95

## Reparametrise probabilities so they are between 0 and 1

$$ p =  {\exp(x) \over 1 + \exp(x)}$$

```python
x1, x2, x3 = sp.symbols('x1 x2 x3')
#p1 = sp.exp(x1) / (1 + sp.exp(x1))
#p2 = sp.exp(x2) / (1 + sp.exp(x2))
p1 = 1 / (1 + sp.exp(-x1))
p2 = 1 / (1 + sp.exp(-x2))
h_sp = sp.simplify(-(p1*sp.log(p1) + p2*sp.log(p2) + (1 - p1 - p2)*sp.log(1 - p1 - p2)))

sp.simplify(sp.diff(h_sp, x1))

```

```python
sp.simplify(sp.diff(h_sp, x2))
```

```python
b1, b2, b3 = sp.symbols('b1 b2 b3')
eb_sp = b1 *p1 + b2*p2 + b3*(1-p1-p2)
sp.diff(eb_sp, x1)
```

```python
sp.diff(eb_sp, x2)
```

```python
def logit(p):
    return np.log(p / (1 - p))

def x_to_p(x):
    """ inverse logit"""
    return np.e**(x) / (1 + np.e**(x))

def fun(x):
    lambdas = x_to_lambdas(x_to_p(x))
    return [my_entropy(lambdas) - H,
         np.dot(βs, lambdas) - Eβ]

def jac(x):
    block = np.log( (1 - np.e**(x[0] + x[1]) )   / (np.e**(x[0])+np.e**(x[1])+np.e**(x[0]+x[1]) + 1))
    num0 = (-np.log(  np.e**x[0] / ( np.e**x[0] + 1)) + block  )*np.e**x[0]
    den0 = np.e**(2*x[0]) + 2*np.e**(x[0]) + 1
    num1 =(-np.log(  np.e**x[1] / ( np.e**x[1] + 1)) + block  )*np.e**x[1]
    den1 =np.e**(2*x[1]) + 2*np.e**(x[1]) + 1

    dh_dx = np.array([num0/den0, num1/den1])

    deb_0 = ((βs[0] - βs[2])*np.e**(-x[0])) / (1 + np.e**(-x[0]))**2
    deb_1 = ((βs[1] - βs[2])*np.e**(-x[1])) / (1 + np.e**(-x[1]))**2
    deb_dx = np.array([deb_0, deb_1])

    return np.array([dh_dx, deb_dx])

sol = optimize.root(fun, logit(np.array([0.1, 0.01])), jac=jac)
print(sol.message)

lambdas_sol = x_to_lambdas(x_to_p(sol.x))
print(lambdas_sol)
print("Values: ", H, Eβ)
my_entropy(lambdas_sol), np.dot(βs, lambdas_sol)
```

```python

```

```python
%matplotlib inline
import matplotlib.pyplot as plt
x = np.linspace(-100, 100, 1000)
y = np.e**x / ( 1 + np.e**x)
y2 = y = 1 / ( 1 + np.e**(-x))
plt.plot(x, y)
plt.plot(x, y2)
plt.plot(x, y -y2, label="hola")
plt.legend()
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
y = 1 / ( 1 + np.e**(-x))

plt.plot(x, y)
```

```python

```