+ This notebook is part of lecture 27 *Positive definite matrices and minima* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, Matrix, symbols, Derivative, diff
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

```python
a, b, c, d, lamda, x1, x2, x3 = symbols('a b c d lamda x1 x2 x3')
```

# Tests for positive definite matrices
# Tests for minimum **x**TA**x**>0
# Ellipsoids in ℝn

## When is a symmetric matrix positive definite

* Let's first consider the 2×2 matrix

* Tests for complete definitiveness
    * λ1>0, λ2>0
    * *a*>0, *ac*-*b*2
    * The pivots larger than zero
    $$ {a}>0;\quad\frac{{ac}-{b}^{2}}{a}>0 $$
    * **x**TA**x**>0

* Let's look at some example matrices

```python
A = Matrix([[2, 6], [6, a]])
A
```

* The first question is what value of *a* would make this symmetric matrix positive definite
* The second would be, which of the tests above would you use

* The second question first
    * Seems the determinant tests would suffice
    * We need 2*a*-36>0
* The first question is the answered
    * *a* must therefor be larger than 18

* Let's play around by making *a* equal to 18

```python
A = Matrix([[2, 6], [6, 18]])
A
```

```python
A.charpoly(lamda)
```

```python
A.eigenvals()
```

* One of the eigenvalues is zero (after all, it is a singular matrix now and one eigenvalues must be zero)
* It is a 2×2 matrix and we must have two eigenvalues
* The other eigenvalue must equal the trace of A, which is 20 (therefor there was no need to calculate the eigenvalues, we could just reason and read it off)
* We'll call this matrix *positive semi-definite*

* Notice that the pivot test would not have helped
$$ \frac{(6)(2)-{6}^{2}}{{2}} $$

* Let's look at **x**TA**x**>0 (where **x** is any correctly-sized vector)

```python
x_vect = Matrix([x1, x2])
x_vect
```

```python
f = x_vect.transpose() * A * x_vect
f
```

```python
f.expand() # Expanding the expression shows it is no quadratic (not linear anymore)
```

* For A to be positive definite, this quadratic must be positive for all values of *x*i

* Below I use some 3D plotting
* Not too clear to see, but note that nowhere does plot go below zero on the *z*-axis

```python
import scipy as sp
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

%matplotlib inline
```

```python
fig = plt.figure(figsize = (10, 8))
ax = Axes3D(fig)
x = sp.linspace(-2, 2, 100)
y = sp.linspace(-2, 2, 100)
[x, y] = sp.meshgrid(x, y)
z = 2 * x ** 2 + 12 * x * y + 18 * y ** 2

ax.plot_wireframe(x, y, z, rstride = 5, cstride = 5)
plt.show();
```

* We can construct a matrix with a value for *a* (same matrix as above), which will clearly not be positive definite

```python
A = Matrix([[2, 6], [6, 7]])
A
```

```python
f = x_vect.transpose() * A * x_vect
f.expand()
```

```python
fig = plt.figure(figsize = (10, 8))
ax = Axes3D(fig)
x = sp.linspace(-20, 20, 100)
y = sp.linspace(-20, 20, 100)
[x, y] = sp.meshgrid(x, y)
z = 2 * x ** 2 + 12 * x * y + 7 * y ** 2

ax.plot_wireframe(x, y, z, rstride = 5, cstride = 5)
plt.show();
```

* I've saved a separate rendition of this which is rotated so that you can see, we are dipping below *z*=0

```python
Image(filename ='figure_1.png', width = 800)
```

* Clearly now, for some values of *x*i the *z* value is less than zero

* Now for an example which is clearly definite positive

```python
A = Matrix([[2, 6], [6, 26]])
A
```

```python
f = x_vect.transpose() * A * x_vect
f.expand()
```

```python
fig = plt.figure(figsize = (10, 8))
ax = Axes3D(fig)
x = sp.linspace(-2, 2, 100)
y = sp.linspace(-2, 2, 100)
[x, y] = sp.meshgrid(x, y)
z = 2 * x ** 2 + 12 * x * y + 26 * y ** 2

ax.plot_wireframe(x, y, z, rstride = 5, cstride = 5)
plt.show();
```

## Minina

* We use the following function from our symmetric matrix above
$$ {f}\left({{x}_{1},{x}_{2}}\right)=2{x}_{1}^{2}+12{{x}_{1}}{{x}_{2}}+20{x}_{2}^{2} $$

* Completing the square we have the following
$$ {f}\left({{x}_{1},{x}_{2}}\right)=2\left({x}_{1}+3{x}_{2}\right)^{2}+2{x}_{2} $$
* From this we can see that we are dealing with all positive values irrespective of the values of the variables

```python
(2 * (x1 + 3 * x2) ** 2).expand() # Just checking if we are correct
```

* Setting the equation equal to (a positive) value will *cut through* the plot and result in an ellipse
* *Cutting though* a saddle point results in a hyperbola

```python
# Rewriting the function above as a computer variable
function = 2 * (x1 + 3 * x2) ** 2 + 2 * x2 ** 2
function
```

```python
# Derivative(f, variable with whicg respect to partial derivative is taken, order)
Derivative(function, x1) # Printing the partial derivative to the screen
```

```python
Derivative(function, x1).doit() # The .doit() method execute the partial derivative
```

```python
diff(function, x2, 1) # Alternative method of doing the partial derivative
```

* Solving for the two variables in two equations using an augmented matrix

```python
M = Matrix([[4, 12, 0], [12, 40, 0]])
M.rref()
```

* Let's look at this if we cut through the *x1z*-plane (that is *xy* = 0) and the *x2z*-plane (that is *x1*=0)
$$ f\left( { x }_{ 1 },{ x }_{ 2 } \right) =2{ \left( { x }_{ 1 }+3{ x }_{ 2 } \right)  }^{ 2 }+2{ x }_{ 2 }^{ 2 }\\ f\left( { x }_{ 1 },0 \right) =2{ \left( { x }_{ 1 } \right)  }^{ 2 }\\ f\left( 0,{ x }_{ 2 } \right) =2{ \left( 3{ x }_{ 2 } \right)  }^{ 2 }+2{ x }_{ 2 }^{ 2 }=8{ x }_{ 2 }^{ 2 } $$

* Let's look at the two plots

```python
x = sp.linspace(-2, 2, 100)

plt.figure(figsize = (10, 8))
plt.plot(x, 2 * x **2)
plt.show();
```

```python
y = sp.linspace(-2, 2, 100)

plt.figure(figsize = (10, 8))
plt.plot(y, 8 * x **2)
plt.show();
```

* If you reconstruct this in your mind's eye, you can see that we are dealing with a bowl shape
* Remember from calculus that an extrema is a first derivative set to zero
* We look at the second derivative to know if we are dealing with a minimum or a maximum

```python
diff(2 * x1 ** 2, x1, 2) # Taking a derivative twice
```

```python
diff(8 * x2 ** 2, x2, 2)
```

* These second derivatives are both positive and we have a minimum

* Let's go for a setup again which will not be positive definite
$$ {f}\left({{x}_{1},{x}_{2}}\right)=2{x}_{1}^{2}+12{{x}_{1}}{{x}_{2}}+7{x}_{2}^{2} $$
* Completing the square will have the following equation
$$ {f}\left({{x}_{1},{x}_{2}}\right)=2\left({x}_{1}+3{x}_{2}\right)^{2}-11{x}_{2}^{2} $$

```python
function = 2 * (x1 + 3 * x2) ** 2 - 11 * x2 ** 2
function, function.expand() # Just checking to see if our completion of the square was correct
```

```python
Derivative(function, x1), diff(function, x1)
```

```python
Derivative(function, x2), diff(function, x2)
```

```python
M = Matrix([[4, 12, 0], [12, 14, 0]])
M.rref()
```

* Again, an extrema at (0,0)

* In your mind's eye we clearly have a saddle point at (*x*1,*x*2)=(0,0)

* Let's look at *x*1=-*x*2
$$ f\left( { x }_{ 1 },{ x }_{ 2 } \right) =2{ x }_{ 1 }^{ 2 }+12{ x }_{ 1 }{ x }_{ 2 }+7{ x }_{ 2 }^{ 2 }\\ f\left( { x }_{ 1 },{ -x }_{ 1 } \right) =2{ x }_{ 1 }^{ 2 }-12{ x }_{ 1 }{ x }_{ 1 }+7{ x }_{ 1 }^{ 2 }\\ f\left( { x }_{ 1 },{ -x }_{ 1 } \right) =2{ x }_{ 1 }^{ 2 }-12{ x }_{ 1 }^{ 2 }+7{ x }_{ 1 }^{ 2 }\\ f\left( { x }_{ 1 },{ -x }_{ 1 } \right) =-3{ x }_{ 1 }^{ 2 } $$
* We are creating a 450 plane and values for *z* will be negative here

* This is what makes the matrix *non*-positive definite
* **x**TA**x** will result in an equation which we can use to show always, not-always positive (and the marginal case described above)

* So positive definite is the matrix equivalent of the first and second derivative in calculus (which looks at the shape of the plot, i.e extrema)

* For a 2×2 we are thus looking for the following
$$ \begin{bmatrix} \frac { \partial  }{ \partial x\partial x }  & \frac { \partial  }{ \partial x\partial y }  \\ \frac { \partial  }{ \partial y\partial x }  & \frac { \partial  }{ \partial y\partial y }  \end{bmatrix} $$

## The pivots, the multiplier and completing the square

* Let's take the matrix below
* We know it's symmetric and positive definite
* We also saw that for **x**TA**x** we had to complete the square to show *z*>0
* This can easily be done by looking at the pivots and the multiplier

```python
A = Matrix([[2, 6], [6, 20]])
A
```

```python
L, U, _ = A.LUdecomposition()
L, U
```

* Note how the pivots are 2 and 2 and the multiplier was 3
* Now look at the completed square equation

```python
(x_vect.transpose() * A * x_vect).expand()
```

$$ 2{ x }_{ 1 }^{ 2 }+12{ x }_{ 1 }{ x }_{ 2 }+20{ x }_{ 2 }^{ 2 }\\ 2{ \left( { x }_{ 1 }+3{ x }_{ 2 } \right)  }^{ 2 }+2{ x }_{ 2 }^{ 2 } $$

* So, we wanted squares, but we are also interested in what goes on *outside* the squares, i.e. the pivots (2 and 2 in our example)
* Positive pivots give sum of squares, everything positive means there is a minimum (everything goes up)
* We can extend this for any *n*×*n* symmetric matrices

* Let's look again at the matrix of second derivatives we had above
$$ \begin{bmatrix} \frac { \partial  }{ \partial x\partial x }  & \frac { \partial  }{ \partial x\partial y }  \\ \frac { \partial  }{ \partial y\partial x }  & \frac { \partial  }{ \partial y\partial y }  \end{bmatrix} $$
* *f*xx and *f*yy have to be positive (for a minimum) and they have to be larger than the product of the other two, *f*xy and *f*yx

```python
function = 2 * x1 ** 2 + 12 * x1 * x2 + 20 * x2 ** 2
function
```

```python
fxx = diff(function, x1, 2)
fxy = diff(function, x1, x2)
fyx = diff(function, x2, x1)
fyy = diff(function, x2, 2)
```

```python
deriv_matr = Matrix([[fxx, fxy], [fyx, fyy]])
deriv_matr
```

```python
deriv_matr.det()
```

* Setting the first partial derivatives equal to zero finds the extrema
* The condition above sets *everything positive* (positive definite)

* Let's the look at the following

```python
function = 2 * x1 ** 2 + 12 * x1 * x2 + 7 * x2 ** 2
function
```

```python
fxx = diff(function, x1, 2)
fxy = diff(function, x1, x2)
fyx = diff(function, x2, x1)
fyy = diff(function, x2, 2)
```

```python
deriv_matr = Matrix([[fxx, fxy], [fyx, fyy]])
deriv_matr
```

```python
deriv_matr.det()
```

* Although the matrix of second derivatives are all positive entries, the determinant is negative
* Not all conditions are met for the original matrix to be positive definite

## Let's step this up to 3×3 symmetric matrices

```python
A = Matrix([[2, -1, 0], [-1, 2, -1], [0, -1, 2]])
A
```

```python
A.transpose() == A # Test to see of A is symmetric
```

* Is this symmetric matrix positive definite?

* Let's start by looking at the determinant (and *sub*-determinants)

```python
A.det() # determinant of the whole matrix
```

* All the submatrices will be the following
$$ \left[ 2 \right] ,\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix},\begin{bmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 2 \end{bmatrix} $$
* There determinant are all positive (2, 3, 4)

* Let's look at the pivots

```python
L, U, _ = A.LUdecomposition()
L, U
```

* The pivots are all positive
$$ {2}, \quad \frac{3}{2}, \quad \frac{4}{3} $$

* Notice how the first determinant was 2 (also the first *sub*-determinant)
* The product of the first two pivots must equal the 2nd determinant (which is 3) and must therefor be 3/2
* The product of the first three (all) pivots must equal the 4th determinant (which is 4) and must therefor be 4/3

* Let's look at the eigenvalues

```python
A.eigenvals()
```

* Again, all positive
* So far, so good
* Just as a reminder, remember that the sum of the eigenvalues must equal the trace (sum of the entries on the main diagonal) and multiplying them must equal the determinant

* Let's look at **x**TA**x**

```python
x_vect = Matrix([x1, x2, x3])
x_vect
```

```python
(x_vect.transpose() * A * x_vect).expand()
```

```python
function = 2 * x1 ** 2 - 2 * x1 * x2 + 2 * x2 ** 2 - 2 * x2 * x3 + 2 * x3 ** 2
function # A cubic equation
```

* We can construct this as follows
* The main diagonal entries are the constant of the squared variables (2, 2, and 2)
* There is a -1 and a -1 in the *12* and *21* *row-column* positions, whose sum is -2 and which then belongs to the *x*1*x*2 (or *x*2*x*1)
* The *13* and *31* entries are both zero, so there will be no *x*1*x*3 coefficient
* The *23* and *32* entries are both -1, so again, a coefficient of -2 for *x*2*x*3

* This matrix represents a plot in 4D space, so we can't draw it
* We can construct it as the sum of three squares though
* The three squares will be made up of the three pivots (for their coefficients)
* They (and obviously the squared values) are all positive and therefor we will only have *f*(*x*,*y*,*z*) values which are positive

* Cutting through this 4D space (which is difficult to visualize) as say *f*(*x*,*y*,*z*) will give an ellipsoid (lopsided football)
* A sphere would have three equal eigenvalues
* A *football*-shape would have two identical eigenvalues and the third different
* The lopsided ellipsoid would all three eigenvalues different as in this case
* The half-lengths the axes of these shapes are 1 over the eigenvalues
* Diagonalization will give the principle axis theorem
$$ {Q}{\Lambda}{Q}^{T} $$

```python
fxx = diff(function, x1, x1)
fxy = diff(function, x1, x2)
fxz = diff(function, x1, x3)
fyx = diff(function, x2, x1)
fyy = diff(function, x2, x2)
fyz = diff(function, x2, x3)
fzx = diff(function, x3, x1)
fzy = diff(function, x3, x2)
fzz = diff(function, x3, x3)
```

```python
deriv_matr = Matrix([[fxx, fxy, fxz], [fyx, fyy, fyz], [fzx, fzy, fzz]])
deriv_matr
```

```python
deriv_matr.det()
```

* The determinant (and all *sub*-determinants) are positive

## Example problems

### Example problem 1

* For which values of *c* will the following matrix be positive definite and *semi*-positive definite
$$ \begin{bmatrix} 2 & -1 & -1 \\ -1 & 2 & -1 \\ -1 & -1 & 2+c \end{bmatrix} $$

#### Solution

* Let's try the determinant test first

```python
A = Matrix([[2, -1, -1], [-1, 2, -1], [-1, -1, 2 + c]])
A
```

```python
A.det()
```

* All the *sub*-determinants are positive, being 2, 3 and then 3*c* for *c*>0

* Let's look at the pivot test

```python
L, U, _ = A.LUdecomposition()
L, U
```

* Again, all the pivots (in U) are positive for *c* > 0
* So for positive definite we have *c*>0 and for semi-definite we have *c*=0

```python
A.eigenvals()
```

* The energy or completing the square test

```python
(x_vect.transpose() * A * x_vect).expand()
```

* For *x*3 we have (*c*+2)(*x*32)
* Remember, though, that for the squares of the **x**-components we must have the entries along the main diagonal of A as their coefficients; this *c* + 2 = 2 and hence, again, *c*=0

* For interest's sake, we will have the following completed square equation
$$ 2\left({x}-\frac{1}{2}{y}-\frac{1}{2}{z}\right)^{2}+\frac{3}{2}\left({y}-{z}\right)^{2}+{c}{z}^{2} $$

* Now the coefficients come from the values along the diagonal of U
* The -½ values come from the multipliers as seen in column 1 of L
* The +1 and -1 for (*y*-*z*)2 come from column 2 of L
* The +1 in front of *z*2 come from column 3 of L (actually every set of ()'s contains and *x*, *y* and *z*, some coefficients (from L) are just zero)
* Be that as it may, the squared equation as it stands will only equal zero if *x*=*y*=*z*=0
* For a value of more than zero, *c* must be positive
* Please note that by *x*,*y* and *z* I am referring to *x*1, *x*2 and *x*3

* If *x*=0 we have the following matrix

```python
A = Matrix([[2, -1, -1], [-1, 2, -1], [-1, -1, 2]])
A
```

```python
A.det()
```

* A singular matrix, again only possible if all variables equal zero

```python

```


# Modeling and Simulation in Python

Chapter 7: Thermal systems

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

We'll start with the same code we saw last time: the magic command that tells Jupyter where to put the figures, and the import statement that gets the function defined in the `modsim` module.

```python
# If you want the figures to appear in the notebook,
# and you want to interact with them, use
# %matplotlib notebook

# If you want the figures to appear in the notebook,
# and you don't want to interact with them, use
# %matplotlib inline

# If you want the figures to appear in separate windows, use
# %matplotlib qt|

# tempo switch from one to another, you have to select Kernel->Restart

%matplotlib inline

from modsim import *
```

### The coffee cooling problem.

I'll use a `State` object to store the initial temperature.

```python
T_init = UNITS.Quantity(90, UNITS.degC)
init = State(temp=T_init)
init
```

      value

      temp
      90 degC

And a `System` object to contain the system parameters.

```python
T_env = UNITS.Quantity(22, UNITS.degC)
r = 0.01 / UNITS.min
t0 = 0 * UNITS.min
t_end = 30 * UNITS.min
dt = 1 * UNITS.min

coffee = System(init=init, T_env=T_env, r=r, t0=t0, t_end=t_end, dt=dt)
coffee
```

      value

      init
      temp    90 degC
dtype: object

      T_env
      22 degC

      r
      0.01 1 / minute

      t0
      0 minute

      t_end
      30 minute

      dt
      1 minute

The `update` function implements Newton's law of cooling.

```python
def update(system, state):
    """Update the thermal transfer model.

    system: System object
    state: State (temp)

    returns: State (temp)
    """
    T = state.temp
    dT = - system.r * (T - system.T_env) * system.dt
    return State(temp=T+dT)
```

Here's how it works.

```python
update(coffee, init)
```

      value

      temp
      89.32 degC

```python
system = coffee
df = TimeFrame(columns=system.init.index)
df.loc[float(system.t0.magnitude)] = system.init
```

```python
array(1)
```

    array(1)

```python
def run_simulation(system, update_func):
    """Runs a simulation of the system.

    Add a DataFrame to the System: results

    system: System object
    update_func: function that updates state
    """
    df = TimeFrame(columns=system.init.index)
    t0 = float(system.t0.magnitude)
    t_end = float(system.t_end.magnitude)

    df.loc[t0] = system.init

    for i in arange(t0, t_end):
        df.loc[i+1] = update_func(system, df.loc[i])

    system.results = df
```

```python
run_simulation(coffee, update)
coffee.results
```

      temp

      0.0
      90 degC

      1.0
      89.32 degC

      2.0
      88.6468 degC

      3.0
      87.980332 degC

      4.0
      87.32052868000001 degC

      5.0
      86.66732339320001 degC

      6.0
      86.020650159268 degC

      7.0
      85.38044365767531 degC

      8.0
      84.74663922109856 degC

      9.0
      84.11917282888757 degC

      10.0
      83.4979811005987 degC

      11.0
      82.88300128959271 degC

      12.0
      82.27417127669679 degC

      13.0
      81.67142956392982 degC

      14.0
      81.07471526829052 degC

      15.0
      80.48396811560761 degC

      16.0
      79.89912843445154 degC

      17.0
      79.32013715010703 degC

      18.0
      78.74693577860596 degC

      19.0
      78.17946642081989 degC

      20.0
      77.6176717566117 degC

      21.0
      77.06149503904558 degC

      22.0
      76.51088008865513 degC

      23.0
      75.96577128776858 degC

      24.0
      75.42611357489089 degC

      25.0
      74.89185243914199 degC

      26.0
      74.36293391475057 degC

      27.0
      73.83930457560307 degC

      28.0
      73.32091152984704 degC

      29.0
      72.80770241454857 degC

      30.0
      72.2996253904031 degC

```python
plot(coffee.results.temp, label='coffee')
decorate(xlabel='Time (minutes)',
         ylabel='Temperature (C)')
```

```python
def final_temp(system):
    """Final temperature.

    system: System object with results.

    returns: temperature (C)
    """
    # TODO: if there are no results, return init.temp?
    df = system.results
    return df.temp[system.t_end]
```

```python
def run(volume, T_init, r, t_end):
    init = State(temp=T_init)
    system = System(volume=volume, init=init, T_env=22,
                    r=r, t0=0, t_end=t_end)
    run_simulation(system, update)
    return system
```

```python
coffee = run(volume=300, T_init=90, r=0.01, t_end=30)
final_temp(coffee)
```

    72.2996253904031

```python
def func(x):
    return (x-1) * (x-2) * (x-3)
```

```python
fsolve(func, x0=0)
```

    array([ 1.])

```python
fsolve(func, 1.9)
```

    array([ 2.])

```python
fsolve(func, 2.9)
```

    array([ 3.])

```python
fsolve(func, 1.5)
```

    array([ 3.])

```python
def error_func1(r):
    """Runs a simulation and returns the `error`.

    r: thermal insulation factor

    returns: difference between final temp and 70C
    """
    system = run(volume=300, T_init=90, r=r, t_end=30)
    return final_temp(system) - 70
```

```python
error_func1(r=0.01)
```

    2.2996253904030937

```python
solution = fsolve(error_func1, 0.01, xtol=1e-8)
r_coffee = solution[0]
r_coffee
```

```python
coffee = run(volume=300, T_init=90, r=r_coffee, t_end=30)
final_temp(coffee)
```

    70.0000000000064

**Exercise:**

```python
milk = run(volume=50, T_init=5, r=0.1, t_end=15)
final_temp(milk)
```

    18.499850754390966

```python
def error_func2(r):
    """Runs a simulation and returns the `error`.

    r: thermal insulation factor

    returns: difference between final temp and 20C
    """
    system = run(volume=50, T_init=5, r=r, t_end=15)
    return final_temp(system) - 20
```

```python
error_func2(r=0.1)
```

    -1.500149245609034

```python
solution = fsolve(error_func2, 0.1)
r_milk = solution[0]
r_milk
```

    0.13296078935465339

```python
milk = run(volume=50, T_init=5, r=r_milk, t_end=15)
final_temp(milk)
```

    19.999999999999613

```python
milk = run(volume=50, T_init=5, r=r_milk, t_end=30)
final_temp(milk)
```

    21.76470588235285

```python
plot(coffee.results.temp, label='coffee')
plot(milk.results.temp, '--', label='milk')
decorate(xlabel='Time (minutes)',
         ylabel='Temperature (C)',
         loc='center left')

savefig('chap07-fig01.pdf')
```

```python
sympy.init_printing()

from sympy import symbols, Eq, solve

C1, C2, T1, T2, T = symbols('C1 C2 T1 T2 T')

eq = Eq(C1 * (T - T1) + C2 * (T - T2), 0)

solve(eq, T)
```

```python
def mix(s1, s2):
    assert s1.t_end == s2.t_end

    volume = s1.volume + s2.volume

    temp = (s1.volume * final_temp(s1) +
            s2.volume * final_temp(s2)) / volume

    init = State(temp=temp)
    mixture = System(volume=volume, init=init, T_env=22,
                    r=s1.r, t0=s1.t_end, t_end=30)

    return mixture
```

```python
coffee = run(volume=300, T_init=90, r=r_coffee, t_end=30)
milk = run(volume=50, T_init=5, r=r_milk, t_end=30)
final_temp(coffee), final_temp(milk)
```

```python
mix_last = mix(coffee, milk)
mix_last.init.temp
```

```python
coffee = run(volume=300, T_init=90, r=r_coffee, t_end=1)
milk = run(volume=50, T_init=5, r=r_milk, t_end=1)
mix_first = mix(coffee, milk)
mix_first.init.temp
```

```python
run_simulation(mix_first, update)
final_temp(mix_first)
```

```python
def run_and_mix(t_add):
    coffee = run(volume=300, T_init=90, r=r_coffee, t_end=t_add)
    milk = run(volume=50, T_init=5, r=r_milk, t_end=t_add)
    mixture = mix(coffee, milk)
    run_simulation(mixture, update)
    return final_temp(mixture)
```

```python
run_and_mix(1)
```

```python
run_and_mix(15)
```

```python
run_and_mix(30)
```

```python
sweep = Sweep()
for t_add in range(1, 30):
    temp = run_and_mix(t_add)
    sweep[t_add] = temp
```

```python
plot(sweep, color='purple')
decorate(xlabel='Time added (min)',
         ylabel='Final temperature (C)',
         legend=False)

savefig('chap07-fig02.pdf')
```

```python
from sympy.printing import latex

def show(expr, show_latex=True):
    if show_latex:
        print(latex(expr))
    return expr
```

```python
from sympy import symbols, Function, diff, Eq, dsolve

T_init, T_env, r, t = symbols('T_init T_env r t')
T = symbols('T', cls=Function)

eqn = Eq(diff(T(t), t), -r * (T(t) - T_env))
eqn
```

```python
solution_eq = dsolve(eqn)
show(solution_eq)
```

```python
general = solution_eq.rhs
general
```

```python
at0 = general.subs(t, 0)
at0
```

```python
solutions = solve(Eq(at0, T_init), C1)
value_of_C1 = solutions[0]
value_of_C1
```

```python
particular = general.subs(C1, value_of_C1)
show(particular)
```

```python
t_end, T_end = symbols('t_end T_end')
```

```python
at_end = particular.subs(t, t_end)
at_end
```

```python
solutions = solve(Eq(at_end, T_end), r)
value_of_r = solutions[0]
show(value_of_r)
```

```python
subs = dict(t_end=30, T_end=70, T_init=90, T_env=22)
r_coffee2 = value_of_r.evalf(subs=subs)
type(r_coffee2)
```

    sympy.core.numbers.Float

```python
r_coffee2 = float(r_coffee2)
r_coffee2, type(r_coffee2)
```

    (0.011610223142273859, float)

```python
def run_analysis(system):
    T_init = system.init.temp
    T_env = system.T_env

    ts = arange(system.t0, system.t_end+1)
    Ts = T_env + (T_init - T_env) * np.exp(-system.r * ts)
    series = TimeSeries(Ts, index=ts)

    system.results = TimeFrame(series, columns=['temp'])
```

```python
init = State(temp=90)
coffee2 = System(volume=300, init=init, T_env=22,
                 r=r_coffee2, t0=0, t_end=30)
run_analysis(coffee2)
final_temp(coffee2)
```

```python
coffee = run(volume=300, T_init=90, r=r_coffee, t_end=30)
final_temp(coffee)
```

```python
coffee.results - coffee2.results
```

      temp

      0
      0

      1
      0

      2
      0

      3
      0

      4
      0

      5
      0

      6
      0

      7
      0

      8
      0

      9
      0

      10
      0

      11
      0

      12
      0

      13
      -1.42109e-14

      14
      -1.42109e-14

      15
      0

      16
      -1.42109e-14

      17
      -1.42109e-14

      18
      -1.42109e-14

      19
      0

      20
      -1.42109e-14

      21
      0

      22
      0

      23
      0

      24
      0

      25
      0

      26
      -1.42109e-14

      27
      0

      28
      -1.42109e-14

      29
      0

      30
      0

```python

```


```python
%pylab inline
import sympy
sympy.init_printing()
```

    Populating the interactive namespace from numpy and matplotlib

```python
def plot_complex_function(Y, sigma_min, sigma_max, omega_max, log_omega, plot_conj):
    sigma_eval=np.linspace(sigma_min, sigma_max, 200)

    # constants
    omega_min = 1e-3
    nlevels = 20  # levels of contour
    nlevels_outline = 10  # levels of contour

    # validate input
    assert np.all(omega_max > 0)

    # symbolic function
    sigma, omega = sympy.symbols('sigma, omega', real=True, positive=True)
    s_binomial = sigma + sympy.I*omega
    Y_arg = sympy.arg(Y.subs(s, s_binomial))
    Y_mag = sympy.Abs(Y.subs(s, s_binomial))
    f_Y_euler = sympy.lambdify((sigma, omega), (Y_arg, Y_mag))

    # compute poles and zeros
    poles = sympy.solve(sympy.denom(Y))
    zeros = sympy.solve(sympy.numer(Y))
    re_poles = np.array(np.real(poles), dtype=float)
    im_poles = np.array(np.imag(poles), dtype=float)
    re_zeros = np.array(np.real(zeros), dtype=float)
    im_zeros = np.array(np.imag(zeros), dtype=float)

    # evaluate
    if log_omega:
        omega_eval=np.logspace(np.log10(omega_min), np.log10(omega_max), 200)
        yplot = plt.semilogy
        xplot = plt.semilogx
        im_zeros = np.where(im_zeros < omega_min, omega_min, im_zeros)
        im_poles = np.where(im_poles < omega_min, omega_min, im_poles)
    else:
        omega_eval=np.linspace(omega_min, omega_max, 100)
        omega_both = np.concatenate([-np.flip(omega_eval), omega_eval])
        yplot = plt.plot
        xplot = plt.plot

    if plot_conj and not log_omega:
        Sigma, Omega = np.meshgrid(sigma_eval, omega_both)
    else:
        Sigma, Omega = np.meshgrid(sigma_eval, omega_eval)

    Z = f_Y_euler(Sigma, Omega)
    angle = np.rad2deg(Z[0])
    mag = 20*np.log10(np.abs(Z[1]))

    # angle G(jw)
    Z_jw = f_Y_euler(0, omega_eval)
    angle_jw = np.rad2deg(Z_jw[0])
    mag_jw = 20*np.log10(np.abs(Z_jw[1]))

    # figures
    fig = plt.figure(figsize=(20, 10))

    # angle of G(S)
    ax = plt.subplot(221)
    CS = ax.contourf(Sigma, Omega, angle, nlevels,cmap=plt.cm.hsv)
    cbar = plt.colorbar(CS)
    CS = ax.contour(Sigma, Omega, angle, nlevels_outline, colors='k', linestyles='--')
    cbar.add_lines(CS)
    ax.set_title(r'$\angle G(s),\ deg$');
    ax.set_xlabel(r'$\sigma$')
    ax.set_ylabel(r'$\omega,\ rad/s$')
    pole_props = {'marker':'x', 'color':'white', 'markeredgewidth':3, 'markersize':20, 'markerfacecolor':'none'}
    zero_props = {'marker':'o', 'color':'white', 'markeredgewidth':3, 'markersize':20, 'markerfacecolor':'none'}
    h_pole = yplot(re_poles, im_poles, **pole_props)
    h_zero = yplot(re_zeros, im_zeros, **zero_props)
    h_jw = yplot(0*omega_eval, omega_eval, 'g-', linewidth=5, alpha=0.5)
    plt.legend([h_zero[0], h_pole[0], h_jw[0]], ['zero', 'pole', 'jw'], loc='best', ncol=3)
    plt.grid()

    # mag of G(S)
    ax = plt.subplot(222)
    CS2 = ax.contourf(Sigma, Omega, mag, nlevels, cmap=plt.cm.coolwarm)
    cbar = plt.colorbar(CS2)
    CS2 = ax.contour(Sigma, Omega, mag, nlevels_outline, colors='k', linestyles='--')
    cbar.add_lines(CS2)
    ax.set_title(r'$|G(s)|,\ dB$');
    ax.set_xlabel(r'$\sigma$')
    ax.set_ylabel(r'$\omega,\ rad/s$')
    h_pole = yplot(re_poles, im_poles, **pole_props)
    h_zero = yplot(re_zeros, im_zeros, **zero_props)
    h_jw = yplot(0*omega_eval, omega_eval, 'g-', linewidth=5, alpha=0.5)
    plt.legend([h_zero[0], h_pole[0], h_jw[0]], ['zero', 'pole', 'jw'], loc='best', ncol=3)
    plt.grid()

    # magnitude of G(jw)
    ax = plt.subplot(223)
    ax.set_title(r'$\angle G(j\omega),\ deg$');
    ax.set_xlabel(r'$\omega,\ rad/s$')
    xplot(omega_eval, angle_jw)
    xplot(omega_eval, angle_jw)
    ax.set_ylabel(r'$\angle G(j\omega),\ deg$');
    plt.grid()

    ax = plt.subplot(224)
    ax.set_title(r'$|G(j\omega)|,\ dB$');
    ax.set_xlabel(r'$\omega,\ rad/s$')
    xplot(omega_eval, mag_jw)
    xplot(omega_eval, mag_jw)
    ax.set_ylabel(r'$|G(j\omega)|,\ dB$');
    plt.grid()
```

```python
t, sigma, omega = sympy.symbols('t, sigma, omega')
s = sympy.symbols('s', complex=True)

Y = 1/(s+1)
Y
```

```python
plot_complex_function(Y, sigma_min=-10, sigma_max=1, omega_max=10, log_omega=False, plot_conj=True)
```

```python

```

```python

```


```python
%matplotlib inline
import numpy as np
import pylab as pl
import sympy as sym
from sympy.functions import Abs
#from sympy import Abs, Symbol, S
```

## Goals of today:

- Check how good or bad are the estimates given in the theoretical lecture
- Compare Equispaced with Chebyshev
- Compute errors, plot error tables
- Implement Bernstein interpolation
- Compare Bernstein interpolation with Lagrangian interpolation

## Lagrangian part

The estimate we want to check:

$$
|| f -p || \leq ||f^{n+1}||_\infty \frac{||w(x)||_\infty}{(n+1)!}
$$

in order to do so we need to define, simbolic and numerical functions. [`Sympy`]() is a very useful package to handle symbolic expressions and to export them to numerical functions. A the beginnnig of this notebook it is imported with the command: `import sympy as sym`. Let's define a simple function:

**Hint:** Try to play around, derivatives integrals...

```python
x = np.linspace(-10, 10, 2**10 + 1)
t = sym.symbols('t')

#We can define a function in these two equivalent ways

def runge(x):
    return 1./(1. + x**2)

runge = lambda x:1. / (1 + x**2)

#We can now plot it

y = runge(x)
_ = pl.plot(x,y)

def l_infty(y):
    return abs(y).max()  # takes the abs of y values and then gets the max

l = np.linalg.norm(y, ord=np.inf)

print(l)

yy = np.random.rand(1000000)

print(len(yy))

l_infty = lambda y: np.linalg.norm(y, ord = np.inf)

l_infty(yy)

l_infty_0 = lambda y: abs(y).max()

def l_infty_1(y):
    m = -1.0
    for i in range(len(y)):
        m = max(m,abs(y[i]))

def l_infty_2(y):
    m = -1.0
    for i in y:
        m = max(m,abs(i)

%timeit l_infty(yy)
%timeit l_infty_0(yy) # the numpy solution is the best: it using vectorized operations
%timeit l_infty_1(yy)
%timeit l_infty_2(yy)

```

```python
# Contruct a symbolic function...
t = sym.var('t')
fs = 1.0/(1.0+t**2)

fs.diff(t,1)

# This will no work...
fs(x)
```

To make this function *digestible* by numpy we use the simple command `nf = sym.lambdify(t,f, 'numpy')`. This allows the function nf to be called with numpy arguments.

```python
nf = sym.lambdify(t, fs, "numpy")
nfprime = sym.lambdify(t, fs.diff(t,1), 'numpy')

_=pl.plot(x,nf(x))
_=pl.plot(x,nfprime(x))
```

```python
#def my_derivatives(f, n, t=t):
 #   return sym.lambdify(t,f.diff(t,n),'numpy')

my_derivatives = lambda f,n: sym.lambdify(t,f.diff(t,n), 'numpy')

fpp = my_derivatives(fs,2)

_=pl.plot(x,nf(x))
_=pl.plot(x,fpp(x))
```

```python
# Check for two functions..

function_set = [fs, sym.sin(2*sym.pi*t)]

for my_f in function_set:
    print('**************')
    print my_f
    for i in range(5):
        print l_infty(my_derivatives(my_f,i)(x))

```

    **************
    1.0/(t**2 + 1.0)
    1.0
    0.649412355588
    2.0
    4.66644172318
    24.0
    **************
    sin(2*pi*t)
    1.0
    6.28318530718
    39.4784176044
    248.050213442
    1558.54545654

We aim at controlling all of the pieces of the inequality above, plot how terms behave with the degree, and see what happens `:)`

Good thing is to start from the beginning and control the term $||f-p||_\infty$. We recall that:
$$
p = \mathcal{L}^n f := \sum_{i=0}^n f(x_i) l^n_i(x),
$$

with

$$
l^n_i(x) := \prod_{j=0, j\neq i}^n \frac{(x-x_j)}{(x_i-x_j)}  \qquad
i = 0, \dots, n.
$$

Let's implment this guy. We want to fill the matrix `Ln` with $n+1$ rows and as many colums as the number of points where we evaluate the funtion.

$$
\text{Ln}_{ij}:= l_i(x_j)
$$

so that

$$
\text{Ln}_{ij} f(q_i) = \sum_{i=0}^n l_i(x_j) f(q_i) 0 (\mathcal{L}^nf)(x_j)
$$

A good idea would be to collect the few operations in a function, like this one:
```python
def lagrangian_interpolation_matrix(x,q):
    ...
    return Ln
```
so that we can recall it whenever we need it.

**Hint:** I wouldn't call myself a good programmer, but I do my best to be like that. First construct the code in the main section of your program, run it, check that it works, then collect the precious commmands you wrote in an function.

### Step 0

```python
n = 3
q = np.linspace(-5,5,n+1)

Ln = np.zeros((n+1, len(x)))

for i in range(n+1):
    Ln[i] = np.ones_like(x)
    for j in range(n+1):
        if j!=i:
            Ln[i] *= (x - q[j])/(q[i]-q[j])

#Equivalently

#Ln[i] = np.product([(x - q[j])/(q[i]-q[j]) for j in range(n+1) if j is not i], axis = 0)

_ = pl.plot(x, Ln.T)
```

## Step 1

Now we transform this into a function that takes the points where we want to compute the
matrix, and the interpolation points we use to define the basis.

```python
def lagrangian_interpolation_matrix(x,q):
    Ln = np.zeros((len(q), len(x)))

    for i in range(len(q)):
        Ln[i] = np.ones_like(x)
        for j in range(len(q)):
            if j!=i:
                Ln[i] *= (x - q[j])/(q[i]-q[j])
    return Ln

def lagrangian_interpolation_matrix_one_liner(x,q):
    Ln = np.zeros((len(q), len(x)))
    for i in range(len(q)):
        Ln[i] = np.product([(x - q[j])/(q[i]-q[j]) for j in range(len(q)) if j is not i], axis = 0)
    return Ln
```

```python
Error = lagrangian_interpolation_matrix(x,q) - lagrangian_interpolation_matrix_one_liner(x,q)

print 'Error:', np.linalg.norm(Error)
```

    Error: 0.0

From the previous lecture we know that the mathemathical expression:
$$
(\mathcal{L}^nf)(x_i) := \sum_{j=0}^n f(q_j) l^n_j(x_i) = (\text{Ln}^T f)_i
$$

Can be easyly translated into the `numpy` line:
```python
Ln.T.dot(f(x))
```
Let's give it a try:

```python
#f = lambda x: np.sin(2*np.pi*x)
f = lambda x: np.sin(x)

n = 3
q = np.linspace(-5,5,n+1)
Ln = lagrangian_interpolation_matrix(x,q)

_ = pl.plot(x, f(x))
_ = pl.plot(x, Ln.T.dot(f(q)))
_ = pl.plot(q, f(q),'ro')

err = f(x) - Ln.T.dot(f(q))
_ = pl.plot(x, err)

_ = pl.plot(q, f(q),'ro')

Error = l_infty(err)
print 'Error:', Error
```

We need a very simple function to evaluate the norm infinity of a function $||\cdot||_\infty$:

```python
def linfty_norm(fun):
    # Your function here
    return
```

Now compute the following

$||f-p||_\infty = $ `error`

$||f^{n+1}||_\infty = $ `nth_der`

$w(x) =\prod_{i=0}^n (x-q_i), \quad ||w(x)||_\infty = $ `w`

```python
w = lambda x, q: np.product([ x - q[i] for i in range(len(q))], axis = 0)

q = np.linspace(-5,5,10)
_ = pl.plot(x, w(x,q))

Ln = lagrangian_interpolation_matrix(x,q)
Error = l_infty(f(x) - Ln.T.dot(f(q)))

fs = sym.sin(2*sym.pi*t)
fp = my_derivatives(fs, len(q))

nth_der = l_infty(fp(q))
w_infty = l_infty(w(x,q))

UpperEstimate = nth_der * w_infty/np.math.factorial(len(q))
print UpperEstimate
```

```python
fs = sym.sin(2*sym.pi*t)

points = range(2,15)
UpperEstimate = []

for n in points:
    q = np.linspace(-5,5,n)

    Ln = lagrangian_interpolation_matrix(x,q)
    Error = l_infty(f(x) - Ln.T.dot(f(q)))

    fp = my_derivatives(fs, len(q))
    nth_der = l_infty(fp(q))

    w_infty = l_infty(w(x,q))

    UpperEstimate = nth_der * w_infty/np.math.factorial(len(q))
print UpperEstimate

_= pl.semilogy(points, UpperEstimate)
```

If I didn't mess the code this a good spot to play aroud with the function to be checked:

```python
# Test on one f
```

```python
# Make it a python function call
```

```python

```


# Naive-Bayes Classification
## Definition
> Prerequisites to increase your understanding
> - Programming knowledge
> - Statistical knowledge
> - Jupyter notebook knowledge

Naive Bayes is a relatively simple classification algorithm based on probability and uses Bayes Theorom with an independence assumption among the features in the data. The fundamental idea of Naive Bayes is that it computes the probability of every class, which we want to reveal, based on the probability of every feature in the data.

According to Naive Bayes algorithm, we are going to assume that every feature in the data is in an independent condition on the outcome probability of each separate class. Let's assume that we are doing a car classification and we have a data such as;

| buying   | maint    | doors    | persons  | lug-boot | safety   | class    |
| :------- | :------- | :------- | :------- | :------- | :------- | :------- |
| vvhigh   | vhigh    | 2        | 2        | small    | low      | unacc    |

**Description of dataset:**
* CAR                      car acceptability
    * PRICE                  overall price
        * _buying_               buying price
        * _maint_                price of the maintenance
* TECH                   technical characteristics
    * COMFORT              comfort
        * _doors_              number of doors
        * _persons_            capacity in terms of persons to carry
        * _lug-boot_           the size of luggage boot
    * _safety_               estimated safety of the car

Naive Bayes assumes that features above occur independently of each other.

In machine learning, Naive Bayes is advantageous against other commonly used classification algorithms because of its simplicity, speed and accuracy on small datasets and it also enables us to make classification despite missing information. Naive Bayes is a supervised learning algorithm because it needs to be trained with a labeled dataset.

## Bayes Theorem
Consider two events, $A$ and $B$. For example, $A$ is a set of car features, which are $A \in \{ vvhigh, vhigh, 2, 2, small, low \}$,and $B$ is a set of car classes that are $B \in \{ unacc, acc, good, vgood \}$

* $A \cap B$ means the intersection of $A$ and $B$.
* $P(A \mid B)$ is read as probability of A given B.

When we know that $B$ is given (Event $B$ has occurred), it means our sample space is $B$ that is the right figure. Now we are trying to compute the probability of also occuring $A$ at the same time (the conditional probability of $A$). It is obvious that we are trying to find the probability of $A \cap B$ given that we are in the space of $B$.

\begin{equation}
P(A \mid B) = \frac{P(A \cap B)}{P(B)}
\end{equation}

We can rewrite $P(A \cap B)$ as $P(A, B)$. Two of these mean the probability of $A$ and $B$ at the same time. So the new form of the equation is :

\begin{equation}
P(A \mid B) = \frac{P(A, B)}{P(B)}
\end{equation}

For the probability of $A$ and $B$, we can deduce equations below from the figure above.

\begin{align}
& P(A, B) = P(B, A) = P(A \mid B)P(B) \\
& P(A, B) = P(B, A) = P(B \mid A)P(A)
\end{align}

Let's look at the new form of the equation putting the second form of $P(A, B)$:

\begin{equation}
P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}
\end{equation}

This equation is known as **Bayes Theorem**.
* $P(A \mid B)$ : posterior that is the probability of $A$ when it is known that $B$ is given
* $P(B)$ : evidence that is the marginal probability of $B$
* $P(B \mid A)$ : likelihood
* $P(A)$ : prior probability that is marginal probabiliy of $A$

## Naive-Bayes Formulation
Suppose we have a dataset which each observation belongs to a class from the finite set $C = \{ c_1, c_2, ..., c_n \}$ and each observation constitutes from a few features $F = \{ f_1, f_2, ..., f_b \}$. If we could compute the probabilities of $P(c_1 | F), P(c_2 | F), ..., P(c_n | F)$ then we could predict the class for a new observation $i$ to be one of those which have the highest probability.

To compute the conditional probabilities, we can use Bayes Theorem;

\begin{equation}
P(c_i \mid f_1, f_2, \dots ,f_b) = \frac{P(f_1, f_2, \dots ,f_b \mid c_i)P(c_i)}{P(f_1, f_2, \dots ,f_b)}
\end{equation}

As you know, Naive-Bayes supposes that all features are in independent conditions, therefore we can rewrite this equation like;

\begin{equation}
P(c_i \mid f_1, f_2, \dots ,f_b) = \frac{P(f_1 \mid c_i)P(f_2 \mid c_i) \dots P(f_b \mid c_i)P(c_i)}{P(f_1, f_2, \dots ,f_b)}
\end{equation}

The final form of equation is

\begin{align}
& \text{for} \; i = 1, 2, \dots , n \\
& P(c_i \mid f_1, f_2, \dots ,f_b) = P(c_i) \frac{\Pi_{j=1}^b P(f_j \mid c_i)}{P(f_1, f_2, \dots ,f_b)}
\end{align}

Since $P(f_1, f_2, \dots ,f_b)$ is a constant, we can use the classification rule below.

\begin{align}
& P(c_i \mid f_1, f_2, \dots ,f_b) \propto P(c_i) \Pi_{j=1}^b P(f_j \mid c_i)
\end{align}

When a set of features came, we are going to compute the possibilities of each class for the feature set by using the equation above.

Let's look at how we can code it.

```python
# for data manipulation
import pandas as pd
# for array operations
import numpy as np

# read dataset from csv file
dataset = pd.read_csv("car-eval.csv")
#print(dataset)
dataset.describe()
```

      buying
      maint
      doors
      persons
      lug_boot
      safety
      clazz

      count
      1728
      1728
      1728
      1728
      1728
      1728
      1728

      unique
      4
      4
      4
      3
      3
      3
      4

      top
      high
      high
      4
      more
      med
      high
      unacc

      freq
      432
      432
      432
      576
      576
      576
      1210

```python
# if dataset.index % test_indis == 0
# then it is going to be used as test dataset
# they will not be attended into the train dataset
test_indis = 79

test_dataset = dataset[dataset.index % test_indis == 0]
train_dataset = dataset[dataset.index % test_indis != 0]

# total count of sample space
total = len(train_dataset)
```

Let's look at the probabilities of each item in the data one by one.

```python
# unique values of clazz column
keys_clazz = np.array(train_dataset["clazz"].unique())

# let's initiate a matrix to hold counts (buying X clazz)
p_clazz = np.zeros((len(keys_clazz)))

# zip only buying and clazz values
for u in train_dataset["clazz"]:
    # get index of current clazz value
    index_clazz = np.where(keys_clazz == u)

    # increment 1 the count of the intersection of buying x clazz pair.
    p_clazz[index_clazz] += 1

# let's normalize the possibilities
p_clazz = p_clazz / np.sum(p_clazz, axis=0, keepdims=True)

# we create a pandas dataframe to visualize the table more familiar
df_p_clazz = pd.DataFrame(p_clazz, keys_clazz)

# possibilities of class values
df_p_clazz
```

      0

      unacc
      0.700469

      acc
      0.222743

      vgood
      0.036928

      good
      0.039859

```python
# unique values of buying column
keys_buying = np.array(train_dataset["buying"].unique())

# unique values of clazz column
keys_clazz = np.array(train_dataset["clazz"].unique())

# let's initiate a matrix to hold counts (buying X clazz)
p_buying_given_clazz = np.zeros((len(keys_buying), len(keys_clazz)))

# zip only buying and clazz values
for u in zip(train_dataset["buying"], train_dataset["clazz"]):
    # get index of current buying value
    index_buying = np.where(keys_buying == u[0])

    # get index of current clazz value
    index_clazz = np.where(keys_clazz == u[1])

    # increment 1 the count of the intersection of buying x clazz pair.
    p_buying_given_clazz[[index_buying], [index_clazz]] += 1

# we create a pandas dataframe to visualize the table more familiar
df_p_buying_given_clazz = pd.DataFrame(p_buying_given_clazz, keys_buying, keys_clazz)

# counts of buying given class
df_p_buying_given_clazz
```

      unacc
      acc
      vgood
      good

      vhigh
      355.0
      71.0
      0.0
      0.0

      high
      320.0
      107.0
      0.0
      0.0

      med
      264.0
      115.0
      25.0
      22.0

      low
      256.0
      87.0
      38.0
      46.0

```python
# let's normalize the possibilities
p_buying_given_clazz = p_buying_given_clazz / np.sum(p_buying_given_clazz, axis=0, keepdims=True)

# we create a pandas dataframe to visualize the table more familiar
df_p_buying_given_clazz = pd.DataFrame(p_buying_given_clazz, keys_buying, keys_clazz)

# possibilities of buying values given class
df_p_buying_given_clazz
```

      unacc
      acc
      vgood
      good

      vhigh
      0.297071
      0.186842
      0.000000
      0.000000

      high
      0.267782
      0.281579
      0.000000
      0.000000

      med
      0.220921
      0.302632
      0.396825
      0.323529

      low
      0.214226
      0.228947
      0.603175
      0.676471

```python
# unique values of maint column
keys_maint = np.array(train_dataset["maint"].unique())

# unique values of clazz column
keys_clazz = np.array(train_dataset["clazz"].unique())

# let's initiate a matrix to hold counts (maint X clazz)
p_maint_given_clazz = np.zeros((len(keys_maint), len(keys_clazz)))

# zip only maint and clazz values
for u in zip(train_dataset["maint"], train_dataset["clazz"]):
    # get index of current maint value
    index_maint = np.where(keys_maint == u[0])

    # get index of current clazz value
    index_clazz = np.where(keys_clazz == u[1])

    # increment 1 the count of the intersection of maint x clazz pair.
    p_maint_given_clazz[[index_maint], [index_clazz]] += 1

# let's normalize the possibilities
p_maint_given_clazz = p_maint_given_clazz / np.sum(p_maint_given_clazz, axis=0, keepdims=True)

# we create a pandas dataframe to visualize the table more familiar
df_p_maint_given_clazz = pd.DataFrame(p_maint_given_clazz, keys_maint, keys_clazz)

# possibilities of maint values given class
df_p_maint_given_clazz
```

      unacc
      acc
      vgood
      good

      vhigh
      0.297071
      0.186842
      0.000000
      0.000000

      high
      0.259414
      0.271053
      0.206349
      0.000000

      med
      0.220921
      0.302632
      0.380952
      0.338235

      low
      0.222594
      0.239474
      0.412698
      0.661765

```python
# unique values of doors column
keys_doors = np.array(train_dataset["doors"].unique())

# unique values of clazz column
keys_clazz = np.array(train_dataset["clazz"].unique())

# let's initiate a matrix to hold counts (doors X clazz)
p_doors_given_clazz = np.zeros((len(keys_doors), len(keys_clazz)))

# zip only doors and clazz values
for u in zip(train_dataset["doors"], train_dataset["clazz"]):
    # get index of current doors value
    index_doors = np.where(keys_doors == u[0])

    # get index of current clazz value
    index_clazz = np.where(keys_clazz == u[1])

    # increment 1 the count of the intersection of doors x clazz pair.
    p_doors_given_clazz[[index_doors], [index_clazz]] += 1

# let's normalize the possibilities
p_doors_given_clazz = p_doors_given_clazz / np.sum(p_doors_given_clazz, axis=0, keepdims=True)

# we create a pandas dataframe to visualize the table more familiar
df_p_doors_given_clazz = pd.DataFrame(p_doors_given_clazz, keys_doors, keys_clazz)

# possibilities of doors values given class
df_p_doors_given_clazz
```

      unacc
      acc
      vgood
      good

      2
      0.268619
      0.213158
      0.142857
      0.220588

      3
      0.247699
      0.257895
      0.238095
      0.264706

      4
      0.241841
      0.265789
      0.301587
      0.250000

      5more
      0.241841
      0.263158
      0.317460
      0.264706

```python
# unique values of persons column
keys_persons = np.array(train_dataset["persons"].unique())

# unique values of clazz column
keys_clazz = np.array(train_dataset["clazz"].unique())

# let's initiate a matrix to hold counts (persons X clazz)
p_persons_given_clazz = np.zeros((len(keys_persons), len(keys_clazz)))

# zip only persons and clazz values
for u in zip(train_dataset["persons"], train_dataset["clazz"]):
    # get index of current persons value
    index_persons = np.where(keys_persons == u[0])

    # get index of current clazz value
    index_clazz = np.where(keys_clazz == u[1])

    # increment 1 the count of the intersection of persons x clazz pair.
    p_persons_given_clazz[[index_persons], [index_clazz]] += 1

# let's normalize the possibilities
p_persons_given_clazz = p_persons_given_clazz / np.sum(p_persons_given_clazz, axis=0, keepdims=True)

# we create a pandas dataframe to visualize the table more familiar
df_p_persons_given_clazz = pd.DataFrame(p_persons_given_clazz, keys_persons, keys_clazz)

# possibilities of persons values given class
df_p_persons_given_clazz
```

      unacc
      acc
      vgood
      good

      2
      0.477824
      0.000000
      0.000000
      0.000000

      4
      0.257741
      0.513158
      0.460317
      0.529412

      more
      0.264435
      0.486842
      0.539683
      0.470588

```python
# unique values of lugboot column
keys_lugboot = np.array(train_dataset["lug_boot"].unique())

# unique values of clazz column
keys_clazz = np.array(train_dataset["clazz"].unique())

# let's initiate a matrix to hold counts (lugboot X clazz)
p_lugboot_given_clazz = np.zeros((len(keys_lugboot), len(keys_clazz)))

# zip only lugboot and clazz values
for u in zip(train_dataset["lug_boot"], train_dataset["clazz"]):
    # get index of current lugboot value
    index_lugboot = np.where(keys_lugboot == u[0])

    # get index of current clazz value
    index_clazz = np.where(keys_clazz == u[1])

    # increment 1 the count of the intersection of lugboot x clazz pair.
    p_lugboot_given_clazz[[index_lugboot], [index_clazz]] += 1

# let's normalize the possibilities
p_lugboot_given_clazz = p_lugboot_given_clazz / np.sum(p_lugboot_given_clazz, axis=0, keepdims=True)

# we create a pandas dataframe to visualize the table more familiar
df_p_lugboot_given_clazz = pd.DataFrame(p_lugboot_given_clazz, keys_lugboot, keys_clazz)

# possibilities of lugboot values given class
df_p_lugboot_given_clazz
```

      unacc
      acc
      vgood
      good

      small
      0.372385
      0.271053
      0.000000
      0.308824

      med
      0.323013
      0.355263
      0.380952
      0.338235

      big
      0.304603
      0.373684
      0.619048
      0.352941

```python
# unique values of safety column
keys_safety = np.array(train_dataset["safety"].unique())

# unique values of clazz column
keys_clazz = np.array(train_dataset["clazz"].unique())

# let's initiate a matrix to hold counts (safety X clazz)
p_safety_given_clazz = np.zeros((len(keys_safety), len(keys_clazz)))

# zip only safety and clazz values
for u in zip(train_dataset["safety"], train_dataset["clazz"]):
    # get index of current safety value
    index_safety = np.where(keys_safety == u[0])

    # get index of current clazz value
    index_clazz = np.where(keys_clazz == u[1])

    # increment 1 the count of the intersection of safety x clazz pair.
    p_safety_given_clazz[[index_safety], [index_clazz]] += 1

# let's normalize the possibilities
p_safety_given_clazz = p_safety_given_clazz / np.sum(p_safety_given_clazz, axis=0, keepdims=True)

# we create a pandas dataframe to visualize the table more familiar
df_p_safety_given_clazz = pd.DataFrame(p_safety_given_clazz, keys_safety, keys_clazz)

# possibilities of safety values given class
df_p_safety_given_clazz
```

      unacc
      acc
      vgood
      good

      med
      0.294561
      0.471053
      0.0
      0.558824

      high
      0.230126
      0.528947
      1.0
      0.441176

      low
      0.475314
      0.000000
      0.0
      0.000000

```python
# let's compute marginal probabilities of buying values
p_buying = np.sum(p_clazz * p_buying_given_clazz, axis=1)

# we create a pandas dataframe to visualize the table more familiar
df_p_buying = pd.DataFrame(p_buying, keys_buying)

# possibilities of buying values
df_p_buying
```

      0

      vhigh
      0.249707

      high
      0.250293

      med
      0.249707

      low
      0.250293

```python
# let's compute marginal probabilities of maint values
p_maint = np.sum(p_clazz * p_maint_given_clazz, axis=1)

# we create a pandas dataframe to visualize the table more familiar
df_p_maint = pd.DataFrame(p_maint, keys_maint)

# possibilities of maint values
df_p_maint
```

      0

      vhigh
      0.249707

      high
      0.249707

      med
      0.249707

      low
      0.250879

```python
# let's compute marginal probabilities of doors values
p_doors = np.sum(p_clazz * p_doors_given_clazz, axis=1)

# we create a pandas dataframe to visualize the table more familiar
df_p_doors = pd.DataFrame(p_doors, keys_doors)

# possibilities of doors values
df_p_doors
```

      0

      2
      0.249707

      3
      0.250293

      4
      0.249707

      5more
      0.250293

```python
# let's compute marginal probabilities of persons values
p_persons = np.sum(p_clazz * p_persons_given_clazz, axis=1)

# we create a pandas dataframe to visualize the table more familiar
df_p_persons = pd.DataFrame(p_persons, keys_persons)

# possibilities of persons values
df_p_persons
```

      0

      2
      0.334701

      4
      0.332943

      more
      0.332356

```python
# let's compute marginal probabilities of lug_boot values
p_lugboot = np.sum(p_clazz * p_lugboot_given_clazz, axis=1)

# we create a pandas dataframe to visualize the table more familiar
df_p_lugboot = pd.DataFrame(p_lugboot, keys_lugboot)

# possibilities of lug_boot values
df_p_lugboot
```

      0

      small
      0.333529

      med
      0.332943

      big
      0.333529

```python
# let's compute marginal probabilities of safety values
p_safety = np.sum(p_clazz * p_safety_given_clazz, axis=1)

# we create a pandas dataframe to visualize the table more familiar
df_p_safety = pd.DataFrame(p_safety, keys_safety)

# possibilities of safety values
df_p_safety
```

      0

      med
      0.333529

      high
      0.333529

      low
      0.332943

```python
correct = 0
for idx, item in test_dataset.iterrows():
    alpha = 1
    denominator = 0
    possibilities = {}
    for cls in keys_clazz:
        p = 1
        p *= (df_p_buying_given_clazz[cls][item.buying])
        p *= (df_p_maint_given_clazz[cls][item.maint])
        p *= (df_p_doors_given_clazz[cls][item.doors])
        p *= (df_p_persons_given_clazz[cls][item.persons])
        p *= (df_p_lugboot_given_clazz[cls][item.lug_boot])
        p *= (df_p_safety_given_clazz[cls][item.safety])
        p *= df_p_clazz[0][cls]

        denominator += p

        possibilities[cls] = p
    # end of for loop

    possibilities.update({k: v / denominator for k, v in possibilities.items()})

    #print(possibilities)
    correct += max(possibilities, key=possibilities.get) == item.clazz
# end of for loop

print("accuracy: %d%%" % ((correct / len(test_dataset)) * 100))
```

    accuracy: 81%

## References
* UCL Machine Learning Repository, https://archive.ics.uci.edu/ml/index.php (22/10/2017)
* Scikit-learn, Naive Bayes, http://scikit-learn.org/stable/modules/naive_bayes.html (22/10/2017)


## Universal Approximation Theorem:

Let $\phi(\cdot)$ be a non-constant, bounded and monotone-increasing continuous function.  Let $I_{m_0}$ denote the $m_0$-dimensional unit hypercube $[0, 1]^{m_0}$.  The space of continuous functions on $I_{m_0}$ is denoted by $C(I_{m_0})$.  Then, given any function $f \ni C(I_{m_0})$ and $\epsilon > 0$, there exists an integer $m_1$ and sets of real constants $\alpha_i, \beta_i,$ and $w_{ij}$, where $i = 1, \ldots, m_1$ and $j = 1, \ldots, m_0$ such that we may define
\begin{equation}
F(x_1, \ldots, x_{m_0}) = \sum_{i=1}^{m_1} \alpha_i \phi\left( \sum_{j=1}^{m_0} w_{ij}x_j + b_i\right)
\end{equation}
as an approximation realization of the function $f(\cdot)$: that is,
\begin{equation}
\left| F(x_1, \ldots, x_{m_0}) - f(x_1, \ldots, x_{m_0}) \right| < \epsilon
\end{equation}
for all $x_1, x_2, \ldots, x_{m_0}$ that like in the input space.}}

* Essentially, the Universal Approximation Theorem states that a single hidden layer is sufficient for a multilayer perceptron to compute a uniform $\epsilon$ approximation to a given training set - provided you have the *right* number of neurons and the *right* activation function.  (However, this does not say that a single hidden layer is optimal with regards to learning time, generalization, etc.)

## Background for Error Back-Propagation}

* Error Back-Propagation is based on *gradient descent*.
* Let's review/learn gradient descent:

*Method of Gradient/Steepest Descent:*

*move in direction opposite to the gradient vector, $g = \bigtriangledown E(\mathbf{w})$
\begin{eqnarray}
w(n+1) &=& w(n) - \eta g(n)\\
\Delta w(n) &=& w(n+1) - w(n)\\
\Delta w(n) &=& - \eta g(n) \quad \text{ Error correction rule }
\end{eqnarray}
* Show that using steepest descent, $E(\mathbf{w}(n+1)) < E(\mathbf{w}(n)) $
*  Recall: Taylor Series Expansion: $f(x) = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + ....$
* Approximate $E(\mathbf{w}(n+1))$ with Taylor Series around $w(n)$
\begin{eqnarray}
E(w(n+1)) &\approx& E(w(n)) + \Delta E(w(n))(w(n+1) - w(n))\\
&\approx& E(w(n)) + g^T(n)(\Delta w(n))\\
&\approx& E(w(n)) - \eta g^T(n)g(n)\\
&\approx& E(w(n)) - \eta \left\| g(n) \right\|^2
\end{eqnarray}
* For positive, small $\eta$, the cost function is decreased

## Error Back-propagation

* There are many approaches to train a neural network.
* One of the most commonly used is the *Error Back-Propagation Algorithm*.

* Two kinds of signals:
    1. Function Signals: presumed to perform useful function at the output of the network, also called input signal
    2.  Error Signals: propagates backwards, involves an error-dependent function

* Each hidden or output neuron performs two computations:
    1. Computation of function signal going out of this neuron
    2.  Computation of an estimate of the gradient vector

* First let's consider the output layer...

* Given a training set, $\left\{ \mathbf{x}_n, d_n\right\}_{n = 1}^N$, we want to find the parameters of our network that minimizes the squared error:
 \begin{equation}
 E(w) = \frac{1}{2} \sum_{n=1}^N (d_n - y_n)^2
 \end{equation}

* What is a common optimization approach to estimate the parameters that minimize an objective/error function? *gradient descent*
* To use gradient descent, what do we need?  The analytic form of the gradient.

 \begin{eqnarray}
 \frac{ \partial E}{\partial w_i} &=& \frac{\partial}{\partial w_i} \left[ \frac{1}{2} \sum_{n=1}^N (d_n - y_n)^2 \right]\\
 &=&  \frac{1}{2} \sum_{n=1}^N  \frac{\partial}{\partial w_i}  (d_n - y_n)^2 \\
 &=&  \frac{1}{2} \sum_{n=1}^N   2(d_n - y_n) \frac{\partial}{\partial w_i} (d_n - y_n) \\
  &=&  \sum_{n=1}^N   (d_n - y_n) \left( \frac{\partial}{\partial w_i} d_n -  \frac{\partial}{\partial w_i} y_n \right) \\
    &=&  \sum_{n=1}^N   (d_n - y_n) \left(  -  \frac{\partial }{\partial w_i} y_n \right)
 \end{eqnarray}

* What is $y_n$ in terms of $w_i$?  (At first let's assume we have no hidden layers, only the output layer to deal with)

\begin{equation}
y_n = \phi(v_n) = \phi(\mathbf{w}^T \mathbf{x}_n)
\end{equation}

* Going back to computing our gradient...
 \begin{eqnarray}
    &=&  \sum_{n=1}^N   (d_n - y_n) \left(  -  \frac{\partial }{\partial w_i} y_n \right) \\
    &=&  \sum_{n=1}^N  - (d_n - y_n)   \frac{\partial y_n}{\partial v_n} \frac{\partial v_n}{\partial w_i}
 \end{eqnarray}

* So, $\frac{\partial y_n}{\partial v_n}$ will depend what form of an activation function we use.  If we use the sigmoid: $y_n = \frac{1}{1 + \exp(-\alpha v_n)}$, then *what is* $\frac{\partial y_n}{\partial v_n}$ ?

 \begin{eqnarray}
 \frac{\partial y_n}{\partial v_n} &=& \frac{\partial \phi(v_n)}{\partial v_n}\\
  &=& \frac{ \partial }{\partial v_n}  \frac{1}{1 + \exp(-\alpha v_n)} \\
  &=& \frac{  \left(1 + \exp(- \alpha v_n)\right)\left(\frac{ \partial }{\partial v_n} 1\right) - \left(1\right)\left( \frac{ \partial }{\partial v_n}  1 + \exp(- \alpha v_n) \right)}{(1 + \exp(-\alpha v_n))^2}\\
    &=& \frac{  - \frac{ \partial }{\partial v_n}  (1 + \exp(- \alpha v_n) ) }{(1 + \exp(-\alpha v_n))^2}\\
    &=& \frac{  -1  }{(1 + \exp(-\alpha v_n))^2} \exp(-\alpha v_n)(-\alpha)\\
    &=&\frac{  1  }{1 + \exp(-\alpha v_n)} \frac{  1  }{1 + \exp(- \alpha v_n)} \exp(-\alpha v_n)\\
        &=&\frac{  1  }{1 + \exp(-\alpha v_n)} \frac{  \exp(-\alpha v_n)  }{1 + \exp(-\alpha v_n)}\\
        &=& y_n (1-y_n)
 \end{eqnarray}

* Going back to computing our gradient...
 \begin{eqnarray}
    &=&  \sum_{n=1}^N  - (d_n - y_n)   \frac{\partial y_n}{\partial v_n} \frac{\partial v_n}{\partial w_i} \\
     &=&  \sum_{n=1}^N  - (d_n - y_n)   y_n (1-y_n) \frac{\partial v_n}{\partial w_i}\\
     &=&  \sum_{n=1}^N  - (d_n - y_n)   y_n (1-y_n) \frac{\partial }{\partial w_i} \mathbf{w}^T \mathbf{x}_n\\
     &=&  \sum_{n=1}^N  - (d_n - y_n)   y_n (1-y_n) x_{ni}
 \end{eqnarray}

* *Now that we have the gradient, how do we use this to update the output layer weights in our MLP?*
* *How will this update equation  (for the output layer) change if the network is a multilayer perceptron with hidden units?*
* *Can you write this in vector form to update all weights simultaneously?*

## Backpropagation through hidden layers

* Now to address hidden layers... We have to deal with the credit assignment problem.

* Suppose we want to update $w_{ji}$ where $j$ is a hidden layer.

* The error objective function overall $N$ data points is

\begin{equation}
E(n) = \frac{1}{2}\sum_{n=1}^N e_n^2 = \sum_{n=1}^N \left(d_n - y_n \right)^2  = \sum_{n=1}^N \left(d_n - \phi_l(v_l(n)) \right)^2
\end{equation}

\begin{eqnarray}
\frac{\partial E(n)}{\partial w_{lj}} &=& \frac{\partial E(n)}{\partial e_l(n)} \frac{\partial e_l(n)}{\partial y_l(n)}\frac{\partial y_l(n)}{\partial v_{l}(n)}\frac{\partial v_l(n)}{\partial w_{lj}}\\
&=&  [e_l][-1][\phi^{\prime}(v_l(n))][y_{jl}(n)] \\
\end{eqnarray}

* Let's define a *local gradient* $\delta_l(n)$:

\begin{eqnarray}
\delta_l(n) &=& - \frac{\partial E(n)}{\partial v_l(n)}\\
 &=&  e_l(n) \phi^{\prime}(v_l(n))
 \end{eqnarray}

* Similarly,

\begin{eqnarray}
\delta_j(n) &=& - \frac{\partial E(n)}{\partial v_j(n)}\\
&=& - \frac{\partial E(n)}{\partial y_j(n)}\frac{\partial y_j(n)}{\partial v_j(n)} \\
&=& - \frac{\partial E(n)}{\partial y_j(n)} \phi^{\prime}(v_j(n))
 \end{eqnarray}

* Note that:
 \begin{eqnarray}
 \frac{\partial E(n)}{\partial y_j(n)} &=&  \sum_{l}e_l(n)\frac{\partial e_l(n)}{\partial y_j(n)} \\
 &=&  \sum_{l}e_l(n)\frac{\partial e_l(n)}{\partial v_l(n)} \frac{\partial v_l(n)}{\partial y_j(n)} \\
  &=&  \sum_{l}e_l(n)[-\phi^{\prime}(v_l(n))][w_{kj}(n)]
\end{eqnarray}

* So,

 \begin{eqnarray}
 \delta_j(n) &=& - \frac{\partial E(n)}{\partial y_j(n)} \phi^{\prime}(v_j(n)) \\
 &=& - \left[ \sum_{l}e_l(n)[-\phi^{\prime}(v_l(n))][w_{kj}(n)] \right]  \phi^{\prime}(v_j(n))\\
 &=& \phi^{\prime}(v_j(n)) \sum_l \delta_l(n)w_{kj}(n)
\end{eqnarray}

* So, you can write the gradient at a hidden neuron in terms of the local gradient and the connected neurons in the next layer

\begin{equation}
\delta w_{ij}(n) = \eta \delta_j(n) y_i(n)
\end{equation}

```python

```

Lecture17_Backpropagation/Lecture 17 - Backpropagation.ipynb


```python
from __future__ import print_function
import math
import sisl
import numpy as np
import matplotlib.pyplot as plt
from functools import partial
%matplotlib inline
```

One of the key ideas behind `sisl` was the interaction of a DFT Hamiltonian and the user.
In this example we will highlight a unique implementation in TBtrans which enables ***any*** kind of user intervention.

The idea is a transformation of the Green function calculation from:
\begin{equation}
   \mathbf G^{-1}(E) = \mathbf S (E + i\eta) - \mathbf H - \sum_i\boldsymbol\Sigma_i
\end{equation}
to
\begin{equation}
   \mathbf G^{-1}(E) = \mathbf S (E + i\eta) - \mathbf H - \delta\mathbf H - \sum_i\boldsymbol\Sigma_i - \delta\boldsymbol \Sigma
\end{equation}
where $\delta\mathbf H$ and $\delta\boldsymbol\Sigma$ can be of any type, i.e. complex and/or real.
The only (important!) difference between $\delta\mathbf H$ and $\delta\boldsymbol \Sigma$ is that the former enters the calculation of bond-currents, while the latter does not.

Since TBtrans by it-self does not allow complex Hamiltonians the above is a way to leviate this restriction. One feature this may be used for is by applying magnetic fields.

In the following we will use Peierls substitution on a square tight-binding model:
\begin{equation}
   \mathbf H(\Phi) = \mathbf H(0)e^{i \Phi \delta x^- \cdot \delta y^+ / 2},
\end{equation}
where $\Phi$ is the magnetic flux (in proper units), $\delta x^- = x_j - x_i$ and $\delta y^+=y_j + y_i$, with $i$ and $j$ being atomic indices. If you are interested in this substitution, please search litterature.

---

First create a square lattice and define the on-site and nearest neighbour couplings

```python
square = sisl.Geometry([0,0,0], sisl.Atom(1, R=1.0), sc=sisl.SuperCell(1, nsc=[3, 3, 1]))
on, nn = 4, -1
```

```python
H_minimal = sisl.Hamiltonian(square)
H_minimal.construct([[0.1, 1.1], [on, nn]])
H_elec = H_minimal.tile(100, 1).tile(2, 0)
H_elec.set_nsc([3, 1, 1])
H_elec.write('ELEC.nc')
```

```python
H = H_elec.tile(50, 0)

# Make a constriction
geom = H.geom.translate( -H.geom.center(what='xyz') )
# This constriction is based on an example in the kwant project (called qhe). We however make a slight modification.
# Remove some atoms, this will create a constriction of 100 - 40 * 2 = 20 Ang with a Gaussian edge profile
remove = (np.abs(geom.xyz[:, 1]) > 50 - 37.5 * np.exp( -(geom.xyz[:, 0] / 12) **2 )).nonzero()[0]
# To reduce computations we find the atoms in the constriction such that we can
# limit the calculation region.
device = (np.abs(geom.remove(remove).xyz[:, 0]) < .6).nonzero()[0]
geom.remove(remove).write('test.xyz')

# Pretty print a range of atoms that is the smallest device region
print(sisl.utils.list2str(device))

H = H.remove(remove)
H.write('DEVICE.nc')
```

The above printed list of atoms should be inserted in the `RUN.fdf` in the `TBT.Atoms.Device`. This is important when one is *only* interested in the transmission, and does not care about the density of states. You are always encouraged to select the minimal device region to 1) speed up computations and 2) drastically reduce memory requirements.
In this regard you should read in the TBtrans manual about the additional flag `TBT.Atoms.Device.Connect` (you may try, as an additional exercise to set this flag to true and check the difference from the previous calculation).

Now we have $\mathbf H(0)$ with *no* phases due to magnetic fields. As the magnetic field is changing the Hamiltonian, and thus enters the bond-current calculations, we have to use the $\delta\mathbf H$ term (and *not* $\delta\boldsymbol\Sigma$).

The first thing we need to calculate is $\delta x^- \cdot\delta y^+$.
Since we already have the Hamiltonian we can utilize the connections by looping the coupling elements (in a sparse matrix/graph this is called *edges*). This is *much* cheaper than trying to figure out which atoms are neighbouring.

```python
device = H.geom
xy = sisl.Hamiltonian(device)
for ia in device:
    # Get all connecting elements (excluding it-self)
    edges = H.edges(ia)
    # Calculate the vector between edges and ia:
    #    xyz[edges, :] - xyz[ia, :]
    Rij = device.Rij(ia, edges)
    # Now calculate the product:
    #    (xj - xi) * (yj + yi)
    # Notice that we correct to +yi by adding it twice
    xy[ia, edges] = Rij[:, 0] * (Rij[:, 1] + 2 * device.xyz[ia, 1])
    xy[ia, ia] = 0.
xy.finalize() # this is only because it will speed up the following calculations
```

Now we have the coupling dependent phase factor, $\delta x^-\cdot\delta y^+$, and all we need to calculate is $\delta\mathbf H$ that transforms $\mathbf H(0) \to \mathbf H(\Phi)$.
This is done easily as a `Hamiltonian` allows basic element wise operations, i.e. `+`, `-`, `*`, `/` and `**` (the power function).
Your task is to insert the correct mathematical equation below, such that `dH` contains $\delta \mathbf H$ for $\mathbf H(\Phi) = \mathbf H(0) + \delta\mathbf H$.
To help you I have inserted the exponential function. To finalize the equation, you need three terms: `nn`, `xy` and `rec_phi`.
<!-- dH = math.e ** (0.5j / rec_phi * xy) * nn - nn-->

```python
rec_phis = np.arange(1, 51, 4)
print('Calulating (of {}):'.format(len(rec_phis)), end='')
for i, rec_phi in enumerate(rec_phis):
    print(' {}'.format(i+1), end=',')
    # Calculate H(Phi)
    # TODO: insert the correct mathematical equation below to calculate the correct phase
    #dH = math.e ** (0.5j  ... ) ...
    with sisl.get_sile('M_{}.dH.nc'.format(rec_phi), mode='w') as fh:
        fh.write_delta(dH)
```

## Exercises

- Calculate all physical quantities for all different applied magnetic fields.
   Before running the calculations, search the manual on how to save the self-energies (*HINT* out-of-core). By default, TBtrans calculates the self-energies as they are needed. However, if one has the same electrodes, same $k$-grid and same $E$-points for several different runs (as in this case) one can with benefit calculate the self-energies *once*, and then reuse them in subsequent calculations.

   To ease the calculation of all magnetic fields
   To help you a script `run.sh` is located in this directory. Carefully read it to infer which option specifies the $\delta \mathbf H$ term.

   Since this example has 14 different setups, each with 51 energy points, it will take some time. Around 30 seconds for the first (includes self-energy calculation), and around 10 seconds for all subsequent setups. So be patient. :)
- Secondly, read in all output into the workbook in a list.
- **TIME (*HARD*)**:
    Choose an energy-point for $B=0$ and $B>0$ such that you have a finite transmission ($T>0$). Next, extract the bond-currents and calculate the transmission using the law of current conservation (Kirchhoffs 1st law: $T_{\mathrm{in}} \equiv T_{\mathrm{out}}$). *HINT*: select a line of atoms and calculate the bond-currents flowing between this line and its neighbouring line of atoms. Assert that the transmission is the same using both methods.

```python
# Create short-hand function
gs = sisl.get_sile
# No magnetic field
tbt0 = gs('siesta.TBT.nc')
# All magnetic fields in increasing order
tbts = [gs('M_{}/siesta.TBT.nc'.format(rec_phi)) for rec_phi in rec_phis]
```

- Plot the transmission function for all applied fields in the full energy range.

```python
# Do trick with easy plotting utility
E = tbt0.E[:]
Eplot = partial(plt.plot, E)

for rec_phi, tbt in zip(rec_phis, tbts):
    Eplot(tbt.transmission(), '--', label='{}'.format(rec_phi));
Eplot(tbt0.transmission(), 'k');
plt.xlim([E.min(), E.max()]); plt.ylim([0, None]);
plt.xlabel('Energy [eV]'); plt.ylabel('Transmisson'); plt.legend(bbox_to_anchor=(1, 1), loc=2);
```

- Make a contour plot of the transmission vs. $\Phi$ and $E$

```python
T = np.stack([tbt.transmission() for tbt in tbts])
T[T < 1e-9] = 0 # small numbers are difficult to interpolate (they tend to blow up), so remove them
plt.contourf(rec_phis, E, T.T, 20); plt.colorbar(label='Transmission');
plt.ylabel('Energy [eV]'); plt.xlabel(r'$1/\Phi$');
```

- **TIME** Choose a given magnetic field and create a different set of constriction widths and plot $T(E, \Phi)$ for different widths, fix $\Phi$ at a fairly large value (copy codes in `In [4-7]` and adapt), you may decide the Gaussian profile and change if you want.

```python

```

```python

```

## Learned lessons

- Advanced construction of geometries by removing subsets of atoms (both for Hamiltonian and geometries)
- Creation of $\delta\mathbf H$ terms. Note that *exactly* the same method is used for the $\delta\boldsymbol\Sigma$ terms, the only difference is how it is specified in the fdf-file (`TBT.dH` vs. `TBT.dSE`). Please look in the manual and figure out what the difference is between the two methods?
- Supplying fdf-flags to TBtrans on the command line to override flags in the input files.
- Inform TBtrans to store the self-energies on disk to re-use them in later calculations.
- Inform TBtrans to make all output into a sub-folder (and if it does not exist, create it)
- Adding complex valued Hamiltonians, we have not uncovered everything as the $\delta$ files may contain $k$-resolved and/or $E$-resolved $\delta$-terms for full control, but the principles are the same. Search the documentation for `deltancSileTBtrans`.

```python

```