# Hydrodynamics Systems with a General EOS

This notebook explores the eigensystem of the Euler equations augmented with an additional thermodynamic variable to describe a general equation of state in the reconstruction of interface states

```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.0 (Python 3.5.2-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

```python
from sympy.abc import rho, tau, alpha
rho, tau, c, h, p = symbols("rho tau c h p", real=True, positive=True)
re = symbols(r"(\rho{}e)", real=True, positive=True)
ge = symbols(r"\gamma_e", real=True, positive=True)
alpha, u = symbols("alpha u", real=True)
```

The two routines below simplify the analysis of the eigensystem by simultaneously finding the set of orthonormal left and right eigenvectors of a matrix from the primitive form of the hydro equations

```python
plus = u + c
zero = u
minus = u - c

class Eigenvector(object):
    def __init__(self, name, ev, r, l=None):
        self.name = name
        if name == "minus":
            self.d = 0
        elif name == "zero":
            self.d = 1
        elif name == "plus":
            self.d = 2
        else:
            self.d = None

        self.ev = ev
        self.l = l
        self.r = r

    def __lt__(self, other):
        return self.d < other.d

    def __str__(self):
        return "{} wave, r = {}, l = {}".format(self.eigenvalue, self.r, self.l)

def eigensystem(A, suba=None, subb=None):
    # get the left and right eigenvectors that diagonalize the system.
    # it is best to use sympy diagonalize() for this purpose than getting
    # the left and right eigenvectors independently.

    e = []

    R, D = A.diagonalize()

    # the columns of R are the right eigenvectors and the diagonal
    # element of D is the corresponding eigenvalues

    for n in range(A.shape[0]):
        r = R.col(n)
        ev = D[n,n]

        #print("here", r, ev)
        if suba is not None and subb is not None:
            ev = ev.subs(suba, subb)

        # which eigenvalue are we?
        if simplify(ev - minus) == 0:
            name = "minus"
        elif simplify(ev - plus) == 0:
            name = "plus"
        elif simplify(ev - zero) == 0:
            name = "zero"
        else:
            return None

        # normalize the right eigenvector
        v = r[0]
        if v != 0:
            r = r/v

        if suba is not None and subb is not None:
            r = simplify(r.subs(suba, subb))

        e.append(Eigenvector(name=name, ev=ev, r=r))

    # now sort the system from smallest (u-c) to largest (u+c)
    e.sort()

    # now let's construct the R with this sorting
    for n in range(A.shape[0]):
        R[:,n] = e[n].r

    # the left eigenvector matrix, L, is just the inverse
    L = R**-1

    for n in range(A.shape[0]):
        e[n].l = L.row(n)

    return e
```

# Euler Equations with $(\rho e)$

The Euler equations in primitive variable form, $q = (\rho, u, p, (\rho e))^\intercal$ appear as:
\begin{align}
\frac{\partial \rho}{\partial t} &= -u \frac{\partial \rho}{\partial x} - \rho \frac{\partial u}{\partial x} \\
\frac{\partial u}{\partial t} &= -u \frac{\partial u}{\partial x} - \frac{1}{\rho} \frac{\partial p}{\partial x} \\
\frac{\partial p}{\partial t} &= -u \frac{\partial p}{\partial x} - \rho c^2 \frac{\partial u}{\partial x} \\
\frac{\partial (\rho e)}{\partial t} &= -u \frac{\partial (\rho e)}{\partial x} - \rho h \frac{\partial u}{\partial x}
\end{align}

In vector form, we have:
$$q_t + A(q) q_x = 0$$

with the matrix $A(q)$:

$$A(q) = \left ( \begin{array}{cccc} u  & \rho     & 0    & 0\\
                                  0  &  u       & 1/\rho  & 0\\
                                  0  & \rho c^2 & u       & 0\\
                                  0  & \rho h & 0       & u\end{array} \right )
$$

The sound speed is related to the adiabatic index, $\Gamma_1$, as $c^2 = \Gamma_1 p /\rho$.

We can represent this matrix symbolically in SymPy and explore its eigensystem.

```python
q = Matrix([rho, u, p, re]).transpose()
A = Matrix([[u, rho, 0, 0], [0, u, rho**-1, 0], [0, c**2 * rho, u, 0], [0, rho*h, 0, u]])
A
```

The eigenvalues are the speeds at which information propagates with.  SymPy returns them as a
dictionary, giving the multiplicity for each eigenvalue.

```python
A.eigenvals()
```

We see that there are 2 eigenvalues $u$ -- the addition of $(\rho e)$ to the system adds this degeneracy.

## Eigenvectors

The right eigenvectors are defined for a given eigenvalue, $\lambda$, as:
$$A r = \lambda r$$
and the left eigenvectors satisfy:
$$l A = \lambda l$$

Note that the left and right eigenvectors are orthogonal to those corresponding to a different eigenvalue, and usually normalized so:
$$l^i \cdot r^j = \delta_{ij}$$

```python
# we use the helper rountines above to find the orthogonal left and right eigenvectors
eigen = eigensystem(A)
```

```python
# printing them out for inspection
from sympy.printing.mathml import mathml
for e in eigen:
    print(e.name)
    display(e.r, e.l)
```

## $\beta$'s and final update

The final interface state is writen by projecting the jump in primitive variables, $\Delta q$, into characteristic variables (as $l \cdot \Delta q$), and then adding up all the jumps that reach the interface.

The convention is to write $\beta^\nu = l^\nu \cdot \Delta q$, where the superscript identifies which eigenvalue (and corresponding eigenvectors) we are considering.  Note, that often a reference state is used, and the jump, $\Delta q$, will be the difference with respect to this reference state.  For PPM, the $\Delta q$ will take the form of the integral under the parabola over the range that each wave can reach.

The final interface state is then:
$$
q_\mathrm{int} = q - \sum_\nu \beta^\nu r^\nu
$$

The tracing projects the primitive variables into characteristic variables by defining
$$
\Delta q = \left ( \begin{array}{c} \Delta \rho \\ \Delta u \\ \Delta p \\ \Delta (\rho e) \end{array} \right )
$$
and then
$\beta^\nu = l^\nu \cdot \Delta q$

```python
from sympy.abc import delta

dr = symbols(r"\Delta\rho")
du = symbols(r"\Delta{}u")
dp = symbols(r"\Delta{}p")
dre = symbols(r"\Delta(\rho{}e)")

rhoi = symbols(r"\rho_\mathrm{int}")
ui = symbols(r"u_\mathrm{int}")
pri = symbols(r"p_\mathrm{int}")
rei = symbols(r"(\rho{}e)_\mathrm{int}")

# this is the jump
dq = Matrix([[dr, du, dp, dre]]).transpose()

# this is the interface state vector
qint = Matrix([[rhoi, ui, pri, rei]]).transpose()
```

Now compute the $\beta$s

```python
betas = [symbols(r"\beta^-")]
for n in range(len([e for e in eigen if e.d == 1])):
    betas += [symbols(r"\beta^0_{}".format(n))]
betas += [symbols(r"\beta^+")]

for n, e in enumerate(eigen):
    print(e.name)
    beta = e.l.dot(dq)
    display(Eq(betas[n], simplify(beta)))
```

and now the final interface states

```python
for n in range(len(eigen)):
    rhs = q[n]
    for m in range(len(eigen)):
        rhs -= betas[m]*eigen[m].r[n]
    display(Eq(qint[n],rhs))
```

-----

# Euler Equations with $(\gamma_e)$

We can define $\gamma_e = p/(\rho e) + 1$ and differentiate it to get:
$$
\frac{\partial \gamma_e}{\partial t} = -u \frac{\partial \gamma_e}{\partial x} + (\gamma_e - 1)(\gamma_e - \Gamma_1) \frac{\partial u}{\partial x}
$$
The original CG paper used $\tau = 1/\rho$ in place of density.  With this, the continuity equation becomes:
$$
\frac{\partial \tau}{\partial t} = -u\frac{\partial \tau}{\partial x}  + \tau \frac{\partial u}{\partial x}
$$

The Euler equations with this set of primitive variables, $q = (\tau, u, p, \gamma_e)^\intercal$ appear as:
\begin{align}
\frac{\partial \tau}{\partial t} &= -u\frac{\partial \tau}{\partial x}  + \tau \frac{\partial u}{\partial x} \\
\frac{\partial u}{\partial t} &= -u \frac{\partial u}{\partial x} - \frac{1}{\rho} \frac{\partial p}{\partial x} \\
\frac{\partial p}{\partial t} &= -u \frac{\partial p}{\partial x} - \rho c^2 \frac{\partial u}{\partial x} \\
\frac{\partial \gamma_e}{\partial t} &= -u \frac{\partial \gamma_e}{\partial x} + (\gamma_e - 1)(\gamma_e - \Gamma_1) \frac{\partial u}{\partial x}
\end{align}

For convenience, we define
$$
\alpha = (\gamma_e - 1)(\gamma_e - \Gamma_1)
$$
and then in vector form, we have:
$$q_t + A(q) q_x = 0$$

with the matrix $A(q)$:

$$A(q) = \left ( \begin{array}{cccc} u  & -\tau     & 0    & 0\\
                                  0  &  u       & \tau  & 0\\
                                  0  & c^2/\tau & u       & 0\\
                                  0  & -\alpha & 0       & u\end{array} \right )
$$

We can represent this matrix symbolically in SymPy and explore its eigensystem.

```python
q = Matrix([[tau, u, p, ge]]).transpose()

A = Matrix([[u, -tau, 0, 0], [0, u, tau, 0], [0, c**2/tau, u, 0], [0, -alpha, 0, u]])
A
```

The eigenvalues are the speeds at which information propagates with.  SymPy returns them as a
dictionary, giving the multiplicity for each eigenvalue.

```python
A.eigenvals()
```

We see that there are 2 eigenvalues $u$ -- the addition of $\gamma_e$ to the system adds this degeneracy.

## Eigenvectors

```python
# we use the helper rountines above to find the orthogonal left and right eigenvectors
eigen = eigensystem(A)
```

```python
# printing them out for inspection
from sympy.printing.mathml import mathml
for e in eigen:
    print(e.name)
    display(e.r, e.l)
```

## $\beta$'s and final update

The tracing projects the primitive variables into characteristic variables by defining
$$
\Delta q = \left ( \begin{array}{c} \Delta \tau \\ \Delta u \\ \Delta p \\ \Delta \gamma_e \end{array} \right )
$$
and then
$\beta^\nu = l^\nu \cdot \Delta q$

```python
dtau = symbols(r"\Delta\tau")
dge = symbols(r"\Delta\gamma_e")

dq = Matrix([[dtau, du, dp, dge]]).transpose()

taui = symbols(r"\tau_\mathrm{int}")
gei = symbols(r"(\gamma_e)_\mathrm{int}")

qint = Matrix([[taui, ui, pri, gei]]).transpose()
```

```python
betas = [symbols(r"\beta^-")]
for n in range(len([e for e in eigen if e.d == 1])):
    betas += [symbols(r"\beta^0_{}".format(n))]
betas += [symbols(r"\beta^+")]

for n, e in enumerate(eigen):
    print(e.name)
    beta = e.l.dot(dq)
    display(Eq(betas[n], simplify(beta)))
```

and now the final interface states

```python
for n in range(len(eigen)):
    rhs = q[n]
    for m in range(len(eigen)):
        rhs -= betas[m]*eigen[m].r[n]
    display(Eq(qint[n],rhs))
```

------

# Gray FLD Radiation Euler Equations with $(\rho e)$

Following Zhang et al. (2011), the equations of gray radiation hydrodynamics with primitive
variables $q = (\rho, u, p, (\rho e)_g, E_r)^\intercal$ are:
\begin{align}
\frac{\partial \rho}{\partial t} &= -u\frac{\partial \rho}{\partial x}  - \rho \frac{\partial u}{\partial x} \\
\frac{\partial u}{\partial t} &= -u \frac{\partial u}{\partial x} - \frac{1}{\rho} \frac{\partial p}{\partial x}
    - \frac{\lambda_f}{\rho} \frac{\partial E_r}{\partial x}\\
\frac{\partial p}{\partial t} &= -u \frac{\partial p}{\partial x} - \rho c_g^2 \frac{\partial u}{\partial x} \\
\frac{\partial (\rho e)_g}{\partial t} &= -u \frac{\partial (\rho e)_g}{\partial x} - \rho h \frac{\partial u}{\partial x} \\
\frac{\partial E_r}{\partial t} &= -\frac{3-f}{2} E_r \frac{\partial u}{\partial x} - \left ( \frac{3-f}{2} - \lambda_f\right ) u \frac{\partial E_r}{\partial x}
\end{align}
where $(\rho e)_g$ is the gas internal energy density, $h_g$ is the gas specific enthalpy, $c_g$ is the gas sound speed (obeying $c_g^2 = \Gamma_1 p /\rho$), $E_r$ is the radiation energy density, $f$ is the Eddington factor, and $\lambda_f$ is the flux limiter.

Following Zhang et al., we make the approximation that
$$
\frac{3-f}{2} = \lambda_f + 1
$$
and then in vector form, we have:
$$q_t + A(q) q_x = 0$$

with the matrix $A(q)$:

$$A(q) = \left ( \begin{array}{ccccc} u  & \rho     & 0    & 0    & 0 \\
                                      0  &  u       & 1/\rho & 0   & \lambda_f/\rho\\
                                      0  & \rho c_g^2 & u       & 0 & 0 \\
                                      0  & \rho h_g   & 0       & u & 0 \\
                                      0  & (3-f)E_r/2   & 0       & 0 & u\end{array} \right )
$$

We can represent this matrix symbolically in SymPy and explore its eigensystem.

```python
cg = symbols(r"c_g", real=True, positive=True)
hg = symbols(r"h_g", real=True, positive=True)
Er = symbols(r"E_r", real=True, positive=True)
lf = symbols(r"\lambda_f", real=True)
f = symbols("f", real=True, positive=True)

q = Matrix([[rho, u, p, re, Er]]).transpose()
A = Matrix([[u, rho, 0, 0, 0],
            [0, u, rho**-1, 0, lf/rho],
            [0, rho*cg**2, u, 0, 0],
            [0, rho*hg, 0, u, 0],
            [0, (lf+1)*Er, 0, 0, u]])
A
```

The eigenvalues are the speeds at which information propagates with.  SymPy returns them as a
dictionary, giving the multiplicity for each eigenvalue.

```python
A.eigenvals()
```

We see that there are 3 eigenvalues $u$.  We identify the total sound speed (radiation + gas) as:
$$c^2 = c_g^2 + (\lambda_f + 1)\frac{\lambda_f E_r}{\rho}$$
We can simplify these by substituting in that relationship

```python
cc = c**2 - (lf +1)*lf*Er/rho

evs = A.eigenvals()
for e in evs.keys():
    display(powsimp(simplify(e.subs(cg**2, cc))))
```

We see that these are the same form of the eigenvalues for the pure hydrodynamics system

## Eigenvectors

```python
# we use the helper rountines above to find the orthogonal left and right eigenvectors
eigen = eigensystem(A, suba=cg, subb=sqrt(cc))
```

```python
# printing them out for inspection
for e in eigen:
    print(e.name)
    display(e.r)
    display(simplify(e.l))
```

## $\beta$'s and final update

The tracing projects the primitive variables into characteristic variables by defining
$\beta^\nu = l^\nu \cdot \Delta q$

```python
dEr = symbols(r"\Delta{}E_r")
Eri = symbols(r"{E_r}_\mathrm{int}")
dq = Matrix([[dr, du, dp, dre, dEr]]).transpose()

qint = Matrix([[rhoi, ui, pri, rei, Eri]])
```

```python
betas = [symbols(r"\beta^-")]
for n in range(len([e for e in eigen if e.d == 1])):
    betas += [symbols(r"\beta^0_{}".format(n))]
betas += [symbols(r"\beta^+")]

for n, e in enumerate(eigen):
    print(e.name)
    beta = e.l.dot(dq)
    display(Eq(betas[n], simplify(beta)))
```

and now the final interface states

```python
for n in range(len(eigen)):
    rhs = q[n]
    for m in range(len(eigen)):
        rhs -= betas[m]*eigen[m].r[n]
    display(Eq(qint[n],rhs))
```

```python

```

# Gray FLD Radiation Euler Equations with $(\gamma_e)$

We now look at the same system with a different auxillary thermodynamic variable (as we did with pure hydro), using $q = (\tau, u, p, {\gamma_e}_g, E_r)^\intercal$:
\begin{align}
\frac{\partial \tau}{\partial t} &= -u\frac{\partial \tau}{\partial x}  + \tau \frac{\partial u}{\partial x} \\
\frac{\partial u}{\partial t} &= -u \frac{\partial u}{\partial x} - \tau \frac{\partial p}{\partial x}
    - \tau \lambda_f \frac{\partial E_r}{\partial x}\\
\frac{\partial p}{\partial t} &= -u \frac{\partial p}{\partial x} - \frac{c_g^2}{\tau} \frac{\partial u}{\partial x} \\
\frac{\partial {\gamma_e}_g}{\partial t} &= -u \frac{\partial {\gamma_e}_g}{\partial x} + \alpha \frac{\partial u}{\partial x} \\
\frac{\partial E_r}{\partial t} &= -\frac{3-f}{2} E_r \frac{\partial u}{\partial x} - \left ( \frac{3-f}{2} - \lambda_f\right ) u \frac{\partial E_r}{\partial x}
\end{align}
here, ${\gamma_e}_g$ is defined solely in terms of the gas pressure and energy, the remaining variables have the same meaning as above.  We again make the approximation that
$$
\frac{3-f}{2} = \lambda_f + 1
$$
and then in vector form, we have:
$$q_t + A(q) q_x = 0$$

with the matrix $A(q)$:

$$A(q) = \left ( \begin{array}{ccccc} u  & -\tau    & 0    & 0    & 0 \\
                                      0  &  u       & \tau & 0   & \tau \lambda_f\\
                                      0  & c_g^2/\tau & u       & 0 & 0 \\
                                      0  & -\alpha   & 0       & u & 0 \\
                                      0  & (3-f)E_r/2   & 0       & 0 & u\end{array} \right )
$$

We can represent this matrix symbolically in SymPy and explore its eigensystem.

```python
geg = symbols(r"{\gamma_e}_g", real=True, positive=True)
Er = symbols(r"E_r", real=True, positive=True)
lf = symbols(r"\lambda_f", real=True, positive=True)
cg = symbols(r"c_g", real=True, positive=True)

q = Matrix([[tau, u, p, geg, Er]]).transpose()
A = Matrix([[u, -tau, 0, 0, 0],
            [0, u, tau, 0, tau*lf],
            [0, cg**2/tau, u, 0, 0],
            [0, -alpha, 0, u, 0],
            [0, (lf+1)*Er, 0, 0, u]])
A
```

The eigenvalues are the speeds at which information propagates with.  SymPy returns them as a
dictionary, giving the multiplicity for each eigenvalue.

```python
A.eigenvals()
```

We see that there are 3 eigenvalues $u$.  We identify the total sound speed (radiation + gas) as:
$$c^2 = c_g^2 + (\lambda_f + 1)\lambda_f E_r\tau$$
We can simplify these by substituting in that relationship

```python
cc = c**2 - (lf +1)*lf*Er*tau

evs = A.eigenvals()
for e in evs.keys():
    display(powsimp(simplify(e.subs(cg**2, cc))))
```

We see that these are the same form of the eigenvalues for the pure hydrodynamics system

## Eigenvectors

```python
# we use the helper rountines above to find the orthogonal left and right eigenvectors
eigen = eigensystem(A, suba=cg, subb=sqrt(cc))
```

```python
# printing them out for inspection
for e in eigen:
    print(e.name)
    display(e.r)
    display(simplify(e.l))
```

## $\beta$'s and final update

The tracing projects the primitive variables into characteristic variables by defining
$\beta^\nu = l^\nu \cdot \Delta q$

```python
dgeg = symbols(r"\Delta{\gamma_e}_g")
dEr = symbols(r"\Delta{}E_r")
dq = Matrix([[dtau, du, dp, dgeg, dEr]]).transpose()

Eri = symbols(r"{E_r}_\mathrm{int}")
qint = Matrix([[taui, ui, pri, gei, Eri]])
```

```python
betas = [symbols(r"\beta^-")]
for n in range(len([e for e in eigen if e.d == 1])):
    betas += [symbols(r"\beta^0_{}".format(n))]
betas += [symbols(r"\beta^+")]

for n, e in enumerate(eigen):
    print(e.name)
    beta = e.l.dot(dq)
    display(Eq(betas[n], simplify(beta)))
```

and now the final interface states

```python
for n in range(len(eigen)):
    rhs = q[n]
    for m in range(len(eigen)):
        rhs -= betas[m]*eigen[m].r[n]
    display(Eq(qint[n],rhs))
```

```python

```


# Project 4, deadline  November 18

**[Computational Physics I FYS3150/FYS4150](http://www.uio.no/studier/emner/matnat/fys/FYS3150/index-eng.html)**, Department of Physics, University of Oslo, Norway

Date: **Fall semester 2019**

## Studies of phase transitions in magnetic systems

### Introduction

The aim of this project is to study a widely popular model to simulate phase transitions, the so-called Ising model in two dimensions. At a given critical temperature, this model exhbits a phase transition from a magnetic phase (a system with a finite magnetic moment) to a phase with zero magnetization.
This is a so-called binary system where the objects at each lattice site can only take two values. These could be $0$ and $1$ or other values. Here we will use spins pointing up or down as the model for our system. But we could replace the spins with blue and green balls for example.
The [Ising model](https://en.wikipedia.org/wiki/Ising_model) has been extremely popular, with applications spanning from studies of phase transitions to simulations in statistics. In one and two dimensions its has analytical solutions to several expectation values and it gives a qualitatively  good underatanding of several types of phase transitions.

In its simplest form
the energy of the Ising model is expressed as, without an externally applied magnetic field,

$$
E=-J\sum_{< kl >}^{N}s_ks_l
$$

with
$s_k=\pm 1$. The quantity $N$ represents the total number of spins and $J$ is a coupling
constant expressing the strength of the interaction between
neighboring spins.  The symbol $<kl>$ indicates that we sum over
nearest neighbors only. We will assume that we have a ferromagnetic
ordering, viz $J> 0$.  We will use periodic boundary conditions and
the Metropolis algorithm only. The material on the Ising model can be found in chapter 13 of the lecture notes. The Metropolis algorithm is discussed in chapter 12.

**For this project you can hand in collaborative reports and programs.**
This project (together with projects 3 and 5) counts 1/3 of the final mark.

### Project 4a): A simple $2\times 2$ lattice, analytical expressions

Assume we have only two spins in each dimension, that is $L=2$.
Find the analytical expression
for the partition function and the corresponding
expectations values for the energy $E$, the mean absolute value of the magnetic moment $\vert M\vert$ (we will refer to this as the mean magnetization),
the specific heat $C_V$ and the susceptibility $\chi$
as functions of  $T$ using periodic boundary conditions. These results will serve as benchmark calculations for our next steps.

### Project 4b): Writing a code for the Ising model

Write now a code for the Ising model which computes the mean energy
$E$, mean magnetization
$\vert M\vert$, the specific heat $C_V$ and the susceptibility $\chi$
as functions of  $T$ using periodic boundary conditions for
$L=2$ in the $x$ and $y$ directions.
Compare your results with the expressions from a)
for  a  temperature $T=1.0$ (in units of $kT/J$).

How many Monte Carlo cycles do you need in order to achieve a
good agreeement?

### Project 4c): When is the most likely state reached?

We choose now a square lattice with $L=20$ spins in the $x$ and $y$ directions.

In the previous exercise  we did not study carefully how many Monte Carlo cycles were needed in order to reach the most likely state. Here
we want to perform a study of the time (here it corresponds to the number
of Monte Carlo sweeps of the lattice) one needs before one reaches an equilibrium situation
and can start computing various expectations values. Our
first attempt is a rough and plain graphical
one, where we plot various expectations values as functions of the number of Monte Carlo cycles.

Choose first a temperature of $T=1.0$ (in units of $kT/J$) and study the
mean energy and magnetisation (absolute value) as functions of the number of Monte Carlo cycles. Let the number of Monte Carlo cycles (sweeps per lattice)
represent time.
Use both an ordered (all spins pointing in one direction) and a random
spin orientation as starting configuration.
How many Monte Carlo cycles do you need before you reach an equilibrium situation?
Repeat this analysis for $T=2.4$.
Can you, based on these values estimate an equilibration time?
Make also a plot of the total number of accepted configurations
as function of the total number of Monte Carlo cycles. How does the number of
accepted configurations behave as function of temperature $T$?

### Project 4d): Analyzing the probability distribution

Compute thereafter the probability
$P(E)$ for the previous system with $L=20$ and the same temperatures, that is at $T=1.0$ and $T=2.4$.
You compute this probability by simply counting the number of times a
given energy appears in your computation. Start the computation after
the steady state situation has been reached.
Compare your results with the computed variance in energy
$\sigma^2_E$ and discuss the behavior you observe.

### Studies of phase transitions

Near $T_C$ we can characterize the behavior of many physical quantities
by a power law behavior.
As an example, for the Ising class of models,
the mean magnetization is given by

$$
\langle M(T) \rangle \sim \left(T-T_C\right)^{\beta},
$$

where $\beta=1/8$ is a so-called critical exponent. A similar relation
applies to the heat capacity

$$
C_V(T) \sim \left|T_C-T\right|^{\alpha},
$$

and the susceptibility

$$
\begin{equation}
  \chi(T) \sim \left|T_C-T\right|^{\gamma},
\label{_auto1} \tag{1}
\end{equation}
$$

with $\alpha = 0$ and $\gamma = 7/4$.
Another important quantity is the correlation length, which is expected
to be of the order of the lattice spacing for $T>> T_C$. Because the spins
become more and more correlated as $T$ approaches $T_C$, the correlation
length increases as we get closer to the critical temperature. The divergent
behavior of $\xi$ near $T_C$
is

$$
\begin{equation}
  \xi(T) \sim \left|T_C-T\right|^{-\nu}.
\label{eq:xi} \tag{2}
\end{equation}
$$

A second-order phase transition is characterized by a
correlation length which spans the whole system.
Since we are always limited to a finite lattice, $\xi$ will
be proportional with the size of the lattice.
Through so-called finite size scaling relations
it is possible to relate the behavior at finite lattices with the
results for an infinitely large lattice.
The critical temperature scales then as

$$
\begin{equation}
 T_C(L)-T_C(L=\infty) = aL^{-1/\nu},
\label{eq:tc} \tag{3}
\end{equation}
$$

with  $a$ a constant and  $\nu$ defined in Eq. ([2](#eq:xi)).
We set $T=T_C$ and obtain a mean magnetisation

$$
\begin{equation}
  \langle {\cal M}(T) \rangle \sim \left(T-T_C\right)^{\beta}
  \rightarrow L^{-\beta/\nu},
\label{eq:scale1} \tag{4}
\end{equation}
$$

a heat capacity

$$
\begin{equation}
  C_V(T) \sim \left|T_C-T\right|^{-\gamma} \rightarrow L^{\alpha/\nu},
\label{eq:scale2} \tag{5}
\end{equation}
$$

and susceptibility

$$
\begin{equation}
  \chi(T) \sim \left|T_C-T\right|^{-\alpha} \rightarrow L^{\gamma/\nu}.
\label{eq:scale3} \tag{6}
\end{equation}
$$

### Project 4e): Numerical studies of phase transitions

We wish to study the behavior of the Ising model
in two dimensions close to the critical temperature as a function of
the lattice size $L\times L$. Calculate the expectation values for
$\langle E\rangle$ and $\langle \vert M\vert \rangle$, the specific heat
$C_V$ and the susceptibility $\chi$ as functions of $T$ for $L=40$,
$L=60$, $L=80$ and $L=100$ for $T\in [2.0,2.3]$ with a step in
temperature $\Delta T=0.05$ or smaller.  You may find it convenient to narrow the domain for $T$.

Plot $\langle E\rangle$,
$\langle \vert M\vert\rangle$, $C_V$ and $\chi$ as functions of $T$. Can
you see an indication of a phase transition?  Use the absolute value
$\langle \vert M\vert\rangle$ when you evaluate $\chi$.  For these production runs you should
parallelize the code using MPI (recommended). Alternatively OpenMP can be used. Use optimization flags when compiling. Perform a timing analysis of some selected runs in order to see that you get an optimal speedup when parallelizing your code.

### Project 4f): Extracting the critical temperature

Use Eq. ([3](#eq:tc)) and the exact result
$\nu=1$ in order to estimate $T_C$ in the thermodynamic limit
$L\rightarrow \infty$
using your simulations with $L=40$, $L=60$, $L=80$ and $L=100$
The exact result for the critical temperature ([after Lars Onsager](http://journals.aps.org/pr/abstract/10.1103/PhysRev.65.117)) is
$kT_C/J=2/ln(1+\sqrt{2})\approx 2.269$ with $\nu=1$.

## Background literature

If you wish to read more about the Ising model and statistical physics here are three suggestions.

  * [M. Plischke and B. Bergersen](http://www.worldscientific.com/worldscibooks/10.1142/5660), *Equilibrium Statistical Physics*, World Scientific, see chapters 5 and 6.

  * [D. P. Landau and K. Binder](http://www.cambridge.org/no/academic/subjects/physics/computational-science-and-modelling/guide-monte-carlo-simulations-statistical-physics-4th-edition?format=HB), *A Guide to Monte Carlo Simulations in Statistical Physics*, Cambridge, see chapters 2,3 and 4.

  * [M. E. J. Newman and T. Barkema](https://global.oup.com/academic/product/monte-carlo-methods-in-statistical-physics-9780198517979?cc=no&lang=en&), *Monte Carlo Methods in Statistical Physics*, Oxford, see chapters 3 and 4.

## Introduction to numerical projects

Here follows a brief recipe and recommendation on how to write a report for each
project.

  * Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  * Include the source code of your program. Comment your program properly.

  * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  * Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  * Try to give an interpretation of you results in your answers to  the problems.

  * Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  * Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.

## Format for electronic delivery of report and programs

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

  * Use Devilry to hand in your projects, log in  at  <http://devilry.ifi.uio.no> with your normal UiO username and password and choose either 'fys3150' or 'fys4150'. There you can load up the files within the deadline.

  * Upload **only** the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  Do not include library files which are available at the course homepage, unless you have made specific changes to them.

  * In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parametxers.

  * In this and all later projects, you should include tests (for example unit tests) of your code(s).

  * Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your Devilry domain and are only visible to you and the teachers of the course.

Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students.

## How to install openmpi and/or OpenMP on your PC/laptop

If you use your own laptop, for linux/ubuntu users, you need to install two packages (alternatively use the synaptic package manager)

          sudo apt-get install libopenmpi-dev
          sudo apt-get install openmpi-bin

For OS X users, install brew (after having installed xcode and gcc, needed for the
gfortran compiler of openmpi) and then run

        brew install open-mpi

When compiling from the command line, depending on your choice of programming language you need to compile and link as for example

        mpic++ -O3 -o <executable> <programname.cpp>

if you use c++ (alternatively mpicxx) and

        mpif90 -O3 -o <executable> <programname.f90>

if you use Fortran90.

When running an executable, run as

        mpirun -n 10 ./<executable>

where -n indicates the number of processes, 10 here.

With openmpi installed, when using Qt, add to your .pro file the instructions at [Svenn-Arne Dragly's site](http://dragly.org/2012/03/14/developing-mpi-applications-in-qt-creator/)

You may need to tell Qt where opempi is stored.

For the machines at the computer lab, openmpi is located  at /usr/lib64/openmpi/bin
Add to your .bashrc file the following

        export PATH=/usr/lib64/openmpi/bin:$PATH

For Windows users we recommend to follow the instructions at the [Open MPI site](https://www.open-mpi.org/software/ompi/v1.6/ms-windows.php).

If you use OpenMP, for linux users, compile and link with for example

        c++ -O3 -fopenmp -o <executable>  <programname.cpp>

For OS X users, you need to install libomp using brew, that is

        brew install clang-omp

and then compile and link with for example

        c++ -O3 -lomp -o <executable>  <programname.cpp>

If you program in Fortran and use **gfortran**, compile as for example

        gfortran -O3 -fopenmp -o <executable>  <programname.f90>

If you have access to Intel's **ifort** compiler, compile as

        ifort -O3 -fopenmp -o <executable>  <programname.f90>


#  CS-109A Introduction to Data Science

## Lab 11: Neural Network Basics - Introduction to `tf.keras`

**Harvard University**

**Fall 2019**

**Instructors:** Pavlos Protopapas, Kevin Rader, Chris Tanner

**Lab Instructors:** Chris Tanner and Eleni Kaxiras.

**Authors:** Eleni Kaxiras, David Sondak, and Pavlos Protopapas.

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import pandas as pd
%matplotlib inline
from PIL import Image
```

```python
from __future__ import absolute_import, division, print_function, unicode_literals

# TensorFlow and tf.keras
import tensorflow as tf

tf.keras.backend.clear_session()  # For easy reset of notebook state.

print(tf.__version__)  # You should see a 2.0.0 here!
```

    2.0.0

#### Instructions for running `tf.keras` with Tensorflow 2.0:

1. Create a `conda` virtual environment by cloning an existing one that you know works
```
conda create --name myclone --clone myenv
```

2. Go to [https://www.tensorflow.org/install/pip](https://www.tensorflow.org/install/pip) and follow instructions for your machine.

3. In a nutshell:
```
pip install --upgrade pip
pip install tensorflow==2.0.0
```

All references to Keras should be written as `tf.keras`.  For example:

```
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

tf.keras.models.Sequential
tf.keras.layers.Dense, tf.keras.layers.Activation,
tf.keras.layers.Dropout, tf.keras.layers.Flatten, tf.keras.layers.Reshape
tf.keras.optimizers.SGD
tf.keras.preprocessing.image.ImageDataGenerator
tf.keras.regularizers
tf.keras.datasets.mnist
```

You could avoid the long names by using
```
from tensorflow import keras
from tensorflow.keras import layers
```
These imports do not work on some systems, however, because they pick up previous versions of `keras` and `tensorflow`. That is why I avoid them in this lab.

## Learning Goals
In this lab we will understand the basics of neural networks and how to start using a deep learning library called `keras`. By the end of this lab, you should:

- Understand how a simple neural network works and code some of its functionality from scratch.
- Be able to think and do calculations in matrix notation. Also think of vectors and arrays as tensors.
- Know how to install and run `tf.keras`.
- Implement a simple real world example using a neural network.

## Part 1: Neural Networks 101

Suppose we have an input vector $X=${$x_1, x_2, ... x_L$} to a $k$-layered network.

Each layer has its own number of nodes. For the first layer in our drawing that number is $J$. We can store the weights for each node in a vector $\mathbf{W} \in \mathbb{R}^{JxL+1}$ (accounting for bias).  Similarly, we can store the biases from each node in a vector $\mathbf{b} \in \mathbb{R}^{I}$.  The affine transformation is then written as $$\mathbf{a} = \mathbf{W^T}X + \mathbf{b}$$
  What we then do is "absorb" $\mathbf{b}$ into $X$ by adding a column of ones to $X$. Our $X$ matrix than becomes $\mathbf{X} \in \mathbb{R}^{JxL+1}$ and our equation:

$$\mathbf{a} = \mathbf{W^T}_{plusones}X$$
We have that $\mathbf{a} \in \mathbb{R}^{J}$ as well.  Next we evaluate the output from each node.  We write $$\mathbf{u} = \sigma\left(\mathbf{a}\right)$$ where $\mathbf{u}\in\mathbb{R}^{J}$.  We can think of $\sigma$ operating on each individual element of $\mathbf{a}$ separately or in matrix notation. If we denote each component of $\mathbf{a}$ by $a_{j}$ then we can write $$u_{j} = \sigma\left(a_{j}\right), \quad j = 1, ... J.$$
 In our code we will implement all these equations in matrix notation.
`tf.keras` (Tensorflow) and `numpy` perform the calculations in matrix format.

Image source: *"Modern Mathematical Methods for Computational Science and Engineering"* Efthimios Kaxiras and Athanassios Fokas.

Let's assume that we have 3 input points (L = 3), two hidden layers ($k=2$), and 2 nodes in each layer ($J=2$)

### Input Layer

$𝑋$={$𝑥_1,𝑥_2,x_3$}

### First Hidden Layer

\begin{equation}
 \begin{aligned}
a^{(1)}_1 = w^{(1)}_{10} + w^{(1)}_{11}x_1 + w^{(1)}_{12}x_2 + w^{(1)}_{13}x_3 \\
a^{(1)}_2 = w^{(1)}_{20} + w^{(1)}_{21}x_1 + w^{(1)}_{22}x_2 + w^{(1)}_{23}x_3 \\
\end{aligned}
\end{equation}

 All this in matrix notation: $$\mathbf{a} = \mathbf{W^T}X$$

 NOTE: in $X$ we have added a column of ones to account for the bias

**Then the sigmoid is applied**:
\begin{equation}
 \begin{aligned}
u^{(1)}_1 = \sigma(a^{(1)}_1) \\
u^{(1)}_2 = \sigma(a^{(1)}_2) \\
\end{aligned}
\end{equation}

or in matrix notation: $$\mathbf{u} = \sigma\left(\mathbf{a}\right)$$

### Second Hidden Layer

\begin{equation}
 \begin{aligned}
a^{(2)}_1 = w^{(2)}_{10} + w^{(2)}_{11}u^{(1)}_1 + w^{(2)}_{12}u^{(1)}_2 + w^{(2)}_{13}u^{(1)}_3 \\
a^{(2)}_2 = w^{(2)}_{20} + w^{(2)}_{21}u^{(1)}_1 + w^{(2)}_{22}u^{(1)}_2 + w^{(2)}_{23}u^{(1)}_3 \\
\end{aligned}
\end{equation}

**Then the sigmoid is applied**:

\begin{equation}
 \begin{aligned}
u^{(2)}_1 = \sigma(a^{(2)}_1) \\
u^{(2)}_2 = \sigma(a^{(2)}_2) \\
\end{aligned}
\end{equation}

### Output Layer

#### If the output is categorical:

For example with four classes ($M=4$): $Y$={$y_1, y_2, y_3, y_4$}, we have the affine and then the sigmoid is lastly applied:

\begin{equation}
 \begin{aligned}
a^{(3)}_1 = w^{(3)}_{10} + w^{(3)}_{11}u^{(2)}_1 + w^{(3)}_{12}u^{(2)}_2 \\
a^{(3)}_2 = w^{(3)}_{20} + w^{(3)}_{21}u^{(2)}_1 + w^{(3)}_{22}u^{(2)}_2 \\
a^{(3)}_3 = w^{(3)}_{30} + w^{(3)}_{31}u^{(2)}_1 + w^{(3)}_{32}u^{(2)}_2 \\
a^{(3)}_4 = w^{(3)}_{40} + w^{(3)}_{41}u^{(2)}_1 + w^{(3)}_{42}u^{(2)}_2 \\
\end{aligned}
\end{equation}

\begin{equation}
 \begin{aligned}
y_1 = \sigma(a^{(3)}_1) \\
y_2 = \sigma(a^{(3)}_2) \\
y_3 = \sigma(a^{(3)}_3) \\
y_3 = \sigma(a^{(3)}_4) \\
\end{aligned}
\end{equation}
$\sigma$ will be softmax in the case of multiple classes and sigmoid for binary.

#### If the output is a number (regression):

We have a single y as output:

\begin{equation}
 \begin{aligned}
y = w^{(3)}_{10}+ w^{(3)}_{11}u^{(2)}_1 + w^{(3)}_{12}u^{(2)}_2 + w^{(3)}_{13}u^{(2)}_3 \\
\end{aligned}
\end{equation}

#### Matrix Multiplication and constant addition

```python
a = np.array([[1, 0], [0, 1], [2, 3]])
b = np.array([[4, 1, 1], [2, 2, 1]])
print(np.matrix(a))
print('------')
print(np.matrix(b))
```

    [[1 0]
     [0 1]
     [2 3]]
    ------
    [[4 1 1]
     [2 2 1]]

```python
# both Tensorflow and numpy take care of transposing.
c = tf.matmul(a, b) # the tensorflow way
print(c)
d = np.dot(a, b) # the numpy way
print(d)
```

    tf.Tensor(
    [[ 4  1  1]
     [ 2  2  1]
     [14  8  5]], shape=(3, 3), dtype=int64)
    [[ 4  1  1]
     [ 2  2  1]
     [14  8  5]]

```python
# how do we add the constant in the matrix
a = [[1, 0], [0, 1]]
ones = np.ones((len(a),1))
a = np.append(a, ones, axis=1)
a
```

    array([[1., 0., 1.],
           [0., 1., 1.]])

1. In class exercise : Plot the sigmoid

Define the `sigmoid` and the `tanh`. For `tanh` you may use `np.tanh` and for the `sigmoid` use the general equation:
\begin{align}
\sigma = \dfrac{1}{1+e^{-2(x-c)/a}} \qquad\text{(1.1)}
\textrm{}
\end{align}

Generate a list of 500 $x$ points from -5 to 5 and plot both functions. What do you observe? What do variables $c$ and $a$ do?

```python
# your code here
```

```python
# %load solutions/sigmoid.py

# The smaller the `a`, the sharper the function is.
# Variable `c` moves the function along the x axis
def sigmoid(x,c,a):
    z = ((x-c)/a)
    return 1.0 / (1.0 + np.exp(-z))

x = np.linspace(-5.0, 5.0, 500) # input points
c = 1.
a = 0.5
plt.plot(x, sigmoid(x, c, a), label='sigmoid')
plt.plot(x, np.tanh(x), label='tanh')
plt.grid();
plt.legend();
```

2. In class exercise: Approximate a Gaussian function using a node and manually adjusting the weights. Start with one layer with one node and move to two nodes.

The task is to approximate (learn) a function $f\left(x\right)$ given some input $x$.  For demonstration purposes, the function we will try to learn is a Gaussian function:
\begin{align}
f\left(x\right) = e^{-x^{2}}
\textrm{}
\end{align}

Even though we represent the input $x$ as a vector on the computer, you should think of it as a single input.

#### 2.1 Start by plotting the above function using the $x$ dataset you created earlier

```python
x = np.linspace(-5.0, 5.0, 500) # input points
def gaussian(x):
    return np.exp(-x*x)

f = gaussian(x)
plt.plot(x, f, label='gaussian')
plt.legend()
```

```python
f.shape
```

    (500,)

#### 2.2 Now, let's code the single node as per the image above.

Write a function named `affine` that does the transformation. The definition is provided below. Then create a simpler sigmoid with just one variable. We choose a **sigmoid** activation function and specifically the **logistic** function.  Sigmoids are a family of functions and the logistic function is just one member in that family. $$\sigma\left(z\right) = \dfrac{1}{1 + e^{-z}}.$$

Define both functions in code.

```python
def affine(x, w, b):
    """Return affine transformation of x

    INPUTS
    ======
    x: A numpy array of points in x
    w: An array representing the weight of the perceptron
    b: An array representing the biases of the perceptron

    RETURN
    ======
    z: A numpy array of points after the affine transformation
       z = wx + b
    """

    # Code goes here
    return z
```

```python
# your code here

```

```python
# %load solutions/affine-sigmoid.py
def affine(x, w, b):
    return w * x + b

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))
```

And now we plot the activation function and the true function. What do you think will happen if you change $w$ and $b$?

```python
w = [-5.0, 0.1, 5.0] # Create a list of weights
b = [0.0, -1.0, 1.0] # Create a list of biases

fig, ax = plt.subplots(1,1, figsize=(9,5))
SIZE = 16

# plot our true function, the gaussian
ax.plot(x, f, lw=4, ls='-.', label='True function')

# plot 3 "networks"
for wi, bi in zip(w, b):
    h = sigmoid(affine(x, wi, bi))
    ax.plot(x, h, lw=4, label=r'$w = {0}$, $b = {1}$'.format(wi,bi))

ax.set_title('Single neuron network', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

We didn't do an exhaustive search of the weights and biases, but it sure looks like this single perceptron is never going to match the actual function.  Again, we shouldn't be suprised about this.  The output layer of the network is simple the logistic function, which can only have so much flexibility.

Let's try to make our network more flexible by using **more nodes**!

### Multiple Perceptrons in a Single Layer
It appears that a single neuron is somewhat limited in what it can accomplish.  What if we expand the number of nodes/neurons in our network?  We have two obvious choices here.  One option is to add depth to the network by putting layers next to each other.  The other option is to stack neurons on top of each other in the same layer.  Now the network has some width, but is still only one layer deep.

```python
x = np.linspace(-5.0, 5.0, 500) # input points
f = np.exp(-x*x) # data

w = np.array([3.5, -3.5])
b = np.array([3.5, 3.5])

# Affine transformations
z1 = w[0] * x + b[0]
z2 = w[1] * x + b[1]

# Node outputs
h1 = sigmoid(z1)
h2 = sigmoid(z2)
```

Now let's plot things and see what they look like.

```python
fig, ax = plt.subplots(1,1, figsize=(9,5))

ax.plot(x, f, lw=4, ls = '-.', label='True function')
ax.plot(x, h1, lw=4, label='First neuron')
ax.plot(x, h2, lw=4, label='Second neuron')

# Set title
ax.set_title('Comparison of Neuron Outputs', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

Just as we expected.  Some sigmoids.  Of course, to get the network prediction we must combine these two sigmoid curves somehow.  First we'll just add $h_{1}$ and $h_{2}$ without any weights to see what happens.

#### Note
We are **not** doing classification here.  We are trying to predict an actual function.  The sigmoid activation is convenient when doing classification because you need to go from $0$ to $1$.  However, when learning a function, we don't have as good of a reason to choose a sigmoid.

```python
# Network output
wout = np.ones(2) # Set the output weights to unity to begin
bout = -1 # bias
yout = wout[0] * h1 + wout[1] * h2 + bout
```

And plot.

```python
fig, ax = plt.subplots(1,1, figsize=(9,5))

ax.plot(x, f, ls='-.', lw=4, label=r'True function')
ax.plot(x, yout, lw=4, label=r'$y_{out} = h_{1} + h_{2}$')

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

Very cool!  The two nodes interact with each other to produce a pretty complicated-looking function.  It still doesn't match the true function, but now we have some hope.  In fact, it's starting to look a little bit like a Gaussian!

We can do better.  There are three obvious options at this point:
1. Change the number of nodes
2. Change the activation functions
3. Change the weights

#### We will leave this simple example for some other time! Let's move on to fashion items!

## Part 2: Tensors, Fashion, and Reese Witherspoon

We can think of tensors as multidimensional arrays of real numerical values; their job is to generalize matrices to multiple dimensions. While tensors first emerged in the 20th century, they have since been applied to numerous other disciplines, including machine learning. Tensor decomposition/factorization can solve, among other, problems in unsupervised learning settings, temporal and multirelational data. For those of you that will get to handle images for Convolutional Neural Networks, it's a good idea to have the understanding of tensors of rank 3.

We will use the following naming conventions:

- scalar = just a number = rank 0 tensor  ($a$ ∈ $F$,)

- vector = 1D array = rank 1 tensor ( $x = (\;x_1,...,x_i\;)⊤$ ∈ $F^n$ )

- matrix = 2D array = rank 2 tensor ( $\textbf{X} = [a_{ij}] ∈ F^{m×n}$ )

- 3D array = rank 3 tensor ( $\mathscr{X} =[t_{i,j,k}]∈F^{m×n×l}$ )

- $\mathscr{N}$D array = rank $\mathscr{N}$ tensor ( $\mathscr{T} =[t_{i1},...,t_{i\mathscr{N}}]∈F^{n_1×...×n_\mathscr{N}}$ ) <-- Things start to get complicated here...

#### Tensor indexing
We can create subarrays by fixing some of the given tensor’s indices. We can create a vector by fixing all but one index. A 2D matrix is created when fixing all but two indices. For example, for a third order tensor the vectors are

$\mathscr{X}[:,j,k]$ = $\mathscr{X}[j,k]$ (column),

$\mathscr{X}[i,:,k]$ = $\mathscr{X}[i,k]$ (row), and

$\mathscr{X}[i,j,:]$ = $\mathscr{X}[i,j]$ (tube)

#### Tensor multiplication
We can multiply one matrix with another as long as the sizes are compatible ((n × m) × (m × p) = n × p), and also multiply an entire matrix by a constant. Numpy `numpy.dot` performs a matrix multiplication which is straightforward when we have 2D or 1D arrays. But what about > 3D arrays? The function will choose according to the matching dimentions but if we want to choose we should use `tensordot`, but, again, we **do not need tensordot** for this class.

### Reese Witherspoon

This image is from the dataset [Labeled Faces in the Wild](http://vis-www.cs.umass.edu/lfw/person/Reese_Witherspoon.html) used for machine learning training. Images are 24-bit RGB images (height, width, channels) with 8 bits for each of R, G, B channel. Explore and print the array.

```python
# load and show the image
FILE = '../fig/Reese_Witherspoon.jpg'
img = mpimg.imread(FILE)
imgplot = plt.imshow(img)
```

```python
print(f'The image is a: {type(img)} of shape {img.shape}')
img[3:5, 3:5, :]
```

#### Slicing tensors: slice along each axis

```python
# we want to show each color channel
fig, axes = plt.subplots(1, 3, figsize=(10,10))
for i, subplot in zip(range(3), axes):
    temp = np.zeros(img.shape, dtype='uint8')
    temp[:,:,i] = img[:,:,i]
    subplot.imshow(temp)
    subplot.set_axis_off()
plt.show()
```

#### Multiplying Images with a scalar (just for fun, does not really help us in any way)

```python
temp = img
temp = temp * 2
plt.imshow(temp)
```

For more on image manipulation by `matplotlib` see: [matplotlib-images](https://matplotlib.org/3.1.1/tutorials/introductory/images.html)

### Anatomy of an Artificial Neural Network

In Part 1 we hand-made a neural network by writing some simple python functions.  We focused on a regression problem where we tried to learn a function. We practiced using the logistic activation function in a network with multiple nodes, but a single or two hidden layers.  Some of the key observations were:
* Increasing the number of nodes allows us to represent more complicated functions
* The weights and biases have a very big impact on the solution
* Finding the "correct" weights and biases is really hard to do manually
* There must be a better method for determining the weights and biases automatically

We also didn't assess the effects of different activation functions or different network depths.

###
https://www.tensorflow.org/guide/keras

`tf.keras` is TensorFlow's high-level API for building and training deep learning models. It's used for fast prototyping, state-of-the-art research, and production. `Keras` is a library created by François Chollet. After Google released Tensorflow 2.0, the creators of `keras` recommend that "Keras users who use multi-backend Keras with the TensorFlow backend switch to `tf.keras` in TensorFlow 2.0. `tf.keras` is better maintained and has better integration with TensorFlow features".

#### IMPORTANT:  In `Keras` everything starts with a Tensor of N samples as input and ends with a Tensor of N samples as output.

### The 3 parts of an ANN

- **Part 1: the input layer** (our dataset)
- **Part 2: the internal architecture or hidden layers** (the number of layers, the activation functions, the learnable parameters and other hyperparameters)
- **Part 3: the output layer** (what we want from the network)

In the rest of the lab we will practice with end-to-end neural network training

1. Load the data
2. Define the layers of the model.
3. Compile the model.
4. Fit the model to the train set (also using a validation set).
5. Evaluate the model on the test set.
6. Plot metrics such as accuracy.
7. Predict on random images from test set.
8. Predict on  a random image from the web!

```python
seed = 7
np.random.seed(seed)
```

### Fashion MNIST

MNIST, the set of handwritten digits is considered the Drosophila of Machine Learning. It has been overused, though, so we will try a slight modification to it.

**Fashion-MNIST** is a dataset of clothing article images (created by [Zalando](https://github.com/zalandoresearch/fashion-mnist)), consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a **28 x 28** grayscale image, associated with a label from **10 classes**. The creators intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. Each pixel is 8 bits so its value ranges from 0 to 255.

Let's load and look at it!

#### 1. Load the data

```python
%%time
# get the data from keras
fashion_mnist = tf.keras.datasets.fashion_mnist

# load the data splitted in train and test! how nice!
(x_train, y_train),(x_test, y_test) = fashion_mnist.load_data()

# normalize the data by dividing with pixel intensity
# (each pixel is 8 bits so its value ranges from 0 to 255)
x_train, x_test = x_train / 255.0, x_test / 255.0

# classes are named 0-9 so define names for plotting clarity
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

# plot
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(x_train[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[y_train[i]])
plt.show()
```

```python
plt.imshow(x_train[3], cmap=plt.cm.binary)
```

```python
x_train.shape, x_test.shape
```

    ((60000, 28, 28), (10000, 28, 28))

```python
y_train.shape
```

    (60000,)

#### 2. Define the layers of the model.

```python
# type together
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(154, activation='relu'),
  tf.keras.layers.Dense(64, activation='relu'),
  #tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])
```

#### 3. Compile the model

```python
loss_fn =
optimizer =

model.compile(optimizer=optimizer,
              loss=loss_fn,
             )
```

```python
model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    flatten (Flatten)            (None, 784)               0
    _________________________________________________________________
    dense (Dense)                (None, 154)               120890
    _________________________________________________________________
    dense_1 (Dense)              (None, 64)                9920
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                650
    =================================================================
    Total params: 131,460
    Trainable params: 131,460
    Non-trainable params: 0
    _________________________________________________________________

```python
tf.keras.utils.plot_model(
    model,
    #to_file='model.png', # if you want to save the image
    show_shapes=True, # True for more details than you need
    show_layer_names=True,
    rankdir='TB',
    expand_nested=False,
    dpi=96
)
```

[Everything you wanted to know about a Keras Model and were afraid to ask](https://www.tensorflow.org/api_docs/python/tf/keras/Model)

#### 4. Fit the model to the train set (also using a validation set)

This is the part that takes the longest.

-----------------------------------------------------------
**ep·och**

noun: epoch; plural noun: epochs. A period of time in history or a person's life, typically one marked by notable events or particular characteristics. Examples: "the Victorian epoch", "my Neural Netwok's epochs".

-----------------------------------------------------------

```python
%%time

# the core of the network training
history = model.fit(x_train, y_train, validation_split=0.33, epochs=50,
                    verbose=2)
```

#### Save the model

You can save the model so you do not have `.fit` everytime you reset the kernel in the notebook. Network training is expensive!

For more details on this see [https://www.tensorflow.org/guide/keras/save_and_serialize](https://www.tensorflow.org/guide/keras/save_and_serialize)

```python
# save the model so you do not have to run the code everytime
model.save('fashion_model.h5')

# Recreate the exact same model purely from the file
#model = tf.keras.models.load_model('fashion_model.h5')
```

#### 5. Evaluate the model on the test set.

```python
test_loss, test_accuracy = model.evaluate(x_test, y_test,  verbose=0)
print(f'Test accuracy={test_accuracy}')
```

#### 6. We learn a lot by studying History! Plot metrics such as accuracy.

You can learn a lot about neural networks by observing how they perform while training. You can issue `kallbacks` in `keras`. The networks's performance is stored in a `keras` callback aptly named `history` which can be plotted.

```python
print(history.history.keys())
```

```python
# plot accuracy and loss for the test set
fig, ax = plt.subplots(1,2, figsize=(20,6))

ax[0].plot(history.history['accuracy'])
ax[0].plot(history.history['val_accuracy'])
ax[0].set_title('Model accuracy')
ax[0].set_ylabel('accuracy')
ax[0].set_xlabel('epoch')
ax[0].legend(['train', 'val'], loc='best')

ax[1].plot(history.history['loss'])
ax[1].plot(history.history['val_loss'])
ax[1].set_title('Model loss')
ax[1].set_ylabel('loss')
ax[1].set_xlabel('epoch')
ax[1].legend(['train', 'val'], loc='best')
```

#### 7. Now let's use the Network for what it was meant to do: Predict!

```python
predictions = model.predict(x_test)
```

```python
predictions[0]
```

```python
np.argmax(predictions[0]), class_names[np.argmax(predictions[0])]
```

Let's see if our network predicted right! Is the first item what was predicted?

```python
plt.figure()
plt.imshow(x_test[0], cmap=plt.cm.binary)
plt.xlabel(class_names[y_test[0]])
plt.colorbar()
```

**Correct!!** Now let's see how confident our model is by plotting the probability values:

```python
# code source: https://www.tensorflow.org/tutorials/keras/classification
def plot_image(i, predictions_array, true_label, img):
    predictions_array, true_label, img = predictions_array, true_label[i], img[i]
    plt.grid(False)
    plt.xticks([])
    plt.yticks([])

    plt.imshow(img, cmap=plt.cm.binary)

    predicted_label = np.argmax(predictions_array)
    if predicted_label == true_label:
        color = 'blue'
    else:
        color = 'red'

    plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

def plot_value_array(i, predictions_array, true_label):
    predictions_array, true_label = predictions_array, true_label[i]
    plt.grid(False)
    plt.xticks(range(10))
    plt.yticks([])
    thisplot = plt.bar(range(10), predictions_array, color="#777777")
    plt.ylim([0, 1])
    predicted_label = np.argmax(predictions_array)

    thisplot[predicted_label].set_color('red')
    thisplot[true_label].set_color('blue')
```

```python
i = 406
plt.figure(figsize=(6,3))
plt.subplot(1,2,1)
plot_image(i, predictions[i], y_test, x_test)
plt.subplot(1,2,2)
plot_value_array(i, predictions[i],  y_test)
plt.show()
```

#### 8. Predicting in the real world

Let's see if our network can generalize beyond the MNIST fashion dataset. Let's give it an random googled image of a boot. Does it have to be a clothing item resembling the MNIST fashion dataset? Can it be a puppy?

Download an image from the internet and resize it to 28x28.

`tf.keras` models are optimized to make predictions on a batch, or collection, of examples at once. Accordingly, even though you're using a single image, you need to add it to a list: