```python
!pip install qiskit
```

    Collecting qiskit
      Downloading qiskit-0.34.2.tar.gz (13 kB)
    Collecting qiskit-terra==0.19.2
      Downloading qiskit_terra-0.19.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)
         |████████████████████████████████| 6.5 MB 8.1 MB/s
    Collecting qiskit-aer==0.10.3
      Downloading qiskit_aer-0.10.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (18.0 MB)
         |████████████████████████████████| 18.0 MB 89 kB/s
    Collecting qiskit-ibmq-provider==0.18.3
      Downloading qiskit_ibmq_provider-0.18.3-py3-none-any.whl (238 kB)
         |████████████████████████████████| 238 kB 69.1 MB/s
    Collecting qiskit-ignis==0.7.0
      Downloading qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
         |████████████████████████████████| 200 kB 50.5 MB/s
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.21.5)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.4.1)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.3.1-py3-none-any.whl (54 kB)
         |████████████████████████████████| 54 kB 2.6 MB/s
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.8.2)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.23.0)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.24.3)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.11.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
         |████████████████████████████████| 1.6 MB 31.7 MB/s
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.7.1)
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (5.4.8)
    Collecting stevedore>=3.0.0
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.1 MB/s
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 5.6 MB/s
    Collecting symengine>=0.8
      Downloading symengine-0.9.2-cp37-cp37m-manylinux2010_x86_64.whl (37.5 MB)
         |████████████████████████████████| 37.5 MB 1.1 MB/s
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
         |████████████████████████████████| 943 kB 41.9 MB/s
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (0.3.4)
    Collecting scipy>=1.0
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
         |████████████████████████████████| 38.1 MB 1.1 MB/s
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (3.0.4)
    Collecting cryptography>=1.3
      Downloading cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
         |████████████████████████████████| 3.6 MB 42.9 MB/s
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (2.21)
    Collecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.8.1-py2.py3-none-any.whl (113 kB)
         |████████████████████████████████| 113 kB 54.3 MB/s
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (4.11.2)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.10.0.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.7.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.19.2->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.34.2-py3-none-any.whl size=11805 sha256=64ff390fec8fba196230a7e0d3b04a8434eeddf6e12f50a45340dc7cc5079ce5
      Stored in directory: /root/.cache/pip/wheels/62/77/65/cda6eedfdd2a525bd3f479a4386930ae3088a1eb01f8c944ed
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=f71e40f2e8fdb99dba2daf46a25efb63e5c37f77ce0094e5ecd0704d8bd844af
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, scipy, retworkx, python-constraint, ply, ntlm-auth, cryptography, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed cryptography-36.0.2 ntlm-auth-1.5.0 pbr-5.8.1 ply-3.11 python-constraint-1.4.0 qiskit-0.34.2 qiskit-aer-0.10.3 qiskit-ibmq-provider-0.18.3 qiskit-ignis-0.7.0 qiskit-terra-0.19.2 requests-ntlm-1.1.0 retworkx-0.11.0 scipy-1.7.3 stevedore-3.5.0 symengine-0.9.2 tweedledum-1.1.1 websocket-client-1.3.1

```python
from qiskit import QuantumCircuit, assemble, Aer
```

```python
qc = QuantumCircuit(2)
qc.cx(0,1)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f1bb01b8960>

```python
from qiskit.visualization import plot_histogram
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
!pip install pylatexenc

```

    Requirement already satisfied: pylatexenc in /usr/local/lib/python3.7/dist-packages (2.10)

```python
qc.draw()
```

q_0: ──■──
     ┌─┴─┐
q_1: ┤ X ├
     └───┘

# **ENCODING THE INPUT**

```python
qc = QuantumCircuit(4,2)
qc.x(0) ##Flips the 1st qubit to 1
qc.x(1) ##Flips the 2nd qubit to 1, as we want to perform '1+1'
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f1ba63d73c0>

# **Using CNOT to perform XOR operations on 2nd qubit**

```python
qc.cx(0,2)
qc.cx(1,2)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f1ba631de10>

# **Extracting the outputs**

```python
## Extracting XOR values of 2nd qubit
qc.measure(2,0)
qc.measure(3,1)
qc.draw()
```

     ┌───┐
q_0: ┤ X ├──■──────────
     ├───┤  │
q_1: ┤ X ├──┼────■─────
     └───┘┌─┴─┐┌─┴─┐┌─┐
q_2: ─────┤ X ├┤ X ├┤M├
      ┌─┐ └───┘└───┘└╥┘
q_3: ─┤M├────────────╫─
      └╥┘            ║
c: 2/══╩═════════════╩═
       1             0

# **Adding ccx Tofolli gate**

```python
## use ccx to write the AND of inputs at qubit 3
qc.ccx(0,1,3)
qc.measure(3,1) ## Extract the value of AND operation from qubit 3
qc.draw()
```

     ┌───┐
q_0: ┤ X ├──■────────────■─────
     ├───┤  │            │
q_1: ┤ X ├──┼────■───────■─────
     └───┘┌─┴─┐┌─┴─┐┌─┐  │
q_2: ─────┤ X ├┤ X ├┤M├──┼─────
      ┌─┐ └───┘└───┘└╥┘┌─┴─┐┌─┐
q_3: ─┤M├────────────╫─┤ X ├┤M├
      └╥┘            ║ └───┘└╥┘
c: 2/══╩═════════════╩═══════╩═
       1             0       1

# **Running the circuit on Quantum Simulator**

```python
from qiskit import execute
counts = execute(qc, Aer.get_backend('aer_simulator')).result().get_counts()
plot_histogram(counts)
```

Therefore, the outpu comes out to be **1 0** , which is the additive result of '1+1'

```python

```


#  CS-109A Introduction to Data Science

## Lab 11: Neural Network Basics - Introduction to `tf.keras`

**Harvard University**

**Fall 2019**

**Instructors:** Pavlos Protopapas, Kevin Rader, Chris Tanner

**Lab Instructors:** Chris Tanner and Eleni Kaxiras.

**Authors:** Eleni Kaxiras, David Sondak, and Pavlos Protopapas.

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import pandas as pd
%matplotlib inline

from PIL import Image
```

```python
from __future__ import absolute_import, division, print_function, unicode_literals

# TensorFlow and tf.keras
import tensorflow as tf

tf.keras.backend.clear_session()  # For easy reset of notebook state.

print(tf.__version__)  # You should see a 2.0.0 here!
```

    2.0.0

#### Instructions for running `tf.keras` with Tensorflow 2.0:

1. Create a `conda` virtual environment by cloning an existing one that you know works
```
conda create --name myclone --clone myenv
```

2. Go to [https://www.tensorflow.org/install/pip](https://www.tensorflow.org/install/pip) and follow instructions for your machine.

3. In a nutshell:
```
pip install --upgrade pip
pip install tensorflow==2.0.0
```

All references to Keras should be written as `tf.keras`.  For example:

```
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

tf.keras.models.Sequential
tf.keras.layers.Dense, tf.keras.layers.Activation,
tf.keras.layers.Dropout, tf.keras.layers.Flatten, tf.keras.layers.Reshape
tf.keras.optimizers.SGD
tf.keras.preprocessing.image.ImageDataGenerator
tf.keras.regularizers
tf.keras.datasets.mnist
```

You could avoid the long names by using
```
from tensorflow import keras
from tensorflow.keras import layers
```
These imports do not work on some systems, however, because they pick up previous versions of `keras` and `tensorflow`. That is why I avoid them in this lab.

## Learning Goals
In this lab we will understand the basics of neural networks and how to start using a deep learning library called `keras`. By the end of this lab, you should:

- Understand how a simple neural network works and code some of its functionality from scratch.
- Be able to think and do calculations in matrix notation. Also think of vectors and arrays as tensors.
- Know how to install and run `tf.keras`.
- Implement a simple real world example using a neural network.

## Part 1: Neural Networks 101

Suppose we have an input vector $X=${$x_1, x_2, ... x_L$} to a $k$-layered network.

Each layer has its own number of nodes. For the first layer in our drawing that number is $J$. We can store the weights for each node in a vector $\mathbf{W} \in \mathbb{R}^{JxL+1}$ (accounting for bias).  Similarly, we can store the biases from each node in a vector $\mathbf{b} \in \mathbb{R}^{I}$.  The affine transformation is then written as $$\mathbf{a} = \mathbf{W^T}X + \mathbf{b}$$
  What we then do is "absorb" $\mathbf{b}$ into $X$ by adding a column of ones to $X$. Our $X$ matrix than becomes $\mathbf{X} \in \mathbb{R}^{JxL+1}$ and our equation:

$$\mathbf{a} = \mathbf{W^T}_{plusones}X$$
We have that $\mathbf{a} \in \mathbb{R}^{J}$ as well.  Next we evaluate the output from each node.  We write $$\mathbf{u} = \sigma\left(\mathbf{a}\right)$$ where $\mathbf{u}\in\mathbb{R}^{J}$.  We can think of $\sigma$ operating on each individual element of $\mathbf{a}$ separately or in matrix notation. If we denote each component of $\mathbf{a}$ by $a_{j}$ then we can write $$u_{j} = \sigma\left(a_{j}\right), \quad j = 1, ... J.$$
 In our code we will implement all these equations in matrix notation.
`tf.keras` (Tensorflow) and `numpy` perform the calculations in matrix format.

Image source: *"Modern Mathematical Methods for Computational Science and Engineering"* Efthimios Kaxiras and Athanassios Fokas.

Let's assume that we have 3 input points (L = 3), two hidden layers ($k=2$), and 2 nodes in each layer ($J=2$)

### Input Layer

$𝑋$={$𝑥_1,𝑥_2,x_3$}

### First Hidden Layer

\begin{equation}
 \begin{aligned}
a^{(1)}_1 = w^{(1)}_{10} + w^{(1)}_{11}x_1 + w^{(1)}_{12}x_2 + w^{(1)}_{13}x_3 \\
a^{(1)}_2 = w^{(1)}_{20} + w^{(1)}_{21}x_1 + w^{(1)}_{22}x_2 + w^{(1)}_{23}x_3 \\
\end{aligned}
\end{equation}

 All this in matrix notation: $$\mathbf{a} = \mathbf{W^T}X$$

 NOTE: in $X$ we have added a column of ones to account for the bias

**Then the sigmoid is applied**:
\begin{equation}
 \begin{aligned}
u^{(1)}_1 = \sigma(a^{(1)}_1) \\
u^{(1)}_2 = \sigma(a^{(1)}_2) \\
\end{aligned}
\end{equation}

or in matrix notation: $$\mathbf{u} = \sigma\left(\mathbf{a}\right)$$

### Second Hidden Layer

\begin{equation}
 \begin{aligned}
a^{(2)}_1 = w^{(2)}_{10} + w^{(2)}_{11}u^{(1)}_1 + w^{(2)}_{12}u^{(1)}_2 + w^{(2)}_{13}u^{(1)}_3 \\
a^{(2)}_2 = w^{(2)}_{20} + w^{(2)}_{21}u^{(1)}_1 + w^{(2)}_{22}u^{(1)}_2 + w^{(2)}_{23}u^{(1)}_3 \\
\end{aligned}
\end{equation}

**Then the sigmoid is applied**:

\begin{equation}
 \begin{aligned}
u^{(2)}_1 = \sigma(a^{(2)}_1) \\
u^{(2)}_2 = \sigma(a^{(2)}_2) \\
\end{aligned}
\end{equation}

### Output Layer

#### If the output is categorical:

For example with four classes ($M=4$): $Y$={$y_1, y_2, y_3, y_4$}, we have the affine and then the sigmoid is lastly applied:

\begin{equation}
 \begin{aligned}
a^{(3)}_1 = w^{(3)}_{10} + w^{(3)}_{11}u^{(2)}_1 + w^{(3)}_{12}u^{(2)}_2 \\
a^{(3)}_2 = w^{(3)}_{20} + w^{(3)}_{21}u^{(2)}_1 + w^{(3)}_{22}u^{(2)}_2 \\
a^{(3)}_3 = w^{(3)}_{30} + w^{(3)}_{31}u^{(2)}_1 + w^{(3)}_{32}u^{(2)}_2 \\
a^{(3)}_4 = w^{(3)}_{40} + w^{(3)}_{41}u^{(2)}_1 + w^{(3)}_{42}u^{(2)}_2 \\
\end{aligned}
\end{equation}

\begin{equation}
 \begin{aligned}
y_1 = \sigma(a^{(3)}_1) \\
y_2 = \sigma(a^{(3)}_2) \\
y_3 = \sigma(a^{(3)}_3) \\
y_3 = \sigma(a^{(3)}_4) \\
\end{aligned}
\end{equation}
$\sigma$ will be softmax in the case of multiple classes and sigmoid for binary.

#### If the output is a number (regression):

We have a single y as output:

\begin{equation}
 \begin{aligned}
y = w^{(3)}_{10}+ w^{(3)}_{11}u^{(2)}_1 + w^{(3)}_{12}u^{(2)}_2 + w^{(3)}_{13}u^{(2)}_3 \\
\end{aligned}
\end{equation}

#### Matrix Multiplication and constant addition

```python
a = np.array([[1, 0], [0, 1], [2, 3]])
b = np.array([[4, 1, 1], [2, 2, 1]])
print(np.matrix(a))
print('------')
print(np.matrix(b))
```

    [[1 0]
     [0 1]
     [2 3]]
    ------
    [[4 1 1]
     [2 2 1]]

```python
# both Tensorflow and numpy take care of transposing.
c = tf.matmul(a, b) # the tensorflow way
print(c)
d = np.dot(a, b) # the numpy way
print(d)
```

    tf.Tensor(
    [[ 4  1  1]
     [ 2  2  1]
     [14  8  5]], shape=(3, 3), dtype=int64)
    [[ 4  1  1]
     [ 2  2  1]
     [14  8  5]]

```python
# how do we add the constant in the matrix
a = [[1, 0], [0, 1]]
ones = np.ones((len(a),1))
a = np.append(a, ones, axis=1)
a
```

    array([[1., 0., 1.],
           [0., 1., 1.]])

1. In class exercise : Plot the sigmoid

Define the `sigmoid` and the `tanh`. For `tanh` you may use `np.tanh` and for the `sigmoid` use the general equation:
\begin{align}
\sigma = \dfrac{1}{1+e^{-2(x-c)/a}} \qquad\text{(1.1)}
\textrm{}
\end{align}

Generate a list of 500 $x$ points from -5 to 5 and plot both functions. What do you observe? What do variables $c$ and $a$ do?

```python
# your code here
```

```python
# %load solutions/sigmoid.py

# The smaller the `a`, the sharper the function is.
# Variable `c` moves the function along the x axis
def sigmoid(x,c,a):
    z = ((x-c)/a)
    return 1.0 / (1.0 + np.exp(-z))

x = np.linspace(-5.0, 5.0, 500) # input points
c = 1.
a = 0.5
plt.plot(x, sigmoid(x, c, a), label='sigmoid')
plt.plot(x, np.tanh(x), label='tanh')
plt.grid();
plt.legend();
```

2. In class exercise: Approximate a Gaussian function using a node and manually adjusting the weights. Start with one layer with one node and move to two nodes.

The task is to approximate (learn) a function $f\left(x\right)$ given some input $x$.  For demonstration purposes, the function we will try to learn is a Gaussian function:
\begin{align}
f\left(x\right) = e^{-x^{2}}
\textrm{}
\end{align}

Even though we represent the input $x$ as a vector on the computer, you should think of it as a single input.

#### 2.1 Start by plotting the above function using the $x$ dataset you created earlier

```python
x = np.linspace(-5.0, 5.0, 500) # input points
def gaussian(x):
    return np.exp(-x*x)

f = gaussian(x)
plt.plot(x, f, label='gaussian')
plt.legend()
```

```python
f.shape
```

    (500,)

#### 2.2 Now, let's code the single node as per the image above.

Write a function named `affine` that does the transformation. The definition is provided below. Then create a simpler sigmoid with just one variable. We choose a **sigmoid** activation function and specifically the **logistic** function.  Sigmoids are a family of functions and the logistic function is just one member in that family. $$\sigma\left(z\right) = \dfrac{1}{1 + e^{-z}}.$$

Define both functions in code.

```python
def affine(x, w, b):
    """Return affine transformation of x

    INPUTS
    ======
    x: A numpy array of points in x
    w: An array representing the weight of the perceptron
    b: An array representing the biases of the perceptron

    RETURN
    ======
    z: A numpy array of points after the affine transformation
       z = wx + b
    """

    # Code goes here
    return z
```

```python
# your code here

```

```python
# %load solutions/affine-sigmoid.py
def affine(x, w, b):
    return w * x + b

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))
```

And now we plot the activation function and the true function. What do you think will happen if you change $w$ and $b$?

```python
w = [-5.0, 0.1, 5.0] # Create a list of weights
b = [0.0, -1.0, 1.0] # Create a list of biases

fig, ax = plt.subplots(1,1, figsize=(9,5))
SIZE = 16

# plot our true function, the gaussian
ax.plot(x, f, lw=4, ls='-.', label='True function')

# plot 3 "networks"
for wi, bi in zip(w, b):
    h = sigmoid(affine(x, wi, bi))
    ax.plot(x, h, lw=4, label=r'$w = {0}$, $b = {1}$'.format(wi,bi))

ax.set_title('Single neuron network', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

We didn't do an exhaustive search of the weights and biases, but it sure looks like this single perceptron is never going to match the actual function.  Again, we shouldn't be suprised about this.  The output layer of the network is simple the logistic function, which can only have so much flexibility.

Let's try to make our network more flexible by using **more nodes**!

### Multiple Perceptrons in a Single Layer
It appears that a single neuron is somewhat limited in what it can accomplish.  What if we expand the number of nodes/neurons in our network?  We have two obvious choices here.  One option is to add depth to the network by putting layers next to each other.  The other option is to stack neurons on top of each other in the same layer.  Now the network has some width, but is still only one layer deep.

```python
x = np.linspace(-5.0, 5.0, 500) # input points
f = np.exp(-x*x) # data

w = np.array([3.5, -3.5])
b = np.array([3.5, 3.5])

# Affine transformations
z1 = w[0] * x + b[0]
z2 = w[1] * x + b[1]

# Node outputs
h1 = sigmoid(z1)
h2 = sigmoid(z2)
```

Now let's plot things and see what they look like.

```python
fig, ax = plt.subplots(1,1, figsize=(9,5))

ax.plot(x, f, lw=4, ls = '-.', label='True function')
ax.plot(x, h1, lw=4, label='First neuron')
ax.plot(x, h2, lw=4, label='Second neuron')

# Set title
ax.set_title('Comparison of Neuron Outputs', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

Just as we expected.  Some sigmoids.  Of course, to get the network prediction we must combine these two sigmoid curves somehow.  First we'll just add $h_{1}$ and $h_{2}$ without any weights to see what happens.

#### Note
We are **not** doing classification here.  We are trying to predict an actual function.  The sigmoid activation is convenient when doing classification because you need to go from $0$ to $1$.  However, when learning a function, we don't have as good of a reason to choose a sigmoid.

```python
# Network output
wout = np.ones(2) # Set the output weights to unity to begin
bout = -1 # bias
yout = wout[0] * h1 + wout[1] * h2 + bout
```

And plot.

```python
fig, ax = plt.subplots(1,1, figsize=(9,5))

ax.plot(x, f, ls='-.', lw=4, label=r'True function')
ax.plot(x, yout, lw=4, label=r'$y_{out} = h_{1} + h_{2}$')

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

Very cool!  The two nodes interact with each other to produce a pretty complicated-looking function.  It still doesn't match the true function, but now we have some hope.  In fact, it's starting to look a little bit like a Gaussian!

We can do better.  There are three obvious options at this point:
1. Change the number of nodes
2. Change the activation functions
3. Change the weights

#### We will leave this simple example for some other time! Let's move on to fashion items!

## Part 2: Tensors, Fashion, and Reese Witherspoon

We can think of tensors as multidimensional arrays of real numerical values; their job is to generalize matrices to multiple dimensions. While tensors first emerged in the 20th century, they have since been applied to numerous other disciplines, including machine learning. Tensor decomposition/factorization can solve, among other, problems in unsupervised learning settings, temporal and multirelational data. For those of you that will get to handle images for Convolutional Neural Networks, it's a good idea to have the understanding of tensors of rank 3.

We will use the following naming conventions:

- scalar = just a number = rank 0 tensor  ($a$ ∈ $F$,)

- vector = 1D array = rank 1 tensor ( $x = (\;x_1,...,x_i\;)⊤$ ∈ $F^n$ )

- matrix = 2D array = rank 2 tensor ( $\textbf{X} = [a_{ij}] ∈ F^{m×n}$ )

- 3D array = rank 3 tensor ( $\mathscr{X} =[t_{i,j,k}]∈F^{m×n×l}$ )

- $\mathscr{N}$D array = rank $\mathscr{N}$ tensor ( $\mathscr{T} =[t_{i1},...,t_{i\mathscr{N}}]∈F^{n_1×...×n_\mathscr{N}}$ ) <-- Things start to get complicated here...

#### Tensor indexing
We can create subarrays by fixing some of the given tensor’s indices. We can create a vector by fixing all but one index. A 2D matrix is created when fixing all but two indices. For example, for a third order tensor the vectors are

$\mathscr{X}[:,j,k]$ = $\mathscr{X}[j,k]$ (column),

$\mathscr{X}[i,:,k]$ = $\mathscr{X}[i,k]$ (row), and

$\mathscr{X}[i,j,:]$ = $\mathscr{X}[i,j]$ (tube)

#### Tensor multiplication
We can multiply one matrix with another as long as the sizes are compatible ((n × m) × (m × p) = n × p), and also multiply an entire matrix by a constant. Numpy `numpy.dot` performs a matrix multiplication which is straightforward when we have 2D or 1D arrays. But what about > 3D arrays? The function will choose according to the matching dimentions but if we want to choose we should use `tensordot`, but, again, we **do not need tensordot** for this class.

### Reese Witherspoon

This image is from the dataset [Labeled Faces in the Wild](http://vis-www.cs.umass.edu/lfw/person/Reese_Witherspoon.html) used for machine learning training. Images are 24-bit RGB images (height, width, channels) with 8 bits for each of R, G, B channel. Explore and print the array.

```python
# load and show the image
FILE = '../fig/Reese_Witherspoon.jpg'
img = mpimg.imread(FILE)
imgplot = plt.imshow(img)
```

```python
print(f'The image is a: {type(img)} of shape {img.shape}')
img[3:5, 3:5, :]
```

    The image is a: <class 'numpy.ndarray'> of shape (150, 150, 3)

    array([[[241, 241, 241],
            [242, 242, 242]],

           [[241, 241, 241],
            [242, 242, 242]]], dtype=uint8)

#### Slicing tensors: slice along each axis

```python
# we want to show each color channel
fig, axes = plt.subplots(1, 3, figsize=(10,10))
for i, subplot in zip(range(3), axes):
    temp = np.zeros(img.shape, dtype='uint8')
    temp[:,:,i] = img[:,:,i]
    subplot.imshow(temp)
    subplot.set_axis_off()
plt.show()
```

#### Multiplying Images with a scalar (just for fun, does not really help us in any way)

```python
temp = img
temp = temp * 2
plt.imshow(temp)
```

For more on image manipulation by `matplotlib` see: [matplotlib-images](https://matplotlib.org/3.1.1/tutorials/introductory/images.html)

### Anatomy of an Artificial Neural Network

In Part 1 we hand-made a neural network by writing some simple python functions.  We focused on a regression problem where we tried to learn a function. We practiced using the logistic activation function in a network with multiple nodes, but a single or two hidden layers.  Some of the key observations were:
* Increasing the number of nodes allows us to represent more complicated functions
* The weights and biases have a very big impact on the solution
* Finding the "correct" weights and biases is really hard to do manually
* There must be a better method for determining the weights and biases automatically

We also didn't assess the effects of different activation functions or different network depths.

###
https://www.tensorflow.org/guide/keras

`tf.keras` is TensorFlow's high-level API for building and training deep learning models. It's used for fast prototyping, state-of-the-art research, and production. `Keras` is a library created by François Chollet. After Google released Tensorflow 2.0, the creators of `keras` recommend that "Keras users who use multi-backend Keras with the TensorFlow backend switch to `tf.keras` in TensorFlow 2.0. `tf.keras` is better maintained and has better integration with TensorFlow features".

#### IMPORTANT:  In `Keras` everything starts with a Tensor of N samples as input and ends with a Tensor of N samples as output.

### The 3 parts of an ANN

- **Part 1: the input layer** (our dataset)
- **Part 2: the internal architecture or hidden layers** (the number of layers, the activation functions, the learnable parameters and other hyperparameters)
- **Part 3: the output layer** (what we want from the network)

In the rest of the lab we will practice with end-to-end neural network training

1. Load the data
2. Define the layers of the model.
3. Compile the model.
4. Fit the model to the train set (also using a validation set).
5. Evaluate the model on the test set.
6. Plot metrics such as accuracy.
7. Predict on random images from test set.
8. Predict on  a random image from the web!

```python
seed = 7
np.random.seed(seed)
```

### Fashion MNIST

MNIST, the set of handwritten digits is considered the Drosophila of Machine Learning. It has been overused, though, so we will try a slight modification to it.

**Fashion-MNIST** is a dataset of clothing article images (created by [Zalando](https://github.com/zalandoresearch/fashion-mnist)), consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a **28 x 28** grayscale image, associated with a label from **10 classes**. The creators intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. Each pixel is 8 bits so its value ranges from 0 to 255.

Let's load and look at it!

#### 1. Load the data

```python
x_train.shape
```

    (60000, 28, 28)

```python
%%time
# get the data from keras
fashion_mnist = tf.keras.datasets.fashion_mnist

# load the data splitted in train and test! how nice!
(x_train, y_train),(x_test, y_test) = fashion_mnist.load_data()

# normalize the data by dividing with pixel intensity
# (each pixel is 8 bits so its value ranges from 0 to 255)
x_train, x_test = x_train / 255.0, x_test / 255.0

# classes are named 0-9 so define names for plotting clarity
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

# plot
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(x_train[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[y_train[i]])
plt.show()
```

```python
plt.imshow(x_train[3], cmap=plt.cm.binary)
```

```python
x_train.shape, x_test.shape
```

    ((60000, 28, 28), (10000, 28, 28))

```python
y_train.shape
```

    (60000,)

#### 2. Define the layers of the model.

```python
# type together
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(154, activation='relu'),
  tf.keras.layers.Dense(64, activation='relu'),
  #tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])
```

#### 3. Compile the model

```python
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.Adam()

model.compile(optimizer=optimizer,
              loss=loss_fn,
              metrics=['accuracy'])
```

```python
model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    flatten (Flatten)            (None, 784)               0
    _________________________________________________________________
    dense (Dense)                (None, 154)               120890
    _________________________________________________________________
    dense_1 (Dense)              (None, 64)                9920
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                650
    =================================================================
    Total params: 131,460
    Trainable params: 131,460
    Non-trainable params: 0
    _________________________________________________________________

```python
tf.keras.utils.plot_model(
    model,
    #to_file='model.png', # if you want to save the image
    show_shapes=True, # True for more details than you need
    show_layer_names=True,
    rankdir='TB',
    expand_nested=False,
    dpi=96
)
```

[Everything you wanted to know about a Keras Model and were afraid to ask](https://www.tensorflow.org/api_docs/python/tf/keras/Model)

#### 4. Fit the model to the train set (also using a validation set)

This is the part that takes the longest.

-----------------------------------------------------------
**ep·och**

noun: epoch; plural noun: epochs. A period of time in history or a person's life, typically one marked by notable events or particular characteristics. Examples: "the Victorian epoch", "my Neural Netwok's epochs".

-----------------------------------------------------------

```python
%%time

# the core of the network training
history = model.fit(x_train, y_train, validation_split=0.33, epochs=50,
                    verbose=2)
```

    Train on 40199 samples, validate on 19801 samples
    Epoch 1/50
    40199/40199 - 3s - loss: 0.5265 - accuracy: 0.8121 - val_loss: 0.4014 - val_accuracy: 0.8543
    Epoch 2/50
    40199/40199 - 3s - loss: 0.3875 - accuracy: 0.8587 - val_loss: 0.3988 - val_accuracy: 0.8521
    Epoch 3/50
    40199/40199 - 3s - loss: 0.3502 - accuracy: 0.8711 - val_loss: 0.3711 - val_accuracy: 0.8645
    Epoch 4/50
    40199/40199 - 3s - loss: 0.3199 - accuracy: 0.8818 - val_loss: 0.3884 - val_accuracy: 0.8573
    Epoch 5/50
    40199/40199 - 3s - loss: 0.3066 - accuracy: 0.8870 - val_loss: 0.3463 - val_accuracy: 0.8763
    Epoch 6/50
    40199/40199 - 3s - loss: 0.2861 - accuracy: 0.8931 - val_loss: 0.3719 - val_accuracy: 0.8655
    Epoch 7/50
    40199/40199 - 3s - loss: 0.2775 - accuracy: 0.8964 - val_loss: 0.3487 - val_accuracy: 0.8752
    Epoch 8/50
    40199/40199 - 3s - loss: 0.2641 - accuracy: 0.9008 - val_loss: 0.3437 - val_accuracy: 0.8779
    Epoch 9/50
    40199/40199 - 3s - loss: 0.2528 - accuracy: 0.9043 - val_loss: 0.3454 - val_accuracy: 0.8809
    Epoch 10/50
    40199/40199 - 3s - loss: 0.2430 - accuracy: 0.9093 - val_loss: 0.3273 - val_accuracy: 0.8857
    Epoch 11/50
    40199/40199 - 3s - loss: 0.2335 - accuracy: 0.9127 - val_loss: 0.3207 - val_accuracy: 0.8866
    Epoch 12/50
    40199/40199 - 3s - loss: 0.2259 - accuracy: 0.9141 - val_loss: 0.3352 - val_accuracy: 0.8876
    Epoch 13/50
    40199/40199 - 3s - loss: 0.2189 - accuracy: 0.9175 - val_loss: 0.3281 - val_accuracy: 0.8915
    Epoch 14/50
    40199/40199 - 3s - loss: 0.2128 - accuracy: 0.9184 - val_loss: 0.3394 - val_accuracy: 0.8850
    Epoch 15/50
    40199/40199 - 3s - loss: 0.2038 - accuracy: 0.9215 - val_loss: 0.3257 - val_accuracy: 0.8934
    Epoch 16/50
    40199/40199 - 3s - loss: 0.1982 - accuracy: 0.9249 - val_loss: 0.3501 - val_accuracy: 0.8895
    Epoch 17/50
    40199/40199 - 3s - loss: 0.1912 - accuracy: 0.9277 - val_loss: 0.3758 - val_accuracy: 0.8849
    Epoch 18/50
    40199/40199 - 3s - loss: 0.1828 - accuracy: 0.9295 - val_loss: 0.3625 - val_accuracy: 0.8889
    Epoch 19/50
    40199/40199 - 3s - loss: 0.1817 - accuracy: 0.9315 - val_loss: 0.3517 - val_accuracy: 0.8911
    Epoch 20/50
    40199/40199 - 3s - loss: 0.1747 - accuracy: 0.9332 - val_loss: 0.3463 - val_accuracy: 0.8929
    Epoch 21/50
    40199/40199 - 3s - loss: 0.1707 - accuracy: 0.9349 - val_loss: 0.3691 - val_accuracy: 0.8892
    Epoch 22/50
    40199/40199 - 3s - loss: 0.1651 - accuracy: 0.9372 - val_loss: 0.3567 - val_accuracy: 0.8932
    Epoch 23/50
    40199/40199 - 3s - loss: 0.1611 - accuracy: 0.9374 - val_loss: 0.3737 - val_accuracy: 0.8914
    Epoch 24/50
    40199/40199 - 3s - loss: 0.1582 - accuracy: 0.9397 - val_loss: 0.3833 - val_accuracy: 0.8906
    Epoch 25/50
    40199/40199 - 3s - loss: 0.1518 - accuracy: 0.9416 - val_loss: 0.3730 - val_accuracy: 0.8921
    Epoch 26/50
    40199/40199 - 3s - loss: 0.1485 - accuracy: 0.9429 - val_loss: 0.4085 - val_accuracy: 0.8897
    Epoch 27/50
    40199/40199 - 3s - loss: 0.1415 - accuracy: 0.9462 - val_loss: 0.3862 - val_accuracy: 0.8909
    Epoch 28/50
    40199/40199 - 3s - loss: 0.1433 - accuracy: 0.9451 - val_loss: 0.4204 - val_accuracy: 0.8907
    Epoch 29/50
    40199/40199 - 3s - loss: 0.1374 - accuracy: 0.9483 - val_loss: 0.3937 - val_accuracy: 0.8920
    Epoch 30/50
    40199/40199 - 3s - loss: 0.1330 - accuracy: 0.9498 - val_loss: 0.4073 - val_accuracy: 0.8887
    Epoch 31/50
    40199/40199 - 3s - loss: 0.1339 - accuracy: 0.9493 - val_loss: 0.4131 - val_accuracy: 0.8891
    Epoch 32/50
    40199/40199 - 3s - loss: 0.1288 - accuracy: 0.9514 - val_loss: 0.4089 - val_accuracy: 0.8919
    Epoch 33/50
    40199/40199 - 3s - loss: 0.1239 - accuracy: 0.9526 - val_loss: 0.4195 - val_accuracy: 0.8904
    Epoch 34/50
    40199/40199 - 3s - loss: 0.1228 - accuracy: 0.9529 - val_loss: 0.4848 - val_accuracy: 0.8873
    Epoch 35/50
    40199/40199 - 3s - loss: 0.1183 - accuracy: 0.9547 - val_loss: 0.4405 - val_accuracy: 0.8924
    Epoch 36/50
    40199/40199 - 3s - loss: 0.1176 - accuracy: 0.9555 - val_loss: 0.4518 - val_accuracy: 0.8895
    Epoch 37/50
    40199/40199 - 3s - loss: 0.1174 - accuracy: 0.9552 - val_loss: 0.4882 - val_accuracy: 0.8883
    Epoch 38/50
    40199/40199 - 3s - loss: 0.1101 - accuracy: 0.9581 - val_loss: 0.4646 - val_accuracy: 0.8947
    Epoch 39/50
    40199/40199 - 3s - loss: 0.1050 - accuracy: 0.9599 - val_loss: 0.4875 - val_accuracy: 0.8919
    Epoch 40/50
    40199/40199 - 3s - loss: 0.1089 - accuracy: 0.9599 - val_loss: 0.4824 - val_accuracy: 0.8900
    Epoch 41/50
    40199/40199 - 3s - loss: 0.1047 - accuracy: 0.9583 - val_loss: 0.5133 - val_accuracy: 0.8872
    Epoch 42/50
    40199/40199 - 3s - loss: 0.1056 - accuracy: 0.9598 - val_loss: 0.4922 - val_accuracy: 0.8908
    Epoch 43/50
    40199/40199 - 3s - loss: 0.0965 - accuracy: 0.9635 - val_loss: 0.5100 - val_accuracy: 0.8891
    Epoch 44/50
    40199/40199 - 3s - loss: 0.1012 - accuracy: 0.9617 - val_loss: 0.5292 - val_accuracy: 0.8891
    Epoch 45/50
    40199/40199 - 3s - loss: 0.0968 - accuracy: 0.9631 - val_loss: 0.5449 - val_accuracy: 0.8865
    Epoch 46/50
    40199/40199 - 3s - loss: 0.0926 - accuracy: 0.9642 - val_loss: 0.5455 - val_accuracy: 0.8904
    Epoch 47/50
    40199/40199 - 3s - loss: 0.0945 - accuracy: 0.9634 - val_loss: 0.5249 - val_accuracy: 0.8909
    Epoch 48/50
    40199/40199 - 3s - loss: 0.0915 - accuracy: 0.9649 - val_loss: 0.5712 - val_accuracy: 0.8861
    Epoch 49/50
    40199/40199 - 3s - loss: 0.0887 - accuracy: 0.9670 - val_loss: 0.5586 - val_accuracy: 0.8849
    Epoch 50/50
    40199/40199 - 3s - loss: 0.0873 - accuracy: 0.9670 - val_loss: 0.5836 - val_accuracy: 0.8896
    CPU times: user 3min 50s, sys: 26.9 s, total: 4min 17s
    Wall time: 2min 21s

#### Save the model

You can save the model so you do not have `.fit` everytime you reset the kernel in the notebook. Network training is expensive!

For more details on this see [https://www.tensorflow.org/guide/keras/save_and_serialize](https://www.tensorflow.org/guide/keras/save_and_serialize)

```python
# save the model so you do not have to run the code everytime
model.save('fashion_model.h5')

# Recreate the exact same model purely from the file
#model = tf.keras.models.load_model('fashion_model.h5')
```

#### 5. Evaluate the model on the test set.

```python
test_loss, test_accuracy = model.evaluate(x_test, y_test, verbose=0)
print(f'Test accuracy={test_accuracy}')
```

    Test accuracy=0.8804000020027161

#### 6. We learn a lot by studying History! Plot metrics such as accuracy.

You can learn a lot about neural networks by observing how they perform while training. You can issue `kallbacks` in `keras`. The networks's performance is stored in a `keras` callback aptly named `history` which can be plotted.

```python
print(history.history.keys())
```

    dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

```python
# plot accuracy and loss for the test set
fig, ax = plt.subplots(1,2, figsize=(20,6))

ax[0].plot(history.history['accuracy'])
ax[0].plot(history.history['val_accuracy'])
ax[0].set_title('Model accuracy')
ax[0].set_ylabel('accuracy')
ax[0].set_xlabel('epoch')
ax[0].legend(['train', 'val'], loc='best')

ax[1].plot(history.history['loss'])
ax[1].plot(history.history['val_loss'])
ax[1].set_title('Model loss')
ax[1].set_ylabel('loss')
ax[1].set_xlabel('epoch')
ax[1].legend(['train', 'val'], loc='best')
```

#### 7. Now let's use the Network for what it was meant to do: Predict!

```python
predictions = model.predict(x_test)
```

```python
predictions[0]
```

    array([2.3347583e-26, 5.1650537e-17, 7.8453709e-22, 4.0654924e-26,
           6.6475328e-18, 1.3495652e-14, 1.9801722e-15, 1.6511808e-06,
           1.7612138e-24, 9.9999833e-01], dtype=float32)

```python
np.argmax(predictions[0]), class_names[np.argmax(predictions[0])]
```

    (9, 'Ankle boot')

Let's see if our network predicted right! Is the first item what was predicted?

```python
plt.figure()
plt.imshow(x_test[0], cmap=plt.cm.binary)
plt.xlabel(class_names[y_test[0]])
plt.colorbar()
```

**Correct!!** Now let's see how confident our model is by plotting the probability values:

```python
# code source: https://www.tensorflow.org/tutorials/keras/classification
def plot_image(i, predictions_array, true_label, img):
    predictions_array, true_label, img = predictions_array, true_label[i], img[i]
    plt.grid(False)
    plt.xticks([])
    plt.yticks([])

    plt.imshow(img, cmap=plt.cm.binary)

    predicted_label = np.argmax(predictions_array)
    if predicted_label == true_label:
        color = 'blue'
    else:
        color = 'red'

    plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

def plot_value_array(i, predictions_array, true_label):
    predictions_array, true_label = predictions_array, true_label[i]
    plt.grid(False)
    plt.xticks(range(10))
    plt.yticks([])
    thisplot = plt.bar(range(10), predictions_array, color="#777777")
    plt.ylim([0, 1])
    predicted_label = np.argmax(predictions_array)

    thisplot[predicted_label].set_color('red')
    thisplot[true_label].set_color('blue')
```

```python
i = 406
plt.figure(figsize=(6,3))
plt.subplot(1,2,1)
plot_image(i, predictions[i], y_test, x_test)
plt.subplot(1,2,2)
plot_value_array(i, predictions[i],  y_test)
plt.show()
```

#### 8. Predicting in the real world

Let's see if our network can generalize beyond the MNIST fashion dataset. Let's give it an random googled image of a boot. Does it have to be a clothing item resembling the MNIST fashion dataset? Can it be a puppy?

Download an image from the internet and resize it to 28x28.

`tf.keras` models are optimized to make predictions on a batch, or collection, of examples at once. Accordingly, even though you're using a single image, you need to add it to a list:

```python

```


# "Entropy and Complex Systems"
> "In this blog post we will look at entropy in 3 classical settings: Thermodynamics, Information Theory and Statistics. We will show that while under certain conditions they have the same functional form this is not true in general. Moreover we will look at how we can define entropy in relation to complex systems."
- toc: true
- author: Lewis Cole (2020)
- branch: master
- badges: false
- comments: false
- categories: [Entropy, Thermodynamics, Information-Theory, Statistics, Theory, Probability, Complexity, Complex-Systems]
- hide: false
- search_exclude: false
- image: https://github.com/lewiscoleblog/blog/raw/master/images/Entropy/entropy.jpg

In this blog post we shall be looking at the concept of "entropy" which appears under a number of guises. We will look at a number of areas in which entropy arises and look at similiarities and differences. In the interests of brevity this will be a whistle-stop tour of each area and we will not cover all the details. Each area has enough results/theorems/etc. to cover a series of blog posts by themselves. In this blog post we are simply concerned with the concept of entropy.

(Note: This blog post follows the research by Rudolf Hanel and Stefan Thurner and in particular the first half of the 6th chapter of the book: "Inroduction to the Theory of Complex Systems" by Stefan Thurner, Rudolf Hanel and Peter Klimek)

## Entropy Thermodynamics and Statistical Mechanics
When we mention "entropy" what many people first think of is a measure of "disorder" or "uncertainty" of a system. We imagine a system whereby an individual configuration is called a microstate. A microstate can be associated with a global behaviour called a macrostate. For each macrostate if we take (the logarithm of) the number of microstates that lead to the macrostate then we have the "entropy" quantity (Boltzmann entropy). A macrostate that has more generating microstates is considered to be more "disordered". To take a simple example taking water ($H_2 O$) in solid (ice) form the individual molecules are (somewhat) forced into position in the crystaline structure, as such there are relatively few different orientations of molecules and hence a low entropy. As we move into the liquid phase the molecules move around more freely and so there are many more positions which the molecules can take, leading to a higher entropy. In the gaseous phase the molecules are whizzing around with higher energies and even more orientations are possible and so the entropy increases yet further.

The first definition of entropy comes from Rudolf Clausius in the mid 1800s through work in thermodynamics. He asserts that internal energy ($U$), temperature ($T$), pressure ($P$) and volume ($V$) are related via entropy ($S$) as:
$$ dU = T dS - P dV $$
Through this definition we find that entropy is a way of relating the macroscopic behaviours of a system. This holds for reversible processes that are quasi-stationary (essentially: there can be an absorbing state that we reach almost-surely but initial conditions are such that the system will take a long time to reach it). We sometimes re-arrange this formulation slightly to give:
$$\frac{dQ}{T} = dS$$
Where $Q$ is the heat added/removed to/from the system. There are a couple of extra properties we require of entropy, firstly it must be additive. If for example we take 2 identical systems and combine them the entropy must double, in other words:
$$ S_{A+B} = S_A + S_B $$
For systems $A$ and $B$. Also we have that the second law of thermodynamics must hold - that is: for a closed system entropy must never be decreasing:
$$dS \geq 0$$
Which means if we know the macrostate at a given time then the state at a later time must be of equal or higher entropy. Moreover if the closed system is in equilibrium: $dS = 0$.

Another definition of entropy via statistical mechanics is:
$$ S_B = k_B log \Omega $$
Where $k_B$ is a unit-correcting constant and $\Omega$ is the number of microstates consistent with a macrostate. If the microstates are not equally probable we can use the Gibbs-Boltzmann form of entropy:
$$S_B = -k_b \sum_i p_i ln p_i $$
This can be shown to be equivalent to Clausius entropy by invoking the Boltzmann distribution, that is:
$$ p_i = \frac{1}{Z} exp\left( \frac{-E_i}{k_B T} \right) $$
Where $E_i$ is the energy corresponding to the microstate, $T$ is the temperature and $Z$ is the partition function (normalizing constant).

To show the equivalency between Gibbs-Boltzmann and Clausius entropy formulations:
\begin{align}
dS_B &= -k_b \sum_i dp_i (ln p_i)  \\
     &= -k_b \sum_i dp_i \left(\frac{-E_i}{k_B T} - lnZ\right) \\
     &= \sum_i \frac{E_i dp_i}{T} \\
     &= \sum_i \frac{d(E_i p_i) - (dE_i)p_i}{T} \\
     &= \frac{dE - dW}{T} \\
     &= \frac{dQ}{T}
\end{align}
Which is equivalent to above. In the last line we used the first law of thermodynamics: $dE = dW + dQ$ - the change in energy is the work done plus the change in heat. We also used $\sum d(E_i p_i) = dE$ (average energy) and $\sum d(E_i)p_i = dW$ (work done). For this to hold we require that the system is in a thermal equilibrium.

This equivalence is quite an impressive and powerful result, we now have a mechanism by which we can relate the macrostates of a system to the microstates - that is we habe a link between the mircoscopic behaviour (particle behaviour) to macroscopic behaviour (phase of matter). The key assumption however is that the system is in **equilibrium**.

## Entropy in Information Theory
Another area where entropy appears is in information theory. Put very simply information theory is concerned with the problem of encoding messages and how much information is required to encode a message so that it can be decoded again with a certian level of accuracy. If we know what messages we want to send and a noise-rate then we can work out what capacity of channel we would require.

In the context of information theory entropy is the term used to quantify "uncertainty" in a message/signal. High entropy represents a large uncertainty and vice versa. Typically this is taken to be the Shannon entropy:
$$ S_S = - \sum_i p_i ln p_i $$
Which looks somewhat familiar now we have looked at Boltzmann entropy ($S_B$) however we should note these represent fundamentally different things and any resemblance is merely coincidence.

Why is Shannon entropy a good measure in this setting? Let's consider an example where we have $N$ signals with associated probabilites: $(p_1, p_2, ... , P_N)$. If we have a determinstic case: $(p_1=1, p_2=0,...,p_N=0)$ then the Shannon entropy is zero. Noting:
$$\lim_{x \to 0} xlnx = 0$$
and
$$1ln1=0$$
But also we can see that Shannon entropy is maximised for $p_i = \frac{1}{N}$ - uniform distribution. This fits with our notion of "maximum uncertainty". We also have for two systems $(A=(p_1, p_2, ... , P_N), B=(q_1, q_2, ..., Q_M)$ then:
$$S_S(AB) = S_S(A) + S_S(B | A) $$
Where $(B|A)$ represents the distribution of $B$ conditional on $A$ - if $A$ and $B$ are fully independent then $(B|A)=B$ (for example flipping two coins, the probability of heads on one coin will not impact the probability of heads on the other - unless you taped the coins together or something equally silly).

By defining entropy in this way we can derive Shannon's noisy channel theorem (we will not prove this here) in layman's terms this states that: "we can create a coding scheme for a message that allows it to be transmitted (essentially) error free if the information capacity of the channel is greater than the source entropy." We have been a little sloppy in not defining all the terms here but we only need the gist for our purposes. We can see that the entropy is an important concept.

However, we have not applied any restriction to our "source" of the signals/messages. Since we are dealing with uncertainty we typically rely on a probabilistic description. Further for the statement above to hold for $S_S$ we require the information source process to be:
1. Markov
2. Ergodic
The first criteria states that the next signal to be sent depends only on the current signal, not the entire history of sent signals. The second criteria ensures that the process exhibits stationarity (rates for individual signals do not vary in time) and moreover there is no state we can start a trajectory in that makes it impossible to reach any other state. This assumption underlies most of the classic information theory results.

However Shannon also proposed 3 axioms (SK1, SK2, SK4) representing fundamental requirements that an entropy measure $S(p)$ should have for dealing with **Markov-Ergodic** ("simple") information sources. Later Khinchin added a 4th (SK3).

- SK1. - Entropy $S(p)$ must be continuous and depend only on arguments $p_i$ - probability of individual states and nothing else
- SK2. - $S(p)$ must take it's maximum for $p_i = \frac{1}{N}$
- SK3. - Adding an additional state with zero probability does not change the entropy value
- SK4. - The composition of 2 systems should result in: $S(AB) = S(A) + S(B|A)$ - if the systems are fully independent this becomes $S(AB) = S(A) + S(B)$

Using these axioms we find that any functional $S(p)$ satisfying these axioms must be of the form:
$$ S(p) = - k \sum_i p_i ln p_i $$
Which is nothing more than Shannon entropy upto a multiplicative constant.

## Statistical Inference
In statistical inference we are trying to answer the question: what is the best model that captures some data? Or given some model what is the most likely outcome? There are a number of methods we can use to do this, one of which is the "maximum entropy principle" which loosely states that the distribution with the highest entropy given our current data is our "best guess" for the distribution that generated it. This was originally popularized by Jaynes in the late 1950s.

To illustrate the method let's imagine that we have $N$ possible states of a system. We have observations $k=(k_1, k_2, ... , k_N)$ for how often each has occured in our data. The process generating the data has "true" distribution $q=(q_1, q_2, ... , q_N)$ with $\sum_i q_i = 1$. We then have the probability of our observation (assuming independence) following a multinomial distribution:
$$P(k|q) = \frac{N!}{k_1! k_2!...k_N!} q_1^{k_1} q_2^{k_2} ... q_N^{k_N}$$
Which we notice we can factorize as:
$$ P(k|q) = M(k) G(k|q) $$
With:
\begin{align}
M(k) &= \frac{N!}{k_1! k_2!...k_N!} \\
G(q|k) &= q_1^{k_1} q_2^{k_2} ... q_N^{k_N}
\end{align}
We call the function $M(k)$ the multiplicity, and $G(q|k)$ the probability. If we take logarithms and divide by N this formula becomes:
$$ \frac{1}{N} ln P(k|q) = \frac{1}{N} ln M(k) + \frac{1}{N} ln G(q|k) $$
Where the left hand side of the equation is called the relative entropy (or Kullback-Leibler divergence). The right hand term containing $M(k)$ is called the entropy and the final term the cross entropy. Via Stirling's approximation we have:
$$M(k) \approx \frac{N^N}{k_1^{k_1}k_2^{k_2}...k_N^{k_N}}$$
If we denote the sample average:
$$p_i = \frac{k_i}{N} $$
Then via Stirling:
$$ S_J = \frac{1}{N} ln M(k) = - \sum_i p_i ln p_i $$
Which is the (now familiar) entropy funcational. We can also substitute $p_i$ into the formula above, in a similar way we can relate the relative entropy, entropy and cross entropy via:
$$ - \sum_i p_i ln \frac{p_i}{q_i} = - \sum_i p_i ln p_i + \sum_i p_i ln q_i $$
We can make the assumption $q_i = exp^{-\alpha - \beta \epsilon_i}$. And then:
$$ \frac{1}{N} ln P(k|q) = - \sum_i p_i ln p_i - \alpha \sum_i p_i - \beta \sum_i p_i \epsilon_i $$
By differentiating and setting to zero we end up with a set of equations we can solve for $p^*_i$ that is the maximum entropy distribution:
$$ 0 = \frac{\partial}{\partial p_i} ln P(k|q) = \frac{\partial}{\partial p_i}  \left(- \sum_i p_i ln p_i - \alpha \sum_i p_i - \beta \sum_i p_i \epsilon_i  \right)  $$
The constants $\alpha$ and $\beta$ represent Lagrangian multipliers here, $\alpha$ ensures normalization of $p_i$ and $\beta$ ensures the first moment (mean) is correct. If we have additional information (e.g. we know higher order moments) we can build this in via additional Lagrangian multipliers.

We can see that entropy in this sense has the same functional as before. The key assumption however is that the process is **multinomial**.

## Equivalence of Entropy
We have now seen 3 classic examples of where the term entropy is used - the Boltzmann, Shannon and Jaynes entropy functionals. Since the functionals are identical (upto a multiplicative constant) it is natural for us to consider them equivalent. This is a mistake, although they are all called "entropy" and have the same form they represent different things.

It is worth noting that for each energy functional to make sense we make some pretty cavalier assumptions about the underlying systems:
1. Boltzmann entropy assumes a process is in **equilibrium**
2. Shannon entropy assumes an **ergodic** process
3. Jaynes entropy assumes a **multinomial** process

But what happens if we remove these assumptions?

## Entropy and Complexity
Complex systems by their nature do not follow the assumptions above. For example most complex systems are non-ergodic. In particular they may be "evolutionary" systems that show path dependent behaviour which violates ergodicity. They may also display long term "memory" of previous states, which again is non-ergodic. So what can we say in these cases?

Using the Shannon-Khinchin axioms above we can see the most restrictive is $SK4$ - this is a requirement built off the properties of an ergodic system. What if we simply drop this axiom? Thurner-Hanel-et. al. have shown we can do a lot with the following axioms:
- SK1. - Entropy $S(p)$ must be continuous and depend only on arguments $p_i$ - probability of individual states and nothing else
- SK2. - $S(p)$ must take it's maximum for $p_i = \frac{1}{N}$
- SK3. - Adding an additional state with zero probability does not change the entropy value
- $S(p)$ takes a trace form: $S(p) = \sum_i g(p_i)$

We can re-write these axioms further by taking the form: $S(p) = \sum_i g(p_i)$ and noting that:
- $g(.)$ must be continuous (by SK1)
- $g(.)$ must be concave (by SK2)
- $g(0) = 0$ (by SK3)

If we define:
$$ S_g(N) = \sum_i g\left( \frac{1}{N} \right) = N g\left( \frac{1}{N} \right) $$

We can then elicit scaling behaviour:
\begin{align}
\lim_{N \to \infty} \frac{S_g(N \lambda)}{S_g(N)} &= \lambda^{1-c} \\
\lim_{N \to \infty} \frac{S_g(N^{1+a})}{S_g(N)N^{a(1-c)}} &= (1+a)^d
\end{align}
We can then see there are 2 constants that define scaling laws for entropy following this trace form. We have  $0 \leq c \leq 1$ and $d$ real valued. If $c>1$ then SK2 is violated and if $c<0$ SK3 is violated. Moreover it has been shown that an entropy funcitonal can be of the form:
\begin{align}
S_{c,d}(p) &= \frac{r}{c} A^{-d} e^A \left( \sum_i \Gamma(1+d, A - c ln p_i) - rp_i \right) \\
A &= \frac{cdr}{1 - (1-c)r} \\
\Gamma(a,b) &= \int_b^{\infty} t^{a-1} e^{-t} dt
\end{align}
Where $c$ and $d$ are the same as in the scaling properties above. There is an additional constant $r$ taking a range of values dependant on $c$ and $d$ to ensure the expression makes sense, it does not affect the scaling behaviour. We can thus use constants $c$ and $d$ to create equivalence classes of complex systems.

(Note: the expression of $S_{c,d}$ is not unique and simpler expressions exist, however this is a general form that allows for the widest range of $c$ and $d$ values available).

What happens if we take $c=1$ and $d=1$ as an example? Then $A=r$ and via some work (and the fact $\Gamma(2,x) = e^{-x}(x+1)$) we get:
$$ S_{1,1}(p) = (r+1) - re^r - \sum_i p_i ln p_i $$
Which we see as equivalent to the Boltzmann/Shannon/Jaynes entropy above (the additive constant non-withstanding which is a consequence of allowing SK4 to be relaxed - careful selection of $r$ can remove this constant).

So far we have limited ourselves to a trace form function: $S_{c,d}(p) = \sum_i g_{c,d}(p_i)$ - however this is not a requirement. We can use entropy functionals that are not of trace form and still use the same $(c,d)$ criteria for equivalence classes. The table below shows some entropy functionals proposed and their equivalence class:

|Name----------------------|Entropy-Functional-Form-----------------------------------------|------c------|--d--|
|:-------------------------|:---------------------------------------------------------------|:-----------:|:---:|
| Boltzmann  | $-\sum_i p_i ln p_i $         | 1 | 1 |
| Reyni      | $ \frac{1}{1-\alpha}ln \sum_i p_i^{\alpha} $   | 1 | 1 |
| Tsallis    | $ \frac{1 - \sum_i p_i^q}{q-1} $   | $q$ | 0 |
| Abe | $-  \frac{\sum_i p_i^q - p_i^{1/q}}{q - 1/q}$ | $q$ | 0 |
| Sharma-Mittal | $ - \sum_i p_i^r (p_i^{\kappa} - p_i^{-\kappa}) / 2\kappa $ | $r-\kappa$ | 0 |
| Landsber-Vedral      | $\left( \left( \sum_i p_i^q \right)^{-1} - 1 \right) / (q-1)$   | $2-q$ | 0 |
| Exponential | $\sum_i p_i \left( 1 - e^{ \frac{p_i-1}{p_i}} \right) $| 1 | 0 |
| Anteonodo-Plastino | $\sum_i \left( \Gamma \left( \frac{\eta+1}{\eta}, -lnp_i \right) -p_i \Gamma \left( \frac{\eta+1}{\eta} \right) \right)$ | 1 | $\frac{1}{\eta}$ |
| Shafee | $-\sum_i p_i^{\beta} lnp_i$ | $\beta$ | 1 |
| Hanel-Thurner | $\frac{r}{c} A^{-d} e^A \left( \sum_i \Gamma(1+d, A - c ln p_i) - rp_i \right)$ | c | d |

From this we can see that equivalence classes can contain functionals that appear very different from each-other. For example with the right choice of parameter Landsber-Vedral displays the same scaling as  the Sharma-Mittal.

For any trace form of entropy we can denote the generalized logarithm of that entropy as:
$$ \Lambda(p_i) = - \frac{d}{dp_i} g(p_i) $$

We can apply scaling to $g$ to ensure $\Lambda(1) = 0$ and $\Lambda ' (1) = 1$ (this does not change the equivalence class of the entropy functional). This determines the probability distribution function of the system via:
$$p(x) = \Lambda^{-1}(-x) $$

Given:
$$S_{c,d}(p) = \frac{r}{c} A^{-d} e^A \left( \sum_i \Gamma(1+d, A - c ln p_i) - rp_i \right)$$
Then:
$$\Lambda_{c,d,r}(x) = r \left( 1 - x^{c-1} \left(1 - \frac{1-(1-c)r}{dr} ln(x) \right)^d \right) $$
Which we can invert to give:
$$ p_{c,d,r}(x) \sim exp \left( -\frac{d}{1-c} W_k \left( \frac{(1-c)r}{1-(1-c)r} exp \left(\frac{(1-c)r}{1-(1-c)r}\right) \left(1 + \frac{x}{r}\right)^{\frac{1}{d}} \right) \right) $$
Where $W_k$ is the Lambert-W function, which only has real solutions for $k=0$ ($d\geq 0$) or $k=-1$ ($d < 0$).

By taking $c=d=r=1$ we regain the Boltzmann distribution:
$$p_{1,1,1}(x) = e^{-x}$$

Similarly for $c=1$ and we have a stretched exponential distribution:
$$p_{1,d,r}(x) = e^{-dr\left(\left(1+ \frac{x}{r}\right)^{\frac{1}{d}}-1\right)}$$

For $d=0$ and $r=\frac{1}{1-c}$ we hae a power-law:
$$p_{c,0,r} = (1+(1-c)x)^{\frac{-1}{1-c}}$$

In general we find that the distribution functions will look very similar to the power-law case. This demonstrates the uniquity of power-law relations in non-ergodic systems.

## Conclusion
In this blog post we have seen that the 3 "classic" uses of entropy (Boltzmann, Shannon and Jaynes) are in fact distinct despite having the same functional form. This leads to much confusion. Moreover we have seen that the degeneracy in functional form arises out of some pretty strong assumptions on the underlying process namely: **equilibrium**, **Markov-Ergodicity** and **multinomial distributions**. For some systems these assumptions are either true or reasonable approximations of reality, however in some cases these aren't good assumptions and using a Boltzmann entropy functional is asking for trouble.

We also looked at the Shannon-Khinchin axioms for defining a "good" entropy function. We saw that by dropping $SK4$ we were no longer bound to Ergodicity. As such we considered the scaling behaviour of an entropy functional and found 2 key forms of scaling $c$ and $d$, which we could use to create equivlance classes of entropy functionals and non-Ergodic systems.

By further assuming a trace form ($S(p) = \sum_i g(p_i)$) of entropy functional we found a general form for the entropy functional. By taking the generalized logarithm of this function we find that the distribution function for the system must follow a Lambert-W exponential form. For all intents and purposes this leads to power-law behaviour which acts as justification for the ubiquity of power-laws in non-Ergodic systems.

In this blog we talk about "modelling" a lot. The result that non-Ergodic systems follow a distribution that is (at least somewhat like) a power-law suggests that our default modelling assumption for statistical systems should be a power-law type distribution unless we can specifically prove otherwise.

_____
## References
Most of the work in this area (that I know about) is from Stefan Thurner and Rudolf Hanel. Some suggested reading is:
1. Introduction to the Theory of Complex Systems - Thurner, Hamel, Klimek  (This blog post follows the first half of chapter 6 of this book)
2. https://arxiv.org/abs/1310.5959 - Generalized (c,d)-entropy and aging random walks - Hanel, Thurner
3. https://arxiv.org/abs/1211.2257 - Generalized entropies and logarithms and their duality relations - Hanel, Thurner, Gell-Mann
4. https://arxiv.org/abs/1005.0138 - A comprehensive classification of complex statistical systems - Hanel, Thurner
5. https://arxiv.org/pdf/1705.07714.pdf - The three faces of entropy for complex systems - Thurner, Corominas-Murtra, Hanel


```python
import pygame
from pygame.color import THECOLORS
import pymunk
from pymunk.vec2d import Vec2d
from pymunk.pygame_util import draw
```

    Loading chipmunk for Linux (64bit) [/home/rebcabin/anaconda3/envs/self-driving-car/lib/python3.5/site-packages/pymunk/libchipmunk64.so]

```python
width = 1000
height = 700
pygame.init()
screen = pygame.display.set_mode((width, height))
clock = pygame.time.Clock()
screen.set_alpha(None)
```

```python
def clear_screen(color=THECOLORS['black']):
    screen.fill(color)
    pygame.display.flip()
```

```python
def draw_puck(p_0=Vec2d(width/2, height/2), puck_color=THECOLORS['red']):
    dont_fill_bit = 0
    clear_screen()
    r = 100
    pygame.draw.circle(screen, puck_color, p_0.int_tuple, r, dont_fill_bit)
    pygame.display.flip()
draw_puck()
```

```python
def anim_puck(
        steps=100,
        dt=1,
        p_0=Vec2d(width/2, height/2),
        v_0=Vec2d(1, 1/2),
        puck_color=THECOLORS['red']):
    for i in range(steps):
        clear_screen()
        p_i = p_0 + (i * dt * v_0)
        pygame.draw.circle(screen, puck_color, p_i.int_tuple, r, dont_fill_bit)
        pygame.display.flip()
anim_puck()
```

## PREDICT WALL STRIKES

```python
import sympy
```

Function to return a parametric function that produces all the points along a line:

```python
def parametric_line(p0: Vec2d, p1: Vec2d):
    d = (p1 - p0).get_length()
    return lambda t: p0 + t * (p1 - p0) / d
```

```python
import numpy.random as rndm
```

```python
def random_points(n):
    return [Vec2d(p[0]*width, p[1]*height) for p in rndm.random((n, 2))]
```

https://goo.gl/PR24H2
https://goo.gl/x3dEM6

```python
from typing import List, Tuple
```

```python
def convex_hull(points: List[Vec2d]) -> List[Tuple[int, int]]:
    """Computes the convex hull of a set of 2D points.

    Input: an iterable sequence of Vec2ds(x, y) representing the points.
    Output: a list of integer vertices of the convex hull in counter-clockwise
    order, starting from the vertex with the lexicographically smallest
    coordinates. Implements Andrew's monotone chain algorithm.
    O(n log n) complexity in time."""

    # Sort the points lexicographically (tuples are compared lexicographically).
    # Remove duplicates to detect the case we have just one unique point.
    points = sorted(set([p.int_tuple for p in points]))

    # Boring case: no points or a single point, possibly repeated multiple times.
    if len(points) <= 1:
        return points

    # 2D cross product of OA and OB vectors, i.e. z-component of their 3D cross product.
    # Returns a positive value, if OAB makes a counter-clockwise turn,
    # negative for clockwise turn, and zero if the points are collinear.
    def cross(o, a, b):
        return (a[0] - o[0]) * (b[1] - o[1]) - (a[1] - o[1]) * (b[0] - o[0])

    # Build lower hull
    lower = []
    for p in points:
        while len(lower) >= 2 and cross(lower[-2], lower[-1], p) <= 0:
            lower.pop()
        lower.append(p)

    # Build upper hull
    upper = []
    for p in reversed(points):
        while len(upper) >= 2 and cross(upper[-2], upper[-1], p) <= 0:
            upper.pop()
        upper.append(p)

    # Concatenation of the lower and upper hulls gives the convex hull.
    # Last point of each list is omitted because it is repeated at the
    # beginning of the other list.
    return lower[:-1] + upper[:-1]
```

```python
def screen_cage(pad=10):
    return [Vec2d(pad, pad),
            Vec2d(pad, height-pad-1),
            Vec2d(width-pad-1, height-pad-1),
            Vec2d(width-pad-1, pad)]
```

```python
def draw_points(int_tuples: List[Tuple[int, int]],
                color=THECOLORS['yellow']):
    pygame.draw.polygon(screen, color, int_tuples, 1)
    pygame.display.flip()
anim_puck()
# draw_points(convex_hull(random_points(15)))
draw_points([p.int_tuple for p in screen_cage()], THECOLORS['green'])
```

```python
def colinear_point_and_parameter(u: Vec2d, v: Vec2d, p: Vec2d) -> Tuple[Vec2d, float]:
    vmu_squared = (v - u).dot(v - u)
    t = (p - u).dot(v - u) / vmu_squared
    q = u + t * (v - u)
    return (q, t)
def draw_col_pt_param(u=Vec2d(10, height - 10 - 1),
                      v=Vec2d(width - 10 - 1, height - 10 - 1),
                      p=Vec2d(width/2 + 99, height/2 + 99/2),
                      point_color=THECOLORS['black'],
                      line_color=THECOLORS['magenta']):
    q, t = colinear_point_and_parameter(u, v, p)
    pygame.draw.line(screen, point_color, q.int_tuple, q.int_tuple)
    pygame.draw.line(screen, line_color, p.int_tuple, q.int_tuple)
draw_col_pt_param()
```

```python
def predict_hit(line_p_0: Vec2d,
                line_p_1: Vec2d,
                puck_center: Vec2d=Vec2d(width/2, height/2),
                puck_radius: float=100.0,
                puck_vel: Vec2d=Vec2d(1, 1/2),
                dt: float=1.0):
    pass
```

```python

```


# Open Boundary Conditions: A Survey #
An introduction to lateral boundary conditions using a simple shallow water equation model.
Closed, periodic, passive and active boundaries are considered.

```python
''' some basic imports, numpy to do math and the rest for plotting
and import our helper module with "everything but" the boundary conditions
I include def and return lines for those functions here, you can find
the whole functions in swe_module.py
'''

import matplotlib.pyplot as plt
import numpy as np

import swe_module as sw

%matplotlib inline
```

Our model system is a simple flat bottom box with a single layer of fluid.  The shallow water equation system.  We will neglect any explicit viscosity, and the nonlinear advection terms but include the Coriolis Force.  If your research is generally too small to feel the Coriolis force, set your box small and it will have a negligible impact.  We will force the fluid with a Gaussian addition (rain or runoff) in the centre of the domain.

So the governing equations are
\begin{equation}
\frac{\partial u}{\partial t} - f v = - g \frac{\partial \eta}{\partial x}
\end{equation}
\begin{equation}
\frac{\partial v}{\partial t} + f u = - g \frac{\partial \eta}{\partial y}
\end{equation}
\begin{equation}
\frac{\partial \eta}{\partial t} + H \left( \frac{\partial u}{\partial x} + \frac{\partial v}{\partial y} \right) = 0
\end{equation}
where $(u, v)$ is the horizontal velocity in $(x, y)$ directions, $f$ is the Coriolis parameter, $g = 9.8$m s$^{-2}$ is the acceleration due to gravity and $H$ is the depth of the fluid.

The box has width and length $length$ and depth $H$.

```python
# Nice small system, keep npts small
npts = 32
# Standard Parameters
g = 9.8 # m/s2
# Choose your latitude
lat = 40 # degrees N
f = 2 * 7.27e-5 * np.sin(np.deg2rad(lat))
# Choose your Box Size
length = 2000e3 # m
H = 200 # m
```

The shallow water equations allow shallow water surface waves.  Without rotation these waves have wavespeed, $c = (gH)^{1/2}$.  With rotation two types of waves are possible, Poincare waves and coastally trapped Kelvin waves.  Kelvin waves have the same wave speed.  Poincare waves are slower but approach $c$ as wavelengths decrease.

We will solve the system on an Arakawa C-Grid with the u-velocity staggered 1/2 grid point in x from the $\eta$ point and the v-velocity staggered 1/2 grid point in y.  Like this:
v
|
|
$\eta$ $-$ $-$ u

We will use an explicit leap-frog scheme and so must choose our timestep, $dt$, small enough to meet the CFL condition. The leap-frog scheme requires two-previous steps so we need two sets of arrays. For one set of boundary conditions we require yet another previous time step, so we will use three sets of arrays.

```python
# Derived Quantities
dx = length/npts
wavespeed = np.sqrt(g*H)
dt = 0.25*dx/wavespeed
```

## Initialization

*Set up our arrays*

```python
def set_arrays(npts):
return eta_now, u_now, v_now, eta_prev,
           u_prev, v_prev, eta_next, u_next, v_next
```

## Internal Forcing

Our forcing will be an initial surface height increase of Gaussian shape, offset so its in the southwest quadrant of the box.

*Initialize Gaussian*
```python
def initialize(magnitude, gausshalfwidth, eta):
   return eta_init
```

*A 2-dimensional Gaussian if you want it*
```python
def initialize_simple(magnitude, gausshalfwidth, eta):
return eta_init
```

## External Forcing ##
The external forcing is a long wave length wave coming in from the west.

### Large scale wave ###
$\eta = \eta_o \sin(kx - \omega t)$

$k = 2\pi/(10 L)$

$c = \sqrt(gH)$

$\omega = c/k$

$\left( \frac {\partial^2}{\partial t^2} + f^2\right) u = - g \left(
\frac {\partial^2 \eta}{\partial t \partial x} + f \frac{\partial \eta}{\partial y}\right)$

$\left( \frac {\partial^2}{\partial t^2} + f^2\right) v = - g \left(
\frac {\partial^2 \eta}{\partial t \partial y} - f \frac{\partial \eta}{\partial x}\right)$

*Incoming Large Scale Wave*
```python
def incoming_wave(t, npts, dx):
    return eta_wave, u_wave, v_wave
```

## Time Stepping

To start the code, we do a predictor-corrector which requires an Euler step

*Euler Step*
```python
def euler(eta, u, v, idt, f, g, H, dx):
    return eta_next, u_next, v_next
```

*Leap-frog Step*
```python
def leapfrog(eta, u, v, etap, up, vp, idt, f, g, H, dx):
    return eta_next, u_next, v_next
```

## Boundary Conditions ##

The equations above neglect lateral viscosity and advection.  These approximations, plus the choice of the c-grid, greatly reduce the number of boundary conditions we need.  We only need to set the normal velocity at each of the four edges of the box.

The figure below shows the western wall of the box

$u_o$ is a real boundary point.  $v_o$ and $\eta_o$ are ghost points.  Neither are needed in the "stencil" to find interior points.  So we just don't use them or set them.

```python
filename = 'grid_lateral.png'
from IPython.display import Image
Image(filename=filename)
```

```python
def boundary_conditions_periodic(u, v, **kwargs):
    '''Periodic boundary conditions wrap the solution,
    copying the first internal point on the eastern boundary
    to the boundary point on the western boundary'''
    u[0] = u[-2]
    u[-1] = u[1]
    v[:, 0] = v[:, -2]
    v[:, -1] = v[:, 1]
    return u, v
```

```python
def boundary_conditions_freeslip(u, v, **kwargs):
    '''These conditions impose wall boundaries with zero flow
    through the walls'''
    u[0] = 0
    u[-1] = 0
    v[:, 0] = 0
    v[:, -1] = 0
    return u, v
```

```python
def boundary_conditions_zerogradient(u, v, **kwargs):
    '''A simple, not particularly good, open boundary type where
    the value outside the boundary is the same as inside'''
    u[0] = u[1]
    u[-1] = u[-2]
    v[:, 0] = v[:, 1]
    v[:, -1] = v[:, -2]
    return u, v
```

```python
def boundary_conditions_cnstgradient(u, v, **kwargs):
    '''A simple open boundary type where the value outside the boundary
    is different from the value inside the boundary by the same amount
    as the value inside is from the next value'''
    u[0] = 2 * u[1] - u[2]
    u[-1] = 2 * u[-2] - u[-3]
    v[:, 0] = 2 * v[:, 1] - v[:, 2]
    v[:, -1] = 2 * v[:, -2] - v[:, -3]
    return u, v
```

```python
def boundary_conditions_sommerfield(u_next, v_next, u_now, v_now, u_prev,
                                    v_prev, dx, dt, wavespeed, **kwargs):
    '''A wave radiation boundary condition.  Signals are moved out at the
    assumed wavespeed'''
    # cn is c*dt/dx or normalized wavespeed
    cn = -wavespeed * dt / dx
    u_next[0] = (1 + cn) / (1 - cn) * u_prev[0] - 2 * cn / (1 - cn) * u_now[1]
    u_next[-1] = (1 + cn) / (1 - cn) * u_prev[-1] - 2 * cn / (1 - cn
                                                              ) * u_now[-2]
    v_next[:, 0] = (1 + cn) / (1 - cn) * v_prev[:, 0] - 2 * cn / (
        1 - cn) * v_now[:, 1]
    v_next[:, -1] = (1 + cn) / (1 - cn) * v_prev[:, -1] - 2 * cn / (
        1 - cn) * v_now[:, -2]
    return u_next, v_next
```

```python
def boundary_conditions_orlanski(u_next, v_next, u_now, v_now, u_prev, v_prev,
                                 dx, dt, **kwargs):
    '''The classic wave radiation boundary condition.  A wave speed is estimated.
    If that wavespeed is toward the boundary, signals are moved out at that
    wavespeed. There is a limited on the wavespeed.'''
    # cn is c*dt/dx or normalized wavespeed
    epsilon = 1e-12
    # c here is c*dt/dx
    cn = (u_next[1] - u_prev[1]) / (
        u_next[1] + u_prev[1] - 2 * u_now[2] + epsilon)
    for j in range(u_next.shape[1]):
        if cn[j] > 0:
            cn[j] = 0
        elif cn[j] < -1:
            cn[j] = -1
    u_next[0] = (1 + cn) / (1 - cn) * u_prev[0] - 2 * cn / (1 - cn) * u_now[1]

    cn = (u_next[-2] - u_prev[-2]) / (
        u_next[-2] + u_prev[-2] - 2 * u_now[-3] + epsilon)
    for j in range(u_next.shape[1]):
        if cn[j] > 0:
            cn[j] = 0
        elif cn[j] < -1:
            cn[j] = -1
    u_next[-1] = (1 + cn) / (1 - cn) * u_prev[-1] - 2 * cn / (1 - cn) * u_now[-2]

    c = (v_next[:, 1] - v_prev[:, 1]) / (
        v_next[:, 1] + v_prev[:, 1] - 2 * v_now[:, 2] + epsilon)
    for i in range(v_next.shape[0]):
        if cn[i] > 0:
            cn[i] = 0
        elif cn[i] < -1:
            cn[i] = -1
    v_next[:, 0] = (1 + cn) / (1 - cn) * v_prev[:, 0] - 2 * cn / (1 - cn
                                                               ) * v_now[:, 1]

    cn = (v_next[:, -2] - v_prev[:, -2]) / (
        v_next[:, -2] + v_prev[:, -2] - 2 * v_now[:, -3] + epsilon)
    for i in range(v_next.shape[0]):
        if cn[i] > 0:
            cn[i] = 0
        elif cn[i] < -1:
            cn[i] = -1
    v_next[:, -1] = (1 + cn) / (1 - cn) * v_prev[:, -1] - 2 * cn / (
        1 - cn) * v_now[:, -2]

    return u_next, v_next
```

```python
def boundary_conditions_frs_sommerfield(u_next, v_next, u_now, v_now, u_prev,
                                        v_prev, dx, dt, wavespeed, eta_next,
                                        rim, t, **kwargs):
    '''Combined boundary conditions.  Sommerfield radiation and then
    Relaxation to an applied external field'''

    cn = -wavespeed * dt / dx

    u_next[-1] = (1 + cn) / (1 - cn) * u_prev[-1] - 2 * cn / (1 - cn
                                                              ) * u_now[-2]

    v_next[:, 0] = (1 + cn) / (1 - cn) * v_prev[:, 0] - 2 * cn / (
        1 - cn) * v_now[:, 1]

    v_next[:, -1] = (1 + cn) / (1 - cn) * v_prev[:, -1] - 2 * cn / (
        1 - cn) * v_now[:, -2]

    weight = np.array([np.tanh((np.arange(rim)) * 0.5), ] *
                      u_next.shape[0]).transpose()

    eta_wave, u_wave, v_wave = sw.incoming_wave(t, u_next.shape[0], dx, f, g,
                                                wavespeed)
    u_next[:rim] = u_wave[:rim] * (1 - weight) + u_next[:rim] * weight
    v_next[:rim] = v_wave[:rim] * (1 - weight) + v_next[:rim] * weight
    eta_next[:rim] = eta_wave[:rim] * (1 - weight) + eta_next[:rim] * weight

    return eta_next, u_next, v_next
```

```python
def boundary_conditions_flather_sommerfield(
        u_next, v_next, u_now, v_now, u_prev, v_prev, dx, dt, eta_next,
        wavespeed, rim, t, eta_now):
    '''Combined boundary conditions.  Sommerfield radiation and Flather conditions
    for an applied external barotropic surface height field'''

    cn = -wavespeed * dt / dx

    u_next[-1] = (1 + cn) / (1 - cn) * u_prev[-1] - 2 * cn / (1 - cn
                                                              ) * u_now[-2]

    v_next[:, 0] = (1 + cn) / (1 - cn) * v_prev[:, 0] - 2 * cn / (
        1 - cn) * v_now[:, 1]

    v_next[:, -1] = (1 + cn) / (1 - cn) * v_prev[:, -1] - 2 * cn / (
        1 - cn) * v_now[:, -2]

    eta_wave, u_wave, v_wave = sw.incoming_wave(t, u_next.shape[0], dx, f, g,
                                                wavespeed)

    u_next[0] = u_wave[0] - np.sqrt(g / H) * (eta_now[1] - eta_wave[0])

    return eta_next, u_next, v_next
```

```python
def boundary_conditions(kind, u_next, v_next, u_now, v_now, u_prev, v_prev, dx,
                        dt, wavespeed, eta_next, rim, t, eta_now):
    '''Generalized driver for boundary condtions'''
    bc_funcs = {
        'periodic': boundary_conditions_periodic,
        'freeslip': boundary_conditions_freeslip,
        'zerogradient': boundary_conditions_zerogradient,
        'cnstgradient': boundary_conditions_cnstgradient,
        'sommerfield': boundary_conditions_sommerfield,
        'orlanski': boundary_conditions_orlanski,
        'frs_sommerfield': boundary_conditions_frs_sommerfield,
        'flather_sommerfield': boundary_conditions_flather_sommerfield,
    }
    with_eta = ['frs_sommerfield', 'flather_sommerfield']
    if kind in with_eta:
        eta_next, u_next, v_next = bc_funcs[kind](u_next,
                                                  v_next,
                                                  u_now=u_now,
                                                  v_now=v_now,
                                                  u_prev=u_prev,
                                                  v_prev=v_prev,
                                                  dx=dx,
                                                  dt=dt,
                                                  wavespeed=wavespeed,
                                                  eta_next=eta_next,
                                                  rim=rim,
                                                  t=t,
                                                  eta_now=eta_now)
    else:
        u_next, v_next = bc_funcs[kind](u_next,
                                        v_next,
                                        u_now=u_now,
                                        v_now=v_now,
                                        u_prev=u_prev,
                                        v_prev=v_prev,
                                        dx=dx,
                                        dt=dt,
                                        wavespeed=wavespeed,
                                        eta_next=eta_next,
                                        rim=rim,
                                        t=t,
                                        eta_now=eta_now)
    return eta_next, u_next, v_next
```

# Main Program #

```python
# Chose how many time steps
ntime = 10
# Chose boundary conditions
boundary_type = 'periodic'
rim = 3

# initialize array
eta_now, u_now, v_now, eta_prev, u_prev, v_prev, eta_next, u_next, v_next = sw.set_arrays(
    npts)

# add our internal forcing
eta_prev = sw.initialize(magnitude=1.0, gausshalfwidth=3.5, eta=eta_now)

# take an Euler step from 0 to dt
eta_now, u_now, v_now = sw.euler(eta_prev, u_prev, v_prev, dt, f, g, H, dx)
# and boundary conditions
eta_now, u_now, v_now = boundary_conditions(
    boundary_type, u_now, v_now, u_now, v_now, u_prev, v_prev, dx, dt / 2,
    wavespeed, eta_now, rim, dt, eta_prev)

# Average to give us the 1/2 dt value in now
for var_now, var_prev in zip([eta_now, u_now, v_now],
                             [eta_prev, u_prev, v_prev]):
    var_now = 0.5 * (var_now + var_prev)

# Take a first leap-frog step
eta_next, u_next, v_next = sw.leapfrog(eta_now, u_now, v_now, eta_prev, u_prev,
                                       v_prev, dt, f, g, H, dx)
# Increment Time
t = dt
# Boundary Conditions
eta_next, u_next, v_next = boundary_conditions(
    boundary_type, u_next, v_next, u_now, v_now, u_prev, v_prev, dx, dt / 2,
    wavespeed, eta_next, rim, t, eta_now)

# and now we are started we can loop
for i in range(ntime):
    eta_prev, u_prev, v_prev = eta_now, u_now, v_now
    eta_now, u_now, v_now = eta_next, u_next, v_next

    eta_next, u_next, v_next = sw.leapfrog(eta_now, u_now, v_now, eta_prev,
                                           u_prev, v_prev, 2 * dt, f, g, H, dx)
    t = t + dt
    eta_next, u_next, v_next = boundary_conditions(
        boundary_type, u_next, v_next, u_now, v_now, u_prev, v_prev, dx, dt,
        wavespeed, eta_next, rim, t, eta_now)

    # every 101 time steps average and restart to remove the leap frog
    # computational mode
    if (np.mod(i, 101) == 0):
        for var_now, var_next in zip([eta_now, u_now, v_now],
                                     [eta_next, u_next, v_next]):
            var_next = 0.5 * (var_now + var_next)
        eta_next, u_next, v_next = sw.leapfrog(
            eta_next, u_next, v_next, eta_now, u_now, v_now, dt, f, g, H, dx)
        eta_next, u_next, v_next = boundary_conditions(
            boundary_type, u_next, v_next, u_next, v_next, u_now, v_now, dx,
            dt, wavespeed, eta_next, rim, t, eta_next)
```

# Plotting #

```python
sw.make_plot(npts, dx, eta_next, u_next, v_next)
```

```python
sw.make_line_plots(eta_next, u_next, v_next, 'eastwest', ii=7)
```

# The Questions #

## A. Simple Boundary Conditions ##

I have included a suite of simple boundary conditions:
* periodic
* freeslip
* zerograd
* cnstgrad

For each of these boundary conditions, open a cell below.  Choose markdown and answer the following questions (the first one **before** you run the code).  You may wish to change the length of time you run for "ntime"
* What do you expect to see at the boundary?
* What do you see at the boundary when you run the code?
* What would you use these boundary conditions for?

## B. Radiation Boundary Conditions ##

I have included two radiation type boundary conditions:
* sommerfield which uses the known wavespeed
* orlanski which calculates the wavespeed

Open a cell below and answer these questions:
* What should radiation boundary conditions do?
* Which of the two methods works better?  In what way is it better?
* Under which conditions, would you use the other method?

# C. Bringing Information In #

I have included two types of boundary conditions that bring external information in.
* frs
* flather

In each case, I am bringing in a large wave from the west.  Note that you still have your internal forcing.

Open a cell below and answer these questions:
* Ideally what should happen?
* Do you get what you expect?

# D. Toward the Real World #

The major simplification that matters here, is that this system has only one wave speed.

* Why does adding multiple wave speeds make open boundaries more difficult?

```python

```