# Optimization Sample: Multi-ship loading

In this example, we will take our learnings from the ship-loading sample and generalize to load-balancing between any number of ships. In addition, we'll see how we can make use of the parameter free-solvers to guide us on a selection of parameters which we can use with the parametrized solvers.

## Pre-requisites

1. [Create an Azure Quantum Workspace](https://github.com/MicrosoftDocs/quantum-docs-private/wiki/Create-quantum-workspaces-with-the-Azure-portal)
2. [Install the `azure-quantum` Python module](https://github.com/MicrosoftDocs/quantum-docs-private/wiki/Use-the-Python-SDK-for-Quantum-Inspired-Optimization).
3. [Complete the ship-loading sample](https://github.com/microsoft/qio-samples/tree/main/samples/ship-loading)

## The Problem

To tackle this problem we will use a PUBO format. As a reminder, these are [cost functions](https://docs.microsoft.com/en-us/azure/quantum/optimization-concepts-cost-functions) where the variables take the values of either 0 or 1 (rather than -1 or 1 for an Ising cost function).

In order to balance containers between multiple ships, one option is to define a cost function that:
1. Penalizes variance from a theoretical equal distribution (where an equal distribution is the total weight of the containers divided by the number of ships), and,
2. Penalizes the assignment of the same container on multiple ships

We will create two sub cost-functions $H1$ and $H2$ that we will then sum to evaluate the total cost of a solution. Let's begin with the first cost function, $H1$.

## Penalize variance from equal distribution between ships

Suppose we had 3 containers with respective weights $W0$, $W1$, $W2$, and we define an equal distribution of the container weights to be:

$$EqDistrib = (W0 + W1 + W2) / 3$$

A way to penalize a large variance from the equal distribution for a given ship is to express it in the following way:

$$(W0 + W1 + W2 - EqDistrib)^2$$

Let's take the following example:

|                 |               |
|-----------------|---------------|
|Container weights| 1, 5, 9, 7, 3 |
|Total weight     |      25       |
|Ships            |     A, B, C   |
|EqualDistrib     | 25 / 3 = 8.33 |

Suppose we were to assign those containers to the ships listed, we can calculate the variance from the equal distribution for each of the given ships, shown in the rightmost column:

| Ships\Containers | 1 | 5 | 9 | 7 | 3 |            |        |
|------------------|---|---|---|---|---|------------|--------|
| A                | 0 | 0 | 9 | 0 | 0 | (9-8.33)^2 |= 0.4489|
| B                | 0 | 5 | 0 | 0 | 3 |(5+3-8.33)^2|= 0.1089|
| C                | 1 | 0 | 0 | 7 | 0 |(1+7-8.33)^2|= 0.1089|

As we need to represent our problem in a binary format we need to "encode" the presence ($x_i=1$) or absence ($x_i=0$) of a given container on a ship. To do this, we need to have a label for the weight of each container on each ship. The table below shows how we assign this continuous index by repeating the list of container weights for each ship and assigning a single list of weight labels across all three ships:

|| Ship A |     |     |     |     |  Ship B |     |     |     |     | Ship C   |        |        |        |        |
|---|--------|-----|-----|-----|-----|--------|-----|-----|-----|-----|----------|--------|--------|--------|--------|
|Container weight| 1 |  5  |  9  |  7  |  3  | 1 |  5  |  9  |  7  |  3  | 1 | 5 | 9 | 7 | 3  |
|Weight label|*w0*|*w1*|*w2*|*w3*|*w4*| *w5*|*w6*|*w7*|*w8*|*w9*|*w10*|*w11*|*w12*|*w13*|*w14*|

The cost function $H1$ becomes:

$$ H1 = H_{A} + H_{B} + H_{C} $$

where:

$$ H_{A} = (w_0 x_0 + w_1 x_1 + w_2 x_2 + w_3 x_3 + w_4 x_4 - EqDistrib)^2 $$

$$ H_{B} = (w_5 x_5 + w_6 x_6 + w_7 x_7 + w_8 x_8 + w_9 x_9 - EqDistrib)^2 $$

and

$$ H_{C} = (w_{10} x_{10} + w_{11} x_{11} + w_{12} x_{12} + w_{13} x_{13} + w_{14} x_{14} - EqDistrib)^2 $$

We can expand the above and group the common terms, for example if we expand $H_{A}$, we get:

$$
\begin{align}
H_{A} &= (\sum_i(w_i x_i) - EqDistrib)^2\\
&= (w_0 x_0 + w_1 x_1 + w_2 x_2 + w_3 x_3 + w_4 x_4 - EqDistrib)^2\\
\end{align}
$$

To simplify things for the expansion, let's rename the variables as follows:

$$
\begin{align}
w_0 x_0 &= a \\
w_1 x_1 &= b \\
w_2 x_2 &= c \\
w_3 x_3 &= d \\
w_4 x_4 &= e \\
EqDistrib &= f \\
\end{align}
$$

So now we have:

$$
\begin{align}
H_{A} &= (\sum_i(w_i x_i) - EqDistrib)^2\\
&= (w_0 x_0 + w_1 x_1 + w_2 x_2 + w_3 x_3 + w_4 x_4 - EqDistrib)^2\\
&= (a + b + c + d + e - f)^2\\
&= a^2 + b^2 + c^2 + d^2 + e^2 + f^2 + 2(ab + ac + ad + ae + bc + bd + be + cd + ce + de) - 2(af + bf + cf + df + ef)
\end{align}
$$

Substituting our original values back in, this gives us the following:

$$
\begin{align}
H_{A} &= (\sum_i(w_i x_i) - EqDistrib)^2\\
&= w_0^2 x_0^2 + w_1^2 x_1^2 + w_2^2 x_2^2 + w_3^2 x_3^2 + w_4^2 x_4^2 + EqDistrib ^2 + 2(w_0 x_0 \cdot w_1 x_1 + w_0 x_0 \cdot w_2 x_2 + w_0 x_0 \cdot w_3 x_3 + w_0 x_0 \cdot w_4 x_4 + w_1 x_1 \cdot w_2 x_2 + w_1 x_1 \cdot w_3 x_3 + w_1 x_1 \cdot w_4 x_4 + w_2 x_2 \cdot w_3 x_3 + w_2 x_2 \cdot w_4 x_4 + w_3 x_3 \cdot w_4 x_4) - 2(w_0 x_0 \cdot EqDistrib +  w_1 x_1 \cdot EqDistrib + w_2 x_2 \cdot EqDistrib + w_3 x_3 \cdot EqDistrib + w_4 x_4 \cdot EqDistrib)
\end{align}
$$

We can do the same for $H_{B}$, and $H_{C}$.

## Penalize the assignment of the same container on multiple ships

Using the containers weight encoding above, we can devise a cost function such as this one for the first container:

$$ H_{D} = (w_0 x_0 + w_5 x_5 + w_{10} x_{10} - w_0)^2 $$

As $w_0$, $w_5$ and $w_{10}$ are actually the same value (it is the same container represented across multiple ships) we have:
$$ H_{D} = (w_0 x_0 + w_0 x_5 + w_0 x_{10} - w_0)^2 $$

If we expand and group the common terms, we get the following:

$$ H_{D} = {w_0}^2 {x_0}^2 + {w_0}^2 {x_5}^2 + {w_0}^2 {x_{10}}^2 + {w_0}^2 +
2 ({w_0}^2 x_0 x_5 + {w_0}^2 x_0 x_{10} + {w_0}^2 x_5 x_{10})
- 2({w_0}^2 x_0 + {w_0}^2 x_5 + {w_0}^2 x_{10}) $$

We can then repeat the above for each container across all ships:

So in addition to:

$$ H_{D} = (w_0 x_0 + w_0 x_5 + w_0 x_{10} - w_0)^2 $$

we also have:

$$ H_{E} = (w_1 x_1 + w_1 x_6 + w_1 x_{11} - w_1)^2 $$
$$ H_{F} = (w_2 x_2 + w_2 x_7 + w_2 x_{12} - w_2)^2 $$
$$ H_{G} = (w_3 x_3 + w_3 x_8 + w_3 x_{13} - w_3)^2 $$
$$ H_{H} = (w_4 x_4 + w_4 x_9 + w_4 x_{14} - w_4)^2 $$

Grouping these together into a single cost function, $H2$, we get:

$$ H2 = H_{D} + H_{E} + H_{F}+ H_{G}+ H_{H} $$

which we can expand and group the terms as we did with $H_D$ above.

## Combining our cost functions

The final part of the problem definition is to combine our cost functions $H1$ and $H2$:

$$H = H1 + H2 $$

You will notice that $H1$ and $H2$ have common indices $[i,i]/[m,m]$ and $[i]/[m]$. We will need to be careful to not duplicate them, but sum them, in our final list of terms describing the cost function.

## Solving the Problem in Python

First, we must instantiate a `Workspace` object which allows you to connect to the Workspace you've previously deployed in Azure. Be sure to fill in the settings below which can be retrieved by running `az quantum workspace show`.

```python
# This allows you to connect to the Workspace you've previously deployed in Azure.
# Be sure to fill in the settings below which can be retrieved by running 'az quantum workspace show' in the terminal.
from azure.quantum import Workspace

# Copy the settings for your workspace below
workspace = Workspace (
    subscription_id = "",
    resource_group = "",
    name = "",
    location = ""
)
```

First, let's define a function to add terms according to our definition of the H1 cost function, where we penalized the variance from an equal distribution between the ships.

```python
def AddTermsWeightVarianceCost(start, end, containers, EqDistrib):
    terms: List[Term] = []
    for i,w in enumerate(containers[start:end+1], start):
        # -2*Wi*EqDistrib.xi -2Wi^2.xi (weight variance cost + duplicate container cost)
        terms.append(Term(w=-2*w*EqDistrib - 2*w*w, indices=[i]))
        # Wi^2.xi^2 + Wi^2.xi^2 (weight variance cost + duplicate container cost)
        terms.append(Term(w=2*w*w, indices=[i,i]))

    for c in combinations(range(start, end+1), 2):
        w0 = containers[c[0]]
        w1 = containers[c[1]]
        # 2*Wi*Wj (weight variance cost)
        terms.append(Term(w=2*w0*w1, indices=[c[0],c[1]]))

    return terms
```

Next, let's incorporate our definition of the second part of our cost function, H2, where we wished to penalize the assignment of the same container on multiple ships.

```python
def AddTermsDuplicateContainerCost(start, end, containers):
    terms: List[Term] = []

    # The following is integrated into AddTermsWeightVarianceCost to reduce the number of Terms and speed-up Terms generation
    # for c in combinations(range(start, end+1), 1):
    #     w = containers[c[0]][0]
    #     i1 = containers[c[0]][1]
    #     terms.append(Term(w=w*w, indices=[i1,i1]))              # Wi^2

    # 2.w^2.x_i.x_j terms
    for c in combinations(range(start, end+1), 2):
        w = containers[c[0]][0]
        i1 = containers[c[0]][1]
        i2 = containers[c[1]][1]
        terms.append(Term(w=2*w*w, indices=[i1,i2]))            # Term(w=2*Wm^2, [m,n])

    # The following is integrated into AddTermsWeightVarianceCost to reduce the number of Terms and speed-up Terms generation
    # # for c in combinations(range(start, end+1), 1):
    #     w = containers[c[0]][0]
    #     i1 = containers[c[0]][1]
    #     terms.append(Term(w=-2*w*w, indices=[i1]))              # -2*Wi^2

    # w^2 term
    terms.append(Term(w=containers[start][0]*containers[start][0], indices=[]))

    return terms
```

Combining these together, we can create our cost function definition of the multi-ship problem.

```python
from typing import List
from azure.quantum.optimization import Term
import numpy as np
from itertools import combinations

def createProblemForContainerWeights(containerWeights: List[int], Ships) -> List[Term]:

    terms: List[Term] = []
    containersWithinShip: List[int] = []
    containersAcrossShips: List[int, int] = []
    totalWeight = 0
    EqDistrib = 0

    for c in range (len(containerWeights)):
        totalWeight = totalWeight + containerWeights[c]
    EqDistrib = totalWeight / len(Ships)
    print(Ships)
    print(containerWeights)
    print("Total Weight:", totalWeight)
    print("Equal weight distribution:", EqDistrib)

    # Create container weights in this format:
    # 1  5  9  7  3  - 1  5  9  7  3  - 1   5   9   7   3
    # W0 W1 W2 W3 W4   W5 W6 W7 W8 W9   W10 W11 W12 W13 W14
    containersWithinShip = containerWeights*len(Ships)

    # Create container weights in this format:
    # 1  1  1  5  5  5  9  9  9  7  7  7  3  3  3
    for i in range(len(containerWeights)):
        for j in range(len(Ships)):
            k = i + j*len(containerWeights)
            containersAcrossShips.append([containersWithinShip[i], k])

    for split in np.array_split(range(len(containersWithinShip)), len(Ships)):
        terms = terms + AddTermsWeightVarianceCost(split[0], split[-1], containersWithinShip, EqDistrib)

    for split in np.array_split(range(len(containersAcrossShips)), len(containerWeights)):
        terms = terms + AddTermsDuplicateContainerCost(split[0], split[-1], containersAcrossShips)

    return terms
```

Before we solve our problem, let's create a function to help us visualize the results of the solver:

```python
def visualize_result(result, containers, ships, target):
    print("\rResult received from: ", target)
    nb_ships = len(ships)
    try:
        config = result['configuration']
        config = list(config.values())
        for ship, sub_config in enumerate(np.array_split(config, nb_ships)):
            shipWeight = 0
            for c,b in enumerate(sub_config):
                shipWeight = shipWeight + b*containers[c]
            print(f'Ship {ships[ship]}: \t' + ''.join(f'{b*containers[c]}' for c,b in enumerate(sub_config)) + ' - ' + str(shipWeight))
    except:
        print('No Configuration')
    try:
        print('Cost: {}'.format(result['cost']))
    except:
        print('No Cost')
    try:
        print('Parameters: {}'.format(result['parameters']))
    except:
        print('No Parameter')
```

Lastly, let's define a function that will submit our problem to Azure Quantum and visualize the results.

```python
def SolveMyProblem(problem, s):
    try:
        # Optimize the problem
        print("Optimizing with:", s.target)
        Job = s.submit(problem)
        Job.wait_until_completed()
        duration = Job.details.end_execution_time - Job.details.begin_execution_time
        if (Job.details.status == "Succeeded"):
            visualize_result(Job.get_results(), containerWeights*len(Ships), Ships, s.target)
            print("Execution duration: ", duration)
        else:
            print("\rJob ID", Job.id, "failed")
        return Job.id
    except BaseException as e:
        print(e)
```

## Submitting our problem to Azure Quantum

Now, we can put it all together. In the following steps we'll:
1. Define the list of containers and their weights
2. Instantiate the problem, creating our list of terms
3. Submit these terms to a parameter-free solver
4. Use the parameters returned to submit the problem to a parametrized solver

```python
# This array contains a list of the weights of the containers:
containerWeights = [3, 8, 3, 4, 1, 5, 2, 2, 7, 9, 5, 4, 8, 9, 4, 6, 8, 7, 6, 2, 2, 9, 4, 6, 3, 8, 5, 7, 2, 4, 9, 4]
Ships = ["A", "B", "C", "D", "E"]
```

```python
# Create the Terms for this list of containers:
terms = createProblemForContainerWeights(containerWeights,Ships)
```

```python
from azure.quantum.optimization import Problem, ProblemType

# Create the Problem to submit to the solver:
nbTerms = len(terms)
problemName = f'Balancing {str(len(containerWeights))} containers between {str(len(Ships))} Ships ({nbTerms:,} terms)'
print(problemName)
problem = Problem(name=problemName, problem_type=ProblemType.pubo, terms=terms)
```

```python
from azure.quantum.optimization import SimulatedAnnealing

# Try to call a solver with different timeout value and see if it affects the results
jobid = SolveMyProblem(problem, SimulatedAnnealing(workspace, timeout=10))
# jobid = SolveMyProblem(problem, SimulatedAnnealing(workspace, timeout=20))
# jobid = SolveMyProblem(problem, SimulatedAnnealing(workspace, timeout=30))
# jobid = SolveMyProblem(problem, SimulatedAnnealing(workspace))
```

```python
# Try using the parameters returned by the parameter free versions and observe the significant performance improvement

# First use the job id to view the parameters selected by the parameter free solver
job = workspace.get_job(jobid)
results = job.get_results()

print(results)
```

```python
# From the results, let's extract the beta_start, beta_stop, restarts, and sweeps parameters selected

beta_start = results["parameters"]["beta_start"]
beta_stop = results["parameters"]["beta_stop"]
restarts = results["parameters"]["restarts"]
sweeps = results["parameters"]["sweeps"]
```

```python
# Now let's call the solver again, this time the parametrized version, using these parameters

jobid = SolveMyProblem(problem, SimulatedAnnealing(workspace, timeout=5, beta_start=beta_start, beta_stop=beta_stop, restarts=restarts, sweeps=sweeps))

```

```python
from azure.quantum.optimization import ParallelTempering, Tabu, HardwarePlatform, QuantumMonteCarlo

# Here's how we could experiment with different solvers from Microsoft

jobid = SolveMyProblem(problem, SimulatedAnnealing(workspace, platform=HardwarePlatform.FPGA, timeout=5))
# jobid = SolveMyProblem(problem, Tabu(workspace, timeout=5))
# jobid = SolveMyProblem(problem, ParallelTempering(workspace, timeout=60))
# jobid = SolveMyProblem(problem, QuantumMonteCarlo(workspace))
```

```python
from azure.quantum.optimization.oneqbit import PathRelinkingSolver

# And how we can submit the same jobs to solvers by third-party providers, such as 1QBit
# Note: PathRelinkingSolver is only available if the 1QBit provider is enabled in your quantum workspace

# jobid = SolveMyProblem(problem, PathRelinkingSolver(workspace))
```

```python

```


# Practical 2b:  Optimisation for Deep Learning

© Deep Learning Indaba. Apache License 2.0.

## Introduction

In this practical, we will take a *deep* dive into an essential part of deep learning, and machine learning in general, **optimisation**. We'll take a look at the tools that allow as to turn a random collection of weights into a state-of-the-art model for any number of applications. More specifically, we'll implement a few standard optimisation algorithms for  finding the minimum of [Rosenbrock's banana function](https://en.wikipedia.org/wiki/Rosenbrock_function) and then we'll try them out on FashionMNIST.

## Learning Objectives

* Understand **what optimisation algorithms are**, and **how they are used** in the context of deep learning.
* Understand gradient descent, stochastic gradient descent, and mini-batch stochastic gradient descent.
* Understand the roles of batch size, learning rate and other hyper-parameters.
* Implement, using TF2.0, gradient descent and a few variations of it.
* Understand the strengths and weaknesses of the various optimisation algorithms covered in this practical.

```python
#@title Imports (RUN ME!) { display-mode: "form" }

!pip install tensorflow-gpu==2.0.0-beta0 > /dev/null 2>&1

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from IPython import display
%matplotlib inline

display.clear_output()

print("TensorFlow executing eagerly: {}".format(tf.executing_eagerly()))
```

$$
\newcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\vechat}[1]{\hat{\mathbf{#1}}}
\newcommand{\x}{\vec{x}}
\newcommand{\utheta}{θ}
\newcommand{\th}{\vec{\utheta}}
\newcommand{\y}{\vec{y}}
\newcommand{\b}{\vec{b}}
\newcommand{\W}{\textrm{W}}
\newcommand{\L}{\mathcal{L}}
\newcommand{\xhat}{\vechat{x}}
\newcommand{\yhat}{\vechat{y}}
\newcommand{\bhat}{\vechat{b}}
\newcommand{\What}{\hat{\W}}
\newcommand{\partialfrac}[2]{\frac{\partial{#1}}{\partial{#2}}}
\newcommand{\ipartialfrac}[2]{{\partial{#1}}/{\partial{#2}}}
\newcommand{\dydx}{\partialfrac{\y}{\x}}
\newcommand{\dld}[1]{\partialfrac{\L}{#1}}
\newcommand{\dldx}{\dld{\x}}
\newcommand{\dldy}{\dld{\y}}
\newcommand{\dldw}{\dld{W}}
\newcommand{\idld}[1]{\ipartialfrac{\L}{#1}}
\newcommand{\idldx}{\idld{\x}}
\newcommand{\idldy}{\idld{\y}}
\newcommand{\idydx}{\ipartialfrac{\y}{\x}}
\newcommand{\red}[1]{\color{red}{#1}}
\newcommand{\green}[1]{\color{green}{#1}}
\newcommand{\blue}[1]{\color{blue}{#1}}
\newcommand{\because}[1]{&& \triangleright \textrm{#1}}
\newcommand{\relu}[1]{\textrm{relu}({#1})}
\newcommand{\step}[1]{\textrm{step}({#1})}
\newcommand{\gap}{\hspace{0.5mm}}
\newcommand{\gapp}{\hspace{1mm}}
\newcommand{\ngap}{\hspace{-0.5mm}}
\newcommand{\ngapp}{\hspace{-1mm}}
$$

## Rosenbrock's Banana Function 🍌

In practice, when evaluating the performance of various optimisation algorithms and hyper-parameters, what we really care about is the performance on a wide range of real-world problems. However, we are not easily able to visualize what our algorithms are doing because the loss landscape for even simple neural networks trained on just about any real-world dataset will be very high-dimensional.

To solve this problem, we will use Rosenbrock's (Banana) Function as a playground for investigating how these optimisation algorithms work. The banana function is easy to visualize because it is a function that takes a 2D ($x$ and $y$) input and returns a scalar output. It is defined as:

\begin{equation}
f(x,y) = (a-x)^2+b\times(y-x^2)^2
\end{equation}

where typical values for $a$ and $b$ are $1$ and $100$, respectively. For this practical we'll use $a = 1$ and $b = 20$. The global minimum of this function is at $(a, a^2)$ or $(1, 1)$ in our case.

We can easily define this function using TensorFlow:

```python
def rosenbrock_banana(x, y, a=1., b=20.):
  return tf.math.pow(a - x, 2.) + b * tf.math.pow(y - tf.math.pow(x, 2.), 2.)
```

Let's try visualizing the 🍌, first using a contour plot:

```python
#@title Helper functions (RUN ME) (double click to unhide/hide the code)

def gen_2d_loss_surface(loss_func,
             n_x=100, # number of discretization points along the x-axis
             n_y=100, # number of discretization points along the x-axis
             min_x=-2., max_x=2., # extreme points in the x-axis
             min_y=-0.2, max_y=1.3 # extreme points in the y-axis
            ):

  # create a mesh of points at which to evaluate our function
  X, Y = np.meshgrid(np.linspace(min_x, max_x, n_x),
                     np.linspace(min_y, max_y, n_y))
  # evaluate the func at all of the points
  Z = loss_func(X, Y).numpy()

  return X, Y, Z

def make_contour_plot(X, Y, Z, levels=None):
  if levels == None:
    # generate 20 levels on a log scale
    levels = np.insert(np.logspace(0, 2.6, 20, True, base=10), 0, 0)

  fig = plt.figure(figsize=(9.84, 3))
  ax = fig.gca()

  ax.contour(X, Y, Z, levels, alpha=0.5)
  ax.contourf(X, Y, Z, levels, alpha=0.2)
  ax.set_xlabel('x')
  ax.set_ylabel('y')

  return fig, ax

def make_surface_plot(X, Y, Z, elevation=0, azimuth_angle=0, levels=None):

  if levels == None:
    # generate 20 levels on a log scale
    levels = np.insert(np.logspace(0, 2.6, 20, True, base=10), 0, 0)

  fig = plt.figure(figsize=(10,6))
  ax = fig.gca(projection='3d')
  ax.view_init(elevation, azimuth_angle)
  ax.plot_surface(X, Y, Z, cmap='viridis', alpha=0.2)
  ax.contour(X, Y, Z, levels, cmap='viridis', alpha=0.5)

  ax.set_xlabel('x')
  ax.set_ylabel('y')

  return fig, ax

```

```python
X, Y, Z = gen_2d_loss_surface(rosenbrock_banana)
fig, ax = make_contour_plot(X, Y, Z)

# add a marker to show the minimum
ax.plot(1, 1, 'r*', ms=30, label='minimum')
ax.legend()

fig.show()
```

And now with a surface plot:

```python
#@title {run: "auto"}

elevation = 62 #@param {type:"slider", min:0, max:360, step:1}
azimuth_angle = 117 #@param {type:"slider", min:0, max:360, step:1}

X, Y, Z = gen_2d_loss_surface(rosenbrock_banana)
fig, ax = make_surface_plot(X, Y, Z, elevation, azimuth_angle)

ax.plot([1], [1], 'r*', zs=[0], zdir='z', ms=30, label='minimum')
ax.legend()

fig.show()
```

As you can see, this is called the *banana* function because it contains a banana-shaped valley. Within the valley, we have a **global minimum**. Finding the valley is relatively easy, but finding the global minimum is difficult, which makes this a useful function for testing optimisation algorithms.

## Optimization

### What is optimization?

Optimization is the process of comparing different items and selecting the best one, based on some metric. Thus, the algorithms we are going to look at try do this for us. So, we can define an optimization algorithm as a program (or series of instructions) that try to find the best item when compared to other items using a given method of comparison (or metric).

### How do we use it in machine learning?

Optimization algorithms are used in machine learning to compare different values for the parameters of a model and to try to find the best one. For the banana function, the optimization algorithm tries to find the x and y values that lead to the lowest value of the function (here our metric is: "lower values are better"). More practically, optimization algorithms are used to try to find the best values for our neural network weights and biases. In this case, we use a "Loss function" as a metric and try to select the weights and biases that lead to the lowest loss function value. But, we don't need to worry about that yet. First, we need to build up some tools that will let us do this.

## Gradients

### What are gradient vectors?

*Note: feel free to skip this if you already understand what a gradient vector represents.*

Let's start with a scalar function $f$, which maps the input $x$ to the output $y$: $$y = \green{f(x)}$$

Here, $f$ could be a polynomial, exponential, or whatever your favorite kind of mathematical function is.

Let's now  _approximate_ the function $\green f$ around a particular point $x_0$ with a straight-line function $\blue {d\ngap f}$:

$$\blue{d\ngap f(x)} = f(x_0) + \red{f'(x_0)}(x - x_0)$$

**Reminder:** the equation for a straight-line function can be given as: $y = m x + c$, where $m$ is the slope of the line and $c$ raises the function. If we want the function to go through a given point $(x_0, y_0)$, then we can change the equation to the following: $y = m (x - x_0) + y_0$. Notice, we just shifted the line along the $x$-axis and made sure the function is raised by $y_0$.

As the picture above shows, we can use the derivative $f' = \frac{dy}{dx}$ to construct the best linear approximation to a function $f$ around a specific point $x_0$. Specifically, the derivative gives us the _slope_. If the function above was horizontal at $x_0$, you can see that the gradient $\red{f'(x_0)}$ would be zero.

Notice the _crucial_ fact that the gradient $\red{f'(x_0)}$ points in the _direction_ in which $\green{f(x)}$ _increases_, and the magnitude tells us _how quickly_ it increases. Here, the gradient is positive, which tells us the function is increasing to the *right*, and the large magnitude tells us it is increasing relatively quickly.

#### What did we just do?

Before we continue, let's make sure we know how this fits into the bigger picture. We just approximated some function (any function) $\green f$ with a straight line. Doing this gave us some important information: the direction in which the function is increasing $\red{f'(x_0)}$. In other words, what direction we should move $x_0$ in to find a higher value of our function $\green f$ - and an idea of how much we should move $x_0$.

In optimization, this is useful because we can use our function $\green f$ to *compare* different items, and we can use the direction $\red{f'(x_0)}$ to get *better* items. In this case, $\green f$ is our *metric* and $x_0$ is our current "item". We then follow the direction $\red{f'(x_0)}$ to get a *better* item than $x_0$.

Below is an interactive version of the graphic. Play around with $x_0$ and build an intuition of how the gradient $\red{f'(x_0)}$ changes with the slope of the function $\green{f(x)}$.

```python
#@title Helper functions (RUN ME) (double click to unhide/hide the code)
def f(x):
  return -np.cos(x)

def tangent_f(x):
  return np.sin(x)

def df(x, x_0):
  return tangent_f(x_0) * (x - x_0) + f(x_0)

def perpindicular_unit_f(x_0):
  slope_f = tangent_f(x_0)
  y_0 = f(x_0)

  x_1 = slope_f / np.sqrt(2) + x_0
  y_1 = -x_1 / slope_f + y_0 + x_0 / slope_f

  return [[x_0, x_1], [y_0, y_1]]

def interactive_gradient_visual(x_0):
  # change the fontsize for better visibility
  init_size = plt.rcParams["font.size"] # store initial font size
  plt.rcParams.update({'font.size': 22}) # update the size

  plt.figure(figsize=(12, 8))

  x = np.linspace(-np.pi, 2 * np.pi)
  f_x = f(x)

  # plot f(x)
  plt.plot(x, f_x, label=r"$f(x)$", color="green")

  # add a point showing where x_0 falls on f(x)
  plt.plot(x_0, f(x_0), marker="o", color="black")

  # plot the tangent line to f(x) at x_0
  plt.plot(x, df(x, x_0), linestyle="--", color="cornflowerblue", label=r"$df(x)$")

  # plot the normal vector to the tangent
  perp_unit_vector = perpindicular_unit_f(x_0)
  plt.plot(perp_unit_vector[0], perp_unit_vector[1], color="dimgray")

  # drop a vertical line from x_0
  plt.plot([x_0, x_0], [f(x_0), -3.1], color="silver")

  # plot the positive direction of change vector
  [[x_0, x_1], [y_0, y_1]] = perp_unit_vector
  dx = x_1 - x_0
  dy = 0 # y_1 - y_1
  arrow = plt.arrow(
      x_0, y_1, dx, dy,
      color="red", label=r"$f'(x_0)$",
      lw=3, head_width=np.abs(x_1 - x_0)/10, length_includes_head=True
  )
  plt.plot([x_0, x_1], [y_1, y_1], color="red", label=r"$f'(x_0)$")

  plt.legend(loc="upper left")
  plt.xlim(-3.1, 6.2)
  plt.ylim(-3.1, 3.1)
  plt.xlabel(r"$x_0$")
  plt.show()

  # reset to initial font size
  plt.rcParams.update({'font.size': init_size})
```

```python
#@title Double click to unhide/hide the code {run: "auto"}
x_0 = 1.25 #@param {type:"slider", min:-3.1, max:6.2, step:0.01}

interactive_gradient_visual(x_0)
```

Again, notice how the gradient $\red{f'(x_0)}$ (the red arrow) always points in the direction where the funciton $\green{f(x)}$ is increasing. Also notice how the magnitude (length of the arrow) of the gradient $\red{f'(x_0)}$ increases when the slope increases.

### Moving to higher dimensions

In fact, a similar thing happens if we are working with a function whose input is a *vector* rather than a scalar. The same idea applies, but we must replace the multiplication between the *scalar* derivative $\red{ f'(x_0) }$ and the *scalar* difference $(x - x_0)$, with a dot product between the *vector* gradient $\red{\nabla f(\vec x_0)}$ and the _vector_ difference $(\vec x - \vec x_0)$:

$$\blue{d\hspace{-0.3ex}f(\x)} = f(\mathbf x_0) + \red{\nabla f(\mathbf x_0)}\gap\cdot \gap (\mathbf x - \mathbf x_0)$$

As the picture above shows, this vector gradient represents the slope of the *plane* that approximates the surface of $f$ near the point $x_0$. The gradient points in the direction of steepest ascent ("the uphill direction"), and its magnitude tells us the steepness of the plane (so zero gradient means the plane is flat).

**Note**: the gradient points in the direction of *increase* of the function. For gradient descent (discussed next), we will instead be travelling in the direction of *decrease* of the loss. This means that our updates will simply be based on the negative of the gradient.

## Gradient Descent

Gradient descent is the simplest of training algorithms commonly used in deep learning. However, it gives excellent results in many cases, and also forms the basis for many other powerful optimization methods - such as Momentum, RMSProp, and Adam - which we will look at later in this practical. Mathematically we can describe gradient descent as follows:

\begin{equation}
\mathbf{θ}= \mathbf{θ} −\eta \times \nabla_\mathbf{θ} J(\mathbf{θ})
\end{equation}

where $\mathbf{θ}$ are the parameters of the model, $\eta$ (eta) is the learning rate, $J(\mathbf{θ})$ is the loss (also called $\mathcal{L}$), and $\nabla_\theta J(\mathbf{θ})$ is the **gradient** of the loss with respect to the parameters - similarly to $\red{\nabla f(\mathbf x_0)}$ in the above example. This equation tells us that to update each of the parameters, we scale the gradient for each parameter by the learning rate and subtract it from the corresponding parameter. Or, in pseudo-code:

```
for each epoch:
  grad = calc_grad(loss_func, data, params)  # Calculate gradient of loss wrt parameters
  params = params - learning_rate * grad
```

You might have noticed that we are sweeping a lot of critical details under the rug here! Firstly we are assuming that we can easily calculate the gradients using some `calc_grad` function, and secondly, we are ignoring the issue of batch size.

**For those that have not heard of batch size before:** the batch size is the number of training data points we use to calculate an estimate of the gradient. There will be more on this soon.

Luckily for us, TensorFlow addresses the first detail thanks to **automatic differentiation** (AD). With AD, calculating the gradients is about as simple as calling a `calc_grad` function, which means that we do not need to worry about the details of *how* to calculate the gradients. We don't need to think much about implementing derivatives for each of our operations, or the backpropagation algorithm, for example. If you want to know more about how this all works, you should check out the *Build your own TensorFlow* tutorial.

In practice, batch size is simply a hyper-parameter that we can tune. However, there are three cases that are worth knowing about:

1.   Using a batch size of $n$, where $n$ is the number of training examples is known as **Batch Gradient Descent (BGD)**. In this case, all of the data is used to calculate the gradient at each step. This results in the *most accurate* estimate of the gradient. If the learning rate is not too high and that there are no [saddle points](https://en.wikipedia.org/wiki/Saddle_point), BGD is guaranteed to converge to:
    * the global minimum for [convex](https://en.wikipedia.org/wiki/Convex_function) optimisation surfaces
    * a local minimum for non-convex surfaces.

    However, a downside of BGD is that it is **not compatible with online learning** - where we get new examples during training.

2.   Using a batch size of 1, called **Stochastic Gradient Descent (SGD)**. In this case, only a single data point is used to calculate an estimate of the gradient. Thus, the estimate is very noisy, and we are not guaranteed to find a minimum (local or global). However, SGD still performs very well in practice and allows for online learning. It also turns out that having noisy estimates of the gradient acts as a form of **regularisation** which can prevent over-fitting. Finally, because we are performing gradient descent with one example at a time, we need much less memory. Not having enough memory can be a significant issue when using BGD.

3.   Using a batch size of $m < n$ is called **Mini-batch Gradient Descent** and is a compromise between batch and stochastic gradient descent. We use $m$ examples to calculate an estimate of the gradient. Thus, we still have *some* noise in the gradient estimate, we can tune the batch size to make good use of memory, and the variance of the gradient estimate is greatly reduced - which leads to better convergence to local or global minima.

In deep learning, we almost always use mini-batch gradient descent. However, it is often referred to simply as SGD.

### Optional extra reading: choosing a batch size

You might be wondering how one chooses which batch size to use for a given problem.

One approach is to choose as big a batch size as possible. The reason we might want to do this is that a larger batch size means that our model will train more quickly. This speedup is because modern computer hardware, especially GPUs, are designed with parallelism in mind. In other words, by having a larger batch size, we can take better advantage of our hardware. So in practice, we often choose the largest batch size that will fit into memory.

[According to some research](https://arxiv.org/pdf/1803.09820.pdf), using a larger batch size allows you to select a higher learning rate - which means that your model will train faster. On the other hand, choosing too large a learning rate can lead to not converging to a minimum. So, we still have to tune the batch size.

You can read more about these issues in this [blog post](https://blog.janestreet.com/does-batch-size-matter/) as well as the paper linked above — the paper, in particular, discusses strategies for choosing various hyper-parameters related to optimisation in deep learning.

Let's look at the one dimensional straight-line aproximation example from above and see how the batch size affects the calculated gradient. In the plot below, we see the function $\green{f(x)}$, the true tangent line $\blue{df(x)}$, the true gradient $\blue{\tilde{f'}(x_0)}$, the calculated tangent line $\red{\tilde{df}(x)}$ and the calculated gradient $\red{\tilde{f'}(x_0)}$. In this case, we get the calculated tangent line $\red{\tilde{df}(x)}$ and gradient $\red{\tilde{f'}(x_0)}$ by using *batch_size* number of datapoints to calculate an estimate.

Change the values for $x_0$ and *batch_size* and see what you can learn.

```python
#@title Helper functions (RUN ME) (double click to unhide/hide the code)
def f(x):
  return -np.cos(x)

def tangent_f(x):
  return np.sin(x)

def df(x, x_0):
  return tangent_f(x_0) * (x - x_0) + f(x_0)

def perpindicular_unit_f(x_0):
  slope_f = tangent_f(x_0)
  y_0 = f(x_0)

  x_1 = slope_f / np.sqrt(2) + x_0
  y_1 = -x_1 / slope_f + y_0 + x_0 / slope_f

  return [[x_0, x_1], [y_0, y_1]]

def noisy_df(x, x_0, noisy_x_0):
  return tangent_f(noisy_x_0) * (x - x_0) + f(x_0)

def noisy_perpindicular_unit_f(x_0, noisy_x_0):
  slope_f = tangent_f(noisy_x_0)
  y_0 = f(x_0)

  x_1 = slope_f / np.sqrt(2) + x_0
  y_1 = -x_1 / slope_f + y_0 + x_0 / slope_f

  return [[x_0, x_1], [y_0, y_1]]

def interactive_noisy_gradient_visual(x_0, noisy_x_0):
  # change the fontsize for better visibility
  init_size = plt.rcParams["font.size"] # store initial font size
  plt.rcParams.update({'font.size': 22}) # update the size

  plt.figure(figsize=(12, 8))

  x = np.linspace(-np.pi, 2 * np.pi)
  f_x = f(x)
  y_0 = f(x_0)

  # plot f(x)
  plt.plot(x, f_x, label=r"$f(x)$", color="green")

  # add a point showing where x_0 falls on f(x)
  plt.plot(x_0, f(x_0), marker="o", color="black")

  # plot the tangent line to f(x) at x_0
  plt.plot(x, df(x, x_0), linestyle="--", color="cornflowerblue", label=r"$df(x)$")

  # drop a vertical line from x_0
  plt.plot([x_0, x_0], [f(x_0), -3.1], color="silver")

  # plot the noisy tangent line to f(x) at x_0
  plt.plot(x, noisy_df(x, x_0, noisy_x_0), linestyle="--", color="red", label=r"$\widetilde{df}(x)$")

  # plot the normal vector to the tangent
  [[x_0, x_1], [y_0, y_1]] = perpindicular_unit_f(x_0)
  plt.plot([x_0, x_1], [y_0, y_1], color="dimgray")

  # plot the positive direction of change vector
  dx = x_1 - x_0
  dy = 0 # y_1 - y_1
  arrow = plt.arrow(
      x_0, y_1, dx, dy,
      color="blue", label=r"$f'(x_0)$",
      lw=3, head_width=np.abs(x_1 - x_0)/10, length_includes_head=True
  )
  plt.plot([x_0, x_1], [y_1, y_1], color="blue", label=r"$f'(x_0)$")

  # plot the noisy normal vector to the tangent
  [[noisy_x_0, x_1], [noisy_y_0, y_1]] = noisy_perpindicular_unit_f(x_0, noisy_x_0)
  plt.plot([x_0, x_1], [y_0, y_1], color="dimgray")

  # plot the noisy positive direction of change vector
  dx = x_1 - x_0
  dy = 0 # y_1 - y_1
  arrow = plt.arrow(
      x_0, y_1, dx, dy,
      color="red", label=r"$f'(x_0)$",
      lw=3, head_width=np.abs(x_1 - x_0)/10, length_includes_head=True
  )
  plt.plot([x_0, x_1], [y_1, y_1], color="red", label=r"$\widetilde{f'}(x_0)$")

  plt.legend(loc="upper left")
  plt.xlim(-3.1, 6.2)
  plt.ylim(-3.1, 3.1)
  plt.xlabel(r"$x_0$")
  plt.show()

  # reset to initial font size
  plt.rcParams.update({'font.size': init_size})

def interactive_batch_size_visual(x_0, batch_size):
  np.random.seed(0)
  noisy_x_0 = x_0 + np.mean(np.random.normal(loc=0, scale=0.5, size=batch_size))
  interactive_noisy_gradient_visual(x_0, noisy_x_0)
```

```python
#@title Double click to unhide/hide the code {run: "auto"}
x_0 = 2.7 #@param {type:"slider", min:-3.1, max:6.2, step:0.01}
batch_size = 20 #@param {type:"slider", min:1, max:256, step:1}

interactive_batch_size_visual(x_0, batch_size)
```

**Exercise:** What are the general relationships you notice between the accuracy of the calculated gradient $\red{\tilde{f'}(x_0)}$ and (a) the batch size and (b) the funciton $\green{f(x)}$?

### Implementing SGD

SGD is incredibly simple to implement, and as you'll see later can perform very well!

```python
def SGD_update(params, grads, states, hyper_params):
  # hyper-param typical values: learning_rate=0.01
  # SGD doesn't have any state, however, the algorithms
  # we will look at later do!
  for param, grad in zip(params, grads):
    param.assign_sub(hyper_params['lr'] * grad)
```

And now we can visualize each step of the SGD optimization:

**Excercise:** Take a look at the following code and make sure that you understand it. Do you see how the code relates to the pseudo-code and equation above? For example, where do you find `calc_grad`? Once you understand the code, you should hide it again to make it easier to change the sliders and see the results.

```python
#@title Helper functions (RUN ME) (double click to unhide/hide the code)

def optimize_banana(update_func, params, states, hyper_params):
  # plot the loss surface, minimum value and starting point
  X, Y, Z = gen_2d_loss_surface(rosenbrock_banana)
  fig, ax = make_contour_plot(X, Y, Z)
  ax.plot(1, 1, 'r*', ms=30, label='minimum')
  ax.plot(start_x, start_y, 'b*', ms=20, label='start')

  for epoch in range(epochs):
    with tf.GradientTape() as tape:
      # we are trying to minimize the output of the 🍌 func
      loss = rosenbrock_banana(x, y)
    # calculate the gradients of the loss with respect to the params
    grads = tape.gradient(loss, params)

    # save the old x and y values for the plot
    old_x = params[0].numpy()
    old_y = params[1].numpy()

    # update the parameters using SGD
    update_func(params, grads, states, hyper_params)

    # plot the change in x and y for each update step
    ax.annotate('', xy=(x.numpy(), y.numpy()),
                xytext=(old_x, old_y),
              arrowprops={'arrowstyle': '->', 'color': 'k', 'lw': 1},
                   va='center', ha='center')

  ax.plot(x.numpy(), y.numpy(), 'g*', ms=20, label='end')
  ax.legend()

  fig.show()
```

```python
#@title Double click to unhide/hide the code {run: "auto"}
start_x = -1 #@param {type:"slider", min:-2, max:2, step:0.1}
start_y = 0.73334 #@param {type:"slider", min:-0.26666, max:1.0666, step:0.1}
learning_rate = 0.015 #@param {type:"slider", min:0, max:0.02, step:0.0005}
epochs = 150 #@param {type:"slider", min:1, max:150, step:1}

x = tf.Variable(start_x, dtype='float32')
y = tf.Variable(start_y, dtype='float32')
params = [x, y]
states = []
hyper_params = {"lr": learning_rate}

optimize_banana(SGD_update, params, states, hyper_params)
```

**Partner Excercise:** Tweak the starting position (x and y), as well as the learning rate and the number of epochs. See if you can get to the global minimum. What do you notice about the behaviour of SGD in the 🍌 valley?

## SGD with Momentum

Hopefully, you've noticed a bit of an issue with SGD. When the gradient is small (i.e. in the 🍌) the changes to the parameters become very small and progress towards the minimum is very slow. One solution to this problem is to add a momentum term to our optimization step:

\begin{align}
\Delta \mathbf{θ} &= \gamma \Delta \mathbf{θ} + \eta \nabla_\mathbf{θ} J(\mathbf{θ}) \\
\mathbf{θ} &= \mathbf{θ} − \Delta \mathbf{θ}
\end{align}

where $\Delta \mathbf{θ}$ is the change in parameters $\mathbf{θ}$ at each step and is made up of a mixture between the gradients at a given step and the change from the previous step. $\gamma$ is called the *momentum* term, and $\eta$ is called the learning rate, as before.

The reason that this method is called *momentum* is that we can compare it colloquially to SGD as follows:

> Gradient descent is a person walking down a hill. They follow the steepest path downwards; their progress is slow, but steady. Momentum is a heavy ball rolling down the same hill. The added inertia acts both as a smoother and an accelerator, dampening oscillations and causing it to barrel through narrow valleys, small humps and local minima.

In other words, **the momentum term speeds up optimisation if the direction of change stays more or less the same and reduces oscillations when the direction of change goes back and forth**.

We can also describe momentum with some simple pseudo-code:

```
change = 0
for each epoch:
  grad = calc_grad(loss_func, data, params)
  change = momentum * change + learning_rate * grad
  params = params - change
```

### Implementing Momentum

```python
def momentum_update(params, grads, states, hyper_params):
  # hyper-param typical values: learning_rate=0.01, momentum=0.9
  changes = states['changes']
  for param, grad in zip(params, grads):
    changes[param].assign(hyper_params['momentum'] * changes[param] +
                          hyper_params['lr'] * grad)
    param.assign_sub(changes[param])
```

```python
#@title Double click to unhide/hide the code{run: "auto"}
start_x = -1 #@param {type:"slider", min:-2, max:2, step:0.1}
start_y = 0.73334 #@param {type:"slider", min:-0.26666, max:1.0666, step:0.1}
learning_rate = 0.01 #@param {type:"slider", min:0, max:0.02,step:0.0005}
momentum = 0.8 #@param {type:"slider", min:0, max:0.99, step:0.01}
epochs = 50 #@param {type:"slider", min:1, max:150,step:1}

x = tf.Variable(start_x, dtype='float32')
y = tf.Variable(start_y, dtype='float32')
params = [x, y]
changes = {param: tf.Variable(0., dtype='float32') for param in params}
hyper_params = {"lr": learning_rate, "momentum": momentum}
states = {"changes": changes}

optimize_banana(momentum_update, params, states, hyper_params)
```

**Partner Excercise:** Play around with the various parameters and see if you can get to the minimum. Compare the performance of momentum in the 🍌 to that of SGD.

**Partner Excercise:** Why do we see oscillations for high enough momentum values?

**Partner Excercise:** Do you see any relationship between the amount of momentum and the learning rate?

## RMSProp (Root Mean Square Propagation)

So far, we have chosen a learning rate $\eta$, and it has been multiplied by the whole gradient. Thus, each element in the gradient vector has had the same learning rate applied to it at every step. However, we can imagine that:

1.   Each weight might not need to vary by the same amount.
2.   The amount we want to change each parameter might change throughout the optimisation process.

**Excercise:** can you think of examples of when these two cases might apply?

RMSProp is a method that addresses these issues. It can be described with the following formulae:

\begin{align}
\mathbf{v} &= \gamma \mathbf{v} + (1 - \gamma) (\nabla_\mathbf{θ} J(\mathbf{θ}))^2 \\
\mathbf{θ} &= \mathbf{θ} − \frac{\eta}{\sqrt{\mathbf{v}} + \mathbf{\epsilon}} \nabla_\mathbf{θ} J(\mathbf{θ})
\end{align}

where each element of $\mathbf{v}$ is an estimate of the square of the gradient for a specific parameter, calculated using a rolling average, $\gamma$ is a forgetting factor for the rolling average, and $\mathbf{\epsilon}$ is a small number added for numerical stability.

In words, RMSProp is scaling down the learning rate for each gradient by rolling average of the most recent gradients for that parameter. Importantly **each parameter has its own learning rate, which changes over time**.

We can also describe RMSPRop using pseudo-code:

```
average = 0
for each epoch:
  grad = calc_grad(loss_func, data, params)
  average = gamma * average + (1 - gamma) * pow(grad, 2)
  params = params - learning_rate / sqrt(average) * grad
 ```

### Implementing RMSProp

```python
def RMSProp_update(params, grads, states, hyper_params):
  # hyper-param typical values: learning_rate=0.001, gamma=0.9, eps=1e-8
  averages = states['averages']
  for param, grad in zip(params, grads):
    averages[param].assign(hyper_params['gamma']*averages[param] +
                           (1 - hyper_params['gamma'])*tf.math.pow(grad, 2))
    param.assign_sub(hyper_params['lr']/(tf.sqrt(averages[param]) + hyper_params['eps'])*grad)
```

```python
#@title Double click to unhide/hide the code {run: "auto"}
start_x = -1 #@param {type:"slider", min:-2, max:2, step:0.1}
start_y = 0.73334 #@param {type:"slider", min:-0.26666, max:1.0666, step:0.1}
learning_rate = 0.01 #@param {type:"slider", min:0, max:0.02,step:0.0005}
gamma = 0.9 #@param {type:"slider", min:0.01, max:0.99, step:0.01}
epochs = 150 #@param {type:"slider", min:1, max:150,step:1}

x = tf.Variable(start_x, dtype='float32')
y = tf.Variable(start_y, dtype='float32')
params = [x, y]
averages = {param: tf.Variable(0., dtype='float32') for param in params}
states = {"averages": averages}
hyper_params = {"lr": learning_rate, "gamma": gamma, "eps": 1e-8}

optimize_banana(RMSProp_update, params, states, hyper_params)
```

**Partner Excercise:** Play around with the various parameters and see if you can get to the minimum. Compare the performance of RMSProp with momentum and SGD, particularly in the 🍌.

## Adam (Adaptive moment estimation)

Adam combines the ideas of momentum and adaptive learning rates that we have explored above. More specifically, in addition to storing the rolling averages of the *squared* gradients and using them to control the learning rate for each parameter, like RMSProp, it also stores the rolling averages of the gradients themselves and uses them like momentum. Mathematically, we can describe Adam as follows:

\begin{align}
\mathbf m &= β_1 \mathbf{m} + (1 − β_1)  \nabla_\mathbf{θ} J(\mathbf{θ}) \\
\mathbf v &= β_2 \mathbf{v} + (1 − β_2) (\nabla_\mathbf{θ} J(\mathbf{θ}))^2 \\
\mathbf{\hat{m}} &= \frac{\mathbf{m}}{1 − β_1^t} \\
\mathbf{\hat{v}} &= \frac{\mathbf{v}}{1 − β_2^t} \\
\mathbf{θ} &= \mathbf{θ} − \frac{\eta}{\sqrt{\mathbf{\hat{v}}} + \mathbf{\epsilon}} \mathbf{\hat{m}}
\end{align}

where $\mathbf{m}$ is a rolling estimate of the gradient, $\mathbf{v}$ is a rolling estimate of the squared gradient, and $\mathbf{\hat{m}}$ and $\mathbf{\hat{v}}$ are bias-corrected estimates. As before $\eta$ and $\epsilon$ are the learning rate and a numerical stability term.

The name Adam comes from the fact that $\mathbf{m}$ and $\mathbf{v}$ are estimates of the first moment (the mean) and the second moment (the uncentered variance) of the gradient. The reason that we need bias corrected versions of the estimates is that they are initialized to be zero vectors, which means that when the training starts, the estimates are biased towards zero. This problem is especially relevant when $β_1$ and $β_2$ are close to 1.

The pseudo-code for Adam is slightly longer than the other methods we've looked at but should be reasonably simple to understand. If something doesn't make sense, go back and look at momentum and RMSProp.

```
first_moment = 0
second_moment = 0
t = 0
for each epoch:
  t = t + 1
  grad = calc_grad(loss_func, data, params)
  first_moment  = beta1 * first_moment  + (1 - beta1) * grad
  second_moment = beta2 * second_moment + (1 - beta2) * pow(grad, 2)
  first_moment_unbiased  = first_moment  / (1 - pow(beta1, t))
  second_moment_unbiased = second_moment / (1 - pow(beta2, t))
  params = params - (learning_rate / sqrt(second_moment_unbiased)) * first_moment_unbiased
```

### Implementing Adam

Implementing Adam is left as an excercise, we've set up the schafolding for you but you'll have to get your hands dirty! So skip this section for now, and come back later!

```python
def Adam_update(params, grads, states, hyper_params):
  # hyper-param typical values: learning_rate=0.001, beta1=0.9, beta2=0.999, eps=1e-8
  t = states["t"]
  t.assign_add(1.0)
  first_moments = states["first_moments"]
  second_moments = states["second_moments"]

  for param, grad in zip(params, grads):
    # come back here later!

```

```python
#@title Double click to unhide/hide the code {run: "auto"}
start_x = -1 #@param {type:"slider", min:-2, max:2, step:0.1}
start_y = 0.73334 #@param {type:"slider", min:-0.26666, max:1.0666, step:0.1}
learning_rate = 0.01 #@param {type:"slider", min:0, max:0.02,step:0.0005}
beta1 = 0.8 #@param {type:"slider", min:0.01, max:0.99, step:0.01}
beta2 = 0.8 #@param {type:"slider", min:0.01, max:0.999, step:0.001}
epochs = 150 #@param {type:"slider", min:1, max:150,step:1}

x = tf.Variable(start_x, dtype='float32')
y = tf.Variable(start_y, dtype='float32')
params = [x, y]
first_moments = {param: tf.Variable(0., dtype='float32')
                 for param in params}
second_moments = {param: tf.Variable(0., dtype='float32')
                                     for param in params}
t = tf.Variable(0., dtype='float32')
states = {"first_moments": first_moments,
          "second_moments": second_moments, "t": t}
hyper_params = {"lr": learning_rate, "beta1": beta1, "beta2": beta2, "eps": 1e-8}

optimize_banana(Adam_update, params, states, hyper_params)
```

## Learning Rate Decay

One of the advantages of methods such as Adam and RMSProp is that the effective learning rate for each parameter is no longer a constant during training. This behaviour is desirable because we often want to reduce the learning rate as we near the global minimum so that we do not overshoot it. Another simple strategy for solving this problem is *learning rate decay*. With learning rate decay, we progressively reduce the learning rate during the training process. For example, we might decay our learning rate as follows:

\begin{equation}
  \eta = \eta \times \frac{1}{1 + \delta \times t}
\end{equation}

where $\eta$ is the learning rate, $\delta$ is the decay rate, and $t$ is the number of the current training epoch. This method will give us a learning rate that looks something like this:

```python
initial_learning_rate = 0.01
epochs = 100
decay_rate = initial_learning_rate/epochs
learning_rates = [initial_learning_rate]

for t in range(epochs):
  previous_learning_rate = learning_rates[t]
  learning_rate_at_t = previous_learning_rate * 1/(1 + decay_rate * t)
  learning_rates.append(learning_rate_at_t)

plt.plot(learning_rates)
plt.show()
```

In practice, while we do often use the simple decay scheme shown above, it is also common to use more complicated *learning rate schedules* such as exponential decay and step decay:

```python
# exponential decay
initial_learning_rate = 0.01
epochs = 100
decay_rate = 0.01
learning_rates = []

for t in range(epochs):
  learning_rate_at_t = initial_learning_rate * np.exp(-decay_rate * t)
  learning_rates.append(learning_rate_at_t)

plt.plot(learning_rates)
plt.show()
```

```python
# step decay
initial_learning_rate = 0.01
epochs = 100
epochs_wait = 10
decay_rate = 0.5
learning_rates = []

for t in range(epochs):
  exponent = np.floor((1 + t) / epochs_wait) # this part gives us the "steps"
  learning_rate_at_t = initial_learning_rate * np.power(decay_rate, exponent)
  learning_rates.append(learning_rate_at_t)

plt.plot(learning_rates)
plt.show()
```

Any of these learning rate decay methods are compatible with the optimisation methods described above. The choices of whether or not to use learning rate decay and if so, which method to use, are hyper-parameters that we can tune.

## Putting it all into practice

Putting all of this into practice is very simple! Keras provides us with a high-level API that makes using any of the optimization methods or learning rate schedules as easy as changing a single line of code. Of course, if you are defining a custom training loop using `tf.GradientTape` then the code we've used above can easily be converted to work for any model and dataset.

One of the advantages of using Keras is that it provides us with reasonable default values for all of the hyper-parameters of the optimization algorithms. Let's use Keras to train a simple MLP on FashionMNIST so that we can compare the optimization algorithms we've looked at in a more realistic setting.

As a quick reminder, FashionMNIST contains 28x28 grayscale images from 10 different types of clothing. Let's take a quick look:

```python
fashion_mnist = tf.keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

text_labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)

    img_index = np.random.randint(0, 50000)
    plt.imshow(train_images[img_index], cmap="gray_r")
    plt.xlabel(text_labels[train_labels[img_index]])
```

Before we train on the data, we want to do a little pre-processing. We won't go into detail about how and why we are doing the pre-processing, but if you want to know more, you should take a look at the *Deep Feedforward Networks* practical.

```python
batch_size = 128

train_ds = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
# Divide image values and cast to float so that they end up as a floating point number between 0 and 1
train_ds = train_ds.map(lambda x, y: (tf.cast(x, tf.float32) / 255.0, tf.cast(y, tf.int32)))
# Shuffle the examples.
train_ds = train_ds.shuffle(buffer_size=batch_size * 10)
# Now "chunk" the examples into batches
train_ds = train_ds.batch(batch_size)

test_ds = tf.data.Dataset.from_tensor_slices((test_images, test_labels))
test_ds = test_ds.map(lambda x, y: (tf.cast(x, tf.float32) / 255.0, tf.cast(y, tf.int32)))
test_ds = test_ds.batch(batch_size)
```

Now let's define a simple MLP:

```python
def build_mlp():
  model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
  ])

  return model

model = build_mlp()

model.summary()
```

And finally, lets train the MLP using a few different optimizers and compare the results:

```python
#@title Helper functions (double click to unhide/hide the code)

def make_loss_plots(losses):
  plt.close()
  for label, loss_vals in losses.items():
    plt.plot(loss_vals, label=label)
  plt.legend()
  plt.show()
```

```python
losses = {}
tf.random.set_seed(0)

# SGD
model = build_mlp()
model.compile(optimizer='sgd',
              # we can use the string shortcut if we
              # don't want to change the hyper-parameters
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
sgd_hist = model.fit(train_ds, epochs=5,
          validation_data=test_ds)

losses['SGD'] = sgd_hist.history["val_loss"]
make_loss_plots(losses)

# Momentum
model = build_mlp()
model.compile(optimizer=tf.keras.optimizers.SGD(lr=0.01, momentum=0.9),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
momentum_hist = model.fit(train_ds, epochs=5,
          validation_data=test_ds)

losses['Momentum'] = momentum_hist.history["val_loss"]
make_loss_plots(losses)

# RMSProp
model = build_mlp()
model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.001, rho=0.9),
              # rho is the symbol used for the forget factor in Keras
              # we used gamma (γ) in this practical
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
rmsprop_hist = model.fit(train_ds, epochs=5,
          validation_data=test_ds)

losses['RMSProp'] = rmsprop_hist.history["val_loss"]
make_loss_plots(losses)

# Adam
model = build_mlp()
model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
adam_hist = model.fit(train_ds, epochs=5,
          validation_data=test_ds)

losses['Adam'] = adam_hist.history["val_loss"]
make_loss_plots(losses)

# SGD with learning rate decay
model = build_mlp()
model.compile(optimizer=tf.keras.optimizers.SGD(lr=0.5, momentum=0.0, decay=0.01),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
sgd_decay_hist = model.fit(train_ds, epochs=5,
          validation_data=test_ds)

losses['SGD with decay'] = sgd_decay_hist.history["val_loss"]
make_loss_plots(losses)

# Momentum with learning rate decay
model = build_mlp()
model.compile(optimizer=tf.keras.optimizers.SGD(lr=0.5, momentum=0.9, decay=0.01),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
sgd_decay_hist = model.fit(train_ds, epochs=5,
          validation_data=test_ds)

losses['Momentum with decay'] = sgd_decay_hist.history["val_loss"]
make_loss_plots(losses)
```

**Exercise:** Play with the parameters of the various optimizers and see how that affects this particular problem.

**Note:** There is an element of randomness each time we train our models; this means that we should be careful when making judgements about which methods are best for this problem.

## Optional extra reading: second-order methods

All of the optimization methods we've looked at in this practical are what we call first order methods: they calculate a straight line estimate of the gradient and take a step in that direction. However, this straight line estimate throws away some useful information about the curvature of our loss surface. In other words, a first order method gives us information about whether or not our loss is increasing or deacresing in a given direction, but a second order method tells us how much it is increasing or decreasing. With this extra information, second order methods can find a minima in fewer steps, and have less trouble with getting stuck in saddle points.

Unfortunately, the benefits of second-order methods come at a cost. As you might have guessed from the names, a first-order method calculates the first derivative of our loss function, while a second-order method also calculates the second derivative. Calculating these second derivatives (the Hessian matrix) is computationally expensive, which means that second-order methods are often slower in practice. For this reason, we usually do not see second-order methods in deep learning. However, fast approximations for second-order methods are an active area of research!

Examples of second-order methods include [Newton's method](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization) and [BFGS](https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm).

## Conclusion:

### What are optimizers, and how do we use them?

Optimizers are algorithms that try to find the best item when compared to other items using a given method of comparison (or metric). In deep learning, we use them to try to find the best values for our weights and biases.

### What method should you use?

There are **no hard rules** for which methods you should use. It will always depend on your particular model and dataset. However, some guidelines are worth keeping in mind:

1. Adam typically works very well in a large number of settings and is usually a good first choice.
2. RMSProp can often outperform Adam for RNNs as well as in RL. If you are working in either of these domains, then it might be worth trying RMSProp.
3. SGD and SGD with momentum often work just as well as more sophisticated methods like Adam. Don't think that they aren't worth trying out just because they are simple.

## Tasks

1. **[All]** Combine the implementations for momentum and RMSProp to implement Adam. Experiment with the parameters of Adam and compare it to the previous methods we have looked at.
2. **[All]** Augment any of the optimization methods with learning rate decay. You can choose any of the learning rate decay methods. Play around with the parameters. Compare the performance of the optimization method with and without learning rate decay.
3. **[Optional, Intermediate]** Implement the other two learning rate decay methods and compare all three decay methods with one another.
4. **[Optional, Advanced]** Implement Nesterov Momentum. You can read more about it [here](http://ruder.io/optimizing-gradient-descent/).
5. **[Optional, Advanced]** Implement Nadam. You can read more about it [here](http://ruder.io/optimizing-gradient-descent/) - it also works particularly well in deep RL.

## Extra resources

* A [blog post](http://fa.bianp.net/teaching/2018/eecs227at/gradient_descent.html) from Fabian Pedregosa on gradient descent [**Highly Recommended**].
* [Distil.pub post](https://distill.pub/2017/momentum/) by Gabriel Goh on why momentum works [**Highly Recommended**].
* [Sebastian Ruder's blog](http://ruder.io/optimizing-gradient-descent/) on gradient descent algorithms.
* Deep Dive into Deep Learning chapter on [Optimization Algorithms](http://d2l.ai/chapter_optimization/index.html).
* Keras optimizer [docs](https://keras.io/optimizers/).


# Gas compressibility
We now investigate correlations for gas compressibility factors $Z$.
The key sources are:
1. Ding-Yu Peng & Donald B. Robinson, **A New Two-Constant Equation of State**, *Ind. Eng. Chem. Fundam., Vol. 15, No. 1*, 1976
[link](https://pubs.acs.org/doi/abs/10.1021/i160057a011)
1. Azad Jarrahian, Babak Aghel, Ehsan Heidaryan, **On the viscosity of natural gas**, *Fuel 150 Pp 609–618*, 2015 [link](https://www.sciencedirect.com/science/article/pii/S0016236115001970)

## Extensions to non-ideal gases
We start by decomposing pressure into a repulsive and attractive component motivated by non-ideal kinetic theory and an inter-molecular attractive force.

$$ P = P_R + P_A = \frac{RT}{V-b} - \frac{a}{g(V)} $$
where $b$ is related to the size of the 'hard' molecules (independent of $T$), $a$ is a representation of the inter-molecular attraction (and is a function of $T$). $g(V)$ is a function of the molar volume, $V$. Peng and Robinson suggest that:
$$P = \frac{RT}{V-b} - \frac{a(T)}{V(V+b)+b(V-b)} ~~~~~~~~~~~~~~~\textsf{Eqn: PR}$$

>  *Compare Van der Waal's forces which can be written as: $\left (P+\frac{n^2a}{V^2} \right ) \left (V-nb \right )=nRT $.
This has the analytic solution $a=\frac{27n^2R^2T_c^2}{64P_c}(4-3n^2)$ and $b=\frac{RT_c}{8P_c}(4-3n^2)$. For $n=1$ we have $a=0.421875\frac{R^2T_c^2}{P_c}$ and $b=0.125\frac{RT_c}{P_c}$; compare the PR values obtained below.*

We introduce $Z = \frac{PV}{RT}$ we can write:
$$Z^3 - (1-B)Z^2 + (A-3B^2 -2B) Z - (AB -B^2 - B^3) = 0$$
where
$ A = \frac{aP}{(RT)^2}, B = \frac{bP}{RT} $

Of course, we can write this as:
$$
Z^3 - Z^2 + [ B Z^2 + (A-3B^2 -2B) Z - (AB -B^2 - B^3) ] = 0
$$
which is consistant with the statement that $Z = 1$ for an ideal gas (because $ a = b = 0$ for an *ideal* gas).

At the critical point we have:
$$ \left. \frac{\partial P}{\partial V} \right |_T = 0 ~~\textsf{and}~~  \left. \frac{\partial^2 P}{\partial V^2} \right |_T = 0$$ which leads to:
$$
a(T_c) = 0.45724 \frac{R^2T_c^2}{P_c} , ~~b(T_c) = 0.07780 \frac{RT_c}{P_c} ~~\textsf{and}~~ Z_c = 0.307
$$

At other temperatures it is reasonable to write: $b(T) = b(T_c)$ (a constant)
and $a(T) = a(T_c)\times\alpha(T_r,\omega)$

where $T_r = T/T_c$ is the reduced temperature and $\omega$ is an 'acentric' factor; clearly we need $\alpha(1,\omega) \equiv 1 ~~~~\forall~ \omega$.

Comparison with experimental data gives:
\begin{align}
\sqrt{\alpha} = 1 + \kappa \left(1 -T_r^{\frac{1}{2}}\right)
\end{align}
with:
$$\kappa = 0.37464 + 1.54226\omega - 0.26992 \omega^2 $$
Values of $\omega$ can be found from this graph taken from Peng and Robinson (Figure 2).
_(This data is also available in tabular form at: [Reid et al.](http://www.kaylaiacovino.com/Petrology_Tools/Critical_Constants_and_Acentric_Factors.htm))_

This derivation shows __the Law of Corresponding States__: all fluids behave similarly when described in terms of their reduced temperature and pressure ($T_r$ and $P_r$).

__Status__ 28/4/2020
-   Summarised the Peng and Robinson Paper.
-   Verified the math and aritmetic at the critical point.
   -      included notes on Van der Waal's equation
-   Computed the compressibilities of methane and ethane (correctly?).
Possible Extensions and Future Work.
    -   Talk about mixtures (including mixtures with $CO_2$ and $H_2S$).
    Wichert, E. and Aziz, K. 1972. **Calculate Z's for Sour Gases.** Hydrocarbon Processing 51 (May): 119–122.
    -   Talk about additional correlations.
    -   Add sheet about equations of state and gas viscosity $\mu_g$.
    -   Clean-up the code and comments (in progress)

## Compressibility of Methane and Ethane for the Frigg pipeline conditions.

```python
import math
from scipy import optimize

def PRZP3(Z, XARGS =[0.4572355289212813, 0.07779607390386772, 1]):
    """
    Evaluate the Peng-Robinson polynomial for the non-dimensional compressibility (Z):
    P(Z,A,B) = Z^3 -(1-B)Z^2 + (A-3B^2-2B)Z-(AB-B^2-B^3)
    Try:
    A=B=0 (the ideal gas case) => Z=0
    and
    PolyZ(0.307,0.45724,0.07780) =0 The coefficients in the PR paper evaluated at the critical point.
    """
    Acap=XARGS[0]; Bcap=XARGS[1]; DEBUG=XARGS[2]
    PRZ3 = Z**3 - (1-Bcap)*Z**2 + (Acap-3*Bcap**2 - 2*Bcap)*Z -( Acap*Bcap - Bcap**2 - Bcap**3)
    if DEBUG: print("Z=",Z," A=",Acap," B=",Bcap,"PRCubic=",PRZ3)
    return PRZ3
Acap=Bcap=0.0; Z=1.0
XARGS= [Acap, Bcap,1]
print("Ideal Gas (PR cubic should be 0.0):",PRZP3(1.0, XARGS))
Acap=0.4572355289212813 # optimal value x[1], from below
Bcap=0.07779607390386772 # optimal value x[2], from below
XARGS= [Acap, Bcap,1]
print("XARGS: ",XARGS[0],XARGS[1],XARGS[2])
print("Critical Point (PR cubic should be 0.0):",PRZP3(0.30740131,XARGS))

R = 8.314413 # The ideal gas constant
print("\nCompressibility of Methane")
#R. C. Reid, J. M. Prausnitz, & B. E. Poling, 1987, The Properties of Gases & Liquids, 4th Ed. New York: McGraw-Hill
#    http://www.kaylaiacovino.com/Petrology_Tools/Critical_Constants_and_Acentric_Factors.htm

omega = 0.0115              # acentric factor for methane
kappa = 0.37464+1.54226*omega-0.26992*omega**2
Tc = 191.15; Pc = 4.641E6   # Methane critical point properties
T= 313 ; P = 13.01E6        # Frigg pipeline conditions (Case 9) Frigg - MCP01
Tr = T/Tc; Pr = P/Pc        #Compute reduced temperature and pressure
print("Case 9: T =:",T,"K, P =",P,"Pa")
alpha = (1+kappa*(1-math.sqrt(Tr)))**2
Acap = (0.4572355289212813*Pr*alpha)/Tr**2 # Use improved values for A and B
Bcap = (0.07779607390386772*Pr)/Tr
zx=optimize.fsolve(PRZP3,[1], [Acap, Bcap,0],full_output=0)
print("The compressibility of methane at Frigg conditions is:", zx)
#
#
print("\nCompressibility of Ethane")
omega = 0.099              # acentric factor for ethane
kappa = 0.37464+1.54226*omega-0.26992*omega**2
Tc = 305.4; Pc = 4.88E6    # Ethane critical point properties
T= 313 ; P = 13.01E6       # Frigg pipeline conditions (Case 9) Frigg - MCP01
Tr = T/Tc; Pr = P/Pc       #Compute reduced temperature and pressure
print("Case 9: T =:",T,"K, P =",P,"Pa")
alpha = (1+kappa*(1-math.sqrt(Tr)))**2
Acap = (0.4572355289212813*Pr*alpha)/Tr**2   # Use improved values for A and B
Bcap = (0.07779607390386772*Pr)/Tr
zx=optimize.fsolve(PRZP3,1, [Acap, Bcap,0],full_output=0)
print("The compressibility of ethane at Frigg conditions is:", zx)
```

    Z= 1.0  A= 0.0  B= 0.0 PRCubic= 0.0
    Ideal Gas (PR cubic should be 0.0): 0.0
    XARGS:  0.4572355289212813 0.07779607390386772 1
    Z= 0.30740131  A= 0.4572355289212813  B= 0.07779607390386772 PRCubic= -3.5388358909926865e-15
    Critical Point (PR cubic should be 0.0): -3.5388358909926865e-15

    Compressibility of Methane
    Case 9: T =: 313 K, P = 13010000.0 Pa
    The compressibility of methane at Frigg conditions is: [0.83924289]

    Compressibility of Ethane
    Case 9: T =: 313 K, P = 13010000.0 Pa
    The compressibility of ethane at Frigg conditions is: [0.39788793]

#### Other resources
For methane: [USGS web page](https://pubs.usgs.gov/of/2005/1451/equation.html):
$\omega = 0.0115, T_c = 191.15K, P_c = 4.641 MPa$
The ideal gas constant $R = 8.314413 J mol^{-1}K^{-1}$

For enthalpy diagrams of other fluids see this [on-line worksheet](https://www.questconsult.com/thermo/enthalpy_pressure_diagram.html):

This data is available in tabular form at: [Reid et al.](http://www.kaylaiacovino.com/Petrology_Tools/Critical_Constants_and_Acentric_Factors.htm)

Taken from:
R. C. Reid, J. M. Prausnitz, and B. E. Poling, 1987, The Properties of Gases and Liquids, 4th Ed. New York: McGraw-Hill

 #### Other relevant web sites
1.  https://engineeringunits.com/compressibility-factor-calculator-z/
1.  https://petrowiki.org/Real_gases - poor
3.  https://checalc.com/solved/naturalgasZ.html
4.  https://pubs.usgs.gov/of/2005/1451/equation.html (Peng-Robinson equation of state)
5.  http://www.kaylaiacovino.com/Petrology_Tools/Critical_Constants_and_Acentric_Factors.htm
1.  https://engineeringunits.com/compressibility-factor-z-factor/ (Deals with mixtures e.g. Kay's rule - worth a look.)
1.  http://www.peacesoftware.de/einigewerte/methan_e.html (Compressibility factors - and other thermodynamic properties - for a few interesting gases.)

## Notes on Peng Robinson coefficients at the critical point

First let's evaluate the basic polynomial:
$ PR(Z) \equiv Z^3 - Z^2 + [ B Z^2 + (A-3B^2 -2B) Z - (AB -B^2 - B^3) ] = 0 $

```python
def PolyZ(Z, A, B):
    """
    Evaluate the Peng-Robinson polynomial for the non-dimensional compressibility (Z):
    P(Z,A,B) = Z^3 -(1-B)Z^2 + (A-3B^2-2B)Z-(AB-B^2-B^3)
    Validation tests:
    A=B=0 (the ideal gas case) => Z=0
    and
    PolyZ(0.307,0.45724,0.07780) =0 The coefficients in the PR paper evaluated at the critical point.
    """
    PZ = Z**3 - (1-B)*Z**2 + (A-3*B**2 - 2*B) *Z -( A*B - B**2 - B**3)
    print("Z=",Z," A=",A," B=",B,"ANS=",PZ)
    return PZ
print("Ideal Gas ",PolyZ(1,0,0))
print("Critical Point ",PolyZ(0.307,0.45724,0.07780))
```

    Z= 1  A= 0  B= 0 ANS= 0
    Ideal Gas  0
    Z= 0.307  A= 0.45724  B= 0.0778 ANS= -2.691488000014508e-06
    Critical Point  -2.691488000014508e-06

#### Notes on PR coefficients at the critical point - the first derivative

We differentiate Eqn:PR with respect to V:

\begin{align}
\left . \frac{\partial P}{\partial V} \right |_T &= - \frac{RT}{(V-b)^2} + 2a \frac{(V + b)}{(V^2+2bV-b^2)^2}\\
\end{align}

which can be written in normalised form as:
\begin{align}
- PositiveFn \times \frac{P^4}{(RT)^5}\left .\frac{\partial P}{\partial V} \right |_T = \{ 2A (Z+B)(Z-B)^2 - (Z^2 + 2BZ -B^2)^2 \}
\end{align}

Evaluating this at the critical point we have:
$
\left .\frac{\partial P}{\partial V} \right |_T = 0
$
which is satisfied by the coefficients and value given above.

```python
def DPolyZ(Z,A,B):
    """
    This function evaluates the RHS of the above equation for DP/DV.
    Usage: DPolyZ(Z-compressibility, A, B - non-dimensional constants from Peng Robinson)
    Validation test: It should be zero at the critical point.
    """
    DPZ = 2*A*(Z+B)*(Z-B)**2 - (Z**2 + 2*B*Z - B**2)**2
    return DPZ
print("Gradient at Critical Point ",DPolyZ(0.307,0.45724,0.07780))
print("Gradient at Critical Point ",DPolyZ(3.07401e-01,4.57236e-01,7.77961e-02))
```

    Gradient at Critical Point  -7.684081830341971e-07
    Gradient at Critical Point  1.290313487653516e-08

#### Notes on PR coefficients at the critical point - the second derivative

We differentiate Eqn:PR **twice** wrt V:

\begin{align}
\left . \frac{\partial^2 P}{\partial V^2} \right |_T &= \frac{2RT}{(V-b)^3} - 2a \frac{(3V^2+6BV + 5b^2)}{(V^2+2bV-b^2)^3}\\
\end{align}

which can be written in normalised form as:
\begin{align}
- 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝐹𝑛 \times \frac{RT^2}{P^3}\left .\frac{\partial^2 P}{\partial V^2} \right |_T &=
2\left \{ -A \frac{3Z^2+6BZ+5B^2}{(Z^2+2BZ - B^2)^3}  + \frac{1}{(Z-B)^3} \right \}\\
&=\left \{ \frac{1}{(Z-B)^3} - A \frac{3Z^2+6BZ+5B^2}{(Z^2+2BZ - B^2)^3} \right \}
\end{align}

Evaluating this at the critical point we have:
$
\left .\frac{\partial^2 P}{\partial V^2} \right |_T = 0
$
which is satisfied by the coefficients and value given above.

```python
def D2PolyZ(Z,A,B):
    """A function to evaluate a multiple of D^2P/DV^2 of the PR equation"""
    D2PZ = 1/(Z-B)**3 - A*(3*Z**2+6*B*Z+5*B**2)/(Z**2+2*B*Z-B**2)**3
    return D2PZ
print("Second Gradient at Critical Point ",D2PolyZ(0.307,0.45724,0.07780),"That's quite small?")
print("Second Gradient at Critical Point ",D2PolyZ(3.07401e-01,4.57236e-01,7.77961e-02),"Very sensitive to rounding!")
```

    Second Gradient at Critical Point  0.04366953294452003 That's quite small?
    Second Gradient at Critical Point  -1.7917987406690372e-05 Very sensitive to rounding!

```python
from scipy import optimize
def func(x):
    """
    A function to evaluate a scaled function of P, DP/DV, D2P/DV2
    when written in the Peng-Robinson notation of:
    Z,A,B
    """
    DEBUG = 0 # 0 => FALSE
#First switch to normal (PR) notation
    Z=x[0]; A=x[1] ; B=x[2]
    F=[1.0,1.0,1.0] # Initialise to some easy to recognize value
#    print("Z,A,B: ",Z,A,B)
#Calculate P
    F[0] = Z**3 - (1-B)*Z**2 +(A-3*B**2-2*B)*Z - (A*B-B**2-B**3)
#Calculate DP/DV
    F[1] = 2*A*(Z+B)*(Z-B)**2 - (Z**2 + 2*B*Z - B**2)**2
#Calculate D2P/DV2
    F[2] = 1/(Z-B)**3 - A*(3*Z**2+6*B*Z+5*B**2)/(Z**2+2*B*Z-B**2)**3

    if DEBUG: print("Z,A,B: ",'{:12.5e}'.format(Z),'{:12.5e}'.format(A),'{:12.5e}'.format(B),
          "   F: ",'{:8.2e}'.format(F[0])," DP/DV:",'{:8.1e}'.format(F[1]),"D2P/DV2:",'{:8.1e}'.format(F[2]))

    return F

print("Validation - compressibility for an ideal gas:")
x=x0 = [1.0,0,0]
print("x: ",x, x[0],x[1],x[2])
x=x0 = [0.307,0.45724,0.07780]
print("Validation for PR values (close to critical point) x: ",x, x[0],x[1],x[2])

FN = func(x0)
print("                                   F: ",FN)

print("Behaviour at the critical point - confirms PR result (Eqns. 9,10,11)")
x=optimize.fsolve(func,x0,full_output=0)
print("optimize.fsolve [Z,A,B]",x,"[PR(Z),DP,D^2P]",func(x))
fstring = '{:8.2E}'
print("2.5 optimize.fsolve [Z,A,B]",x,"[PR(Z),DP,D^2P]",end=' ');
for i in range (0,3): print('{:8.2e}'.format(func(x)[i]), end=' ')
print()
print("4   optimize.fsolve [Z,A,B]",x,"[PR(Z),DP,D^2P]",func(x))
```

    Validation - compressibility for an ideal gas:
    x:  [1.0, 0, 0] 1.0 0 0
    Validation for PR values (close to critical point) x:  [0.307, 0.45724, 0.0778] 0.307 0.45724 0.0778
                                       F:  [-2.691488000014508e-06, -7.684081830341971e-07, 0.04366953294452003]
    Behaviour at the critical point - confirms PR result (Eqns. 9,10,11)
    optimize.fsolve [Z,A,B] [0.30740131 0.45723553 0.07779607] [PR(Z),DP,D^2P] [-3.5388358909926865e-15, 8.396061623727746e-16, -3.1874947126198094e-11]
    2.5 optimize.fsolve [Z,A,B] [0.30740131 0.45723553 0.07779607] [PR(Z),DP,D^2P] -3.54e-15 8.40e-16 -3.19e-11
    4   optimize.fsolve [Z,A,B] [0.30740131 0.45723553 0.07779607] [PR(Z),DP,D^2P] [-3.5388358909926865e-15, 8.396061623727746e-16, -3.1874947126198094e-11]

### Summary

These results show that the coefficients given in the Peng Robinson paper follow directly form the analysis of the gas behaviour around the critical point. Although the results are sensitive to exact values four decimal places for $b(T_cr)$ has a significant effect on the value of $\left . \frac{\partial ^2 P}{\partial V^2} \right |_T$

### CNGA compressibility factor

The California Natural Gas Association gives the following simple formula for the compressibility of natural gas:
$$
Z = \left [ 1+ \frac{3.44400\mathrm{E}5 \times \overline{P}_{psi~g} \times 10^{1.785\gamma}}{\overline{T}_R^{~3.825}} \right ]^{~-1}
$$
in field units(!):
$\overline{P}_{psi~g}$ -- average gas pressure in psig; $\overline{T}_R$ -- average gas temperature in $ ^\circ R$ and $\gamma$ -- specific gravity of the gas. This gives reasonable accuracy for pressures over 100psig ($\approx 0.69MPa$); below this pressure we can take $Z \equiv 1$.

Converting this to SI units, we obtain:
$$
Z =
\left [ 1+  \frac{344\,400}{6894.75728} \left ( \frac{5}{9}\right )^{3.825} \frac{  \overline{P}_{Pa~g} \times 10^{1.785\gamma}}{\overline{T}_{K}^{~3.825}} \right ]^{~-1}
  = \left [ 1+ \frac{ 5.273838529 \times\overline{P}_{Pa~g} \times 10^{1.785\gamma}}{\overline{T}_{K}^{~3.825}} \right ]^{~-1}
$$
where: $\overline{P}_{Pa~g}$ -- average gas (gauge) pressure in Pa; $\overline{T}_K$ -- average gas temperature in $K$ and $\gamma$ -- specific gravity of the gas.

```python
import math
from scipy import optimize
def PengRComp(T, P, omega = 0.0115,Tc = 191.15, Pc = 4.641E6, DEBUG=0):
    """
    A function to evaluate the compressibility of a gas using Peng Robinson.
    Default values are for methane.
    Usage: PengRComp(Temp[K], Press[Pa}, omega(acentric factor),Temp(crit pt), Pres(crit pt), DEBUG(0=>False))
    """

    kappa = 0.37464+1.54226*omega-0.26992*omega**2

    Tr = T/Tc; Pr = P/Pc        #Compute reduced temperature and pressure
    alpha = (1+kappa*(1-math.sqrt(Tr)))**2
    Acap = (0.4572355289212813*Pr*alpha)/Tr**2 # Use improved values for A and B
    Bcap = (0.07779607390386772*Pr)/Tr
    PRZ=optimize.fsolve(PRZP3,1,[Acap,Bcap,DEBUG],full_output=0)
    if DEBUG: print("PengRComp - omega: ",omega, "T: ", T," P: ", P," Z: ", PRZ)
    return PRZ
T= 313 ; P = 13.01E6
PRZX=PengRComp(T,P,DEBUG=0)
print(PRZX, "Should be: 0.83924289")

```

    [0.83924289] Should be: 0.83924289

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
# evenly sampled time at 200ms intervals
t = np.arange(-.1, 2.0, 0.05)
zero = t*0
# red dashes, blue squares and green triangles
print("Acap: ",Acap," Bcap: ", Bcap)
plt.plot(t, PRZP3(t,[Acap,Bcap,0]),t,zero,)
#plt.ylabel('PR polynomial')
#plt.xlabel('Compressibility - Z')
plt.show()
```

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
# evenly sampled time at 200ms intervals
t = np.arange(-1.0,1.0, 0.05)
TEMP = 313+50*t
PRTEMP=zero = t*0
for index in range (len(TEMP)):
    PRTEMP[index]=PengRComp(TEMP[index],P,DEBUG=0)
# red dashes, blue squares and green triangles
print("Acap: ",Acap," Bcap: ", Bcap)
plt.plot(TEMP, PRTEMP,TEMP,zero,)
plt.xlabel('Temperature (K)')
plt.ylabel('Compressibility - Z')
plt.show()
```

```python

```


```python
import sympy as sp
import numpy as np
import numpy.linalg as nl
import fractions
np.set_printoptions(formatter={'all':lambda x: str(fractions.Fraction(x).limit_denominator())})
```

```python
sp.init_printing(use_latex='mathjax')
```

# Characteristic polynomial

for a 2x2 the formula is:

l**2 - ((a + d) * l) + (a * d - b * c)

```python
def reduce(matrix):
    matrix = np.array(matrix)
    a = matrix[0,0]
    b = matrix[0,1]
    c = matrix[1,0]
    d = matrix[1,1]
    l = sp.symbols('l')
    expr = l**2 - ((a + d) * l) + (a * d - b * c)
    expr = sp.simplify(expr)
    return expr
```

```python
def testEigenVector(matrix, lambda1, lambda2, *testVectors):
    result = []
    for vector in testVectors:
        vector = np.array(vector)
        dotP = np.dot(matrix, vector)
        for l in [lambda1, lambda2]:
            dotL = int(l) * vector
            if np.allclose(dotP, dotL):
               result.append(vector)

    return result
```

# New Matrix

```python
A = [[1,0], [0,2]]
```

```python
expr = reduce(A)
expr
```

$$l^{2} - 3 l + 2$$

```python
solutions = sp.solve(expr)
solutions
```

$$\left [ 1, \quad 2\right ]$$

```python
eigenvalues, eigenvectors = nl.eig(A)
eigenvectors
```

    array([[1, 0],
           [0, 1]])

```python
testEigenVector(A, solutions[0], solutions[1], [0, 2], [1, -1], [0, 3], [1,0])
```

    [array([0, 2]), array([0, 3]), array([1, 0])]

# New Matrix

```python
A = [[3,4], [0,5]]
```

```python
expr = reduce(A)
expr
```

$$l^{2} - 8 l + 15$$

```python
solutions = sp.solve(expr)
solutions
```

$$\left [ 3, \quad 5\right ]$$

```python
eigenvalues, eigenvectors = nl.eig(A)
eigenvectors
```

    array([[1, 832040/930249],
           [0, 416020/930249]])

```python
testEigenVector(A, solutions[0], solutions[1], [-1, -1/2], [0,0], [2,1], [3,0])
```

    [array([-1, -1/2]), array([0, 0]), array([0, 0]), array([2, 1]), array([3, 0])]

# New Matrix

```python
A = [[1,0],[-1,4]]
```

```python
expr = reduce(A)
expr
```

$$l^{2} - 5 l + 4$$

```python
solutions = sp.solve(expr)
solutions
```

$$\left [ 1, \quad 4\right ]$$

```python
eigenvalues, eigenvectors = nl.eig(A)
eigenvectors
```

    array([[0, 932975/983442],
           [1, 275423/870964]])

```python
testEigenVector(A, solutions[0], solutions[1], [0, 1], [3,2], [3,1], [3,-1])
```

    [array([0, 1]), array([3, 1])]

# New Matrix

```python
A = [[-3, 8], [2, 3]]
```

```python
expr = reduce(A)
expr
```

$$l^{2} - 25$$

```python
solutions = sp.solve(expr)
solutions
```

$$\left [ -5, \quad 5\right ]$$

```python
eigenvalues, eigenvectors = nl.eig(A)
eigenvectors
```

    array([[-283009/291719, -470832/665857],
           [213329/879578, -470832/665857]])

```python
testEigenVector(A, solutions[0], solutions[1], [0, 2], [-1, -1], [4,-1], [1,1])
```

    [array([-1, -1]), array([4, -1]), array([1, 1])]

# New Matrix

```python
A = [[5, 4], [-4, -3]]
```

```python
expr = reduce(A)
expr
```

$$l^{2} - 2 l + 1$$

```python
solutions = sp.solve(expr)
solutions
```

$$\left [ 1\right ]$$

# New Matrix

```python
A = [[-2, -3], [1,1]]
```

```python
expr = reduce(A)
expr
```

$$l^{2} + l + 1$$

```python
solutions = sp.solve(expr)
solutions
```

$$\left [ - \frac{1}{2} - \frac{\sqrt{3} i}{2}, \quad - \frac{1}{2} + \frac{\sqrt{3} i}{2}\right ]$$

```python

```

# New Matrix

```python
A = [[3/2, -1], [-1/2, 1/2]]
```

```python
expr = reduce(A)
expr
```

$$l^{2} - 2.0 l + 0.25$$

# New Matrix

```python
A = np.array([[3/2, -1], [-1/2, 1/2]])
```

```python
expr = reduce(A)
expr
```

$$l^{2} - 2.0 l + 0.25$$

```python
solutions = sp.solve(expr)
solutions
```

$$\left [ 0.133974596215561, \quad 1.86602540378444\right ]$$

```python
1 - np.sqrt(3) / 2
```

$$0.1339745962155614$$

```python
1 + np.sqrt(3) / 2
```

$$1.8660254037844386$$

```python
testEigenVector(
    A,
    solutions[0],
    solutions[1],
    [-1-np.sqrt(3), 1], [-1+np.sqrt(3), 1],
    [1-np.sqrt(5), 1], [1+np.sqrt(5), 1],
    [1-np.sqrt(3), 1], [1+np.sqrt(3), 1],
    [-1-np.sqrt(5), 1], [-1+np.sqrt(5), 1])
```

$$\left [ \right ]$$

```python
A @ A
```

    array([[11/4, -2],
           [-1, 3/4]])

```python

```

Certification 1/Week5.1 - characteristic polynomial.ipynb


Copyright 2021 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

# Statistics - Session 1 - Exercises

Below are a few simple exercises or examples on probability to be discussed in the breakout rooms.

## Exercise 1 - Bayes rule for COVID diagnosis

Ref: [slides](https://docs.google.com/presentation/d/1UbAAhmCDJPnUsbKtaiEZnvwSUDYO5XG89fynGQAEnX4/edit?resourcekey=0-sYwZZm77lY9ZV1Uabe227Q#slide=id.gd0c69af6f6_0_56).

Consider estimating if someone has COVID $H=1$ or not $H=0$ on the basis of a PCR test. The test can either return a positive result $Y=1$ or a negative result $Y=0$. The reliability of the test is given by the following confusion matrix:

Observation

0
1

Truth
0
True Negative Rate (TNR) = Specificity
False Positive Rate (FPR) = 1 - TNR

1
False Negative Rate (FNR) = 1 - TPR
True Positive Rate (TPR) = Sensitivity

Using data from the [New York Times](https://www.nytimes.com/2020/08/04/science/coronavirus-bayes-statistics-math.html), we set **sensitivity** to 87.5\% and **specificity** to 97.5\%.

We also need to specify the prior probability $p(H=1)$; known as the **prevalence**. This varies over time and place, but let's pick $p(H=1)=0.1$ as a reasonable estimate.

If you test positive:

\begin{align}
p(H=1|Y=1)
 &= \frac{p(Y=1|H=1) p(H=1)}
{p(Y=1|H=1) p(H=1) + p(Y=1|H=0) p(H=0)}
= 0.795
\end{align}

If you test negative:
\begin{align}
p(H=1|Y=0)
 &= \frac{p(Y=0|H=1) p(H=1)}
{p(Y=0|H=1) p(H=1) + p(Y=0|H=0) p(H=0)}
=0.014
\end{align}

**Exercise**: Using the code below, explore how the resulting probabilities depend on the sensitivity and specificity of the test, as well as on the prevalence.

```python
import numpy as np

def normalize(x):
  return x / np.sum(x)

def posterior_covid(observed, prevalence=None, sensitivity=None):
  # observed = 0 for negative test, 1 for positive test
  # hidden state = 0 if no-covid, 1 if have-covid
  if sensitivity is None:
    sensitivity = 0.875
  specificity = 0.975
  TPR = sensitivity;
  FNR = 1-TPR
  TNR = specificity
  FPR = 1-TNR
  # likelihood(hidden, obs)
  likelihood_fn = np.array([[TNR, FPR], [FNR, TPR]])
  # prior(hidden)
  if prevalence is None:
    prevalence = 0.1
  prior = np.array([1-prevalence, prevalence])
  likelihood = likelihood_fn[:, observed].T
  posterior = normalize(prior * likelihood)
  return posterior
```

For a prevalence of $p(H=1)=0.1$

```python
print(posterior_covid(1)[1]*100)
print(posterior_covid(0)[1]*100)

```

    79.54545454545453
    1.4044943820224722

For a prevalence of $p(H=1)=0.01$

```python
print(posterior_covid(1, 0.01)[1]*100)
print(posterior_covid(0, 0.01)[1]*100)
```

    26.119402985074615
    0.12933264355923438

## Exercise 2 - Simpson's paradox in Covid-19 case mortality Italy vs. China

**Discussion**: Discuss this section and see if you can understand it intuitively. How would you notice that you need to consider Simpson's paradox? Can you apply Bayes' rule to understand the conditions when Simpson's paradox occurs?

Ref:
* [book pml1](https://probml.github.io/pml-book/book1.html), C.4.6, page 747f,
* [this paper](https://arxiv.org/abs/2005.07180),
* [wikipedia](https://en.wikipedia.org/wiki/Simpson%27s_paradox).

Simpson’s paradox says that a statistical trend or relationship that appears in several different groups of data can disappear or reverse sign when these groups are combined.

This results in counterintuitive behavior if we misinterpret claims of statistical dependence in a causal way.

A visualization of the paradox is given in the figure below. Overall, we see that $y$ decreases with $x$, but within each subpopulation, $y$ increases with $x$.

For a recent real-world example of Simpson’s paradox in the context of covid19, consider [this paper](https://arxiv.org/abs/2005.07180).

The reason for this is that there are more older people in Italy:

## Exercise 3 - Drawing balls from a bucket

Ref. [E.T. Jaynes Paper](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.46.1264&rep=rep1&type=pdf)

Consider the following reasoning about conditional probabilities when drawing colored balls from a bucket.

### Question 1 - Is the following computation valid?

There are 10 balls in a bucket, 1 red and 9 white ones. We randomly draw one, and then another.

For the first draw, the probability to pick the red ball is

$$P(R) = \frac{1}{10}$$

The probability to pick a white ball is

$$P(W) = \frac{9}{10}$$

For the second draw, if the first draw picked a white ball, the probability of picking the red ball is now

$$P(R|W) = \frac{1}{9}$$

However, if the first ball was the only red one, the probability to pick a white one next becomes 1:

$$P(W|R) = 1$$

Let's check if this is in accordance with Bayes' Rule:

$$P(W|R) = \frac{P(R|W) P(W)}{P(R)} = \frac{\frac{1}{9}\frac{9}{10}}{\frac{1}{10}} = \frac{\frac{1}{10}}{\frac{1}{10}} = 1$$ as we expect.

**Question 1**: Discuss whether this is indeed a correct application of Bayes rule.

**Hint**: Are the random variables $W$ and $R$ indeed the same ones in all expressions $P(R|W)$ or $R(W|R)$?

### More questions to consider:

**Question 2**: What are the probabilities for the color of the second drawn ball if we put away the first ball without looking at it?

**Question 3**: What are the probabilities for the color of the first ball if we look at it only *after* we have drawn the second ball?

**Question 4**: Do the two previous questions even make sense? Why? Why not?

## Exercise 4 - Choice of Basis in a Density Operator

Ref.: [Quantum Computation and Quantum Information](https://www.amazon.com/Quantum-Computation-Information-10th-Anniversary/dp/1107002176), page 103.

Remember how we introduced density operators to represent a probability distribution over some quantum states.

(Following is excerpt from ref above.)

It is, however, a fallacy to assume that the eigenvectors and eigenvalues of a density operator have a special significmace wiith regard to the ensemble of states represented by the density operator..

For example, one might suppose that a quantum system with the density matrix

$$\rho = \frac{3}{4} |0\rangle\langle0| + \frac{1}{4} |1\rangle\langle1|$$

must be in state $|0\rangle$ with probability $\frac{3}{4}$ and in state $|1\rangle$ with probability $\frac{1}{4}$. However, this is not necessarily the case. Let's define

$$|a\rangle = \sqrt{\frac{3}{4}}|0\rangle + \sqrt{\frac{1}{4}}|1\rangle$$
$$|b\rangle = \sqrt{\frac{3}{4}}|0\rangle - \sqrt{\frac{1}{4}}|1\rangle$$

and the quantum system is prepared to be in state $|a\rangle$ with probability $\frac{1}{2}$ and in state $|b\rangle$ with probability \frac{1}{2}. The resulting density matrix is

$$\rho = \frac{1}{2} |a\rangle\langle a| + \frac{1}{2} |b\rangle\langle b| = \frac{3}{4} |0\rangle\langle0| + \frac{1}{4} |1\rangle\langle1|$$

That is, two different mixed states give rise to the same density matrix.

**Exercise**: Verify those are indeed the same density operators.

**Discussion**: What does this mean for the "reality" of the probability distribution expressed by the densiity matrix? How does this relate to the "reality" of the probability distribution of the color of the ball in the previous exercise?

## References and further reading

* [Quantum Computation and Quantum Information](https://www.amazon.com/Quantum-Computation-Information-10th-Anniversary/dp/1107002176) (book)
* E.T. Jaynes [Clearing up Mysteries - The Original Goal](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.46.1264&rep=rep1&type=pdf) (paper)
* E.T. Jaynes [Probability Theory. The Logic of Science](https://www.amazon.com/Probability-Theory-Science-T-Jaynes/dp/0521592712) (textbook)


jacobian, hessian

```python
% matplotlib inline

import sympy as sy
import numpy as np
import scipy as sp
import math
import matplotlib.pyplot as plt
import matplotlib as mpl

style_name = 'bmh' #bmh
mpl.style.use(style_name)
np.set_printoptions(precision=4, linewidth =150)

style = plt.style.library[style_name]
style_colors = [ c['color'] for c in style['axes.prop_cycle'] ]

sy.init_printing(use_latex='mathjax')

```

```python
x_1, x_2 = sy.symbols('x_1 x_2')
y = x_1 ** 3 + x_2 ** 3 + 2 * x_1 ** 2 + 3 * x_2 ** 2 - x_1 * x_2 + 2 * x_1 + 4 * x_2

# jacobian
x1_diff = sy.diff(y, x_1)
x2_diff = sy.diff(y, x_2)

f_f1_diff = sy.lambdify((x_1, x_2), x1_diff, 'numpy')
f_f2_diff = sy.lambdify((x_1, x_2), x2_diff, 'numpy')
# lamdify 미분값을 알고 싶을 때 쓰는 함수. 도함수 자체를 도출해주는 것.

np.array([f_f1_diff(1,2), f_f2_diff(1,2)])
```

    array([ 7, 27])

```python
# hessian 은 대칭행렬
x1_x1_diff = sy.diff(y, x_1, x_1)
x1_x2_diff = sy.diff(y, x_1, x_2)
x2_x1_diff = sy.diff(y, x_2, x_1)
x2_x2_diff = sy.diff(y, x_2, x_2)

f_00_diff = sy.lambdify((x_1, x_2), x1_x1_diff, 'numpy')
f_01_diff = sy.lambdify((x_1, x_2), x1_x2_diff, 'numpy')
f_10_diff = sy.lambdify((x_1, x_2), x2_x1_diff, 'numpy')
f_11_diff = sy.lambdify((x_1, x_2), x2_x2_diff, 'numpy')

np.array([[f_00_diff(1,2), f_01_diff(1,2)],[f_10_diff(1,2), f_11_diff(1,2)]])
```

    array([[10, -1],
           [-1, 18]])

tayler series

$$ T_f(x) = \sum_{n=0}^{\infty}{\frac{f^{n}(a)}{n!}}(x-a)^{n} $$
$$
T_f(x,y) = \sum_{k=0}^{\infty}\sum_{i=0}^{k}{\frac{(x - a)^{k-i}(y - b)^i}{(k - i)!i!}}\left.
{\frac{\partial^kf}{\partial x^{k-i}\partial y^i}}\right|_{(a,b)}
$$
- 단변수
$$
\begin{align}
& T_f(x) = f(x^*) + \frac{d f(x^*)}{dx} (x - x^*) + \frac{1}{2!} \frac{d^2 f(x^*)}{dx^2}(x - x^*)^2 + R\\[1pt]
& (R \approx error)
 \end{align}
$$

- 다변수
$$
\begin{align}
& T_f(x, y) = f(x^*, y^*) + \frac{\partial f}{\partial x}(x - x^*) + \frac{\partial f}{\partial y}(y - y^*) + \\[1pt]
& \frac{1}{2} \left[ \frac{\partial^2 f}{\partial x^2}(x - x^*)^2 + 2 \frac{\partial^2 f}{\partial x \partial y} (x - x^*)(y - y^*) + \frac{\partial^2 f}{\partial y^2}(y - y^*)^2 \right] + R\\[1pt]
& (R \approx error)
\end{align}
$$

- 그라디언트와 헤시안으로 표현
    - 일변수
$$
\Delta f = f'(x^*)d + {\frac{1}{2}}f''(x^*)d^2 + R \;\; ,(d = x - x^*)
$$
    - 다변수
$$
\Delta f = \triangledown f\left(\mathbf{x}^*\right)^{\text{T}}\mathbf{d} + \frac{1}{2}\mathbf{d}^{\text{T}} \mathbf{H}\left(\mathbf{x}^*\right)\mathbf{d} + R
$$

- 테일러 급수(Taylor series) 또는 테일러 전개(Taylor expansion)는 어떤 미지의 함수 f(x)를 아래 식과 같이 근사 다항함수로 표현하는 것을 말합니다.
- x = a 근처에서만 성립한다는 점입니다. 즉, x가 a에서 멀어지면 멀어질수록 f(x) = p(x)로 놓는 것은 큰 오차를 갖게 됩니다. 한편, 근사다항식의 차수는 높으면 높을수록 f(x)를 좀더 잘 근사하게 됩니다.

- 출처: [다크프로그래머](http://darkpgmr.tistory.com/59)
- 설명: [2변수함수의 테일러정리](http://math.kongju.ac.kr/calculus/data/chap9/s6/s6.htm)

```python
from sympy.functions import sin, cos, ln

plt.style.use("ggplot")

def factorial(n):
    if n <= 0:
        return 1
    else:
        return n * factorial(n - 1)

def taylor(function, x0, n, x = sy.Symbol('x')):
    # x0 : 근사값을 구하려고 하는 지점
    # n : 미분차수
    i = 0
    p = 0
    while i <= n:
        p += (function.diff(x, i).subs(x, x0)) / (factorial(i)) * (x - x0) ** i
        i += 1
    return p
    #

def plot(f, x0 = 0, n = 18, by = 3, x_lims = [-10, 10], y_lims = [-10, 10], npoints = 800, x = sy.Symbol('x')):
    x1 = np.linspace(x_lims[0], x_lims[1], npoints)
    # x=0 에서(맥클로린) 차수를 올려가며 테일러 급수로 근사한 그래프 그리기
    for j in range(1, n + 1, by):
        func = taylor(f, x0, j)
        taylor_lambda = sy.lambdify(x, func, "numpy")
        print('Taylor expansion at n=' + str(j), func)
        plt.plot(x1, taylor_lambda(x1), label = 'Order '+ str(j))

    # 실제 함수 그래프
    func_lambda = sy.lambdify(x, f, "numpy")
    plt.plot(x1, func_lambda(x1), label = 'f(x)', color = 'black', linestyle = '--')

    plt.xlim(x_lims)
    plt.ylim(y_lims)
    plt.xlabel('x')
    plt.ylabel('y')
    plt.legend()
    plt.grid(True)
    plt.title('Taylor series approximation')
    plt.show()
```

```python
x = sy.Symbol('x')
f = x ** 2 + 2 * x + 1
f.diff(x, 1)
```

$$2 x + 2$$

```python
x = sy.Symbol('x')
# f = ln(1 + x)
f = sin(x)
plot(f)
```

### 수치 알고리즘 일반개념

- 최적화 수치 알고리즘은 일반적으로 다음 단계를 따름

    - 단계 1. 타당성있는 출발점 $\mathbf{x}^{(0)}$ 추정, $k=0$
    - 단계 2. 탐색방향 $\mathbf{d}^{(k)}$를 계산
    - 단계 3. 수렴 검토
        - 지역 최소를 위한 1계 필요조건
$$
\color{Red}{f'(x^*) =0}
$$
        - 또한 추가적으로 x*에서 함수를 두번미분한 결과값도 양수가 되어야겠지
        - 필요조건을 만족시킨다면 그점은
            - 지역 최소
            - 지역 최대
            - 변곡점

    - 단계 4. 양의 이동거리 $\alpha_k$ 계산
    - 단계 5. 새로운 설계 변수 계산 $\mathbf{x}^{(k+1)} = \mathbf{x}^{(k)} + \alpha_k \mathbf{d}^{(k)}$, $k=k+1$
        - (벡터의 덧셈으로 원래 벡터에서 뱡향이 달라진 새로운 벡터를 생성) 단계 2로

- 따라서   $\alpha_k$와   $\mathbf{d}^{(k)}$ 계산이 중요

#### 경사도 수치계산 실습

$$
\mu = 0,\; \sigma^2 = 0.2\\
f(x) = {\frac{1}{\sigma\sqrt{2\pi}}}exp\left(-{\frac{(x - \mu)^2}{2\sigma^2}}\right)
$$

```python
x = sy.symbols('x')
mu, var = 0, 0.2
s = sy.sqrt(var)

1 / (sy.sqrt(2 * np.pi * var)) * sy.exp(-1 * (((x - mu)**2) / (2 * var)))

```

$$- 4.46031029038193 x e^{- 2.5 x^{2}}$$

```python

sy.diff(1 / (sy.sqrt(2 * np.pi * var)) * sy.exp(-1 * (x - mu) ** 2 / (2 * var)), x)
```

$$- 4.46031029038193 x e^{- 2.5 x^{2}}$$

```python
plt.rcParams["figure.figsize"] = (15,7)

x = sy.symbols('x')
mu, var = 0, 0.2

def f(x):
    """
    실습용 함수 정의
    """
    s = np.sqrt(var)
    return 1 / (np.sqrt(2 * np.pi * var)) * np.exp(-1 * (x - mu) ** 2 / (2 * var))

def df_anal(x):
    """
    sympy의 결과로 얻은 도함수
    """
    return -1 * 4.46031029038193 * x * np.exp(-2.5 * x**2)

def df_numer(x):
    """
    수치적으로 도함수의 값을 계산하는 함수
    전방, 후방, 중앙차분법 오차 시각화하기
    """
    h = 0.1
    return (f(x + h * 0.5) - f(x - 0.5 * h)) / h

x = np.linspace(-5, 5, 200)

plt.plot(x, f(x), lw=3, color=style_colors[0], label=r"$f(x) = \frac{4x}{(x^2  + 1)}$")
plt.plot(x, df_anal(x), lw=10, color=style_colors[1], alpha=0.3 , label=r"$\frac{df}{dx}$")
plt.plot(x, df_numer(x), color=style_colors[1], lw=3, label=r"Numerical derivative")
plt.legend(fontsize=20)

plt.suptitle("Numerical derivative", fontsize=20)
plt.show()
```