```
# This mounts your Google Drive to the Colab VM.
from google.colab import drive
drive.mount('/content/drive')

# TODO: Enter the foldername in your Drive where you have saved the unzipped
# assignment folder, e.g. 'cs231n/assignments/assignment3/'
FOLDERNAME = '/Colab Notebooks/Stanford_CS231n/assignment3'
assert FOLDERNAME is not None, "[!] Enter the foldername."

# Now that we've mounted your Drive, this ensures that
# the Python interpreter of the Colab VM can load
# python files from within it.
import sys
sys.path.append('/content/drive/My Drive/{}'.format(FOLDERNAME))

# This downloads the COCO dataset to your Drive
# if it doesn't already exist.
%cd /content/drive/My\ Drive/$FOLDERNAME/cs231n/datasets/
!bash get_datasets.sh
%cd /content/drive/My\ Drive/$FOLDERNAME
```

# Image Captioning with Transformers
You have now implemented a vanilla RNN and for the task of image captioning. In this notebook you will implement key pieces of a transformer decoder to accomplish the same task.

**NOTE:** This notebook will be primarily written in PyTorch rather than NumPy, unlike the RNN notebook.

```
# Setup cell.
import time, os, json
import numpy as np
import matplotlib.pyplot as plt

from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.transformer_layers import *
from cs231n.captioning_solver_transformer import CaptioningSolverTransformer
from cs231n.classifiers.transformer import CaptioningTransformer
from cs231n.coco_utils import load_coco_data, sample_coco_minibatch, decode_captions
from cs231n.image_utils import image_from_url

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # Set default size of plots.
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))
```

# COCO Dataset
As in the previous notebooks, we will use the COCO dataset for captioning.

```
# Load COCO data from disk into a dictionary.
data = load_coco_data(pca_features=True)

# Print out all the keys and values from the data dictionary.
for k, v in data.items():
    if type(v) == np.ndarray:
        print(k, type(v), v.shape, v.dtype)
    else:
        print(k, type(v), len(v))
```

# Transformer
As you have seen, RNNs are incredibly powerful but often slow to train. Further, RNNs struggle to encode long-range dependencies (though LSTMs are one way of mitigating the issue). In 2017, Vaswani et al introduced the Transformer in their paper ["Attention Is All You Need"](https://arxiv.org/abs/1706.03762) to a) introduce parallelism and b) allow models to learn long-range dependencies. The paper not only led to famous models like BERT and GPT in the natural language processing community, but also an explosion of interest across fields, including vision. While here we introduce the model in the context of image captioning, the idea of attention itself is much more general.

# Transformer: Multi-Headed Attention

### Dot-Product Attention

Recall that attention can be viewed as an operation on a query $q\in\mathbb{R}^d$, a set of value vectors $\{v_1,\dots,v_n\}, v_i\in\mathbb{R}^d$, and a set of key vectors $\{k_1,\dots,k_n\}, k_i \in \mathbb{R}^d$, specified as

\begin{align}
c = \sum_{i=1}^{n} v_i \alpha_i &\alpha_i = \frac{\exp(k_i^\top q)}{\sum_{j=1}^{n} \exp(k_j^\top q)} \\
\end{align}

where $\alpha_i$ are frequently called the "attention weights", and the output $c\in\mathbb{R}^d$ is a correspondingly weighted average over the value vectors.

### Self-Attention
In Transformers, we perform self-attention, which means that the values, keys and query are derived from the input $X \in \mathbb{R}^{\ell \times d}$, where $\ell$ is our sequence length. Specifically, we learn parameter matrices $V,K,Q \in \mathbb{R}^{d\times d}$ to map our input $X$ as follows:

\begin{align}
v_i = Vx_i\ \ i \in \{1,\dots,\ell\}\\
k_i = Kx_i\ \ i \in \{1,\dots,\ell\}\\
q_i = Qx_i\ \ i \in \{1,\dots,\ell\}
\end{align}

### Multi-Headed Scaled Dot-Product Attention
In the case of multi-headed attention, we learn a parameter matrix for each head, which gives the model more expressivity to attend to different parts of the input. Let $h$ be number of heads, and $Y_i$ be the attention output of head $i$. Thus we learn individual matrices $Q_i$, $K_i$ and $V_i$. To keep our overall computation the same as the single-headed case, we choose $Q_i \in \mathbb{R}^{d\times d/h}$, $K_i \in \mathbb{R}^{d\times d/h}$ and $V_i \in \mathbb{R}^{d\times d/h}$. Adding in a scaling term $\frac{1}{\sqrt{d/h}}$ to our simple dot-product attention above, we have

\begin{equation} \label{qkv_eqn}
Y_i = \text{softmax}\bigg(\frac{(XQ_i)(XK_i)^\top}{\sqrt{d/h}}\bigg)(XV_i)
\end{equation}

where $Y_i\in\mathbb{R}^{\ell \times d/h}$, where $\ell$ is our sequence length.

In our implementation, we then apply dropout here (though in practice it could be used at any step):

$$Y_i = \text{dropout}(Y_i)$$

Finally, then the output of the self-attention is a linear transformation of the concatenation of the heads:

\begin{equation}
Y = [Y_1;\dots;Y_h]A
\end{equation}

were $A \in\mathbb{R}^{d\times d}$ and $[Y_1;\dots;Y_h]\in\mathbb{R}^{\ell \times d}$.

Implement multi-headed scaled dot-product attention in the `MultiHeadAttention` class in the file `cs231n/transformer_layers.py`. The code below will check your implementation. The relative error should be less than 1e-3.

```
torch.manual_seed(231)

# Choose dimensions such that they are all unique for easier debugging:
# Specifically, the following values correspond to N=1, H=2, T=3, E//H=4, and E=8.
batch_size = 1
sequence_length = 3
embed_dim = 8
attn = MultiHeadAttention(embed_dim, num_heads=2)

# Self-attention.
data = torch.randn(batch_size, sequence_length, embed_dim)
self_attn_output = attn(query=data, key=data, value=data)

# Masked self-attention.
mask = torch.randn(sequence_length, sequence_length) < 0.5
masked_self_attn_output = attn(query=data, key=data, value=data, attn_mask=mask)

# Attention using two inputs.
other_data = torch.randn(batch_size, sequence_length, embed_dim)
attn_output = attn(query=data, key=other_data, value=other_data)

expected_self_attn_output = np.asarray([[
[-0.2494,  0.1396,  0.4323, -0.2411, -0.1547,  0.2329, -0.1936,
          -0.1444],
         [-0.1997,  0.1746,  0.7377, -0.3549, -0.2657,  0.2693, -0.2541,
          -0.2476],
         [-0.0625,  0.1503,  0.7572, -0.3974, -0.1681,  0.2168, -0.2478,
          -0.3038]]])

expected_masked_self_attn_output = np.asarray([[
[-0.1347,  0.1934,  0.8628, -0.4903, -0.2614,  0.2798, -0.2586,
          -0.3019],
         [-0.1013,  0.3111,  0.5783, -0.3248, -0.3842,  0.1482, -0.3628,
          -0.1496],
         [-0.2071,  0.1669,  0.7097, -0.3152, -0.3136,  0.2520, -0.2774,
          -0.2208]]])

expected_attn_output = np.asarray([[
[-0.1980,  0.4083,  0.1968, -0.3477,  0.0321,  0.4258, -0.8972,
          -0.2744],
         [-0.1603,  0.4155,  0.2295, -0.3485, -0.0341,  0.3929, -0.8248,
          -0.2767],
         [-0.0908,  0.4113,  0.3017, -0.3539, -0.1020,  0.3784, -0.7189,
          -0.2912]]])

print('self_attn_output error: ', rel_error(expected_self_attn_output, self_attn_output.detach().numpy()))
print('masked_self_attn_output error: ', rel_error(expected_masked_self_attn_output, masked_self_attn_output.detach().numpy()))
print('attn_output error: ', rel_error(expected_attn_output, attn_output.detach().numpy()))
```

# Positional Encoding

While transformers are able to easily attend to any part of their input, the attention mechanism has no concept of token order. However, for many tasks (especially natural language processing), relative token order is very important. To recover this, the authors add a positional encoding to the embeddings of individual word tokens.

Let us define a matrix $P \in \mathbb{R}^{l\times d}$, where $P_{ij} = $

$$
\begin{cases}
\text{sin}\left(i \cdot 10000^{-\frac{j}{d}}\right) & \text{if j is even} \\
\text{cos}\left(i \cdot 10000^{-\frac{(j-1)}{d}}\right) & \text{otherwise} \\
\end{cases}
$$

Rather than directly passing an input $X \in \mathbb{R}^{l\times d}$ to our network, we instead pass $X + P$.

Implement this layer in `PositionalEncoding` in `cs231n/transformer_layers.py`. Once you are done, run the following to perform a simple test of your implementation. You should see errors on the order of `e-3` or less.

```
torch.manual_seed(231)

batch_size = 1
sequence_length = 2
embed_dim = 6
data = torch.randn(batch_size, sequence_length, embed_dim)

pos_encoder = PositionalEncoding(embed_dim)
output = pos_encoder(data)

expected_pe_output = np.asarray([[[-1.2340,  1.1127,  1.6978, -0.0865, -0.0000,  1.2728],
                                  [ 0.9028, -0.4781,  0.5535,  0.8133,  1.2644,  1.7034]]])

print('pe_output error: ', rel_error(expected_pe_output, output.detach().numpy()))
```

# Inline Question 1

Several key design decisions were made in designing the scaled dot product attention we introduced above. Explain why the following choices were beneficial:
1. Using multiple attention heads as opposed to one.
2. Dividing by $\sqrt{d/h}$ before applying the softmax function. Recall that $d$ is the feature dimension and $h$ is the number of heads.
3. Adding a linear transformation to the output of the attention operation. What would happen if we were to stack attention operations directly?

Only one or two sentences per choice is necessary, but be sure to be specific in addressing what would have happened without each given implementation detail, why such a situation would be suboptimal, and how the proposed implementation improves the situation.

**Your Answer:**

# Overfit Transformer Captioning Model on Small Data
Run the following to overfit the Transformer-based captioning model on the same small dataset as we used for the RNN previously.

```
torch.manual_seed(231)
np.random.seed(231)

data = load_coco_data(max_train=50)

transformer = CaptioningTransformer(
          word_to_idx=data['word_to_idx'],
          input_dim=data['train_features'].shape[1],
          wordvec_dim=256,
          num_heads=2,
          num_layers=2,
          max_length=30
        )

transformer_solver = CaptioningSolverTransformer(transformer, data, idx_to_word=data['idx_to_word'],
           num_epochs=100,
           batch_size=25,
           learning_rate=0.001,
           verbose=True, print_every=10,
         )

transformer_solver.train()

# Plot the training losses.
plt.plot(transformer_solver.loss_history)
plt.xlabel('Iteration')
plt.ylabel('Loss')
plt.title('Training loss history')
plt.show()
```

Print final training loss. You should see a final loss of less than 0.03.

```
print('Final loss: ', transformer_solver.loss_history[-1])
```

# Transformer Sampling at Test Time
The sampling code has been written for you. You can simply run the following to compare with the previous results with the RNN. As before the training results should be much better than the validation set results, given how little data we trained on.

```
# If you get an error, the URL just no longer exists, so don't worry!
# You can re-sample as many times as you want.
for split in ['train', 'val']:
    minibatch = sample_coco_minibatch(data, split=split, batch_size=2)
    gt_captions, features, urls = minibatch
    gt_captions = decode_captions(gt_captions, data['idx_to_word'])

    sample_captions = transformer.sample(features, max_length=30)
    sample_captions = decode_captions(sample_captions, data['idx_to_word'])

    for gt_caption, sample_caption, url in zip(gt_captions, sample_captions, urls):
        img = image_from_url(url)
        # Skip missing URLs.
        if img is None: continue
        plt.imshow(img)
        plt.title('%s\n%s\nGT:%s' % (split, sample_caption, gt_caption))
        plt.axis('off')
        plt.show()
```

```

```


#  CS-109A Introduction to Data Science

## Lab 11: Neural Network Basics - Introduction to `tf.keras`

**Harvard University**

**Fall 2019**

**Instructors:** Pavlos Protopapas, Kevin Rader, Chris Tanner

**Lab Instructors:** Chris Tanner and Eleni Kaxiras.

**Authors:** Eleni Kaxiras, David Sondak, and Pavlos Protopapas.

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import pandas as pd
%matplotlib inline

from PIL import Image
```

```python
from __future__ import absolute_import, division, print_function, unicode_literals

# TensorFlow and tf.keras
import tensorflow as tf

tf.keras.backend.clear_session()  # For easy reset of notebook state.

print(tf.__version__)  # You should see a 2.0.0 here!
```

    2.0.0

#### Instructions for running `tf.keras` with Tensorflow 2.0:

1. Create a `conda` virtual environment by cloning an existing one that you know works
```
conda create --name myclone --clone myenv
```

2. Go to [https://www.tensorflow.org/install/pip](https://www.tensorflow.org/install/pip) and follow instructions for your machine.

3. In a nutshell:
```
pip install --upgrade pip
pip install tensorflow==2.0.0
```
All references to Keras should be written as `tf.keras`.  For example:

```
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

tf.keras.models.Sequential
tf.keras.layers.Dense, tf.keras.layers.Activation,
tf.keras.layers.Dropout, tf.keras.layers.Flatten, tf.keras.layers.Reshape
tf.keras.optimizers.SGD
tf.keras.preprocessing.image.ImageDataGenerator
tf.keras.regularizers
tf.keras.datasets.mnist
```

You could avoid the long names by using
```
from tensorflow import keras
from tensorflow.keras import layers
```
These imports do not work on some systems, however, because they pick up previous versions of `keras` and `tensorflow`. That is why I avoid them in this lab.

## Learning Goals
In this lab we will understand the basics of neural networks and how to start using a deep learning library called `keras`. By the end of this lab, you should:

- Understand how a simple neural network works and code some of its functionality from scratch.
- Be able to think and do calculations in matrix notation. Also think of vectors and arrays as tensors.
- Know how to install and run `tf.keras`.
- Implement a simple real world example using a neural network.

## Part 1: Neural Networks 101

Suppose we have an input vector $X=${$x_1, x_2, ... x_L$} to a $k$-layered network.

Each layer has its own number of nodes. For the first layer in our drawing that number is $J$. We can store the weights for each node in a vector $\mathbf{W} \in \mathbb{R}^{JxL+1}$ (accounting for bias).  Similarly, we can store the biases from each node in a vector $\mathbf{b} \in \mathbb{R}^{I}$.  The affine transformation is then written as $$\mathbf{a} = \mathbf{W^T}X + \mathbf{b}$$
  What we then do is "absorb" $\mathbf{b}$ into $X$ by adding a column of ones to $X$. Our $X$ matrix than becomes $\mathbf{X} \in \mathbb{R}^{JxL+1}$ and our equation:

$$\mathbf{a} = \mathbf{W^T}_{plusones}X$$
We have that $\mathbf{a} \in \mathbb{R}^{J}$ as well.  Next we evaluate the output from each node.  We write $$\mathbf{u} = \sigma\left(\mathbf{a}\right)$$ where $\mathbf{u}\in\mathbb{R}^{J}$.  We can think of $\sigma$ operating on each individual element of $\mathbf{a}$ separately or in matrix notation. If we denote each component of $\mathbf{a}$ by $a_{j}$ then we can write $$u_{j} = \sigma\left(a_{j}\right), \quad j = 1, ... J.$$
 In our code we will implement all these equations in matrix notation.
`tf.keras` (Tensorflow) and `numpy` perform the calculations in matrix format.

*Image source: "Modern Mathematical Methods for Computational Science and Engineering" Efthimios Kaxiras and Athanassios Fokas.*

Let's assume that we have 3 input points (L = 3), two hidden layers ($k=2$), and 2 nodes in each layer ($J=2$)

### Input Layer

$𝑋$={$𝑥_1,𝑥_2,x_3$}

### First Hidden Layer

\begin{equation}
 \begin{aligned}
a^{(1)}_1 = w^{(1)}_{10} + w^{(1)}_{11}x_1 + w^{(1)}_{12}x_2 + w^{(1)}_{13}x_3 \\
a^{(1)}_2 = w^{(1)}_{20} + w^{(1)}_{21}x_1 + w^{(1)}_{22}x_2 + w^{(1)}_{23}x_3 \\
\end{aligned}
\end{equation}

 All this in matrix notation: $$\mathbf{a} = \mathbf{W^T}X$$

 NOTE: in $X$ we have added a column of ones to account for the bias

**Then the sigmoid is applied**:
\begin{equation}
 \begin{aligned}
u^{(1)}_1 = \sigma(a^{(1)}_1) \\
u^{(1)}_2 = \sigma(a^{(1)}_2) \\
\end{aligned}
\end{equation}

or in matrix notation: $$\mathbf{u} = \sigma\left(\mathbf{a}\right)$$

### Second Hidden Layer

\begin{equation}
 \begin{aligned}
a^{(2)}_1 = w^{(2)}_{10} + w^{(2)}_{11}u^{(1)}_1 + w^{(2)}_{12}u^{(1)}_2 + w^{(2)}_{13}u^{(1)}_3 \\
a^{(2)}_2 = w^{(2)}_{20} + w^{(2)}_{21}u^{(1)}_1 + w^{(2)}_{22}u^{(1)}_2 + w^{(2)}_{23}u^{(1)}_3 \\
\end{aligned}
\end{equation}

**Then the sigmoid is applied**:

\begin{equation}
 \begin{aligned}
u^{(2)}_1 = \sigma(a^{(2)}_1) \\
u^{(2)}_2 = \sigma(a^{(2)}_2) \\
\end{aligned}
\end{equation}

### Output Layer

#### If the output is categorical:

For example with four classes ($M=4$): $Y$={$y_1, y_2, y_3, y_4$}, we have the affine and then the sigmoid is lastly applied:

\begin{equation}
 \begin{aligned}
a^{(3)}_1 = w^{(3)}_{10} + w^{(3)}_{11}u^{(2)}_1 + w^{(3)}_{12}u^{(2)}_2 \\
a^{(3)}_2 = w^{(3)}_{20} + w^{(3)}_{21}u^{(2)}_1 + w^{(3)}_{22}u^{(2)}_2 \\
a^{(3)}_3 = w^{(3)}_{30} + w^{(3)}_{31}u^{(2)}_1 + w^{(3)}_{32}u^{(2)}_2 \\
a^{(3)}_4 = w^{(3)}_{40} + w^{(3)}_{41}u^{(2)}_1 + w^{(3)}_{42}u^{(2)}_2 \\
\end{aligned}
\end{equation}

\begin{equation}
 \begin{aligned}
y_1 = \sigma(a^{(3)}_1) \\
y_2 = \sigma(a^{(3)}_2) \\
y_3 = \sigma(a^{(3)}_3) \\
y_3 = \sigma(a^{(3)}_4) \\
\end{aligned}
\end{equation}
$\sigma$ will be softmax in the case of multiple classes and sigmoid for binary.

#### If the output is a number (regression):

We have a single y as output:

\begin{equation}
 \begin{aligned}
y = w^{(3)}_{10}+ w^{(3)}_{11}u^{(2)}_1 + w^{(3)}_{12}u^{(2)}_2 + w^{(3)}_{13}u^{(2)}_3 \\
\end{aligned}
\end{equation}

#### Matrix Multiplication and constant addition

```python
# make two arrays and multiply them using np.dot(a, b) or tf.matmul(a, b)
```

```python
# both Tensorflow and numpy take care of transposing.

```

```python
# how do we add the constant in the matrix

```

In class exercise : Plot the sigmoid

Define the `sigmoid` and the `tanh`. For `tanh` you may use `np.tanh` and for the `sigmoid` use the general equation:
\begin{align}
\sigma = \dfrac{1}{1+e^{-2(x-c)/a}} \qquad\text{(1.1)}
\textrm{}
\end{align}

Generate a list of 500 $x$ points from -5 to 5 and plot both functions. What do you observe? What do variables $c$ and $a$ do?

```python
# your code here
```

```python
# %load solutions/sigmoid.py
# The smaller the `a`, the sharper the function is.
# Variable `c` moves the function along the x axis
def sigmoid(x,c,a):
    z = ((x-c)/a)
    return 1.0 / (1.0 + np.exp(-z))

x = np.linspace(-5.0, 5.0, 500) # input points
c = 1.
a = 0.5
plt.plot(x, sigmoid(x, c, a), label='sigmoid')
plt.plot(x, np.tanh(x), label='tanh')
plt.grid();
plt.legend();

```

2. In class exercise: Approximate a Gaussian function using a node and manually adjusting the weights. Start with one layer with one node and move to two nodes.

The task is to approximate (learn) a function $f\left(x\right)$ given some input $x$.  For demonstration purposes, the function we will try to learn is a Gaussian function:
\begin{align}
f\left(x\right) = e^{-x^{2}}
\textrm{}
\end{align}

Even though we represent the input $x$ as a vector on the computer, you should think of it as a single input.

#### 2.1 Start by plotting the above function using the $x$ dataset

```python
x = np.linspace(-5.0, 5.0, 500) # input points
def gaussian(x):
    return np.exp(-x*x)

f = gaussian(x)
plt.plot(x, f, label='gaussian')
plt.legend()
```

```python
f.shape
```

    (500,)

#### 2.2 Now, let's code the single node as per the image above.

Write a function named `affine` that does the transformation. The definition is provided below. Then create a simpler sigmoid with just one variable. We choose a **sigmoid** activation function and specifically the **logistic** function.  Sigmoids are a family of functions and the logistic function is just one member in that family. $$\sigma\left(z\right) = \dfrac{1}{1 + e^{-z}}.$$

Define both functions in code.

```python
def affine(x, w, b):
    """Return affine transformation of x

    INPUTS
    ======
    x: A numpy array of points in x
    w: An array representing the weight of the perceptron
    b: An array representing the biases of the perceptron

    RETURN
    ======
    z: A numpy array of points after the affine transformation
       z = wx + b
    """

    # Code goes here
    return z
```

```python
# your code here

```

```python
# %load solutions/affine-sigmoid.py
def affine(x, w, b):
    return w * x + b

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))
```

And now we plot the activation function and the true function. What do you think will happen if you change $w$ and $b$?

```python
w = [.., .., ..] # Create a list of weights to try
b = [.., .., ..] # Create a list of biases

fig, ax = plt.subplots(1,1, figsize=(9,5))
SIZE = 16

# plot our true function, the gaussian
ax.plot(x, f, lw=4, ls='-.', label='True function')

# plot 3 "networks"
for wi, bi in zip(w, b):
    h = sigmoid(affine(x, wi, bi))
    ax.plot(x, h, lw=4, label=r'$w = {0}$, $b = {1}$'.format(wi,bi))

ax.set_title('Single neuron network', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

We didn't do an exhaustive search of the weights and biases, but it sure looks like this single perceptron is never going to match the actual function.  Again, we shouldn't be suprised about this.  The output layer of the network is simple the logistic function, which can only have so much flexibility.

Let's try to make our network more flexible by using **more nodes**!

### Multiple Perceptrons in a Single Layer
It appears that a single neuron is somewhat limited in what it can accomplish.  What if we expand the number of nodes/neurons in our network?  We have two obvious choices here.  One option is to add depth to the network by putting layers next to each other.  The other option is to stack neurons on top of each other in the same layer.  Now the network has some width, but is still only one layer deep.

```python
x = np.linspace(-5.0, 5.0, 500) # input points
f = np.exp(-x*x) # data

w = np.array([..., ...]) # ENTER TWO VALUES HERE
b = np.array([..., ...]) # HERE TOO

# Affine transformations
z1 = w[0] * x + b[0]
z2 = w[1] * x + b[1]

# Node outputs
h1 = sigmoid(z1)
h2 = sigmoid(z2)
```

Now let's plot things and see what they look like.

```python
fig, ax = plt.subplots(1,1, figsize=(9,5))

ax.plot(x, f, lw=4, ls = '-.', label='True function')
ax.plot(x, h1, lw=4, label='First neuron')
ax.plot(x, h2, lw=4, label='Second neuron')

# Set title
ax.set_title('Comparison of Neuron Outputs', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

Just as we expected.  Some sigmoids.  Of course, to get the network prediction we must combine these two sigmoid curves somehow.  First we'll just add $h_{1}$ and $h_{2}$ without any weights to see what happens.

#### Note
We are **not** doing classification here.  We are trying to predict an actual function.  The sigmoid activation is convenient when doing classification because you need to go from $0$ to $1$.  However, when learning a function, we don't have as good of a reason to choose a sigmoid.

```python
# Network output
wout = np.ones(2) # Set the output weights to unity to begin
bout = -1 # bias
yout = wout[0] * h1 + wout[1] * h2 + bout
```

And plot.

```python
fig, ax = plt.subplots(1,1, figsize=(9,5))

ax.plot(x, f, ls='-.', lw=4, label=r'True function')
ax.plot(x, yout, lw=4, label=r'$y_{out} = h_{1} + h_{2}$')

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

Very cool!  The two nodes interact with each other to produce a pretty complicated-looking function.  It still doesn't match the true function, but now we have some hope.  In fact, it's starting to look a little bit like a Gaussian!

We can do better.  There are three obvious options at this point:
1. Change the number of nodes
2. Change the activation functions
3. Change the weights

#### We will leave this simple example for some other time! Let's move on to fashion items!

## Part 2: Tensors, Fashion, and Reese Witherspoon

We can think of tensors as multidimensional arrays of real numerical values; their job is to generalize matrices to multiple dimensions. While tensors first emerged in the 20th century, they have since been applied to numerous other disciplines, including machine learning. Tensor decomposition/factorization can solve, among other, problems in unsupervised learning settings, temporal and multirelational data. For those of you that will get to handle images for Convolutional Neural Networks, it's a good idea to have the understanding of tensors of rank 3.

We will use the following naming conventions:

- scalar = just a number = rank 0 tensor  ($a$ ∈ $F$,)

- vector = 1D array = rank 1 tensor ( $x = (\;x_1,...,x_i\;)⊤$ ∈ $F^n$ )

- matrix = 2D array = rank 2 tensor ( $\textbf{X} = [a_{ij}] ∈ F^{m×n}$ )

- 3D array = rank 3 tensor ( $\mathscr{X} =[t_{i,j,k}]∈F^{m×n×l}$ )

- $\mathscr{N}$D array = rank $\mathscr{N}$ tensor ( $\mathscr{T} =[t_{i1},...,t_{i\mathscr{N}}]∈F^{n_1×...×n_\mathscr{N}}$ ) <-- Things start to get complicated here...

#### Tensor indexing
We can create subarrays by fixing some of the given tensor’s indices. We can create a vector by fixing all but one index. A 2D matrix is created when fixing all but two indices. For example, for a third order tensor the vectors are

$\mathscr{X}[:,j,k]$ = $\mathscr{X}[j,k]$ (column),

$\mathscr{X}[i,:,k]$ = $\mathscr{X}[i,k]$ (row), and

$\mathscr{X}[i,j,:]$ = $\mathscr{X}[i,j]$ (tube)

#### Tensor multiplication
We can multiply one matrix with another as long as the sizes are compatible ((n × m) × (m × p) = n × p), and also multiply an entire matrix by a constant. Numpy `numpy.dot` performs a matrix multiplication which is straightforward when we have 2D or 1D arrays. But what about > 3D arrays? The function will choose according to the matching dimentions but if we want to choose we should use `tensordot`, but, again, we **do not need tensordot** for this class.

### Reese Witherspoon

This image is from the dataset [Labeled Faces in the Wild](http://vis-www.cs.umass.edu/lfw/person/Reese_Witherspoon.html) used for machine learning training. Images are 24-bit RGB images (height, width, channels) with 8 bits for each of R, G, B channel. Explore and print the array.

```python
# load and show the image
FILE = '../fig/Reese_Witherspoon.jpg'
#img = np.array(Image.open(FILE)) # this works as well
img = mpimg.imread(FILE)
imgplot = plt.imshow(img)
```

```python
print(f'The image is a: {type(img)} of shape {img.shape}')
img[3:5, 3:5, :]
```

    The image is a: <class 'numpy.ndarray'> of shape (150, 150, 3)

    array([[[241, 241, 241],
            [242, 242, 242]],

           [[241, 241, 241],
            [242, 242, 242]]], dtype=uint8)

#### Slicing tensors: slice along each axis

```python
# we want to show each color channel
fig, axes = plt.subplots(1, 3, figsize=(10,10))
for i, subplot in zip(range(3), axes):
    temp = np.zeros(img.shape, dtype='uint8')
    temp[:,:,i] = img[:,:,i]
    subplot.imshow(temp)
    subplot.set_axis_off()
plt.show()
```

#### Multiplying Images with a scalar (just for fun, does not really help us in any way)

```python
temp = img
temp = temp * 2
plt.imshow(temp)
```

For more on image manipulation by `matplotlib` see: [matplotlib-images](https://matplotlib.org/3.1.1/tutorials/introductory/images.html)

### Anatomy of an Artificial Neural Network

In Part 1 we hand-made a neural network by writing some simple python functions.  We focused on a regression problem where we tried to learn a function. We practiced using the logistic activation function in a network with multiple nodes, but a single or two hidden layers.  Some of the key observations were:
* Increasing the number of nodes allows us to represent more complicated functions
* The weights and biases have a very big impact on the solution
* Finding the "correct" weights and biases is really hard to do manually
* There must be a better method for determining the weights and biases automatically

We also didn't assess the effects of different activation functions or different network depths.

###
https://www.tensorflow.org/guide/keras

`tf.keras` is TensorFlow's high-level API for building and training deep learning models. It's used for fast prototyping, state-of-the-art research, and production. `Keras` is a library created by François Chollet. After Google released Tensorflow 2.0, the creators of `keras` recommend that "Keras users who use multi-backend Keras with the TensorFlow backend switch to `tf.keras` in TensorFlow 2.0. `tf.keras` is better maintained and has better integration with TensorFlow features".

#### IMPORTANT:  In `Keras` everything starts with a Tensor of N samples as input and ends with a Tensor of N samples as output.

### The 3 parts of an ANN

- **Part 1: the input layer** (our dataset)
- **Part 2: the internal architecture or hidden layers** (the number of layers, the activation functions, the learnable parameters and other hyperparameters)
- **Part 3: the output layer** (what we want from the network)

In the rest of the lab we will practice with end-to-end neural network training

1. Load the data
2. Define the layers of the model.
3. Compile the model.
4. Fit the model to the train set (also using a validation set).
5. Evaluate the model on the test set.
6. Plot metrics such as accuracy.
7. Predict on random images from test set.
8. Predict on  a random image from the web!

```python
seed = 7
np.random.seed(seed)
```

### Fashion MNIST

MNIST, the set of handwritten digits is considered the Drosophila of Machine Learning. It has been overused, though, so we will try a slight modification to it.

**Fashion-MNIST** is a dataset of clothing article images (created by [Zalando](https://github.com/zalandoresearch/fashion-mnist)), consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a **28 x 28** grayscale image, associated with a label from **10 classes**. The creators intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. Each pixel is 8 bits so its value ranges from 0 to 255.

Let's load and look at it!

#### 1. Load the data

```python
%%time
# get the data from keras

# load the data splitted in train and test! how nice!

# normalize the data by dividing with pixel intensity
# (each pixel is 8 bits so its value ranges from 0 to 255)

# classes are named 0-9 so define names for plotting clarity
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

# plot
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(x_train[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[y_train[i]])
plt.show()
```

```python
plt.imshow(x_train[3], cmap=plt.cm.binary)
```

```python
x_train.shape, x_test.shape
```

```python
y_train.shape
```

#### 2. Define the layers of the model.

```python

```

#### 3. Compile the model

```python

```

```python
model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    flatten (Flatten)            (None, 784)               0
    _________________________________________________________________
    dense (Dense)                (None, 154)               120890
    _________________________________________________________________
    dense_1 (Dense)              (None, 64)                9920
    _________________________________________________________________
    dropout (Dropout)            (None, 64)                0
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                650
    =================================================================
    Total params: 131,460
    Trainable params: 131,460
    Non-trainable params: 0
    _________________________________________________________________

```python
tf.keras.utils.plot_model(
    model,
    #to_file='model.png', # if you want to save the image
    show_shapes=True, # True for more details than you need
    show_layer_names=True,
    rankdir='TB',
    expand_nested=False,
    dpi=96
)
```

[Everything you wanted to know about a Keras Model and were afraid to ask](https://www.tensorflow.org/api_docs/python/tf/keras/Model)

#### 4. Fit the model to the train set (also using a validation set)

This is the part that takes the longest.

-----------------------------------------------------------
**ep·och**

noun: epoch; plural noun: epochs. A period of time in history or a person's life, typically one marked by notable events or particular characteristics. Examples: "the Victorian epoch", "my Neural Netwok's epochs".

-----------------------------------------------------------

```python
%%time

# the core of the network training

```

#### Save the model

You can save the model so you do not have `.fit` everytime you reset the kernel in the notebook. Network training is expensive!

For more details on this see [https://www.tensorflow.org/guide/keras/save_and_serialize](https://www.tensorflow.org/guide/keras/save_and_serialize)

```python
# save the model so you do not have to run the code everytime
model.save('fashion_model.h5')

# Recreate the exact same model purely from the file
#model = tf.keras.models.load_model('fashion_model.h5')
```

#### 5. Evaluate the model on the test set.

```python
test_loss, test_accuracy = model.evaluate(x_test, y_test, verbose=0)
print(f'Test accuracy={test_accuracy}')
```

    Test accuracy=0.8866999745368958

#### 6. We learn a lot by studying History! Plot metrics such as accuracy.

You can learn a lot about neural networks by observing how they perform while training. You can issue `kallbacks` in `keras`. The networks's performance is stored in a `keras` callback aptly named `history` which can be plotted.

```python
print(history.history.keys())
```

    dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

```python
# plot accuracy and loss for the test set
fig, ax = plt.subplots(1,2, figsize=(20,6))

ax[0].plot(history.history['accuracy'])
ax[0].plot(history.history['val_accuracy'])
ax[0].set_title('Model accuracy')
ax[0].set_ylabel('accuracy')
ax[0].set_xlabel('epoch')
ax[0].legend(['train', 'val'], loc='best')

ax[1].plot(history.history['loss'])
ax[1].plot(history.history['val_loss'])
ax[1].set_title('Model loss')
ax[1].set_ylabel('loss')
ax[1].set_xlabel('epoch')
ax[1].legend(['train', 'val'], loc='best')
```

#### 7. Now let's use the Network for what it was meant to do: Predict!

```python
predictions = model.predict(x_test)
```

```python
predictions[0]
```

```python
np.argmax(predictions[0]), class_names[np.argmax(predictions[0])]
```

Let's see if our network predicted right! Is the first item what was predicted?

```python
plt.figure()
plt.imshow(x_test[0], cmap=plt.cm.binary)
plt.xlabel(class_names[y_test[0]])
plt.colorbar()
```

**Correct!!** Now let's see how confident our model is by plotting the probability values:

```python
# code source: https://www.tensorflow.org/tutorials/keras/classification
def plot_image(i, predictions_array, true_label, img):
    predictions_array, true_label, img = predictions_array, true_label[i], img[i]
    plt.grid(False)
    plt.xticks([])
    plt.yticks([])

    plt.imshow(img, cmap=plt.cm.binary)

    predicted_label = np.argmax(predictions_array)
    if predicted_label == true_label:
        color = 'blue'
    else:
        color = 'red'

    plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

def plot_value_array(i, predictions_array, true_label):
    predictions_array, true_label = predictions_array, true_label[i]
    plt.grid(False)
    plt.xticks(range(10))
    plt.yticks([])
    thisplot = plt.bar(range(10), predictions_array, color="#777777")
    plt.ylim([0, 1])
    predicted_label = np.argmax(predictions_array)

    thisplot[predicted_label].set_color('red')
    thisplot[true_label].set_color('blue')
```

```python
i = 406
plt.figure(figsize=(6,3))
plt.subplot(1,2,1)
plot_image(i, predictions[i], y_test, x_test)
plt.subplot(1,2,2)
plot_value_array(i, predictions[i],  y_test)
plt.show()
```

#### 8. Predicting in the real world

Let's see if our network can generalize beyond the MNIST fashion dataset. Let's give it an random googled image of a boot. Does it have to be a clothing item resembling the MNIST fashion dataset? Can it be a puppy?

**Download an image from the internet and resize it to 28x28.**

```python
# your code here.
# Resize it to 28 x 28 and one channel (you could do this outside the notebook)
# make into one channel and see .shape

```

`tf.keras` models are optimized to make predictions on a batch, or collection, of examples at once. Accordingly, even though you're using a single image, you need to add it to a list:

```python
# your code here
# Add the image to a batch where it's the only member.

```

```python
# your code here
# print the prediction

```


# Two-body problems

This module covers solution methods for two-body problems in the **perifocal frame**:
given some information about an orbit, how can we find the new position and velocity
after some change in true anomaly or after some time?

This will cover two-body problem solutions using Lagrange coefficients,
Kepler problems (involving time), the Kepler problem with universal variables,
and solving the orbital equation of motion numerically from initial conditions.

```python
import numpy as np
%matplotlib inline
from matplotlib import pyplot as plt
```

```python
# these lines are only for helping improve the display
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('pdf', 'png')
plt.rcParams['figure.dpi']= 150
plt.rcParams['savefig.dpi'] = 150
```

## Two-body problems in perifocal frame

**The problem:** given the initial position vector $\vec{r}_0$ and initial velocity vector $\vec{v}_0$ (the state vectors),
known at some instant in time, find the new state vectors $\vec{r}$ and $\vec{v}$ after the true anomaly changes by $\Delta \theta$.

:::{figure,myclass} fig-twobody

Diagram of two-body problem with state vectors.
:::

Since the state vectors $\vec{r}$ and $\vec{v}$ will always remain in the same two-dimensional plane,
they must be able to be expressed as linear combinations of
$\vec{r}_0$ and $\vec{v}_0$:

$$
\begin{align}
\vec{r} &= f \vec{r}_0 + g \vec{v}_0 \\
\vec{v} &= \dot{f} \vec{r}_0 + \dot{g} \vec{v}_0 \;,
\end{align}
$$

where $f, g, \dot{f}, \dot{g}$ are the **Lagrange coefficients**, which are constant scalars.

Note that by substituting these into the definition of specific angular momentum $\vec{h}$,
which must remain constant, we can obtain the **Lagrange identity**:

$$
\begin{align}
\vec{h} = \vec{r}_0 \times \vec{v}_0 &= \vec{r} \times \vec{v} \\
&= \left( f \vec{r}_0 + g \vec{v}_0 \right) \times \left( \dot{f} \vec{r}_0 + \dot{g} \vec{v}_0 \right) \\
&= f \dot{f} \vec{r}_0 \times \vec{r}_0 + f \dot{g} \vec{r}_0 \times \vec{v}_0 + g \dot{f} \vec{v}_0 \times \vec{r}_0 + g \dot{g} \vec{v}_0 \times \vec{v}_0 \\
&= f \dot{g} \vec{r}_0 \times \vec{v}_0 + g \dot{f} \vec{v}_0 \times \vec{r}_0 \\
&= f \dot{g} \vec{r}_0 \times \vec{v}_0 - g \dot{f} \vec{r}_0 \times \vec{v}_0 \\
\vec{r}_0 \times \vec{v}_0 &= \left( f \dot{g} - g \dot{f} \right) \vec{r}_0 \times \vec{v}_0 \\
\therefore f \dot{g} - g \dot{f} &= 1 \;,
\end{align}
$$

which provides a potentially helpful relationship between the coefficients—given three of them, you can easily find the fourth.

(sec:lagrange)=
### Lagrange coefficients

Via a complicated derivation, we can find closed-form expressions for the Lagrange coefficients:

$$
\begin{align}
f &= 1 - \frac{\mu r}{h^2} \left( 1 - \cos \Delta \theta \right) \\
g &= \frac{r r_0}{h} \sin \Delta \theta \\
\dot{f} &= \frac{\mu}{h} \frac{1 - \cos \Delta \theta}{\sin \Delta \theta} \left[ \frac{\mu}{h^2} \left( 1 - \cos \Delta \theta \right) - \frac{1}{r_0} - \frac{1}{r} \right] \\
\dot{g} &= 1 - \frac{\mu r_0}{h^2} \left( 1 - \cos \Delta \theta \right) \;,
\end{align}
$$ (lagrange)

where $r_0 = |\vec{r}_0| = \sqrt{\vec{r}_0 \cdot \vec{r}_0}$ is the magnitude of the initial position vector,
$v_0 = |\vec{v}_0| = \sqrt{\vec{v}_0 \cdot \vec{v}_0}$ is the magnitude of the initial velocity vector,
$v_{r0} = \vec{r}_0 \cdot \vec{v}_0 / r_0$ is the radial component of the initial velocity vector (the projection along the apse line),
$h = |\vec{h}|$ is the magnitude of the specific angular momentum,
and the new radial distance is

$$
r = \frac{h^2}{\mu} \frac{1}{1 + \left( \frac{h^2}{\mu r_0} - 1 \right) \cos \Delta \theta - \frac{h v_{r0}}{\mu} \sin \Delta \theta} \;.
$$

```{note}
While examples may show elliptical orbits, this approach applies regardless of orbit.
```

With these equations, we can now solve the problem.

## Example workflow

The workflow for solving a problem follows these steps:

1. Calculate the magnitudes and necessary quantities: $r_0 = |\vec{r}_0|$, $v_0 = |\vec{v}_0|$,
$\vec{h} = \vec{r} \times \vec{v}$, $h = |\vec{h}|$, and $v_{r0} = \frac{ \vec{r}_0 \cdot \vec{v}_0 }{ |r_0| }$.

2. An optional step is to determine the initial true anomaly, and from
   that the eccentricity of the orbit $e$ (which would tell us what
   kind of orbit this is):

   $$
   \begin{align}
   \theta_0 &= \tan^{-1} \left( \frac{r_{0y}}{r_{0x}} \right) \\
   r_0 &= \frac{h^2 / \mu}{1 + e \cos \theta_0 } \\
   \rightarrow e &= \left( \frac{h^2 / \mu}{r_0} - 1 \right) / \cos \theta_0
   \end{align}
   $$

3. Calculate the new radial distance:

   $$
   r = \frac{h^2}{\mu} \frac{1}{1 + \left( \frac{h^2}{\mu r_0} - 1 \right) \cos \Delta \theta - \frac{h v_{r0}}{\mu} \sin \Delta \theta }
   $$

4. Calculate the Lagrange coefficients $f, g, \dot{f}, \dot{g}$, using Equation {eq}`lagrange`.

5. Finally, calculate the new position and velocity vectors:

   $$
   \begin{align}
   \vec{r} &= f \vec{r}_0 + g \vec{v}_0 \\
   \vec{v} &= \dot{f} \vec{r}_0 + \dot{g} \vec{v}_0
   \end{align}
   $$

Here's example code in Matlab and Python for doing these steps:

````{tabbed} matlab

```{code-block} matlab

mu = 398.6e3; % km^3/s^2
r0 = [. . .];
v0 = [. . .];
delta_theta = .;

% 1. Calculate magnitudes
r0_mag = sqrt(dot(r0, r0));
v0_mag = sqrt(dot(v0, v0));

h = cross(r0, v0);
h_mag = sqrt(dot(h, h));
v_r0 = dot(r0, v0) / r0_mag;

% 2. Get eccentricity
theta_0 = atan(r0(2) / r0(1));
e = ((h_mag^2/(mu * r0_mag)) - 1) / cos(theta_0);

% 3. Get new distance
r_mag = (h_mag^2/mu) / (1 + ((h_mag^2/(mu*r0_mag)) - 1)*cos(delta_theta) - h_mag*v_r0 * sin(delta_theta)/mu);

% 4. Get Lagrange coefficients
f = 1 - mu * r_mag * (1 - cos(delta_theta)) / h_mag^2;
g = r_mag * r0_mag * sin(delta_theta) / h_mag;
fdot = (mu / h_mag)*(1 - cos(delta_theta))*( mu*(1 - cos(delta_theta))/h_mag^2 - (1/r0_mag) - (1/r_mag)) / sin(delta_theta);
gdot = 1 - mu*r0_mag*(1 - cos(delta_theta)) / h_mag^2;

% 5. Calculate new vectors:
r = f*r0 + g*v0;
v = fdot*r0 + gdot*v0;

```
````

````{tabbed} python

```{code-block} python
import numpy as np

mu = 398.6e3 # km^3/s^2
r0 = np.array([. . .])
v0 = np.array([. . .])
delta_theta = .

# 1. Calculate magnitudes
r0_mag = np.sqrt(np.dot(r0, r0))
v0_mag = np.sqrt(np.dot(v0, v0))

h = np.cross(r0, v0)
h_mag = np.sqrt(np.dot(h, h))
v_r0 = np.dot(r0, v0) / r0_mag

# 2. Get eccentricity
theta_0 = np.arctan(r0[1] / r0[0])
e = ((h_mag^2/(mu * r0_mag)) - 1) / np.cos(theta_0)

# 3. Get new distance
r_mag = (h_mag^2/mu) / (
    1 + ((h_mag^2/(mu*r0_mag)) - 1)*np.cos(delta_theta) - h_mag*v_r0 * np.sin(delta_theta)/mu
    )

# 4. Get Lagrange coefficients
f = 1 - mu * r_mag * (1 - np.cos(delta_theta)) / h_mag^2
g = r_mag * r0_mag * np.sin(delta_theta) / h_mag
fdot = (mu / h_mag)*(1 - np.cos(delta_theta))*(
    mu*(1 - np.cos(delta_theta))/h_mag^2 - (1/r0_mag) - (1/r_mag)
    ) / np.sin(delta_theta)
gdot = 1 - mu*r0_mag*(1 - np.cos(delta_theta)) / h_mag^2

# 5. Calculate new vectors:
r = f*r0 + g*v0
v = fdot*r0 + gdot*v0
```
````

## Kepler problems

Kepler problems are two-body problems that involve time:
find the time of flight given information about an orbit and a change in true anomaly,
or find the change in true anomaly (and new state vectors) given a time of flight.

### Example: find time of flight

**The problem:** Given the orbital eccentricity $e$, the initial position vector
$\vec{r} = \vec{r}_p$ (the position starts at perigee), and the change in
true anomaly $\Delta \theta$, find the time of flight $\Delta t$.
(The solution procedure does not depend on the initial position being at perigee,
this is just to simplify things a bit.)

:::{figure,myclass} fig-kepler-time

Diagram of Kepler problem for finding time of flight.
:::

This problem requires a different approach. We can relate the transverse/angular
component of velocity to the change in true anomaly:

$$
\begin{align}
V_{\theta} = \frac{h}{r} &= r \dot{\theta} = r \frac{d\theta}{dt} \\
r^2 \, d\theta &= h \, dt \\
\text{but } r &= \frac{h^2 / \mu}{1 + e \cos \theta} \\
\rightarrow \left( \frac{h^2 / \mu}{1 + e \cos \theta} \right)^2 d\theta &= h \, dt \\
\int_{\theta_0 = 0}^{\theta = \Delta \theta} \left( \frac{1}{1 + e \cos \theta} \right)^2 d\theta &= \int_0^t \frac{\mu^2}{h^3} dt \\
\end{align}
$$ (kepler_integral)

This is the **Kepler integral**, and it has a closed-form solution!
(Fortunately, the bounds of the integral do not matter—just the $\Delta \theta$.)

However, depending on the value of eccentricity $e$, the solution takes on different forms.
So this approach depends on knowing eccentricity! We will cover a more-general method that
does not require this information soon.

For an elliptic orbit, the solution to the Kepler integral is

$$
\begin{align}
\frac{2 \pi}{T} t &= M_e = E - e \sin E \\
\tan \frac{E}{2} &= \sqrt{\frac{1-e}{1+e}} \tan \frac{\Delta \theta}{2} \;,
\end{align}
$$

where $T$ is the period of an elliptical orbit,
$E$ is the eccentric anomaly (which *must* be in radians), and
$M_e$ is the mean anomaly.

:::{warning}
Make sure you use the correct form of the equations based on the value of eccentricity!
And make sure your anomaly values are in **radians**!
:::

The solution procedure for solving this problem is then:

1. Calculate the magnitude of specific angular momentum $h$ using the known perigee distance:

   $$
   r_p = \frac{h^2 / \mu}{1 + e} \rightarrow h
   $$

2. Calculate the apogee distance, and then use this to get the semi-major axis:

   $$
   \begin{align}
   r_a &= \frac{h^2 / \mu}{1 - e} \\
   a &= \frac{r_p + r_a}{2}
   \end{align}
   $$

3. For an elliptic orbit, calculate the period:

   $$
   T = \frac{2\pi}{\sqrt{\mu}} a^{3/2}
   $$

4. Calculate the eccentric anomaly $E$, and then use this to get $t$.

## Example: find change in true anomaly

**The problem:** now, given the eccentricity $e$, initial position vector $\vec{r}=\vec{r}_p$,
and the time of flight $\Delta t$, find the associated change in true anomaly $\Delta \theta$, and then the new state vectors ($\vec{r}$ and $\vec{v}$).

As we will quickly see, this problem is a bit trickier, but still solveable.

The following solution steps for an **elliptic** orbit, since the equations only apply
for this case, but the procedure will be the same for other orbits:

1. Calculate the semi-major axis $a$ and orbital period $T$ using the perigee distance.

2. Calculate the mean anomaly using the known time of flight:

   $$
   M_e = \frac{2\pi}{T} t
   $$

   Next, we need to to find the eccentric anomaly $E$, using the $M_e = E - e \sin E$,
   but this is a transcendental equation with no closed-form solution.
   So, we need to use an iterative solution procedure, for example using the
   `fzero()` function in Matlab or the `scipy.optimize.root_scalar()` function in Python
   (which requires [SciPy](https://www.scipy.org)).
   Either way, we need a good initial guess, otherwise even these robust algorithms
   may not converge to the correct solution.

3. Use the **Prussing estimate** as a reasonable first guess for $E$:

   $$
   E_0 = M_e + \frac{e}{2} \;.
   $$

   Then, use an equation solver to find $E$:

   ````{tabbed} matlab
   ```{code-block} matlab
   E0 = M_e + e/2;
   E = fzero(@f, E0);

   function zero = f(E)
     M_e = 2 * pi * t / T;
     zero = M_e - (E - e * sin(E));
   end
   ```
   ````

   ````{tabbed} python
   ```{code-block} python
   import numpy as np
   from scipy.optimize import root_scalar

   def eccentric_anomaly(E):
       M_e = 2 * np.pi * t / T
       return M_e - (E - e * np.sin(E))

   E0 = M_e + e/2
   sol = root_scalar(eccentric_anomaly, x0=E0)
   ```
   ````

4. Use $E$ to find the value of the change in true anomaly:

   $$
   \tan \frac{\Delta \theta}{2} = \sqrt{\frac{1+e}{1-e}} \tan \frac{E}{2} \;.
   $$

5. Then, with $\Delta \theta$, you can find the new state vectors by treating this like a two-body problem and using the {ref}`sec:lagrange`. Since this example starts at perigee, we know the initial position and velocity vectors:

   $$
   \begin{align}
   \vec{r}_0 &= \vec{r}_p = \left[ r_p, 0, 0 \right] \\
   \vec{v}_0 &= \left[ 0, \frac{h}{r_p}, 0 \right]
   \end{align}
   $$

## Universal variable approach

The above approaches are all limited by the need to know the eccentricity of the orbit,
which in general you would not know. The **universal variable** approach is a more-general
approach that applies consistently regardless of orbital eccentricity.

**The problem:** Given initial position vector $\vec{r}_0$,
initial velocity vector $\vec{v}_0$, and some time of flight $\Delta t$,
find the new state vectors $\vec{r}$ and $\vec{v}$.

This approach is based on the definition of the universal anomaly $\chi$, using:

$$
\dot{\chi} \equiv \frac{\sqrt{\mu}}{r} \;,
$$

known as the Sundmann transformation.

Through a complicated derivation, this leads to the universal variable formulation of Kepler's equation:

$$
\sqrt{\mu} \Delta t = \frac{r_0 v_{r0}}{\sqrt{\mu}} \chi^2 C \left( \alpha \chi^2 \right) + (1 - \alpha r_0) \chi^3 S \left( \alpha \chi^2 \right) + r_0 \chi \;,
$$

which is a highly nonlinear transcendental equation, where
$\alpha = \frac{1}{a} = \frac{2}{r_0} - \frac{v_0^2}{\mu}$.
But, this is solveable via iterative methods!

However, as above, we need a good first guess to help the solvers converge,
and we also need to define the functions $C()$ and $S()$.

1. First, we can get a first guess for $\chi$ using the **Chobotov approximation**:

   $$
   \chi_0 = \frac{\sqrt{\mu}}{|a|} \Delta t \;.
   $$

2. Then, we need to define the **Stumpff functions**:

   $$
   \begin{align}
   C(z) &= \frac{\cosh \left(\sqrt{-z}\right) - 1}{-z} \\
   S(z) &= \left\lvert \frac{ \sinh \left(\sqrt{-z}\right) - \sqrt{-z}}{ \left(\sqrt{-z}\right)^3 } \right\rvert \;,
   \end{align}
   $$

   where $z = \alpha \chi^2$. Take care to include the absolute value when evaluating $S(z)$, which must be positive.

3. Then, solve iteratively for $\chi$, using (for example) `fzero()` in Matlab or `scipy.optimize.root_scalar()` in Python.

4. Once you obtain $\chi$, you can solve for the Lagrange coefficients $f$ and $g$ with the **universal Lagrange equations**:

   $$
   \begin{align}
   f &= 1 - \frac{\chi^2}{r_0} C \left( \alpha \chi^2 \right) \\
   g &= \Delta t - \frac{1}{\sqrt{\mu}} \chi^3 S \left( \alpha \chi^2 \right) \;.
   \end{align}
   $$

   Then, calculate the new position vector: $\vec{r} = f \vec{r}_0 + g \vec{v}_0$.

5. Calculate the magnitude of $\vec{r}$: $|\vec{r}|$.

6. Then, we can get $\dot{f}$ and $\dot{g}$ to find the velocity vector:

   $$
   \begin{align}
   \dot{f} &= \frac{\sqrt{\mu}}{r r_0} \left[ \alpha \chi^3 S \left( \alpha \chi^2 \right) - \chi \right] \\
   \dot{g} &= 1 - \frac{\chi^2}{r} C \left( \alpha \chi^2 \right) \\
   \vec{v} &= \dot{f} \vec{r}_0 + \dot{g} \vec{v}_0 \;.
   \end{align}
   $$

## Solve equation of motion numerically

An alternate strategy for finding the position and velocity of an orbiting body after some time of flight, given the initial position and velocity, is to numerically integrate the orbital equation of motion:

$$
\vec{ \ddot{r} } + \frac{\mu}{r^3} \vec{r} = 0 \;,
$$

based on initial conditions $\vec{r}_0$ and $\vec{v}_0$ defined at an initial time $t_0$.
Integrating this forward in time would give us the position as a function of time:
$x(t)$, $y(t)$, and $z(t)$.

If you recall the methods for integrating second-order ordinary differential equations (ODEs),
we can solve this by decomposing it into six first-order ODEs:

$$
\begin{align}
z_1 &= x \rightarrow \dot{z}_1 = z_4 \\
z_2 &= y \rightarrow \dot{z}_2 = z_5 \\
z_3 &= z \rightarrow \dot{z}_3 = z_6 \\
z_4 &= \dot{x} \rightarrow \dot{z}_4 = \ddot{x} = -\frac{\mu}{r^3} z_1 \\
z_5 &= \dot{y} \rightarrow \dot{z}_5 = \ddot{y} = -\frac{\mu}{r^3} z_2 \\
z_6 &= \dot{z} \rightarrow \dot{z}_6 = \ddot{z} = -\frac{\mu}{r^3} z_3 \;,
\end{align}
$$

where the radial position magnitude is $r = \sqrt{z_1^2 + z_2^2 + z_3^2}$.

We can then integrate using `ode45()` in Matlab, or `solve_ivp()` in Python/SciPy:

````{tabbed} matlab

```{code-block} matlab

clear, clf, clc;

r0 = [. . .];
v0 = [. . .];
T = .;

options = odeset('RelTol', 1e-6);

[T, Z] = ode45(@rhs, [0 2*T], [r0, v0], options);
plot3(Z(:,1), Z(:,2), Z(:,3));

% function for ODE system
function dzdt = rhs(t, z)
  mu = 398.6e3; % km^3/s^2
  r = sqrt(z(1)^2 + z(2)^2 + z(3)^2);
  dzdt = zeros(6,1);
  dzdt(1) = z(4);
  dzdt(2) = z(5);
  dzdt(3) = z(6);
  dzdt(4) = (-mu/r^3) * z(1);
  dzdt(5) = (-mu/r^3) * z(2);
  dzdt(6) = (-mu/r^3) * z(3);
end
```
````

````{tabbed} python

```{code-block} python
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt

def rhs(t, z):
    # 3D orbital motion ODE
    mu = 398.6e3 # km^3/s^2
    r = np.sqrt(z[0]**2 + z[1]**2 + z[2]**2)
    dzdt = np.zeros(6)
    dzdt[0] = z[3]
    dzdt[1] = z[4]
    dzdt[2] = z[5]
    dzdt[3] = (-mu/r**3) * z[0]
    dzdt[4] = (-mu/r**3) * z[1]
    dzdt[5] = (-mu/r**3) * z[2]
    return dzdt

r0 = [. . .]
v0 = [. . .]
T = .

sol = solve_ivp(rhs, [0, 2*T], np.array(r0 + v0))

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot3D(sol.y[0,:], sol.y[1,:], sol.y[2.:])
```
````

## Summary of Kepler's equations

:::{figure,myclass} fig-anomaly

Diagram showing the geometric meaning of the mean and eccentric anomaly.
Original sources: [PNG by Brews ohare](https://commons.wikimedia.org/wiki/File:Anomalies.PNG),
[SVG by CheCheDaWaff](https://commons.wikimedia.org/wiki/File:Mean_Anomaly.svg).
Shared under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0)
:::

### Elliptic orbit

$$
\begin{align}
M_e &= \frac{2 \pi}{T} t = \sqrt{\frac{\mu}{a^3}} t \\
M_e &= E - e \sin E \\
\tan \frac{\theta}{2} &= \sqrt{\frac{1+e}{1-e}} \tan \frac{E}{2} \\
r &= a \left( 1 - e \cos E \right) \\
v^2 &= \mu \left( \frac{2}{r} - \frac{1}{a} \right) \\
r_p &= a (1 - e) \;,
\end{align}
$$

where $M_e$ is the mean anomaly and $E$ is the eccentric anomaly.

### Hyperbolic orbit

```{margin} Typo in SMAD!
Be careful, there is a typo in the equation relating hyperbolic anomaly and true anomaly
in *Space Mission Engineering: The New SMAD*.
```

$$
\begin{align}
M_h &= \sqrt{\frac{\mu}{(-a)^3}} \, t = \frac{\mu^2}{h^3} \left( e^2 - 1 \right)^{3/2} t\\
M_h &= e \sinh F - F \\
\tan \frac{\theta}{2} &= \sqrt{\frac{e+1}{e-1}} \tanh \frac{F}{2} \\
r &= a \left( 1 - e \cosh F \right) \\
v^2 &= \mu \left( \frac{2}{r} - \frac{1}{a} \right) \\
r_p &= a (1 - e) \;,
\end{align}
$$

where $M_h$ is the mean anomaly and $F$ is the hyperbolic anomaly.

```python

```


# The Bayesian Bootstrap Is Not a Free Lunch

Some recent work has suggested that we can solve computationally difficult, multi-modal Bayesian posterior calculations with optimization and bootstrap sampling.  There are many variations such methods; for shorthand I will simply refer to them collectively as Bayesian bootstrap (BB) methods.  It is true that BB can be practically quite informative, and, in certain problems, much more useful than any other out-of-the-box Bayesian methods which fail utterly to explore multiple modes.  Nevertheless, I will argue that BB methods do not, in general, produce an accurate approximation of a Bayesian posterior.

I will argue that when we are actually interested in a particular multi-modal parametric posterior but simply cannot sample from it, then BB methods will not, in general, approximate the desired posterior.  Of course, the BB is the right thing to do when we actually do have a proper Dirichlet process prior on a space of data distributions, and are interested in the output of a particular optimization problem as a posterior quantity.  But naively replacing a parametric Bayesian posterior quantity with a BB quantity risks (a) misinterpreting what is meant by an optimum and (b) misinterpreting what is meant by uncertainty.  I stress that I do not mean to argue that the BB is not useful.  A flawed answer can better than no answer at all.  But a flawed answer is even more useful when its potential shortcomings are well understood, and my aim is to clearly and intuitively illustrate these potential shortcomings.

### Model 1

I will begin with a toy model which is simple and contrived.  Its simplicity is such that we can clearly see what the right Bayesian answer should be, and how the BB fails to replicate it.  I will argue in future, less simple examples that its contrivance is not so severe, and that the failure of the BB in this simple model illustrate its potential failures in more complex cases.

Suppose we observe a vector of scalar, real-valued data $x=(x_1,...,x_N)$, drawn from the following model:

$$
\begin{align}
p(\tau) &= \mathrm{Gamma}(\tau | 1, 1)\\
p(x | \tau) &= \prod_{n=1}^N \mathcal{N}(x_n | \tau, 1).
\end{align}
$$

The unkown mean parameter $\tau$ is positive, and all other aspects of the data generating process are known.  As stated, this is a simple problem.  Now, let us suppose that we are actually interested in a quantity $\theta \in \mathbb{R}$, given by

$$
\begin{align}
z &\in \{-1, 1\}\\
p(z=1) &= 0.8\\
\theta &= z \sqrt{\tau}.
\end{align}
$$

Obviously, $|\theta| = \tau$, and $p(\theta  > 0 | x) = 0.8$ because the data $x$ is independent of $z$, the sign of $\theta$.

Effectively, the sign of $\theta$ is identified only by the prior, and the posterior $p(\theta \vert x)$ has both a positive mode and a negative mode, which it occupies with respective probabilities $0.8$ and $0.2$.  We can sample from the posterior of $p(\theta | x)$ by sampling from $p(\tau | x)$, drawing $z$ from its prior, and setting $\theta = z \tau$.

Can the BB reproduce a reasonable approximation of the posterior $p(\theta | x)$ in this trivial case?  Below, I will show that it cannot.  The reason is twofold and general.  Recall that the probability probability of a lying near a particular mode is determined by the posterior mass under the mode.  For example, in the present case, $p(\theta > 0 | x) = \int_{0}^{\infty} p(\theta | x) d\theta = 0.8$.  However, in general,

- The frequency with which an optimum finds a mode is not, in general, determined by the relative mass of a mode and
- The variability of an optimum induced by the bootstrap is not, in general, determined by the relative mass of a mode.

```R
library(tidyverse)
library(rstan)
library(gridExtra)
library(numDeriv)
library(repr)

# Change plot size to 4 x 3
options(repr.plot.width=7, repr.plot.height=4)

options(mc.cores=4)
rstan_options(auto_write=TRUE)

setwd("/home/rgiordan/Documents/git_repos/variational_bayes/bayesian_bootstrap")
```

    ── Attaching packages ─────────────────────────────────────── tidyverse 1.2.1 ──
    ✔ ggplot2 3.1.0     ✔ purrr   0.2.5
    ✔ tibble  1.4.2     ✔ dplyr   0.7.8
    ✔ tidyr   0.8.1     ✔ stringr 1.3.1
    ✔ readr   1.1.1     ✔ forcats 0.3.0
    ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
    ✖ dplyr::filter() masks stats::filter()
    ✖ dplyr::lag()    masks stats::lag()
    Loading required package: StanHeaders
    rstan (Version 2.18.2, GitRev: 2e1f913d3ca3)
    For execution on a local, multicore CPU with excess RAM we recommend calling
    options(mc.cores = parallel::detectCores()).
    To avoid recompilation of unchanged Stan programs, we recommend calling
    rstan_options(auto_write = TRUE)

    Attaching package: ‘rstan’

    The following object is masked from ‘package:tidyr’:

        extract

    Attaching package: ‘gridExtra’

    The following object is masked from ‘package:dplyr’:

        combine

```R
EvalLogPost <- function(theta, fit, z_prob) {
  upars <- unconstrain_pars(fit, list(tau=theta ^ 2))
  lp <- log_prob(fit, upars) + ifelse(theta > 0, log(z_prob), log(1 - z_prob))
}

GetThetaGridLp <- function(fit, z_prob, theta_max=NA) {
  draws <- extract(fit)
  if (!is.finite(theta_max)) {
      theta_max <- 1.3 * max(abs(draws$theta))
  }
  theta_grid <- seq(-1 * theta_max, theta_max, length.out=1000)
  lp <- sapply(theta_grid, EvalLogPost, fit, z_prob)
  return(data.frame(theta=theta_grid, lp=lp))
}

DrawAndFit <- function(N, z_prob, tau_prior_shape=1, tau_prior_rate=1, x_sd=1) {
  tau_true <- rgamma(1, shape=tau_prior_shape, rate=tau_prior_rate)
  x <- rnorm(N, mean=tau_true, sd=x_sd)
  # Also save tau_true even though Stan doesn't use it.
  bb_data <- list(x=x, N=N, prob=z_prob, x_sd=x_sd,
                  tau_prior_shape=tau_prior_shape, tau_prior_rate=tau_prior_rate,
                  tau_true=tau_true)

  # The Stan model will exhibit some divergent transitions near 0, but that's
  # not a real problem here.
  fit <- stan(file='multimodal_v1.stan', data=bb_data)
  return(list(fit=fit, bb_data=bb_data))
}
```

Let's draw a dataset and look at the posterior of $\theta$.

```R
z_prob <- 0.8
initial_fit <- DrawAndFit(N=50, z_prob=z_prob)
fit <- initial_fit$fit
```

    hash mismatch so recompiling; make sure Stan code ends with a blank line

```R
draws <- extract(fit)
cat("P(\\theta > 0) \\approx ", mean(draws$theta > 0), "\n")

theta_lp <- GetThetaGridLp(fit, z_prob)
grid.arrange(
  ggplot(theta_lp) +
    geom_line(aes(x=theta, y=lp)) +
    ggtitle("Log posterior")
,
  ggplot(theta_lp) +
    geom_line(aes(x=theta, y=exp(lp))) +
    ggtitle("Posterior")
,
    ggplot() +
      geom_histogram(aes(x=draws$theta)) +
      ggtitle("Draws")
, ncol=3
)
```

Now, let's see whether the Bayesian bootstrap will reproduce $p(\theta > 0 | x) = 0.8$.

First of all, note that reasonable optimizers (e.g. well-conditined gradient ascent) will find the positive mode if the initial value is positive, and the negative mode if the initial value is negative.  So, irrespective of the noise induced by the bootstrap, the BB estimate of $P(\theta > 0)$ will be precisely the probability that the initial value for optimization is positive.  With uniform random restarts, this probability will be $0.5$, not $0.8$.

A resonable question is whether drawing bootstrap samples of $x$ can overcome this obvious shortcoming.  Let's take a look.

```R
DrawBootstrapSample <- function(x) {
    return(x[sample.int(length(x), replace=TRUE)])
}

GetBootstrapFit <- function(initial_fit) {
  x_boot <- DrawBootstrapSample(initial_fit$bb_data$x)
  bb_data_boot <- initial_fit$bb_data
  bb_data_boot$x <- x_boot

  # We only need the fit object, not the draws, so set samples to be very low and
  # suppress output and warnings.
  options(warn=-1)
  invisible(capture.output(
      fit_boot <- stan(file='multimodal_v1.stan',
                       data=bb_data_boot, iter=1, chains=1)))
  options(warn=0)
  return(list(x_boot=x_boot, fit_boot=fit_boot))
}

```

```R
theta_max <- 2 * max(abs(draws$theta))
mean_abs_theta <- mean(abs(draws$theta))

boot_lp_df <- data.frame()
for (b in 1:10) {
    print(b)
    boot_fit <- GetBootstrapFit(initial_fit)
    theta_lp <- GetThetaGridLp(boot_fit$fit, z_prob, theta_max)
    boot_lp_df <- bind_rows(boot_lp_df, theta_lp %>% mutate(b=as.character(b)))
}
```

    [1] 1
    [1] 2
    [1] 3
    [1] 4
    [1] 5
    [1] 6
    [1] 7
    [1] 8
    [1] 9
    [1] 10

As we can see, the bootstrap variability is never such that the lower mode is larger than the higher mode.  Consequently, if our optimization method finds the global optimum rather than a local optimum, it will estimate $p(\theta > 0 | x) = 1$.

```R
boot_lp_df <-
    boot_lp_df %>%
    group_by(b) %>%
    mutate(lp_norm=lp - log(sum(exp(lp)))) # Were this to be a density you would need the delta x

ggplot(boot_lp_df) +
    geom_line(aes(x=theta, y=exp(lp_norm), group=b), alpha=0.2) +
    xlim(-2.5, 2.5)
```

Let's check this formally by actually optimizing a number of bootstrap samples.

```R
GetMAP <- function(fit, init_val=NA, width=NA, pos=TRUE) {
  # init_val and width are in the log space.
  x_sign <- (sign(pos) * 2 - 1)
  Objective <- function(log_x) {
    x <- x_sign * exp(log_x)
    # The log probability doesn't matter because we're only finding the optimum in
    # one of the two modes.
    return(-1 * EvalLogPost(x, fit, 0.5))
  }
  if (!is.finite(init_val)) {
    init_val  <- log(mean(sqrt(extract(fit, "tau")$tau)))
  }
  if (!is.finite(width)) {
    width <- 4 * sd(log(sqrt(extract(fit, "tau")$tau)))
  }
  opt <- optim(init_val, Objective, method="Brent",
               upper=init_val + width, lower=init_val - width)
  opt$par_theta <- x_sign * exp(opt$par)
  return(opt)
}

GetBootstrapResult <- function(fit_boot, z_prob, init_val=NA, width=NA) {
  boot_opt <- GetMAP(fit_boot, pos=TRUE, init_val=init_val, width=width)
  lp_neg <- EvalLogPost(-1 * boot_opt$par_theta, fit_boot, z_prob)
  lp_pos <- EvalLogPost(boot_opt$par_theta, fit_boot, z_prob)
  lp_hess_neg <- hessian(
      function(x) EvalLogPost(x, fit_boot, z_prob),
      -1 * boot_opt$par_theta)[1, 1]
  lp_hess_pos <- hessian(
      function(x) EvalLogPost(x, fit_boot, z_prob),
      boot_opt$par_theta)[1, 1]

  return(data.frame(
    theta_opt=ifelse(lp_pos > lp_neg, 1, -1) * boot_opt$par_theta,
    lp_pos=lp_pos,
    lp_neg=lp_neg,
    lp_hess_neg=lp_hess_neg,
    lp_hess_pos=lp_hess_pos
  ))
}
```

```R
init_val  <- log(mean(sqrt(draws$tau)))
width <- 4 * sd(log(sqrt(draws$tau)))

B <- 50
boot_df <- data.frame()
boot_time <- Sys.time()

for (b in 1:B) {
  fit_boot <- GetBootstrapFit(initial_fit)$fit_boot
  boot_df <- bind_rows(
      boot_df,
      GetBootstrapResult(fit_boot, z_prob, init_val=init_val, width=width))
}
boot_time <- Sys.time() - boot_time

```

```R
cat("BB estimate of P(\\thetasign(boot_df$theta_opt): ", mean(boot_df$theta_opt > 0), "\n")
```

    BB estimate of P(\thetasign(boot_df$theta_opt):  1

### Summary

In summary, there are two flavors of the BB: one which finds a local optimum, and one which finds a global optimum.  The former estimates $p(\theta > 0 | x) \approx 0.5$, and the latter estimates $p(\theta > 0 | x) \approx 1.0$.  Neither is a good estimate of the true $p(\theta > 0 | x) = 0.8$.

What has gone wrong?  The value chosen by the BB depends on two things:
- The domain of attraction and the distribution of the starting point
- The variability of the height of the optima under data resampling.

Neither of these quantites necessarily have anything to do with the real posterior probability, which is the mass under the mode.

### Model

For reference, here is a dump of the Stan model.

```R
cat(scan("multimodal_v1.stan", what="character", sep="\n"), sep="\n")
```

    data {
      int<lower=0> N;
      real x[N];
      real<lower=0> x_sd;
      real<lower=0, upper=1> prob;
      real<lower=0> tau_prior_shape;
      real<lower=0> tau_prior_rate;
    }
    parameters {
      real<lower=0> tau;
    }
    model {
      tau ~ gamma(tau_prior_shape, tau_prior_rate);
      x ~ normal(tau, x_sd);
    }
    generated quantities {
      real theta;
      int z;
      z = 2 * bernoulli_rng(prob) - 1;
      theta = z * sqrt(tau);
    }