# Almgren and Chriss Model For Optimal Execution of Portfolio Transactions

### Introduction

We consider the execution of portfolio transactions with the aim of minimizing a combination of risk and transaction costs arising from permanent and temporary market impact. As an example, assume that you have a certain number of stocks that you want to sell within a given time frame. If you place this sell order directly to the market as it is, transaction costs may rise due to temporary market impact. On the other hand, if you split up into pieces in time, cost may rise due to volatility in the stock price.

[Almgren and Chriss](https://cims.nyu.edu/~almgren/papers/optliq.pdf) provided a solution to this problem by assuming the permanent and temporary market impact functions are linear functions of the rate of trading, and that stock prices follow a discrete arithmetic random walk.

In this notebook, we will take a look at the model used by Almgren and Chriss to solve the optimal liquidation problem. We will start by stating the formal definitions of *trading trajectory*, *trading list*, and *trading strategy* for liquidating a single stock.

### Trading Trajectory, Trading List, and Trading Strategy

We define trading trajectory, trading list, and trading strategy just as Almgren and Chriss did in their [paper](https://cims.nyu.edu/~almgren/papers/optliq.pdf). Suppose we hold $X$ shares of a stock that we want to liquidate before time $T$. Divide $T$ into $N$ intervals of length $\tau=\frac{T}{N}$ and define:

- $t_k = k\tau$ to be discrete times, where $k = 0,..,N$.

- A **trading trajectory** to be the list $(x_0,..,x_N)$, where $x_k$ is the number of shares we plan to hold at time $t_k$. We require that our initial position $x_0 = X$, and that at liquidation time $T$, $x_N = 0$.

- A **trading list** to be $(n_1,..,n_N)$, $n_k =  x_{k-1} - x_k$ as the number of shares that we will sell between times $t_{k-1}$ and $t_k$.

- A **trading strategy** as a rule for determining $n_k$ from the information available at time $t_{k-1}$.

Below, we can see a visual example of a trading trajectory, for $N = 12$.

## Price Dynamics

We will assume that the stock price evolves according to a discrete arithmetic random walk:

\begin{equation}
S_k = S_{k-1} + \sigma \tau^{1/2} \xi_k
\end{equation}

for $k = 1,..,N$ and where:

\begin{equation}
S_k = \text{ stock price at time $k$}\hspace{21.6cm}\\
\sigma = \text{ standard deviation of the fluctuations in stock price}\hspace{16.3cm}\\
\tau = \text{ length of discrete time interval}\hspace{20.2cm}\\
\xi_k = \text{ draws from independent random variables}\hspace{17.8cm}
\end{equation}

We will denote the initial stock price as $S_0$. The role of $\xi_k$ is to simulate random price fluctuations using random numbers drawn from a Normal Gaussian distribution with zero mean and unit variance. The code below shows us what this price model looks like, for an initial stock price of $S_0 =$ \$50 dollars, a standard deviation of price fluctuations of $\sigma = 0.379$, and a discrete time interval of $\tau = 1$.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Set the number of days to follow the stock price
n_days = 100

# Plot the stock price as a function of time
utils.plot_price_model(seed = 0, num_days = n_days)
```

## Market Impact

As we learned previously the price of a stock is affected by market impact that occurs every time we sell a stock. In their model, Almgren and Chriss distinguish between two types of market impact, permanent and temporary market impact. We will now add these two factors into our price model.

### Permanent Impact

Permanent market impact refers to changes in the equilibrium price of a stock as a direct function of our trading. Permanent market impact is called *permanent* because its effect persists for the entire liquidation period, $T$. We will denote the permanent price impact as $g(v)$, and will add it to our price model:

\begin{equation}
S_k = S_{k-1} + \sigma \tau^{1/2} \xi_k - \tau g\left(\frac{n_k}{\tau}\right)
\end{equation}

Here, we assumed the permanent impact function, $g(v)$, is a linear function of the trading rate, $v = n_k / \tau$. We will take $g(v)$ to have the form:

\begin{equation}
g(v) = \gamma \left(\frac{n_k}{\tau}\right)
\end{equation}

where $\gamma$ is a constant and has units of (\$/share${}^2$). Replacing this in the above equation we get:

\begin{equation}
S_k = S_{k-1} + \sigma \tau^{1/2} \xi_k - \gamma n_k
\end{equation}

With this form, we can see that for each $n$ shares that we sell, we will depress the stock price permanently by $n\gamma$, regardless of the time we take to sell the stocks.

### Temporary Impact

Temporary market impact refers to temporary imbalances in supply and demand caused by our trading. This leads to temporary price movements away from equilibrium. Temporary market impact is called *temporary* because its effect
dissipates by the next trading period. We will denote the temporary price impact as $h(v)$. Given this, the actual stock price at time $k$ is given by:

\begin{equation}
\tilde{S_k} = S_{k-1} - h\left(\frac{n_k}{\tau}\right)
\end{equation}

Where, we have again assumed the temporary impact function, $h(v)$, is a linear function of the trading rate, $v = n_k / \tau$. We will take $h(v)$ to have the form:

\begin{equation}
h(v) = \epsilon \mbox{ sign}(n_k) + \eta \left(\frac{n_k}{\tau}\right)
\end{equation}

where $\epsilon$ and $\eta$ are constants with units (\$/share) and (\$ time/share${}^2$), respectively. It is important to note that $h(v)$ does not affect the price $S_k$.

## Capture

We define the **Capture** to be the total profits resulting from trading along a particular trading trajectory, upon completion of all trades. We can compute the capture via:

\begin{equation}
\sum\limits_{k=1}^{N} n_k \tilde{S_k} = X S_0 + \sum\limits_{k=1}^{N} \left(\sigma \tau^{1/2} \xi_k - \tau g\left(\frac{n_k}{\tau}\right)\right) x_k - \sum\limits_{k=1}^{N} n_k h\left(\frac{n_k}{\tau}\right)
\end{equation}

As we can see this is the sum of the product of the number of shares $n_k$ that we sell in each time interval, times the effective price per share $\tilde{S_k}$ received on that sale.

## Implementation Shortfall

We define the **Implementation Shortfall** as the total cost of trading and is given by:

\begin{equation}
I_s = X S_0 - \sum_{k = 1}^N n_k \tilde{S_k}
\end{equation}

This is what we seek to minimize when determining the best trading strategy!

Note that since $\xi_k$ is random, so is the implementation shortfall. Therefore, we have to frame the minimization problem in terms of the expectation value of the shortfall and its corresponding variance. We'll refer to $E(x)$ as the expected shortfall and $V(x)$ as the variance of the shortfall. Simplifying the above equation for $I_s$, is easy to see that:

\begin{equation}
E(x) = \sum\limits_{k=1}^{N} \tau x_k g\left(\frac{n_k}{\tau}\right) + \sum\limits_{k=1}^{N} n_k h\left(\frac{n_k}{\tau}\right)
\end{equation}

and

\begin{equation}
V(x) = \sigma^2 \sum\limits_{k=1}^{N} \tau {x_k}^2
\end{equation}

The units of $E(x)$ are dollars and the units of $V(x)$ are dollars squared. So now, we can reframe our minimization problem in terms of $E(x)$ and $V(x)$.
For a given level of variance of shortfall, $V(x)$, we seek to minimize the expectation of shortfall, $E(x)$. In the next section we will see how to solve this problem.

## Utility Function

Our goal now is to find the strategy that has the minimum expected shortfall $E(x)$ for a given maximum level of variance $V(x) \ge 0$. This constrained optimization problem can be solved by introducing a Lagrange multiplier $\lambda$. Therefore, our problem reduces to finding the trading strategy that minimizes the **Utility Function** $U(x)$:

\begin{equation}
U(x) = E(x) + \lambda V(x)
\end{equation}

The parameter $\lambda$ is referred to as **trader’s risk aversion** and controls how much we penalize the variance relative to the expected shortfall.

The intuition of this utility function can be thought of as follows. Consider a stock which exhibits high price volatility and thus a high risk of price movement away from the equilibrium price. A risk averse trader would prefer to trade a large portion of the volume immediately, causing a known price impact, rather than risk trading in small increments at successively adverse prices. Alternatively, if the price is expected to be stable over the liquidation period, the trader would rather split the trade into smaller sizes to avoid price impact. This trade-off between speed of execution and risk of price movement is ultimately what governs the structure of the resulting trade list.

# Optimal Trading Strategy

Almgren and Chriss solved the above problem and showed that for each value
of risk aversion there is a uniquely determined optimal execution strategy. The details of their derivation is discussed in their [paper](https://cims.nyu.edu/~almgren/papers/optliq.pdf). Here, we will just state the general solution.

The optimal trajectory is given by:

\begin{equation}
x_j = \frac{\sinh \left( \kappa \left( T-t_j\right)\right)}{ \sinh (\kappa T)}X, \hspace{1cm}\text{ for } j=0,...,N
\end{equation}

and the associated trading list:

\begin{equation}
n_j = \frac{2 \sinh \left(\frac{1}{2} \kappa \tau \right)}{ \sinh \left(\kappa T\right) } \cosh \left(\kappa \left(T - t_{j-\frac{1}{2}}\right)\right) X, \hspace{1cm}\text{ for } j=1,...,N
\end{equation}

where $t_{j-1/2} = (j-\frac{1}{2}) \tau$. The expected shortfall and variance of the optimal trading strategy are given by:

In the above equations $\kappa$ is given by:

\begin{align*}
&\kappa = \frac{1}{\tau}\cosh^{-1}\left(\frac{\tau^2}{2}\tilde{\kappa}^2 + 1\right)
\end{align*}

where:

\begin{align*}
&\tilde{\kappa}^2 = \frac{\lambda \sigma^2}{\tilde{\eta}} = \frac{\lambda \sigma^2}{\eta \left(1-\frac{\gamma \tau}{2 \eta}\right)}
\end{align*}

```python

```


```python
%pylab inline
import pandas as pd
import seaborn as sns
import json
from scipy.linalg import norm

sns.set_context('paper', font_scale=2)
sns.set_style('whitegrid')
np.random.seed(42)
```

    Populating the interactive namespace from numpy and matplotlib

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['e', 'sample', 'norm']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

```python
df = pd.read_csv('data/wdbc.data', header=None)
df = df.drop(columns=[0])
output = np.array(df.loc[:, 1])
df  = df.drop(columns=[1])

```

```python
df.shape
```

    (569, 30)

```python
df_matrix = df.as_matrix()
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:1: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
      """Entry point for launching an IPython kernel.

```python
np.sum(df_matrix)
```

    1056474.4596355997

# Normalize

```python
df_normalized = df_matrix - df_matrix.mean()
df_normalized = df_normalized/np.sqrt((df_normalized * df_normalized).sum(axis=1))[:, np.newaxis]
```

```python
df_normalized
```

    array([[-0.01998381, -0.02344792,  0.02772619, ..., -0.02805213,
            -0.0279635 , -0.02811882],
           [-0.01797171, -0.01918952,  0.03088423, ..., -0.02683737,
            -0.02679866, -0.02687955],
           [-0.0208045 , -0.02003544,  0.03357715, ..., -0.03039166,
            -0.03033334, -0.03046828],
           ...,
           [-0.03326271, -0.02483149,  0.03408422, ..., -0.04535005,
            -0.04529129, -0.04539676],
           [-0.01915379, -0.01510415,  0.03627945, ..., -0.02858672,
            -0.02852006, -0.02865213],
           [-0.141273  , -0.09747973, -0.03646145, ..., -0.16152542,
            -0.16077614, -0.16134172]])

# Problem (a), (b)

In all of our experiments we will perform 100 random partitions of the data for
train/test and then report the final result as the average over these tria

\begin{align*}
(\hat{w}, \hat{b}) &= \text{arg} \min_{w,b} f(\vec{w}, b) := \frac{1}{N} \sum_{i=1}^N -y_i(\vec{w}^T\vec{x}_i + b) + \log(1+ \exp{(\vec{w}^T\vec{x}_i + b)} + \frac{\lambda^2}{2N} ||\vec{w}||_{l_2}^2
\end{align*}

Consider the derivatives:

\begin{align}
\frac{\partial f(\vec{w}, b)}{\partial b} &= \frac{1}{N} \sum_{i=1}^N -y_i + \frac{\exp{(\vec{w}^T\vec{x}_i + b)}}{1+ \exp{(\vec{w}^T\vec{x}_i + b)}}\\
&= \frac{1}{N} \sum_{i=1}^N \frac{1}{1+ \exp{(-(\vec{w}^T\vec{x}_i + b))}}-y_i
\end{align}

For $j \geq 1$ and learning rate $\mu$:

\begin{align}
\frac{\partial f(\vec{w}, b)}{\partial w_j} &= \frac{1}{N} \sum_{i=1}^N -y_i(x_j) + \frac{\exp{(\vec{w}^T\vec{x}_i + b)}}{1+ \exp{(\vec{w}^T\vec{x}_i + b)}}x_j + \frac{\lambda}{N} w_j\\
&=  \sum_{i=1}^N \big(\frac{1}{1+ \exp{(-(\vec{w}^T\vec{x}_i + b))}}-y_i\big)x_j + \frac{\lambda}{N}w_j
\end{align}

Gradient descent:

\begin{align*}
b^{(t+1)} &= b^{(t)} - \frac{\mu}{N} \sum_{i=1}^N \big(\frac{1}{1+ \exp{(-(\vec{w}^T\vec{x}_i + b))}}-y_i\big)\\
w^{(t+1)}_j &= w^{(t)}_j - \frac{\mu}{N} \sum_{i=1}^N \big(\frac{1}{1+ \exp{(-(\vec{w}^T\vec{x}_i + b))}}-y_i\big)x_{ij} + \frac{\lambda}{N}w_j^{(t)}\\
\end{align*}

where $x_{ij}$ refers to the $j^{th}$ feature of the $i^{th}$ sample. In our case.
$1 \leq j \leq 30$ and $i$ ranges over the number of samples.

This looks super confusing, let's resort to matrix notations., We construct an augmented matrix $X$ by appending a column of 1s in the beginning that will capture $b$.

$X = \begin{pmatrix}1 & x_{11} & x_{12} & \cdots & x_{1M}\\
1 & x_{21} & x_{22} & \cdots & x_{2M}\\
\vdots\\
1 & x_{N1} & x_{N2} & \cdots & x_{NM}\\
\end{pmatrix}_{N\times(M+1)}$

Our iteration vector is $\vec{w}_{1\times(M+1)} = \begin{pmatrix} b \\ w_1 \\ w_2 \\ \vdots \\ w_M \end{pmatrix}_{(M+1)\times 1}$, thus $X\vec{w}$ has dimensions $N \times 1$ which is same as our target vector $\vec{y}$

Let's also define an augmented  vector $\vec{v} = \begin{pmatrix} 0 \\ w_1 \\ w_2 \\ \vdots \\ w_M \end{pmatrix}_{(M+1)\times 1}$

And the update simplifies to:

\begin{align*}
\vec{w}_{(M+1)\times 1}^{(t+1)} &= \vec{w}^{(t+1)}_{(M+1)\times 1} - \frac{\mu}{N} X^T(X\vec{w}^{(t+1)}-\vec{y}) + \frac{\lambda}{M} \vec{v}^{t}
\end{align*}

```python
def split_train_test(X, y, n_train=500):
    nrows = X.shape[0]
    indices = numpy.random.permutation(nrows)
    training_idx, test_idx = indices[:n_train], indices[n_train:]
    X_train, X_test = X[training_idx,:], X[test_idx,:]
    y_train, y_test = y[training_idx], y[test_idx]
    return (X_train, y_train, X_test, y_test)

def sigmoid(z):
    return 1/(1+np.exp(-z))

def calc_error(z, y_true):
    y_predicted = sigmoid(z)
    y_predicted[y_predicted>=0.5] = 1
    y_predicted[y_predicted<0.5] = 0
    return np.mean(np.abs(y_predicted-y_true))

def calc_misclassified(z, y_true):
    y_predicted = sigmoid(z)
    y_predicted[y_predicted>=0.5] = 1
    y_predicted[y_predicted<0.5] = 0
    return np.sum(np.abs(y_predicted-y_true))

def logistic_regression(X, y,
                        X_test=None,
                        y_test=None,
                        num_steps=500,
                        learning_rate=1e-6,
                        regularizer_lambda=0.01):
    intercept = np.ones((X.shape[0], 1))
    X = np.hstack((intercept, X))
    if X_test is not None:
        intercept = np.ones((X_test.shape[0], 1))
        X_test= np.hstack((intercept, X_test))
    weights = np.zeros(X.shape[1])
    regularization_indicator = np.ones(X.shape[1])
    regularization_indicator[0] =  0
    regularization_weights = np.multiply(regularization_indicator, weights)
    training_errors = []
    training_misclassified = []
    testing_errors = []
    testing_misclassified = []
    error_ratios = []
    for step in range(num_steps):
        scores = np.dot(X, weights)
        predictions = sigmoid(scores)
        error = predictions - y
        gradient = np.dot(X.T, error)
        regularization_weights = np.multiply(regularization_indicator, weights)
        regularization = regularizer_lambda * regularization_weights

        weights = weights - (learning_rate * gradient) + regularization
        ll = log_likelihood(X, y, weights, regularizer_lambda)
        error_ratio = norm(gradient, 2)**2/(1+ll)
        error_ratios.append(error_ratio)

        if y_test is not None:
            scores_test = np.dot(X_test, weights)
            testing_error = calc_error(scores_test, y_test)
            testing_errors.append(testing_error)
            testing_misclassified.append(calc_misclassified(scores_test, y_test))

        training_error = calc_error(scores, y)
        training_errors.append(training_error)
        training_misclassified.append(calc_misclassified(scores, y))

    return weights, training_errors, testing_errors, training_misclassified, testing_misclassified, error_ratios

def log_likelihood(X, y, weights, regularizer_lambda):
    scores = np.dot(X, weights)
    ll = np.sum( -y*scores + np.log(1 + np.exp(scores)) ) + regularizer_lambda * norm(weights, 2)**2
    return ll

def logistic_regression_convergence(X, y,
                                    X_test=None,
                                    y_test=None,
                                    max_error=1e-5,
                                    learning_rate=1e-6,
                                    regularizer_lambda=0.01,
                                    momentum=None,
                                    eta=0.1):
    intercept = np.ones((X.shape[0], 1))
    X = np.hstack((intercept, X))
    if X_test is not None:
        intercept = np.ones((X_test.shape[0], 1))
        X_test= np.hstack((intercept, X_test))
    weights = np.zeros(X.shape[1])
    regularization_indicator = np.ones(X.shape[1])
    regularization_indicator[0] =  0
    regularization_weights = np.multiply(regularization_indicator, weights)
    training_errors = []
    training_misclassified = []
    testing_errors = []
    testing_misclassified = []
    error_ratio = 1e5
    steps = 0
    weights_t = [weights]
    error_ratios = []
    while error_ratio > max_error:
        steps += 1
        scores = np.dot(X, weights)
        predictions = sigmoid(scores)
        error = predictions - y
        gradient = np.dot(X.T, error)
        regularization_weights = np.multiply(regularization_indicator, weights)
        regularization = regularizer_lambda * regularization_weights

        if momentum is None:
            weights = weights - (learning_rate * gradient) + regularization
        elif momentum is 'heavy_ball':
            old_weights = weights.copy()
            if steps>=2:
                weights = weights - (learning_rate * gradient) + regularization +\
                eta * (weights_t[steps-1]-weights_t[steps-2])
            else:
                weights = weights - (learning_rate * gradient) + regularization
        elif momentum is 'nesterov':
            if steps>=2:
                scores = np.dot(X, weights+eta*(weights_t[steps-1]-weights_t[steps-2]))
                predictions = sigmoid(scores)
                error = predictions - y
                gradient = np.dot(X.T, error)
                regularization_weights = np.multiply(regularization_indicator, weights)
                regularization = regularizer_lambda * regularization_weights
                weights = weights - (learning_rate * gradient) + regularization +\
                eta * (weights_t[steps-1]-weights_t[steps-2])
            else:
                weights = weights - (learning_rate * gradient) + regularization

        weights_t.append(weights)
        ll = log_likelihood(X, y, weights, regularizer_lambda)
        error_ratio = norm(gradient, 2)**2/(1+ll)
        error_ratios.append(error_ratio)
        if y_test is not None:
            scores_test = np.dot(X_test, weights)
            testing_error = calc_error(scores_test, y_test)
            testing_errors.append(testing_error)
            testing_misclassified.append(calc_misclassified(scores_test, y_test))

        training_error = calc_error(scores, y)
        training_errors.append(training_error)
        training_misclassified.append(calc_misclassified(scores, y))

    return weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios
```

```python
def logistic_regression_convergence_steps(X, y,
                                          X_test=None,
                                          y_test=None,
                                          max_steps=500,
                                          learning_rate=1e-6,
                                          regularizer_lambda=0.01,
                                          momentum=None,
                                          eta=0.1):
    intercept = np.ones((X.shape[0], 1))
    X = np.hstack((intercept, X))
    if X_test is not None:
        intercept = np.ones((X_test.shape[0], 1))
        X_test= np.hstack((intercept, X_test))
    weights = np.zeros(X.shape[1])
    regularization_indicator = np.ones(X.shape[1])
    regularization_indicator[0] =  0
    regularization_weights = np.multiply(regularization_indicator, weights)
    training_errors = []
    training_misclassified = []
    testing_errors = []
    testing_misclassified = []
    error_ratio = 1e5
    steps = 0
    weights_t = [weights]
    error_ratios = []

    while steps < max_steps:
        steps += 1
        scores = np.dot(X, weights)
        predictions = sigmoid(scores)
        error = predictions - y
        gradient = np.dot(X.T, error)
        regularization_weights = np.multiply(regularization_indicator, weights)
        regularization = regularizer_lambda * regularization_weights

        if momentum is None:
            weights = weights - (learning_rate * gradient) + regularization
        elif momentum is 'heavy_ball':
            old_weights = weights.copy()
            if steps>=2:
                weights = weights - (learning_rate * gradient) + regularization +\
                eta * (weights_t[steps-1]-weights_t[steps-2])
            else:
                weights = weights - (learning_rate * gradient) + regularization
        elif momentum is 'nesterov':
            if steps>=2:
                scores = np.dot(X, weights+eta*(weights_t[steps-1]-weights_t[steps-2]))
                predictions = sigmoid(scores)
                error = predictions - y
                gradient = np.dot(X.T, error)
                regularization_weights = np.multiply(regularization_indicator, weights)
                regularization = regularizer_lambda * regularization_weights
                weights = weights - (learning_rate * gradient) + regularization +\
                eta * (weights_t[steps-1]-weights_t[steps-2])
            else:
                weights = weights - (learning_rate * gradient) + regularization

        weights_t.append(weights)
        ll = log_likelihood(X, y, weights, regularizer_lambda)
        error_ratio = norm(gradient, 2)**2/(1+ll)
        error_ratios.append(error_ratio)
        if y_test is not None:
            scores_test = np.dot(X_test, weights)
            testing_error = calc_error(scores_test, y_test)
            testing_errors.append(testing_error)
            testing_misclassified.append(calc_misclassified(scores_test, y_test))

        training_error = calc_error(scores, y)
        training_errors.append(training_error)
        training_misclassified.append(calc_misclassified(scores, y))

    return weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios
```

```python

X_train = []
y_train = []

X_test = []
y_test = []

N_iter = 100
lamda_w = 0.01
T = 500

for i in range(N_iter):
    train_x, train_label, test_x, test_label = split_train_test(df_normalized, output)
    X_train.append(train_x)
    y_train.append(train_label)
    X_test.append(test_x)
    y_test.append(test_label)

```

# Problem (c)
Now that we have normalized data, partition it into train/test sets at random 100 times
as discussed earlier. In each trial learn the model by solving (1) with λ = 0.01. To
due this run gradient descent for T = 500 iterations and then use the trained model to
make predictions on the test data and calculate the average error (average number of
miss-classified patients on the test data) for each trial. Report the average over the 100
trials. The value of the step size you use does not matter too much. However, make sure
that the algorithm has converged

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []

for i in range(100):
    weights, training_errors, testing_errors, training_misclassified, testing_misclassified, error_ratios = logistic_regression(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                num_steps=500,
                                                                                learning_rate=0.01,
                                                                                regularizer_lambda=0.01)
    all_weights.append(weights)
    all_training_errors.append(training_errors)
    all_testing_errors.append(testing_errors)
    all_training_misclassified.append(training_misclassified)
    all_testing_misclassified.append(testing_misclassified)
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.

```python
vanilla_logistic = {'weights': all_weights,
                    'training_errors': all_training_errors,
                    'testing_errors': all_testing_errors,
                    'training_misclassified': all_training_misclassified,
                    'testing_misclassified': all_testing_misclassified
                   }
```

```python
vanilla_df = []
for index, error in enumerate(vanilla_logistic['training_errors']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Training'])
for index, error in enumerate(vanilla_logistic['testing_errors']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Testing'])
vanilla_df = pd.DataFrame(vanilla_df, columns=['sample', 'Error', 'Step', 'Data Type'])
fig, ax = plt.subplots(figsize=(12,8))
sns.lineplot(x="Step",
             y="Error",
             hue="Data Type",
             style="Data Type",
             markers=False,
             dashes=True,
             err_style="bars",
             ci=None,
             data=vanilla_df,
             ax=ax)
ax.set_title('Error [0-1] for vanilla logistic reg.')
fig.tight_layout()
fig.savefig('vanilla_error.pdf')
```

```python
vanilla_df = []
for index, error in enumerate(vanilla_logistic['training_misclassified']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Training (total: 500)'])
for index, error in enumerate(vanilla_logistic['testing_misclassified']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Testing (total: 69)'])
vanilla_df = pd.DataFrame(vanilla_df, columns=['sample', "# Misclassified", 'Step', 'Data Type'])

fig, ax = plt.subplots(figsize=(12,8))
sns.lineplot(x="Step",
             y="# Misclassified",
             hue="Data Type",
             style="Data Type",
             markers=False,
             dashes=True,
             err_style="bars",
             ci=None,
             data=vanilla_df,
             ax=ax)
ax.set_title('# of Misclassified samples for vanilla logistic reg. (Problem 3c)')
fig.tight_layout()

fig.savefig('vanilla_testing_misclassified.pdf')
```

# Problem (d)

Perform the experiment with the same step size you used before but now report the
number of iterations it takes to get to an accuracy of 10−6
calculated via the first iteration t when the following inequality holds
$||f(w_t, b_t)||^2_{l_2} \leq 10^{−6}(1 + |f(w_t, b_t)|)$

The number you should report is the average of this number over the 100 trials

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_steps = []
all_error_ratios = []
for i in range(100):
    weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios = logistic_regression_convergence(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                max_error=1e-5,
                                                                                learning_rate=0.1,
                                                                                regularizer_lambda=0.01)

    all_weights.append(weights)
    all_training_errors.append(training_errors)
    all_testing_errors.append(testing_errors)
    all_training_misclassified.append(training_misclassified)
    all_testing_misclassified.append(testing_misclassified)
    all_steps.append(steps)
    all_error_ratios.append(error_ratios)
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:70: RuntimeWarning: overflow encountered in exp

```python
steps_logistic = {'weights': all_weights,
                  'training_errors': all_training_errors,
                  'testing_errors': all_testing_errors,
                  'training_misclassified': all_training_misclassified,
                  'testing_misclassified': all_testing_misclassified,
                  'steps': all_steps
                 }
```

```python
fig = plt.figure()
ax = sns.violinplot(steps_logistic['steps'])
ax.set_title('# of steps for convergence (Problem 3d)')
fig.tight_layout()
fig.savefig('number_of_steps_3d.pdf')

```

# Problem (e)

Perform the experiment of part (d) but now add a momentum term (1) using the heavy
ball method and (2) using Nesterov’s accelerated scheme. In both cases keep the same
step size as part (d) but fine tune the momentum parameter to get the smallest number
of iterations for convergence based on the stopping criteria (2) (again averaged over the
100 trials). Draw the convergence of the three algorithms gradient descent, heavy ball,
Nesterov’s accelerated scheme for one trial. That is, draw the ratio
k∇f(wt, bt)k2`2(1 + |f(wt, bt)|) as a function of the iteration number t = 1, 2, . . . , 500. Which algorithm would you use
and why

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_error_ratios = []
min_steps = 1e5
for eta in np.linspace(0.0, 1, 20):
    all_steps = []
    for i in range(100):

        weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios = logistic_regression_convergence(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                max_error=1e-5,
                                                                                learning_rate=0.1,
                                                                                regularizer_lambda=0.01,
                                                                                momentum='heavy_ball',
                                                                                eta=eta)
        all_steps.append(steps)
    if np.mean(all_steps) < min_steps:
        min_steps = np.mean(all_steps)
        opt_eta_heavy_ball = eta
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:70: RuntimeWarning: overflow encountered in exp

```python
opt_eta_heavy_ball
```

    1.0

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_steps = []
all_error_ratios = []
for i in range(100):
    weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios = logistic_regression_convergence(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                max_error=1e-5,
                                                                                learning_rate=0.1,
                                                                                regularizer_lambda=0.01,
                                                                                momentum='heavy_ball',
                                                                                eta=opt_eta_heavy_ball)
    all_weights.append(weights)
    all_training_errors.append(training_errors)
    all_testing_errors.append(testing_errors)
    all_training_misclassified.append(training_misclassified)
    all_testing_misclassified.append(testing_misclassified)
    all_steps.append(steps)
    all_error_ratios.append(error_ratios)
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:70: RuntimeWarning: overflow encountered in exp

```python
heavy_ball_logistic = {'weights': all_weights,
                       'training_errors': all_training_errors,
                       'testing_errors': all_testing_errors,
                       'training_misclassified': all_training_misclassified,
                       'testing_misclassified': all_testing_misclassified,
                       'steps': all_steps,
                       'error_ratios': all_error_ratios,
                       'opt_eta': opt_eta_heavy_ball
                      }
```

```python
vanilla_df = []
for index, error in enumerate(heavy_ball_logistic['training_errors']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Training'])
for index, error in enumerate(heavy_ball_logistic['testing_errors']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Testing'])
vanilla_df = pd.DataFrame(vanilla_df, columns=['sample', 'Error', 'Step', 'Data Type'])
fig, ax = plt.subplots(figsize=(12,8))
sns.lineplot(x="Step",
             y="Error",
             hue="Data Type",
             style="Data Type",
             markers=False,
             dashes=True,
             err_style="bars",
             ci=None,
             data=vanilla_df,
             ax=ax)
ax.set_title('Error [0-1] for heavy ball logistic reg. (eta=1) | Problem 3(e)')
fig.tight_layout()
fig.savefig('heavy_ball_opt_error.pdf')
```

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_error_ratios = []
min_steps = 1e5
for eta in np.linspace(0.0, 1, 20):
    all_steps = []
    for i in range(100):

        weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios = logistic_regression_convergence(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                max_error=1e-5,
                                                                                learning_rate=0.1,
                                                                                regularizer_lambda=0.01,
                                                                                momentum='nesterov',
                                                                                eta=eta)
        all_steps.append(steps)
    if np.mean(all_steps) < min_steps:
        min_steps = np.mean(all_steps)
        opt_eta_nesterov = eta
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:70: RuntimeWarning: overflow encountered in exp

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_steps = []
all_error_ratios = []
for i in range(100):
    weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios = logistic_regression_convergence(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                max_error=1e-5,
                                                                                learning_rate=0.1,
                                                                                regularizer_lambda=0.01,
                                                                                momentum='nesterov',
                                                                                eta=opt_eta_nesterov)
    all_weights.append(weights)
    all_training_errors.append(training_errors)
    all_testing_errors.append(testing_errors)
    all_training_misclassified.append(training_misclassified)
    all_testing_misclassified.append(testing_misclassified)
    all_steps.append(steps)
    all_error_ratios.append(error_ratios)
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:70: RuntimeWarning: overflow encountered in exp

```python
nesterov_logistic = {'weights': all_weights,
                     'training_errors': all_training_errors,
                     'testing_errors': all_testing_errors,
                     'training_misclassified': all_training_misclassified,
                     'testing_misclassified': all_testing_misclassified,
                     'steps': all_steps,
                     'error_ratios': all_error_ratios,
                     'opt_eta': opt_eta_nesterov
                    }
```

```python
opt_eta_nesterov
```

    1.0

```python
vanilla_df = []
for index, error in enumerate(nesterov_logistic['training_errors']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Training'])
for index, error in enumerate(nesterov_logistic['testing_errors']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Testing'])
vanilla_df = pd.DataFrame(vanilla_df, columns=['sample', 'Error', 'Step', 'Data Type'])
fig, ax = plt.subplots(figsize=(12,8))
sns.lineplot(x="Step",
             y="Error",
             hue="Data Type",
             style="Data Type",
             markers=False,
             dashes=True,
             err_style="bars",
             ci=None,
             data=vanilla_df,
             ax=ax)
ax.set_title('Error [0-1] for heavy ball logistic reg. (eta=1) | Problem 3(e)')
fig.tight_layout()
fig.savefig('nesterov_opt_error.pdf')
```

# For 500 steps

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_steps = []
all_error_ratios = []
for i in range(100):
    weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios = logistic_regression_convergence_steps(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                max_steps=500,
                                                                                learning_rate=0.1,
                                                                                regularizer_lambda=0.01,
                                                                                momentum='nesterov',
                                                                                eta=opt_eta_nesterov)
    all_weights.append(weights)
    all_training_errors.append(training_errors)
    all_testing_errors.append(testing_errors)
    all_training_misclassified.append(training_misclassified)
    all_testing_misclassified.append(testing_misclassified)
    all_steps.append(steps)
    all_error_ratios.append(error_ratios)
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:70: RuntimeWarning: overflow encountered in exp

```python
nesterov_logistic_steps = {'weights': all_weights,
                           'training_errors': all_training_errors,
                           'testing_errors': all_testing_errors,
                           'training_misclassified': all_training_misclassified,
                           'testing_misclassified': all_testing_misclassified,
                           'steps': all_steps,
                           'error_ratios': all_error_ratios,
                           'opt_eta': opt_eta_nesterov
                    }
```

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_steps = []
all_error_ratios = []
for i in range(100):
    weights, training_errors, testing_errors, training_misclassified, testing_misclassified, steps, error_ratios = logistic_regression_convergence_steps(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                max_steps=500,
                                                                                learning_rate=0.1,
                                                                                regularizer_lambda=0.01,
                                                                                momentum='heavy_ball',
                                                                                eta=opt_eta_heavy_ball)
    all_weights.append(weights)
    all_training_errors.append(training_errors)
    all_testing_errors.append(testing_errors)
    all_training_misclassified.append(training_misclassified)
    all_testing_misclassified.append(testing_misclassified)
    all_steps.append(steps)
    all_error_ratios.append(error_ratios)
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:70: RuntimeWarning: overflow encountered in exp

```python
heavy_ball_logistic_steps = {'weights': all_weights,
                           'training_errors': all_training_errors,
                           'testing_errors': all_testing_errors,
                           'training_misclassified': all_training_misclassified,
                           'testing_misclassified': all_testing_misclassified,
                           'steps': all_steps,
                           'error_ratios': all_error_ratios,
                           'opt_eta': opt_eta_nesterov
                    }
```

```python
all_weights = []
all_training_errors = []
all_testing_errors = []
all_training_misclassified = []
all_testing_misclassified = []
all_error_ratios = []
for i in range(100):
    weights, training_errors, testing_errors, training_misclassified, testing_misclassified, error_ratios = logistic_regression(X_train[i],
                                                                                y_train[i],
                                                                                X_test[i],
                                                                                y_test[i],
                                                                                num_steps=500,
                                                                                learning_rate=0.01,
                                                                                regularizer_lambda=0.01)
    all_weights.append(weights)
    all_training_errors.append(training_errors)
    all_testing_errors.append(testing_errors)
    all_training_misclassified.append(training_misclassified)
    all_testing_misclassified.append(testing_misclassified)
    all_error_ratios.append(error_ratios)
```

    /home/cmb-panasas2/skchoudh/software_frozen/anaconda27/envs/riboraptor/lib/python3.5/site-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.

```python
vanilla_logistic_steps = {'weights': all_weights,
                           'training_errors': all_training_errors,
                           'testing_errors': all_testing_errors,
                           'training_misclassified': all_training_misclassified,
                           'testing_misclassified': all_testing_misclassified,
                           'error_ratios': all_error_ratios,
                    }
```

```python
vanilla_df = []
for index, error in enumerate(vanilla_logistic_steps['error_ratios']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Gradient Descent'])
for index, error in enumerate(heavy_ball_logistic_steps['error_ratios']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Heavy Ball'])
for index, error in enumerate(nesterov_logistic_steps['error_ratios']):
    sample = 's{}'.format(index)
    for t, e in enumerate(error):
        vanilla_df.append([sample, e, t+1, 'Nesterov'])

vanilla_df = pd.DataFrame(vanilla_df, columns=['sample', 'Error', 'Step', 'Method'])
fig, ax = plt.subplots(figsize=(12,8))
sns.lineplot(x="Step",
             y="Error",
             hue="Method",
             style="Method",
             markers=False,
             dashes=True,
             err_style="bars",
             ci=None,
             data=vanilla_df,
             ax=ax)
ax.set_title('Error ratios for GD vs Heavyball vs Nesterov')
fig.tight_layout()
fig.savefig('error_ratios_gd_vs_h_vs_n.pdf')
```

```python
fig, ax = plt.subplots(figsize=(12,8))
sns.lineplot(x="Step",
             y="Error",
             hue="Method",
             style="Method",
             markers=False,
             dashes=True,
             err_style="bars",
             ci=None,
             data=vanilla_df,
             ax=ax)
ax.set_xlim(0,100)
ax.set_title('Error ratios for GD vs Heavyball vs Nesterov')
fig.tight_layout()
fig.savefig('error_ratios_gd_vs_h_vs_n_100.pdf')
```

```python

```


\newcommand{\vt}[1]{\boldsymbol{#1}}
\newcommand{\uv}[1]{\hat{\boldsymbol{#1}}}
\newcommand{\tv}[1]{\tilde{\boldsymbol{#1}}}
\newcommand{\dvg}{\operatorname{div}}
\newcommand{\grd}{\operatorname{grad}}
$$
\renewcommand{\vt}[1]{\boldsymbol{#1}}
\renewcommand{\uv}[1]{\hat{\boldsymbol{#1}}}
\renewcommand{\tv}[1]{\tilde{\boldsymbol{#1}}}
\renewcommand{\dvg}{\operatorname{div}}
\renewcommand{\grd}{\operatorname{grad}}
$$

# Integration of $R^n$ on triangles

We are interested in the computation of

\begin{equation}
\int_T R^n dy,
\end{equation}

where $T$ is a triangle, $R=|x-y|$, and $n = -3,-1,0,1,2,3,..$. This package is based on [1] and papers building on this paper, notably [2,3].

[[1]](http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1143304&tag=1) D. Wilton, S. Rao, A. Glisson, D. Schaubert, O. Al-Bundak, and C. Butler, “Potential integrals for uniform and linear source distributions on polygonal and polyhedral domains,” IEEE Transactions on Antennas and Propagation, vol. 32, no. 3, pp. 276–281, Mar. 1984.

[2]R. D. Graglia, “On the numerical integration of the linear shape functions times the 3-D Green’s function or its gradient on a plane triangle,” IEEE Transactions on Antennas and Propagation, vol. 41, no. 10, pp. 1448–1455, Oct. 1993.

[3] P. Yla-Oijala and M. Taskinen, “Calculation of CFIE impedance matrix elements with RWG and n times;RWG functions,” IEEE Transactions on Antennas and Propagation, vol. 51, no. 8, pp. 1837–1846, Aug. 2003.

Following Wilton[1], we start with writing the integrand as the surface divergence of a radially symmetric vector function. Assume

\begin{equation}
R^n = \dvg_{y} \left( F(P;h) \vt{P} \right)
\end{equation}

Here, $\vt{P}$ is the projection onto the plane supporting $T$ of $\vt{y} - \vt{x}$, and $h$ is the signed normal coordinate of $\vt{x}-\vt{y}$, i.e. $\vt{x}-\vt{y} = h\uv{n} - \vt{P}$ (with $\vt{n}$ a normal on $T$ that will induce a counter-clockwise ordering of $\partial T$).

The Ansatz above is equivalent to:

\begin{equation}
\left(P^2+h^2\right)^{n/2} = F'(P;h) P + F(P;h) 2
\end{equation}

Multiplying with the integrating factor $P$ leads to

\begin{equation}
P \left(P^2+h^2\right)^{n/2} = \left( P^2 F(P;h) \right)'
\end{equation}

or,

\begin{align}
F(P;h) =& \frac{1}{P^2} \int Q \left(Q^2+h^2\right)^{n/2} dQ \\
=& \frac{1}{P^2} \frac{1}{(n+2)}  \left( P^2 + h^2 \right)^{\frac{n+2}{2}}
\end{align}

and finally

\begin{equation}
R^n = \dvg \left( \frac{1}{P^2} \frac{1}{(n+2)} \left( P^2 + h^2 \right)^{\frac{n+2}{2}} \vt{P} \right)
\end{equation}

## Computation by Recursion

Using this the integral we want to compute reduces to:

\begin{equation}
\lim_{\epsilon \rightarrow 0} \frac{1}{n+2} \int_{\partial(T \setminus B_{\epsilon})} \frac{ \left( P^2 + h^2 \right)^{\frac{n+2}{2}}}{P^2} \uv{m} \cdot \vt{P} dy - \frac{1}{n+2} \lim_{\epsilon \rightarrow 0} \int_{C_{\epsilon}} \frac{ \left( P^2 + h^2 \right)^{\frac{n+2}{2}}}{P^2} \uv{m} \cdot \vt{P} dy
\end{equation}

The integration path of the contour in the second term is $\partial (B_{\epsilon}(\xi) \cap T)$. For $\xi \notin T$ the contour is void, for $\xi \in T$ it is a circle, for $\xi$ in the interior of one of the sides comprising $\partial T$ it is half a circle, and for $\xi$ in the vertices of $T$ it is a circle spanning the angle formed at those vertices w.r.t the interior of $T$. If $\xi \in \partial T$, the integral in the first term is meant in the Cauchy principle value sense.

The second term is a contribution of a small circlular area of radius $\epsilon$ centered on $\xi$. This region needs to be excluded prior to applying the surfacic divergence theorem because the potential is not smooth there. It can easily be evaluated to

\begin{align}
S_n =& - \frac{1}{n+2} \lim_{\epsilon \rightarrow 0} \int_{C_{\epsilon}} \frac{ \left( P^2 + h^2 \right)^{\frac{n+2}{2}}}{P^2} \uv{m} \cdot \vt{P} dy \\
=& -\frac{\alpha}{n+2} |h|^{n+2}
\end{align}

The boundary of the triangle is the disjoint union of the three segments: $\partial T = \cup_{i=1}^3 \partial_i T$. On $\partial_i T$, $\uv{m} \cdot \vt{P}$ takes on the constant value denoted $p_i$ (the sign of $p_i$ depends on the position of $\vt{\xi}$ with respect to the oriented line $\partial_i T$). The segment $\partial_i T$ is parametrised by

\begin{equation}
\partial_i T(s) = \vt{y}_{0,i} + s \uv{t}_i, \quad s \in [a_i,b_i]
\end{equation}

where $\vt{y}_{0,i}$ is the orthogonal projection of $\vt{x}$ on $\partial T_i$, and $\uv{t}_i$ is the unit vector along $\partial_i T$.

The integral reduces to:

\begin{equation}
\frac{1}{n+2} \sum_{i=1}^3 p_i \int_{a_i}^{b_i} \frac{\left(s^2 + p_i^2 + h^2\right)^{\frac{n+2}{2}}}{s^2 + p_i^2} ds
\end{equation}

Let us now focus on a single term:

\begin{equation}
I_n = \frac{p}{n+2} \int_{a}^{b} \frac{(s^2+p^2+h^2)^{\frac{n+2}{2}}}{s^2+p^2} ds
\end{equation}

where indices $i$ where omitted.

For $n = -3$ we get

\begin{align}
I_{-3} =& -p \int_{a}^{b} \frac{(s^2+p^2+h^2)^{-\frac{1}{2}}}{s^2+p^2} ds \\
=& -\frac{1}{h} \tan^{-1} \frac{hs}{p \sqrt{s^2+p^2+h^2}}
\end{align}

*Note*: Here we used the dare I say non-trivial primitve:

\begin{equation}
\int \frac{ab}{\left( s^2 + a^2 \right) \sqrt{s^2 + a^2 + b^2}} = \tan^{-1} \frac{bs}{a \sqrt{s^2 + a^2 + b^2}}
\end{equation}

See e.g. [Wolfram Alpha](https://www.wolframalpha.com/input/?i=diff%28atan%28b*s%2F%28a*sqrt%28s%5E2%2Ba%5E2%2Bb%5E2%29%29%29,s%29).

For n = 0 the integral becomes

\begin{align}
I_0 =& \frac{1}{2} p \int_a^b \frac{s^2+p^2+h^2}{s^2+p^2} ds \\
=& \frac{1}{2} p \left( s + \frac{h^2}{p} \tan^{-1} \frac{s}{p} \right)
\end{align}

In the case $n > 0$ the integrals can be computed by recursion.

\begin{align}
I_n =& \frac{p}{n+2} \int \frac{(s^2+p^2+h^2)^{\frac{n+2}{2}}}{s^2+p^2} ds \\
=& \frac{p}{n+2} \int \frac{(s^2+p^2+h^2)^{n/2}\left(s^2+p^2+h^2\right)}{s^2+p^2} ds \\
=& \frac{p}{n+2} \left( \int \left( s^2+p^2+h^2 \right)^{n/2} ds + h^2 \frac{n}{p} I_{n-2}\right) \\
=& \frac{p}{n+2} J_n(s) + h^2 \frac{n}{n+2} I_{n-2} \\
\end{align}

The first term can be computed by recursion. Consider the sequence of integrals:

\begin{align}
L_n =& \int \left( t^2 + 1 \right)^{n/2} dt \\
=& \int \cosh^{n+1} (\alpha) d\alpha \quad \mbox{where } t = \sinh \alpha \\
=& \cosh^n \alpha \sinh \alpha - n \int \cosh^{n-1} \alpha \sinh^2 \alpha \\
=& \left(1 + t^2\right)^{\frac{n}{2}} t - n \int \cosh^{n-1} \alpha (\cosh^2 \alpha - 1) \\
=& \left(1 + t^2\right)^{\frac{n}{2}} t - n \left( L_n - L_{n-2} \right)
\end{align}

From which

\begin{equation}
L_n = \frac{1}{n+1} \left( \left(1 + t^2\right)^{\frac{n}{2}} t + n J_{n-2} \right)
\end{equation}

The recursion is bootstrapped from

\begin{equation}
L_{-1} = \log \left( t + \sqrt{1 + t^2} \right)
\end{equation}

And so we get for the sequence $J_n$:

\begin{align}
J_{n}(s) =& \int \left( s^2 + p^2 + h^2 \right)^{\frac{n}{2}} ds \\
=& \left( p^2 + h^2 \right)^{\frac{n+1}{2}} \int (t^2+1)^{n/2} dt, \quad t = \frac{s}{\sqrt{p^2+h^2}} \\
=& \left( p^2 + h^2 \right)^{\frac{n+1}{2}} L_n(t) \\
=& \frac{1}{n+1} \left( p^2 + h^2 \right)^{\frac{n+1}{2}} \left( \left(1 + t^2\right)^{\frac{n}{2}} t + n L_{n-2} \right) \\
=& \frac{1}{n+1} \left( s \left( s^2 + p^2 + h^2 \right)^{\frac{n}{2}} + n (p^2+h^2) J_{n-2}(s) \right)
\end{align}

bootstrapped by

\begin{equation}
J_{-1}(s) = \log \left( s + \sqrt{s^2 + p^2 + h^2} \right)
\end{equation}

This recursion was already mentioned in [3].

In the case where $p=0$ and $h=0$ we will use the limiting value

\begin{equation}
\left. J_{-1}(s) \right|_a^b = \lim_{\epsilon \rightarrow 0} \log \left( \frac{b+\sqrt{b^2+\epsilon^2}}{a + \sqrt{a^2 +\epsilon^2}} \right) = \left\{ \begin{array}{cl}
\log \left( \frac{b}{a} \right), & \mbox{when } b > 0 \\
\log \left( \frac{a}{b} \right), & \mbox{when } b < 0 \\
\end{array} \right.
\end{equation}

*note*: The limit is only finite if $a$ and $b$ have the same sign. This is guaranteed because the singularity $\vt{x}$ is not allowed to be on the boundary (even though it is allowed to be on the extension of these segments).

The first recursion gives the integral for $n = -1$, i.e. one of the values computed in [1]:

\begin{align}
I_{-1} =& p \int_{a}^{b} \frac{(s^2+p^2+h^2)^{1/2}}{s^2+p^2} ds \\
=& p \int_{a}^{b} \left( \frac{1}{\sqrt{s^2+p^2+h^2}} + \frac{h^2}{\sqrt{s^2+p^2+h^2}(s^2+p^2)} \right) ds \\
=& p \left. \log (s + \sqrt{s^2+p^2+h^2}) \right|_a^b + h \left. \tan^{-1} \frac{hs}{p \sqrt{s^2+p^2+h^2}} \right|_a^b
\end{align}

### Combining the contour and singularity contributions

For small $p$, the argument of $\tan^{-1}$ becomes very large. Moreover, the contributions stemming from the integral over $\partial T$ and those stemming from the region around $\xi$ cancel in the final computation of $\sum_i I_{n,i} + S_n$.

In [1] it is suggested to combine both contributions using the addition property of the $\tan^{-1}$. In this section this will be generalised for arbitrary powers of $R$. Not only will this result in the stable computation of the integrals, it will also eliminate branches from the code.

Using the computations from previous section we have

\begin{align}
\int \frac{h}{R^3} =& - \sum_{i=1}^3 \left. \tan^{-1} \frac{hs}{p_i \sqrt{s^2+p_i^2+h^2}} \right|_{a_i}^{b_i} + \alpha \sigma(h) \\
=& - \sigma(h) \left( \sum_{i=1}^3 \left. \tan^{-1} \frac{|h|s}{p_i \sqrt{s^2+p_i^2+h^2}} \right|_{a_i}^{b_i} - \alpha \right) \\
\end{align}

The interior angle spanned by $\partial T$ as seen by $\xi$ can be written as:

\begin{equation}
a = \sum_{i=1}^3 \left. \tan^{-1} \frac{s}{p_i} \right|_{a_i}^{b_i}
\end{equation}

The [addition theorem](https://en.wikipedia.org/wiki/Inverse_trigonometric_functions#Arctangent_addition_formula) for $\tan^{-1}$ applied to this situtation allows to combine corresponding terms from $\sum_i I_{n,i}$ and $S_n$:

\begin{equation}
\tan^{-1} \frac{s}{p_i} - \tan^{-1} \frac{|h|s}{p_i \sqrt{s^2+p_i^2+h^2}} = \tan^{-1} \frac{p_i s}{(p_i^2 + h^2) + |h| \sqrt{s^2+p_i^2+h^2}}
\end{equation}

Note that

\begin{equation}
S_n = h^2 \frac{n}{n+2} S_{n-2}
\end{equation}

and comparing this with the second term in the recursion for $I_n$ ($n$ odd):

\begin{equation}
I_{n,i} = ... + h^2 \frac{n}{n+2} I_{n-2,i}
\end{equation}

tells us that once the merging of the two contributions is done for $n=-3$, the recursion will generate the correct answers for all other powers of $R$.

Whenever $p=0$, the $\tan^{-1}$ will be set to zero. Not only will this result in more efficient code in that case, it will also void division by zero when in addition also $h=0$.

For $n = 0$ the situation is even better:

\begin{align}
I_{0} =& \frac{1}{2} \sum_{i=1}^{3} \left( p_i \left. s \right|_{a_i}^{b_i} + h^2 \left. \tan^{-1} \frac{s}{p} \right|_{a_i}^{b_i} \right) -\frac{1}{2} h^2 \alpha \\
=& \frac{1}{2} \sum_{i=1}^{3} p_i \left. s \right|_{a_i}^{b_i}
\end{align}

Again because the second term in the recursion for $I_n$ and the recursion for the singularity contribution behave identically, the $\tan^{-1}$ will cancel  in the computation of integrals of all powers of $R$.

## The vector case

In this section we will compute integrals of the form

\begin{equation}
\int \frac{\vt{y} - \vt{\xi}}{R^n} dy, \quad n=-3,-1,0,1,2,...
\end{equation}

In this case we will look for scalar potentials such that their surfacic gradient is the above integrand:

\begin{equation}
\frac{\vt{y} - \vt{\xi}}{R^n} = \grd_T F(P;h)
\end{equation}

or

\begin{equation}
\frac{1}{P} F'(P;h) \vt{P} = \left( P^2 + h^2 \right)^{n/2} \vt{P}
\end{equation}

from which

\begin{align}
F(p) =& \int \left(P^2 + h^2 \right)^{n/2} dP \\
=& \frac{1}{2} \int_{Q = P^2+h^2} Q^{n/2} dQ \\
=& \frac{1}{n+2} \left( P^2 + h^2 \right)^{\frac{n+2}{2}}.
\end{align}

So we conclude

\begin{equation}
R^{n} = \grd_T \left( \frac{1}{n+2} \left( P^2 + h^2 \right)^{\frac{n+2}{2}} \right)
\end{equation}

The integral we are looking for thus can be written as

\begin{align}
\int_T \frac{\vt{y} - \vt{\xi}}{R^n} dy =& \lim_{\epsilon \rightarrow 0} \left( \int_{\partial T \setminus B_{\epsilon}} \frac{\uv{m}}{n+2} \left( P^2 + h^2 \right)^{\frac{n+2}{2}} dy + \int_{T \cap C_{\epsilon}} \frac{\uv{m}}{n+2} \left( P^2 + h^2 \right)^{\frac{n+2}{2}} dy \right)
\end{align}

For $n \geq -1$ the second term clearly disappears when $\epsilon \rightarrow 0$. Also when $n=-3$ and $h \neq 0$ the singular contribution disappers. It is only the case where $n=-3$ and $h = 0$ that needs a more careful analysis of the second term. If $\xi \notin \partial T$, the integral vanishes because contributions on opposite sides of $\vt{\xi}$ cancel. When $\xi \in \partial T$ this is not necessarily the case. However, we can safely assume this case will never occur because in that case the integral $\int_T (\vt{y}-\vt{\xi}) R^{-3}$ diverges anyway.

In conclusion either the integral diverges or the second term is zero. We can focus or efforts on the first term.

Like we did for the scalar case, the integral can be written as

\begin{equation}
\int_T \frac{\vt{y} - \vt{\xi}}{R^n} dy = \sum_{i=1}^{3} \uv{m}_i \left. K_n(s,p_i,h) \right|_{a_i}^{b_i}
\end{equation}

with

\begin{equation}
K_n(s,p,h) = \frac{1}{n+2} \int_a^b \left( s^2 + p^2 + h^2 \right)^{\frac{n+2}{2}} = \frac{1}{n+2} J_{n+2}\left(s,p,h\right)
\end{equation}

which can be easily expressed in terms of quantities we already encountered in our previous caluclations. In other words computation of the vector integrals comes at almost no computational overhead.

## Extension to Triangle/Disk intersections

In the Space-Time Galerkin discretisation of time domain retarded potential boundary integral equations, the computation of integrals over the following class of domains is required:

\begin{equation}
D = T \cap B(\xi,Q)
\end{equation}

where $T$ is a triangle and $B(\xi,Q)$ is the disk with centre $\xi$ (assumed to lie in the plane supporting $T$) and radius $Q$.

When given a triangle by its three vertices and a disk by its center and radius, the construction of a boundary representation of such a domain is nothing short of a nightmare, and will not be discussed in this document.

We start from the alternative surfacic potential representation for the kernel:

\begin{equation}
R^n = \dvg \left( \frac{1}{(n+2) P^2} \left( \left( P^2 + h^2 \right)^{\frac{n+2}{2}} - \left(h^2 \right)^{\frac{n+2}{2}} \right) \vt{P} \right)
\end{equation}

Note that the term we added has vanishing divergence and thus will not affect the value of the integral. The reason for this alternative potential is that it is regular at $P=0$ and so we do not have to keep track of the singularity term separately.

The boundary $\partial D$ is made up from straight segments and arcs. Integration of the potential along the straight segments is dealt with as explained in the previous sections. For an arc $C$ the contribution to the integral looks like:

\begin{align}
A_n =& \frac{1}{n+2} \int_{C} \frac{\left( P^2 + h^2 \right)^{\frac{n+2}{2}} - \left( h^2 \right)^{\frac{n+2}{2}}}{P^2} \uv{m} \cdot \vt{P} dy \\
=& \frac{\alpha}{n+2} \left( \left( p^2 + h^2 \right)^{\frac{n+2}{2}} - \left(h^2\right)^{\frac{n+2}{2}} \right)
\end{align}

where $\alpha$ is the signed angle spanned by the arc (positive for arcs traversed in counterclockwise sense, negative otherwise). From this expression we get the following recursion:

\begin{equation}
A_n = \frac{\alpha}{n+2} \left[ \left( \frac{n}{\alpha} A_{n-2} + \left( h^2 \right)^{\frac{n}{2}} \right) \left( p^2 + h^2 \right) - \left( h^2 \right)^{\frac{n+2}{2}} \right]
\end{equation}

Bootstrapped by

\begin{align}
A_{-3} =& -\alpha \left( \left( p^2 + h^2 \right)^{-\frac{1}{2}} - \left( h^2 \right)^{-\frac{1}{2}} \right) \\
A_{0} =& \frac{\alpha}{2} p^2
\end{align}

For the vector case the contribution from a single arc reads

\begin{align}
B_n =& \int_C \uv{m} \left( \frac{1}{n+2} \left( P^2 + h^2 \right)^{\frac{n+2}{2}} \right) dy \\
=& \frac{1}{n+2} \left( p^2 + h^2 \right)^{\frac{n+2}{2}} \int_C \uv{m} dy
\end{align}

*Note*: In the vector case we need to be consistent in our choice for the potential $\Phi$ since the quantity we integrate over is $\uv{m} \Phi$ which does not necessarily cancel at the inter-sector boundary. In contrast, for the scalar case, the quantity $\uv{m} \cdot (F(P) \vt{P})$ vanishes along the inter-sector boundary and does $F(P)$ can be chosen freely in each sector. So if we would leave out the second term in the surfacic potential, we would have to do so also in the computation of the segment contributions, forcing us to consider what happens in the neighborhood of $\xi$. For certain configurations of $D$ this point might not even be in the integration domain. The resulting algorithm becomes inelegant and prone to numerical and logical errors.

*Note*: for contributions of arcs and circles we can assume that $p \neq 0$. This ensures for example that no divisions by zero occur in the computation of $h A_{-3}$

*Note* In the case of a full circle $\alpha = \pm 2\pi$, $B_n$ vanishes.

```julia
using WiltonInts84
using FixedSizeArrays
```

```julia
using PyPlot
```

```julia
p1 = Vec(0.0, 0.0, 0.0)
p2 = Vec(1.0, 0.0, 0.0)
p3 = Vec(0.0, 1.0, 0.0)
h = 0.1
```

    0.1

```julia
x = -0.5 : 0.01 : 1.5
y = -0.5 : 0.01 : 1.5
```

    -0.5:0.01:1.5

```julia
t, T = 0.15, 1
r, R = max(0,t-T), max(0,t)
```

    (0.0,0.15)

```julia
M = zeros(typeof(t), length(x), length(y))
for i in eachindex(x)
    for j in eachindex(y)
        c = Vec(x[i],y[j],h)
        try
            I, K = wiltonints(p1,p2,p3,c,r,R,Val{0})
            M[i,j] = I[2]
        catch
            #@show i j
            #@show c
        end
    end
end
```

```julia
imshow(M); colorbar()
```

The contour computation is not 100% robust at this time. In practice this is not a serious issue, since the coordinates of $\vt{x}$ depend on the location of the e.g. Gauss-Legendre quadrature points used for the testing integration in the variational method this toolbox is meant for. These coordinates are *weird* numbers as in highly unlikely to hit the geometric rountine's boundary cases.


# Expectation Maximization for latent variable models

In all the notebooks we've seen so far, we have made the assumption that the observations correspond directly to realizations of a random variable. Take the case of linear regression: we are given observations of the random variable $t$ (plus some noise), which is the target value for a given value of the input $\mathbf{x}$. Under some criterion, we find the best parameters $\boldsymbol{\theta}$ of a model $y(\mathbf{x}, \boldsymbol{\theta})$ that is able to explain the observations and yield predictions for new inputs.

In the more general case, we have a dataset of observations $\mathcal{D} = \lbrace \mathbf{x_1}, ..., \mathbf{x_N}\rbrace$. We hypothesize that each observation is drawn from a probability distribution $p(\mathbf{x_i}\vert\boldsymbol{\theta})$  with parameters $\boldsymbol{\theta}$. It is sometimes useful to think of this as a probabilistic graphical model, where nodes represent random variables and edges encode dependency relationships between them. In this case, the graph looks as follows:

In this graph, we show that we have $N$ observations by enclosing the random variables within a *plate*. This also represents the fact that we assume the observations to be independent.

For many situations this model works, that is, the model is able to explain the data and can be used to make predictions for new observations. In other cases, however, this model is not expressive enough.

Imagine that we have a single dimensional variable $x$ of which we observe some samples. Our first hypothesis is that $p(x\vert\boldsymbol{\theta})$ is a normal distribution, so we proceed to find the mean and variance of this distribution using maximum likelihood estimation (MLE):

```python
%matplotlib inline
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
from data.synthetic import gaussian_mixture

X = gaussian_mixture(200)
# The MLE estimates are the sample mean and standard deviation
mean = np.mean(X)
std = np.std(X)

# Plot fit on top of histogram
fig, ax1 = plt.subplots()
ax1.hist(X, alpha=0.4)
ax1.set_ylabel('Counts')
ax1.set_xlabel('x')

ax2 = ax1.twinx()
x = np.linspace(-4, 4, 100)
ax2.plot(x, norm.pdf(x))
ax2.set_ylim([0, 0.5])
ax2.set_ylabel('Probability density');
```

Clearly, once we have actually examined the data, we realize that a single normal distribution is not a good model. The data seems to come from a multimodal distribution with two components, which a single Gaussian is not able to capture. In this case we are better off by changing our model to a *mixture model*, a model that mixes two or more distributions. For the example above, it would seem that a mixture of two components, centered at -2 and 2 would be a better fit.

Under this idea, our hypothesis is the next: there are $K$ components in the mixture model. We start by selecting a component $k$ with some probability $\pi_k$ and, given the selected component, we then draw a sample $x$ from a normal distribution $\mathcal{N}(x\vert\mu_k, \sigma_k^2)$.

We can think of the component as a discrete random variable $\mathbf{z}$ that can take values from 1 up to $K$. Therefore, to each sample $x$ there is an associated value of $\mathbf{z}$. Since we do not observe $\mathbf{z}$, we call it a **latent variable**. If we collapse the parameters $\pi_k$, $\mu_k$ and $\sigma_k$ into a single parameter $\boldsymbol{\theta}$, the graphical model for the general case is now the following:

Note how the model emphasizes the fact that each observation has an associated value of the latent variable. Also note that since $z$ is not observed, its node is not shaded.

We can now proceed to find the parameters of our new model, using MLE (or maximum a posteriori, if we have priors on the parameters). However, as it usually is the case, we have traded tractability for expressiveness by introducing latent variables. If we attempt to maximize the log-likelihood, we find that we need to maximize

$$
\begin{align}
\sum_{n=1}^N\log \sum_{z}p(x_n\vert z_n, \boldsymbol{\theta})p(z_n\vert\boldsymbol{\theta})
\end{align}
$$

which due to the summation inside the logarithm, does not result in a closed form solution. An alternative is to use the **Expectation Maximization** algorithm, an iterative procedure that can be used to find maximum likelihood estimates of the parameters, which we motivate next.

Let $\mathbf{X}$ and $\mathbf{Z}$ denote the set of observed and latent variables, respectively, for which we have defined a joint parametric probability distribution

$$
p(\mathbf{X}, \mathbf{Z}\vert\boldsymbol{\theta}) = p(\mathbf{X}\vert\mathbf{Z},\boldsymbol{\theta})p(\mathbf{Z}\vert\boldsymbol{\theta})
$$

It can be shown [1, 2] that for any distribution $q(\mathbf{Z})$ we can decompose the log-likelihood as

$$
\log p(\mathbf{X}\vert\boldsymbol{\theta}) = \mathcal{L}(q, \boldsymbol{\theta}) + \text{KL}(q(\mathbf{Z})\Vert p(\mathbf{Z\vert\mathbf{X}, \boldsymbol{\theta}}))\tag{1}
$$

where $\text{KL}$ is the Kullback-Leibler divergence, and $\mathcal{L}(q,\boldsymbol{\theta})$ is known as the Evidence Lower Bound (ELBO), because since the KL divergence is always non-negative, it is a lower bound for $p(\mathbf{X}\vert\boldsymbol{\theta})$. The ELBO is defined as

$$
\mathcal{L}(q, \boldsymbol{\theta}) = \mathbb{E}_q[\log p(\mathbf{X},\mathbf{Z}\vert\boldsymbol{\theta})] - \mathbb{E}_q[\log q(\mathbf{Z})]\tag{2}
$$

Note that these expectations are taken with respect to the distribution $q(\mathbf{Z})$.

We can now use this decomposition in the EM algorithm to define two steps:

- **E step:** Initialize the parameters with some value $\boldsymbol{\theta}^\prime$. In equation 1, close the gap between the lower bound and the likelihood by making the KL divergence equal to zero. We achieve this by setting $q(\mathbf{Z})$ equal to the posterior $p(\mathbf{Z}\vert\mathbf{X},\boldsymbol{\theta}^\prime)$, which usually involves using Bayes' theorem to calculate

  $$
  p(\mathbf{Z}\vert\mathbf{X},\boldsymbol{\theta}^\prime) =
  \frac{p(\mathbf{X}\vert\mathbf{Z},\boldsymbol{\theta}^\prime)p(\mathbf{Z}\vert\boldsymbol{\theta}^\prime)}{p(\mathbf{X}\vert\boldsymbol{\theta}^\prime)}
  $$

- **M step:** now that the likelihood is equal to the lower bound, maximize the lower bound in equation 2 with respect to the parameters. We find

  $$
  \boldsymbol{\theta}^{\text{new}} = \arg\max_{\boldsymbol{\theta}} \mathbb{E}_q[\log p(\mathbf{X},\mathbf{Z}\vert\boldsymbol{\theta})]
  $$

  where we have dropped the second term in equation 2 as it does not depend on the parameters, and the expectation is calculated with respect to $q(\mathbf{Z}) = p(\mathbf{Z}\vert\mathbf{X},\boldsymbol{\theta}^\prime)$. In this step we calculate the derivatives with respect to the parameters and set them to zero to find the maximizing values.

This process is repeated until convergence.

## A practical example

We will use this idea to fit a mixture model from a subset of the famous MNIST dataset. In this dataset each image is of size 28 by 28, containing a handwritten number between 0 and 9, although for simplicity we will take only the digits 4, 5, and 2.

We will process the images so that they are binary, so that a pixel can be either 1 or 0. The images are flattened, so that a digit is represented by a vector of 28 $\times$ 28 = 784 values.

```python
from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

# Fetch data from OpenML.org
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)

# Take three digits only
sample_indices = []
for i, label in enumerate(y):
    if label in ['4', '5', '2']:
        sample_indices.append(i)

X = X[sample_indices]
y = y[sample_indices]

# Binarize
X_bin = np.zeros(X.shape)
X_bin[X > np.max(X)/2] = 1

# Visualize some digits
plt.figure(figsize=(14, 1))
for i in range(10):
    plt.subplot(1, 10, i+1)
    plt.imshow(X_bin[i].reshape((28, 28)), cmap='bone')
    plt.axis('off')
```

Our hypothesis is that each pixel $x_i$ in the image $\mathbf{x}$ is a Bernoulli random variable with probability $\mu_i$ of being 1, and so we define the vector $\boldsymbol{\mu}$ as containing the probabilities for each pixel.

The probabilities can be different depending on whether the number is a 2, a 4 or a 5, so we will define 3 components for the mixture model. This means that there will be 3 mean vectors $\boldsymbol{\mu}_i$ and each component will have a probability $\pi_k$. These are the parameters that we will find using the EM algorithm, which I have implemented in the [`BernoulliMixture`](https://github.com/dfdazac/machine-learning-1/blob/master/mixture_models/bernoulli_mixture.py) class.

```python
from mixture_models.bernoulli_mixture import BernoulliMixture

model = BernoulliMixture(dimensions=784, n_components=3, verbose=True)
model.fit(X_bin)
```

    Iteration 9/100,  convergence: 0.000088
    Terminating on convergence.

We can now observe the means and the mixing coefficients of the mixture model after convergence of the EM algorithm, which are stored in the `model.mu` attribute:

```python
plt.figure(figsize=(10, 4))
for i, component_mean in enumerate(model.mu):
    plt.subplot(1, 3, i + 1)
    plt.imshow(component_mean.reshape((28, 28)), cmap='Blues')
    plt.title(r'$\pi_{:d}$ = {:.3f}'.format(i + 1, model.pi[i]))
    plt.axis('off')
```

We can see that the means correspond to the three digits in the dataset. In this particular case, given a value of 1 for the latent variable $\mathbf{z}$ (corresponding to digit 4), the observation (the image) is a sample from the distribution whose mean is given by the one to the left in the plots above. The mixing coefficients give us an idea of the proportion of instances of each digit in the dataset used to train the model.

As we have seen, introducing latent variables has allowed us to specify complex probability distributions that are likely to provide more predictive power in some applications. Other examples include mixtures of Gaussians for clustering applications, where usually an algorithm like K-means fails at capturing different covariances for each cluster; and non-trivial probabilistic models of real life data, such as click models [2].

It is important to note that the EM algorithm is not the ultimate way to find parameters of models with latent variables. On one hand, the method is sensitive to the initialization of the parameters and might end up at a local maximum of the log-likelihood. On the other hand, we have assumed that we can calculate the posterior $p(\mathbf{Z}\vert\mathbf{X},\boldsymbol{\theta})$, which sometimes is not possible for more elaborate models. In such cases it is necessary to resort to approximate methods such as sampling and variational inference. This last area has led to many recent advances, including the Variational Autoencoder, which I would like to discuss in the near feature.

### References

[1] Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. The MIT Press.

[2] Dan Piponi, "Expectation-Maximization with Less Arbitrariness", blog post at [http://blog.sigfpe.com/2016/10/expectation-maximization-with-less.html](http://blog.sigfpe.com/2016/10/expectation-maximization-with-less.html).

[2] Chuklin, Aleksandr, Ilya Markov, and Maarten de Rijke. "Click models for web search." Synthesis Lectures on Information Concepts, Retrieval, and Services 7, no. 3 (2015): 1-115.


# Code and Physical Units

## GRMHD Units in terms of EOS

$\newcommand{\rhoCode}{{\tilde{\rho}}}$

$\newcommand{\MCode}{{\tilde{M}}}$ $\newcommand{\rCode}{{\tilde{r}}}$ $\newcommand{\PCode}{{\tilde{P}}}$$\newcommand{\tCode}{{\tilde{t}}}$$\newcommand{\Mfid}{{M_{\rm fid}}}$$\newcommand{\MfidBar}{\bar{M}_{\rm fid}}$$\newcommand{\Mbar}{\bar{M}}$
$\newcommand{\rBar}{\bar{r}}$$\newcommand{\tBar}{\bar{t}}$
In GRMHD, we can set an equation of state of the form
\begin{equation}
P = K\rho^{1+1/n}
\end{equation}
Taking $c_s^2 = \partial P/\partial \rho = (1+1/n) K\rho^{1/n}$.  This gives for some fidicial $\rho_0$
\begin{equation}
c_{s,0}^2 = \left(1 + \frac 1 n\right)K\rho_0^{1/n}.
\end{equation}
Selecting $c_s^2 = c^2\left(1 + 1/n\right)$, we have
\begin{equation}
\rho_0 = \left(\frac {c^2}{K}\right)^n
\end{equation}
This is equivalent to setting the isothermal sound speed to $c$.  With this definition of $\rho_0$, we can write
\begin{equation}
P = \rho_0c^2\left(\frac{\rho}{\rho_0}\right)^{1+1/n}
\end{equation}
which allows us to define the dimensionless density $\rhoCode = \rho/\rho_0$ and dimensionless pressure $\PCode = P/\rho_0 c^2$
\begin{equation}
\PCode = \rhoCode^{1+1/n},
\end{equation}
where we adopt code units where $c=1$.  These dimensionless pressure and density are in $G=c=1$ units and can be used in GRMHD code including inclusion in the spacetime solver via $T_{\mu\nu}$.  Note that this sets $K=1$ in these units.

To find a dimensionless mass, $\MCode$, dimensionless distance, $\rCode$, and dimensionless time, $\tCode$, we note
$GM/rc^2$ is dimensionless
\begin{equation}
\frac{GM}{rc^2} = \frac{G\rho_0 r^2}{c^2} = \frac{Gc^{2n-2}}{K^n}r^2 \rightarrow \rCode = \frac{\sqrt{G}c^{n-1}}{K^{n/2}} r = \frac r {r_0},
\end{equation}
where $r_0 = K^{n/2}/\sqrt{G}c^{n-1}$.  Then
\begin{eqnarray}
\tCode &=& \frac{t}{t_0} = \frac{t}{r_0/c} = \frac{\sqrt{G}c^n}{K^{n/2}} t \\
\MCode &=& \frac{M}{M_0} = \frac{M}{\rho_0 r_0^3} = M\frac{K^n}{c^{2n}}\frac{G^{3/2}c^{3(n-1)}}{K^{3n/2}} = \frac{G^{3/2}c^{n-3}}{K^{n/2}} M,
\end{eqnarray}
Hence, we have
\begin{eqnarray}
\rho_0 &=& \left(\frac{K}{c^2}\right)^n\\
r_0 &=& \frac{c^{n+1}}{\sqrt{G}K^{n/2}}\\
t_0 &=& \frac{c^{n}}{\sqrt{G}K^{n/2}}\\
M_0 &=& \frac{c^{n+3}}{G^{3/2}K^{n/2}}
\end{eqnarray}

## Mapping to SENR or any NR code

So we will need a $\Mfid$ which is define such that the (SENR) code units $\MfidBar = 1$ or in other words in SENR codes units:
\begin{equation}
\Mbar = \frac{M}{\Mfid}
\end{equation}
In these units:
\begin{eqnarray}
\rBar &=& \frac{c^2}{G\Mfid} r\\
\tBar &=& \frac{c^3}{G\Mfid} t
\end{eqnarray}
At some level $\Mfid$ is arbitrary, so we can select $M_0 = \Mfid$.  In this case, this means that $\rBar = \rCode$, $\tBar = \tCode$, and $\Mbar = \MCode$, which fixes all the quantities. This comes at a cost the $\bar{M}_{\rm ADM}$ is not something nice like 1 or 2, but the choice is consistent.

### Zach's comments:

Sound speed $c_s$ is defined as

$$\frac{\partial P}{\partial \rho} = c_s^2,$$

so if we have a polytropic EOS, where

$$P = K \rho^{(1 + 1/n)},$$

then

\begin{align}
\frac{\partial P}{\partial \rho} &= c_s^2 \\
&= \left(1 + \frac{1}{n}\right) K \rho^{1/n}.
\end{align}

Let's adopt the notation

$$[\rho] = \text{"the units of $\rho$"}$$

Using this notation and the fact that $n$ is dimensionless, the above expression implies

\begin{align}
\left[\rho^{1/n}\right] &= \left[\frac{c_s^2}{K}\right] \\
\implies \left[\rho\right] &= \left[\frac{c_s^2}{K}\right]^n
\end{align}

I think you found the inverse to be true.

# TOV Solver as illustration
The TOV equations are
\begin{eqnarray}
\frac{dP}{dr} &=& -\mu\frac{GM}{r^2}\left(1 + \frac P {\mu c^2}\right)\left(1 + \frac {4\pi r^3 P}{Mc^2}\right)\left(1 - \frac {2GM}{rc^2}\right)^{-1}\\
\frac{dM}{dr} &=& 4\pi \mu r^2,
\end{eqnarray}
Here $M$ is the rest mass measure by a distant observer when we take $r\rightarrow \infty$.  Note this is different from the mass measured by integrating the density over the volume
\begin{equation}
M' = \int_0^{\infty} \frac{4\pi r^2\mu}{\sqrt{1 - \frac {2 GM}{rc^2}}} dr
\end{equation}
Additionally and annoyingly, $\mu = \rho h$ is the mass-energy density.  Alot of the literature uses $\rho$ for this, which is incredibly annoying.

$\newcommand{\muCode}{{\tilde{\mu}}}$

In dimensionless units they are
\begin{eqnarray}
\frac{d\PCode}{d\rCode} &=& -\frac {\left(\muCode + \PCode\right)\left(\MCode + 4\pi \rCode^3 \PCode\right)}{\rCode^2\left(1 - \frac {2\MCode}{\rCode}\right)}\\
\frac{d\MCode}{d\rCode} &=& 4\pi \muCode\rCode^2
\end{eqnarray}

At this point, we need to discuss how to numerically integrate these models.  First we pick a central baryonic mass density $\rhoCode_{0,c}$, then we compute a central pressure $\PCode_c$ and central mass-energy density $\muCode_c$.  At $\rCode=0$, we assume that $\muCode=\muCode_c$ is a constant and so
\begin{eqnarray}
\frac{d\PCode}{d\rCode} &=& -\frac {\left(\muCode_c + \PCode_c\right)\left(\MCode(\rCode \ll 1) + 4\pi \rCode^3 \PCode_c\right)}{\rCode^2\left(1 - \frac {2\MCode(\rCode \ll 1)}{\rCode}\right)}\\
\frac{d\MCode}{d\rCode} &=& 4\pi \muCode_c\rCode^2 \rightarrow \MCode(\rCode \ll 1) = \frac{4\pi}{3} \muCode_c \rCode^3
\end{eqnarray}

## Another dimensionless prescription

Let consider an alternative formulation where rather than setting $K=1$, we set the characteristic mass $\Mfid = M_0$.  In this case,
\begin{eqnarray}
r_0 &=& \frac{GM_0}{c^2} \\
t_0 &=& \frac{GM_0}{c^3} \\
\rho_0 &=& \frac{M_0}{r_0^3} = \frac{c^6}{G^3 M_0^2} = 6.17\times 10^{17}\left(\frac {M_0} {1 M_{\odot}}\right)^{-2}
\end{eqnarray}
In this case we can define $\rhoCode = \rho/\rho_0$, $\rCode = r/r_0$, $t_0 = t/t_0$. The only remaining thing to do is to define $\PCode$.  Lets define $P_0'$ to be the pressure in dimensionful units at $\rho_0$ (density in units of $1/M_0^2$):
\begin{equation}
P = P_0'\rhoCode^{1+1/n} \rightarrow P_0' = K\rho_0^{1+1/n},
\end{equation}
So defining $P_0 = \rho_0 c^2$, we have
\begin{equation}
\PCode = \frac{P}{P_0} = \frac{K\rho_0^{1/n}}{c^2}\rhoCode^{1+1/n} = \PCode_0\rhoCode^{1+1/n}
\end{equation}
If we take $K=1$ and define $\rho_0$ such that the $\PCode_0 = 1$, we recover the results above.
Finally for $\muCode = \rhoCode + \PCode/n$

## metric for TOV equation

The metric for the TOV equation (taken) from wikipedia is
\begin{equation}
ds^2 = - c^2 e^\nu dt^2 + \left(1 - \frac{2GM}{rc^2}\right)^{-1} dr^2 + r^2 d\Omega^2
\end{equation}
where $M$ is defined as above, the mass as measured by a distant observer.  The equation for $\nu$ is
\begin{equation}
\frac{d\nu}{dr} = -\left(\frac {2}{P +\mu}\right)\frac{dP}{dr}
\end{equation}
with the boundary condition
\begin{equation}
\exp(\nu) = \left(1-\frac {2Gm(R)}{Rc^2}\right)
\end{equation}

Lets write this in dimensionless units:
\begin{equation}
ds^2 = \exp(\nu) d\tCode^2 - \left(1 - \frac{2\MCode}{\rCode}\right)^{-1} d\rCode^2 + \rCode^2 d\Omega^2
\end{equation}
\begin{equation}
\frac{d\nu}{d\rCode} = -\left(\frac {2}{\PCode +\muCode}\right)\frac{d\PCode}{d\rCode}
\end{equation}
and BC:
\begin{equation}
\exp(\nu) = \left(1-\frac {2\MCode}{\rCode}\right)
\end{equation}

```python
import numpy as np
import scipy.integrate as si
import math
import matplotlib.pyplot as pl
n = 1.
rho_central = 0.129285

P0 = 1. # ZACH NOTES: CHANGED FROM 100.
gamma = 1. + 1./n
gam1 = gamma - 1.

def pressure( rho) :
    return P0*rho**gamma

def rhs( r, y) :
# In \tilde units
#
    P = y[0]
    m   = y[1]
    nu  = y[2]
    rbar = y[3]

    rho = (P/P0)**(1./gamma)
    mu = rho + P/gam1
    dPdr = 0.
    drbardr = 0.
    if( r < 1e-4 or m <= 0.) :
        m = 4*math.pi/3. * mu*r**3
        dPdr = -(mu + P)*(4.*math.pi/3.*r*mu + 4.*math.pi*r*P)/(1.-8.*math.pi*mu*r*r)
        drbardr = 1./(1. - 8.*math.pi*mu*r*r)**0.5
    else :
        dPdr = -(mu + P)*(m + 4.*math.pi*r**3*P)/(r*r*(1.-2.*m/r))
        drbardr = 1./(1. - 2.*m/r)**0.5*rbar/r

    dmdr = 4.*math.pi*r*r*mu
    dnudr = -2./(P + mu)*dPdr
    return [dPdr, dmdr, dnudr, drbardr]

def integrateStar( P, showPlot = False, dumpData = False, compareFile="TOV/output_EinsteinToolkitTOVSolver.txt") :
    integrator = si.ode(rhs).set_integrator('dop853')
    y0 = [P, 0., 0., 0.]
    integrator.set_initial_value(y0,0.)
    dr = 1e-5
    P = y0[0]

    PArr = []
    rArr = []
    mArr = []
    nuArr = []
    rbarArr = []
    r = 0.

    while integrator.successful() and P > 1e-9*y0[0] :
        P, m, nu, rbar = integrator.integrate(r + dr)
        r = integrator.t

        dPdr, dmdr, dnudr, drbardr = rhs( r+dr, [P,m,nu,rbar])
        dr = 0.1*min(abs(P/dPdr), abs(m/dmdr))
        dr = min(dr, 1e-2)
        PArr.append(P)
        rArr.append(r)
        mArr.append(m)
        nuArr.append(nu)
        rbarArr.append( rbar)

    M = mArr[-1]
    R = rArr[-1]

    nuArr_np = np.array(nuArr)
    # Rescale solution to nu so that it satisfies BC: exp(nu(R))=exp(nutilde-nu(r=R)) * (1 - 2m(R)/R)
    #   Thus, nu(R) = (nutilde - nu(r=R)) + log(1 - 2*m(R)/R)
    nuArr_np = nuArr_np - nuArr_np[-1] + math.log(1.-2.*mArr[-1]/rArr[-1])

    rArrExtend_np = 10.**(np.arange(0.01,5.0,0.01))*rArr[-1]

    rArr.extend(rArrExtend_np)
    mArr.extend(rArrExtend_np*0. + M)
    PArr.extend(rArrExtend_np*0.)
    phiArr_np = np.append( np.exp(nuArr_np), 1. - 2.*M/rArrExtend_np)
    rbarArr.extend( 0.5*(np.sqrt(rArrExtend_np*rArrExtend_np-2.*M*rArrExtend_np) + rArrExtend_np - M))
    # Appending a Python array does what one would reasonably expect.
    #   Appending a numpy array allocates space for a new array with size+1,
    #   then copies the data over... over and over... super inefficient.
    mArr_np       = np.array(mArr)
    rArr_np       = np.array(rArr)
    PArr_np       = np.array(PArr)
    rbarArr_np    = np.array(rbarArr)
    rhoArr_np     = (PArr_np/P0)**(1./gamma)
    confFactor_np = rArr_np/rbarArr_np
    #confFactor_np = (1.0 / 12.0) * np.log(1.0/(1.0 - 2.0*mArr_np/rArr_np))
    Grr_np        = 1.0/(1.0 - 2.0*mArr_np/rArr_np)
    Gtt_np        = phiArr_np
    if( showPlot) :
        r,rbar,rprop,rho,m,phi = np.loadtxt( compareFile, usecols=[0,1,2,3,4,5],unpack=True)
        pl.plot(rArr_np[rArr_np < r[-1]], rbarArr_np[rArr_np < r[-1]],lw=2,color="black")
        #pl.plot(r, rbar, lw=2,color="red")

        pl.show()

        if( dumpData) :
            np.savetxt( "output.txt", zip(rArr_np,rhoArr_np,PArr_np,mArr_np,phiArr_np,confFactor_np,rbarArr_np), fmt="%.15e")
            np.savetxt( "metric.txt", zip(rArr_np, Grr_np, Gtt_np),fmt="%.15e")
        #        np.savetxt( "output.txt", zip(rArr,rhoArr,mArr,phiArr), fmt="%12.7e")

#    return rArr[-1], mArr[-1], phiArr[-1]
    return R, M

mass = []
radius = []

R_TOV,M_TOV = integrateStar(pressure(rho_central), showPlot=True, dumpData=True)
print("Just generated a TOV star with r= "+str(R_TOV)+" , m = "+str(M_TOV)+" , m/r = "+str(M_TOV/R_TOV)+" .")

#for rho0 in np.arange(0.01, 1., 0.01):
#    r,m = integrateStar(pressure(rho0))
#    mass.append(m)
#    radius.append(r)

#print(mass, radius)
#pl.clf()
#pl.plot(radius,mass)
#pl.show()
```

    <Figure size 640x480 with 1 Axes>

    Just generated a TOV star with r= 0.956568142523 , m = 0.14050303285288188 , m/r = 0.1468824086931645 .

```python
# Generate the Sedov Problem

rArr_np = np.arange(0.01,5.,0.01)
rbarArr_np = rArr_np
rhoArr_np = np.ones(rArr_np.size)*0.1
mArr_np = 4.*np.pi/3.*rArr_np**3*rhoArr_np
PArr_np = rhoArr_np*1e-6
PArr_np[rArr_np < 0.5] = 1e-2
phiArr_np = np.ones(rArr_np.size)
confFactor_np = rArr_np/rbarArr_np
np.savetxt( "sedov.txt", zip(rArr_np,rhoArr_np,PArr_np,mArr_np,phiArr_np,confFactor_np,rbarArr_np), fmt="%.15e")

pl.semilogx(rArr_np, rhoArr_np)
pl.show()
```

## Convert metric to be in terms of ADM quantities

Above, the line element was written:
$$
ds^2 = - c^2 e^\nu dt^2 + \left(1 - \frac{2GM}{rc^2}\right)^{-1} dr^2 + r^2 d\Omega^2.
$$

In terms of $G=c=1$ units adopted by NRPy+, this becomes:
$$
ds^2 = - e^\nu dt^2 + \left(1 - \frac{2M}{r}\right)^{-1} dr^2 + r^2 d\Omega^2.
$$

The ADM 3+1 line element for this diagonal metric in spherical coordinates is given by:
$$
ds^2 = (-\alpha^2 + \beta_k \beta^k) dt^2 + \gamma_{rr} dr^2 + \gamma_{\theta\theta} d\theta^2+ \gamma_{\phi\phi} d\phi^2,
$$

from which we can immediately read off the ADM quantities:
\begin{align}
\alpha &= e^{\nu/2} \\
\beta^k &= 0 \\
\gamma_{rr} &= \left(1 - \frac{2M}{r}\right)^{-1}\\
\gamma_{\theta\theta} &= r^2 \\
\gamma_{\phi\phi} &= r^2 \sin^2 \theta \\
\end{align}

## Convert to cartesian coordinates

The above metric is given in spherical coordinates and we need everything in cartesian coordinates.  Given this the
transformation to cartesian coordinates is
\begin{equation}
g_{\mu\nu} = \Lambda^{\mu'}_{\mu} \Lambda^{\nu'}_{\nu} g_{\mu'\nu'},
\end{equation}
where $\Lambda^{\mu'}_{\mu}$ is the Jacobian defined as
\begin{equation}
\Lambda^{\mu'}_{\mu} = \frac{\partial x'^{\mu'}}{\partial x^{\mu}}
\end{equation}
In this particular case $x'$ is in spherical coordinates and $x$ is in Cartesian coordinates.

```python
import sympy as sp
import NRPy_param_funcs as par
from outputC import *
import indexedexp as ixp
import reference_metric as rfm

# The ADM & BSSN formalisms only work in 3D; they are 3+1 decompositions of Einstein's equations.
#    To implement axisymmetry or spherical symmetry, simply set all spatial derivatives in
#    the relevant angular directions to zero; DO NOT SET DIM TO ANYTHING BUT 3.
# Step 0: Set spatial dimension (must be 3 for BSSN)
DIM = 3

# Set the desired *output* coordinate system to Cylindrical:
par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
rfm.reference_metric()

CoordType_in = "Spherical"

r_th_ph_or_Cart_xyz_of_xx = []
if CoordType_in == "Spherical":
    r_th_ph_or_Cart_xyz_of_xx = rfm.xxSph
elif CoordType_in == "Cartesian":
    r_th_ph_or_Cart_xyz_of_xx = rfm.xxCart

Jac_dUSphorCart_dDrfmUD = ixp.zerorank2()
for i in range(DIM):
    for j in range(DIM):
        Jac_dUSphorCart_dDrfmUD[i][j] = sp.diff(r_th_ph_or_Cart_xyz_of_xx[i],rfm.xx[j])

Jac_dUrfm_dDSphorCartUD, dummyDET = ixp.generic_matrix_inverter3x3(Jac_dUSphorCart_dDrfmUD)

betaU   = ixp.zerorank1()
gammaDD = ixp.zerorank2()
gammaSphDD = ixp.zerorank2()
grr, gthth, gphph = sp.symbols("grr gthth gphph")
gammaSphDD[0][0] = grr
gammaSphDD[1][1] = gthth
gammaSphDD[2][2] = gphph
betaSphU = ixp.zerorank1()
for i in range(DIM):
    for j in range(DIM):
        betaU[i] += Jac_dUrfm_dDSphorCartUD[i][j] * betaSphU[j]
        for k in range(DIM):
            for l in range(DIM):
                gammaDD[i][j] += Jac_dUSphorCart_dDrfmUD[k][i]*Jac_dUSphorCart_dDrfmUD[l][j] * gammaSphDD[k][l]

outputC([gammaDD[0][0], gammaDD[0][1], gammaDD[0][2], gammaDD[1][1], gammaDD[1][2], gammaDD[2][2]],
              ["mi.gamDDxx", "mi.gamDDxy", "mi.gamDDxz", "mi.gamDDyy", "mi.gamDDyz","mi.gamDDzz"], filename="NRPY+spherical_to_cartesian_metric.h")
```

```python

```


## Problem Sheet Question 2a

The general form of the population growth differential equation
\begin{equation} y^{'}=y-t, \ \ (0 \leq t \leq 2) \end{equation}
with the initial condition
\begin{equation}y(0)=2.\end{equation}
For N=4
with the analytic (exact) solution
\begin{equation} y= e^{t}+t+1.\end{equation}

# 3-step Adams Bashforth

The 3-step Adams Bashforth difference equation is
\begin{equation}w_{i+1} = w_{i} + \frac{h}{12}(23f(t_i,w_i)-16f(t_{i-1},w_{i-1})+5f(t_{i-2},w_{i-2})) \end{equation}
where
\begin{equation}w_{i+1} = w_{i} + \frac{h}{12}(23(w_i-t_i)-(w_{i-1}-t_{i-1})+5(w_{i-2}-t_{i-2})). \end{equation}

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

```python
def myfun_ty(t,y):
    return y-t

#PLOTS
def Adams_Bashforth_3step(N,IC):

    x_end=2
    x_start=0
    INTITIAL_CONDITION=IC
    h=x_end/(N)
    N=N+2;
    t=np.zeros(N)
    w=np.zeros(N)

    Analytic_Solution=np.zeros(N)
    k=0
    w[0]=INTITIAL_CONDITION
    Analytic_Solution[0]=INTITIAL_CONDITION
    t[0]=x_start
    t[1]=x_start+1*h
    t[2]=x_start+2*h
    w[1]=math.exp(t[1])+t[1]+1
    w[2]=math.exp(t[2])+t[2]+1
    Analytic_Solution[1]=math.exp(t[1])+t[1]+1
    Analytic_Solution[2]=math.exp(t[2])+t[1]+1

    for k in range (2,N-1):
        w[k+1]=w[k]+h/12.0*(23*myfun_ty(t[k],w[k])-16*myfun_ty(t[k-1],w[k-1])+5*myfun_ty(t[k-2],w[k-2]))
        t[k+1]=t[k]+h
        Analytic_Solution[k+1]=2*math.exp(t[k+1])+t[k+1]+1

    fig = plt.figure(figsize=(10,4))
    # --- left hand plot
    ax = fig.add_subplot(1,3,1)
    plt.plot(t,w,'o:',color='blue')
    #ax.legend(loc='best')
    plt.title('Numerical Solution h=%s'%(h))

    # --- right hand plot
    ax = fig.add_subplot(1,3,2)
    plt.plot(t,Analytic_Solution,color='black')
    plt.title('Analytic Solution')

    ax = fig.add_subplot(1,3,3)
    plt.plot(t,Analytic_Solution-w,'o:',color='red')
    plt.title('Error')
 # --- title, explanatory text and save

    # --- title, explanatory text and save
    fig.suptitle(r"$y'=t-y$", fontsize=20)
    plt.tight_layout()
    plt.subplots_adjust(top=0.85)
    print(t)
    print(Analytic_Solution)
    print(w)
```

```python
Adams_Bashforth_3step(4,2)
```

```python

```

Chapter 04 - Multistep Methods/4_Problem_Sheet/402a_Problem_Sheet.ipynb