# Polynomial Regression

## Setup

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

Define the convenience function *polynomial_features* that implements the polynomial feature mapping/transformation $\boldsymbol{\phi}(\cdot)$ for a $D$-degree polynomial

\begin{align}
    \boldsymbol{\phi}(x) & \triangleq
    \begin{bmatrix}
        x^D     \\
        x^{D-1} \\
        \vdots  \\
        x^2     \\
        x       \\
        1
    \end{bmatrix}
\end{align}

For a vector $\mathbf{x} \in \mathbb{R}^N$ as input argument, it constructs and returns the matrix

\begin{align}
    \boldsymbol{\Phi} & \triangleq
    \begin{bmatrix}
        \boldsymbol{\phi}^T(x_1) \\
        \boldsymbol{\phi}^T(x_2) \\
        \vdots  \\
        \boldsymbol{\phi}^T(x_N) \\
    \end{bmatrix} =
    \begin{bmatrix}
        x_1^D & x_1^{D-1} & \cdots & x_1^2 & x_1 & 1 \\
        x_2^D & x_2^{D-1} & \cdots & x_2^2 & x_2 & 1 \\
        \vdots  \\
        x_N^D & x_N^{D-1} & \cdots & x_N^2 & x_N & 1
    \end{bmatrix} \in \mathbb{R}^{N \times (D+1)}
\end{align}

```python
def polynomial_features(x, degree):
    """
    Construct design matrix for polynomial regression of degree `polydegree`

    Inputs
    ------
    x: numpy.ndarray, ndims=1
        A 1-dimensional numpy array of length `N` contianing `x` values

    degree: int
        An integer specifying the maximum degree of polynomial to be constructed
        in the design matrix

    Returns
    -------
    Phi: np.ndarray, ndims=2
        A 2-dimensional numpy array of shape `(N, polydegree+1)` representing the
        design matrix. Columns are:
        `[x**polydegree, x**(polydegree-1) ... x**2, x**1, 1]`
    """
    N = x.shape[0]
    Phi = np.ones((N, degree + 1))
    for deg in range(1, degree+1):
        Phi[:, -deg-1] = Phi[:, -deg] * x

    return Phi
```

## Specification of true model

We want the true model to be the 3rd degree polynomial in $x$

\begin{align}
    & f(x|\mathbf{w}_{\mathrm{true}}) = \frac{A}{3} x^3 - \frac{A(a+b)}{2} x^2 + A a b x = \boldsymbol{\phi}^T(x)  \mathbf{w}_{\mathrm{true}}
\end{align}

where the polynomial coefficients $\mathbf{w}_{\mathrm{true}}$ are given as

\begin{align}
    \mathbf{w}_{\mathrm{true}} & \triangleq
    \begin{bmatrix}
        \frac{A}{3} \\
        -\frac{A (a+b)}{2} \\
        a b A \\
        0
    \end{bmatrix}
\end{align}

and the polynomial feature map $\boldsymbol{\phi}(\cdot)$ is given as

\begin{align}
    \boldsymbol{\phi}(x) & \triangleq
    \begin{bmatrix}
        x^3 \\
        x^2 \\
        x   \\
        1
    \end{bmatrix}
\end{align}

```python
## True model (polynomial) specification

# The parameters that follow influence the shape of the polynomial (feel free to change)
a = 1.0    # First stationary point of polynomial
b = 1.5    # Second stationary point of polynomial
A = 2.0    # Scaling factor

# Polynomial coefficients (do not change)
w_true = np.zeros(4)
w_true[0] = A / 3.0;
w_true[1] = - A * (a + b) / 2.0
w_true[2] = A * a * b
w_true[3] = 0.2

w_true
```

## True Data Generation Process

We will feed the polynomial regression models noisy data obtained from the model above

For any $x$ value $x_i$ we will compute the corresponding target $y_i$ as:

\begin{align*}
y_i &= \phi(x_i) \cdot w_i  + \epsilon_i, \quad \epsilon_i \stackrel{\text{iid}}{\sim} N(0, \sigma)
\end{align*}

The parameter $\sigma$ controls the strength of the noise

Below we sample oa grid of x on $[0, 3]$, compute $\phi(x)$, then add random noise to compute $y$

```python
np.random.seed(42)  # for reproducible resultss

N = 100
xmin, xmax = 0, 3
x = xmin + (xmax - xmin) * np.random.rand(N)

sigma = 0.05
epsilon = np.random.randn(100) * np.sqrt(sigma)

## Construct the target values
Phi = polynomial_features(x, 3)
y = Phi @ w_true + epsilon

# also construct data for plotting later
x_plot = np.linspace(xmin, xmax, 40)
y_true_plot = polynomial_features(x_plot, 3) @ w_true
```

## Data Partitioning

The available data will be partitioned into a *training* set and a *hold-out* set (i.e. samples not used for training/fitting). In particular, the hold-out set will be used to obtain an honest estimate of how good the model performs on previously unseen samples (i.e. samples other than ones used for training).

```python
## Partition data into training and holdout set

# Number of training samples (feel free to change)
N_train = 10

N_holdout = N - N_train

x_train = x[:N_train]
y_train = y[:N_train]

x_holdout = x[N_train:]
y_holdout = y[N_train:]
```

## Model Training/Fitting

```python
# Specify the degree of the polynomial that we plan to fit (feel free to change)
polydegree = 3   # must be integer >=0

# Construct design matrix for polynomial regression (with intercept term)
# for the training data
Phi_train = polynomial_features(x_train, polydegree)

# Fit/train the model by computing the optimal coefficient (weight) vector
Phi_train_dagger = np.linalg.pinv(Phi_train)
w_star = np.dot(Phi_train_dagger, y_train)

# Compute the (optimal) predicted outputs for the training inputs
y_hat_train = np.dot(Phi_train, w_star)

# Compute the (minimum) MSE on the training set
res_train = y_train - y_hat_train   # these are the optimal regression residuals
MSE_train = np.linalg.norm(res_train,2)**2 / N_train
```

## Hold-Out Set Predictions

```python
# Construct design matrix for polynomial regression (with intercept term)
# for the hold-out data
Phi_holdout = polynomial_features(x_holdout, polydegree)

# Compute the (optimal) predicted outputs for the hould-out inputs
# using the optimal coefficients obtained by fitting the model
y_hat_holdout = np.dot(Phi_holdout, w_star)

# Compute the (minimum) MSE on the hold-out set
res_holdout = y_holdout - y_hat_holdout
MSE_holdout = np.linalg.norm(res_holdout,2)**2 / N_holdout
```

## Reporting

```python
print('The optimal coefficients are:\n')
print(w_star)

print('\n')

print('The (minimum) MSE of the fitted model as computed on the training set is\nMSE_train={:.3f}\n'.format(MSE_train))
print('The MSE of the fitted model as computed on the hold-out set is\nMSE_holdout={:.3f}\n'.format(MSE_holdout))
print('The noise variance of the true model is\nsigmaSquared_e={:.3f}\n'.format(sigma))
```

## Graphing of Results

```python
## Create data for plotting the fitted polynomial curve
Phi_plot = polynomial_features(x_plot, polydegree)
y_hat_plot_fitted = np.dot(Phi_plot, w_star)

```

```python
# Resize figure to make it bigger
fig, ax = plt.subplots(figsize=(13, 8))

ax.plot(x_plot, y_hat_plot_fitted, 'r', lw=3, label="Fitted model")
ax.plot(x_plot, y_true_plot, 'g', lw=3, label="True model")
ax.scatter(x_holdout, y_holdout, s=100, facecolors='none', edgecolors='b', label="Hold-Out data")
ax.scatter(x_train, y_train, s=100, facecolors='r', edgecolors='k', label="train data")

# Annotate graph
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend(loc='lower right')
ax.set_title((
    "Fitted vs. True Model Responses: "
    f"Ntrain={N_train:d} deg={polydegree:d} MSEtrain={MSE_train:.3f}, "
    f"MSEholdout={MSE_holdout:.3f}, NoiseVar={sigma:.3f}"
))
```

## Using sklearn

Below we will show an example of how to use scikit-learn to:

1. construct the polynomial feature matrix
2. Fit the polynomial regression
3. Compute the MSE on training and holdout sets

We will wrap this process up in a function that takes as an input the degree of the polynomial as well as the number of points to use for training

This function will then be used to construct an interactive GUI where we will get sliders for the degree and number of training points, and as we adjust the sliders a plot of the model fit updates automatically

Feel free to experiment with the two sliders and try to gain some intutions for when overfitting will and won't occur

```python
from sklearn import preprocessing, linear_model, pipeline, metrics, model_selection
from ipywidgets import widgets
```

```python
def polyreg_demo(degree=8, n_train=10):
    # define model
    model = pipeline.make_pipeline(
        preprocessing.PolynomialFeatures(degree=degree),
        linear_model.LinearRegression()
    )

    X = x[:, None]  # convert to 2d
    test_size = X.shape[0] - n_train
    split = model_selection.train_test_split(X, y, train_size=n_train, test_size=test_size, random_state=12)
    X_train, X_test, y_train, y_test = split

    # fit model
    model.fit(X_train, y_train)
    yhat = model.predict(x_plot[:, None])

    # compute metrics
    MSE_train = metrics.mean_squared_error(y_train, model.predict(X_train))
    MSE_holdout = metrics.mean_squared_error(y_test, model.predict(X_test))

    # make the plot
    fig, ax = plt.subplots(figsize=(11, 8))
    ax.plot(x_plot, yhat, "k-", lw=3, label="Fitted Model")
    ax.scatter(x_holdout, y_holdout, color="b", s=60, alpha=0.5, label="Hold-Out Data")
    ax.scatter(X_train.flatten(), y_train, color="r", s=80, alpha=0.7, label="Training Data")
    ax.set_ylim((-20, 20))
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.legend(loc='upper left')
    ax.set_title((
        "Fitted vs. True Model Responses: "
        f"Ntrain={N_train:d} deg={degree:d} MSEtrain={MSE_train:.3f}, "
        f"MSEholdout={MSE_holdout:.3f}, NoiseVar={sigma:.3f}"
    ))

    return fig

widgets.interactive(polyreg_demo, degree=(1, 10, 1), n_train=(5, 95, 1))
```

```python
fig = polyreg_demo(0, 10)
fig.set_size_inches((11, 5))
ax = fig.axes[0]
ax.set_ylim((-2, 5))
fig.savefig("underfit.pdf")
fig.savefig("../overfitting_model_selection/underfitting.pdf")
```

## Using Keras (Optional)

We have had a few questions about how Linear Regression relates to a neural network in keras

In this section we will show how you can do Linear Regression using keras

Note that this is like using a blowtorch when a match would do, but it will illustrate the following:

- How you can view a LinearRegression as a `Dense` neural network layer

- This will be the first example of an _iterative_ approach to finding coefficients for a linear regression. As mentioned in the lectures this the most numerically stable approach and often the only computationaly reasonable approach as either the number of samples or the number of features gets very large

### Theory

Recall that a multi-layer-perceptron (MLP) is specific kind of neural network that is constructed by stacking one or more `Dense` layers, separated by non-linear activation functions

An example of a 2 hidden layer MLP in keras is:

```python
from tensorflow import keras

model = keras.Sequential()
model.add(keras.layers.Dense(10), activation="relu")  # hidden layer 1
model.add(keras.layers.Dense(5), activation="relu")   # hidden layer 2
model.add(keras.layers.Dense(1), activation=None)     # output layer -- assuming we have 1 target to predict
```

Each dense layer in the network has the following structure:

$$\text{output} = \mathbf{input} \mathbf{W} + \mathbf{b}$$

where $\mathbf{input}$ is the input to the layer, $\mathbf{W}$ is a matrix of weights or coefficients and $\mathbf{b}$ is a vector of intercepts

The number of columns in the matrix $\mathbf{W}$ and the number of elements in the vector $\mathbf{b}$ are the number of neurons in the hidden layer

In our example, the $\mathbf{W}$ for the first hidden layer would have shape $n_{\text{features}} \times 10$

The output of this first hidden layer would he $n_{\text{samples}} \times 10$

The $\mathbf{W}$ for the second hidden layer would have shape $10 \times 5$ -- there are $5$ rows because that is how many neurons there are in the hidden layer

The $\mathbf{b}$ fot the second layer would have $5$ elements and the output would be of shape $n_{\text{samples}} \times 5$, and so on until the last layer...

Given an input sample $\mathbf{x}$, we can compute the prediction of the network as follows:

$$\begin{align*}
\text{output}_1 &= \text{relu}(\mathbf{x} \mathbf{W}_1 + \mathbf{b}_1) \\
\text{output}_2 &= \text{relu}(\text{output}_1 \mathbf{W}_2 + \mathbf{b}_2) \\
\hat{y} &= \text{output}_2 \mathbf{W}_{\text{final}} + \mathbf{b}_{\text{final}} \\
        &= (\text{relu}(\text{relu}(\mathbf{x} \mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2) \mathbf{W}_{\text{out}} + \mathbf{b}_{\text{out}}
\end{align*}$$

### Special Case: Linear Regression

We can consider a special case where there are no hidden layers and no activation function

The keras model will be written as:

```python
model = keras.Sequential()
model.add(keras.layers.Dense(1), activation=None)   # output layer -- assuming we have 1 target to predict
```

and the prediction for an input sample $\mathbf{x}$ is

$$\hat{y} = \mathbf{x}\mathbf{W} + \mathbf{b},$$

which is exactly our linear regression equation!

Let's test this out with our polynomial regression

```python
from tensorflow import keras

model = keras.Sequential()
model.add(keras.layers.Dense(1, use_bias=False, activation=None))   # output layer -- assuming we have 1 target to predict

model.compile(optimizer="rmsprop", loss="mse")  # note we use mse as loss!
```

Now let's fit a degree 3 polynomial regression for 10 epochs

```python
X3 = polynomial_features(x_train, 3)
X3_holdout = polynomial_features(x_holdout, 3)
```

```python
model.fit(X3, y_train, epochs=10, validation_data=(X3_holdout, y_holdout));
```

Notice that keras is reporting the `loss` after each pass through the training data

This is the MSE with the current parameters

It is changing because the parameters are being updated (stay tuned for later in the course and you'll learn _how_ theose parameters are being updated)

Also, notice that the MSE after 10 epocs is much larger than the optimal MSE of 0.015 we found using the closed form optimal solution from linear regression

Let's train the model for many more epochs and see if we can close that gap

```python
history = model.fit(X3, y_train, epochs=4000, verbose=False, initial_epoch=11, validation_data=(X3_holdout, y_holdout))
```

```python
fig, ax = plt.subplots(figsize=(8, 6))
x = np.arange(len(history.history["loss"]))
ax.plot(x, history.history["loss"], label="train loss")
ax.plot(x, history.history["val_loss"], label="Validation loss")
ax.hlines(MSE_train, xmin=x[0], xmax=x[-1], alpha=0.5, lw=4, color="red")
ax.set_ylabel("MSE")
ax.set_xlabel("Epoch")
ax.legend()

msg = "The final MSE_train and MSE_validation are {} and {}"
print(msg.format(history.history["loss"][-1], history.history["val_loss"][-1]))
```

Now the training MSE is much closer to the optimum than before


### Задание 1

Напишите уравнение параболы, проходящей через три точки (x,y):
(1,2), (3,10), (5,1)

\begin{equation}
y = ax^2 + bx + c
\end{equation}

\begin{equation}
\left\{ \begin{array}{ll}
 2 = a \cdot 1^2 + b \cdot 1 + c \\
 10 = a \cdot 3^2 + b \cdot 3 + c \\
 1 = a \cdot 5^2 + b \cdot 5 + c
  \end{array} \right.
\end{equation}

```python
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
A = np.matrix([[1, 1, 1], [9, 3, 1], [25, 5, 1]], dtype=float)
A
```

    matrix([[ 1.,  1.,  1.],
            [ 9.,  3.,  1.],
            [25.,  5.,  1.]])

```python
B = np.matrix([[2], [10], [1]])
B
```

    matrix([[ 2],
            [10],
            [ 1]])

\begin{equation}
X = A^{-1} B
\end{equation}

```python
X = np.dot(np.linalg.inv(A), B)
X
```

    matrix([[-2.125],
            [12.5  ],
            [-8.375]])

\begin{equation}
y = -2.125x^2 + 12.5x - 8.375
\end{equation}

```python
x = np.linspace(-10, 10, 1000)
```

```python
x1 = [1, 3, 5]
y1 = [2, 10, 1]
```

```python
plt.grid(color='g', linestyle='-', linewidth=1)
plt.xlim((0,6))
plt.ylim((-1,11))
plt.plot(x, (-2.125)*(x**2) + 12.5*x - 8.375)
plt.scatter(x1, y1)
plt.show()
```

### Задание 2

Известно, что свежий огурец на 99% состоит из воды. Месяц назад взвесили мешок со свежими огурцами. Получилось, что огурцов ровно 100 кг. Мешок убрали, а через месяц снова взвесили. Огурцы за это время усохли, и теперь вода составляет уже только 98% их веса. Сколько теперь (в кг) весят огурцы?

100 кг = 99 кг воды + 1 кг сухой массы

через месяц \
x кг воды = 98 / 2 = 49

Ответ: 49 + 1 = 50 кг

### Задание 3. Определение логарифма. Решить уравнения:

1
\begin{equation}
2^x = 256, x = \log_{2} 256 = 8
\end{equation}

2
\begin{equation}
2^x = 300, x = 8.23
\end{equation}

3
\begin{equation}
\log_{8} 2^{8x-4} = 4
\end{equation}

\begin{equation}
(8x-4) \log_8 2 = 4
\end{equation}

\begin{equation}
x = \frac{1}{2 \log_8 2} + \frac{1}{2} = 2
\end{equation}

4
\begin{equation}
3^{\log_9 5x-5} = 5
\end{equation}

\begin{equation}
\log_9 {3^{\log_9 5x-5}} = \log_9 5
\end{equation}

\begin{equation}
\log_9 (5x-5) \cdot \log_9 3 = \log_9 5
\end{equation}

\begin{equation}
\log_9 (5x-5) = \frac{\log_9 5}{\log_9 3}
\end{equation}

\begin{equation}
\frac{\log_3 (5x-5)}{2} = \log_3 5
\end{equation}

\begin{equation}
5x - 5 = 10, x = 3
\end{equation}

5
\begin{equation}
x^{\log_3 x+1} = 9
\end{equation}

\begin{equation}
\log_3 x^{\log_3 x+1} = \log_3 9
\end{equation}

\begin{equation}
(\log_3 x+1) \log_3 x = \log_3 9
\end{equation}

\begin{equation}
t = \log_3 x
\end{equation}

\begin{equation}
t^2 + t - 2 = 0
\end{equation}

\begin{equation}
t_{1,2} = [1, -2]
\end{equation}

\begin{equation}
\log_3 x = 1, \log_3 x = -2
\end{equation}

\begin{equation}
x_1 = 3, x_2 = \frac{1}{9}
\end{equation}

### Задание 4. Свойства логарифмов. Вычислить:

6
\begin{equation}
\log_4 16 = 2
\end{equation}

7
\begin{equation}
\log_5 \frac{1}{25} = -2
\end{equation}

8
\begin{equation}
\log_{25} 5 = 0.5
\end{equation}

9
\begin{equation}
\log_3 \sqrt{27} = \frac{3}{2}
\end{equation}

10
\begin{equation}
\log_2 12 - \log_2 3 = \log_2 \frac{12}{3} = 2
\end{equation}

11
\begin{equation}
\log_6 12 + \log_6 3 = \log_6 36 = 2
\end{equation}

12
\begin{equation}
e^{\ln 5} = e^{\log_e 5} = 5
\end{equation}

13
\begin{equation}
\frac{\log_2 225}{\log_2 15} = \log_{15} 225 = 2
\end{equation}

14
\begin{equation}
9^{\log_3 \sqrt{5}} = 3^{2\frac{1}{2}\log_3 5} = 5
\end{equation}


```python
import numpy as np
import sympy
import matplotlib.pyplot as plt
from scipy import integrate
from scipy.special import legendre
from numpy.polynomial.legendre import Legendre

# Legendre polynomial
def leg(n, x):
    return Legendre(np.concatenate((np.zeros(n), np.array([1]))))(x)

def hbasis(i,x):
#  Evaluates the function Ni(y) at x
#  In function L(y) i-2 is the degree of the legendre polynomial
    if i==0:
        Ni=0.5 *(1-x)
    elif i==1:
        Ni=0.5 *(1+x)
    else:
        Ni=(np.sqrt(1/(4*(i+1)-6)))*(leg(i,x)-leg(i-2,x))
    return Ni

def stifness_matrix(p):
#evaluates the elemental stifness matrix of size (p+1)x(p+1)
    K=np.zeros((p+1,p+1))
    K[0,0]=K[1,1]=0.5
    K[0,1]=K[1,0]=-0.5
    if p>=1:
        for i in range(2,p+1):
            K[i,i]=1
    return K

def mass_matrix(p):
# Evaluates the elemental mass matrix of size (p+1)x(p+1)
    G=np.zeros((p+1,p+1))
    G[0,0]=G[1,1]=2/3
    G[0,1]=G[1,0]=1/3
    if p>=2:
        G[0,2]=G[1,2]=G[2,0]=G[2,1]=-1/np.sqrt(6)
        for i in range(2,p+1):
            G[i,i]=2/((2*(i+1)-1)*((2*(i+1)-5)))
    if p>=3:
        G[0,3]=G[3,0]=1/3*np.sqrt(10)
        G[1,3]=G[3,1]=-1/3*np.sqrt(10)
        for i in range(2,p+1):
            if i+2<p+1:
                G[i,i+2]=G[i+2,i]=(-1)/(((2*(i+1)-1)*np.sqrt(((2*(i+1)-3)*((2*(i+1)+1))))))
    return G

def load_vector(p,f,l,c):
#Evaluates the element load vector
    vals=[]
    for i in range(p+1):
        g = lambda x : f(x,l,c)*hbasis(i,x)
        vals.append(integrate.quad(g,-1,1)[0])
    b=np.array(vals)
    return b

def fem_solution(p,f,x,l,c):
# Evaluates the finite element method solution and the coifficients that generate the solution
    K=stifness_matrix(p)[2:,2:]
    G=mass_matrix(p)[2:,2:]
    A=K+c*G
    b=load_vector(p,f,l,c)[2:]
    a=np.linalg.solve(A,b)
    ufe_array = np.zeros((p-1, x.shape[0]))
    for i in range(2,p+1):
        for j,val in enumerate(x):
            ufe_array[i-2,j] = a[i-2]*hbasis(i,val)
    ufe_values = np.array([np.sum(ufe_array[:,j]) for j in range(x.shape[0])])
    return a, ufe_values

def energy_norm(p,f,x,l,c):
# Evaluates the energy norm of the finite element method solution and it finds
# the percentage of the error
    a=fem_solution(p,f,x,l,c)[0]
    enorm = np.dot(a, np.array(load_vector(p,f,l,c)[2:]))
    y = lambda x : ((1-x)*(1+x)**l)*f(x,l,c)
    enorm_uex=integrate.quad(y,-1,1)[0]
    err=np.sqrt(abs(enorm-enorm_uex)/abs(enorm_uex))
    return enorm, err

def main():
    f = lambda x,l,c : (l*(x+1)**(l-2))*((1+l)*x-l+3)+c*((x+1)**(l)*(1-x))
    l_list=[8.7,4.4,2.9,1.2]
    fig, axes = plt.subplots(1,4, figsize=(15,7))
    fig.tight_layout()
    for i,l in enumerate(l_list):
        c=1
        p_list=[2,3,4,5,6,7,8]
        errors=[]
        x = np.linspace(-1,1,1001)
        for p in p_list:
            errors.append(energy_norm(p,f,x,l,c)[1])
        slope=((np.log(errors[5])-np.log(errors[6]))/(np.log(p_list[5])-np.log(p_list[6])))
        print(f"The slope in graph {i+1} is {slope}")
        axes[i].loglog(np.array(p_list)-1,errors)
        axes[i].set_xlabel("log(DOF)")
        axes[i].set_ylabel("log(E)")
        axes[i].set_title(f"graph{i+1}")

    l=1.1
    c=2
    p=4
    x = np.linspace(-1,1,1001)
    u_Ex_fun = lambda x : (1-x)*((1+x)**l)
    u_Fe = fem_solution(p,f,x,l,c)[1]
    u_Ex_values = [u_Ex_fun(val) for val in x]
    fig , axes =plt.subplots(figsize=(10,10))
    axes.plot(x,u_Fe, color="red")
    axes.plot(x,u_Ex_values, color="blue")
    axes.set_ylim(0,2)
    axes.set_xlabel("x")
    axes.set_ylabel("y")
    axes.set_title(r'Plots of $u_{FE}$ and $u_{EX}$')
    axes.legend([r'$u_{FE}$','$u_{EX}$'])
    fig , axes =plt.subplots(figsize=(10,10))
    axes.plot(x,abs(u_Ex_values-u_Fe),color='black')
    axes.set_xlabel("x")
    axes.set_ylabel("y")
    axes.set_title(r'Error $|u_{FE}-u_{EX}|$')

```

```python
if __name__ == "__main__":
    main()
```

```python

```


```python
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
import numpy as np
from scipy import stats
from scipy.integrate import quad
import scipy.linalg as la
import seaborn as sns
from functools import partial
```

# Abbreviated lecture notes

```python
np.random.seed(123)
```

## Monte Carlo integration

We want to find some integral

$$I = \int{f(x)} \, dx$$

Consider the expectation of a function $g(x)$ with respect to some distribution $p(x)$. By definition, we have

$$
E[g(x)] = \int{g(x) \, p(x) \, dx}
$$

If we choose $g(x) = f(x)/p(x)$, then we have

$$
\begin{align}
E[g(x)] &= \int{\frac{f(x}{p(x)} \, p(x) \, dx} \\
&= \int{f(x) dx} \\
&= I
\end{align}
$$

By the law of large numbers, the average converges on the expectation, so we have

$$
I \approx \bar{g_n} = \frac{1}{n} \sum_{i=1}^n g(x_i)
$$

If $f(x)$ is a proper integral (i.e. bounded), and $p(x)$ is the uniform distribution, then $g(x) = f(x)$ and this is known as ordinary Monte Carlo. If the integral of $f(x)$ is improper, then we need to use another distribution with the same support as $f(x)$.

### Steps

Draw $n$ random samples $x_i$ from some probability distribution $p(x)$ with thee same support as the function $f(x)$. Then calculate

$$
I \approx \frac{1}{n}\sum_{i=1}^n \frac{f(x_i)}{p(x_i)}
$$

### Example 1

```python
x = np.linspace(-3,3,100)
dist = stats.norm(0,1)
a = -2
b = 0
plt.plot(x, dist.pdf(x))
plt.fill_between(np.linspace(a,b,100), dist.pdf(np.linspace(a,b,100)), alpha=0.5)
plt.text(b+0.1, 0.1, 'p=%.4f' % (dist.cdf(b) - dist.cdf(a)), fontsize=14)
pass
```

#### Monte Carlo integral

If we cannot sample directly from the target distribution $N(0,1)$ but can evaluate it at any point.

Since $p(x)$ is $U(a, b)$, $p(x) = \frac{1}{b-a}$. So we want to calculate

$$
\frac{1}{n} \sum_{i=1}^n (b-a) f(x)
$$

```python
n = 10000
x = np.random.uniform(a, b, n)
np.mean((b-a)*dist.pdf(x))
```

    0.47631158517865363

### Example 2

```python
def f(x):
    return x * np.cos(71*x) + np.sin(13*x)
```

```python
x = np.linspace(0, 1, 100)
plt.plot(x, f(x))
pass
```

#### Monte Carlo integral

```python
n = int(1e6)
x = f(np.random.random(n))
y = 1.0/n * np.sum(x)
y
```

    0.020342363216455394

Check

```python
y, err = quad(f, 0, 1.0)
y
```

    0.02025493910239419

### How many Monte Carlo samples do we need?

We are often interested in knowing how many iterations it takes for Monte Carlo integration to "converge". To do this, we would like some estimate of the variance, and it is useful to inspect such plots. One simple way to get confidence intervals for the plot of Monte Carlo estimate against number of iterations is simply to do many such simulations. This might be expensive, so we show instead how to estimate convergence using bootstrap.

We find the Monte Carlo samples once for 100 steps

```python
n = 100
x = f(np.random.uniform(a, b, n))
```

```python
x.shape
```

    (100,)

```python
x[:5]
```

    array([-0.92731533, -0.54132811, -0.85450052, -0.57789469, -0.6274338 ])

A single MC integral

```python
y = 1.0/n * np.sum(x)
y
```

    -0.06417233839386707

Use sampling without replacement to make copies

```python
reps = 1000
xb = np.random.choice(x, (n, reps), replace=True)
```

```python
xb.shape
```

    (100, 1000)

Each column is a bootstrap replicate of the Monte Carlo samples

```python
xb[:5, 0]
```

    array([ 1.47414827,  1.76137976, -0.85450052,  1.87386139, -0.02133597])

```python
xb[:5, 1]
```

    array([-1.01541359, -0.90344323,  0.68736292,  0.14541681,  0.98707994])

```python
xb[:5, 2]
```

    array([-0.54132811, -0.81092802, -1.01712687, -1.07677473,  0.98707994])

Calculate the Monte Carlo estimate among the replicates for Monte Carlo sequences of length 1, 2, ..., n

```python
yb = 1/np.arange(1, n+1)[:, None] * np.cumsum(xb, axis=0)
```

```python
yb.shape
```

    (100, 1000)

Calculate the ($\alpha/2$, 1-$\alpha/2$) intervals for plotting the CI envelope at each step 1, 2, ..., n

```python
upper, lower = np.percentile(yb, [2.5, 97.5], axis=1)
```

```python
plt.plot(np.arange(1, n+1)[:, None], yb, c='grey', alpha=0.02)
plt.plot(np.arange(1, n+1), yb[:, 0], c='red', linewidth=1)
plt.plot(np.arange(1, n+1), upper, 'b', np.arange(1, n+1), lower, 'b')
pass
```

## Markov chain Monte Carlo (MCMC)

In Bayesian statistics, we want to estimate the posterior distribution, but this is often intractable due to the high-dimensional integral in the denominator (marginal likelihood). A few other ideas we have encountered that are also relevant here are Monte Carlo integration with independent samples and the use of proposal distributions (e.g. rejection and importance sampling). As we have seen from the Monte Carlo integration lectures, we can approximate the posterior $p(\theta | X)$ if we can somehow draw many samples that come from the posterior distribution. With vanilla Monte Carlo integration, the samples are independent draws from the posterior distribution.

With MCMC, we draw samples from a (simple) proposal distribution so that each draw depends only on the state of the previous draw (i.e. the samples form a Markov chain). Under certain conditions, the Markov chain will have a unique stationary distribution. In addition, not all samples are used - instead we set up acceptance criteria for each draw based on comparing successive states with respect to a target distribution that ensure that the stationary distribution is the posterior distribution of interest. The nice thing is that this target distribution only needs to be proportional to the posterior distribution, which means we don't need to evaluate the potentially intractable marginal likelihood, which is just a normalizing constant. We can find such a target distribution easily, since `posterior` $\propto$ `likelihood` $\times$ `prior`. After some time, the Markov chain of accepted draws will converge to the stationary distribution, and we can use those samples as (correlated) draws from the posterior distribution, and find functions of the posterior distribution in the same way as for vanilla Monte Carlo integration.

All MCMC sampling schemes have the following structure:

Start with some state $x_k$

Repeat

1. Draw a sample $x_{k+1}$ from a proposal distribution $p(x_k)$
2. Calculate the acceptance probability $A(x)$ - in general, this is a function of the proposal and target distributions at $x_k$ and $x_{k+1}$
3. Choose a standard random uniform number $r$
4. If $r < A(x)$ set state to be $x_{k+1}$, otherwise keep state as $x_{k}$

### Metropolis-Hastings

To carry out the Metropolis-Hastings algorithm, we need to draw random samples from the following distributions

- the standard uniform distribution
- a proposal distribution $p(x)$
- the target distribution $g(x)$ which is proportional to the posterior probability

Given an initial guess for $\theta$ with positive probability of being drawn, the Metropolis-Hastings algorithm proceeds as follows

- Choose a new proposed value ($\theta_p$) such that  $\theta_p = \theta + \Delta\theta$ where $\Delta \theta \sim p$
- If $p$ is a symmetrical distribution, calculate the ratio

$$
\rho = \frac{g(\theta_p \ | \ X)}{g(\theta \ | \ X)}
$$

where $g$ is the posterior probability.

- If the proposal distribution is not symmetrical, we need to weight the acceptance probability to maintain detailed balance (reversibility) of the stationary distribution, and instead calculate

$$
\rho = \frac{g(\theta_p \ | \ X) p(\theta \ | \ \theta_p)}{g(\theta \ | \ X) p(\theta_p \ | \ \theta)}
$$

Since we are taking ratios, the denominator cancels any distribution proportional to $g$ will also work - so we can use

$$
\rho = \frac{g(X | \theta_p ) p(\theta_p)}{g(X | \theta ) p(\theta)}
$$

- If $\rho \ge 1$, then set $\theta = \theta_p$
- If $\rho \lt 1$, then set $\theta = \theta_p$ with probability $\rho$, otherwise set $\theta = \theta$ (this is where we use the standard uniform distribution)
- Repeat the earlier steps

After some number of iterations $k$, the samples $\theta_{k+1}, \theta_{k+2}, \dots$ will be samples from the posterior distributions. Here are initial concepts to help your intuition about why this is so:

- We accept a proposed move to $\theta_{k+1}$ whenever the density of the (unnormalized) target distribution at $\theta_{k+1}$ is larger than the value of $\theta_k$ - so $\theta$ will more often be found in places where the target distribution is denser
- If this was all we accepted, $\theta$ would get stuck at a local mode of the target distribution, so we also accept occasional moves to lower density regions - it turns out that the correct probability of doing so is given by the ratio $\rho$
- The acceptance criteria only looks at ratios of the target distribution, so the denominator cancels out and does not matter - that is why we only need samples from a distribution proportional to the posterior distribution
- So, $\theta$ will be expected to bounce around in such a way that its spends its time in places proportional to the density of the posterior distribution - that is, $\theta$ is a draw from the posterior distribution.

Additional notes:

Different proposal distributions can be used for Metropolis-Hastings:

- The independence sampler uses a proposal distribution that is independent of the current value of $\theta$. In this case the proposal distribution needs to be similar to the posterior distribution for efficiency, while ensuring that the acceptance ratio is bounded in the tail region of the posterior.
- The random walk sampler (used in this example) takes a random step centered at the current value of $\theta$ - efficiency is a trade-off between small step size with high probability of acceptance and large step sizes with low probability of acceptance. Note (picture will be sketched in class) that the random walk may take a long time to traverse narrow regions of the probability distribution. Changing the step size (e.g. scaling $\Sigma$ for a multivariate normal proposal distribution) so that a target proportion of proposals are accepted is known as *tuning*.
- Much research is being conducted on different proposal distributions for efficient sampling of the posterior distribution.

### Example 1

Using MCMC to generate standard normal samples from uniform samples.

- The proposal density $p(x)$ is uniform on an interval centered at current state (hence symmetrical)
- The target density $q(x)$ is $N(0,1)$ which is proportional to $e^{-x^2/2}$
- The acceptance function $A(x)$ is
$$
\frac{e^{-x_{k+1}^2/2}}{e^{-x_{k}^2/2}} = e^{-(x_{k+1}^2-x_{k}^2)/2}
$$

```python
n = 10000
xs = np.zeros(n)
x = 0
for i in range(n):
    p = np.random.uniform(x-1, x+1)
    a = np.exp(-(p**2-x**2)/2)
    r = np.random.rand()
    if r < a:
        x = p
    xs[i] = x
```

```python
plt.hist(xs, 25, histtype='step', normed=True)
xp = np.linspace(xs.min(), xs.max(), 100)
plt.plot(xp, stats.norm().pdf(xp))
pass
```

### Gibbs sampler

Suppose we have a vector of parameters $\theta = (\theta_1, \theta_2, \dots, \theta_k)$, and we want to estimate the joint posterior distribution $p(\theta | X)$. Suppose we can find and draw random samples from all the conditional distributions

$$
p(\theta_1 | \theta_2, \dots \theta_k, X) \\
p(\theta_2 | \theta_1, \dots \theta_k, X) \\
\dots \\
p(\theta_k | \theta_1, \theta_2, \dots, X)
$$

With Gibbs sampling, the Markov chain is constructed by sampling from the conditional distribution for each parameter $\theta_i$ in turn, treating all other parameters as observed. When we have finished iterating over all parameters, we are said to have completed one cycle of the Gibbs sampler. Since hierarchical models are typically set up as products of conditional distributions, the Gibbs sampler is ubiquitous in Bayesian modeling. Where it is difficult to sample from a conditional distribution, we can sample using a Metropolis-Hastings algorithm instead - this is known as Metropolis within Gibbs.

Gibbs sampling is a type of random walk through parameter space, and hence can be thought of as a Metropolis-Hastings algorithm with a special proposal distribution. At each iteration in the cycle, we are drawing a proposal for a new value of a particular parameter, where the proposal distribution *is* the conditional posterior probability of that parameter. This means that the proposal move is *always* accepted. Hence, if we can draw samples from the conditional distributions, Gibbs sampling can be much more efficient than regular Metropolis-Hastings.
More formally, we want to show that

$$
\frac{P(y) \, q(y \to x)}{P(x) \, q(x \to y)} = 1
$$

We start by noting that $P(x_{-i})$ is the same as $P(y_{-i})$ since apart from the component $i$, the old state and the proposed new state are identical in Gibbs sampling. We also recall that

$$P(x_i \mid x_{-i}) \, P(x_{-i}) = P(x_i, x_{-i}) = P(x)$$

by definition of conditional probability. So we have

$$
\begin{align}
\frac{P(y) \, q(y \to x)}{P(x) \, q(x \to y)} &= \frac{P(y_i \mid y_{-1}) \, P(y_{-i})\, P(x_i \mid x_{-i}) }{P(x_i \mid x_{-i}) \, P(x_{-i})\, P(y_i \mid y_{-1})} &= 1
\end{align}
$$

**Advantages of Gibbs sampling**

- No need to tune proposal distribution
- Proposals are always accepted

**Disadvantages of Gibbs sampling**

- Need to be able to derive conditional probability distributions
- Need to be able to (cheaply) draw random samples from conditional probability distributions
- Can be very slow if parameters are correlated because you cannot take "diagonal" steps (draw picture to illustrate)

#### Example: Bivariate normal distribution

Suppose we want to draw samples from a bivariate normal distribution $(X, Y)$ with correlation $\rho$

$$
\mu =
\begin{bmatrix}
0 \\ 0
\end{bmatrix}
$$

$$
\Sigma =
\begin{bmatrix}
1 & \rho \\
\rho & 1
\end{bmatrix}
$$

Then the conditional distributions are given by

$$
p(X \vert y) = N(\rho y, 1-\rho^2)
$$

and

$$
p(Y \vert x) = N(\rho x, 1-\rho^2)
$$

and the Gibbs sampler is

```
Start with initial guess for y
Repeat
    x = N(rho*y, 1-rho**2)
    y = N(rho*x, 1-rho**2)
```

```python
rho = -0.8
niter = 1000
xs = np.zeros((niter, 2))
y = 0
for i in range(niter):
    x = np.random.normal(rho*y, 1-rho**2)
    y = np.random.normal(rho*x, 1-rho**2)
    xs[i,:] = x,y
```

```python
dist = stats.multivariate_normal([0,0], np.array([[1,rho],[rho,1]]))

x = np.linspace(-3.0, 3.0, 100)
y = np.linspace(-3.0, 2.0, 100)
X, Y = np.meshgrid(x, y)
Z = dist.pdf(np.c_[X.ravel(), Y.ravel()]).reshape(X.shape)
```

```python
plt.contour(X, Y, Z)
plt.scatter(xs[:, 0], xs[:, 1], s=2)
plt.axis('square')
pass
```

### Hamiltonian Monte Carlo (HMC)

HMC uses an auxiliary variable corresponding to the momentum of particles in a potential energy well to generate proposal distributions that can make use of gradient information in the posterior distribution. For reversibility to be maintained, the total energy of the particle has to be conserved - hence we are interested in Hamiltonian systems. The main attraction of HMC is that it works much better than other methods when variables of interest are highly correlated. Because we have to solve problems involving momentum, we need to understand how to numerically solve differential equations in a way that is both accurate (i.e. second order) and preserves total energy (necessary for a Hamiltonian system).

Example adapted from [MCMC: Hamiltonian Monte Carlo (a.k.a. Hybrid Monte Carlo)](https://theclevermachine.wordpress.com/2012/11/18/mcmc-hamiltonian-monte-carlo-a-k-a-hybrid-monte-carlo/)

#### Hamiltonian systems

In a Hamiltonian system, we consider particles with position $x$ and momentum (or velocity if we assume unit mass) $v$. The total energy of the system $H(x, v) = K(v) + U(x)$, where $K$ is the kinetic energy and $U$ is the potential energy, is conserved. Such a system satisfies the following Hamiltonian equations

$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta H}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta H}{dx}
\end{align}
$$

Since $K$ depends only on $v$ and $U$ depends only on $x$, we have
$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta K}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta U}{dx}
\end{align}
$$

#### Harmonic oscillator

We will consider solving a classical Hamiltonian system - that of a undamped spring governed by the second order differential equation

$$
x'' + x = 0
$$

We convert this to two first order ODEs by using a dummy variable $x' = v$ to get

$$
\begin{align}
x' &= v \\
v' &= -x
\end{align}
$$

From the Hamiltonian equations above, this is equivalent to a system with kinetic energy $K(v) = \frac{1}{2}v^2$ and potential energy $U(x) = \frac{1}{2}x^2$.

Writing in matrix form,

$$
A = \pmatrix{ x' \\ v' } = \pmatrix{0 & 1 \\ -1 & 0} \pmatrix{x \\ v}
$$

and in general, for the state vector $x$,

$$
x' = Ax
$$

We note that $A$ is anti- or skew-symmetric ($A^T = -A$), and hence has purely imaginary eigenvalues. Solving $|A - \lambda I = 0$, we see that the eigenvalues and eigenvectors are $i, \pmatrix{1\\i}$ and $-i, \pmatrix{1\\-i}$. Since the eigenvalues are pure imaginary, we see that the solution for the initial conditions $(x,v) = (1, 0)$ is $x(t) = e^{it}$ and the orbit just goes around a circle with a period of $2\pi$, neither growing nor decaying. Another weay of seeing this is that the Hamiltonian $H(u, v)$ or sum of potential ($U(x)) = \frac{1}{2}x^2$) and kinetic energy ($K(v) = \frac{1}{2}v^2$) is constant, i.e. in vector form, $(x^T x) = \text{constant}$.

#### Finite difference methods

We want to find a finite difference approximation to $u' = Au$ that is **accurate** and **preserves total energy**. If total energy is not preserved, the orbit will either spiral in towards zero or outwards away from the unit circle. If the accuracy is poor, the orbit will not be close to its starting value after $t = 2\pi$. This gives us an easy way to visualize how good our numerical scheme is. We can also compare the numerical scheme to the Taylor series to evaluate its accuracy.

##### The leapfrog method

The leapfrog method uses a second order difference to update $u_n$. The algorithm simplifies to the following explicit scheme:

- First take one half-step for v
- Then take a full step for u
- Then take one final half step for v

It performs as well as the trapezoidal method, with the advantage of being an explicit scheme and cheaper to calculate, so the leapfrog method is used in HMC.

```python
def leapfrog(A, u, N):
    orbit = np.zeros((N,2))

    dt = 2*np.pi/N
    for i in range(N):
        u[1] = u[1] + dt/2 * A[1] @ u
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt/2 * A[1] @ u
        orbit[i] = u
    return orbit
```

If we don't care about the intermediate steps, it is more efficient to just take 1/2 steps at the beginning and end

```python
def leapfrog2(A, u, N):
    dt = 2*np.pi/N

    u[1] = u[1] + dt/2 * A[1] @ u
    for i in range(N-1):
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt * A[1] @ u

    u[0] = u[0] + dt * A[0] @ u
    u[1] = u[1] + dt/2 * A[1] @ u
    return u
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
```

##### Orbit

```python
orbit = leapfrog(A, u, N)
```

```python
plt.plot([p @ p for p in orbit])
pass
```

##### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.0025229913808033464

##### Conservation of energy

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

#### From Hamiltonians to probability distributions

The physical analogy considers the negative log likelihood of the target distribution $p(x)$ to correspond to a potential energy well, with a collection of particles moving on the surface of the well. The state of each particle is given only by its position and momentum (or velocity if we assume unit mass for each particle). In a Hamiltonian system, the total energy $H(x,, v) = U(x) + K(v)$ is conserved. From statistical mechanics, the probability of each state is related to the total energy of the system

$$
\begin{align}
p(x, v) & \propto e^{-H(x, v)} \\
&= e^{-U(x) - K(v)} \\
&= e^{-P(x)}e^{-K(v)} \\
& \propto p(x) \, p(v)
\end{align}
$$

Since the joint distribution factorizes $p(x, v) = p(x)\, p(v)$, we can select an initial random $v$ for a particle, numerically integrate using a finite difference method such as the leapfrog and then use the updated $x^*$ as the new proposal. The acceptance ratio for the new $x^*$ is

$$
\frac{ e^{ -U(x^*)-K(v^*) }} { e^{-U(x)-K(v)} } = e^{U(x)-U(x^*)+K(x)-K(x^*)}
$$

If our finite difference scheme was exact, the acceptance ration would be 1 since energy is conserved with Hamiltonian dynamics. However, as we have seen, the leapfrog method does not conserve energy perfectly and an accept/reject step is still needed.

#### Example of HMC

We will explore how HMC works when the target distribution is bivariate normal centered at zero

$$
x \sim N(0, \Sigma)
$$

In practice of course, the target distribution will be the posterior distribution and depend on both data and distributional parameters.

The potential energy or negative log likelihood is proportional to
$$
U(x) = \frac{x^T\Sigma^{-1} x}{2}
$$

The kinetic energy is given by
$$
K(v) = \frac{v^T v}{2}
$$

where the initial $v_0$ is chosen at random from the unit normal at each step.

To find the time updates, we use the Hamiltonian equations and find the first derivatives of total energy with respect to $x$ and $v$

$$
\begin{align}
x' &= \frac{\delta K}{\delta v} &= v \\
v' &= -\frac{\delta U}{\delta x} &= -\Sigma^{-1} x \\
\end{align}
$$

giving us the block matrix

$$
A = \pmatrix{0 & 1 \\ -\Sigma^{-1} & 0}
$$

By using the first derivatives, we are making use of the gradient information on the log posterior to guide the proposal distribution.

#### This is what the target distribution should look like

```python
sigma = np.array([[1,0.8],[0.8,1]])
mu = np.zeros(2)

dist = stats.multivariate_normal(mu, sigma)

x = np.linspace(-3.0, 3.0, 100)
y = np.linspace(-3.0, 2.0, 100)
X, Y = np.meshgrid(x, y)
Z = dist.pdf(np.c_[X.ravel(), Y.ravel()]).reshape(X.shape)

plt.contour(X, Y, Z)
plt.axis('square')
pass
```

#### This is the HMC posterior

```python
def E(A, u0, v0, u, v):
    """Total energy."""
    return (u0 @ tau @ u0 + v0 @ v0) - (u @ tau@u + v @ v)
```

```python
def leapfrog(A, u, v, h, N):
    """Leapfrog finite difference scheme."""
    v = v - h/2 * A @ u
    for i in range(N-1):
        u = u + h * v
        v = v - h * A @ u

    u = u + h * v
    v = v - h/2 * A @ u

    return u, v
```

```python
niter = 100
h = 0.01
N = 100

A = np.array([[0,1],[-1,0]])

tau = la.inv(sigma)

orbit = np.zeros((niter+1, 2))
u = np.array([-2,2])
orbit[0] = u
for k in range(niter):
    v0 = np.random.normal(0,1,2)
    u, v = leapfrog(tau, u, v0, h, N)

    # accept-reject
    u0 = orbit[k]
    a = np.exp(E(A, u0, v0, u, v))
    r = np.random.rand()

    if r < a:
        orbit[k+1] = u
    else:
        orbit[k+1] = u0
```

```python
sns.kdeplot(orbit[:, 0], orbit[:, 1])
plt.plot(orbit[:,0], orbit[:,1], alpha=0.2)
plt.scatter(orbit[:1,0], orbit[:1,1],  c='red', s=30)
plt.scatter(orbit[1:,0], orbit[1:,1],  c=np.arange(niter)[::-1], cmap='Reds')
plt.text(-1.8,2,'$x_0$', fontsize=14)
plt.axis('square')
pass
```

### Hierarchical models

Hierarchical models have the following structure - first we specify that the data come from a distribution with parameters $\theta$

$$
X \sim f(X\ | \ \theta)
$$

and that the parameters themselves come from another distribution with hyperparameters $\lambda$

$$
\theta \sim g(\theta \ | \ \lambda)
$$

and finally that $\lambda$ comes from a prior distribution

$$
\lambda \sim h(\lambda)
$$

More levels of hierarchy are possible - i.e you can specify hyper-hyperparameters for the distribution of $\lambda$ and so on.

The essential idea of the hierarchical model is because the $\theta$s are not independent but rather are drawn from a common distribution with parameter $\lambda$, we can share information across the $\theta$s by also estimating $\lambda$ at the same time.

As an example, suppose have data about the proportion of heads after some number of tosses from several coins, and we want to estimate the bias of each coin. We also know that the coins come from the same mint and so might share some common manufacturing defect. There are two extreme approaches - we could estimate the bias of each coin from its coin toss data independently of all the others, or we could pool the results together and estimate the same bias for all coins. Hierarchical models provide a compromise where we shrink individual estimates towards a common estimate.

Note that because of the conditionally independent structure of hierarchical models, Gibbs sampling is often a natural choice for the MCMC sampling strategy.

#### Gibbs sampler example from [Robert and Casella, 10.17](http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-21239-5)

Suppose we have data of the number of failures ($y_i$) for each of 10 pumps in a nuclear plant. We also have the times ($_i$) at which each pump was observed. We want to model the number of failures with a Poisson likelihood, where the expected number of failure $\lambda_i$ differs for each pump. Since the time which we observed each pump is different, we need to scale each $\lambda_i$ by its observed time $t_i$.

We now specify the hierarchical model - note change of notation from the overview above - that $\theta$ is $\lambda$ (parameter) and $\lambda$ is $\beta$ (hyperparameter) simply because $\lambda$ is traditional for the Poisson distribution parameter.

The likelihood $f$ is
$$
\prod_{i=1}^{10} \text{Poisson}(\lambda_i t_i)
$$

We let the prior $g$ for $\lambda$ be

$$
\lambda \sim \text{Gamma}(\alpha, \beta)
$$
with $\alpha = 1.8$ (an improper prior whose integral does not sum to 1)

and let the hyperprior $h$ for $\beta$ to be

$$
\beta \sim \text{Gamma}(\gamma, \delta)
$$

with $\gamma = 0.01$ and $\delta = 1$.

There are 11 unknown parameters (10 $\lambda$s and $\beta$) in this hierarchical model.

The posterior is
$$
p(\lambda, \beta \ | \ y, t) = \prod_{i=1}^{10} \text{Poisson}(\lambda_i t_i) \times \text{Gamma}(\alpha, \beta) \times \text{Gamma}(\gamma, \delta)
$$

with the conditional distributions needed for Gibbs sampling given by

$$
p(\lambda_i \ | \ \lambda_{-i}, \beta, y, t) = \text{Gamma}(y_i + \alpha, t_i + \beta)
$$

and

$$
p(\beta \ | \ \lambda, y, t) = \text{Gamma}(10\alpha + \gamma, \delta + \sum_{i=1}^10 \lambda_i)
$$

```python
from numpy.random import gamma as rgamma # rename so we can use gamma for parameter name
```

```python
def lambda_update(alpha, beta, y, t):
    return rgamma(size=len(y), shape=y+alpha, scale=1.0/(t+beta))

def beta_update(alpha, gamma, delta, lambd, y):
    return rgamma(size=1, shape=len(y) * alpha + gamma, scale=1.0/(delta + lambd.sum()))

def gibbs(niter, y, t, alpha, gamma, delta):
    lambdas_ = np.zeros((niter, len(y)), np.float)
    betas_ = np.zeros(niter, np.float)

    lambda_ = y/t

    for i in range(niter):
        beta_ = beta_update(alpha, gamma, delta, lambda_, y)
        lambda_ = lambda_update(alpha, beta_, y, t)

        betas_[i] = beta_
        lambdas_[i,:] = lambda_

    return betas_, lambdas_
```

```python
alpha = 1.8
gamma = 0.01
delta = 1.0
beta0 = 1
y = np.array([5, 1, 5, 14, 3, 19, 1, 1, 4, 22], np.int)
t = np.array([94.32, 15.72, 62.88, 125.76, 5.24, 31.44, 1.05, 1.05, 2.10, 10.48], np.float)
niter = 1000
```

```python
betas, lambdas = gibbs(niter, y, t, alpha, gamma, delta)
print('%.3f' % betas.mean())
print('%.3f' % betas.std(ddof=1))
print(lambdas.mean(axis=0))
print(lambdas.std(ddof=1, axis=0))
```

    2.413
    0.688
    [0.07170175 0.15396935 0.10465642 0.12387877 0.64406695 0.61179004
     0.84305929 0.85584759 1.33038796 1.85643359]
    [0.02764277 0.09410763 0.04150302 0.03185053 0.2946104  0.13278886
     0.57377361 0.52315636 0.60987176 0.4113526 ]

```python
plt.figure(figsize=(8, 16))
for i in range(len(lambdas.T)):
    plt.subplot(5,2,i+1)
    plt.plot(lambdas[::10, i]);
    plt.title('Trace for $\lambda$%d' % i)
plt.tight_layout()
```

```python

```


# Central Limit Theorem (CLT) and Normality Testing
My main inspiration for this notebook is [MIT's edX Lecture](https://courses.edx.org/courses/MITx/6.041x_1/1T2015/courseware/Unit_8_Limit_theorems_and_classical_statistics/Lec__19_The_Central_Limit_Theorem__CLT_/) (you might need an edX account to view it). In particular, I wanted to demonstrate the CLT by continuously summing samples of random variables from a non-normal distribution, _i.e._ an exponential or uniform distribution.

The question arises: how many times do you need to sum samples from a non-normal distribution to get "close enough" to a normal distribution? I explore a number of [normality tests](#stat_testing).

In math terms, let $X$ be a random variable:

$$
\begin{align} S_n &= X_1 + X_2 + \dots + X_n \\[1.5ex]
Z_n &= \frac{S_n - n \mu}{\sigma \sqrt{n}} \end{align}
$$

where $X_i$s are i.i.d.

The CLT states:

Let $Z$ be a standard normal RV: $Z \sim \mathcal{N}(\mu = 0, \sigma = 1)$.

$$ \forall z: \lim_{n \to \infty} P(Z_n \le z) = P(Z \le z) $$

At what value of $n$ does $P(Z_n \le z) \approx P(Z \le z)$?

# Imports and Definitions

```python
%matplotlib inline
import warnings
warnings.simplefilter('once', UserWarning)

import numpy as np
import matplotlib.pyplot as plt

from cycler import cycler # for plot colors
from scipy import stats

np.random.seed(56) # ensure reproducibility
plt.close('all')

# Globals
samples = int(5e3) # n > 5000 gives warnings about p-values
res = 1e-4

# Define standard normal distribution
N = stats.norm(loc=0, scale=1)
```

## Define the Test Distribution
Feel free to experiment with any of these [scipy.stats.rv_continuous](https://docs.scipy.org/doc/scipy/reference/stats.html) distributions! Distributions that are asymmetrical or have fat tails will give a more pronounced effect.

```python
# Uniform
# a = 1
# b = 9 # s.t. pdf = 1/8 = 0.125
# dist = stats.uniform(loc=a, scale=b-a)     # ~ dist[loc, loc+scale]

# Exponential
lam = 1
dist = stats.expon(scale=1/lam)
# scale = 1/lambda for f(x) = lambda * exp(-lambda*x)
```

# Convolve Samples From A Distribution

My major open question in this simple function arises in the normalization factors. Do $\mu$ = `dist.mean()` and $\sigma^2$ = `dist.var()` need to be known from the underlying distribution of our sample (which is not normally known)?

```python
def convolve_dist(dist, n, samples=1000, norm_out=True):
    """Convolve a distribution n times.
    For a random variable X,
        Sn = X1 + X2 + ... + Xn

    Parameters
    ----------
    dist : rv_continuous
        continuous distrubution object, i.e. scipy.stats.norm
    n : int
        number of convolutions to make
    samples : int, optional, default=1000
        number of samples to draw for each convolution
    norm_out : boolean, optional, default=True
        normalize output to Z-score: (S - n*dist.mean()) / np.sqrt(n*dist.var())

    Returns
    -------
    out : ndarray, shape (samples,)
        if norm_out, out = Zn values, otherwise out = Sn values
    """
    Sn = np.zeros(samples)
    for i in range(n):
        # Draw from distribution and add to sum
        Sn += dist.rvs(size=samples)

    if norm_out:
        Zn = (Sn - n*dist.mean()) / np.sqrt(n*dist.var()) # normalize Sn
        return Zn
    else:
        return Sn
```

## Plot the pdf of the test distribution

```python
# Draw samples on the range where pdf has support
x = np.linspace(dist.ppf(res), dist.ppf(1-res), 100)

fig = plt.figure(1)
fig.clf()
ax = plt.gca()
ax.set_title('Test Distribution')
ax.plot(x, dist.pdf(x), 'r-', label='test pdf')

# Draw from the distribution and display the histogram
r = dist.rvs(size=1000)
ax.hist(r, density=True, bins=25, histtype='stepfilled', alpha=0.2, label='samples')
ax.legend(loc='lower right')
plt.show()
```

## Demonstrate CLT
The following plot shows our test distribution vs. a standard normal for values of $n \in \{1, 2, 10, 30\}$. The convolution gets astoundingly close to normal at $n = 30$, even for the heavily skewed exponential distribution.

```python
#------------------------------------------------------------------------------
#        Plots vs. n
#------------------------------------------------------------------------------
# Plot histogram of samples vs normal distribution
fig = plt.figure(2, figsize=(11,9))

xN = np.linspace(N.ppf(res), N.ppf(1-res), 1000)

n_arr = [1, 2, 10, 30]
for i in range(len(n_arr)):
    # Convolve the pdfs
    Zn = convolve_dist(dist, n=n_arr[i], samples=samples)
    Nn = stats.gaussian_kde(Zn)   # compare to actual normal

    # Plot vs standard normal distribution
    ax = fig.add_subplot(2, 2, i+1)
    sns.distplot(Zn, kde=False, norm_hist=True, ax=ax)
    ax.plot(xN, Nn.pdf(xN), 'C0', label='$Z_n$ KDE')
    ax.plot(xN, N.pdf(xN), 'C3', label='$\mathcal{N}(0,1)$')
    ax.set_xlim([-4, 4])
    # ax.set_ylim([0, 1.25*max(Nn.pdf(xN))])
    ax.set_title("n = {}".format(n_arr[i]))

fig.suptitle(("Central Limit Theorem, $N_{{samples}}$ = {}\n" + \
             "$S_n = X_1 + \dots + X_n$").format(samples))
ax.legend(loc='lower right')
plt.show()
```

## Determine $n$ Using Normality Tests
Tests employed are:
* [Kolmogorov-Smirnov (K-S) Test `kstest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kstest.html#scipy.stats.kstest)
* [Shapiro-Wilk Test `shapiro`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html#scipy.stats.shapiro)
* [D'Agostino-Pearson Test `normaltest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.normaltest.html#scipy.stats.normaltest)
* [Anderson-Darling Test `anderson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.anderson.html#scipy.stats.anderson)

In all but the Anderson test, the null hypothesis is that the sample is drawn from the reference distribution (standard normal in this case). The Shaprio and D'Agostino tests are specific to normality testing. The others may be used to compare with _any_ reference distribution.

The $D$ statistic for the K-S test should approach 0 for a true normal distribution. The $W$ statistic for the Shapiro-Wilk test should approach 1, so I have plotted $1-W$ to compare with the other statistics. The $K$ statistic of the D'Agostino-Pearson test (`normaltest`) is not bound, so I have scaled it by its maximum value for comparison with the other statistics.

**Each test is plotted vs. $n$ convolutions of the test distribution, so $n = 1$ is just an exponential distribution (or whatever you chose for the test).**

I got some odd results with a [chi-squared test `chisquare`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.html#scipy.stats.chisquare) (lines commented out below).

```python
#------------------------------------------------------------------------------
#        Determine n to match N(0,1)
#------------------------------------------------------------------------------
# Draw from distrubution until we approach a standard normal
MAX_N = 100
#thresh = 1 - 1e-3
score = np.inf
n = 1
D = np.empty(MAX_N)
W = np.empty(MAX_N)
A = np.empty(MAX_N)
K = np.empty(MAX_N)
p = np.empty(MAX_N)
# X2 = np.empty(MAX_N)
D.fill(np.nan)
A.fill(np.nan)
W.fill(np.nan)
K.fill(np.nan)
p.fill(np.nan)
# X2.fill(np.nan)
Zn = []
while n < MAX_N:
    # Compute convolution
    Zn.append(convolve_dist(dist, n=n, samples=samples))
    # Test if convolution is equivalent to normal distribution
    D[n], p[n] = stats.kstest(Zn[-1], 'norm')
    W[n], _ = stats.shapiro(Zn[-1])
    A[n], cv, sig = stats.anderson(Zn[-1], dist='norm')
    K[n], _ = stats.normaltest(Zn[-1])
    # # Chi-squared test requires bins of data
    # Zn_hist, _ = np.histogram(Zn[-1], bins=100, density=True)
    # N_hist, _ = np.histogram(N.rvs(size=100000), bins=100, density=True)
    # X2[n], _ = stats.chisquare(f_obs=Zn_hist, f_exp=N_hist)
    # # Possible test if we've reached an acceptable threshold value:
    #if W[n] > thresh:
    #    break
    n += 1

# Plot test statistics vs. n
plt.figure(9, figsize=(11,9))
plt.clf()
ax = plt.gca()
ax.plot(np.arange(MAX_N), D, c='C3', label='$D$ statistic')
ax.plot(np.arange(MAX_N), 1-W, c='C2', label='$W$ statistic')
# ax.plot(np.arange(MAX_N), X2/np.nanmax(X2), c='C4', label='$\chi^2$ statistic')
ax.plot(np.arange(MAX_N), K/np.nanmax(K), c='C1', label='$K^2$ statistic')
ax.plot(np.arange(MAX_N), p, c='C0', label='$p$-value', zorder=0, alpha=0.5)
# ax.set_yscale('log')
ax.set_title('Test Statistics vs. $n$ convolutions, {} samples per convolution'.format(samples))
ax.set_xlabel('Number of convolved distributions')
ax.set_ylabel('Statistic')
# ax.set_ylim([0, 2])
ax.legend(loc='upper right')
plt.show()
```

### Results
We note each of the statistics starts at a large value, then decays rapidly towards 0 as we approach a normal distribution. The $p$-value gets quite noisy for large $n$ values. I am unsure why that is the case, and of how to interpret the $p$-value in conjunction with the test statistics.

### Anderson-Darling Test
This test statistic has a slightly different interpretation. If $A^2$ is larger than a given threshold, the null hypothesis that the data come from the chosen (normal) distribution can be rejected at corresponding significance level. The Anderson test is ([according to Wikipedia](https://en.wikipedia.org/wiki/Anderson–Darling_test)) more sensitive in the tails of the distribution.

For an exponential test distribution, the test statistic is _larger_ than all of the critical values until about $n = 95$, so we reject the null hypothesis that the data come from a normal distribution. It seems the Anderson test is **most stringent** when performing normality testing.

```python
# Plot A^2 statistic (Anderson-Darling test)
# If A^2 is larger than critical value for corresponding significance level,
# the null hypothesis that the data come from the chosen distribution can be
# rejected
plt.figure(10, figsize=(11,9))
ax = plt.gca()
ax.plot(np.arange(MAX_N), A, c='C1', label='$A^2$ statistic')
# Use greys for threshold values
ax.set_prop_cycle(cycler('color',
                         [plt.cm.bone(i) for i in np.linspace(0, 0.75, 5)]))
for i in range(5):
    ax.plot(np.array([0, n]), cv[i]*np.array([1, 1]),
            label='Threshold {}%'.format(sig[i]))
ax.set_yscale('log')
ax.set_title('Test Statistics vs. $n$')
ax.set_xlabel('Number of convolved distributions')
ax.set_ylabel('Statistic')
ax.legend(loc='upper right')
plt.show()
```

## Q-Q Plot
Lastly, we can plot the [quartile-quartile plot](https://en.wikipedia.org/wiki/Q–Q_plot) of the samples from our test distribution vs. the expected values from a normal distribution. The darker colors are higher $n$ values, which show the approach to the straight red line (true standard normal).

```python
plt.figure(11, figsize=(11,9))
ax = plt.gca()
colors = [plt.cm.bone(i) for i in np.linspace(0, 0.9, len(Zn))][::-1]
for i in range(len(Zn)):
    result = stats.probplot(Zn[i], dist='norm', plot=ax) # Q-Q plot
    ax.get_lines()[2*i].set_markeredgecolor('none')
    ax.get_lines()[2*i].set_markerfacecolor(colors[i])
    # Turn off all but last fit line
    if i < len(Zn)-1:
        ax.get_lines()[2*i+1].set_linestyle('none')
plt.show()
```

notebooks/CLT and Statistical Testing.ipynb


# The JuMP ecosystem for mathematical optimization

## 2019 INFORMS-ALIO meeting

## Minimum # of Passports to Visit all Countries?
[](https://www.passportindex.org)

199 passports = $10^{33}$ times the age of the universe to enumerate at $10^{17}$ flops!

```julia
import Pkg
Pkg.activate(@__DIR__)
Pkg.instantiate()
```

      Updating registry at `~/.julia/registries/General`
      Updating git-repo `https://github.com/JuliaRegistries/General.git`

Download data from https://github.com/ilyankou/passport-index-dataset

```julia
;git clone https://github.com/ilyankou/passport-index-dataset.git
```

    Cloning into 'passport-index-dataset'...

```julia
using DelimitedFiles
data = readdlm(joinpath("passport-index-dataset","passport-index-matrix.csv"),',')
cntr = data[2:end,1]
vf = (x ->  x == -1 || x == 3 ? 1 : 0).(data[2:end,2:end]);
```

## (Constrained) Mathematical Optimization and JuMP

$$
\begin{alignedat}{3}
&\min_{x,y} &\quad \sum_{\mathit{cntr} \;\in\; \mathit{World}} \mathit{pass}_{\,\mathit{cntr}} \\
&\text{s.t.}&\quad  \sum_{\mathit{cntr} \;\in\; \mathit{World}}\mathit{vf}(\mathit{cntr},\mathit{dst}) \cdot \mathit{pass}_{\,\mathit{cntr}} &\geq 1\quad  &\quad& \forall \; \mathit{dst} \;\in \; \mathit{World}\\
 &&\mathit{pass}_{\,\mathit{cntr}}  &\in \{0,1\}&\quad& \forall \; \mathit{cntr}\in \; \mathit{World}.
\end{alignedat}
$$

```julia
using JuMP, GLPK
model = Model(with_optimizer(GLPK.Optimizer))
@variable(model, pass[1:length(cntr)], Bin)
@constraint(model, [j=1:length(cntr)], sum( vf[i,j]*pass[i] for i in 1:length(cntr)) >= 1)
@objective(model, Min, sum(pass))
JuMP.optimize!(model)
print(JuMP.objective_value(model)," passports: ",join(cntr[findall(JuMP.value.(pass) .== 1)],", "))
```

    23.0 passports: Afghanistan, Angola, Australia, Austria, Comoros, Congo, Eritrea, Gambia, Georgia, Hong Kong, India, Iraq, Kenya, Madagascar, Maldives, North Korea, Papua New Guinea, Singapore, Somalia, Sri Lanka, Tunisia, United Arab Emirates, United States

# A step by step example
Let's see how we translate a simple, 2 variable LP to JuMP code.

$$
\begin{align*}
&\max_{x,y}& \quad x + 2y \\
&\text{s.t.}&\quad x + y &\leq 1 \\
&&0\leq x, y &\leq 1
\end{align*}
$$

Load JuMP, MathOptInterface (MOI), and GLPK (GNU LP/MIP solver):

```julia
using JuMP
using MathOptInterface # Replaces MathProgBase
# shortcuts
const MOI = MathOptInterface

using GLPK # Loading the GLPK module for using its solver
```

Construct a model object (a container for variables, constraints, solver options, etc.):

```julia
model = Model(with_optimizer(GLPK.Optimizer, msg_lev = 4));
# Old syntax: model = Model(solver=GLPKSolverLP(msg_lev = 4)))
```

Define variables $0\leq x, y \leq 1$:

```julia
@variable(model, 0 <= x <= 1)
@variable(model, 0 <= y <= 1)
```

$$ y $$

Add constraint $x + y \leq 1$:

```julia
@constraint(model, x + y <= 1)
```

$ x + y \leq 1.0 $

Add objective $\max x + 2y$:

```julia
@objective(model, Max, x + 2y)
```

$$ x + 2 y $$

To solve the optimization problem, call the `optimize` function.

```julia
JuMP.optimize!(model) # Old syntax: status = JuMP.solve(model)
```

    GLPK Simplex Optimizer, v4.64
    1 row, 2 columns, 2 non-zeros
    *     0: obj =  -0.000000000e+00 inf =   0.000e+00 (2)
    *     2: obj =   2.000000000e+00 inf =   0.000e+00 (0)
    OPTIMAL LP SOLUTION FOUND

We can then check the status of the optimization call.

```julia
@show JuMP.has_values(model)
@show JuMP.termination_status(model) == MOI.OPTIMAL
@show JuMP.primal_status(model) == MOI.FEASIBLE_POINT
@show JuMP.dual_status(model) == MOI.FEASIBLE_POINT
```

    JuMP.has_values(model) = true
    JuMP.termination_status(model) == MOI.OPTIMAL = true
    JuMP.primal_status(model) == MOI.FEASIBLE_POINT = true
    JuMP.dual_status(model) == MOI.FEASIBLE_POINT = true

    true

# New Solver Status

Much more details than old `:Optimal, :Unbounded, :Infeasible, :UserLimit, :Error, :NotSolved`

```julia
@show JuMP.termination_status(model) == MOI.OPTIMAL
```

```julia
display(typeof(MOI.OPTIMAL))
```

    Enum MathOptInterface.TerminationStatusCode:
    OPTIMIZE_NOT_CALLED = 0
    OPTIMAL = 1
    INFEASIBLE = 2
    DUAL_INFEASIBLE = 3
    LOCALLY_SOLVED = 4
    LOCALLY_INFEASIBLE = 5
    INFEASIBLE_OR_UNBOUNDED = 6
    ALMOST_OPTIMAL = 7
    ALMOST_INFEASIBLE = 8
    ALMOST_DUAL_INFEASIBLE = 9
    ALMOST_LOCALLY_SOLVED = 10
    ITERATION_LIMIT = 11
    TIME_LIMIT = 12
    NODE_LIMIT = 13
    SOLUTION_LIMIT = 14
    MEMORY_LIMIT = 15
    OBJECTIVE_LIMIT = 16
    NORM_LIMIT = 17
    OTHER_LIMIT = 18
    SLOW_PROGRESS = 19
    NUMERICAL_ERROR = 20
    INVALID_MODEL = 21
    INVALID_OPTION = 22
    INTERRUPTED = 23
    OTHER_ERROR = 24

```julia
@show JuMP.primal_status(model) == MOI.FEASIBLE_POINT
```

```julia
display(typeof(MOI.FEASIBLE_POINT))
```

    Enum MathOptInterface.ResultStatusCode:
    NO_SOLUTION = 0
    FEASIBLE_POINT = 1
    NEARLY_FEASIBLE_POINT = 2
    INFEASIBLE_POINT = 3
    INFEASIBILITY_CERTIFICATE = 4
    NEARLY_INFEASIBILITY_CERTIFICATE = 5
    UNKNOWN_RESULT_STATUS = 6
    OTHER_RESULT_STATUS = 7

We can also inspect the solution values and optimal cost:

```julia
@show JuMP.value(x)              # Old syntax: getvalue(x)
@show JuMP.value(y)              # Old syntax: getvalue(y)
@show JuMP.objective_value(model)       # Old syntax: getobjectivevalue(model)
```

    JuMP.value(x) = 0.0
    JuMP.value(y) = 1.0
    JuMP.objective_value(model) = 2.0

    2.0

I can also "name" constraints for later reference.

```julia
model = Model(with_optimizer(GLPK.Optimizer, msg_lev = 0))
@variable(model, 0 <= x <= 1)
@variable(model, 0 <= y <= 1)
@constraint(model, inequality, x + y <= 1)     # <=============== constraint can be referenced later as "inequality"
@objective(model, Max, x + 2y)
JuMP.optimize!(model)
@show JuMP.termination_status(model) == MOI.OPTIMAL
```

    JuMP.termination_status(model) == MOI.OPTIMAL = true

    true

constraint references  can by used to delete them

```julia
JuMP.delete(model, inequality)
JuMP.optimize!(model)
@show JuMP.termination_status(model) == MOI.OPTIMAL
@show JuMP.objective_value(model)
```

    JuMP.termination_status(model) == MOI.OPTIMAL = true
    JuMP.objective_value(model) = 3.0

    3.0

Constraint references can be used to modify problem (see [MOI](MOI.ipynb#Model-modifications)) and to get duals (see [Topics notebook](Topics.ipynb#Duality)).

## Collections of variables/constraints and summation in JuMP

You can also create collections of variables like $x_i \geq 0 \quad \forall \; i\in\{1,\ldots,10\}$

```julia
model = Model()
@variable(model, x[1:10] >= 0)
```

    10-element Array{VariableRef,1}:
     x[1]
     x[2]
     x[3]
     x[4]
     x[5]
     x[6]
     x[7]
     x[8]
     x[9]
     x[10]

Also multidimensional indexing, separated by commas:

```julia
@variable(model, y[1:10,["red","blue"]] <= 1)
```

    2-dimensional DenseAxisArray{VariableRef,2,...} with index sets:
        Dimension 1, 1:10
        Dimension 2, ["red", "blue"]
    And data, a 10×2 Array{VariableRef,2}:
     y[1,red]   y[1,blue]
     y[2,red]   y[2,blue]
     y[3,red]   y[3,blue]
     y[4,red]   y[4,blue]
     y[5,red]   y[5,blue]
     y[6,red]   y[6,blue]
     y[7,red]   y[7,blue]
     y[8,red]   y[8,blue]
     y[9,red]   y[9,blue]
     y[10,red]  y[10,blue]

and more complicated expressions like $\quad
i \leq z_{ij} \leq u_j \;\;\; \forall i \in \{1,...,10\}, j \in \{i+1, ..., 10\}
$:

```julia
u = rand(10)
@variable(model, i <= z[i=1:10,j=(i+1):10] <= u[j])
```

    JuMP.Containers.SparseAxisArray{VariableRef,2,Tuple{Any,Any}} with 45 entries:
      [8, 10]  =  z[8,10]
      [3, 6 ]  =  z[3,6]
      [6, 9 ]  =  z[6,9]
      [8, 9 ]  =  z[8,9]
      [1, 10]  =  z[1,10]
      [4, 5 ]  =  z[4,5]
      [2, 4 ]  =  z[2,4]
      [4, 9 ]  =  z[4,9]
      [1, 2 ]  =  z[1,2]
      [3, 4 ]  =  z[3,4]
               ⋮
      [4, 6 ]  =  z[4,6]
      [2, 10]  =  z[2,10]
      [1, 9 ]  =  z[1,9]
      [6, 10]  =  z[6,10]
      [1, 8 ]  =  z[1,8]
      [7, 10]  =  z[7,10]
      [6, 7 ]  =  z[6,7]
      [3, 10]  =  z[3,10]
      [3, 8 ]  =  z[3,8]
      [1, 5 ]  =  z[1,5]
      [3, 5 ]  =  z[3,5]

To specify conditions on the indexing, you can add conditionals inside the ``[...]`` block, separated by a semicolon:

```julia
@variable(model, w[i=1:10, c=["red","blue"]; iseven(i) || c == "red"] >= 0)
```

    JuMP.Containers.SparseAxisArray{VariableRef,2,Tuple{Any,Any}} with 15 entries:
      [10, blue]  =  w[10,blue]
      [7, red  ]  =  w[7,red]
      [8, red  ]  =  w[8,red]
      [10, red ]  =  w[10,red]
      [6, blue ]  =  w[6,blue]
      [1, red  ]  =  w[1,red]
      [5, red  ]  =  w[5,red]
      [2, blue ]  =  w[2,blue]
      [6, red  ]  =  w[6,red]
      [3, red  ]  =  w[3,red]
      [8, blue ]  =  w[8,blue]
      [4, red  ]  =  w[4,red]
      [4, blue ]  =  w[4,blue]
      [2, red  ]  =  w[2,red]
      [9, red  ]  =  w[9,red]

Also easy to create constrainta like $ \sum _{i = 1}^{10} x_i \leq 1$:

```julia
@constraint(model, sum(x[i] for i in 1:10) <= 1)
```

$ x_{1} + x_{2} + x_{3} + x_{4} + x_{5} + x_{6} + x_{7} + x_{8} + x_{9} + x_{10} \leq 1.0 $

Or more complicated ones like
$
\sum_{\substack{i\in\{1,...,10\}\\
                c\in\{"red","blue"\}}}
       coef(c) \cdot y_{ic} = 1
$

```julia
coef = Dict("red" => 2, "blue" => 3)
@constraint(model, sum(coef[c]*y[i,c] for i in 1:10, c in ["red","blue"]) == 1)
```

$ 2 y_{1,red} + 3 y_{1,blue} + 2 y_{2,red} + 3 y_{2,blue} + 2 y_{3,red} + 3 y_{3,blue} + 2 y_{4,red} + 3 y_{4,blue} + 2 y_{5,red} + 3 y_{5,blue} + 2 y_{6,red} + 3 y_{6,blue} + 2 y_{7,red} + 3 y_{7,blue} + 2 y_{8,red} + 3 y_{8,blue} + 2 y_{9,red} + 3 y_{9,blue} + 2 y_{10,red} + 3 y_{10,blue} = 1.0 $

or $
\sum_{i = 1}^{10} \sum_{j = i+1}^{10}
       i \cdot j \cdot z_{ij} \leq
\sum_{\substack{i\in\{1,...,10\},
                c\in\{"red","blue"\} \\
                \text{s.t. } iseven(i) \text{ or } c = "red"}}
       i^2 \cdot w_{ic}
$:

```julia
@constraint(model, sum(i*j*z[i,j] for i in 1:10, j in (i+1):10) <=
               sum(i^2*w[i,c] for i in 1:10, c in ["red","blue"] if iseven(i) || c == "red"))
```

$ 2 z_{1,2} + 3 z_{1,3} + 4 z_{1,4} + 5 z_{1,5} + 6 z_{1,6} + 7 z_{1,7} + 8 z_{1,8} + 9 z_{1,9} + 10 z_{1,10} + 6 z_{2,3} + 8 z_{2,4} + 10 z_{2,5} + 12 z_{2,6} + 14 z_{2,7} + 16 z_{2,8} + 18 z_{2,9} + 20 z_{2,10} + 12 z_{3,4} + 15 z_{3,5} + 18 z_{3,6} + 21 z_{3,7} + 24 z_{3,8} + 27 z_{3,9} + 30 z_{3,10} + 20 z_{4,5} + 24 z_{4,6} + 28 z_{4,7} + 32 z_{4,8} + 36 z_{4,9} + 40 z_{4,10} + 30 z_{5,6} + 35 z_{5,7} + 40 z_{5,8} + 45 z_{5,9} + 50 z_{5,10} + 42 z_{6,7} + 48 z_{6,8} + 54 z_{6,9} + 60 z_{6,10} + 56 z_{7,8} + 63 z_{7,9} + 70 z_{7,10} + 72 z_{8,9} + 80 z_{8,10} + 90 z_{9,10} - w_{1,red} - 4 w_{2,red} - 4 w_{2,blue} - 9 w_{3,red} - 16 w_{4,red} - 16 w_{4,blue} - 25 w_{5,red} - 36 w_{6,red} - 36 w_{6,blue} - 49 w_{7,red} - 64 w_{8,red} - 64 w_{8,blue} - 81 w_{9,red} - 100 w_{10,red} - 100 w_{10,blue} \leq 0.0 $

Can also do collections of constraints (named or unamed):
    $$
\begin{align}
x_i &\leq 0.9 \quad \forall i \in \{1,2,3\} \quad\text{ (large bounds)}\\
x_i &\leq 0.5 \quad \forall i \in \{4,5,6\}
\end{align}
$$

```julia
@constraint(model,largebounds[i=1:3], x[i] <= 0.9)
@constraint(model,[i=4:6], x[i] <= 0.5)
```

    1-dimensional DenseAxisArray{ConstraintRef{Model,C,Shape} where Shape<:AbstractShape where C,1,...} with index sets:
        Dimension 1, 4:6
    And data, a 3-element Array{ConstraintRef{Model,C,Shape} where Shape<:AbstractShape where C,1}:
     x[4] ≤ 0.5
     x[5] ≤ 0.5
     x[6] ≤ 0.5

# Classes of Constraints Beyond Linear Inequalities

Broadcasted and two sided linear inequalities:

```julia
A = [1.0 2.0; 3.0 4.0]
l = [4.0, 5.0]
u = [2.0, 3.0]
model = Model()
@variable(model, x[1:2])
@constraint(model, l .<= A*x .<= u)
```

    2-element Array{ConstraintRef{Model,MathOptInterface.ConstraintIndex{MathOptInterface.ScalarAffineFunction{Float64},MathOptInterface.Interval{Float64}},ScalarShape},1}:
     x[1] + 2 x[2] ∈ [4.0, 2.0]
     3 x[1] + 4 x[2] ∈ [5.0, 3.0]

## Quadratic Inequalities:

Both convex:

```julia
model = Model()
@variable(model, x[1:2])
@constraint(model, x[1]^2 + x[2] <= 1)
```

$ x_{1}^2 + x_{2} \leq 1.0 $

and non-convex:

```julia
@constraint(model, x[1]*x[2] - 1.0 == 0.0)
```

$ x_{1}\times x_{2} = 1.0 $

## Conic constraints including...

Semidefinite constraints:

```julia
model = Model()                         # using CSDP; model = Model(with_optimizer(CSDP.CSDPOptimizer))
@variable(model, y[1:2,1:2], Symmetric)
@constraint(model, y in PSDCone())
@variable(model, t)
@variable(model, w)
@SDconstraint(model,  [t 1; 1 -w] ⪰ [1 t; t -2])
```

$ \begin{bmatrix}
t - 1 & -t + 1\\
-t + 1 & -w + 2\\
\end{bmatrix} \in PSDCone() $

Second order cone constraints:
$$
\begin{equation}
\left\| Ax+u \right\|_2 \leq t
\end{equation}
$$

```julia
model = Model()
@variable(model, x[1:2])
@variable(model, t)
@constraint(model, [t;A*x+u] in SecondOrderCone())
```

$ [t, x_{1} + 2 x_{2} + 2, 3 x_{1} + 4 x_{2} + 3] \in MathOptInterface.SecondOrderCone(3) $

Rotated second order cone constraints:
$$
\begin{equation}
\left\| Ax+u \right\|_2 \leq t \cdot w,\quad w\geq 0
\end{equation}
$$

```julia
model = Model()
@variable(model, x[1:2])
@variable(model, t)
@variable(model, w)
@constraint(model, [t;w;A*x+u] in RotatedSecondOrderCone())
```

$ [t, w, x_{1} + 2 x_{2} + 2, 3 x_{1} + 4 x_{2} + 3] \in MathOptInterface.RotatedSecondOrderCone(4) $

# Also "derivative based" nonlinear constraints

Remains unchanged:

```julia
using Ipopt

model = Model(with_optimizer(Ipopt.Optimizer))
@variable(model, x)
@variable(model, y)

@NLobjective(model, Min, (1-x)^2 + 100(y-x^2)^2)
@NLconstraint(model, exp(x)+sin(x) <=0)

JuMP.optimize!(model)
```

    ******************************************************************************
    This program contains Ipopt, a library for large-scale nonlinear optimization.
     Ipopt is released as open source code under the Eclipse Public License (EPL).
             For more information visit http://projects.coin-or.org/Ipopt
    ******************************************************************************

    This is Ipopt version 3.12.10, running with linear solver mumps.
    NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

    Number of nonzeros in equality constraint Jacobian...:        0
    Number of nonzeros in inequality constraint Jacobian.:        1
    Number of nonzeros in Lagrangian Hessian.............:        4

    Total number of variables............................:        2
                         variables with only lower bounds:        0
                    variables with lower and upper bounds:        0
                         variables with only upper bounds:        0
    Total number of equality constraints.................:        0
    Total number of inequality constraints...............:        1
            inequality constraints with only lower bounds:        0
       inequality constraints with lower and upper bounds:        0
            inequality constraints with only upper bounds:        1

    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
       0  1.0000000e+00 1.00e+00 0.00e+00  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
       1  1.1271001e+01 6.01e-02 6.56e+01  -1.0 5.46e-01    -  1.00e+00 1.00e+00h  1
       2  3.1638662e+00 4.28e-02 1.79e+01  -1.0 3.17e-01    -  1.00e+00 7.13e-01f  1
       3  2.5851172e+00 0.00e+00 1.08e+00  -1.0 1.41e-01    -  1.00e+00 1.00e+00f  1
       4  2.6099717e+00 0.00e+00 1.25e-02  -1.0 1.21e-02    -  1.00e+00 1.00e+00h  1
       5  2.5289573e+00 0.00e+00 1.55e-01  -2.5 3.43e-02    -  1.00e+00 1.00e+00f  1
       6  2.5262625e+00 0.00e+00 3.78e-04  -2.5 1.51e-03    -  1.00e+00 1.00e+00h  1
       7  2.5235888e+00 0.00e+00 1.70e-04  -3.8 1.17e-03    -  1.00e+00 1.00e+00f  1
       8  2.5234381e+00 0.00e+00 5.52e-07  -5.7 6.62e-05    -  1.00e+00 1.00e+00h  1
       9  2.5234363e+00 0.00e+00 8.16e-11  -8.6 8.09e-07    -  1.00e+00 1.00e+00h  1

    Number of Iterations....: 9

                                       (scaled)                 (unscaled)
    Objective...............:   2.5234362583018233e+00    2.5234362583018233e+00
    Dual infeasibility......:   8.1634699000687760e-11    8.1634699000687760e-11
    Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
    Complementarity.........:   2.5074330278896033e-09    2.5074330278896033e-09
    Overall NLP error.......:   2.5074330278896033e-09    2.5074330278896033e-09

    Number of objective function evaluations             = 10
    Number of objective gradient evaluations             = 10
    Number of equality constraint evaluations            = 0
    Number of inequality constraint evaluations          = 10
    Number of equality constraint Jacobian evaluations   = 0
    Number of inequality constraint Jacobian evaluations = 10
    Number of Lagrangian Hessian evaluations             = 9
    Total CPU secs in IPOPT (w/o function evaluations)   =      2.360
    Total CPU secs in NLP function evaluations           =      1.220

    EXIT: Optimal Solution Found.

```julia

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, C. Cooper. Based on [CFDPython](https://github.com/barbagroup/CFDPython), (c)2013 L.A. Barba, also under CC-BY license.

# Space & Time

## 1-D Diffusion

Welcome back! This is the third IPython Notebook of the series *Space and Time — Introduction of Finite-difference solutions of PDEs*, the second module of ["Practical Numerical Methods with Python"](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about).

In the previous IPython notebooks of this series, we studied the numerical solution of the linear and non-linear convection equations using the finite-difference method, and learned about the CFL condition. Now, we will look at the one-dimensional diffusion equation:

\begin{equation}\frac{\partial u}{\partial t}= \nu \frac{\partial^2 u}{\partial x^2}\end{equation}

where $\nu$ is a constant known as the *diffusion coefficient*.

The first thing you should notice is that this equation has a second-order derivative. We first need to learn what to do with it!

### Discretizing 2nd-order derivatives

The second-order derivative can be represented geometrically as the line tangent to the curve given by the first derivative.  We will discretize the second-order derivative with a Central Difference scheme: a combination of forward difference and backward difference of the first derivative.  Consider the Taylor expansion of $u_{i+1}$ and $u_{i-1}$ around $u_i$:

$u_{i+1} = u_i + \Delta x \frac{\partial u}{\partial x}\big|_i + \frac{\Delta x^2}{2!} \frac{\partial ^2 u}{\partial x^2}\big|_i + \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\big|_i + {\mathcal O}(\Delta x^4)$

$u_{i-1} = u_i - \Delta x \frac{\partial u}{\partial x}\big|_i + \frac{\Delta x^2}{2!} \frac{\partial ^2 u}{\partial x^2}\big|_i - \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\big|_i + {\mathcal O}(\Delta x^4)$

If we add these two expansions, the odd-numbered derivatives will cancel out.  Neglecting any terms of ${\mathcal O}(\Delta x^4)$ or higher (and really, those are very small), we can rearrange the sum of these two expansions to solve for the second-derivative.

$u_{i+1} + u_{i-1} = 2u_i+\Delta x^2 \frac{\partial ^2 u}{\partial x^2}\big|_i + {\mathcal O}(\Delta x^4)$

And finally:

\begin{equation}\frac{\partial ^2 u}{\partial x^2}=\frac{u_{i+1}-2u_{i}+u_{i-1}}{\Delta x^2} + {\mathcal O}(\Delta x^2)\end{equation}

The central difference approximation of the 2nd-order derivative is 2nd-order accurate.

### Back to diffusion

We can now write the discretized version of the diffusion equation in 1D:

\begin{equation}\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\nu\frac{u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n}}{\Delta x^2}\end{equation}

As before, we notice that once we have an initial condition, the only unknown is $u_{i}^{n+1}$, so we re-arrange the equation to isolate this term:

\begin{equation}u_{i}^{n+1}=u_{i}^{n}+\frac{\nu\Delta t}{\Delta x^2}(u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n})\end{equation}

This discrete equation allows us to write a program that advances a solution in time—but we need an initial condition. Let's continue using our favorite: the hat function. So, at $t=0$, $u=2$ in the interval $0.5\le x\le 1$ and $u=1$ everywhere else.

### Stability of the diffusion equation

The diffusion equation is not free of stability constraints. Just like the linear and non-linear convection equations, there are a set of discretization parameters $\Delta x$ and $\Delta t$ that will make the numerical solution blow up. For the diffusion equation and the discretization used here, the stability condition for diffusion is

$$
\begin{equation}
\nu \frac{\Delta t}{\Delta x^2} \leq \frac{1}{2}
\end{equation}
$$

### And solve!

 We are ready to number-crunch!

The next two code cells initialize the problem by loading the needed libraries, then defining the solution parameters and initial condition. This time, we don't let the user choose just *any* $\Delta t$, though; we have decided this is not safe: people just like to blow things up. Instead, the code calculates a value of $\Delta t$ that will be in the stable range, according to the spatial discretization chosen! You can now experiment with different solution parameters to see how the numerical solution changes, but it won't blow up.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

```python
nx = 41
dx = 2./(nx-1)
nt = 20
nu = 0.3   #the value of viscosity
sigma = .2
dt = sigma*dx**2/nu

x = numpy.linspace(0,2,nx)
ubound = numpy.where(x >= 0.5)
lbound = numpy.where(x <= 1)

u = numpy.ones(nx)
u[numpy.intersect1d(lbound, ubound)] = 2

un = numpy.ones(nx)
```

```python
for n in range(nt):
    un = u.copy()
    u[1:-1] = un[1:-1] + nu*dt/dx**2*(un[2:] -2*un[1:-1] +un[0:-2])

pyplot.plot(x, u, color='#003366', ls='--', lw=3)
pyplot.ylim(0,2.5);
```

## Animations

Looking at before-and-after plots of the wave in motion is helpful, but it's even better if we can see it changing!

First, let's import the `matplotlib` animation module and also a special IPython display method called `HTML` (more on this in a bit).

##### Note:
You will also have to install a video encoder/decoder named `ffmpeg`.

If you use Linux or OSX, you can install `ffmpeg` using `conda`:

```console
conda install -c conda-forge ffmpeg
```

If you use Windows, installation instructions can be found here:
http://adaptivesamples.com/how-to-install-ffmpeg-on-windows/

```python
from matplotlib import animation
from IPython.display import HTML
```

Now we need to reset our initial conditions!

```python
nt = 50

u = numpy.ones(nx)
u[numpy.intersect1d(lbound, ubound)] = 2

un = numpy.ones(nx)
```

Now we have to create an animation.  This takes a few steps, but it's actually not hard to do!  First, we need to create a `figure` and add `axes`.  The `figure` defines the plot we want to animate.  The `axes` lets us create elements of the plot that we can change interactively.  Then we create a "blank" line that we will update at each iteration. Like this:

```python
fig = pyplot.figure(figsize=(8,5))
ax = pyplot.axes(xlim=(0,2), ylim=(1,2.5))
line = ax.plot([], [], color='#003366', ls='--', lw=3)[0]
```

**Note**: The `[0]` after the `ax.plot()` command is required, since `ax.plot` can (optionally) return several values.  Since we're only creating one line, we ask it for the "zeroth" (and only...) line.

Now that our `figure` is set up, we can define a function to evolve our solution plot.  This will be exactly the same as the code for the numerical solution, except that we also want to "update" the data of the solution line at each iteration.

```python
def diffusion(i):
    line.set_data(x,u)

    un = u.copy()
    u[1:-1] = un[1:-1] + nu*dt/dx**2*(un[2:] -2*un[1:-1] +un[0:-2])
```

We initialize our animation using `animation.FuncAnimation()` and save the animation object as a variable named `anim`.  We're going to pass the following arguments:

*  `fig`: the name of our figure
*  `diffusion`: the name of our solver function
*  `frames`: the number of frames to draw (which we set equal to `nt`)
*  `interval`: the number of milliseconds each frame appears for

```python
anim = animation.FuncAnimation(fig, diffusion,
                               frames=nt, interval=100)
```

Ok!  Time to display the animation.  We use the `HTML` display method that we imported above and the `to_html5_video` method to make it web compatible.

```python
HTML(anim.to_html5_video())
```

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


Traçar um esboço do gráfico e obter uma equação da parábola que satisfaça as condições dadas.

26. Vértice $V(-2,3)$; eixo $x+2=0$, passando pelo ponto $P(2,0)$

Sabendo que o eixo da parábola é representado por $x = -2$ pode-se perceber que seu eixo é paralelo a $y$ e sua função é representada por $(x-h)^2 = 2p(y-k)$

Descobrindo o valor de $p$ substituindo $V(-2, 3)$ e $P(2,0)$

$(2-(-2)^2 = 2 \cdot p \cdot (0 - 3)$

$4^2 = -6p$

$16 = -6p$

$p = -\frac{8}{3}$

Substituindo os pontos na equação da parábola

$(x-(-2))^2 = 2 \cdot -\frac{8}{3}(y-3)$

$(x+2)^2 = -\frac{16}{3}(y-3)$

$x^2 + 4x + 4 = -\frac{16}{3}y + \frac{48}{3}$

$x^2 + 4x + \frac{16}{3}y - \frac{48}{3} + 4 = 0$

$x^2 + 4x + \frac{16}{3}y - \frac{36}{3} = 0$

Multiplicando por $3$

$3x^2 + 12x + 16 - 36 = 0$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((x+2)**2, -16/3*(y-3)), (x,-20,20), (y,-20,20),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/26.ipynb