# Homework #7: Due October 28th, 2021, 11:59pm
## Hamiltonian Monte Carlo for Bayesian Logistic Regression

**AM 207: Advanced Scientific Computing**

**Instructor: Weiwei Pan**

**Fall 2021**

**Name: Jiahui Tang**

**Students collaborators: Shi Le Wong, Xin Zeng**

### Instructions:

**Submission Format:** Use this notebook as a template to complete your homework. Please intersperse text blocks (using Markdown cells) amongst `python` code and results -- format your submission for maximum readability. Your assignments will be graded for correctness as well as clarity of exposition and presentation -- a “right” answer by itself without an explanation or is presented with a difficult to follow format will receive no credit.

**Code Check:** Before submitting, you must do a "Restart and Run All" under "Kernel" in the Jupyter or colab menu. Portions of your submission that contains syntactic or run-time errors will not be graded.

**Libraries and packages:** Unless a problems specifically asks you to implement from scratch, you are welcomed to use any `python` library package in the standard Anaconda distribution.

```python
import autograd.numpy as np
from autograd import grad, elementwise_grad
from autograd.misc.optimizers import adam, sgd
from autograd import scipy as sp
import pandas as pd
import numpy
from sklearn.datasets import fetch_openml
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.utils import check_random_state
from sklearn.preprocessing import PolynomialFeatures
from pymc3 import Model
import pymc3 as pm
import matplotlib.pyplot as plt
import sys
%matplotlib inline
```

```python
def plot_decision_boundary(x, y, models, ax, poly_degree=1, test_points=None, shaded=True):
    '''
    plot_decision_boundary plots the training data and the decision boundary of the classifier.
    input:
       x - a numpy array of size N x 2, each row is a patient, each column is a biomarker
       y - a numpy array of length N, each entry is either 0 (no cancer) or 1 (cancerous)
       models - an array of classification models
       ax - axis to plot on
       poly_degree - the degree of polynomial features used to fit the model
       test_points - test data
       shaded - whether or not the two sides of the decision boundary are shaded
    returns:
       ax - the axis with the scatter plot

    '''
    # Plot data
    ax.scatter(x[y == 1, 0], x[y == 1, 1], alpha=0.2, c='red', label='class 1')
    ax.scatter(x[y == 0, 0], x[y == 0, 1], alpha=0.2, c='blue', label='class 0')

    # Create mesh
    interval = np.arange(-6, 6, 0.1)
    n = np.size(interval)
    x1, x2 = np.meshgrid(interval, interval)
    x1 = x1.reshape(-1, 1)
    x2 = x2.reshape(-1, 1)
    xx = np.concatenate((x1, x2), axis=1)

    # Predict on mesh points
    if(poly_degree > 1):
        polynomial_features = PolynomialFeatures(degree=poly_degree, include_bias=False)
        xx = polynomial_features.fit_transform(xx)

    if len(models) > 1:
        alpha_line = 0.1
        linewidths=0.1
    else:
        alpha_line = 0.8
        linewidths=0.5

    i = 0

    for model in models:
        yy = model.predict(xx)
        yy = yy.reshape((n, n))

        # Plot decision surface
        x1 = x1.reshape(n, n)
        x2 = x2.reshape(n, n)
        if shaded:
            ax.contourf(x1, x2, yy, alpha=0.1 * 1. / (i + 1)**2, cmap='bwr')
        ax.contour(x1, x2, yy, colors='black', linewidths=linewidths, alpha=alpha_line)

        i += 1

    if test_points is not None:
        for i in range(len(test_points)):
            pt = test_points[i]
            if i == 0:
                ax.scatter(pt[0], pt[1], alpha=1., s=50, color='black', label='test data')
            else:
                ax.scatter(pt[0], pt[1], alpha=1., s=50, color='black')

    ax.set_xlim((-5.5, 5.5))
    ax.set_ylim((-5.5, 5.5))
    ax.set_xlabel('x_1')
    ax.set_ylabel('x_2')
    ax.legend(loc='best')
    return ax

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

class Bayesian_logistic_regression:
    def __init__(self, intercept, slopes):
        self.intercept = intercept
        self.slopes = slopes

    def predict(self, x):
        y = sigmoid(x.dot(self.slopes) + self.intercept)
        return (y > 0.5).astype(np.int_)

    def predict_proba(self, x):
        return sigmoid(x.dot(self.slopes) + self.intercept)
```

## Problem Description: Inference for Bayesian Logistic Regression Models
In In-Class Exercise 10.13, we studied the problem of how to evaluate the classification probabilites outputed by a Logistic Regression model using calibration plots. Calibration plots are computed using observed data (data that is part of or similar to our training set). In this homework, we evaluate the usefulness of the predictive uncertainties Logistic Regression models and Bayesian Logistic Regression models for test data that are dissimilar to our training data. These test data points are called ***out of distribution (OOD) points***. Just as in Homework #2, we want the predictive uncertainties from our models to help us distinguish ***in-distribution*** test data (test data that are similar to data on which we trained our model) and OOD test data. Again, in many safety-critical applications of ML, we want human-experts to override model decisions if the model is operating on extremely unfamiliar data.

### Part I: Interpreting the Classification Probabilites of Logistic Regression as Uncertainties

**Exercise 1: (Classifying OOD MNIST Digits)** In In-Class Exercise 10.13, we interpreted the output of a Logistic Regression model - the classification probability - as an indicator of the confidence of the model when it is classifying a specific data point (e.g. when the probability is close to 0.5, we say that the model is not confident in classifying the data point as class 1 or 0). Explain why this is a reasonable interpretation of the classification probability.

In this Exercise, build a Logistic Regression model to distinguish between digital images of handwritten digits (this is the MNIST dataset):

` LogisticRegression(C=1., penalty='l2', solver='saga', tol=0.1)`

For simplicity, we only look at images of digits: 1 and 6.

In this dataset, the classes are balanced (we take 600 images of each digit). Report the accuracy of your trained classifier, based on this, decide whether or not this is a good model.

Next, for the predefined set of six test images,
`test_point_1`, ..., `test_point_6`,
report the classification probabiliy for each image under your trained model. Interpreting these probabilities as uncertainty, i.e. model confidence, do these predictive uncertainties make sense? **Hint:** compare the predictive uncertainties to the visualizations of these test images.

So is this classifier a good model? Does your analysis of the model's performance via its predictive accuracy agree with your analysis of the model's performance via its predictive uncertainties? Conclude whether or not you would feel comfortable using this model in a real-life setting?

```python
# Load data from https://www.openml.org/d/554
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)

```

```python
# Pick out two classes of digits: 1, 6 and take a subset of samples
X_subset = X[((y == '1') | (y == '6'))][0:600]
y_subset = y[((y == '1') | (y == '6'))][0:600]

# Encode the label '1' with y=1, and the label '6' with y=0
y_subset[y_subset == '1'] = '1'
y_subset[y_subset == '6'] = '0'
y_subset = y_subset.astype(int)

```

```python
# Randomize our data
random_state = check_random_state(0)
permutation = random_state.permutation(X_subset.shape[0])
X_subset = X_subset.iloc[permutation].values
y_subset = y_subset.iloc[permutation].values
X_subset = X_subset.reshape((X_subset.shape[0], -1))
```

```python
# Visualize some samples from the training dataset
fig, ax = plt.subplots(2, 3, figsize=(15, 10))
for i in range(3):
    ax[0, i].imshow(X_subset[y_subset == 0][i].reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
    ax[1, i].imshow(X_subset[y_subset == 1][i].reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
plt.show()
```

```python
# Test images for evaluating your classifier
test_point_1 = X_subset[y_subset == 0][10]
test_point_2 = X_subset[y_subset == 1][10]
test_point_3 = X[y=='3'].iloc[20].values
test_point_4 = np.random.normal(200, 10, size=(28 * 28, ))
test_point_5 = test_point_2 + np.random.normal(300, 10, size=(28 * 28, )) - test_point_1
test_point_6 = test_point_2 + test_point_1 + np.random.normal(100, 10, size=(28 * 28, ))

```

```python
# Visualize some samples from the test dataset
fig, ax = plt.subplots(2, 3, figsize=(15, 10))
ax[0, 0].imshow(test_point_1.reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
ax[0, 0].set_title('test image 1')
ax[0, 1].imshow(test_point_2.reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
ax[0, 1].set_title('test image 2')
ax[0, 2].imshow(test_point_3.reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
ax[0, 2].set_title('test image 3')
ax[1, 0].imshow(test_point_4.reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
ax[1, 0].set_title('test image 4')
ax[1, 1].imshow(test_point_5.reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
ax[1, 1].set_title('test image 5')
ax[1, 2].imshow(test_point_6.reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest')
ax[1, 2].set_title('test image 6')
plt.show()
```

```python
logreg = LogisticRegression(C=1., penalty='l2', solver='saga', tol=0.1)
logreg.fit(X_subset, y_subset)
print('Accuracy of logistic regression on training dataset is : %.4f'%logreg.score(X_subset, y_subset))
```

    Accuracy of logistic regression on training dataset is : 0.9950

```python
test_lst = [test_point_1, test_point_2,test_point_3,test_point_4,test_point_5,test_point_6]

prob = np.vstack([logreg.predict_proba(i.reshape(1, -1)) for i in test_lst])
pred = np.vstack([logreg.predict(i.reshape(1,-1)) for i in test_lst])
df = pd.DataFrame(prob, columns=['Prob of 6 (CLASS 0)', 'Prob of 1 (CLASS 1)'])
df['prediction'] = pred

df
```

      Prob of 6 (CLASS 0)
      Prob of 1 (CLASS 1)
      prediction

      0
      0.998424
      0.001576
      0

      1
      0.001475
      0.998525
      1

      2
      0.168727
      0.831273
      1

      3
      0.996468
      0.003532
      0

      4
      0.008245
      0.991755
      1

      5
      0.934672
      0.065328
      0

**Interpretation:**

1. *Explain why this is a reasonable interpretation of the classification probability.*

When the classification probability is high in one class, e.g. 0.9, the model showing the probability of the obervation being in that class is high, which means that our model is pretty confident in classifying this observation.
However, if the classification probability is around 0.5, it means the model think it is equally likely for this prediction to be either in class 0 or class 1. The model is like making a random guess when classifying this observation, and not confident in classifying the data point as class 1 or 0. Thus this is a reasonable interpretation of the classification probability.

2. *Model Interpretation*

Our model achieves 99.5% accuracy, which seems to be highly efficient and accurate. However, as the model is only trained on number 1 and 6, when it comes to out of distribution (OOD) test data points, it may not achieve same accuracy.

Looking at predictions on test data points.
* It classifies point 1 as `6`, point 2 as `1` all with almost 99.8% probability. This makes sense as both of them are 6 and 1 points taking from x subset.
* It classifies point 3 as `1`, point 4 as `6` with also high confidence. This doesn't make sense. As point 3 is an image with number `3`, and point 4 is produced by random normal noise.
* It further classifes point 5 as `1` and point 6 as `6` with 99% and 93% probability each. However, point 5 is made by taking test point 1 off from test point 2; and point 6 is made by adding test point 1 on top of test point 2, both of them has number 1 and 6 overlays each other.
* All of these predictions are with high probability and high confidence. However, these predictive uncertainties make no sense, by comparing the predictive uncertainties to the visualizations of these test images.

Thus this classifier could not be considered a good model when comes to OOD data. Our analysis of the model's performance via its predictive accuracy does not always agree with the analysis of the model's performance via its predictive uncertainties. The accuracy is quite high (99%), however, the predictive uncertainty is not something we expected when it comes to OOD data. We expect it to be less certain when facing something they never see before, but it turns out to be quite certain, and also making wrong prediction.
I would not feel comfortable using this model in a real-life setting, it could give certain prediction even when facing something we never seen before, without giving us enough warning of these new types of data input.

As said above in question background, we want the predictive uncertainties from our models to help us distinguish in-distribution test data (test data that are similar to data on which we trained our model) and OOD test data. In many safety-critical applications of ML, we want human-experts to override model decisions if the model is operating on extremely unfamiliar data. This model doesn't seem to give any signal warning us if the model is operating on extremely unfamiliar data, but instead give us a quite certain prediction. Thus, I don't feel very safe to deploy this model in real life setting.

**Exercise 2: (Classifying OOD Data for a Toy Dataset)** In this Exercise, build a Logistic Regression model to distinguish between two classes in our toy classification dataset:

` LogisticRegression(C=1., penalty='l2', solver='saga', tol=0.1)`

In this dataset, the classes are balanced (we have 100 points in each class). Report the accuracy of your trained classifier, based on this, decide whether or not this is a good model.

Next, for the predefined set of test points,
`test_points`,
report the classification probabiliy for each test point under your trained model. Interpreting these probabilities as uncertainty, i.e. model confidence, do these predictive uncertainties make sense? **Hint:** compare the predictive uncertainties to the visualizations of these test points, identify which of these test points are in-distribution and which ones are OOD.

Relate your analysis of the behaviour of Logistic Regression on this toy dataset to its performance on MNIST. Can you come up with some hypotheses for exactly when the classification probability indicates an inappropriate amount of certainty when predicting on test points?

```python
# Generate a toy dataset for classification
samples = 100
class_0 = np.random.multivariate_normal([-1, -1], 0.5 * np.eye(2), samples)
class_1 = np.random.multivariate_normal([1, 1], 0.5 * np.eye(2), samples)
x = np.vstack((class_0, class_1))
y = np.array([0] * 100 + [1] * 100)

```

```python
# Define test data points
test_points = [(-5, -5), (-5, 3.5), (5, 5), (5, -3.5), (-1, 1), (-1, -1.5), (1, 1.5)]

```

```python
# Visualize our test data points
fig, ax = plt.subplots(1, 1, figsize=(10, 5))
ax.scatter(x[y == 1, 0], x[y == 1, 1], alpha=0.3, color='red', label='class 1')
ax.scatter(x[y == 0, 0], x[y == 0, 1], alpha=0.3, color='blue', label='class 0')
for i in range(len(test_points)):
    pt = test_points[i]
    if i == 0:
        ax.scatter(pt[0], pt[1], alpha=1., s=50, color='black', label='test data')
    else:
        ax.scatter(pt[0], pt[1], alpha=1., s=50, color='black')
ax.set_xlabel('x_1')
ax.set_ylabel('x_2')
ax.set_title('toy classification data with test data points')
ax.legend(loc='best')
plt.show()

```

```python
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

logreg = LogisticRegression(C=1., penalty='l2', solver='saga', tol=0.1)

logreg.fit(x_train, y_train)
print('Accuracy of logistic regression on training dataset is : %.4f'%logreg.score(x_train, y_train))
print('Accuracy of logistic regression on training dataset is : %.4f'%logreg.score(x_test, y_test))

```

    Accuracy of logistic regression on training dataset is : 0.9750
    Accuracy of logistic regression on training dataset is : 0.9500

```python
# Plot the decision boundary for the learned model
fig, ax = plt.subplots(1, 1, figsize=(10, 5))
ax = plot_decision_boundary(x, y, [logreg], ax, poly_degree=1, test_points=test_points, shaded=True)
ax.set_xlabel('x_1')
ax.set_ylabel('x_2')
ax.legend(loc='best')
plt.show()
```

```python
prob = np.vstack([logreg.predict_proba(np.array(i).reshape(1, -1)) for i in test_points])
pred = np.vstack([logreg.predict(np.array(i).reshape(1,-1)) for i in test_points])

df = pd.DataFrame(prob, columns=['Prob of CLASS 0', 'Prob of CLASS 1'])
df['prediction'] = pred
df.index = [str(i) for i in test_points]

df

```

      Prob of CLASS 0
      Prob of CLASS 1
      prediction

      (-5, -5)
      1.000000e+00
      7.164770e-11
      0

      (-5, 3.5)
      9.915977e-01
      8.402293e-03
      0

      (5, 5)
      7.483858e-11
      1.000000e+00
      1

      (5, -3.5)
      8.773210e-03
      9.912268e-01
      1

      (-1, 1)
      5.782342e-01
      4.217658e-01
      0

      (-1, -1.5)
      9.969291e-01
      3.070869e-03
      0

      (1, 1.5)
      3.207193e-03
      9.967928e-01
      1

**Interpretation:**

Our model achieves 97% and 95% accuracy in training and testing set, which seems to be highly efficient and accurate, it seems to be a fairly good model.

Looking at predictions on pre-defined test data points.
* It classifies 4 OOD data point `(-5, -5)`,`(-5, 3.5)`,`(5, 5)`, `(5, -3.5)` with high certainty and confidence.
* However, it classifiy in-distribution data point `(-1, 1)` with low confidence, as it sits on the decision boundary.

The uncertainty doesn't make sense because the OOD test points might be classified by the logistic model with high confidency, while in-distribution test points might be classified with low confidency.
However, if we dive deep into the visualization of our data and test points, it is reasonable to hypothesize that the test points near the decision boundary will be classified with low confidency, while points away from the dicision boundary will be classified with high confidency.

If the OOD data have some real life meaning, and needs to have higher uncertainty, then this classifier could not be considered a good model when comes to OOD data. Our analysis of the model's performance via its predictive accuracy does not always agree with the analysis of the model's performance via its predictive uncertainties. The accuracy is quite high (98%), however, the predictive uncertainty is not something we expected when it comes to OOD data. We expect it to be less certain when facing something they never see before, but it turns out to be quite certain.

However, if there's no real life meaning but we only want to classify points by a hyperplane, then this certainty is totally fine, as these OOD points are further away from decision boundary.

Therefore, if a point is near the decision boundary, the classification probability might be appropriate to indicate uncertainty. However, if a point is far away from the decision boundary, it could be inappropriate to indicate uncertainty using classification probability. We need a second thought with a grain of salt here, to look into downstream task and consider underlying real life meaning of these OOD data points.

---

### Part II: Interpreting the Uncertainties of Bayesian Logistic Regression

**Exercise 3: (Classifying OOD Data for a Toy Dataset with Bayesian Logistic Regression)**
In Exercise 2 and 3, we examined one source of uncertainty from a logistic model: the classification probability. There is another source of uncertainty: uncertainty over the model itself (which decision boundaries can fit the dataset).

Use your implementation of HMC from In-Class Exercise 10.20 to sample from the posterior of the following Bayesian Logistic Regression model for our toy dataset:

\begin{align}
w_i &\sim \mathcal{N}(0, 10)\\
Y^{(n)} &\sim Ber(\text{sigm}(w_2 X_1^{(n)} + w_1 X_2^{(n)} + w_0))
\end{align}

where $w_i$, $Y^{(n)}$, $X_i^{(n)}$ are real scalar valued random variables, and where the data consists of the observations $(x_1^{(n)}, x_2^{(n)}, y^{(n)})$ in our toy dataset $\mathcal{D}$.

We suggest using the following settings for your HMC:

```python
#initialize the position vector for HMC sampling with the weights of the trained neural network
position_init = np.hstack((logistic.coef_.flatten(), logistic.intercept_))
position_init = position_init.reshape((1, 3))
#leap-frog step size
step_size = 1e-1
#leap-frog steps
leapfrog_steps = 20
#number of total samples after burn-in
total_samples = 5000
#percentage of samples to burn
burn_in = 0.1
#thinning factor
thinning_factor = 1
```

where `logistic.coef_` and `logistic.intercept_` are the learned parameters of your `sklearn` Logistic regression model you trained in Exercise 3.

Visualize the decision boundaries corresponding to the posterior samples from your Bayesian Logistic Regression model (see In-Class Exercise 10.13). What do the distribution of decision boundaries tell you about the model's predictive uncertainties for each of the test points (particularly the OOD points)?

```python
def leap_frog(position_init, momentum_init, step_size, leapfrog_steps, potential_energy, kinetic_energy):
        # initialize position
        position = position_init
        # initialize momentum
        momentum = momentum_init

        # compute the gradient of potential energy
        potential_energy_grad = grad(potential_energy)

        # compute the gradient of kinetic energy
        kinetic_energy_grad = grad(kinetic_energy)

        # full leap frog steps
        for _ in range(leapfrog_steps - 1):
            #print("position: ", position)
            #print("momentum: ", momentum)

            # half step update of momentum
            momentum = momentum - step_size/2 * potential_energy_grad(position)

            # full step update of position
            position = position + step_size * kinetic_energy_grad(momentum)

            # half step update of momentum
            momentum = momentum - step_size/2 * potential_energy_grad(position)

            assert not np.any(np.isnan(position))
            assert not np.any(np.isnan(momentum))

        # Reverse momentum
        momentum = -momentum
        return position, momentum

def hmc(position_current, momentum_current, step_size, leapfrog_steps, potential_energy, kinetic_energy, momentum_sampler):

        ### Define total energy
        total_energy = lambda q, p: potential_energy(q) + kinetic_energy(p)

        ### Sample momentum
        # Make sure that the distribution from which you sample the momentum matches
        # with your kinetic energy function
        momentum_current = momentum_sampler(10)

        ### Simulate Hamiltonian dynamics using Leap Frog
        position_proposal, momentum_proposal = leap_frog(position_current.copy(), momentum_current.copy(), step_size, leapfrog_steps, potential_energy, kinetic_energy)

        # compute total energy in current position and proposal position
        current_total_energy = total_energy(position_current, momentum_current)
        proposal_total_energy = total_energy(position_proposal, momentum_proposal)

        ### Metropolis Hastings Step
        # comute accept probability
        accept_prob = min(1, np.exp(current_total_energy - proposal_total_energy))

        # accept proposal with accept probability
        if np.random.rand() < accept_prob:
            accept = 1
            position_current = np.copy(position_proposal)
            momentum_current = np.copy(momentum_proposal)
        else:
            accept = 0

        return position_current, momentum_current, accept
```

```python
x_new = np.hstack((x, np.ones(x.shape[0]).reshape(-1,1)))
x_new.shape

```

    (200, 3)

```python
### Setting up the energy functions for HMC
m = 10
D = 3
M = lambda m: m * np.identity(D)

# bernoulli pmf with y = 1 is p**1(1-p)**0=p

potential_energy = lambda q: -(sp.stats.multivariate_normal.logpdf(q, np.zeros(D), 10 * np.identity(D)) + np.sum(np.log(1-sigmoid(np.dot(q,x_new[y==0,:].T))))+ np.sum(np.log(sigmoid(np.dot(q, x_new[y==1,:].T)))))

kinetic_energy = lambda p: 0.5*(np.dot(p,np.matmul(M(1/m),p.T))) + 0.5*np.log(np.linalg.det(M(m))) + D/2*np.log(2*np.pi)

momentum_sampler = lambda m: np.random.multivariate_normal(mean=np.zeros(D), cov=M(m))

### Setting up hyperparameters of HMC
#leap-frog step size
step_size = 1e-1
#leap-frog steps
leapfrog_steps = 20
#number of total samples after burn-in
total_trials = 5000
#percentage of samples to burn
burn_in = int(0.1 * 5000)
#thinning factor
thinning_factor = 1

#initialize the position vector for HMC sampling with the weights of the trained neural network
position_current = np.hstack((logreg.coef_.flatten(), logreg.intercept_))
position_current = position_current.reshape((1, 3))
momentum_current = np.array(([0.1,0.1,0.1]))#momentum_sampler(m)
```

```python
samples = [position_current]
accepts = 0.

for t in range(total_trials):
    position_current, momentum_current, accept = hmc(position_current, momentum_sampler(m), step_size, leapfrog_steps, potential_energy, kinetic_energy, momentum_sampler)
    samples.append(position_current)
    accepts += accept
    if t % 100 == 0 and t > 0:
        print('{} iterations, accept rate: {}'.format(t, accepts / (t + 1.)))

accept_rate = accepts / total_trials
samples = np.array(samples[burn_in::thinning_factor])
```

    100 iterations, accept rate: 1.0
    200 iterations, accept rate: 1.0
    300 iterations, accept rate: 1.0
    400 iterations, accept rate: 1.0
    500 iterations, accept rate: 1.0
    600 iterations, accept rate: 1.0
    700 iterations, accept rate: 1.0
    800 iterations, accept rate: 1.0
    900 iterations, accept rate: 1.0
    1000 iterations, accept rate: 1.0
    1100 iterations, accept rate: 1.0
    1200 iterations, accept rate: 1.0
    1300 iterations, accept rate: 1.0
    1400 iterations, accept rate: 1.0
    1500 iterations, accept rate: 1.0
    1600 iterations, accept rate: 1.0
    1700 iterations, accept rate: 1.0
    1800 iterations, accept rate: 1.0
    1900 iterations, accept rate: 1.0
    2000 iterations, accept rate: 0.9995002498750625
    2100 iterations, accept rate: 0.9995240361732508
    2200 iterations, accept rate: 0.9995456610631531
    2300 iterations, accept rate: 0.9991308126901347
    2400 iterations, accept rate: 0.9991670137442732
    2500 iterations, accept rate: 0.9992003198720512
    2600 iterations, accept rate: 0.9992310649750096
    2700 iterations, accept rate: 0.9992595335061089
    2800 iterations, accept rate: 0.9992859692966798
    2900 iterations, accept rate: 0.9993105825577387
    3000 iterations, accept rate: 0.9993335554815062
    3100 iterations, accept rate: 0.9993550467591099
    3200 iterations, accept rate: 0.9993751952514839
    3300 iterations, accept rate: 0.9993941229930324
    3400 iterations, accept rate: 0.9994119376653925
    3500 iterations, accept rate: 0.9994287346472437
    3600 iterations, accept rate: 0.9994445987225771
    3700 iterations, accept rate: 0.9994596055120237
    3800 iterations, accept rate: 0.9994738226782426
    3900 iterations, accept rate: 0.9994873109459113
    4000 iterations, accept rate: 0.9995001249687578
    4100 iterations, accept rate: 0.9995123140697391
    4200 iterations, accept rate: 0.9995239228755058
    4300 iterations, accept rate: 0.9995349918623576
    4400 iterations, accept rate: 0.9995455578277664
    4500 iterations, accept rate: 0.9995556542990447
    4600 iterations, accept rate: 0.9995653118887199
    4700 iterations, accept rate: 0.9995745586045522
    4800 iterations, accept rate: 0.9995834201208081
    4900 iterations, accept rate: 0.9995919200163232

```python
# Build a model for each sample in the posterior
posterior_samples_index = np.random.choice(len(samples),size=100, replace=False)
models = []

for index in posterior_samples_index:
    model_samples = samples[index, :, :]
    model = Bayesian_logistic_regression(model_samples[:,2],model_samples[:,:2].T)
    models.append(model)
fig, ax = plt.subplots(1, 1, figsize=(10, 5))
plot_decision_boundary(x, y, models, ax, poly_degree=1,test_points=test_points, shaded= True)
```

What do the distribution of decision boundaries tell you about the model's predictive uncertainties for each of the test points (particularly the OOD points)?

**Interpretation:**

From the decision boundaries, it is reasonable to conclude that, the test points lie at dense decision boundaries area have high predictive uncertainty, while the test points lie further away from decision boundaries area have low predictive uncertainty.

These decision boundaries are not enough to distinguish the OOD points from in-distribution points. For example, the test point 1,2,3,4 are far away from decision boundaries and have high confidence and low uncertainty, but they are OOD points. And the test point 5 that lie on the decision boundaries and has high epistemic uncertainty, but it is in-distribution.

**Exercise 4: (Epistemic Uncertainty from Bayesian Logistic Regression)** Since the posterior $p(\mathbf{w} | \mathcal{D}))$ is a distribution over multiple models (i.e. decision boundaries) and each model outputs a classification probability for each test data point, the posterior $p(\mathbf{w} | \mathcal{D}))$ defines a distribution of classification probabilities for each test point. The variance of the distribution over classification probabilities is one way of quantifying the amount of ***epistemic uncertainty*** - uncertainty due to lack of observed training data - of the model at a particular test point.

Estimate the average epistemic uncertainty, under the posterior $p(\mathbf{w} | \mathcal{D}))$, of the data points in the toy data set, then estimate the epistemic uncertainty of each test data point. Interpret these uncertainties, in particular, explain to what extend can the epistemic uncertainties of this Bayesian Logistic Regression model be used to distinguish in-distribution points from OOD points.

Relate the amount of epistemic uncertainty at a particular data point to the decision boundaries in the posterior $p(\mathbf{w} | \mathcal{D}))$. Using this geometric relationship, explain whether or not the epistemic uncertainties of our Bayesian Logistic Regression model is capable of distinguishing **all** possible OOD points.

```python
# Compute the average epistemic uncertainty over the trianing data
epistemic_uncertainties = []
for i in x:
    epistemic_uncertainties.append(np.var([model.predict_proba(np.array(i).reshape((1, -1))) for model in models]))

avg_uncertainty = np.mean(epistemic_uncertainties)
print('The estimated average epistemic uncertainty of training data points is: %.4f'%avg_uncertainty)

```

    The estimated average epistemic uncertainty of training data points is: 0.0012

```python
# Compute the epistemic uncertainty for test points
for i in test_points:
    test_epistemic_uncertainties = np.var([model.predict_proba(np.array(i).reshape((1, -1))) for model in models])
    print("for point "+ str(i) + ', the estimated epistemic uncertainty is: ' + str(round(test_epistemic_uncertainties,4)))

```

    for point (-5, -5), the estimated epistemic uncertainty is: 0.0
    for point (-5, 3.5), the estimated epistemic uncertainty is: 0.0124
    for point (5, 5), the estimated epistemic uncertainty is: 0.0
    for point (5, -3.5), the estimated epistemic uncertainty is: 0.0173
    for point (-1, 1), the estimated epistemic uncertainty is: 0.0343
    for point (-1, -1.5), the estimated epistemic uncertainty is: 0.0
    for point (1, 1.5), the estimated epistemic uncertainty is: 0.0

Interpret these uncertainties, in particular, explain to what extend can the epistemic uncertainties of this Bayesian Logistic Regression model be used to distinguish in-distribution points from OOD points.

Relate the amount of epistemic uncertainty at a particular data point to the decision boundaries in the posterior  $p(\mathbf{w} | \mathcal{D}))$. Using this geometric relationship, explain whether or not the epistemic uncertainties of our Bayesian Logistic Regression model is capable of distinguishing all possible OOD points.

**Interpretation:**

The estimated average epistemic uncertainty of training data points is 0.0012.

For point `(-1, 1)`, the estimated epistemic uncertainty is 0.0343, which is 28 times higher than average epistemic uncertainty, as it sits on a dense region of multiple decision boundaries.

For point `(-5, -5)` and `(5, 5)` and `(-1, -1.5)`, `(1, 1.5)`, their epistemic uncertainty is close to zero, as they are away from decision boundary.

However, from the above four points, half of them are OOD data and half of them are in distribution data. It seems we are unable to distinguish OOD points with in distribution data even with epistemic uncertainty under the posterior of Bayeisan logistic regression. To obtain high epistemic uncertainties for OOD points, we may want it to be close to decision boundaries, in a sense that posterior should curve around these point or let the point sits on top of the boundary. We may need to introduce non-linearity into our decision boundary.

**Exercise 5: (Classifying OOD Data for a Toy Dataset with Bayesian Non-linear Logistic Regression)**
Just as we can cast Bayesian Non-linear Regression as a non-linear transformation of the data followed by Bayesian Linear Regression, we can perform Bayesian Logistic Regression with non-linear decision boundaries by first applying a non-linear feature map to the data and then fitting a Bayesian Logistic Regression model on the features.

Perform Bayesian Logistic Regression using a degree 2 polynomial feature map, $\phi$, on the toy dataset. Use your implementation of HMC from In-Class Exercise 10.20 to sample from the posterior of the following Bayesian Logistic Regression model for our toy dataset:

\begin{align}
\mathbf{w} &\sim \mathcal{N}(0, 10\, \mathbf{I})\\
Y^{(n)} &\sim Ber(\text{sigm}(\mathbf{w}^\top \phi(\mathbf{X}^{(n)}) ))
\end{align}

Visualize the decision boundaries corresponding to the samples in the posterior. Are the epistemic uncertainties of this Bayesian model able to distinguish in-distribution test points from OOD ones? Relate the amount of epistemic uncertainty at a particular data point to the decision boundaries in the posterior $p(\mathbf{w} | \mathcal{D}))$. Using this geometric relationship, explain whether or not the epistemic uncertainties of our Bayesian Logistic Regression model is capable of distinguishing **all** possible OOD points.

```python
# degree 2 polynomial feature map
poly = PolynomialFeatures(2, include_bias=False)
x_poly = poly.fit_transform(x)
#x_poly_new = np.hstack((x_poly, np.ones(x.shape[0]).reshape(-1,1)))
x_poly_new = np.hstack((x_poly, np.ones((x_poly.shape[0], 1), dtype=x_poly.dtype)))
```

```python
logreg_poly = LogisticRegression(C=1., penalty='l2', solver='saga', tol=0.1)

logreg_poly.fit(x_poly, y)
print('Accuracy of logistic regression on training dataset is : %.4f'%logreg.score(x, y))
```

    Accuracy of logistic regression on training dataset is : 0.9700

```python
x_poly_new.shape
```

    (200, 6)

```python
### Setting up the energy functions for HMC
m = 10
D = 6
M = lambda m: m * np.identity(D)

# bernoulli pmf with y = 1 is p**1(1-p)**0=p
potential_energy = lambda q: -(sp.stats.multivariate_normal.logpdf(q, np.zeros(D), 10 * np.identity(D)) + np.sum(np.log(1-sigmoid(np.dot(q, x_poly_new[y==0,:].T)))) + np.sum(np.log(sigmoid(np.dot(q, x_poly_new[y==1,:].T)))))

kinetic_energy = lambda p: 0.5*(np.dot(p,np.matmul(M(1/m),p.T))) + 0.5*np.log(np.linalg.det(M(m))) + D/2*np.log(2*np.pi)

momentum_sampler = lambda m: np.random.multivariate_normal(mean=np.zeros(D), cov=M(m))

### Setting up hyperparameters of HMC
#leap-frog step size
step_size = 1e-1
#leap-frog steps
leapfrog_steps = 20
#number of total samples after burn-in
total_trials = 5000
#percentage of samples to burn
burn_in = int(0.1 * 5000)
#thinning factor
thinning_factor = 1

#initialize the position vector for HMC sampling with the weights of the trained neural network
position_current = np.hstack((logreg_poly.coef_.flatten(), logreg_poly.intercept_))
position_current = position_current.reshape((1, 6))
momentum_current = np.array(([0.1,0.1,0.1,0.1,0.1,0.1]))#momentum_sampler(m)

```

```python
### Running HMC to sample from a target distribution
samples = [position_current]
accepts = 0.

for t in range(total_trials):
    position_current, momentum_current, accept = hmc(position_current, momentum_current, step_size, leapfrog_steps, potential_energy, kinetic_energy, momentum_sampler)
    samples.append(position_current)
    accepts += accept
    if t % 100 == 0 and t > 0:
        print('{} iterations, accept rate: {}'.format(t, accepts / (t + 1.)))

accept_rate = accepts / total_trials
samples = np.array(samples[burn_in::thinning_factor])
```

    100 iterations, accept rate: 1.0
    200 iterations, accept rate: 0.9950248756218906
    300 iterations, accept rate: 0.9966777408637874
    400 iterations, accept rate: 0.9975062344139651
    500 iterations, accept rate: 0.998003992015968
    600 iterations, accept rate: 0.9983361064891847
    700 iterations, accept rate: 0.9985734664764622
    800 iterations, accept rate: 0.9987515605493134
    900 iterations, accept rate: 0.9988901220865705
    1000 iterations, accept rate: 0.999000999000999
    1100 iterations, accept rate: 0.9990917347865577
    1200 iterations, accept rate: 0.9991673605328892
    1300 iterations, accept rate: 0.9992313604919293
    1400 iterations, accept rate: 0.9992862241256245
    1500 iterations, accept rate: 0.9993337774816788
    1600 iterations, accept rate: 0.9993753903810119
    1700 iterations, accept rate: 0.9994121105232217
    1800 iterations, accept rate: 0.9994447529150472
    1900 iterations, accept rate: 0.9994739610731194
    2000 iterations, accept rate: 0.9995002498750625
    2100 iterations, accept rate: 0.9990480723465016
    2200 iterations, accept rate: 0.9990913221263062
    2300 iterations, accept rate: 0.9991308126901347
    2400 iterations, accept rate: 0.9991670137442732
    2500 iterations, accept rate: 0.9992003198720512
    2600 iterations, accept rate: 0.9992310649750096
    2700 iterations, accept rate: 0.9992595335061089
    2800 iterations, accept rate: 0.9989289539450197
    2900 iterations, accept rate: 0.9986211651154774
    3000 iterations, accept rate: 0.9983338887037654
    3100 iterations, accept rate: 0.9983876168977749
    3200 iterations, accept rate: 0.9984379881287098
    3300 iterations, accept rate: 0.998485307482581
    3400 iterations, accept rate: 0.9985298441634813
    3500 iterations, accept rate: 0.9985718366181091
    3600 iterations, accept rate: 0.9986114968064427
    3700 iterations, accept rate: 0.9986490137800594
    3800 iterations, accept rate: 0.9986845566956064
    3900 iterations, accept rate: 0.9987182773647783
    4000 iterations, accept rate: 0.9987503124218945
    4100 iterations, accept rate: 0.9987807851743478
    4200 iterations, accept rate: 0.9988098071887646
    4300 iterations, accept rate: 0.998837479655894
    4400 iterations, accept rate: 0.998863894569416
    4500 iterations, accept rate: 0.9988891357476116
    4600 iterations, accept rate: 0.9989132797217997
    4700 iterations, accept rate: 0.9989363965113806
    4800 iterations, accept rate: 0.9989585503020204
    4900 iterations, accept rate: 0.998979800040808

```python
# Build a model for each sample in the posterior
posterior_samples_index = np.random.choice(len(samples),size=100, replace=False)
models = []

for index in posterior_samples_index:
    model_samples = samples[index, :, :]
    model = Bayesian_logistic_regression(model_samples[:,5],model_samples[:,:5].T)
    models.append(model)
fig, ax = plt.subplots(1, 1, figsize=(10, 5))
plot_decision_boundary(x, y, models, ax, poly_degree=2,test_points=test_points, shaded= True)
```

```python
# Compute the average epistemic uncertainty over the training data
epistemic_uncertainties = []
for i in x_poly:
    epistemic_uncertainties.append(np.var([model.predict_proba(np.array(i).reshape((1, -1))) for model in models]))

avg_uncertainty = np.mean(epistemic_uncertainties)
print('The estimated average epistemic uncertainty of training data points is: %.4f'%avg_uncertainty)
```

    The estimated average epistemic uncertainty of training data points is: 0.0018

```python
# Compute the epistemic uncertainty for test points

test_points_poly = poly.fit_transform(np.array(test_points))

for i,j in zip(test_points_poly,test_points):
    test_epistemic_uncertainties = np.var([model.predict_proba(np.array(i).reshape((1, -1))) for model in models])
    print("for point "+ str(j) + ', the estimated epistemic uncertainty is: ' + str(round(test_epistemic_uncertainties,4)))

```

    for point (-5, -5), the estimated epistemic uncertainty is: 0.1333
    for point (-5, 3.5), the estimated epistemic uncertainty is: 0.2084
    for point (5, 5), the estimated epistemic uncertainty is: 0.1768
    for point (5, -3.5), the estimated epistemic uncertainty is: 0.0839
    for point (-1, 1), the estimated epistemic uncertainty is: 0.0778
    for point (-1, -1.5), the estimated epistemic uncertainty is: 0.0
    for point (1, 1.5), the estimated epistemic uncertainty is: 0.0

Are the epistemic uncertainties of this Bayesian model able to distinguish in-distribution test points from OOD ones? Relate the amount of epistemic uncertainty at a particular data point to the decision boundaries in the posterior  $p(\mathbf{w} | \mathcal{D}))$ . Using this geometric relationship, explain whether or not the epistemic uncertainties of our Bayesian Logistic Regression model is capable of distinguishing all possible OOD points.

**Interpretation:**

From the above plot, the model with polynomial feature have nonlinear decision boundary, thus, the points distance to decision boundary varied a lot comparing to previous example.

The estimated average epistemic uncertainty of training data points is 0.0018.

We see that estimated epistemic uncertainty for point (1, 1.5) and (-1, -1.5) are all close to zero, showing high confidence and low uncertainty. As they are now far away from the curve decision boundary in this case.
For the OOD points (-5, 3.5), (-5, -5) and (5, 5), they are now having higher uncertainty with 0.13 and 0.18, which is about 100 times higher than average uncertainty. It could be say that OOD points are now distinguishable from the training data since they have way higher epistemic uncertainties.

For point (5, -3.5), it also sit besides different decision boundaries where the posterior gives difference probabilites prediction, however, the boundary are not as messy as the region around (-5, 3.5), thus (5,-3.5) now have high variance, but lower than point (-5,3.5) based on different prediction probability. The varaince of prediction probability that rises from different decision boundaries determine the uncertainty level, and it doesn't depend on whether the data come from in distribution or OOD. It depends more on where the curves of decision boundary lie, and their relative distance to the points.

Still, we are unable to distinguish OOD points with in distribution data even with epistemic uncertainty under the posterior of Bayeisan logistic regression with degree 2 polynomials.

**Exercise 6: (Classifying OOD Data for a Toy Dataset with Bootstrap Non-linear Logistic Regression)** Perform bootstrap Logistic Regression using a degree 2 polynomial feature map, $\phi$, on the toy dataset. Obtain 100 bootstrap Logistic Regression models:
`LogisticRegression(C=1., penalty='l2', solver='saga', tol=0.1)`.

Visualize the decision boundaries corresponding to the samples in the bootstrap ensemble. Are the epistemic uncertainties of this ensemble able to distinguish in-distribution test points from OOD ones? Relate the amount of epistemic uncertainty at a particular data point to the decision boundaries of the models in the ensemble. Using this geometric relationship, explain whether or not the epistemic uncertainties of our bootstrap ensemble is capable of distinguishing **all** possible OOD points.

```python
def bootstrap_logreg(sample_numbers):
    models = []
    for i in range(sample_numbers):
        indices = np.random.choice(np.arange(0, len(y)), size=len(y))
        bootX, bootY = x_poly[indices], y[indices]
        logreg = LogisticRegression(C=1., penalty='l2', solver='saga', tol=0.1)
        logreg.fit(bootX, bootY)
        models.append(logreg)
    return models
```

```python
models = bootstrap_logreg(100)

# Visualize the decision boundaries of the bootstrap models
fig, ax = plt.subplots(1, 1, figsize=(10, 5))

ax = plot_decision_boundary(x, y, models, ax, poly_degree=2, test_points=test_points, shaded = True)
ax.set_xlabel('x_1')
ax.set_ylabel('x_2')
ax.legend(loc='best')
plt.show()

```

```python
# Compute the average epistemic uncertainty over the training data
epistemic_uncertainties = []
for i in x_poly:
    epistemic_uncertainties.append(np.var([model.predict_proba(np.array(i).reshape((1, -1))) for model in models]))

avg_uncertainty = np.mean(epistemic_uncertainties)
print('The estimated average epistemic uncertainty of training data points is: %.4f'%avg_uncertainty)
```

    The estimated average epistemic uncertainty of training data points is: 0.1824

```python
# Compute the epistemic uncertainty for test points

test_points_poly = poly.fit_transform(np.array(test_points))

for i,j in zip(test_points_poly,test_points):
    test_epistemic_uncertainties = np.var([model.predict_proba(np.array(i).reshape((1, -1))) for model in models])
    print("for point "+ str(j) + ', the estimated epistemic uncertainty is: ' + str(round(test_epistemic_uncertainties,4)))

```

    for point (-5, -5), the estimated epistemic uncertainty is: 0.25
    for point (-5, 3.5), the estimated epistemic uncertainty is: 0.1398
    for point (5, 5), the estimated epistemic uncertainty is: 0.25
    for point (5, -3.5), the estimated epistemic uncertainty is: 0.1787
    for point (-1, 1), the estimated epistemic uncertainty is: 0.0036
    for point (-1, -1.5), the estimated epistemic uncertainty is: 0.2309
    for point (1, 1.5), the estimated epistemic uncertainty is: 0.2304

Are the epistemic uncertainties of this ensemble able to distinguish in-distribution test points from OOD ones? Relate the amount of epistemic uncertainty at a particular data point to the decision boundaries of the models in the ensemble. Using this geometric relationship, explain whether or not the epistemic uncertainties of our bootstrap ensemble is capable of distinguishing all possible OOD points.

**Interpretation:**

From the above plot, the model with polynomial feature and bootstrap logistic regression also have nonlinear decision boundary, thus, the points distance to decision boundary varied a lot comparing to previous example.

The estimated average epistemic uncertainty of training data points is 0.1824.

We see that similarly, estimated epistemic uncertainty for point (-5, -5), (5, 5), (1, 1.5) and (-1, -1.5) are now all pretty high, close to `0.23-0.25`, showing low confidence and high uncertainty, regardless of whether they are in distribution or OOD. Half of them are OOD data and half of them are in distribution data. For point (-1, 1), it sits in highly densed area of all bootstrap models' decision boundary, instead of sitting somewhere where decision boundaries in the posterior gives difference probabilites prediction, thus it now become the point with lowest estimated epistemic uncertainty, with only `0.0036`. The varaince of prediction probability that rises from different decision boundaries determine the uncertainty level, and it doesn't depend on whether the data come from in distribution or OOD. It depends more on where the curves of decision boundary lie, and their relative distance to the points.

Still, we are unable to distinguish OOD points with in distribution data even with epistemic uncertainty under the posterior of Bayeisan logistic regression with degree 2 polynomials and bootstrap models.

### Part III: Broader Impact Analysis

**Exercise 7: (The Need for Useful Uncertainties in Classification)** Explain why OOD detection may be crucial for classifiers deployed in risk-adverse applications. Be concrete: choose a real-life domain application, identify the affected communities and the real-life consequences of an incorrect classification, explain *exactly* how OOD detection can help mitigate the risk of an incorrect classification. Feel free to use the following list of articles to help construct your answer.

**Reference:**
1. [Facial Recognition Is Accurate, if You’re a White Guy](https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html)
2. [IBM Builds a More Diverse Million-Face Dataset to Help Reduce Bias in AI](https://techcrunch.com/2019/01/29/ibm-builds-a-more-diverse-million-face-dataset-to-help-reduce-bias-in-ai/)
3. [Wrongfully Accused by an Algorithm](https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html)
4. [Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis](https://www.pnas.org/content/pnas/early/2020/05/19/1919012117.full.pdf)

**Answer:**

OOD detections is crucial for classifier deployed in risk-adverse application, from the linked articles above, we know that models could be used in real life domain such as facial recognition in crime detection and law enforcement. For example, a faulty facial recognition match led to a Michigan man’s arrest for a crime he did not commit. It shows that the training data does not have enough representative samples, or may be lacking data for some specific minority groups. Thus, when it comes to an OOD data point that's not seen before in training data, the model may instead give extremely confidence prediction with low uncertainty. For example, it may classify someone who the model have not seen before with high possibility and high certainty of committing crimes.

In this case, affected communities are citizens, especially minority groups who suffered from imblanace training data problem. An incorrect classification may lead to wrong law enforcement and arrest of an innocent citizen. Thus, OOD detection could help to mitigate the risk of an incorrect classification by introducing human factors, and let an experienced human decision maker to override the model's decision.

**Exercise 8: (Comparing Bayesian and Frequnetist Epistemic Uncertainties)** In Homework #2, we saw that the epistemic uncertainties of frequentist and Bayesian uncertainties can be desirable or undesirable depending on the number of observations and the choices of hyperparameters (e.g. polynomial degree, variance of the prior in the Bayesian model, regularization strength of the models in the ensemble). For Bayesian and frequentist ensembles of classifiers, based on empirical results above, conjecture which factors affect the quality of epistemic uncertainties of Bayesian and bootstrap ensemble classifiers, respectively. Support your answer with what we know about the properties of Logistic Regression models.

In order to obtain epistemic uncertainties that can distinguish in-distribution and OOD points, what settings of hyperparameters do we need for Bayesian and bootstrap ensemble classifiers, respectively?

**Answer:**

For Bayesian Model, polynomial degree, variance of the prior in the Bayesian model, regularization strength of the models in the ensemble will all affect the level of epistemic uncertainty. By changing those parameters, the shape of decision boundaries will vary a lot (linear to non-linear etc), and thus affecting the estimate variance of predictive classification probability. By setting different values, if we could make shape of decision boundary surround in distribution points, and make OOD points further away to decision boundaries, we could probably control the estimate variance of predictive classification probability, making high uncertainty for OOD data and low uncertainty for in distribution data.

To be more specific,
* Lower regularization: As high regularization might penalize curvy decision boundaries with complicated parameters, however, we want our decision boundary to curve around data.
* Higher Variance on prior: As higher variance enable us to change shape of boundaries in posterior.
* Higher degree of polynomial: To make the non-linearity curve around our data.

For frequentist ensembles of classifiers, there's no prior, thus only the polynominal degree, and the regularization strength of the models will affect the quality of epistemic uncertainty. The reason is similar as above, thes shape of decision boundries will change and help indicate uncertainty. And we also want higher degree of polynomial and lower regularization.

Thus, to conclude, for Bayesian classifiers, setting appropriate polynomial degree,regularization strength and prior variance can all be helpful to obtain epistemic uncertainties that can detect OOD points;
For frequentist ensembles of classifiers, setting appropriate polynomial degree and regularization strength may help distinguish in-distribution points from OOD points.

**Exercise 9: (Is OOD Detection Enough for Risk Adverse Applications?)** In the risk adverse real-life application you identified in Exercise 1, is accurate OOD detection enough to mitigate unwanted outcomes of deploying our model? That is, when classifiers are used in real-life applications where incorrect predictions can have disasterous consequence, what specific failure modes can these models experience? Are all of these failure modes addressed by proper detection of OOD inputs?

**Answer:**

Accurate OOD detection may not be enough to mitigate unwanted outcomes of deploying our model. When classifiers are used in real-life applications where incorrect predictions can have disasterous consequence, there are still many specific failure modes can these models experience even with detection of OOD inputs:

* Data Imbalance, Sparsity & Data Quality: If there's pre-existed systematic bias, such as minority with higher crime rate in our real life example, and there's no large pool of data to correct these bias, then the model might incorporating these bias by design, as it rises in the data collection step. This could not always be addressed even with detection of OOD inputs.

* Hidden Covariate Shift Problem: When the distribution of input data shifts between the training environment and live environment, and undetectable under OOD input detection.

* Confounding factors: Even if the model identified some major features that contributing to outcome, it could be the case that there are confounding factors that highly correlated to what features we observed. As correlation does not imply casual, we should not implement policy or law enforcement based on model performance and feature selection. There could be other factors that the model hasn't be able to capture. And depends on downstream task, we need to take special care on what level of policy we are going to enforce. This problem also could not be simply addressed by OOD input detection.


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also http://splines.readthedocs.io/.

# Non-Uniform Cubic Hermite Splines

Uniform spline: each segment is defined on the same unit interval, regardless how far apart (spatially) their start and end points are.
Longer elements show a higher velocity.
But the tangent vectors can be defined in a way that the velocity is smoothly changing from one segment to the next ($C^1$ continuity).

Idea: just use a larger time interval for longer segments?
But this doesn't work!
If we make the incoming and outgoing tangent vector the same, there is a sudden change in speed at the segment border ($G^1$ continuity):

TODO: image

For non-unit intervals we'll have to go back to the drawing board!
It will turn out that the size of the time interval $\Delta_0$ has to scale the tangent vectors for this to keep working.

Plan: substitute $t \to \frac{t - t_0}{t_1 - t_0}$ and do the same as for uniform Hermine splines.

```python
%matplotlib inline
from IPython.display import display
```

```python
import sympy as sp
sp.init_printing(order='rev-lex')
```

```python
from utility import NamedExpression, NamedMatrix
```

```python
t, t0, t1 = sp.symbols('t t:2')
```

```python
delta = t1 - t0, sp.S('Delta0')
sp.Eq(*delta)
```

```python
coefficients = sp.Matrix(sp.symbols('abm:4')[::-1])
b_monomial = sp.Matrix([t**3, t**2, t, 1]).T
b_monomial.dot(coefficients)
```

```python
p = _.subs(t, (t - t0) / (t1 - t0))
p.subs(*delta)
```

```python
velocity = p.diff(t)
velocity.subs(*delta)
```

The start and end point are the same as in the uniform case, since the amount of time in the interval doesn't affect them:

```python
x0 = NamedExpression('xbm0', p.subs(t, t0))
x1 = NamedExpression('xbm1', p.subs(t, t1))
display(x0, x1)
```

The velocities are similar to the uniform case, but they are divided by the interval $\Delta_0 = t_1 - t_0$:

```python
xd0 = NamedExpression('xdotbm0', velocity.subs(t, t0))
xd1 = NamedExpression('xdotbm1', velocity.subs(t, t1).factor())
display(xd0.subs(*delta), xd1.subs(*delta))
```

As before, we can now try to extract the matrix values from the equations:

```python
M_H = NamedMatrix(r'{M_\text{H}}', 4, 4)
```

```python
control_values_H = NamedMatrix(sp.Matrix([x0.name, x1.name, xd0.name, xd1.name]),
                               M_H.name.I * coefficients)
control_values_H
```

```python
substitutions = x0, x1, xd0, xd1
```

```python
control_values_H.subs(substitutions).subs(*delta)
```

```python
M_H.I = sp.Matrix([[expr.expand().coeff(c) for c in coefficients]
                   for expr in control_values_H.subs(substitutions).name])
M_H.I.subs(*delta)
```

```python
print(_.expr)
```

```python
M_H.factor().subs(*delta)
```

```python
print(_.expr)
```

```python
b_H = NamedMatrix(r'{b_\text{H}}', b_monomial * M_H.expr)
b_H.factor().subs(*delta).simplify().T
```

So those are the new basis functions.
They are the same as in the uniform case, except that the last two (the ones corresponding to the the tangent vectors at the start and the end) are scaled by $\Delta_0$.

With non-uniform splines, the coefficients are different for each segment,
because generally the values $\Delta_i$ are different.

## Plotting Non-Uniform Hermite Splines

```python
import matplotlib.pyplot as plt
import numpy as np
```

```python
def hermite_curve(control_values, begin, end):
    coeffs = sp.lambdify([t0, t1], M_H.expr)(begin, end) @ control_values

    def evaluate(t):
        t = np.expand_dims(t, -1)
        t = (t - begin) / (end - begin)
        return t**[3, 2, 1, 0] @ coeffs

    return evaluate
```

```python
def plot_hermite(control_values, t_values, ax=None):
    if ax is None:
        ax = plt.gca()
    control_values = np.asarray(control_values)
    c = hermite_curve(control_values, t_values[0], t_values[-1])
    xyuv = np.column_stack(np.vsplit(control_values, 2))
    # Move beginning of second arrow:
    xyuv[1, :2] -= xyuv[1, 2:]
    ax.quiver(*xyuv.T, angles='xy', scale_units='xy', scale=1, color='lightgrey')
    ax.plot(*c(t_values).T, '.')
    ax.axis('equal')
```

```python
t_values = np.linspace(0, 1, 15)
```

For unit intervals, it's the same as before:

```python
plot_hermite([[0, 0], [5, 2], [1, 2], [1, -2]], t_values)
plot_hermite([[5, 2], [6, 1], [1, -2], [5, 1]], t_values)
```

```python
plot_hermite([[0, 0], [5, 2], [1, 2], [1, -2]], np.linspace(0, 3, 45, endpoint=True))
plot_hermite([[5, 2], [6, 1], [1, -2], [5, 1]], np.linspace(0, 2/3, 10, endpoint=True))
```

```python

```


                                             MVA - MCMC
                                             TP2 - EM-Algo
                                        Achari Berrada Youssef

# Exercice 1: Discrete distributions:

```python
# Import the libraries :
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
from scipy.stats import multivariate_normal as mn # To evaluate a multivariate normal distribution
from sklearn.cluster import KMeans
```

```python
def sample_discrete (pmf):
    # Function to sample from a distribution vector p
    x = np.random.uniform(0,1)
    i = 1
    s = pmf[i-1]
    while   s < x  :
        i = i + 1
        s = s + pmf[i-1]
    return i ;
```

```python
def sample_N (pmf, N ):
    # Function to sample from a probability mass function pmf N samples
    l = []
    for i in range(0,N):
        l.append(sample_discrete(pmf))
    return l;
```

```python
def gen_pmf(n) :
    # Function to generate a propability mass function of length K
    pmf = np.random.uniform(0,1,n)
    pmf = pmf / pmf.sum()
    return pmf ;
```

#### Define the parameters :

```python
n = 40 # size of the dataset
N = 10000 # Number of iid random values generated
```

Generate $\text{pmf} = (p_i)_{1\leq i \leq n} $ the probability mass function of $n$ distincts real numbers $X = \{ x_1,\dots,x_n \}$:

```python
pmf = gen_pmf(n) # Probability mass function
```

Generate $ (X_i)_{1 \leq i \leq n } $ i.i.d. random variables having the distribution of $X$ :

```python
X = sample_N(pmf,N)
```

```python
bins_hist = np.linspace(1, n+1, n+1,endpoint = True)
bins_bar = np.linspace(1, n, n,endpoint = True)
```

```python
# Comparison of the empirical and true distribution :
plt.hist(X, normed = 1,bins = bins_hist, facecolor='green',width=0.5); # Empirical distribution
plt.bar(bins_bar,pmf,edgecolor='r',fc = (0,0,0,0),width=0.5); # True Probability Mass Function
plt.legend({'Probability Mass Function','Empirical distribution'})
```

# Exercice 2 : Gaussian mixture model and the EM algorithm :

## Part 1 :
Generate gaussian mixture according to the Hierarchical model :
\begin{align}
 \mathbb{P}_{\theta} (Z_i  = j ) & = \alpha_j  \\
 X_i | \theta , {Z_i = j }  & \sim  \mathcal{N}(\mu_j,\sigma_j^2 )
 \end{align}

```python
n = 1000 # Size of the data set.
M = 3 # Number of components of the GMM.
T = 20 # Number of iteration of the EM step.
```

Fix the parameters to estimate for $M = 3$ :

```python
# Priors on the components
alpha = np.array([0.3,0.3,0.4])
# Means of the Gaussians distribution
Mu = np.zeros([M,2])
Mu[0] = np.array([0.2,0.9])
Mu[1] = np.array([0.5,0.1])
Mu[2] = np.array([0.8,0.9])
# Covariance Matrices of the Gaussians distribution
# Function to generate n semidefinite of size 2
Sigma = np.zeros([M,2,2])
Sigma[0] = np.array([[0.02,0.001],[0.001,0.1]])
Sigma[1] = np.array([[0.05,0.0002],[0.0002,0.01]])
Sigma[2] = np.array([[0.01,0],[0,0.01]])
```

```python
# Sample n points from the hierarchical model :
def gmm_sample(n,M,alpha,Mu,Sigma):
    Z = sample_N(alpha,n)
    X = np.zeros([n,2])
    y = np.zeros(n,dtype=int)
    for i in range(n) :
        X[i] = np.random.multivariate_normal(Mu[Z[i]-1], Sigma[Z[i]-1])
        y[i] = Z[i]
    return X,y
```

### Generate the data points :

```python
X,y = gmm_sample(n,M,alpha,Mu,Sigma)
```

```python
COLORS = ['red','green','yellow']
C = []
for i in range(n):
    C.append(COLORS[y[i]-1])
```

### Plot the data points :
We plot points generated from a hierarchical model with 3 components :

```python
plt.scatter(X[:,0],X[:,1],c=C )
plt.title(u"GM Model - 3 Components")
plt.xlabel(u"1st principal component")
plt.ylabel(u"2nd principal component")
```

## Part 2 :
Expectation-Minimization algorithm to estimate the parameters $\theta = (\alpha_i, \{ \mu_i, \Sigma_i\})_{i = 1..M}$:

```python
def computeQ(X,alpha_em,Mu_em,Sigma_em):
    # Compute Q of size n*M
    n = X.shape[0]
    M = Mu_em.shape[0]
    Q = np.zeros([n,M])
    for i in range(M):
        rvi = mn(mean=Mu_em[i],cov=Sigma_em[i])
        Q[:,i] = rvi.pdf(X)
    Q = np.multiply(Q,alpha_em)
    Q = np.divide(Q.T,np.sum(Q,axis=1))
    Q = Q.T

    return Q;
```

```python
def loglikelihood(X,Q,alpha_em,Mu_em,Sigma_em):
    # Compute the loglikelihood
    n = X.shape[0]
    M = Mu_em.shape[0]
    L = 0
    I = np.argmax(Q,axis=1)
    rv = []
    for k in range(M):
        rv.append(mn(mean=Mu_em[k],cov=Sigma_em[k]))
    for i in range(n):
        L = L + np.log(rv[I[i]].pdf(X[i]) * alpha_em[I[i]])
    return L
```

```python
def EM(X,M,T):
    # Function to implement the EM algorithm of M component on sample
    n = X.shape[0] # Size of the Data
    # Initialization of \theta :
    # Innitialization with kMeans implemented with Scikit learn :
    kmeans = KMeans(n_clusters=M, random_state=0).fit(X) ;
    Mu_em  = kmeans.cluster_centers_
    Sigma_em = np.ones([M,2,2])
    alpha_em = np.zeros(M)
    ## Log likelihood :
    L=[]
    for k in range(M):
        Sigma_em[k] = np.cov(X[kmeans.labels_ == k].T)
        alpha_em[k] = np.sum(kmeans.labels_ == k) ;
    alpha_em = np.divide(alpha_em,n)
    # Set Q as a matrix of probability of size n * M
    # Iterate from 1 to T the EM Algorithm
    if T == 0 : return alpha_em, Mu_em, Sigma_em
    for t in range(T):
        #print Sigma_em
        # Expectation step
        Q = computeQ(X,alpha_em,Mu_em,Sigma_em)
        # Maximization step
        # Compute the means :
        for k in range(M) :
            Mu_em[k] = np.divide(np.dot( X.transpose() , Q[:,k]), np.sum(Q[:,k]) )
        # Compute the Covariance matrices :
        for k in range(M) :
            X_k = X - Mu_em[k]
            x = np.multiply(X_k.T , Q[:,k] )
            Sigma_em[k] = np.divide(np.dot(x ,X_k),np.sum(Q[:,k]))
        # Compute the priors on the GM Components
        for k in range(M) :
            alpha_em[k] = np.sum(Q[:,k]) / np.sum(Q)
        L.append(loglikelihood(X,Q,alpha_em,Mu_em,Sigma_em))
    return alpha_em, Mu_em, Sigma_em,L
```

```python
alpha_em, Mu_em, Sigma_em, L = EM(X,M,T)
```

### Compute the loglikelihood :

```python
plt.plot(L)
plt.title("Plot of the loglikelihood")
plt.ylabel("L")
plt.xlabel("T iteration")
```

### Compare the EM parameters obtained :

```python
alpha
```

    array([ 0.3,  0.3,  0.4])

```python
alpha_em
```

    array([ 0.30924689,  0.38492314,  0.30582997])

```python
Mu
```

    array([[ 0.2,  0.9],
           [ 0.5,  0.1],
           [ 0.8,  0.9]])

```python
Mu_em
```

    array([[ 0.4776372 ,  0.09040291],
           [ 0.79352094,  0.91101768],
           [ 0.19662513,  0.9068444 ]])

```python
Sigma
```

    array([[[ 0.02,  0.01],
            [ 0.01,  0.1 ]],

           [[ 0.05,  0.02],
            [ 0.02,  0.01]],

           [[ 0.01,  0.  ],
            [ 0.  ,  0.01]]])

```python
Sigma_em
```

    array([[[ 0.05162482,  0.02095016],
            [ 0.02095016,  0.01048711]],

           [[ 0.00979671, -0.00053315],
            [-0.00053315,  0.01016339]],

           [[ 0.01753074,  0.00998589],
            [ 0.00998589,  0.11149271]]])

### Plot contours of the Gaussians finded by the EM :

```python
## define the grid :
x, y = np.mgrid[-0.3:1.1:.01, -0.35:1.6:.01]
pos = np.empty(x.shape + (2,))
pos[:, :, 0] = x; pos[:, :, 1] = y
for i in range(M):
    rvi = mn(Mu_em[i], Sigma_em[i])
    plt.contour(x, y, rvi.pdf(pos))

plt.scatter(X[:,0],X[:,1],c=C )
plt.title(u"GM Model - 3 Components")
plt.xlabel(u"1st principal component")
plt.ylabel(u"2nd principal component")

```

### Application :

```python
# Load data of the Crude Birth/Death Rate.
CBDR = pd.read_csv('Crude-BirthDeath-Rate.csv', header=0)
X_ = CBDR.values
plt.scatter(X_[:,0],X_[:,1])
plt.title(u"Crude Birth / Death Rate ")
plt.xlabel(u"Birth Rate")
plt.ylabel(u"Death Rate")
```

The data points are distributed as a Banana shape and therefore it not perfectly adapted for a Gaussian Distribution.
Let's see what happens :

```python
## define new adapted grid :
x_, y_ = np.mgrid[10:60:1, 0:30:1]
pos_ = np.empty(x.shape + (2,))
pos_[:, :, 0] = x; pos_[:, :, 1] = y
```

### M=1

```python
T_ = 50
M_ = 1
a_, Mu_, Si_,_ = EM(X_,M_,T_)
for i in range(M_):
    rvi = mn(Mu_[i], Si_[i])
    plt.contour(x_, y_, rvi.pdf(pos_))
plt.scatter(X_[:,0],X_[:,1] )
plt.title(u"GM Model - 1 Components")
plt.xlabel(u"1st principal component")
plt.ylabel(u"2nd principal component")
```

### M = 2

```python
T_ = 50
M_ = 2
a_, Mu_, Si_,_ = EM(X_,M_,T_)
for i in range(M_):
    rvi = mn(Mu_[i], Si_[i])
    plt.contour(x_, y_, rvi.pdf(pos_))

plt.scatter(X_[:,0],X_[:,1] )
plt.title(u"GM Model - 2 Components")
plt.xlabel(u"1st principal component")
plt.ylabel(u"2nd principal component")
```

### M = 3

```python
T_ = 50
M_ = 3
a_, Mu_, Si_,L = EM(X_,M_,T_)
for i in range(M_):
    rvi = mn(Mu_[i], Si_[i])
    plt.contour(x_, y_, rvi.pdf(pos_))

plt.scatter(X_[:,0],X_[:,1] )
plt.title(u"GM Model - 3 Components")
plt.xlabel(u"1st principal component")
plt.ylabel(u"2nd principal component")
```

### M =4

```python
T_ = 50
M_ = 4
a_, Mu_, Si_,_ = EM(X_,M_,T_)
for i in range(M_):
    rvi = mn(Mu_[i], Si_[i])
    plt.contour(x_, y_, rvi.pdf(pos_))

plt.scatter(X_[:,0],X_[:,1], )
plt.title(u"GM Model - 4 Components")
plt.xlabel(u"1st principal component")
plt.ylabel(u"2nd principal component")
```

It seems that the model with 2 and 3 Components are the best adapted for the data.


```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
from sympy.physics.vector import ReferenceFrame
from sympy.physics.vector import curl, divergence

R = ReferenceFrame('R')

VecF = R[1]**2 * R[2] * R.x - R[0]*R[1] * R.y + R[2]**2 * R.z

print(VecF)
```

    R_y**2*R_z*R.x - R_x*R_y*R.y + R_z**2*R.z

```python
Figure = plt.figure()
ax = Figure.gca(projection = '3d')

x, y, z = np.meshgrid(np.arange(-1, 1, 0.2),
                      np.arange(-1, 1, 0.2),
                      np.arange(-1, 1, 0.2))

u = 2*x*y
v = x**2 + 2*z
w = y + 1

ax.quiver3D(x,y,z,u,v,w,length =0.1)

plt.show()
```

```python
# Calculate the Divergence
F = divergence(VecF,R)
F

# Calculate the Curl
G = curl(VecF,R)
G
```

    R_y**2*R.y + (-2*R_y*R_z - R_y)*R.z

```python

```


```python
import numpy as np
```

```python
x = np.ones(5)
```

```python
x
```

    array([1., 1., 1., 1., 1.])

```python
print(x)
```

    [1. 1. 1. 1. 1.]

```python
def f(x):
    return 2+x
```

```python
f
```

    <function __main__.f>

```python
print(f)
```

    <function f at 0x7f8834126598>

```python
repr(f)
```

    '<function f at 0x7f8834126598>'

```python
repr(x)
```

    'array([1., 1., 1., 1., 1.])'

```python
str(x)
```

    '[1. 1. 1. 1. 1.]'

```python
str(f)
```

    '<function f at 0x7f8834126598>'

```python
print(x)
```

    [1. 1. 1. 1. 1.]

```python
x.__str__()
```

    '[1. 1. 1. 1. 1.]'

```python
x.__repr__()
```

    'array([1., 1., 1., 1., 1.])'

```python
import sympy as sm
```

```python
x, y = sm.symbols('x, y')
```

```python
x
```

$\displaystyle x$

```python
x + y
```

$\displaystyle x + y$

```python
repr(x)
```

    'x'

```python
repr(x + y)
```

    'x + y'

```python
str(x + y)
```

    'x + y'

```python
x = np.ones((2, 2))
```

```python
x
```

    array([[1., 1.],
           [1., 1.]])

```python
x[0, 0]
```

    1.0

```python
a = [1, 2, 3]
type(a)
```

    list

```python
a
```

    [1, 2, 3]

```python
y = ['a', 1, 9.0]
```

```python
y
```

    ['a', 1, 9.0]

```python
import numpy as np
```

```python
np.array(y)
```

    array(['a', '1', '9.0'], dtype='<U3')

```python
my_list = []
for i in range(20):
    print(i)
    my_list.append(i**2)
```

    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19

```python
my_list
```

    [0,
     1,
     4,
     9,
     16,
     25,
     36,
     49,
     64,
     81,
     100,
     121,
     144,
     169,
     196,
     225,
     256,
     289,
     324,
     361]

```python
my_array = np.array(my_list)
```

```python
my_array
```

    array([  0,   1,   4,   9,  16,  25,  36,  49,  64,  81, 100, 121, 144,
           169, 196, 225, 256, 289, 324, 361])

```python
my_list * 2
```

    [0,
     1,
     4,
     9,
     16,
     25,
     36,
     49,
     64,
     81,
     100,
     121,
     144,
     169,
     196,
     225,
     256,
     289,
     324,
     361,
     0,
     1,
     4,
     9,
     16,
     25,
     36,
     49,
     64,
     81,
     100,
     121,
     144,
     169,
     196,
     225,
     256,
     289,
     324,
     361]

```python
my_array * 2
```

    array([  0,   2,   8,  18,  32,  50,  72,  98, 128, 162, 200, 242, 288,
           338, 392, 450, 512, 578, 648, 722])

```python
np.random.random(5)
```

    array([0.64327535, 0.64564919, 0.78193309, 0.03636536, 0.37608049])

```python
a = np.random.random(5)
b = np.random.random(5)
```

```python
a
```

    array([0.83327995, 0.2396199 , 0.74695168, 0.06965017, 0.39579413])

```python
b
```

    array([0.42479002, 0.95725464, 0.4073338 , 0.30301574, 0.3486592 ])

```python
np.sqrt(a**2 + b**2)
```

    array([0.93530852, 0.98678982, 0.85079823, 0.31091749, 0.52746206])

```python
for thing in ['a', 'b', 'c']:
    print(thing)
```

    a
    b
    c

```python
for i, thing in enumerate(['a', 'b', 'c']):
    print(i)
    print(thing)
```

    0
    a
    1
    b
    2
    c

```python
from utils2 import period2freq, freq2period
```

```python
period2freq(5.0)
```

    1.2566370614359172

```python
period2freq?
```

    Signature: period2freq(period)
    Docstring: Returns frequency in rad/s given period in seconds.
    File:      ~/eng122-2020w-notebook-02/utils2.py
    Type:      function

```python

```


# Quantificação de elementos acústicos

Neste notebook vamos avaliar a quantificação de diversos elementos acústicos. Entre eles:
- compliância
- tubo aberto-fechado
- tubo aberto-aberto
- tubo aberto-aberto de pequenas dimensões
- impedância de radiação (pistão no baffle infinito)
- impedância de radiação (pistão no cilindro)

```python
import numpy as np
from scipy import special
from matplotlib import pyplot as plt
plt.rcParams.update({'font.size': 14})
```

# 1. Compliância acústica

A compliância está associada a um volume de ar, $V_s$ [m$^{3}$].

\begin{equation}
C_a = \frac{V_s}{\rho_0 c_{0}^{2}}
\end{equation}

```python
def ac_compliance(vol, rho0 = 1.21, c0 = 343):
    Ca = vol/(rho0*c0**2)
    return Ca
```

```python
Vs = 0.6/1000
Ca = ac_compliance(Vs)
print(r'Compliância é {} [m^3/Pa]'.format(Ca))
```

    Compliância é 4.214806488750787e-09 [m^3/Pa]

# 2. Tubo aberto-fechado

Resulta numa compliância em série com uma massa acústica.

\begin{equation}
Z_a = \frac{1}{j \omega}\frac{\rho_0 c_{0}^{2}}{\pi a^2 l'} \ + \ j \omega \frac{\rho_0 l'}{3 \pi a^2}
\end{equation}
em que $l'$ é o comprimento corrigido do tubo e é

- para um tubo flangeado $l'= l + 0.85 a$
- para um tubo não-flangeado $l'= l + 0.61 a$

```python
def ac_octube(a, l, freq, flanged = False, rho0 = 1.21, c0 = 343):
    if flanged:
        ll = l + 0.85*a
    else:
        ll = l + 0.61*a
    # compliance and mass
    Ca = np.pi*(a**2)*ll/(rho0*c0**2)
    Ma = rho0*ll/(3*np.pi*a**2)
    # impedance
    omega = 2 *np.pi*freq
    Za = 1/(1j*omega*Ca) + 1j*omega*Ma
    return Ca, Ma, Za
```

```python
a = 0.01
l = 0.2

freq = np.arange(10, 10000, 1)

Ca, Ma, Za = ac_octube(a, l, freq, flanged = False)

print(r'Compliância é {} [m^3/Pa]'.format(Ca))
print(r'Massa é {} [kg/m^4]'.format(Ma))

plt.figure(figsize=(7,5))
plt.semilogx(freq, np.abs(Za), '-k', linewidth = 1, label = r'$|Z_a|$')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|Z_a|$ [Pa s/m$^3$]')
plt.xlim((freq[0], freq[-1]))
plt.tight_layout()
```

# 3. Tubo aberto-aberto

Resulta em uma massa acústica.

\begin{equation}
Z_a = j \omega \frac{\rho_0 l'}{\pi a^2}
\end{equation}
em que $l'$ é o comprimento corrigido do tubo.

```python
def ac_ootube(a, l, freq, flanged = False, rho0 = 1.21, c0 = 343):
    if flanged:
        ll = l + 0.85*a
    else:
        ll = l + 0.61*a
    # mass
    Ma = rho0*ll/(np.pi*a**2)
    # impedance
    omega = 2 *np.pi*freq
    Za = 1j*omega*Ma
    return Ma, Za
```

```python
a = 0.01
l = 0.06

freq = np.arange(10, 10000, 1)

Ma, Za = ac_ootube(a, l, freq, flanged = False)

print(r'Massa é {} [kg/m^4]'.format(Ma))
```

    Massa é 254.58743006865757 [kg/m^4]

# 4. Tubo aberto-aberto de pequenas dimensões

Resulta em uma resistência acústica em série com uma massa acústica.

\begin{equation}
Z_a = \frac{8 \eta l'}{\pi a^4} \ + \ j \omega \frac{4}{3}\frac{\rho_0 l'}{\pi a^2}
\end{equation}
em que $l'$ é o comprimento corrigido do tubo.

```python
def ac_oo_smalltube(a, l, freq, flanged = False, rho0 = 1.21, c0 = 343, eta = 1.84e-5):
    if flanged:
        ll = l + 0.85*a
    else:
        ll = l + 0.61*a
    # compliance and mass
    Ra = 8*eta*ll/(np.pi*a**4)
    Ma = 4*rho0*ll/(3*np.pi*a**2)
    # impedance
    omega = 2 *np.pi*freq
    Za = Ra + 1j*omega*Ma
    return Ra, Ma, Za
```

```python
a = 0.01
l = 0.06

freq = np.arange(10, 10000, 1)

Ra, Ma, Za = ac_oo_smalltube(a, l, freq, flanged = False)

print(r'Resistência é {} [Pa s/m^3]'.format(Ra))
print(r'Massa é {} [kg/m^4]'.format(Ma))
```

    Resistência é 309.71297277773886 [Pa s/m^3]
    Massa é 339.4499067582101 [kg/m^4]

# 5. Impedância de radiação (pistão no baffle infinito)

Para um pistão no baffle infinito, a impedância de radiação é dada por

\begin{equation}
Z_a = \frac{\rho_0 c_0}{\pi a^2} [1-\frac{2 \ J_1(2k_0 a)}{2k_0 a} + j \frac{2 \ S_1(2k_0 a)}{2k_0 a}]
\end{equation}
em que $J_1(x)$ é a função de Bessel do primeiro tipo e $S_1(x)$ é a função de Struve do primeiro tipo - ambas de ordem 1.

Alternativamente, a impedância de radiação pode ser modelada pelo circuito a seguir.

Com

\begin{equation}
R_{A1} = \large\frac{0.1404\rho_{0}c_{0}}{a^{2}}
\end{equation}

\begin{equation}
R_{A2} = \large\frac{\rho_{0}c_{0}}{\pi a^{2}}
\end{equation}

\begin{equation}
C_{A1} = \large\frac{5.94a^{3}}{\rho_{0}c_{0}^{2}}
\end{equation}

\begin{equation}
M_{A1} = \large\frac{0.27\rho_{0}}{a}
\end{equation}

```python
def zrad_piston_infbaffle(a, freq, rho0 = 1.21, c0 = 343):
    # Equivalent circuit
    Ra1 = (0.1404 * rho0 * c0) / (a**2)
    Ra2 = (rho0 * c0)/(np.pi * a**2)
    Ca1 = 5.94 *a**3/(rho0 * c0**2)
    Ma1 = (0.27 * rho0) / a
    # Equivalent impedance
    omega = 2*np.pi*freq
    Zrad_eq=(-(omega**2)*Ma1*Ra1*Ra2*Ca1+1j*omega*Ma1*(Ra1+Ra2))/\
        (Ra1+Ra2-(omega**2)*Ma1*Ra1*Ca1+1j*omega*(Ra1*Ra2*Ca1+Ma1))

    # Radiation impedance
    k0 = omega/c0;
    #s = pi*a^2;
    Jka = special.jv(1,2*k0*a)
    Ska = special.struve(1,2*k0*a)

    R1 = 1 - 2*Jka/(2*k0*a);
    X1 = 2*Ska/(2*k0*a);

    Zrad = (rho0*c0/(np.pi*a**2))*(R1 + 1j*X1)
    return Ra1, Ra2, Ca1, Ma1, Zrad_eq, Zrad
```

```python
a = (6/2)*(2.54e-2)
freq = np.arange(10, 10000, 1)

Ra1, Ra2, Ca1, Ma1, Zrad_eq, Zrad = zrad_piston_infbaffle(a, freq)

print(r'Ra1 é {} [Pa s/m^3]'.format(Ra1))
print(r'Ra2 é {} [Pa s/m^3]'.format(Ra2))
print(r'Ca1 é {} [m^3/Pa]'.format(Ca1))
print(r'Ma1 é {} [kg/m^4]'.format(Ma1))

plt.figure(figsize=(12,6))
plt.subplot(2,2,1)
plt.semilogx(freq, np.real(Zrad), linewidth = 1, label = 'Exato')
plt.semilogx(freq, np.real(Zrad_eq), linewidth = 1, label = 'Equivalente')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\Re e \{Z_a\}$ [Pa s/m$^3$]')
plt.xlim((freq[0], freq[-1]))

plt.subplot(2,2,2)
plt.semilogx(freq, np.imag(Zrad), linewidth = 1)
plt.semilogx(freq, np.imag(Zrad_eq), linewidth = 1)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\Im m \{Z_a\}$ [Pa s/m$^3$]')
plt.xlim((freq[0], freq[-1]))

plt.subplot(2,2,3)
plt.semilogx(freq, np.abs(Zrad), linewidth = 1)
plt.semilogx(freq, np.abs(Zrad_eq), linewidth = 1)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|Z_a|$ [Pa s/m$^3$]')
plt.xlim((freq[0], freq[-1]))

plt.subplot(2,2,4)
plt.semilogx(freq, np.angle(Zrad), linewidth = 1)
plt.semilogx(freq, np.angle(Zrad_eq), linewidth = 1)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\angle Z_a$ [rad]')
plt.xlim((freq[0], freq[-1]))
plt.tight_layout()
```

# 5. Impedância de radiação (pistão na extremidade de um cilindro)

Para um pistão na extremidade de um cilindro, a impedância pelo mesmo circuito do exemplo anterior, com

\begin{equation}
R_{A1} = \large\frac{0.504\rho_{0}c_{0}}{\pi a^{2}}
\end{equation}

\begin{equation}
R_{A2} = \large\frac{\rho_{0}c_{0}}{\pi a^{2}}
\end{equation}

\begin{equation}
C_{A1} = \large\frac{5.44a^{3}}{\rho_{0}c_{0}^{2}}
\end{equation}

\begin{equation}
M_{A1} = \large\frac{0.1952\rho_{0}}{a}
\end{equation}

```python
def zrad_cylinder(a, freq, rho0 = 1.21, c0 = 343):
    # Equivalent circuit
    Ra1 = (0.504 * rho0 * c0) / (np.pi * a**2)
    Ra2 = (rho0 * c0) / (np.pi * a**2)
    Ca1 = (5.44 * a**3) / (rho0 * c0**2)
    Ma1 = (0.1952 * rho0) / a
    # Equivalent impedance
    omega = 2*np.pi*freq
    Zrad_eq=(-(omega**2)*Ma1*Ra1*Ra2*Ca1+1j*omega*Ma1*(Ra1+Ra2))/\
        (Ra1+Ra2-(omega**2)*Ma1*Ra1*Ca1+1j*omega*(Ra1*Ra2*Ca1+Ma1))

    return Ra1, Ra2, Ca1, Ma1, Zrad_eq
```

```python
a = 0.5*(1/2)*(2.54e-2)
freq = np.arange(10, 10000, 1)

Ra1, Ra2, Ca1, Ma1, Zrad_eq = zrad_cylinder(a, freq)

print(r'Ra1 é {} [Pa s/m^3]'.format(Ra1))
print(r'Ra2 é {} [Pa s/m^3]'.format(Ra2))
print(r'Ca1 é {} [m^3/Pa]'.format(Ca1))
print(r'Ma1 é {} [kg/m^4]'.format(Ma1))

plt.figure(figsize=(12,6))
plt.subplot(2,2,1)
plt.semilogx(freq, np.real(Zrad_eq), linewidth = 1, label = 'Equivalente')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\Re e \{Z_a\}$ [Pa s/m$^3$]')
plt.xlim((freq[0], freq[-1]))

plt.subplot(2,2,2)
plt.semilogx(freq, np.imag(Zrad_eq), linewidth = 1)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\Im m \{Z_a\}$ [Pa s/m$^3$]')
plt.xlim((freq[0], freq[-1]))

plt.subplot(2,2,3)
plt.semilogx(freq, np.abs(Zrad_eq), linewidth = 1)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$|Z_a|$ [Pa s/m$^3$]')
plt.xlim((freq[0], freq[-1]))

plt.subplot(2,2,4)
plt.semilogx(freq, np.angle(Zrad_eq), linewidth = 1)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [Hz]')
plt.ylabel(r'$\angle Z_a$ [rad]')
plt.xlim((freq[0], freq[-1]))
plt.tight_layout()
```

# Ex.2 Muffler

```python
freq = np.arange(10, 10000, 1)
# Câmara de expansão
a1 = 0.03
S1=np.pi*a1**2
l1=0.05
VA1=S1*l1
Ca = ac_compliance(VA1)
print(r'Compliância é {} [m^3/Pa]'.format(Ca))

# Tubo de entrada
at1=0.005
lt1=0.05
Rat1, Mat1, _ = ac_oo_smalltube(at1, lt1, freq, flanged = False)
print(r'Rat1 é {} [Pa s/m^3]'.format(Rat1))
print(r'Mat1 é {} [kg/m^4]'.format(Mat1))

# Tubo de saída
at2=0.005
lt2=0.05
Rat2, Mat2, _ = ac_oo_smalltube(at2, lt2, freq, flanged = False)

Rr1, Rr2, Cr1, Mr1, _ = zrad_cylinder(at2, freq)
print(r'Rat2 é {} [Pa s/m^3]'.format(Rat2))
print(r'Mat2 é {} [kg/m^4]'.format(Mat2))

print(r'Rr1 é {} [Pa s/m^3]'.format(Rr1))
print(r'Rr2 é {} [Pa s/m^3]'.format(Rr2))
print(r'Cr1 é {} [m^3/Pa]'.format(Cr1))
print(r'Mr1 é {} [kg/m^4]'.format(Mr1))

```

# Caixa dutada

```python
freq = np.arange(10, 10000, 1)

# Volumes
VA1 = 6/1000
Ca1 = ac_compliance(VA1)
print(r'Ca1 é {} [m^3/Pa]'.format(Ca1))

VA2 = 18/1000
Ca2 = ac_compliance(VA2)
print(r'Ca2 é {} [m^3/Pa]'.format(Ca2))

# Tubo
at=0.04
lt=0.2
Rat, Mat, Zeq = ac_oo_smalltube(at, lt, freq, flanged = False)

Rr1, Rr2, Cr1, Mr1, _, _ = zrad_piston_infbaffle(at, freq)
print(r'Rat2 é {} [Pa s/m^3]'.format(Rat))
print(r'Mat2 é {} [kg/m^4]'.format(Mat))

print(r'Rr1 é {} [Pa s/m^3]'.format(Rr1))
print(r'Rr2 é {} [Pa s/m^3]'.format(Rr2))
print(r'Cr1 é {} [m^3/Pa]'.format(Cr1))
print(r'Mr1 é {} [kg/m^4]'.format(Mr1))
```

    Ca1 é 4.214806488750787e-08 [m^3/Pa]
    Ca2 é 1.2644419466252358e-07 [m^3/Pa]
    Rat2 é 4.107152461429451 [Pa s/m^3]
    Mat2 é 72.02397794680631 [kg/m^4]
    Rr1 é 36418.88249999999 [Pa s/m^3]
    Rr2 é 82567.59503928665 [Pa s/m^3]
    Cr1 é 2.670501391272499e-09 [m^3/Pa]
    Mr1 é 8.1675 [kg/m^4]


```python
import numpy as np
import math
from scipy import stats
import matplotlib as mpl
import matplotlib.pyplot as plt
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
%matplotlib inline
plt.style.use('seaborn-whitegrid')
mpl.style.use('seaborn')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
import Abatement_functions as func
```

# Abatement Project: Adding end-of-pipe technologies to a CGE

Consider the simple setup:
* A sector uses inputs $(x,y)$ in production of a commodity/service $H$.
* In using the input $x$ there is an associated emission of something 'bad'. Let $e^0$ denote emissions prior to abatement and $e$ actual emissions.
* There is a pre-abatement known emission coefficient in the use of $x$. Denote this $\eta$. Thus we have the simple relation:
    $$\begin{align}
    e^0 = \eta x^0,
    \end{align}$$
    where $x^0$ is the use of the input $x$, in absence of any regulation on emissions.
* Emissions are taxed at a rate $\tau$.

In general we think of three abatement channels:

* **Output reduction**: Given input-mix and abatement technology, a lower output lowers the use of the dirty input $x$ and thus emissions.
* **Input substitution**: Given the output level and abatement technology, input can be mixed with lower use of $x$ (consequently more use of $y$ here), emissions can be lowered.
* **Abatement technology**: Given the output level and input-mix, targeted abatement equipment can lower emissions directly at some cost.

We can further think of **abatement technology** in two different ways:

* **End-of-pipe abatement:** Lowers emissions without altering the optimal input-mix (think of a filter on an engine that burns fossil fuels).
* **Input-displacing abatement technology:** A discrete change in technology that alters the production function altogether (i.e. altering a production process).

In the following we will refer to *abatement technology* as an *end-of-pipe* type of technology, as the *input-displacing* type would involve a discrete choice between different production functions, which presents more of a challenge.

## 1: Abatement as discrete set of end-of-pipe technologies

### 1.1: Representing technology data

We assume that a technology dataset gives $T$ ways of abating emissions. Each technology $t$ is summed up by the information:
* $q_t \in [0,1]$ is the share of emissions that are abated, if the equipment is installed.
* $c_t\in R_+$ is unit cost of abating emissions.

This gives rise to the optimal abatement $(A)$ as a function of the tax rate $(\tau)$, as well as the total abatement costs $(C)$:

$$\begin{align}
    A(\tau) &= \eta x \sum_{t=1}^{T} q_t * \mathbf{1}_{c_t<\tau}, && \mathbf{1}_{c_t<\tau} = \left\lbrace \begin{array}{ll} 0, & c_t\geq \tau \\ 1, & c_t<\tau \end{array} \right.  \tag{D1}\\
    C(\tau) &= \eta x \sum_{t=1}^{T} q_t c_t * \mathbf{1}_{c_t<\tau} \tag{D2}
\end{align}$$

**Draw and plot a data set for technologies:**

```python
T = 10; # Number of technologies
abate_upper = 0.9 # Upper limit of total abatement (in percentages)
c_upper = 1 # Most expensive abatement technology
seed=21 # Set seed so the random draw does not change every time we run this cell.
labels = ['Tax on emissions', 'Abatement'] # Names on axes in plot below.
sample_c, sample_q_sum, sample_q = func.draw_sample(T,abate_upper,c_upper,seed) # draw samples of unit costs and abatement potential.
```

```python
func.plot_stepcurve(sample_c,sample_q_sum,labels) # Plot abatement function.
```

### 1.2: Smoothing out the step-function

In an older version we told a story about how this step function looks like, if instead of constant unit costs of abatement $c_t$, there were a continuum of heterogeneous firms applying the abatement technologies. Either through difference in abilities or compatibility with the technology, we can think of the heterogeneity resulting in varying unit costs $c_t^i$ for each $i$ firm in the continuum. Alternatively we can think of it as a *smoothing device*, facilitating the use of gradient-based solvers.

The idea is to replace the abatement and cost functions with a smoothed stepwise curve of the form:

$$\begin{align}
    A_t(\tau) &= \eta x q_t \int_{\underline{s}}^{\tau} dF(c_t^i), \tag{S1} \\
              &= \eta x q_t F_t(\tau) \\
    C_t(\tau) &= \eta x q_t \int_{\underline{s}}^{\tau} c_t^i dF(c_t^i), \tag{S2} \\
              &= \eta x q_t F_t(\tau) \mathbb{E}\left[c_t^i|c_t^i<\tau\right].
\end{align}$$

where $F(\cdot)$ represents some continuously differentiable distribution with support on $[\underline{s},\bar{s}]$ (bounded support is not a necessary condition). Relevant functions that approximate the step-shape well are the *normal* and *log-normal* distributions.

In the following we use the log-normal distribution in which case the functions are given by:

$$\begin{align}
    ln(c_t^i) &\sim N\left(ln(c_t)-\dfrac{\sigma^2}{2},\mbox{ }\sigma^2\right) \tag{S3} \\
    A_t(\tau) &= \eta x q_t \underbrace{\Phi\left(\dfrac{ln(\tau)-ln(c_t)+\sigma^2/2}{\sigma}\right)}_{F_t^q(\tau)} \tag{S4} \\
    C_t(\tau) &= \eta x q_t c_t \underbrace{\Phi\left(\dfrac{ln(\tau)-ln(c_t)-\sigma^2/2}{\sigma}\right)}_{F_t^c(\tau)} \tag{S5}
\end{align}$$

(Alternatively we could use the normal distribution. This has some nice properties when it comes to the $A_t(\tau)$ function, but is a more complicated function for $C_t(\tau)$. There are several other readily available smoothing functions out there though.)

**Plot the smoothed function varying $\sigma$**

```python
n_tau_grid = 500
n_sigma_grid = 100;
sigma_grid = np.linspace(0.0001,0.25,n_sigma_grid)
tau_grid = np.linspace(0.01,c_upper*1.1,n_tau_grid)
A_share, C_share = func.smooth_sample(sample_c,tau_grid,sigma_grid)
```

```python
A = np.sum((A_share * sample_q[:,None,None]), axis=0)
func.interactive_sigma(A,tau_grid,sigma_grid,int((n_sigma_grid/2)),'Abatement')
```

    interactive(children=(FloatSlider(value=0.1263121212121212, description='$\\sigma$', max=0.25, min=0.0001, ste…

Note that with a log-normal distribution, the $\sigma$ values should not be constant across technologies, as there is a clear skewness in the smoothing (more smooth for larger $\tau$).

## 2: Application in the world's simplest CGE model

### 2.1: The Setup:

In a shocking twist of events we learn that the production function of the polluting firm is of the CES type. With abatement technology and cost functions as discussed above, the profit maximization problem that the firm is facing is given by:

$$\begin{align}
    \max_{x,y,H}\mbox{ }\Pi = pH - \tau e -C-p_xx-p_yy, \tag{CES-1}
\end{align}$$

subject to the constraints:

$$\begin{align}
 H =& \left(\mu_x^{1/\theta}x^{\frac{\theta-1}{\theta}}+\mu_y^{1/\theta}y^{\frac{\theta-1}{\theta}}\right)^{\frac{\theta}{\theta-1}} \tag{CES-2} \\
 e =& \eta x \left(1-\sum_{t=1}^T q_tF_t^q(\tau) \right) \tag{CES-3} \\
 C =& \eta x \sum_{t=1}^T q_tc_tF_t^c(\tau) \tag{CES-4}
\end{align}$$

The resulting first order conditions for optimality are then given by:

$$\begin{align}
    x &= \mu_x H \left(\dfrac{p}{\hat{p}_x}\right)^{\theta} \tag{CES-5} \\
    y &= \mu_y H \left(\dfrac{p}{p_y}\right)^{\theta} \tag{CES_6} \\
    \hat{p}_x &= p_x+\eta\left(\tau-\sum_{t=1}^Tq_t\Big[\tau F_t^q(\tau)-c_tF_t^c(\tau)\Big]\right) \tag{CES-7}
\end{align} $$

along with (CES-2)-(CES-4).

### 2.2: Emission taxes increases adjusted relative price of applying dirty input

The additional effects on the relevant price on the use of the dirty input $x$ is now:
1. Emission tax on marginal pollution of size $\eta \tau$.
2. A gain (relative to paying the full tax on all emissions), from applying profitable abatement equipment:

$$\begin{align}
    \text{Abatement gain } = \eta \sum_{t=1}^Tq_t\Big[\tau F_t^q(\tau)-c_t F_t^c(\tau)\Big]\geq 0.
\end{align}$$
The inequality comes from the fact that

$$\begin{align}
    c_tF_t^c(\tau) = \mathbb{E}\left[c_t|c_t<\tau\right] F_t^q(\tau),
\end{align}$$

which per construction is lower than $\tau F_t^q(\tau)$. However, it is also straightforward to see that the higher the tax $\tau$, the higher is the 'adjusted' price on the dirty input:

$$\begin{align}
    \hat{p}_x \geq p_x, && \text{as } && \tau \geq \sum_{t=1}^T q_t\Big[\tau F_t^q(\tau)-c_tF_t^c(\tau)\Big].
\end{align}$$

This follows from the fact that:

$$\begin{align}
    \sum_t^T q_t \leq& 1
\end{align}$$

and

$$\begin{align}
   0\leq  \tau F_t^q(\tau)-c_tF_t^c(\tau) = F_t^q(\tau)\left(\tau-\mathbb{E}\left[c_t|c_t<\tau\right]\right) \leq \tau.
\end{align}$$

### 2.3: The abatement gain is fully crowded out by adjusted output prices, i.e. there are still zero profits with perfect competition

Evaluating the profit function in (CES-1), substituting initially for (CES-5) and (CES-6) yields:

$$\begin{align}
    \Pi^*   &= pH-\tau e-C-\mu_x H p^{\theta} \dfrac{p_x}{(\hat{p}_x)^{\theta}}-\mu_y H p^{\theta}p_y^{1-\theta} \\
            &= H\left[p-p^{\theta}\left(\mu_x \dfrac{p_x}{\left(\hat{p}_x\right)^{\theta}}+\mu_yp_y^{1-\theta}\right)\right]-\tau e- C
\end{align}$$

Focus on the last part concerning emissions. Substitute for (CES-3)-(CES-4) (abatement functions) to get:

$$\begin{align}
    \tau e + C &= \tau \eta x \left(1-\sum_{t=1}^T q_tF_t^q (\tau)\right)+\eta x \sum_{t=1}^T q_tc_tF_t^c(\tau). \\
                &= \eta x \left(\tau - \sum_{t=1}^T q_t \left[\tau F_t^q(\tau)-c_tF_t^c(\tau)\right]\right)
\end{align}$$

Using (CES-7) we can rewrite this as:

$$\begin{align}
    \tau e + C = x \left(\hat{p}_x-p_x\right).
\end{align}$$

Plugging this back into the expression for $\Pi^*$ we then have:

$$\begin{align}
    \Pi^* = H\left[p-p^{\theta}\left(\mu_x\dfrac{p_x}{(\hat{p}_x)^{\theta}}+\mu_yp_y^{1-\theta}\right)\right]-x\left(\hat{p}_x-p_x\right).
\end{align}$$

Finally substituting for $x$ using (CES-5) this yields the maximized profit function:

$$\begin{align}
    \Pi^* &= H\left[p-p^{\theta}\left(\mu_x\dfrac{p_x}{(\hat{p}_x)^{\theta}}+\mu_yp_y^{1-\theta}\right)\right]-\mu_x H \left(\dfrac{p}{\hat{p}_x}\right)^{\theta}\left(\hat{p}_x-p_x\right) \\
    & = H\left[p-p^{\theta}\left(\mu_x\left(\hat{p}_x\right)^{1-\theta}+\mu_y\left(p_y\right)^{1-\theta}\right)\right]
\end{align}$$

With the *usual* CES price index of (Combine production function with the (CES-5)-(CES-6) to show)

$$\begin{align}
    p = \left(\mu_x\left(\hat{p}_x\right)^{1-\theta}+\mu_y\left(p_y\right)^{1-\theta}\right)^{\frac{1}{1-\theta}},
\end{align}$$

this yields exactly zero profits $\Pi^* = 0$.

## 3: Further analysis and plots

Here we investigate and plot a number of the functions derived above:

* The *abatement gain* function,
* The *adjusted relative price on dirty good*,
* Total emissions and abatement, (more interesting things?)

all as a function of the tax rate $\tau$, and with interactive features for parameters $(\mu_x,\theta,\sigma)$.

### 3.1: Abatement gain function

Start with settings:

```python
# Fixed values:
eta,px,py,muy = 2,5,4,1
# Grids:
n_tau_grid,n_sigma_grid,n_mux_grid,n_theta_grid = 100,10,6,6
tau_grid = np.linspace(0.01,c_upper*1.1,n_tau_grid)
sigma_grid = np.linspace(0.0001,0.25,n_sigma_grid)
mux_grid = np.linspace(0.5, 2.5,n_mux_grid)
theta_grid = np.linspace(0.25, 1.25, n_theta_grid)
```

Used already drawn abatement technology data set. Recall only $(\sigma)$ parameter enters the function. Note that the abatement gain function per construction is lower than $\tau$.

```python
A_share, C_share = func.smooth_sample(sample_c,tau_grid,sigma_grid) # 3-dimensions: Technology (T), tau grid, sigma grid.
A = np.sum(( A_share * sample_q[:,None,None] * tau_grid[None,:,None]), axis=0)
C = np.sum(( C_share * sample_q[:,None,None] * sample_c[:,None,None]), axis=0)
Abatement_gain = eta*(A-C)
func.interactive_sigma(Abatement_gain,tau_grid,sigma_grid,int((n_sigma_grid/2)),'Abatement gain')
```

    interactive(children=(FloatSlider(value=0.13893333333333333, description='$\\sigma$', max=0.25, min=0.0001, st…

### 3.2: The Relative price on the 'dirty' input factor

Recall that the price was given by:

$$\begin{align}
\hat{p}_x &= p_x+\eta\left(\tau-\sum_{t=1}^Tq_t\Big[\tau F_t^q(\tau)-c_tF_t^c(\tau)\Big]\right) = p_x + \eta \tau - \text{Abatement gain}
\end{align}$$

```python
px_hat = px + eta * tau_grid[:,None]-Abatement_gain
func.interactive_sigma(px_hat,tau_grid,sigma_grid,int((n_sigma_grid/2)),'$\hat{p}_x$')
```

    interactive(children=(FloatSlider(value=0.13893333333333333, description='$\\sigma$', max=0.25, min=0.0001, st…

### 3.3: Emissions and abatement