# Lab 4 (Part A): Regularized Linear Regression

<div class="alert alert-block alert-danger">

__IMPORTANT__
Please complete this Jupyter Notebook file and upload it to blackboard __before 20 February 2020__.
</div>

In this Lab, you will implement regularized linear regression and use it to study models with different bias-variance properties. Before starting, we strongly recommend reading the slides of lectures 4.1 and 4.2.

In this part of the Lab, you will first implement regularized linear regression to predict the amount of water owing out of a dam using the change of water level in a reservoir. Then, you will examine the effects of bias vs. variance.

## Loading the data
We have a file `water-level-dataset.mat` which contains the dataset for our linear regression problem. The `.mat` file can be loaded in Python using `scipy.io.loadmat(..)` which returns a dictionary that contains 6 arrays. The follwing Python code load these arrays into the variables: `X`, `y`, `Xtest`, `ytest`, `Xval`, `yval`. So, our data is divided into three parts:
- A training set that your model will learn on: `X`, `y`
- A cross validation set for determining the regularization parameter: `Xval`, `yval`
- A test set for evaluating performance. These are *unseen* examples which your model did not see during training: `Xtest`, `ytest`.

Read the following code and print a small subset of these arrays to see what they look like.

```python
%matplotlib notebook
%matplotlib inline
from scipy.io import loadmat

mat = loadmat("datasets/water-level-dataset.mat")

# X and y correspond to a training set that your model will learn on.
X = mat["X"]
y = mat["y"].reshape(len(X))

# Xval and yval correspond to a cross validation set for determining the regularization parameter.
Xval = mat["Xval"]
yval = mat["yval"].reshape(len(Xval))

# Xtest and ytest correspond to a test set for evaluating performance. These
# are unseen examples which your model will not see during training
Xtest = mat["Xtest"]
ytest = mat["ytest"].reshape(len(Xtest))

""" TODO:
You can print here a small subset of X, y, Xtest, ytest, Xval and Xval to see what they look like.
"""
# ...
print("X:",X[:5], "\ny:",y[:5], "\nXtest:",Xtest[:5], "\nytest:",ytest[:5], "\nXval:",Xval[:5], "\nyval:",yval[:5])
```

    X: [[-15.93675813]
     [-29.15297922]
     [ 36.18954863]
     [ 37.49218733]
     [-48.05882945]]
    y: [ 2.13431051  1.17325668 34.35910918 36.83795516  2.80896507]
    Xtest: [[-33.31800399]
     [-37.91216403]
     [-51.20693795]
     [ -6.13259585]
     [ 21.26118327]]
    ytest: [ 3.31688953  5.39768952  0.13042984  6.1925982  17.08848712]
    Xval: [[-16.74653578]
     [-14.57747075]
     [ 34.51575866]
     [-47.01007574]
     [ 36.97511905]]
    yval: [ 4.17020201  4.0672628  31.87306758 10.62365619 31.83602128]

## Visualizing the data
We will begin by visualizing the dataset containing historical records on the change in the water level, $x$, and the amount of water owing out of the dam, $y$. Note that the input data $X$ consists of only one feature corresponding to the "change in water level". Produce a scatter plot showing this feature vs. the output corresponding to the "water flowing out of the dam". Your plot should look like the following figure:

```python
import matplotlib.pylab as plt

""" TODO:
Using the training dataset X and y, produce a scatter plot showing the "change in water level" on
the x-axis and the "water flowing out of the dam" on the y-axis, as shown in the previous figure.
"""
# ...
fig, ax = plt.subplots()
ax.scatter(X, y)

ax.set_xlabel = "Change in water level"

ax.set_ylabel = "Water flowing out of the dam"

ax.set_title  = "This is a plot"
plt.legend()

```

## Adding a first column of ones to the dataset
As usual, before starting the implementation of our linear regression model, it might be helpful to use a modified version of our dataset which has an additional first column of ones.

```python
import numpy as np

# This function takes a matrix as argument and returns a new matrix with an additional first column (of ones)
def add_all_ones_column(X):
    n, d = X.shape # dimension of the matrix X (n lines, d columns)
    XX = np.ones((n, d+1)) # new matrix of all ones with one additional column
    XX[:, 1:] = X # set X starting from column 1 (keep only column 0 unchanged)
    return XX

# The following line creates a new data matrix X_new with an additional first column (of ones)
X_new = add_all_ones_column(X) # for the training set
Xval_new = add_all_ones_column(Xval) # for the validation set

""" TODO:
You can print a small subset of X_new here to see how it looks like
"""
# ...
print(X_new[:5])
```

    [[  1.         -15.93675813]
     [  1.         -29.15297922]
     [  1.          36.18954863]
     [  1.          37.49218733]
     [  1.         -48.05882945]]

## Regularized linear regression cost function
Recall that regularized linear regression has the following cost function:
$$E(\theta) = \frac{1}{2n} \left ( \sum_{i=1}^{n} ( h_\theta(x^{(i)}) - y^{(i)} )^2 \right ) + \frac{\lambda}{2n} \left ( \sum_{j=1}^{n} \theta_j^2 \right ),$$
where $\lambda$ is a regularization parameter which controls the degree of regularization (thus, help preventing overfitting). The regularization term puts a penalty on the overal cost function $E$. As the magnitudes of the model parameters $\theta_j$ increase, the penalty increases as well. Note that you should not regularize the $\theta_0$ term.

Complete the following code to write a function to calculate the regularized linear regression cost function. If possible, try to vectorize your code and avoid writing loops. When you are finished, call your cost function using $\theta$ initialized at `np.array([1, 1])` and $\lambda = 1$. You should then expect to see an output of about $303.993$.

```python
""" TODO:
Write the definition of the regularized linear regression cost function.
If possible, try to vectorize your code and avoid writing loops. The
last argument lmd corresponds to the regularization term lambda.
"""
n = len(X)

def h(theta, x):
    return theta.T @ x

def E(theta, X, y, lmd):
    return (1/(2*n))*((h(theta,X.T) - y) @ (h(theta,X.T) - y).T) + (lmd/(2*n))*theta[1:]@ theta[1:]

""" TODO:
Call your cost function with theta initialized to an array of ones, and
lambda = 1. You should expect to see an output of about 303.993.
"""
theta = np.array([1, 1])
lmd = 1
print(E(theta, X_new,y, lmd))
```

    303.9931922202643

## Regularized linear regression gradient
The partial derivative of regularized linear regression's cost for $\theta_j$ is defined as:
$$
\frac{\partial E(\theta)}{\partial \theta_0} = \frac{1}{n} \sum_{i=1}^{n} ( h_\theta(x^{(i)}) - y^{(i)} ) ~ x_j^{(i)}
\quad \quad \quad \quad \quad \quad \text{for } j = 0
$$

$$
\frac{\partial E(\theta)}{\partial \theta_j} = \left ( \frac{1}{n} \sum_{i=1}^{n} ( h_\theta(x^{(i)}) - y^{(i)} ) ~ x_j^{(i)} \right ) + \frac{\lambda}{n} \theta_j
\quad \quad \text{for } j \geq 1
$$

Complete the definition of the function `gradE(..)` in the code below, to calculate the gradient of the cost function $\nabla E(\theta)$. The function must return an array of the same length as $\theta$. When you are finished, call your gradient function using $\theta$ initialized at `np.array([1, 1])` and $\lambda = 1$. You should then expect to see a gradient of `[-15.30, 598.250]`.

```python
""" TODO:
Write the definition of the gradient function. It should return an array containing
the derivative of the cost function with respect to each parameter theta[j].
"""
def gradE(theta, X, y, lmd):
    return (1/n)*(h(theta,X.T)-y)@X + (lmd/n)*np.array([0] + list(theta[1:]))

""" TODO:
Call your gradient function with theta initialized to an array of ones, and
lambda = 1. You should expect to see a gradient of approximatly [-15.30  598.250].
"""
np.array([1, 1])
lmd = 1
print( gradE(theta, X_new, y, lmd) )
```

    [-15.30301567 598.25074417]

## Fitting linear regression
Once your cost function and gradient are working correctly, the following code will help you to compute the optimal values of $\theta$. We use `scipy.optimize.minimize(..)` to optimize the cost function.

In this part, we set the regularization parameter $\lambda$ to zero. Because our current implementation of linear regression is trying to fit a 2-dimensional $\theta$, regularization will not be incredibly helpful for a $\theta$ of such low dimension. In the later sections of this Lab, you will be using polynomial regression with regularization.

For more information about `scipy.optimize.minimize(..)`, you can visit:
- https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize
- https://docs.scipy.org/doc/scipy/reference/optimize.html

```python
import scipy.optimize as op

theta = np.array([0, 0])  # Some initial parameters vector
lmd = 0 # We set lambda to zero this time.
print("Initial cost: ", E(theta, X_new, y, lmd))

""" TODO:
Use op.minimize(..) to minimize the cost function E. Then, print the
optimal parameter vector theta and the final cost.
"""
res = op.minimize(E, theta, (X_new, y, lmd), 'TNC', gradE)
# Then, the optimal parameters can be accessed with: theta = res.x
theta = res.x
print("theta:", theta)
```

    Initial cost:  140.95412088055392
    theta: [13.08790351  0.36777923]

## Plotting the linear regression model
One you get the optimal parameters of $\theta$, you can call the function `plot_linear_fit(X, y, theta)` defined in the code below to plot the original dataset and the best fit line. Read the code carefully to see how such a line is ploted using the $\theta$ values. You will get a plot similar to the follwing figure.

The best fit line tells us that the model is not a good fit to the data because the data has a non-linear pattern. While visualizing the best fit as shown is one possible way to debug your learning algorithm, it is not always easy to visualize the data and model. In the next section, you will implement a function to generate learning curves that can help you debug your learning algorithm even if it is not easy to visualize the data.

```python
# This is a function that plots the original dataset (X, y) and the best fit line:
def plot_linear_fit(X, y, theta):
    fig, ax = plt.subplots()

    # Plottin the training data:
    ax.scatter(X[:, 0], y, marker="x", color="red")
    ax.set_xlabel("Change in water level (x)")
    ax.set_ylabel("Water flowing out of the dam (y)")

    # Plotting the line:
    x_min, x_max = np.min(X[:, 0]), np.max(X[:, 0])
    plot_x = np.arange(x_min, x_max) # range of values for the x axis
    plot_y = theta[0] + theta[1] * plot_x
    ax.plot(plot_x, plot_y, color="green", label="Best fit line")

    ax.set_title("Plot of the training data and best fit line")
    plt.legend()
    fig.show()

""" TODO:
Call the function plot_linear_fit(X, y, theta) with
the optimal theta parameters that you got previously.
"""
plot_linear_fit(X, y, theta)
```

## Bias-variance
An important concept in machine learning is the bias-variance tradeoff. Models with high bias are not complex enough for the data and tend to underfit, while models with high variance overfit to the training data. In this part of the Lab, you will plot training and test errors on a learning curve to diagnose bias-variance problems.

You will now implement code to generate the learning curves that will be useful in debugging learning algorithms. Recall that a learning curve plots training and cross validation error as a function of training set size. Your job is to fill in the code below so that it returns a vector of errors for the training set and cross validation set.

To plot the learning curve, we need a training and cross validation set error for different training set sizes. To obtain different training set sizes, you should use different subsets of the original training set $X$. Specifically, for a training set size of $i$, you should use the first $i$ examples (i.e., `X[1:i]` and `y[1:i]`).

You can use the `scipy.optimize.minimize(..)` function to find the $\theta$ parameters. After learning the $\theta$ parameters, you should compute the error on the training and cross validation sets. Recall that the training error for a dataset is defined as:

$$E(\theta) = \frac{1}{2n} \sum_{i=1}^{n} ( h_\theta(x^{(i)}) - y^{(i)} )^2$$

In particular, note that the training error does not include the regularization term. One way to compute the training error is to use your existing cost function `E(..)` and set $\lambda$ to $0$ only when using it to compute the training error and cross validation error. When you are computing the training set error, make sure you compute it on the training subset (i.e., `X[0:i]` and `y[0:i]`) (instead of the entire training set). However, for the cross validation error, you should compute it over the entire cross validation set. You should store the computed errors in the vectors error train and error val.

When you are finished, print the learning curves and produce a plot similar to the following figure. From the figure, you can observe that both the train error and cross validation error are quite high even when the number of training examples is increased. This reflects a **high bias** problem in the model - the linear regression model is too simple and is unable to fit our dataset well. In the next section, you will implement polynomial regression to fit a better model for this dataset.

```python
# This function takes some training data, an initial theta and a regularization term
# lmd, and returns the optimal prameters vector theta of a linear regression. It uses
# scipy.optimize.minimize(..), the cost function E(..) and the gradient gradE(..)
def trainLinearReg(X, y, theta_init, lmd):
    res = op.minimize(E, theta_init, (X, y, lmd), 'TNC', gradE)
    return res.x # the best/final parameters vector theta

""" TODO:
Inside a loop which iterates over the range of n (number of training examples), find the optimal theta by
calling trainLinearReg(..) using the first i examples (X_new[:i], y[:i]), then compute the corresponding
training error err_train (using X_new[:i], y[:i]) and the validation error err_val (using Xval_new, yval) ...
"""
theta = [1,1]
lmd = 0
errs_train, errs_val = [], [] # Lists to save the training and validation errors
for i in range(2, len(y)): # Start from 2 examples at least

    # TODO: find the optimal theta parameters using the i first training examples
    theta = trainLinearReg(X_new[:i], y[:i],theta, lmd )

    # TODO: compute the training error using the optimal theta and the i first training examples
    errs_train.append(E(theta, X_new[:i], y[:i], lmd))

    # TODO: compute the validation error using the optimal theta and the validation dataset
    errs_val.append(E(theta, Xval_new, yval, lmd))

""" TODO:
Complete the code below to plot the learning curve
"""
fig, ax = plt.subplots()

# TODO: plot the "number of training examples" vs the training errors
ax.plot(range(2, len(y)),errs_train ,label="Training error")

# TODO: plot the "number of training examples" vs the validation errors
ax.plot(range(2, len(y)), errs_val,label="Validation error")

ax.set_xlabel("Number of training examples")
ax.set_ylabel("Error")
ax.set_title("Linear regression learning curve")
ax.legend()
fig.show()
```

## Polynomial regression
The problem with our linear model was that it was too simple for the data and resulted in underfitting (high bias). In this part of the exercise, you will address this problem by adding more features. For using polynomial regression, our hypothesis has the form:
$$
\begin{align}
h_\theta(x) &= \theta_0 + \theta_1 * (\text{waterLevel}) + \theta_2 * (\text{waterLevel})^2 + \dots + \theta_p * (\text{waterLevel})^p \\
&= \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_p x_p
\end{align}
$$
Notice that by defining $x_1 = (\text{waterLevel}), x_2 = (\text{waterLevel})^2, \dots, x_p = (\text{waterLevel})^p$, we obtain a linear regression model where the features are the various powers of the original value (waterLevel). Note that you don't have to account for the zero'th power in this function.

Now, you will add more features using the higher powers of the existing feature $x$ in the dataset. Your task in this part is to complete the following code so that the function `polyFeatures(..)` maps the original training set $X$ of size $n \times 1$ into its higher powers. Specifically, when a training set $X$ of size $n \times 1$ is passed into the function, the function should return a $n \times p$ matrix `X_poly`.

```python
""" TODO:
Write the definition of the function polyFeatures(X, p) which takes
as arguement the original training set X and the degree of the polynomial p.
The function should return a new set X_poly of p features (i.e. matrix of len(X) rows and p columns).
"""
def polyFeatures(X, p):
    return np.array([[x**(i+1) for i in range(p)] for x in X[:,0]])

""" TODO:
You can call your polyFeatures(...) function here on some exampls to check if it works correctly.
"""
print(polyFeatures(X[:5],3))
```

    [[-1.59367581e+01  2.53980260e+02 -4.04762197e+03]
     [-2.91529792e+01  8.49896197e+02 -2.47770062e+04]
     [ 3.61895486e+01  1.30968343e+03  4.73968522e+04]
     [ 3.74921873e+01  1.40566411e+03  5.27014222e+04]
     [-4.80588295e+01  2.30965109e+03 -1.10999128e+05]]

After you have completed the function `polyFeatures`, we will proceed to train polynomial regression using your linear regression cost function. Keep in mind that even though we have polynomial terms in our feature vector, we are still solving a linear regression optimization problem. The polynomial terms have simply turned into features that we can use for linear regression. We are using the same cost function and gradient that you wrote for the earlier part of this Lab.

In the Python code below, we are using a polynomial of degree 8. The function `polyFeatures(..)` is therefore applied with $p=8$ to the training set `X`, the validation set `Xval` and the test set `Xtest`, to get the corresponding projected data `X_poly`, `Xval_poly` and `Xtest_poly`.

It turns out that if we run the training directly on the projected data `X_poly`, it will not work well as the features would be badly scaled (e.g., an example with $x_1 = 40$ will have a feature $x_8 = 40^8$). Therefore, we need to use feature normalization. In the following Python code, we first normalize the features of the training set and add to it an additional first column of ones to get `X_poly_normalized`. The parameters used for normalization are stored in variables `mu` and `sigma`. Then, we train the linear regression model (i.e. find the optimal parameters vector $\theta$) by calling the function `trainLinearReg(..)` using our new training set `X_poly_normalized`, `y`, an initial $\theta$ vector of dimension $p+1$ (as we have $p$ features now), and a value of $\lambda$ set to $0$ for now (no regularization). Then, we plot two figures which looks like follows. From the first figure, you should see that the polynomial fit is able to follow the data-points very well - thus, obtaining a low training error. However, the polynomial fit is very complex and even drops off at the extremes. This is an indicator that the polynomial regression model is overfitting the training data and will not generalize well. To better understand the problems with the unregularized ($\lambda = 0$) model, you can see from the second figure that the learning curve shows the same effect where the training error is low, but the validation error is high. There is a gap between the training and cross validation errors, indicating a high variance problem.

We have already implemented all this for you. You just need to read the code and then run it.

```python
# The regularization parameter lmd
lmd = 0.0

# Using polyFeatures(..) with p=8 to map the original data to a higher dimension
p = 8
X_poly = polyFeatures(X, p) # on the training set
Xval_poly = polyFeatures(Xval, p) # on the validation set
Xtest_poly = polyFeatures(Xtest, p) # on the test set

# mean vector and standard deviation vector
mu = np.mean(X_poly, axis=0)
sigma = np.std(X_poly, axis=0)

# normalizing the training set and validation set using mu and sigma and adding an aditional first column of ones
X_poly_normalized = add_all_ones_column((X_poly - mu) / sigma)
Xval_poly_normalized = add_all_ones_column((Xval_poly - mu) / sigma)

# Training to find the optimal parameters vector theta
theta_init = np.zeros(p+1) # we p features, so we need p+1 parameters (as we have theta_0)
theta = trainLinearReg(X_poly_normalized, y, theta_init, lmd)

# Plotting the dataset and the polynomial regression curve
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))
ax1.set_xlabel("Change in water level (x)")
ax1.set_ylabel("Water flowing out of the dam (y)")
ax1.set_title(r"Polynomial Regression Fit ($\lambda = {}$)".format(lmd))

ax2.set_xlabel("Number of training examples")
ax2.set_ylabel("Error")
ax2.set_title(r"Polynomial Regression Learning Curve ($\lambda = {}$)".format(lmd))

x_plot = np.linspace(-60, 40, 100)
x_plot_poly = polyFeatures(x_plot.reshape(len(x_plot), 1), p)
x_plot_poly_normalized = add_all_ones_column((x_plot_poly - mu) / sigma)
y_plot = x_plot_poly_normalized @ theta
ax1.plot(x_plot, y_plot, linestyle="--", color="green", label="Polynomial fit")
ax1.scatter(X[:, 0], y, marker="x", color="red", label = "Training examples")

# Plotting the learning curves using the training and validation sets
errs_train, errs_val = [], []
for i in range(2, len(y)):
    theta = trainLinearReg(X_poly_normalized[:i], y[:i], theta_init, lmd)
    errs_train.append( E(theta, X_poly_normalized[:i], y[:i], 0) )
    errs_val.append( E(theta, Xval_poly_normalized, yval, 0) )

ax2.plot(range(2, len(y)), errs_train, label="Training error")
ax2.plot(range(2, len(y)), errs_val, label="Validation error")

ax1.legend()
ax2.legend()
plt.show()
```

## Adjusting the regularization parameter $\lambda$
In this section, you will get to observe how the regularization parameter affects the bias-variance of regularized polynomial regression. You should now modify the $\lambda$ parameter (i.e. variable `lmd`) in previous code and try $\lambda = 1$ and $\lambda = 50$ (you could also try with other values of $\lambda$). For each of these values, the above code should generate two figures (the polynomial fit to the data and the learning curve).

For $\lambda = 1$, you should see a polynomial fit that follows the data trend well and a learning curve showing that both the validation and training error converge to a relatively low value. This shows the $\lambda = 1$ regularized polynomial regression model does not have the high-bias or high-variance problems. In fact, it achieves a good trade-off between bias and variance. For $\lambda = 50$, you should see a polynomial fit that does not follow the data well. In this case, there is too much regularization and the model is unable to fit the training data.

## Selecting $\lambda$ using the validation set
From the previous section, you observed that the value of $\lambda$ can significantly affect the results of regularized polynomial regression on the training and cross validation set. In particular, a model without regularization ($\lambda = 0$) fits the training set well, but does not generalize. Conversely, a model with too much regularization (e.g. $\lambda = 50$) does not fit the data well. A good choice of $\lambda$ (e.g., $\lambda = 1$ for this dataset) can provide a good fit to the data.

In this section, you will implement an automated method to select the $\lambda$ parameter. Concretely, you will use a cross validation set to evaluate how good each $\lambda$ value is. After selecting the best $\lambda$ value using the validation set, we can then evaluate the model on the test set to estimate how well the model will perform on actual unseen data.

Your task is to complete the code below. Specifically, you should use the `trainLinearReg(..)` function to train the model using different values of $\lambda$ and compute the training error and validation error. You should try $\lambda$ in the following range: `[0, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10]`. You should also plot a curve that shows the validation error and training error with respect to the different values of $\lambda$. You should see a plot similar to the following figure, which allows you to select which $\lambda$ parameter to use. In this figure, we can see that the best value of $\lambda$ is around $3$ (as it gives the smallest validation error).

*Note:* Due to randomness in the training and validation splits of the dataset, the cross validation error can sometimes be lower than the training error.

```python
lmd_range = [0, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10]
errs_train, errs_val = [], []

""" TODO:
For each value of lmd you should call trainLinearReg(..) to train a model using the training set
X_poly_normalized, y, then compute the training error (on the training set: X_poly_normalized, y)
and the validation error (on the validation set: Xval_poly_normalized, yval).
"""
for lmd in lmd_range:
    theta = trainLinearReg(X_poly_normalized, y, theta_init,lmd)
    errs_train.append(E(theta, X_poly_normalized, y, lmd))
    errs_val.append(E(theta,Xval_poly_normalized, yval, lmd))

""" TODO:
Plot a curve that shows the validation error and training error with respect to lambda
"""
fig, ax = plt.subplots()

ax.plot(lmd_range, errs_train, color="blue", label="Training")
ax.plot(lmd_range,errs_val, color="orange",label="Validation")

ax.set_xlabel("lambda")
ax.set_ylabel("Error")
ax.legend()
fig.show()

```

## Computing test set error
In the previous section, you implemented code to compute the validation error for various values of the regularization parameter $\theta$. However, to get a better indication of the model's performance in the real world, it is important to evaluate the "*final*" model on a test set that was not used in any part of training (that is, it was neither used to select the regularization parameter $\lambda$, nor to learn the model parameters $\theta$).

In the code below, you should compute the test error using the best value of $\theta$ you found (using $\lambda = 3$).

```python
""" TODO:
Find the best parameters vector theta by training the polynomial regression
model using lambda = 3. Then compute the error on the test set.
Note: don't forget to normalize the test set before.
"""
lmd = 3
theta = trainLinearReg(X_poly_normalized, y, theta_init,lmd)
Xtest_poly_normalized = add_all_ones_column((Xtest_poly - mu) / sigma)
err_test = E(theta, Xtest_poly_normalized, ytest,lmd)
print("err_test =", err_test)
```

    err_test = 16.66360225546146

## (OPTIONAL) Performing a 10-fold-cross-validation

This section is optional.

Combine the training and validation sets into one set `Xall`, `yall`. Note: in Python you can combine two datasets A and B (two numpy arrays) into one dataset using `AB = np.append(A, B, axis=0)`.

Implement 10-fold-cross-validation to evaluate to find the best value for the hyperparameter $\lambda$. This will be the value that gives you the best generalization error.

Finally, use this best value of $\lambda$ to train a model on the whole dataset `Xall`, `yall`, and then test this model on the test dataset `Xtest`, `ytest` and compute the test error.

```python
""" TODO
Plot the learning curves based on randomly selected examples as described above.
"""
lmd_range = [0, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10]
p = 8

# Combine X with Xval
Xall = np.append(X,Xval,axis = 0)

# Normalize
Xall_poly = polyFeatures(Xall, p)
mu = np.mean(Xall_poly, axis=0)
sigma = np.std(Xall_poly, axis=0)
Xall_poly_normalized = add_all_ones_column((Xall_poly - mu) / sigma)

# Combine y with yval
yall = np.append(y,yval,axis = 0)

# Perform a 10-fold-cross-validation to find the best value for the hyperparameter lambda
def calc_mean_error(lmd):
    error = []
    for i in range(0, len(Xall_poly_normalized), k):
        Xall_poly_subset= Xall_poly_normalized[i:i+k]
        y_subset = yall[i:i+k]

        theta = trainLinearReg(Xall_poly_subset, y_subset, theta_init,lmd)

        error.append(E(theta, X_poly_normalized, y,lmd))
    return error

k = 10
mean_error = []
for lmd in lmd_range:

    mean_error.append(np.mean(calc_mean_error(lmd)))
    print(f"Mean error: {mean_error[-1]} Lambda: {lmd}")

optimalLmd = lmd_range[np.argmin(mean_error)]

theta = trainLinearReg(Xall_poly_normalized, yall, theta_init, optimalLmd)
error = E(theta, Xtest_poly_normalized, ytest,optimalLmd)

print(f"\nError {error} Lambda: {optimalLmd}")
```

    Mean error: 77.2803315741191 Lambda: 0
    Mean error: 18.685748753877043 Lambda: 0.003
    Mean error: 13.178456256439663 Lambda: 0.01
    Mean error: 10.727771926606307 Lambda: 0.03
    Mean error: 13.06554306040346 Lambda: 0.1
    Mean error: 16.98698788328017 Lambda: 0.3
    Mean error: 22.77291746862979 Lambda: 1
    Mean error: 32.64318835558437 Lambda: 3
    Mean error: 51.336836994837796 Lambda: 10

    Error 14.870496778091727 Lambda: 0.03

```python

```

Labs/Lab4_Generalization/Lab 4 - Part A - Regularized Linear Regression.ipynb


# Lambda School Data Science Module 141
## Statistics, Probability, and Inference

## Prepare - examine what's available in SciPy

As we delve into statistics, we'll be using more libraries - in particular the [stats package from SciPy](https://docs.scipy.org/doc/scipy/reference/tutorial/stats.html).

```
from scipy import stats
dir(stats)
```

    ['__all__',
     '__builtins__',
     '__cached__',
     '__doc__',
     '__file__',
     '__loader__',
     '__name__',
     '__package__',
     '__path__',
     '__spec__',
     '_binned_statistic',
     '_constants',
     '_continuous_distns',
     '_discrete_distns',
     '_distn_infrastructure',
     '_distr_params',
     '_multivariate',
     '_stats',
     '_stats_mstats_common',
     '_tukeylambda_stats',
     'absolute_import',
     'alpha',
     'anderson',
     'anderson_ksamp',
     'anglit',
     'ansari',
     'arcsine',
     'argus',
     'bartlett',
     'bayes_mvs',
     'bernoulli',
     'beta',
     'betaprime',
     'binned_statistic',
     'binned_statistic_2d',
     'binned_statistic_dd',
     'binom',
     'binom_test',
     'boltzmann',
     'boxcox',
     'boxcox_llf',
     'boxcox_normmax',
     'boxcox_normplot',
     'bradford',
     'burr',
     'burr12',
     'cauchy',
     'chi',
     'chi2',
     'chi2_contingency',
     'chisquare',
     'circmean',
     'circstd',
     'circvar',
     'combine_pvalues',
     'contingency',
     'cosine',
     'crystalball',
     'cumfreq',
     'describe',
     'dgamma',
     'dirichlet',
     'distributions',
     'division',
     'dlaplace',
     'dweibull',
     'energy_distance',
     'entropy',
     'erlang',
     'expon',
     'exponnorm',
     'exponpow',
     'exponweib',
     'f',
     'f_oneway',
     'fatiguelife',
     'find_repeats',
     'fisher_exact',
     'fisk',
     'fligner',
     'foldcauchy',
     'foldnorm',
     'frechet_l',
     'frechet_r',
     'friedmanchisquare',
     'gamma',
     'gausshyper',
     'gaussian_kde',
     'genexpon',
     'genextreme',
     'gengamma',
     'genhalflogistic',
     'genlogistic',
     'gennorm',
     'genpareto',
     'geom',
     'gilbrat',
     'gmean',
     'gompertz',
     'gumbel_l',
     'gumbel_r',
     'halfcauchy',
     'halfgennorm',
     'halflogistic',
     'halfnorm',
     'hmean',
     'hypergeom',
     'hypsecant',
     'invgamma',
     'invgauss',
     'invweibull',
     'invwishart',
     'iqr',
     'itemfreq',
     'jarque_bera',
     'johnsonsb',
     'johnsonsu',
     'kappa3',
     'kappa4',
     'kde',
     'kendalltau',
     'kruskal',
     'ks_2samp',
     'ksone',
     'kstat',
     'kstatvar',
     'kstest',
     'kstwobign',
     'kurtosis',
     'kurtosistest',
     'laplace',
     'levene',
     'levy',
     'levy_l',
     'levy_stable',
     'linregress',
     'loggamma',
     'logistic',
     'loglaplace',
     'lognorm',
     'logser',
     'lomax',
     'mannwhitneyu',
     'matrix_normal',
     'maxwell',
     'median_test',
     'mielke',
     'mode',
     'moment',
     'mood',
     'morestats',
     'moyal',
     'mstats',
     'mstats_basic',
     'mstats_extras',
     'multinomial',
     'multivariate_normal',
     'mvn',
     'mvsdist',
     'nakagami',
     'nbinom',
     'ncf',
     'nct',
     'ncx2',
     'norm',
     'normaltest',
     'norminvgauss',
     'obrientransform',
     'ortho_group',
     'pareto',
     'pearson3',
     'pearsonr',
     'percentileofscore',
     'planck',
     'pointbiserialr',
     'poisson',
     'power_divergence',
     'powerlaw',
     'powerlognorm',
     'powernorm',
     'ppcc_max',
     'ppcc_plot',
     'print_function',
     'probplot',
     'randint',
     'random_correlation',
     'rankdata',
     'ranksums',
     'rayleigh',
     'rdist',
     'recipinvgauss',
     'reciprocal',
     'relfreq',
     'rice',
     'rv_continuous',
     'rv_discrete',
     'rv_histogram',
     'scoreatpercentile',
     'sem',
     'semicircular',
     'shapiro',
     'sigmaclip',
     'skellam',
     'skew',
     'skewnorm',
     'skewtest',
     'spearmanr',
     'special_ortho_group',
     'statlib',
     'stats',
     't',
     'test',
     'theilslopes',
     'tiecorrect',
     'tmax',
     'tmean',
     'tmin',
     'trapz',
     'triang',
     'trim1',
     'trim_mean',
     'trimboth',
     'truncexpon',
     'truncnorm',
     'tsem',
     'tstd',
     'ttest_1samp',
     'ttest_ind',
     'ttest_ind_from_stats',
     'ttest_rel',
     'tukeylambda',
     'tvar',
     'uniform',
     'unitary_group',
     'variation',
     'vonmises',
     'vonmises_line',
     'wald',
     'wasserstein_distance',
     'weibull_max',
     'weibull_min',
     'weightedtau',
     'wilcoxon',
     'wishart',
     'wrapcauchy',
     'zipf',
     'zmap',
     'zscore']

```
# As usual, lots of stuff here! There's our friend, the normal distribution
norm = stats.norm()
print(norm.mean())
print(norm.std())
print(norm.var())
```

    0.0
    1.0
    1.0

```
# And a new friend - t
t1 = stats.t(5)  # 5 is df "shape" parameter
print(t1.mean())
print(t1.std())
print(t1.var())
```

    0.0
    1.2909944487358056
    1.6666666666666667

*(Picture from [Wikipedia](https://en.wikipedia.org/wiki/Student's_t-distribution#/media/File:Student_t_pdf.svg))*

The t-distribution is "normal-ish" - the larger the parameter (which reflects its degrees of freedom - more input data/features will increase it), the closer to true normal.

```
t2 = stats.t(30)  # Will be closer to normal
print(t2.mean())
print(t2.std())
print(t2.var())
```

    0.0
    1.0350983390135313
    1.0714285714285714

Why is it different from normal? To better reflect the tendencies of small data and situations with unknown population standard deviation. In other words, the normal distribution is still the nice pure ideal in the limit (thanks to the central limit theorem), but the t-distribution is much more useful in many real-world situations.

History sidenote - this is "Student":

*(Picture from [Wikipedia](https://en.wikipedia.org/wiki/File:William_Sealy_Gosset.jpg))*

His real name is William Sealy Gosset, and he published under the pen name "Student" because he was not an academic. He was a brewer, working at Guinness and using trial and error to determine the best ways to yield barley. He's also proof that, even 100 years ago, you don't need official credentials to do real data science!

## Live Lecture - let's perform and interpret a t-test

We'll generate our own data, so we can know and alter the "ground truth" that the t-test should find. We will learn about p-values and how to interpret "statistical significance" based on the output of a hypothesis test. We will also dig a bit deeper into how the test statistic is calculated based on the sample error, and visually what it looks like to have 1 or 2 "tailed" t-tests.

```
# TODO - during class, but please help!
lambda_heights = [72, 72, 77, 72, 73, 67, 64, 58, 63, 78]

import pandas as pd
df = pd.DataFrame({'heights': lambda_heights})

df.head(10)
```

      heights

      0
      72

      1
      72

      2
      77

      3
      72

      4
      73

      5
      67

      6
      64

      7
      58

      8
      63

      9
      78

```
df['heights'].mean()
```

    69.6

```
import random
random.seed(10)

population = []

for _ in range(1000):
  population.append(random.randint(0,1))

print(population)
print(len(population))
```

    [0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
    1000

```
sample = random.sample(population, 100)

print(sample)
print(len(sample))
```

    [0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0]
    100

```
df = pd.DataFrame({'likes_coke': sample})
df.head()

df['likes_coke'].mean()
```

    0.46

```
df.plot.hist();
```

```
df['likes_coke'].describe()
```

    count    100.000000
    mean       0.460000
    std        0.500908
    min        0.000000
    25%        0.000000
    50%        0.000000
    75%        1.000000
    max        1.000000
    Name: likes_coke, dtype: float64

```
import numpy as np

def mean(list):
  average = np.sum(list)/len(list)
  return average

print('Population Mean:', mean(population))
print('Sample Mean', mean(sample))
```

    Population Mean: 0.507
    Sample Mean 0.46

```
def variance(list):
  n = len(list)
  return np.sum((list - mean(list))**2)/(n-1)

variance(df['likes_coke'])
```

    0.25090909090909086

```
def stddev(list):
  var = variance(list)
  return var**(1/2)

stddev(df['likes_coke'])
```

    0.5009082659620331

```
n = len(df['likes_coke'])
t_stat = (mean(df['likes_coke']) - .5)/(stddev(df['likes_coke'])/n**(1/2))

print(t_stat)
```

    -0.7985494095046901

```
# dir(scipy.stats)
```

```
stats.ttest_1samp(df['likes_coke'], .5)
```

    Ttest_1sampResult(statistic=-0.7985494095046901, pvalue=0.4264632540527489)

\begin{align}
t = \frac{\text{variance between groups}}{\text{variance within groups}}
\end{align}

## Assignment - apply the t-test to real data

Your assignment is to determine which issues have "statistically significant" differences between political parties in this [1980s congressional voting data](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records). The data consists of 435 instances (one for each congressperson), a class (democrat or republican), and 16 binary attributes (yes or no for voting for or against certain issues). Be aware - there are missing values!

Your goals:

1. Load and clean the data (or determine the best method to drop observations when running tests)
2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.01
3. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.01
4. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)

Note that this data will involve *2 sample* t-tests, because you're comparing averages across two groups (republicans and democrats) rather than a single group against a null hypothesis.

Stretch goals:

1. Refactor your code into functions so it's easy to rerun with arbitrary variables
2. Apply hypothesis testing to your personal project data (for the purposes of this notebook you can type a summary of the hypothesis you formed and tested)

```
# TODO - your code here!
votes = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data', header=None)

votes.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16

      0
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      y
      ?
      y
      y
      y
      n
      y

      1
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      n
      n
      y
      y
      y
      n
      ?

      2
      democrat
      ?
      y
      y
      ?
      y
      y
      n
      n
      n
      n
      y
      n
      y
      y
      n
      n

      3
      democrat
      n
      y
      y
      n
      ?
      y
      n
      n
      n
      n
      y
      n
      y
      n
      n
      y

      4
      democrat
      y
      y
      y
      n
      y
      y
      n
      n
      n
      n
      y
      ?
      y
      y
      y
      y

```
votes.columns
```

    Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16], dtype='int64')

```
votes = votes.rename(columns={0: 'party:', 1: 'handicapped-infants:', 2: 'water-project-cost-sharing:', 3: 'adoption-of-the-budget-resolution:',
                      4: 'physician-fee-freeze:', 5: 'el-salvador-aid:', 6: 'religious-groups-in-schools:', 7: 'anti-satellite-test-ban:',
                      8: 'aid-to-nicaraguan-contras:', 9: 'mx-missile:', 10: 'immigration:', 11: 'synfuels-corporation-cutback:', 12: 'education-spending:',
                      13: 'superfund-right-to-sue:', 14: 'crime:', 15: 'duty-free-exports:', 16: 'export-administration-act-south-africa:'})

votes.head()
```

      party:
      handicapped-infants:
      water-project-cost-sharing:
      adoption-of-the-budget-resolution:
      physician-fee-freeze:
      el-salvador-aid:
      religious-groups-in-schools:
      anti-satellite-test-ban:
      aid-to-nicaraguan-contras:
      mx-missile:
      immigration:
      synfuels-corporation-cutback:
      education-spending:
      superfund-right-to-sue:
      crime:
      duty-free-exports:
      export-administration-act-south-africa:

      0
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      y
      ?
      y
      y
      y
      n
      y

      1
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      n
      n
      y
      y
      y
      n
      ?

      2
      democrat
      ?
      y
      y
      ?
      y
      y
      n
      n
      n
      n
      y
      n
      y
      y
      n
      n

      3
      democrat
      n
      y
      y
      n
      ?
      y
      n
      n
      n
      n
      y
      n
      y
      n
      n
      y

      4
      democrat
      y
      y
      y
      n
      y
      y
      n
      n
      n
      n
      y
      ?
      y
      y
      y
      y

```
votes = votes.replace('?', np.nan)

votes.isnull().sum()
```

    party:                                       0
    handicapped-infants:                        12
    water-project-cost-sharing:                 48
    adoption-of-the-budget-resolution:          11
    physician-fee-freeze:                       11
    el-salvador-aid:                            15
    religious-groups-in-schools:                11
    anti-satellite-test-ban:                    14
    aid-to-nicaraguan-contras:                  15
    mx-missile:                                 22
    immigration:                                 7
    synfuels-corporation-cutback:               21
    education-spending:                         31
    superfund-right-to-sue:                     25
    crime:                                      17
    duty-free-exports:                          28
    export-administration-act-south-africa:    104
    dtype: int64

```
votes = votes.replace({np.nan: 0, 'n': -1, 'y': 1})

votes.isnull().sum()
```

    party:                                     0
    handicapped-infants:                       0
    water-project-cost-sharing:                0
    adoption-of-the-budget-resolution:         0
    physician-fee-freeze:                      0
    el-salvador-aid:                           0
    religious-groups-in-schools:               0
    anti-satellite-test-ban:                   0
    aid-to-nicaraguan-contras:                 0
    mx-missile:                                0
    immigration:                               0
    synfuels-corporation-cutback:              0
    education-spending:                        0
    superfund-right-to-sue:                    0
    crime:                                     0
    duty-free-exports:                         0
    export-administration-act-south-africa:    0
    dtype: int64

```
votes.head()
```

      party:
      handicapped-infants:
      water-project-cost-sharing:
      adoption-of-the-budget-resolution:
      physician-fee-freeze:
      el-salvador-aid:
      religious-groups-in-schools:
      anti-satellite-test-ban:
      aid-to-nicaraguan-contras:
      mx-missile:
      immigration:
      synfuels-corporation-cutback:
      education-spending:
      superfund-right-to-sue:
      crime:
      duty-free-exports:
      export-administration-act-south-africa:

      0
      republican
      -1
      1
      -1
      1
      1
      1
      -1
      -1
      -1
      1
      0
      1
      1
      1
      -1
      1

      1
      republican
      -1
      1
      -1
      1
      1
      1
      -1
      -1
      -1
      -1
      -1
      1
      1
      1
      -1
      0

      2
      democrat
      0
      1
      1
      0
      1
      1
      -1
      -1
      -1
      -1
      1
      -1
      1
      1
      -1
      -1

      3
      democrat
      -1
      1
      1
      -1
      0
      1
      -1
      -1
      -1
      -1
      1
      -1
      1
      -1
      -1
      1

      4
      democrat
      1
      1
      1
      -1
      1
      1
      -1
      -1
      -1
      -1
      1
      0
      1
      1
      1
      1

```
immigration = votes[['party:', 'immigration:']]

immigration.head(100)
```

      party:
      immigration:

      0
      republican
      1

      1
      republican
      -1

      2
      democrat
      -1

      3
      democrat
      -1

      4
      democrat
      -1

      5
      democrat
      -1

      6
      democrat
      -1

      7
      republican
      -1

      8
      republican
      -1

      9
      democrat
      -1

      10
      republican
      -1

      11
      republican
      -1

      12
      democrat
      -1

      13
      democrat
      1

      14
      republican
      -1

      15
      republican
      1

      16
      democrat
      1

      17
      democrat
      -1

      18
      republican
      -1

      19
      democrat
      -1

      20
      democrat
      -1

      21
      democrat
      -1

      22
      democrat
      -1

      23
      democrat
      -1

      24
      democrat
      -1

      25
      democrat
      1

      26
      democrat
      -1

      27
      democrat
      -1

      28
      republican
      -1

      29
      democrat
      -1

      ...
      ...
      ...

      70
      democrat
      -1

      71
      republican
      1

      72
      democrat
      -1

      73
      republican
      1

      74
      democrat
      1

      75
      democrat
      1

      76
      democrat
      1

      77
      democrat
      1

      78
      democrat
      1

      79
      republican
      1

      80
      democrat
      1

      81
      democrat
      1

      82
      republican
      1

      83
      republican
      -1

      84
      republican
      1

      85
      democrat
      1

      86
      republican
      1

      87
      republican
      -1

      88
      democrat
      1

      89
      republican
      1

      90
      democrat
      1

      91
      democrat
      1

      92
      democrat
      1

      93
      democrat
      1

      94
      democrat
      -1

      95
      democrat
      -1

      96
      democrat
      -1

      97
      democrat
      1

      98
      democrat
      1

      99
      republican
      1

100 rows × 2 columns

```
immigration.shape
```

    (435, 2)

```
immigration_ct = pd.crosstab(immigration['immigration:'],
                     immigration['party:'], margins=True, margins_name='Total')

immigration_ct
```

      party:
      democrat
      republican
      Total

      immigration:

      -1
      139
      73
      212

      0
      4
      3
      7

      1
      124
      92
      216

      Total
      267
      168
      435

```
immigration.columns
```

    Index(['party:', 'immigration:'], dtype='object')

```

```

```
rep_immigration = votes.loc[votes['party:'] == 'republican', ['party:','immigration:']]

rep_immigration.head()
```

      party:
      immigration:

      0
      republican
      1

      1
      republican
      -1

      7
      republican
      -1

      8
      republican
      -1

      10
      republican
      -1

```
stats.ttest_ind(rep_immigration['immigration:'], votes['immigration:'])
```

    Ttest_indResult(statistic=1.1536316734149077, pvalue=0.24910977397659076)

```
stats.ttest_1samp(rep_immigration['immigration:'], .5)
```

    Ttest_1sampResult(statistic=-5.078336591867216, pvalue=1.0088210778028215e-06)

```
dem_immigration = votes.loc[votes['party:'] == 'democrat', ['party:', 'immigration:']]

dem_immigration.head()
```

      party:
      immigration:

      2
      democrat
      -1

      3
      democrat
      -1

      4
      democrat
      -1

      5
      democrat
      -1

      6
      democrat
      -1

```
stats.ttest_1samp(dem_immigration['immigration:'], .5)
```

    Ttest_1sampResult(statistic=-9.154416352898474, pvalue=1.496387072993042e-17)

```
stats.ttest_ind(dem_immigration['immigration:'], votes['immigration:'])
```

    Ttest_indResult(statistic=-0.8468980514996853, pvalue=0.39734138999289603)

```
budget = votes[['party:', 'adoption-of-the-budget-resolution:']]

budget.head()
```

      party:
      adoption-of-the-budget-resolution:

      0
      republican
      -1

      1
      republican
      -1

      2
      democrat
      1

      3
      democrat
      1

      4
      democrat
      1

```
budget_ct = pd.crosstab(budget['adoption-of-the-budget-resolution:'],
                               budget['party:'])

budget_ct
```

      party:
      democrat
      republican

      adoption-of-the-budget-resolution:

      -1
      29
      142

      0
      7
      4

      1
      231
      22

```
stats.ttest_1samp(budget_ct, .5)
```

    Ttest_1sampResult(statistic=array([1.24152202, 1.28137594]), pvalue=array([0.34026678, 0.32855397]))

```
education = votes[['party:', 'education-spending:']]

education.head()
```

      party:
      education-spending:

      0
      republican
      1

      1
      republican
      1

      2
      democrat
      -1

      3
      democrat
      -1

      4
      democrat
      0

```
education_ct = pd.crosstab(education['education-spending:'],
                           education['party:'])

education_ct
```

      party:
      democrat
      republican

      education-spending:

      -1
      213
      20

      0
      18
      13

      1
      36
      135

```
stats.ttest_1samp(education_ct, .5)
```

    Ttest_1sampResult(statistic=array([1.42243256, 1.4032283 ]), pvalue=array([0.2908474, 0.2956556]))

```
crime = votes[['party:', 'crime:']]

crime.head()
```

      party:
      crime:

      0
      republican
      1

      1
      republican
      1

      2
      democrat
      1

      3
      democrat
      -1

      4
      democrat
      1

```
crime_ct = pd.crosstab(crime['crime:'],
                      crime['party:'])
crime_ct
```

      party:
      democrat
      republican

      crime:

      -1
      167
      3

      0
      10
      7

      1
      90
      158

```

```

```
stats.ttest_1samp(crime_ct, .5)
```

    Ttest_1sampResult(statistic=array([1.9525754 , 1.08795647]), pvalue=array([0.19011298, 0.39025407]))

# Resources

- https://homepage.divms.uiowa.edu/~mbognar/applets/t.html
- https://rpsychologist.com/d3/tdist/
- https://gallery.shinyapps.io/tdist/
- https://en.wikipedia.org/wiki/Standard_deviation#Sample_standard_deviation_of_metabolic_rate_of_northern_fulmars


```python
import numpy as np
import math as m
import matplotlib.pyplot as plt
import sympy

def cel_to_kel(D):
    K = D + 273.15
    return K

def kel_to_cel(K):
    D = K - 273.15
    return D
```

# Useful constants

```python
sigma = 5.67*10**(-8)
```

# 5.5

```python
D = 12e-3
Taq = 1150
Tref = 400
Tinf = 325
h = 20
k = 40
rho = 7800
c = 600

Lc = D/2/3

Bi = h*Lc/k

print('Biot = {}'.format(Bi))

V = np.pi * D**3 / 6

As = np.pi * D**2

theta = Tref - Tinf

theta_i = Taq - Tinf

t  = (rho*V*c)/(h*As) * np.log(theta_i/theta)

print('tempo = {}'.format(t))
```

    Biot = 0.001
    tempo = 1122.2149876696374

# 5.16

```python
k = 60
rho = 7850
c = 430
L = 10e-3
R_fx_tc = 0.01
Tinf = 1300
Ti = 300
Tsi = 1200
h = 25

U = 1/(1/h + R_fx_tc)

print('Coeficiente global de trans. cal. = {}'.format(U))

Bi = U*L/k

print('Biot = {}'.format(Bi))

theta = Tsi - Tinf

theta_i = Ti - Tinf

t  = rho * L * c/ U * np.log(theta_i/theta)

print('tempo = {}'.format(t))

Tso = (h*Tinf + Tsi/R_fx_tc)/(h+1/R_fx_tc)

print('Temperatura superficie externa = {} K'.format(Tso))
```

    Coeficiente global de trans. cal. = 20.0
    Biot = 0.0033333333333333335
    tempo = 3886.187990700701
    Temperatura superficie externa = 1220.0 K

```python

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.

# Spreading out

You've reached the fourth module of the open course [**"Practical Numerical Methods with Python"**](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about), titled *Spreading out: Parabolic PDEs*. We hope that you are enjoying the ride of [#numericalmooc](https://twitter.com/hashtag/numericalmooc) so far!

We introduced finite-difference methods for partial differential equations (PDEs) in the [second module](https://github.com/numerical-mooc/numerical-mooc/tree/master/lessons/02_spacetime), and looked at convection problems in more depth in [module 3](https://github.com/numerical-mooc/numerical-mooc/tree/master/lessons/03_wave). Now we'll look at solving problems dominated by diffusion.

Why do we separate the discussion of how to solve convection-dominated and diffusion-dominated problems, you might ask? It's all about the harmony between mathematical model and numerical method. Convection and diffusion are inherently different physical processes.

The titles of the course modules are meant to spark your imagination:

* _Riding the wave_—imagine a surfer on a tall wave, moving fast towards the beach ... convection implies transport, speed, direction. The physics has a directional bias, and we discovered that numerical methods should be compatible with that. That's why we use _upwind_ methods for convection, and we pay attention to problems where waves move in opposite directions, needing special schemes.

* _Spreading out_—now imagine a drop of food dye in a cup of water, slowly spreading in all directions until all the liquid takes a uniform color. [Diffusion](http://en.wikipedia.org/wiki/Diffusion) spreads the concentration of something around (atoms, people, ideas, dirt, anything!). Since it is not a directional process, we need numerical methods that are isotropic (like central differences).

```python
from IPython.display import Image
Image(url='http://upload.wikimedia.org/wikipedia/commons/f/f9/Blausen_0315_Diffusion.png')
```

## Parabolic PDEs

You already met the simplest parabolic PDE—the [1-D diffusion equation](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_03_1DDiffusion.ipynb)—in module 2. Its main feature is that it has a second-order derivative in space. Here it is again:

$$
\frac{\partial u}{\partial t} = \alpha \frac{\partial^2 u}{\partial x^2}
$$

Check out the article on [parabolic PDEs](http://en.wikipedia.org/wiki/Parabolic_partial_differential_equation) in Wikipedia. Now compare with the diffusion equation above, with the two independent variables here being $x, t$. You'll see that with no mixed derivatives, and only one second-order derivative (in the spatial variable $x$), it satisfies the condition of a parabolic PDE. Work it out on a piece of paper if you need to.

In the previous module, discussing hyperbolic conservation laws, we learned that solutions have characteristics: information travels along certain paths on space-time phase space. In contrast, parabolic equations don't have characteristics, because any local change in the initial condition will eventually affect the entire domain, although its effect will be felt at smaller intensity with longer distances. This is typical of diffusion processes.

In this first lesson of the module, we first review the 1D diffusion equation and then take a deeper look into the issue of boundary conditions. In the next notebook, we'll introduce _implicit discretizations_ for the first time, which will take us to the land of linear solvers. In the third lesson we'll graduate to two dimensions—more boundary condition and stability issues will come up. We'll then study 2D implicit methods, and go into Crank-Nicolson method: perhaps the most popular of them all. _Enjoy!_

## Heat conduction

Heat conduction is a diffusive process. Let's remind ourselves of the heat equation in one spatial dimension:

\begin{equation}
\frac{\partial T}{\partial t} = \alpha \frac{\partial^2 T}{\partial x^2}
\end{equation}

Here, $\alpha$ is the thermal diffusivity, a property of the material, and $T$ is the temperature.

In the [third lesson of module 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_03_1DDiffusion.ipynb), we discretized the diffusion equation  with a forward-time, centered-space scheme, subject to the following stability constraint:

\begin{equation}
\alpha \frac{\Delta t}{(\Delta x)^2} \leq \frac{1}{2}
\end{equation}

Let's look into it more deeply now, using a 1D temperature-evolution problem.

### Problem set up

Say we have a graphite rod, with [thermal diffusivity](http://en.wikipedia.org/wiki/Thermal_diffusivity) of $\alpha=1.22\times10^{-3} {\rm m}^2/{\rm s}$, length $L=1{\rm m}$, and temperature $T=0{\rm C}$ everywhere. At time $t=0$, we raise the temperature on the left-side end, $x=0$, to $T=100{\rm C}$, and hold it there. *How will the temperature evolve in the rod?*

As usual, start by importing some libraries and setting up the discretization. We'll begin by using a spatial grid with 51 points and advance for 100 time steps, using a forward-time/centered scheme.

.
#### Figure 1. Graphite rod, temperature fixed on ends.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

```python
L = 1
nt = 100
nx = 51
alpha = 1.22e-3

dx = L/(nx-1)

Ti = numpy.zeros(nx)
Ti[0] = 100
```

Remember the forward-time, centered-space discretization? You should work it out on a piece of paper yourself (if you can't do it without looking it up, it means you need to do this more!).

\begin{equation}\frac{T_{i}^{n+1}-T_{i}^{n}}{\Delta t}=\alpha\frac{T_{i+1}^{n}-2T_{i}^{n}+T_{i-1}^{n}}{\Delta x^2}\end{equation}

To obtain the temperature at the next time step, $T^{n+1}_i$, from the known information at the current time step, we compute

\begin{equation}T_{i}^{n+1}=T_{i}^{n}+\frac{\alpha\Delta t}{\Delta x^2}(T_{i+1}^{n}-2T_{i}^{n}+T_{i-1}^{n})\end{equation}

Check the [third notebook of module 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_03_1DDiffusion.ipynb), if you need to refresh your memory!

The following function implements this numerical scheme:

```python
def ftcs(T, nt, dt, dx, alpha):
    """Solves the diffusion equation with forward-time, centered scheme

    Parameters:
    ----------
    T: array of float
        Initial temperature profile
    nt: int
        Number of time steps
    dt: float
        Time step size
    dx: float
        Mesh size
    alpha: float
        Diffusion coefficient (thermal diffusivity)

    Returns:
    -------
    T: array of float
        Temperature profile after nt time steps with forward in time scheme

    """

    for n in range(nt):
        Tn = T.copy()
        T[1:-1] = Tn[1:-1] + alpha*dt/dx**2*(Tn[2:] -2*Tn[1:-1] + Tn[0:-2])

    return T
```

We are all set to run! First, let's use a time step `dt` that satisfies the stability constraint

```python
sigma = 1/2.0
dt = sigma * dx*dx/alpha
T = ftcs(Ti.copy(), nt, dt, dx, alpha)
```

##### Note
-----
We are sending a *copy* of `Ti` to the function `ftcs`.  Sending a copy insures that the value of `Ti` remains unchanged for us to reuse.

Confused?  Take a look at [Lesson 0](./04_00_Python_Function_Quirks.ipynb) for a more in-depth explanation.

 -----

Now plot the solution!

```python
pyplot.plot(numpy.linspace(0,1,nx), T, color='#003366', ls='-', lw=3)
pyplot.ylim(0,100)
pyplot.xlabel('Length of Rod')
pyplot.ylabel('Temperature');
```

## Boundary Conditions

In the rod problem above, we stated that the left-hand side of the rod is held to a temperature $T = 100{\rm C}$.  This is an example of a *boundary condition* (BC): a rule that defines how the problem interacts with the borders of the domain.  In this case, the domain spans the length of the rod, from $0 \leq x \leq 1$.

There are many types of boundary conditions, and they have an important effect on the solution of the problem. For example, in [module 2, lesson 4](http://nbviewer.ipython.org/urls/raw.github.com/numerical-mooc/numerical-mooc/master/lessons/02_spacetime/02_04_1DBurgers.ipynb) on Burgers' Equation, we used a periodic boundary condition to "connect" the right-hand and left-hand borders of the domain.

We need to discuss boundary conditions in a little more detail, and this is good place to do it. Read on!

### Dirichlet boundary

The rod example above uses a *Dirichlet* BC on the left-hand side.  A Dirichlet boundary is one in which the border is held to a specific value of the solution variable.

What about the right-hand end of the rod $(x=1)$?  In the discretization we set up above, the problem hasn't evolved for long enough time for the heat to travel the full length of the rod.  Let's increase the number of timesteps, `nt`, and see what happens.

```python
nt = 1000
T = ftcs(Ti.copy(), nt, dt, dx, alpha)
```

```python
pyplot.plot(numpy.linspace(0,1,nx), T, color='#003366', ls='-', lw=3)
pyplot.ylim(0,100)
pyplot.xlabel('Length of Rod')
pyplot.ylabel('Temperature');
```

Hmmm ... it looks like we're pinning the right BC to the value $T=0{\rm C}$, do you agree?
Now study the code for the `ftbs` function, above, and try to figure out what happens at the right-most point of the spatial grid.

Did you figure it out?

It never updates the value at `T[-1]`!  That value is set to zero in the initial condition and stays that way throughout the entire simulation.  We effectively have a Dirichlet boundary at *both* ends.  The left-hand side of the rod is held to a temperature of $T = 100{\rm C}$ and the right-hand side is held to $T = 0{\rm C}$  Because both end tempera- tures are "pinned" to set values, the temperature distribution within the rod will relax (after enough time) to a linear temperature gradient across the length of the rod.

Once the solution is relaxed, no number of extra timesteps will make it change.  This is an example of a *steady-state solution* and we'll learn more about those in Module 5 of the course.

Dirichlet BCs show up in many engineering applications, among them thermo- dynamics (e.g., setting a surface temperature) and fluid dynamics (e.g., the no-slip conditions at walls in viscous fluids).

### Neumann boundary

Another commonly used BC is the *Neumann* boundary: rather than specifying the value of the solution at the border, they specify the value of the *derivative* of the solution at the border.

In our example, if we apply a Neumann boundary to the right-hand end of the rod, we can represent that mathematically as

\begin{equation}
\left. \frac{\partial T}{\partial x} \right|_{x = 1} = q(t)
\end{equation}

How can we enforce that in code?  One easy way is using a finite-difference discretization of the derivative at the boundary.  At time step $n$, for $N$ points, that would be

\begin{equation}
\left. \frac{\partial T}{\partial x} \right|_{x=1} \approx \frac{T^n_N - T^n_{N-1}}{\Delta x} = q(t)
\end{equation}

In the context of heat conduction, the space derivative of temperature is the heat flux density $q$, the amount of heat per unit time, per unit area.

For example, if the rod at $x=1$ has some insulating material, no heat is going to be able to get out through that end, and the Neumann boundary condition is:

\begin{equation}
\left. \frac{\partial T}{\partial x} \right|_{x=1} = 0
\end{equation}

Think about what the equation is saying: the change in temperature when moving in the $x$ direction is zero at the right-most edge of the rod. That means that the temperature should be equal on the last two spatial grid points.

To enforce this Neumann boundary condition at the right-hand end of the rod, we add the following line of code:

```Python
T[-1] = T[-2]
```

That is, the temperature at the boundary (`T[-1]`) is equal to the temperature directly to its left (`T[-2]`).  The spatial gradient is zero and a Neumann condition is satisfied there.

The Dirichlet conditions in the rod example were enforced without any extra input from us, but the Neumann condition will require an update at each iteration of the loop.

Finally, Dirichlet and Neumann BCs can sometimes be mixed to better represent the physics of a problem.

Let's revisit the heated-rod problem.  This time, the temperature at $x = 0$ will remain fixed at $T = 100{\rm C}$—that's the Dirichlet condition— and a Neumann BC applies at $x = 1$.

```python
def ftcs_mixed(T, nt, dt, dx, alpha):
    """Solves the diffusion equation with forward-time, centered scheme using
    Dirichlet b.c. at left boundary and Neumann b.c. at right boundary

    Parameters:
    ----------
    u: array of float
        Initial temperature profile
    nt: int
        Number of time steps
    dt: float
        Time step size
    dx: float
        Mesh size
    alpha: float
        Diffusion coefficient (thermal diffusivity)

    Returns:
    -------
    u: array of float
        Temperature profile after nt time steps with forward in time scheme

    """
    for n in range(nt):
        Tn = T.copy()
        T[1:-1] = Tn[1:-1] + alpha*dt/dx**2*(Tn[2:] -2*Tn[1:-1] + Tn[0:-2])
        T[-1] = T[-2]

    return T
```

```python
nt = 1000
T = ftcs_mixed(Ti.copy(), nt, dt, dx, alpha)
```

```python
pyplot.plot(numpy.linspace(0,1,nx), T, color='#003366', ls='-', lw=3)
pyplot.ylim(0,100)
pyplot.title('Rod with LHS Dirichlet B.C. and RHS Neumann B.C.\n')
pyplot.xlabel('Length of Rod')
pyplot.ylabel('Temperature');
```

Now heat is accumulating in the domain. Our insulator works!  If you increase the number of timesteps `nt` further, you will see that the outflow temperature at $x = 1$ continues to increase.

### Explicit schemes and BCs

The numerical schemes we've learned so far are called _explicit_, because we compute the updated solution at $t^{n+1}$ using only known information at time $t^n$. This is simple but has limitations, particularly with the small sizes of time step necessary to obtain stability.

Here's another issue with explicit schemes. Figure 2 shows the superposed stencil of several grid points over three time steps computed using the forward-time, centered scheme. You know that to calculate $T_i^{n+1}$ you use the information from the grid points $i-1, i, i+1$ at the previous time step. Think about what happens at the boundary: any change in the boundary condition will feed into the solution only at the next time step, not immediately. But this contradicts the physics of the problem, as any change on the boundary should be felt right away under the diffusion equation. To include boundary effects on the same time level, we can use an _implicit_ update (subject of the next lesson).

.
#### Figure 2. Stencil with explicit scheme over 3 time steps.

### Explicit schemes and time step

Let's replace the parameter $\sigma=\alpha \frac{\Delta t}{(\Delta x)^2}$ into Equation 4 and rearrange it a little bit, grouping together the terms at grid point $i$ on the right-hand side:

\begin{equation}
T_{i}^{n+1} = \sigma T_{i-1}^{n}+(1- 2 \sigma) T_{i}^{n} + \sigma T_{i+1}^{n}
\end{equation}

It's helpful to look at a sketch of the stencil for the heat equation with the weights added in for the contribution of each grid point. See Figure 3, which shows the weights for two values of $\sigma$: $\frac{1}{2}$ and $1$. Notice that with $\sigma=\frac{1}{2}$, the solution variable at the next time step is _independent_ from its own value at the present step, which seems a bit weird. And with $\sigma=1$, the current value affects negatively the future value—now _that's_ a red flag!

In fact, the solution will develop growing errors with $\sigma>\frac{1}{2}$, i.e., become unstable. This limits the time step that can be used quite significantly, because $\Delta t \propto (\Delta x)^2$. This is a serious cost of _explicit_ methods and an incentive to consider _implicit_ alternatives (next lesson).

.
#### Figure 3. Stencils for the heat equation with $\sigma= \frac{1}{2}, 1$.

---
###### The cell below loads the style of the notebook

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

</style>


Massimo Nocentini

November 9, 2016: splitting from "big" notebook

Abstract

Theory of matrix functions applied to Fibonacci matrix and a matrix with a eigenvalue with multiplicity greater than 1.

```python
from sympy import *
from sympy.abc import n, i, N, x, lamda, phi, z, j, r, k, a

from commons import *
from matrix_functions import *
from sequences import *
import functions_catalog

init_printing()
```

# Fibonacci matrix

```python
F = define(Symbol(r'\mathcal{F}'), Matrix([[1, 1], [1, 0]]))
F
```

```python
m = F.rhs.rows
```

```python
eigendata = spectrum(F)
eigendata
```

```python
data, eigenvals, multiplicities = eigendata.rhs
```

```python
Phi_poly = Phi_poly_ctor(deg=m-1)
Phi_poly
```

```python
Phi_polynomials = component_polynomials(eigendata)
Phi_polynomials
```

```python
cmatrices = component_matrices(F, Phi_polynomials)
cmatrices
```

```python
Z_11 = cmatrices[1,1].rhs
```

```python
assert (F.rhs * Z_11 - Z_11 * F.rhs).simplify() == zeros(2)
```

```python
assert (Z_11*Z_11 - Z_11).subs(eigenvals).applyfunc(simplify) == zeros(2)
```

```python
Zi1 = list(cm.rhs.as_immutable() for (i, j), cm in cmatrices.items() if j == 1)

s = zeros(m)
for Z in Zi1:
    s += Z
s, s.simplify(), s.subs(eigenvals).applyfunc(simplify)
```

```python
v = IndexedBase('v')
v_vector = Matrix(m, 1, lambda i, _: v[i])
M_space_ctor = M_space(cmatrices)
M_space_v = M_space_ctor(v_vector)
M_space_v
```

```python
i = 1
eq = Eq(F.rhs*M_space_v[i][1].rhs, M_space_v[i][1].rhs.applyfunc(lambda k: k * data[i][0]))
eq
```

```python
assert (eq.lhs.applyfunc(lambda i: i.subs(eigenvals).ratsimp()) ==
        eq.rhs.applyfunc(lambda i: i.subs(eigenvals).ratsimp()))
```

```python
GEs = generalized_eigenvectors_matrices(M_space_v)
GEs # actually, not necessary for Jordan Normal Form computation
```

```python
relations = generalized_eigenvectors_relations(eigendata)
```

```python
eqs = relations(F.rhs, M_space_v,post=lambda i: i.subs(eigenvals).ratsimp())
eqs
```

```python
miniblocks = Jordan_blocks(eigendata)
miniblocks
```

```python
X, J = Jordan_normalform(eigendata, matrices=(F.rhs, M_space_v, miniblocks))
```

```python
X
```

```python
J
```

```python
assert ((F.rhs*X.rhs).applyfunc(lambda i: i.subs(eigenvals).ratsimp()) ==
        (X.rhs*J.rhs).applyfunc(lambda i: i.subs(eigenvals).ratsimp()))
```

```python
fq = (X.rhs**(-1)*F.rhs*X.rhs).applyfunc(lambda i: i.subs(eigenvals).simplify())
fq
```

```python
x = symbols('x', positive=True)
assert (fq[0, 0].subs({v[0]:1, v[1]:x}).radsimp() == eigenvals[data[1][0]].radsimp())
```

## `power` function

```python
f = Function('f')
f_power = define(let=f(z), be=z**r)
f_power
```

```python
g_power = Hermite_interpolation_polynomial(f_power, eigendata, Phi_polynomials)
g_power
```

```python
g_power = g_power.subs(eigenvals)
g_power
```

```python
with lift_to_matrix_function(g_power) as G_power:
    m_power = G_power(F)
m_power
```

```python
m_power.rhs.subs(eigenvals).subs({r:8}).applyfunc(simplify)
```

```python
F.rhs**r
```

```python
_.subs({r:8}).applyfunc(simplify)
```

# A multiplicity greater than 1

```python
m = define(Symbol(r'\mathcal{A}'), Matrix([
        [1, 0, 0],
        [1, 1, -1],
        [-2, 0, 3],
    ]))
m
```

```python
eigendata = spectrum(m)
eigendata
```

```python
data, eigenvals, multiplicities = eigendata.rhs
```

```python
m_bar = 3 # degree of \Xi minimal polynomial
Phi_poly = Phi_poly_ctor(deg=m_bar-1)
Phi_poly
```

```python
Phi_polynomials = component_polynomials(eigendata)
Phi_polynomials
```

```python
cmatrices = component_matrices(m, Phi_polynomials)
cmatrices
```

```python
Zi1 = list(cm.rhs.as_immutable() for (i, j), cm in cmatrices.items() if j == 1)

s = zeros(m.rhs.rows)
for Z in Zi1:
    s += Z
s, s.simplify(), s.subs(eigenvals)
```

## `inverse` function

```python
f, h = Function('f'), Function('h')
f_inverse = define(let=f(z), be=1/(z))
f_inverse
```

```python
g_inverse = Hermite_interpolation_polynomial(f_inverse, eigendata, Phi_polynomials)
g_inverse
```

```python
g_inverse = g_inverse.subs(eigenvals)
g_inverse
```

```python
with lift_to_matrix_function(g_inverse) as G_inverse:
    m_inverse = G_inverse(m)
m_inverse
```

```python
m.rhs**(-1)
```

---

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.


# Part D: Comparison of toroidal meniscus models with different profile shapes

## Introduction

So far all the capillary entry pressures for the percoaltion examples were calculated using the ``Standard`` physics model which is the ``Washburn`` model for straight walled capillary tubes. This has been shown to be a bad model for fibrous media where the walls of throats are converging and diverging. In the study [Capillary Hysteresis in Neutrally Wettable Fibrous Media: A Pore Network Study of a Fuel Cell Electrode](http://link.springer.com/10.1007/s11242-017-0973-2) percolation in fibrous media was simulated using a meniscus model that assumed the contrictions between fibers are similar to a toroid:

This model was first proposed by Purcell and treats the inner solid profile as a circle. As the fluid invades through the center of the torus the meniscus is pinned to the surface and the "effective" contact angle becomes influenced by the converging diverging geometry and is a function of the filling angle $\alpha$. The shape of the meniscus as the invading phase moves upwards through the torus with key model parameters is shown below.

Different intrinsic contact angles through invading phase are shown above: (a) 60$^\circ$, (b) 90$^\circ$ and (c) 120$^\circ$. All scenarios clearly show an inflection of the meniscus curvature signifying a switch in the sign of the capillary pressure from negative to positive. This inflection is predicted to occur for all contact angles by the model with varying filling angle. The capillary pressure can be shown to be:

$P_C = -2\sigma cos(\theta-\alpha))/(r+R(1-cos(\alpha))$

A consequence of the circular solid profile is that all fluid behaves as non-wetting fluid because $\alpha$ can range from -90$^\circ$ to 90$^\circ$ degrees and so even if $\theta$ is 0 then the meniscus is still pinned at zero capillary pressure at the very furthest part of the throat where the $\alpha$ is 90$^\circ$

Considering other shapes of solid profile this situation can be avoided. It will be shown by reformulating the Purcell model in a more general way that allows for a flexible defintion of the solid profile that filling angle can be limited to values below 90 and allow for spontaneous imbibition (percolation threshold below zero) of highly wetting fluids.

## Set up
We will set up a trivially small network with one throat to demonstrate the use of the meniscus model. Here we do the imports and define a few functions for plotting.

```python
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import sympy as syp
from sympy import lambdify, symbols
from sympy import atan as sym_atan
from sympy import cos as sym_cos
from sympy import sin as sym_sin
from sympy import sqrt as sym_sqrt
from sympy import pi as sym_pi
from ipywidgets import interact, fixed
from IPython.display import display
import warnings
np.random.seed(10)
warnings.simplefilter(action='ignore')
matplotlib.rcParams['figure.figsize'] = (5, 5)
```

```python
theta = 60
fiberRad = 5e-6
throatRad = 1e-5
max_bulge = 1e-5
```

Now we define our two pore network and add the meniscus model in several modes: 'max' returns the maximum pressure experienced by the meniscus as it transitions through the throat, i.e. the burst entry pressure. 'touch' is the pressure at which the meniscus has protruded past the throat center a distance defined by the 'touch_length' dictionary key. In network simulations this could be set to the pore_diameter. Finally the 'men' mode accepts a target_Pc parameter and returns all the mensicus information required for assessing cooperative filling or plotting.

```python
import openpnm as op
import openpnm.models.physics as pm
net = op.network.Cubic(shape=[2, 1, 1], spacing=5e-5)
geo = op.geometry.StickAndBall(network=net,
                               pores=net.pores(),
                               throats=net.throats())
phase = op.phases.Water(network=net)
phase['pore.contact_angle'] = theta
phys = op.physics.Standard(network=net,
                           phase=phase,
                           geometry=geo)
geo['throat.diameter'] = throatRad*2
geo['throat.touch_length'] = max_bulge
```

We define a plotting function that uses the meniscus data:
$\alpha$ is filling angle as defined above,
$radius$ is the radius of curvature of the mensicus,
$center$ is the position of the centre of curvature relative to the throat center along the axis of the throat,
$\gamma$ is the angle between the throat axis and the line joining the meniscus center and meniscus contact point.

```python
def plot_meniscus(target_Pc, meniscus_model=None, ax=None):
    throatRad = geo['throat.diameter'][0]/2
    theta = np.deg2rad(phys['pore.contact_angle'][0])
    throat_a = phys['throat.scale_a']
    throat_b = phys['throat.scale_b']
    x_points = np.arange(-0.99, 0.99, 0.01)*throat_a
    if ax is None:
        fig, ax = plt.subplots()
    if meniscus_model.__name__ == 'purcell':
        # Parameters for plotting fibers
        x, R, rt, s, t = syp.symbols('x, R, rt, s, t')
        y = R*syp.sqrt(1- (x/R)**2)
        r = rt + (R-y)
        rx = syp.lambdify((x, R, rt), r, 'numpy')
        ax.plot(x_points, rx(x_points, fiberRad, throatRad), 'k-');
        ax.plot(x_points, -rx(x_points, fiberRad, throatRad), 'k-');
        phys.add_model(propname='throat.meniscus',
                       model=meniscus_model,
                       mode='men',
                       r_toroid=fiberRad,
                       target_Pc=target_Pc)
    elif meniscus_model.__name__ == 'sinusoidal':
        x, a, b, rt, sigma, theta = syp.symbols('x, a, b, rt, sigma, theta')
        y = (sym_cos(sym_pi*x/(2*a)))*b
        r = rt + (b-y)
        rx = lambdify((x, a, b, rt), r, 'numpy')
        ax.plot(x_points, rx(x_points, throat_a, throat_b, throatRad), 'k-');
        ax.plot(x_points, -rx(x_points, throat_a, throat_b, throatRad), 'k-');
        phys.add_model(propname='throat.meniscus',
                       model=meniscus_model,
                       mode='men',
                       r_toroid=fiberRad,
                       target_Pc=target_Pc)
    else:
        # General Ellipse
        x, a, b, rt, sigma, theta = syp.symbols('x, a, b, rt, sigma, theta')
        profile_equation = phys.models['throat.entry_pressure']['profile_equation']
        if profile_equation == 'elliptical':
            y = sym_sqrt(1 - (x/a)**2)*b
        elif profile_equation == 'sinusoidal':
            y = (sym_cos(sym_pi*x/(2*a)))*b
        r = rt + (b-y)
        rx = lambdify((x, a, b, rt), r, 'numpy')
        ax.plot(x_points, rx(x_points, throat_a, throat_b, throatRad), 'k-');
        ax.plot(x_points, -rx(x_points, throat_a, throat_b, throatRad), 'k-');
        phys.add_model(propname='throat.meniscus',
                       model=meniscus_model,
                       profile_equation=profile_equation,
                       mode='men',
                       target_Pc=target_Pc)
    men_data = {}
    men_data['alpha'] = phys['throat.meniscus.alpha']
    men_data['gamma'] = phys['throat.meniscus.gamma']
    men_data['radius'] = phys['throat.meniscus.radius']
    men_data['center'] = phys['throat.meniscus.center']
    arc_cen = men_data['center']
    arc_rad = men_data['radius']
    arc_angle = men_data['gamma']
    angles = np.linspace(-arc_angle, arc_angle, 100)
    arcx = arc_cen + arc_rad*np.cos(angles)
    arcy = arc_rad*np.sin(angles)
    ax.plot(arcx, arcy, 'b-')
    ax.scatter(phys['throat.meniscus.pos'], phys['throat.meniscus.rx']);
    ax.axis('equal')
    ax.ticklabel_format(style='sci', axis='both', scilimits=(-6,-6))
    return ax
```

# Circular (Purcell)

```python
circular_model = pm.meniscus.purcell

phys.add_model(propname='throat.max',
               model=circular_model,
               mode='max',
               r_toroid=fiberRad)
phys.add_model(propname='throat.touch',
               model=circular_model,
               mode='touch',
               r_toroid=fiberRad)
phys.add_model(propname='throat.meniscus',
               model=circular_model,
               mode='men',
               r_toroid=fiberRad,
               target_Pc=1000)
touch_Pc = phys['throat.touch'][0]
print('Pressure at maximum bulge', np.around(touch_Pc, 0))
max_Pc_circle = phys['throat.max'][0]
print('Circular profile critical entry pressure', np.around(max_Pc_circle, 0))
```

    Pressure at maximum bulge 7213.0
    Circular profile critical entry pressure 8165.0

We can see that the touch_Pc calculated earlier, corresponds with the tip of the meniscus exceeding the max_bulge parameter. Try changing this and re-running to see what happens.

```python
# NBVAL_IGNORE_OUTPUT
ax = plot_meniscus(target_Pc=touch_Pc, meniscus_model=circular_model)
ax.plot([max_bulge, max_bulge], [-throatRad, throatRad], 'r--');
```

```python
# NBVAL_IGNORE_OUTPUT
ax = plot_meniscus(target_Pc=max_Pc_circle, meniscus_model=circular_model)
```

We can interact with the mensicus model by changing the target_Pc parameter.

```python
# NBVAL_IGNORE_OUTPUT
interact(plot_meniscus,
         target_Pc=(-2000, max_Pc_circle, 1),
         meniscus_model=fixed(circular_model),
         ax=fixed(None));
```

    interactive(children=(FloatSlider(value=3082.0, description='target_Pc', max=8165.324889242946, min=-2000.0, s…

Here we can see that the critical entry pressure for the circular profile is positive, even though the intrinsic contact angle is highly non-wetting

# Sinusoidal

Now we can start to compare the different meniscus models:

```python
sinusoidal_model = pm.meniscus.sinusoidal
```

```python
display(sinusoidal_model)
```

    <function openpnm.models.physics.meniscus.sinusoidal(target, mode='max', target_Pc=None, num_points=1000.0, r_toroid=5e-06, throat_diameter='throat.diameter', pore_diameter='pore.diameter', touch_length='throat.touch_length', surface_tension='pore.surface_tension', contact_angle='pore.contact_angle')>

```python
phys.add_model(propname='throat.meniscus',
               model=sinusoidal_model,
               mode='men',
               r_toroid=fiberRad,
               target_Pc=1000)
```

The equation for the solid sinusoidal profile is:

```python
x, a, b, rt, sigma, theta = syp.symbols('x, a, b, rt, sigma, theta')
y = (sym_cos(sym_pi*x/(2*a)))*b
r = rt + b-y
r
```

$\displaystyle - b \cos{\left(\frac{\pi x}{2 a} \right)} + b + rt$

```python
# Derivative of profile
rprime = r.diff(x)
rprime
```

$\displaystyle \frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a}$

```python
# Filling angle
alpha = sym_atan(rprime)
alpha
```

$\displaystyle \operatorname{atan}{\left(\frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a} \right)}$

```python
# angle between y axis, meniscus center and meniscus contact point
eta = sym_pi - (theta + alpha)
eta
```

$\displaystyle - \theta - \operatorname{atan}{\left(\frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a} \right)} + \pi$

```python
# angle between x axis, meniscus center and meniscus contact point
gamma = sym_pi/2 - eta
gamma
```

$\displaystyle \theta + \operatorname{atan}{\left(\frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a} \right)} - \frac{\pi}{2}$

```python
# Radius of curvature of meniscus
rm = r/sym_cos(eta)
rm
```

$\displaystyle - \frac{- b \cos{\left(\frac{\pi x}{2 a} \right)} + b + rt}{\cos{\left(\theta + \operatorname{atan}{\left(\frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a} \right)} \right)}}$

```python
# distance along x-axis from center of curvature to meniscus contact point
d = rm*sym_sin(eta)
d
```

$\displaystyle - \frac{\left(- b \cos{\left(\frac{\pi x}{2 a} \right)} + b + rt\right) \sin{\left(\theta + \operatorname{atan}{\left(\frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a} \right)} \right)}}{\cos{\left(\theta + \operatorname{atan}{\left(\frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a} \right)} \right)}}$

```python
# Capillary Pressure
p = 2*sigma/rm
p
```

$\displaystyle - \frac{2 \sigma \cos{\left(\theta + \operatorname{atan}{\left(\frac{\pi b \sin{\left(\frac{\pi x}{2 a} \right)}}{2 a} \right)} \right)}}{- b \cos{\left(\frac{\pi x}{2 a} \right)} + b + rt}$

```python
phys.add_model(propname='throat.max',
               model=sinusoidal_model,
               mode='max',
               r_toroid=fiberRad)
phys.add_model(propname='throat.touch',
               model=sinusoidal_model,
               mode='touch',
               r_toroid=fiberRad)
max_Pc_sin = phys['throat.max'][0]
print(max_Pc_sin)
```

    4729.770413396985

```python
# NBVAL_IGNORE_OUTPUT
plot_meniscus(target_Pc=max_Pc_sin, meniscus_model=sinusoidal_model);
```

```python
# NBVAL_IGNORE_OUTPUT
interact(plot_meniscus,
         target_Pc=(-2000, max_Pc_sin, 1),
         meniscus_model=fixed(sinusoidal_model),
         ax=fixed(None));
```

    interactive(children=(FloatSlider(value=1364.0, description='target_Pc', max=4729.770413396985, min=-2000.0, s…

Now the crtical entry pressure is negative signifying that spontaneous imbibition will occur

# General Elliptical

Similarly we can define an elliptical profile and use the same method to determine the capillary pressure:

```python
y = sym_sqrt(1 - (x/a)**2)*b
y
```

$\displaystyle b \sqrt{1 - \frac{x^{2}}{a^{2}}}$

In-fact this is the model that OpenPNM uses for Purcell as well with a = b = fiber radius

```python
# Scale ellipse in x direction
phys['throat.scale_a'] = fiberRad
# Scale ellipse in y direction
phys['throat.scale_b'] = fiberRad
general_model = pm.meniscus.general_toroidal
phys.add_model(propname='throat.entry_pressure',
               model=general_model,
               profile_equation='elliptical',
               mode='max')
max_Pc_ellipse = phys['throat.entry_pressure'][0]
print(max_Pc_ellipse)
```

    8165.324889242946

```python
# NBVAL_IGNORE_OUTPUT
plot_meniscus(target_Pc=max_Pc_ellipse, meniscus_model=general_model);
```

```python
max_Pc_ellipse
```

    8165.324889242946

```python
# NBVAL_IGNORE_OUTPUT
interact(plot_meniscus,
         target_Pc=(-2000, max_Pc_ellipse, 1),
         meniscus_model=fixed(general_model),
         ax=fixed(None));
```

    interactive(children=(FloatSlider(value=3082.0, description='target_Pc', max=8165.324889242946, min=-2000.0, s…

The two scale factors can now be used to determine a wide range of capillary behaviours with one general model. Below we run the model for a range of scaling factors showing the effect on the sign and magnitude of the entry pressure.

```python
# NBVAL_IGNORE_OUTPUT
bs = np.linspace(0.2, 1.0, 4)*throatRad
phys['throat.scale_a'] = throatRad
elliptical_pressures = []
sinusoidal_pressures = []
fig, (ax1, ax2) = plt.subplots(2, len(bs), figsize=(10, 10))
for i in range(len(bs)):
    phys['throat.scale_b'] = bs[i]
    phys.add_model(propname='throat.entry_pressure',
                   model=general_model,
                   profile_equation='elliptical',
                   mode='max',
                   num_points=1000)
    Pc = phys['throat.entry_pressure']
    elliptical_pressures.append(Pc)
    plot_meniscus(target_Pc=Pc, meniscus_model=general_model, ax=ax1[i])
for i in range(len(bs)):
    phys['throat.scale_b'] = bs[i]
    phys.add_model(propname='throat.entry_pressure',
                   model=general_model,
                   profile_equation='sinusoidal',
                   mode='max',
                   num_points=1000)
    Pc = phys['throat.entry_pressure']
    sinusoidal_pressures.append(Pc)
    plot_meniscus(target_Pc=Pc, meniscus_model=general_model, ax=ax2[i])
```

```python
# NBVAL_IGNORE_OUTPUT
plt.figure()
plt.plot(bs/throatRad, elliptical_pressures, 'g-');
plt.plot(bs/throatRad, sinusoidal_pressures, 'r-');
```

Here we can see that the two different shaped profiles lead to quite different capiallary behaviour. The elliptical profile always resuls in positive pressure and the meniscus is basically pinned to the end of the throat where highest pressure occurs as alpha always reaches 90. Whereas the sinusiodal model allows for spontaneous imbibition where a breakthrough may occur at negative capillary pressure for wetting fluids if the wall angle is shallow.