$\newcommand{\tensor}[1]{\boldsymbol{#1}}$

$\newcommand{\tensorthree}[1]{\mathbfcal{#1}}$
$\newcommand{\tensorfour}[1]{\mathbb{#1}}$

$\newcommand{\dar}{\, \text{d} a^r}$

$\newcommand{\dvr}{\, \text{d} v^r}$
$\newcommand{\dv}{\, \text{d} v}$

$\newcommand{\dt}{\, \text{d} t}$
$\newcommand{\dthe}{\, \text{d} \theta}$

$\newcommand{\tr}{\operatorname{tr}}$

## The nonlinear heat equation solver in FEniCS

```python
from IPython.display import display_pretty, display_html, display_jpeg, display_png, display_json, display_latex, display_svg

from IPython.display import Image
Image(url='http://python.org/images/python-logo.gif')
```

### Functionals

The total **deformation gradient** can be can be decomposed as:

\begin{equation}
    \tensor{F} = \tensor{F}_e ~ \tensor{F}_{\theta}
\end{equation}

The analysis restricted to isotropic materials, for which the thermal part of the deformation gradient is:

\begin{equation}
    \tensor{F}_{\theta} = \vartheta \left( \theta \right) ~ \tensor{I}
\end{equation}

The scalar $\theta = \upsilon \left( \theta \right)$ is the **thermal stretch ratio** in any material direction. In this case, the elastic and thermal Green strains become:

\begin{equation}
    \tensor{E}_{e} = \frac{1}{\vartheta^2} \left( \tensor{E} - \tensor{E}_{\theta} \right), \qquad \tensor{E}_{\theta} = \frac{1}{2} \left( \vartheta^2 - 1 \right) \tensor{I}
\end{equation}

The relationship holds:

\begin{equation}
    \tensor{I} + 2 \tensor{E} = \vartheta^2 \left( \tensor{I} + 2 \tensor{E}_{e} \right)
\end{equation}

Since the thermal stretch ratio $\vartheta$ and the coefficient of thermal expansion $\alpha$ are related by

\begin{equation}
    \alpha \left( \theta \right) = \frac{1}{\vartheta} \frac{\text{d} \vartheta}{\dthe}
\end{equation}

the rate of elastic strain can be written as

\begin{equation}
    \dot{\tensor{E}}_{e} = \frac{1}{\vartheta^2 \left( \theta \right)} \left[ \dot{\tensor{E}} - \alpha \left( \theta \right) \left( \tensor{I} + 2 \tensor{E} \right) \dot{\theta} \right]
\end{equation}

### Stress Response

Within the model of the multiplicative decomposition, the Helmholtz free energy can be conveniently split into two
parts:

\begin{equation}
    {\varphi} \left( \tensor{u}, \theta \right) = {\varphi}_{e} \left( \tensor{E}_{e}, \theta \right) + {\varphi}_{\theta} \left( \theta \right)
\end{equation}

```python

```

where ${\varphi}_{e}$ is an isotropic function of the elastic strain $\tensor{E}_{e}$ and the temperature $\theta$. This decomposition is physically appealing because the function ${\varphi}_{e}$ can be taken as one of the
well-known strain energy functions of the isothermal finite-strain elasticity, except that the coefficients of the strain-dependent terms are the functions of temperature, while the function ${\varphi}_{\theta}$ can be separately adjusted in accord with experimental data for the specific heat.

The time-rate of the free energy

\begin{equation}
    \dot{\varphi} = \frac{\text{d} \varphi \left( \tensor{u}, \theta \right)}{\dt} = \frac{\partial {\varphi}_{e}}{\partial \tensor{E}_{e}} : \dot{\tensor{E}_{e}} + \frac{\partial {\varphi}_{e}}{\partial \theta} ~ \dot{\theta} + \frac{\text{d} \varphi_{\theta}}{\dthe} ~ \dot{\theta}
\end{equation}

there follows

\begin{equation}
    \dot{\varphi} = \frac{1}{\vartheta^2} \frac{\partial {\varphi}_{e}}{\partial \tensor{E}_{e}} : \dot{\tensor{E}_{e}} - \left[ \frac{\alpha}{\vartheta^2} \frac{\partial {\varphi}_{e}}{\partial \tensor{E}_{e}}: \left( \tensor{I} + 2 \tensor{E} \right) - \frac{\partial {\varphi}_{e}}{\partial \theta} - \frac{\text{d} \varphi_{\theta}}{\dthe} \right] ~ \dot{\theta}
\end{equation}

```python

```

The comparison with the energy equation:

\begin{equation}
    \dot{\varphi} = \frac{1}{\varrho^r} \tensor{S} : \dot{\tensor{E}} - \eta \dot{\theta}
\end{equation}

establishes the constitutive relations for the symmetric second Piola–Kirchhoff stress tensor $\tensor{S}$ and the specific entropy $\eta$. These
are:

\begin{equation}
    \tensor{S} = \frac{\varrho^r}{\vartheta^2} \frac{\partial {\varphi}_{e}}{\partial \tensor{E}_{e}}
\end{equation}

\begin{equation}
    \eta = \alpha \frac{\partial {\varphi}_{e}}{\partial \tensor{E}_{e}} : \left( \tensor{I} + 2 \tensor{E} \right) - \frac{\partial {\varphi}_{e}}{\partial \theta} - \frac{\text{d} \varphi_{\theta}}{\dthe}
\end{equation}

\begin{equation}
    \varrho^r = \upsilon^3 \varrho^{\theta}
\end{equation}

\begin{equation}
    \tensor{S} = \vartheta ~ \tensor{S}_{e}, \qquad \tensor{S}_{e} = \varrho^{\theta} \frac{\partial {\varphi}_{e}}{\partial \tensor{E}_{e}}
\end{equation}

\begin{equation}
    \varrho^{\theta} \frac{\partial {\varphi}_{e}}{\partial \tensor{E}_{e}} = \frac{1}{2} \lambda \left( \theta \right) \left( \tr{\tensor{E}_{e}} \right)^{2} + \mu \left( \theta \right) ~ \tensor{E}_{e} : \tensor{E}_{e}
\end{equation}

where $\lambda \left( \theta \right)$ and $\mu \left( \theta \right)$ are the temperature-dependent Lamé moduli. It follows that

\begin{equation}
    \tensor{S}_{e} = \tensorfour{C} \left( \tensor{u}, \theta \right) : \tensor{E}_{e}
\end{equation}

into $\tensor{S} = \vartheta \tensor{S}_{e}$, the stress response becomes

\begin{equation}
    \tensor{S} = \frac{1}{\vartheta} \left[ \lambda \left( \tr{\tensor{E}} \right) \tensor{I} + 2 \mu \tensor{E} \right] - \frac{3}{2} \left[ \vartheta - \frac{1}{\vartheta} \right] \kappa \tensor{I},
\end{equation}

where $\kappa$ refers to the temperature-dependent bulk modulus. This is an exact expression for the thermoelastic stress response in the case of quadratic representation of $\varphi_e$ e in terms of the finite elastic strain $\tensor{E}_{e}$. If the Lamé moduli are taken to be temperature-independent, and if the approximation $\upsilon \approx 1 + \alpha^{r} \left( \theta - \theta^{r} \right)$ is used ($\alpha^{r}$ o being the coefficient of linear thermal expansion at $\theta - \theta^{r}$), that reduces to

\begin{equation}
    \tensor{S} = \lambda^{r} \left( \tr{\tensor{E}} \right) \tensor{I} + 2 \mu^{r} \tensor{E} - 3 \alpha^{r} \left( \theta - \theta^{r} \right) \kappa^{r} \tensor{I}
\end{equation}

### Entropy expression

In the case of quadratic strain energy representation, there is a relationship $\varrho^r {\varphi}_{e} = \vartheta ^ 3 ~ \tensor{S}_{e} : \tensor{E}_{e}/2 $, so that

\begin{equation}
    \varrho^r \frac{\partial {\varphi}_{e}}{\partial \theta} = \frac{3}{2} \vartheta^2 \frac{\text{d} \varphi_{\theta}}{\dthe} \tensor{S}_{e} : \tensor{E}_{e} + \frac{1}{2} \vartheta^3 \frac{\partial \tensor{S}_{e}}{\partial \theta} : \tensor{E}_{e}
\end{equation}

\begin{equation}
    \varrho^r \frac{\partial {\varphi}_{e}}{\partial \theta} = \frac{3}{2} \alpha \left[ \tensor{S} : \tensor{E} - \frac{1}{2} \left( \vartheta^2 - 1 \right) ~ \tr{\tensor{S}} \right] + \frac{1}{2} \vartheta^3 \frac{\partial {\tensor{S}}_{e}}{\partial \theta} : \tensor{E}_{e}
\end{equation}

The coefficient of thermal expansion $\alpha$ can be readily verified that

\begin{equation}
    \vartheta \frac{\partial {\tensor{S}}_{e}}{\partial \theta} = \frac{\partial {\tensor{S}}}{\partial \theta} + \alpha \left( \tensor{S} + 3 \vartheta \kappa \tensor{I} \right)
\end{equation}

\begin{equation}
    \vartheta^{3} \frac{\partial {\tensor{S}}_{e}}{\partial \theta} : \tensor{E}_{e} = \frac{\partial {\tensor{S}}}{\partial \theta} : \left[ \tensor{E} - \frac{\left( \vartheta^{2} - 1 \right)}{2} ~ \tensor{I} \right] + \alpha \left[ \tensor{S} : \tensor{E} + \frac{\left( 1 + \vartheta^{2} \right)}{2} \tr{\tensor{S}} \right]
\end{equation}

\begin{equation}
    \varrho^r \frac{\partial {\varphi}_{e}}{\partial \theta} = 2 \alpha ~ \tensor{S} : \tensor{E} + \frac{\alpha \left( 2 - \vartheta^{2} \right)}{2} \tr{\tensor{S}} + \frac{1}{2} \frac{\partial {\tensor{S}}}{\partial \theta} : \left[ \tensor{E} - \frac{\left( \vartheta^{2} - 1 \right)}{2} ~ \tensor{I} \right]
\end{equation}

\begin{equation}
    \eta = \frac{1}{2 \varrho^r} \left[ 3 \vartheta \alpha \kappa \tensor{I} - \frac{\partial {\tensor{S}}}{\partial \theta} \right] : \left[ \tensor{E} - \frac{\left( \vartheta^{2} - 1 \right)}{2} ~ \tensor{I} \right] - \frac{\text{d} \varphi_{\theta}}{\dthe}
\end{equation}

Recalling the standard expression for the latent heat $\tensor{\varepsilon}$, we finally have

\begin{equation}
    \eta = \frac{1}{2} \left( \frac{\tensor{\varepsilon}}{\theta} + \frac{\vartheta \alpha \kappa}{\varrho^r} \tensor{I} \right) : \left( \tensor{E} - \frac{\left( \vartheta^{2} - 1 \right)}{2} ~ \tensor{I} \right) - \frac{\text{d} \varphi_{\theta}}{\dthe}
\end{equation}

This is an exact expression for the entropy $\eta$ within the approximation used for the elastic strain energy function. The second-order tensor of the latent heat $\tensor{\varepsilon}$ can be calculated as

\begin{equation}
    \tensor{\varepsilon} = - \frac{\theta}{\varrho^r} \frac{\partial {\tensor{S}}}{\partial \theta} = - \frac{\theta}{\varrho^r} \left( \vartheta \frac{\partial {\tensor{S}}}{\partial \theta} -\alpha \left( \tensor{S} + 3 \vartheta \kappa \tensor{I} \right) \right)
\end{equation}

which gives

\begin{equation}
    \tensor{\varepsilon} =  \frac{\theta}{\varrho^r} \left( \alpha \left( \tensor{S} + 3 \vartheta \kappa \tensor{I} \right) - \frac{1}{\vartheta} \frac{\text{d} \tensorfour{C}}{\dthe} : \left( \tensor{E} - \frac{\left( \vartheta^{2} - 1 \right)}{2} ~ \tensor{I} \right) \right)
\end{equation}

If the elastic moduli are independent of the temperature, and if the stress components are much smaller than the elastic bulk modulus, then the specific heat becomes $\tensor{\varepsilon} = 3 \vartheta \alpha \theta \kappa \tensor{I} / {\varrho^r}$, while the entropy expression reduces to

\begin{equation}
    \eta = \frac{3}{\varrho^r} \vartheta \alpha \kappa \left( \tr{\tensor{E}} - \frac{3}{2} \left( \vartheta^2 - 1 \right) \right) - \frac{\text{d} \varphi_{\theta}}{\dthe}
\end{equation}

The function $\varphi_{\theta}$ can be selected according to experimental data for the specific heat $c_{E} = \theta \partial \eta / \partial \theta$. For example, if we take

\begin{equation}
    \varphi_{\theta} = -\frac{1}{2} \left( \frac{c_{E}}{\theta^r} + \frac{9 \left( \alpha^r \right)^2 \kappa^r}{\varrho^r} \right) \left( \theta - \theta^r \right)^2
\end{equation}

then becomes

\begin{equation}
    \eta = \frac{3}{\varrho^r} \alpha^r \kappa^r \tr{\tensor{E}} + \frac{c_{E}}{\theta^r} \left( \theta - \theta^r \right)
\end{equation}

which is in agreement with the classical result from the linearized theory of thermoelasticity. The approximations $\alpha \approx \alpha^r$ and $\vartheta \approx 1 + \alpha^r \left( \theta - \theta^r \right)$ are used in the above derivation.

```python

```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Stratified Sphere Radar Cross Section
***

Referring to Section 7.4.1.5, Mie gives the exact solution for scattering from a sphere.  The solution is composed of vector wave functions defined in a spherical coordinate system.  The terms of the Mie series are obtained from boundary value techniques.  Therefore, the Mie formulation may be employed regardless of the composition of the sphere.  To calculate the radar cross section of a sphere, use the Mie formulation along with far field approximations to give (Equations 7.42 and 7.43)

\begin{align}
    S_1(\theta_o) &= \sum\limits_{n=1}^\infty(j)^{n+1}\Big[A_n\frac{P_n^1(\cos\theta_o)}{\sin\theta_o} - jB_n\frac{d}{d\theta_o}P_n^1(\cos\theta_o)\Big], \\ \nonumber \\
    S_2(\theta_o) &= \sum\limits_{n=1}^\infty(j)^{n+1}\Big[A_n\frac{d}{d\theta_o}\frac{P_n^1(\cos\theta_o)}{\sin\theta_o} - jB_nP_n^1(\cos\theta_o)\Big],
\end{align}

where $P_n^1$ is the associated Legendre polynomial and may be calculated using the SciPy implementation ***scipy.special.lpmn(m, n, z)***.  $S_1(\theta_o)$ and $S_2(\theta_o)$ are the complex far-field scattered radiation values for the $\hat{\theta}$ and $\hat{\phi}$ directions.  The radar cross section for the $\hat{\theta}$ and $\hat{\phi}$ polarization states is then found to be (Equations 7.44 and 7.45)

\begin{align}\label{eq:rcs_mie}
    \sigma_\theta &= \frac{4\pi}{k_0^2}S_1(\theta_o)\cos^2(\phi_0) \hspace{0.5in} \text{(m^2)}, \\ \nonumber \\
    \sigma_\phi &= \frac{4\pi}{k_0^2}S_2(\theta_o)\sin^2(\phi_0) \hspace{0.5in} \text{(m}^2\text{)}.
\end{align}

For the $N$-layer concentric sphere, use the Mie coefficients of the following form (Equations 7.46 and 7.47)

\begin{align}\label{eq:mie_coefficients_layered}
    A_n = -(j)^n &\frac{2n+1}{n(n+1)}\frac{k_0a_0J_n(k_0a_0) + jZ_n(k_0a_0)(k_0a_0J_n^\prime(k_0a_0)}{k_0a_0H_n(k_0a_0) + jZ_n(k_0a_0)(k_0a_0H_n^\prime(k_0a_0)},  \\ \nonumber \\
    B_n = (j)^n &\frac{2n+1}{n(n+1)}\frac{k_0a_0J_n(k_0a_0) + jY_n(k_0a_0)(k_0a_0J_n^\prime(k_0a_0)}{k_0a_0H_n(k_0a_0) + jY_n(k_0a_0)(k_0a_0H_n^\prime(k_0a_0)}.
\end{align}

Ruck et al. showed that the modal surface impedance and admittance can be derived using an iterative technique similar to the method used for transmission lines.  To begin, the impedance at the interface between the core and the first layer, $Z_n^N$, is determined independently.  Then, the impedance at the second interface, $Z_n^{N-1}$, is determined from $Z_n^N$.  This process continues until the impedance at the outermost surface, $Z_n^0$, is found.  Then $Z_n(k_0a_0) = j(Z_n^0/\eta)$.  Following the same process for the admittance, $Y_n(k_0a_0)$ may also be calculated.
The impedance and admittance are used in the Mie coefficients of (7.46) for the scattering radiation calculation in (7.42).  Finally, the radar cross section is obtained from (7.44).

***

Begin by getting the library path

```python
import lib_path
```

Set the operating frequency (Hz), the radii (m), the relative permeabilities, the relative permittivities, the number of modes and the flag for perfectly conducting core

```python
frequency = 1e9

radius = [1.0, 1.25]

mu_r = [1.0, 1.0]

eps_r = [1.0, 4.0]

number_of_modes = 60

pec = True
```

Size the ordered arrays

```python
from numpy import ones

nr = len(radius)

mu = ones(nr + 1)

eps = ones(nr + 1)

ra = ones(nr)
```

Set up the parameters in the correct order

```python
i = 0

for r in radius:

    ra[nr - 1 - i] = float(r)

    i += 1

i = 0

for m, e in zip(mu_r, eps_r):

    mu[nr - i] = float(m)

    eps[nr - i] = float(e)

    i += 1
```

Set the observation angles (radians) using the `linspace` routine from `scipy`

```python
from numpy import linspace

from scipy.constants import pi

observation_angle = linspace(0, pi, 721)
```

Calculate the coefficients for the sphere

```python
from Libs.rcs.stratified_sphere import coefficients

An, Bn = coefficients(frequency, eps, mu, ra, number_of_modes, pec)
```

Calculate the radar cross section (m^2) for the stratified sphere

```python
from Libs.rcs.stratified_sphere import radar_cross_section

from numpy import array

et = array([radar_cross_section(frequency, oa, 0, An, Bn) for oa in observation_angle])

ep = array([radar_cross_section(frequency, oa, 0.5 * pi, An, Bn) for oa in observation_angle])
```

Display the radar cross section (dBsm) for the stratified sphere using the `matplotlib` routines

```python
from matplotlib import pyplot as plt

from numpy import log10, degrees

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(degrees(observation_angle), 20.0 * log10(abs(ep[:, 1])), '', label='TE')

plt.plot(degrees(observation_angle), 20.0 * log10(abs(et[:, 0])), '--', label='TM')

# Set the plot title and labels

plt.title('RCS vs Bistatic Angle', size=14)

plt.ylabel('RCS (dBsm)', size=12)

plt.xlabel('Observation Angle (deg)', size=12)

plt.ylim(min(20.0 * log10(abs(et[:,0]))) - 3, max(20.0 * log10(abs(et[:,0]))) + 3)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the legend

plt.legend(loc='upper left', prop={'size': 10})
```

```python

```


# The GLM, part 2: inference
In this notebook, we'll continue with the GLM, focusing on statistical tests (i.e., inference) of parameters. Note that there are two notebooks this week: this one, `glm_part2_inference.ipynb`, and `design_of_experiments.ipynb`. Please do this one first.

Last week, you learned how to estimate parameters of the GLM and how to interpret them. This week, we'll focus on statistical inference of those estimated parameters (and design of experiment, in another notebook). Importantly, we are going to introduce the most important formula in the context of univariate fMRI analyses: the formula for the *t-value*. Make sure you understand this formula, as we will continue to discuss it in the next weeks.

**What you'll learn**: after this week's lab ...
* you know how the different parts of the t-value formula and how they relate to your data and experiment;
* you are able to calculate t-values and corresponding p-value of parameters from a GLM;

**Estimated time needed to complete**: 1-3 hours

```python
# First some imports
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv
%matplotlib inline
```

## Introduction
From your statistics classes, you might remember that many software packages (e.g. SPSS, R, SAS) do not only return beta-parameters of linear regression models, but also *t*-values and *p*-values associated with those parameters. Like beta-parameters, these statistics evaluate whether a beta-parameter (or combination of beta-parameters) differs significantly from 0 (or in fMRI terms: whether a voxel activates/deactivates significantly in response to one or more experimental factors).

In univariate (activation-based) fMRI studies, we need statistics to evaluate the estimated parameters in context of the *uncertainty* of their estimation. As we'll discuss later in more detail, interpreting (and performing inference about) the magnitude of GLM parameters without their associated uncertainty is rarely warranted in univariate fMRI studies. To illustrate the problem with this, let's look at an example.

In this example, we try to predict someone's height (in meters; $\mathbf{y}$) using someone's weight (in kilos; $\mathbf{X}$). (Note that the data is not necessarily representative of the true relationship between height and weight.)

Anyway, let's run a linear regression using weight (in kilos) as a predictor for height (in meters).

```python
data = np.load('weight_height_data.npz')
X, y = data['X'], data['y']

plt.figure(figsize=(10, 6))
plt.scatter(X, y)
plt.title('Relation between weight and height (in meters)', y=1.05, fontsize=20)
plt.xlabel('Weight (kg)', fontsize=20)
plt.ylabel('Height (meters)', fontsize=20)

Xn = np.hstack((np.ones((y.size, 1)), X))
beta = inv(Xn.T @ Xn) @ Xn.T @ y
y_hat = Xn @ beta
mse = np.mean((y_hat - y) ** 2)
plt.plot([X.min(), X.max()], [Xn.min(axis=0) @ beta, Xn.max(axis=0) @ beta], ls='-', c='r')
plt.xlim((X.min(), X.max()))
plt.text(70, 1.9, r'$\hat{\beta}_{weight} = %.5f$' % beta[1], fontsize=18)
plt.text(70, 1.8, r'$MSE = %.5f$' % mse, fontsize=18)
plt.grid()

plt.show()
```

Well, quite a modest beta-parameter on the one hand, but on the other hand the Mean Squared Error is also quite low.
Now, to illustrate the problem of interpretating 'raw' beta-weights, let's rephrase our objective of predicting height based on weight: we'll try to predict *height in centimeters* based on weight (still in kilos). So, what we'll do is just rescale the data points of $\mathbf{y}$ (height in meters) so that they reflect height in centimeters. We can simply do this by multipling our $\mathbf{y}$ by 100.

```python
y_cm = y * 100
```

Now, you wouldn't expect our model to change, right? We only rescaled our target ... As you'll see below, this actually changes a lot!

    ToDo (0 points): Run linear regression like the previous code block, but with <tt>y_cm</tt> instead of <tt>y</tt> as the target variable. You can use the same design (<tt>Xn</tt>). Calculate the beta-parameter and MSE (store them in the variables <tt>beta_cm</tt> and <tt>mse_cm</tt>).

```python
''' Implement the ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
print(beta_cm)
```

```python
''' Tests the above ToDo'''
np.testing.assert_almost_equal(beta_cm, beta * 100, decimal=4)
np.testing.assert_almost_equal(mse_cm, mse * 10000, decimal=4)
print("Well done!")
```

If you did it correctly, when you compare the beta-parameters between the two models (one where $y$ is in meters, and one where $y$ is in centimeters), you see a massive difference — a 100 fold difference to be exact\*! This is a nice example where you see that the (raw) value of the beta-parameter is completely dependent on the scale of your variables. (Actually, you could either rescale $\mathbf{X}$ or $\mathbf{y}$); both will have a similar effect on your estimated beta-parameter.)

    ToThink (0 points): Note that the MSE is a 10,000 times larger in the model with <tt>y_cm</tt> compared to <tt>y</tt> (in meters). From your understanding of how MSE is calculated, do you understand why?

ToThink (2 points): By now, you know that the scale of the data (either $X$ or $y$) influences the magnitude of the raw parameter estimates. One could argue that this is not relevant for fMRI data because all data (i.e. different voxels in the brain) all measure the same type of signal, so their scale shouldn't differ that much. This, however, is a false assumption.

Think of (at least) two reasons why voxels might differ in their scale and write them down in the text cell below.

YOUR ANSWER HERE

## How to compute *t*-values and *p*-values
So, you've seen that interpreting beta-parameters by themselves is useless because their value depends very much on the scale of your variables. But how should we, then, interpret the effects of our predictors on our target-variable? From the plots above, you probably guessed already that it has something to do with the MSE of our model (or, more generally, the model fit). That is indeed the case. As you might have noticed, not only the beta-parameters depend on the scale of your data, the errors (residuals) depend on the scale as well. In other words, not only the *effect* (beta-values) but also the *noise* (errors, MSE) depend on the scale of the variables!

### *t*-values
In fact, the key to getting interpretable effects of our predictors is to divide ("normalize") our beta-parameter(s) by some quantity that summarizes how well our model describes the data. This quantity is the **standard error of the beta-parameter**, usually denoted by $\mathrm{SE}_{\beta}$. The standard error of the beta-parameter can be computed by taking the square root of the **variance of the beta-parameter**. If we'd divide our beta-estimate with it's standard error, we compute a statistic you are all familiar with: the *t*-statistic! Formally:

\begin{align}
t_{\hat{\beta}} = \frac{\hat{\beta}}{\mathrm{SE}_{\hat{\beta}}} = \frac{\hat{\beta}}{\sqrt{\mathrm{var}(\hat{\beta})}}
\end{align}

    ToThink (0 points): Suppose that I know the $\mathrm{SE}$ of a particular beta-parameter. How can I derive the variance of that parameter (i.e., how do I go from the $\mathrm{SE}$ to the variance)? And yes, the answer is as straightforward as you'd think.

Another way to think about it is that the t-value is the "effect" ($\hat{\beta}$) divided by your (un)certainty or confidence in the effect ($\mathrm{SE}_{\hat{\beta}}$). In a way, you can think of t-values as "uncertainty-normalized" effects.

So, what drives (statistical) uncertainty about "effects" (here: $\hat{\beta}$ parameters)? To find out, let's dissect the uncertainty term, $\mathrm{SE}_{\hat{\beta}}$, a little more. The standard error of a parameter can interpreted conceptually as the "unexplained variance of the model" (or *noise*) multiplied with the "design variance" (or: *the variance of the parameter due to the design*). In this lab, we won't explain what *design variance* means or how to compute this, as this is the topic of the second notebook of this week (`design_of_experiments`).

For now, we treat "design variance", here, as some known (constant) value given the design matrix ($\mathbf{X}$). So, with this information, we can construct a conceptual formula for the standard error of our parameter(s):

\begin{align}
\mathbf{SE}_{\hat{\beta}} = \sqrt{\mathrm{noise} \cdot \mathrm{design\ variance}}
\end{align}

Now we also create a "conceptual formula" for the *t*-statistic:

\begin{align}
t_{\hat{\beta}} = \frac{\hat{\beta}}{\mathrm{SE}_{\hat{\beta}}} = \frac{\mathrm{effect}}{\sqrt{\mathrm{noise} \cdot \mathrm{design\ variance}}}
\end{align}

**This (conceptual) formula involving effects, noise, and design variance is probably the most important concept of this course**. The effects (*t*-values) we measure in GLM analyses of fMRI data depend on two things: the effect measured ($\hat{\beta}$) and the (un)certainty of the effect ($SE_{\hat{\beta}}$), of which the latter term can be divided into the unexplained variance ("noise") and the design variance (uncertainty of the parameter due to the design).

These two terms (noise and design variance) will be central to the next couple of weeks of this course. In this week's second notebook (topic: design of experiments), we'll focus on how to optimize our *t*-values by minimizing the "design variance" term. Next week (topic: preprocessing), we'll focus on how to (further) optimize our *t*-values by minimizing the error/noise.

While we're going to ignore the design variance for now, we are, however, going to learn how to calculate the "noise" term.

In fact, the noise term is *very* similar to the MSE, but instead of taking the *mean* of the squared residuals, we sum the squared residuals ("sums of squared erros", SSE) and divide it by the model's degrees of freedom (DF). People usually use the $\hat{\sigma}^{2}$ symbol for this noise term:

\begin{align}
\mathrm{noise} = \hat{\sigma}^{2} = \frac{\sum_{i=1}^{N}(\hat{y_{i}} - y_{i})^2}{\mathrm{df}}
\end{align}

where the degrees of freedom (df) are defined as the number of samples ($N$) minus the number of predictors *including the intercept* ($P$):

\begin{align}
\mathrm{df} = N - P
\end{align}

So, the formula of the *t*-statistic becomes:

\begin{align}
t_{\hat{\beta}} = \frac{\hat{\beta}}{\sqrt{\frac{\sum_{i=1}^{N}(\hat{y_{i}} - y_{i})^2}{N - P} \cdot \mathrm{design\ variance}}}
\end{align}

Alright, enough formulas. Let's see how we can compute these terms in Python. We're going to calculate the *t*-statistic of the weight-predictor for both models (the meter and the centimeter model) to see whether we can show that essentially the (normalized) effect of weight on height in meters is the same as the effect on height in centimeters; in other words, we are going to investigate whether the conversion to *t*-values "normalizes" the beta-parameters.

First, we'll create a function for you to calculate the design-variance. You *don't* have to understand how this works; we're going to explain this to you in detail next week.

```python
def design_variance(X, which_predictor=1):
    ''' Returns the design variance of a predictor (or contrast) in X.

    Parameters
    ----------
    X : numpy array
        Array of shape (N, P)
    which_predictor : int or list/array
        The index of the predictor you want the design var from.
        Note that 0 refers to the intercept!
        Alternatively, "which_predictor" can be a contrast-vector
        (which will be discussed later this lab).

    Returns
    -------
    des_var : float
        Design variance of the specified predictor/contrast from X.
    '''

    is_single = isinstance(which_predictor, int)
    if is_single:
        idx = which_predictor
    else:
        idx = np.array(which_predictor) != 0

    c = np.zeros(X.shape[1])
    c[idx] = 1 if is_single == 1 else which_predictor[idx]
    des_var = c.dot(np.linalg.inv(X.T.dot(X))).dot(c.T)
    return des_var
```

So, if you want the design variance of the 'weight' parameter in the varianble `Xn` from before, you do:

```python
# use which_predictor=1, because the weight-column in Xn is at index 1 (index 0 = intercept)
design_variance_weight_predictor = design_variance(Xn, which_predictor=1)
print("Design variance of weight predictor is: %.6f " % design_variance_weight_predictor)
```

Alright, now we only need to calculate our noise-term ($\hat{\sigma}^2$):

```python
# Let's just redo the linear regression (for clarity)
beta_meter = inv(Xn.T @ Xn) @ Xn.T @ y
y_hat_meter = Xn @ beta_meter

N = y.size
P = Xn.shape[1]
df = (N - P)
print("Degrees of freedom: %i" % df)
sigma_hat = np.sum((y - y_hat_meter) ** 2) / df
print("Sigma-hat (noise) is: %.3f" % sigma_hat)
design_variance_weight = design_variance(Xn, 1)
```

Now we can calculate the *t*-value:

```python
t_meter = beta_meter[1] / np.sqrt(sigma_hat * design_variance_weight)
print("The t-value for the weight-parameter (beta = %.3f) is: %.3f" % (beta_meter[1], t_meter))
```

That's it! There's not much more to calculating *t*-values in linear regression. Now it's up to you to do the same thing and calculate the *t*-value for the model of height in centimeters, and check if it is the same as the *t*-value for the weight parameter in the model with height in meters.

    ToDo (1 point): Calculate the t-statistic for the beta from the centimeter-model you calculated earlier. Store the value in a new variable named <tt>t_centimeter</tt>. Note: you don't have to calculate the design variance again (because <tt>X</tt> hasn't changed!) — you can reuse the variable <tt>design_variance_weight</tt>.

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
try:
    np.testing.assert_almost_equal(t_centimeter, t_meter)
except AssertionError as e:
    print("The t-value using height in centimeters is not the same as when using height in meters!")
    raise(e)

print("Well done!")
```

### P-values
As you can see, calculating *t*-values solves the "problem" of uninterpretable beta-parameters!

Now, the last thing you need to know is how to calculate the statistical significance of your *t*-value, or in other words, how you calculate the corresponding *p*-value. You probably remember that the *p*-value corresponds to the area under the curve of a *t*-distribution associated with your observed *t*-value *and more extreme values*:

*Image credits: Frank Dietrich and Mike Collins, Northern Kentucky University*

The function `stats.t.sf(t_value, df)` from the `scipy` package does exactly this. Importantly, this function *always* returns the right-tailed p-value. For negative t-values, however, you'd want the left-tailed *p*-value. One way to remedy this, is to always pass the absolute value of your *t*-value - `np.abs(t_value)` to the `stats.t.sf()` function. Also, the `stats.t.sf()` function by default returns the one-sided *p*-value. If you'd want the two-sided *p*-value, you can simply multiply the returned *p*-value by two to get the corresponding two-sided *p*-value.

Let's see how we'd do that in practice:

```python
from scipy import stats

# take the absolute by np.abs(t)
p_value = stats.t.sf(np.abs(t_meter), df) * 2 # multiply by two to create a two-tailed p-value
print('The p-value corresponding to t(%i) = %.3f is: %.8f' % (df, t_meter, p_value))
```

## Contrasts
We're almost done! We're really at 99% of what you should know about the GLM and fMRI analysis (except for some important caveats that have to do with GLM assumptions, that we'll discuss next week). The only major concept that we need to discuss is **contrasts**. Contrasts are basically follow-up statistical tests of GLM parameters, with which you can implement any (linear) statistical test that you are familiar with. *t*-tests, *F*-tests, ANCOVAs — they can all be realized with the GLM and the right contrast(s). (Again, if you want to know more about this equivalence between the GLM and common statistical tests, check out this [blog post](https://lindeloev.github.io/tests-as-linear/).) Importantly, the choice of contrast should reflect the hypothesis that you want to test.

### *t*-tests
T-tests in the GLM can be implemented in two general ways:

**1. Using a contrast of a parameters "against baseline"**

This type of contrast basically tests the hypothesis: "Does my predictor(s) have *any* effect on my dependent variable?" In other words, it tests the following hypothesis:
* $H_{0}: \beta = 0$        (our null-hypothesis, i.e. no effect)
* $H_{a}: \beta \neq 0$     (our two-sided alternative hypothesis, i.e. *some* effect)

Note that a directional alternative hypothesis is also possible, i.e., $H_{a}: \beta > 0$ or $H_{a}: \beta < 0$.

**2. Using a contrast between parameters**

This type of contrast basically tests hypotheses such as "Does predictor 1 have a larger effect on my dependent variable than predictor 2?". In other words, it tests the following hypothesis:
* $H_{0}: \beta_{1} - \beta_{2} = 0$ (our null-hypothesis, i.e. there is no difference)
* $H_{a}: \beta_{1} - \beta_{2} \neq 0$     (our alternative hypotehsis, i.e. there is some difference)

Let's look at an example of how we would evaluate a simple hypothesis that a beta has an *some* effect on the dependent variable. Say we'd have an experimental design with 6 conditions:

* condition 1: images of **male** faces with a **happy** expression
* condition 2: images of **male** faces with a **sad** expression
* condition 3: images of **male** faces with a **neutral** expression
* condition 4: images of **female** faces with a **happy** expression
* condition 5: images of **female** faces with a **sad** expression
* condition 6: images of **female** faces with a **neutral** expression

Let's assume we have fMRI data from a run with 100 volumes. We then have a target-signal of shape ($100,$) and a design-matrix (after convolution with a canonical HRF) of shape ($100 \times 7$) (the first predictor is the intercept!). We load in this data below:

```python
data = np.load('data_contrast_example.npz')
X, y = data['X'], data['y']

print("Shape of X: %s" % (X.shape,))
print("Shape of y: %s" % (y.shape,))
```

After performing linear regression with these 6 predictors (after convolving the stimulus-onset times with an HRF, etc. etc.), you end up with 7 beta values:

```python
betas = inv(X.T @ X) @ X.T @ y
betas = betas.squeeze()  # remove singleton dimension; this is important for later
print("Betas corresonding to our 6 conditions (and intercept):\n%r" % betas.T)
```

The first beta corresponds to the intercept, the second beta to the male/happy predictor, the third beta to the male/sad predictor, etc. etc. Now, suppose that we'd like to test whether images of male faces with a sad expression have an influence on voxel activity (our dependent variable).

The first thing you need to do is extract this particular beta value from the array with beta values (I know this sounds really trivial, but bear with me):

```python
beta_male_sad = betas[2]
print("The 'extracted' beta is %.3f" % beta_male_sad)
```

In neuroimaging analyses, however, this is usually done slightly differently: using **contrast-vectors**. Basically, it specifies your specific hypothesis about your beta(s) of interest in a vector. Before explaining it in more detail, let's look at it in a code example:

```python
# Again, we'd want to test whether the beta of "male_sad" is different from 0
contrast_vector = np.array([0, 0, 1, 0, 0, 0, 0])
contrast = (betas * contrast_vector).sum() # we simply elementwise multiply the contrast-vector with the betas and sum it!
print('The beta-contrast is: %.3f' % contrast)
```

"Wow, what a tedious way to just select the third value of the beta-array", you might think. And, in a way, this is indeed somewhat tedious for a contrast against baseline. But let's look at a case where you would want to investigate whether two betas are different - let's say whether male sad faces have a larger effect on our voxel than male happy faces. Again, you *could* do this:

```python
beta_difference = betas[2] - betas[1]
print("Difference between betas: %.3f" % beta_difference)
```

... but you could also use a contrast-vector:

```python
contrast_vector = np.array([0, -1, 1, 0, 0, 0, 0])
contrast = (betas * contrast_vector).sum()
print('The contrast between beta 2 and beta 1 is: %.3f' % contrast)
print('This is exactly the same as beta[2] - beta[1]: %.3f' % (betas[2]-betas[1]))
```

"Alright, so using contrast-vectors is just a fancy way of extracting and subtracting betas from each other ...", you might think. In a way, that's true. But you have to realize that once the hypotheses you want to test become more complicated, using contrast-vectors actually starts to make sense.

Let's look at some more elaborate hypotheses. First, let's test whether male faces lead to higher voxel activity than female faces, *regardless of emotion*:

```python
# male faces > female faces
contrast_vector = [0, 1, 1, 1, -1, -1, -1]
male_female_contrast = (contrast_vector * betas).sum()
print("Male - female contrast (regardless of expression): %.2f" % male_female_contrast)
```

... or whether emotional faces (regardless of *which* exact emotion) lead to higher activity than neutral faces:

```python
# Emotion (regardless of which emotion, i.e., regardless of sad/happy) - neutral
contrast_vector = np.array([0, 1, 1, -2, 1, 1, -2])
emo_neutral_contrast = (contrast_vector * betas).sum()
print("Emotion - neutral contrast (regardless of which emotion): %.2f" % emo_neutral_contrast)
```

See how contrast-vectors come in handy when calculating (more intricate) comparisons? In the male-female contrast, for example, instead 'manually' picking out the betas of 'sad_male' and 'happy_male', averaging them, and subtracting their average beta from the average 'female' betas ('happy_female', 'sad_female'), you can
specify a contrast-vector, multiply it with your betas, and sum them. That's it.

ToThink (1 point): In the last contrast (<tt>emo_neural_contrast</tt>), we set all the "emotional" predictors (sad/happy) to 1, but the neutral predictors to minus 2 ... Why are these set to -2 and not -1? Write your answer below.

YOUR ANSWER HERE

ToDo (1 point): Create a contrast vector for the hypothesis: sad faces (regardless whether it's male or female) activate this voxel more than neutral faces (regardless of whether it's male/female). Multiply this contrast vector with the betas and store the result in a variable named <tt>contrast_todo</tt>.

```python
# Implement the sad - neutral contrast here:

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_3 import test_contrast_todo_1
test_contrast_todo_1(betas, contrast_todo)
```

We're not only telling you about contrasts because we think it's an elegant way of computing beta-comparisons, but also because virtually every major neuroimaging software package uses them, so that you can specify what hypotheses you exactly want to test! You'll also see this when we're going to work with FSL (in week 5) to perform automated whole-brain linear regression analyses.

Knowing how contrast-vectors work, we now can extend our formula for *t*-tests of beta-parameters such that they can describe **every possible test** (not only *t*-tests, but also ANOVAs, *F*-tests, etc.) of betas "against baseline" or between betas that you can think of:

Our "old" formula of the *t*-test of a beta-parameter:
\begin{align}
t_{\hat{\beta}} = \frac{\hat{\beta}_{j}}{\mathrm{SE}_{\hat{\beta}}}
\end{align}

And now our "generalized" version of the *t*-test of *any* contrast/hypothesis:

\begin{align}
t_{\mathbf{c}\hat{\beta}} = \frac{\sum_{j=1}^{P}{c_{j}\hat{\beta}_{j}}}{\mathrm{SE}_{\mathbf{c}\hat{\beta}}}
\end{align}

in which $\mathbf{c}$ represents the entire contrast-vector, and $c_{j}$ represents the $j^{\mathrm{th}}$ value in our contrast vector. By the way, we can simplify the (notation of the) numerator a little bit using some matrix algebra. Remember that multiplying two (equal length) vectors with each other and then summing the values together is the same thing as the (inner) "dot product" between the two vectors?

This means that you can also evaluate this elementwise multiplication and sum of the contrast-vector and the betas using the dot-product:

\begin{align}
t_{\mathbf{c}\hat{\beta}} = \frac{\mathbf{c}\hat{\beta}}{\mathrm{SE}_{\mathbf{c}\hat{\beta}}}
\end{align}

    ToDo (0 points): Convince yourself that the elementwise multiplication and sum is mathematically exactly the same as the dot product! Below, we initialized a hypothetical vector with beta-values (<tt>some_betas</tt>) and a hypothetical contrast-vector (<tt>some_cvec</tt>). First, implement the "multiply and sum" approach and then implement the "dot product" approach. You should find that it gives you exactly the same value: -3.34

```python
some_betas = np.array([1.23, 2.95, 3.33, 4.19])
some_cvec = np.array([1, 1, -1, -1])

# Try to implement both approaches and convince yourself that it's
# mathematically the same!

```

So, you need the contrast vector in the *numerator* of the *t*-value formula (i.e., $\mathbf{c}\hat{\beta}$), but it turns out that you actually also need the contrast-vector in the denominator, because it's part of the calculation of design variance. Again, we will discuss how this works exactly in the next notebook. In the function `design_variance`, it is also possible to calculate design variance for a particular contrast (not just a single predictor) by passing a contrast vector to the `which_predictor` argument.

We'll show this below:

```python
# E.g., get design-variance of happy/male - sad/male
c_vec = np.array([0, 1, -1, 0, 0, 0, 0])  # our contrast vector!
dvar = design_variance(X, which_predictor=c_vec)  # pass c_vec to which_predictor
print("Design variance of happy/male - sad/male: %.3f" % dvar)
```

For the rest of ToDos this lab, make sure to pass your contrast-vector to the `design_variance` function in order to calculate it correctly.

Now you know enough to do it yourself!

ToDo (2 points):

Calculate the *t*-value and *p*-value for the hypothesis "sad faces have a larger effect than happy faces (regardless of gender) on our dependent variable" (i.e. voxel activity). In other words, test the hypothesis: $\beta_{sad} - \beta_{happy} \neq 0$ (note that this is a two-sided test!).

Store the *t*-value and *p*-value in the variables <tt>tval_todo</tt> and <tt>pval_todo</tt> respectively. We reload the variables below (we'll call them <tt>X_new</tt> and <tt>y_new</tt>) to make sure you're working with the correct data. Note that the <tt>X_new</tt> variable already contains an intercept; the other six columns correspond to the different predictors (male/hapy, male/sad, etc.). In summary, you have to do the following:

- (you don't have to calculate the betas; this has already been done (stored in the variable <tt>betas</tt>)
- calculate "sigma-hat" ($\mathrm{SSE} / \mathrm{df}$)
- calculate design-variance (use the <tt>design_variance</tt> function with a proper contrast-vector)
- calculate the contrast ($\mathbf{c}\hat{\beta}$)
- calculate the t-value and p-value

```python
data = np.load('data_contrast_example.npz')
X_new, y_new = data['X'], data['y']

print("Shape of X: %s" % (X_new.shape,))
print("Shape of y: %s" % (y_new.shape,))

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Part 1 of testing the above ToDo. '''
from niedu.tests.nii.week_3 import test_contrast_todo_2
test_contrast_todo_2(X_new, y_new, betas, tval_todo, pval_todo)
print("Well done!")
```

### *F*-tests on contrasts
In the previous section we discussed how to calculate *t*-values for single contrasts. However, sometimes you might have a hypothesis about multiple contrasts at the same time. This may sound weird, but let's consider an experiment.

Suppose you have data from an experiment in which you showed images circles which were either blue, red, or green. In that case, you have three predictors. Then, you could have very specific question, like "Do blue circles activate a voxel significantly compared to baseline", which corresponds to the following null and alternative hypothesis:

* $H_{0}: \beta_{blue} = 0$ (our null-hypothesis, i.e. there is no activation compared to baseline)
* $H_{a}: \beta_{blue} > 0$   (our alternative hypothesis, i.e. blue activates relative to baseline)

However, you can also have a more general question, like "Does the presentation of *any* circle (regardless of color) activate a voxel compared to baseline?". This question represents the following null and alternative hypothesis:

* $H_{0}: \beta_{blue} = \beta_{red} = \beta_{green} = 0$
* $H_{a}: (\beta_{blue} > 0) \vee (\beta_{red} > 0) \vee (\beta_{green} > 0)$

The $\vee$ symbol in the alternative hypothesis means "or". So the alternative hypothesis nicely illustrates our question: is there *any* condition (circle) that activates a voxel more than baseline? This hypothesis-test might sound familiar, because it encompasses the *F*-test! In other words, an *F*-test tests *a collection of contrasts* together. In the example here, the *F*-test tests the following contrasts together (ignoring the intercept) of our beta-parameters:

* `[1, 0, 0]` ($\mathrm{red} > 0$)
* `[0, 1, 0]` ($\mathrm{blue} > 0$)
* `[0, 0, 1]` ($\mathrm{green} > 0$)

Thus, a *F*-test basically tests this contrast-*matrix* all at once! Therefore, the *F*-tests is a type of "omnibus test"!

Now, let's look at the math behind the *F*-statistic. The *F*-statistic for set of $K$ contrasts (i.e., the number of rows in the contrast-matrix) is defined as follows:

\begin{align}
F = (\mathbf{c}\hat{\beta})^{T}[K\mathbf{c}((X^{T}X)^{-1}\hat{\sigma}^{2})\mathbf{c}^{T}]^{-1}(\mathbf{c}\hat{\beta})
\end{align}

With a little imagination, you can see how the *F*-test is an extension of the *t*-test of a single contrast to accomodate testing a set of contrasts together. Don't worry, you don't have to understand how the formula for the *F*-statistic works mathematically and you don't have to implement this in Python. But you *do* need to understand what type of hypothesis an *F*-test tests!

Let's practice this in a ToDo!

ToDo (1 point)

Remember the temporal basis sets from before? Suppose we have an experiment with two conditions ("A" and "B") and suppose we've created a design matrix based on convolution with a single-gamma basis set (with a canonical HRF, its temporal derivative, and its dispersion derivative). Together with the intercept, the design matrix thus has 7 columns (2 conditions * 3 HRF + intercept).

The order of the columns is as follows:
* column 1: intercept
* column 2: canonical HRF "A"
* column 3: temporal deriv "A"
* column 4: dispersion deriv "A"
* column 5: canonical HRF "B"
* column 6: temporal deriv "B"
* column 7: dispersion deriv "B"

Suppose I want to test whether there is *any* difference in response to condition "A" ($A > 0$) compared to baseline, and *I don't care what element of the HRF caused it*. I can use an F-test for this. What would the corresponding contrast-*matrix* (in which each row represents a different contrast) look like?

We've created an 'empty' (all-zeros) 2D matrix below with three rows. It's up to you to fill in the matrix such that it can be used to test the above question/hypothesis.

```python
# Fill in the correct values!
contrast_matrix = np.array([

    [0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0]

])

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_3 import test_definition_ftest

test_definition_ftest(contrast_matrix)
print("Well done!")
```

### Summary
Alright, now you know basically everything about how to perform a univariate fMRI analysis!

"Wait, that's it?", you might ask (or not). Well, yeah, regular univariate analyses as you might read about in scientific journals do basically what you've just learned, but then not on a single voxel, but on each voxel in the brain separately. Basically just a gigantic for-loop across voxels in which everytime the same design ($\mathbf{X}$) is used to predict a new voxel-signal ($\mathbf{y}$). Afterwards, the *t*-values of the contrast (hypothesis) you're interested in are plotted back onto a brain, color-code it (high t-values yellow, low t-values red), and voilà, you have your pretty brain plot.

    ToThink (1 point): More explained variance (i.e., a smaller "sums of squared error" term) does not always mean that your t-value is higher. Explain how this might happen.

YOUR ANSWER HERE

    ToDo (2 points): Suppose that, within the hypothesized face-experiment explained earlier, you want to know which parts of the brain show (significantly) more activity during periods without stimuli (i.e., no faces were shown, i.e., "rest") than during periods with stimuli. Define a contrast vector which would test this hypothesis and store it in a variable <tt>cvec_rest</tt>. Remember: the original face experiment had 7 predictors (the first one being the intercept, followed by 6 face predictors).

```python
# Implement the assignment here

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_3 import test_rest_vs_stim_contrast
test_rest_vs_stim_contrast(cvec_rest)
```

    Tip!
    Before handing in your notebooks, we recommend restarting your kernel (Kernel → Restart & Clear Ouput) and running all your cells again (manually, or by Cell → Run all). By running all your cells one by one (from "top" to "bottom" of the notebook), you may spot potential errors that are caused by accidentally overwriting your variables or running your cells out of order (e.g., defining the variable 'x' in cell 28 which you then use in cell 15).


# Unrestricted Open-Shell Hartree-Fock

In the first two tutorials in this module, we wrote programs which implement a closed-shell formulation of Hartree-Fock theory using restricted orbitals, aptly named Restricted Hartree-Fock (RHF).  In this tutorial, we will abandon strictly closed-shell systems and the notion of restricted orbitals, in favor of a more general theory known as Unrestricted Hartree-Fock (UHF) which can accommodate more diverse molecules.  In UHF, the orbitals occupied by spin up ($\alpha$) electrons and those occupied by spin down ($\beta$) electrons no longer have the same spatial component, e.g.,

$$\chi_i({\bf x}) = \begin{cases}\psi^{\alpha}_j({\bf r})\alpha(\omega) \\ \psi^{\beta}_j({\bf r})\beta(\omega)\end{cases},$$

meaning that they will not have the same orbital energy.  This relaxation of orbital constraints allows for more variational flexibility, which leads to UHF always being able to find a lower total energy solution than RHF.

## I. Theoretical Overview
In UHF, we seek to solve the coupled equations

\begin{align}
{\bf F}^{\alpha}{\bf C}^{\alpha} &= {\bf SC}^{\alpha}{\bf\epsilon}^{\alpha} \\
{\bf F}^{\beta}{\bf C}^{\beta} &= {\bf SC}^{\beta}{\bf\epsilon}^{\beta},
\end{align}

which are the unrestricted generalizations of the restricted Roothan equations, called the Pople-Nesbitt equations.  Here, the one-electron Fock matrices are given by

\begin{align}
F_{\mu\nu}^{\alpha} &= H_{\mu\nu} + (\mu\,\nu\mid\lambda\,\sigma)[D_{\lambda\sigma}^{\alpha} + D_{\lambda\sigma}^{\beta}] - (\mu\,\lambda\,\mid\nu\,\sigma)D_{\lambda\sigma}^{\beta}\\
F_{\mu\nu}^{\beta} &= H_{\mu\nu} + (\mu\,\nu\mid\,\lambda\,\sigma)[D_{\lambda\sigma}^{\alpha} + D_{\lambda\sigma}^{\beta}] - (\mu\,\lambda\,\mid\nu\,\sigma)D_{\lambda\sigma}^{\alpha},
\end{align}

where the density matrices $D_{\lambda\sigma}^{\alpha}$ and $D_{\lambda\sigma}^{\beta}$ are given by

\begin{align}
D_{\lambda\sigma}^{\alpha} &= C_{\sigma i}^{\alpha}C_{\lambda i}^{\alpha}\\
D_{\lambda\sigma}^{\beta} &= C_{\sigma i}^{\beta}C_{\lambda i}^{\beta}.
\end{align}

Unlike for RHF, the orbital coefficient matrices ${\bf C}^{\alpha}$ and ${\bf C}^{\beta}$ are of dimension $M\times N^{\alpha}$ and $M\times N^{\beta}$, where $M$ is the number of AO basis functions and $N^{\alpha}$ ($N^{\beta}$) is the number of $\alpha$ ($\beta$) electrons.  The total UHF energy is given by

\begin{align}
E^{\rm UHF}_{\rm total} &= E^{\rm UHF}_{\rm elec} + E^{\rm BO}_{\rm nuc},\;\;{\rm with}\\
E^{\rm UHF}_{\rm elec} &= \frac{1}{2}[({\bf D}^{\alpha} + {\bf D}^{\beta}){\bf H} +
{\bf D}^{\alpha}{\bf F}^{\alpha} + {\bf D}^{\beta}{\bf F}^{\beta}].
\end{align}

## II. Implementation

In any SCF program, there will be several common elements which can be abstracted from the program itself into separate modules, classes, or functions to 'clean up' the code that will need to be written explicitly; examples of this concept can be seen throughout the Psi4NumPy reference implementations.  For the purposes of this tutorial, we can achieve some degree of code cleanup without sacrificing readabilitiy and clarity by focusing on abstracting only the parts of the code which are both
- Lengthy subroutines, and
- Used repeatedly.

In our UHF program, let's use what we've learned in the last tutorial by also implementing DIIS convergence accelleration for our SCF iterations.  With this in mind, two subroutines in particular would benefit from abstraction are

1. Orthogonalize & diagonalize Fock matrix
2. Extrapolate previous trial vectors for new DIIS solution vector

Before we start writing our UHF program, let's try to write functions which can perform the above tasks so that we can use them in our implementation of UHF.  Recall that defining functions in Python has the following syntax:
~~~python
def function_name(*args **kwargs):
    # function block
    return return_values
~~~
A thorough discussion of defining functions in Python can be found [here](https://docs.python.org/2/tutorial/controlflow.html#defining-functions "Go to Python docs").  First, let's write a function which can diagonalize the Fock matrix and return the orbital coefficient matrix **C** and the density matrix **D**.  From our RHF tutorial, this subroutine is executed with:
~~~python
F_p =  A.dot(F).dot(A)
e, C_p = np.linalg.eigh(F_p)
C = A.dot(C_p)
C_occ = C[:, :ndocc]
D = np.einsum('pi,qi->pq', C_occ, C_occ)
~~~
Examining this code block, there are three quantities which must be specified beforehand:
- Fock matrix, **F**
- Orthogonalization matrix, ${\bf A} = {\bf S}^{-1/2}$
- Number of doubly occupied orbitals, `ndocc`

However, since the orthogonalization matrix **A** is a static quantity (only built once, then left alone) we may choose to leave **A** as a *global* quantity, instead of an argument to our function.  In the cell below, using the code snippet given above, write a function `diag_F()` which takes **F** and the number of orbitals `norb` as arguments, and returns **C** and **D**:

```python
# ==> Define function to diagonalize F <==
def diag_F(F, norb):
    F_p = A.dot(F).dot(A)
    e, C_p = np.linalg.eigh(F_p)
    C = A.dot(C_p)
    C_occ = C[:, :norb]
    D = np.einsum('pi,qi->pq', C_occ, C_occ)
    return (C, D)
```

Next, let's write a function to perform DIIS extrapolation and generate a new solution vector.  Recall that the DIIS-accellerated SCF algorithm is:
#### Algorithm 1: DIIS within a generic SCF Iteration
1. Compute **F**, append to list of previous trial vectors
2. Compute AO orbital gradient **r**, append to list of previous residual vectors
3. Compute RHF energy
3. Check convergence criteria
    - If RMSD of **r** sufficiently small, and
    - If change in SCF energy sufficiently small, break
4. Build **B** matrix from previous AO gradient vectors
5. Solve Pulay equation for coefficients $\{c_i\}$
6. Compute DIIS solution vector **F_DIIS** from $\{c_i\}$ and previous trial vectors
7. Compute new orbital guess with **F_DIIS**

In our function, we will perform steps 4-6 of the above algorithm.  What information will we need to provide our function in order to do so?  To build **B** (step 4 above) in the previous tutorial, we used:
~~~python
# Build B matrix
B_dim = len(F_list) + 1
B = np.empty((B_dim, B_dim))
B[-1, :] = -1
B[:, -1] = -1
B[-1, -1] = 0
for i in xrange(len(F_list)):
    for j in xrange(len(F_list)):
        B[i, j] = np.einsum('ij,ij->', DIIS_RESID[i], DIIS_RESID[j])
~~~
Here, we see that we must have all previous DIIS residual vectors (`DIIS_RESID`), as well as knowledge about how many previous trial vectors there are (for the dimension of **B**).  To solve the Pulay equation (step 5 above):
~~~python
# Build RHS of Pulay equation
rhs = np.zeros((B_dim))
rhs[-1] = -1

# Solve Pulay equation for c_i's with NumPy
coeff = np.linalg.solve(B, rhs)
~~~
For this step, we only need the dimension of **B** (which we computed in step 4 above) and a NumPy routine, so this step doesn't require any additional arguments.  Finally, to build the DIIS Fock matrix (step 6):
~~~python
# Build DIIS Fock matrix
F = np.zeros_like(F_list[0])
for x in xrange(coeff.shape[0] - 1):
    F += coeff[x] * F_list[x]
~~~
Clearly, for this step, we need to know all the previous trial vectors (`F_list`) and the coefficients we generated in the previous step.  In the cell below, write a funciton `diis_xtrap()` according to Algorithm 1 steps 4-6, using the above code snippets, which takes a list of previous trial vectors `F_list` and residual vectors `DIIS_RESID` as arguments and returns the new DIIS solution vector `F_DIIS`:

```python
# ==> Build DIIS Extrapolation Function <==
def diis_xtrap(F_list, DIIS_RESID):
    # Build B matrix
    B_dim = len(F_list) + 1
    B = np.empty((B_dim, B_dim))
    B[-1, :] = -1
    B[:, -1] = -1
    B[-1, -1] = 0
    for i in range(len(F_list)):
        for j in range(len(F_list)):
            B[i, j] = np.einsum('ij,ij->', DIIS_RESID[i], DIIS_RESID[j])

    # Build RHS of Pulay equation
    rhs = np.zeros((B_dim))
    rhs[-1] = -1

    # Solve Pulay equation for c_i's with NumPy
    coeff = np.linalg.solve(B, rhs)

    # Build DIIS Fock matrix
    F_DIIS = np.zeros_like(F_list[0])
    for x in range(coeff.shape[0] - 1):
        F_DIIS += coeff[x] * F_list[x]

    return F_DIIS
```

We are now ready to begin writing our UHF program!  Let's begin by importing  Psi4  and NumPy, and defining our molecule & basic options:

```python
# ==> Import Psi4 & NumPy <==
import psi4
import numpy as np
```

```python
# ==> Set Basic Psi4 Options <==
# Memory specification
psi4.set_memory(int(5e8))
numpy_memory = 2

# Set output file
psi4.core.set_output_file('output.dat', False)

# Define Physicist's water -- don't forget C1 symmetry!
mol = psi4.geometry("""
O
H 1 1.1
H 1 1.1 2 104
symmetry c1
""")

# Set computation options
psi4.set_options({'guess': 'core',
                  'basis': 'cc-pvdz',
                  'scf_type': 'pk',
                  'e_convergence': 1e-8,
                  'reference': 'uhf'})
```

You may notice that in the above `psi4.set_options()` block, there are two additional options -- namely, `'guess': 'core'` and `'reference': 'uhf'`.  These options make sure that when we ultimately check our program against  Psi4, the options  Psi4  uses are identical to our implementation.  Next, let's define the options for our UHF program; we can borrow these options from our RHF implementation with DIIS accelleration that we completed in our last tutorial.

```python
# ==> Set default program options <==
# Maximum SCF iterations
MAXITER = 40
# Energy convergence criterion
E_conv = 1.0e-6
D_conv = 1.0e-3
```

Static quantities like the ERI tensor, core Hamiltonian, and orthogonalization matrix have exactly the same form in UHF as in RHF.  Unlike in RHF, however, we will need the number of $\alpha$ and $\beta$ electrons.  Fortunately, both these values are available through querying the Wavefunction object.  In the cell below, generate these static objects and compute each of the following:
- Number of basis functions, `nbf`
- Number of alpha electrons, `nalpha`
- Number of beta electrons, `nbeta`
- Number of doubly occupied orbitals, `ndocc` (Hint: In UHF, there can be unpaired electrons!)

```python
# ==> Compute static 1e- and 2e- quantities with Psi4 <==
# Class instantiation
wfn = psi4.core.Wavefunction.build(mol, psi4.core.get_global_option('basis'))
mints = psi4.core.MintsHelper(wfn.basisset())

# Overlap matrix
S = np.asarray(mints.ao_overlap())

# Number of basis Functions, alpha & beta orbitals, and # doubly occupied orbitals
nbf = wfn.nso()
nalpha = wfn.nalpha()
nbeta = wfn.nbeta()
ndocc = min(nalpha, nbeta)

print('Number of basis functions: %d' % (nbf))
print('Number of singly occupied orbitals: %d' % (abs(nalpha - nbeta)))
print('Number of doubly occupied orbitals: %d' % (ndocc))

# Memory check for ERI tensor
I_size = (nbf**4) * 8.e-9
print('\nSize of the ERI tensor will be {:4.2f} GB.'.format(I_size))
memory_footprint = I_size * 1.5
if I_size > numpy_memory:
    psi4.core.clean()
    raise Exception("Estimated memory utilization (%4.2f GB) exceeds allotted memory \
                     limit of %4.2f GB." % (memory_footprint, numpy_memory))

# Build ERI Tensor
I = np.asarray(mints.ao_eri())

# Build core Hamiltonian
T = np.asarray(mints.ao_kinetic())
V = np.asarray(mints.ao_potential())
H = T + V

# Construct AO orthogonalization matrix A
A = mints.ao_overlap()
A.power(-0.5, 1.e-16)
A = np.asarray(A)
```

    Number of basis functions: 24
    Number of singly occupied orbitals: 0
    Number of doubly occupied orbitals: 5

    Size of the ERI tensor will be 0.00 GB.

Unlike the static quantities above, the CORE guess in UHF is slightly different than in RHF.  Since the $\alpha$ and $\beta$ electrons do not share spatial orbitals, we must construct a guess for *each* of the $\alpha$ and $\beta$ orbitals and densities.  In the cell below, using the function `diag_F()`, construct the CORE guesses and compute the nuclear repulsion energy:

(Hint: The number of $\alpha$ orbitals is the same as the number of $\alpha$ electrons!)

```python
# ==> Build alpha & beta CORE guess <==
Ca, Da = diag_F(H, nalpha)
Cb, Db = diag_F(H, nbeta)

# Get nuclear repulsion energy
E_nuc = mol.nuclear_repulsion_energy()
```

We are almost ready to perform our SCF iterations; beforehand, however, we must initiate variables for the current & previous SCF energies, and the lists to hold previous residual vectors and trial vectors for the DIIS procedure.  Since, in UHF, there are Fock matrices ${\bf F}^{\alpha}$ and ${\bf F}^{\beta}$ for both $\alpha$ and $\beta$ orbitals, we must apply DIIS to each of these matrices separately.  In the cell below, define empty lists to hold previous Fock matrices and residual vectors for both $\alpha$ and $\beta$ orbitals:

```python
# ==> Pre-Iteration Setup <==
# SCF & Previous Energy
SCF_E = 0.0
E_old = 0.0
```

We are now ready to write the SCF iterations.  The algorithm for UHF-SCF iteration, with DIIS convergence accelleration, is:
#### Algorithm 2: DIIS within UHF-SCF Iteration
1. Build ${\bf F}^{\alpha}$ and ${\bf F}^{\beta}$, append to trial vector lists
2. Compute the DIIS residual for $\alpha$ and $\beta$, append to residual vector lists
3. Compute UHF energy
4. Convergence check
    - If average of RMSD of $\alpha$ and $\beta$ residual sufficiently small, and
    - If change in UHF energy sufficiently small, break
5. DIIS extrapolation of ${\bf F}^{\alpha}$ and ${\bf F}^{\beta}$ to form new solution vector
6. Compute new ${\alpha}$ and ${\beta}$ orbital & density guesses

In the cell below, write the UHF-SCF iteration according to Algorithm 2:

(Hint: Use your functions `diis_xtrap()` and `diag_F` for Algorithm 2 steps 5 & 6, respectively)

```python
# Trial & Residual Vector Lists -- one each for alpha & beta
F_list_a = []
F_list_b = []
R_list_a = []
R_list_b = []

# ==> UHF-SCF Iterations <==
print('==> Starting SCF Iterations <==\n')

# Begin Iterations
for scf_iter in range(MAXITER):
    # Build Fa & Fb matrices
    Ja = np.einsum('pqrs,rs->pq', I, Da)
    Jb = np.einsum('pqrs,rs->pq', I, Db)
    Ka = np.einsum('prqs,rs->pq', I, Da)
    Kb = np.einsum('prqs,rs->pq', I, Db)
    Fa = H + (Ja + Jb) - Ka
    Fb = H + (Ja + Jb) - Kb

    # Compute DIIS residual for Fa & Fb
    diis_r_a = A.dot(Fa.dot(Da).dot(S) - S.dot(Da).dot(Fa)).dot(A)
    diis_r_b = A.dot(Fb.dot(Db).dot(S) - S.dot(Db).dot(Fb)).dot(A)

    # Append trial & residual vectors to lists
    F_list_a.append(Fa)
    F_list_b.append(Fb)
    R_list_a.append(diis_r_a)
    R_list_b.append(diis_r_b)

    # Compute UHF Energy
    SCF_E = np.einsum('pq,pq->', (Da + Db), H)
    SCF_E += np.einsum('pq,pq->', Da, Fa)
    SCF_E += np.einsum('pq,pq->', Db, Fb)
    SCF_E *= 0.5
    SCF_E += E_nuc

    dE = SCF_E - E_old
    dRMS = 0.5 * (np.mean(diis_r_a**2)**0.5 + np.mean(diis_r_b**2)**0.5)
    print('SCF Iteration %3d: Energy = %4.16f dE = % 1.5E dRMS = %1.5E' % (scf_iter, SCF_E, dE, dRMS))

    # Convergence Check
    if (abs(dE) < E_conv) and (dRMS < D_conv):
        break
    E_old = SCF_E

    # DIIS Extrapolation
    if scf_iter >= 2:
        Fa = diis_xtrap(F_list_a, R_list_a)
        Fb = diis_xtrap(F_list_b, R_list_b)

    # Compute new orbital guess
    Ca, Da = diag_F(Fa, nalpha)
    Cb, Db = diag_F(Fb, nbeta)

    # MAXITER exceeded?
    if (scf_iter == MAXITER):
        psi4.core.clean()
        raise Exception("Maximum number of SCF iterations exceeded.")

# Post iterations
print('\nSCF converged.')
print('Final UHF Energy: %.8f [Eh]' % SCF_E)
```

    ==> Starting SCF Iterations <==

    SCF Iteration   0: Energy = -74.1207806468836452 dE =  0.00000E+00 dRMS = 8.64677E-02
    SCF Iteration   1: Energy = -74.8671819457688485 dE = -7.46401E-01 dRMS = 6.52840E-02
    SCF Iteration   2: Energy = -75.4149087803903342 dE = -5.47727E-01 dRMS = 5.21690E-02
    SCF Iteration   3: Energy = -75.9800488561561309 dE = -5.65140E-01 dRMS = 6.34267E-03
    SCF Iteration   4: Energy = -75.9894383301614340 dE = -9.38947E-03 dRMS = 5.45826E-04
    SCF Iteration   5: Energy = -75.9897683674259383 dE = -3.30037E-04 dRMS = 1.70671E-04
    SCF Iteration   6: Energy = -75.9897948623176376 dE = -2.64949E-05 dRMS = 4.28126E-05
    SCF Iteration   7: Energy = -75.9897957712875609 dE = -9.08970E-07 dRMS = 5.40285E-06

    SCF converged.
    Final UHF Energy: -75.98979577 [Eh]

Congratulations! You've written your very own Unrestricted Hartree-Fock program with DIIS convergence accelleration!  Finally, let's check your final UHF energy against  Psi4:

```python
# Compare to Psi4
SCF_E_psi = psi4.energy('SCF')
psi4.compare_values(SCF_E_psi, SCF_E, 6, 'SCF Energy')
```

    	SCF Energy........................................................PASSED

## References
1. A. Szabo and N. S. Ostlund, *Modern Quantum Chemistry*, Introduction to Advanced Electronic Structure Theory. Courier Corporation, 1996.
2. I. N. Levine, *Quantum Chemistry*. Prentice-Hall, New Jersey, 5th edition, 2000.
3. T. Helgaker, P. Jorgensen, and J. Olsen, *Molecular Electronic Structure Theory*, John Wiley & Sons Inc, 2000.

```python

```


This is the second in a series of posts on testing scientific software. For this to make sense, you'll need to have skimmed [the motivation and background](http://ianhawke.github.io/blog/close-enough.html). The [first in the series](http://ianhawke.github.io/blog/close-enough-part-1.html) assumed we only cared about the answer, and whether we'd implemented the expected algorithm wasn't important.

$$
\newcommand{\dt}{\Delta t}
\newcommand{\udt}[1]{u^{({#1})}(T)}
\newcommand{\sm}[1]{s_m^{({#1})}}
$$
Let's suppose that instead of caring about the answer $u$ we care about the algorithm behaving in the right fashion, $\udt{\dt} = u(T) + c_1 \dt^{s_e}$.

Given three pieces of data we can measure the slope:

$$
\begin{align}
  s_m & = \log_2 \left| \frac{\udt{4\dt}-\udt{2\dt}}{\udt{2\dt}-\udt{\dt}} \right| \\
  & = \log_2 \left| \frac{u(T) + c_1 4^{s_e} \dt^{s_e} - u(T) + c_1 2^{s_e} \dt^{s_e} + {\cal O}(\dt^{s_e+1})}{u(T) + c_1 2^{s_e} \dt^{s_e} - u(T) + c_1 \dt^{s_e} + {\cal O}(\dt^{s_e+1)}} \right| \\
  & = \log_2 \left| 2^{s_e} + {\cal O}(\dt) \right| \\
  & = s_e + {\cal O}(\dt).
\end{align}
$$

In the [first post in the series](http://ianhawke.github.io/blog/close-enough-part-1.html) we applied Richardson extrapolation to the solution, getting an extrapolated answer and an interval within which we expected the true solution to lie. Now we can apply the same Richardson extrapolation tricks to our measured slope. If the measured slope is denoted $\sm{\dt}$ then we can use Richardson extrapolation as before to compute

$$
\begin{align}
  s_R &= 2\sm{\dt} - \sm{2\dt}, \\ E_R & = s_R - \sm{\dt}.
\end{align}
$$

We then state that our algorithm is *close enough* to correct if $s_e$ lies in the interval $s_R \pm E_R$.

```python
from math import sin, cos, log, ceil
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

We first need the code implementing Euler's method [the full phugoid model notebook](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb).

```python
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 30.0   # trim velocity in m s^{-1}
C_D = 1/40.  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = v_t     # start at the trim velocity (or add a delta)
theta0 = 0.0 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 1000.0  # initial altitude
```

```python
def f(u):
    """Returns the right-hand side of the phugoid system of equations.

    Parameters
    ----------
    u : array of float
        array containing the solution at time n.

    Returns
    -------
    dudt : array of float
        array containing the RHS given u.
    """

    v = u[0]
    theta = u[1]
    x = u[2]
    y = u[3]
    return numpy.array([-g*sin(theta) - C_D/C_L*g/v_t**2*v**2,
                      -g*cos(theta)/v + g/v_t**2*v,
                      v*cos(theta),
                      v*sin(theta)])
```

```python
def euler_step(u, f, dt):
    """Returns the solution at the next time-step using Euler's method.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        approximate solution at the next time step.
    """

    return u + dt * f(u)
```

We then construct a sequence of solutions in order to perform Richardson extrapolation, exactly as in the [first post](http://ianhawke.github.io/blog/close-enough-part-1.html). Here we need four, rather than three, solutions, so that we can compute the slope at two different resolutions.

```python
T = 100.0
dt_values = numpy.array([0.001*2**i for i in range(-1,3)])
v_values = numpy.zeros_like(dt_values)
for i, dt in enumerate(dt_values):
    N = int(T/dt)+1
    t = numpy.linspace(0.0, T, N)
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])
    for n in range(N-1):
        u[n+1] = euler_step(u[n], f, dt)
    v_values[i] = u[-1,0]
```

We can now measure the convergence rate slope twice:

```python
s_m = numpy.zeros(2)
for i in range(2):
    s_m[i] = log(abs((v_values[2+i]-v_values[1+i])/(v_values[1+i]-v_values[0+i]))) / log(2.0)
    print("Measured convergence rate (base dt {}) is {:.7g}.".format(dt_values[i], s_m[i]))
```

    Measured convergence rate (base dt 0.0005) is 1.011621.
    Measured convergence rate (base dt 0.001) is 1.023266.

We then use Richardson extrapolation applied to the measured convergence rates (at finite resolution) to get the "*actual*" convergence rate and its accuracy:

```python
s_exact = 2*s_m[0]-s_m[1]
error_e = abs(s_exact - s_m[0])
print("The convergence rate lies in the interval [{:.7g},{:.7g}]".format(
        s_exact-error_e, s_exact+error_e))
```

    The convergence rate lies in the interval [0.9883297,1.011621]

We see that $\sm{\dt} \simeq 1.023$ is *close enough* when $\sm{\dt/2} \simeq 1.012$, as the interval is consistent with $s_e=1$.

## Finding the limits

Once again, we want to know what the limiting values are. In this case we're interested in fixing $s_e$ and $\sm{\dt}$, and finding the value of $\sm{\dt/2}$ that is close enough.

So the model we'll use is that $s_m = s_e + c_1 \dt$, as above, so we need

$$
\begin{equation}
  2 \sm{\dt/2} - \sm{\dt} - | \sm{\dt/2} - \sm{\dt} | \le s_e \le 2 \sm{\dt/2} - \sm{\dt} + | \sm{\dt/2} - \sm{\dt} |.
\end{equation}
$$

In the case where $\sm{\dt/2} < \sm{\dt}$  this gives

$$
\begin{equation}
  3 \sm{\dt/2} - 2\sm{\dt} \le s_e \le \sm{\dt/2} \quad \implies \quad \sm{\dt/2} \le \tfrac{1}{3} \left( 2 \sm{\dt} + s_e \right).
\end{equation}
$$

In the case where $\sm{\dt/2} > \sm{\dt}$  this gives

$$
\begin{equation}
  \sm{\dt/2} \le s_e \le 3 \sm{\dt/2} - 2 \sm{\dt} \quad \implies \quad \sm{\dt/2} \ge \tfrac{1}{3} \left( 2 \sm{\dt} + s_e \right).
\end{equation}
$$

We can write this in terms of the *error* in the slope as

$$
\begin{equation}
  | \sm{\dt/2} - s_e | \le \frac{2}{3} | \sm{\dt} - s_e |;
\end{equation}
$$

that is, the error must go down by at least a factor of $1/3$ when the step size is cut in half.

So, if we fix $s_e=1$ (as for Euler's method), we can see the *upper* bound on $\sm{\dt/2}$ given $s_m = 1.023$ is:

```python
s_e = 1.0
s_mdthalf_max = (2.0*s_m[1] + s_e)/3.0
print("Upper bound when s_e=1 and s_m (coarse resolution)=1.023: "
      "{:.7f}.".format(s_mdthalf_max))
```

    Upper bound when s_e=1 and s_m (coarse resolution)=1.023: 1.0155107.

In general, for Euler's method, the bound varies linearly:

```python
s_m_values = numpy.linspace(0.5, 1.5)
bound = (2.0*s_m_values + 1.0)/3.0
pyplot.figure(figsize=(8,6))
pyplot.fill_between(s_m_values, numpy.ones_like(s_m_values), bound,
                    facecolor='green', alpha=0.5)
#pyplot.ylim(1.0, 2.5)
pyplot.xlim(0.5, 1.5)
pyplot.xlabel(r"$s_m^{(\Delta t)}$")
pyplot.ylabel(r"$s_m^{(\Delta t / 2)}$");
```

## Summary

If we want the algorithm to behave in the way expected of Euler's method, then measuring a single convergence rate is useful but not sufficient. Instead we must measure the convergence rate twice, once with base timestep $\dt$ and once with base timestep $\dt/2$. The difference between the measured slope $\sm{\dt}$ and the expected slope $s_e=1$ must go down by at least $1/3$ when the timestep is reduced to $\dt/2$.


Text provided under a Creative Commons Attribution license, CC-BY.  All code is made available under the FSF-approved BSD-3 license.  (c) Lorena A. Barba, Gilbert F. Forsyth 2017. Thanks to NSF for support via CAREER award #1149784.
[@LorenaABarba](https://twitter.com/LorenaABarba)

[@ggruszczynski](https://github.com/ggruszczynski)

```python
import timeit
import numpy as np
import matplotlib.pyplot as plt   #and the useful plotting library
```

Diffusion part 1: the fundamental solution
-----
***

The one-dimensional diffusion equation is:

$$\frac{\partial u}{\partial t}= \nu \frac{\partial^2 u}{\partial x^2}$$

The first thing you should notice is that —unlike the previous two simple equations we have studied— this equation has a second-order derivative. We first need to learn what to do with it!

### Discretizing $\frac{\partial ^2 u}{\partial x^2}$

The second-order derivative can be represented geometrically as the line tangent to the curve given by the first derivative.  We will discretize the second-order derivative with a Central Difference scheme: a combination of Forward Difference and Backward Difference of the first derivative.  Consider the Taylor expansion of $u_{i+1}$ and $u_{i-1}$ around $u_i$:

$u_{i+1} = u_i + \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i + \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

$u_{i-1} = u_i - \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i - \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

If we add these two expansions, you can see that the odd-numbered derivative terms will cancel each other out.  If we neglect any terms of $O(\Delta x^4)$ or higher (and really, those are very small), then we can rearrange the sum of these two expansions to solve for our second-derivative.

$u_{i+1} + u_{i-1} = 2u_i+\Delta x^2 \frac{\partial ^2 u}{\partial x^2}\bigg|_i + O(\Delta x^4)$

Then rearrange to solve for $\frac{\partial ^2 u}{\partial x^2}\bigg|_i$ and the result is:

$$\frac{\partial ^2 u}{\partial x^2}=\frac{u_{i+1}-2u_{i}+u_{i-1}}{\Delta x^2} + O(\Delta x^4)$$

### Discretizing both $\frac{\partial u}{\partial t}$ and $\frac{\partial ^2 u}{\partial x^2}$

We can now write the discretized version of the diffusion equation in 1D:

$$\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\nu\frac{u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n}}{\Delta x^2}$$

As before, we notice that once we have an initial condition, the only unknown is $u_{i}^{n+1}$, so we re-arrange the equation solving for our unknown:

$$u_{i}^{n+1}=u_{i}^{n}+\underbrace{\frac{\nu\Delta t}{\Delta x^2}}_{\beta}(u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n})$$

$$u_{i}^{n+1}=\beta u_{i-1}^{n} + u_{i}^{n}(1- 2 \beta) +\beta u_{i+1}^{n}$$

The above discrete equation allows us to write a program to advance a solution in time. But we need an initial condition. Let's continue using our favorite: the hat function. So, at $t=0$, $u=2$ in the interval $0.5\le x\le 1$ and $u=1$ everywhere else. We are ready to number-crunch!

```python
nx = 128
domain_length = 64
dx = domain_length / (nx-1)
xspace = np.linspace(0, domain_length, nx)

nt = 200                # the number of timesteps we want to calculate
nu = 5                  # the value of viscosity
sigma = .2              # sigma is a parameter, we'll learn more about it later
dt = sigma * dx**2 / nu # dt is defined using sigma ... more later!

u_IC = 0*np.ones(nx)                 # numpy function ones()
u_IC[int((nx-1)/4):int(nx/2 + 1)] = 1 # setting u = 2 between 0.5 and 1 as per our I.C.s

plt.plot(xspace, u_IC)
```

```python
def calc_diffusion_FD_naive(IC,nx,nt,nu,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time
    beta = nu * dt / dx**2

    for n in range(nt):  #iterate through time
        un = u.copy() #copy the existing values of u into un
        for i in range(0, nx):
            # this is slow (index operations & branching)
            if i == nx-1:
                u[i] = beta*un[i-1]+ (1-2*beta)*un[i] + beta*un[0] # periodic BC
            else:
                u[i] = beta*un[i-1]+ (1-2*beta)*un[i] + beta*un[i+1]
    return u

def calc_diffusion_FD(IC,nx,nt,nu,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time
    beta = nu * dt / dx**2

    c_ind = np.arange(0, nx)
    l_ind = np.roll(c_ind, -1)
    r_ind = np.roll(c_ind, 1)

    for n in range(nt):  #iterate through time
        un = u.copy() # copy the existing values of u into un

        lap_u = un[l_ind] - 2 * un[c_ind] + un[r_ind]  # periodic BC
        u = un + beta* lap_u
    return u

starttime = timeit.default_timer()
u_FD = calc_diffusion_FD(u_IC,nx,nt,nu,dt)
print("The time difference is :", timeit.default_timer() - starttime)
plt.plot(xspace, u_FD)
```

# The convolution - part I

<https://numpy.org/doc/stable/reference/generated/numpy.convolve.html>

<https://en.wikipedia.org/wiki/Convolution>

The discrete convolution operation is known as:

$$
(a * v)[n]= \sum^{\infty}_{m=-\infty} a[m]v[n-m]
$$

Notice, that single step of the explicit algorithm implemented before can be expressed as convolution with a $[\beta,1-2 \beta,\beta]$ filter.
To compute more time steps, one have to convolve many times.

### Task

Solve the diffusion equation by convolving the initial condition with the filter in each iteration.

```python
def calc_diffusion_iterate_convolutions(IC,nx,nt,nu,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time
    beta = nu * dt / dx**2
    filtr = np.array([beta,1-2*beta,beta])
    for n in range(nt):  #iterate through time
        un = u.copy() ##copy the existing values of u into un
        u = np.convolve(filtr,un, 'same')
    return u

u_iter_conv = calc_diffusion_iterate_convolutions(u_IC,nx,nt,nu,dt)
plt.plot(xspace, u_iter_conv)
```

# The convolution - part II

The fundamental solution of the heat equation is the Gaussian function (impulse responce).

Consider "diffusion" of a single particle.
The probability of finding a particle after T time steps follows the normal (a.k.a Gaussian) distribution.
To compute it, one can convolve the initial position with a Gaussian.
The result will be equivalent with repeated convolutions with small filter.

This means that convolving with a Gaussian tells us the solution to the diffusion equation after
a fixed amount of time. This is the same as low pass filtering an image. So smoothing, low pass
filtering, diffusion, all mean the same thing.

### Task

Solve the diffusion equation by convolving the initial condition with the Gaussian.

```python
def calc_diffusion_single_convolution(IC,x,nt,nu,dt):
    u = IC.copy()
    def get_gaussian(x, alfa, t):
        g = -(x-domain_length/2.)**2
        g /=(4*alfa*t)
        g = np.exp(g)
        g /= np.sqrt(4*np.pi*alfa*t)
        g *= domain_length/(nx-1) # normalize --> sum(g)=1
        return g

    time_spot = dt*nt
    fundamental_solution = get_gaussian(x, nu, time_spot)

    u = np.convolve(fundamental_solution, u, 'same')
    # plt.plot(x, fundamental_solution, marker='v', linestyle="", markevery=5)
    return u, fundamental_solution

u_single_conv, fs = calc_diffusion_single_convolution(u_IC,xspace,nt,nu,dt)
plt.plot(xspace, u_single_conv)

```

```python
# Now plot the solutions obtained using 3 different approaches on the same plot

plt.rcParams.update({'font.size': 16})
figure, axis = plt.subplots(1, 1, figsize=(10, 8))
plt.subplots_adjust(hspace=1)
axis.set_title('Diffusion')
axis.plot(xspace, u_FD, label=r'$u_{FD}$', linewidth="3")
axis.plot(xspace, u_iter_conv, label=r'$u_{multiple \; convolutions}$',  marker='o', linestyle="", markevery=1)
axis.plot(xspace, u_single_conv, label=r'$u_{convolution}$', marker='x', linestyle="", markevery=1)
axis.set_xlabel('x')
axis.set_ylabel('Concentration')
axis.legend(loc="upper right")
```

# Analytical solution: Advection - Diffusion of a Gaussian Hill

In case of an isotropic diffusion, the analytical solution describing evolution of a Gaussian Hill can be expressed as

$$
C(\boldsymbol{x}, t)=\frac{\left(2\pi\sigma_{0}^{2}\right)^{D/2} }{\left(2\pi(\sigma_{0}^{2} + 2 k t)\right)^{D/2}}
C_0 \exp \left(-\frac{\left(\boldsymbol{x}-\boldsymbol{x}_{0}-\boldsymbol{u} t\right)^{2}}{2\left(\sigma_{0}^{2}+ 2 k t\right)}\right)
$$

where:

* $C_0$ - initial concentration,
* $D$ - number of dimensions,
* $t$ - time,
* $k$ - conductivity,
* $\boldsymbol{u}$ - velocity of advection
* $\sigma_{0}$ the initial variance of the distribution.

## Task

1) Implement the `GaussianHillAnal` class.
It shall have a method `get_concentration_ND(self, X, t)`, which will return the concentration at given time and space.

2) Benchmark the FD code against analytical solution.

```python
from sympy.matrices import Matrix
import sympy as sp

class GaussianHillAnal:
    def __init__(self, C0, X0, Sigma2_0, k, U, D):
        """
        :param C0: initial concentration
        :param X0: initial position of the hill's centre = Matrix([x0, y0])
        :param U:  velocity = Matrix([ux, uy])
        :param Sigma2_0: initial width of the Gaussian Hill
        :param k: conductivity
        :param dimenions: number of dimensions
        """
        self.C0 = C0
        self.X0 = X0
        self.U = U
        self.Sigma2_0 = Sigma2_0
        self.k = k
        self.dim = D

    def get_concentration_ND(self, X, t):
        decay = 2.*self.k*t
        L = X - self.X0 - self.U*t
        C = self.C0
        C *= pow(2. * np.pi * self.Sigma2_0, self.dim / 2.)
        C /= pow(2. * np.pi * (self.Sigma2_0 + decay), self.dim / 2.)
        C *= sp.exp(-(L.dot(L)) / (2.*(self.Sigma2_0 + decay)))
        return C

```

```python
time_0    = dt*nt/2     # initial contidion for FD
time_spot = dt*nt       # time to be simulated (by FD and analytically)

X0 = Matrix([domain_length/2.]) # center of the hill
C0 = 1.                 # concentration
variance = 30           # initial variance
reference_level = 0

T_0 = np.zeros(nx)
T_anal = np.zeros(nx)

gha = GaussianHillAnal(C0, X0, variance, nu, Matrix([0]), D=1)

for i in range(nx):
    T_0[i] = reference_level + gha.get_concentration_ND(Matrix([xspace[i]]), time_0)
    T_anal[i] = reference_level + gha.get_concentration_ND(Matrix([xspace[i]]), time_spot)

T_FD = calc_diffusion_FD(T_0,nx,nt,nu,dt)
T_single_conv, fs = calc_diffusion_single_convolution(T_0,xspace,nt,nu,dt)

plt.rcParams.update({'font.size': 16})
figure, axis = plt.subplots(1, 1, figsize=(8, 6))
plt.subplots_adjust(hspace=1)
axis.set_title('Diffusion of a Gaussian Hill')
axis.plot(xspace, T_0, label=r'$T_{0}$')
axis.plot(xspace, T_anal, label=r'$T_{anal}$')
axis.plot(xspace, T_FD, label=r'$T_{FD}$', marker='x', linestyle="", markevery=1)
axis.plot(xspace, T_single_conv, label=r'$T_{conv}$', marker='v', linestyle="", markevery=1)
axis.set_xlabel('x')
axis.set_ylabel('Concentration')
axis.legend(loc="upper right")

```

## Questions:

* How do you find the FD solution compared to analytical one? Experiment with different dx, dt.

* How would you asses that your mesh is fine enought in a real CFD simulation (without analytical solution)?

## Answers

* Mesh convergence study

# Learn More

Inspiration

<http://www.cs.umd.edu/~djacobs/CMSC828seg/Diffusion.pdf>

<https://web.math.ucsb.edu/~helena/teaching/math124b/heat.pdf>

You should have completed Steps [1](./01_Step_1.ipynb) and [2](./02_Step_2.ipynb) before continuing. This Jupyter notebook continues the presentation of the **12 steps to Navier–Stokes**, the practical module taught in the interactive CFD class of [Prof. Lorena Barba](http://lorenabarba.com).

For a careful walk-through of the discretization of the diffusion equation with finite differences (and all steps from 1 to 4), watch **Video Lesson 4** by Prof. Barba on YouTube.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('y2WaK7_iMRI')
```