```javascript
%%javascript
$('#appmode-leave').hide();
$('#copy-binder-link').hide();
$('#visit-repo-link').hide();
```

# Crystal Violet - numerical lab

### Video credit:  Creative Studios at The University of Texas at El Paso

```python
# Crystal Violet - numerical lab
from IPython.display import YouTubeVideo
# Video credit:  Creative Studios at The University of Texas at El Paso
YouTubeVideo('tcv5Xk9ZXpg')
```

## Introduction to the experiment
In this experiment a UV-Vis spectrophotometer will be used to measure the absorbance as the reaction
between crystal violet and hydroxide proceeds.
The absorbance versus time data will be used to determine the rate of the reaction with respect to both crystal violet and hydroxide ions.
Crystal violet has many uses, which include its use as a dye, as a constituent in inks for printing, as an antibacterial and antifungal, to treat mouth ulcers and for the development of fingerprints in forensics.
The dye has problems; the process requires the colour to be set in a highly basic washing soda and it has found that the dye looses it’s colour over time during this process.
The goal of this experiment is to investigate the decolourisation of the dye by studying the role of sodium hydroxide in the kinetics of crystal violet decolourisation.

## Theory
The rate (or velocity) $\nu\ [mol L^{-1} s{-1}]$ of a chemical reaction can be expressed in terms of the loss of reactants or the formation of products. For the reaction:

\begin{equation}
n_aA + n_bB + \dots \to n_xX + n_yY + \dots
\end{equation}

the rate can be written as

\begin{equation}
\nu = -\frac{1}{n_a}\frac{\mathrm{d[A]}}{\mathrm{d}t} = -\frac{1}{n_b}\frac{\mathrm{d[B]}}{\mathrm{d}t} = \frac{1}{n_x}\frac{\mathrm{d[X]}}{\mathrm{d}t} = \frac{1}{n_y}\frac{\mathrm{d[Y]}}{\mathrm{d}t} = \dots
\end{equation}

where $n_a$, $n_b$, $n_x$ and $n_y$, and [A], [B], [X] and [Y] are the stoichiometric coefficients and molar concentrations of the reactants and products.
Note that the rate is always a positive quantity, hence the equation have different sign to account for the \emph{loss} of reactants, or the \emph{formation} of products.

The rate, $\nu$, of a chemical reaction is frequently found to be proportional to the concentrations of the reacting species.

\begin{equation}
\nu = k_r \mathrm{[A]}^a \mathrm{[B]}^b \mathrm{[C]}^c
\end{equation}

Where $k_r$ is the rate constant and $a$, $b$, and $c$ denote the order of the reaction with
respect to reactants A, B and C.
The sum of the individual orders of the reactants is the overall order of the reaction.
Most reactions are a complicated series of elementary reactions, and the measured rate is generally the rate of the slowest step.
Therefore knowledge of the order of each reactant with respect to the overall reaction often allows the specific reaction mechanisms to be theorized.
The rate of any reaction can vary with time, and therefore it is impossible to define a general rate of a reaction.
Rather an instantaneous rate can be determined for any particular time.
If one can determine the concentation of a reactant (or product) as a function of time, this instantaneous rate will be given by the tangent of the time-concentration curve.

The time-concentration curve corresponds to the \emph{integrated} rate law and it is often more useful that the instantaneous rate.

| Reaction Order    | Differential form | Intergrated form  |
|:------------------|:-----------------:|------------------:|
| Zeroth order   | $\nu=k_r\mathrm{[A]}^0$ | $\mathrm{[A]} = \mathrm{[A]}_0 - k_r t$ |
| First order    | $\nu=k_r\mathrm{[A]}^1$ | $\ln\mathrm{[A]} = \ln\mathrm{[A]}_0 - k_r t$ |
| Second order   | $\nu=k_r\mathrm{[A]}^2$ | $\frac{1}{\mathrm{[A]}} = \frac{1}{\mathrm{[A]}_0} + k_r t$ |

where the subscript 0 denoted the initial concentration and $t$ is the time of when the concentration, [A], is measured.

For a **zeroth order** reaction the rate is constant and independent of the concentration of the reactant, [A]. Therefore a plot of the concentration of the reactant vs. time will be linear with a negative slope. The slope can be used to find the rate constant $k_r$, which will have units of (concentration/time).

For a **first order** reaction the rate is directly proportional to the concentration of the reactant, [A]. Therefore a plot of the natural log of the concentration of the reactant vs. time will be linear with a negative slope. The slope can be used to find the rate constant $k_r$, which will have units of (time)$^{-1}.

For a **second order** reaction the rate is directly proportional to the square of the concentration of the reactant, [A]. Therefore a plot of the reciprocal of the concentration of the reactant vs. time will be linear with a positive slope. The slope can be used to find the rate constant $k_r$, which will have units of (concentration*time)$^{-1}.
Second order reactions can arise when a reaction is first order in two separate reactants. The method of isolation described next, applies to this scenario.

### Isolation method
When an elementary reaction involves more than one reactant,

\begin{equation}
n_aA + n_bB + n_cC \to n_xX + n_yY + \dots
\end{equation}

the most general form of the instantaneous rate equation includes the concentration of all these species

\begin{equation}
\nu = k_r \mathrm{[A]}^a \mathrm{[B]}^b \mathrm{[C]}^c
\end{equation}

It is then much more difficult to analyse the rate equation in a way similar to that described above for simple zero, first and second order processes.
The analysis can be greatly simplified by **isolating** one reactant.
In the isolation method, all concentrations the rate equation are kept (as much as possible to) constant, except for one.
This is usually achieved by having all but one reactant present in large excess, so the concentration hardly changes during the reaction.
In this case, the limiting reagent  (for example, A) is said to be isolated, and its concentration is measured as a function of time to determine the kinetics of the reaction. Then we can rewrite the rate equation as

\begin{equation}
\nu = k^\dagger_r \mathrm{[A]}^a
\end{equation}

Where $k^\dagger_r$ is a new “pseudo rate constant” incorporating the values of [B]$^b$ and [C]$^c$.
Note that the value of $k^\dagger_r$ depends on the chosen “constant” values of [B] and [C]..

\begin{equation}
k^\dagger_r = k_r \mathrm{[B]}^b \mathrm{[C]}^c
\end{equation}

Now that A is **isolated**, we can apply the methods of analysis for a single reactant to
any complicated reaction and determine the order of that reaction with respect to A and $k^\dagger_r$ .
This is done by plotting the relationships for zero, first and second orders, and choosing which plot fits best (*i.e.* which is the most linear).

To find $b$, $c$ and $k_r$ we can look at a logarithmic form of the pseudo-rate constant

\begin{equation}
\log k^\dagger_r = b \log\mathrm{[B]} + c \log\mathrm{[C]} + \log k_r
\end{equation}

This means that if $k^\dagger_r$ is found for different values of [B] (while [C] is kept constant and the conditions are still met for isolation of A), then $b$ can be found from a plot
of $\log k^\dagger_r$ versus $\log\mathrm{[B]}$.
A near-integer value of slope can be rounded to the integer, as the order of reaction is expected to be an integer.
Similarly, if $k^\dagger_r$ is found for different values of [C] (while [B] is kept constant and the conditions are still met for isolation of A), then $c$ can be found from a plot of $\log k^\dagger_r$ versus $\log\mathrm{[C]}$. A near-integer value of slope can be rounded to the integer, as the order of reaction is expected to be an integer.
Finally, now that $b$, and $c$ have been found, $k_r$ can be found using the intercepts of the $\log–\log$ plots.

## The Reaction between Crystal violet and hydroxide
Crystal violet, CV, and hydroxide react according to the following reaction:

\begin{equation}
\mathrm{CV^+} + \mathrm{OH^-} \to \mathrm{CVOH}
\end{equation}

The reaction rate can therefore be expressed in the form:

\begin{equation}
\nu = k_r \mathrm{[CV^+]}^a \mathrm{[OH^-]}^b
\end{equation}

The method of isolation is used by isolating crystal violet and using it to evaluate the order of the reaction with respect to the reactants separately.
The rate of the reaction can therefore be written as:

\begin{equation}
\nu = k^\dagger_r \mathrm{[CV^+]}^a
\end{equation}

where

\begin{equation}
k^\dagger_r = k_r \mathrm{[OH^-]}^b
\end{equation}

## Relating absorbance and concentration
Since the absorbance, A, of the crystal violet is monitored as a function of time, it is not necessary to know the actual concentration of the crystal violet at any particular time. Beer’s law can be used to relate the absorbance, Ab and the crystal violet concentration, [CV+] by the equation:

\begin{equation}
Ab = \varepsilon\ l\ \mathrm{[CV^+]}
\end{equation}

Where $\varepsilon$ is the molar absorptivity of crystal violet at the wavelength of the maximum absorption peak, $l$ is the length of the cuvette ($l$ = 1cm) and $\mathrm{[CV+]}$ is the crystal violet concentration.
The absorbance, $Ab$, may therefore be used as a proxy of the crystal violet concentration in all calculations.

The integrated form equations for zero, first and second order can be rearranged and [CV+]o/[CV+]t substituted for Abo/Abt. This will give theoretical expressions that relate the absorbance values to time and to the rate constant kr’.

### Refences

Background information from:

Atkins P & de Paula J, Atkins’ Physical Chemistry, ninth edition, Oxford University Press, Oxford

Experiment:

Randall J, Holmquist D & Volz D 2010, Chemistry with Vernier, American Chemical Society
J. Chem. Educ. (2015), 92, pp 1692–1695. DOI: 10.1021/ed500876y

## Aims
1. Determine the rate law
2. Determine the rate constant
3. Determine the activation energy

## Virtual experiment procedure
1. Choose a weight for you samples (benzioc acid, Sucrose and naphthalene)
2. Choose the amount of water in the calorimeter

## Numerical skills

1. fitting of portions of data
2. error analysis

## Pre-lab questions
1. Rewrite the integrated form equations for zero, first and second order in terms of absorption.
    1. What is the main consequence of using the absorbcance instead of the concentration in the calculations ?
    2. would this affect the results for the reaction order and activation energy ?
2. Find literature values for the rate law and activation energy for the reaction between crystal violet and hydroxide.
3. In the **real** experiment the samples will be prepared by mixing comparable amounts of liquid taken from a $2.5\times10^{-5}$ M stock solution of CV and a 0.25 M stock solution of NaOH.
If you run two batches of experiments where you alternatively keep the amount of one of the two solutions constant, which species is **isolated** in the two sets?

##  Questions to be answered in the lab report
1. What is the rate law for the reaction between Crystal Violet and hydroxide ?
    1. what is the order of the reaction with respect to [CV$^+$] ?
    2. what is the order of the reaction with respect to [OH$^-$] ?
2. What is the activation energy for the reaction ?
3. How do your estimates compare with literature values ?

## Launch virtual experiment
- [Crystal Violet virtual lab](virtualExperiment.ipynb)


# Deep Learning and Applied AI project
## *Growing 3D Neural Cellar Automaton*

```python
# Clone the github repo on content folder
# of google colab machine
!git clone https://github.com/francesco-cubito97/DLAI_project_3D_NCA

# Create the model checkpoints folder
# in a more stable memory, like google drive
from google.colab import drive
drive.mount("/content/gdrive")
```

    Cloning into 'DLAI_project_3D_NCA'...
    remote: Enumerating objects: 123, done.
    remote: Counting objects: 100% (30/30), done.
    remote: Compressing objects: 100% (30/30), done.
    remote: Total 123 (delta 18), reused 0 (delta 0), pack-reused 93
    Receiving objects: 100% (123/123), 10.70 MiB | 7.01 MiB/s, done.
    Resolving deltas: 100% (50/50), done.
    Mounted at /content/gdrive

```python
MAIN_FOLDER = "/content/gdrive/MyDrive/DLAI_project_3D_NCA"
OBJECTS_FOLDER = "/content/DLAI_project_3D_NCA/Objects"
BINVOX_PATH = "/content/DLAI_project_3D_NCA"
# Here will save the checkpoints
CHECKPOINTS_FOLDER = MAIN_FOLDER + "/Checkpoints"

# Here I will save gifs
GIFS_FOLDER = MAIN_FOLDER + "/Gifs"

# Here I will save losses to generate plots
LOG_FOLDER = MAIN_FOLDER + "/Logs"

# Here there are some pretrained models
PRETRAINED_FOLDER = "/content/DLAI_project_3D_NCA/Pretrained_models"

import sys
sys.path.append(MAIN_FOLDER)
sys.path.append(BINVOX_PATH)
```

```python
# Create all folder and clear all old contents if exist
!mkdir -p $MAIN_FOLDER && rm -f $MAIN_FOLDER/*
!mkdir -p $CHECKPOINTS_FOLDER && rm -f $CHECKPOINTS_FOLDER/*
!mkdir -p $GIFS_FOLDER && rm -f $GIFS_FOLDER/*
!mkdir -p $LOG_FOLDER && rm -f $LOG_FOLDER/*
```

    rm: cannot remove '/content/gdrive/MyDrive/DLAI_project_3D_NCA/Checkpoints': Is a directory
    rm: cannot remove '/content/gdrive/MyDrive/DLAI_project_3D_NCA/Gifs': Is a directory
    rm: cannot remove '/content/gdrive/MyDrive/DLAI_project_3D_NCA/Logs': Is a directory

```python
# Import all needed components
import numpy as np
import pandas as pd
from IPython.display import Image
import imageio
import os
from typing import Optional, Callable, Dict, Union

# Deep Learning libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
from torchvision import datasets, models, transforms

# Visualization libraries
from torchsummary import summary
import binvox_rw #Taken from https://github.com/dimatura/binvox-rw-py
import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib.colors import to_rgb
%matplotlib inline
sns.set()
# Reproducibility stuff
import random
seed = 7
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
torch.cuda.manual_seed(seed)

# This can be set also to False with performance improvement
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
```

```python
# Global variables
BINVOX_PATHS = {
    "cow": "/cow_minecraft_14x8x16.binvox",
    "fox": "/fox_minecraft_7x10x16.binvox",
    "wolf": "/wolf_minecraft_16x5x9.binvox",
}
for p in BINVOX_PATHS:
    BINVOX_PATHS[p] = OBJECTS_FOLDER + BINVOX_PATHS[p]
    print(f"Element: {p}, path:{BINVOX_PATHS[p]}")

# Some useful color
COLORS = {
    "cow": to_rgb("#f2cB68"),
    "fox": to_rgb("#df5d5d"),
    "wolf": to_rgb("#42b0c1")
}

# Training variables
N_CHANNELS = 32
DIM = 16
PADDING = 4
TOT_ITERATIONS = 16000
DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(f'Using device: {DEVICE}')
```

    Element: cow, path:/content/DLAI_project_3D_NCA/Objects/cow_minecraft_14x8x16.binvox
    Element: fox, path:/content/DLAI_project_3D_NCA/Objects/fox_minecraft_7x10x16.binvox
    Element: wolf, path:/content/DLAI_project_3D_NCA/Objects/wolf_minecraft_16x5x9.binvox
    Using device: cuda:0

```python
def loadBinvox(path):
    '''
    Load binvox model and convert it into numpy array
    for simplicity

        Params
        ------
        - path: path of the model

        Returns
        -------
        - voxels numpy array
    '''
    with open(path, 'rb') as f:
        voxels = binvox_rw.read_as_3d_array(f)

    return np.array(voxels.data, dtype=bool)

def plotVoxels(voxels, title : str="Voxels models", figsize : tuple=(10, 10), c=None) -> None:
    '''
    Plot voxels starting from numpy array

        Params
        ------
        - voxels: numpy array of boolean voxels
        - figsize: dimension in inches of image
        - c: color to use in rgba. Can be a single float, a list of 3/4 float in [0, 1] or [0, 255] range or a 4D array of RGB/RGBA colors, one for each voxel

        Returns
        -------
        - `None`
    '''
    fig = plt.figure(figsize=figsize)

    ax = fig.add_subplot(projection="3d")

    if c is not None: colors = c # Set to middle gray
    else: colors = [0.5, 0.5, 0.5]

    ax.voxels(voxels, facecolor=colors, edgecolor=[1.0, 1.0, 1.0])

    # Change point of view of camera
    #ax.set_title(f"View {abs(i*(360//tot))} degree")
    #ax.view_init(0, i*(360//tot))

    fig.suptitle(title, fontsize=figsize[0]*4)
    plt.show()

def createBox(obj):
    '''
    Calculate the box around a 3d object

        Params
        ------
        - obj: space in which lies the 3d object

        Returns
        -------
        - 3d box object
    '''
    # Get coordinates of the object
    coord = (obj > 0).nonzero()

    # Calculate where each dimension starts and ends
    # for the object
    start_x = torch.min(coord[:, 0])
    size_x = torch.max(coord[:, 0]) - start_x + 1

    start_y = torch.min(coord[:, 1])
    size_y = torch.max(coord[:, 1]) - start_y + 1

    start_z = torch.min(coord[:, 2])
    size_z = torch.max(coord[:, 2]) - start_z + 1

    # Create the parallelepiped with the defined dimensions
    x = slice(start_x, start_x + size_x)
    y = slice(start_y, start_y + size_y)
    z = slice(start_z, start_z + size_z)

    parallelepiped = torch.zeros_like(obj)
    parallelepiped[x, y, z] = 1.0

    return parallelepiped.clone().detach()

def centerObj(obj):
    '''
    Centralize a 3d object in the space in which it lies

        Params
        ------
        - obj: the object that should be centralized

        Returns
        -------
        - The centered object
    '''
    # Get center of the space
    center_s = torch.tensor([obj.shape[0]//2, obj.shape[1]//2, obj.shape[2]//2])

    ## To center the object we need to count how many
    ## vertices of distance there are between the
    ## center position and the center of the parallelepiped

    # Get the box around the object
    p = createBox(obj)

    # Get the parallelepiped coordinates
    coord = (p > 0).nonzero()

    # Sum all voxels coordinates and average to get centroid
    sum = coord.sum(dim=0)
    centroid = torch.div(sum, coord.shape[0], rounding_mode="floor")

    # Get the shift value in voxels between the two centers
    shift = tuple(center_s - centroid)

    # Move the object to the center of the space
    # based on the box center calculated
    obj = torch.roll(obj, shift, (0, 1, 2))

    # Get the parallelepiped coordinates
    coord = (obj > 0).nonzero()

    # Average to get centroid
    sum = coord.sum(dim=0)
    centroid = torch.div(sum, coord.shape[0], rounding_mode="floor")

    return obj

# One temptative to speed up the training
def erosion3D(obj):
    '''
    Morphological 3d erosion of object

        Params
        ------
        - obj: the object that should be eroded

        Returns
        -------
        - Eroded object
    '''
    # Create a copy of the object
    o = obj.clone().detach()
    # Create the 3D average pool operation
    avgP1 = nn.AvgPool3d((1, 1, 3), stride=(1, 1, 1))
    avgP2 = nn.AvgPool3d((1, 3, 1), stride=(1, 1, 1))
    avgP3 = nn.AvgPool3d((3, 1, 1), stride=(1, 1, 1))
    # Return the mask of elements
    mask1 = F.pad(avgP1(o) < 1.0, [1, 1, 0, 0, 0, 0])
    mask2 = F.pad(avgP2(o) < 1.0, [0, 0, 1, 1, 0, 0])
    mask3 = F.pad(avgP3(o) < 1.0, [0, 0, 0, 0, 1, 1])

    print(mask1.shape, mask2.shape, mask3.shape)

    # Invert the mask
    mask = ~(mask1 & mask2 & mask3)

    # Multiply the mask to the original
    # object to erode it
    return o * mask.float()
```

```python
# Loading all binvox models and visualizing them
target_objects = {
    "cow": None,
    "fox": None,
    "wolf": None
}

o = "cow"
vx = loadBinvox(BINVOX_PATHS[o])
# Rotate to adjust
target_objects[o] = torch.tensor(np.rot90(vx, k=1, axes=(2, 1)).copy())
target_objects[o] = F.pad(target_objects[o], [PADDING, PADDING, PADDING, PADDING, PADDING, PADDING])
target_objects[o] = centerObj(target_objects[o]).float()

o = "fox"
vx = loadBinvox(BINVOX_PATHS[o])
target_objects[o] = torch.tensor(np.rot90(vx, k=1, axes=(0, 1)).copy())
target_objects[o] = F.pad(target_objects[o], [PADDING, PADDING, PADDING, PADDING, PADDING, PADDING])
target_objects[o] = centerObj(target_objects[o]).float()

o = "wolf"
vx = loadBinvox(BINVOX_PATHS[o])
target_objects[o] = torch.tensor(np.rot90(np.rot90(vx, k=2, axes=(0, 1)), k=1, axes=(1, 2)).copy())
target_objects[o] = F.pad(target_objects[o], [PADDING, PADDING, PADDING, PADDING, PADDING, PADDING])
target_objects[o] = centerObj(target_objects[o]).float()

for key in target_objects:
    print(target_objects[key].shape)

```

    torch.Size([24, 24, 24])
    torch.Size([24, 24, 24])
    torch.Size([24, 24, 24])

```python
# Visualize the result
for key in target_objects:
    plotVoxels(target_objects[key].numpy(), figsize=(5, 5), c=COLORS[key])
    print("Object shape", target_objects[key].shape)
```

## Project details
Now the steps are consequential. We should obtain a neural network able to output at each time step a new configuration of the final object in 3D voxels, each one with a certain probability of existence that is dependent, like in the original paper, on a value that we will call $\alpha$, that will represent the live itself.
In our case:
\begin{align}
       0 \le \alpha \le 1
\end{align}
and  will have the folling meaning:
- $0\le \alpha < 0.1$: dead cell
- $0.1\le \alpha < 0.5$: growing cell
- $0.5\le \alpha \le1$: mature cell

The network will be composed of an initial layer that will make a 3D convolutional operation with *Sobel Filter* (taken from [here](https://https://stackoverflow.com/questions/7330746/implement-3d-sobel-operator)):
>
>
>

The central cell will be passed directly, without any filter (identity filter).

```python
#@title Sobel filters and identity filter
# 3D Sobel filters to make a comparison with the one
# found during training
SF_X = torch.tensor([
                        [[-1, 0, 1],
                         [-2, 0, 2],
                         [-1, 0, 1]],
                        [[-2, 0, 2],
                         [-4, 0, 4],
                         [-2, 0, 2]],
                        [[-1, 0, 1],
                         [-2, 0, 2],
                         [-1, 0, 1]],
                         ]).float()
# Normalize the values for the application of the convolutions
# with the (filter.x * filter.y * filter.z) neighbors
neighbors = SF_X.shape[0] * SF_X.shape[1] * SF_X.shape[2] - 1

SF_X = SF_X/neighbors
SF_Y = torch.einsum("xyz -> xzy", SF_X)
SF_Z = torch.einsum("xyz -> zyx", SF_X)

print(SF_X , "\n")
print(SF_Y , "\n")
print(SF_Z , "\n")

# Identity filter
IDF = torch.tensor([0, 1, 0]).float()
IDF = torch.einsum("a, b, c -> abc", IDF, IDF, IDF)
print(IDF)
```

    tensor([[[-0.0385,  0.0000,  0.0385],
             [-0.0769,  0.0000,  0.0769],
             [-0.0385,  0.0000,  0.0385]],

            [[-0.0769,  0.0000,  0.0769],
             [-0.1538,  0.0000,  0.1538],
             [-0.0769,  0.0000,  0.0769]],

            [[-0.0385,  0.0000,  0.0385],
             [-0.0769,  0.0000,  0.0769],
             [-0.0385,  0.0000,  0.0385]]])

    tensor([[[-0.0385, -0.0769, -0.0385],
             [ 0.0000,  0.0000,  0.0000],
             [ 0.0385,  0.0769,  0.0385]],

            [[-0.0769, -0.1538, -0.0769],
             [ 0.0000,  0.0000,  0.0000],
             [ 0.0769,  0.1538,  0.0769]],

            [[-0.0385, -0.0769, -0.0385],
             [ 0.0000,  0.0000,  0.0000],
             [ 0.0385,  0.0769,  0.0385]]])

    tensor([[[-0.0385, -0.0769, -0.0385],
             [-0.0769, -0.1538, -0.0769],
             [-0.0385, -0.0769, -0.0385]],

            [[ 0.0000,  0.0000,  0.0000],
             [ 0.0000,  0.0000,  0.0000],
             [ 0.0000,  0.0000,  0.0000]],

            [[ 0.0385,  0.0769,  0.0385],
             [ 0.0769,  0.1538,  0.0769],
             [ 0.0385,  0.0769,  0.0385]]])

    tensor([[[0., 0., 0.],
             [0., 0., 0.],
             [0., 0., 0.]],

            [[0., 0., 0.],
             [0., 1., 0.],
             [0., 0., 0.]],

            [[0., 0., 0.],
             [0., 0., 0.],
             [0., 0., 0.]]])

```python
def randVoxel(obj):
    '''
    Select randomly a voxel in a 3d object

        Params
        ------
        - obj: the object in which pick the random voxel

        Returns
        -------
        - `None`
    '''
    # Get the list of possible coordinates where to pick the random value
    coord = (obj > 0.0).nonzero()

    # Pick the seed between coordinates
    n = torch.randint(coord.shape[0], []).item()

    return tuple(coord[n])

def initCA(dim, channels, obj, random_seed=False):
    '''
    Initialize virtual 3D cells lattice

        Params
        ------
        - dim: dimension of the 3d space
        - channels: channels of the 3d space
        - obj: the object from which pick the seed voxel
        - random_seed: use a random cell in the object

        Returns
        -------
        - Tensor of cells with seed
    '''
    # Create the lattice of dim x dim x dim dimension
    cells = torch.zeros((channels, dim, dim, dim), dtype=torch.float32)

    # Get the random voxel
    if random_seed:
        vx = randVoxel(obj)

        x, y, z = vx[0], vx[1], vx[2]
        # The seed cell will have all features values equal to one
        cells[:, x, y, z] = 1.0

    # Return only the central seed
    else:
        cells[:, dim//2, dim//2, dim//2] = 1.0

    return cells

# Cells space
dim = DIM+2*PADDING
cells = initCA(dim, N_CHANNELS, target_objects["cow"])
cellsN = torch.einsum("cdhw -> dhwc", cells)

# Extract the alpha channel
alpha = cells[:1, ...].clone().detach()
alphaN = torch.einsum("cdhw -> dhwc", alpha)

# Set RGBA colors of cells. Died cells wiil have alpha = 0.0
colorsN = torch.zeros((dim, dim, dim, 4))
colorsN[..., :3] = torch.tensor(COLORS["cow"]) # To have a gray color
colorsN[..., 3:] = alphaN
print(cells[cells > 0].shape)

ax = plt.figure(figsize=(10, 10)).add_subplot(projection='3d')
ax.voxels(cellsN[..., 0].numpy(), facecolors=colorsN, edgecolor="white")
plt.show()
```

```python
# Utils
def countParameters(model: torch.nn.Module) -> int:
    """
    Counts the number of trainable parameters of a module

        Params
        ------
        - model: model that contains the parameters to count

        Returns
        -------
        - the number of parameters in the model
    """
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

def getLivingMask(x):
    """
    Create a mask of alive cells seeing at alpha channel

        Params
        ------
        - x: input 3d object

        Returns
        -------
        - boolean mask of living cells
    """
    # The alpha values are in the first channel
    alpha = x[:, :1, :, :, :].clone()
    # Return a boolean mask that is true in 3x3x3 neighbors
    # maximum with an alpha value higher than 0.1, so if the maximum neighbor
    # cell is mature
    return F.max_pool3d(alpha, kernel_size=3, stride=1, padding=1) > 0.1

def normWeights(weight):
    '''
    Normalize weights to avoid exploding gradients problem

        Params
        ------
        - weight: weights data to be normalized

        Returns
        -------
        - normalized weights
    '''
    w = weight.view(1, -1).clone().detach()
    with torch.no_grad():
        # Inplace division
        w.div_(torch.norm(w, dim=1, keepdim=True))

    return w.view(weight.shape)

def weightsInit(smodel, slow_train=False):
    '''
    Initialize weights 3D convolutional layers in sequential component

        Params
        ------
        - smodel: sequential model
        - slow_train: init the last layer with zeros to slow the convergency

        Returns
        -------
        - `None`
    '''
    for idx, layer in enumerate(smodel):
        if isinstance(layer, nn.Conv3d) and idx != 4:
            print(f"Layer {layer} initialized with He initialization")
            nn.init.kaiming_uniform_(layer.weight, mode='fan_in', nonlinearity='leaky_relu')

        elif idx == 4:
            # Initialize the last 1x1x1 convolutional layer with zeros
            # otherwise the network will not learn to construct step-by-step
            # like a Cellular Automaton does
            print(f"Layer {layer} initialized with zeros")
            nn.init.zeros_(layer.weight)

```

```python
# Create 3D CA module like an RNN
# to use backpropagation through time
class CA3D(nn.Module):
    def __init__(self, input_channels: int, kernel_size: int, output_size: int, depthwise_filter=None, device="cuda") -> None:
        '''
        CA3D Neural Network initialzation

            Params
            ------
            - input_channels: number of input channels
            - kernel_size: dimension of filters in depthwise convolutions
            - output_size: number of channels in output
            - depthwise_filter: kernel values of 3x3x3 depthwise filter
            - device: device in which the module is executed

            Returns
            -------
            - `None`
        '''
        super(CA3D, self).__init__()
        self.input_channels = input_channels
        self.kernel_size = kernel_size
        self.output_size = output_size
        self.fire_rate = 0.5
        self.device = device

        # Must be trained end-to-end
        # I would like to obtain the same shape of output that
        # I would obtain using Sobel Filters
        self.identity_kernel = IDF.view(1, 1, self.kernel_size, self.kernel_size, self.kernel_size).repeat(self.input_channels, 1, 1, 1, 1).to(self.device)

        # The result from this filter must be concatenated with the result
        # obtained applying the identity filter
        self.depthwise_filter = nn.Conv3d(in_channels=input_channels, out_channels=3*input_channels,
                                         kernel_size=kernel_size, padding="same",
                                         groups=input_channels, bias=False)

        if depthwise_filter is not None:
            # Use an already filled filter
            self.depthwise_filter.weight.data = depthwise_filter
            self.depthwise_filter.requires_grad_(False)

        self.dmodel = nn.Sequential(
            # First 3d convolution: 128 -> 192, 1x1 conv
            nn.Conv3d(input_channels*4, input_channels*6, kernel_size=1, bias=False),
            # Activation function
            nn.LeakyReLU(),
            # Second 3d convolution: 192 -> 96, 1x1 conv
            nn.Conv3d(input_channels*6, input_channels*3, kernel_size=1, bias=False),
            # Activation function
            nn.LeakyReLU(),
            # Third 3d convolution: 96 -> 32, 1x1 conv
            nn.Conv3d(input_channels*3, output_size, kernel_size=1, bias=False)
        )

    def neighPerc(self, x):
        '''
        3x3x3 neighbors perception using gradients plus identity value

            Params
            ------
            - x: 3d model in input

            Returns
            -------
            - Perception tensor
        '''
        # Apply the Depthwise 3d convolution
        # First apply identity filter depthwise
        id_res = F.conv3d(x, self.identity_kernel, padding="same", groups=self.input_channels)

        # Apply the filter trained depthwise
        filter_res = self.depthwise_filter(x)

        # Concatenate the two results
        perception = torch.concat([id_res, filter_res], 1)

        return perception

    def forward(self, x, fire_rate=None, perception_result=False, bef_alive=False):
        '''
        Forward-pass through the network to create computational graph

            Params
            ------
            - x: Input 3d object
            - fire_rate: updatable cells minumum value

            Returns
            -------
            - Output 3d object
        '''
        pre_life_mask = getLivingMask(x)

        # Depthwise convolutions over 3x3x3 neighborhood
        perception = self.neighPerc(x)

        # Perception of neighbors gradients values
        neigh_grad = self.dmodel(perception)

        if fire_rate is None:
            fire_rate = self.fire_rate
        # Stochastic update of the alpha channel
        # Get a boolean mask to update only cells
        # with a value smaller than or equal to fire_rate
        update_mask = torch.rand(x[:, :1, ...].shape, device=self.device) <= fire_rate

        # Control the update only of the alpha channel
        x = x + (neigh_grad * update_mask.float())

        post_life_mask = getLivingMask(x)
        # Alive mask, maintain alive only the
        # cells alive before and after computations
        life_mask = pre_life_mask & post_life_mask
        out = x * life_mask.float()

         # Visulize the perception
        if perception_result and bef_alive:
            return perception, x, out

        return out
```

```python
#@title Choose the model to train
model_type = "Trainable perception filter" #@param ["Trainable perception filter", "Fixed perception filter"]
# Model with trainable initial filter
# Instantiate the model
if model_type == "Trainable perception filter":
    mod = "tpf"
    ca3d = CA3D(input_channels=N_CHANNELS, kernel_size=3, output_size=N_CHANNELS, device=DEVICE)
else:
    # Compose sobel filters
    mod = "fpf"
    l = []
    for f in [SF_X, SF_Y, SF_Z]:
        l.append(f.view(1, 1, 3, 3, 3).repeat(N_CHANNELS, 1, 1, 1, 1))
    sobel_filter = torch.concat(l, 0)

    ca3d = CA3D(input_channels=N_CHANNELS, kernel_size=3, output_size=N_CHANNELS, depthwise_filter=sobel_filter, device=DEVICE)

# Initialize last three layers of model
weightsInit(ca3d.dmodel)

ca3d = ca3d.to(DEVICE)

# Visualize layers and output dimensions
print(ca3d)
print(f"Number of parameters: {countParameters(ca3d)}")
print("\n\n")
summary(ca3d, cells.shape, batch_size=1)
```

    Layer Conv3d(128, 192, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False) initialized with He initialization
    Layer Conv3d(192, 96, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False) initialized with He initialization
    Layer Conv3d(96, 32, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False) initialized with zeros
    CA3D(
      (depthwise_filter): Conv3d(32, 96, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=same, groups=32, bias=False)
      (dmodel): Sequential(
        (0): Conv3d(128, 192, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
        (1): LeakyReLU(negative_slope=0.01)
        (2): Conv3d(192, 96, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
        (3): LeakyReLU(negative_slope=0.01)
        (4): Conv3d(96, 32, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
      )
    )
    Number of parameters: 48672

    ----------------------------------------------------------------
            Layer (type)               Output Shape         Param #
    ================================================================
                Conv3d-1        [1, 96, 24, 24, 24]           2,592
                Conv3d-2       [1, 192, 24, 24, 24]          24,576
             LeakyReLU-3       [1, 192, 24, 24, 24]               0
                Conv3d-4        [1, 96, 24, 24, 24]          18,432
             LeakyReLU-5        [1, 96, 24, 24, 24]               0
                Conv3d-6        [1, 32, 24, 24, 24]           3,072
    ================================================================
    Total params: 48,672
    Trainable params: 48,672
    Non-trainable params: 0
    ----------------------------------------------------------------
    Input size (MB): 1.69
    Forward/backward pass size (MB): 74.25
    Params size (MB): 0.19
    Estimated Total Size (MB): 76.12
    ----------------------------------------------------------------

## **Training phase**

```python
#@title Choose the target
obj = "fox" #@param ["cow", "fox", "wolf"]
target = target_objects[obj][None, None, :, :, :].clone().detach()

plotVoxels(target[0, 0].numpy(), title="Target object", c=COLORS[obj])
print(f"Target object shape: {target.shape}, dtype: {target.dtype}")

# Initialize the cells
seed = initCA(target.shape[2], N_CHANNELS, target[0, 0])[None, ...]
plotVoxels(seed[0, 0].numpy(), title="Initial state", c=COLORS[obj])
print(f"Seed shape: {seed.shape}, dtype: {seed.dtype}")

# Loss criterion
lossFunc = nn.L1Loss()
def error(x, target):
    return lossFunc(torch.clamp(x[:, 0], min=0.0, max=1.0), target[:, 0])

print(f"Initial loss value: {error(seed, target)}")

# Optimizer with L2-regularization and momentum
lr = 5e-3 if mod == "fpf" else 3e-3
wd = 1e-5
m = 0.8
opt = optim.SGD(ca3d.parameters(), lr=lr, weight_decay=wd)

```

```python
def loadCheckpoint(path, model, opt):
    '''
    Load a model checkpoint specifing the path

        Params
        ------
        - path: path of the model checkpoint
        - model: reference to the model
        - opt: reference to the optimazer

        Returns
        -------
        - epoch and loss of the model checkpoint
    '''
    checkpoint = torch.load(path)
    model.load_state_dict(checkpoint['model_state_dict'])
    opt.load_state_dict(checkpoint['optimizer_state_dict'])

    return checkpoint['n_iter'], checkpoint['loss']

def saveCheckpoint(path, model_sd, opt_sd, n_iter, loss):
    '''
    Save a model checkpoint into a specified path location

        Params
        ------
        - path: path in which save the modelcheckpoint
        - model_sd: model state dictionary
        - opt_sd: optimizer state dictionary
        - n_iter: last number iteration of the model
        - loss: loss at last iteration

        Returns
        -------
        - `None`
    '''
    torch.save({
        'n_iter': n_iter,
        'model_state_dict': model_sd,
        'optimizer_state_dict': opt_sd,
        'loss': loss
    }, path)
```

```python
def trainStep(x, target, model: torch.nn.Module, opt: torch.optim.Optimizer,  device: str = "cuda"):
    '''
    Training iteration of the model

        Params
        ------
        - x: Input 3d object
        - target: Target 3d object
        - model: model to train
        - opt: optimizer
        - device: type of device in which make calculous

        Returns
        -------
        - Model output and loss reached
    '''
    model.train()

    # Send to device
    x, target = x.to(device), target.to(device)
    # Try with a random number of iterations
    n_iter = torch.randint(85, 100, []).item()

    # Create a local variable
    _x = x.clone().detach()
    _x.requires_grad_(True)

    for i in range(n_iter):
        # Iterate the model without backpropagate
        # to use always the same set of learned rules
        # (otherwise calling backpropagation the rules changes)
        _x = model(_x)

        # Truncated Backpropagation through time
        # Every 5 iterations backpropagate
        '''
        if (i+1)%5 == 0:
            opt.zero_grad()
            loss = error(_x, target)

            #if(i+1)%50: print(f"Internal loss: {loss}")
            #wandb.log({"Internal_loss": loss})

            loss.backward()
            # Avoid exploding gradient
            #nn.utils.clip_grad_norm_(model.parameters(), .7)

            opt.step()

            _x = _x.clone().detach()
            _x.requires_grad_(True)
            #print("Done!")'''

    # Each #iter steps
    # Calculate the loss and backpropagate through time
    opt.zero_grad()
    loss = error(_x, target)

    loss.backward()
    # Avoid exploding gradient
    #nn.utils.clip_grad_norm_(model.parameters(), .7)
    opt.step()

    return _x.clone().detach(), loss.clone().detach()
```

```python
# Training iterations
x = seed.clone().detach()
x.requires_grad_(True)

losses = []

checks = [1000, 1500, 2000, 3000, 4000, 6000, 8000, 12000, 16000]
i = 0

for n_iter in range(1, TOT_ITERATIONS+1):
    # Start each new iteration with single cell
    out, loss = trainStep(x, target, model=ca3d, opt=opt, device=DEVICE)

    # Print the reached loss
    print(f"At iteration: {n_iter: 4d} Reached loss: {loss}\n")

    losses.append(loss.item())

    # Early stopping
    if(loss.item() <= 0.001):
        print("Reached early stopping point!")
        saveCheckpoint(f"{CHECKPOINTS_FOLDER}/checkpoint_{mod}_{n_iter}_{obj}.pth", ca3d.state_dict(), opt.state_dict(), n_iter, loss)

        # Save loss and empty list
        torch.save(torch.tensor(losses), f"{LOG_FOLDER}/losses_{mod}_{n_iter}_{obj}.pt")
        break

    # Save checkpoint
    if n_iter%checks[i] == 0:
        i += 1
        saveCheckpoint(f"{CHECKPOINTS_FOLDER}/checkpoint_{mod}_{n_iter}_{obj}.pth", ca3d.state_dict(), opt.state_dict(), n_iter, loss)

        # Save loss and empty list
        torch.save(torch.tensor(losses), f"{LOG_FOLDER}/losses_{mod}_{n_iter}_{obj}.pt")
        losses = []
```

```python
# Format the path
loss_conc = np.array([])
for c in range(i):
    path = f"{LOG_FOLDER}/losses_{mod}_{checks[c]}_{obj}.pt"
    inp = torch.load(path).to(DEVICE)
    loss_conc = np.concatenate((loss_conc, inp.cpu().numpy()))

print(loss_conc.shape)

```

    (1000,)

```python
# Plot the result of training
from cycler import cycler
ax = plt.figure(figsize=(10, 8)).subplots()
ax.set_prop_cycle(cycler(color=["tab:blue", "tab:orange"]))
ax.plot(loss_conc, linestyle=(0, (3, 10, 1, 10)), linewidth=1.5)

ax.set_title(f"Losses plot {mod} filter {obj}")

l = f"Fixed {obj}" if mod == "fpf" else f"Trained {obj}"
plt.legend([l], ncol=2, loc='upper left')
plt.savefig(f"{MAIN_FOLDER}/plot_loss_cow.png")
```

## **Model Evaluation**

```python
# Load the last network that should be the most robust
if i != 0:
    _iter, _loss = loadCheckpoint(f"{CHECKPOINTS_FOLDER}/checkpoint_{mod}_{checks[i-1]}_{obj}.pth", ca3d, opt)

    print(f"Checkpoint saved at iteration number: {_iter}, Loss: {_loss}")
else:
    print("Error! Execute training section at least 1000 iterations or \nskip this section and use pretrained model")
```

    Checkpoint saved at iteration number: 1000, Loss: 0.017115643247961998

```python
#@title To use pretrained models
mod = "fpf" #@param ["fpf", "tpf"]
obj = "cow" #@param ["cow", "fox"]
_iter, _loss = loadCheckpoint(f"{PRETRAINED_FOLDER}/checkpoint_{mod}_{16000}_{obj}.pth", ca3d, opt)

print(f"Checkpoint saved at iteration number: {_iter}, Loss: {_loss}")
```

    Checkpoint saved at iteration number: 16000, Loss: 0.005899389274418354

```python
# Visualize the capability to represent the cow
def eval_model(model, saved_out, save_intermediate=False):
    '''
    Evaluate the model
        Params
        ------
        - model: the model to be evaluated
        - saved_out: list of saved output tensors
        - save_intermediate: boolean, if true the function returns perception tensor, updating tensor and final output at 80th time step

        Returns
        -------
        - the modified saved_out
    '''
    # Iterate to 100 times to create the object
    for i in range(1, 100):

        model.eval()

        with torch.no_grad():
            inp = torch.load(saved_out[-1]).to(DEVICE)
            if i == 80 and save_intermediate==True:
                p, b, out_ext = ca3d(inp, perception_result=True, bef_alive=True)
            else: out = ca3d(inp)

            filename = f"{i}"
            torch.save(out.cpu().clone().detach(), filename)
            saved_out.append(filename)

            del out
            del inp
        torch.cuda.empty_cache()

    print(f"Number of saved outputs: {len(saved_out)}")
    if save_intermediate:
        return p, b, out_ext

```

```python
# Visualize the outputs
def visualizeOutputs(saved_out,  dim, iter_gif=100):
    for i in range(len(saved_out)):
        cells = torch.zeros((1, N_CHANNELS, dim, dim, dim))
        # A cell will exist only if its values is different from zero
        alpha = torch.load(saved_out[i%iter_gif])[0, :1, ...].cpu()

        # Clamp largest/smallest values
        alpha = torch.clamp(alpha, min=0, max=1)
        cells[0, :1, ...] = alpha
        cellsN = cells.numpy()[0, 0, ...]

        # Adapt alpha channel to be visualizable
        alpha = torch.einsum("cdhw -> dhwc", alpha)
        alphaN = alpha.numpy()

        colorsN = np.zeros((dim, dim, dim, 4))
        colorsN[..., :3]  = COLORS[obj] # This is to obtain gray in r,g,b
        colorsN[..., 3:4] = alphaN

        # The first channel represent the alpha and will be
        # the unique relevant for us
        ax = plt.figure().add_subplot(projection="3d")
        ax.voxels(cellsN, facecolors=colorsN, edgecolors=colorsN)

        # Save the frame
        plt.savefig(saved_out[i%iter_gif] + ".png")
        plt.close()

        # Create a gif with each model saved images
        if (i+1)%iter_gif == 0:
            print(f"Creating the gif for model_n_{i+1}...")
            # Build gif
            with imageio.get_writer(f'{GIFS_FOLDER}/{obj}_model_{mod}_n_{i+1}.gif', mode='I') as writer:
                for k in range(iter_gif):
                    # Extract path and free saved_x
                    path = saved_out[0]
                    del saved_out[0]

                    # Compose the gif
                    try:
                        image = imageio.imread(path + ".png")
                        writer.append_data(image)
                        # Remove useless files from disk
                        os.remove(path)
                        os.remove(path + ".png")
                    except:
                        print(f"Not found file: {path}.png at iteration {i} and gif iteration {k}")

            print(f"Done! Now saved are {len(saved_out)}")
```

```python
# Visualize the capability to represent the object
x = seed.clone().detach().to(DEVICE)

# Iterate to 100 times to create the object
torch.save(x.cpu().clone().detach(), "0")
saved_out = []
saved_out.append("0")

eval_model(ca3d, saved_out)
```

    Number of saved outputs: 100

```python
# Await some minutes and the result will be
# created and saved in "/content/gdrive/MyDrive/DLAI_project_3D_NCA/Gifs"
# location on google drive
visualizeOutputs(saved_out, dim=DIM+2*PADDING)
```

    Creating the gif for model_n_100...
    Done! Now saved are 0


```julia
"""Tutorial: CEPA0 and CCD"""

__author__    = ["D. Menendez", "Adam S. Abbott"]
__credit__    = ["D. Menendez", "Adam S. Abbott", "Justin M. Turney"]

__copyright__ = "(c) 2014-2020, The Psi4Julia Developers"
__license__   = "BSD-3-Clause"
__date__      = "2020-08-02"
```

    "2020-08-02"

# Introduction
In this tutorial, we will implement the coupled-electron pair approximation (CEPA0) and coupled-cluster doubles (CCD) methods using our spin orbital framework covered in the [previous tutorial](8a_Intro_to_spin_orbital_postHF.ipynb).

### I. Coupled Cluster Theory

In single reference coupled cluster theory, dynamic correlation is acquired by operating an exponential operator on some reference determinant, such as a Hartree-Fock wavefunction, to obtain the coupled cluster wavefunction given by:

\begin{equation}
\mid \mathrm{\Psi_{CC}} \rangle = \exp(\hat{T}) \mid \mathrm{\Phi} \rangle
\end{equation}

where $\hat{T} = T_1 + T_2 + ... + T_n$ is the sum of "cluster operators" which act on our reference wavefunction to excite electrons from occupied ($i, j, k$...) to virtual ($a, b, c$...) orbitals. In second quantization, these cluster operators are expressed as:

\begin{equation}
T_k = \left(\frac{1}{k!}\right)^2 \sum_{\substack{i_1 \ldots i_k \\ a_1 \ldots a_k }} t_{i_1 \ldots i_k}^{a_1 \ldots a_k}   a_{a_1}^{\dagger} \ldots a_{a_k}^{\dagger} a_{i_k} \ldots a_{i_1}
\end{equation}

where $t$ is the $t$-amplitude, and $a^{\dagger}$ and $a$ are creation and annihilation operators.

### II. Coupled Cluster Doubles
For CCD, we only include the doubles cluster operator:

\begin{equation}
\mid \mathrm{\Psi_{CCD}} \rangle = \exp(T_2) \mid \mathrm{\Phi} \rangle
\end{equation}

The CCD Schrödinger equation is

\begin{equation}
\hat{H} \mid \mathrm{\Psi_{CCD}} \rangle = E \mid \mathrm{\Psi_{CCD}}\rangle
\end{equation}

The details will not be covered here, but if we project the CCD Schrödinger equation on the left by our Hartree-Fock reference determinant $ \langle \mathrm{\Phi}\mid $, assuming intermediate normalization $\langle \Phi \mid \mathrm{\Psi_{CCD}} \rangle = 1$, we obtain:

\begin{equation}
 \langle \Phi \mid \hat{H} \space \exp(T_2) \mid \Phi \rangle = E
\end{equation}

which is most easily evaluated with a diagrammatic application of Wick's theorem. Assuming Brillouin's theorem applies (that is, our reference is a Hartree-Fock wavefunction) we obtain:

\begin{equation}
E_{\mathrm{CCD}} = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}
\end{equation}

A somewhat more involved derivation is that of the $t$-amplitudes. These are obtained in a similar fashion to the energy expression, this time projecting the CCD Schrödinger equation on the left by a doubly-excited reference determinant $ \langle\Phi_{ij}^{ab}\mid $:

\begin{equation}
\langle\Phi_{ij}^{ab}\mid \hat{H} \space \exp(T_2) \mid \Phi \rangle
\end{equation}

I will spare you the details of solving this expectation value as well. But, if one evaluates the diagrams via Wick's theorem and simplifies, the $t$-amplitudes are given by:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} - \tfrac{1}{2}\hat{P}_{(a \space / \space b)} \bar{g}_{kl}^{cd} t_{ac}^{ij} t_{bd}^{kl} - \tfrac{1}{2}  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ab}^{ik} t_{cd}^{jl} + \tfrac{1}{4} \bar{g}_{kl}^{cd} t_{cd}^{ij} t_{ab}^{kl} +  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ac}^{ik} t_{bd}^{jl} \right)
\end{equation}

where $(\mathcal{E}_{ab}^{ij})^{-1}$ is the orbital energy denominator, more familiarly known as

\begin{equation}
(\mathcal{E}_{ab}^{ij})^{-1} = \frac{1}{\epsilon_i + \epsilon_j - \epsilon_a - \epsilon_b}
\end{equation}

and $\bar{g}_{pq}^{rs}$ is the antisymmetrized two-electron integral in physicist's notation $\langle pq \mid\mid rs \rangle$. $\hat{P}$ is the *antisymmetric permutation operator*. This operator acts on a term to produce the sum of the permutations of the indicated indices, with an appropriate sign factor. Its effect is best illustrated by an example. Consider the fourth term, which is really four terms in one.

$\hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk}$ produces:

1. The original: $ \quad \bar{g}_{ak}^{ic} t_{bc}^{jk} \\ $

2. Permuation of $a$ and $b$: $ \quad  \textrm{-} \bar{g}_{bk}^{ic} t_{ac}^{jk} \\ $

3. Permuation of $i$ and $j$: $ \quad \, \, \textrm{-} \bar{g}_{ak}^{jc} t_{bc}^{ik} \\ $

4. Permuation of $a$ and $b$, $i$ and $j$: $ \quad \bar{g}_{bk}^{jc} t_{ac}^{ik} \\ $

Note that each permutation adds a sign change. This shorthand notation keeps the equation in a more manageable form.

Since the $t$-amplitudes and the energy depend on $t$-amplitudes, we must iteratively solve these equations until they reach self consistency, and the energy converges to some threshold.

### III. Retrieving MP2 and CEPA0 from the CCD equations
It is interesting to note that if we only consider the first term of the expression for the doubles amplitude $t_{ab}^{ij}$ and plug it into the energy expression, we obtain the MP2 energy expression:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \bar{g}_{ab}^{ij}
\end{equation}

\begin{equation}
E_{\mathrm{MP2}} = \tfrac{1}{4}  \bar{g}_{ij}^{ab} t_{ab}^{ij}  = \tfrac{1}{4} \bar{g}_{ij}^{ab} \bar{g}_{ab}^{ij}   (\mathcal{E}_{ab}^{ij})^{-1}
\end{equation}

Furthermore, if we leave out the quadratic terms in the CCD amplitude equation (terms containing two $t$-amplitudes), we obtain the coupled electron-pair approximation (CEPA0):
\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} \right)
\end{equation}

The CEPA0 energy expression is identical:

\begin{equation}
E_{\mathrm{CEPA0}} = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}
\end{equation}

Using our spin orbital setup for the MO coefficients, orbital energies, and two-electron integrals used in the [previous tutorial](8a_Intro_to_spin_orbital_postHF.ipynb), we are equipped to program the expressions for the CEPA0 and CCD correlation energy.

### Implementation: CEPA0 and CCD
As usual, we import Psi4, NumPy, and TensorOperations, and set the appropriate options.

```julia
# ==> Import statements & Global Options <==
using PyCall: pyimport
psi4 = pyimport("psi4")
np   = pyimport("numpy")
using TensorOperations: @tensor
using Formatting: printfmt

psi4.set_memory(Int(2e9))
numpy_memory = 2
psi4.core.set_output_file("output.dat", false)
```

      Memory set to   1.863 GiB by Python driver.

```julia
# ==> Molecule & Psi4 Options Definitions <==
mol = psi4.geometry("""
0 1
O
H 1 1.1
H 1 1.1 2 104
symmetry c1
""")

psi4.set_options(Dict("basis"         => "6-31g",
                      "scf_type"      => "pk",
                      "reference"     => "rhf",
                      "mp2_type"      => "conv",
                      "e_convergence" => 1e-8,
                      "d_convergence" => 1e-8))
```

Note that since we are using a spin orbital setup, we are free to use any Hartree-Fock reference we want. Here we choose RHF. For convenience, we let Psi4 take care of the Hartree-Fock procedure, and return the wavefunction object.

```julia
# Get the SCF wavefunction & energies
scf_e, scf_wfn = psi4.energy("scf", return_wfn=true)
```

    (-75.95252904632221, PyObject <psi4.core.RHF object at 0x14d493110>)

Load in information about the basis set and orbitals using MintsHelper and the wavefunction:

```julia
mints = psi4.core.MintsHelper(scf_wfn.basisset())
nbf = mints.nbf()           # number of basis functions
nso = 2nbf                  # number of spin orbitals
nalpha = scf_wfn.nalpha()   # number of alpha electrons
nbeta = scf_wfn.nbeta()     # number of beta electrons
nocc = nalpha + nbeta       # number of occupied orbitals
nvirt = 2nbf - nocc         # number of virtual orbitals
```

    16

Spin-block our MO coefficients and two-electron integrals, just like in the spin orbital MP2 code:

```julia
Ca = np.asarray(scf_wfn.Ca())
Cb = np.asarray(scf_wfn.Cb())
C = [Ca zero(Ca); zero(Cb) Cb]; # direct sum

# Result: | Ca  0 |
#         | 0   Cb|
```

```julia
# Get the two electron integrals using MintsHelper
I = np.asarray(mints.ao_eri())

"""
Function that spin blocks two-electron integrals
Using `np.kron`, we project I into the space of the 2x2 identity, tranpose the result
and project into the space of the 2x2 identity again. This doubles the size of each axis.
The result is our two electron integral tensor in the spin orbital form.
"""
function spin_block_tei(I)
    identity = [ 1.0 0.0; 0.0 1.0]
    I = np.kron(identity, I)
    np.kron(identity, permutedims(I, reverse(1:4)))
end

# Spin-block the two electron integral array
I_spinblock = spin_block_tei(I);
```

Convert two-electron integrals to antisymmetrized physicist's notation:

```julia
# Converts chemist's notation to physicist's notation, and antisymmetrize
# (pq|rs) ↦ ⟨pr|qs⟩
# Physicist's notation
tmp = permutedims(I_spinblock, (1, 3, 2, 4))
# Antisymmetrize:
# ⟨pr||qs⟩ = ⟨pr|qs⟩ - ⟨pr|sq⟩
gao = tmp - permutedims(tmp, (1, 2, 4, 3));
```

Obtain the orbital energies, append them, and sort the columns of our MO coefficient matrix according to the increasing order of orbital energies.

```julia
# Get orbital energies
eps_a = np.asarray(scf_wfn.epsilon_a())
eps_b = np.asarray(scf_wfn.epsilon_b())
eps = vcat(eps_a, eps_b)

# Before sorting the orbital energies, we can use their current arrangement to sort the columns
# of C. Currently, each element i of eps corresponds to the column i of C, but we want both
# eps and columns of C to be in increasing order of orbital energies

# Sort the columns of C according to the order of increasing orbital energies
C = C[:, sortperm(eps)]

# Sort orbital energies in increasing order
sort!(eps);
```

Finally, we transform our two-electron integrals to the MO basis. Here, we denote the integrals as `gmo` to differentiate from the chemist's notation integrals `I_mo`.

```julia
# Transform gao, which is the spin-blocked 4d array of physicist's notation,
# antisymmetric two-electron integrals, into the MO basis using MO coefficients
gmo = @tensor begin
   gmo[P,Q,R,S] := gao[p,Q,R,S] * C[p,P]
   gmo[p,Q,R,S] := gmo[p,q,R,S] * C[q,Q]
   gmo[p,q,R,S] := gmo[p,q,r,S] * C[r,R]
   gmo[p,q,r,S] := gmo[p,q,r,s] * C[s,S]
end
nothing
```

Construct the 4-dimensional array of orbital energy denominators:

```julia
# Define slices, create 4 dimensional orbital energy denominator tensor
n = [CartesianIndex()]
o = [p ≤ nocc for p in 1:nso]
v = [p > nocc for p in 1:nso]
e_denom = @. inv(-eps[v, n, n, n] - eps[n, v, n, n] + eps[n, n, o, n] + eps[n, n, n, o]);
```

We now have everything we need to construct our $t$-amplitudes and iteratively solve for our CEPA0 and CCD energy. To build the $t$-amplitudes, we first construct an empty 4-dimensional array to store them.

```julia
# Create space to store t amplitudes
t_amp = zeros(nvirt, nvirt, nocc, nocc);
```

# Implementation: CEPA0
First we will program CEPA0. Recall the expression for the $t$-amplitudes:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} \right)
\end{equation}

These terms translate naturally into code using Julia's `@tensor` function. To access only the occupied and virtual indices of `gmo` we use our slices defined above. The permutation operator terms can be easily obtained by transposing the original result accordingly. To construct each iteration's $t$-amplitude:

~~~julia
mp2    = @view gmo[v, v, o, o]
@tensor cepa1[ a,b,i,j] := 0.5(gmo[v,v,v,v])[a,b,c,d] * t_amp[c,d,i,j]
@tensor cepa2[ a,b,i,j] := 0.5(gmo[o,o,o,o])[k,l,i,j] * t_amp[a,b,k,l]
@tensor cepa3a[a,b,i,j] :=    (gmo[v,o,o,v])[a,k,i,c] * t_amp[b,c,j,k]
cepa3b = -permutedims(cepa3a, (2, 1, 3, 4)) # a <-> b
cepa3c = -permutedims(cepa3a, (1, 2, 4, 3)) #          i <-> j
cepa3d =  permutedims(cepa3a, (2, 1, 4, 3)) # a <-> b, i <-> j
cepa3  =  cepa3a + cepa3b + cepa3c + cepa3d

t_amp_new = @. e_denom * (mp2 + cepa1 + cepa2 + cepa3)
~~~

To evaluate the energy, $E_{\mathrm{CEPA0}} = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}$,

~~~julia
E_CEPA0 = 1/4 * @tensor scalar((gmo[o,o,v,v])[i,j,a,b] * t_amp_new[a,b,i,j])
~~~

Putting it all together, we initialize the energy, set the max iterations, and iterate the energy until it converges to our convergence criterion:

```julia
# Initialize energy
E_CEPA0 = let E_CEPA0 = 0.0, gmo=gmo, o=o,v=v, e_denom = e_denom, t_amp = t_amp

   MAXITER = 50

   for cc_iter in 1:MAXITER
       E_old = E_CEPA0

       # Collect terms
       mp2      = @view gmo[v,v,o,o]
       @tensor cepa1[ a,b,i,j] := 0.5(gmo[v,v,v,v])[a,b,c,d] * t_amp[c,d,i,j]
       @tensor cepa2[ a,b,i,j] := 0.5(gmo[o,o,o,o])[k,l,i,j] * t_amp[a,b,k,l]
       @tensor cepa3a[a,b,i,j] :=    (gmo[v,o,o,v])[a,k,i,c] * t_amp[b,c,j,k]
       cepa3b = -permutedims(cepa3a, (2, 1, 3, 4))
       cepa3c = -permutedims(cepa3a, (1, 2, 4, 3))
       cepa3d =  permutedims(cepa3a, (2, 1, 4, 3))
       cepa3  =  cepa3a + cepa3b + cepa3c + cepa3d

       # Update t amplitude
       t_amp_new = @. e_denom * (mp2 + cepa1 + cepa2 + cepa3)

       # Evaluate Energy
       E_CEPA0 = 1/4 * @tensor scalar((gmo[o,o,v,v])[i,j,a,b] * t_amp_new[a,b,i,j])
       t_amp = t_amp_new
       dE = E_CEPA0 - E_old
       printfmt("CEPA0 Iteration {1:3d}: Energy = {2:4.12f} dE = {3:1.5e}\n", cc_iter, E_CEPA0, dE)

       if abs(dE) < 1.e-8
           @info "CEPA0 Iterations have converged!"
           break
       end

       if cc_iter == MAXITER
           psi4.core.clean()
           error("Maximum number of iterations exceeded.")
       end
   end
   E_CEPA0
end

printfmt("\nCEPA0 Correlation Energy: {:5.15f}\n", E_CEPA0)
printfmt("CEPA0 Total Energy: {:5.15f}\n", E_CEPA0 + scf_e)
```

    CEPA0 Iteration   1: Energy = -0.142119840107 dE = -1.42120e-01
    CEPA0 Iteration   2: Energy = -0.142244391124 dE = -1.24551e-04
    CEPA0 Iteration   3: Energy = -0.146403555808 dE = -4.15916e-03
    CEPA0 Iteration   4: Energy = -0.147737944685 dE = -1.33439e-03
    CEPA0 Iteration   5: Energy = -0.148357998476 dE = -6.20054e-04
    CEPA0 Iteration   6: Energy = -0.148640319256 dE = -2.82321e-04
    CEPA0 Iteration   7: Energy = -0.148774677462 dE = -1.34358e-04
    CEPA0 Iteration   8: Energy = -0.148840007175 dE = -6.53297e-05
    CEPA0 Iteration   9: Energy = -0.148872387868 dE = -3.23807e-05
    CEPA0 Iteration  10: Energy = -0.148888687346 dE = -1.62995e-05
    CEPA0 Iteration  11: Energy = -0.148897003346 dE = -8.31600e-06
    CEPA0 Iteration  12: Energy = -0.148901297751 dE = -4.29440e-06
    CEPA0 Iteration  13: Energy = -0.148903540226 dE = -2.24248e-06
    CEPA0 Iteration  14: Energy = -0.148904723489 dE = -1.18326e-06
    CEPA0 Iteration  15: Energy = -0.148905354026 dE = -6.30537e-07
    CEPA0 Iteration  16: Energy = -0.148905693171 dE = -3.39145e-07
    CEPA0 Iteration  17: Energy = -0.148905877194 dE = -1.84023e-07
    CEPA0 Iteration  18: Energy = -0.148905977873 dE = -1.00678e-07
    CEPA0 Iteration  19: Energy = -0.148906033377 dE = -5.55039e-08
    CEPA0 Iteration  20: Energy = -0.148906064192 dE = -3.08158e-08
    CEPA0 Iteration  21: Energy = -0.148906081412 dE = -1.72194e-08
    CEPA0 Iteration  22: Energy = -0.148906091090 dE = -9.67815e-09

    CEPA0 Correlation Energy: -0.148906091090053
    CEPA0 Total Energy: -76.101435137412267

    ┌ Info: CEPA0 Iterations have converged!
    └ @ Main In[13]:29

Since `t_amp` is initialized to zero, the very first iteration should be the MP2 correlation energy. We can check the final CEPA0 energy with Psi4. The method is called `lccd`, or linear CCD, since CEPA0 omits the terms with two cluster amplitudes.

```julia
psi4.compare_values(psi4.energy("lccd"), E_CEPA0 + scf_e, 6, "CEPA0 Energy")
```

    	CEPA0 Energy......................................................PASSED

    true

# Implementation: CCD

To code CCD, we only have to add in the last four terms in our expression for the $t$-amplitudes:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} - \underline{\tfrac{1}{2}\hat{P}_{(a \space / \space b)} \bar{g}_{kl}^{cd} t_{ac}^{ij} t_{bd}^{kl} - \tfrac{1}{2}  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ab}^{ik} t_{cd}^{jl} + \tfrac{1}{4} \bar{g}_{kl}^{cd} t_{cd}^{ij} t_{ab}^{kl} +  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ac}^{ik} t_{bd}^{jl}} \right)
\end{equation}

which we readily translate into `@tensor`'s:

~~~julia
@tensor ccd1a[a,b,i,j] := (gmo[o,o,v,v])[k,l,c,d] * t_amp[a,c,i,j] * t_amp[b,d,k,l]
ccd1b  = -permutedims(ccd1a, (2, 1, 3, 4))
ccd1   = -0.5(ccd1a + ccd1b)

@tensor ccd2a[a,b,i,j] := (gmo[o,o,v,v])[k,l,c,d] * t_amp[a,b,i,k] * t_amp[c,d,j,l]
ccd2b  = -permutedims(ccd2a, (1, 2, 4, 3))
ccd2   = -0.5(ccd2a + ccd2b)

@tensor ccd3[a,b,i,j] := 1/4 * (gmo[o,o,v,v])[k,l,c,d] * t_amp[c,d,i,j] * t_amp[a,b,k,l]

@tensor ccd4a[a,b,i,j] := (gmo[o,o,v,v])[ k,l,c,d] * t_amp[a,c,i,k] * t_amp[b,d,j,l]
ccd4b  = -permutedims(ccd4a, (1, 2, 4, 3))
ccd4   = (ccd4a + ccd4b)
~~~

and the energy expression is identical to CEPA0:
\begin{equation}
E_{CCD } = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}
\end{equation}

Adding the above terms to our CEPA0 code will compute the CCD correlation energy (may take a minute or two to run):

```julia
# Initialize energy
E_CCD = let E_CCD = 0.0, o=o,v=v, e_denom=e_denom, t_amp=t_amp

   MAXITER = 50

   # Create space to store t amplitudes
   t_amp = zeros(nvirt, nvirt, nocc, nocc)
   for cc_iter in 1:MAXITER
       E_old = E_CCD

       # Collect terms
       mp2      = @view gmo[v,v,o,o]
       @tensor cepa1[ a,b,i,j] := 0.5(gmo[v,v,v,v])[a,b,c,d] * t_amp[c,d,i,j]
       @tensor cepa2[ a,b,i,j] := 0.5(gmo[o,o,o,o])[k,l,i,j] * t_amp[a,b,k,l]
       @tensor cepa3a[a,b,i,j] :=    (gmo[v,o,o,v])[a,k,i,c] * t_amp[b,c,j,k]
       cepa3b = -permutedims(cepa3a, (2, 1, 3, 4))
       cepa3c = -permutedims(cepa3a, (1, 2, 4, 3))
       cepa3d =  permutedims(cepa3a, (2, 1, 4, 3))
       cepa3  =  cepa3a + cepa3b + cepa3c + cepa3d

       @tensor ccd1a_ref[a,b,i,j] := (gmo[o,o,v,v])[k,l,c,d] * t_amp[a,c,i,j] * t_amp[b,d,k,l]
       @tensor ccd1a_tmp[c,b]     := (gmo[o,o,v,v])[k,l,c,d] * t_amp[b,d,k,l]
       @tensor ccd1a[a,b,i,j]     := ccd1a_tmp[c,b]    * t_amp[a,c,i,j]
       println(isapprox(ccd1a_ref, ccd1a))

       ccd1b  = -permutedims(ccd1a, (2, 1, 3, 4))
       ccd1   = -0.5(ccd1a + ccd1b)

       @tensor ccd2a_ref[a,b,i,j] := (gmo[o,o,v,v])[k,l,c,d] * t_amp[a,b,i,k] * t_amp[c,d,j,l]
       @tensor ccd2a_tmp[j,k]     := (gmo[o,o,v,v])[k,l,c,d] * t_amp[c,d,j,l]
       @tensor ccd2a[a,b,i,j]     := ccd2a_tmp[j,k]    * t_amp[a,b,i,k]
       println(isapprox(ccd2a_ref, ccd2a))

       ccd2b  = -permutedims(ccd2a, (1, 2, 4, 3))
       ccd2   = -0.5(ccd2a + ccd2b)

       @tensor ccd3_ref[a,b,i,j] := 1/4 * (gmo[o,o,v,v])[k,l,c,d] * t_amp[c,d,i,j] * t_amp[a,b,k,l]
       @tensor ccd3_tmp[k,l,i,j] :=       (gmo[o,o,v,v])[k,l,c,d] * t_amp[c,d,i,j]
       @tensor ccd3[a,b,i,j]     := 1/4 * ccd3_tmp[k,l,i,j] * t_amp[a,b,k,l]
       println(isapprox(ccd3_ref, ccd3))

       @tensor ccd4a_ref[a,b,i,j] := (gmo[o,o,v,v])[ k,l,c,d] * t_amp[a,c,i,k] * t_amp[b,d,j,l]
       @tensor ccd4a_tmp[l,a,i,d] := (gmo[o,o,v,v])[ k,l,c,d] * t_amp[a,c,i,k]
       @tensor ccd4a[a,b,i,j]     := ccd4a_tmp[l,a,i,d] * t_amp[b,d,j,l]
       println(isapprox(ccd4a_ref, ccd4a))

       ccd4b  = -permutedims(ccd4a, (1, 2, 4, 3))
       ccd4   = ccd4a + ccd4b

       # Update Amplitude
       t_amp_new = @. e_denom * (mp2 + cepa1 + cepa2 + cepa3 + ccd1 + ccd2 + ccd3 + ccd4)

       # Evaluate Energy
       E_CCD = 1/4 * @tensor scalar((gmo[o,o,v,v])[i,j,a,b] * t_amp_new[a,b,i,j])
       t_amp = t_amp_new
       dE = E_CCD - E_old
       printfmt("CCD Iteration {1:3d}: Energy = {2:4.12f} dE = {3:1.5e}\n", cc_iter, E_CCD, dE)

       if abs(dE) < 1.e-8
           @info "CCD Iterations have converged!"
           break
       end

       if cc_iter == MAXITER
           psi4.core.clean()
           error("Maximum number of iterations exceeded.")
       end
   end
   E_CCD
end

printfmt("\nCCD Correlation Energy:    {:15.12f}\n", E_CCD)
printfmt("CCD Total Energy:         {:15.12f}\n", E_CCD + scf_e)
```

    true
    true
    true
    true
    CCD Iteration   1: Energy = -0.142119840107 dE = -1.42120e-01
    true
    true
    true
    true
    CCD Iteration   2: Energy = -0.142920457961 dE = -8.00618e-04
    true
    true
    true
    true
    CCD Iteration   3: Energy = -0.146174466311 dE = -3.25401e-03
    true
    true
    true
    true
    CCD Iteration   4: Energy = -0.147222337053 dE = -1.04787e-03
    true
    true
    true
    true
    CCD Iteration   5: Energy = -0.147660207822 dE = -4.37871e-04
    true
    true
    true
    true
    CCD Iteration   6: Energy = -0.147845022862 dE = -1.84815e-04
    true
    true
    true
    true
    CCD Iteration   7: Energy = -0.147926013534 dE = -8.09907e-05
    true
    true
    true
    true
    CCD Iteration   8: Energy = -0.147962311493 dE = -3.62980e-05
    true
    true
    true
    true
    CCD Iteration   9: Energy = -0.147978892019 dE = -1.65805e-05
    true
    true
    true
    true
    CCD Iteration  10: Energy = -0.147986584027 dE = -7.69201e-06
    true
    true
    true
    true
    CCD Iteration  11: Energy = -0.147990200750 dE = -3.61672e-06
    true
    true
    true
    true
    CCD Iteration  12: Energy = -0.147991921640 dE = -1.72089e-06
    true
    true
    true
    true
    CCD Iteration  13: Energy = -0.147992749316 dE = -8.27677e-07
    true
    true
    true
    true
    CCD Iteration  14: Energy = -0.147993151327 dE = -4.02011e-07
    true
    true
    true
    true
    CCD Iteration  15: Energy = -0.147993348360 dE = -1.97033e-07
    true
    true
    true
    true
    CCD Iteration  16: Energy = -0.147993445735 dE = -9.73748e-08
    true
    true
    true
    true
    CCD Iteration  17: Energy = -0.147993494226 dE = -4.84909e-08
    true
    true
    true
    true
    CCD Iteration  18: Energy = -0.147993518542 dE = -2.43158e-08
    true
    true
    true
    true
    CCD Iteration  19: Energy = -0.147993530812 dE = -1.22701e-08
    true
    true
    true
    true
    CCD Iteration  20: Energy = -0.147993537039 dE = -6.22685e-09

    CCD Correlation Energy:    -0.147993537039
    CCD Total Energy:         -76.100522583361

    ┌ Info: CCD Iterations have converged!
    └ @ Main In[15]:60

Unfortunately, Psi4 does not have a CCD code to compare this to. However, Psi4 does have Bruekner CCD, an orbital-optimized variant of CCD. We can qualitatively compare our energies to this energy. The Bruekner-CCD energy should be a little lower than our CCD energy due to the orbital optimization procedure.

```julia
psi4_bccd = psi4.energy("bccd", ref_wfn = scf_wfn)
printfmt("\nPsi4 BCCD Correlation Energy:    {:15.12f}\n", psi4_bccd - scf_e)
printfmt("Psi4 BCCD Total Energy:         {:15.12f}\n", psi4_bccd)
```

    Psi4 BCCD Correlation Energy:    -0.149207663736
    Psi4 BCCD Total Energy:         -76.101736710059

## References

1. Modern review of coupled-cluster theory, included diagrammatic derivations of the CCD equations:
	> [[Bartlett and Musial:2007](https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.79.291)] Rodney J. Bartlett and Monika Musial,  "Coupled-cluster theory in quantum chemistry" *Rev. Mod. Phys.* **79**, 291 (2007)

2. Background on CEPA:
    >Kutzelnigg, Werner 1977 *Methods of Electronic Structure Theory* ed. H. F. Schaefer III (Plenum, New York), p 129

3. More CEPA:
    > [Koch and Kutzelnigg:1981](https://link.springer.com/article/10.1007/BF00553396) S. Koch and W. Kutzelnigg, *Theor. Chim. Acta* **59**, 387 (1981).

4. Original CCD Paper:
    > [Čížek:1966](http://aip.scitation.org/doi/abs/10.1063/1.1727484) Jiří Čížek, "On the Correlation Problem in Atomic and Molecular Systems. Calculation of Wavefunction Components in Ursell‐Type Expansion Using Quantum‐Field Theoretical Methods" *J. Chem. Phys* **45**, 4256 (1966)

5. Useful notes on diagrams applied to post-HF methods:
    > A. V. Copan, "Diagram notation" accessed with https://github.com/CCQC/chem-8950/tree/master/2017


```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Philosopher', sans-serif;
    font-weight: 400;
    font-size: 2.2em;
    line-height: 100%;
    color: rgb(0, 80, 120);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Philosopher', serif;
    font-weight: 400;
    font-size: 1.9em;
    line-height: 100%;
    color: rgb(245,179,64);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Philosopher', serif;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: italic;
    color: rgb(94,127,192);
}

.text_cell_render h4 {
    font-family: 'Philosopher', serif;
}

.text_cell_render h5 {
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 16pt;
    color: grey;
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h6 {
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 10pt;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 100%;
}

</style>

```python
from sympy import init_printing, Matrix, symbols
init_printing()
```

# Independence
# Spanning
# Basis
# Dimension

## Independence

Sticking with the theme of linearly independent vectors, we note that vectors are indeed linearly independent under the following conditions:

1. If no combination of the vectors results in the zero vector (other than the trivial cases of a scalar multiple of $0$ of each of the vectors)
2. For the respective dimensional space, they do not lie on a line, plane, or hyperplane through the origin

Let's consider the example of the matrix (of coefficients), named `A` below.

```python
A= Matrix([[1, 2, 4], [3, 1, 4]])
A
```

### How many vectors in the nullspace?

This is a matrix with a rank of $2$ ($2$ pivots) and $3$ unknowns and $2$ rows.  Thus, $\text{rank}\left(A\right)=m=2$ (a full row rank).  We are left with $n-r$ free variables, i.e. $3 - 2 = 1$.  Importantly, this means that we will have one vector in the nullspace.

```python
A.rref()  # Reduced row-echelon form
```

```python
A.nullspace()  # Null -sapce vector
```

We note the two equations in three unknowns (set to the zero vector) in (1) below.

$$\begin{align}&{x_1}+2{x_2}+4{x_3}=0\\&3{x_1}+{x_2}+4{x_3}=0\end{align}\tag{1}$$

After Gauss-Jordan elimination we have (2).

$$\begin{align}&{x_1}+\frac{4}{5}{x_3}=0\\&{x_2}+\frac{8}{5}{x_3}=0\end{align}\tag{2}$$

For the null space, we set $x_3=1$.  From this follows that ${x_2}=-\frac{8}{5}$ and then ${x_1}=-\frac{4}{5}$ confirming the results of the `.nullspace()` method above.

### Another way to state independence

Consider the columns of any matrix $A$ as vectors $\underline{v}_1,\underline{v}_2,\ldots,\underline{v}_n$.  If $\text{rank}\left(A\right)=n$ (the number of columns) then the nullspace only contains the zero vector and the column vectors are linearly independent.

## Spanning

We have introduced the concept of _spanning_ a (sub)space.  If we have a set of linearly independent vectors such that all their linear combinations (including the zero vectors) _fill_ a (sub)space, we state that is _spans_ that (sub)space.  We are particularly interested in a set of (column) vectors (in a matrix) that are linearly independent and span a (sub)space, because this leads us to the next topic of _basis vectors_.

## Basis

Basis vectors (in a space $W$) are vectors with the properties, (a) they are linearly independent and (b) they span the space (linear combinations of them fill the space).

Up until now we looked at columns in a matrix $A$.  It is more common in textbooks to look at a space first and ask about basis vectors, spanning vectors, dimension, and so on.

So let's look at $\mathbb{R}^3$.  The obvious set of basis vectors are shown in (3).

$$\hat{i},\quad\hat{j},\quad\hat{k}\tag{3}$$

What about the vectors in (4).  Are they linearly independent and do they span $\mathbb{R}^3$?

$$\begin{bmatrix}1\\1\\2\end{bmatrix},\quad\begin{bmatrix}2\\2\\5\end{bmatrix}\tag{4}$$

```python
A = Matrix([[1, 2], [1, 2], [2, 5]])
A
```

```python
A.rref()
```

Here we have $\text{rank}\left(A\right)=2$, $n=2$, and $n-r=0$, i.e. $0$ vectors in the nullspace.  They cannot possibly be a basis for $\mathbb{R}^3$ and do not span $\mathbb{R}^3$.

```python
A.nullspace()
```

All their linear combinations will only fill a plane through the origin.  Their (trivial) zero combination does result in the zero vector, though, so they do fill a subspace of $\mathbb{R}^3$.

If we added a column vector that is a linear combination of the original two columns, it will also fall in the plane.  There will be a vector in the nullspace other than the zero vector, though.

```python
A = Matrix([[1, 2, 3], [1, 2, 3], [2, 5, 7]])  # Adding a linear combination of columns one and two
A.nullspace()  # Calculation the nullspace
```

```python
A.rref()
```

We see that we have a row with zero values.  This means that we have a column without a pivot and thus a free variable.

Let's add another, just for fun.  Here we duplicate the first row.

```python
A = Matrix([[1, 2, 3], [1, 2, 3], [2, 5, 8]])
A
```

```python
A.rref()
```

Again, a column without a pivot and sure enough, we'll find a vector (other than the zero vector) in the nullspace.

```python
A.nullspace()
```

### The special case of a square matrix

If we have a square matrix, we need only look at it's determinant: _Is it invertible_?  (More about matrix inverses later).

```python
A.det() # .det() calculates the determinant
```

The determinant is $0$ (as expected) and we have a vector in the nullspace.

## Dimension

Given a (sub)space, every basis for that (sub)space has the same number of vectors (there are usually more than one basis for every (sub)space.  This called the _dimension_ of the (sub)space.

## Example

* Consider the column space

```python
A = Matrix([[1, 2, 3, 1], [1, 1, 2, 1], [1, 2, 3, 1]])
A
```

There are $n=4$ unknowns, $m=3$ unknowns.  We note that column $1$ = column $4$.  We note that with $4$ unknowns we are dealing with $\mathbb{R}^4$.  In essence, there are at most three independent columns, thus the matrix cannot be a basis for $\mathbb{R}^4$.

```python
A.nullspace()
```

```python
A.rref()
```

As we can see here, (5), columns three and four have free variables, i.e. no pivots.

$$\begin{align}&{x}_{1}+0{x}_{2}+{x}_{3}+{x}_{4}=0\\&0{x}_{1}+1{x}_{2}+{x}_{3}+{0}_{4}=0\\&{x}_{4}={c}_{2}\\&{x}_{3}={c}_{1}\\ \therefore \quad &{x}_{2}=-{c}_{1}\\ \therefore \quad &{x}_{1}=-{c}_{1}-{c}_{2}\end{align}\tag{5}$$

We express this more fully in (6).

$$  \begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ { x }_{ 3 } \\ { x }_{ 4 } \end{bmatrix}=\begin{bmatrix} -{ c }_{ 1 }-{ c }_{ 2 } \\ -{ c }_{ 1 } \\ { c }_{ 1 } \\ { c }_{ 2 } \end{bmatrix}=\begin{bmatrix} -{ c }_{ 1 } \\ -{ c }_{ 1 } \\ { c }_{ 1 } \\ 0 \end{bmatrix}+\begin{bmatrix} -{ c }_{ 2 } \\ 0 \\ 0 \\ { c }_{ 2 } \end{bmatrix}={ c }_{ 1 }\begin{bmatrix} -1 \\ -1 \\ 1 \\ 0 \end{bmatrix}+{ c }_{ 2 }\begin{bmatrix} -1 \\ 0 \\ 0 \\ 1 \end{bmatrix} \tag {6} $$

The rank of matrix $A$ is $2$ (it is the number of pivot columns).  This matrix space thus have two basis vectors (column vectors $1$ and $2$) and we say the dimension of this space is $2$.  Remember, a matrix has a rank, which is the dimension of a column space (the column space representing the space 'produced' by the column vectors).  We talk about the rank of a matrix, $\text{rank}\left(A\right)$ and the column space of a matrix, $\text{C}\left(A\right)$.

In summary, we have two basis above (they span a space).  Any two vectors that are not linearly dependent will also span this space, they can't help but to, $\text{dim}\text{C}\left(A\right)=r$.  The nullspace will have $n-r$ vectors (the dimension of the nullspace equal the number of free variables).

```python

```

Mathematics/Linear Algebra/0.0 MIT-18.06 - Jupyter/Lecture_9_Independence_Spanning_Basis_Dimension.ipynb


```python
from IPython.core.display import HTML
css_file = './custom.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

###### Content provided under a Creative Commons Attribution license, CC-BY 4.0; code under MIT License. (c)2014 [David I. Ketcheson](http://davidketcheson.info)

##### version 0.1 - May 2014

# Hyperbolic Conservation Laws

\begin{equation*}
\newcommand{Dx}{\Delta x}
\newcommand{Dt}{\Delta t}
\newcommand{imh}{{i-1/2}}
\newcommand{iph}{{i+1/2}}
\end{equation*}
Many models of wave phenomena are governed by *hyperbolic conservation laws*.  In this short course, we will learn about hyperbolic conservation laws and their numerical solution.

## Conservation of mass

Imagine a fluid flowing in a narrow tube.  We'll use $q$ to indicate the density of the fluid and $u$ to indicate its velocity.  Both of these are functions of space and time: $q = q(x,t)$; $u=u(x,t)$.  The total mass in the section of tube $[x_1,x_2]$ is

\begin{equation}
\int_{x_1}^{x_2} q(x,t) dx.
\end{equation}

This total mass can change in time due to fluid flowing in or out of this section of the tube.  We call the rate of flow the *flux*, and represent it with the function $f(q)$.  Thus the net rate of flow of mass into (or out of) the interval $[x_1,x_2]$ at time $t$ is

$$f(q(x_1,t)) - f(q(x_2,t)).$$

We just said that this rate of flow must equal the time rate of change of total mass; i.e.

$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = f(q(x_1,t)) - f(q(x_2,t)).$$

Now since $\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx = f(q(x_2,t)) - f(q(x_1,t))$, we can rewrite this as

$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = -\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx.$$

Under certain smoothness assumptions on $q$, we can move the time derivative inside the integral.  We'll also put everything on the left side, to obtain

$$\int_{x_1}^{x_2} \left(\frac{\partial}{\partial t}q(x,t) + \frac{\partial}{\partial x} f(q)\right) dx = 0.$$

Since this integral is zero for *any* choice of $x_1,x_2$, it must be that the integrand (the expression in parentheses) is actually zero *everywhere*!  Therefore we can write the **differential conservation law**

$$q_t + f_x = 0.$$

Here and throughout the course, we use subscripts to denote partial derivatives.
This equation expresses the fact that the total mass is conserved -- since locally the mass can change only due to a net inflow or outflow.

## Advection

In order to solve the conservation law above, we need an expression for the flux, $f$.  The rate of flow is just mass times velocity: $f=u q$.  Thus we obtain the **continuity equation**

$$q_t + (uq)_x = 0.$$

In general, we need another equation to determine the velocity $u(x,t)$.  In [Lesson 4](Lesson_04_Fluid_dynamics.ipynb) we'll look at the full equations of fluid dynamics, but for now let's consider the simplest case, in which all of the fluid flows at a single, constant velocity $u(x,t)=a$.  Then the continuity equation becomes the **advection equation**

$$q_t + a q_x = 0.$$

This equation has a very simple solution.  If we are given the density $q(x,0)=q_0(x)$ at time zero, then the solution is just

$$q(x,t) = q_0(x-at).$$

Let's plot the solution of the advection equation on the interval $[0,1]$ for the initial condition
$$q_0(x) = e^{-2(x-1/2)^2}.$$

First, let's import all the modules we'll need.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import animation
from clawpack.visclaw.JSAnimation import IPython_display
```

Next, we'll set up a grid and the initial condition:

```python
x = np.linspace(0,1,1000)  # Spatial grid
t = np.linspace(0,1)       # Temporal grid
a = 1.0                    # Advection speed

def q_0(x):                # Initial condition
    return np.exp(-200.*(x-0.2)**2)
```

Finally, let's make an animation of the solution.  It will take a few moments to run this code.  For now, you don't need to worry about understanding all of the plotting code below.  Just play with the animation until you have a feel for how the solution behaves.

```python
fig = plt.figure(figsize=(8,4))      # Create an empty figure
ax  = plt.axes()
line, = ax.plot([], [],linewidth=2)  # Create an empty line plot
plt.axis((0,1,-0.1,1.1))             # Set the bounds of the plot

def plot_q(t):
    line.set_data(x,q_0(x-a*t))  # Replace the line plot with the solution at time t

animation.FuncAnimation(fig, plot_q, frames=t)  # Animate the solution
```

    <input id="_anim_slider d475256bf6f6494" type="range" style="width:350px" name="points" min="0" max="1" step="1" value="0" onchange="anim d475256bf6f6494.set_frame(parseInt(this.value));"></input>

    <button onclick="anim d475256bf6f6494.slower()">–</button>
    <button onclick="anim d475256bf6f6494.first_frame()"></button>
    <button onclick="anim d475256bf6f6494.previous_frame()"></button>
    <button onclick="anim d475256bf6f6494.reverse_animation()"></button>
    <button onclick="anim d475256bf6f6494.pause_animation()"></button>
    <button onclick="anim d475256bf6f6494.play_animation()"></button>
    <button onclick="anim d475256bf6f6494.next_frame()"></button>
    <button onclick="anim d475256bf6f6494.last_frame()"></button>
    <button onclick="anim d475256bf6f6494.faster()">+</button>
  <form action="#n" name="_anim_loop_select d475256bf6f6494" class="anim_control">

    <input id="_frame_no d475256bf6f6494" type="textbox" size="1" onchange="anim d475256bf6f6494.set_frame(parseInt(this.value));" onpaste="this.onchange();" oninput="this.onchange();"></input>

    <input type="radio" name="state" value="once" > Once </input>
    <input type="radio" name="state" value="loop" checked> Loop </input>
    <input type="radio" name="state" value="reflect" > Reflect </input>
  </form>

As you can see, the initial pulse just moves to the right at speed $a$ as time advances.  This isn't very interesting, but it captures the most important feature of hyperbolic equations: waves travel at finite speed.

## Characteristics

Notice that the solution value is constant along the line $x-at=x_0$, in the $x-t$ plane, for each value of $x_0$.  These lines are called **characteristics**; they are the trajectories along which solution information is transmitted.  The value $a$ is referred to as the **characteristic velocity**.  The code below plots some of these characteristics.

When we learn about more complicated conservation laws, we'll see that information still travels along characteristics, but those characteristics aren't necessarily straight lines.

```python
fig = plt.figure(figsize=(8,4))
ax  = plt.axes()

for x_0 in np.linspace(0,1,10):
    ax.plot(x,(x-x_0)/a,'-k')
plt.ylim(0,1)
```

## A finite volume method for advection

We can easily solve the advection equation exactly.  But the advection equation is a prototype for more complicated conservation laws that we will only be able to solve approximately by using numerical methods.  In order to better understand these methods, we will discuss them first in the context of the advection equation.

For simplicity, we'll suppose that we wish to solve the advection equation on the interval $[0,1]$.  We introduce a set of equally spaced *grid cells* of width $\Dx$, and write $x_i$ to mean the center of cell $i$.  Thus the first cell is the interval $[0,\Dx]$ and $x_1=\Dx/2$.  We will also write $x_\imh$ or $x_\iph$ to denote the left or right boundary of cell $i$, respectively.

We write $Q_i$ to denote the *average* value of the solution over cell $i$:

$$Q_i = \frac{1}{\Dx} \int_{x_\imh}^{x_\iph} q \ dx.$$

The simplest finite volume method is obtained by supposing that the solution is actually *equal* to $Q_i$ over all of cell $i$.

Suppose $a>0$.  Then the flux into cell $i$ from the left is $a Q_{i-1}$ and the flux out of cell $i$ to the right is $a Q_i$.  Then our integral conservation law reads

$$Q_i'(t) = -\frac{a}{\Dx}\left(Q_i - Q_{i-1}\right).$$

Applying a forward difference in time we obtain the *upwind method*

$$Q^{n+1}_i = Q^n_i -\frac{a}{\Dx}\left(Q_i - Q_{i-1}\right).$$

We call this the upwind method because the solution behaves as if it were being blown by a wind to the right, and the method uses the value $Q_{i-1}$ from the upwind direction.

Here is a bit of Python code to solve the advection equation using the upwind method.

```python
a = 1.0     # advection speed

m = 50      # number of cells
dx = 1./m   # Size of 1 grid cell
x = np.arange(-dx/2, 1.+dx/2, dx)  # Cell centers, including ghost cells

t = 0.      # Initial time
T = 0.5     # Final time
dt = 0.8 * dx / a  # Time step

Q = np.exp(-200*(x-0.2)**2)    # Initial data
Qnew = np.empty(Q.shape)

while t < T:

    # Extrapolation at boundaries:
    Qnew[0] = Q[1]
    Qnew[-1] = Q[-2]

    for i in range(1,len(x)):
        Qnew[i] = Q[i] - a*dt/dx * (Q[i]-Q[i-1])

    Q = Qnew.copy()
    t = t + dt

plt.plot(x,Q,linewidth = 2)
plt.title('t = '+str(t));
```

Notice how we set up a grid that contains an extra cell at each end, outside of the problem domain $[0,1]$.  These are called **ghost cells** and are often useful in handling the solution at the grid boundaries.

  The technique we have used to set the ghost cell values above, by copying the last value inside the grid to the ghost cells, is known as **zero-order extrapolation**.  It is useful for allowing waves to pass out of the domain (so-called *non-reflecting* boundaries).  Note that we don't actually need the ghost cell at the right end for the upwind method, but for other methods we will.

The upwind method is simple, but it is not very accurate.  Notice how the computed solution becomes wider and shorter over time.  This behavior is referred to as *dissipation*.

### Exercise

Now do the following with the code above:

1. Set $m=1000$ or more and notice that it takes some time to compute the solution.  Rewrite the inner loop (over $i$) as a single line with no loop, using numpy slicing.  For large values of $m$, the code with slicing is much faster.
1. Notice that the last step of the simulation goes past time $T$.  Modify the code so that the last step is adjusted to exactly reach $T$.
2. Change the code so that animation of the solution versus time is plotted.  You will want to accumulate frames of the solution in a list and then use the same kind of code we used above to animate the exact solution.
3. Add some code to plot the exact solution.

*Extra credit*: change the left boundary condition so that there is a sinusoidal wave coming in from the left:
$$u(0,t) = \sin(20 \pi t).$$
What do you notice about the sinusoid as it moves into the domain?

```python

```

After making it through the exercise above, you should feel pretty comfortable with the basics of scientific programming in Python.

## The CFL condition

Take a look at the line of code that sets the time step:
```python
    dt = 0.8 * dx / a
```
You might be wondering where that formula came from.  Rearranging that equation, we have
$$a \frac{\Delta t}{\Delta x} = 0.8.$$
The quantity $\nu = a \frac{\Delta t}{\Delta x}$ is the distance the exact solution moves during each time step, in units of grid cells.  It is referred to as the *CFL number* or just the *Courant number* after the authors Courant, Friedrichs and Lewy who [established its importance](http://www.stat.uchicago.edu/~lekheng/courses/302/classics/courant-friedrichs-lewy.pdf).  Try the following values of $\nu$ in the code above, and compare the results with those you obtained already using $\nu=0.8$.
1. $\nu = 1.0$
2. $\nu = 1.5$
3. $\nu = 0.1$

Finally, try setting $a$ to a negative value.  What happens?

The results you have observed can be explained as follows.  Over a time step of size $\Dt$, the solution moves by an amount $a \Dt$.  So $q(x_i,t_n)$ should be given exactly by $q(x_i-a\Dt,t_{n-1})$.  This is referred to as the *domain of dependence* of the solution.

The upwind method uses the values $Q_i^{n-1}$ and $Q_{i-1}^{n-1}$ to compute $Q_i^n$.  The points $(x_{i-1},t_n)$ and $(x_i,t_n)$ (as well as the locations of solution values they depend, and the ones those depend on, and so forth) are the *numerical domain of dependence*.

For this to work, the true domain of dependence $x_i-a\Dt$ must lie within the numerical domain of dependence, as $\Dt,\Dx \to 0$.  This is known as the [CFL condition](http://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition).

For the upwind method, that means that we must have
$$x_i - \Dx \le x_i - a \Dt \le x_i$$
or in other words
$$0 \le a \frac{\Dt}{\Dx} \le 1.$$

If the CFL condition is violated, the information that is used by the numerical method doesn't include the true information that influences the exact solution, so the numerical solution cannot be convergent.

## The Lax-Friedrichs method

The upwind method gets its name from the fact that it uses the value $U_{i-1}$ and not $U_{i+1}$.  Assuming $a>0$, the correct solution value $U_i^{n+1}$ should come from a point to the left of $x_i$ at time $t_n$ (i.e., the wind blows to the right, so $x_{i-1}$ is *upwind* of $x_i$).

This bias is fine for the advection equation, where we know everything moves in the same direction.  But for more complicated conservation laws, things may move in either direction.  It will be useful to have a method that uses information from both directions.  The simplest such method is known as the **Lax-Friedrichs** method.  For the conservation law $q_t + f(q)_x$, this method is

$$Q_i^{n+1} = \frac{1}{2}(Q_{i-1}^n + Q_{i+1}^n) - \frac{\Dt}{2\Dx}\left(f(Q_{i+1}^n) - f(Q_{i-1}^n)\right).$$

Notice that the flux difference term clearly approximates $f(q)_x$.  Meanwhile, the value of $q$ itself is approximated by taking the average of two neighboring values.  This average makes this method dissipative too (but it ensures that the solution is stable).

### Exercise

1. What does the CFL condition imply for the time step when using the Lax-Friedrichs method?

2. In the cell below, implement the Lax-Friedrichs method for advection.

```python

```

*Extra credit*: Compute the norm of the difference between the approximate and exact solution.  How does it change if you decrease $\Dx$?

## Accuracy

The methods we have used so far (i.e., the *upwind method* and the *Lax-Friedrichs method*) are both dissipative.  Furthermore, both of these methods are only *first order accurate*, meaning that if we reduce the values of $\Dt$ and $\Dx$ by a factor of two, the overall error decreases only by a factor of two.  In [Lesson 3](Lesson_03_High-resolution_methods.ipynb), we will learn about more accurate methods.  But first, in [Lesson 2](Lesson_02_Traffic.ipynb) we'll look at a model for traffic flow.