# [Introductory applied machine learning (INFR10069)](https://www.learn.ed.ac.uk/webapps/blackboard/execute/content/blankPage?cmd=view&content_id=_2651677_1&course_id=_53633_1)

# Lab 5: Neural Networks

*by [James Owers](https://jamesowers.github.io/), University of Edinburgh 2017*

1. [Introduction](#Introduction)
    * [Lab Outline](#Lab-Outline)
    * [The Data](#The-Data)
1. [Part 1 - Introducing the Neural Network Model](#Part-1---Introducing-the-Neural-Network-Model)
    * [Resources to Watch and Read pt. 1](##Resources-to-Watch-and-Read-pt.-1)
    * [Model Design](#Model-Design)
    * [The Cost Space](#The-Cost-Space)
1. [Part 2 - Fitting the Model & Optimisation](#Part-2---Fitting-the-Model-&-Optimisation)
    * [Resources to Watch and Read pt. 2](#Resources-to-Watch-and-Read-pt.-2)
    * [Finding the Best Parameters](#Finding-the-Best-Parameters)
    * [Gradient Descent](#Gradient-Descent)
    * [Backpropagation](#Backpropagation)
1. [Part 3 - Implementation From Scratch](#Part-3---Implementation-From-Scratch!)
1. [Part 4 - Implementation With Sklearn](#Part-4---Implementation-with-Sklearn)
1. [Moar?!](#Please-sir...I-want-some-more)

## Import packages

```python
# https://docs.python.org/2/library/__future__.html
# make printing and division act like python 3
from __future__ import division, print_function

# General
import sys
import os
import copy
from IPython.display import Image, HTML

# Data structures
import numpy as np
import pandas as pd

# Modelling
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
from scipy.optimize import check_grad

# Plotting
import matplotlib.pyplot as plt
import seaborn as sns

# Local module adjacent to this notebook
import iaml
from iaml.data import load_letters

# http://ipython.readthedocs.io/en/stable/interactive/magics.html
%matplotlib inline
```

## Introduction

This lab:
1. introduces a simple neural network model in a supervised learning setting
1. provides impetus to understand the fitting procedure of that, and other networks
1. encourages you to implement a model from scratch
1. models the same problem with the sklearn package
1. makes you think about what you've done!

It does not discuss in detail:
1. any of the plethora of different activation functions you can use e.g. RELUs, SELUs, Tanh, ...
1. how to initialise the parameters and why that matters
1. issues with the fitting process e.g. local optima, and how to avoid them e.g. learning rate schedulers, momentum, RMSProp, Adam, cyclic learning rates
1. issues with model complexity e.g. overfitting, and solutions such as dropout, regularisation, or using [shedloads of data](https://what-if.xkcd.com/63/)
1. other tricks for speeding up and stablising fitting such as batch sizes, weight norm, layer norm
1. deep networks and their tricks like skip connections, pooling, convolutions
1. nor other more complex architectures like CNNs, RNNs, LSTMs, GANs, etc. etc.
1. many, many, MANY other things (that probably were published, like, [yesterday](https://arxiv.org/abs/1711.04340v1))

However, if you understand what is in this notebook well, **you will have the ability to understand [all of these things](https://i.imgflip.com/1zn8p9.jpg)**.

### Lab outline

I provide you with a function that creates data then link you to some excellent resources to learn the basics. These resources are superb, short, and free. I highly, highly recommend setting aside a couple of hours to give them a good watch/read and, at the very least, use them for reference.

After you have had a crack at the problems, I'll release the solutions. The solutions, particularly to part 3, walk you through the process of coding a simple neural neural network in detail.

Parts 3 & 4 are practical, parts 1 & 2 are links to external resources to read. Whilst I recommend you soak up some context first with 1 & 2, feel free to jump in at the deep end and get your hands dirty with part 3 or 4.

### The Data

Throughout this lab we are going to be using a simple classification example: the TC classification problem (not to be confused with the real [TC](https://www.youtube.com/watch?v=NToYkBYezZA)). This is a small toy problem where we, initially, try to distinguish between 3x3 grids that look like Ts and Cs. Let's create the dataset and have a look...

I have written a function `load_letters()` to generate synthetic data. For now, you will use the data generated below, but later you have opportunity to play with generating different data if you like. The function is located in the `iaml` module adjacent to this notebook - feel free to check out the code but I advise you **do not edit it**. Run (and don't edit) the next few cells to create and observe the data.

```python
bounds = [-1, 1]
X, y, y_labels = load_letters(categories=['T', 'C'],
                              num_obs=[50, 50],
                              bounds=bounds,
                              beta_params=[[1, 8], [8, 1]],
                              shuffle=True,
                              random_state=42)
```

Let's print the data (I'm just creating a Pandas DataFrame for display, I probably wont use this object again)

```python
pd.set_option("max_rows", 10)
df = pd.DataFrame(
    np.hstack(
        [np.around(X,2),
         y[:, np.newaxis],
         np.array([y_labels[ii] for ii in y])[:, np.newaxis]
        ]
    ),
    columns = ['x{}'.format(ii) for ii in range(9)] + ['Class (numeric)', 'Class Label']
)
df
```

```python
pd.reset_option("max_rows")
```

The data are arranged as vectors for your convenience, but they're really `3 x 3` images. Here's a function to plot them.

```python
def plot_grid(x, shape=None, **heatmap_params):
    """Function for reshaping and plotting vector data.
    If shape not given, assumed square.
    """
    if shape is None:
        width = int(np.sqrt(len(x)))
        if width == np.sqrt(len(x)):
            shape = (width, width)
        else:
            print('Data not square, supply shape argument')
    sns.heatmap(x.reshape(shape), annot=True, **heatmap_params)
```

```python
for ii in range(3):
    plt.figure()
    plot_grid(X[ii], vmin=bounds[0], vmax=bounds[1], cmap='Greys')
    plt.title('Observation {}: Class = {} (numeric label {})'.format(ii, y_labels[y[ii]], y[ii]))
    plt.show()
```

Finally, let's make the train and test split. Again, don't alter this code.

```python
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=42)
```

```python
X_train, X_valid, y_train, y_valid = train_test_split(
    X_train, y_train, test_size=0.33, random_state=42)
```

```python
[dd.shape for dd in [X_train, X_valid, X_test, y_train, y_valid, y_test]]
```

## Part 1 - Introducing the Neural Network Model

### Resources to Watch and Read pt. 1

**Reading/watching time:** 30 minutes

First, watch this video from 3 Blue 1 Brown: [But what *is* a Neural Network? | Deep learning, chapter 1](https://www.youtube.com/watch?v=aircAruvnKk)

If you prefer reading, try 2 sections of Nielsen's Book Chapter 1:
* [Sigmoid Neurons](http://neuralnetworksanddeeplearning.com/chap1.html#sigmoid_neurons)
* and [The Architecture of Neural Networks](http://neuralnetworksanddeeplearning.com/chap1.html#the_architecture_of_neural_networks)

### Model Design

Just so as there's something in this notebook to quickly reference - here's a nice illustration of what's going on in a neural net. Within the calculation of the $z$'s you'll see the learned **parameters**: $w$'s and $b$'s - these are the weights and biases respectively. *N.B. I omit the bias $b$ parameters in the Part 3 implementation.* The functions $g$ are the activation functions.

### The Cost Space

When we talk about the cost space, loss$^*$ space, or cost surface, we are talking about a function that changes with respect to the parameters. This function determines how well the network is performing - a low cost is good, a high cost is bad. A simple example for two parameters is shown below. **Our goal is to update the parameters such that we find the global minimum of the cost function.**

$^*$ 'loss' and 'cost' are interchangeable terms - you'll see them both around but I try to stick to 'cost'!

N.B. The cost function is often referred to with different letters e.g. $J(w)$, $C(\theta)$, $\mathcal{L}(x)$, and $E(w)$

## Part 2 - Fitting the Model & Optimisation

### Resources to Watch and Read pt. 2

**Watching/reading time:** ~1 hour

First, watch these two videos from 3 Blue 1 Brown:
1. [Gradient descent, how neural networks learn | Deep learning, chapter 2](https://www.youtube.com/watch?v=IHZwWFHWa-w)
2. [What is backpropagation and what is it actually doing? | Deep learning, chapter 3](https://www.youtube.com/watch?v=Ilg3gGewQ5U)

This will take you just over half an hour (if you watch at 1x speed). They are really excellent and well worth the time investment.

Again, if you prefer reading try Nielsen's section [Learning with Gradient Descent](http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent)

### Finding the Best Parameters

So, we've got a function, let's call it $C(\theta)$ that puts a number on how well the neural network is doing. We provide the function with the parameters $\theta$ and it spits out the cost$^*$. We could just randomly chose values for $\theta$ and select the ones that result in the best cost...but that might take a long time. We'd also need to define a way to randomly select parameters as well. What if the best parameter setting is very unlikely to be selected?

**Calculus to the rescue!** The cost $C(\theta)$ is a function and, whilst we can't see the surface without evaluating it everywhere (expensive!), we can calculate the derivative with respect to the parameters $\frac{\partial C(\theta)}{\partial \theta}$. The derivative **tells you how the function value changes if you change $\theta$**.

For example, imagine $\theta$ is 1D and I tell you that $\frac{\partial C(\theta)}{\partial \theta} = 10\theta$. This means that if I increase $theta$ by 2, the cost function will go up by 20. Which way will you update $\theta$? You want to *decrease* the cost, so you would want to *decrease* $\theta$ by some amount.

The only thing we need to do is choose a cost function $C(\theta)$ that has a derivative function $\frac{\partial C(\theta)}{\partial \theta}$...and that is easy!

$^*$It's much easier if you imagine $\theta$ as just one number to start with, but the maths is basically the same as $\theta$ becomes a vector (or matrix) of numbers.

### Gradient Descent

So how do we actually update the parameters?! All update the parameters in the opposite direction to the gradient; you always try to take a step 'downhill'. Here's the formula:

$$
\theta \leftarrow \theta - \eta \frac{\partial C(\theta)}{\partial \theta}
$$

where "$\leftarrow$" means "update from", and $\eta$ is the "learning rate" - a hyperparameter you can choose. If you increase $\eta$ you make bigger updates to $\theta$, and vice versa.

There are many more complicated ways to update the parameters using the gradient of the cost function, but they all have this same starting point.

Below is an example cost surface. A few things to note:

* The axes should be labelled $\theta_0$ (1, -1.5) and $\theta_1$ (-1, 1) on the 'flat' axes, and $C(\theta)$ (-4, 4) on the vertical axis
* The surface is shown - we don't have direct access to this in reality. To show it, the creator has queried the cost function *at every [$\theta_0$, $\theta_1$] location* and plotted it
* The animated balls rolling along the surface are different gradient descent algorithms - each frame of the GIF shows one update. The equation shown above is SGD - the GIF highlights a potential issue with the algorithm!

Visualisation by [Alec Radford](https://blog.openai.com/tag/alec-radford/), summarised excellently in [this blog post](http://ruder.io/optimizing-gradient-descent/).

### Backpropagation

**Reading/watching time:** 1 hour

Right...it's time for some derivatives. If you've been liking the videos - go ahead and watch the next in the series:

1. [Backpropagation calculus | Appendix to deep learning chapter 3](https://www.youtube.com/watch?v=tIeHLnjs5U8)

If you have time, I recommend now having a crack at reading half of [Nielsen Chapter 2](http://neuralnetworksanddeeplearning.com/chap2.html), up to and including the section entitled [The Backpropagation Algorithm](http://neuralnetworksanddeeplearning.com/chap2.html#the_backpropagation_algorithm).

I'm just going to write out some derivatives you're going to find useful for Part 3 below:

$$
\begin{align}
z^{(L)}                             &= W^{(L)}a^{(L-1)}  \\
\frac{\partial z^{(L)}}{\partial W} &= a^{(L-1)}
\end{align}
$$

$$
\begin{align}
\text{linear}[z]                             &= z \\
\frac{\partial \text{linear}[z]}{\partial z} &= 1 \\
\end{align}
$$

$$
\begin{align}
\text{sigmoid}[z] = \sigma[z]                &= \frac{1}{1 + e^{-z}} = \frac{e^{z}}{e^{z} + 1}\\
\frac{\partial \sigma[z]}{\partial z}        &= \frac{e^{z}}{e^{z} + 1} - (\frac{e^{z}}{e^{z} + 1})^2 \\
                                             &= \frac{e^{z}}{e^{z} + 1} ( 1 - \frac{e^{z}}{e^{z} + 1} ) \\
                                             &= \sigma[z] (1 - \sigma[z])
\end{align}
$$

$$
\begin{align}
\text{crossentropy}[y, a] = C[y, a]       &= - \frac{1}{N} \sum_{i=1}^N y_i \log a_i + (1-y_i)\log(1-a_i) \\
\frac{\partial C[y_i, a_i]}{\partial a_i} &=  \frac{1 - y_i}{1 - a_i} + \frac{y_i}{a_i}
\end{align}
$$

And finally, this is all backpropagation really is...
$$
\begin{align}
\frac{\partial C[y_i, a_i]}{\partial w_j} &=  \frac{\partial a_i}{\partial w_j}\frac{\partial C[y_i, a_i]}{\partial a_i}\\
                                          &=  \frac{\partial z_k}{\partial w_j}\frac{\partial a_i}{\partial z_k}\frac{\partial C[y_i, a_i]}{\partial a_i}\\
\end{align}
$$

Challenge: derive these yourself.

#### Reading extension

For more on gradient based optimisers [check out this blog post](http://ruder.io/optimizing-gradient-descent/)

For another look at backpropagation - try [Christopher Olah's blog](http://colah.github.io/posts/2015-08-Backprop/)

## Part 3 - Implementation From Scratch!

### ========== Question 3.1 ==========

First thing is first: **don't get stuck on this**. I recommend you attempt this question for an hour and, if you don't get anywhere, move on to Question 3.2. You can even move straight on to Part 4. It's exactly the same problem addressed here in 3.1, but using sklearn instead of coding it yourself.

#### Model Specification

We are going to fit a very small neural network to classify the TC data. Here is the specification of the model:

1. Input of size 9
1. Hidden layer of size 3
    * Linear activation function
1. Output layer of size 1
    * Logistic activation function

As for the **cost function**: use Cross-Entropy. However, if you're getting bogged down with derivatives, feel free to try squared error to start with (this is what Nielsen and 3 Blue 1 Brown start with in their tutorials). Squared error is [not necessarily the right cost function to use](https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/) but it will still work!

For a given input vector $x$, we can predict an output probability $a^{(2)}$ (were the $^{(2)}$ indicates the layer number, *not a power* - I'm following 3 Blue 1 Brown notation as best I can) using the following formula:

$$
\begin{align}
a^{(2)} &= f^{(2)}[z^{(2)}] \\
        &= f^{(2)}[W^{(2)}a^{(1)}] \\
        &= f^{(2)}[W^{(2)}f^{(1)}[z^{(1)}]] \\
        &= f^{(2)}[W^{(2)}f^{(1)}[W^{(1)}a^{(0)}]] \\
        &= f^{(2)}[W^{(2)}f^{(1)}[W^{(1)}x]] \\
        &= \sigma[W^{(2)}(W^{(1)}x)]
\end{align}
$$

where:

* $f^{(2)}$ is the activation function of the output layer (a sigmoid function $\sigma[]$)
* $f^{(1)}$ is the activation function of the hidden layer (the identity - 'linear activation')
* $W^{(2)}$ and $W^{(1)}$ are the parameters to learn
* $a^{(L)} = f^{(L)}[z^{(L)}]$ are the activations **exiting** layer $^{(L)}$
* $z^{(L)} = W^{(L)}a^{(L-1)}$ is the pre-activation weighted sum calculated **within** layer $^{(L)}$

The formula for the Cross-Entropy cost function is:

$$
C(a) = - \frac{1}{N} \sum_{i=1}^N y_i \log a_i + (1-y_i)\log(1-a_i)
$$

Notice how only one term in the sum is ever non-zero because $y_i$ is only ever 0 or 1. In our case, $N$ is the number of data observations in the dataset.

##### Parameters

The parameters of the model are two matrices:

1. $W^{(2)}$ - $3 \times 9$ matrix
    * used within the hidden layer (the $1^{st}$ layer) to get $z^{(1)} = W^{(1)}x$ for some $9 \times 1$ input vector $x$. $z^{(1)}$ is thus $3 \times 1$.
1. $W^{(1)}$ - $1 \times 3$ matrix
    * used within the output layer (the $2^{nd}$ layer) to get $z^{(2)} = W^{(2)}a^{(1)}$ for some $3 \times 1$ input vector $a^{(1)}$. $z^{(2)}$ is thus $1 \times 1$.

**Note that I'm not asking you to fit *bias parameters*.**

You'll often see parameters referred to as $\theta$, it's a catch all term. In our case it's just a list of all the weights, $\theta = [W^{(1)}, W^{(2)}]$. **We have 3 x 9 + 3 x 1 = 30 parameters to learn in total.**

##### Advice

You can use any of the equations and code I've given you or linked you to in this lab but **you do not have to!** You're free to code as you please. Personally, since this is a simple example, I did not do anything fancy (I didn't create any objects with methods and attributes). I simply:
* created a list containing the two parameter matrices `theta = [W1, W2]`
* created a function to do prediction (the forward pass)
* created a function to do the backward pass (updating the weights)
    * This is the tricky bit - I coded functions that are the [relevant derivatives](#http://localhost:8888/notebooks/10_Lab_5_Neural_Networks.ipynb#Backpropagation), and wrote code to iteratively pass back the 'deltas' - (I think Nielsen's equations [here](http://neuralnetworksanddeeplearning.com/chap2.html#the_backpropagation_algorithm) are very useful)
* wrote a training loop which called these two main functions
    * each epoch calls the forward pass to predict, then the backward pass to update the parameters.

When the training was finished, my "model" was simply the parameters I had fitted, along with the 'forward pass' function - a function which uses those weights to predict a probability for any input data.

**You do not have to code it up like me**, you can do it however you like! The point of this part is for you to explore, code up all the equations, understand how to calculate the loss, and how to use that loss to update the parameters of the model by backpropagation.

**Debugging**: You're probably going to have issues particularly in the backprop section. You are welcome to make use of the `scipy.optimize.check_grad()` function. This takes as input a function f, g: a function that is (supposed to be) the function's derivative.

If you didn't watch it already, now is a great time to take 10 minutes and watch [Backpropagation calculus | Appendix to deep learning chapter 3](https://www.youtube.com/watch?v=tIeHLnjs5U8)

#### ===== What you actually need to do for this question! =====

Write a training loop which uses gradient descent to learn the parameters. Each iteration of the loop is called an **epoch**. Run your code for *no more than 100 epochs*. You should be able to achieve 100% accuracy on this problem.

In this case, for simplicity, you may initialise the weights to be samples from a normal distribution mean 0 variance 1, but please note that this [is not necessarily good practice](https://intoli.com/blog/neural-network-initialization/)!

**Do not code up a grid search for the learning rate hyperparameter**. You may instead play with the learning rate manually until you are happy. Try small values first like 0.0001 (if your backprop code is correct you **should** see your cost decreasing every epoch). Since this problem is so simple, a range of values should work. Again, with real data, you *must* do a search over hyperparameters, but here we are focussed on *coding* a working model.

To test whether or not what you have written has worked, please output the following:
1. After the training loop:
    1. plot a graph of training and validation loss against epoch number
    1. print or plot the final parameters you have learned using a Hinton diagram - feel free to use [code you can find online](http://bfy.tw/F74s)
    1. pick one weight parameter and produce a plot of its value against epoch number
        * Extension: do that for all the weights **leaving one specific input node** (i.e. the weights for one pixel of the input data)
    1. use your model to:
        1. print a few of the validation data examples and their predicted probabilities
        1. print the output for a T and C with no noise (you can make that input data yourself)
        1. print the output of a few random binary vectors i.e. 9x1 vectors of only 0s and 1s (again, you can make that input data yourself)

1. Within the training loop:
    1. print the training and validation crossentropy loss **and** percentage accuracy every epoch
    1. save the value of the training and validation losses for every epoch [for the plot after the loop]
    1. save the value of a weight parameter of your choice [for the plot after the loop]

#### ===== Example outputs =====

Below I give you some examples of what I'd like you to produce. **I produced these using a learning rate of 0.003, 100 epochs, and weights initialised with N(0,1) with a random seed of 42**. I found that you could learn faster i.e. you can use a larger learning rate, but I wanted to make smooth plots for you.

You don't need to produce plots exactly like this, you can do them how you like, but try and display the same information. You can also use my plots for checking (if you use the same settings as me).

##### 1A

##### 1B

##### 1C

##### 1D

##### 2A

```python
# Your code goes here
```

### ========== Question 3.2 ==========

Did you need a network this large to do this classification task? Give the values for the parameters of a network with no hidden layers, one output node, and an output activation function of a sigmoid that would get 100% accuracy. This network only has 9 parameters.

*Your answer goes here*

### ========== Question 3.3 ==========

You should recognise the model described in question 3.2. What is it?

*Your answer goes here*

### ========== Question 3.4 ==========

Why did I create input data, `X`, that was between [-1, 1] i.e. why wasn't it between [0, 1] like normal?! Would the model specified in question 3.1 above have worked if `X` was in [0, 1]? Explain why or why not.

*Hint: if you're stuck, you can try it out by generating some new data and trying to fit it.*

*Your answer goes here*

### ========== Question 3.5 [EXTENSION] ==========

Create a dataset which makes the problem harder. Have a look at the dataset generation code. You can use the arguments to create data with:
* more letters (make the problem a multiclass classification)
    * You'll need to implement the multiclass version of the sigmoid for the output activation function - [the softmax](https://en.wikipedia.org/wiki/Softmax_function) (and of course it's derivative)
* increase the noise on the data

Some other things you could implement:
* include rotated letters in the data
* make larger data (bigger than 3x3)
* make the letters non-centred e.g. 5x5 data with 3x3 letters in 1 of 9 different places

You'll probably need to adapt the code you wrote in 3.1, but you can probably copy and paste most of it. For an additional challenge: introduce [bias parameters](http://neuralnetworksanddeeplearning.com/chap1.html) and create your `X` data in range [0, 1] (i.e. set the bounds argument to [0, 1])...

Some other things to try if you get code happy:
* Implement stochastic gradient descent updates (updating parameters every training example, as opposed to every epoch) - tip: randomise data order each epoch
* Implement batch gradient descent updates - tip: randomise data order each epoch

**Requirements**:
1. Describe the modelling problem and your input data. Plot some examples of the data
1. Write down the model specification (I should be able to reproduce your model with this description):
    * number of nodes in each layer
    * a description of the parameters to learn (and a total number of parameters)
    * the activation functions used for each layer
    * cost function used
1. All the outputs asked for in Question 3.1: loss per epoch plot, final parameters, a weight against epoch plot, and example predictions

*Your answer goes here*

```python
# Your code goes here
```

## Part 4 - Implementation with Sklearn

### ========== Question 4.1 ==========

If you did Question 3.1, this should be a breeze! Use the same data and perform the same modelling task. This time you can use Sklearn's Neural Network object [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier).

Before you begin, read the [introduction](http://scikit-learn.org/stable/modules/neural_networks_supervised.html) (sections 1.17.1 and 1.17.2 at a minimum, 1.17.5, 1.17.6, 1.17.7 are recommended).

```python
# Your code goes here
```

### ========== Question 4.2 ==========

The learned parameters are stored in the fitted sklearn [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier) object **as two separate attributes**.

1. Print the parameters learned by your fitted model
1. Print the total number of parameters learned

Look at the number of parameters described in question 3.1 (you do not need to have done this question 3.1 - just read its description). Below the code:

1. Explain why the number of parameters learned by sklearn is different from the number specified in 3.1?

```python
# Your code goes here
```

*Your answer goes here*

# [Please sir...I want some more](https://www.youtube.com/watch?v=Ex2r86G0sdc)

Well done, you successfully covered the basics of Neural Networks!

If you enjoyed this lab, you'll love another course @ Edinburgh: [Machine Learning Practical](https://github.com/CSTR-Edinburgh/mlpractical). Check it out.

### Next steps

The first thing to do, if you haven't already, is do the extension question 3.5. **In particular, you should implement bias parameters in your model code**.

Next, go back to the very top of the notebook where I detail things I will not cover. Pick some words you don't understand (perhaps along with the keyword 'example' or 'introduction') and have fun reading/watching some tutorials about them online. Code up what you have learned; if you can code it up without peeking, you know you have understood it very well indeed. Another good "starter for 10" google is "a review of neural networks for [images|text|music|bat detection|captioning images|generation|...]".

Here are some things that you might find fun to read:
* [Visualising networks learning](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=5&networkShape=3&seed=0.42978&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false)
* [Trying to understand what features are learned by Deep Nets](https://distill.pub/2017/feature-visualization/)
* [Modelling sound waves](https://deepmind.com/blog/wavenet-generative-model-raw-audio/)
   * ...and using that to [encode instruments](https://magenta.tensorflow.org/nsynth)
* An [Introduction to LSTMs](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) and their [unreasonable effectiveness](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
* How to encode the entire meaning of a word [in a few numbers](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)
* [Convolutions for text data?!](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)

### Learning resources

Also:
* [there](http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/)
* [are](http://neuralnetworksanddeeplearning.com/chap1.html)
* [literally](https://www.coursera.org/learn/machine-learning)
* [so](https://www.coursera.org/learn/neural-networks)
* [many](http://deeplearning.net/)
* [learning](http://datasciencemasters.org/)
* [resources](https://metacademy.org/graphs/concepts/backpropagation)
* [online!](http://www.deeplearningbook.org/)

(about neural nets etc.)

In all seriousness, make sure you check out [metacademy](https://metacademy.org/). You can search for a topic and it gives you a list of free resources, an estimated time you need to understand it, and prerequisite topics.

# Attributions

Parts of this lab were inspired by  D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel distributed processing: Explorations
in the microstructure of cognition, vol. 1, MIT Press, Cambridge, MA, USA, 1986,
pp. 318–362.

Thanks also to:
* [3 Blue 1 Brown](https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw)
* [Michael Nielsen](http://neuralnetworksanddeeplearning.com)
* [Christopher Olah](http://colah.github.io/)

for producing some excellent visualisations and learning resources and providing them free of charge.

Additionally, many thanks to the developers of open source software, in particular:
* [Numpy](http://www.numpy.org/)
* [Scipy](https://www.scipy.org/)
* [Sklearn](http://scikit-learn.org/stable/)
* [Matplotlib](https://matplotlib.org/)
* [Jupyter](http://jupyter.org/)
* and of course [Python](https://www.python.org/) itself!

your work is invaluable and appreciated.

# Credits

This lab was created by [James Owers](https://jamesowers.github.io/) in November 2017 and reviewed by [Patric Fulop](https://www.inf.ed.ac.uk/people/students/Patric_Fulop.html).


# How does the SFF method work?

Vanderburg and Johnson (2014) introduced a method for "Self Flat Fielding" by tracking how the lightcurve changes with motion of the spacecraft:

[A Technique for Extracting Highly Precise Photometry for the Two-Wheeled Kepler Mission](http://adsabs.harvard.edu/abs/2014PASP..126..948V)

In this notebook we replicate the K2SFF method following the same example source, #60021426, as that in the publication.  We aim to demystify the technique, which is extremely popular within the K2 community.  We have focused on reproducibility, so that we achieve the same result at the publication.

The Vanderburg & Johnson 2014 paper uses data from the Kepler two-wheel "Concept Engineering Test", predating campaign 0, and sometimes called campaign *"eng"* or abbreviated CET.  This vestigal "campaign" lacks some of the standardization of later K2 campaigns--- it was much shorter, only about 9 days long, it lacks some of the standard quality flags, targets have non-traditional EPIC IDs, and other quirks.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from astropy.io import fits
import pandas as pd
```

## Retrieve the K2SFF data for ENG test source `60021426`

First we will retrieve data and inspect the mask used in the paper.

```python
path = 'http://archive.stsci.edu/hlsps/k2sff/cet/060000000/21426/hlsp_k2sff_k2_lightcurve_060021426-cet_kepler_v1_llc.fits'
vdb_fits = fits.open(path)
```

The `BESTAPER` keyword explains which aperture was chosen as the "best" by Vanderburg & Johnson 2014.  The FITS header for that slice contains the metadata needed to reproduce the mask.

```python
keys = ['MASKTYPE', 'MASKINDE', 'NPIXSAP']
_ = [print(key, ' : ', vdb_fits['BESTAPER'].header[key]) for key in keys]
```

    MASKTYPE  :  PRF FIT
    MASKINDE  :  4
    NPIXSAP  :  29.0

We want the *exact same* mask as Vanderburg & Johnson 2014, but the publication version and MAST version differ!

Publication version:

MAST Version:

Aperture 7 should yield a bigger mask, more similar to what was used in the paper.

```python
VDB_J_mask = vdb_fits['PRF_APER_TBL'].data[7,:, :] == True
VDB_J_mask.sum()
```

    53

Save the mask for easy use in our next notebook.

```python
np.save('VDB_J_2014_mask.npy', VDB_J_mask)
```

## Manually reproduce with the Vanderburg-provided diagnostic data
Retrieve the Vanderburg-provided diagnostic data for the Kepler ENG testing.
Uncomment the line below to retrieve the data programmatically, or manually get the linked file in a browser and save it to this directory.

```python
#! wget https://www.cfa.harvard.edu/~avanderb/k2/ep60021426alldiagnostics.csv
```

```python
df = pd.read_csv('ep60021426alldiagnostics.csv',index_col=False)
df.head()
```

      BJD - 2454833
      Raw Flux
      Corrected Flux
      X-centroid
      Y-centroid
      arclength
      Correction
      Thrusters On

      0
      1862.502368
      0.995119
      0.995985
      25.135097
      24.661074
      2.327480
      0.999130
      1.0

      1
      1862.522801
      0.997313
      0.996767
      25.289752
      24.418689
      1.175322
      1.000548
      1.0

      2
      1862.543235
      0.996713
      0.996136
      25.288052
      24.429406
      1.214627
      1.000580
      0.0

      3
      1862.563668
      0.996930
      0.996277
      25.275216
      24.448405
      1.306617
      1.000656
      0.0

      4
      1862.584102
      0.996862
      0.996228
      25.253864
      24.480184
      1.460259
      1.000636
      0.0

We can mean-subtract the provided $x-y$ centroids, assigning them column and row identifiers, then rotate the coordinates into their major and minor axes.

```python
col = df[' X-centroid'].values
col = col - np.mean(col)
row = df[' Y-centroid'].values
row = row - np.mean(row)
```

```python
def _get_eigen_vectors(centroid_col, centroid_row):
    '''get the eigenvalues and eigenvectors given centroid x, y positions'''
    centroids = np.array([centroid_col, centroid_row])
    eig_val, eig_vec = np.linalg.eigh(np.cov(centroids))
    return eig_val, eig_vec
```

```python
def _rotate(eig_vec, centroid_col, centroid_row):
    '''rotate the centroids into their predominant linear axis'''
    centroids = np.array([centroid_col, centroid_row])
    return np.dot(eig_vec, centroids)
```

```python
eig_val, eig_vec = _get_eigen_vectors(col, row)
v1, v2 = eig_vec
```

The major axis is the latter.

```python
platescale = 4.0 # The Kepler plate scale; has units of arcseconds / pixel
```

```python
plt.figure(figsize=(5, 6))
plt.plot(col * platescale, row * platescale, 'ko', ms=4)
plt.plot(col * platescale, row * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel('X position [arcseconds]')
plt.ylabel('Y position [arcseconds]')
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, v1[0]], [0, v1[1]], color='blue', lw=3)
plt.plot([0, v2[0]], [0, v2[1]], color='blue', lw=3);
```

Following the form of **Figure 2** of Vanderburg & Johsnon 2014.

```python
rot_colp, rot_rowp = _rotate(eig_vec, col, row) #units in pixels
```

You can rotate into the new reference frame.

```python
plt.figure(figsize=(5, 6))
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ko', ms=4)
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel("X' position [arcseconds]")
plt.ylabel("Y' position [arcseconds]")
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, 1], [0, 0], color='blue')
plt.plot([0, 0], [0, 1], color='blue');
```

We need to calculate the arclength using:
\begin{equation}s= \int_{x'_0}^{x'_1}\sqrt{1+\left( \frac{dy'_p}{dx'}\right)^2} dx'\end{equation}

where $x^\prime_0$ is the transformed $x$ coordinate of the point with the smallest $x^\prime$ position, and $y^\prime_p$ is the best--fit polynomial function.

Fit a $5^{th}$ order polynomial to the rotated coordinates.

```python
z = np.polyfit(rot_rowp, rot_colp, 5)
p5 = np.poly1d(z)
p5_deriv = p5.deriv()
```

```python
x0_prime = np.min(rot_rowp)
xmax_prime = np.max(rot_rowp)
x_dense = np.linspace(x0_prime, xmax_prime, 2000)
```

```python
plt.plot(rot_rowp, rot_colp, '.')
plt.plot(x_dense, p5(x_dense))
plt.ylabel('Position along minor axis (pixels)')
plt.xlabel('Position along major axis (pixels)')
plt.title('Performance of polynomial regression')
plt.ylim(-0.1, 0.1);
```

We see evidence for a [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff), suggesting some modest opportunity for improvement.

```python
@np.vectorize
def arclength(x):
    '''Input x1_prime, get out arclength'''
    gi = x_dense <x
    s_integrand = np.sqrt(1 + p5_deriv(x_dense[gi]) ** 2)
    s = np.trapz(s_integrand, x=x_dense[gi])
    return s
```

Let's double check that we compute the same arclength as the published paper.

```python
aspect_ratio = plt.figaspect(1)
plt.figure(figsize=aspect_ratio)
plt.plot(df[' arclength'], arclength(rot_rowp)*4.0, '.')
plt.xlabel('$s$  (Vanderburg & Johnson 2014)')
plt.ylabel('$s$  (This work)')
plt.plot([0, 4], [0, 4], 'k--');
```

Yes, we compute arclength correctly.

Now we apply a **high-pass filter** to the raw lightcurve data.  We follow the original paper by using *BSplines* with 1.5 day breakpoints.  You can also apply data exclusion at this stage.

```python
from scipy.interpolate import BSpline
from scipy import interpolate
```

```python
times, raw_fluxes = df['BJD - 2454833'].values, df[' Raw Flux'].values
```

We find the weighted least square spline for a given set of knots, $t$.  We supply interior knots as knots on the ends are added automatically, as stated in the `interpolate.splrep()` docstring.

```python
interior_knots = np.arange(times[0]+1.5, times[0]+6, 1.5)
t,c,k = interpolate.splrep(times, raw_fluxes, s=0, task=-1, t=interior_knots)
bspl = BSpline(t,c,k)
```

```python
plt.plot(times, raw_fluxes, '.')
plt.plot(times, bspl(times))
plt.xlabel('$t$ (days)')
plt.ylabel('Raw Flux');
```

The Spline fit looks good, so we can normalize the flux by the long-term trend.
Plot the normalized flux versus arclength to see the position-dependent flux.

```python
fluxes = raw_fluxes/bspl(times)
```

Mask the data by keeping only the good samples.

```python
bi = df[' Thrusters On'].values == 1.0
gi = df[' Thrusters On'].values == 0.0
clean_fluxes = fluxes[gi]
al = arclength(rot_rowp[gi]) * platescale
```

```python
sorted_inds = np.argsort(al)
```

We will follow the paper by interpolating **flux versus arclength position** in 15 bins of means, which is a *piecewise linear fit*.

```python
knots = np.array([np.min(al)]+
                 [np.median(splt) for splt in np.array_split(al[sorted_inds], 15)]+
                 [np.max(al)])
```

```python
bin_means = np.array([clean_fluxes[sorted_inds][0]]+
                     [np.mean(splt) for splt in np.array_split(clean_fluxes[sorted_inds], 15)]+
                     [clean_fluxes[sorted_inds][-1]])
```

```python
zz = np.polyfit(al, clean_fluxes,6)
sff = np.poly1d(zz)
al_dense = np.linspace(0, 4, 1000)
interp_func = interpolate.interp1d(knots, bin_means)
```

```python
plt.figure(figsize=(5, 6))
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'ko', ms=4)
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'o', color='#3498db', ms=3)
plt.plot(arclength(rot_rowp[bi])*4.0, fluxes[bi], 'o', color='r', ms=3)
plt.plot(np.sort(al), interp_func(np.sort(al)), '-', color='#e67e22')

plt.xticks([0, 1,2, 3, 4])
plt.minorticks_on()
plt.xlabel('Arclength [arcseconds]')
plt.ylabel('Relative Brightness')
plt.title('EPIC 60021426, Kp =10.3')
plt.xlim(0,4)
plt.ylim(0.997, 1.002);
```

Following **Figure 4** of Vanderburg & Johnson 2014.

Apply the Self Flat Field (SFF) correction:

```python
corr_fluxes = clean_fluxes / interp_func(al)
```

```python
plt.figure(figsize=(10,6))

dy = 0.004
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'ko', ms=4)
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'o', color='#3498db', ms=3)
plt.plot(df['BJD - 2454833'][bi], df[' Raw Flux'][bi]+dy, 'o', color='r', ms=3)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='k', ms = 4)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='#e67e22', ms = 3)

plt.xlabel('BJD - 2454833')
plt.ylabel('Relative Brightness')

plt.xlim(1862, 1870)
plt.ylim(0.994, 1.008);
```

Following **Figure 5** of Vanderburg & Johnson 2015.

*The end.*


```python
"""Tutorial: SCF first hyperpolarizability"""

__author__    = "Eric J. Berquist"
__credit__    = ["Eric J. Berquist"]

__copyright__ = "(c) 2014-2018, The Psi4NumPy Developers"
__license__   = "BSD-3-Clause"
__date__      = "2017-12-19"
```

# SCF First Hyperpolarizability

## Introduction

In Tutorial 6a, the calculation of linear response properties from analytic derivative theory is presented, the foundation of which are the coupled-perturbed Hartree-Fock (CPHF) or coupled-perturbed self-consistent field (CPSCF) equations. Starting from analytic derivative theory provides a convenient physical picture: how does the total energy of a system change under the influence of one or more internal or external perturbations? Continuing the case of an external electric field, the total energy of a system can be represented with a series expansion:

$$
E(\mathbf{E}) = \sum_{n=0}^{\infty} \frac{1}{n!}E^{(n)}(\mathbf{a})\cdot(\mathbf{E}-\mathbf{a})^{n},
$$

where the electric field is $\mathbf{E} = \vec{E} = (E_x,E_y,E_z)$ and $\mathbf{a}$ is the expansion point. In practice, we always expand around $\mathbf{a} = \mathbf{0}$, so it is a Maclaurin series:

$$
E(\mathbf{E}) = \sum_{n=0}^{\infty} \frac{1}{n!}\mathbf{E}^{(n)}(\mathbf{0})\cdot\mathbf{E}^{n}.
$$

Expanding the above to the first 4 explicit terms gives

$$
E(\mathbf{E}) \approx E^{(0)}(\mathbf{0}) + \mathbf{E}^{(1)}(\mathbf{0})\cdot\mathbf{E} + \frac{1}{2}\mathbf{E}^{(2)}(\mathbf{0})\cdot\mathbf{E}^{2} + \frac{1}{6}\mathbf{E}^{(3)}(\mathbf{0})\cdot\mathbf{E}^{3},
$$

where we identify

\begin{align}
E^{(0)} &\rightarrow \textrm{the unperturbed ground-state energy} \\
E_{a}^{(1)} &\rightarrow -\mu_{a},\textrm{the dipole moment} \\
E_{ab}^{(2)} &\rightarrow -\alpha_{ab},\textrm{the polarizability} \\
E_{abc}^{(3)} &\rightarrow -\beta_{abc},\textrm{the first hyperpolarizability}
\end{align}

The first hyperpolarizability is the leading-order term that describes the _nonlinear_ response of a system to an external electric field. Each term in the series expansion increases the rank of the coefficient by one: the ground-state energy is a scalar, the dipole is a length 3 vector, the polarizability is a 3-by-3 matrix, and the first hyperpolarizability is a 3-by-3-by-3 tensor.

Translated into the language of analytic derivative theory, is it represented as $\beta_{abc} = \left.\frac{\partial^{3} E}{\partial E_a \partial E_b \partial E_c}\right|_{\mathbf{E}=\mathbf{0}}$, though it is not yet clear how to take derivatives of the energy beyond what is presented in tutorial 6a. Additionally, nothing has been stated about time dependence; everything to this point has been the static case, where the strength of fields do not vary with time. We will first incorporate time dependence, and equations for derivative theory will result.

### Notation

Before going further, some notational conventions should be mentioned. When used as field indices, $a,b,c,\dots \in \{x,y,z\}$, the three Cartesian directions.

For matrix indices, $\mu,\nu,\lambda,\sigma,\dots$ label atomic orbitals (AOs)/basis functions, $i,j,k,l,\dots$ label occupied molecular orbitals (MOs), $a,b,c,d,\dots$ label unoccupied/virtual MOs, and $p,q,r,s,\dots$ label all MOs. Einstein summation is used, so repeated indices are contracted over.

## Derivation

Again, write the total Hamiltonian as the sum of unperturbed and perturbed components

\begin{align}
\hat{H}(\mathbf{E},t) &= \hat{H}^{(0)} + \hat{V}(\mathbf{E},t) \\
\hat{V}(\mathbf{E},t) &= -\mathbf{\mu} \cdot \mathbf{E}(e^{\pm i \omega t} + 1)
\end{align}

where part of the external field now oscillates with some characteristic frequency $\omega$. This can be incorporated into the time-dependent Schrodinger equation, which for a stationary state obeys

\begin{align}
\left[ \hat{H}^{(0)} + \hat{V}(\mathbf{E},t) - i\frac{\partial}{\partial t} \right] \psi(t) &= 0, \\
FC - i \frac{\partial}{\partial t} SC &= SC\epsilon, \\
\frac{\partial}{\partial t} C^{\dagger} S C &= 0,
\end{align}

where the full definition of the Fock matrix is

$$
F_{\mu\nu} = h_{\mu\nu} + D_{\lambda\sigma}[2J_{\mu\nu\lambda\sigma} - K_{\mu\nu\lambda\sigma}]
$$

and the density matrix is defined as

$$
D_{\mu\nu} = C_{\mu p}n_{pq}C_{\nu q}^{\dagger},
$$

where the diagonal occupation number matrix $n_{ii} = 2$ and $n_{aa} = 0$ for RHF.

In general, the MO coeffients are perturbation- and time-dependent, but the basis functions themselves are not. This means that when the series expansion for the perturbation above is performed on other quantities, only $F$, $C$, $\epsilon$, and $D$ are affected. For example, the Lagrangian multiplier matrix $\epsilon$ can be expanded as

$$
\epsilon(\mathbf{E}) = \epsilon^0 + E_a\epsilon^a + \frac{1}{2!}E_aE_b\epsilon^{ab} + \frac{1}{3!}E_aE_bE_c\epsilon^{abc} + \cdots
$$

where $a,b,c,...\in\{x,y,z\}$, and

\begin{align}
\epsilon^{a} &= e^{\pm i \omega t} \epsilon^{a}(\pm\omega) + \epsilon^{a}(0), \\
\epsilon^{ab} &= e^{\pm 2 i \omega t} \epsilon^{ab}(\pm\omega,\pm\omega) + e^{\pm i \omega t} \{ \epsilon^{ab}(0,\pm\omega) + \epsilon^{ab}(\pm\omega,0)\} + \epsilon^{ab}(\pm\omega,\mp\omega) + \epsilon^{ab}(0,0), \\
\epsilon^{abc} &= e^{\pm 3 i \omega t} \epsilon^{abc}(\pm\omega,\pm\omega,\pm\omega) + e^{\pm 2 i \omega t} \{\epsilon^{abc}(0,\pm\omega,\pm\omega) + \epsilon^{abc}(\pm\omega,0,\pm\omega) + \epsilon^{abc}(\pm\omega,\pm\omega,0)\} + e^{\pm i \omega t} \{\epsilon^{abc}(\pm\omega,\pm\omega,\mp\omega) + \epsilon^{abc}(\pm\omega,\mp\omega,\pm\omega) + \epsilon^{abc}(\mp\omega,\pm\omega,\pm\omega)\} + e^{\pm i \omega t} \{\epsilon^{abc}(0,0,\pm\omega) + \epsilon^{abc}(0,\pm\omega,0) + \epsilon^{abc}(\pm\omega,0,0)\} + \{\epsilon^{abc}(0,\pm\omega,\mp\omega) + \epsilon^{abc}(\pm\omega,0,\mp\omega) + \epsilon^{abc}(\pm\omega,\mp\omega,0)\} + \epsilon^{abc}(0,0,0),
\end{align}

showing that each order of the expansion consists of all possible phase combinations. For the first hyperpolarizability, only quantities with at most two field indices are required. Each permutationally unique subterm of the expansion corresponds to a different physical observable:

* $(0) \rightarrow$ static polarizability $\rightarrow \alpha(0;0) = -Tr[H^{a} D^{b}(0)]$
* $(\pm\omega) \rightarrow$ dynamic polarizability $\rightarrow \alpha(\mp\omega;\pm\omega) = -Tr[H^{a} D^{b}(\pm\omega)]$

* $(0,0) \rightarrow$ static (first) hyperpolarizability $\rightarrow \beta(0;0,0) = -Tr[H^{a} D^{bc}(0,0)]$
* $(0,\pm\omega) \rightarrow$ electrooptic Pockels effect (EOPE) $\rightarrow \beta(\mp \omega;0,\pm\omega) = -Tr[H^{a} D^{bc}(0,\pm\omega)]$
* $(\pm\omega,\pm\omega) \rightarrow$ second harmonic generation (SHG) $\rightarrow \beta(\mp 2\omega;\pm\omega,\pm\omega) = -Tr[H^{a} D^{bc}(\pm\omega,\pm\omega)]$
* $(\pm\omega,\mp\omega) \rightarrow$ optical rectification $\rightarrow \beta(0;\pm\omega,\mp\omega) = -Tr[H^{a} D^{bc}(\pm\omega,\mp\omega)]$

where each property is calculated as the trace over the AO-basis dipole matrices $H^{a}$ with the appropriate perturbed density. The task now comes down to calculating the necessary perturbed density for the phenomenon of interest. The second-order densities required for the four different first hyperpolarizabilities are (Karna eqs. III-2a to III-2d)

\begin{align}
D^{ab}(\pm\omega,\pm\omega) &= C^{ab}(\pm\omega,\pm\omega) n C^{0\dagger} + C^{a}(\pm\omega) n C^{b\dagger}(\mp\omega) + C^{b}(\pm\omega) n C^{a\dagger}(\mp\omega) + C^{0} n C^{ab\dagger}(\mp\omega,\mp\omega), \\
D^{ab}(0,\pm\omega) &= C^{ab}(0,\pm\omega) n C^{0\dagger} + C^{a}(0) n C^{b\dagger}(\mp\omega) + C^{b}(\pm\omega) n C^{a\dagger}(0) + C^{0} n C^{ab\dagger}(0,\mp\omega), \\
D^{ab}(\pm\omega,\mp\omega) &= C^{ab}(\pm\omega,\mp\omega) n C^{0\dagger} + C^{a}(\pm\omega) n C^{b\dagger}(\pm\omega) + C^{b}(\mp\omega) n C^{a\dagger}(\mp\omega) + C^{0} n C^{ab\dagger}(\mp\omega,\pm\omega), \\
D^{ab}(0,0) &= C^{ab}(0,0) n C^{0\dagger} + C^{a}(0) n C^{b\dagger}(0) + C^{b}(0) n C^{a\dagger}(0) + C^{0} n C^{ab\dagger}(0,0).
\end{align}

Already a few important insights about the equations are revealed: Each perturbation index always carries its respective frequency, and the positive and negative frequencies are related by the Hermitian adjoint (except for $C(-\omega) = -C^{0} U^{\dagger}(+\omega)$, Karna eq. 40). We also see the appearance of terms like $C^{ab}$, which will require $U^{ab}$ originating from the second-order CPHF. Computationally, this is undesirable due to the increased number of iterative calculations that must be performed, so we borrow a trick that most prominently appears in perturbation theory.

### Wigner's $2n + 1$ rule

From Schaefer, page 25:

> When the wavefunction is determined up to the $n$th order, the expectation value (electronic energy) of the the system is resolved, according to the results of perturbation theory, up to the $(2n+1)$st order. This principle is called Wigner's $2n+1$ theorem [29-31].

$$
\begin{array}{cccc}
\hline
 & \text{CI: MO/CI space} & \text{MCSCF: MO/CI space} & \text{RHF: MO space} \\
\hline
\text{Energy}, E & C_{\mu}^{i} , C_{I} & C_{\mu}^{i}, C_{I} & C_{\mu}^{i} \\
\text{First Derivative}, \frac{\partial E}{\partial a} & U^{a} , C_{I} & C_{\mu}^{i}, C_{I} & C_{\mu}^{i}\\
\text{Second Derivative}, \frac{\partial^{2} E}{\partial a \partial b} & U^{ab} , \frac{\partial C_{I}}{\partial a} & U^{a}, \frac{\partial C_{I}}{\partial a} & U^{a} \\
\text{Third Derivative}, \frac{\partial^{3} E}{\partial a \partial b \partial c} & U^{abc} , \frac{\partial C_{I}}{\partial a} & U^{a}, \frac{\partial C_{I}}{\partial a} & U^{a} \\
\text{Fourth Derivative}, \frac{\partial^{4} E}{\partial a \partial b \partial c \partial d} & U^{abcd} , \frac{\partial^{2} C_{I}}{\partial a \partial b} & U^{ab}, \frac{\partial^{2} C_{I}}{\partial a \partial b} & U^{ab} \\
\text{Fifth Derivative}, \frac{\partial^{5} E}{\partial a \partial b \partial c \partial d \partial e} & U^{abcde} , \frac{\partial^{2} C_{I}}{\partial a \partial b} & U^{ab}, \frac{\partial^{2} C_{I}}{\partial a \partial b} & U^{ab} \\
\hline
\end{array}
$$

Since the first hyperpolarizability is calculated as a third derivative of the energy, perturbed coefficients with only one field index should be required. From the above table, we can also see why SCF gradients ($\frac{\partial E}{\partial R_A}$, where $R_A$ is the $A$-th Cartesian component of nucleus $R$) avoid the need to solve for $U$ matrices.

## Final expressions

To this point, most work has been in the AO basis, but it is conceptually easier to work in the MO basis, in particular due to the use of the $\epsilon$ equations (Karna eq. 34)

$$
\epsilon^{a}(\pm\omega) = G^{a}(\pm\omega) + \epsilon^{0} U^{a}(\pm\omega) - U^{a}(\pm\omega) \epsilon^{0} \pm \omega U^{a}(\pm\omega),
$$

where the $G$ matrices are the MO-basis Fock matrices

$$
G^{ab\dots} = C^{0\dagger} F^{ab\dots} C^{0},
$$

and the $U$ matrices are the MO-basis perturbation parameters

$$
C^{ab\dots} = C^{0} U^{ab\dots},
$$

which will be discussed in the implementation. The final expression for the static hyperpolarizability is (Karna eq. VII-4)

$$
\beta_{abc}(0; 0, 0) = Tr[n \{  U^{a}(0) G^{b}(0) U^{c}(0) + U^{c}(0) G^{b}(0) U^{a}(0) + U^{b}(0) G^{c}(0) U^{a}(0) + U^{a}(0) G^{c}(0) U^{b}(0) + U^{c}(0) G^{a}(0) U^{b}(0) + U^{b}(0) G^{a}(0) U^{c}(0)\} ] - Tr[n \{ U^{a}(0) U^{c}(0) \epsilon^{b}(0) + U^{c}(0) U^{a}(0) \epsilon^{b}(0) + U^{b}(0) U^{a}(0) \epsilon^{c}(0) + U^{a}(0) U^{b}(0) \epsilon^{c}(0) + U^{c}(0) U^{b}(0) \epsilon^{a}(0) + U^{b}(0) U^{c}(0) \epsilon^{a}(0) \} ].
$$

By noticing that each term corresponds to a unique permutation of the field indices, it can be rewritten as

$$
\beta_{abc} = Tr\left[ n \sum \mathcal{P}(d,e,f) U^{d} G^{e} U^{f} \right] - Tr\left[ n \sum \mathcal{P}(d,e,f) U^{d} U^{e} \epsilon^{f} \right],
$$

where the permutation indices are initially assigned as $d = a, e = b, f = c$. The frequency notation has also been dropped, since each $abc$ (and therefore each $def$) will always carry the appropriate field index, making this the most general form of the first hyperpolarizability. If the indices $abc$ are also permuted, then all 27 components of the first hyperpolarizability tensor will be computed.

## Computational Procedure

The basic quantities we need are the matrices $C, \mu, F, \epsilon$, and $U$. The MO coefficients $C$ are already obtained from the ground-state calculation, along with $\epsilon^{0}$ (the MO energies) and $F^{0}$ (the AO-basis Fock matrix). The dipole matrices $\mu$ are needed for the linear response (polarizability) calculation, which results in response vectors that compose the off-diagonal blocks of the $U^{a}$ matrix. $G^{a}$ is obtained from $F^{a}$, which comes from performing a single Fock build with the perturbed density $D^{a}$. Finally, $\epsilon^{a}$ can be constructed.

Although the expressions so far are general for any frequency and first-order non-linear optical response, the tutorial implementation will cover the static case. For second-harmonic generation, see the reference implementation.

```python
import numpy as np
np.set_printoptions(3, linewidth=100, suppress=True)    # when we inspect the vectors/matrices,
                                                        # use a prettier format for printing
import psi4
```

The energy and density convergence criteria are tightened from defaults, as response properties are sensitive to the quality of the ground-state wavefunction.

```python
mol = psi4.geometry('''
    O
    H  1  0.9435
    H  1  0.9435  2  105.9443
    symmetry c1
''')
psi4.set_options({
    "basis": "aug-cc-pVDZ",
    "scf_type": "direct",
    "df_scf_guess": False,
    "e_convergence": 1e-9,
    "d_convergence": 1e-9,
})
```

```python
# This is to enable testing outside of the notebook environment.
import sys
try:
    get_ipython()
    sys.path.append('../../Response-Theory/Self-Consistent-Field')
except NameError:
    import os.path
    dirname = os.path.dirname(os.path.abspath(__file__))
    sys.path.append(os.path.join(dirname, '../../Response-Theory/Self-Consistent-Field'))

from helper_CPHF import helper_CPHF
```

The helper encapsulates the solution of the ground-state wavefunction followed by the frequency-(in)dependent linear response equations,

$$
\left[
\begin{pmatrix}
\mathbf{A} & \mathbf{B} \\
\mathbf{B}^{*} & \mathbf{A}^{*}
\end{pmatrix}
- \omega_{f}
\begin{pmatrix}
\mathbf{\Sigma} & \mathbf{\Delta} \\
-\mathbf{\Delta}^{*} & -\mathbf{\Sigma}^{*}
\end{pmatrix}
\right]
\begin{pmatrix}
\mathbf{X} \\
\mathbf{Y}
\end{pmatrix}
=
\begin{pmatrix}
\mathbf{V} \\
-\mathbf{V}^{*}
\end{pmatrix}
,
$$

either directly (via matrix inversion in the MO basis) or iteratively (via repeated matrix-vector products using Fock builds). For a HF/DFT reference with canonical orbitals, the above equations reduce to

$$
\left[
\begin{pmatrix}
\mathbf{A} & \mathbf{B} \\
\mathbf{B} & \mathbf{A}
\end{pmatrix}
- \omega_{f}
\begin{pmatrix}
\mathbf{1} & \mathbf{0} \\
\mathbf{0} & -\mathbf{1}
\end{pmatrix}
\right]
\begin{pmatrix}
\mathbf{X} \\
\mathbf{Y}
\end{pmatrix}
=
\begin{pmatrix}
\mathbf{V} \\
-\mathbf{V}
\end{pmatrix}
.
$$

In the static limit $(\omega_f = 0)$, the whole superoverlap matrix vanishes, and the CPHF equations can be reduced to those used in tutorial 6a.

```python
solver = helper_CPHF(mol)
solver.run()
```

    Number of occupied orbitals: 5
    Number of basis functions: 41

    Tensor sizes:
    ERI tensor           0.02 GB.
    oNNN MO tensor       0.00 GB.
    ovov Hessian tensor  0.00 GB.

    Forming Hessian...
    ...formed Hessian in 0.473 seconds.

    Inverting Hessian...
    ...inverted Hessian in 0.007 seconds.

Because the calculation of $\beta$ requires $U^{a}$, we also obtain linear response properties from a quadratic response calculation. This holds for any order of response, where lower-order response functions are automatically obtained from higher-order response calculations.

```python
print(np.around(solver.polar, 4))
```

    [[ 7.2587 -0.      0.    ]
     [-0.      8.7969  0.    ]
     [ 0.      0.      7.854 ]]

```python
# epsilon^{0}
moenergies = solver.epsilon
C = np.asarray(solver.C)
Co = solver.Co
Cv = solver.Cv
nbf, norb = C.shape
nocc = Co.shape[1]
nvir = norb - nocc
nov = nocc * nvir
# the response vectors X_x, X_y, X_z; Y_x, Y_y, Y_z not needed separately for static response
x = np.asarray(solver.x)
ncomp = x.shape[0]
# reuse the AO-basis dipole integrals
integrals_ao = np.asarray([np.asarray(dipole_ao_component)
                           for dipole_ao_component in solver.tmp_dipoles])
print("dimension of response vectors from linear response: {}".format(x.shape))
# for dynamic response, this will be (2 * nov)
assert x.shape[1] == nov
```

    dimension of response vectors from linear response: (3, 180)

The foundation of the CPHF equations is that the right-hand side $\mathbf{V}$ is a perturbation on the wavefunction causing single excitations from the occupied orbitals to virtual orbitals, the coefficents of which are in the response vectors $\mathbf{X}$; the vectors $\mathbf{Y}$ describe single deexcitations. Because the full (square) $\mathbf{U}$ matrices are required, all MO-based quantities must be of shape $[N_{orb}, N_{orb}]$ rather than $[N_{occ}, N_{vir}]$.

```python
# form full MO-basis dipole integrals
integrals_mo = np.empty(shape=(ncomp, norb, norb))
for i in range(ncomp):
    integrals_mo[i, ...] = (C.T).dot(integrals_ao[i, ...]).dot(C)
```

Similarly, $\mathbf{X}$ and $\mathbf{Y}$ form the off-diagonal blocks of the $\mathbf{U}$ matrices. They are usually stored as in DALTON, where each vector is of length $2N_{ov}$, with $\mathbf{X}$ on top of $\mathbf{Y}$.

```python
# repack response vectors to [norb, norb]; 1/2 is due to X + Y
U = np.zeros_like(integrals_mo)
for i in range(ncomp):
    U[i, :nocc, nocc:] = 0.5 * x[i, ...].reshape(nocc, nvir)
    U[i, nocc:, :nocc] = -0.5 * x[i, ...].reshape(nocc, nvir).T
```

A minor implementation detail: because this was not a frequency-dependent calculation, only $\mathbf{X}+\mathbf{Y}$ needs to be calculated; as they are identical, this leads to the prefactor of 1/2.

```python
# form G matrices from perturbation and generalized Fock matrices; do
# one more Fock build for each response vector
jk = psi4.core.JK.build(solver.scf_wfn.basisset())
jk.initialize()
G = np.empty_like(U)
R = psi4.core.Matrix(nbf, nocc)
npR = np.asarray(R)
for i in range(ncomp):
    V = integrals_mo[i, ...]

    # eqn. (III-1b)
    # Note: this simplified handling of the response vector
    # transformation for the Fock build is insufficient for
    # frequency-dependent response. 1/2 is due to X + Y
    jk.C_clear()
    L = Co
    npR[...] = x[i, ...].reshape(nocc, nvir).dot(np.asarray(Cv).T).T
    jk.C_left_add(L)
    jk.C_right_add(R)
    jk.compute()
    J = 0.5 * np.asarray(jk.J()[0])
    K = 0.5 * np.asarray(jk.K()[0])

    # eqn. (21b)
    F = (C.T).dot(4 * J - K.T - K).dot(C)
    G[i, ...] = V + F

# form epsilon matrices, eqn. (34)
E = G.copy()
omega = 0
for i in range(ncomp):
    eoU = (moenergies[..., np.newaxis] + omega) * U[i, ...]
    Ue = U[i, ...] * moenergies[np.newaxis, ...]
    E[i, ...] += (eoU - Ue)

# Assume some symmetry and calculate only part of the tensor.
# eqn. (VII-4)
hyperpolarizability = np.zeros(shape=(6, 3))
off1 = [0, 1, 2, 0, 0, 1]
off2 = [0, 1, 2, 1, 2, 2]
for r in range(6):
    b = off1[r]
    c = off2[r]
    for a in range(3):
        tl1 = 2 * np.trace(U[a, ...].dot(G[b, ...]).dot(U[c, ...])[:nocc, :nocc])
        tl2 = 2 * np.trace(U[a, ...].dot(G[c, ...]).dot(U[b, ...])[:nocc, :nocc])
        tl3 = 2 * np.trace(U[c, ...].dot(G[a, ...]).dot(U[b, ...])[:nocc, :nocc])
        tr1 = np.trace(U[c, ...].dot(U[b, ...]).dot(E[a, ...])[:nocc, :nocc])
        tr2 = np.trace(U[b, ...].dot(U[c, ...]).dot(E[a, ...])[:nocc, :nocc])
        tr3 = np.trace(U[c, ...].dot(U[a, ...]).dot(E[b, ...])[:nocc, :nocc])
        tr4 = np.trace(U[a, ...].dot(U[c, ...]).dot(E[b, ...])[:nocc, :nocc])
        tr5 = np.trace(U[b, ...].dot(U[a, ...]).dot(E[c, ...])[:nocc, :nocc])
        tr6 = np.trace(U[a, ...].dot(U[b, ...]).dot(E[c, ...])[:nocc, :nocc])
        tl = tl1 + tl2 + tl3
        tr = tr1 + tr2 + tr3 + tr4 + tr5 + tr6
        hyperpolarizability[r, a] = -2 * (tl - tr)
```

```python
ref_static = np.array([
    [ 0.00000001,   0.00000000,  -0.10826460],
    [ 0.00000000,   0.00000000, -11.22412215],
    [ 0.00000000,   0.00000000,  -4.36450397],
    [ 0.00000000,   0.00000000,  -0.00000001],
    [-0.10826460,  -0.00000001,   0.00000000],
    [-0.00000001, -11.22412215,   0.00000000]
])
assert np.allclose(ref_static, hyperpolarizability, rtol=0.0, atol=1.0e-3)
print('\nFirst dipole hyperpolarizability (static):')
print(hyperpolarizability)
```

    First dipole hyperpolarizability (static):
    [[ -0.       -0.       -0.10826]
     [ -0.       -0.      -11.22412]
     [ -0.       -0.       -4.3645 ]
     [ -0.       -0.        0.     ]
     [ -0.10826   0.       -0.     ]
     [  0.      -11.22412  -0.     ]]

## References

### Primary equations and implementation

* [Karna:1991:487][1]: Karna, Shashi P.; Dupuis, Michel. Frequency dependent nonlinear optical properties of molecules: Formulation and implementation in the HONDO program _J. Comput. Chem._ **12** 487-504 (1991)

### $2n + 1$ rule

* [Yamaguchi, Yukio; Goddard, John D.; Osamura, Yoshihiro; Schaefer III, Henry F. _A New Dimension to Quantum Chemistry: Analytic Derivative Methods in Ab Initio Molecular Electronic Structure Theory (International Series of Monographs on Chemistry);_ Oxford University Press: 1994.][2]
* [Epstein, Saul T. General Remainder Theorem _J. Chem. Phys._ **48** 4725 (1968)][3]
* [Epstein, Saul T. Constraints and the $V^{2n+1}$ theorem _Chem. Phys. Lett._ **70** 311 (1980)][4]

### Additional reading

* [Neese, Frank. Prediction of molecular properties and molecular spectroscopy with density functional theory: From fundamental theory to exchange-coupling _Coord. Chem. Rev._ **253** 526-563 (2009)][5]

  [1]: https://doi.org/10.1002/jcc.540120409
  [2]: https://isbnsearch.org/isbn/9780195070286
  [3]: https://doi.org/10.1063/1.1668053
  [4]: https://doi.org/10.1016/0009-2614(80)85340-1
  [5]: https://doi.org/10.1016/j.ccr.2008.05.014


# Materials data science: descriptors and machine learning

Welcome to the materials data science lesson. In this session, we will demonstrate how to use `matminer`, `pandas` and `scikit-learn` for machine learning materials properties.

The lesson is split into four sections:
1. Data retrieval and basic analysis of pandas `DataFrame` objects.
2. Generating machine learnable descriptors.
3. Training, testing and visualizing machine learning methods with `scikit-learn` and `plotly.express`.

Many more tutorials on how to use matminer (beyond the scope of this workshop) are available in the `matminer_examples` repository, available [here](https://github.com/hackingmaterials/matminer_examples).

## Machine learning workflow

Firstly, what does a typical machine learning workflow look like? The overall process can be summarized as:
1. Take raw inputs, such as a list of compositions, and an associated target property to learn.
2. Convert the raw inputs into *descriptors* or *features* that can be learned by machine learning algorithms.
3. Train a machine learning model on the data.
4. Plot and analyze the performance of the model.

Typically, questions asked by a new practitioner in the field include:
- Where do we get the raw data from?
- How do we convert the raw data into learnable features?
- How can we plot and interpret the results of a model?

The `matminer` package has been developed to help make machine learning of materials properties easy and hassle free. The aim of matminer is to connect materials data with data mining algorithms and data visualization.

## Part 1: Data retrieval and filtering

Matminer interfaces with many materials databases,  including:
- Materials Project
- Citrine
- AFLOW
- Materials Data Facility (MDF)
- Materials Platform for Data Science (MPDS)

In addition, it also includes datasets from published literature. Matminer hosts a repository of **42 (and growing)** datasets which comes from published and peer-reviewed machine learning investigations of materials properties or publications of high-throughput computing studies.

In this section, we will show how to access and manipulate the pre-formatted datasets from the published literature. More information on accessing other materials databases are detailed in the [matminer_examples](https://github.com/hackingmaterials/matminer_examples) repository.

A list of the literature-based datasets can be printed using the `get_available_datasets()` function. This also prints information about what the dataset contains, such as the number of samples, the target properties, and how the data was obtained (e.g., via theory or experiment).

```python
from matminer.datasets import get_available_datasets

get_available_datasets()
```

All datasets can be loaded using the `load_dataset()` function and the database name. To save installation space, the datasets are not automatically downloaded when matminer is installed. Instead, the first time the dataset is loaded, it will be downloaded from the internet and stored in the matminer installation directory.

Let's say we're interested in the `dielectric_constant` dataset,  which contains 1,056 structures with dielectric properties calculated with DFPT-PBE. We can download it with the `load_dataset` function.

We'll set an environment variable `MATMINER_DATA` which will tell matminer to download all our dataset to a directory `./data`. If you are running this locally, you usually don't need to set this variable as matminer will download the dataset directly to your matminer source code folder.

```python
%env MATMINER_DATA data

from matminer.datasets import load_dataset

df = load_dataset("dielectric_constant")
```

We can get some more detailed information about this dataset using the `get_all_dataset_info(<dataset>)` function from matminer.

```python
from matminer.datasets import get_all_dataset_info

print(get_all_dataset_info("dielectric_constant"))
```

### Manipulating and examining pandas `DataFrame` objects

The datasets are made available as pandas `DataFrame` objects. You can think of these as a type of "spreadsheet" object in Python. DataFrames have several useful methods you can use to explore and clean the data, some of which we'll explore below.

**Dataframes are central objects for retrieving and holding data, adding features, training models, and visualizing with graphs in Python.** In the rest of the lesson we will be working with them a lot.

#### Inspecting the dataset

The `head()` function prints a summary of the first few rows of a data set. You can scroll across to see more columns. From this, it is easy to see the types of data available in in the dataset.

```python
df.head()
```

Sometimes, if a dataset is very large, you will be unable to see all the available columns. Instead, you can see the full list of columns using the `columns` attribute:

```python
df.columns
```

A pandas `DataFrame` includes a function called `describe()` that helps determine statistics for the various numerical/categorical columns in the data. Note that the `describe()` function only describes numerical columns by default.

Sometimes, the `describe()` function will reveal outliers that indicate mistakes in the data.

```python
df.describe()
```

#### Indexing the dataset

We can access a particular column of `DataFrame` by indexing the object using the column name. For example:

```python
df["band_gap"]
```

You can also access multiple columns by indexing with a list of column names rather than a single column name:

Alternatively, we can access a particular row of a `Dataframe` using the `iloc` attribute.

```python
df.iloc[100]
```

#### Filtering the dataset

Pandas `DataFrame` objects make it very easy to filter the data based on a specific column. We can use the typical Python comparison operators (==, >, >=, <, etc) to filter numerical values. For example, let's find all entries where the cell volume is greater than 580. We do this by filtering on the `volume` column.

Note that we first produce a *boolean mask* – a series of `True` and `False` depending on the comparison. We can then use the mask to filter the `DataFrame`.

```python
mask = df["volume"] >= 580
df[mask]
```

We can use this method of filtering to clean our dataset. For example, if we only wanted our dataset to only include nonmetals (materials with a non-zero band gap), we can do this easily by filtering the `band_gap` column.

```python
mask = df["band_gap"] > 0
nonmetal_df = df[mask]
nonmetal_df
```

Often, a dataset contains many additional columns that are not necessary for machine learning. Before we can train a model on the data, we need to remove any extraneous columns. We can remove whole columns from the dataset using the `drop()` function. This function can be used to drop both rows and columns.

The function takes a list of items to drop. For columns, this is column names whereas for rows it is the row number. Finally, the `axis` option specifies whether the data to drop is columns (`1`) or rows (`0`).

For example, to remove the `nsites`, `space_group`, `e_electronic`, and `e_total` columns, we can run:

```python
cleaned_df = df.drop(["nsites", "space_group", "e_electronic", "e_total"], axis=1)
```

Let's examine the cleaned `DataFrame` to see that the columns have been removed.

```python
cleaned_df.head()
```

You can alternatively *select* multiple columns by passing in a list of column names as an index.

For example, if we're only interested in the `band_gap` and `structure` columns, we can index with `["band_gap", "structure"]`

```python
df[["band_gap", "structure"]]
```

#### Generating new columns

Pandas `DataFrame` objects also make it easy to perform simple calculations on the data. Think of this as using formulas in Excel spreadsheets. All fundamental Python math operators (such as +, -, /, and \*) can be used.

For example, the dielectric dataset contains the electronic contribution to the dielectric constant ($\epsilon_\mathrm{electronic}$, in the `poly_electronic` column) and the total (static) dielectric constant ($\epsilon_\mathrm{total}$, in the `poly_total` column). The ionic contribution to the dataset is given by:

$$
\epsilon_\mathrm{ionic} = \epsilon_\mathrm{total} - \epsilon_\mathrm{electronic}
$$

Below, we calculate the ionic contribution to the dielectric constant and store it in a new column called `poly_ionic`. This is as simple as assigning the data to the new column, even if the column doesn't already exist.

```python
df["poly_ionic"] = df["poly_total"] - df["poly_electronic"]
```

Let's check the new data was added correctly.

```python
df.head()
```

## Part 2: Generating descriptors for machine learning

In this section, we will learn a bit about how to generate machine-learning descriptors from materials objects in pymatgen. First, we'll generate some descriptors with matminer's "featurizers" classes. Next, we'll use some of what we learned about dataframes in the previous section to examine our descriptors and prepare them for input to machine learning models.

### Featurizers transform materials primitives into machine-learnable features

The general idea of featurizers is that they accept a materials primitive (e.g., pymatgen Composition) and output a vector. For example:

\begin{align}
f(\mathrm{Fe}_2\mathrm{O}_3) \rightarrow [1.5, 7.8, 9.1, 0.09]
\end{align}

#### Matminer contains featurizers for the following pymatgen objects:
* Composition
* Crystal structure
* Crystal sites
* Bandstructure
* Density of states

#### Depending on the featurizer, the features returned may be:
* numerical, categorical, or mixed vectors
* matrices
* other pymatgen objects (for further processing)

#### Featurizers play nice with dataframes
Since most of the time we are working with pandas dataframes, all featurizers work natively with pandas dataframes. We'll provide examples of this later in the lesson.

#### Featurizers present in matminer
Matminer hosts over 60 featurizers, most of which are implemented from methods published in peer reviewed papers. You can find a full list of featurizers on the [matminer website](https://hackingmaterials.lbl.gov/matminer/featurizer_summary.html). All featurizers have parallelization and convenient error tolerance built into their core methods.

In this lesson, we'll go over the main methods present in all featurizers. By the end of this unit, you will be able to generate descriptors for a wide range of materials informatics problems using one common software interface.

### The `featurize` method and basics

The core method of any matminer is "featurize". This method accepts a materials object and returns a machine learning vector or matrix. Let's see an example on a pymatgen composition:

```python
from pymatgen.core import Composition

fe2o3 = Composition("Fe2O3")
```

As a trivial example, we'll get the element fractions with the `ElementFraction` featurizer.

```python
from matminer.featurizers.composition.element import ElementFraction

ef = ElementFraction()
```

Now we can featurize our composition.

```python
element_fractions = ef.featurize(fe2o3)

print(element_fractions)
```

We've managed to generate features for learning, but what do they mean? One way to check is by reading the `Features` section in the documentation of any featurizer... but a much easier way is to use the `feature_labels()` method.

```python
element_fraction_labels = ef.feature_labels()
print(element_fraction_labels)
```

We now see the labels in the order that we generated the features.

```python
print(element_fraction_labels[7], element_fractions[7])
print(element_fraction_labels[25], element_fractions[25])
```

### Featurizing  dataframes

We just generated some descriptors and their labels from an individual sample but most of the time our data is in pandas dataframes. Fortunately, matminer featurizers implement a `featurize_dataframe()` method which interacts natively with dataframes.

Let's grab a new dataset from matminer and use our `ElementFraction` featurizer on it.

First, we download a dataset as we did in the previous unit. In this example, we'll download a dataset of super hard materials.

```python
from matminer.datasets.dataset_retrieval import load_dataset

df = load_dataset("brgoch_superhard_training")
df.head()
```

Next, we can use the `featurize_dataframe()` method (implemented by all featurizers) to apply ElementFraction to all of our data at once. The only required arguments are the dataframe as input and the input column name (in this case it is `composition`). `featurize_dataframe()` is parallelized by default using multiprocessing.

If we look at the dataframe, now we can see our new feature columns.

```python
df = ef.featurize_dataframe(df, "composition")

df.head()
```

### Structure Featurizers

We can use the same syntax for other kinds of featurizers. Let's now assign descriptors to a structure. We do this with the same syntax as the composition featurizers. First, let's load a dataset containing structures.

```python
df = load_dataset("phonon_dielectric_mp")

df.head()
```

Let's calculate some basic density features of these structures using `DensityFeatures`.

```python
from matminer.featurizers.structure import DensityFeatures

densityf = DensityFeatures()
densityf.feature_labels()
```

These are the features we will get. Now we use `featurize_dataframe()` to generate these features for all the samples in the dataframe. Since we are using the structures as input to the featurizer, we select the "structure" column.

Let's examine the dataframe and see the structural features.

```python
df = densityf.featurize_dataframe(df, "structure")

df.head()
```

### Conversion Featurizers

In addition to Bandstructure/DOS/Structure/Composition featurizers, matminer also provides a featurizer interface for converting between pymatgen objects (e.g., assinging oxidation states to compositions) in a fault-tolerant fashion. These featurizers are found in `matminer.featurizers.conversion` and work with the same `featurize`/`featurize_dataframe` etc. syntax as the other featurizers.

The dataset we loaded previously only contains a `formula` column with string objects. To convert this data into a `composition` column containing pymatgen `Composition` objects, we can use the `StrToComposition` conversion featurizer on the `formula` column.

```python
from matminer.featurizers.conversions import StrToComposition

stc = StrToComposition()
df = stc.featurize_dataframe(df, "formula", pbar=False)
```

We can see a new `composition` column has been added to the dataframe.

```python
df.head()
```

### Advanced capabilities

There are powerful functionalities of Featurizers which are worth quickly mentioning before we go practice (and _many_ more not mentioned here).

#### Dealing with Errors

Often, data is messy and certain featurizers will encounter errors. Set `ignore_errors=True` in `featurize_dataframe()` to skip errors; if you'd like to see the errors returned in an additional column, also set `return_errors=True`.

#### Citing the authors

Many featurizers are implemented using methods found in peer reviewed studies. Please cite these original works using the `citations()` method, which returns the BibTex-formatted references in a Python list. For example:

```python
from matminer.featurizers.structure.bonding import GlobalInstabilityIndex

GlobalInstabilityIndex().citations()
```

## Part 3: Machine learning models

In parts 1 and 2, we demonstrated how to download a dataset and add machine learnable features. In part 3, we show how to train a machine learning model on a dataset and analyze the results.

### Scikit-Learn
This unit makes extensive use of the [scikit-learn](https://scikit-learn.org/stable/) package, an open-source python package for  machine learning. Matminer has been designed to make machine learning with scikit-learn as easy as possible. Other machine learning packages exist, such as [TensorFlow](https://www.tensorflow.org), which implement neural network architectures. These packages can also be used with `matminer` but are outside the scope of this workshop.

### Load and prepare a pre-featurized model

First, let's load a dataset that we can use for machine learning. In advance, we've added some composition and structure features to the `elastic_tensor_2015` dataset used in exercises 1 and 2.

```python
import os
from matminer.utils.io import load_dataframe_from_json

df = load_dataframe_from_json(os.path.join("resources", "elastic_tensor_2015_featurized.json"))
df.head()
```

We first need to split the dataset into the "target" property, and the "features" used for learning. In this model, we will be using the bulk modulus (`K_VRH`) as the target property. We use the `values` attribute of the dataframe to give the target properties a numpy array, rather than pandas `Series` object.

```python
y = df['K_VRH'].values

print(y)
```

The machine learning algorithm can only use numerical features for training. Accordingly, we need to remove any non-numerical columns from our dataset. Additionally, we want to remove the `K_VRH` column from the set of features, as the model should not know about the target property in advance.

The dataset loaded above, includes `structure`, `formula`, and `composition` columns that were previously used to generate the machine learnable features. Let's remove them using the pandas `drop()` function, discussed in unit 1. Remember, `axis=1` indicates we are dropping columns rather than rows.

```python
X = df.drop(["structure", "formula", "composition", "K_VRH"], axis=1)
```

We can see all the descriptors in model using the `column` attribute.

```python
print("There are {} possible descriptors:".format(len(X.columns)))
print(X.columns)
```

### Try a random forest model using scikit-learn

The `scikit-learn` library makes it easy to use our generated features for training machine learning models. It implements a variety of different regression models and contains tools for cross-validation.

In the interests of time, in this example we will only trial a single model but it is good practice to trial multiple models to see which performs best for your machine learning problem. A good "starting" model is the random forest model. Let's create a random forest model.

```python
from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(n_estimators=100, random_state=1)
```

Notice we created the model with the number of estimators (`n_estimators`) set to `100`. `n_estimators` is an example of a machine learning *hyper-parameter*. Most models contain many tunable hyper-parameters. To obtain good performance, it is necessary to fine tune these parameters for each individual machine learning problem. There is currently no simple way to know in advance what hyper-parameters will be optimal. Usually, a trial and error approach is used.

We can now train our model to use the input features (`X`) to predict the target property (`y`). This is achieved using the `fit()` function.

```python
rf.fit(X, y)
```

That's it, we have trained our first machine learning model!

### Evaluating model performance

Next, we need to assess how the model is performing. To do this, we first ask the model to predict the bulk modulus for every entry in our original dataframe.

```python
y_pred = rf.predict(X)
```

Next, we can check the accuracy of our model by looking at the *root mean squared error* of our predictions. Scikit-learn provides a `mean_squared_error()` function to calculate the mean squared error. We then take the square-root of this to obtain our final performance metric.

```python
import numpy as np
from sklearn.metrics import mean_squared_error

mse = mean_squared_error(y, y_pred)
print('training RMSE = {:.3f} GPa'.format(np.sqrt(mse)))
```

An RMSE of 7.2 GPa looks very reasonable! However, as the model was trained and evaluated on exactly the same data, this is not a true estimate of how the model will perform for unseen materials (the primary purpose of machine learning studies).

#### Cross validation

To obtain a more accurate estimate of prediction performance and validate that we are not over-fitting, we need to check the **cross-validation score** rather than the fitting score.

In cross-validation, the data is partitioned randomly into $n$ "splits" (in this case 10), each containing roughly the same number of samples. The model is trained on $n-1$ splits (the training set) and the model performance evaluated by comparing the actual and predicted values for the final split (the testing set). In total, this process is repeated $n$ times, such that each split is at some point used as the testing set. The cross-validation score is the average score across all testing sets.

There are a number of ways to partition the data into splits. In this example, we use the `KFold` method and select the number of splits to be 10. I.e., 90 % of the data will be used as the training set, with 10 % used as the testing set.

```python
from sklearn.model_selection import KFold

kfold = KFold(n_splits=10, random_state=1, shuffle=True)
```

Note, we set `random_state=1` to ensure every attendee gets the same answer for their model.

Finally, obtaining the cross validation score can be automated using the Scikit-Learn `cross_val_score()` function. This function requires a machine learning model, the input features, and target property as arguments. Note, we pass the `kfold` object as the`cv` argument, to make `cross_val_score()` use the correct test/train splits.

For each split, the model will be trained from scratch, before the performance is evaluated. As we have to train and predict 10 times, cross validation can often take some time to perform. In our case, the model is quite small, so the process only takes about a minute. The final cross validation score is the average across all splits.

```python
from sklearn.model_selection import cross_val_score

scores = cross_val_score(rf, X, y, scoring='neg_mean_squared_error', cv=kfold)

rmse_scores = [np.sqrt(abs(s)) for s in scores]
print('Mean RMSE: {:.3f}'.format(np.mean(rmse_scores)))
```

Notice that our RMSE has almost tripled as now it reflects the true predictive power of the model. However, a root-mean-squared error of ~19 GPa is still not bad!

### Visualizing model performance

We can visualize the predictive performance of our model by plotting the our predictions against the actual value, for each sample in the test set for all test/train splits. First, we get the predicted values of the testing set for each split using the `cross_val_predict` method. This is similar to the `cross_val_score` method, except it returns the actual predictions, rather than the model score.

```python
from sklearn.model_selection import cross_val_predict

y_pred = cross_val_predict(rf, X, y, cv=kfold)
```

Let's now add our predicted values to our dataframe and calculate an absolute percentage error for each sample.

We can do this conveniently for all of our samples with the dataframe columns.

If we scroll to the end of the dataframe, our predicted `K_VRH` and percentage errors are given for each sample. This might allow us to examine manually which samples are performing well and which are performing poorly.

```python
df["K_VRH predicted"] = y_pred
df["percentage_error"] = (df["K_VRH"] - df["K_VRH predicted"]).abs()/df["K_VRH"] * 100

df
```

A more convient way of examining our model's performance is by creating a graph comparing our cross-validation predicted bulk modulus to the actual bulk modulus for every sample. Here, we use `plotly.express` from the Plotly package to create our graphs.

Plotly Express is designed to create many kinds of plots directly from dataframes. Since we already have our data inside a dataframe, we can specify the column names to tell Plotly the data we'd like to show.

We make two series of data:
- First, a reference line indicating "perfect" peformance of the model.
- Second, a scatter plot of the predicted K_VRH vs the actual K_VRH for every sample.

```python
import plotly.express as px
import plotly.graph_objects as go

reference_line = go.Scatter(
    x=[0, 400],
    y=[0, 400],
    line=dict(color="black", dash="dash"),
    mode="lines",
    showlegend=False
)

fig = px.scatter(
    df,
    x="K_VRH",
    y="K_VRH predicted",
    hover_name="formula",
    color="percentage_error",
    color_continuous_scale=px.colors.sequential.Bluered,
)

fig.add_trace(reference_line)
fig.show()
```

Not too bad! However, there are definitely some outliers (you can hover over the points with your mouse to see what they are).

### Model interpretation

An important aspect of machine learning is being able to understand why a model is making certain predictions. Random forest models are particularly amenable to interpretation as they possess a `feature_importances` attribute, which contains the importance of each feature in deciding the final prediction. Let's look at the feature importances of our model.

```python
rf.feature_importances_
```

To make sense of this, we need to know which feature each number corresponds to. We can use `PlotlyFig` to plot the importances of the 5 most important features.

```python
importances = rf.feature_importances_
included = X.columns.values
indices = np.argsort(importances)[::-1]

fig_bar = px.bar(
    x=included[indices][0:5],
    y=importances[indices][0:5],
    title="Feature Importances of Random Forest",
    labels={"x": "Feature", "y": "Importance"}
)
fig_bar.show()

```

## Bonus: Curated ML datasets with Matbench

If you are interested in comparing your machine learning algorithms with the state of the art, matminer also offers access to a curated set of 13 benchmarking datasets called Matbench, which have been used to benchmark SoTA algorithms like RooSt, CGCNN, CRABNet, MEGNet, Automatminer, and more.

The Matbench datasets span a wide variety of materials informatics tasks such as:

- Predicting materials properties given **only composition**, or given **composition _and_ structure**
- Predicting a **wide array of target properties**, such as elastic constants, dielectric constants, formation energies, and steel yield strength
- **Data-sparse** tasks (300 samples) and (relatively) **data-rich** tasks (100k+ samples)
- Both regression and classification tasks

The full set of datasets is given in the table below:

| Task name | Task type | Target column (unit) | Input type | Samples | MAD (regression) or Fraction True (classification) | Links |
|-------|-------|-------|-------|-------|-------|-------|
| `matbench_steels` | regression | `yield strength` (MPa) | composition | 312 | 229.3743 | [download](https://ml.materialsproject.org/projects/matbench_steels.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_steels) |
| `matbench_jdft2d` | regression | `exfoliation_en` (meV/atom) | structure | 636 | 67.2020 | [download](https://ml.materialsproject.org/projects/matbench_jdft2d.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_jdft2d) |
| `matbench_phonons` | regression | `last phdos peak` (cm^-1) | structure | 1,265 | 323.7870 | [download](https://ml.materialsproject.org/projects/matbench_phonons.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_phonons) |
| `matbench_expt_gap` | regression | `gap expt` (eV) | composition | 4,604 | 1.1432 | [download](https://ml.materialsproject.org/projects/matbench_expt_gap.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_expt_gap) |
| `matbench_dielectric` | regression | `n` (unitless) | structure | 4,764 | 0.8085 | [download](https://ml.materialsproject.org/projects/matbench_dielectric.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_dielectric) |
| `matbench_expt_is_metal` | classification | `is_metal` | composition | 4,921 | 0.4981 | [download](https://ml.materialsproject.org/projects/matbench_expt_is_metal.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_expt_is_metal) |
| `matbench_glass` | classification | `gfa` | composition | 5,680 | 0.7104 | [download](https://ml.materialsproject.org/projects/matbench_glass.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_glass) |
| `matbench_log_gvrh` | regression | `log10(G_VRH)` (log10(GPa)) | structure | 10,987 | 0.2931 | [download](https://ml.materialsproject.org/projects/matbench_log_gvrh.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_log_gvrh) |
| `matbench_log_kvrh` | regression | `log10(K_VRH)` (log10(GPa)) | structure | 10,987 | 0.2897 | [download](https://ml.materialsproject.org/projects/matbench_log_kvrh.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_log_kvrh) |
| `matbench_perovskites` | regression | `e_form` (eV/unit cell) | structure | 18,928 | 0.5660 | [download](https://ml.materialsproject.org/projects/matbench_perovskites.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_perovskites) |
| `matbench_mp_gap` | regression | `gap pbe` (eV) | structure | 106,113 | 1.3271 | [download](https://ml.materialsproject.org/projects/matbench_mp_gap.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_mp_gap) |
| `matbench_mp_is_metal` | classification | `is_metal` | structure | 106,113 | 0.4349 | [download](https://ml.materialsproject.org/projects/matbench_mp_is_metal.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_mp_is_metal) |
| `matbench_mp_e_form` | regression | `e_form` (eV/atom) | structure | 132,752 | 1.0059 | [download](https://ml.materialsproject.org/projects/matbench_mp_e_form.json.gz), [interactive](https://ml.materialsproject.org/projects/matbench_mp_e_form) |

You can click on the "download" links above to download the raw files, or alternatively click on "interactive" to explore the datasets on MPContribs (**note: you must be logged in to MPContribs to see the data**).

### Loading Matbench datasets with Matminer

We can easily load and examine any of these datasets using the exact same `load_dataset` function we have used throughout this lesson in matminer. Let's look at the smallest of the matbench datasets, `matbench_steels`, which has 312 samples and contains chemical compositions and yield strengths in MPa as the target property.

```python
from matminer.datasets import load_dataset

df = load_dataset("matbench_steels")
```

All Matbench datasets are reduced to the minimum possible format, including a single column for input (either structure or composition) and a single target property (a number for regression or a `bool` for binary classification).

If we use the `get_all_dataset_info` we can examine some more about where this data came from, how it was edited into this form, as well as the meaning of columns.

```python
from matminer.datasets import get_all_dataset_info

info = get_all_dataset_info("matbench_steels")

print(info)
```

## The Matbench Leaderboard and Benchmarking Code

We host an online benchmark leaderboard - similar to an "ImageNet" for materials science - at the following URL:

### [https://hackingmaterials.lbl.gov/matbench](https://hackingmaterials.lbl.gov/matbench)

Which contains comprehensive data on various SoTA algorithm's performance across tasks in Matbench. On the website you can find:

- **A general purpose leaderboard comparing only the most-widely applicable algorithms**
- **Individual per-task (per-dataset) leaderboards for comparing any ML model on a particular task**
- Comprehensive breakdowns of cross-validation performance, statistics, and metadata for every model
- **Access to individual sample predictions for each and every submission**

### General purpose leaderboard

| Task name | Samples | Algorithm | Verified MAE (unit) or ROCAUC | Notes |
|------------------|---------|-----------|----------------------|-------|
| matbench_steels | 312 | [AMMExpress v2020](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_automatminer_expressv2020) | **97.4929 (MPa)** |  |
| matbench_jdft2d | 636 | [AMMExpress v2020](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_automatminer_expressv2020) | **39.8497 (meV/atom)** |  |
| matbench_phonons | 1,265 | [CrabNet](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_CrabNet) | **55.1114 (cm^-1)** |  |
| matbench_expt_gap | 4,604 | [CrabNet](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_CrabNet) | **0.3463 (eV)** |  |
| matbench_dielectric | 4,764 | [AMMExpress v2020](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_automatminer_expressv2020) | **0.3150 (unitless)** |  |
| matbench_expt_is_metal | 4,921 | [AMMExpress v2020](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_automatminer_expressv2020) | **0.9209** |  |
| matbench_glass | 5,680 | [AMMExpress v2020](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_automatminer_expressv2020) | **0.8607** |  |
| matbench_log_gvrh | 10,987 | [AMMExpress v2020](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_automatminer_expressv2020) | **0.0874 (log10(GPa))** |  |
| matbench_log_kvrh | 10,987 | [AMMExpress v2020](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_automatminer_expressv2020) | **0.0647 (log10(GPa))** |  |
| matbench_perovskites | 18,928 | [CGCNN v2019](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_cgcnnv2019) | **0.0452 (eV/unit cell)** | structure required |
| matbench_mp_gap | 106,113 | [CrabNet](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_CrabNet) | **0.2655 (eV)** |  |
| matbench_mp_is_metal | 106,113 | [CGCNN v2019](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_cgcnnv2019) | **0.9520** | structure required |
| matbench_mp_e_form | 132,752 | [CGCNN v2019](https://hackingmaterials.github.io/matbench/Full%20Benchmark%20Data/matbench_v0.1_cgcnnv2019) | **0.0337 (eV/atom)** | structure required |

### Submit your algorithm

Although outside the scope of this lesson, you can submit your own algorithms to Matbench via Github and **appear on the leaderboards**! For more information on how to do this, see the [documentation on how to record results and submit to the leaderboard](https://hackingmaterials.lbl.gov/matbench/How%20To%20Use/1install/).