# Welcome to Session 01

This is a [Google Colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb) notebook file. Python programs are run directly in the browser—a great way to learn and use TensorFlow. To follow this tutorial, run the notebook in Google Colab by clicking the button at the top of this page.

1. In Colab, connect to a Python runtime: At the top-right of the menu bar, select *CONNECT*.
2. Run all the notebook code cells: Select *Runtime* > *Run all*.

```python
#@title ## Mounting Gdrive

USE_G_COLAB = True #@param {type:"boolean"}

if USE_G_COLAB:
    from google.colab import drive

    drive.mount('/content/drive', force_remount=True)
```

    Mounted at /content/drive

```python
#@title ## Project Root

root_dir = ''

if USE_G_COLAB:
    root_dir = '/content/drive/My Drive/workshops/2019_07_21/sessions_01/' #@param {type:"string"}
```

```python
#@title ## Installing requried packages

#@markdown ---
#@markdown - [TensorFlow 2.0-beta](https://www.tensorflow.org/install/gpu)
#@markdown - [Watermark](https://github.com/rasbt/watermark)
!pip install -q tensorflow-gpu==2.0.0-beta1
!pip install -qU watermark
```

```python
#@title ## Custom Matplotlib Style
mpl_style = "https://gist.githubusercontent.com/m3hrdadfi/af8aca01094afb7d3e5b46de9ad8d509/raw/871ec5d721a3b438c3c896718ea4aafc91ea9744/gadfly.mplstyle" #@param {type:"string"}

!wget -q $mpl_style -O /root/.config/matplotlib/matplotlibrc
```

```python
#@title ## General Paramas

#@markdown > A random seed is a number used to initialize a pseudorandom number generator. For a seed to be used in a pseudorandom number generator, it does not need to be random
RANDOM_SEED = 141 #@param {type:"integer"}
```

# Overview

$z_2 = XW_1$

$a_2 = f(z_2)$

$z_3 = a_2W_2$

$\hat{y} = softmax(z_3)$ # classification

*where*:
* $X$ = inputs | $\in \mathbb{R}^{NXD}$ ($D$ is the number of features)
* $W_1$ = 1st layer weights | $\in \mathbb{R}^{DXH}$ ($H$ is the number of hidden units in layer 1)
* $z_2$ = outputs from first layer's weights  $\in \mathbb{R}^{NXH}$
* $f$ = non-linear activation function
* $a_2$ = activation applied first layer's outputs | $\in \mathbb{R}^{NXH}$
* $W_2$ = 2nd layer weights | $\in \mathbb{R}^{HXC}$ ($C$ is the number of classes)
* $\hat{y}$ = prediction | $\in \mathbb{R}^{NXC}$ ($N$ is the number of samples)

This is a simple two-layer MLP.

* **Objective:**  Predict the probability of class $y$ given the inputs $X$. Non-linearity is introduced to model the complex, non-linear data.
* **Advantages:**
  * Can model non-linear patterns in the data really well.
* **Disadvantages:**
  * Overfits easily.
  * Computationally intensive as network increases in size.
  * Not easily interpretable.
* **Miscellaneous:** Future neural network architectures that we'll see use the MLP as a modular unit for feed forward operations (affine transformation (XW) followed by a non-linear operation).

## 7 Common Nonlinear Activation Functions  and How to Choose an Activation Function

#### SIGMOID / LOGISTIC

*ADVANTAGES*
- Smooth gradient, preventing “jumps” in output values.
- Output values bound between 0 and 1, normalizing the output of each neuron.
- Clear predictions—For X above 2 or below -2, tends to bring the Y value (the prediction) to the edge of the curve, very close to 1 or 0. This enables clear predictions.

*DISADVANTAGES*
- Vanishing gradient—for very high or very low values of X, there is almost no change to the prediction, causing a vanishing gradient problem. This can result in the network refusing to learn further, or being too slow to reach an accurate prediction.
- Outputs not zero centered.
- Computationally expensive

#### TANH / HYPERBOLIC TANGENT

*ADVANTAGES*
- Zero centered—making it easier to model inputs that have strongly negative, neutral, and strongly positive values.
- Otherwise like the Sigmoid function.

*DISADVANTAGES*
- Like the Sigmoid function

#### RELU (RECTIFIED LINEAR UNIT)

*ADVANTAGES*
- Computationally efficient—allows the network to converge very quickly
- Non-linear—although it looks like a linear function, ReLU has a derivative function and allows for backpropagation.

*DISADVANTAGES*
- The Dying ReLU problem—when inputs approach zero, or are negative, the gradient of the function becomes zero, the network cannot perform backpropagation and cannot learn.
- Like the Sigmoid function

#### LEAKY RELU

*ADVANTAGES*
- Prevents dying ReLU problem—this variation of ReLU has a small positive slope in the negative area, so it does enable backpropagation, even for negative input values
Otherwise like ReLU

*DISADVANTAGES*
Results not consistent—leaky ReLU does not provide consistent predictions for negative input values.

#### PARAMETRIC RELU

*ADVANTAGES*
- Allows the negative slope to be learned—unlike leaky ReLU, this function provides the slope of the negative part of the function as an argument. It is, therefore, possible to perform backpropagation and learn the most appropriate value of α.
Otherwise like ReLU

*DISADVANTAGES*
May perform differently for different problems.

$ f(x) = max( \alpha x,x)$

#### SOFTMAX

*ADVANTAGES*
- Able to handle multiple classes only one class in other activation functions—normalizes the outputs for each class between 0 and 1, and divides by their sum, giving the probability of the input value being in a specific class.
- Useful for output neurons—typically Softmax is used only for the output layer, for neural networks that need to classify inputs into multiple categories.

$ \frac{ e^{ z_{j} } }{   \sum_{k=1}^c e^{ z_{k} } } $

#### SWISH

Swish is a new, self-gated activation function discovered by researchers at Google. According to their paper, it performs better than ReLU with a similar level of computational efficiency. In experiments on ImageNet with identical models running ReLU and Swish, the new function achieved top -1 classification accuracy 0.6-0.9% higher.

# Training

*Steps*:

1. Randomly initialize the model's weights $W$ (we'll cover more effective initalization strategies in future lessons).
2. Feed inputs $X$ into the model to do the forward pass and receive the probabilities.
3. Compare the predictions $\hat{y}$ (ex.  [0.3, 0.3, 0.4]]) with the actual target values $y$ (ex. class 2 would look like [0, 0, 1]) with the objective (cost) function to determine loss $J$. A common objective function for classification tasks is cross-entropy loss.
  * $z_2 = XW_1$
  * $a_2 = max(0, z_2)$ # ReLU activation
  * $z_3 = a_2W_2$
  * $\hat{y} = softmax(z_3)$
  * $J(\theta) = - \sum_i y_i ln (\hat{y_i}) $
4. Calculate the gradient of loss $J(\theta)$ w.r.t to the model weights.
   * $ \frac{\partial{J}}{\partial{W_{2j}}} = a_2\hat{y}, \frac{\partial{J}}{\partial{W_{2y}}} = a_2(\hat{y}-1) $
   * $ \frac{\partial{J}}{\partial{W_1}} = \frac{\partial{J}}{\partial{\hat{y}}} \frac{\partial{\hat{y}}}{\partial{a_2}}  \frac{\partial{a_2}}{\partial{z_2}}  \frac{\partial{z_2}}{\partial{W_1}}  = W_2(\partial{scores})(\partial{ReLU})X $

5. Apply backpropagation to update the weights $W$ using gradient descent. The updates will penalize the probabiltiy for the incorrect classes (j) and encourage a higher probability for the correct class (y).
  * $W_i = W_i - \alpha\frac{\partial{J}}{\partial{W_i}}$
6. Repeat steps 2 - 4 until model performs well.

# Terminologies

## Neural Network Learning as Optimization
A deep learning neural network learns to map a set of inputs to a set of outputs from training data.

We cannot calculate the perfect weights for a neural network; there are too many unknowns. Instead, the problem of learning is cast as a search or optimization problem and an algorithm is used to navigate the space of possible sets of weights the model may use in order to make good or good enough predictions.

Typically, a neural network model is trained using the stochastic gradient descent optimization algorithm and weights are updated using the backpropagation of error algorithm.

The “gradient” in gradient descent refers to an error gradient. The model with a given set of weights is used to make predictions and the error for those predictions is calculated.

The gradient descent algorithm seeks to change the weights so that the next evaluation reduces the error, meaning the optimization algorithm is navigating down the gradient (or slope) of error.

Now that we know that training neural nets solves an optimization problem, we can look at how the error of a given set of weights is calculated.

## What Is a Loss Function and Loss?

In the context of an optimization algorithm, the function used to evaluate a candidate solution (i.e. a set of weights) is referred to as the objective function.

We may seek to maximize or minimize the objective function, meaning that we are searching for a candidate solution that has the highest or lowest score respectively.

Typically, with neural networks, we seek to minimize the error. As such, the objective function is often referred to as a cost function or a loss function and the value calculated by the loss function is referred to as simply “loss.”
The cost or loss function has an important job in that it must faithfully distill all aspects of the model down into a single number in such a way that improvements in that number are a sign of a better model.
We will review best practice or default values for each problem type with regard to the output layer and loss function.

## Regression Problem

A problem where you predict a real-value quantity.

Output Layer Configuration: One node with a linear activation unit.
Loss Function: Mean Squared Error (MSE).

## Binary Classification Problem

A problem where you classify an example as belonging to one of two classes.

The problem is framed as predicting the likelihood of an example belonging to class one, e.g. the class that you assign the integer value 1, whereas the other class is assigned the value 0.

Output Layer Configuration: One node with a sigmoid activation unit.
Loss Function: Cross-Entropy, also referred to as Logarithmic loss.

## Multi-Class Classification Problem

A problem where you classify an example as belonging to one of more than two classes.

The problem is framed as predicting the likelihood of an example belonging to each class.

Output Layer Configuration: One node for each class using the softmax activation function.
Loss Function: Cross-Entropy, also referred to as Logarithmic loss.

## How to Implement Loss Functions

In order to make the loss functions concrete, this section explains how each of the main types of loss function works.

## Mean Squared Error Loss

Mean Squared Error loss, or MSE for short, is calculated as the average of the squared differences between the predicted and actual values.

The result is always positive regardless of the sign of the predicted and actual values and a perfect value is 0.0. The loss value is minimized, although it can be used in a maximization optimization process by making the score negative.

## Cross-Entropy Loss (or Log Loss)

Cross-entropy loss is often simply referred to as “cross-entropy,” “logarithmic loss,” “logistic loss,” or “log loss” for short.

Each predicted probability is compared to the actual class output value (0 or 1) and a score is calculated that penalizes the probability based on the distance from the expected value. The penalty is logarithmic, offering a small score for small differences (0.1 or 0.2) and enormous score for a large difference (0.9 or 1.0).

Cross-entropy loss is minimized, where smaller values represent a better model than larger values. A model that predicts perfect probabilities has a cross entropy or log loss of 0.0.

Cross-entropy for a binary or two class prediction problem is actually calculated as the average cross entropy across all examples.

## An overview of gradient descent optimization algorithms

Gradient descent is one of the most popular algorithms to perform optimization and by far the most common way to optimize neural networks. At the same time, every state-of-the-art Deep Learning library contains implementations of various algorithms to optimize gradient descent. These algorithms, however, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by.

## Gradient descent variants

There are three variants of gradient descent, which differ in how much data we use to compute the gradient of the objective function. Depending on the amount of data, we make a trade-off between the accuracy of the parameter update and the time it takes to perform an update.

## Batch gradient descent

Vanilla gradient descent, aka batch gradient descent, computes the gradient of the cost function w.r.t. to the parameters θ for the entire training dataset:

$ \theta =  \theta - \eta    \nabla _{ \theta } J( \theta )$

    We then update our parameters in the opposite direction of the gradients with the learning rate determining how big of an update we perform. Batch gradient descent is guaranteed to converge to the global minimum for convex error surfaces and to a local minimum for non-convex surfaces.
     As we need to calculate the gradients for the whole dataset to perform just one update, batch gradient descent can be very slow and is intractable for datasets that don't fit in memory. Batch gradient descent also doesn't allow us to update our model online, i.e. with new examples on-the-fly.

## Stochastic gradient descent

Stochastic gradient descent (SGD) in contrast performs a parameter update for each training example x(i) and label y(i):

$\theta =  \theta - \eta    \nabla _{ \theta } J( \theta;  x^{i};  y^{i})$

 Batch gradient descent performs redundant computations for large datasets, as it recomputes gradients for similar examples before each parameter update. SGD does away with this redundancy by performing one update at a time. It is therefore usually much faster and can also be used to learn online.
 While batch gradient descent converges to the minimum of the basin the parameters are placed in, SGD's fluctuation, on the one hand, enables it to jump to new and potentially better local minima. On the other hand, this ultimately complicates convergence to the exact minimum, as SGD will keep overshooting. However, it has been shown that when we slowly decrease the learning rate, SGD shows the same convergence behaviour as batch gradient descent, almost certainly converging to a local or the global minimum for non-convex and convex optimization respectively.

## Mini-batch gradient descent

Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of n training examples:

$\theta =  \theta - \eta    \nabla _{ \theta } J( \theta;  x^{i:i+n};  y^{i:i+n})$

This way, it a) reduces the variance of the parameter updates, which can lead to more stable convergence; and b) can make use of highly optimized matrix optimizations common to state-of-the-art deep learning libraries that make computing the gradient w.r.t. a mini-batch very efficient. Common mini-batch sizes range between 50 and 256, but can vary for different applications. Mini-batch gradient descent is typically the algorithm of choice when training a neural network and the term SGD usually is employed also when mini-batches are used.

## Gradient descent optimization algorithms
- Momentum
- Nesterov accelerated gradient (NAG)
- Adagrad
- Adadelta
- RMSprop
- Adam
- AdaMax
- Nadam
- AMSGrad

## Which optimizer to use?

So, which optimizer should you now use? If your input data is sparse, then you likely achieve the best results using one of the adaptive learning-rate methods. An additional benefit is that you won't need to tune the learning rate but likely achieve the best results with the default value.

In summary, RMSprop is an extension of Adagrad that deals with its radically diminishing learning rates. It is identical to Adadelta, except that Adadelta uses the RMS of parameter updates in the numinator update rule. Adam, finally, adds bias-correction and momentum to RMSprop. Insofar, RMSprop, Adadelta, and Adam are very similar algorithms that do well in similar circumstances. It is shown that its bias-correction helps Adam slightly outperform RMSprop towards the end of optimization as gradients become sparser. Insofar, Adam might be the best overall choice.

## Hyperparameters

Model optimization is one of the toughest challenges in the implementation of machine learning solutions. Entire branches of machine learning and deep learning theory have been dedicated to the optimization of models. Typically, we think about model optimization as a process of regularly modifying the code of the model in order to minimize the testing error. However, deep learning optimization often entails fine tuning elements that live outside the model but that can heavily influence its behavior. Deep learning often refers to those hidden elements as hyperparameters as they are one of the most critical components of any machine learning application.

#### Some Examples of Hyperparameters

The number and diversity of hyperparameters in machine learning algorithms is very specific to each model. However, there some classic hyperparameters that we should always keep our eyes on and that should help you think about this aspect of machine learning solutions:
· Learning Rate: The mother of all hyperparameters, the learning rate quantifies the learning progress of a model in a way that can be used to optimize its capacity.
· Number of Hidden Units: A classic hyperparameter in deep learning algorithms, the number of hidden units is key to regulate the representational capacity of a model.
· Convolution Kernel Width: In convolutional Neural Networks(CNNs), the Kernel Width influences the number of parameters in a model which, in turns, influences its capacity.

## Metrics to Evaluate your Machine Learning Algorithm

Evaluating your machine learning algorithm is an essential part of any project. Your model may give you satisfying results when evaluated using a metric say accuracy_score but may give poor results when evaluated against other metrics such as logarithmic_loss or any other such metric. Most of the times we use classification accuracy to measure the performance of our model, however it is not enough to truly judge our model. In this post, we will cover different types of evaluation metrics available.

## Classification Accuracy

Classification Accuracy is what we usually mean, when we use the term accuracy. It is the ratio of number of correct predictions to the total number of input samples.

$Accuracy = \frac{Number of Correct predictions}{Total Number of Predictions made}$

It works well only if there are equal number of samples belonging to each class.
For example, consider that there are 98% samples of class A and 2% samples of class B in our training set. Then our model can easily get 98% training accuracy by simply predicting every training sample belonging to class A.
When the same model is tested on a test set with 60% samples of class A and 40% samples of class B, then the test accuracy would drop down to 60%. Classification Accuracy is great, but gives us the false sense of achieving high accuracy.
The real problem arises, when the cost of misclassification of the minor class samples are very high. If we deal with a rare but fatal disease, the cost of failing to diagnose the disease of a sick person is much higher than the cost of sending a healthy person to more tests.

## Confusion Matrix

Confusion Matrix as the name suggests gives us a matrix as output and describes the complete performance of the model.
Lets assume we have a binary classification problem. We have some samples belonging to two classes : YES or NO. Also, we have our own classifier which predicts a class for a given input sample. On testing our model on 165 samples ,we get the following result.

  $LogLoss = - \frac{1}{n} \sum\limits_{i=1}^n [y_i \cdot log_e(\hat{y_i}) + (1-y_i) \cdot log_e(1-\hat{y_i}) ]$

where,
y_ij, indicates whether sample i belongs to class j or not
p_ij, indicates the probability of sample i belonging to class j
Log Loss has no upper bound and it exists on the range Log Loss nearer to 0 indicates higher accuracy, whereas if the Log Loss is away from 0 then it indicates lower accuracy.
In general, minimising Log Loss gives greater accuracy for the classifier.

| n=165 | Predicted: NO | Predicted: YES |
|---|---|---|
| Actual: NO | 50 | 10 |
| Actual: YES | 5 | 100 |

There are 4 important terms :
- True Positives : The cases in which we predicted YES and the actual output was also YES.
- True Negatives : The cases in which we predicted NO and the actual output was NO.
- False Positives : The cases in which we predicted YES and the actual output was NO.
- False Negatives : The cases in which we predicted NO and the actual output was YES.
Accuracy for the matrix can be calculated by taking average of the values lying across the “main diagonal” i.e

$Accuracy = \frac{TruePositive+ TrueNegative}{Total Number of Samples}$

 $Accuracy = \frac{100+50}{165} = 0.91$

Confusion Matrix forms the basis for the other types of metrics.

## Area Under Curve

Area Under Curve(AUC) is one of the most widely used metrics for evaluation. It is used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example. Before defining AUC, let us understand two basic terms :
- True Positive Rate (Sensitivity) : True Positive Rate is defined as TP/ (FN+TP). True Positive Rate corresponds to the proportion of positive data points that are correctly considered as positive, with respect to all positive data points.

$TruePositiveRate = \frac{TruePositive}{FalseNegative+TruePositive}$

- False Positive Rate (Specificity) : False Positive Rate is defined as FP / (FP+TN). False Positive Rate corresponds to the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points.

$FalsePositiveRate = \frac{FalsePositive}{TrueNegative+FalsePositive}$

  False Positive Rate and True Positive Rate both have values in the range [0, 1]. FPR and TPR bot hare computed at threshold values such as (0.00, 0.02, 0.04, …., 1.00) and a graph is drawn. AUC is the area under the curve of plot False Positive Rate vs True Positive Rate at different points in [0, 1].

As evident, AUC has a range of [0, 1]. The greater the value, the better is the performance of our model.

## F1 Score

F1 Score is used to measure a test’s accuracy
F1 Score is the Harmonic Mean between precision and recall. The range for F1 Score is [0, 1]. It tells you how precise your classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances).
High precision but lower recall, gives you an extremely accurate, but it then misses a large number of instances that are difficult to classify. The greater the F1 Score, the better is the performance of our model. Mathematically, it can be expressed as :

$F1 = 2*\frac{1}{\frac{1}{precision}+\frac{1}{recall}}$

- Precision : It is the number of correct positive results divided by the number of positive results predicted by the classifier.

$Precision = 2*\frac{TruePositive}{TruePositive+FalsePositive}$

Recall : It is the number of correct positive results divided by the number of all relevant samples (all samples that should have been identified as positive).

$Recall = 2*\frac{TruePositive}{TruePositive+FalsePositive}$

# Preparation

```python
#@title ## Import requried packages
from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf

import numpy as np

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

import matplotlib as mpl
import matplotlib.pyplot as plt

import itertools
from datetime import datetime
from tqdm import tqdm
import os

%matplotlib inline
mpl.rc_file(mpl.matplotlib_fname())
```

```python
%reload_ext watermark
%watermark -m -n -p tensorflow,numpy,matplotlib,sklearn -g
```

    Sun Jul 21 2019

    tensorflow 2.0.0-beta1
    numpy 1.16.4
    matplotlib 3.0.3
    sklearn 0.21.2

    compiler   : GCC 8.0.1 20180414 (experimental) [trunk revision 259383
    system     : Linux
    release    : 4.14.79+
    machine    : x86_64
    processor  : x86_64
    CPU cores  : 2
    interpreter: 64bit
    Git hash   :

```python
#@title ## Random seed

np.random.seed(RANDOM_SEED)
```

# Data

We're going to first generate some non-linear data for a classification task.

**A spiral**: is a curve which emanates from a point, moving farther away as it revolves around the point.

\begin{align}
 Spiral =\begin{cases} X_{1\theta} = r_{\theta} \cos(\theta) \\X_{2\theta} = r_{\theta} \sin(\theta)\end{cases}
\end{align}

```python
#@title ## Generate non-linear data [Spiral](https://en.wikipedia.org/wiki/Spiral)

def generate_data(num_samples, dimensions, num_classes):
    """ Generate non-linear dataset.

    Args:
        num_samples (int): number of samples which we want to produce.
        dimensions (int): the dimension of the data which we plan to produce ex. 2d.
        num_classes (int): number of classes which the samples are going to generate ex. 2#.

    Examples:
        >>> x, y = generate_data(2, 2, 2)
        (array([[ 0.        ,  0.        ],
        [-0.712528  , -0.70164368],
        [-0.        , -0.        ],
        [ 0.90006129, -0.43576333]]), array([0, 0, 1, 1], dtype=uint8))

    Returns:
        x (ndarray): a numpy array in a shape like this (num_samples, dimensions).
        y (ndarray): a numpy array in a shape like this (num_samples, num_classes).
    """

    x_origin = np.zeros((num_samples * num_classes, dimensions))
    y = np.zeros(num_samples * num_classes, dtype='uint8')

    for i in tqdm(range(num_classes), position=0):
        idx = range(num_samples * i, num_samples * (i + 1))
        radius = np.linspace(0.0, 1, num_samples)
        theta = np.linspace(i * 4, (i + 1) * 4, num_samples) + np.random.randn(num_samples) * 0.2

        x_origin[idx] = np.c_[radius * np.sin(theta), radius * np.cos(theta)]
        y[idx] = i

    x = np.hstack([x_origin])

    return x, y
```

```python
#@title ## Generate x, y

num_samples = 500 #@param {type:"integer"}
dimensions = 2 #@param {type:"integer"}
num_classes = 3 #@param {type:"integer"}

x, y = generate_data(num_samples, dimensions, num_classes)

print()
print('x: %s' % str(x.shape))
print('y: %s' % str(y.shape))
```

    100%|██████████| 3/3 [00:00<00:00, 1181.49it/s]

    x: (1500, 2)
    y: (1500,)

```python
#@title ## Visualize data

xfig = 12.0 #@param {type:"number"}
yfig = 8.0 #@param {type:"number"}

plt.figure(figsize=(xfig, yfig))
plt.title('Generated non-linear data')
plt.scatter(x[:, 0], x[:, 1], c=y)
plt.show()
```

## One-hot Encoding

Consider an array of 5 labels out of a set of 3 classes {0, 1, 2}:

```python
> labels
array([0, 2, 1, 2, 0])
```

**`to_categorical`** converts this into a matrix with as many
columns as there are classes. The number of rows
stays the same.

```python
> to_categorical(labels)
array([[ 1.,  0.,  0.],
       [ 0.,  0.,  1.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.],
       [ 1.,  0.,  0.]], dtype=float32)
```

---

**to_categorical**

```
tf.keras.utils.to_categorical(y, num_classes=None, dtype='float32')
```
Converts a class vector (integers) to binary class matrix.

**Arguments:**

- **`y`**: class vector to be converted into a matrix (integers from 0 to num_classes).
- **`num_classes`**: total number of classes.
- **`dtype`**: The data type expected by the input, as a string (float32, float64, int32...)

```python
#@title ## Preprocessing
```

# Linear Model

Before we get to our neural network, we're going to implement a linear model (logistic regression). We want to see why linear models won't suffice for our dataset.

```python
#@title ## Build linear model [Logistic Model](https://en.wikipedia.org/wiki/Multinomial_logistic_regression)

def build_linear(n_units, n_features):
    model = None
    return model
```

```python
#@title ## Create linear model
```

```python
#@title ## Fit linear model.
```

```python
#@title ## Evaluating
```

```python
#@title ## Plotting
```

```python
#@title ## Multiclass decision boundary

def plot_mc_decision_boundary(model, x, y, steps=1000):
    """ Plot multiclass decision boundary,

    Args:
        model (keras.Model): a keras model.
        x (ndarray): a numpy array.
        y (ndarray): a numpy array.
        steps (integer): number of linear spaces.

    Returns:
        None
    """
    x_min, x_max = x[:, 0].min() - 0.1, x[:, 0].max() + 0.1
    y_min, y_max = x[:, 1].min() - 0.1, x[:, 1].max() + 0.1

    x_span = np.linspace(x_min, x_max, steps)
    y_span = np.linspace(y_min, y_max, steps)
    xx, yy = np.meshgrid(x_span, y_span)

    y_pred = model.predict(np.c_[xx.ravel(), yy.ravel()])
    y_pred = np.argmax(y_pred, axis=1)
    y_pred = y_pred.reshape(xx.shape)

    plt.contourf(xx, yy, y_pred, alpha=0.5)
    plt.scatter(x[:, 0], x[:, 1], c=y)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())

    return None
```

```python
#@title ## Visualize the decision boundary
```

```python
#@title ## Plot confusion matrix

def plot_confusion_matrix(cm, classes):
    """ Plot confusion matrix

    Args:
        cm (ndarray): the input of confusion matrix.
        classes (integer): number of classes.

    Returns:
        None
    """
    cmap=plt.cm.Blues
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title("Confusion Matrix")
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)
    plt.grid(False)

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt), horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.tight_layout()

    return None
```

## Classification Report

`sklearn.metrics.classification_report`

Build a text report showing the main classification metrics

**Params:**
- `y_true` : 1d array-like, or label indicator array / sparse matrix
- `y_pred` : 1d array-like, or label indicator array / sparse matrix

**Output:**
```bash
              precision    recall  f1-score   support

           1       1.00      0.67      0.80         3
           2       0.00      0.00      0.00         0
           3       0.00      0.00      0.00         0

   micro avg       1.00      0.67      0.80         3
   macro avg       0.33      0.22      0.27         3
weighted avg       1.00      0.67      0.80         3
```

```python
#@title ## Confusion matrix
```

# Non-linear Model

Now let's see how the MLP performs on the data. Note that the only difference is the addition of the non-linear activation function (we use $ReLU$ which is just $max(0, z))$.

```python
#@title ## Build non-linear model [MLP Model](https://en.wikipedia.org/wiki/Multilayer_perceptron)

def build_non_linear(n_classes, n_units, n_features):
    model = None

    return model
```

```python
#@title ## Create non-linear model
```

```python
#@title ## Fit non-linear model.
```

```python
#@title ## Evaluating
```

```python
#@title ## Plotting
```

```python
#@title ## Visualize the decision boundary
```

```python
#@title ## Confusion matrix
```

# Overfitting

Though neural networks are great at capturing non-linear relationships they are highly susceptible to overfitting to the training data and failing to generalize on test data. Just take a look at the example below where we generate completely random data and are able to fit a model with [$2*N*C + D$](https://arxiv.org/abs/1611.03530) hidden units. The training performance is great but the overfitting leads to very poor test performance. We'll be covering strategies to tackle overfitting in future lessons.

Let me define overfitting more formally. Overfitting refers to a model that models the “training data” too well. Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data.

# How do you know your NN is overfitting?
In practice, detecting that our model is overfitting is difficult. It’s not uncommon that our trained model is already in production and then we start to realize that something is wrong. In fact, it is only by confronting new data that you can make sure that everything is working properly. However, during the training we should try to reproduce the real conditions as much as possible. For this reason, it is good practice to divide our dataset into three parts - training set, dev set (also known as cross-validation or hold-out) and test set. Our model learns by seeing only the first of these parts. Hold-out is used to track our progress and draw conclusions to optimise the model. While, we use a test set at the end of the training process to evaluate the performance of our model. Using completely new data allows us to get an unbiased opinion on how well our algorithm works.

It is very important to make sure that your cross-validation and test set come from the same distribution as well as that they accurately reflect data that we expect to receive in the future. Only then we can be sure that the decisions we make during the learning process bring us closer to a better solution. I know what you are thinking about… “How should I divide my dataset?” Until recently, one of the most frequently recommended splits was 60/20/20, but in the era of big data, when our dataset can count millions of entries, those fixed proportions are no longer appropriate. In short, everything depends on the size of the dataset we work with. If we have millions of entries at our disposal, perhaps it would be better idea to divide them in 98/1/1 ratio. Our dev and test sets should be simply large enough to give us high confidence in the performance of our model.

# Bias and Variance
To give us a better understanding of this some how complex issue, we will use a simple example, that hopefully allow us to develop a valuable intuition. Our dataset consisting of two classes of points, located in a two-dimensional space.

The first model in the top right corner is very simple and therefore has a high bias, i.e. it is not able to find all significant links between features and result. This is understandable - our dataset has a lot of noise in it, and therefore simple linear regression, is not able to deal with it effectively. Neural networks performed much better, but the first one (shown in the lower left corner) fitted into the data too closely, which made it work significantly worse on the hold-out set. This means that it has a high variance - it fits into the noise and not into the intended output. This undesirable effect was mitigated, in the last model, by the use of regularisation.

# Ways to prevent overfitting

### L1 and L2 Regularizations

One of the first methods we should try when we need to reduce overfitting is regularisation. It involves adding an extra element to the loss function, which punishes our model for being too complex or, in simple words, for using too high values in the weight matrix. This way we try to limit its flexibility, but also encourage it to build solutions based on multiple features. Two popular versions of this method are L1 - Least Absolute Deviations (LAD) and L2 - Least Square Errors (LS). Equations describing these regularisations are given below.
In most cases the use of L1 is preferable, because it reduces the weight values of less important features to zero, very often eliminating them completely from the calculations. In a way, it is a built-in mechanism for automatic featur selection. Moreover, L2 does not perform very well on datasets with a large number of outliers. The use of value squares results in the model minimizing the impact of outliers at the expense of more popular examples.

$$
\begin{array}{rlrl}{J_{L 1}(W, b)} & {=\frac{1}{m} \sum_{i=1}^{m} L\left(\hat{y}^{(i)}, y^{(i)}\right)+\lambda\|w\|_{1}} & {\|w\|_{1}} & {=\sum_{j=1}^{n_{x}}\left|w_{j}\right|} \\ {J_{L 2}(W, b)} & {=\frac{1}{m} \sum_{i=1}^{m} L\left(\hat{y}^{(i)}, y^{(i)}\right)+\lambda\|w\|_{2}} & {\|w\|_{2}} & {=\sum_{j=1}^{n_{x}} w_{j}^{2}}\end{array}
$$

Increasing the λ value also increases the regularisation effect. Models with a very low λ coefficient value are very “turbulent”.

# Dropout
Another very popular method of regularization of neural networks is dropout. This idea is actually very simple - every unit of our neural network (except those belonging to the output layer) is given the probability p of being temporarily ignored in calculations. Hyper parameter p is called dropout rate and very often its default value is set to 0.5. Then, in each iteration, we randomly select the neurons that we drop according to the assigned probability. As a result, each time we work with a smaller neural network. The visualization below shows an example of a neural network subjected to a dropout. We can see how in each iteration random neurons from second and fourth layer are deactivated.

# Early Stopping
The graph below shows the change in accuracy values calculated on the test and cross-validation sets during subsequent iterations of learning process. We see right away that the model we get at the end is not the best we could have possibly create. To be honest, it is much worse than what we have had after 150 epochs. Why not interrupt the learning process before the model starts overfitting? This observation inspired one of the popular overfitting reduction method, namely early stopping.

```python
#@title ## Generate random x, y

ov_num_samples = 40 #@param {type:"integer"}
ov_dimensions = 2 #@param {type:"integer"}
ov_num_classes = 3 #@param {type:"integer"}

ov_x = np.random.randn(ov_num_samples * ov_num_classes, ov_dimensions)
ov_y = np.array([[i] * ov_num_samples for i in range(ov_num_classes)])
ov_y = ov_y.flatten()

print()
print('x: %s' % str(ov_x.shape))
print('y: %s' % str(ov_y.shape))
```

    x: (120, 2)
    y: (120,)

```python
#@title ## Preprocessing
```

```python
#@title ## Create non-linear model
```

```python
#@title ## Fit overfie model.
```

```python
#@title ## Evaluating
```

```python
#@title ## Plotting
```

```python
#@title ## Visualize the decision boundary
```

```python
#@title ## Confusion matrix
```

# Regularization

```python
#@title ## Build non-linear model [MLP Model](https://en.wikipedia.org/wiki/Multilayer_perceptron)

def build_non_linear_with_regularization(n_classes, n_units, n_features):
    model = None

    return model
```

```python
#@title ## Create non-linear model
```

```python
#@title ## Fit non-linear model with regularization.
```

```python
#@title ## Evaluating
```

```python
#@title ## Plotting
```

```python
#@title ## Visualize the decision boundary
```

```python
#@title ## Confusion matrix
```

# Dropout

A great technique to overcome overfitting is to increase the size of your data but this isn't always an option. Fortuntely, there are methods like regularization and dropout that can help create a more robust model. We've already seen regularization and we can easily add it in our optimizer to use it in PyTorch.

Dropout is a technique (used only during training) that allows us to zero the outputs of neurons. We do this for p% of the total neurons in each layer and it changes every batch. Dropout prevents units from co-adapting too much to the data and acts as a sampling strategy since we drop a different set of neurons each time.

```python
#@title ## Build non-linear model [MLP Model](https://en.wikipedia.org/wiki/Multilayer_perceptron)

def build_non_linear_with_dropout(n_classes, n_units, n_features):
    model = None

    return model
```

```python
#@title ## Create non-linear model
```

```python
#@title ## Fit non-linear model with dropout.
```

```python
#@title ## Evaluating
```

```python
#@title ## Plotting
```

```python
#@title ## Visualize the decision boundary
```

```python
#@title ## Confusion matrix
```

# Visualization

```python
#@title ## Build non-linear model [MLP Model](https://en.wikipedia.org/wiki/Multilayer_perceptron)

def build_non_linear_v2(n_classes, n_units, n_features):
    model = None

    return model
```

```python
#@title ## Create non-linear model
```

```python
#@title # Summary of the model
```

```python
#@title ## Model Arch
```

```python
#@title ## Tensorboard callbacks
```

```python
#@title ## Fit non-linear model.
```

```python
#@title ## Plotting
```

```python
#@title ## Visualize the decision boundary
```

```python
#@title ## Confusion matrix
```

# Tensorboard

```python
# %load_ext tensorboard
```

```python
# %tensorboard --logdir logs/scalars
```

```python

```


# Stochastic Processes:
Data Analysis and Computer Simulation

# Brownian motion 3: data analysis

# 3. Interacting Brownian particles

# 3.1. Necessary changes for interacting Brownian particles

## Periodic boundary conditions

## Inter-particle interaction

$$
U=\sum_{r_{ij}<2\sigma}\phi(r_{ij})=\sum_{r_{ij}<2\sigma}4\epsilon\left(\frac{\sigma}{r_{ij}} \right)^{12},\ \ \ \
\mathbf{F}_{i}=-\frac{\partial U}{\partial \mathbf{r}_{ij}}
=\sum_{r_{ij}<2\sigma}48\epsilon\left(\frac{\sigma}{r_{ij}} \right)^{12} \frac{\mathbf{r}_{ij}}{r^2_{ij}}\tag{I1, I2}
$$

$$
\mathbf{r}_{ij} = \mathbf{r}_j - \mathbf{r}_i,\ \ \ r_{ij}=|\mathbf{r}_{ij}|
$$

# 3.2. Simulation code for interacting Brownian particles with animation

## Import libraries

```python
% matplotlib nbagg
import numpy as np # import numpy library as np
import matplotlib.pyplot as plt # import pyplot library as plt
import matplotlib.animation as animation # import animation modules from matplotlib
from   mpl_toolkits.mplot3d import Axes3D # import Axes3D from mpl_toolkits.mplot3d
from   numpy import newaxis # import newaxis used for inter-particle force calculations
plt.style.use('ggplot') # use "ggplot" style for graphs
```

## Define `init` function for `FuncAnimation`

```python
def init():
    global R,V,W,F,Rs,Vs,Ws,time # define global variables
    initconf()          # create random particle configuration without overlaps
    V[:,:] = 0.0        # initialize all the variables to zero
    W[:,:] = 0.0        # initialize all the variables to zero
    F[:,:] = 0.0        # initialize all the variables to zero
    Rs[:,:,:] = 0.0     # initialize all the variables to zero
    Rs[0,:,:] = R[:,:]  # store initial particle positions in Rs
    Vs[:,:,:] = 0.0     # initialize all the variables to zero
    Ws[:,:,:] = 0.0     # initialize all the variables to zero
    time[:]   = 0.0     # initialize all the variables to zero
    title.set_text(r'') # empty title
    line.set_data([],[]) # set line data to show the trajectory of particle n in 2d (x,y)
    line.set_3d_properties([]) # add z-data separately for 3d plot
    particles.set_data([],[]) # set position current (x,y) position data for all particles
    particles.set_3d_properties([]) # add current z data of particles to get 3d plot
    return particles,title,line # return listed objects that will be drawn by FuncAnimation
```

## Define `animate` function for `FuncAnimation`

The Euler method used in previous `animate` function is replaced with the Leap-Frog method in the following new `animate` function because of the numerical instability taking place with the former method for the present case of interacting Brownian particles.

```python
def animate(i):
    global R,V,W,F,Rs,Vs,Ws,time # define global variables
    time[i]=i*dt # store time in each step in an array time
    particleforces() # compute inter-particle force F by examining all nump^2 pairs
    W = std*np.random.randn(nump,dim) # generate an array of random forces accordingly to Eqs.(F10) and (F11)
#    V = V*(1-zeta/m*dt)+F/m*dt+W/m # update velocity via Eq.(I11) using the L-F method
    V = (V*(1-zeta/2/m*dt)+F/m*dt+W/m)/(1+zeta/2/m*dt) # update velocity via Eq.(I11) using the L-F method
    R = R+V*dt # update position via Eq.(I12) using the L-F method
    Rs[i,:,:]=R # accumulate particle positions at each step in an array Rs
    Vs[i,:,:]=V # accumulate particle velocitys at each step in an array Vs
    Ws[i,:,:]=W # accumulate random forces at each step in an array Ws
    title.set_text(r"t = "+str(time[i])+"/"+str((nums-1)*dt))  # set the title to display the current time
    line.set_data(Rs[:i+1,n,0],Rs[:i+1,n,1]) # set the line in 2D (x,y)
    line.set_3d_properties(Rs[:i+1,n,2]) # add z axis to set the line in 3D
    particles.set_data(pbc(R[:,0],box[0]), pbc(R[:,1],box[1])) # set the current position of all the particles in 2d (x,y) with PBC
    particles.set_3d_properties(pbc(R[:,2],box[2])) # add z axis to set the particle in 3D with PBC
    return particles,title,line # return listed objects that will be drawn by FuncAnimation
```

## Newly defined functions for interacting Brownian particles

```python
def pbc(r, lbox): # enforce Periodic Boundary Conditions for all positions
    return np.fmod(r+lbox,lbox)
def distance(r1,r2,lbox): # Compute distance vector R2 - R1 with PBC
    return r2-r1-np.around((r2-r1)/lbox)*lbox
def fij(r2,rij): # calculate Fij=dU/drij
#    f=-24*eps*(2*(r2/sig**2)**(-6)-(r2/sig**2)**(-3))/r2*rij # Lennard-Jones potential
    f=-48*eps*((r2/sig**2)**(-6))/r2*rij # soft-core potential
    return f
def particleforces(): # compute inter-particle force F by examining all nump^2 pairs
    global F
    F[:,:] = 0.0
    for n in range(nump): # repeat below for all particles
        rij = distance(R[n,:], R, box) # distance vectors rij=R_i-R_j for all i (1 <= i <= nump)
        r2  = np.linalg.norm(rij, axis=1)**2 # square distance rij**2
        nei = (r2 < (2.0*sig)**2)  # list neighbor particles of j
        nei[n] = False # ignore self pair (i=j)
        F[n,:] = np.sum(fij(r2[nei, newaxis], rij[nei,:]), axis=0) # total force on particle j
def initconf():  # create random particle configuration without overlapping
    global R,V,W,F,Rs,Vs,Ws,time
    for n in range(nump): # repeat below from n=0 to nump-1
        nn=0 # set overlap true to perform while loop below for the n-th particle
        while nn == 0: # repeat the loop below while overlap is true (nn==0)
            R[n,:]=np.random.rand(dim)*box # generate a position candidate for n using uniform random number.
            nn = 1 # initialize overlap as false
            for l in range(n): # examine overlap generated positions (from l=0 to n-1)
                rij = distance(R[n,:],R[l,:],box) # calculate distance vector rij=R_l-R_n
                r2  = np.linalg.norm(rij)**2 # calculate the squared distance rij**2
                if r2 < (0.90*sig)**2: # check if the distance is smaller than threshold
                # Yes -> perform below (nn=0) -> repeat while loop, No (nn=1) -> exit while loop
                    nn = 0 # set overlap true
```

## Set parameters and initialize variables

```python
dim  = 3    # system dimension (x,y,z)
nump = 100  # number of interacting Brownian particles to simulate
nums = 4096 # number of simulation steps
dt   = 0.01 # set time increment, \Delta t
zeta = 1.0  # set friction constant, \zeta
m    = 1.0  # set particle mass, m
kBT  = 1.0  # set thermal energy, k_B T
std  = np.sqrt(2*kBT*zeta*dt) # calculate std for \Delta W via Eq.(F11)
sig  = 1.0  # unit of length of inter-particle potential
eps  = 1.0  # unit of energy inter-particle potential
vf   = 0.001 ##volume fraction of particles < 0.45
boxl = np.power(nump*np.pi/6/vf,1/3) # calculate the side length of unit cell
print('Volume fraction =',vf,'  boxl =',boxl) # print vf and boxl
box  = np.array([boxl,boxl,boxl])*sig # set array box[dim]
np.random.seed(0) # initialize random number generator with a seed=0
R = np.zeros([nump,dim]) # array to store current positions and set initial condition Eq.(F12)
V = np.zeros([nump,dim]) # array to store current velocities and set initial condition Eq.(F12)
W = np.zeros([nump,dim]) # array to store current random forcces
F = np.zeros([nump,dim]) # rray to store current particle orcces
Rs = np.zeros([nums,nump,dim]) # array to store positions at all steps
Vs = np.zeros([nums,nump,dim]) # array to store velocities at all steps
Ws = np.zeros([nums,nump,dim]) # array to store random forces at all steps
time = np.zeros([nums]) # an array to store time at all steps
```

    Volume fraction = 0.001   boxl = 37.4110192682

## Perform and animate the simulation using `FuncAnimation`

```python
fig = plt.figure(figsize=(10,10)) # set fig with its size 10 x 10 inch
ax = fig.add_subplot(111,projection='3d') # creates an additional axis to the standard 2D axes
ax.set_xlim(0.0,box[0]) # set x-range
ax.set_ylim(0.0,box[1]) # set y-range
ax.set_zlim(0.0,box[2]) # set z-range
ax.set_xlabel(r"x",fontsize=20) # set x-lavel
ax.set_ylabel(r"y",fontsize=20) # set y-lavel
ax.set_zlabel(r"z",fontsize=20) # set z-lavel
ax.view_init(elev=12,azim=120)  # set view point
particles, = ax.plot([],[],[],linestyle='None',color='r',marker='o',ms=250/box[0],alpha=0.5) # define object particles
title = ax.text(0.,0.,0.,r'',transform=ax.transAxes,va='center') # define object title
line, = ax.plot([],[],[],'b',lw=2,alpha=0.8) # define object line
n = 0  # trajectry line is plotted for the n-th particle
anim = animation.FuncAnimation(fig,func=animate,init_func=init,
            frames=nums,interval=5,blit=True,repeat=False)
## If you have ffmpeg installed on your machine
## you can save the animation by uncomment the last line
## You may install ffmpeg by typing the following command in command prompt
## conda install -c menpo ffmpeg
##
#anim.save('movie.mp4',fps=20,dpi=400)
```

    <IPython.core.display.Javascript object>

# 3.3. The mean square displacement and the diffusion constant

```python
# mean square displacement vs time
msd = np.zeros([nums])
for i in range(nums):
    for n in range(nump):
        msd[i]=msd[i]+np.linalg.norm(Rs[i,n,:]-Rs[0,n,:])**2 # (R(t) - R(0))^2
    msd[i] = msd[i]/nump
dmsd = np.trapz(msd, dx=dt)/(3*(nums*dt)**2)
print('D_0 =',kBT/zeta,'(Theoretical)')
print('D =',dmsd,'(Simulation via MSD)')
print('Volume fraction =',vf) ### print vf
fig, ax = plt.subplots(figsize=(7.5,7.5))
ax.set_xlabel(r"$t$", fontsize=20)
ax.set_ylabel(r"mean square displacement", fontsize=16)
ax.plot(time,6*kBT/zeta*time,'r',lw=6,label=r'$6Dt={6k_BT t}/{\zeta}$')
ax.plot(time,msd,'b',lw=4, label=r'$\langle R^2(t)\rangle$')
ax.legend(fontsize=16, loc=4)
plt.show()
```

    D_0 = 1.0 (Theoretical)
    D = 0.885382884029 (Simulation via MSD)
    Volume fraction = 0.001

    <IPython.core.display.Javascript object>

# References

- The SciPy.org website, https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html

- Listed in the major unsolved problems in physics, https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_physics#Condensed_matter_physics

## Summary of simulation method

### Original differential equation
$i$: particle index

\begin{equation}
\frac{d\mathbf{R}_i(t)}{dt}=\mathbf{V}_i(t)\tag{I3}
\end{equation}

\begin{equation}
m\frac{d\mathbf{V}_i(t)}{dt}=\color{black}{-\zeta\mathbf{V}_i(t)}+\color{black}{\mathbf{F}_i(t)} +\color{black}{\mathbf{F}_R(t)}
\tag{I4}
\end{equation}
$\hspace{80mm}$with
\begin{equation}
\langle \mathbf{F}_R(t)\rangle=\mathbf{0}
\tag{I5}
\end{equation}

\begin{equation}
\langle \mathbf{F}_R(t)\mathbf{F}_R(0)\rangle = {2k_B T\zeta}\mathbf{I}\delta(t)
\tag{I6}
\end{equation}

### Euler method
$i$: index of particles,
$n$: time step
$$
\mathbf{V}_{i,n+1}
=\left(1-\frac{\zeta}{m}\Delta t\right)\mathbf{V}_{i,n}  + \frac{1}{m}\mathbf{F}_{i,n}\Delta t + \frac{1}{m} {\Delta \mathbf{W}_n}
\tag{I7}
$$

$$
\mathbf{R}_{i,n+1}=\mathbf{R}_{i,n}+\mathbf{V}_{i,n} \Delta t \hspace{15mm}\tag{I8}
$$
$\hspace{80mm}$with
\begin{equation}
\langle \Delta \mathbf{W}_n\rangle=\mathbf{0}
\tag{I9}
\end{equation}

\begin{equation}
\langle \Delta \mathbf{W}_n\Delta \mathbf{W}_l\rangle = {2k_B T\zeta}\Delta t\mathbf{I}\delta_{nl}
\tag{I10}
\end{equation}

### Leap-Frog method
$i$: index of particles,
$n$: time step
$$
\mathbf{V}_{i,n+\frac{1}{2}}
=\mathbf{V}_{i,n-\frac{1}{2}}-\frac{\zeta}{m}\Delta t\mathbf{V}_{i,n}  + \frac{1}{m}\mathbf{F}_{i,n}\Delta t + \frac{1}{m} {\Delta \mathbf{W}_n} \\
\hspace{45mm}=\mathbf{V}_{i,n-\frac{1}{2}}-\frac{\zeta}{m}\Delta t\frac{1}{2}\left(\mathbf{V}_{i,n+\frac{1}{2}}+\mathbf{V}_{i,n-\frac{1}{2}}\right)  + \frac{1}{m}\mathbf{F}_{i,n}\Delta t + \frac{1}{m} {\Delta \mathbf{W}_n}
$$

$$
\therefore\ \ \mathbf{V}_{i,n+\frac{1}{2}}
=\left(\left(1-\frac{\zeta}{2m}\Delta t\right)\mathbf{V}_{i,n-\frac{1}{2}} + \frac{1}{m}\mathbf{F}_{i,n}\Delta t + \frac{1}{m} {\Delta \mathbf{W}_n}\right)\left({1+\frac{\zeta}{2m}\Delta t}\right)^{-1} \tag{I11}
$$

$$
\mathbf{R}_{i,n+1}=\mathbf{R}_{i,n}+\mathbf{V}_{i,n+\frac{1}{2}} \Delta t \hspace{15mm}\tag{I12}
$$
$\hspace{80mm}$with
\begin{equation}
\langle \Delta \mathbf{W}_n\rangle=\mathbf{0}
\tag{I13}
\end{equation}

\begin{equation}
\langle \Delta \mathbf{W}_n\Delta \mathbf{W}_l\rangle = {2k_B T\zeta}\Delta t\mathbf{I}\delta_{nl}
\tag{I14}
\end{equation}

### 2nd order Runge-Kutta method
$i$: index of particles,
$n$: time step

$$
\mathbf{V}'_{i,n+\frac{1}{2}}
=\mathbf{V}_{i,n}-\frac{\zeta}{m}\frac{\Delta t}{2}\mathbf{V}_{i,n}  + \frac{1}{m}\mathbf{F}_{i,n}\frac{\Delta t}{2}
=\left(1-\frac{\zeta}{m}\frac{\Delta t}{2}\right)\mathbf{V}_{i,n}
+ \frac{1}{m}\mathbf{F}_{i,n}\frac{\Delta t}{2} \tag{I15}
$$

$$
\mathbf{R}'_{i,n+\frac{1}{2}}=\mathbf{R}_{i,n}+\mathbf{V}_{i,n} \frac{\Delta t}{2} \hspace{15mm}\tag{I16}
$$

$$
\mathbf{V}_{n,i+1}
=\mathbf{V}_{i,n}-\frac{\zeta}{m}\Delta t\mathbf{V}'_{n,i+\frac{1}{2}} + \frac{1}{m}\Delta t\mathbf{F}_{i,n+\frac{1}{2}} + \frac{1}{m} {\Delta \mathbf{W}_i}
\tag{I17}
$$

$$
\mathbf{R}_{n,i+1}=\mathbf{R}_{n,i}+\mathbf{V}'_{n,i+\frac{1}{2}} \Delta t \hspace{15mm}\tag{I18}
$$

$\hspace{80mm}$with
\begin{equation}
\langle \Delta \mathbf{W}_n\rangle=\mathbf{0}
\tag{I19}
\end{equation}

\begin{equation}
\langle \Delta \mathbf{W}_n\Delta \mathbf{W}_l\rangle = {2k_B T\zeta}\Delta t\mathbf{I}\delta_{nl}
\tag{I20}
\end{equation}


```python
%run common-imports.ipynb
```

<link href="https://fonts.googleapis.com/css?family=Lora:400,700|Montserrat:300" rel="stylesheet">

<link href="https://fonts.googleapis.com/css2?family=Crimson+Pro&family=Literata&display=swap" rel="stylesheet">

#  Dataset-5

## Prerequisites

prior notebooks:

* Notebook 1: `univariate-1.ipynb`
* Notebook 2: `dataset-2.ipynb`
* Notebook 3: `dataset-3.ipynb`
* Notebook 4: `univariate-2.ipynb`
* Notebook 5: `univariate-3.ipynb`
* Notebook 6: `dataset-4.ipynb`

## Lab Goals

* will explore this data, observe its statistical characteristics, visualize it.
* will take a systematic approach to build  linear and polynomial regression models to make prediction on the data.

## Outcome

This lab dealing with polynomial regression, and under its limitations when dealing with data more appropriately modeled with transcendental functions.

```python
#source = 'https://raw.githubusercontent.com/'
data = pd.read_csv("../datasets/dataset-5.csv")
```

```python
data.sample(5)
```

      x
      y

      6252
      9.541218
      0.202160

      4684
      9.348196
      0.218139

      1731
      0.551262
      1.046148

      4742
      1.020358
      1.531527

      4521
      0.275044
      0.599263

#### Descriptive statistics

```python
data.describe(include="all").transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      x
      10000.0
      4.944755
      2.863460
      0.002700
      2.463689
      4.915809
      7.399909
      9.999281

      y
      10000.0
      0.975080
      0.571116
      0.006741
      0.430866
      0.912415
      1.539281
      1.839397

#### Missing Values Analysis

```python
data.isnull().sum()
```

    x    0
    y    0
    dtype: int64

 From the above results, no missing values at all. we don't address this issue.

#### Pandas Profiling

```python
# data.profile_report()
```

## Data Visualization

### Plotting using matplotlib

Observe that the data distinctly exhibits nonlinearity of relationship between $x$ and $y$. **what the correlation is between the variables?**

```python
plt.scatter(data['x'], data['y'], alpha=0.5, s=150, color='salmon')
plt.title(r'\textbf{Scatter-plot of $y$ vs $x$}')
plt.xlabel(r'$x\longrightarrow$');
plt.ylabel(r'$y\longrightarrow$');
```

# Regression

From the figure above, it is apparent that a simple linear regression model is unlikely to work. However, let us first build a simple linear regression model for this dataset, in order to get a baseline performance.

As usual, we will first separate out the predictor from the target, and then split the data into a training and test set.

```python
X, y = data[['x']], data['y']
```

```python
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42,train_size=0.5)
```

Observe that we have called the predictor dataframe as `X` (note the capitalization), and the target as simply the array of values: `y`. While we could have named the variables anything, it is a well established practice in python based data science notebooks to follow this particular naming convention. It makes is much easier for other data scientists to read a notebook if it follows the well established naming conventions.

## Split the data into test and train subsets

In order to know how good at predicting are the regression models we will build, we need to take a subset of the data, and sort of "hide it under the pillow", and not allow the algorithm to see it during the training process. Then we can check if the model is making good predictions by comparing the said predictions against the actual target values.

Once again, observe how we follow the well established naming conventions for the training and test subsets of the data:

## Build a linear regression model

Let us now build a regression model, and fit it to the dataset. For this, we instantiate a `LinearRegression` object named `model` using the constructor. Then we fit the `model` to the available training dataset.

```python
model = LinearRegression();
model.fit(X_train, y_train);
```

Recall that a linear regression model is given by the equation:

\begin{equation} y =  \beta_0 + \beta_1 x + \epsilon\end{equation}

where:
$\epsilon$ is the irreducible error term, so that the model is essentially:

\begin{equation} \hat{y} =  \beta_0 + \beta_1 x \end{equation}

What values of $\beta_0$ (the intercept) and $\beta_1$ (the slope) is this model predicting? We can inspect this as follows:

```python
print (f'Intercept: {model.intercept_}, Slope: {model.coef_}')
```

    Intercept: 1.7711852235155838, Slope: [-0.15990193]

Look back at the data visualization, and see if this agrees with your own estimates.

## Predictions from the model

Now, let us use the model to make predictions on the **test** data, something the model has not seen so far. By comparing the predictions to the actual values, will get a sense of how well the model has learned to generalize from the data.

```python
yhat = model.predict(X_test)
print("Mean Squared Error: %.2f"
      % mean_squared_error(y_test, yhat))
r2 = r2_score(y_test, yhat)
print(rf"Coefficient of Determination (R^2):{r2}")
```

    Mean Squared Error: 0.12
    Coefficient of Determination (R^2):0.6161335362432641

The coefficient of determination, $R^2$ indicates a dismal model! Should we plod on to the next step of model verification: namely the residual analysis.

## Residual Analysis

will start by plotting the residuals from the predictions. Recall that the residual from the prediction $\hat{y}_i$ on a particular datum $(x_i, y_i)$ is defined as:

\begin{equation}\mathbf{ r_i = \hat{y}_i  - y_i }\end{equation}

will use the `yellowbrick` library for looking at the model characteristics.

```python
from yellowbrick.regressor import residuals_plot
viz = residuals_plot(model, X_train, y_train, X_test, y_test,
                     train_color='orange', test_color='IndianRed', train_alpha=0.3, test_alpha=0.5)
```

The residuals display a striking pattern! The presence of a pattern in the residuals in a clear indication that the model has failed to capture some essential characteristics of the relationship between $x$ and $y$.

## Visualization of the model predictions

As a final step, let us visualize the predictions of the model, and superimpose it on the actual data. This should give us a sense of how well the model is working.

```python

X = pd.DataFrame(data={'x': np.linspace(data.x.min(), data.x.max(), 1000)})
yhat = model.predict(X)
```

```python
fig, ax = plt.subplots(figsize=(20,10))
ax.scatter(data['x'], data['y'], alpha=0.5, s=150, color='salmon')
ax.plot(X.x, yhat, 'r--.', label="Model Predictions")
ax.legend(loc='best');
```

This simple linear model has, as one would have expected by now, failed terribly!

##### Polynomial Regression

from the prior notebooks adding polynomial features to the dataset, and then performing the regression can be an effective tool.

**since we observe 5 distinct bends in the data, we should expect a polynomial of degree no less than 6 to fit the data.** With some experimentation, we find that a polynomial of degree 6 proves reasonably effective.

merged the steps of:
* create polynomial degrees of the input
* fit a linear model to the data
* make predictions on test data
* print model diagnostics

```python
degree     = 6
polynomial = PolynomialFeatures(degree)
X_poly = polynomial.fit_transform(X_train)
model = LinearRegression()
# Now, train the model
model.fit(X_poly, y_train)
print ("The cofficients: {}".format(model.coef_))
X_poly_test = polynomial.fit_transform(X_test)
yhat = model.predict(X_poly_test)
print("Mean squared error: %.2f"
      % mean_squared_error(y_test, yhat))
r2 = r2_score(y_test, yhat)
print(rf"Coefficient of Determination (R^2):{r2}")
```

    The cofficients: [ 0.00000000e+00  2.41758892e+00 -1.13045501e+00  2.37878473e-01
     -2.73961903e-02  1.67614403e-03 -4.25161284e-05]
    Mean squared error: 0.00
    Coefficient of Determination (R^2):0.9999743750656844

The coefficient of determination seems astonishingly good! Let us now proceed with the residual analysis.

## Residual Analysis

We will start by plotting the residuals from the predictions. Recall that the residual from the prediction $\hat{y}_i$ on a particular datum $(x_i, y_i)$ is defined as:

\begin{equation}\mathbf{ r_i = \hat{y}_i  - y_i }\end{equation}

We will use the `yellowbrick` library for looking at the model characteristics.

```python
from yellowbrick.regressor import residuals_plot
viz = residuals_plot(model, X_poly, y_train, X_poly_test, y_test,
                     train_color='orange', test_color='IndianRed', train_alpha=0.3, test_alpha=0.5)
```

Despite the excellent coefficient of determination, the figure above shows a distinct pattern in the residual, clearly indicative of the fact that the polynomial model is perhaps not representative of the underlying data.

The y-axis range range of the residuals are practically zero.

## Visualization of the model predictions

let's visualize the predictions of the model, and superimpose it on the actual data. This should give us a sense of how well the model is working.

```python

X = pd.DataFrame(data={'x': np.linspace(data.x.min(), data.x.max(), 1000)})
yhat = model.predict(polynomial.transform(X))

fig, ax = plt.subplots(figsize=(20,10))
ax.scatter(data['x'], data['y'], alpha=0.5, s=150, color='salmon')
ax.plot(X.x, yhat, 'r--.', label="Model Predictions")
ax.legend(loc='best');
```

The model predictions agree exceedingly well with the data! From this we conlude that this is indeed an effective model: though it may not have captured well the underlying generative force, it is a fairly effective approximation.

when there are some patterns in the residuals, consider this a very effective model for the following reasons:

* the coefficient of determination is excellent
* the actual residuals are vanishingly small
* the model predictions when visualized over the data show a high degree of fidelity.

# Conclusion

For this dataset, polynomial regression has proved reasonably effective. keep the polynomial to six or so, we do not see pronounced Runge's phenomenon. However, besyond that, the oscillations at the periphery begin to get pronounced.

Even the model the data with a very high degree polynomial, there are still very small residuals but with a pronounced pattern in the residual plot. This leads us to believe that this data was generated with some transcendental functions involved.

(Indeed, it so happens that this data was generated with a Gamma function, which is a transcendental.)


# Part 1; Final Exam CHEM 3PA3; Winter 2021

👨🏫 **Overview:**
This is Part 1 of 3 Parts of your Final Exam/Project.
- Part 1. Questions on the Course Material. (40%)
- Part 2. Applying the Course material. (40%)
- Part 3. Discussing your exam and the Course Material. (20%)

**You must submit your Jupyter notebooks for Part 1 and Part 2 at least 48 hours prior to your appointment for Part 3.** You will be given your grade on Part 1 and Part 2 before the oral exam, so that you know what your status is. *For late submission of Part 1 and/or Part 2, I will deduct 2 points per hour.

📜 **Instructions:**
Answer the following questions. These are worth 40% of your exam, and each question is worth 2 points. (Some questions have multiple parts.) There is also a bonus problem.
- You can upload files for mathematical answers or type them in Markdown.
- You can use the notebook as a calculator for numerical problems; but you can also just type in your answer computed offline.
- You may find these [sheets containing reference data and mathematical formulas/identities](https://github.com/PaulWAyers/IntroQChem/blob/main/documents/ReferenceConstantsConversionsMath.pdf?raw=true) useful.
- You may also find it useful to know that the [speed of light, in atomic units](https://en.wikipedia.org/wiki/Hartree_atomic_units), is 137.036 a.u..

📖 **Rules for "Open Book" Exam:**
**Like all other portions of this exam, this part of the exam is open notes and open library. It is "open internet search" but you (obviously!) can't post questions on an internet discussion board or homework/problem/exam help site. You are not allowed to communicate with your classmates or any other human being (except me) about these questions or your responses, and this includes human beings (singular or plural, known or anonymous) online.**

☑ **Checklist Before Submission:**
- Make sure you fill in any place that says YOUR CODE HERE or "YOUR ANSWER HERE".
- Remove "Not Implemented Error" where appropriate.
- **Please put your name, username (the prefix to your @mcmaster.ca e-mail), and student ID number in the cell below.**
- Before you turn these problems in, make sure everything runs as expected.
  - First, restart the kernel (in the menubar, select Kernel → Restart).
  - Then run all cells (in the menubar, select Cell → Run All).
- Turn in your exam as a Jupyter notebook.

```python
import numpy as np
import scipy
from scipy import constants
```

## ✍ Write the Time-Dependent Schrödinger equation for one particle, in three dimensions, confined by the time-dependent potential $V(x,y,z,t)$.
**Note:** Write out the specific Hamiltonian; do not just write $\hat{H}$. Report your answer in the below markdown cell. You can drag-and-drop a photo/screenshot/image of your answer to the cell, but before submitting please confirm that the image is appropriately embedded.

YOUR ANSWER HERE

## ✍ Write the Time-Independent Schrödinger equation for a one-electron atom with atomic number $Z$ in atomic units.
**Note:** Report your answer in the below Markdown Cell.

YOUR ANSWER HERE

## 🖩 Compute properties of a photon with wavelength 486.1 nm.
The photon emitted when a 4p electron in the Hydrogen atom deexcites into a 2s orbital has wavelength 486.1 nm. Compute its
- frequency in Hz
- wavenumber, $\bar{\nu}$, in $\text{cm}^{-1}$.
- momentum in m kg/s
- energy in Joules.

```python
def p1():
    # Give your answers as Floats. I have initialized the variables to None.

    ### START YOUR CODE HERE
    frequency = None         #Frequency in Hz.
    wavenumber = None        #Wavenumber, nu-bar, in cm^-1
    momentum = None          #Momentum in SI units.
    energy = None            #Energy in SI units.
    ### END YOUR CODE HERE

    return frequency, wavenumber, momentum, energy

```

```python
frequency, wavenumber, momentum, energy = p1()

print(f"The frequency of a photon with wavelength 486.1 nm is {frequency} Hz.")
print(f"The wavenumber, nu-bar, of a photon with wavelength  486.1 nm is {wavenumber} cm^-1.")
print(f"The momentum of a photon with wavelength  486.1 nm is {momentum} kg m/s.")
print(f"The energy of a photon with wavelength  486.1 nm is {energy} Joules.")

assert(isinstance(frequency,float)), "Type Error: The frequency should be a float."
assert(isinstance(wavenumber,float)), "Type Error: The wavenumber should be a float."
assert(isinstance(momentum,float)), "Type Error: The momentum should be a float."
assert(isinstance(energy,float)), "Type Error: The energy should be a float."

```

## 🧮 How many quantum numbers are needed to label the eigenstates of the following Hamiltonian:

$$
\hat{H} =  -\frac{\hbar^2}{2m}\frac{d^2}{dx_1^2} -\frac{\hbar^2}{2m}\frac{d^2}{dy_1^2}
-\frac{\hbar^2}{2m}\frac{d^2}{dx_2^2}-\frac{\hbar^2}{2m}\frac{d^2}{dy_2^2}
+ x_1^2 + y_1^2 + x_2^2 + y_2^2
+ e^{-2\left(x_1^2 + y_1^2\right)} + e^{-2\left(x_2^2 + y_2^2\right)}
$$

```python
def p2():
    #Give your answer as an integer. I have initialized the variable to None

    ### START YOUR CODE HERE
    number_of_quantum_nos = None
    ### END YOUR CODE HERE

    return number_of_quantum_nos

```

```python
number_of_quantum_nos = p2()

print(f'The states of this Hamiltonian are labelled with {number_of_quantum_nos} quantum numbers.')

assert isinstance(number_of_quantum_nos,int), "Type error: The answer should be an integer."

```

## 🔀Below is a list of terms in the molecular Hamiltonian, together with their mathematical expressions.
Match the verbal description of the term to its mathematical expression; the mathematical expressions are written in atomic units. Report your answer as a dictionary. The keys and values are:

**Keys:**
`electronic kinetic energy`, `nuclear kinetic energy`, `electron-electron repulsion potential`, `nucleus-nucleus repulsion potential`, `electron-nuclear attraction potential`

**Values**:
"eq1", "eq2", "eq3", "eq4", and "eq5", where

`eq1`:  $\sum_{a=1}^{A-1} \sum_{b=a+1}^A \frac{Z_a Z_b}{\left| \mathbf{x}_a - \mathbf{x}_b \right|}$

`eq2`:  $\sum_{a=1}^A -\tfrac{1}{2m_a} \nabla^2_{\mathbf{x}_a}$

`eq2`:  $\sum_{i=1}^N \sum_{a=1}^A -\frac{Z_a}{\left| \mathbf{r}_i - \mathbf{x}_a \right|}$

`eq4`:  $\sum_{i=1}^N -\tfrac{1}{2} \nabla^2_{\mathbf{r}_i}$

`eq5`:  $\sum_{i=1}^{N-1} \sum_{j=i+1}^N \frac{1}{\left| \mathbf{r}_i - \mathbf{r}_j \right|}$

For example, the following is an example of an *incorrect* answer with the correct format:
```
hamiltonian_terms = {'electronic kinetic energy':'eq1', 'nuclear kinetic energy':'eq2', 'electron-electron repulsion potential':'eq3', 'nucleus-nucleus repulsion potential':'eq4', 'electron-nuclear attraction potential':'eq5'}
```

```python
def p3():
    # Enter your answer as a dictionary called hamiltonian_terms.
    # The below dictionary has the keys typed in already for you; you need to type in the values eq1, eq2, eq3, eq4, eq5
    #      instead of the place-holder values listed here.

    ### START YOUR CODE HERE
    hamiltonian_terms = {'electronic kinetic energy':'eq#',
                         'nuclear kinetic energy':'eq#',
                         'electron-electron repulsion potential':'eq#',
                         'nucleus-nucleus repulsion potential':'eq#',
                         'electron-nuclear attraction potential':'eq#'}
    ### END YOUR CODE HERE
    return hamiltonian_terms

```

```python
hamiltonian_terms = p3()
print(hamiltonian_terms)

assert len(hamiltonian_terms) == 5, "There should be 5 key:value pairs in the hamiltonian_terms dictionary!"

k = ['electron-electron repulsion potential', 'electron-nuclear attraction potential',
     'electronic kinetic energy', 'nuclear kinetic energy', 'nucleus-nucleus repulsion potential']
v = ['eq1', 'eq2', 'eq3', 'eq4', 'eq5']
assert sorted(hamiltonian_terms.keys()) == k, "At least one key does not match the original dictionary"
assert sorted(hamiltonian_terms.values()) == v, "At least one value does not match the equation labels"

```

## 🔀Below are lists of different Hamiltonians we considered in the course, together with the names of functions that are related to their solutions.
Match each operator with the type of (special) functions that are its eigenfunctions. Report your answer as a dictionary. The keys and values are:

**Keys:**

`Ham1`:  $ -\frac{\hbar^2}{2m} \left(\frac{1}{r^2} \frac{d^2}{dr^2} + \frac{1}{r} \frac{d}{dr} - \frac{l^2}{r^2}\right) $

`Ham2`:  $ - \hbar^2 \left[\frac{1}{\sin \theta}\frac{d}{d\theta}\sin\theta\frac{d}{d\theta} + \frac{1}{\sin^2 \theta} \frac{d^2}{d\phi^2}\right] $

`Ham3`:  $ -\frac{1}{2} \left( \frac{d^2}{dr^2} + \frac{2}{r} \frac{d}{dr}\right) + \frac{l(l+1)}{2r^2} - \frac{Z}{r}$

`Ham4`:  $ \frac{d^2}{dx^2} $

**Values**:
`associated Laguerre polynomials`, `sines and cosines`, `Bessel functions`, `spherical harmonics`

For example, the following is an example of an *incorrect* answer with the correct format:
```
eigenfunctions = {'Ham1':'associated Laguerre Polynomials', 'Ham2':'sines and cosines', 'Ham3':'spherical harmonics', 'Ham4':'Bessel functions'}
```

```python
def p4():
    # Enter your answer as a dictionary called hamiltonian_terms.
    # The below dictionary has the values typed in already for you; you need to type in the keys Ham1, Ham2, Ham3, Ham4
    #      instead of the place-holder values listed here.

    ### START YOUR CODE HERE
    eigenfunctions = {'Ham#':'associated Laguerre Polynomials',
                      'Ham#':'sines and cosines',
                      'Ham#':'spherical harmonics',
                      'Ham#':'Bessel functions'}
    ### END YOUR CODE HERE

    return eigenfunctions
```

```python
eigenfunctions = p4()
print(eigenfunctions)

assert len(eigenfunctions) == 4, "There should be 4 key:value pairs in the eigenfunctions dictionary!"

v = ['Bessel functions', 'associated Laguerre Polynomials', 'sines and cosines', 'spherical harmonics']
k = ['Ham1', 'Ham2', 'Ham3', 'Ham4']
assert sorted(eigenfunctions.keys()) == k, "At least one Hamiltonian label does not match the original dictionary"
assert sorted(eigenfunctions.values()) == v, "At least one special function name does not match the original dictionary"

```

## 🖩 What is the zero-point energy for an electron confined to a 2-dimensional circular disk of radius $4 \text{ a.u.}$:
- in Hartree atomic units?
- in kJ/mol (kiloJoules per mole)?

```python
def p5():
    # You can use the code box as a calculator if you like; either way, report your answer as float(s).
    # I've initialized the answers to None.

    ### START YOUR CODE HERE
    zero_pt_E_au = None                  #Energy in Hartree (atomic units)
    zero_pt_E_kJperMol = None            #Energy in kJ/mol (SI units)
    ### END YOUR CODE HERE

    return zero_pt_E_au, zero_pt_E_kJperMol

```

```python
zero_pt_E_au, zero_pt_E_kJperMol = p5()
print(f'The zero-point energy of a electron confined to a disk of radius 4 Bohr is {zero_pt_E_au} Hartree.')
print(f'The zero-point energy of a electron confined to a disk of radius 4 Bohr is {zero_pt_E_kJperMol} kJ/mol.')

# basic tests:
assert(isinstance(zero_pt_E_au,float)), "The answer should be an float"
assert(isinstance(zero_pt_E_kJperMol,float)), "The answer should be an integer"

```

## ✍ Write a Slater determinant for the excited-state, $1\text{s}^2 2\text{s}^1 3\text{s}^1$ configuration of the Beryllium atom.
Assume that the unpaired electrons have the same spin. You may denote the spatial orbitals as $\phi_{1s}(\mathbf{r})$, $\phi_{2s}(\mathbf{r})$ and $\phi_{3s}(\mathbf{r})$.

YOUR ANSWER HERE

## 🧮 Compute properties of a ${}^{7}\text{F}$ state.
The ground-state term symbol for Samarium (Sm, atomic number 62) is ${}^{7}\text{F}_0$.
- How many nominally spatial and spin states are associated with the ${}^{7}\text{F}$ term symbol? Answer with an integer.
- In addition to J=0, what other J values are possible for the ${}^7{\text{F}}$ term symbol? Answer with a set or list of integers.

For example, for a triplet P state, we would have
```
multiplicity_3P = 9
Jvalues_3P = [0,1,2]
```

```python
def p6():
    ### START YOUR CODE HERE
    # Multiplicity of septuplet F state of Samarium (integer)
    multiplity_7F = None

    # J values associated with septuplet F state of Samarium (set or list of integers)
    Jvalues_7F = None
    ###END YOUR CODE HERE

    return multiplity_7F, Jvalues_7F

```

```python
multiplity_7F, Jvalues_7F = p6()
print(f'A septuplet-F term is {multiplicity_7F}-fold degenerate and has J values {Jvalues_7F}.')

assert(isinstance(multiplicity_7F,int)), "The multiplicity should be an integer"
assert(isinstance(Jvalues_7F,set) or isinstance(Jvalues_7F,list) or isinstance(Jvalues_7F,tuple)), "The list of J values should be a list of integers."
assert(len(Jvalues_7F) >= 1)

```

## ✍ Solve the Schrödinger equation for a muon, $\mu^-$, bound to Neon nucleus.
In [muonic atoms](https://en.wikipedia.org/wiki/Exotic_atom), a muon replaces an electron. The muon and the electron are both fermions, and they have the same charge. However, the mass of the muon is $207$ times the mass of the electron, $m_\mu = 206.8 m_e$.

Assuming the Born-Oppenheimer approximation is valid and that relativistic effects can be neglected, **what are the energy eigenfunctions and eigenvalues for a system composed of one $\mu^-$ bound to a Neon nucleus.**

YOUR ANSWER HERE

## ✍ Write an expression for the lowest-energy molecular orbital of $\text{H}_3^{+2}$.
You may assume that the system is well-described by the separated-atom limit, and that the Hydrogen atoms are arranged as an equilateral triangle ($\text{D}_{3h}$ symmetry). You do not need to normalize the molecular orbital.

YOUR ANSWER HERE

## The next two questions refer to the Laplacian operator.
In this question you will show, mathematically, that two properties of the Laplacian operator are true. It is OK (but not essential) to assume that you are interested in the Laplacian operator in 1 dimension, $\nabla^2 = \frac{d^2}{dx^2}$.
Note: You may assume that the wavefunction, and its first first and second derivatives, vanish at the ends of the region of integration. However, there are also ways to answer these questions without explicitly invoking those assumptions.

### ✍ Show that $\nabla^2$ is a linear, Hermitian operator.

YOUR ANSWER HERE

### ✍ Show that $\nabla^2$ is negative definite.
That is, show that for *any* $\Psi(\mathbf{r})$, it is *always* true that

$$
\langle \Psi | \nabla^2 | \Psi \rangle < 0
$$

YOUR ANSWER HERE

## Expectation values of $r^k$ and related properties in Hydrogenic atoms.
The goal of this problem is to explore the radial distribution function and its expectation values for hydrogenic atoms. A key integral that will be useful is

$$
\int_0^{\infty} r^n e^{-\beta r} dr = \frac{n!}{\beta^{n+1}}
$$

It is also useful to know [Kramer's recursion relation](https://phlconnect.ched.gov.ph/admin/uploads/81dc9bdb52d04dc20036dbd8313ed055/53-hydrogen-4.pdf) for radial expectation values. If $\psi_{nlm}(r,\theta,\phi)$ denotes a normalized hydrogenic wavefunction, then Kramer's recursion states that:

$$
\langle \psi_{nlm} | r^k | \psi_{nlm} \rangle = a_{n}(k,Z) \langle \psi_{nlm} | r^{k-1} | \psi_{nlm} \rangle + b_{nl}(k,Z) \langle \psi_{nlm} | r^{k-2} | \psi_{nlm} \rangle
$$

where

$$
\begin{align}
a_{n}(k,Z) &=  \frac{n^2(2k+1)}{(k+1)Z}\\
b_{nl}(k,Z) &= \frac{n^2 k \left(k^2 - (2l+1)^2\right)}{4Z^2(k+1)}
\end{align}
$$

A detailed derivation of this relation can be found [here](http://www.blazartheory.com/files/notes/qmnotes/Kramers__Relation.pdf). You will not *need* to use this relation in this problem; it's a matter of preference whether you use this relation or perform integrals.

Throughout this problem you may use atomic units.

### ✍ Expectation value of $r^{-1}$ for Hydrogenic Atoms.
What is the expectation value of $r^{-1}$ for a Hydrogenic atom. That is, what is the expectation value of

$$
\langle \psi_{nlm} | r^{-1} | \psi_{nlm} \rangle
$$

for an electron bound to a nuclear of charge Z. *Report your answer as a float, in atomic units.*

YOUR ANSWER HERE

### 🖩 Expectation value of $r^2$ for a 4f orbital in the hydrogen atom.
What is the expectation value of $r^2$ for the 4f orbital in the hydrogen atom (Z=1). That is, what is the expectation value of

$$
\langle \psi_{43m} | r^2 | \psi_{43m} \rangle
$$

in the Hydrogen atom? *Report your answer as a float, in atomic units.*

If you wish, you can use the markdown cell at the end to explain your answer.

```python
def p14():
    # Expectation value of r^2 for a 4f orbital in the hydrogen atom (Z=1) in atomic units (Bohr^2).
    # I have initialized the variable to None. Give your answer in atomic units (Bohr).

    ### START YOUR CODE HERE
    avg_r2 = None
    ### END YOUR CODE HERE

    return avg_r2

```

```python
print(f'The expectation value of r^2 for an electron in the 4f orbital of the hydrogen atom is {p14()} Bohr^2.')

```

YOUR ANSWER HERE

### 🖩 What is the most probable distance to find an electron from the nucleus in the 4f orbital of a hydrogen atom?
Report your answer in Bohr (atomic units).

If you wish, you can use the markdown cell at the end to explain your answer.

```python
def p15():
    # Most probable distance at which to observe an electron in the 4f orbital from the nucleus of the hydrogen atom.
    # I have initialized the variable to None. Give your answer in atomic units (Bohr).

    ### START YOUR CODE HERE
    dist_mostprob = None
    ### END YOUR CODE HERE

    return dist_mostprob
```

```python
print(f'The most probable value of the distance from the nucleus for a 4f electron in Hydrogen is {p15()} Bohr.')

```

YOUR ANSWER HERE

### 🖩 Heisenberg Uncertainty Principle
The Heisenberg Uncertainty Principle between momentum and position in 3 dimensions states that

$$
\sigma_r^2 \sigma_p^2 \ge  \tfrac{3}{4} \hbar^2
$$

where

$$
\sigma_r^2 = \langle \psi_{n,n-1,m} | r^2 | \psi_{n,n-1,m} \rangle - \langle \psi_{n,n-1,m} | r | \psi_{n,n-1,m} \rangle^2 \\
\sigma_p^2 = \langle \psi_{n,n-1,m} | \hat{p}^2 | \psi_{n,n-1,m} \rangle - \langle \psi_{n,n-1,m} | \hat{p} | \psi_{n,n-1,m} \rangle^2
$$

To reassure yourself that this is true, compute $\sigma_r^2 \sigma_p^2$ for the 4f state of the hydrogen atom. *Report your answer as a float, in atomic units.*

If you wish, you can use the markdown cell at the end to explain your answer.

```python
def p16():
    # Write the left-hand-side of the Heisenberg Uncertainty relation, which should be bigger than 3/2 hbar.
    # Give your answer in atomic units.

    ### START YOUR CODE HERE
    test_Heisenberg = None
    ### END YOUR CODE HERE

    return test_Heisenberg
```

```python
print(f'The product of the variance in position and momentum for an electron in the 4f orbital of Hydrogen is {p16()} times hbar.')
assert(test_Heisenberg >= 0.75), "The Heisenberg Uncertainty Principle has been violated!"

```

YOUR ANSWER HERE

### 💰 Bonus: Derive an expression for $\sigma_r^2 \sigma_p^2$ for *any* state of a Hydrogenic atom with $l=n-1$.

YOUR ANSWER HERE

## Eigenfunctions and Eigenvalues for a two-dimensional Quantum Dot.
Modern displays often use quantum dot technology, where one (or more) electrons are confined to a region within a material. For an electron confined to a rectangular region in two dimensions, it is reasonable to approximate its motion perpendicular to the rectangle with harmonic confinement. The time-independent Schrödinger equation, eigenfunctions, and eigenvalues for a harmonically confined electron are, in atomic units:

$$
\left(-\tfrac{1}{2}\tfrac{d^2}{dz^2} + \tfrac{1}{2}\kappa z^2\right) \psi_k(z) = E_k \psi_k(z)
$$

where the eigenenergies are

$$
E_k = \sqrt{\kappa}\left(k+\tfrac{1}{2}\right) \qquad \qquad k=0,1,2,\ldots
$$

and the eigenfunctions are given in terms of the [Hermite polynomials](https://en.wikipedia.org/wiki/Hermite_polynomials), $H_k(z)$, as:

$$
\psi_k^{(\text{harm. osc.})}(z) =\frac{1}{2^k k!}\sqrt[4]{\frac{\kappa}{\pi}}
e^{-\sqrt{\kappa}z^2/2}H_k\left(\sqrt[4]{\kappa}z\right)
$$

The Hamiltonian for an electron confined to a rectangular region, in atomic units, is then:

$$
\hat{H} = -\frac{1}{2} \frac{d^2}{dx^2} -\frac{1}{2} \frac{d^2}{dy^2} -\frac{1}{2} \frac{d^2}{dz^2}
 + V_{a_x}(x) + V_{a_y}(y) + \frac{1}{2} \kappa z^2
$$

where

$$
V_a(x) =
\begin{cases}
    +\infty & x\leq 0\\
    0       & 0\lt x \lt a\\
    +\infty & a \leq x
\end{cases}
$$

### ✍ Write expressions for the eigenvalues and eigenfunctions of this system.

YOUR ANSWER HERE

###  🖩 What is the wavelength that corresponds to the lowest-energy excitation when $a_x = 16$, $a_y = 9$, and $k_z = 4$?
Report your answer in nm.

If you wish, you can use the markdown cell at the end to explain your answer.

```python
def p20():
    # Report the wavelength of the lowest-energy excitation in nm. I have initialized the variable to None.

    ### START YOUR CODE HERE
    wavelength_qdot_nm = None
    ### END YOUR CODE HERE

    return wavelength_qdot_nm
```

```python
print(f'The wavelength associated with the lowest-energy excitation of the specified quantum dot is {p20()} nm.')

```

YOUR ANSWER HERE

## ✍ What is the form of the eigenfunctions for the Kratzer-Fues Potential
The Kratzer-Fues potential is a reasonable model for a diatomic molecule rotating and vibrating in 3 dimensions, or even a ion-pair complex (e.g., an single ion pair from an ionic solvent in the gas phase). It has the form

$$
V_{\text{Kratzer-Fues}}(r) = \frac{a}{r^2} - \frac{b}{r} \qquad \qquad a>0 \text{ and } b>0
$$

and so the Schrödinger equation has the form:

$$
\left(-\frac{\hbar^2}{2m}\nabla^2 +\frac{a}{r^2} - \frac{b}{r} \right) \psi(\mathbf{r}) = E\psi(\mathbf{r})
$$

Using separation of variables, write the *form* of the solutions for the Kratzer-Fues potential. What is the radial Schrödinger equation for this system?

YOUR ANSWER HERE

### 💰 Bonus: Write an expression for the energy eigenvalues and the radial eigenfunctions of the Kratzer-Fues potential. This isn't 100% trivial, but try to see how far you can get.

YOUR ANSWER HERE