#  CS-109B Introduction to Data Science
## Lab 3: Optimization in Artificial Neural Networks

**Harvard University**

**Spring 2019**

**Lab instructor** Eleni Kaxiras

**Instructors:** Pavlos Protopapas and Mark Glickman

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

## Learning Goals
In this lab, we'll explore ways to optimize the loss function of a Multilayer Learning Perceptor (MLP) by tuning the model hyperparameters. We'll also explore the use of cross-validation as a technique for checking potential values for these hyperparameters.

By the end of this lab, you should:
- Be familiar with the use of `sklearn`'s `optimize` function.
- Be able to identify the hyperparameters that go into the training of a MLP.
- Be familiar with the implementation in `keras` of various optimization techniques.
- Apply cross-validation to check for multiple values of hyperparameters.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import minimize

%matplotlib inline
```

## Part 1:  Beale's function
### First let's look at function optimization in `scipy.optimize`, using Beale's function as an example

Optimizing a function $f: A\rightarrow R$, from some set A to the real numbers is finding an element $x_0\,\epsilon\, A$ such that $f(x_0)\leq f(x)$ for all $x\,\epsilon\, A$ (finding the minimum) or such that $f(x_0)\geq f(x)$ for all $x\,\epsilon\, A$ (finding the maximum).

To illustrate our point we will use a function of two parameters.  Our goal is to optimize over these 2 parameters. We can extend to higher dimensions by plotting pairs of parameters against each other.

The Wikipedia article on Test functions for optimization has a few functions that are useful for evaluating optimization algorithms. Here is Beale's function:

$f(x,y)$ = $(1.5−x+xy)^2+(2.25−x+xy^2)^2+(2.625−x+xy^3)^2$

\[source:[Wikipedia](https://en.wikipedia.org/wiki/Test_functions_for_optimization)\]

We already know that this function has a minimum at [3.0, 0.5]. Let's see if `scipy` will find it.

```python
# define Beale's function which we want to minimize
def objective(X):
    x = X[0]; y = X[1]
    return (1.5 - x + x*y)**2 + (2.25 - x + x*y**2)**2 + (2.625 - x + x*y**3)**2
```

```python
# function boundaries
xmin, xmax, xstep = -4.5, 4.5, .9
ymin, ymax, ystep = -4.5, 4.5, .9
```

```python
# Let's create some points
x1, y1 = np.meshgrid(np.arange(xmin, xmax + xstep, xstep), np.arange(ymin, ymax + ystep, ystep))
```

Let's make an initial guess

```python
# initial guess
x0 = [4., 4.]
f0 = objective(x0)
print (f0)
```

    68891.203125

```python
bnds = ((xmin, xmax), (ymin, ymax))
minimum = minimize(objective, x0, bounds=bnds)
```

```python
print(minimum)
```

          fun: 2.068025638865627e-12
     hess_inv: <2x2 LbfgsInvHessProduct with dtype=float64>
          jac: array([-1.55969780e-06,  9.89837957e-06])
      message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
         nfev: 60
          nit: 14
       status: 0
      success: True
            x: array([3.00000257, 0.50000085])

```python
real_min = [3.0, 0.5]
print (f'The answer, {minimum.x}, is very close to the optimum as we know it, which is {real_min}')
print (f'The value of the objective for {real_min} is {objective(real_min)}')
```

    The answer, [3.00000257 0.50000085], is very close to the optimum as we know it, which is [3.0, 0.5]
    The value of the objective for [3.0, 0.5] is 0.0

## Part 2: Optimization in neural networks

In general: **Learning Representation --> Objective function --> Optimization algorithm**

A neural network can be defined as a framework that combines inputs and tries to guess the output. If we are lucky enough to have some results, called "the ground truth", to compare the outputs produced by the network, we can calculate the **error**. So the network guesses, calculates some error function, guesses again, trying to minimize this error, guesses again, until the error does not go down any more. This is optimization.

In neural networks the most common used optimization algorithms, are flavors of **GD (gradient descent)**. The *objective function* used in gradient descent is the *loss function* which we want to minimize .

### A `keras` Refresher

`Keras` is a Python library for deep learning that can run on top of both Theano or
TensorFlow, two powerful Python libraries for fast numerical computing created and released by Facebook and Google, respectevely.

Keras was developed to make developing deep learning models as fast and easy as
possible for research and practical applications. It runs on Python 2.7 or 3.5 and can seamlessly execute on GPUs and CPUs.

Keras is built on the idea of a model. At its core we have a sequence of layers called
the `Sequential` model which is a linear stack of layers. Keras also provides the `functional API`, a way to define complex models, such as multi-output models, directed acyclic graphs, or models with shared layers.

We can summarize the construction of deep learning models in Keras using the Sequential model as follows:
1. **Define your model**: create a `Sequential` model and add layers.
2. **Compile your model**: specify loss function and optimizers and call the `.compile()` function.
3. **Fit your model**: train the model on data by calling the `.fit()` function.
4. **Make predictions**: use the model to generate predictions on new data by calling functions such as `.evaluate()` or `.predict()`.

### Callbacks: taking a peek into our model while it's training

You can look at what is happening in various stages of your model by using `callbacks`. A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callbacks (as the keyword argument callbacks) to the `.fit()` method of the Sequential or Model classes. The relevant methods of the callbacks will then be called at each stage of the training.

- A callback function you are already familiar with is `keras.callbacks.History()`. This is automatically included in `.fit()`.

- Another very useful one is `keras.callbacks.ModelCheckpoint` which saves the model with its weights at a certain point in the training. This can prove useful if your model is running for a long time and a system failure happens. Not all is lost then. It's a good practice to save the model weights only when an improvement is observed as measured by the `acc`, for example.

- `keras.callbacks.EarlyStopping` stops the training when a monitored quantity has stopped improving.

- `keras.callbacks.LearningRateScheduler` will change the learning rate during training.

We will apply some callbacks later.

For full documentation on `callbacks` see https://keras.io/callbacks/

### What are the steps to optimizing our network?

```python
import tensorflow as tf
import keras
from keras import layers
from keras import models
from keras import utils
from keras.layers import Dense
from keras.models import Sequential
from keras.layers import Flatten
from keras.layers import Dropout
from keras.layers import Activation
from keras.regularizers import l2
from keras.optimizers import SGD
from keras.optimizers import RMSprop
from keras import datasets

from keras import losses
from sklearn.utils import shuffle

print(tf.VERSION)
print(tf.keras.__version__)

# fix random seed for reproducibility
np.random.seed(5)
```

    1.12.0
    2.1.6-tf

    Using TensorFlow backend.

### Step 1 -  Deciding on the network topology (not really considered optimization but is obviously very important)

We will use the MNIST dataset which consists of grayscale images of handwritten digits (0-9) whose dimension is 28x28 pixels. Each pixel is 8 bits so its value ranges from 0 to 255.

```python
#mnist = tf.keras.datasets.mnist
mnist = keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train.shape, y_train.shape
```

    ((60000, 28, 28), (60000,))

Each label is a number between 0 and 9

```python
print(y_train)
```

    [5 0 4 ... 5 6 8]

Let's look at some 10 of the images

```python
plt.figure(figsize=(10,10))
for i in range(10):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(x_train[i], cmap=plt.cm.binary)
    plt.xlabel(y_train[i])
```

```python
x_train[45].shape
x_train[45, 15:20, 15:20]
```

    array([[ 11, 198, 231,  41,   0],
           [ 82, 252, 204,   0,   0],
           [253, 253, 141,   0,   0],
           [252, 220,  36,   0,   0],
           [252,  96,   0,   0,   0]], dtype=uint8)

```python
print(f'We have {x_train.shape[0]} train samples')
print(f'We have {x_test.shape[0]} test samples')
```

    We have 60000 train samples
    We have 10000 test samples

#### Preprocessing the data

To run our NN we need to pre-process the data

* First we need to make the 2D image arrays into 1D (flatten them). We can either perform this by using array reshaping with `numpy.reshape()` or the `keras`' method for this: a layer called `tf.keras.layers.Flatten` which transforms the format of the images from a 2d-array (of 28 by 28 pixels), to a 1D-array of 28 * 28 = 784 pixels.

* Then we need to normalize the pixel values (give them values between 0 and 1) using the following transformation:

\begin{align}
x := \dfrac{x - x_{min}}{x_{max} - x_{min}}
\textrm{}
\end{align}

In our case $x_{min} = 0$ and $x_{max} = 255$ so the formula becomes simply $x := {x}/255$

```python
# normalize the data
x_train, x_test = x_train / 255.0, x_test / 255.0
```

```python
# reshape the data into 1D vectors
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)

num_classes = 10
```

```python
x_train.shape[1]
```

    784

Now let's prepare our class vector (y) to a binary class matrix, e.g. for use with categorical_crossentropy.

```python
# Convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
```

```python
y_train[0]
```

    array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)

Now we are ready to build the model!

### Step 2 - Adjusting the `learning rate`

One of the most common optimization algorithm is Stochastic Gradient Descent (SGD). The hyperparameters that can be optimized in SGD are `learning rate`, `momentum`, `decay` and `nesterov`.

`Learning rate` controls the weight at the end of each batch, and `momentum` controls how much to let the previous update influence the current weight update. `Decay` indicates the learning rate decay over each update, and `nesterov` takes the value True or False depending on if we want to apply Nesterov momentum. Typical values for those hyperparameters are lr=0.01, decay=1e-6, momentum=0.9, and nesterov=True.

The learning rate hyperparameter goes into the `optimizer` function which we will see below.

Let's implement a learning rate adaptation schedule in `Keras`. We'll start with SGD and a learning rate value of 0.1. We will then train the model for 40
epochs and set the decay argument to 0.002 (0.1/50). We also include a momentum value of 0.8 since that seems to work well when using an adaptive learning rate.

```python
epochs = 60
learning_rate = 0.1
decay_rate = learning_rate / epochs
momentum = 0.8

sgd = SGD(lr=learning_rate, momentum=momentum, decay=decay_rate, nesterov=False)
```

```python
# build the model
input_dim = x_train.shape[1]

lr_model = Sequential()
lr_model.add(Dense(64, activation=tf.nn.relu, kernel_initializer='uniform',
                input_dim = input_dim)) # fully-connected layer with 64 hidden units
lr_model.add(Dense(64, kernel_initializer='uniform', activation=tf.nn.relu))
lr_model.add(Dense(num_classes, kernel_initializer='uniform', activation=tf.nn.softmax))

# compile the model
lr_model.compile(loss='categorical_crossentropy',
              optimizer=sgd,
              metrics=['acc'])
```

```python
%%time
# Fit the model
batch_size = 28

lr_model_history = lr_model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/60
    60000/60000 [==============================] - 3s 43us/step - loss: 0.2856 - acc: 0.9072 - val_loss: 0.1419 - val_acc: 0.9559
    Epoch 2/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.1043 - acc: 0.9684 - val_loss: 0.1052 - val_acc: 0.9663
    Epoch 3/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0778 - acc: 0.9759 - val_loss: 0.0948 - val_acc: 0.9715
    Epoch 4/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0652 - acc: 0.9798 - val_loss: 0.0866 - val_acc: 0.9728
    Epoch 5/60
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0571 - acc: 0.9828 - val_loss: 0.0861 - val_acc: 0.9729
    Epoch 6/60
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0516 - acc: 0.9843 - val_loss: 0.0837 - val_acc: 0.9741
    Epoch 7/60
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0473 - acc: 0.9857 - val_loss: 0.0842 - val_acc: 0.9745
    Epoch 8/60
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0439 - acc: 0.9870 - val_loss: 0.0858 - val_acc: 0.9725
    Epoch 9/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0414 - acc: 0.9879 - val_loss: 0.0824 - val_acc: 0.9748
    Epoch 10/60
    60000/60000 [==============================] - 3s 42us/step - loss: 0.0392 - acc: 0.9889 - val_loss: 0.0820 - val_acc: 0.9750
    Epoch 11/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0373 - acc: 0.9895 - val_loss: 0.0824 - val_acc: 0.9745
    Epoch 12/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0355 - acc: 0.9903 - val_loss: 0.0823 - val_acc: 0.9741
    Epoch 13/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0343 - acc: 0.9907 - val_loss: 0.0817 - val_acc: 0.9750
    Epoch 14/60
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0330 - acc: 0.9912 - val_loss: 0.0812 - val_acc: 0.9754
    Epoch 15/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0320 - acc: 0.9912 - val_loss: 0.0809 - val_acc: 0.9757
    Epoch 16/60
    60000/60000 [==============================] - 3s 44us/step - loss: 0.0310 - acc: 0.9917 - val_loss: 0.0815 - val_acc: 0.9761
    Epoch 17/60
    60000/60000 [==============================] - 3s 45us/step - loss: 0.0302 - acc: 0.9919 - val_loss: 0.0814 - val_acc: 0.9759
    Epoch 18/60
    60000/60000 [==============================] - 3s 46us/step - loss: 0.0292 - acc: 0.9924 - val_loss: 0.0815 - val_acc: 0.9755
    Epoch 19/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0286 - acc: 0.9928 - val_loss: 0.0811 - val_acc: 0.9759
    Epoch 20/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0279 - acc: 0.9929 - val_loss: 0.0808 - val_acc: 0.9759
    Epoch 21/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0272 - acc: 0.9931 - val_loss: 0.0819 - val_acc: 0.9767
    Epoch 22/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0267 - acc: 0.9933 - val_loss: 0.0820 - val_acc: 0.9753
    Epoch 23/60
    60000/60000 [==============================] - 3s 42us/step - loss: 0.0261 - acc: 0.9936 - val_loss: 0.0812 - val_acc: 0.9765
    Epoch 24/60
    60000/60000 [==============================] - 3s 47us/step - loss: 0.0256 - acc: 0.9935 - val_loss: 0.0821 - val_acc: 0.9759
    Epoch 25/60
    60000/60000 [==============================] - 3s 50us/step - loss: 0.0250 - acc: 0.9939 - val_loss: 0.0821 - val_acc: 0.9760
    Epoch 26/60
    60000/60000 [==============================] - 3s 47us/step - loss: 0.0246 - acc: 0.9939 - val_loss: 0.0822 - val_acc: 0.9755
    Epoch 27/60
    60000/60000 [==============================] - 3s 45us/step - loss: 0.0242 - acc: 0.9941 - val_loss: 0.0820 - val_acc: 0.9768
    Epoch 28/60
    60000/60000 [==============================] - 3s 49us/step - loss: 0.0238 - acc: 0.9941 - val_loss: 0.0825 - val_acc: 0.9761
    Epoch 29/60
    60000/60000 [==============================] - 3s 48us/step - loss: 0.0233 - acc: 0.9944 - val_loss: 0.0824 - val_acc: 0.9761
    Epoch 30/60
    60000/60000 [==============================] - 3s 46us/step - loss: 0.0231 - acc: 0.9946 - val_loss: 0.0830 - val_acc: 0.9760
    Epoch 31/60
    60000/60000 [==============================] - 3s 48us/step - loss: 0.0228 - acc: 0.9947 - val_loss: 0.0829 - val_acc: 0.9755
    Epoch 32/60
    60000/60000 [==============================] - 3s 48us/step - loss: 0.0224 - acc: 0.9947 - val_loss: 0.0829 - val_acc: 0.9760
    Epoch 33/60
    60000/60000 [==============================] - 3s 48us/step - loss: 0.0220 - acc: 0.9948 - val_loss: 0.0831 - val_acc: 0.9758
    Epoch 34/60
    60000/60000 [==============================] - 3s 51us/step - loss: 0.0218 - acc: 0.9949 - val_loss: 0.0827 - val_acc: 0.9754
    Epoch 35/60
    60000/60000 [==============================] - 3s 44us/step - loss: 0.0215 - acc: 0.9951 - val_loss: 0.0832 - val_acc: 0.9762
    Epoch 36/60
    60000/60000 [==============================] - 3s 45us/step - loss: 0.0213 - acc: 0.9952 - val_loss: 0.0833 - val_acc: 0.9759
    Epoch 37/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0210 - acc: 0.9953 - val_loss: 0.0836 - val_acc: 0.9764
    Epoch 38/60
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0208 - acc: 0.9952 - val_loss: 0.0836 - val_acc: 0.9758
    Epoch 39/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0205 - acc: 0.9954 - val_loss: 0.0835 - val_acc: 0.9760
    Epoch 40/60
    60000/60000 [==============================] - 2s 38us/step - loss: 0.0203 - acc: 0.9955 - val_loss: 0.0838 - val_acc: 0.9761
    Epoch 41/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0201 - acc: 0.9956 - val_loss: 0.0840 - val_acc: 0.9764
    Epoch 42/60
    60000/60000 [==============================] - 3s 47us/step - loss: 0.0198 - acc: 0.9957 - val_loss: 0.0841 - val_acc: 0.9761
    Epoch 43/60
    60000/60000 [==============================] - 3s 47us/step - loss: 0.0196 - acc: 0.9957 - val_loss: 0.0842 - val_acc: 0.9761
    Epoch 44/60
    60000/60000 [==============================] - 3s 43us/step - loss: 0.0194 - acc: 0.9958 - val_loss: 0.0840 - val_acc: 0.9760
    Epoch 45/60
    60000/60000 [==============================] - 2s 42us/step - loss: 0.0193 - acc: 0.9959 - val_loss: 0.0841 - val_acc: 0.9757
    Epoch 46/60
    60000/60000 [==============================] - 3s 42us/step - loss: 0.0190 - acc: 0.9958 - val_loss: 0.0848 - val_acc: 0.9760
    Epoch 47/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0189 - acc: 0.9959 - val_loss: 0.0846 - val_acc: 0.9760
    Epoch 48/60
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0187 - acc: 0.9961 - val_loss: 0.0846 - val_acc: 0.9764
    Epoch 49/60
    60000/60000 [==============================] - 3s 45us/step - loss: 0.0185 - acc: 0.9961 - val_loss: 0.0845 - val_acc: 0.9763
    Epoch 50/60
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0184 - acc: 0.9963 - val_loss: 0.0846 - val_acc: 0.9761
    Epoch 51/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0182 - acc: 0.9962 - val_loss: 0.0846 - val_acc: 0.9762
    Epoch 52/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0181 - acc: 0.9963 - val_loss: 0.0849 - val_acc: 0.9761
    Epoch 53/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0179 - acc: 0.9962 - val_loss: 0.0851 - val_acc: 0.9764
    Epoch 54/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0177 - acc: 0.9964 - val_loss: 0.0852 - val_acc: 0.9768
    Epoch 55/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0176 - acc: 0.9963 - val_loss: 0.0852 - val_acc: 0.9763
    Epoch 56/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0175 - acc: 0.9965 - val_loss: 0.0854 - val_acc: 0.9763
    Epoch 57/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0174 - acc: 0.9965 - val_loss: 0.0854 - val_acc: 0.9763
    Epoch 58/60
    60000/60000 [==============================] - 2s 39us/step - loss: 0.0172 - acc: 0.9965 - val_loss: 0.0855 - val_acc: 0.9763
    Epoch 59/60
    60000/60000 [==============================] - 2s 38us/step - loss: 0.0171 - acc: 0.9966 - val_loss: 0.0855 - val_acc: 0.9763
    Epoch 60/60
    60000/60000 [==============================] - 2s 38us/step - loss: 0.0170 - acc: 0.9967 - val_loss: 0.0853 - val_acc: 0.9757
    CPU times: user 4min 27s, sys: 1min 1s, total: 5min 28s
    Wall time: 2min 32s

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(lr_model_history.history['loss']), 'r', label='train')
ax.plot(np.sqrt(lr_model_history.history['val_loss']), 'b' ,label='val')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Loss', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(lr_model_history.history['acc']), 'r', label='train')
ax.plot(np.sqrt(lr_model_history.history['val_acc']), 'b' ,label='val')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Accuracy', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

**Exercise:** Write a function that performs exponential learning rate decay using `LearningRateScheduler`

$lr = lr0*e^{(-kt)}$ t is the iteration number

### Step 3 - Choosing an `optimizer` and a `loss function`

When constructing a model and using it to make our predictions, for example to assign label scores to images ("cat", "plane", etc), we want to measure our success or failure by defining a "loss" function (or objective function). The goal of optimization is to efficiently calculate the parameters/weights that minimize this loss function. `keras` provides various types of [loss functions](https://github.com/keras-team/keras/blob/master/keras/losses.py).

Sometimes the "loss" function measures the "distance". We can define this "distance" between two data points in various ways suitable to the problem or dataset.

Distance

- Euclidean
- Manhattan
- others such as Hamming which measures distances between strings, for example. The Hamming distance of "carolin" and "cathrin" is 3.

Loss functions
- MSE (for regression)
- categorical cross-entropy (for classification)
- binary cross entropy (for classification)

```python
# build the model
input_dim = x_train.shape[1]

model = Sequential()
model.add(Dense(64, activation=tf.nn.relu, kernel_initializer='uniform',
                input_dim = input_dim)) # fully-connected layer with 64 hidden units
#model.add(Dropout(0.5))
model.add(Dense(64, kernel_initializer='uniform', activation=tf.nn.relu))
#model.add(Dropout(0.5))
model.add(Dense(num_classes, kernel_initializer='uniform', activation=tf.nn.softmax))
```

```python
# defining the parameters for RMSprop (I used the keras defaults here)
rms = RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)

model.compile(loss='categorical_crossentropy',
              optimizer=rms,
              metrics=['acc'])
```

### Step 4 - Deciding on the `batch size` and `number of epochs`

```python
%%time
batch_size = 32
epochs = 20

model_history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/20
    60000/60000 [==============================] - 3s 46us/step - loss: 0.3931 - acc: 0.8850 - val_loss: 0.2082 - val_acc: 0.9400
    Epoch 2/20
    60000/60000 [==============================] - 3s 42us/step - loss: 0.1696 - acc: 0.9502 - val_loss: 0.1334 - val_acc: 0.9595
    Epoch 3/20
    60000/60000 [==============================] - 2s 40us/step - loss: 0.1208 - acc: 0.9641 - val_loss: 0.1162 - val_acc: 0.9677
    Epoch 4/20
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0995 - acc: 0.9710 - val_loss: 0.1068 - val_acc: 0.9712
    Epoch 5/20
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0846 - acc: 0.9751 - val_loss: 0.1035 - val_acc: 0.9728
    Epoch 6/20
    60000/60000 [==============================] - 3s 42us/step - loss: 0.0733 - acc: 0.9784 - val_loss: 0.1203 - val_acc: 0.9713
    Epoch 7/20
    60000/60000 [==============================] - 2s 42us/step - loss: 0.0660 - acc: 0.9807 - val_loss: 0.0993 - val_acc: 0.9735
    Epoch 8/20
    60000/60000 [==============================] - 3s 43us/step - loss: 0.0599 - acc: 0.9834 - val_loss: 0.1143 - val_acc: 0.9713
    Epoch 9/20
    60000/60000 [==============================] - 3s 46us/step - loss: 0.0533 - acc: 0.9851 - val_loss: 0.1217 - val_acc: 0.9695
    Epoch 10/20
    60000/60000 [==============================] - 3s 47us/step - loss: 0.0493 - acc: 0.9866 - val_loss: 0.1109 - val_acc: 0.9735
    Epoch 11/20
    60000/60000 [==============================] - 3s 42us/step - loss: 0.0441 - acc: 0.9875 - val_loss: 0.1186 - val_acc: 0.9742
    Epoch 12/20
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0422 - acc: 0.9883 - val_loss: 0.1222 - val_acc: 0.9725
    Epoch 13/20
    60000/60000 [==============================] - 2s 41us/step - loss: 0.0397 - acc: 0.9897 - val_loss: 0.1327 - val_acc: 0.9725
    Epoch 14/20
    60000/60000 [==============================] - 3s 43us/step - loss: 0.0359 - acc: 0.9905 - val_loss: 0.1365 - val_acc: 0.9724
    Epoch 15/20
    60000/60000 [==============================] - 3s 42us/step - loss: 0.0342 - acc: 0.9910 - val_loss: 0.1445 - val_acc: 0.9748
    Epoch 16/20
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0317 - acc: 0.9915 - val_loss: 0.1420 - val_acc: 0.9758
    Epoch 17/20
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0303 - acc: 0.9920 - val_loss: 0.1534 - val_acc: 0.9740
    Epoch 18/20
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0280 - acc: 0.9928 - val_loss: 0.1549 - val_acc: 0.9727
    Epoch 19/20
    60000/60000 [==============================] - 2s 40us/step - loss: 0.0263 - acc: 0.9938 - val_loss: 0.1691 - val_acc: 0.9731
    Epoch 20/20
    60000/60000 [==============================] - 3s 43us/step - loss: 0.0266 - acc: 0.9934 - val_loss: 0.1504 - val_acc: 0.9759
    CPU times: user 1min 32s, sys: 20.9 s, total: 1min 53s
    Wall time: 50.7 s

```python
from sklearn.metrics import r2_score as r2

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
print('Test R2:', r2(y_test, model.predict(x_test)))
```

    Test loss: 0.1503745241531124
    Test accuracy: 0.9759
    Test R2: 0.9542278387249276

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(model_history.history['acc']), 'r', label='train_acc')
ax.plot(np.sqrt(model_history.history['val_acc']), 'b' ,label='val_acc')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Accuracy', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(model_history.history['loss']), 'r', label='train')
ax.plot(np.sqrt(model_history.history['val_loss']), 'b' ,label='val')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Loss', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

```python
print(model_history.history.keys())
```

    dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])

### Step 5 - Random restarts

This method does not seem to have an implementation in `keras`. We will leave it as a home exercise!

Hint: you can use `keras.callbacks.LearningRateScheduler`.

### Tuning the Hyperparameters using Cross Validation

Now instead of trying different values by hand, we will use GridSearchCV from Scikit-Learn to try out several values for our hyperparameters and compare the results.

To do cross-validation with `keras` we will use the wrappers for the Scikit-Learn API. They provide a way to use Sequential Keras models (single-input only) as part of your Scikit-Learn workflow.

There are two wrappers available:

`keras.wrappers.scikit_learn.KerasClassifier(build_fn=None, **sk_params)`, which implements the Scikit-Learn classifier interface,

`keras.wrappers.scikit_learn.KerasRegressor(build_fn=None, **sk_params)`, which implements the Scikit-Learn regressor interface.

```python
import numpy
from sklearn.model_selection import GridSearchCV
from keras.wrappers.scikit_learn import KerasClassifier
```

#### Trying different weight initializations

```python
# Function to create model, required for KerasClassifier
def create_model(init_mode='uniform'):
    # define model
    model = Sequential()
    model.add(Dense(64, kernel_initializer=init_mode, activation=tf.nn.relu, input_dim=784))
    model.add(Dropout(0.5))
    model.add(Dense(64, kernel_initializer=init_mode, activation=tf.nn.relu))
    model.add(Dropout(0.5))
    model.add(Dense(10, kernel_initializer=init_mode, activation=tf.nn.softmax))
    # compile model
    model.compile(loss='categorical_crossentropy',
              optimizer=RMSprop(),
              metrics=['accuracy'])
    return model
```

```python
%%time
seed = 7
numpy.random.seed(seed)
batch_size = 128
epochs = 10

model_CV = KerasClassifier(build_fn=create_model, epochs=epochs, batch_size=batch_size, verbose=0)
# define the grid search parameters
init_mode = ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform']

param_grid = dict(init_mode=init_mode)
grid = GridSearchCV(estimator=model_CV, param_grid=param_grid, n_jobs=-1, cv=3)
grid_result = grid.fit(x_train, y_train)

# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))
```

    Best: 0.948967 using {'init_mode': 'glorot_uniform'}
    0.946117 (0.002852) with: {'init_mode': 'uniform'}
    0.948400 (0.001192) with: {'init_mode': 'lecun_uniform'}
    0.948117 (0.001310) with: {'init_mode': 'normal'}
    0.112367 (0.002416) with: {'init_mode': 'zero'}
    0.948367 (0.002593) with: {'init_mode': 'glorot_normal'}
    0.948967 (0.002210) with: {'init_mode': 'glorot_uniform'}
    0.947400 (0.002060) with: {'init_mode': 'he_normal'}
    0.946983 (0.002908) with: {'init_mode': 'he_uniform'}
    CPU times: user 21 s, sys: 3.6 s, total: 24.6 s
    Wall time: 1min 36s

### Save Your Neural Network Model to JSON

The Hierarchical Data Format (HDF5) is a data storage format for storing large arrays of data including values for the weights in a neural network.
You can install HDF5 Python module: pip install h5py

Keras gives you the ability to describe and save any model using the JSON format.

```python
from keras.models import model_from_json

# serialize model to JSON
model_json = model.to_json()

with open("model.json", "w") as json_file:
    json_file.write(model_json)

# save weights to HDF5
model.save_weights("model.h5")
print("Model saved")

# when you want to retrieve the model: load json and create model
json_file = open('model.json', 'r')
saved_model = json_file.read()
# close the file as good practice
json_file.close()
model_from_json = model_from_json(saved_model)
# load weights into new model
model_from_json.load_weights("model.h5")
print("Model loaded")
```

### Exercise 1: Perform a GridSearch for Learning rate and number of epochs combined

We will try various small standard learning rates, and momentum values from 0.1 to 0.9 in steps of 0.2.

We want to include the number of epochs in an optimization as there is a dependency between the learning rate, the batch size, and the number of epochs.

```python
#your code here

```


# Logistic Regression

- Logistic regression is a classification algorithm, don’t confuse with the name regression.
- Logistic Regression is basically Supervised ML algorithm that is used for __binary classification__
- In classification problem the values we predict take on only a small number of discrete values. For example, "yes or no", "cat or dog", "cat or dog or bird", etc.
- In logistic regression our main goal is to __predict the probability of the discrete class labels__

Some examples of classification problems:
1. Spam detection
2. Detection of fraudulent transanctions
3. Malignant/Benign tumor

__Assumption of Logistic Regression:__

* Input space can be __separated into two nice ‘regions’, one for each class, by a defined boundary.__

* Your data must be **linearly** separable in __n dimensions__

### Hypothesis for classification problem

In classification problem we want the probability of the outcomes. So we would like f(x) to be between 0 and 1.

\begin{equation}
0
\leq
f_\theta(x)
\leq
1
\end{equation}

* Mathematically, in Linear Regression __hypothesis function $f_{\theta}(x)$ can be >1 or <0__

* For Logistic Regression we need __hypothesis function which can satisfy the math rule:__ $0 \leq f_{\theta}(x)\leq 1$

### Intuitive Example to understand math behind LoR

__Example / Problem statement: __

* We are provided a sample of 100 customers.

* We need to predict the probability whether a customer will buy (y) a particular magazine or not based on their
  age

* As you can see, we’ve a categorical outcome variable, we’ll use logistic regression.

#### Step 1:

To start with logistic regression, I’ll first write the simple linear regression equation with dependent variable

$$ \large f_{\theta}(x) = mx + C ------ (1) $$

#### Step 2:

So, final equation form based on our example will be

$$ \large f(readers\;will\;buy\; magazine\; or\; not) = m * x_{age} + C ------(2)$$

* Now consider Y as function of probability. Why? because we need the chance (probability of the whether buyer will buy the magazine or not)

* This function is established using two things:
  * Probability of Success(p) means reader buys magazine
  * Probability of Failure(1-p) means reader doesn't buy magazine.

* Probability p should meet following criteria:
    1. It must always be positive (since p >= 0)
    2. It must always be less than equals to 1 (since p <= 1)

#### Step 3:

__Since probability must always be positive, we’ll put the linear equation in exponential form.__

For any value of slope and dependent variable, exponent of this equation will never be negative.

$$ \large f_{\theta}(x) = exp(C + mx_{Age}) = e^{(C + mx_{Age})}    ------- (3) $$

Here,

* P  =  Probability of readers' buy magazine or not
* exp OR e =  exponential symbol OR Euler's number.

#### Step 4:

To make the probability less than 1, we must divide p by a number greater than p. This can simply be done by:

$$ \large f_{\theta}(x)  =  \frac{exp(C + mx_{Age})}{exp(C + mx_{Age}) + 1}   =  \frac{ e^{(C + mx_{Age})}}{e^{(C + mx_{Age})} + 1 } = p  ----- (4)$$

#### Step 5:

Now by using equation 1, 2, 3 and 4 we can redefine the probability as:

 $$ p = \frac {e^{g}}{ 1 + e^{g}} ----- (5)$$

 where $ g_{\theta}(x) = mx + C = Decision\_boundary $

 * Here we have just __substitute the C + mx{Age} with Y__

 * Here p is the probability of success. This __Equation (5) is the Logit Function__

```

```

If p is the probability of success, 1-p will be the probability of failure which can be written as:

$$q = 1 - p = 1 - \frac {e^{g}}{ 1 + e^{g}}    --- (6)$$

* Here q is the probability of failure

#### Step 6:

Now let's __divide the equation 5 and 6__ so that we get following equation

$$ \frac {p}{1-p} = e^{g}    --- (7)$$

#### Step 7:

After __taking log on both side__, So that we can get the value of Y

$$ \large log(\frac{p}{1-p}) = g_\theta(x)    --- (8) $$

Now, let's __put the value of Y in equation 8 from equation 2__

$$ \large log(\frac{p}{1-p}) = C + mx_{Age}   --- (9) $$

#### Notes:

* The __equation 9 is used in Logistic Regression and It is called Logit(p). __

* Here __(p/1-p) is called the Odd Ratio (OR) or Odds(p)__

* Whenever the __log of Odd Ratio is found to be positive, then the probability of success is always more than
  50%.__

* You can see __probability never goes below 0 and above 1. __

### Why we take odds instead of probabilities?

Let $P(X)$ denote the probability of an event X occurring. In that case, the __odds ratio (OR(X))__ is defined as

$$ \large \frac{P(X)}{1-P(X)}$$

* It is essentially the ratio of the probability of the event happening, vs. it not happening.

* It is clear that __probability and odds__ convey the __exact same information.__

* __But as $P(X)$ goes from 0 to 1, OR(X) goes from 0 to infinity__

* The advantage is that the __odds defined on 0 to infinity__ which map to __log-odds on -infinity to +infinity__

#### Sigmoid function

We will now see how we arrive at the sigmoid function, starting form the log_odds function:

$log(\frac{y}{1-y}) = g_\theta(x) = \theta^TX$

$y = (1-y)*e^{\theta^TX}$

$y = e^x - y*e^{\theta^TX}$

$y + ye^x = e^{\theta^TX}$

$y*(1 + e^{\theta^TX}) = e^{\theta^TX}$

$y = \frac{e^{\theta^TX}}{1+e^{\theta^TX}}$

$y = \frac{1}{\frac{1}{e^{\theta^TX}} + 1}$

$y = \frac{1}{1 + e^{-\theta^TX}}$

Finally, the Sigmoid function, as derived above, is given below:

$$\sigma(x)=\frac{1}{1+e^{-x}}$$

Sigmoid function is used to restrict the f(x) between 0 and 1.

**Remark:** The derivative of logistic function can be written $\text{d}\sigma = \sigma (1-\sigma)$.

```python
'''Explore the signature of sigmoid function'''
import numpy as np
import matplotlib.pyplot as plt
import math
%matplotlib inline

sigmoid = lambda x: 1/(1 + np.exp(-x))
x = np.linspace(-10,10,100)

plt.plot(x, sigmoid(x), 'o')

# Draw the grid line in background.
plt.grid()

# Title & Subtitle
plt.title('Sigmoid function')

# write the Sigmoid formula
plt.text(4, 0.8, r'$\sigma(x)=\frac{1}{1+e^{-x}}$', fontsize=15)

plt.show()
```

### Why do we need sigmoid function?

* This function will give us the __output value between [0,1]__

* When you __perform derivative of $e^{x}$ it gives you back $e^{x}$ and this function is the only known
  differentiable function with this mathematical property. That makes it more convenient for calculation.__

## Cost functions for Logistic Regression

#### Can we directly use cost function of linear regression which is Sum of sequared Errors?

* The Answer is __No, we can't__

* Linear regression uses mean squared error as its cost function. If this is used for logistic regression, then it
  will be generate a non-convex function of our input intercept values or parameter (theta) values

* Gradient Descent will converge into global minimum for function that is convex.

## Cost function for a value of f(x):

$$ \large \text{Cost function} (f_{\theta}(x))  = \begin{cases}
  -\log(f_{\theta}(x)) & \text{if $y = 1$} \\
  -\log(1\; -\; f_{\theta}(x) ) & \text{if $y = 0$}
\end{cases}$$

$$ \text{Cost function for entire data set with m observation} = J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \text{Cost function} (f_{\theta}(x^{i},y^{i})) $$

### Combine both cases of Cost function in one equation

$$ \large \text{Cost function} (f_{\theta}(x),y)  =
  -y\log(f_{\theta}(x))-(1-y)\log(1\; -\; f_{\theta}(x) )
$$

```python
'''Explore the signature of sigmoid function'''
import numpy as np
import matplotlib.pyplot as plt
import math
%matplotlib inline

sigmoid = lambda x: 1/(1 + np.exp(-x))
x = np.linspace(-10,10,100)

plt.plot(x, [-math.log(x)-math.log(1-x) for x in sigmoid(x)], 'o')

# Draw the grid line in background.
plt.grid()

# Title & Subtitle
plt.title('Loss function for Logistic Regression')

# write the Loss formula
plt.text(-8, 9.5, r'$ -y\log(f_{\theta}(x))-(1-y)\log(1\; -\; f_{\theta}(x) )$', fontsize=15)

plt.show()
```

$$\large J(\theta) = - \frac{1}{m}\sum_{i=1}^{m}[y^{(i)}\log(f_{\theta}(x^{(i)}) + (1-y^{(i)})\log(1-f_{\theta}(x^{(i)}))]$$

__Now we need to minimize this cost function $J(\theta)$__

### Gradient Descent

* In order to minimize the cost function we need to derive its __partial derivative with respect to $\theta$__

$$\large J(\theta) = - \frac{1}{m}\sum_{i=1}^{m}[y^{(i)}\log(\frac{1}{1+e^{-\theta x^{(i)}}}) + (1-y^{(i)})\log(1-\frac{1}{1+e^{-\theta x^{(i)}}})]$$

__Partial derivate that we use for updating our cost function or minimize our error function is given below__

$$\large \frac{\partial J}{\partial \theta}  = \frac{1}{m}\sum_{i=1}^{m}[x^{(i)}_{j}f_{\theta}(x^{(i)}) - y^{(i)}x^{(i)}_{j}]$$

#### Final Gradient descent equation is as below

__Repeat this until we get $min_{\theta}J(\theta)$:__

        {

$$ \large \theta_{j} := \theta_{j} -\alpha \sum_{i=0}^{m}[x^{(i)}_{j}f_{\theta}(x^{(i)}) - y^{(i)}x^{(i)}_{j}] $$

        }

__Here $\alpha$ is Learning Rate __

### Comparing Classification & Regression

|Properties| Supervised Classification (Logistic Regression) | Regression (Linear Regression)|
|:---:|:---: |:---:|
|  Output Type | Discrete Value (Class labels) | Continuous (number) |
|  What are you trying to achieve? | Decision Boundary or How to choose value for parameters theta | Best line of fit |
|  Evaluation | Accuracy | Sum of Sequared Error or $r^{2} Error$|

### Intution of logistic regression

Its about finding the decision boundary.

#### Scenario 1

```python
import matplotlib.pyplot as plt
x1 = np.random.randn(100)
y1 = np.random.randn(100)
x2 = np.random.randn(100)
y2 = np.random.randn(100) + 6
plt.plot(x1, y1, "+", x2, y2, "x")
plt.axis('equal')
plt.show()
```

#### Scenario 2

```python
x1 = np.random.randn(100)
y1 = np.random.randn(100) + 3
x2 = np.random.randn(100) + 3
y2 = np.random.randn(100)
plt.plot(x1, y1, "+", x2, y2, "x")
plt.axis('equal')
plt.show()
```

#### Scenario 3

```python
x1 = np.random.randn(100)
y1 = np.random.randn(100)
x3 = np.random.randn(25) + 3
y3 = np.random.randn(25)
x4 = np.random.randn(25) - 3
y4 = np.random.randn(25)
x5 = np.random.randn(25)
y5 = np.random.randn(25) + 3
x6 = np.random.randn(25)
y6 = np.random.randn(25) - 3
x3 = np.append(x3 , x5)
y3 = np.append(y3, y5)
x4 = np.append(x4, x6)
y4 = np.append(y4, y6)
x2 = np.append(x3, x4)
y2 = np.append(y3, y4)
#x2 = np.append(x3, x6)
#y2 = np.append(y2 , y6)
plt.plot(x1, y1, "+", x2, y2, "x")
plt.axis('equal')
plt.show()
```

```python

```


# $\LaTeX$ Math
This is just markdown that can include latex math.

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

# System Info

```python
import IPython
print(IPython.sys_info())
```

# Data

```python
import pandas as pd
import numpy as np
curve = pd.DataFrame(np.arange(0.0, 2.0, 0.01), columns=list('t'))
curve['x'] = 1 + np.sin(2 * np.pi * curve['t'])
curve
```

# Plot

```python
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [8, 4]
curve.plot.line(x = 't', y = 'x')
```


# Counterfactual Inference in Pyro

Below, we demonstrate a simple model of counterfactual inference in Pyro.  You will see a model relating treatment (cause) to blindness (outcome).  This toy example is technically a structural causal model (SCM).  SCMs enable twin-world counterfactual reasoning.  However, if we are working with a probabilistic programming language like Pyro, we don't have to stick to formal SCMs.  All we need is to have a set of variables from which all other sets of variables in the system are derived deterministically.  These "exogenous" variables are analogous to the initial conditions of a simulation.

The twin-world counterfactual reasoning algorithm is as follows

1. Condition on observed evidence and do inference on the exogenous variables.
2. Apply interventions to the model that reflect the counterfactual setting.
3. Forward sample from the intervention model using the posterior on the exogenous variables.

The intuition is that all the information in the observed world is shunted into the posterior of the exogenous variables, which are then passed over to the counterfactual world and forward probagated.

### Example: Blind treatment

This is an implementation of an example from Peters et al. 2017.

Consider a treatment study, where a company introduced a new medicine for eyes.

Suppose this is the true underlying model for the causal effect of Treatment $T$ ($T=1$ if the treatment was given) and the result $B$ ($B=1$ if the person goes blind).
$$
\begin{align}
N_T &\sim Ber(.5)\\
N_B &\sim Ber(.01) \\
 T &:= N_T \\
 B &:= T * N_B + (1-T)*(1-N_B) \\
\end{align}
$$

Suppose patient with poor eyesight comes to the hospital and goes blind (B=1) after the doctor gives treatment (T=1).

We can ask "what would have happened had the doctor administered treatment T = 0?"

Here is the steps we follow to answer this counterfactual question.

1. Retrieve noise variables given observation.

   We observed $B=T=1$. Plugging that to the equations above

   $$\begin{align}
   1 &= N_T\\
   1 &= 1*N_B + (1-1)*(1-N_B)
   \end{align}$$

	So, $N_T=1$ and $N_B = 1$

2. Intervene on $T$. Put $T=0$, and solve for $B$

$$
\begin{align}
 T &= 0\\
 B &= 0 * 1 + (1-0)*(1-1) = 0 \\
\end{align}
$$

Thus, by this model, person would not have gone blind if the treatment was not given.

### Implementation of the twin-world counterfactual inference algorithm

```python
import pyro
import pyro.distributions as dist
from pyro.infer import Importance, EmpiricalMarginal
from statistics import mean
import torch
import matplotlib.pyplot as plt

%matplotlib inline

exogenous_dists = {
    "N_t": dist.Bernoulli(torch.tensor(.5)),
    "N_b": dist.Bernoulli(torch.tensor(.01))
}

def model(exogenous_dists):
    N_t = pyro.sample("N_t", exogenous_dists["N_t"])
    N_b = pyro.sample("N_b", exogenous_dists["N_b"])
    t = pyro.sample("t", dist.Delta(N_t))
    b = pyro.sample("b", dist.Delta(t * N_b + (1. - t) * (1 - N_b)))
    return t, b

```

#### Condition on the observed world and do inference on the exogenous variables

```python
conditioned = pyro.condition(
    model,
    {'t': torch.tensor(1.), 'b': torch.tensor(1.)}
)
# Sanity check: Exogenous vars become degenerate in posterior
posterior = Importance(
    conditioned,
    num_samples=100).run(exogenous_dists)
N_t_marginal = EmpiricalMarginal(posterior, "N_t")

N_T_samples = [N_t_marginal().item() for _ in range(1000)]
print(mean(N_T_samples))

N_b_marginal = EmpiricalMarginal(posterior, "N_b")

N_b_samples = [N_b_marginal().item() for _ in range(1000)]
print(mean(N_b_samples))
```

    1.0
    1.0

#### Apply counterfactual intervention to the model

```python
cf_model = pyro.do(model, {'t': torch.tensor(0.)})
```

#### Forward propagate posterior of exogenous variables

```python
updated_exogenous_dists = {
    "N_t": dist.Bernoulli(torch.tensor(mean(N_T_samples))),
    "N_b": dist.Bernoulli(torch.tensor(mean(N_b_samples)))
}
samples = [cf_model(updated_exogenous_dists) for _ in range(100)]
b_samples = [b for _, b in samples]
plt.hist(b_samples)

```

Note that the inference here is brittle, and you may have to run this script a few times to get the appropriate output.  This is typically the case with using probabilistic programming to implement twin-world counterfactuals.

Newer languages specifically designed for this problem, like [Omega

## Reference
Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017


# Graph State Examples

## Quantum Cryptography and Certifying Graph States

### Blind Quantum Computing

Blind quantum computing is an area of quantum cryptography which let's a client (Alice), without sohpisticated quantum computing technologies, to transmit her computation to a server (Bob), who can perform her computation on his quantum computer. Blind quantum computing allows one to perform such transaction while protecting privacy. In this notebook we will discuss using graph states for a particular blind quantum computing protocol.

### Stabilizer Groups for Graph States

Before we get into verifying quantum computations, let's review graph states and their stabilizer groups. A **graph state**, as we have defined in the chapter on entanglement and the chapter on error correction, are constructed from a graph. Given some (simple) graph $G$, we construct the graph state by first putting all qubits into the $|+\rangle$ state by applying the Hadamard gate to all of the qubits. Then, for each edge, we apply a corresponding controlled-$Z$ gate between the two qubits corresponding to the two vertices that are connected by that edge. Recall from our previous chapter on error correction and stabilizer codes that graph states have stabilizer generators of the form

\begin{align}
S_v = X_v \otimes \left( \bigotimes_{n \in Adj(v)} Z_u \right)
\end{align}

where $X_v$ is the Pauli-$X$ gate applied to the qubit corresponding to the vertex (node) $v$ of the graph $G$. The set $Adj(v)$ is the set of all vertices connected to $v$ by an edge, i.e. the vertices *adjacent* to $v$. The operators $Z_u$ are the corresponding Pauli-$Z$ gates applied to the corresponding qubits. The set of operators $\{S_v\}_{v \in V}$ gives a complete set of generators for the stabilizer group of the graph state $|G\rangle$. The *group* of operators (gates generated by these is denoted by

\begin{align}
S_G = \langle S_0, S_1, ..., S_N \rangle.
\end{align}

Here $S_j = S_v$ for some vertex $v \in V$, and the number of vertices is $|V|=N$. So we assume we have an index set $\{0, 1, ..., N-1\}$ labeling the vertices of the graph. There are a total of $2^N$ elements in the group $S_G$.

Here we create a class for building arbitrary graph states given by a graph built in [NetworkX](https://networkx.github.io/documentation/stable/tutorial.html).

```python
from qiskit import *
import networkx as nx

class GraphState(QuantumCircuit):

    def __init__(self, graph):
        super().__init__()
        self.graph = graph
        # Create a quantum register based on the number of nodes in G
        self.qreg = QuantumRegister(len(self.graph.nodes))
        self.creg = ClassicalRegister(len(self.graph.nodes))
        # Create a circuit using the quantum register
        self.circuit = QuantumCircuit(self.qreg, self.creg)
        # For each vertex, apply a Hadamard gate
        for vertex in self.graph.nodes:
            self.circuit.h(vertex)
        # For each edge e={x,y} apply a controlled-Z gate on its vertices
        for x, y in self.graph.edges:
            self.circuit.cz(x, y)
        #         create a node dictionary from node to integer index of a qubit
        #         in a Qiskit circuit
        self.node_dict = dict()
        for count, node in enumerate(self.graph.nodes):
            self.node_dict[node] = count

    def add_node(self):
        raise NotImplemented("Disallowed: don't expect to do this after you have the initial graph state")

    def x_measurement(self, qubit, cbit):
        """Measure 'qubit' in the X-basis, and store the result in 'cbit'"""
        self.circuit.h(qubit)
        self.circuit.measure(qubit, cbit)
        self.circuit.h(qubit)

    def apply_stabilizer(self, node):
        """
        applies the stabilizer generator corresponding to node
        :param self:
        :param node: a node in self.graph
        """
        self.circuit.x(self.node_dict[node])
        for neighbor in self.graph.neighbors(node):
            self.circuit.z(self.node_dict[neighbor])

    def __str__(self):
        return self.circuit.draw('text')

    def __eq__(self, other):
        return nx.is_isomorphic(self.graph, other.graph)

    def __hash__(self):
        return hash((self.graph.nodes, self.graph.edges, self.circuit.width()))
```

### Example 1
---

Let's define the Peterson graph in NetworkX to use for an example, and draw the graph.

```python
G = nx.petersen_graph()
nx.draw_shell(G, nlist=[range(5, 10), range(5)], with_labels=True, font_weight='bold')
```

Now, let's create a graph state using this graph $G$.

```python
GS = GraphState(G)
```

We can draw the quantum circuit that is created and see what gates have been applied to prepare the graph state. Each node in the graph corresponds to a qubit in the circuit. The Hadamard gates initialize the qubits to the $|+\rangle = H|0\rangle$ state. The entangling Controlled-$Z$ gates are applied between any two qubits which correspond to nodes connected by an edge in the graph.

```python
GS.circuit.draw('mpl')
```

We can draw the original graph $G$, which is stored in the `GraphState` class as `self.graph`.

```python
nx.draw_shell(GS.graph, nlist=[range(5, 10), range(5)], with_labels=True, font_weight='bold')
```

Now, let's apply a barrier to the circuit, followed by a measurement in the Pauli-$X$ basis, which is done with the method `x_measurement(qbit, cbit)`. This method measure `qbit` and stores the value in `cbit`.

```python
GS.circuit.barrier()
GS.x_measurement(3,3)
GS.circuit.draw('mpl')
```

Now, we can apply one of the stabilizer generators using the `apply_stabilizer` method. Let's apply a barrier then apply the stabilizer generator corresponding to qubit $4$, then apply one more barrier.

```python
GS.circuit.barrier()
GS.apply_stabilizer(4)
GS.circuit.barrier()
GS.circuit.draw('mpl')
```

You can see that the stabilizer

\begin{align}
S_4 = X_4 \otimes \left(\bigotimes_{u \in Adj(4)} Z_u \right)
\end{align}

has been applied, where $Adj(4)$ are the nodes in the graph connected by an edge to node $4$, i.e. nodes $0, 3$, and $9$.

### Example 2
---

Let's create a simpler graph now so that we can test a few more things out.

```python
G2 = nx.Graph()
G2.add_nodes_from([0,1,2])
G2.add_edges_from([(0,1), (1,2)])
nx.draw_spectral(G2, with_labels=True)
```

This gives us the following graph state circuit.

```python
GS2 = GraphState(G2)
GS2.circuit.draw('mpl')
```

We would like to now test out the three stabilizer generators for the stabilizer group of our graph state. The stabilizer generators are

\begin{align}
S_0 = X_0 \otimes Z_1 \otimes I_3 &= X_0Z_1I_3\\
S_1 = Z_0 \otimes X_1 \otimes Z_2 &= Z_0X_1Z_2\\
S_2 = I_0 \otimes Z_1 \otimes X_2 &= I_0Z_1X_2.
\end{align}

Recall that the subscript in the notation above corresponds to the qubit we are operating on with that particular gate, and this also corresponds to a particular node (vertex) in the graph $G_2$. A different labeling of the vertices of our graph will produce a different but equivalent graph state on a QISKit circuit. To test out stabilizers, first let's run the circuit above on the `statevector_simulator` backend in QISKit, and print off the state vector.

```python
# Tell Qiskit how to simulate our circuit
backend = Aer.get_backend('statevector_simulator')
# Do the simulation, returning the state vector
out_state = execute(GS2.circuit, backend).result().get_statevector()
# Display the output state vector
out_state
```

    array([ 0.35355339+0.j,  0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,
            0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,  0.35355339+0.j])

Notice, since we have $3$ qubits we have a $2^3 = 8$ dimensional state vector of amplitudes describing the state of the qubits after we run the graph state circuit. Now, let's apply one of the stabilizers to the circuit and run it again. Here, we will apply $S_0 = X_0Z_1I_2$.

```python
GS2.circuit.barrier()
GS2.circuit.x(0)
GS2.circuit.z(1)
GS2.circuit.draw(output='mpl')
```

Running this on the backend we get a new state vector `out_state2`.

```python
# Tell Qiskit how to simulate our circuit
backend = Aer.get_backend('statevector_simulator')
# Do the simulation, returning the state vector
out_state2 = execute(GS2.circuit, backend).result().get_statevector()
# Display the output state vector
out_state2
```

    array([ 0.35355339+0.j,  0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,
            0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,  0.35355339+0.j])

Remember, we use the `np.isclose()` function in Python with relative tolerance $10^{-5}$ and absolute tolerance $10^{-8}$. If we use the `==` or `np.testing.assert_array_equal()` function, due to round off errors we might get a False outcome.

```python
import numpy as np
np.isclose(out_state, out_state2, rtol=1e-05, atol=1e-08, equal_nan=False)
```

    array([ True,  True,  True,  True,  True,  True,  True,  True])

Since the test showed that these two state vectors are equal (within some rounding error tolerance), we know that

\begin{align}
S_0|G_2\rangle = |G_2\rangle.
\end{align}

In other words, we have verified that $S_0$ is indeed a stabilizer generator for the stabilizer group of the graph state. Let's text the stabilizer generator $S_1 = Z_0X_1Z_2$.

```python
GS2.circuit.barrier()
GS2.circuit.z(0)
GS2.circuit.x(1)
GS2.circuit.z(2)
GS2.circuit.draw(output='mpl')
```

Running this circuit on the backend, we get a new state vector `out_state3`, which we can compare to `out_state` (or to `out_state2`).

```python
# Tell Qiskit how to simulate our circuit
backend = Aer.get_backend('statevector_simulator')
# Do the simulation, returning the state vector
out_state3 = execute(GS2.circuit, backend).result().get_statevector()
# Display the output state vector
out_state3
```

    array([ 0.35355339+0.j,  0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,
            0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,  0.35355339+0.j])

Testing for equality within our chosen error margins we get the following.

```python
np.isclose(out_state, out_state3, rtol=1e-05, atol=1e-08, equal_nan=False)
```

    array([ True,  True,  True,  True,  True,  True,  True,  True])

So, indeed $S_1$ is a stabilizer generator for the stabilizer group of the graph state $|G_2\rangle$. Finally, let's test $S_2 = I_0Z_1X_2$.

```python
GS2.circuit.barrier()
GS2.circuit.z(1)
GS2.circuit.x(2)
GS2.circuit.draw(output='mpl')
```

```python
# Tell Qiskit how to simulate our circuit
backend = Aer.get_backend('statevector_simulator')
# Do the simulation, returning the state vector
out_state4 = execute(GS2.circuit, backend).result().get_statevector()
# Display the output state vector
out_state4
```

    array([ 0.35355339+0.j,  0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,
            0.35355339+0.j,  0.35355339+0.j, -0.35355339+0.j,  0.35355339+0.j])

```python
np.isclose(out_state, out_state4, rtol=1e-05, atol=1e-08, equal_nan=False)
```

    array([ True,  True,  True,  True,  True,  True,  True,  True])

## Verification Protocol

Okay, now that we have a good review of stabilizer generators for the stabilizer groups of graph states, we would like to talk about *blind quantum computing* for quantum cryptography. Suppose we have a client Alice, who wants to send her computation to a server owned by Bob, who has a quantum computer. She wants to do so securely so that her privacy is protected. She starts by preparing $2k+1$ copies of the graph state $|G\rangle$. These copies will be denoted by $|G\rangle^{\otimes 2k+1}$. Alice uses randomly chosen $2k$-copies of $|G\rangle^{\otimes 2k+1}$ to check the stabilizers $S_v$, where $v \in V$ are vertices. The remaining circuit is used for her computation. The details of the protocol are taken from ["Verifiable measurement-only blind quantum computing with stabilizer testing"](https://arxiv.org/pdf/1505.07535.pdf) and are as follows:

### Server's State Preparation Protocol

1. The server (Bob), assumed to be honest, generates $|G\rangle^{\otimes 2k+1}$, i.e. $2k+1$ copies of the $n$-qubit graph state. We assume that the graph $G$ is bipartite (so its vertices can be colored black and white, with no black vertices connect by an edge, and no white vertices connected by an edge). Call the white vertices $W$, and the black vertices $B$. Bob sends each qubit of $|G\rangle^{\otimes 2k+1}$, one by one to the client Alice. We will also have an *"Evil Bob"*, which is assumed to be a dishonest server. Evil Bob can generate any $n(2k + 1)$-qubit state $|\psi\rangle$ instead of $|G\rangle^{\otimes 2k+1}$.

2. Alice divides 2k + 1 blocks of n qubits into three groups *by random choice*.
    - The first group consists of $k$ blocks of $n$ qubits.
    - The second group consists of $k$ blocks of $n$ qubits.
    - The third group consists of a single block of $n$ qubits.

3. Alice uses the third group for her computation. Other blocks are used for the test, which will be explained later.

4. If Alice passes the test, she accepts the result of the computation performed on the third group.

In the figure above, the **Raussendorf-Harrington-Goyal lattice** or the **RHG-lattice** is a *"resource state"*, which is a graph state.

### The Client Alice's Test

For each block of the first and second groups mentioned in the preparation protocol, Alice performs the following test.

1. For each block of the first group, Alice measures qubits of $W$ in the $Z$-basis and qubits of $B$ in the $X$-basis.

2. For each block of the second group, Alice measures qubits of $B$ in the $Z$-basis and qubits of $W$ in the $X$-basis.

3.  If the measurement outcomes in the $X$-basis coincide with the values predicted from the outcomes in the $Z$-basis (in terms of the stabilizer relations), then the test is passed. If any outcome in the $X$-basis that violates the stabilizer relations is obtained, Alice rejects.

## Bipartite Graphs in NetworkX

Before moving on let's look at some examples of using bipartite graphs in NetworkX. In order to create bipartite graphs in NetworkX, we will need the following import.

```python
from networkx.algorithms import bipartite
```

Now, let's create a bipartite graph. This will be the **complete bipartite graph** $K_{4,3}$.

```python
K = nx.Graph()
# Add nodes with the node attribute "bipartite"
K.add_nodes_from([0, 1, 2, 3], bipartite=0)
K.add_nodes_from([4,5,6], bipartite=1)
# Add edges only between nodes of opposite node sets
K.add_edges_from([(0, 4), (0, 5), (0, 6), (1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)])
```

By convention, the vertices labeled "$0$" will be white, and the nodes labeled "$1$" will be black. This corresponds to the set $W$ and $B$ in the description of the blind quantum computing protocol. So long as the graph $B$ is connected, we have a function in NetworkX that separates the $W$ nodes from the $B$ nodes. Just to be safe, let's test that $B$ is indeed connected.

```python
nx.is_connected(K)
```

    True

The following function allows us to separate $W$ nodes from $B$ nodes.

```python
white_nodes, black_nodes = bipartite.sets(K)
```

```python
print(white_nodes)
```

    {0, 1, 2, 3}

```python
print(black_nodes)
```

    {4, 5, 6}

```python
print(bipartite.sets(K))
```

    ({0, 1, 2, 3}, {4, 5, 6})

This will be incredibly helpful if we want to write function that perform different basis measurements on different colored vertices of our graph! As with a typical graph, we can print the edges.

```python
print(K.edges)
```

    [(0, 4), (0, 5), (0, 6), (1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]

```python
nx.draw_circular(K, with_labels=True)
```

### Bipartite $X$-Basis and $Z$-Basis Measurements

Now that we have an idea of how to create and manipulate bipartite graphs in NetworkX, we want to write a function that implements a bipartite graph states. We also want to write two measurement function that will measure in the $X$-basis in the white nodes, and in the $Z$-basis on the black notes. We also want to write a second function that will measure in the $Z$-basis on the white nodes, and in the $X$-basis on the white nodes. This will help us implement the protocols of blind quantum computing mentioned above.

### Bipartite Graph State Creation

To start with, we will need to define a new class to construct graph states for the bipartite case. It will need to handle the new structure and iterate Hadamard gates over two sets of vertices, "$W$" and "$B$". Let's do that now.

```python
from networkx.algorithms import bipartite
from qiskit import *

class BipartiteGraphState(QuantumCircuit):

    def __init__(self, bipartite_graph):
        super().__init__()
        self.graph = bipartite_graph
        # Create a quantum register based on the number of nodes
        # in W + the number of nodes in B (= total number of nodes in G)
        self.white_nodes, self.black_nodes = bipartite.sets(self.graph)
        self.qreg = QuantumRegister(len(self.black_nodes) + len(self.white_nodes))
        self.creg = ClassicalRegister(len(self.black_nodes) + len(self.white_nodes))
        # Create a circuit using the quantum register
        self.circuit = QuantumCircuit(self.qreg, self.creg)
        # For each vertex in W, apply a Hadamard gate
        for vertex in self.white_nodes:
            self.circuit.h(vertex)
        # For each vertex in B, apply a Hadamard gate
        for vertex in self.black_nodes:
            self.circuit.h(vertex)
        # For each edge e={x,y} apply a controlled-Z gate on its vertices
        for x, y in self.graph.edges:
            self.circuit.cz(x, y)
        self.node_dict = self.build_node_dict()

    def build_node_dict(self):
        """
        create a node dictionary from node to integer index of a qubit
        in a Qiskit circuit
        :param self:
        """
        self.node_dict = dict()
        for count, node in enumerate(self.graph.nodes):
            self.node_dict[node] = count

    def x_measurement(self, qubit, cbit):
        """Measure 'qubit' in the X-basis, and store the result in 'cbit'"""
        self.circuit.h(qubit)
        self.circuit.measure(qubit, cbit)
        self.circuit.h(qubit)

    def x_measure_white(self):
        """
        measure the white qubits in the Pauli X-basis
        :param self:
        """
        self.circuit.barrier()
        for vertex in self.black_nodes:
            self.circuit.measure(vertex, vertex)
        self.circuit.barrier()
        for vertex in self.white_nodes:
            self.x_measurement(vertex, vertex)

    def x_measure_black(self):
        """
        measure the black qubits in the Pauli X-basis
        :param self:
        """
        self.circuit.barrier()
        for vertex in self.white_nodes:
            self.circuit.measure(vertex, vertex)
        self.circuit.barrier()
        for vertex in self.black_nodes:
            self.x_measurement(vertex, vertex)

    def apply_stabilizer(self, node):
        """
        applies the stabilizer generator corresponding to node
        :param self:
        :param node: a node in self.graph
        """
        self.circuit.x(self.node_dict[node])
        for neighbor in self.graph.neighbors(node):
            self.circuit.z(self.node_dict[neighbor])
```

Now, let's test this out on a very simple bipartite graph. Let's say $B$ is given by the sets

\begin{align}
W &= \{0, 2\}\\
B &= \{1\}\\
E &= \{(0,1), (1,2)\}.
\end{align}

Let's draw this graph now. It will look the same as the graph $G$ we defined above.

```python
B = nx.Graph()
# Add nodes with the node attribute "bipartite"
B.add_nodes_from([0, 2], bipartite=0)
B.add_nodes_from([1], bipartite=1)
# Add edges only between nodes of opposite node sets
B.add_edges_from([(0,1), (1,2)])
nx.draw_circular(B, with_labels=True)
```

We will divide the nodes into the sets $W$ and $B$ as follows.

```python
black_nodes, white_nodes = bipartite.sets(B)
print(white_nodes)
```

    {1}

```python
print(black_nodes)
```

    {0, 2}

Okay, now let's create the corresponding (bipartite) graph state using the class `BipartiteGraphState()`.

```python
BGS = BipartiteGraphState(B)
```

```python
BGS.circuit.draw(output='mpl')
```

Notice how this graph state is slightly different from, but still equivalent to, the graph state for the bipartite graph $B$ that we defined. The controlled-$Z$ gates are connected in a slightly different way due to the way the vertices (and thus the qubits) have been partitioned. Now, we want to use the method `x_measure_white` that will measure the circuit in the $X$-basis on the white nodes, and in the $Z$-basis on the black nodes.

```python
BGS.x_measure_white()
BGS.circuit.draw('mpl')
```

Now, let's create another instance of the bipartite graph state for the same graph $B$.

```python
BGS2 = BipartiteGraphState(B)
BGS2.circuit.draw(output='mpl')
```

Notice how this graph state is slightly different from, but still equivalent to, the graph state for the bipartite graph $B$ that we defined. The controlled-$Z$ gates are connected in a slightly different way due to the way the vertices (and thus the qubits) have been partitioned. Now, we want to use the method `x_measure_black` that will measure the circuit in the $X$-basis on the black nodes, and in the $Z$-basis on the white nodes.

```python
BGS2.x_measure_black()
BGS2.circuit.draw('mpl')
```

### Example 3 with $K_{4,3}$

Remember the graph $K_{4,3}$ from the example on creating bipartite graphs in NetworkX is created as follows.

```python
K = nx.Graph()
# Add nodes with the node attribute "bipartite"
K.add_nodes_from([0, 1, 2, 3], bipartite=0)
K.add_nodes_from([4,5,6], bipartite=1)
# Add edges only between nodes of opposite node sets
K.add_edges_from([(0, 4), (0, 5), (0, 6), (1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)])
```

The white nodes $W$, and black nodes $B$ can be printed as follows.

```python
white_nodes, black_nodes = bipartite.sets(K)
print(white_nodes)
```

    {0, 1, 2, 3}

```python
print(black_nodes)
```

    {4, 5, 6}

```python
KBGS = BipartiteGraphState(K)
KBGS.circuit.draw(output='mpl')
```

Next, we can measure the white vertices in the $X$-basis, and the black vertices in the $Z$-basis, for a single instance of this graph state using the method `x_measure_white()`.

```python
KBGS.x_measure_white()
KBGS.circuit.draw(output='mpl')
```

On a second copy of the circuit we can measure the white vertices in the $Z$-basis now, and the black vertices in the $X$-basis, using the function `x_measure_black()`.

```python
KBGS2 = BipartiteGraphState(K)
KBGS2.circuit.draw(output='mpl')
```

```python
KBGS2.x_measure_black()
KBGS2.circuit.draw(output='mpl')
```

## References

- ["Verifiable measurement-only blind quantum computing with stabilizer testing"](https://arxiv.org/pdf/1505.07535.pdf)

- [A Simple Protocol for Certifying Graph States and Applications in Quantum Networks](https://arxiv.org/pdf/1801.05057.pdf)

- [Entanglement in Graph States and its Applications](https://arxiv.org/pdf/quant-ph/0602096.pdf)

- [Multi-party entanglement in graph states](https://arxiv.org/pdf/quant-ph/0602096.pdf)

```python

```


# Mass matrix diagonalization (lumping)

```python
from sympy import *
init_session()
```

    IPython console for SymPy 1.0 (Python 3.6.0-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

## Elemental mass matrices

```python
def mass_tet4():
    """Mass matrix for a 4 node tetrahedron"""
    r, s, t = symbols("r s t")
    N = Matrix([1 - r - s - t, r, s, t])
    return (N * N.T).integrate((t, 0, 1 - r - s), (s, 0, 1 - r), (r, 0, 1))

def mass_quad8():
    """Mass matrix for a 8 node quadrilateral"""
    r, s = symbols("r s")
    Haux = Matrix([
        (1 - r**2)*(1 + s),
        (1 - s**2)*(1 - r),
        (1 - r**2)*(1 - s),
        (1 - s**2)*(1 + r)])
    N = S(1)/4*Matrix([
            (1 + r)*(1 + s) - Haux[0] - Haux[3],
            (1 - r)*(1 + s) - Haux[0] - Haux[1],
            (1 - r)*(1 - s) - Haux[1] - Haux[2],
            (1 + r)*(1 - s) - Haux[2] - Haux[3],
           2*Haux[0], 2*Haux[1], 2*Haux[2], 2*Haux[3]])
    return (N * N.T).integrate((s, -1, 1), (r, -1, 1))
```

The elemental mass matrices look like

```python
mass_tet4()
```

```python
mass_quad8()
```

## Lumping

One method for lumping is to sum the matrix per rows, i.e.

$$M^\text{(lumped)}_{ii}= \sum_{j} M_{ij}$$

```python
def row_lump(mass_mat):
    """Matrix lumping by row summing"""
    return diag(*[sum(mass_mat[i, :]) for i in range(mass_mat.shape[0])])
```

One method for lumping is to sum the matrix per rows, i.e.

$$M^\text{(lumped)}_{ii} = c M_{ii}$$

with $c$ adjusted to satisfy $\sum_j M^\text{(lumped)}_{jj} = \int_\Omega \rho d\Omega$. Particularly, we can choose $c = Tr(M)/M_\text{total}$.

```python
def diag_scaling_lump(mass_mat):
    """Matrix lumping by diagonal scaling method"""
    mass = sum(mass_mat)
    trace = mass_mat.trace()
    c = mass/trace
    return diag(*[c*mass_mat[i, i] for i in range(mass_mat.shape[0])])
```

```python
def min_dist_lump(mass_mat):
    """
    Matrix lumping by minimizing the Frobenius norm subject
    to a constraint of conservation of mass.
    """
    num = mass_mat.shape[0]
    mass = sum(mass_mat)
    lamda = symbols("lambda")
    Ms = symbols('M0:%d'%num)
    var = list(Ms)
    mass_diag = diag(*var)
    C = mass_mat - mass_diag
    fun = (C.T*C).trace() + lamda*(mass - sum(mass_diag))
    var.append(lamda)
    grad = [diff(fun, x) for x in var]
    sol = solve(grad, var)
    return diag(*list(sol.values())[:-1])
```

We can compare the methods for the tetrahedron

```python
row_lump(mass_tet4())
```

```python
diag_scaling_lump(mass_tet4())
```

```python
min_dist_lump(mass_tet4())
```

We can compare the methods for the serendipity quadrilaterals. For this type
of element we can't use the row lumping method since it leads to negative
masses.

```python
row_lump(mass_quad8())
```

```python
diag_scaling_lump(mass_quad8())
```

```python
min_dist_lump(mass_quad8())
```

```python

```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
/* Based on Lorena Barba template available at: https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css*/
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```


```python
from sympy import Matrix, S, latex
from sympy.abc import x, y, z
from lmi_sdp.lm import lm_sym_expanded
from lmi_sdp.lmi import ShapeError, NonSymmetricMatrixError, LMI_PSD, \
    LMI_NSD, LMI_PD, LMI_ND, init_lmi_latex_printing
```

### test_LMI_PSD()

```python
m = Matrix([[x, y], [y, z+1]])
lmi = LMI_PSD(m)
assert lmi.lhs == lmi.gts == m
assert lmi.rhs == lmi.lts == S(0)

c = Matrix([[0, 1], [1, 2]])
lmi = LMI_PSD(m, c)
assert lmi.lhs == lmi.gts == m
assert lmi.rhs == lmi.lts == c
```

### test_LMI_canonical()

```python
m = Matrix([[x, y], [y, z+1]])
c = Matrix([[0, 1], [1, 2]])
can_lhs = m-c
can_rhs = 0

can = LMI_PSD(m, c).canonical()
assert can.lhs == can.gts == can_lhs
assert can.rhs == can.lts == can_rhs

can = LMI_PSD(0, c-m).canonical()
assert can.lhs == can.gts == can_lhs
assert can.rhs == can.lts == can_rhs

can = LMI_PD(m, c).canonical()
assert can.lhs == can.gts == can_lhs
assert can.rhs == can.lts == can_rhs

can = LMI_NSD(c, m).canonical()
assert can.lhs == can.gts == can_lhs
assert can.rhs == can.lts == can_rhs
assert isinstance(can, LMI_PSD)

can = LMI_ND(c, m).canonical()
assert can.lhs == can.gts == can_lhs
assert can.rhs == can.lts == can_rhs
assert isinstance(can, LMI_PD)
```

### test_LMI_canonical_same()

```python
m = Matrix([[x, y], [y, z+1]])

lmi = LMI_PSD(m)
assert lmi is lmi.canonical()

lmi = LMI_PD(m)
assert lmi is lmi.canonical()
```

### test_LMI_expanded()

```python
variables = (x, y, z)
m = Matrix([[x, y], [y, z+1]])
c = Matrix([[0, 1], [1, 2]])

expdd = LMI_ND(m+c, c).expanded(variables)
assert expdd == LMI_ND(lm_sym_expanded(m+c, variables), c)

expdd = LMI_PSD(m).expanded(variables)
assert expdd == LMI_PSD(lm_sym_expanded(m, variables))
```

### test_LMI_PSD_exceptions()

```python
try:
    LMI_PSD(Matrix([[1, x], [y, z]]))
except NonSymmetricMatrixError:
    pass
else:
    assert False

try:
    LMI_PSD(Matrix([[1, x], [y, z]]), assert_symmetry=False)
except NonSymmetricMatrixError:
    assert False

try:
    LMI_PSD(Matrix([[x+y]]), Matrix([[x, y], [y, z+1]]))
except ShapeError:
    pass
else:
    assert False
```

### test_lmi_latex_printing()

```python
init_lmi_latex_printing()
lmi = LMI_PSD(Matrix([[x, y], [y, z+1]]))
assert r"\succeq 0" == latex(lmi)[-9:]
```