#  Machine Learning: Basic Principles 2018
## Classification
## Learning goals
In this exercise, you will learn how to formulate and solve a classification problem. A classification problem amounts to finding a good classifier which maps a given data point (e.g. a snapshot taken by an on-board camera) via its features to a particular label. The label indicates to which class or category the data point belongs.
We will implement **logistic regression** and use **Gradient Descent** to find the optimal weights. We will look at binary as well as multiclass classification.

## Exercise Contents

1. Introduction
    * Here we formulate a classification problem.
2. Dataset
    * A description of the dataset.
3. Exercise
    * A total of 5 tasks and 1 demo. Read the task descriptions and answer according to the tasks.
        * 3.1 **Getting Hands on the Data**
        * 3.2 **Logistic Regression**
        * 3.3 **Gradient Descent Step Size**
        * 3.4 **Accuracy - How well did we do?**
        * 3.5 **Multiclass Classification (One vs All)**

### Keywords
`Classification`,`Logistic Regression`, `Sigmoid Function`, `Gradient Descent (GD)`

## 1 Introduction
Suppose you are an intern at the fictive company `Hunda` whose brand-new lawn mower robot uses its on-board camera to find out on which surface it is moving.
Your job is to develop a firmware module which allows the mower robot to classify images generated by the camera according to the categories "I see grass", "I see soil", "I see tiles".

To do so we will use a specific classification method named logistic regression. We will first develop a logistic regression model which can distinguish between "I see grass" and "I do not see grass". After that we will extend this model to also be able to distinguish non-grass images further between soil and tiles images.

In order to develop this module you are provided with a bunch of snapshots that have been labeled according to these three categories manually by the previous summer intern. Thus, we can use this labeled data to train the firmware module.

## 2 Data
The available dataset consists of $N=55$ pictures, saved in the folder named `images`:
* 20 images of grass (`image_1.jpg` to `image_20.jpg`)
* 20 images of soil (`image_21.jpg` to `image_40.jpg`)
* 15 images of tiles (`image_41.jpg` to `image_55.jpg`)

You can use the Python package `PIL` (=the Python Imaging Library) to access the image data. The basics of PIL are demonstrated below, so you can use those functions in the exercise.

```python
from PIL import Image

# Read in an image from a jpg-file and store it in the variable im
im = Image.open("images/image_1.jpg");

# Determine the size of the image
width, height = im.size;
print('width: %d, height: %d' % (width, height))

# Convert the image to RGB
rgb_im = im.convert('RGB');

# Determine the rgb values of the pixel at location (row,col) where both row and col are 0
pixel = rgb_im.getpixel((0,0))
print('Pixel (R, G, B): (%d, %d, %d)' % (pixel[0], pixel[1], pixel[2]))
```

## 3 Exercise
The actual exercise starts from here and it has been divided into 5 tasks and 1 demo:
* 3.1 **Getting Hands on the Data**
* 3.2 **Logistic Regression**
* 3.3 **Gradient Descent Step Size**
* 3.4 **Accuracy - How well did we do?**
* 3.5 **Multiclass Classification (One vs All)**

Your task is to fill in `...` under `### STUDENT TASK ###` in each step.

## 3.1 Getting Hands on the Data
After reading in the RGB images, we have to change them into a form that we can process more easily. Although the images are quite small (less than 3000x3000 pixels) we cannot easily process an image by just stacking the pixels into a vector since it would be of size $3000^2$. Instead, we represent an image by only $d=3$ features, namely the average red, green and blue components (the "redness", "greenness" and "blueness") which are denoted $x_{r}, x_{g}, x_{b}$. Thus, we characterize the $i$th image in our dataset using the feature vector $\mathbf{x}^{(i)} = \big(x_{r}^{(i)},x_{g}^{(i)},x_{b}^{(i)} \big)^{T} \in \mathbb{R}^{3}$. In particular, $x^{(1)}_{r}$ denotes the average red component of the first image in the trainging dataset.

It will be convenient to stack the feature vectors $\mathbf{x}^{(i)} \in \mathbb{R}^{3}$ obtained for all images in our (training) dataset into the feature matrix
\begin{equation*}
    \mathbf{X} = \big(\mathbf{x}^{(1)},\dots,\mathbf{x}^{(55)}\big)^T=\begin{bmatrix}
    x^{(1)}_{r}  & x^{(1)}_{g}  & x^{(1)}_{b} \\
    \vdots & \ddots & \vdots\\
    x^{(55)}_{r} & x^{(55)}_{g} & x^{(55)}_{b}
    \end{bmatrix} \in \mathbb{R}^{N \times 3}
    \label{xm}
    \tag{1}
\end{equation*}

Each image in the training set has a label $y^{(i)}$ where $y^{(i)} = 1$ if the i-th image shows grass while $y^{(i)} = 0$ otherwise (when it shows soil or tiles). It will be convenient to stack the labels into the label vector $\mathbf{y}$.

\begin{equation*}
    \mathbf{y}=\begin{bmatrix}
    y^{(1)}\\
    y^{(2)}\\
    \vdots\\
    y^{(N)}
    \end{bmatrix},\ i \in \{1,\dots,N \}
    \label{vy}
    \tag{2}
\end{equation*}

### Tasks:
- Implement a Python function `get_feature_matrix()` which returns the feature matrix (\ref{xm}) of size $55 \times 3$.
- Implement a Python function `get_labels()` which returns the label vector (\ref{vy}).
    - The vector should contain the **labels** for each image. The label is `1` if the picture shows grass, `0` if it doesn't.
- Implement a Python function `Visualize_data()` to draw three **scatter plots**.
    - Make a scatter plot for each color combination (*Greenness vs Redness*,*Greenness vs. Blueness* and *Redness vs Blueness*).
        - In these plots, mark a datapoint with a cross if it's a grass picture and with a dot if it's not.
    - The plot labels and axes are preconfigured, your job is to setup the correct x and y -values to each `plt.scatter()`-function.
    - These scatter plots are a helpful tool for giving us an idea of the relation between the features and the two labels.

### Tips:
- Since it takes some time to compute the feature matrix $\mathbf{X}$, you might want to save it as .txt, .csv or any other format that is convenient for you. This way you can just reload the matrix from the file without processing the images again.

```python
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt

def get_feature_matrix(N = 55):

    #initialize the feature vector with zeros.
    x_vec = np.zeros((N,3))

    x = []
    ### STUDENT TASK ###
    ## Loop through each picture and each pixel and sum the RGB values into the feature vector matrix.
    ## At last, remember to divide each R, G and B sum with the total pixel count to get the average value.
    ## Hint: Most of the commands required for this task are in the 2.Dataset-section.
    # YOUR CODE HERE
    raise NotImplementedError()
    return x_vec;

def get_labels(N=55):
    y = np.zeros((N,1));
    ### STUDENT TASK ###
    ## Generate the label vector, where 1 is a Grass image and 0 is Non-Grass.
    ## Hint: See the 2.Dataset-section where the picture order is defined.
    # YOUR CODE HERE
    raise NotImplementedError()
    return y

""" VISUALIZE THE DATA """
def Visualize_data(X,y):

    indx_1 = np.where(y == 1) # for grass.
    indx_2 = np.where(y == 0) # for non-grass.

    # Set figure size (width, height)
    fig, axes = plt.subplots(1, 3,figsize=(15, 5))
    # PLOT GREENNESS AGAINST REDNESS
    #Make a scatterplot of the average greenness vs redness.
    #Indicate Grass images by a cross, and others by a dot.
    ### STUDENT TASK ###
    #axes[0].scatter(...,..., c='g', marker ='x', label='Grass')
    #axes[0].scatter(...,..., c='r', marker ='o', label='Soil+Tiles')
    # YOUR CODE HERE
    raise NotImplementedError()
    axes[0].set_xlabel('Greenness of Images')
    axes[0].set_ylabel('Redness of Images')
    axes[0].legend()
    axes[0].set_title(r'$\bf{Figure\ 1.}$Green vs Red')

    # PLOT GREENNESS AGAINST BLUENESS
    #The same as above but now greenness vs blueness.
    ### STUDENT TASK ###
    #axes[1].scatter(..., ..., c='g', marker ='x', label='Grass')
    #axes[1].scatter(..., ..., c='b', marker ='o', label='Soil+Tiles')
    # YOUR CODE HERE
    raise NotImplementedError()
    axes[1].set_xlabel('Greenness of Images')
    axes[1].set_ylabel('Blueness of Images')
    axes[1].legend()
    axes[1].set_title(r'$\bf{Figure\ 2.}$Green vs Blue')

    # PLOT REDNESS AGAINST BLUENESS
    #The same as above but now redness vs blueness.
    ### STUDENT TASK ###
    #axes[2].scatter(..., ..., c='r', marker ='x', label='Grass')
    #axes[2].scatter(..., ..., c='b', marker ='o', label='Soil+Tiles')
    # YOUR CODE HERE
    raise NotImplementedError()
    axes[2].set_xlabel('Redness of Images')
    axes[2].set_ylabel('Blueness of Images')
    axes[2].legend()
    axes[2].set_title(r'$\bf{Figure\ 3.}$Red vs Blue')
    plt.tight_layout()
    return axes

```

```python
y = get_labels()
X = get_feature_matrix()

# Full Vector
# Let s label : Grass = 1 , Soil = 0, Tiles = 0
assert X.shape == (55,3)
axes = Visualize_data(X,y)
```

## 3.2 Logistic Regression
Our goal is to find out the label $y$ of an image, with $y=1$ if the image shows grass and $y=0$ otherwise, based on its features $\mathbf{x}$. Similar to linear regression, logistic regression applies a linear function of the form $h^{(\mathbf{w})}(\mathbf{x})= \mathbf{w}^{T} \mathbf{x}$ to predict the label $y$ based on the features $\mathbf{x} = \big(x_{r},x_{g},x_{b} \big)^{T} \in \mathbb{R}^{3}$ of the image. In contrast to linear regression, which uses the squared error loss, here we use the **logistic loss** to measure the quality of a particular classifier $h^{(\mathbf{w})}$. The logistic loss is defined as:

\begin{equation*}
    \mathcal{L}\big((\mathbf{x},y),h^{(\mathbf{w})}\big) = -y\log\big(\sigma(h^{(\mathbf{w})}(\mathbf{x}))\big)-(1-y)\log\big(1-\sigma(h^{(\mathbf{w})}(\mathbf{x}))\big)
    \label{loss}
    \tag{3}
\end{equation*}

where $\sigma$ is the sigmoid function, which is defined as:
\begin{equation*}
    \sigma(z)= \frac{1}{1+{\rm exp}(-z)}.
    \label{sigmoid}
    \tag{4}
\end{equation*}

**Note that the expression \eqref{loss} for the logistic loss applies only if the classes are encoded as $y=1$ and $y=0$. If the two classes are encoded as $y=1$ and $y=-1$, we obtain a different formula for the logistic loss.**

Since we have $N=55$ labeled images, each of them characterized by the features $\mathbf{x}^{(i)}$ and the true label $y^{(i)}$, we can evaluate the logistic loss for all those images to obtain the empirical risk.

\begin{align}
\mathcal{E}(\mathbf{w}) & = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}((\mathbf{x}^{(i)},y^{(i)}),\ h^{(\mathbf{w})}) \nonumber \\
&  = \frac{1}{N} \sum_{i=1}^{N} -y^{(i)}\log\big(\sigma(\mathbf{w}^{T}\mathbf{x}^{(i)})\big)-(1-y^{(i)})\log\big(1-\sigma(\mathbf{w}^{T}\mathbf{x}^{(i)})\big)
   \label{erm}
    \tag{5}
\end{align}

Note that the empirical risk $\mathcal{E}( \mathbf{w})$ is a differentiable convex function of the weight vector $\mathbf{w}$. Therefore, we can use **gradient descent (GD)** to find the weight vector $\mathbf{w}_{\rm opt}$ which minimizes the loss function. In particular, GD constructs a sequence of weight vectors $\mathbf{w}^{(k)}$ by iterating (=repeating) the GD update

\begin{equation*}
    \mathbf{w}^{(k+1)} = \mathbf{w}^{(k)} − \alpha\nabla \mathcal{E}(\mathbf{w}^{(k)})
    \label{gd}
    \tag{6}
\end{equation*}

Assume we run the GD updates for $k$ iterations which results in the weight vector $\mathbf{w}^{(k)}$ and corresponding classifier map $h^{(\mathbf{w}^{(k)})}(\mathbf{x}) = \big(\mathbf{w}^{(k)} \big)^{T} \mathbf{x}$. Using this classifier map, we can compute the predicted label for a new image with features $\mathbf{x}=\big(x_{r},x_{g},x_{b}\big)^{T}\in \mathbb{R}^{3}$ via simple thresholding

\begin{equation*}
    \hat{y} = \begin{cases}
        1 &\text{if}\ \sigma\big(h^{(\mathbf{w}^{(k)})}(\mathbf{x})\big) \geq 1/2\\
        0 &\text{if}\ \sigma\big(h^{(\mathbf{w}^{(k)})}(\mathbf{x})\big) < 1/2
    \end{cases}
    \label{eq_classify}
    \tag{7}
\end{equation*}

### Tasks
- Implement a Python function `sigmoid_function(z)` which returns the function value (\ref{sigmoid}) of the sigmoid function.
    - In our case $z$ is the dot product of the weight vector and the feature vector.
- Implement a Python function `gradient(X,y,w)` which takes as input the feature matrix $\mathbf{X}$, the label vector $\mathbf{y}$ and the weight vector $\mathbf{w}$. The function should return the gradient of the empirical risk $\mathcal{E}(\mathbf{w})$ (\ref{erm}) at the vector $\mathbf{w}$.
- Implement a Python function `logisticRegression_func(X,y,step_size, K)` which takes as input the feature matrix $\mathbf{X}$, the label vector $\mathbf{y}$, the GD step size `step_size` and the number `K` of GD steps. The function should deliver the final GD iterate $\mathbf{w}^{(K)}$ and a vector of length K whose $k^{th}$ entry is the empirical risk $\mathcal{E}(\mathbf{w}^{(k)})$.
- Implement a Python function `predict_output()` which takes as input the feature matrix $\mathbf{X}$ and the optimal weight vector $\mathbf{w}$ and returns a vector of length $N$ containing the predicted labels $\mathbf{\hat{y}}$ (see Eq. (\ref{eq_classify})).

```python
def sigmoid_func(z):
    ### STUDENT TASK ###
    # sigmoid = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return sigmoid

def gradient(X,y,w):
    ### STUDENT TASK ###
    # grad = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return grad

def logisticRegression_func(X,y,step_size, K):
    N = X.shape[0]
    d = X.shape[1]
    # Initialize w as 1xd array.
    w = np.zeros((1,d))
    loss = float('inf')
    loss_list = []
    for i in range(K):
        ### STUDENT TASK ###
        # YOUR CODE HERE
        raise NotImplementedError()
    return loss_list, w

""" Predict Output """
def predict_output(X,w):
    ### STUDENT TASK ###
    # YOUR CODE HERE
    raise NotImplementedError()
    return y

```

```python
# Execute this cell. Do not modify.
step_size = 1e-5
num_iter = 3000
e_list, w_opt = logisticRegression_func(X,y,step_size,num_iter)
print ('The optimal weight vector is:', w_opt)
y_hat = predict_output(X,w_opt)

```

## 3.3 Gradient Descent Step Size
The performance of GD depends crucially on the step size $\alpha$. This task requires you to investigate how different choices for the step size influence the behavior of GD.

### Tasks
- Implement a Python function `visualize_error()` which iterates over different step sizes given in array `step_sizes=[0.1,0.5,1,5,10,16]`.
    - Use `logisticRegression_func()` which you implemented in task`3.2: Logistic Regression` with each step size, it should return a list of errors.
    - Plot each of these error lists. You should see two plots with 6 descending lines.
    - Figure out which step size has the fastest convergence and mark the corresponding curve in red (i.e. by changing `best=None` to right step size. For example if you think that the best step size is 1, choose `best=1`). Now you should see a plot where one line is red and rest of them are blue.

```python
def visualize_error(X, y, step_sizes, best = None, num_iter = 2000):
    plt.figure(figsize=(12, 4))
    fig, axes = plt.subplots(1, 2,figsize=(12, 4))
    for step in step_sizes:
        ### STUDENT TASK ###
        # Plot Error against Step Size
        # loss_list, w_opt =
        # YOUR CODE HERE
        raise NotImplementedError()
        n = len(loss_list) # Size of list remains the same.
        x_axes = np.linspace(0,n,n,endpoint=False)
        axes[0].plot(x_axes, loss_list, label=step)
    axes[0].set_xlabel('Number of Iterations')
    axes[0].set_ylabel('Loss Function')
    axes[0].legend()
    axes[0].set_title(r'$\bf{Figure\ 4.}$Converge of GD')

    for step in step_sizes:
        ### STUDENT TASK ###
        # Plot Error against Step Size.
        # Now mark the best converge in red. Use value from best as a correct step size.
        # loss_list, w_opt =
        # YOUR CODE HERE
        raise NotImplementedError()
        n = len(loss_list) # Size of list remains the same.
        x_axes = np.linspace(0,n,n,endpoint=False)
        if step == best:
            axes[1].plot(x_axes, loss_list, label=step, color="red")
        else:
            axes[1].plot(x_axes, loss_list, label=step, color="blue")

    axes[1].set_xlabel('Number of Iterations')
    axes[1].set_ylabel('Loss Function')
    axes[1].legend()
    axes[1].set_title(r'$\bf{Figure\ 5.}$Converge of GD')
    plt.tight_layout()
    return best, axes

### STUDENT TASK ###
# Change best=None into step size from the list that provides the fastest converge. e.g best=1
res0_1, axes = visualize_error(X/255, y, best=None, step_sizes=[0.1,0.5,1,5,10,16])
# YOUR CODE HERE
raise NotImplementedError()
```

```python

```

## 3.4 Accuracy - How well did we do?
In order to assess how well our model works, we calculate the accuracy achieved by the classifier $h^{(\mathbf{w})}$ obtained from task 3.3. We do this by computing the fraction of correctly labeled images, i.e., for which the true label $y^{(i)}$ is equal to the predicted label $\hat{y}^{(i)}$:
\begin{equation*}
    \text{Accuracy} =\dfrac{1}{N} \sum_{i=1}^{N} \mathcal{I}(\hat{y}^{(i)} = y^{(i)})
    \label{acc}
    \tag{8}
\end{equation*}

### Tasks
- Implement a Python function `calculate_accuracy(y,y_hat)` according to (Eq. \ref{acc}) which takes as inputs $\mathbf{y}$ and $\mathbf{\hat{y}}$ and returns the accuracy (as percentage).

```python
def calculate_accuracy(y,y_hat):
    ### STUDENT TASK ###
    # YOUR CODE HERE
    raise NotImplementedError()
    return accuracy
print ('Accuracy of the result is: %f%%' % calculate_accuracy(y,y_hat))

```

```python

```

## 3.5: Multiclass Classification ("One vs All" or "One vs Rest")
We have succesfully implemented a logistic regression model which classifies the current on-board image as showing grass or not. We will now extend this model to know how to classify the image according to the three categories "grass", "soil" or "tiles".

So is all of our previous work on classifying images into "grass" vs. "no grass" for nothing? Nope! Adapting a binary classification method (using two different label values) to this multiclass task is straightforward using the **“one vs all” technique**. The idea is quite simple: split the multiclass problem into three subproblems, each subproblem being a binary classification problem as considered in task `3.3 Logistic Regression`. In particular, we can solve the problem of classifiying images into classes "grass", "soil" or "tiles" by solving the three subproblems:
* subproblem 1: classify images into "grass"(y=1) vs. "no grass"(y=0)
* subproblem 2: classify images into "soil"(y=1) vs. "no soil"(y=0)
* subproblem 3: classify images into "tiles"(y=1) vs. "no tiles"(y=0)

We can apply the method developed under `3.2: Logistic Regression`, to solve each of these subproblems. The first subproblem has already been solved in Sec. 3.3, so we only need to solve the latter two subproblems. Note that for the three subproblems we use the same feature vector $\mathbf{x}^{(i)}=\big(x_{r}^{(i)},x_{g}^{(i)},x_{b}^{(i)}\big)^{T}$ but a different label $y^{(i)}$ for the $i$th image. In particular, the first image (which shows grass) has label $y^{(1)}=1$ in subproblem 1 but $y^{(1)}=0$ in subproblem 2 and subproblem 3.

For each subproblem we compute the optimal weight vector $\mathbf{w}^{(\rm grass)}$,$\mathbf{w}^{(\rm soil)}$, $\mathbf{w}^{(\rm tiles)}$ by solving the empirical risk minimization problem (5).

Assume we want to classify a new image with features $\mathbf{x}=(x_{r},x_{g},x_{b})^{T}$ yielding the following predictor values :
- $h^{(\mathbf{w}^{(\rm grass)})}(\mathbf{x}) = 2$ ("grass vs. no grass")
- $h^{(\mathbf{w}^{(\rm soil)})}(\mathbf{x}) = 1$ ("soil vs. no soil")
- $h^{(\mathbf{w}^{(\rm tiles)})}(\mathbf{x}) = 5$ ("tiles vs. no tiles")

Thus, the predictor $h^{(\mathbf{w}^{(\rm tiles)})}(x)$ for subproblem 3 (`tiles` vs. `no tiles`) yields the highest confidence. Hence, we classify this image as `tiles`.

### Tasks
- Implement a Python function `get_labels_k()` which takes as input the number of data points $N$ and the index $k$ of the subproblem: $k=0$ means subproblem 1, $k=1$ means subproblem 2 and $k=2$ means subproblem 3, and returns the label vector $\mathbf{y}$ for that category.
    - i.e. either Grass, Soil **or** Tile pictures are `1`, the other 2 are `0`
- Implement a Python function `multiclass()` which implements logistic regression for the three subproblems and then outputs the real labels $\mathbf{y}$, the final predicted labels $\mathbf{\hat{y}}$ and the accuracy.
    - Use the `logistic_regression()` and `sigmoid_func()` of section 3.3 to get the optimal weight vector and the probability of the picture belonging to that class respectively.
    - After you have done logistic regression for each of the three categories, predict the class the picture belongs to by choosing the highest probability of all binary classifiers.
    - Use the function `calculate_accuracy` of section 3.4 to calculate the accuracy.
    - You might want to encode the three classes "Grass", "Soil" and "Tiles" using numbers, e.g. "Grass"=0, "Soil"=1 and "Tiles"=2
    - **Note**: the label vector $\mathbf{y}$ of this function is different than the output from `get_labels_k()`

```python
def get_labels_k(N=55,k=0):
    y = np.zeros((N,1));
    ### STUDENT TASK ###
    ## Generate the label vector which has value 1 for the pictures of the category we are currently looking at (indicated by k)
    ## and 0 for the other two categories.
    ## Hints: See the 2. Dataset-section where the picture order is defined
    # YOUR CODE HERE
    raise NotImplementedError()
    return y

def multiclass():
    y_predict = []
    step_size = 1e-5
    num_iter = 3000
    for i in range(0,3):
        ### STUDENT TASK ###
        # YOUR CODE HERE
        raise NotImplementedError()
    return y, y_hat, accuracy
```

```python
y, y_hat,acc = multiclass()

```

```python

```


(c) Juan Gomez 2019. Thanks to Universidad EAFIT for support. This material is part of the course Introduction to Finite Element Analysis

# Solution of the equlibrium equations and post-processing

## Introduction

Having solved the problem of assembly of the global equilibrium equations for the full finite element mesh we are now ready to find the unknown nodal displacements. We won't be dealing directly with solvers but will use Python's linear algebra modules. With the nodal displacements available we will be ready to compute secondary variables (e.g., stresses and strains).In this Notebook we will also use most of the available subroutines of the finite element code **SolidsPy** (Gomez and Guarin-Zapata, 2018). Most of these subroutines follow the same structure of the implementations conducted so far. **After completing this notebook you should be able to:**

* Identify well-possed systems of global equilibrium equations within the context of the finite element method.

* Perform computations of secondary variables using as input nodal displacements.

* Understand the algorithmic aspects involved in the post-processing and visualization of typical finite element solutions.

## Global equilibrium

In previous sessions it was shown that the force equilibriun equations in terms of consistent nodal forces for the complete finite element model could be written (after discretizing the prinicple of virtual) work like:

$$
\left\{F_\sigma\right\}-\left\{F_V\right\}-\left\{F_t\right\}=0
$$

where:
* $\left\{F_\sigma\right\}$ is the global internal force vector representing element stresses: $\left\{F_\sigma\right\}=\left[K^G\right]\left\{U^G\right\}$.

* $\left\{F_V\right\}$  is the global force vector corresponding to external body forces:$F_V=\int_VN^Tf\operatorname dV$.

* $\left\{F_t\right\}$ is the global force vector corresponding to external surface tractions: $F_t=\int_SN^Tt\operatorname dS.$

At this point it is also clear that the global stiffnes matriz is built through an assembly operation of independent element contributions like:

$$
\left[K^G\right]=\bigwedge_{i=1}^{Numel} k^i .
$$

Defining a total external loads vector $RHS^G$ as:

$$
RHS^G=F_V+F_t
$$

we can write the equlibrium relationship like:

$$
\left[K^G\right]\left\{U^G\right\}=\left\{RHS^G\right\}
$$

and where $\left\{U^G\right\}$ is the vector of global nodal displacements.

If apropriate boundary conditions are imposed upon $\left[K^G\right]$ the system of equations governing the static equilibrium of the system can be solved in the unknown nodal displacements $\left\{U^G\right\}$.

With $\left\{U^G\right\}$ at hand, this global vector of unknown nodal displacements is extended into $\left\{U^C\right\}$ to include also those zero-valued prescribed displacements corresponding to the imposed displacement boundary conditions. This operation is indicated like:

$$
U^G\rightarrow U^C.
$$

In a subsequent, operation the global nodal displacements are distributed or **scattered** to each element using also the data contained in the boundary conditions array **IBC()**. This operation is schematized in the figure below and indicated by:

$$
U^C\rightarrow U^Q
$$

where specific displacements from $U^C$ are extracted to form the displacemnts vector from element $Q$.

In the final post-processing step we proceed one element at a time and use the local nodal displacements vector $U^Q$ to compute strains and stresses using the strain-displacement interpolator $\left[B\right]$ and the constitutive matrix $C$ as follows:

$$
\left\{\varepsilon\right\}\;=\left[B\right]\left\{U^Q\right\}
$$

and

$$
\left\{\sigma\right\}\;=\left[C\right]\left[B\right]\left\{U^Q\right\}.
$$

**Questions:**

**(i) For element $2$ in the simple mesh problem shown in the figure plot a scattering diagram like the one shown in the figure above.**

## A complete solution step wih SolidsPy.

We now use subroutines implemented in previous notebooks and also available in the Python code **SolidsPy** to solve the simple mesh problem studied so far and shown in the figure.

The basic solution steps, after reading the input files correspond to:

* (i) Determination of the boundary conditions array and assembly operator.

* (ii) Assembly of the global matrix of stiffness coeffcients and global external loads vector.

* (iii) Solution of the system of equations.

* (iv) Post-processing.

**Question:**

**¿ If the plate shown in the figure is loaded by vertical nodal forces applied at the top surface indicate a minimum set of prescribed displacement boundary conditions for the system to have a unique solution?**

### SolidsPy modules

To execute most of this notebook the Python finite element code [SolidsPy](https://github.com/AppliedMechanics-EAFIT/SolidsPy) must be properly installed. Here we will import 3 modules, namely:

* assemutil containing assembling subroutines similar to those implemented in previous notebooks.

* solutil containing calls to solvers of linear systems of equations.

* postprocessor containg a series of subroutines for secondary calculation and visualization of finite element solutions.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym
import solidspy.assemutil as ass
import solidspy.postprocesor as pos
import solidspy.solutil as sol
```

```python
def readin():
    nodes = np.loadtxt('files/' + 'snodes.txt', ndmin=2)
    mats = np.loadtxt('files/' + 'smater.txt', ndmin=2)
    elements = np.loadtxt('files/' + 'seles.txt', ndmin=2, dtype=np.int)
    loads = np.loadtxt('files/' + 'sloads.txt', ndmin=2)

    return nodes, mats, elements, loads

nodes, mats, elements, loads = readin()
```

```python
DME, IBC, neq = ass.DME(nodes, elements)
KG = ass.assembler(elements, mats, nodes, neq, DME)
RHSG = ass.loadasem(loads, IBC, neq)
UG = sol.static_sol(KG, RHSG)
UC = pos.complete_disp(IBC, nodes, UG)
E_nodes, S_nodes = pos.strain_nodes(nodes, elements, mats, UC)
pos.fields_plot(elements, nodes, UC, E_nodes=E_nodes, S_nodes=S_nodes)
```

### Glossary of terms

**Global equlibrium:** Satic equlibrium equations in terms of internal and external forces for the complete finite element model.

**Postprocessing:** Set of computational tasks aied at producing visualizations of primary and secondary data using as input the found nodal point displacements.

**Scattering process:** Procedure that distributes nodal point displacements from the global displacement vector to each element in the mesh.

**Solubility conditions:** Minimum reuirements in terms of displacement boundary conditions to guarantee a unique solution of the discrete finite elment equlibrium equations.

## Class activity.

* (i) Write a pseudo-code to perform the scattering process where the nodal displacements from the global vector are passed to each element. Define the input and output parameters, clearly.

* (ii) Identify the subroutine (or piece of code) within the program **SolidsPy** where the scattering process takes place.

* (iii) Refine the mesh for the example problem by adding more elements while keeping the same size for the computational domain and indicate the changes in the displacement for the top surface.

### References

* Bathe, Klaus-Jürgen. (2006) Finite element procedures. Klaus-Jurgen Bathe. Prentice Hall International.

* Juan Gómez, Nicolás Guarín-Zapata (2018). SolidsPy: 2D-Finite Element Analysis with Python, <https://github.com/AppliedMechanics-EAFIT/SolidsPy>.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


# Prime Obsession

A notebook to reproduce a few results from "Prime Obsession" by John Derbyshire.

The key equation (from p328) is:
$$
\begin{equation}
\large{J(x) = Li(x) - \sum_{\rho} x^\rho - \log{2} + \int_x^\infty \frac{\mathrm{d}t}{t(t^2 - 1)\log{t}}}
\end{equation}
$$

Start with the imports for the libraries we are using:

```python
from mpmath import *
import pandas as pd
from sympy import mobius
from tqdm.notebook import trange
mp.dps = 30; mp.pretty = True
```

We use a slightly different definition of the logarithmic integral to `li` defined in `mpmath` (and Mathematica) - see note 128 on p390.

```python
def Li(x, rho=1):
    return ei(rho * log(x))
```

# Calculating $J(20)$

Pages 333 to 343 go through the calculation of $J(20)$. The secondary term is the part that requires the most work. Page 340 has a table of the logarithmic integral for the first 50 pairs of zeros, reproduced here:

```python
for i in range(1, 51):
    z = zetazero(i)
    a = Li(20, z) + Li(20, z.conjugate())
    nprint(a.real, 6)
```

    -0.210768
    0.0215632
    -0.0535991
    -0.00432174
    -0.0868451
    -0.037716
    -0.0046281
    -0.0577894
    -0.0400277
    -0.0595976
    0.0563226
    -0.0274298
    0.0481966
    0.00127986
    0.0128283
    -0.00472225
    0.0361164
    0.0317626
    0.0222196
    -0.037927
    -0.0332852
    -0.00692417
    0.0205354
    -0.0312052
    0.0280167
    0.0188243
    0.0228139
    -0.0301646
    0.0208943
    0.0275883
    0.00801349
    0.0279464
    0.0159041
    -0.0102871
    0.0224912
    -0.00106082
    0.0130158
    -0.0191586
    -0.018169
    -0.0165671
    0.0240114
    -0.0223427
    -0.0225924
    -0.000132221
    -0.0180932
    0.0221559
    -0.017333
    -0.0150514
    0.0206192
    0.0207551

As Derbyshire says on p342 you have to add up thousands of terms to get a few decimal places of accuracy.
Using `zetazero` is slow, so we can use [Andrew Odlyzko's tables of zeros](http://www.dtc.umn.edu/~odlyzko/zeta_tables/index.html) instead.

```python
import gzip

with gzip.open("data/zeros6.gz", "rt") as f:
    lines = f.readlines()
zetazeros = [mpc(0.5, float(s.strip())) for s in lines]
zetazeros[:10]
```

    [(0.5 + 14.1347251420000006305599526968j),
     (0.5 + 21.0220396389999990560681908391j),
     (0.5 + 25.0108575799999997002487361897j),
     (0.5 + 30.4248761260000009087889338844j),
     (0.5 + 32.9350615880000034962904464919j),
     (0.5 + 37.5861781589999992547745932825j),
     (0.5 + 40.9187190119999968374031595886j),
     (0.5 + 43.3270732809999969958880683407j),
     (0.5 + 48.0051508809999987192895787302j),
     (0.5 + 49.77383247800000276583887171j)]

```python
%%time
sec = 0
for i in trange(86000):
    z = zetazeros[i]
    a = Li(20, z) + Li(20, z.conjugate())
    sec += a
nprint(sec.real, 6)
```

      0%|          | 0/86000 [00:00<?, ?it/s]

    -0.370822
    CPU times: user 14.5 s, sys: 209 ms, total: 14.7 s
    Wall time: 14.6 s

We can calculate the other terms to find the value of $J(20)$:

```python
J20 = Li(20) - sec.real - log(2) + quad(lambda t: 1/(t*(t*t-1)*log(t)), [20, inf])
nprint(J20, 5)
```

    9.5833

This is the same as the result on p343.

# Calculating $\pi(1,000,000)$

Pages 343 and 344 have a calculation for $\pi(1,000,000)$, using the formula:

$$
\begin{equation}
\large{\pi(x) = \sum_{N} \frac{\mu(N)}{N}J(\sqrt[N]{x})}
\end{equation}
$$

This series is finite, since $\sqrt[n]{x} < 2$ for large enough $n$, and $J(x) = 0$ for $x < 2$.

```python
x = 1000000
```

Define a helper function that yields values of $N$ (and $y = \sqrt[N]{x}$) for which $y$ is at least 2, and the Möbius function is non-zero:

```python
def Ns(x):
    N = 1
    y = x
    while y >= 2:
        if mobius(N) != 0:
            yield N, y
        N += 1
        y = x ** (1/N)
list(Ns(x))
```

    [(1, 1000000),
     (2, 1000.0),
     (3, 99.99999999999997),
     (5, 15.848931924611136),
     (6, 9.999999999999998),
     (7, 7.196856730011519),
     (10, 3.9810717055349727),
     (11, 3.5111917342151315),
     (13, 2.894266124716751),
     (14, 2.6826957952797255),
     (15, 2.51188643150958),
     (17, 2.2539339047347906),
     (19, 2.0691380811147897)]

Now we can reproduce Table 21-1 on p344.

First, let's calculate the secondary terms, since those are the most compute intensive.

```python
%%time
secondary_terms = []
for N, y in Ns(x):
    sec = 0
    for i in trange(len(zetazeros)):
        z = zetazeros[i]
        a = Li(y, z) + Li(y, z.conjugate())
        sec += a
    sec = -mobius(N) * sec.real / N
    nprint(sec, 5)
    secondary_terms.append(sec)
```

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -29.832

      0%|          | 0/2001052 [00:00<?, ?it/s]

    0.11036

      0%|          | 0/2001052 [00:00<?, ?it/s]

    0.2999

      0%|          | 0/2001052 [00:00<?, ?it/s]

    0.087854

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.023493

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.047367

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.027906

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.0063405

      0%|          | 0/2001052 [00:00<?, ?it/s]

    0.032062

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.01581

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.0036218

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.012723

      0%|          | 0/2001052 [00:00<?, ?it/s]

    -0.022414
    CPU times: user 1h 13min 57s, sys: 58.4 s, total: 1h 14min 55s
    Wall time: 1h 14min 50s

```python
N_values = []
principal_terms = []
log2_terms = []
integral_terms = []
for N, y in Ns(x):
    N_values.append(N)
    principal_terms.append(mobius(N) * Li(y) / N)
    log2_terms.append(-mobius(N) * log(2) / N)
    integral_terms.append(mobius(N) * quad(lambda t: 1/(t*(t*t-1)*log(t)), [y, inf]) / N)
```

```python
d = {
    "N": N_values,
    "Principal term": principal_terms,
    "Secondary term": secondary_terms,
    "Log 2 term": log2_terms,
    "Integral term": integral_terms
}
df = pd.DataFrame(d).set_index("N")
df["Row totals"] = df["Principal term"] + df["Secondary term"] + df["Log 2 term"] + df["Integral term"]
df = df.astype(float)
df.style.format(precision=5)
```

      Principal term
      Secondary term
      Log 2 term
      Integral term
      Row totals

      N

      1
      78627.54916
      -29.83162
      -0.69315
      0.00000
      78597.02440

      2
      -88.80483
      0.11036
      0.34657
      -0.00000
      -88.34789

      3
      -10.04205
      0.29990
      0.23105
      -0.00000
      -9.51110

      5
      -1.69303
      0.08785
      0.13863
      -0.00012
      -1.46667

      6
      1.02760
      -0.02349
      -0.11552
      0.00031
      0.88889

      7
      -0.69393
      -0.04737
      0.09902
      -0.00058
      -0.64286

      10
      0.29539
      -0.02791
      -0.06931
      0.00183
      0.20000

      11
      -0.23615
      -0.00634
      0.06301
      -0.00234
      -0.18182

      13
      -0.15890
      0.03206
      0.05332
      -0.00340
      -0.07692

      14
      0.13281
      -0.01581
      -0.04951
      0.00394
      0.07143

      15
      0.11202
      -0.00362
      -0.04621
      0.00448
      0.06667

      17
      -0.08133
      -0.01272
      0.04077
      -0.00554
      -0.05882

      19
      -0.06013
      -0.02241
      0.03648
      -0.00657
      -0.05263

```python
totals = pd.DataFrame(df.sum(), columns=["Column totals"])
totals.style.format(precision=5)
```

      Column totals

      Principal term
      78527.34662

      Secondary term
      -29.46111

      Log 2 term
      0.03515

      Integral term
      -0.00799

      Row totals
      78497.91267

It's interesting that even with over 2 million zeros, the total is only accurate to one decimal place: it's about 0.09 away from the value of $\pi(1,000,000)$ of 78498.

The calculation in the book has a closer agreement, which suggests more than 2 million zeros were used. (The author doesn't say how many were used.)

```python

```


## Data Analytics

### Basic Univariate Statistics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Data Analytics: Basic Univariate Statistics

Here's a demonstration of calculation of univariate statistics in Python. This demonstration is part of the resources that I include for my courses in Spatial / Subsurface Data Analytics and Geostatistics at the Cockrell School of Engineering and Jackson School of Goesciences at the University of Texas at Austin.

We will cover the following statistics:

#### Measures of Centrality
* Arithmetic Average / Mean
* Median
* Mode (most frequent binned)
* Geometric Mean
* Harmonic Mean
* Power Law Average

#### Measures of Dispersion
* Population Variance
* Sample Variance
* Population Standard Deviation
* Sample Standard Deviation
* Range
* Percentile w. Tail Assumptions
* Interquartile Range

#### Tukey Outlier Test
* Lower Quartile/P25
* Upper Quartile/P75
* Interquartile Range
* Lower Fence
* Upper Fence
* Calculating Outliers

#### Measures of Shape
* Skew
* Excess Kurtosis
* Pearson' Mode Skewness
* Quartile Skew Coefficient

#### Nonparmetric Cumulative Distribution Functions (CDFs)
* plotting a nonparametric CDF
* fitting a parametric distribution and plotting

I have a lecture on these univariate statistics available on [YouTube](https://www.youtube.com/watch?v=wAcbA2cIqec&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=11&t=0s).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  The dataset is available on my GitHub account in my GeoDataSets repository at:

* Tabular data - [2D_MV_200wells.csv](https://github.com/GeostatsGuy/GeoDataSets/blob/master/2D_MV_200wells.csv)

#### Importing Packages

We will need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # plotting
import scipy                              # statistics
import statistics as stats                # statistics like the mode
from scipy.stats import norm              # fitting a Gaussian distribution
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time). Set this to your working directory, with the above mentioned data file.

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Data

Let's load the provided multivariate, spatial dataset.  '2D_MV_200wells.csv' is available at https://github.com/GeostatsGuy/GeoDataSets.  It is a comma delimited file with X and Y coordinates,facies 1 and 2 (1 is sandstone and 2 interbedded sand and mudstone), porosity (fraction), permeability (mDarcy) and acoustic impedance (kg/m2s*10^6). We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it by printing a slice and by utilizing the 'head' DataFrame member function (with a nice and clean format, see below).

```python
df = pd.read_csv("2D_MV_200wells.csv")                      # read a .csv file in as a DataFrame
#print(df.iloc[0:5,:])                                       # display first 4 samples in the table as a preview
df.head()                                                   # we could also use this command for a table preview
```

      X
      Y
      facies_threshold_0.3
      porosity
      permeability
      acoustic_impedance

      0
      565
      1485
      1
      0.1184
      6.170
      2.009

      1
      2585
      1185
      1
      0.1566
      6.275
      2.864

      2
      2065
      2865
      2
      0.1920
      92.297
      3.524

      3
      3575
      2655
      1
      0.1621
      9.048
      2.157

      4
      1835
      35
      1
      0.1766
      7.123
      3.979

Let's extract one of the features, porosity, into a 1D ndarray and do our statistics on porosity.

* then we can use NumPy's statistics methods

```python
por = df['porosity'].values
```

Now let's go through all the univariate statistics listed above one-by-one.

#### Measures of Central Tendency

Let's start with measures of central tendency.

##### The Arithmetic Average / Mean

\begin{equation}
\overline{x} = \frac{1}{n}\sum^n_{i=1} x_i
\end{equation}

```python
por_average = np.average(por)
print('Porosity average is ' + str(round(por_average,2)) + '.')
```

    Porosity average is 0.15.

##### Median

\begin{equation}
P50_x = F^{-1}_{x}(0.50)
\end{equation}

```python
por_median = np.median(por)
print('Porosity median is ' + str(round(por_median,2)) + '.')
```

    Porosity median is 0.15.

##### Mode

The most common value. To do this we should bin the data, like into histogram bins/bars.  To do this we will round the data to the 2nd decimal place.  We are assume bin boundaries, $0.01, 0.02,\ldots, 0.30$.

```python
por_mode = stats.mode(np.round(por,2))
print('Porosity mode is ' + str(round(por_mode,2)) + '.')
```

    Porosity mode is 0.14.

##### Geometric Mean

\begin{equation}
\overline{x}_G = ( \prod^n_{i=1} x_i )^{\frac{1}{n}}
\end{equation}

```python
por_geometric = scipy.stats.mstats.gmean(por)
print('Porosity geometric mean is ' + str(round(por_geometric,2)) + '.')
```

    Porosity geometric mean is 0.15.

##### Harmonic Mean

\begin{equation}
\overline{x}_H = \frac{n}{\sum^n_{i=1} \frac{1}{x_i}}
\end{equation}

```python
por_hmean = scipy.stats.mstats.hmean(por)
print('Porosity harmonic mean is ' + str(round(por_hmean,2)) + '.')
```

    Porosity harmonic mean is 0.14.

##### Power Law Average

\begin{equation}
\overline{x}_p = (\frac{1}{n}\sum^n_{i=1}{x_i^{p}})^\frac{1}{p}
\end{equation}

```python
power = 1.0
por_power = np.average(np.power(por,power))**(1/power)
print('Porosity law mean for p = ' + str(power) + ' is ' + str(round(por_power,2)) + '.')
```

    Porosity law mean for p = 1.0 is 0.15.

#### Measures of Dispersion

##### Population Variance

\begin{equation}
\sigma^2_{x} = \frac{1}{n}\sum^n_{i=1}(x_i - \mu)
\end{equation}

```python
por_varp = stats.pvariance(por)
print('Porosity population variance is ' + str(round(por_varp,4)) + '.')
```

    Porosity population variance is 0.0011.

##### Sample Variance

\begin{equation}
\sigma^2_{x} = \frac{1}{n-1}\sum^n_{i=1}(x_i - \overline{x})^2
\end{equation}

```python
por_var = stats.variance(por)
print('Porosity sample variance is ' + str(round(por_var,4)) + '.')
```

    Porosity sample variance is 0.0011.

##### Population Standard Deviation

\begin{equation}
\sigma_{x} = \sqrt{ \frac{1}{n}\sum^n_{i=1}(x_i - \mu)^2 }
\end{equation}

```python
por_stdp = stats.pstdev(por)
print('Porosity sample variance is ' + str(round(por_stdp,4)) + '.')
```

    Porosity sample variance is 0.0329.

##### Sample Standard Deviation

\begin{equation}
\sigma_{x} = \sqrt{ \frac{1}{n-1}\sum^n_{i=1}(x_i - \mu)^2 }
\end{equation}

```python
por_std = stats.stdev(por)
print('Porosity sample variance is ' + str(round(por_std,4)) + '.')
```

    Porosity sample variance is 0.0329.

##### Range

\begin{equation}
range_x = P100_x - P00_x
\end{equation}

```python
por_range = np.max(por) - np.min(por)
print('Porosity range is ' + str(round(por_range,2)) + '.')
```

    Porosity range is 0.17.

##### Percentile

\begin{equation}
P(p)_x = F^{-1}_{x}(p)
\end{equation}

```python
p_value = 13
por_percentile = np.percentile(por,p_value)
print('Porosity ' + str(int(p_value)) + 'th percentile is ' + str(round(por_percentile,2)) + '.')
```

    Porosity 13th percentile is 0.11.

##### Inter Quartile Range

\begin{equation}
IQR = P(0.75)_x - P(0.25)_x
\end{equation}

```python
por_iqr = scipy.stats.iqr(por)
print('Porosity interquartile range is ' + str(round(por_iqr,2)) + '.')
```

    Porosity interquartile range is 0.04.

#### Tukey Test for Outliers

Let's demonstrate the Tukey test for outliers based on the lower and upper fences.

\begin{equation}
fence_{lower} = P_x(0.25) - 1.5 \times [P_x(0.75) - P_x(0.25)]
\end{equation}

\begin{equation}
fence_{upper} = P_x(0.75) + 1.5 \times [P_x(0.75) - P_x(0.25)]
\end{equation}

Then we declare samples values above the upper fence or below the lower fence as outliers.

```python
p25, p75 = np.percentile(por, [25, 75])
lower_fence = p25 - por_iqr * 1.5
upper_fence = p75 + por_iqr * 1.5
por_outliers = por[np.where((por > upper_fence) | (por < lower_fence))[0]]
print('Porosity outliers by Tukey test include ' + str(por_outliers) + '.')
por_outliers_indices = np.where((por > upper_fence) | (por < lower_fence))[0]
print('Porosity outlier indices by Tukey test are ' + str(por_outliers_indices) + '.')
```

    Porosity outliers by Tukey test include [0.06726 0.05    0.06092].
    Porosity outlier indices by Tukey test are [110 152 198].

#### Measures of Shape

##### Pearson's Mode Skewness

\begin{equation}
skew = \frac{3 (\overline{x} - P50_x)}{\sigma_x}
\end{equation}

```python
por_skew = (por_average - por_median)/por_std
print('Porosity skew is ' + str(round(por_skew,2)) + '.')
```

    Porosity skew is -0.03.

##### Population Skew, 3rd Central Moment

\begin{equation}
\gamma_{x} = \frac{1}{n}\sum^n_{i=1}(x_i - \mu)^3
\end{equation}

```python
por_cm = scipy.stats.moment(por,moment=3)
print('Porosity 3rd cenral moment is ' + str(round(por_cm,7)) + '.')
```

    Porosity 3rd cenral moment is -1.22e-05.

##### Quartile Skew Coefficient

\begin{equation}
QS = \frac{(P75_x - P50_x) - (P50_x - P25_x)}{(P75_x - P25_x)}
\end{equation}

```python
por_qs = ((np.percentile(por,75)-np.percentile(por,50))
          -(np.percentile(por,50)-np.percentile(por,25))) /((np.percentile(por,75))-np.percentile(por,25))
print('Porosity quartile skew coefficient is ' + str(round(por_qs,2)) + '.')
```

    Porosity quartile skew coefficient is 0.14.

#### Plot the Nonparametric CDF

Let's demonstrate plotting a nonparametric cumulative distribution function (CDF) in Python

```python
# sort the data:
por_sort = np.sort(por)

# calculate the cumulative probabilities assuming known tails
p = np.arange(len(por)) / (len(por) - 1)

# plot the cumulative probabilities vs. the sorted porosity values
plt.subplot(122)
plt.scatter(por_sort, p, c = 'red', edgecolors = 'black', s = 10, alpha = 0.7)
plt.xlabel('Porosity (fraction)'); plt.ylabel('Cumulative Probability'); plt.grid();
plt.title('Nonparametric Porosity CDF')
plt.ylim([0,1]); plt.xlim([0,0.25])
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
```

#### Fit a Gaussian Distribution

Let's fit a Gaussian distribution

* we get fancy with Maximuum Likelihood Estimation (MLE) for the Gaussian parametric distribution fit mean and standard deviation

```python
por_values = np.linspace(0.0,0.25,100)
fit_mean, fit_stdev = norm.fit(por,loc = por_average, scale = por_std) # fit MLE of the distribution
cumul_p = norm.cdf(por_values, loc = fit_mean, scale = fit_stdev)

# plot the cumulative probabilities vs. the sorted porosity values
plt.subplot(122)
plt.scatter(por_sort, p, c = 'red', edgecolors = 'black', s = 10, alpha = 0.7)
plt.plot(por_values,cumul_p, c = 'black')
plt.xlabel('Porosity (fraction)'); plt.ylabel('Cumulative Probability'); plt.grid();
plt.title('Nonparametric Porosity CDF')
plt.ylim([0,1]); plt.xlim([0,0.25])
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
```

#### Comments

This was a basic demonstration of univariate statistics in Python.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


Trusted Notebook" width="500 px" align="left">

## _*Comparing Classical and Quantum Finite Automata (QFA)*_

Finite Automaton has been a mathematical model for computation since its invention in the 1940s. The purpose of a Finite State Machine is to recognize patterns within an input taken from some character set and accept or reject the input based on whether the pattern defined by the machine occurs in the input. The machine requires a list of states, the initial state, and the conditions for each transition from state to state. Such classical examples are vending machines, coin-operated turnstiles, elevators, traffic lights, etc.

In the classical algorithm, the sequence begins in the start state, and will only make a transition if the next character in the input string matches the label on the transition from the current to the next state. The machine will continue making transitions on each input character until no move is possible. The string will be accepted if its final state is in the accept state and will be rejected if its final state is anywhere else.

As for Quantum Finite Automata (QFA), the machine works by accepting a finite-length string of letters from a finite alphabet and utilizing quantum properties such as superposition to assign the string a probability of being in either the accept or reject state.

***
### Contributors
Kaitlin Gili, Rudy Raymond

### Qiskit Package Versions

```python
import qiskit
qiskit.__qiskit_version__
```

    {'qiskit': '0.10.3',
     'qiskit-terra': '0.8.1',
     'qiskit-ignis': '0.1.1',
     'qiskit-aer': '0.2.1',
     'qiskit-ibmq-provider': '0.2.2',
     'qiskit-aqua': '0.5.1'}

## Prime Divisibility Algorithm

Let's say that we have a string with $ a^i $ letters and we want to know whether the string is in the language $ L $ where $ L $ = {$ a^i $ | $ i $ is divisble by $ p $} and $ p $ is a prime number. If $ i $ is divisible by $ p $, we want to accept the string into the language, and if not, we want to reject it.
$|0\rangle $ and $ |1\rangle $ serve as our accept and reject states.

Classically, this algorithm requires a minimum of $ log(p) $ bits to store the information, whereas the quantum algorithm only requires $ log(log(p)) $ qubits. For example, using the highest known prime integer, the classical algorithm requires **a minimum of 77,232,917 bits**, whereas the quantum algorithm **only requires 27 qubits**.

## Introduction

The algorithm in this notebook follows that in [Ambainis et al. 1998](https://arxiv.org/pdf/quant-ph/9802062.pdf). We assume that we are given a string and a prime integer. If the user does not input a prime number, the output will be a ValueError. First, we demonstrate a simpler version of the quantum algorithm that uses $ log(p) $ qubits to store the information. Then, we can use this to more easily understand the quantum algorithm that requires only $ log(log(p)) $ qubits.

## The Algorithm for Log(p) Qubits

The algorithm is quite simple as follows.
1. Prepare quantum and classical registers for $ log(p) $ qubits initialized to zero.
$$ |0\ldots 0\rangle $$
2. Prepare $ log(p) $ random numbers k in the range {$ 1 $... $ p-1 $}. These numbers will be used to decrease the probability of a string getting accepted when $ i $ does not divide $ p $.
3. Perform a number of $ i $ Y-Rotations on each qubit, where $ \theta $ is initially zero and $ \Phi $ is the angle of rotation for each unitary. $$ \Phi = \frac{2 \pi k}{p} $$
4. In the final state:
$$ \cos \theta |0\rangle + \sin \theta |1\rangle $$
$$ \theta = \frac{2 \pi k} p {i} $$
5. Measure each of the qubits in the classical register. If $ i $ divides $ p $, $ \cos \theta $ will be one for every qubit and the state will collapse to $ |0\rangle $ to demonstrate an accept state with a probability of one. Otherwise, the output will consist of a small probability of accepting the string into the language and a higher probability of rejecting the string.

## The Circuit

We now implement the QFA Prime Divisibility algorithm with QISKit by first preparing the environment.

```python
# useful additional packages
import random
import math
from sympy.ntheory import isprime

# importing QISKit
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister
from qiskit import Aer, IBMQ, execute

from qiskit.tools.monitor import job_monitor
from qiskit.providers.ibmq import least_busy
from qiskit.tools.visualization import plot_histogram
```

```python
IBMQ.load_accounts()
sim_backend = Aer.get_backend('qasm_simulator')
device_backend = least_busy(IBMQ.backends(operational=True, simulator=False))
```

We then use QISKit to program the algorithm.

```python
#Function that takes in a prime number and a string of letters and returns a quantum circuit
def qfa_algorithm(string, prime):
    if isprime(prime) == False:
        raise ValueError("This number is not a prime") #Raises a ValueError if the input prime number is not prime
    else:
        n = math.ceil((math.log(prime))) #Rounds up to the next integer of the log(prime)
        qr = QuantumRegister(n) #Creates a quantum register of length log(prime) for log(prime) qubits
        cr = ClassicalRegister(n) #Creates a classical register for measurement
        qfaCircuit = QuantumCircuit(qr, cr) #Defining the circuit to take in the values of qr and cr
        for x in range(n): #For each qubit, we want to apply a series of unitary operations with a random int
            random_value = random.randint(1,prime - 1) #Generates the random int for each qubit from {1, prime -1}
            for letter in string: #For each letter in the string, we want to apply the same unitary operation to each qubit
                qfaCircuit.ry((2*math.pi*random_value) / prime, qr[x]) #Applies the Y-Rotation to each qubit
            qfaCircuit.measure(qr[x], cr[x]) #Measures each qubit
        return qfaCircuit #Returns the created quantum circuit
```

The qfa_algorithm function returns the Quantum Circuit qfaCircuit.

## Experiment with Simulators

We can run the above circuit on the simulator.

```python
#A function that returns a string saying if the string is accepted into the language or rejected
def accept(parameter):
    states = list(result.get_counts(parameter))
    for s in states:
        for integer in s:
            if integer == "1":
                return "Reject: the string is not accepted into the language"
    return "Accept: the string is accepted into the language"
```

Insert your own parameters and try even larger prime numbers.

```python
range_lower = 0
range_higher = 36
prime_number = 11
```

```python
for length in range(range_lower,range_higher):
    params = qfa_algorithm("a"* length, prime_number)
    job = execute(params, sim_backend, shots=1000)
    result = job.result()
    print(accept(params), "\n", "Length:",length," " ,result.get_counts(params))
```

    Accept: the string is accepted into the language
     Length: 0   {'000': 1000}
    Reject: the string is not accepted into the language
     Length: 1   {'011': 58, '100': 166, '101': 192, '000': 38, '111': 264, '010': 44, '001': 41, '110': 197}
    Reject: the string is not accepted into the language
     Length: 2   {'011': 85, '100': 163, '101': 240, '000': 128, '111': 99, '010': 51, '001': 158, '110': 76}
    Reject: the string is not accepted into the language
     Length: 3   {'011': 1, '100': 853, '101': 58, '000': 12, '111': 6, '010': 3, '001': 1, '110': 66}
    Reject: the string is not accepted into the language
     Length: 4   {'011': 118, '100': 27, '101': 125, '000': 6, '111': 559, '010': 22, '001': 22, '110': 121}
    Reject: the string is not accepted into the language
     Length: 5   {'011': 59, '100': 14, '101': 66, '000': 137, '111': 10, '010': 17, '001': 693, '110': 4}
    Reject: the string is not accepted into the language
     Length: 6   {'011': 7, '111': 289, '100': 11, '101': 671, '001': 15, '110': 7}
    Reject: the string is not accepted into the language
     Length: 7   {'011': 34, '100': 267, '101': 126, '000': 221, '111': 41, '010': 88, '001': 96, '110': 127}
    Reject: the string is not accepted into the language
     Length: 8   {'011': 26, '100': 260, '101': 343, '000': 46, '111': 144, '010': 25, '001': 70, '110': 86}
    Reject: the string is not accepted into the language
     Length: 9   {'011': 193, '000': 7, '111': 90, '100': 6, '010': 4, '101': 212, '001': 488}
    Reject: the string is not accepted into the language
     Length: 10   {'011': 63, '000': 19, '111': 24, '100': 6, '010': 2, '101': 256, '001': 630}
    Accept: the string is accepted into the language
     Length: 11   {'000': 1000}
    Reject: the string is not accepted into the language
     Length: 12   {'011': 223, '100': 54, '101': 79, '000': 117, '111': 94, '010': 179, '001': 180, '110': 74}
    Reject: the string is not accepted into the language
     Length: 13   {'011': 19, '111': 658, '100': 29, '010': 2, '101': 143, '001': 2, '110': 147}
    Reject: the string is not accepted into the language
     Length: 14   {'011': 21, '100': 228, '101': 322, '000': 144, '111': 22, '010': 19, '001': 223, '110': 21}
    Reject: the string is not accepted into the language
     Length: 15   {'011': 209, '100': 51, '101': 51, '000': 24, '111': 250, '010': 149, '001': 50, '110': 216}
    Reject: the string is not accepted into the language
     Length: 16   {'011': 14, '100': 16, '101': 2, '000': 3, '111': 68, '010': 161, '001': 1, '110': 735}
    Reject: the string is not accepted into the language
     Length: 17   {'011': 384, '100': 3, '101': 124, '000': 3, '111': 155, '010': 9, '001': 319, '110': 3}
    Reject: the string is not accepted into the language
     Length: 18   {'011': 200, '100': 41, '101': 65, '000': 27, '111': 270, '010': 143, '001': 41, '110': 213}
    Reject: the string is not accepted into the language
     Length: 19   {'011': 35, '100': 27, '101': 37, '000': 370, '111': 1, '010': 34, '001': 495, '110': 1}
    Reject: the string is not accepted into the language
     Length: 20   {'011': 5, '100': 323, '101': 426, '000': 67, '111': 34, '010': 6, '001': 113, '110': 26}
    Reject: the string is not accepted into the language
     Length: 21   {'011': 156, '100': 24, '101': 34, '000': 274, '111': 8, '010': 117, '001': 377, '110': 10}
    Accept: the string is accepted into the language
     Length: 22   {'000': 1000}
    Reject: the string is not accepted into the language
     Length: 23   {'011': 57, '000': 16, '111': 26, '100': 3, '010': 1, '101': 239, '001': 658}
    Reject: the string is not accepted into the language
     Length: 24   {'011': 641, '100': 3, '101': 11, '000': 29, '111': 52, '010': 135, '001': 119, '110': 10}
    Reject: the string is not accepted into the language
     Length: 25   {'000': 5, '111': 65, '100': 141, '010': 14, '101': 14, '110': 761}
    Reject: the string is not accepted into the language
     Length: 26   {'011': 174, '111': 788, '010': 6, '101': 15, '001': 2, '110': 15}
    Reject: the string is not accepted into the language
     Length: 27   {'011': 5, '100': 138, '101': 43, '000': 4, '111': 233, '010': 12, '001': 1, '110': 564}
    Reject: the string is not accepted into the language
     Length: 28   {'011': 23, '100': 103, '101': 436, '000': 70, '111': 34, '010': 11, '001': 315, '110': 8}
    Reject: the string is not accepted into the language
     Length: 29   {'011': 202, '100': 13, '101': 51, '000': 115, '111': 14, '010': 56, '001': 544, '110': 5}
    Reject: the string is not accepted into the language
     Length: 30   {'011': 155, '111': 644, '100': 2, '010': 3, '101': 146, '001': 38, '110': 12}
    Reject: the string is not accepted into the language
     Length: 31   {'011': 38, '100': 9, '101': 1, '000': 4, '111': 43, '010': 403, '001': 1, '110': 501}
    Reject: the string is not accepted into the language
     Length: 32   {'011': 203, '100': 7, '101': 3, '000': 11, '111': 100, '010': 475, '001': 3, '110': 198}
    Accept: the string is accepted into the language
     Length: 33   {'000': 1000}
    Reject: the string is not accepted into the language
     Length: 34   {'011': 411, '100': 3, '101': 6, '000': 4, '111': 178, '010': 263, '001': 5, '110': 130}
    Reject: the string is not accepted into the language
     Length: 35   {'011': 684, '111': 280, '010': 12, '101': 7, '001': 9, '110': 8}

### Drawing the circuit of the QFA

Below is the snapshop of the QFA for reading the bitstring of length $3$. It can be seen that there are independent QFAs each of which performs $Y$ rotation for $3$ times.

```python
qfa_algorithm("a"* 3, prime_number).draw(output='mpl')
```

## The Algorithm for Log(Log(p)) Qubits

The algorithm is quite simple as follows.
1. Prepare a quantum register for $ log(log(p)) + 1 $ qubits initialized to zero. The $ log(log(p))$ qubits will act as your control bits and the 1 extra will act as your target bit. Also prepare a classical register for 1 bit to measure the target.
$$ |0\ldots 0\rangle |0\rangle  $$
2. Hadamard the control bits to put them in a superposition so that we can perform multiple QFA's at the same time.
3. For each of $s $ states in the superposition, we can perform an individual QFA with the control qubits acting as the random integer $ k $ from the previous algorithm. Thus, we need $ n $ values from $ 1... log(p)$ for $ k $. For each letter $ i $ in the string, we perform a controlled y-rotation on the target qubit, where $ \theta $ is initially zero and $ \Phi $ is the angle of rotation for each unitary. $$ \Phi = \frac{2 \pi k_{s}}{p} $$
4. The target qubit in the final state:
$$ \cos \theta |0\rangle + \sin \theta |1\rangle $$
$$ \theta = \sum_{s=0}^n \frac{2 \pi k_{s}} p {i} $$
5. Measure the target qubit in the classical register. If $ i $ divides $ p $, $ \cos \theta $ will be one for every QFA and the state of the target will collapse to $ |0\rangle $ to demonstrate an accept state with a probability of one. Otherwise, the output will consist of a small probability of accepting the string into the language and a higher probability of rejecting the string.

## The Circuit

We then use QISKit to program the algorithm.

```python
#Function that takes in a prime number and a string of letters and returns a quantum circuit
def qfa_controlled_algorithm(string, prime):
    if isprime(prime) == False:
        raise ValueError("This number is not a prime") #Raises a ValueError if the input prime number is not prime
    else:
        n = math.ceil((math.log(math.log(prime,2),2))) #Represents log(log(p)) control qubits
        states = 2 ** (n) #Number of states that the qubits can represent/Number of QFA's to be performed
        qr = QuantumRegister(n+1) #Creates a quantum register of log(log(prime)) control qubits + 1 target qubit
        cr = ClassicalRegister(1) #Creates a classical register of log(log(prime)) control qubits + 1 target qubit
        control_qfaCircuit = QuantumCircuit(qr, cr) #Defining the circuit to take in the values of qr and cr
        for q in range(n): #We want to take each control qubit and put them in a superposition by applying a Hadamard Gate
            control_qfaCircuit.h(qr[q])
        for letter in string: #For each letter in the string, we want to apply a series of Controlled Y-rotations
            for q in range(n):
                control_qfaCircuit.cu3(2*math.pi*(2**q)/prime, 0, 0, qr[q], qr[n]) #Controlled Y on Target qubit
        control_qfaCircuit.measure(qr[n], cr[0]) #Measure the target qubit
        return control_qfaCircuit #Returns the created quantum circuit
```

The qfa_algorithm function returns the Quantum Circuit control_qfaCircuit.

## Experiment with Simulators

We can run the above circuit on the simulator.

```python
for length in range(range_lower,range_higher):
    params = qfa_controlled_algorithm("a"* length, prime_number)
    job = execute(params, sim_backend, shots=1000)
    result = job.result()
    print(accept(params), "\n", "Length:",length," " ,result.get_counts(params))
```

    Accept: the string is accepted into the language
     Length: 0   {'0': 1000}
    Reject: the string is not accepted into the language
     Length: 1   {'0': 770, '1': 230}
    Reject: the string is not accepted into the language
     Length: 2   {'0': 464, '1': 536}
    Reject: the string is not accepted into the language
     Length: 3   {'0': 542, '1': 458}
    Reject: the string is not accepted into the language
     Length: 4   {'0': 590, '1': 410}
    Reject: the string is not accepted into the language
     Length: 5   {'0': 542, '1': 458}
    Reject: the string is not accepted into the language
     Length: 6   {'0': 530, '1': 470}
    Reject: the string is not accepted into the language
     Length: 7   {'0': 602, '1': 398}
    Reject: the string is not accepted into the language
     Length: 8   {'0': 556, '1': 444}
    Reject: the string is not accepted into the language
     Length: 9   {'0': 492, '1': 508}
    Reject: the string is not accepted into the language
     Length: 10   {'0': 752, '1': 248}
    Accept: the string is accepted into the language
     Length: 11   {'0': 1000}
    Reject: the string is not accepted into the language
     Length: 12   {'0': 762, '1': 238}
    Reject: the string is not accepted into the language
     Length: 13   {'0': 450, '1': 550}
    Reject: the string is not accepted into the language
     Length: 14   {'0': 524, '1': 476}
    Reject: the string is not accepted into the language
     Length: 15   {'0': 637, '1': 363}
    Reject: the string is not accepted into the language
     Length: 16   {'0': 505, '1': 495}
    Reject: the string is not accepted into the language
     Length: 17   {'0': 520, '1': 480}
    Reject: the string is not accepted into the language
     Length: 18   {'0': 621, '1': 379}
    Reject: the string is not accepted into the language
     Length: 19   {'0': 540, '1': 460}
    Reject: the string is not accepted into the language
     Length: 20   {'0': 484, '1': 516}
    Reject: the string is not accepted into the language
     Length: 21   {'0': 773, '1': 227}
    Accept: the string is accepted into the language
     Length: 22   {'0': 1000}
    Reject: the string is not accepted into the language
     Length: 23   {'0': 773, '1': 227}
    Reject: the string is not accepted into the language
     Length: 24   {'0': 516, '1': 484}
    Reject: the string is not accepted into the language
     Length: 25   {'0': 529, '1': 471}
    Reject: the string is not accepted into the language
     Length: 26   {'0': 599, '1': 401}
    Reject: the string is not accepted into the language
     Length: 27   {'0': 518, '1': 482}
    Reject: the string is not accepted into the language
     Length: 28   {'0': 559, '1': 441}
    Reject: the string is not accepted into the language
     Length: 29   {'0': 620, '1': 380}
    Reject: the string is not accepted into the language
     Length: 30   {'0': 537, '1': 463}
    Reject: the string is not accepted into the language
     Length: 31   {'0': 464, '1': 536}
    Reject: the string is not accepted into the language
     Length: 32   {'0': 766, '1': 234}
    Accept: the string is accepted into the language
     Length: 33   {'0': 1000}
    Reject: the string is not accepted into the language
     Length: 34   {'0': 779, '1': 221}
    Reject: the string is not accepted into the language
     Length: 35   {'0': 481, '1': 519}

### Drawing the circuit of the QFA

Below is the snapshot of the QFA for reading the bitstring of length $3$. It can be seen that there is a superposition of QFAs instead of independent QFAs.

```python
qfa_controlled_algorithm("a"* 3, prime_number).draw(output='mpl')
```

## Experimenting with Real Devices

Real-device backends have errors and if the above QFAs are executed on the noisy backends, errors in rejecting strings that should have been accepted can happen. Let us see how well the real-device backends can realize the QFAs.

Let us look an example when the QFA should reject the bitstring because the length of the bitstring is not divisible by the prime number.

```python
prime_number = 3
length = 2          # set the length so that it is not divisible by the prime_number
print("The length of a is", length, " while the prime number is", prime_number)
qfa1 = qfa_controlled_algorithm("a"* length, prime_number)
```

    The length of a is 2  while the prime number is 3

```python
job = execute(qfa1, backend=device_backend, shots=100)
job_monitor(job)
```

    Job Status: job has successfully run

```python
result = job.result()
plot_histogram(result.get_counts())
```

In the above, we can see that the probability of observing "1" is quite significant. Let us see how the circuit looks like.

```python
qfa1.draw(output='mpl')
```

Now, let us see what happens when the QFAs should accept the input string.

```python
print_number = length = 3 # set the length so that it is divisible by the prime_number
print("The length of a is", length, " while the prime number is", prime_number)
qfa2 = qfa_controlled_algorithm("a"* length, prime_number)
```

    The length of a is 3  while the prime number is 3

```python
job = execute(qfa2, backend=device_backend, shots=100)
job_monitor(job)
```

    Job Status: job has successfully run

```python
result = job.result()
plot_histogram(result.get_counts())
```

The error of rejecting the bitstring is equal to the probability of observing "1" which can be checked from the above histogram. We can see that the noise of real-device backends prevents us to have a correct answer. It is left as future work on how to mitigate errors of the backends in the QFA models.

```python
qfa2.draw(output='mpl')
```


# Implementation of Quantum Annealing in QISKIT

We aim to implement in a circuit model a quantum annealing process that searches for the ground state of an Ising Hamiltonian. This is not an usual task in a real (digital) quantum computer like the ones available through the IBM Q system. It is normally more efficient to physically build a quantum device that implements this Hamiltonian (e.g. the D-Wave machine).

However, explicilty inducing quantum dynamics associated to the quantum Ising Hamiltonian is a pedagogical exercise that will teach you very useful ideas on quantum simulation and variational quantum computing.

The Ising interaction Hamiltonian takes the form

$$ H_{Ising}  = \frac{1}{2} \sum_{j,k} J_{j k} \sigma^z_j \sigma^z_l + \sum_h h^z_j \sigma^z_j $$.

We have to add a quantum term describing the action of a magnetic field in the $x-$direction. Overall, the time dependent Hamiltonian that we aim to implement is:

$$ H(t) = h \sum_j \sigma^x_j  + g(t) \left( \frac{1}{2} \sum_{j,k} J_{j k} \sigma^z_j \sigma^z_l + \sum_h h^z_j \sigma^z_j  \right)$$

The field $h$ is kept constant. The parameter $g(t)$ controls the interaction strength and it is increased from $g(0) = 0$ to $g(T) = g_f$, with $g_f$  large enough such that the z-z interactions have a dominant effect in determining the ground state of the Hamiltonian. We will fix later the functional form of $g(t)$.

The initial quantum state of the spin system is the ground state of $H(t = 0)$,

\begin{eqnarray}
| \psi(0) \rangle &=& \frac{1}{2^{n/2}} \bigotimes_j \left( |0\rangle_j + | 1 \rangle_j \right) =
\nonumber \\
               &=&   \bigotimes_j H_j |0\rangle_j  .
\end{eqnarray}

where we have assumed that $h < 0$ and in the last step we have used the fact that the initial state can be created by means of $n$ Hadamard gates.

```python
#initialization
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import math

# importing Qiskit
from qiskit import IBMQ, BasicAer
from qiskit.providers.ibmq import least_busy
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister, execute

# import basic plot tools
from qiskit.tools.visualization import plot_histogram
```

## Define gates

We start by defining gates. Although this is simple, one has to dedicate some time to come up with an implementation of the Ising interaction gate. If you consider qubits 1 and 2, for example, the gate that we need would act like this:

$$ U_{Ising}(\alpha) =  e^{-i \alpha \sigma^z_1 \sigma^z_2}$$

In a quantum annealing algorithm, $\alpha = J(t) \Delta t$, with $J(t)$ the interaction at time $t$ and $\Delta t$ the time step used in the Trotterization. We also define gates implementing the magnetic fields and the the initialization circuit. For this we use rotation matrices. The quantum evolution induced by the magnetic field in the $x$-direction is implemented with a gate of the form

$$ Rx(\theta) = e^{i (\theta/2) \sigma^x }$$

whereas the magnetic field in the $z$-direction is implemented by the gate

$$ Rz(\theta) = e^{i (\theta/2) \sigma^z }$$

```python
def ising_gate(circuit,qrj,qrl,Jt):
    circuit.cx(qrj,qrl)
    circuit.u1(-2*Jt,qrl)
    circuit.cx(qrj,qrl)
```

### * Exercise 1:

Using pen and paper, show that this "ising_gate" is equivalent to the unitary operation $U_{Ising}$ defined above.

Below we also add definitions for circuits implementing the Trotter steps corresponding to the action of the magnetic fields acting on the qubits in the x and z directions.

```python
def fieldx_gate(circuit,qrj,Bt):
    # We use rotation around x
    circuit.rx(-2*Bt,qrj)

def fieldz_gate(circuit,qrj,Bt):
    circuit.rz(-2*Bt,qrj)
```

```python
def initialize_qa(circuit,qr):
    circuit.h(qr)
```

```python
qr = QuantumRegister(2)
cr = QuantumRegister(2)
circuit = QuantumCircuit(qr)
initialize_qa(circuit,qr)
ising_gate(circuit,qr[0],qr[1],1)
fieldx_gate(circuit,qr,1)
```

The following picture is a visualization te initialization + Ising gate + rotations around x, lead to the following circuit.

The initialization step consists of a series of Hadamard gates that prepare the state

$| \psi(0) \rangle = \frac{1}{2^{n/2}} \bigotimes_j \left( |0\rangle_j + | 1 \rangle_j \right) = \bigotimes_j H_j |0\rangle_j $

Then we have an Ising gate, and finally a rotation around x (corresponding to the action of a magnetic field in the x-direction)

```python
circuit.draw(output='mpl')
```

### *  Exercise 2

Write down a circuit and run it to check numerically that the ising_gate function defined above really works. You can take as a starting point the circuit above. You could, for example, predict the output of such circuit under different combinations of gates and checke that the final measurement agrees with your calculation.

## Quantum annealing circuit

Ideally, our quantum annealing algorithm would work by implementing the following quantum dynamics,

\begin{equation}
|\psi(t) \rangle = U_{qa} |\psi(t) \rangle =
Texp  \left( -i \int_0^T \left( h \sum_{j} \sigma^x_j + g(t) H_{ising} \right) \right) |\psi(0) \rangle
\end{equation}

We have to Trotterize this expression by using an approximation of the form

$$e^{-i H(t_1) \Delta t - i H(t_2) \Delta t } \approx  e^{-i H(t_1) \Delta t} e^{- i H(t_2) \Delta t }, $$

which holds as long as $| [ H(t_1) , H(t_2) ]| \Delta t \ll 1$.

To make things easy, let us choose energy units such that $h = 1$. We will also assume that Ising couplings are of the order of $1$. (More strictly, we have to assume that $\sum_j J_{j k} \ll 1$, that is, if you take the matrix $J$, then column-wise summations are of order of 1 or less). Note that, since $J_{jk}$ is multiplied by $g$, the coupling matrix can always be normalized to satisfy this condition and the extra factor be absorbed in $g$.

Finally we have to choose $g(t)$, which has to interpolate between $g(0) = 0$ and some value $g \gg 1$ over the duration of the quantum annealing evolution, $T$. We choose an exponential evolution of the form,

$$
g(t) = g_0 \left( 1 - e^{- t/T} \right) .
$$

The value $1/T$ is chosen so that at time $t = T$, $g = g_0 (1/e)$. As a value of $g_0$ we can choose $g_0 = 5$ or $10$, so that the interaction term is much larger than the magnetic field term at the end of the evolution.

Finally, we have to deal with Trotterization. For this we need to choose a number of steps $N_{steps}$, such that the time step can be obtained from the total time

$$
\Delta t = \frac{T}{N_{steps}}
$$

Putting all together, the circuit that we need to implement is

\begin{equation}
U_{qa} = \prod_{j=1}^{N_{steps}}
\left(
e^{- i \ h  \sum_j \sigma^x_j} e^{- i \ g(t_n) H_{ising}}.
\right)
\end{equation}

### * Exercise 3:
Compose a circuit that implements the quantum annealing dynamics. (If this is too difficult, just be creative and try to find out a way to use the Ising interaction for something interesting, e.g. creating entangled states)

### * Exercise 4:
Use your quantum circuit simulator to study the efficiency of quantum annealing with respect to, for example, the ramp time.


# Automatic Differentiation is not numerical differentiation

## Instructions:
 - Copy the notebook to your Google Colab account;
 - Run the notebook with Shift-Enter at each cell;
 - Experiment and have fun!

## What is this demo about?

This demo will show you the difference between **Automatic Differentiation** (Autodiff),
**algebraic differentiation** (the one you did in math classes) and **numerical differentiation**
(coarse and shameless approximation to the derivative).

There are two flavors for Autodiff: the forward and the backward modes.
Backpropagation is a particular case of the backward mode,
and we will see an example of the forward mode -- easier to code, and illustrative of the Autodiff behavior.

**For the forward Autodiff we will use the concept of dual numbers.**
A dual number is an extension of the real numbers.  Written out, the form looks similar to a complex number.

## Complex Numbers reviewed
A complex number has the form $$z = a + bi$$ where we *define* the number $i$ so that $i^{2} = -1$.

Sometimes the imaginary unit is notated as $j$ (as in Python).

No real number has this property. Hence, the introduction of complex numbers.

Visually, you can think of a real number as a number lying on a straight line: the real line.
Then, we extend the real field to 2 dimensions. The new axis is called the *imaginary* axis.

```python
def plot_complex_number_plan(complex_numbers_arg):

    import matplotlib.pyplot as pyplot
    import numpy as numpy

    fig, ax = pyplot.subplots()
    pyplot.axis("on")
    pyplot.grid()

    # Left y-axis and bottom x-axis to centre, passing through (0,0)
    ax.spines['left'].set_position('center')
    ax.spines['bottom'].set_position('center')

    # Clean axes
    ax.spines['right'].set_color('none')
    ax.spines['top'].set_color('none')

    # Ticks on left and lower axes
    ax.xaxis.set_ticks_position('bottom')
    ax.yaxis.set_ticks_position('left')

    for x in range(len(complex_numbers_arg)):
        pyplot.plot([0,complex_numbers_arg[x].real],[0,complex_numbers_arg[x].imag],'rs-')

    limit = numpy.max(numpy.ceil(numpy.absolute(complex_numbers_arg)))

    pyplot.xlim((-limit,limit))
    pyplot.ylim((-limit,limit))
    pyplot.arrow(0, -limit, 0, 1.9*limit, head_width=limit/20, head_length=limit/20, fc='k', ec='k')
    pyplot.arrow(-limit, 0, 1.9*limit, 0, head_width=limit/20, head_length=limit/20, fc='k', ec='k')

    #pyplot.ylabel(30*" "+"Im", horizontalalignment='left',rotation=0)
    #pyplot.xlabel('Re')

    ax.set_xlabel(80*' '+'Re')
    ax.set_ylabel(50*' '+'Im')

    pyplot.show()
```

```python
complex_number = [1+4j]
plot_complex_number_plan(complex_number)
```

```python
complex_numbers = [1+4j, -2+2j]
plot_complex_number_plan(complex_numbers)
```

```python
complex_number_1 = 1+4j
complex_number_2 = -2+2j

complex_numbers = [complex_number_1, complex_number_2, (complex_number_1 + complex_number_2)]
plot_complex_number_plan(complex_numbers)
```

### A few properties and operations on Complex Numbers

$$z = a+b i$$

* Real part: $\text{Re}(z) = a$
* Imaginary part: $\text{Im}(z) = b$

* Complex conjugate: $z^{*} = a - bi$.
* Magnitude of a complex number: $\left|z\right|^{2} = zz^{*} = \left(a+bi\right)\left(a-bi\right) = a^{2} + b^{2}$.

* Summation: $z_1 + z_2 = (a_1 + a_2) + (b_1 + b_2)i$
* Multiplication: $z_1z_2 = (a_1+b_1i)(a_2+b_2i) = a_1a_2-\mathbf{b_1b_2} + (a_1b_2 +a_2b_1)i$ *->* **Feedback of the imaginary part to the real part**

```python
complex_number_1 = 1+4j
complex_number_2 = -2+2j

complex_numbers = [complex_number_1, complex_number_2, (complex_number_1 * complex_number_2)]

plot_complex_number_plan(complex_numbers)
```

## Definition of Dual Numbers
A dual number has a real part $a$ and a dual part $b$. We write $$z = a + b \epsilon $$.

We *define* the number $\epsilon \not = 0$, so that $\epsilon^{2} \approx 0$.

**Note that $\epsilon$ is NOT really zero, but is very close to it!**

$\epsilon$ is not a real number. You can imagine that $\epsilon$ is
a tiny number. When you square it, it gets much smaller.

#### Some properties of Dual Numbers:

* Real part: $\text{Re}(z) = a$
* Dual part: $\text{Dual}(z) = b$
* Conjugate:  $z^{*} = a - b \epsilon $.
* Magnitude: $\left|z\right|^{2} = zz^{*} = \left(a+ b \epsilon \right)\left(a- b \epsilon \right) = a^{2}$.
* Summation: $z_1 + z_2 = (a_1+a_2) + (b_1+b_2) \epsilon $.
* Multiplication: $z_1z_2 = (a_1a_2 +  b_1b_2\mathbf{\epsilon^2}) + (a_1b_2 + a_2b_1) \epsilon  = a_1a_2 +
(a_1b_2 + a_2b_1) \epsilon$ *->* **There is NO feedback of the dual part to the real part**

```python
# Motivation: take a small number, and square it
small = 1e-3
small, small**2
```

    (0.001, 1e-06)

## Taylor Series Expansion
A smooth function $f$ can be written as an infinite sum of polynomials

 $$ f(x) = f(a) + \frac{f^\prime(a)}{1!}(x-a) + \frac{f^{\prime\prime}(a)}{2!}(x-a)^2 + \frac{f^{\prime\prime\prime}(a)}{3!}(x-a)^3 + \cdots $$

### Example: $e^x$

$$f(x) = e^x$$

Assume $a = 0$. Then,
$$f(x) = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!}+ \dots$$

```python
def expansion_taylor(n, x):

  import math

  term = 1

  for k in range(1,n+1):
    term += x**k/math.factorial(k)
    #print(term)

  return term
```

```python
def plot_expansion_taylor(n):

  import matplotlib.pyplot as plt
  import numpy as numpy

  x = numpy.arange(-2,2,0.1)
  exp_taylor = [expansion_taylor(n,xi) for xi in x]

  fig, ax = plt.subplots()

  ax.plot(x, numpy.exp(x))
  ax.plot(x, exp_taylor)
  ax.set_ylim([-2,numpy.exp(2)])

  ax.legend(['exp() function', 'Taylor Series'])
  plt.show()
```

```python
plot_expansion_taylor(0)
```

```python
plot_expansion_taylor(1)
```

```python
plot_expansion_taylor(3)
```

```python
plot_expansion_taylor(6)
```

## Taylor Series and the Dual Number

Let's apply the Taylor expansion to $ x = a+b \epsilon$

 $$ f(a+b \epsilon) = f(a) + \frac{f^\prime(a)}{1!} b \epsilon + \frac{f^{\prime\prime}(a)}{2!}b^2 \epsilon^2 + \frac{f^{\prime\prime\prime}(a)}{3!}b^3 \epsilon^2 \epsilon + \cdots $$

And, all terms that include multiplication by $\epsilon^2$ are zero! So, we see that, for all possible $a$ and $b$,

 $$ f(a+b \epsilon) = f(a) + f^\prime(a) b \epsilon $$

### Example
Recall that the derivative of $f(x)=x^{2}$ is $f^{\prime}(x) = 2x$.

Now if we extend $x$ so that it has a real part and a dual part ($x\leftarrow a + b \epsilon $) and evaluate $f$ we have
\begin{align}
  f(a + b \epsilon) &= \left(a + b \epsilon \right)^{2} \\
    &= a^{2} + 2ab\epsilon + \underbrace{b^{2}\epsilon^{2}}_{=0} \\
    &= a^{2} + 2ab\epsilon.
\end{align}

#### Notice that the real part is $f$ evaluated at $a$, and the dual part contains the derivative of our function evaluated at $a$!!

### Example
Evaluate $y = \sin\left(x\right)$ when $x\leftarrow a + b\epsilon$.

We have
\begin{align}
  y & = \sin\left(a + b \epsilon \right) \\
    & = \sin\left(a\right)\cos\left( b \epsilon\right) + \cos\left(a\right)\sin\left( b \epsilon\right).
\end{align}

Expanding $\cos$ and $\sin$ in their Taylor series gives
\begin{align}
  \sin\left(b \epsilon \right) &= b \epsilon  + \dfrac{\left( b \epsilon \right)^{3}}{3!} + \cdots = b \epsilon \\
  \cos\left(b \epsilon \right) &= 1 + \dfrac{\left( b \epsilon \right)^{2}}{2} + \cdots = 1.
\end{align}
Remember that $\epsilon^2 \approx 0$, and $\epsilon \not = 0$.

So we see that
\begin{align}
  y & = \sin\left(a\right) + \cos\left(a\right) b \epsilon.
\end{align}
And once again the real component is the function, and the dual component is the derivative times $b$. Make $b=1$...

**And we have a closed-form computation for the derivative at a point a!!**

##Using Python Autograd

```python
import autograd
import autograd.numpy as autograd_numpy

autograd.grad(autograd_numpy.exp)(4.), autograd_numpy.exp(4.)
```

    (54.598150033144236, 54.598150033144236)

Square root function:

$$ f(x) = \sqrt(x)$$

Derivative:

$$ f^\prime (x) = \frac{1}{2 \sqrt(x)} $$

Try using autograd

```python
def derivative_sqrt(x):

  import numpy

  return 1.0/2/numpy.sqrt(x)
```

```python
def plot_comparison_sqrt():

  import matplotlib.pyplot as plt
  import autograd
  import autograd.numpy as autograd_numpy_for_sqrt

  x = autograd_numpy_for_sqrt.arange(0.1,2,0.05)
  df_dx = [derivative_sqrt(xi) for xi in x]
  auto_d = [autograd.grad(autograd_numpy_for_sqrt.sqrt)(xi) for xi in x]

  fig, ax = plt.subplots()

  ax.plot(x, autograd_numpy_for_sqrt.sqrt(x))
  ax.plot(x, df_dx)
  ax.plot(x, auto_d, 'o')

  ax.legend(['sqrt() function','Analytic (symbolic) derivative','Autodiff'])
  plt.show()
```

```python
plot_comparison_sqrt()
```

Exercises

Take function $f$ defined as
$$
f(x) = e^{x^2}.
$$

1. Compute the derivative of $f$ by hand. Remember that if $ u = x^2 $, then $$\left . \frac{df}{dx}(x) \right|_{x=a} = \left. \frac{df}{du}\frac{du}{dx}(x)\right | _{x=a}.$$
2. Write a Python function receiving as arguments function $ f $, and a step $h$ to approximate the derivative using finite differences, i.e., $$ \text{NumDiff}(f,h) = \frac{f(x+h) - f(x)}{h}.$$
3. Evaluate the functions from exercises 1 and 2 with $x \in [0.2, 0.4]$, for $h \in \{0.1, 10^{-5}, 10^{-15}\}$ and plot the results. What can you conclude?
4. Deduce the expression of $f(a+\epsilon)$. You will need the property $\epsilon^2 \approx 0$ and the Taylor expansion above for $e^x$.
5. Create a class `DualNumber` and equip it with the necessary methods to perform dual number arithmetic, namely addition, multiplication and power of a number `n`.
6.  Write a new Python function to compute the derivative of $f$ using this class. Remember you will be evaluating function $f(x)$ on the dual number $x=a+\epsilon$. Compare the result in a plot also showing `NumDiff` and the symbolic derivative you computed in the first exercise.
7. Now use the Python package `autograd`to compute the exact derivative and plot the result.
8. Use `autograd` to compute the derivative of
$$ f(x) = e^{-(\sin(x) - 4\cos(x))^2} $$

Solution:

1. $2xe^{x^2}$
2. See the code below
3. See the code below
4. $f(a+\epsilon) = e^{a^2+2a\epsilon} = e^{a^2}e^{2a\epsilon}=e^{a^2} (1+ 2a\epsilon)= e^{a^2} + 2ae^{a^2}\epsilon$
5. See the code below
6. See the code below
7. See the code below
8. See the code below

```python
# For exercise #2
def num_diff(f, h):

  return lambda x:(f(x + h) - f(x)) / h
```

```python
# For exercise #2 and #3
def symbol_exponential_derivative(x):

  import numpy as numpy

  return 2 * x * numpy.exp(x ** 2)
```

```python
# For exercise #3
def exercise_3():

  import numpy as numpy
  import matplotlib.pyplot as plt

  x = numpy.linspace(.2, .4, 15)

  def symbol_exponential_function():
    return lambda argument_x:numpy.exp(argument_x**2)

  set_O_fh = [.1, 1e-2, 1e-12, 1e-15]
  n_diff = [num_diff(symbol_exponential_function(), h) for h in set_O_fh]

  for k in range(len(set_O_fh)):
    plt.plot(x, n_diff[k](x), label = 'h={}'.format(set_O_fh[k]))

  plt.plot(x, symbol_exponential_derivative(x), 'o', label='true')
  plt.legend()

  plt.xlabel('x')
  plt.ylabel('Derivative(x)')
```

```python
# For exercise #3
exercise_3()
```

```python
# For exercise #5
""" Dual Number object for AD """
class DualNumber:

  """ Constructor method """
  def __init__(self, real_part_a, dual_part_b=1.0):
    self.real_part = real_part_a
    self.dual_part = dual_part_b

  def __add__(self, o):

    try:
      return DualNumber(self.real_part + o.real_part, self.dual_part + o.dual_part)

    except AttributeError:
      return DualNumber(self.real_part + o, self.dual_part)

  def __mul__(self, o):

    return DualNumber(self.real_part * o.real_part, self.dual_part*o.real_part+o.dual_part*self.real_part)

  def __str__(self):

    return "Real Part: %f, Dual Part: %f" % (self.real_part, self.dual_part)

  def __pow__(self, n):

    import functools

    return functools.reduce(lambda a, b: a*b, [self]*n)
```

```python
## For exercise 6
def dual_number_exp(arg_dual_number):

    import numpy as numpy

    return DualNumber(numpy.exp(arg_dual_number.real_part),
                      numpy.exp(arg_dual_number.real_part)*arg_dual_number.dual_part)
```

```python
real_part = 1.456
dual_number = DualNumber(real_part)

print(dual_number),\
print("f: % f, f\': % f" %(real_part*real_part, 2*real_part)),\
print(dual_number*dual_number), print(dual_number**2)
```

    Real Part: 1.456000, Dual Part: 1.000000
    f:  2.119936, f':  2.912000
    Real Part: 2.119936, Dual Part: 2.912000
    Real Part: 2.119936, Dual Part: 2.912000

    (None, None, None, None)

```python
real_part = 1
dual_number = DualNumber(real_part)

print(dual_number_exp(dual_number**2))
```

    Real Part: 2.718282, Dual Part: 5.436564

Use `autograd` to compute the derivative of
$$ f(x) = e^{-(\sin(x) - 4\cos(x))^2} $$

```python
# For exercise #7 and #8
def exercise_7_and_8():

  import matplotlib.pyplot as pyplot
  import autograd
  import numpy as numpy

  x = numpy.arange(-2.5, 2.5, 0.05)

  def function(arg_x):

    import autograd.numpy as autograd_numpy_module

    return autograd_numpy_module\
        .exp(-(autograd_numpy_module.sin(arg_x)
               -4*autograd_numpy_module.cos(arg_x))**2)

  auto_d = [autograd.grad(function)(x_i) for x_i in x]
  fig, ax = pyplot.subplots()

  ax.plot(x, numpy.exp(-(numpy.sin(x)-4*numpy.cos(x))**2))
  ax.plot(x, auto_d, '--')

  ax.legend(['f', 'AD'])
  pyplot.show()
```

```python
# Exercise #7 and #8
exercise_7_and_8()
```