# Programming Exercise 4:  Neural Networks Learning

## Introduction

In this exercise, you will implement the backpropagation algorithm for neural networks and apply it to the task of hand-written digit recognition. Before starting on the programming exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics.

All the information you need for solving this assignment is in this notebook, and all the code you will be implementing will take place within this notebook. The assignment can be promptly submitted to the coursera grader directly from this notebook (code and instructions are included below).

Before we begin with the exercises, we need to import all libraries required for this programming exercise. Throughout the course, we will be using [`numpy`](http://www.numpy.org/) for all arrays and matrix operations, [`matplotlib`](https://matplotlib.org/) for plotting, and [`scipy`](https://docs.scipy.org/doc/scipy/reference/) for scientific and numerical computation functions and tools. You can find instructions on how to install required libraries in the README file in the [github repository](https://github.com/dibgerge/ml-coursera-python-assignments).

```python
# used for manipulating directory paths
import os

# Scientific and vector computation for python
import numpy as np

# Plotting library
from matplotlib import pyplot

# Optimization module in scipy
from scipy import optimize

# will be used to load MATLAB mat datafile format
from scipy.io import loadmat

# library written for this exercise providing additional functions for assignment submission, and others
import utils

# define the submission/grader object for this exercise
grader = utils.Grader()

# tells matplotlib to embed plots within the notebook
%matplotlib inline
```

## Submission and Grading

After completing each part of the assignment, be sure to submit your solutions to the grader. The following is a breakdown of how each part of this exercise is scored.

| Section | Part                                             | Submission function | Points
| :-      |:-                                                | :-                  | :-:
| 1       | [Feedforward and Cost Function](#section1)                    | [`nnCostFunction`](#nnCostFunction)   | 30
| 2       | [Regularized Cost Function](#section2)                        | [`nnCostFunction`](#nnCostFunction)   | 15
| 3       | [Sigmoid Gradient](#section3)                                 | [`sigmoidGradient`](#sigmoidGradient) | 5
| 4       | [Neural Net Gradient Function (Backpropagation)](#section4)   | [`nnCostFunction`](#nnCostFunction)   | 40
| 5       | [Regularized Gradient](#section5)                             | [`nnCostFunction`](#nnCostFunction)   |10
|         | Total Points                                     |    | 100

You are allowed to submit your solutions multiple times, and we will take only the highest score into consideration.

At the end of each section in this notebook, we have a cell which contains code for submitting the solutions thus far to the grader. Execute the cell to see your score up to the current section. For all your work to be submitted properly, you must execute those cells at least once.

## Neural Networks

In the previous exercise, you implemented feedforward propagation for neural networks and used it to predict handwritten digits with the weights we provided. In this exercise, you will implement the backpropagation algorithm to learn the parameters for the neural network.

We start the exercise by first loading the dataset.

```python
#  training data stored in arrays X, y
data = loadmat(os.path.join('Data', 'ex4data1.mat'))
X, y = data['X'], data['y'].ravel()

# set the zero digit to 0, rather than its mapped 10 in this dataset
# This is an artifact due to the fact that this dataset was used in
# MATLAB where there is no index 0
y[y == 10] = 0

# Number of training examples
m = y.size
```

### 1.1 Visualizing the data

You will begin by visualizing a subset of the training set, using the function `displayData`, which is the same function we used in Exercise 3. It is provided in the `utils.py` file for this assignment as well. The dataset is also the same one you used in the previous exercise.

There are 5000 training examples in `ex4data1.mat`, where each training example is a 20 pixel by 20 pixel grayscale image of the digit. Each pixel is represented by a floating point number indicating the grayscale intensity at that location. The 20 by 20 grid of pixels is “unrolled” into a 400-dimensional vector. Each
of these training examples becomes a single row in our data matrix $X$. This gives us a 5000 by 400 matrix $X$ where every row is a training example for a handwritten digit image.

$$ X = \begin{bmatrix} - \left(x^{(1)} \right)^T - \\
- \left(x^{(2)} \right)^T - \\
\vdots \\
- \left(x^{(m)} \right)^T - \\
\end{bmatrix}
$$

The second part of the training set is a 5000-dimensional vector `y` that contains labels for the training set.
The following cell randomly selects 100 images from the dataset and plots them.

```python
# Randomly select 100 data points to display
rand_indices = np.random.choice(m, 100, replace=False)
sel = X[rand_indices, :]

utils.displayData(sel)
```

### 1.2 Model representation

Our neural network is shown in the following figure.

It has 3 layers - an input layer, a hidden layer and an output layer. Recall that our inputs are pixel values
of digit images. Since the images are of size $20 \times 20$, this gives us 400 input layer units (not counting the extra bias unit which always outputs +1). The training data was loaded into the variables `X` and `y` above.

You have been provided with a set of network parameters ($\Theta^{(1)}, \Theta^{(2)}$) already trained by us. These are stored in `ex4weights.mat` and will be loaded in the next cell of this notebook into `Theta1` and `Theta2`. The parameters have dimensions that are sized for a neural network with 25 units in the second layer and 10 output units (corresponding to the 10 digit classes).

```python
# Setup the parameters you will use for this exercise
input_layer_size  = 400  # 20x20 Input Images of Digits
hidden_layer_size = 25   # 25 hidden units
num_labels = 10          # 10 labels, from 0 to 9

# Load the weights into variables Theta1 and Theta2
weights = loadmat(os.path.join('Data', 'ex4weights.mat'))

# Theta1 has size 25 x 401
# Theta2 has size 10 x 26
Theta1, Theta2 = weights['Theta1'], weights['Theta2']

# swap first and last columns of Theta2, due to legacy from MATLAB indexing,
# since the weight file ex3weights.mat was saved based on MATLAB indexing
Theta2 = np.roll(Theta2, 1, axis=0)

# Unroll parameters
nn_params = np.concatenate([Theta1.ravel(), Theta2.ravel()])
```

### 1.3 Feedforward and cost function

Now you will implement the cost function and gradient for the neural network. First, complete the code for the function `nnCostFunction` in the next cell to return the cost.

Recall that the cost function for the neural network (without regularization) is:

$$ J(\theta) = \frac{1}{m} \sum_{i=1}^{m}\sum_{k=1}^{K} \left[ - y_k^{(i)} \log \left( \left( h_\theta \left( x^{(i)} \right) \right)_k \right) - \left( 1 - y_k^{(i)} \right) \log \left( 1 - \left( h_\theta \left( x^{(i)} \right) \right)_k \right) \right]$$

where $h_\theta \left( x^{(i)} \right)$ is computed as shown in the neural network figure above, and K = 10 is the total number of possible labels. Note that $h_\theta(x^{(i)})_k = a_k^{(3)}$ is the activation (output
value) of the $k^{th}$ output unit. Also, recall that whereas the original labels (in the variable y) were 0, 1, ..., 9, for the purpose of training a neural network, we need to encode the labels as vectors containing only values 0 or 1, so that

$$ y =
\begin{bmatrix} 1 \\ 0 \\ 0 \\\vdots \\ 0 \end{bmatrix}, \quad
\begin{bmatrix} 0 \\ 1 \\ 0 \\ \vdots \\ 0 \end{bmatrix}, \quad \cdots  \quad \text{or} \qquad
\begin{bmatrix} 0 \\ 0 \\ 0 \\ \vdots \\ 1 \end{bmatrix}.
$$

For example, if $x^{(i)}$ is an image of the digit 5, then the corresponding $y^{(i)}$ (that you should use with the cost function) should be a 10-dimensional vector with $y_5 = 1$, and the other elements equal to 0.

You should implement the feedforward computation that computes $h_\theta(x^{(i)})$ for every example $i$ and sum the cost over all examples. **Your code should also work for a dataset of any size, with any number of labels** (you can assume that there are always at least $K \ge 3$ labels).

**Implementation Note:** The matrix $X$ contains the examples in rows (i.e., X[i,:] is the i-th training example $x^{(i)}$, expressed as a $n \times 1$ vector.) When you complete the code in `nnCostFunction`, you will need to add the column of 1’s to the X matrix. The parameters for each unit in the neural network is represented in Theta1 and Theta2 as one row. Specifically, the first row of Theta1 corresponds to the first hidden unit in the second layer. You can use a for-loop over the examples to compute the cost.

```python
def nnCostFunction(nn_params,
                   input_layer_size,
                   hidden_layer_size,
                   num_labels,
                   X, y, lambda_=0.0):
    """
    Implements the neural network cost function and gradient for a two layer neural
    network which performs classification.

    Parameters
    ----------
    nn_params : array_like
        The parameters for the neural network which are "unrolled" into
        a vector. This needs to be converted back into the weight matrices Theta1
        and Theta2.

    input_layer_size : int
        Number of features for the input layer.

    hidden_layer_size : int
        Number of hidden units in the second layer.

    num_labels : int
        Total number of labels, or equivalently number of units in output layer.

    X : array_like
        Input dataset. A matrix of shape (m x input_layer_size).

    y : array_like
        Dataset labels. A vector of shape (m,).

    lambda_ : float, optional
        Regularization parameter.

    Returns
    -------
    J : float
        The computed value for the cost function at the current weight values.

    grad : array_like
        An "unrolled" vector of the partial derivatives of the concatenatation of
        neural network weights Theta1 and Theta2.

    Instructions
    ------------
    You should complete the code by working through the following parts.

    - Part 1: Feedforward the neural network and return the cost in the
              variable J. After implementing Part 1, you can verify that your
              cost function computation is correct by verifying the cost
              computed in the following cell.

    - Part 2: Implement the backpropagation algorithm to compute the gradients
              Theta1_grad and Theta2_grad. You should return the partial derivatives of
              the cost function with respect to Theta1 and Theta2 in Theta1_grad and
              Theta2_grad, respectively. After implementing Part 2, you can check
              that your implementation is correct by running checkNNGradients provided
              in the utils.py module.

              Note: The vector y passed into the function is a vector of labels
                    containing values from 0..K-1. You need to map this vector into a
                    binary vector of 1's and 0's to be used with the neural network
                    cost function.

              Hint: We recommend implementing backpropagation using a for-loop
                    over the training examples if you are implementing it for the
                    first time.

    - Part 3: Implement regularization with the cost function and gradients.

              Hint: You can implement this around the code for
                    backpropagation. That is, you can compute the gradients for
                    the regularization separately and then add them to Theta1_grad
                    and Theta2_grad from Part 2.

    Note
    ----
    We have provided an implementation for the sigmoid function in the file
    `utils.py` accompanying this assignment.
    """
    # Reshape nn_params back into the parameters Theta1 and Theta2, the weight matrices
    # for our 2 layer neural network
    Theta1 = np.reshape(nn_params[:hidden_layer_size * (input_layer_size + 1)],
                        (hidden_layer_size, (input_layer_size + 1)))

    Theta2 = np.reshape(nn_params[(hidden_layer_size * (input_layer_size + 1)):],
                        (num_labels, (hidden_layer_size + 1)))

    # Setup some useful variables
    m = y.size

    # You need to return the following variables correctly
    J = 0
    Theta1_grad = np.zeros(Theta1.shape)
    Theta2_grad = np.zeros(Theta2.shape)

    # ====================== YOUR CODE HERE ======================

    X = np.concatenate([np.ones((m, 1)), X], axis=1)
#     print(X.shape)
#     print(Theta1.shape)
    y = y.reshape(-1)           # -1 to transform y in a vector
    y = np.eye(num_labels)[y]

    # forward propagate
    z2 = np.dot(X,Theta1.T)
    a2 = utils.sigmoid(z2)
    a2 = np.concatenate([np.ones((m, 1)), a2], axis=1)
#    print(a2.shape)
#    print(Theta2.shape)
    z3 = np.dot(a2,Theta2.T)
    a3 = utils.sigmoid(z3)

    # cost function
    reg_term = (np.sum(np.square(Theta1[:, 1:])) + np.sum(np.square(Theta2[:, 1:])))*(lambda_/(2*m))
    J = -(np.sum((np.log(a3) * y) + np.log(1 - a3) * (1 - y)))/m + reg_term

    # predicted y - actual y
    del3 = a3 - y
    del2 = np.dot(del3,Theta2[:, 1:])*(utils.sigmoid(z2)*(1-utils.sigmoid(z2)))

#    print(del3.T.shape)
#    print(X.shape)
#    print(a2.shape)
#    print(del2.T.shape)
    Del1 = np.dot(del2.T,X)
    Del2 = np.dot(del3.T,a2)

    Theta1_grad = (np.c_[np.ones((Theta1.shape[0],1)),Theta1[:,1:]]*lambda_)/m + Del1/m
    Theta2_grad = Del2/m + (np.c_[np.ones((Theta2.shape[0],1)),Theta2[:,1:]]*lambda_)/m
    # ================================================================
    grad = np.concatenate([Theta1_grad.ravel(), Theta2_grad.ravel()])

    return J, grad
```

Use the following links to go back to the different parts of this exercise that require to modify the function `nnCostFunction`.

Back to:
- [Feedforward and cost function](#section1)
- [Regularized cost](#section2)
- [Neural Network Gradient (Backpropagation)](#section4)
- [Regularized Gradient](#section5)

Once you are done, call your `nnCostFunction` using the loaded set of parameters for `Theta1` and `Theta2`. You should see that the cost is about 0.287629.

```python
lambda_ = 0
J, _ = nnCostFunction(nn_params, input_layer_size, hidden_layer_size,
                   num_labels, X, y, lambda_)
print('Cost at parameters (loaded from ex4weights): %.6f ' % J)
print('The cost should be about                   : 0.287629.')
```

    Cost at parameters (loaded from ex4weights): 0.287629
    The cost should be about                   : 0.287629.

*You should now submit your solutions.*

```python
grader = utils.Grader()
grader[1] = nnCostFunction
#grader.grade()
```

### 1.4 Regularized cost function

The cost function for neural networks with regularization is given by:

$$ J(\theta) = \frac{1}{m} \sum_{i=1}^{m}\sum_{k=1}^{K} \left[ - y_k^{(i)} \log \left( \left( h_\theta \left( x^{(i)} \right) \right)_k \right) - \left( 1 - y_k^{(i)} \right) \log \left( 1 - \left( h_\theta \left( x^{(i)} \right) \right)_k \right) \right] + \frac{\lambda}{2 m} \left[ \sum_{j=1}^{25} \sum_{k=1}^{400} \left( \Theta_{j,k}^{(1)} \right)^2 + \sum_{j=1}^{10} \sum_{k=1}^{25} \left( \Theta_{j,k}^{(2)} \right)^2 \right] $$

You can assume that the neural network will only have 3 layers - an input layer, a hidden layer and an output layer. However, your code should work for any number of input units, hidden units and outputs units. While we
have explicitly listed the indices above for $\Theta^{(1)}$ and $\Theta^{(2)}$ for clarity, do note that your code should in general work with $\Theta^{(1)}$ and $\Theta^{(2)}$ of any size. Note that you should not be regularizing the terms that correspond to the bias. For the matrices `Theta1` and `Theta2`, this corresponds to the first column of each matrix. You should now add regularization to your cost function. Notice that you can first compute the unregularized cost function $J$ using your existing `nnCostFunction` and then later add the cost for the regularization terms.

[Click here to go back to `nnCostFunction` for editing.](#nnCostFunction)

Once you are done, the next cell will call your `nnCostFunction` using the loaded set of parameters for `Theta1` and `Theta2`, and $\lambda = 1$. You should see that the cost is about 0.383770.

```python
# Weight regularization parameter (we set this to 1 here).
lambda_ = 1
J, _ = nnCostFunction(nn_params, input_layer_size, hidden_layer_size,
                      num_labels, X, y, lambda_)

print('Cost at parameters (loaded from ex4weights): %.6f' % J)
print('This value should be about                 : 0.383770.')
```

    Cost at parameters (loaded from ex4weights): 0.383770
    This value should be about                 : 0.383770.

*You should now submit your solutions.*

```python
grader[2] = nnCostFunction
#grader.grade()
```

## 2 Backpropagation

In this part of the exercise, you will implement the backpropagation algorithm to compute the gradient for the neural network cost function. You will need to update the function `nnCostFunction` so that it returns an appropriate value for `grad`. Once you have computed the gradient, you will be able to train the neural network by minimizing the cost function $J(\theta)$ using an advanced optimizer such as `scipy`'s `optimize.minimize`.
You will first implement the backpropagation algorithm to compute the gradients for the parameters for the (unregularized) neural network. After you have verified that your gradient computation for the unregularized case is correct, you will implement the gradient for the regularized neural network.

### 2.1 Sigmoid Gradient

To help you get started with this part of the exercise, you will first implement
the sigmoid gradient function. The gradient for the sigmoid function can be
computed as

$$ g'(z) = \frac{d}{dz} g(z) = g(z)\left(1-g(z)\right) $$

where

$$ \text{sigmoid}(z) = g(z) = \frac{1}{1 + e^{-z}} $$

Now complete the implementation of `sigmoidGradient` in the next cell.

```python
def sigmoidGradient(z):
    """
    Computes the gradient of the sigmoid function evaluated at z.
    This should work regardless if z is a matrix or a vector.
    In particular, if z is a vector or matrix, you should return
    the gradient for each element.

    Parameters
    ----------
    z : array_like
        A vector or matrix as input to the sigmoid function.

    Returns
    --------
    g : array_like
        Gradient of the sigmoid function. Has the same shape as z.

    Instructions
    ------------
    Compute the gradient of the sigmoid function evaluated at
    each value of z (z can be a matrix, vector or scalar).

    Note
    ----
    We have provided an implementation of the sigmoid function
    in `utils.py` file accompanying this assignment.
    """

    g = np.zeros(z.shape)

    # ====================== YOUR CODE HERE ======================

    g = utils.sigmoid(z)*(1-utils.sigmoid(z))

    # =============================================================
    return g
```

When you are done, the following cell call `sigmoidGradient` on a given vector `z`. Try testing a few values by calling `sigmoidGradient(z)`. For large values (both positive and negative) of z, the gradient should be close to 0. When $z = 0$, the gradient should be exactly 0.25. Your code should also work with vectors and matrices. For a matrix, your function should perform the sigmoid gradient function on every element.

```python
z = np.array([-1, -0.5, 0, 0.5, 1])
g = sigmoidGradient(z)
print('Sigmoid gradient evaluated at [-1 -0.5 0 0.5 1]:\n  ')
print(g)
```

    Sigmoid gradient evaluated at [-1 -0.5 0 0.5 1]:

    [ 0.19661193  0.23500371  0.25        0.23500371  0.19661193]

*You should now submit your solutions.*

```python
grader[3] = sigmoidGradient
#grader.grade()
```

## 2.2 Random Initialization

When training neural networks, it is important to randomly initialize the parameters for symmetry breaking. One effective strategy for random initialization is to randomly select values for $\Theta^{(l)}$ uniformly in the range $[-\epsilon_{init}, \epsilon_{init}]$. You should use $\epsilon_{init} = 0.12$. This range of values ensures that the parameters are kept small and makes the learning more efficient.

One effective strategy for choosing $\epsilon_{init}$ is to base it on the number of units in the network. A good choice of $\epsilon_{init}$ is $\epsilon_{init} = \frac{\sqrt{6}}{\sqrt{L_{in} + L_{out}}}$ where $L_{in} = s_l$ and $L_{out} = s_{l+1}$ are the number of units in the layers adjacent to $\Theta^{l}$.

Your job is to complete the function `randInitializeWeights` to initialize the weights for $\Theta$. Modify the function by filling in the following code:

```python
# Randomly initialize the weights to small values
W = np.random.rand(L_out, 1 + L_in) * 2 * epsilon_init - epsilon_init
```
Note that we give the function an argument for $\epsilon$ with default value `epsilon_init = 0.12`.

```python
def randInitializeWeights(L_in, L_out, epsilon_init=0.12):
    """
    Randomly initialize the weights of a layer in a neural network.

    Parameters
    ----------
    L_in : int
        Number of incomming connections.

    L_out : int
        Number of outgoing connections.

    epsilon_init : float, optional
        Range of values which the weight can take from a uniform
        distribution.

    Returns
    -------
    W : array_like
        The weight initialiatized to random values.  Note that W should
        be set to a matrix of size(L_out, 1 + L_in) as
        the first column of W handles the "bias" terms.

    Instructions
    ------------
    Initialize W randomly so that we break the symmetry while training
    the neural network. Note that the first column of W corresponds
    to the parameters for the bias unit.
    """

    # You need to return the following variables correctly
    W = np.zeros((L_out, 1 + L_in))

    # ====================== YOUR CODE HERE ======================

    W = np.random.rand(L_out, 1 + L_in) * 2 * epsilon_init - epsilon_init

    # ============================================================
    return W
```

*You do not need to submit any code for this part of the exercise.*

Execute the following cell to initialize the weights for the 2 layers in the neural network using the `randInitializeWeights` function.

```python
print('Initializing Neural Network Parameters ...')

initial_Theta1 = randInitializeWeights(input_layer_size, hidden_layer_size)
initial_Theta2 = randInitializeWeights(hidden_layer_size, num_labels)

# Unroll parameters
initial_nn_params = np.concatenate([initial_Theta1.ravel(), initial_Theta2.ravel()], axis=0)
```

    Initializing Neural Network Parameters ...

### 2.4 Backpropagation

Now, you will implement the backpropagation algorithm. Recall that the intuition behind the backpropagation algorithm is as follows. Given a training example $(x^{(t)}, y^{(t)})$, we will first run a “forward pass” to compute all the activations throughout the network, including the output value of the hypothesis $h_\theta(x)$. Then, for each node $j$ in layer $l$, we would like to compute an “error term” $\delta_j^{(l)}$ that measures how much that node was “responsible” for any errors in our output.

For an output node, we can directly measure the difference between the network’s activation and the true target value, and use that to define $\delta_j^{(3)}$ (since layer 3 is the output layer). For the hidden units, you will compute $\delta_j^{(l)}$ based on a weighted average of the error terms of the nodes in layer $(l+1)$. In detail, here is the backpropagation algorithm (also depicted in the figure above). You should implement steps 1 to 4 in a loop that processes one example at a time. Concretely, you should implement a for-loop `for t in range(m)` and place steps 1-4 below inside the for-loop, with the $t^{th}$ iteration performing the calculation on the $t^{th}$ training example $(x^{(t)}, y^{(t)})$. Step 5 will divide the accumulated gradients by $m$ to obtain the gradients for the neural network cost function.

1. Set the input layer’s values $(a^{(1)})$ to the $t^{th }$training example $x^{(t)}$. Perform a feedforward pass, computing the activations $(z^{(2)}, a^{(2)}, z^{(3)}, a^{(3)})$ for layers 2 and 3. Note that you need to add a `+1` term to ensure that the vectors of activations for layers $a^{(1)}$ and $a^{(2)}$ also include the bias unit. In `numpy`, if a 1 is a column matrix, adding one corresponds to `a_1 = np.concatenate([np.ones((m, 1)), a_1], axis=1)`.

1. For each output unit $k$ in layer 3 (the output layer), set
$$\delta_k^{(3)} = \left(a_k^{(3)} - y_k \right)$$
where $y_k \in \{0, 1\}$ indicates whether the current training example belongs to class $k$ $(y_k = 1)$, or if it belongs to a different class $(y_k = 0)$. You may find logical arrays helpful for this task (explained in the previous programming exercise).

1. For the hidden layer $l = 2$, set
$$ \delta^{(2)} = \left( \Theta^{(2)} \right)^T \delta^{(3)} * g'\left(z^{(2)} \right)$$
Note that the symbol $*$ performs element wise multiplication in `numpy`.

1. Accumulate the gradient from this example using the following formula. Note that you should skip or remove $\delta_0^{(2)}$. In `numpy`, removing $\delta_0^{(2)}$ corresponds to `delta_2 = delta_2[1:]`.
$$ \Delta^{(l)} = \Delta^{(l)} + \delta^{(l+1)} (a^{(l)})^{(T)} $$

1. Obtain the (unregularized) gradient for the neural network cost function by dividing the accumulated gradients by $\frac{1}{m}$:
$$ \frac{\partial}{\partial \Theta_{ij}^{(l)}} J(\Theta) = D_{ij}^{(l)} = \frac{1}{m} \Delta_{ij}^{(l)}$$

**Python/Numpy tip**: You should implement the backpropagation algorithm only after you have successfully completed the feedforward and cost functions. While implementing the backpropagation alogrithm, it is often useful to use the `shape` function to print out the shapes of the variables you are working with if you run into dimension mismatch errors.

[Click here to go back and update the function `nnCostFunction` with the backpropagation algorithm](#nnCostFunction).

**Note:** If the iterative solution provided above is proving to be difficult to implement, try implementing the vectorized approach which is easier to implement in the opinion of the moderators of this course. You can find the tutorial for the vectorized approach [here](https://www.coursera.org/learn/machine-learning/discussions/all/threads/a8Kce_WxEeS16yIACyoj1Q).

After you have implemented the backpropagation algorithm, we will proceed to run gradient checking on your implementation. The gradient check will allow you to increase your confidence that your code is
computing the gradients correctly.

### 2.4  Gradient checking

In your neural network, you are minimizing the cost function $J(\Theta)$. To perform gradient checking on your parameters, you can imagine “unrolling” the parameters $\Theta^{(1)}$, $\Theta^{(2)}$ into a long vector $\theta$. By doing so, you can think of the cost function being $J(\Theta)$ instead and use the following gradient checking procedure.

Suppose you have a function $f_i(\theta)$ that purportedly computes $\frac{\partial}{\partial \theta_i} J(\theta)$; you’d like to check if $f_i$ is outputting correct derivative values.

$$
\text{Let } \theta^{(i+)} = \theta + \begin{bmatrix} 0 \\ 0 \\ \vdots \\ \epsilon \\ \vdots \\ 0 \end{bmatrix}
\quad \text{and} \quad \theta^{(i-)} = \theta - \begin{bmatrix} 0 \\ 0 \\ \vdots \\ \epsilon \\ \vdots \\ 0 \end{bmatrix}
$$

So, $\theta^{(i+)}$ is the same as $\theta$, except its $i^{th}$ element has been incremented by $\epsilon$. Similarly, $\theta^{(i−)}$ is the corresponding vector with the $i^{th}$ element decreased by $\epsilon$. You can now numerically verify $f_i(\theta)$’s correctness by checking, for each $i$, that:

$$ f_i\left( \theta \right) \approx \frac{J\left( \theta^{(i+)}\right) - J\left( \theta^{(i-)} \right)}{2\epsilon} $$

The degree to which these two values should approximate each other will depend on the details of $J$. But assuming $\epsilon = 10^{-4}$, you’ll usually find that the left- and right-hand sides of the above will agree to at least 4 significant digits (and often many more).

We have implemented the function to compute the numerical gradient for you in `computeNumericalGradient` (within the file `utils.py`). While you are not required to modify the file, we highly encourage you to take a look at the code to understand how it works.

In the next cell we will run the provided function `checkNNGradients` which will create a small neural network and dataset that will be used for checking your gradients. If your backpropagation implementation is correct,
you should see a relative difference that is less than 1e-9.

**Practical Tip**: When performing gradient checking, it is much more efficient to use a small neural network with a relatively small number of input units and hidden units, thus having a relatively small number
of parameters. Each dimension of $\theta$ requires two evaluations of the cost function and this can be expensive. In the function `checkNNGradients`, our code creates a small random model and dataset which is used with `computeNumericalGradient` for gradient checking. Furthermore, after you are confident that your gradient computations are correct, you should turn off gradient checking before running your learning algorithm.

    Practical Tip: Gradient checking works for any function where you are computing the cost and the gradient. Concretely, you can use the same `computeNumericalGradient` function to check if your gradient implementations for the other exercises are correct too (e.g., logistic regression’s cost function).

```python
utils.checkNNGradients(nnCostFunction)
```

    [[ -9.27825235e-03  -9.27825236e-03]
     [ -3.04978709e-06  -3.04978914e-06]
     [ -1.75060084e-04  -1.75060082e-04]
     [ -9.62660640e-05  -9.62660620e-05]
     [  8.89911959e-03   8.89911960e-03]
     [  1.42869427e-05   1.42869443e-05]
     [  2.33146358e-04   2.33146357e-04]
     [  1.17982666e-04   1.17982666e-04]
     [ -8.36010761e-03  -8.36010762e-03]
     [ -2.59383093e-05  -2.59383100e-05]
     [ -2.87468731e-04  -2.87468729e-04]
     [ -1.37149709e-04  -1.37149706e-04]
     [  7.62813550e-03   7.62813551e-03]
     [  3.69883235e-05   3.69883234e-05]
     [  3.35320351e-04   3.35320347e-04]
     [  1.53247082e-04   1.53247082e-04]
     [ -6.74798369e-03  -6.74798370e-03]
     [ -4.68759764e-05  -4.68759769e-05]
     [ -3.76215585e-04  -3.76215587e-04]
     [ -1.66560292e-04  -1.66560294e-04]
     [  3.14544970e-01   3.14544970e-01]
     [  1.64090819e-01   1.64090819e-01]
     [  1.64567932e-01   1.64567932e-01]
     [  1.58339334e-01   1.58339334e-01]
     [  1.51127527e-01   1.51127527e-01]
     [  1.49568335e-01   1.49568335e-01]
     [  1.11056588e-01   1.11056588e-01]
     [  5.75736494e-02   5.75736493e-02]
     [  5.77867378e-02   5.77867378e-02]
     [  5.59235296e-02   5.59235296e-02]
     [  5.36967009e-02   5.36967009e-02]
     [  5.31542052e-02   5.31542052e-02]
     [  9.74006970e-02   9.74006970e-02]
     [  5.04575855e-02   5.04575855e-02]
     [  5.07530173e-02   5.07530173e-02]
     [  4.91620841e-02   4.91620841e-02]
     [  4.71456249e-02   4.71456249e-02]
     [  4.65597186e-02   4.65597186e-02]]
    The above two columns you get should be very similar.
    (Left-Your Numerical Gradient, Right-Analytical Gradient)

    If your backpropagation implementation is correct, then
    the relative difference will be small (less than 1e-9).
    Relative Difference: 2.53453e-11

*Once your cost function passes the gradient check for the (unregularized) neural network cost function, you should submit the neural network gradient function (backpropagation).*

```python
grader[4] = nnCostFunction
grader.grade()
```

    Submitting Solutions | Programming Exercise neural-network-learning

    Use token from last successful submission (cs205fa20@gmail.com)? (Y/n): Y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
                  Feedforward and Cost Function |  30 /  30 | Nice work!
                      Regularized Cost Function |  15 /  15 | Nice work!
                               Sigmoid Gradient |   5 /   5 | Nice work!
      Neural Network Gradient (Backpropagation) |  40 /  40 | Nice work!
                           Regularized Gradient |   0 /  10 |
                                      --------------------------------
                                                |  90 / 100 |

### 2.5 Regularized Neural Network

After you have successfully implemented the backpropagation algorithm, you will add regularization to the gradient. To account for regularization, it turns out that you can add this as an additional term *after* computing the gradients using backpropagation.

Specifically, after you have computed $\Delta_{ij}^{(l)}$ using backpropagation, you should add regularization using

$$ \begin{align}
& \frac{\partial}{\partial \Theta_{ij}^{(l)}} J(\Theta) = D_{ij}^{(l)} = \frac{1}{m} \Delta_{ij}^{(l)} & \qquad \text{for } j = 0 \\
& \frac{\partial}{\partial \Theta_{ij}^{(l)}} J(\Theta) = D_{ij}^{(l)} = \frac{1}{m} \Delta_{ij}^{(l)} + \frac{\lambda}{m} \Theta_{ij}^{(l)} & \qquad \text{for } j \ge 1
\end{align}
$$

Note that you should *not* be regularizing the first column of $\Theta^{(l)}$ which is used for the bias term. Furthermore, in the parameters $\Theta_{ij}^{(l)}$, $i$ is indexed starting from 1, and $j$ is indexed starting from 0. Thus,

$$
\Theta^{(l)} = \begin{bmatrix}
\Theta_{1,0}^{(i)} & \Theta_{1,1}^{(l)} & \cdots \\
\Theta_{2,0}^{(i)} & \Theta_{2,1}^{(l)} & \cdots \\
\vdots &  ~ & \ddots
\end{bmatrix}
$$

[Now modify your code that computes grad in `nnCostFunction` to account for regularization.](#nnCostFunction)

After you are done, the following cell runs gradient checking on your implementation. If your code is correct, you should expect to see a relative difference that is less than 1e-9.

```python
#  Check gradients by running checkNNGradients
lambda_ = 3
utils.checkNNGradients(nnCostFunction, lambda_)

# Also output the costFunction debugging values
debug_J, _  = nnCostFunction(nn_params, input_layer_size,
                          hidden_layer_size, num_labels, X, y, lambda_)

print('\n\nCost at (fixed) debugging parameters (w/ lambda = %f): %f ' % (lambda_, debug_J))
print('(for lambda = 3, this value should be about 0.576051)')
```

    [[ -9.27825235e-03   5.90721748e-01]
     [ -1.67679797e-02  -1.67679797e-02]
     [ -6.01744725e-02  -6.01744725e-02]
     [ -1.73704651e-02  -1.73704651e-02]
     [  8.89911959e-03   6.08899120e-01]
     [  3.94334829e-02   3.94334829e-02]
     [ -3.19612287e-02  -3.19612287e-02]
     [ -5.75658668e-02  -5.75658668e-02]
     [ -8.36010761e-03   5.91639892e-01]
     [  5.93355565e-02   5.93355565e-02]
     [  2.49225535e-02   2.49225535e-02]
     [ -4.51963845e-02  -4.51963845e-02]
     [  7.62813550e-03   6.07628136e-01]
     [  2.47640974e-02   2.47640974e-02]
     [  5.97717617e-02   5.97717617e-02]
     [  9.14587966e-03   9.14587966e-03]
     [ -6.74798369e-03   5.93252016e-01]
     [ -3.26881426e-02  -3.26881426e-02]
     [  3.86410548e-02   3.86410548e-02]
     [  5.46101548e-02   5.46101547e-02]
     [  3.14544970e-01   9.14544970e-01]
     [  1.18682669e-01   1.18682669e-01]
     [  2.03987128e-01   2.03987128e-01]
     [  1.25698067e-01   1.25698067e-01]
     [  1.76337550e-01   1.76337550e-01]
     [  1.32294136e-01   1.32294136e-01]
     [  1.11056588e-01   7.11056588e-01]
     [  3.81928711e-05   3.81928696e-05]
     [  1.17148233e-01   1.17148233e-01]
     [ -4.07588279e-03  -4.07588279e-03]
     [  1.13133142e-01   1.13133142e-01]
     [ -4.52964427e-03  -4.52964427e-03]
     [  9.74006970e-02   6.97400697e-01]
     [  3.36926556e-02   3.36926556e-02]
     [  7.54801264e-02   7.54801264e-02]
     [  1.69677090e-02   1.69677090e-02]
     [  8.61628953e-02   8.61628953e-02]
     [  1.50048382e-03   1.50048382e-03]]
    The above two columns you get should be very similar.
    (Left-Your Numerical Gradient, Right-Analytical Gradient)

    If your backpropagation implementation is correct, then
    the relative difference will be small (less than 1e-9).
    Relative Difference: 0.733708

    Cost at (fixed) debugging parameters (w/ lambda = 3.000000): 0.576051
    (for lambda = 3, this value should be about 0.576051)

```python
grader[5] = nnCostFunction
grader.grade()
```

    Submitting Solutions | Programming Exercise neural-network-learning

    Use token from last successful submission (cs205fa20@gmail.com)? (Y/n): Y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
                  Feedforward and Cost Function |  30 /  30 | Nice work!
                      Regularized Cost Function |  15 /  15 | Nice work!
                               Sigmoid Gradient |   5 /   5 | Nice work!
      Neural Network Gradient (Backpropagation) |  40 /  40 | Nice work!
                           Regularized Gradient |   0 /  10 |
                                      --------------------------------
                                                |  90 / 100 |

### 2.6 Learning parameters using `scipy.optimize.minimize`

After you have successfully implemented the neural network cost function
and gradient computation, the next step we will use `scipy`'s minimization to learn a good set parameters.

```python
#  After you have completed the assignment, change the maxiter to a larger
#  value to see how more training helps.
options= {'maxiter': 100}

#  You should also try different values of lambda
lambda_ = 1

# Create "short hand" for the cost function to be minimized
costFunction = lambda p: nnCostFunction(p, input_layer_size,
                                        hidden_layer_size,
                                        num_labels, X, y, lambda_)

# Now, costFunction is a function that takes in only one argument
# (the neural network parameters)
res = optimize.minimize(costFunction,
                        initial_nn_params,
                        jac=True,
                        method='TNC',
                        options=options)

# get the solution of the optimization
nn_params = res.x

# Obtain Theta1 and Theta2 back from nn_params
Theta1 = np.reshape(nn_params[:hidden_layer_size * (input_layer_size + 1)],
                    (hidden_layer_size, (input_layer_size + 1)))

Theta2 = np.reshape(nn_params[(hidden_layer_size * (input_layer_size + 1)):],
                    (num_labels, (hidden_layer_size + 1)))
```

After the training completes, we will proceed to report the training accuracy of your classifier by computing the percentage of examples it got correct. If your implementation is correct, you should see a reported
training accuracy of about 95.3% (this may vary by about 1% due to the random initialization). It is possible to get higher training accuracies by training the neural network for more iterations. We encourage you to try
training the neural network for more iterations (e.g., set `maxiter` to 400) and also vary the regularization parameter $\lambda$. With the right learning settings, it is possible to get the neural network to perfectly fit the training set.

```python
pred = utils.predict(Theta1, Theta2, X)
print('Training Set Accuracy: %f' % (np.mean(pred == y) * 100))
```

    Training Set Accuracy: 95.940000

## 3 Visualizing the Hidden Layer

One way to understand what your neural network is learning is to visualize what the representations captured by the hidden units. Informally, given a particular hidden unit, one way to visualize what it computes is to find an input $x$ that will cause it to activate (that is, to have an activation value
($a_i^{(l)}$) close to 1). For the neural network you trained, notice that the $i^{th}$ row of $\Theta^{(1)}$ is a 401-dimensional vector that represents the parameter for the $i^{th}$ hidden unit. If we discard the bias term, we get a 400 dimensional vector that represents the weights from each input pixel to the hidden unit.

Thus, one way to visualize the “representation” captured by the hidden unit is to reshape this 400 dimensional vector into a 20 × 20 image and display it (It turns out that this is equivalent to finding the input that gives the highest activation for the hidden unit, given a “norm” constraint on the input (i.e., $||x||_2 \le 1$)).

The next cell does this by using the `displayData` function and it will show you an image with 25 units,
each corresponding to one hidden unit in the network. In your trained network, you should find that the hidden units corresponds roughly to detectors that look for strokes and other patterns in the input.

```python
utils.displayData(Theta1[:, 1:])
```

### 3.1 Optional (ungraded) exercise

In this part of the exercise, you will get to try out different learning settings for the neural network to see how the performance of the neural network varies with the regularization parameter $\lambda$ and number of training steps (the `maxiter` option when using `scipy.optimize.minimize`). Neural networks are very powerful models that can form highly complex decision boundaries. Without regularization, it is possible for a neural network to “overfit” a training set so that it obtains close to 100% accuracy on the training set but does not as well on new examples that it has not seen before. You can set the regularization $\lambda$ to a smaller value and the `maxiter` parameter to a higher number of iterations to see this for youself.


# Thorpe Scales

This section is available as a jupyter notebook in the `docs/` directory or at https://github.com/modscripps/mixsea/tree/main/docs.

Here we'll examine some of the different parameter options used for estimating the turbulent dissipation rate, $\epsilon$, from Thorpe scales. The method was first published by Stephen Thorpe– see Thorpe (1977).

**Basic idea**

Thorpe's method for estimating dissipation relies on the Ozmidov scale

\begin{equation}
L_{O}=\alpha \epsilon^{1/2} N^{-3/2},
\end{equation}

defined by dimensional analysis as the scale at which stratification begins to affect turbulence, where $\alpha=\mathcal{O}(1)$ is a proportionality constant and $N$ is a measure of the background stratification (we will see later the importance of its definition).

Thorpe's insight was to identify the Ozmidov scale with the scale $L_{T}$ (now named after him) of the largest vertical overturns (gravitational instability) observed in a density profile. He proposed sorting the unstable profile to produce a stable one, keeping track of individual water parcels vertical displacements to use their root mean square displacement $d(z)$ as a proxy for the overturn scale
\begin{equation}
L_{T} = \sqrt{\overline{d^{2}}}.
\end{equation}

Setting $L_{O} = L_{T}$ and inverting for $\epsilon$ gives
\begin{equation}
\boxed{\epsilon = C_{0} L_{T}^{2} N^{3},}
\end{equation}

where we identify $C_{0} = \alpha^{-2}$ as an $\mathcal{O}(1)$ free parameter, perhaps tuned based on microstructure measurements of $\epsilon$.

***Warning:** The Thorpe scale method is an **event-based** turbulence parameterization, so one must be wary of spatial and temporal intermittency when interpreting results.*

Thus, the Thorpe scale method in practice can be separated three steps: 1) finding the overturns, 2) determining their Thorpe scales by sorting, and 3) determining the background stratification.

## Example

First, we load modules and example data.

```python
import mixsea as mx
import numpy as np
import matplotlib.pyplot as plt

ctd = mx.helpers.read_ctd_testfile()
```

Our example data contain some NaN values, as is common with observational datasets. Below we isolate the good data by removing NaNs.

```python
notnan = np.isfinite(ctd["depth"]) & np.isfinite(ctd["t"]) & np.isfinite(ctd["SP"])

depth = ctd["depth"][notnan]
t = ctd["t"][notnan]
SP = ctd["SP"][notnan]
lon = ctd["lon"][0]
lat = ctd["lat"][0]
```

### A detailed look at the output and diagnostics

Now we run the Thorpe scale function on the good test data, selecting input parameters that are reasonable for the deep ocean.

```python
dnoise = 5e-4  # Noise parameter
alpha = 0.95  # Coefficient relating the Thorpe and Ozmidov scales.
# Background value of epsilon applied where no overturns are detected.
background_eps = np.nan
# Do not use the intermediate profile method
use_ip = False
# Critical value of the overturn ratio
Roc = 0.3

# Calculate Thorpe scales and diagnostics.
eps, N2, diag = mx.overturn.eps_overturn(
    depth,
    t,
    SP,
    lon,
    lat,
    dnoise=dnoise,
    alpha=alpha,
    Roc=Roc,
    background_eps=background_eps,
    use_ip=use_ip,
    return_diagnostics=True,
)
```

The standard output `eps` and `N2` (buoyancy frequency squared) have been quality controlled. The type and strictness of quality control can be set using arguments to `eps_overturn` (see the function docstring for details) In addition, numerous diagnostic variables are output in the dictionary `diag`. The diagnostic dictionary contains output from pre-quality control, as well as, flags which indicate the quality control state and miscellanous other variables that might be useful.

```python
for key in diag:
    print(key)
```

    eps
    N2
    Lt
    thorpe_disp
    sidx
    dens
    dens_sorted
    Ro
    SA_sorted
    CT_sorted
    noise_flag
    N2_flag
    ends_flag
    Ro_flag

Some of the diagnostics are as follows:

* `eps`: the dissipation rate estimated before applying quality control flags.
* `N2`: the buoyancy frequency squared estimated before applying quality control flags.
* `Lt`: the Thorpe scale.
* `thorpe_disp`: the Thorpe displacements.
* `sidx`: the density-sorted indices of the original profile
* `dens`: potential density (this may appear 'steppy' due to the use of different reference pressures at for different pressure ranges).
* `dens_sorted`: sorted potential density.
* `dens_ip`: density from the intermediate profile method. (Does not appear in the list above because the intermediate profile method was not used)
* `CT_ip`: conservative temperature from the intermediate profile method. (Does not appear in the list above because intermediate profile method was not used)
* `Ro`: the overturn ratio.
* `CT_sorted`: sorted conservative temperature.
* `SA_sorted`: sorted absolute salinity.

Some of the quality control flags:

* `noise_flag`: if density/CT difference from the top to bottom of the overturn is less than the noise parameter, this flag will be True.
* `N2_flag`: if buoyancy frequency is negative, this flag will be True.
* `ends_flag`: if an overturn contains the first or last point in the data, this flag will be True.
* `Ro_flag`: if the overturn ratio is less than the critical value, this flag will be True.

If we zoom into a segment, it is easier to understand what these diagnostics mean.

```python
# Plot only in the depth range:
cut = (depth > 4260) & (depth < 4500)
zc = depth[cut]

# finding unsorting indices for intuitive Thorpe displacements
#idx_unsorted = np.argsort(idx_sorted)

#axs[1].plot(thorpe_displacement[idx_unsorted][inoverturn_unsorted], z[inoverturn_unsorted])

fig, axs = plt.subplots(1, 4, sharey=True, figsize=(9, 6))
axs[0].plot(diag["dens"][cut]-1000, zc, label="density", lw=1.)
axs[0].plot(diag["dens_sorted"][cut]-1000, zc, label="sorted density", lw=1.)
axs[1].plot(diag["eps"][cut], zc, label="rejected epsilon", color="r")
axs[1].plot(eps[cut], zc, label="accepted epsilon", color="g")
axs[2].plot(diag["N2"][cut], zc, label="rejected N2", color="r")
axs[2].plot(N2[cut], zc, label="accepted N2", color="g")
axs[3].plot(diag["thorpe_disp"][cut], zc, label="Thorpe displacement")
axs[3].plot(diag["Lt"][cut], zc, label="±Thorpe scale")
axs[3].plot(-diag["Lt"][cut], zc, "C1--")
axs[0].invert_yaxis()
axs[0].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[1].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[2].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[3].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[0].set_ylabel("Depth [m]")
axs[0].set_xlabel("Potential density [kg m$^{-3}$]")
axs[1].set_xlabel(r"$\epsilon$ [W kg$^{-1}$]")
axs[2].set_xlabel(r"$N^2$ [rad$^2$ s$^{-2}$]")
axs[3].set_xlabel("Thorpe displacement [m]")

fig.align_labels()
```

Starting from the left, we have potential density and sorted potential density. Overturning patches should correspond to regions where the potential density is different from the sorted potential density. Next is the dissipation rate, $\epsilon$, plotted in green are the values that have passed quality control and in red those that were rejected by some criteria. The plot looks a bit odd because there are jumps in $\epsilon$. This is caused by small overturns adjacent to large overturns. Ultimately, a lot of small overturns will be rejected by quality control. Third from the left is buoyancy frequency, which can be calculated in a number of different ways. We will explore this in more detail later. Last is the Thorpe displacements and Thorpe length scale. Large negative values atop large positive values represent large overturns; whether this leads to large dissipation depends on the stratification.

The state (true/false) of some of the quality control flags is plotted below for the same segment of data.

```python
fig, axs = plt.subplots(
    1, 5, sharey=True, figsize=(9, 6), gridspec_kw={"width_ratios": [2, 1, 1, 1, 2]}
)
axs[0].plot(diag["thorpe_disp"][cut], zc, label="Thorpe displacement")
axs[0].plot(diag["Lt"][cut], zc, label="Thorpe scale")
axs[0].plot(-diag["Lt"][cut], zc, "C1--")
axs[1].plot(diag["noise_flag"][cut], zc, color="r")
axs[2].plot(diag["N2_flag"][cut], zc, color="r")
axs[3].plot(diag["ends_flag"][cut], zc, color="r")
axs[4].plot(diag["Ro"][cut], zc, label="Ro value")
axs[4].plot(diag["Ro_flag"][cut], zc, color="r", label="Ro flag")
axs[4].axvline(Roc, color="k", ls=":", label="Ro critical")
axs[0].invert_yaxis()
axs[0].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[4].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[0].set_ylabel("Depth [m]")
axs[0].set_xlabel("Thorpe displacement [m]")
axs[1].set_xlabel("Noise flag")
axs[2].set_xlabel("N2 flag")
axs[3].set_xlabel("Ends flag")
axs[4].set_xlabel("Ro flag and value")

fig.align_labels()
```

The noise flag is raised (true) where the density difference (top to bottom) of the overturning region is smaller than the noise parameter. The N2 flag has not been raised anywhere, indicating there are no regions of negative buoyancy frequency. The ends flag is raised for the last overturn since it contains the end point of the data. However, it is just a warning flag and not used to quality control the output. The Ro flag is raised where the overturn ratio is less than the critical value Gargett and Garner (2008). Note that for this demonstration we have set the critical overturn ratio a bit higher than would be normal.

### The intermediate profile method

The intermediate profile method attempts to reduce the influence of random noise when estimating overturns. For technical details, see Ferron et al. (1998) and Gargett and Garner (2008).

What does the method do? Here we construct three intermediate profiles from a segment of temperature data.

```python
acc = 1e-3
hinge = 0

# Restrict ourselves to:
cut = (depth < 4300) & (depth > 4250)

tup = mx.overturn.intermediate_profile(t[cut], acc, hinge, 'up')
tdo = mx.overturn.intermediate_profile(t[cut], acc, hinge, 'down')
tav = mx.overturn.intermediate_profile(t[cut], acc, hinge, 'ave')

fig, ax = plt.subplots(1, 1)
ax.invert_yaxis()
ax.plot(t[cut], depth[cut], label='data')
ax.plot(tup, depth[cut], label='up')
ax.plot(tdo, depth[cut], label='down')
ax.plot(tav, depth[cut], label='ave')
ax.legend()
ax.set_xlabel('Temperature [$^\circ$C]')
ax.set_ylabel('Depth [m]');
```

The method creates a new profile that moves in increments of the argument `acc`, which is a measure of the accuracy of the observations. A region that looks unstable in the data (such as the cold water over light in the region around 4270 m) might be smoothed out in the intermediate profile because the fluctuations are less than the accuracy. The method can be applied top down (`down`), bottom up (`up`) or the average of up and down (`ave`).

Now we investigate what the method does to estimates of $\epsilon$.

```python
dnoise = 5e-4
alpha = 0.95
# Use the intermediate profile method
use_ip = True

eps, N2, diag = mx.overturn.eps_overturn(
    depth,
    t,
    SP,
    lon,
    lat,
    dnoise=dnoise,
    alpha=alpha,
    use_ip=use_ip,
    return_diagnostics=True,
)
```

Plot the same depth range as the case that did not use the method.

```python
cut = (depth > 4260) & (depth < 4500)
zc = depth[cut]

fig, axs = plt.subplots(1, 4, sharey=True, figsize=(9, 6))
axs[0].plot(diag["dens"][cut]-1000, zc, label="density")
axs[0].plot(diag["dens_sorted"][cut]-1000, zc, label="sorted density")
axs[0].plot(diag["dens_ip"][cut]-1000, zc, label="intermediate density")
axs[1].plot(diag["eps"][cut], zc, label="rejected epsilon", color="r")
axs[1].plot(eps[cut], zc, label="accepted epsilon", color="g")
axs[2].plot(diag["N2"][cut], zc, label="rejected N2", color="r")
axs[2].plot(N2[cut], zc, label="accepted N2", color="g")
axs[3].plot(diag["thorpe_disp"][cut], zc, label="Thorpe displacement")
axs[3].plot(diag["Lt"][cut], zc, label="±Thorpe scale")
axs[3].plot(-diag["Lt"][cut], zc, "C1--")
axs[0].invert_yaxis()
axs[0].legend(loc="lower left", bbox_to_anchor=(-0.1, 1))
axs[1].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[2].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[3].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[0].set_ylabel("Depth [m]")
axs[0].set_xlabel("Potential density [kg m$^{-3}$]")
axs[1].set_xlabel(r"$\epsilon$ [W kg$^{-1}$]")
axs[2].set_xlabel(r"$N^2$ [rad$^2$ s$^{-2}$]")
axs[3].set_xlabel("Thorpe displacement [m]")

fig.align_labels()
```

Now we're using the intermediate profile method, the results have changed dramatically. Overturns are diagnosed from the intermediate profile of density, which is different to the actual density. The overturning patches may be of different sizes to those estimated without the intermediate profile. Sorting the intermediate profile results in strange looking Thorpe displacements, because the sorting algorithm struggles when many data points have an identical value. There is an argument to be made the method should not be applied to data that has already been bin averaged (as ours has) and the method may work better on raw CTD data that is more noisy.

Because the intermediate profile method fundamentally changes many aspects of the Thorpe scale method, including the size of detected overturns and the Thorpe displacements, it is not straightforward or necessarily meaningful to raise a flag for rejected overturns. Consequently, most flags are disabled when using the method and so are not shown.

### Compare methods of estimating $N^2$

There are a few different options for calculating buoyancy frequency:

```python
use_ip = False

# Choose buoyancy frequency method
N2_method = "teosp1"
eps_teosp1, N2_teosp1 = mx.overturn.eps_overturn(
    depth, t, SP, lon, lat, use_ip=use_ip, N2_method=N2_method,
)

N2_method = "teos"
eps_teos, N2_teos = mx.overturn.eps_overturn(
    depth, t, SP, lon, lat, dnoise, use_ip=use_ip, N2_method=N2_method,
)

N2_method = "bulk"
eps_bulk, N2_bulk = mx.overturn.eps_overturn(
    depth, t, SP, lon, lat, use_ip=use_ip, N2_method=N2_method,
)

N2_method = "endpt"
eps_endpt, N2_endpt = mx.overturn.eps_overturn(
    depth, t, SP, lon, lat, use_ip=use_ip, N2_method=N2_method,
)
```

A brief explanation of the methods:

* `teos`: This method estimates $N^2$ from the thermodynamic equation of state 2010 (TEOS), using the tempurature and salinity at the first and last point in the overturn.
* `toesp1`: This method is the same as above except it uses the temperature and salinity from the points immediately above and below the overturn (p1 meaning plus 1).
* `bulk`: This method comes from Smyth et al. (2001) and has the advantage of being insensitive to errors in determining the patch size. Buoyancy frequency is proportional to the root mean square density anomaly of an overturn, divided by the Thorpe scale.
* `endpt`: This simple method estimates buoyancy frequency from the potential density gradient across each overturn.

More methods may be added in future versions of `mixsea`.

```python
# Plot only in the depth range:
cut = (depth > 4225) & (depth < 4350)
zc = depth[cut]

fig, axs = plt.subplots(1, 3, sharey=True, figsize=(9, 6))
axs[0].plot(diag["dens"][cut]-1000, zc, label="density")
axs[0].plot(diag["dens_sorted"][cut]-1000, zc, label="sorted density")
axs[1].plot(eps_teosp1[cut], zc, label="teosp1")
axs[1].plot(eps_teos[cut], zc, label="teos")
axs[1].plot(eps_bulk[cut], zc, label="bulk")
axs[1].plot(eps_endpt[cut], zc, label="endpt")
axs[2].plot(N2_teosp1[cut], zc, label="teosp1")
axs[2].plot(N2_teos[cut], zc, label="teos")
axs[2].plot(N2_bulk[cut], zc, label="bulk")
axs[2].plot(N2_endpt[cut], zc, label="endpt")
axs[0].invert_yaxis()
axs[0].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[1].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[2].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[0].set_ylabel("Depth [m]")
axs[0].set_xlabel("Potential density [kg m$^{-3}$]")
axs[1].set_xlabel(r"$\epsilon$ [W kg$^{-1}$]")
axs[2].set_xlabel(r"$N^2$ [rad$^2$ s$^{-2}$]")

fig.align_labels()
```

The different methods can alter the dissipation rate estimate by almost an order of magnitude. The `endpt` and `teos` methods produce such similar results that they are indistinguishable. The `teosp1` method tends to produce higher $N^2$ estimates, because there is often a jump in stratification adjacent to an overturning patch. The `bulk` method tends to give the smallest $N^2$ estimates.

### Thorpe scales from temperature alone

In some cases it may be desirable to use temperature, rather than density, to detect unstable patches. There are two different ways of doing this:

* Provide both temperature and salinity profiles but set `overturns_from_CT = True`. In this case, overturns will be diagnosed from the conservative temperature profile. Buoyancy frequency will be estimated from the equation of state or potential density as specified by `N2_method`.

* Provide as arguments to `eps_overturn` a profile of temperature and a constant salinity. Pseudo potential density will be calculated assuming a constant salinity profile. Overturns will be diagnosed from the pseudo potential density profile. Depending on the `N2_method` chosen, buoyancy frequency may be estimated from pseudo potential density or the equation of state at constant salinity. As such, buoyancy frequency values will be incorrect in the ocean but perhaps useful for estimating $\epsilon$. It is probably a good idea to choose a plausible salinity value, such as 0 psu for a fresh water environment or 35 psu for an ocean environment.

Note that `dnoise` should describe the temperature noise level if `overturns_from_CT = True` and the density noise level at constant salinity if salinity is constant.

```python
use_ip = False
N2_method = "teos"

eps_t, N2_t = mx.overturn.eps_overturn(
    depth, t, 35, lon, lat, use_ip=use_ip, N2_method=N2_method,
)

eps_ts, N2_ts = mx.overturn.eps_overturn(
    depth,
    t,
    SP,
    lon,
    lat,
    dnoise=2e-4,
    use_ip=use_ip,
    N2_method=N2_method,
    overturns_from_CT=True,
)

eps, N2 = mx.overturn.eps_overturn(
    depth, t, SP, lon, lat, dnoise, use_ip=use_ip, N2_method=N2_method,
)

fig, axs = plt.subplots(1, 2, sharey=True, figsize=(6, 6))
axs[0].plot(eps_ts[cut], zc, label="CT")
axs[0].plot(eps_t[cut], zc, label="pseudo potential density")
axs[0].plot(eps[cut], zc, label="potential density (normal case)")
axs[1].plot(N2_ts[cut], zc)
axs[1].plot(N2_t[cut], zc)
axs[1].plot(N2[cut], zc)
axs[0].invert_yaxis()
axs[0].legend(loc="lower left", bbox_to_anchor=(0, 1))
axs[0].set_ylabel("Depth [m]")
axs[0].set_xlabel(r"$\epsilon$ [W kg$^{-1}$]")
axs[1].set_xlabel(r"$N^2$ [rad$^2$ s$^{-2}$]")

fig.align_labels()
```

The smaller noise parameter in the `overturns_from_CT` case means that more overturns are found, however, $\epsilon$ tends to be a bit lower than the normal case.


# Numerical Recipes Workshop 3
For the week of 7-11 October, 2019

This notebook will provide a practical example of root finding for a nonlinear equation.

## The temperature of interstellar dust grains

Understanding the nature of interstellar dust grains is vital to many areas of astronomy, from star formation to measuring the cosmic microwave background (CMB). Many observed properties of interstellar dust are derived from knowing its temperature. In general, dust is well mixed with the gas in the interstellar medium (ISM), but the two are rarely at the same temperature.

The timescales for dust-related processes are very short, so the dust temperature can be calculated by assuming it is in thermal equilibrium at all times. Then, one only needs to balance the various heating and cooling processes, i.e., to find the root of the energy loss equation:

$
\begin{align}
\large
\frac{de}{dt} = \Gamma(T_{dust}) - \Lambda(T_{dust}),
\end{align}
$

where $\Gamma$ and $\Lambda$ are the dust heating and cooling rates, respectively.

Including the relevant heating and cooling processes, this becomes

$
\begin{align}
\large
\frac{de}{dt} = 4 \sigma T_{CMB}^{4} \kappa_{gr} +
\Gamma_{isrf} + \Lambda_{gas/grain}(T_{dust}, T_{gas}, n_{H}) - 4 \sigma T_{dust}^{4} \kappa_{gr},
\end{align}
$

where $\sigma$ is the Stefan-Boltzmann constant, $T_{CMB}$ is the temperature of the CMB, $\kappa_{gr}$ is the dust opacity, $\Gamma_{isrf}$ is the heating from the instellar radiation field, and $\Lambda_{gas/grain}$ is the rate of heat exchange via collisions between the gas and dust. The first term represents heating from the CMB, the second is heating from nearby stars, the third term transfers heat from the hotter to the cooler matter, and the final term is the cooling of the dust by thermal radiation.

The opacity of the dust can be approximated by the piece-wise power-law:

$
\begin{align}
\large
\kappa_{gr}(T_{dust}) \propto \left\{ \begin{array}{ll}
T_{dust}^{2} & , T_{dust} < 200 K,\\
\textrm{constant} & , 200\ K < T_{dust} < 1500\ K,\\
T_{dust}^{-12} & , T_{dust} > 1500\ K.
\end{array} \right.
\end{align}
$

The gas/grain heat transfer rate is given by:

$
\begin{align}
\large
\Lambda_{gas/grain} = 7.2\times10^{-8} n_{H}
\left(\frac{T_{gas}}{1000 K}\right)^{\frac{1}{2}} (1 - 0.8 e^{-75/T_{gas}}) (T_{gas} - T_{dust})\ [erg/s/g],
\end{align}
$

where $n_{H}$ is the number density of the gas.

## Calculating dust temperatures with root finding

The above equations have been coded below with the full heat balance equation implemented as the `gamma_grain` function. Do `help(gamma_grain)` to see how it can be called.

Assuming a constant gas temperature, $T_{gas}$ and gas density, $n_{H}$, calculate the dust temperature, $T_{dust}$, using bisection, the secand method, and the Scipy implementation of [Brent's method](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.brentq.html#scipy.optimize.brentq). Implement your own bisection and secant methods and count the number of steps to reach a solution.

```python
import numpy as np

mh = 1.673735e-24 # g
# Stefan-Boltzmann constant
sigma_b = 5.670373e-5 # erg cm^−2 s^−1 K^−4

def gas_grain(Tgas):
    """
    Return gas/grain heat transfer rate coefficient.
    """

    grain_coef = 1.2e-31 * 1.0e3**-0.5 / mh
    gasgra = grain_coef * Tgas**0.5 * \
        (1.0 - (0.8 * np.exp(-75.0 / Tgas)))
    return gasgra

def kappa_grain(Tdust):
    """
    Return grain mean opacity.
    """

    kgr1   = 4.0e-4
    kgr200 = 16.0
    T_subl = 1500.

    Tdust = np.asarray(Tdust)
    kgr = np.zeros(Tdust.size)

    f1 = Tdust < 200
    if f1.any():
        kgr[f1] = kgr1 * Tdust[f1]**2

    kgr[(Tdust >= 200) & (Tdust < T_subl)] = kgr200

    f2 = Tdust >= T_subl
    if f2.any():
        kgr[f2] = kgr200 * (Tdust[f2] / T_subl)**-12

    return kgr

def gamma_isrf():
    """
    Interstellar radiation field heating rate coefficient.
    """

    return 4.154682e-22 / mh

def gamma_grain(Tdust, Tgas, nh, isrf=1.7, z=0):
    """
    Return the grain heating rate.

    Parameters
    ----------

    Tdust : float
        dust temperature in K
    Tgas : float
        gas temperature in K
    nh : float
        Hydrogen number density in cm^-3
    isrf : float, optional
        interstellar radiation field strengh in Habing units
        default: 1.7 (typical for local interstellar medium)
    z : float, optional
        current redshift, used to set the temperature of the
        Cosmic Microwave Background.
        default: 0
    """

    TCMB = 2.73 * (1 + z)
    my_isrf = isrf * gamma_isrf()

    return my_isrf + \
        4 * sigma_b * kappa_grain(Tdust) * (TCMB**4 - Tdust**4) + \
        (gas_grain(Tgas) * nh * (Tgas - Tdust))
```

```python
### Tgas and nH values
Tgas = 100 # K
nH = 1e3 # cm^-3
```

### Bisection
See if you can implement the bisection method to calculate $T_{dust}$ for a relative tolerance of $10^{-4}$, where the relative tolerance is given by:

$
\begin{align}
rtol = \left|\frac{val_{new} - val_{old}}{val_{old}}\right|.
\end{align}
$

A sensible initial bound is $[T_{CMB}, T_{gas}]$, where $T_{CMB} = 2.73 K$ in the local Universe.

```python
def bisection(low, high, tol):

    while (np.abs(high - low)) >= tol:
        midpoint = (high + low) / 2.0
        above = gamma_grain(high, Tgas, nH) * gamma_grain(midpoint, Tgas, nH)
        below = gamma_grain(midpoint, Tgas, nH) * gamma_grain(low, Tgas, nH)
        if above < 0:
            low = midpoint
        elif below < 0:
            high = midpoint

    return midpoint

answer = bisection(2.73, Tgas, 1e-4)
print(answer)
```

    40.85661255836487

### Secant Method

See if you can implement the secant method for the same tolerance and initial guesses.

```python
def secant(high, low, tol):
    x_
    while(np.abs(high - low)) >= tol:

```

### Brent's Method

Use [Brent's method](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.brentq.html#scipy.optimize.brentq) to calculate $T_{dust}$. After that, try calculating $T_{dust}$ for a range of $n_{H}$ from $1\ cm^{-3}$ to $10^{13} cm^{-3}$ and plotting $T_{dust}$ vs. $n_{H}$.

```python
import scipy.optimize as opt
```

```python

```

```python
# Try a range of nH values.
nH = np.logspace(0, 13, 100)
```

```python

```

```python
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['font.size'] = 14
```

```python
# plot Tdust vs. nH
plt.semilogx(nH, Tdust)
plt.xlabel('$n_{H}$ $cm^{-3}$')
plt.ylabel('$T_{dust}$ $K$')
```


```python
import numpy as np
import sympy as sy
from sympy import I, exp
```

```python
z = sy.symbols("z", real=False)
a,h,r1,s0,s1 = sy.symbols("a,h,r1,s0,s1")
```

```python
pd = 0.76*np.exp(1j*0.28)
print np.conjugate(pd)
A2ndorder = (z-pd)*(z-np.conjugate(pd))
print sy.collect(sy.simplify(sy.expand(A2ndorder)), z)
Acl = A2ndorder*(z-a)
print sy.collect(sy.simplify(sy.expand(Acl)), z)
```

    (0.730402133116-0.210030292909j)
    z**2 - 1.46080426623237*z + 0.5776
    -0.5776*a + z**3 + z**2*(-a - 1.46080426623237) + z*(1.46080426623237*a + 0.5776)

```python
Acp = sy.poly(Acl, z)
Ap = sy.poly((z-1)*(z-0.76), z)
Bp = sy.poly(-.10*z + 0.14, z)
Rp = sy.poly(z+r1, z)
Sp = sy.poly(s0*z + s1, z)
dioph=(Ap*Rp+Bp*Sp-Acp).all_coeffs()

```

```python
sol=sy.solve(dioph, (r1,s0,s1))
print sol[r1]
sol[r1]-1.76-0.10*sol[s1]
```

    -1.92372666904173*a + 1.2932173366584

    -2.55546400366438*a + 0.235249605130106

```python

```

```python
np.exp(-0.15)
```

    0.86070797642505781

```python
argp=np.sqrt(3)*0.15
argp*180/np.pi
```

    14.885880176388383

```python
0.4/(2*np.sqrt(2))
```

    0.1414213562373095

```python
np.exp(-0.28)
```

    0.75578374145572547

```python
0.28*180.0/np.pi
```

    16.04281826366305

```python
hh=0.14
-1+hh+np.exp(-2*hh)
```

    -0.10421625854427452

```python
1-(1+hh)*np.exp(-2*hh)
```

    0.13840653474047282

```python
np.exp(-2*hh)
```

    0.75578374145572547

```python
sy.expand()
```


# Variational Inference

### 1. What's variational inference?

Variational inference is an approximate technique, which is widely used in inference problem in modern statistic. As we all know, inferene problem is especially important in Bayesian statistics, which frames all inference about unknown quntities as a calculation about the posterior. In facr, posterior is usually intractable in modern Bayesian statistics. In order to sovle this kind of problem, Monte Carlo Markov Chain(MCMC) and variational inference are two main methods. MCMC is a technique of sampling, for example, Metropli-Hast algorithm and gibbs sampling, and variational inference(VI) turns inference into optimization problems.

The core idea of VI is to posit a family of distribution and then to find the member of that family which is close to the target, where closeness is measured using the Kullback-Leibler divergence.

#### 1.1 Core idea of variational inference

Suppose observed data $X$ and its latent variables $Z$, the prior of latent variables is $p(z)$ and the data likelihood is $p(x\vert z)$, in many situation, we want to inference latent variables $Z$ through posterior $p(z\vert x)$. In fact, the posterior is intractble due to the normalization term, where we have to integration over all latent variable $Z$.

In this case, MCMC is usually adopted as an approximate computation for posterior inference. However in many situations, like when large data sets are involved or if the model is too complex, more faster approximate techniques is necessary, and VI is another strong alternative.

As in EM, we start by writing the full-data likelihood:

$$
\begin{align*}
p(x,z) = p(z)p(x\vert z)\\
p(z\vert x) = \frac{p(x,z)}{p(x)}
\end{align*}
$$

The difference with EM is that we view $z$ as parameters, not just specific parameters such as cluster membership or missing data.

Thus, we posit a family of approximate distribution $D$ over latent variables. We then try to find the member of family that minimizes the Kullback-Leibler divergence to the true posterior. This turns inference problem to optimization algorithm.

$$
q^{*}(z) = \arg \min_{q(z)\in D} D_{KL}(q(z)\vert\vert p(z\vert x))
$$

The figure below shows the core idea of VI: to approximate posterior $p(z\vert x)$ with $q(z)$. We optimize $q(z)$ for minimal value of KL divergence.

**Comparision of MCMC and VI**

| MCMC | VI  |
| :----------: | :---: |
| More computationally intensive | Less intensive |
| Gaurantess producing asymptotically exact samples from the target distribution | No such gaurantees |
| Slower | Faster, expecially for large data sets and complex distributions |
| Best for precise inference | Useful to explore many scenarios quickly or large data sets |

#### 1.2 How does variational inference work?

The goal of the VI is:

$$
\begin{equation}
q^{*}(z) = \arg \min_{q(z)\in D}D_{KL}(q(z)\vert \vert p(z\vert x))
\end{equation}
$$

In fact, to minimize the $D_{KL}(q(z)\vert \vert p(z\vert x))$ is equivalent to maximize ELBO.

$$
\begin{aligned}
D_{KL} &= \mathbb{E}_{q(z)}[\log q(z)] -\mathbb{E}_{q(z)}[\log p(z\vert x)] \\
&= \mathbb{E}_{q(z)}[\log q(z)]-\mathbb{E}_{q(z)}[\log p(x,z)] + \mathbb{E}_{q(z)}[\log p(x)] \\
\Rightarrow \log p(x) &= \mathbb{E}_{q(z)}[\log p(x,z)] -\mathbb{E}_{q(z)}[\log q(z)] + D_{KL}(q(z)\vert \vert p(z\vert x))  \\
\log p(x) &= \mathrm{ELBO}(q)+ D_{KL}(q(z)\vert \vert p(z\vert x))
\end{aligned}
$$

where $\mathrm{ELBO}(q)= \mathbb{E}_{q(z)}[\log p(x,z)] -\mathbb{E}_{q(z)}[\log q(z)]$, which is called evidence lower bound, and $\mathbb{E}_{q(z)}[\log p(x)] =\log p(x)$  because $p(x)$ is independent with $q(z)$.

The figure is like the one in EM, but the difference with EM is the likelihood $\ln p(X\vert\theta)$ is fixed. The goal of VI is to minimize the KL divergence of $q$ and $p$, thus it's equivalent to maximize the ELBO $\mathcal{L}(q,\theta)$.

Therefore, we can turn our goal to an optimization problem: maximize $\mathrm{ELBO}$ is equivalent to minimize the DL divergence. Our objective function is $\mathrm{ELBO}$:

$$
\begin{align}
\mathrm{ELBO} &= \mathbb{E}_{q(z)}[\log p(x,z)] - \mathbb{E}_{q(z)}[\log q(z)] \\
&= \mathbb{E}_{q(z)}[\log p(x\vert z)]+\mathbb{E}_{q(z)}[\log p(z)]-\mathbb{E}_{q(z)}[\log q(z)] \\
&= \mathbb{E}_{q(z)}[\log p(x\vert z)]-D_{KL}(q(z)\vert p(z))
\end{align}
$$

1. The first term is the expectation of the data likelihood and thus $\mathrm{ELBO}$ encourage distributions put their mass on configurations of latent variables that explain observed data.
2. The second term is the negative KL divergence between the variational distribution and the prior, so the $\mathrm{ELBO}$ force $q(z)$ to close to the prior $p(z)$.

**Hence, maximize** $\mathrm{ELBO}$ ** means to balance the likelihood and prior.**

### 2. Mean-field Theory

In the section 1, we know the core idea of VI and know the goal of VI is to maximize the $\mathrm{ELBO}$. The sucessive question is how to choose the proposed family of distribution. Intuitively, the complexity of family of distributions we choose directly determine the complexity of the optimization problem.

**The more flexibility in the family of distributions, the closer the approximation and the harder the optimization.**

The priciple that we choose the family of distribution is **mean-field theory**.  **What's the mean-field theory?** the latent variables are mutually independent and each governed by a distinct factor in the variational distribution. A generic member of the mean-field variational family is given by the below equation-

$$
\begin{equation}
q(z) = \prod_{j=1}^{m}q(z_{j})
\end{equation}
$$

the latent variable in mean-field theory is mutually independent, so it cannot capture the correlation in the original space. Once the latent variable of the posterior is dependent, the mean-field approximate will be affected. The example is below:

**Notice that we are not making any comment about the conditional independence or lack thereof of posterior parameters. We are merely saying we will find new functions of these parameters such that we can multiply these functions to construct an approximate posterior.**

### 3. CAVI Algorithm

Now, we have goal(maximize $\mathrm{ELBO}$) and choose mean-field family of distribution $q(z)=\prod_{i=1}^{M} q(z_{i})$. **The next question is how to maximize $\mathrm{ELBO}$?** At this time, coordinate ascent variational inference(CAVI) algorithm is introduced.

The figure below is the process of CAVI algorithm:

First of all, we compute the optimal distribution $q_{i}$:

$$
\begin{align}
q &= \mathbb{E}[p(\cdot)] \\
&= exp(\log \mathbb{E}[p(\cdot)]) \\
&\approx exp(\log \mathbb{E}[p(\cdot)]-Var(p(\cdot))/(2\ast \mathbb{E}[p(\cdot)]^{2}) \\
&= exp(\log \mathbb{E}[p(\cdot)])\ast exp(h(p(\cdot))) \\
&< exp(\log \mathbb{E}[p(\cdot)])
\end{align}
$$

Apply second order Taylor expansion, we have $q=exp(\log \mathbb{E}[p(\cdot)])\ast exp(h(p(\cdot)))$, and $q^{\ast}_{j}(z_{j})=\mathbb{E}[p(z_{j}\vert z_{-j}, x)]$, hence $q_{j}$ is propotional to the log conditional expectation:

$$
\begin{equation}
q_{j}^{\ast}(z_{j}) \propto exp(\mathbb{E}_{-j}[\log p(z_{j}\vert z_{-j},x)])
\end{equation}
$$

Due to that the latent variable are multually independent, the RHS can be rewritten as:

$$
\begin{align}
\mathbb{E}_{-j}[\log p(z_{j}\vert z_{-j}, x)] &= \mathbb{E}_{-j}[\log \frac{p(z_{j}, z_{-j}\vert x)}{p(z_{-j})}] \\
&= \mathbb{E}_{-j}[\log p(z_{j}, z_{-j}\vert x)]-E_{-j}[\log p(z_{-j})] \\
&= \mathbb{E}_{-j}[\log p(z_{j}, z_{-j}, x)] - \mathbb{E}[\log p(z_{-j})]-E_{-j}[\log p(x)] \\
&= \mathbb{E}_{-j}[\log p(z_{j}, z_{-j}, z)-const]
\end{align}
$$

Hence, we can conclude that once we know the log expectation of joint distribution, we obatain the optimal variational distribution $q_{j}^{\ast}$:

$$
\begin{align}
q_{j}^{\ast} &\propto exp(\mathbb{E}[\log p(z_{j}, z_{-j}, x)]-const) \\
q_{j}^{\ast}(z_{j}) &\propto exp(\mathbb{E}_{-j}[\log p(z_{j}, z_{-j}, x)])
\end{align}
$$

最后，在平均场变分分布族的假设下，ELBO可以被分解为对每一个隐变量zj的函数。根据隐变量分解后的ELBO中，利用q分布的平均场性质，第一项将联合概率的期望迭代求出，第二项分解了变分分布的期望。当我们最大化qj时，也就最大化了分解后的ELBO。

$$
\begin{align}
\mathrm{ELBO}(q) &= \mathbb{E}_{q(z)}[\log p(z,x)] - \mathbb{E}_{q(z)}[\log q(z)] \\
\mathrm{ELBO}(q_{j}) &= \mathbb{E}_{j}[\mathbb{E}_{-j}[p(z_{j},, z_{-j}, x)]]- \mathbb{E}_{j}[\log q_{j}(z_{j})] + const
\end{align}
$$

### 4. Variational inference and GMM

In this section, CAVI algorithm is used for Mixture of Gaussians model(GMM). It will be helpful to understand how CAVI works.

####  4.1 Joint distribution computation

Given observed data $X=(x_{1},...,x_{n})$ from $K$ independent gaussian distribution with mean $\mu_{k}$. One-hot vector $c_{i}\in \mathbb{R}^{k}$ indicate the distribution to which each data belong. The hyperparameter $\sigma^{2}$ is fixed. latent variables are $\mu, c$. The prior is:

$$
\begin{align}
\mu_{k} \sim \mathcal{N}(0, \sigma^{2}) \\
c_{i} \sim Categorical(\frac{1}{K},...,\frac{1}{K}) \\
x_{i} \sim \mathcal{N}(c_{i}^{T}, \mu, 1)
\end{align}
$$

According to bay theroem, we can compute the joint distribution:

$$
\begin{align}
p(\mu, c,x) &= p(\mu)p(c,x\vert \mu) \\
&= p(\mu)p(c)p(x\vert c, \mu) \\
&=p(\mu)\prod_{i=1}^{n}p(c_{i})p(x_{i}\vert c_{i}, \mu)
\end{align}
$$

once we have joint distribution, we can compute marginal distribution. However, the formulation has no analytical solution, and the computational complexity is $\mathcal{O}(K^{n})$.

$$
\begin{align}
p(x) &= \int \sum_{c}p(\mu, c, x)\mathrm{d}\mu \\
&= \int p(\mu)\prod_{i=1}^{n}\sum_{c_{i}}p(c_{i})p(x_{i}\vert c_{i}, \mu)\mathrm{d}\mu
\end{align}
$$

#### 4.2 GMM and CAVI

Now, we should compute variational ditribution $q(z)$, where $m=(m_{1},...,m_{k}), s^{2}=(s_{1}^{2},...,s_{K}^{2}), \phi=(\phi_{1},...,\phi_{n})$ are variational parameters, hence the formulation of variational distribution is:

$$
\begin{align}
q(z) &= q(\mu,c) \\
&= \prod_{k=1}^{K}q(\mu_{k};m_{k},s_{k}^{2})\cdot \prod_{i=1}^{n}q(c_{i};\phi_{i})
\end{align}
$$

- 1. we can obtain the formulation $\mathrm{ELBO}$, which is a function of $m,s^{2},\phi$.

$$
\begin{align}
\mathrm{ELBO}(q) &= \mathrm{ELBO}(m, s^{2}, \phi) \\
&= \mathbb{E}_{q(z)}[\log p(x\vert z)] + \mathbb{E}_{q(z)}[\log p(z)]-\mathbb{E}_{q(z)}[\log q(z)] \\
&= \sum_{i=1}^{n} \mathbb{E}[\log p(x_{i}\vert c_{i}, \mu;\phi_{i}, m, s^{2})] +(\sum_{k=1}^{K}\mathbb{E}[\log p(\mu_{k};m_{k},s^{2}_{k})]+\sum_{i=1}^{n}\mathbb{E}[\log p(c_{i};\phi_{i})]) \\
&- (\sum_{k=1}^{K}\mathbb{E}[\log q(\mu_{k};m_{k}, s_{k}^{2})]+\sum_{i=1}^{n}\mathbb{E}[\log q(c_{i};\phi_{i}])
\end{align}
$$

- 2. from section 3, we obtain how CAVI algorithm update latent variables. Now, we applyied it into GMM to compute **cluster indicator** $c$ and update $c$, noted $\mu$ is fixed:

    $$
    \begin{align}
        q^{\ast}_{j} &\propto exp(\mathbb{E}_{\mu}[\log p(c_{i}, \mu, x_{i})]) \\
        &\propto exp(\mathbb{E}[\log p(x_{i}\vert c_{i}, \mu)\cdot \log p(c_{i},\mu)]) \\
        &\propto exp(\mathbb{E}_{\mu}[\log p(x_{i}\vert c_{i}, \mu)]+\mathbb{E}_{\mu}[\log p(c_{i}, \mu)]) \\
        &\propto exp(\mathbb{E}_{\mu}[\log p(c_{i}\vert c_{i},\mu)]+\log p(c_{i}))
    \end{align}
    $$

the second term $\log p(c_{i})$ is log prior and it is a constant. Hence, we pay our attention to the first term: the distribution of $c_{i}$ gaussian distribution. In detail, we simplify it due to $c_{i}=(c_{i1},...,c_{ik})$ is one-hot vector, and we have:

$$
\begin{align}
\mathbb{E}[\log p(x_{i}\vert c_{i},\mu)] &= \sum_{k}c_{ik}\mathbb{E}_{\mu_{k}}[\log p(x_{i}\vert \mu_{k})] \\
&= \sum_{k}c_{ik}\mathbb{E}_{\mu_{k}}[-\frac{(x-\mu_{k})^{2}}{2}]+const \\
&= \sum_{k}c_{ik}(\mathbb{E}_{\mu_{k}}[\mu_{k}]x_{i}+\mathbb{E}_{\mu_{k}}[\mu_{k}^{2}]/2) + const
\end{align}
$$

from the formulation above, $\mathbb{E}[\mu_{k}]$ and $\mathbb{E}[\mu_{k}^{2}]$ can be computed. For each data point $i$, parameter $\phi_{ik}$ in the $k$th component of the latent variable $c$. The updated formulation is:

$$
\begin{equation}
\phi_{ik}\propto exp(\mathbb{E}[\mu_{k}]x_{i}+\mathbb{E}[\mu_{k}^{2}]/2)
\end{equation}
$$

smiliarly, we can compute latent variable $\mu$ of GMM. Firstly, we should calculate the optimal variational distribution $q(\mu_{k})$, and the update the parameter $m_{k}, s_{k}^{2}$ of $\mu_{k}$:

$$
\begin{align}
m_{k} &= \frac{\sum_{i}\phi_{ik}\cdot x_{i}}{1/\sigma^{2}+\sum_{i}\phi_{ik}} \\
s_{k}^{2} &= \frac{1}{1/\sigma^{2}+\sum_{i}\phi_{ik}}
\end{align}
$$

The algorithm of GMM and CAVI is below:

## Code of GMM and CAVI

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
import matplotlib.cm as cm
import matplotlib as mpl
import seaborn as sns
sns.set_style('whitegrid')
sns.set_context('poster')
%matplotlib inline
```

```python
import math
from scipy.stats import multivariate_normal as mv
```

```python
# hyperparameters
prior_std = 10

# true parameters
K =  3
mu = []
random.seed(1032)
for i in range(K):
    mu.append(np.random.normal(0, prior_std))

var = 1
var_arr = [1, 1, 1]

color = ['blue', 'green', 'red', 'magenta', 'yellow']

print('the mean of these three cluster:', mu)
```

    the mean of these three cluster: [-1.2119435442168178, -8.478964340090602, -1.3992471444357935]

```python
n = 1000

# simulated data
mixture_selection_data = np.identity(K)
print('the shape of mixture_selection_data:',mixture_selection_data.shape)
N = np.random.choice(K, n)
print('the shaoe of N:', N.shape)
```

    the shape of mixture_selection_data: (3, 3)
    the shaoe of N: (1000,)

```python
from collections import defaultdict
data = []
data_resp = defaultdict(list)

plt.figure(1, figsize=(10,8))
for i in range(n):
    # pick category
    mean = np.dot(mixture_selection_data[N[i]], mu)
    cov = np.dot(mixture_selection_data[N[i]], var_arr) * var

    # get the point from distribution
    x  = np.random.normal(mean, cov)
    data.append(x)
    data_resp[N[i]].append(x)

for k in range(K):
    plt.hist(data_resp[k], bins=30, color=color[k], histtype='step', lw=2)
plt.hist(data, color='k', bins=90, alpha=0.1);
```

```python
def VI(K, prior_std, n, data):
    # initialization
    mu_mean = []
    mu_var = []
    for i in range(K):
        mu_mean.append(np.random.normal(0, prior_std))
        mu_var.append(abs(np.random.normal(0, prior_std)))
    c_est = np.zeros((n, K))
    for i in range(n):
        c_est[i, np.random.choice(K)] = 1
    while(True):
        mu_mean_old = mu_mean[:]

        # mixture model parameters update step
        for j in range(K):
            nr = 0
            dr = 0
            for i in range(n):
                nr += c_est[i, j] * data[i]
                dr += c_est[i, j]
            mu_mean[j] = nr/((1/prior_std**2) + dr)
            mu_var[j] = 1.0/((1/prior_std**2) + dr)

        # categorical vector update step
        for i in range(n):
            cat_vec = []
            for j in range(K):
                cat_vec.append(math.exp(mu_mean[j]*data[i]-(mu_var[j] + mu_mean[j]**2)/2))
            for k in range(K):
                c_est[i, k] = cat_vec[k]/np.sum(np.array(cat_vec))

        if (np.sum((np.array(mu_mean_old)-np.array(mu_mean))**2)<0.000001):
            break

    # compute ELBO or check for convergence components don't maintain any ordering
    mixture_components = list(zip(mu_mean,mu_var))
    mixture_components.sort()
    return mixture_components, c_est
```

```python
# Run the CAVI algorithm
mixture_components, c_est = VI(K, prior_std, n, data)

plt.figure(figsize=(10,8))
for k in range(K):
    plt.hist(data_resp[k], bins=30, color=color[k], alpha=0.2, normed=True);
for k in range(K):
    vals = np.random.normal(mixture_components[k][0], 1, size=1000);
    sns.kdeplot(vals, color='k');
```

```python

```


# DIGITAL IMAGE PROCESSING IN THE SPACE-FREQUENCY DOMAIN

## THE ANISOTROPIC QUALITY INDEX (AQI)

**@author: Salvador Gabarda**

**salvador@optica.csic.es**

This notebook includes Python functions for the Anisotropic Quality Index (AQI), based on the calculation of the anisotropy of images in the space/sepace-frequency domain supported by the Wigner distribution. Details and examples of this image quality assessment measure may be found in

S. Gabarda, G. Cristóbal, "Blind image quality assessment through anisotropy", JOSA A, 2007

https://scholar.google.es/citations?user=jPzi3LEAAAAJ&hl=es&oi=ao

*Created on Dec 05 2017, reviewed Dec 27 2019*

### 1. Definition of image entropy in the spatial/spatial-frequency domain

Given a discrete random variable $X=\lbrace x_1,x_2,...,x_I \rbrace$, the Shannon entropy of such variable can be computed as

\begin{equation}
H(X)=-\sum\limits_{i=1}^I P(x_i) \log P(x_i)
\end{equation}
(1)

where $P(x_i)$ represents the *probability* of event $x_i$.
The Rényi entropy is given by

\begin{equation}
R_\alpha (X)=\frac{1}{1- \alpha} \log_2 \left( \sum\limits_{i=1}^I P^\alpha(x_i) \right)
\end{equation}
(2)

where $ \alpha $ is a real-positive number.

In this application, the Rényi entropy is calculated by means of Eq. (2) through a Wigner Distribution

\begin{equation}
W(n,k)=2 \sum\limits_{m=- \frac{N}{2}}^{\frac{N}{2}-1} z(n+m)z^*(n-m) e^{-i2 \pi k \left(\frac{2m}{N} \right) }
\end{equation}
(3)

In Eq. (3), $z(n)$ represents the gray values of pixel $n$ in a given image $z$. Variables $n$ and $k$ represent the spatial and spatial-frequency discrete variables respectively, and $m$ is a shifting parameter, which is also discrete.

the PWD given by Eq.(3) is normalized by applying the following normalizing rule, as a previous step to the Rényi entropy  calculation:

\begin{equation}
\widetilde{W}(n,k)=\frac{W(n,k)W^*(n,k)}{\sum\limits_{k} \left( W(n,k)W^*(n,k) \right) }=\frac{W^2(n,k)}{\sum\limits_{k}W^2(n,k)}
\end{equation}

In this application, the general case of the Rényi entropy given by Eq. (2), is particularized by identifying $P$ with the normalized PWD in Eq.  (3) to yield

\begin{equation}
R(n)=-\frac{1}{2} \log_2 \left( \sum\limits_{k} \widetilde{W}^3 (n,k) \right)
\end{equation}

Here $\alpha=3$ is normally selected for reducing the interference of cross-terms.

All the above mathematical support may be implemented by the following code. Some convolution operations have been introduced for speeding up the algorithm.

### 2. The code

```python
# import modules
import numpy as np
#import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from scipy.fftpack import fft, ifft
import math
from scipy import signal

def oriented_pattern(seq_length,angle):
    """
    This function originates a pattern that is later used for the orientation of the operational pseudo-Wigner distribution
    computation window,     seq_length: sequence length in pixels,    angle: orientation in degrees.
    """
    angle = np.mod(angle,180)
    # correction to set axes in the image (x: rows, y: columns) to observer cartesian coordinates x,y
    angle = np.mod((angle+90),180)
    angle =math.radians(angle)
    pi = math.pi
    h = int((seq_length/2))
    values = np.r_[float(-h):float(h+1)]
    new_positions = np.zeros([2*h+1, 2])
    for position in range(seq_length):
        if angle >= 0 and angle < pi/4:
            new_positions[position,0] = values[position]+h
            new_positions[position,1] = values[position]*math.tan(angle)+h
        elif angle >= pi/4 and angle < pi/2:
            new_positions[position,0] = values[position]*math.tan(pi/2-angle)+h
            new_positions[position,1] = values[position]+h
        elif angle >= pi/2 and angle < 3*pi/4:
            new_positions[position,0] = values[position]*math.tan(pi/2-angle)+h
            new_positions[position,1] = values[position]+h
        elif angle >= 3*pi/4 and angle <= pi:
            new_positions[position,0] = 1*values[position]+h
            new_positions[position,1] = values[position]*math.tan(angle)+h
        new_points = np.round_(new_positions)
    return new_points.astype(int)

def image_arrangement (test_image,seq_length,angle,indices):
    """
    arrangement operation for time reduction
    """
    rows = test_image.shape[0]
    columns = test_image.shape[1]
    layers = np.zeros(((seq_length,rows,columns)))
    for k in range(seq_length):
        mask = np.zeros((seq_length,seq_length))
        mask[indices[k,0],indices[k,1]] = 1
        layers[k,:,:] = signal.convolve2d(test_image, mask, boundary='symm', mode='same')
    return layers

def layer_product_function(layers,seq_length):
    """
    product function of the Wigner distribution
    """
    faces = layers.shape[0]
    rows =layers.shape[1]
    columns = layers.shape[2]
    layer_product = np.ones(((seq_length-1,rows,columns)))
    layers[faces-1,:,:]=layers[0,:,:]
    for i in range(faces-1):
        layer_product[i,:,:]= layers[i,:,:]*layers[faces-1-i]
    return layer_product

def layer_wigner_distribution(test_image,seq_length,angle):
    """
    Wigner distribution of test_image, seq_lengthe: odd number of pixels, e.g.: 9, angle: degrees, e.g.: 45
    """
    indices = oriented_pattern(seq_length,angle)
    layers = image_arrangement (test_image,seq_length,angle,indices)
    layer_product = layer_product_function(layers,seq_length)
    distribution = fft(layer_product, axis = 0)
    distribution = np.real(distribution)
    # set zero frequency in central position
    Head = distribution[int(seq_length/2):seq_length,:,:]
    Tail = distribution[0:int(seq_length/2),:,:]
    distribution = np.append(Head,Tail, axis = 0)
    return distribution

def renyi_entropy(distribution,order):
    """
    This function calculates the Rényi entropy of an image based on its pseudo-Wigner distribution (distribution).
    The "order" variabe represents the exponential order of the Rényi entropy (3 is the most common value)
    """
    eps = np.finfo(float).eps
    rows = distribution.shape[1]
    columns = distribution.shape[2]
    layers = distribution.shape[0]
    squared_wl = np.ones([layers,rows,columns])
    for layer in range(layers):
        # square distribution local values
        working_layer = distribution[layer,:,:]
        squared_wl[layer,:,:] = np.multiply(working_layer,working_layer)
    squared_wd = squared_wl
    # sum squared wigner distribution along coordinate 1
    sum_sq_wd = np.sum(squared_wd, axis = 0)
    # normalize squared values
    normalized_distribution =np.zeros([layers,rows,columns])
    for layer in range(layers):
        normalized_distribution[layer,:,:] = np.divide(squared_wd[layer,:,:],sum_sq_wd+eps)
    # raise elements to the power defined by input variable "order"
    power_nor_dis = np.power(normalized_distribution,order)
    # sum pixelwise
    entropy_1 = np.sum(power_nor_dis, axis = 0)+eps
    # pixelwise entropy
    entropy_2 =np.log2(entropy_1)
    entropy =(1/(1-order))*entropy_2
    super_threshold_indices = entropy < 0
    entropy[super_threshold_indices] = 0
    entropy = np.nan_to_num(entropy)
    # normalize entropy
    entropy = entropy*(1/np.log2(layers))
    return entropy

def show_wigner_frequencies(distribution):
    """
    Starting from the pseudo-Wigner distribution (distribution) of the input test image, this function gives a visualization
    of the frequency components of such distribution and images are saved in pdf's
    """
    rows = distribution.shape[1]
    columns = distribution.shape[2]
    layers = distribution.shape[0]
    frequencies = np.zeros([layers,rows,columns])
    for layer in range(layers):
        frequency = distribution[layer,:,:]
        min_val =np.amin(frequency)
        frequency = frequency - min_val
        max_val = np.amax(frequency)
        frequency = (1/max_val)*frequency
        #plt.figure()
        frequency = np.uint8(255*frequency)
        #plt.imshow(frequency, interpolation='nearest',cmap='gray')
        #plt.imshow(frequency,cmap='gray')
        name = "wigner_distribution_" + str(layer) + ".pdf"
        msg = "Wigner distribution, frequency #" + str(layer)
        #plt.xlabel(msg)
        #plt.savefig(name)
        frequencies[layer,:,:]= frequency
    return frequencies

def layer_image_anisotropy(test_image,image_name,seq_length,orientations,order):
    """
    This function calculates a parameter that behaves as an objective measure of the quality of the image for Gaussian blur
    and Gaussian noise. It is based on the frequency content given by the pseudo-Wigner distribution.
    """
    entropy_val = np.zeros([orientations])
    for orientation in range(orientations):
        angle = (180/orientations)*orientation
        print( angle, " degrees distribution")
        distribution = layer_wigner_distribution(test_image,seq_length,angle)
        entropy_pixelwise = renyi_entropy(distribution,order)
        entropy = np.mean(entropy_pixelwise)
        print("entropy is %.4f" % entropy)
        entropy_val[orientation] = entropy
    anisotropy = np.var(entropy_val)
    anisotropy = math.sqrt(anisotropy)
    print("\n")
    print("The AQI (image anisotropy) is %.4f" % anisotropy)
    print("\n")
    F = open("AQI test.txt","a")
    F.write(image_name)
    F.write("\n")
    F.write("The Anisotropic Quality Index of the image is %.4f\n" % anisotropy)
    F.close()
    return anisotropy

```

### 3. The test image

By the following code we introduce the test image

```python
import os
os.path
cwd = os.getcwd()
print(cwd)
```

    /home/stefanie/IdeaProjects/MD_CompressedWavelet/notebooks

```python
def input_test_image(subfolder,name):
    total_name = subfolder + name
    input_image = mpimg.imread(total_name)
    image_dimension = len(input_image.shape)
    if image_dimension == 3:
        test_image = (1/3)*(input_image[:,:,0]+
                      input_image[:,:,1]+input_image[:,:,2])
    else:
        test_image = input_image

    # convert image to regular gray levels
    test_image =np.uint8(255*test_image)
    return test_image

```

```python
#%matplotlib inline
# input test image
image_name = 'boat.png'
subfolder = ''
test_image = input_test_image(subfolder,image_name)
# show image version to be processed
#plt.figure(figsize =(6,6))
#plt.imshow(test_image, cmap='gray')
#plt.xlabel("Test image")
#plt.show()
```

### 4. The Wigner distribution

This is an example to show the pseudo-Wigner distribution as a set of images

```python
# calculate Wigner distribution of test image
seq_length = 9
angle = 170.633972
distribution = layer_wigner_distribution(test_image,seq_length,angle)
# show distribution as a set of images
frequencies = show_wigner_frequencies(distribution)

```

### 5. Computation of image entropy

From the Wigner distribution of the test image we can calculate the image entropy. Entropy values are normalized from 0 to 1. The resulting image is saved as "Entropy.pdf" in the current directory. The pixelwise entropy is saved as "Entropy.npy".

```python
# show entropy as image
#%matplotlib inline
order = 3 # this is a commom value
entropy = renyi_entropy(distribution,order)
entropy_global = np.mean(entropy)
print("global image entropy is %.4f" % entropy_global)

```

    global image entropy is 0.0396

```python
#plt.figure(figsize =(6,6))
entropy_im = np.uint8(255*entropy)
#plt.imshow(entropy_im,cmap='gray')
#plt.xlabel("Entropy")
np.save("Entropy",entropy)
#plt.savefig("Entropy_im.pdf")
#plt.show()
```

### 6. Definition of the Anisotropic Quality Index

The 1-D window used to measure entropy by this method may be oriented in any desired direction $\theta$, and the local entropy for pixel $n$ is now denoted as $R(n,\theta)$. This local pixel-wise directional entropy can be constrained to a scalar value for the whole image by averaging all the $R(n,\theta)$ values

\begin{equation}
\overline{R}(\theta)=\frac{1}{M}\sum\limits_{n}R(n,\theta)
\end{equation}

provided that $M$ is the number of pixels in the image. The standard deviation of the expected directional entropy of the image has been defined as an Anisotropic Quality Index (AQI) and adopted as a quality assessment measure:
\begin{equation}
AQI \equiv \sigma(\overline{R}(\theta)),
\end{equation}

From our experimental studies we have shown that the anisotropy of the image based on the above Rényi entropy measure is monotonously decreasing in presence of Gaussian noise. As more Gaussian noise is added to an original, its anisotropy decreases monotonously.  Also, our experiences have shown that as more blur is added to the original, its anisotropy also decreases monotonously. The maximum anisotropy is attained by the original ground truth image.

### 7.1 Single example

This example shows the use of the code for calculating the AQI for a sigle image (the above test image)

```python
orientations = 4
order = 3
seq_length = 9
anisotropy = layer_image_anisotropy(test_image,image_name,seq_length,orientations,order)
```

    0.0  degrees distribution
    entropy is 0.0351
    45.0  degrees distribution
    entropy is 0.0593
    90.0  degrees distribution
    entropy is 0.0452
    135.0  degrees distribution
    entropy is 0.0608

    The AQI (image anisotropy) is 0.0106

### 7.2 Example of application of the AQI for comparing the quality of several related images

The following code runs a test over a set of images from TID2013 image database, corrupted with different amount of *Gaussian blur*, starting from the original and showing the measures in a graphic. The plot reveals how as more blur is added to the image the AQI value get smaller.

**Measures of the AQI are stored in file 'AQI test.txt'**

```python
# initial conditions
orientations = 4
order = 3
seq_length = 9
R = 6 # number of test images
AQI =np.zeros(R)
y_pos = np.zeros(R)
subfolder = './Gaussian blur/'
for i in range(R):
    y_pos[i] = i
    # input test image
    image_name =  'houses_0' + str(i) + '.png'
    print("processing image ",image_name)
    test_image = input_test_image(subfolder,image_name)
    # process image
    anisotropy = layer_image_anisotropy(test_image,image_name,seq_length,orientations,order)

    AQI[i] = anisotropy
    # show processed image
    #plt.figure()
    #plt.imshow(test_image, cmap='gray')
    #label = str(i) + ". " + image_name + ", AQI = " +  "  %.4f" % anisotropy
    #plt.xlabel(label)

# Test plot
#plt.figure(figsize =(7,4))
#plt.bar(y_pos, AQI,width = 1/4, align='center', alpha=1,color = "red")
#plt.ylabel("Anisotropic Quality Index of test images")
#plt.xlabel("Labels of test images")
#plt.show()

```

```python

```

```python

```