## Histograms of Oriented Gradients (HOG)

As we saw with the ORB algorithm, we can use keypoints in images to do keypoint-based matching to detect objects in images. These type of algorithms work great when you want to detect objects that have a lot of consistent internal features that are not affected by the background. For example, these algorithms work well for facial detection because faces have a lot of consistent internal features that don’t get affected by the image background, such as the eyes, nose, and mouth. However, these type of algorithms don’t work so well when attempting to do more general object recognition, say for example, pedestrian detection in images. The reason is that people don’t have consistent internal features, like faces do, because the body shape and style of every person is different (see Fig. 1). This means that every person is going to have a different set of internal features, and so we need something that can more generally describe a person.

Fig. 1. - Pedestrians.

One option is to try to detect pedestrians by their contours instead.  Detecting objects in images by their contours (boundaries) is very challenging because we have to deal with the difficulties brought about by the contrast between the background and the foreground. For example, suppose you wanted to detect a pedestrian in an image that is walking in front of a white building and she is wearing a white coat and black pants (see Fig. 2). We can see in Fig. 2, that since the background of the image is mostly white, the black pants are going to have a very high contrast, but the coat, since it is white as well, is going to have very low contrast. In this case, detecting the edges of pants is going to be easy but detecting the edges of the coat is going to be very difficult. This is where **HOG** comes in. HOG stands for **Histograms of Oriented Gradients** and it was first introduced by Navneet Dalal and Bill Triggs in 2005.

Fig. 2. - High and Low Contrast.

The HOG algorithm works by creating histograms of the distribution of gradient orientations in an image and then normalizing them in a very special way. This special normalization is what makes HOG so effective at detecting the edges of objects even in cases where the contrast is very low. These normalized histograms are put together into a feature vector, known as the HOG descriptor, that can be used to train a machine learning algorithm, such as a Support Vector Machine (SVM), to detect objects in images based on their boundaries (edges). Due to its great success and reliability, HOG has become one of the most widely used algorithms in computer vison for object detection.

In this notebook, you will learn:

* How the HOG algorithm works
* How to use OpenCV to create a HOG descriptor
* How to visualize the HOG descriptor.

# The HOG Algorithm

As its name suggests, the HOG algorithm, is based on creating histograms from the orientation of image gradients. The HOG algorithm is implemented in a series of steps:

1. Given the image of particular object, set a detection window (region of interest) that covers the entire object in the image (see Fig. 3).

2. Calculate the magnitude and direction of the gradient for each individual pixel in the detection window.

3. Divide the detection window into connected *cells* of pixels, with all cells being of the same size (see Fig. 3). The size of the cells is a free parameter and it is usually chosen so as to match the scale of the features that want to be detected. For example, in a 64 x 128 pixel detection window, square cells 6 to 8 pixels wide are suitable for detecting human limbs.

4. Create a Histogram for each cell, by first grouping the gradient directions of all pixels in each cell into a particular number of orientation (angular) bins; and then adding up the gradient magnitudes of the gradients in each angular bin (see Fig. 3). The number of bins in the histogram is a free parameter and it is usually set to 9 angular bins.

5. Group adjacent cells into *blocks* (see Fig. 3). The number of cells in each block is a free parameter and all blocks must be of the same size. The distance between each block (known as the stride) is a free parameter but it is usually set to half the block size, in which case you will get overlapping blocks (*see video below*).  The HOG algorithm has been shown empirically to work better with overlapping blocks.

6. Use the cells contained within each block to normalize the cell histograms in that block (see Fig. 3). If you have overlapping blocks this means that most cells will be normalized with respect to different blocks (*see video below*). Therefore, the same cell may have several different normalizations.

7. Collect all the normalized histograms from all the blocks into a single feature vector called the HOG descriptor.

8. Use the resulting HOG descriptors from many images of the same type of object to train a machine learning algorithm, such as an SVM, to detect those type of objects in images. For example, you could use the HOG descriptors from many images of pedestrians to train an SVM to detect pedestrians in images. The training is done with both positive a negative examples of the object you want detect in the image.

9. Once the SVM has been trained, a sliding window approach is used to try to detect and locate objects in images.  Detecting an object in the image entails finding the part of the image that looks similar to the HOG pattern learned by the SVM.

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# Why The HOG Algorithm Works

As we learned above, HOG creates histograms by adding the magnitude of the gradients in particular orientations in localized portions of the image called *cells*.  By doing this we guarantee that stronger gradients will contribute more to the magnitude of their respective angular bin, while the effects of weak and randomly oriented gradients resulting from noise are minimized. In this manner the histograms tell us the dominant gradient orientation of each cell.

### Dealing with contrast

Now, the magnitude of the dominant orientation can vary widely due to variations in local illumination and the contrast between the background and the foreground.

To account for the background-foreground contrast differences, the HOG algorithm tries to detect edges locally.  In order to do this, it defines groups of cells, called **blocks**, and normalizes the histograms using this local group of cells. By normalizing locally, the HOG algorithm can detect the edges in each block very reliably; this is called **block normalization**.

In addition to using block normalization, the HOG algorithm also uses overlapping blocks to increase its performance. By using overlapping blocks, each cell contributes several independent components to the final HOG descriptor, where each component corresponds to a cell being normalized with respect to a different block. This may seem redundant but, it has been shown empirically that by normalizing each cell several times with respect to different local blocks, the performance of the HOG algorithm increases dramatically.

### Loading Images and Importing Resources

The first step in building our HOG descriptor is to load the required packages into Python and to load our image.

We start by using OpenCV to load an image of a triangle tile. Since, the `cv2.imread()` function loads images as BGR we will convert our image to RGB so we can display it with the correct colors. As usual we will convert our BGR image to Gray Scale for analysis.

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

# Creating The HOG Descriptor

We will be using OpenCV’s `HOGDescriptor` class to create the HOG descriptor. The parameters of the HOG descriptor are setup using the `HOGDescriptor()` function. The parameters of the `HOGDescriptor()` function and their default values are given below:

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

Parameters:

* **win_size** – *Size*
Size of detection window in pixels (*width, height*). Defines the region of interest. Must be an integer multiple of cell size.

* **block_size** – *Size*
Block size in pixels (*width, height*). Defines how many cells are in each block. Must be an integer multiple of cell size and it must be smaller than the detection window. The smaller the block the finer detail you will get.

* **block_stride** – *Size*
Block stride in pixels (*horizontal, vertical*). It must be an integer multiple of cell size. The `block_stride` defines the distance between adjecent blocks, for example, 8 pixels horizontally and 8 pixels vertically. Longer `block_strides` makes the algorithm run faster (because less blocks are evaluated) but the algorithm may not perform as well.

* **cell_size** – *Size*
Cell size in pixels (*width, height*). Determines the size fo your cell. The smaller the cell the finer detail you will get.

* **nbins** – *int*
Number of bins for the histograms. Determines the number of angular bins used to make the histograms. With more bins you capture more gradient directions. HOG uses unsigned gradients, so the angular bins will have values between 0 and 180 degrees.

* **win_sigma** – *double*
Gaussian smoothing window parameter. The performance of the HOG algorithm can be improved by smoothing the pixels near the edges of the blocks by applying a Gaussian spatial window to each pixel before computing the histograms.

* **threshold_L2hys** – *double*
L2-Hys (Lowe-style clipped L2 norm) normalization method shrinkage. The L2-Hys method is used to normalize the blocks and it consists of an L2-norm followed by clipping and a renormalization. The clipping limits the maximum value of the descriptor vector for each block to have the value of the given threshold (0.2 by default). After the clipping the descriptor vector is renormalized as described in *IJCV*, 60(2):91-110, 2004.

* **gamma_correction** – *bool*
Flag to specify whether the gamma correction preprocessing is required or not. Performing gamma correction slightly increases the performance of the HOG algorithm.

* **nlevels** – *int*
Maximum number of detection window increases.

As we can see, the  `cv2.HOGDescriptor()`function supports a wide range of parameters. The first few arguments (`block_size, block_stride, cell_size`, and `nbins`) are probably the ones you are most likely to change. The other parameters can be safely left at their default values and you will get good results.

In the code below, we will use the `cv2.HOGDescriptor()`function to set the cell size, block size, block stride, and the number of bins for the histograms of the HOG descriptor. We will then use `.compute(image)`method to compute the HOG descriptor (feature vector) for the given `image`.

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

    The gray scale image has shape:  (250, 250)

    HOG Descriptor Parameters:

    Window Size: (246, 246)
    Cell Size: (6, 6)
    Block Size: (12, 12)
    Block Stride: (6, 6)
    Number of Bins: 9

# Number of Elements In The HOG Descriptor

The resulting HOG Descriptor (feature vector), contains the normalized histograms from all cells from all blocks in the detection window concatenated in one long vector. Therefore, the size of the HOG feature vector will be given by the total number of blocks in the detection window, multiplied by the number of cells per block, times the number of orientation bins:

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

If we don’t have overlapping blocks (*i.e.* the `block_stride`equals the `block_size`), the total number of blocks can be easily calculated by dividing the size of the detection window by the block size. However, in the general case we have to take into account the fact that we have overlapping blocks. To find the total number of blocks in the general case (*i.e.* for any `block_stride` and `block_size`), we can use the formula given below:

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

Where Total$_x$, is the total number of blocks along the width of the detection window, and Total$_y$, is the total number of blocks along the height of the detection window. This formula for Total$_x$ and Total$_y$, takes into account the extra blocks that result from overlapping. After calculating Total$_x$ and Total$_y$, we can get the total number of blocks in the detection window by multiplying Total$_x$  $\times$ Total$_y$. The above formula can be simplified considerably because the `block_size`, `block_stride`, and `window_size`are all defined in terms of the `cell_size`. By making all the appropriate substitutions and cancelations the above formula reduces to:

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

Where cells$_x$ is the total number of cells along the width of the detection window, and cells$_y$, is the total number of cells along the height of the detection window. And $N_x$ is the horizontal block stride in units of `cell_size` and $N_y$ is the vertical block stride in units of `cell_size`.

Let's calculate what the number of elements for the HOG feature vector should be and check that it matches the shape of the HOG Descriptor calculated above.

```python
print(x_cells)
print(num_cells_per_block[0])
print(h_stride)

# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

    41
    2
    1

    The total number of elements in the HOG Feature Vector should be:  40 x 40 x 2 x 2 x 9 = 57600

    The HOG Descriptor has shape: (57600, 1)

# Visualizing The HOG Descriptor

We can visualize the HOG Descriptor by plotting the histogram associated with each cell as a collection of vectors. To do this, we will plot each bin in the histogram as a single vector whose magnitude is given by the height of the bin and its orientation is given by the angular bin that its associated with. Since any given cell might have multiple histograms associated with it, due to the overlapping blocks, we will choose to average all the histograms for each cell to produce a single histogram for each cell.

OpenCV has no easy way to visualize the HOG Descriptor, so we have to do some manipulation first in order to visualize it. We will start by reshaping the HOG Descriptor in order to make our calculations easier.  We will then compute the average histogram of each cell and finally we will convert the histogram bins into vectors. Once we have the vectors, we plot the corresponding vectors for each cell in an image.

The code below produces an interactive plot so that you can interact with the figure. The figure contains:
* the grayscale image,
* the HOG Descriptor (feature vector),
* a zoomed-in portion of the HOG Descriptor, and
* the histogram of the selected cell.

**You can click anywhere on the gray scale image or the HOG Descriptor image to select a particular cell**. Once you click on either image a *magenta* rectangle will appear showing the cell you selected. The Zoom Window will show you a zoomed in version of the HOG descriptor around the selected cell; and the histogram plot will show you the corresponding histogram for the selected cell. The interactive window also has buttons at the bottom that allow for other functionality, such as panning, and giving you the option to save the figure if desired. The home button returns the figure to its default value.

**NOTE**: If you are running this notebook in the Udacity workspace, there is around a 2 second lag in the interactive plot. This means that if you click in the image to zoom in, it will take about 2 seconds for the plot to refresh.

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

    The feature vector has shape: (57600, 1)
    The reshaped feature vector has shape: (40, 40, 2, 2, 9)
    The average gradient array has shape:  (41, 41, 9)

    <IPython.core.display.Javascript object>

# Understanding The Histograms

Let's take a look at a couple of snapshots of the above figure to see if the histograms for the selected cell make sense. Let's start looking at a cell that is inside a triangle and not near an edge:

Fig. 4. - Histograms Inside a Triangle.

In this case, since the triangle is nearly all of the same color there shouldn't be any dominant gradient in the selected cell. As we can clearly see in the Zoom Window and the histogram, this is indeed the case. We have many gradients but none of them clearly dominates over the other.

Now let’s take a look at a cell that is near a horizontal edge:

Fig. 5. - Histograms Near a Horizontal Edge.

Remember that edges are areas of an image where the intensity changes abruptly. In these cases, we will have a high intensity gradient in some particular direction. This is exactly what we see in the corresponding histogram and Zoom Window for the selected cell.  In the Zoom Window, we can see that the dominant gradient is pointing up, almost at 90 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 90-degree bin in the histogram to dominate strongly over the others. This is in fact what we see.

Now let’s take a look at a cell that is near a vertical edge:

Fig. 6. - Histograms Near a Vertical Edge.

In this case we expect the dominant gradient in the cell to be horizontal, close to 180 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 170-degree bin in the histogram to dominate strongly over the others. This is what we see in the histogram but we also see that there is another dominant gradient in the cell, namely the one in the 10-degree bin. The reason for this, is because the HOG algorithm is using unsigned gradients, which means 0 degrees and 180 degrees are considered the same. Therefore, when the histograms are being created, angles between 160 and 180 degrees, contribute proportionally to both the 10-degree bin and the 170-degree bin. This results in there being two dominant gradients in the cell near the vertical edge instead of just one.

To conclude let’s take a look at a cell that is near a diagonal edge.

Fig. 7. - Histograms Near a Diagonal Edge.

To understand what we are seeing, let’s first remember that gradients have an *x*-component, and a *y*-component, just like vectors. Therefore, the resulting orientation of a gradient is going to be given by the vector sum of its components. For this reason, on vertical edges the gradients are horizontal, because they only have an x-component, as we saw in Figure 4. While on horizontal edges the gradients are vertical, because they only have a y-component, as we saw in Figure 3. Consequently, on diagonal edges, the gradients are also going to be diagonal because both the *x* and *y* components are non-zero.  Since the diagonal edges in the image are close to 45 degrees, we should expect to see a dominant gradient orientation in the 50-degree bin. This is in fact what we see in the histogram but, just like in Figure 4., we see there are two dominant gradients instead of just one. The reason for this is that when the histograms are being created, angles that are near the boundaries of bins, contribute proportionally to the adjacent bins.  For example, a gradient with an angle of 40 degrees, is right in the middle of the 30-degree and 50-degree bin. Therefore, the magnitude of the gradient is split evenly into the 30-degree and 50-degree bin. This results in there being two dominant gradients in the cell near the diagonal edge instead of just one.

Now that you know how HOG is implemented, in the workspace you will find a notebook named *Examples*. In there, you will be able set your own paramters for the HOG descriptor for various images. Have fun!


# Jupyter (iPython) Notebookを使って技術ノート環境を構築する方法

myenigma.hatenablog.com

```python
from sympy import *
```

```python
x=Symbol('x')
```

```python
%matplotlib inline
```

```python
init_printing()
```

```python
expand((x - 3)**5)
```

```python
y=Symbol('y')
expr=cos(x)*ln(y)+2/y
expr
```

```python
solve(expr, x)
```

しかし、下記のJupyter Notebooksのextensionをインストールすることで、

様々な拡張機能が使えるようになり、

画像のドラッグアンドドロップもできるようになります。

インストールは、READMEにある通り、

下記のコマンドでOKです。

(下記のインストールをする場合は

Jupyter Notebookのプロセスはkillしておきましょう)

$ pip install https://github.com/ipython-contrib/IPython-notebook-extensions/archive/master.zip –user

続いて、Jupyter Notebookを起動し、

```python

```

```python

```

```python

```

URLに/nbextensions/を追加して、移動すると

下記のような拡張機能の追加画面に移動します。

あとは、Drag and DropのチェックボタンをONして下さい。

すると、Jupyter Notebookで画像をドラッグアンドドロップすると、

画像が挿入されるようになります。

ちなみに画像ファイルはD＆Dすると

.ipynbファイルと

同じ場所にコピーされます。

目次を見出し情報から自動生成する
上記のIPython-notebook-extensionsの一機能を使うことで、

Markdownの見出し情報から、

上記のように自動で目次を生成することができます。

使い方としては、

先ほどの/nbextensions/の設定画面で、

Table Contents(2)にチェックをいれます。

加えて、同じ設定画面で、

“Add a Table of Contents at the top of the notebook”

にチェックをいれます。

最後に下記のように

目次ウインドウを表示させて、

tボタンを押すと、

自動的に目次情報が先頭のセルに追加されます。

レポートタイトルを入力する方法
残念ながら、今のところ

レポートのタイトルのようなものを

入力するツールは見つかっていませんが、

JupyterのMarkdownのモードでは、

HTMLを入力するとマークアップされた文字を入力できます。

この機能を使うことで、レポートタイトルっぽい文字を表示できました。

例えば、下記のようなhtmlを

JupyterのMarkdownモードのセルに入力すると、

Jupyter Report

Atsushi Sakai

下記のように表示されます。

PDFに出力する
上記の方法である程度のレポート形式の技術ノートを作ることは

できるはずなので、あとはPDFなどに変換すれば、

プログラマ以外の他の人と資料を共有することができます。

Macの場合
Jupyter NotebookのデータをPDFに変換する方法は色々ありますが、

Macで一番シンプルなのは、Jupyter Notebookをブラウザで開いた状態で、

Ctrl-Pで、Macの標準機能でPDFに変換するのが一番簡単だと思います。

下記のように詳細設定の部分で、ヘッダーとフッターに

時刻や、ファイル名、ページなどを入力する設定ができます。

あとは普通にPDFを出力すると、

下記のようにそれなりのレポートがPDFで作成できます。

Jupyterでプレゼン資料を作る方法
下記を参照下さい。

myenigma.hatenablog.com

JupyterのサーバをHerokuの無料枠で構築する方法
下記を参照下さい。

myenigma.hatenablog.com

Jupyter Markdown数式の入力テンプレート
最近、数式の入力はすべてJupyterで実施していますが、

複雑でいつもググっている数式コマンドをメモとして残しておきます。

行列
\begin{equation*}
\begin{bmatrix}
1 & 0 & \cdots \\
0 & 1 & \cdots \\
\vdots & \vdots & \ddots \\
-1 & 0 & \cdots \\
0 & -1 & \cdots \\
\vdots & \vdots & \ddots \\
\end{bmatrix}
\end{equation*}
数式参考リンク
できれば改善してもらいたい部分
もう少しで完全にWordをおさらばできそうなので、

下記のような機能を実現してもらいたいですね。

参考文献管理

図のタイトル入力

自動図番号割り当て

上記のような機能を実現できる拡張などを知っている方は

コメントなどで教えて頂けると幸いです。

(Python使えるから、自分で作ればいいのか。。。)

vimユーザのためのJupyterプラグイン
下記を参照下さい

lambdalisue.hatenablog.com

なにかおかしいので云々。
bookmark

```python

```

```python

```

```python

```

```python

```

```python

```

上記のプラグインを使うことで、

Jupyterのエディタの部分でvimのキーバインドが使えます。

Jupyter Notebook上でコードの処理時間を計測する
Jupyter notebookでは、

%timeit

の後にコードを記述すると、そのコードの計算時間を計測してくれます。

コードやアルゴリズムのレポートに便利ですね。

moqada.hatenablog.com

Jupyter上でモジュールや関数のdocstringを確認する
Jupyter notebookでは、

モジュールや関数の後ろに?(はてな)をつけると、

docstringを表示してくれるので、

マニュアル確認も簡単です。

更にJupyterを学びたい人は
下記の資料がおすすめです。

また、どうしても解決できない問題などが

ある場合は下記のようなQ＆Aサイトで質問してみると、

かなりの確率で回答がもらえると思います。

参考資料
myenigma.hatenablog.com

myenigma.hatenablog.com

myenigma.hatenablog.com

myenigma.hatenablog.com

myenigma.hatenablog.com

MyEnigma Supporters
もしこの記事が参考になり、

ブログをサポートしたいと思われた方は、

こちらからよろしくお願いします。


# Tutorial Notebook 2. Inference and Planning

In this notebook, we will continue on from the last notebook to build a fully fledged active inference agent capable of performing inference and planning using Active Inference in the simple grid-world environment. We will also begin to use some aspects of `pymdp`, although this will mostly be helpful functions for building and sampling from discrete distributions while we will implement the core functionality of the agent ourselves.

First, we simply start out by defining our generative model as we did last time.

## Add `pymdp` module

```python
# This is needed (on my machine at least) due to weird python import issues
import os
import sys
from pathlib import Path

path = Path(os.getcwd())
print(path)
module_path = str(path.parent) + '/'
sys.path.append(module_path)
```

## Imports

```python
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from copy import deepcopy

from pymdp import maths, utils
from pymdp.maths import spm_log_single as log_stable # @NOTE: we use the `spm_log_single` helper function from the `maths` sub-library of pymdp. This is a numerically stable version of np.log()
from pymdp import control
print("imports loaded")
```

## Plotting

```python
state_mapping = {0: (0,0), 1: (1,0), 2: (2,0), 3: (0,1), 4: (1,1), 5:(2,1), 6: (0,2), 7:(1,2), 8:(2,2)}

A = np.eye(9)
def plot_beliefs(Qs, title=""):
    #values = Qs.values[:, 0]
    plt.grid(zorder=0)
    plt.bar(range(Qs.shape[0]), Qs, color='r', zorder=3)
    plt.xticks(range(Qs.shape[0]))
    plt.title(title)
    plt.show()

labels = [state_mapping[i] for i in range(A.shape[1])]
def plot_likelihood(A):
    fig = plt.figure(figsize = (6,6))
    ax = sns.heatmap(A, xticklabels = labels, yticklabels = labels, cbar = False)
    plt.title("Likelihood distribution (A)")
    plt.show()

def plot_empirical_prior(B):
    fig, axes = plt.subplots(3,2, figsize=(8, 10))
    actions = ['UP', 'RIGHT', 'DOWN', 'LEFT', 'STAY']
    count = 0
    for i in range(3):
        for j in range(2):
            if count >= 5:
                break

            g = sns.heatmap(B[:,:,count], cmap="OrRd", linewidth=2.5, cbar=False, ax=axes[i,j])

            g.set_title(actions[count])
            count += 1
    fig.delaxes(axes.flatten()[5])
    plt.tight_layout()
    plt.show()

def plot_transition(B):
    fig, axes = plt.subplots(2,3, figsize = (15,8))
    a = list(actions.keys())
    count = 0
    for i in range(dim-1):
        for j in range(dim):
            if count >= 5:
                break
            g = sns.heatmap(B[:,:,count], cmap = "OrRd", linewidth = 2.5, cbar = False, ax = axes[i,j], xticklabels=labels, yticklabels=labels)
            g.set_title(a[count])
            count +=1
    fig.delaxes(axes.flatten()[5])
    plt.tight_layout()
    plt.show()
```

## Generative model

Here, we setup our generative model which is the same as in the last notebook. This is formed of a likelihood distribution $P(o_t|s_t)$, denoted `A`, and a empirical prior (transition) distribution $P(s_t|s_{t-1},a_{t-1})$, denoted `B`.

Since this was covered in more detail in the previous tutorial, we quickly skip over the details here

```python
# A matrix
A = np.eye(9)
plot_likelihood(A)
```

```python
# construct B matrix

P = {}
dim = 3
actions = {'UP':0, 'RIGHT':1, 'DOWN':2, 'LEFT':3, 'STAY':4}

for state_index, xy_coordinates in state_mapping.items():
    P[state_index] = {a : [] for a in range(len(actions))}
    x, y = xy_coordinates

    '''if your y-coordinate is all the way at the top (i.e. y == 0), you stay in the same place -- otherwise you move one upwards (achieved by subtracting 3 from your linear state index'''
    P[state_index][actions['UP']] = state_index if y == 0 else state_index - dim

    '''f your x-coordinate is all the way to the right (i.e. x == 2), you stay in the same place -- otherwise you move one to the right (achieved by adding 1 to your linear state index)'''
    P[state_index][actions["RIGHT"]] = state_index if x == (dim -1) else state_index+1

    '''if your y-coordinate is all the way at the bottom (i.e. y == 2), you stay in the same place -- otherwise you move one down (achieved by adding 3 to your linear state index)'''
    P[state_index][actions['DOWN']] = state_index if y == (dim -1) else state_index + dim

    ''' if your x-coordinate is all the way at the left (i.e. x == 0), you stay at the same place -- otherwise, you move one to the left (achieved by subtracting 1 from your linear state index)'''
    P[state_index][actions['LEFT']] = state_index if x == 0 else state_index -1

    ''' Stay in the same place (self explanatory) '''
    P[state_index][actions['STAY']] = state_index

num_states = 9
B = np.zeros([num_states, num_states, len(actions)])
for s in range(num_states):
    for a in range(len(actions)):
        ns = int(P[s][a])
        B[ns, s, a] = 1

plot_transition(B)
```

# Create Environment Class

To make things simple we will parcel up the $A$ and $B$ matrices into a class which represents the environment. The environment has two functions `step` which when given an action will update the environment a single step and `reset` which resets the environment back to its initial condition. The API of our simple environment class is similar to the `Env` base class used by `pymdp`, although the `pymdp` version has many more features than we use here.

```python
class GridWorldEnv():

    def __init__(self,A,B):
        self.A = deepcopy(A)
        self.B = deepcopy(B)
        print("B:", B.shape)
        self.state = np.zeros(9)
        # start at state 3
        self.state[2] = 1

    def step(self,a):
        self.state = np.dot(self.B[:,:,a], self.state)
        obs = utils.sample(np.dot(self.A, self.state))
        return obs

    def reset(self):
        self.state =np.zeros(9)
        self.state[2] =1
        obs = utils.sample(np.dot(self.A, self.state))
        return obs

env = GridWorldEnv(A,B)
```

# Inference

Now that we have the generative model setup, we turn to the behaviour of the active inference agent itself. To recap, we assume that this agent receives observations from the environment and can emit actions. Moreover, we assume that this agent has some kind of goal or preferences over the state of the environment it wants to create, and will choose actions in order to increase the probability of observing itself in its preferred state. For the time being, we will not deal with the problem of action selection but only with inference.

The agent receives observations $o_t$ from the environment but does not naturally know the environments true state $x_t$. Thus, the agent must *infer* this state by computing the posterior distribution $p(x_t | o_t)$. It can do this by Bayesian inference using Bayes rule but, as we discussed last time, explicitly computing Bayes rule is often intractable because the marginal likelihood requires the averaging over an infinite number of hypotheses. We therefore need some other way to compute or approximate this posterior. Active inference assumes that this posterior can be approximated through a family of methods called *variational inference* which only approximate the posterior, but are fast and computationally efficient.

# Variational Inference

Variational inference is a set of inference methods which can rapidly and efficiently compute *approximate* posteriors for Bayesian inference problems. The key idea behind variational inference is that instead of trying to compute the true posterior $p(x_t | o_t)$ which may be extremely complex, is that we instead optimize an *approximate posterior*. Specifically, we will define another distribution $q(x_t ; \phi)$ which has some parameters $\phi$ which we then optimize so as to make $q(x_t ; \phi)$ as close as possible to the true distribution. Typically, we choose this $q$ distribution to be some simple distribution which is easy to work with mathematically. If the process works, then we can get the $q$ distribution very close to the true posterior, and as such get a good estimate of the posterior without ever explicitly computing it using Bayes rule.

Mathematically, we can do this by setting up an optimization problem. We have the true posterior $p(x_t | o_t)$, which is unknown, and we have our $q$ distribution which we do know. We then want to optimize the $q$ distribution to make it as *close as possible* to the true posterior $p(x_t | o_t)$. To do this, we first need a way to quantify *how close* two probability distributions are. The way we do this is by using a quantity known as the *Kullback-Leibler (KL) divergence*. This is a metric derived from information theory which lets us quantify the distance (in bits) of two distributions. The KL divergence between two distributions $q(x)$ and $p(x)$ is defined as,
$$
\begin{align}
KL[q(x) || p(x)] = \sum_x q(x) (\ln q(x) - \ln p(x))
\end{align}
$$
Mathematically it can be thought of as the average of the difference of the logarithms of the probabilities assigned by $q$ and $p$ to the states $x$. The KL divergence is smallest when $q(x) = p(x)$ when it is equal to 0, and can grow to be infinitely large which happens wherever $q(x)$ assigns a nonzero probability but $p(x)$ doesn't. In code, we can compute the KL divergence as:

```python
def KL_divergence(q,p):
    return np.sum(q * (log_stable(q) - log_stable(p)))
```

Now that we know about the KL divergence, we can express our variational problem of making our approximate posterior $q(x_t ; \varphi)$ as close as possible to the true posterior as simply minimizing the KL divergence between the two distributions. That is, we can define the optimal approximate distribution as,
$$
\begin{align}
q^*(x_t ; \varphi) = \operatorname{argmin}_{\varphi} \, KL[q(x_t ; \varphi) || p(x_t | o_t)]
\end{align}
$$

And then simply try to optimize this objective so as to find the setting of the variational parameters $\varphi$ that make $q(x_t ; \varphi)$ as close to $p(x_t |o_t)$ as possible. The trouble with this is that our objective actually explicitly contains the true posterior in it and so, since we can't conpute the true posterior, we can't compute this objective either -- so we are stuck!

Variational inference provides a clever way to get around this problem by instead minimizing an *upper bound* on this divergence called the *variational free energy*. Importantly this bound is computable so we can actually optimize it and moreover since it is an upper bound, if we minimize it, we can make $q$ as close as possible to the real bound, thus still managing to obtain a good approximate posterior distribution. Deriving the variational free energy is very simple, we first take our initial objective and apply Bayes rule to the true posterior, and then take out the marginal likelihood term separately
<!-- $$
\begin{align}
 KL[q(x_t | o_t) || p(x_t |  o_t)] &= KL[q(x_t | o_t) || \frac{p(o_t,x_t)}{p(o_t)}] \\
 &= KL[q(x_t | o_t) || p(o_t, x_t)] + \sum_x q(x_t | o_t) \ln p(o_t) \\
 &= KL[q(x_t | o_t) || p(o_t, x_t)] +  \ln p(o_t)
\end{align}
$$ -->

$$
\begin{align}
 KL[q(x_t ; \varphi) || p(x_t |  o_t)] &= KL[q(x_t ; \varphi) || \frac{p(o_t,x_t)}{p(o_t)}] \\
 &= KL[q(x_t ; \varphi) || p(o_t, x_t)] + \sum_x q(x_t ; \varphi) \ln p(o_t) \\
 &= KL[q(x_t ; \varphi) || p(o_t, x_t)] +  \ln p(o_t)
\end{align}
$$

Where in the final line we have used the fact that the sum is over a different variable than the distribution, and the sum of a probability distribution is $1$ -- i.e $\sum_x q(x_t ; \varphi) \ln p(o_t) = \ln p(o_t) * \sum_x q(x_t ; \varphi)$ and $\sum_x q(x_t ; \varphi) = 1$. Specifically, since $ \ln p(o_t)$ is the log of a probability distribution it is always negative, since the probability of a state is always between 0 and 1. This means that we know that this term we have devised $KL[q(x_t ; \varphi) || p(o_t, x_t)]$ is always necessarily greater than our original divergence between the approximate posterior and the true posterior, so it is an *upper bound*. We call this term the *variational free energy* and denote it by $\mathcal{F}$.
$$
\begin{align}
\mathcal{F} = KL[q(x_t ; \varphi) || p(o_t, x_t)]
\end{align}
$$

The free energy here is simply the divergence between the approximate posterior and the *generative model* of the agent. Since we know both the approximate posterior (as we defined it in the first place!) and the generative model, then both terms of this divergence are computable. We thus have our algorithm to approximate the posterior! Since the free energy is an upper bound, if we minimize the free energy, we also implicitly minimize the true divergence between the true and approximate posteriors, which will force the approximate posterior to be close to the true posterior and thus a good approximation! Moreover, since we can compute the free energy, we can actually perform this optimization!

In many cases, we typically perform variational inference by taking the gradients of the free energy with respect to the variational parameteres $\varphi$ and then doing gradient descent on the parameters $\varphi$ that define $q(x_t ; \varphi)$. However, when the distributions are discrete (i.e. Categorical distributions), the parameters of the approximate distribution are simply the probability values for each state (the elements of the vector $q(x)$. For some simple generative models (e.g. the grid-world described here), we can actually solve this optimization problem directly to obtain $q(x)$ in a single step, instead of as a gradient descent.

# Directly solving variational inference in the case of a simple discrete model

To recap, remember that we have turned the problem of computing the posterior distribution $p(x_t | o_t)$ into that of minimizing the variational free energy: $\mathcal{F} = KL[q(x_t) || p(o_t, x_t)]$ with respect to an approximate posterior distribution $q(x_t)$. From now on, we will leave out the variational parameters $\varphi$ when referring to $q(x_t)$, since we are dealing with a single Categorical distribution $q(x_t)$ whose vector elements are identical to the variational parameters, i.e. $\forall_{i} \varphi_i = q(x_t = i)$.

The optimal distribution is simply that particular $q^*(x_t)$ that minimizes the KL divergence. Now, remember from high-school calculus that we can explicitly compute the minimum of a function by taking its derivative and setting it to 0 (i.e. at the minimum the first derivative of the function is 0) (if you don't remember this from calculus, trust me on this). This means that to solve this problem all we need to do is take the derivative of the free energy and set it to 0 and rearrange. First, let's write out the free energy explicitly.

<!-- $$
\begin{align}
\mathcal{F} &= KL[q(x_t | o_t) || p(o_t, x_t)] \\
&= \sum_x q(x_t | o_t) (\ln q(x_t | o_t) - \ln p(o_t,x_t))
\end{align}
$$ -->

$$
\begin{align}
\mathcal{F} &= KL[q(x_t) || p(o_t, x_t)] \\
&= \sum_x q(x_t)(\ln q(x_t) - \ln p(o_t,x_t))
\end{align}
$$
If we then split the generative model up into a likelihood and a prior, we can write it as,

$$
\begin{align}
\mathcal{F} = \sum_x q(x_t) \big[ \ln q(x_t) - \ln p(o_t | x_t)  - \ln p(x_t) \big]
\end{align}
$$
Recall that we have explicitly defined the observation and transition likelihood distributions as the $\textbf{A}$ and $\textbf{B}$ matrices, which play the role of the likelihood and prior distributions, respectively. In particular, we can define the prior over the current timestep $p(x_t)$ to be the "expected prior", given the beliefs about the state at the last timesep $q(x_{t-1})$ the beliefs about the transition dynamics and the past action, i.e.:

$$
\begin{align}
p(x_t) = \mathcal{E}_{q(x_{t-1})}\big[p(x_t | x_{t-1}, a_t) \big]
\end{align}
$$

which can be expressed as a simple matrix vector product

$$
\begin{align}
p(x_t) = \textbf{B}_{a_t}q(x_{t-1})
\end{align}
$$

where $\textbf{B}_{a_t}$ is the component of the $\textbf{B}$ matrix that is conditioned on action $a_t$. In other words, for this simple Markovian model, we assume that "yesterday's posterior is today's prior." For simplicity, we refer to this entire prior term as $\textbf{B}$ below, i.e. we temporarily define $\textbf{B} \equiv \textbf{B}_{a_t}q(x_{t-1})$.

As mentioned above, the posterior beliefs $q(x_t)$ are a vector of probabilities (the variational parameters) which we denote $\textbf{q} = [q_1, q_2, q_3 \dots]$. With this all defined, we can write out the free energy as,

$$
\begin{align}
\mathcal{F} = \sum_x \textbf{q} * \big[ \ln \textbf{q} - \ln \textbf{A}  - \ln \textbf{B} \big]
\end{align}
$$

And for fun, we can explicitly compute it in code:

```python
def compute_free_energy(q,A, B):
    return np.sum(q * (log_stable(q) - log_stable(A) - log_stable(B)))
```

Then, all we need to do is take the derivative of the free energy with respect to the approximate posterior distribution $\textbf{q}$ as follows,
$$
\begin{align}
\frac{\partial \mathcal{F}}{\partial \textbf{q}} = \ln \textbf{q} - \ln \textbf{A} - \ln \textbf{B} - \textbf{1}
\end{align}
$$

Where $\textbf{1}$ is just a vector of ones of equal length to $\textbf{q}$ and comes from the $q \frac{\partial \ln q}{\partial q} = q * \frac{1}{q} = 1$. Thus, if we set this derivative to 0 and rearrange, we can get,
$$
\begin{align}
0 &= \ln \textbf{q} - \ln \textbf{A} - \ln \textbf{B} - \textbf{1} \\
&\implies \textbf{q}^* = \sigma(\ln \textbf{A} + \ln \textbf{B})
\end{align}
$$

Where $\sigma$ is a softmax function $\sigma(x) = \frac{e^x}{\sum_x e^x}$ which ensures that the resulting probability distribution is normalized. This expression lets us compute the optimal approximate posterior instantly as a straightforward function of the current observation $o_t$, $\textbf{A}$, and $\textbf{B}$. We can thus quickly right the code for inference:

```python
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x))

def perform_inference(likelihood, prior):
    return softmax(log_stable(likelihood) + log_stable(prior))
```

Note that the likelihood term is not the entire $\textbf{A}$ matrix, but just the 'row' of the $\textbf{A}$ matrix corresponding to the current observation, i.e. $P(o_t = o \mid x_t)$.

Inference for simple discrete state-space models like these is therefore very simple. All we need to do is have some initial set of beliefs $\textbf{q}_0$ and then update them according to these rules for every observation we get, using the $\textbf{A}$, $\textbf{B}$, the past action $a_t$ and the past posterior $q(x_{t-1})$ to provide the likelihood and prior terms.

# Planning through Active Inference

So far we just have an agent that can perform perform inference in the discrete state space, but inference by itself isn't really that useful. Instead what we really want to do is *planning*. That is, the agent needs to be able to figure out how to emit a series of actions which will take it to a certain goal state. A key part of Active Inference is that this process of planning, or more broadly action selection, can also be solved as a process of variational inference. This is why it is called *Active* Inference, after all.

<!-- However, when starting to think about this, it is not immediately obvious how to turn planning into an inference problem. What are the hypotheses? What are the observations? To turn the problem of planning into an inference problem, we need to introduce two additional concepts. The first is the idea of a *goal state*. To make planning useful, the agent has to *want something*. This is different from just performing objective inference about the state of the world in which there is no goal except to infer correctly. In Active Inference we define the goal state as a separate goal vector denoted $\textbf{C}$. When performing planning we then modify the generative model of the agent so that it no longer reflects the true distribution of observations in the environment, but rather includes the goal vector. We denote this new generative model $\tilde{p}(o_t, x_t) = p(o_t | x_t)\tilde{p}(x_t)$ where we use $\tilde{p}$ to say that this distribution is not a true distribution describing the agents model of the world, but is instead a *desired distribution*. Here, we set the desired distribution to be over the state of the environment $x_t$ such that $\tilde{p}(x_t) = \textbf{C}$. -->

However, when starting to think about this, it is not immediately obvious how to turn planning into an inference problem. What are the hypotheses? What are the observations? To turn the problem of planning into an inference problem, we need to introduce two additional concepts. The first is the idea of a *preferred observations*. To make planning useful, the agent has to *want something*. This is different from just performing objective inference about the state of the world in which there is no goal except to infer correctly. In Active Inference, we define the preferred observations as a separate goal vector denoted $\textbf{C}$. When performing planning we then modify the generative model of the agent so that it no longer reflects the true distribution of observations in the environment, but rather includes the goal vector, which is a *prior over observations*. We denote this new generative model $\tilde{p}(o_t, x_t) = p(x_t | o_t)\tilde{p}(o_t)$ where we use $\tilde{p}$ to say that this distribution is not a true distribution describing the agent's model of the world, but is instead a *preference distribution*. Here, we set the preference distribution to be over the state of the environment $o_t$ such that $\tilde{p}(o_t) = \textbf{C}$.

By changing the generative model in this way, we have effectively changed the inference problem from: *infer the most likely states and actions given the true generative model of the world* to *infer the most likely states and actions given a false model of the world, in which I achieve my goals*. Perhaps more intuitively, we can think of this inference problem as answering the question: *Given that I have achieved my goals, what actions must have I taken to get there?*.

The second thing we need to do to perform planning is to also extend the inference problem to *actions in the future*, since these are the fundamental things that the agent controls which it can use to adjust the environment. We call a sequence of future actions from now (time $t$) until some set future time $T$ a *policy* and denote it $\pi = [a_t, a_{t+1}, a_{t+2} \dots a_T]$. The goal is then to infer the optimal policy $\pi^*$ given the preferences $\textbf{C}$.

However here there is a problem. Typically we would perform variational inference to solve this inference problem, but the variational free energy is not defined over future trajectories of observations, which are not yet known. Instead, we define a new objective which can handle this -- the *Expected Free Energy (EFE)*, which is defined over policies and which we denote as $\mathcal{G}(\pi)$. We define the EFE for a particular timepoint $\tau$ and policy $\pi$ as:

$$
\begin{align}
\mathcal{G}(\pi)_{\tau} = \sum_{o_{\tau}, x_{\tau}} q(x_{\tau} | \pi)q(o_{\tau} | \pi) \big[ \ln q(x_{\tau} | \pi) - \ln \tilde{p}(o_{\tau}, x_{\tau}) \big]
\end{align}
$$

The expected free energy is defined for a single time-step of a trajectory *in the future*, i.e. prior to receiving any observations. The key difference between the standard variational free energy and the expected free energy is that the expected free energy also averages over the *expected observations* $q(o_{\tau} | pi)$ and *expected states* $q(x_{\tau} | \pi)$, where the expectations are conditioned on some policy $\pi$. This is necessary because with the expected free energy, we are evaluating possible future trajectories without a given observation, unlike the variational free energy where we can assume that we have already received the observation.

To get an intuitive handle on what the expected free energy *means* we can decompose it into two more intuitive quantities.

$$
\begin{align}
\mathcal{G}(\pi)_{\tau}&= \sum_{o_{\tau}, x_{\tau}} q(x_{\tau} | \pi)q(o_{\tau} | \pi) \big[ \ln q(x_{\tau} | \pi) - \ln \tilde{p}(o_{\tau}, x_{\tau}) \big] \\
&= -\underbrace{\sum_{o_{\tau}, x_{\tau}} q(x_{\tau} | \pi)q(o_{\tau} | \pi) \big[ \ln p(o_{\tau} | x_{\tau}) \big]}_{\text{Uncertainty}} + \underbrace{KL[q(x_{\tau} | \pi) || \tilde{p}(x_{\tau}) ]}_{\text{Divergence}}
\end{align}
$$

The first term is called *expected uncertainty* or sometimes *novelty* and represents essentially the spread of the observations expected in the future. Since we are choosing actions that *minimize* the whole quantity, we want to take actions that *maximize* novelty. We can think of this as a bonus to aid exploration since active inference agents will preferentially pursue resolveable uncertainty. The second term is the *divergence* term which is the KL divergence between the states expected under a given policy (also known as the posterior predictive density $q(x_\tau | \pi)$) and the goal distribution of the generative model, $\tilde{p}(x)$. This term scores how far away (in an informational sense) the agent expects it will be from the goal, if it were to pursue that policy. Since we are minimizing the expected free energy this term is positive and so is also minimized -- that is by minimizing the expected free energy, we are trying to choose trajectories which will bring the expected states in the future close to the desired states.

Now that we have the expected free energy to score possible trajectories, we now need to infer the optimal policy. A simple approach (which can be derived explicitly although it is somewhat complex) is to say that the posterior probability of a policy is proportional to the (exponentiated) sum of the expected free energy accumulated along the trajectory through the environment created by that policy. While this sounds complex, mathematically we can express it very simply as,
$$
\begin{align}
q(\pi) = \sigma( \sum_{\tau}^T \mathcal{G}(\pi)_{\tau})
\end{align}
$$
<!-- We can then choose which policy we implement for the next timestep by just sampling a policy from $q(\pi)$ and then emitting the first action of that policy. -->
We can then choose which action we choose for the next timestep by computing the marginal probability of each action $P(a)$, given the posterior over actions $q(\pi)$ and some mapping between policies and actions $P(a | \pi)$:

$$
\begin{align}
P(a) = \sum_{\pi}P(a | \pi)q(\pi)
\end{align}
$$

We can sample from this distribution over actions to choose our action for the next timestep.

In our simple grid world environment, policies are identical to actions since we are only planning ahead 1-step in the future, so $\forall_{i} P(a = i | \pi = i) = 1$, which implies $P(a) == q(\pi)$.

While all this may seem exceptionally long and complex, it results in an algorithm which is actually remarkably simple. The algorithm is:

1.) There is an agent with a generative model of the environment ($\textbf{A}$ and $\textbf{B}$ matrices), some initial set of approximate posterior beliefs $q(x_t | o_t)$ and a desired state vector $\textbf{C}$.

2.) The agent receives an observation $o_t$ and computes its posterior beliefs as we did earlier by minimizing the free energy.

3.) The agent now needs to choose what action to make to achieve its goals. It does this by:

 3.1.) First creating a set of potential policies to evaluate.

   3.2.) For each policy in this set, use the generative model to simulate the agent's trajectory in the environment *as if* it had emitted the actions prescribed by the policy

   3.3.) For each future timestep of each future trajectory, compute the expected free energy of that time-step

   3.4.) Sum the expected free energies for each timestep of each trajectory to get a total expected free energy for each possible policy.

   3.5.) Use these total expected free energies to compute the posterior distribution $q(\pi)$ as done above.

   3.6 ) Compute the marginal probability of each action, expected under the policies that include them, and then sample from this "action marginal" to generate an action at the current timestep.

4.) Execute the sampled action to 'step forward' the environmental dynamics and get a new observation. Go back to step 1.

And that's it! We're done. We have the full algorithm to create an active inference agent. Now all we do is show how to translate this algorithm into code for our specific case.

## Beliefs

First we need to setup an initial belief distribution which we will then update according to the observations we will receive.

```python
# setup initial prior beliefs -- uncertain -- completely unknown which state it is in
Qs = np.ones(9) * 1/9
plot_beliefs(Qs)
```

# Preferences

Now we have to encode the agent's preferences, so that it can learn to go to its reward state. In the current context, the agent wants (i.e. expects) to be in the reward location 7.

```python
# C matrix -- desires

REWARD_LOCATION = 7
reward_state = state_mapping[REWARD_LOCATION]
print(reward_state)

C = np.zeros(num_states)
C[REWARD_LOCATION] = 1.
print(C)
plot_beliefs(C)
```

The C matrix is a 1x9 matrix, where each value represents the preference to occupy a given state. We will create a one-hot C matrix, so that the agent only has a preference to be in state 7.

# Implementing the Active Inference Agent

## Evaluate policy

This helper function we evaluate the negative expected free energy for a given policy $-\mathcal{G}(\pi)$. To do this we need to calculate the cumulative expected free energy for that policy. All that entails is looping through the timesteps of the policy, simulate what the environment would do, using our generative model, evolving posterior beliefs and the actions entailed by the policy, and then compute the expected free energy of our (policy-dependent) expectations about the hidden states and observations.

```python
def evaluate_policy(policy, Qs, A, B, C):
    # initialize expected free energy at 0
    G = 0

    # loop over policy
    for t in range(len(policy)):

        # get action entailed by the policy at timestep `t`
        u = int(policy[t])

        # work out expected state, given the action
        Qs_pi = B[:,:,u].dot(Qs)

        # work out expected observations, given the action
        Qo_pi = A.dot(Qs_pi)

        # get entropy
        H = - (A * log_stable(A)).sum(axis = 0)

        # get predicted divergence
        # divergence = np.sum(Qo_pi * (log_stable(Qo_pi) - log_stable(C)), axis=0)
        divergence = KL_divergence(Qo_pi, C)

        # compute the expected uncertainty or ambiguity
        uncertainty = H.dot(Qs_pi)

        # increment the expected free energy counter for the policy, using the expected free energy at this timestep
        G += (divergence + uncertainty)

    return -G
```

## Infer action

This helper function will infer the most likely action. Specifically, it computes steps 3.1 to 3.5 in the active inference algorithm. First, it constructs all possible policies for a given policy length and set of actions. Then it loops through every possible policy and computes the expected free energy of that policy using our previous function, and then computing the policy distribution $q(\pi)$ using the softmax over the expected free energies.

```python
def infer_action(Qs, A, B, C, n_actions, policies):

    # initialize the negative expected free energy
    neg_G = np.zeros(len(policies))

    # loop over every possible policy and compute the EFE of each policy
    for i, policy in enumerate(policies):
        neg_G[i] = evaluate_policy(policy, Qs, A, B, C)

    # get distribution over policies
    Q_pi = maths.softmax(neg_G)

    # initialize probabilites of control states (convert from policies to actions)
    Qu = np.zeros(n_actions)

    # sum probabilites of control states or actions
    for i, policy in enumerate(policies):
        # control state specified by policy
        u = int(policy[0])
        # add probability of policy
        Qu[u] += Q_pi[i]

    # normalize action marginal
    utils.norm_dist(Qu)

    # sample control from action marginal
    u = utils.sample(Qu)

    return u
```

## Main loop

Here we simply implement the main loop of the active inference agent interacting with the environment. Specifically, this essentially implements steps 1-5 of the MDP "program" we discussed in notebook 1. Specifically, for each timestep, the agent infers an action, it emits that action to the environment, the environment is updated and returns an observation to the agent. The agent then infers the new state of the environment given that observation.

```python
# number of time steps
T = 10

#n_actions = env.n_control
n_actions = 5

# length of policies we consider
policy_len = 4

# this function generates all possible combinations of policies
policies = control.construct_policies([B.shape[0]], [n_actions], policy_len)

# reset environment
o = env.reset()

# loop over time
for t in range(T):

    # infer which action to take
    a = infer_action(Qs, A, B, C, n_actions, policies)

    # perform action in the environment and update the environment
    o = env.step(int(a))

    # infer new hidden state (this is the same equation as above but with PyMDP functions)
    likelihood = A[o,:]
    prior = B[:,:,int(a)].dot(Qs)

    Qs = maths.softmax(log_stable(likelihood) + log_stable(prior))

    print(Qs.round(3))
    plot_beliefs(Qs, "Beliefs (Qs) at time {}".format(t))
```

And that's it! In the last two notebooks, we have implemented a basic active inference agent which can successfully navigate around a 3x3 gridworld using active inference. Moreover, we have implemented this all from scratch using mostly basic numpy functions and not using much of the functionality of `pymdp`.

Hopefully after going through this you now understand roughly what active inference is and how it works, as well as ideally have some intuitions about how inference as well as policy selection work "under the hood", as well as learnt a lot about Bayesian and specifically variational inference. In the next notebook, we will focus more on the `pymdp` library itself and demonstrate how `pymdp` provides a useful set of abstractions that allows us to easily create active inference agents, as well as perform inference and policy selection in considerably more complex environments than the one described here.

We will discuss the high level structure of the library and show how its possible to replicate these notebooks in a much smaller amount of code using the `pymdp` abstractions.


# Imports and Simulation Parameters

```python
import numpy as np
import math
import cmath
import scipy
import scipy.integrate
import sys

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

hbar = 1.0 / (2.0 * np.pi)

ZERO_TOLERANCE = 10**-6
```

    /Users/joseph/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

```python
MAX_VIBRATIONAL_STATES = 200

STARTING_GROUND_STATES = 5
STARTING_EXCITED_STATES = 5

time_scale_set = 10  #will divide the highest energy to give us the time step
low_frequency_cycles = 25.0 #will multiply the frequency of the lowest frequency mode to get the max time
```

```python
#See if a factorial_Storage dictionary exists already and if not, create one
try:
    a = factorial_storage
except:
    factorial_storage = {}
```

# Defining Parameters of the System

```python
energy_g = 0
energy_gamma = .1

energy_e = 0
energy_epsilon = .31

Huang_Rhys_Parameter = .80
S = Huang_Rhys_Parameter

#bookkeeping
overlap_storage = {}
electronic_energy_gap = energy_e + .5*energy_epsilon - (energy_g  + .5 * energy_gamma)
min_energy = energy_g + energy_gamma * .5

```

```python
mu_0 = 1.0
```

If we set the central frequency of a pulse at the 0->0 transition, and we decide we care about the ratio of the 0->1 transition to the 0->0 transition and set that to be $\tau$ then the desired pulse width will be
\begin{align}
    \sigma &= \sqrt{-\frac{2 \ln (\tau)}{\omega_{\epsilon}^2}}
\end{align}

```python
def blank_wavefunction(number_ground_states, number_excited_states):
    return np.zeros((number_ground_states + number_excited_states))

def perturbing_function(time):
#     stdev = 30000.0 * dt #very specific to 0->0 transition!
    stdev = 3000.0 * dt #clearly has a small amount of amplitude on 0->1 transition
    center = 6 * stdev
    return np.cos(electronic_energy_gap*(time - center) / hbar)*np.exp( - (time - center)**2 / (2 * stdev**2)) / stdev

def time_function_handle_from_tau(tau_proportion):
    stdev = np.sqrt( -2.0 * np.log(tau_proportion) / (energy_epsilon/hbar)**2)
    center = 6 * stdev
    return center, stdev, lambda t: np.cos(electronic_energy_gap*(t - center) / hbar)*np.exp( - (t - center)**2 / (2 * stdev**2)) / stdev

def time_function_handle_from_tau_and_kappa(tau_proportion, kappa_proportion):
    stdev = np.sqrt( -2.0 * np.log(tau_proportion) / (energy_epsilon/hbar)**2)
    center = 6 * stdev
    return center, stdev, lambda t: kappa_proportion * energy_gamma * np.cos(electronic_energy_gap*(t - center) / hbar)*np.exp( - (t - center)**2 / (2 * stdev**2)) / stdev

def perturbing_function_define_tau(time, tau_proportion):
    center, stdev, f = time_function_handle_from_tau(tau_proportion)
    return f(time)
```

```python

```

```python

```

# Defining Useful functions

$ O_{m}^{n} = \left(-1\right)^{n} \sqrt{\frac{e^{-S}S^{m+n}}{m!n!}} \sum_{j=0}^{\min \left( m,n \right)} \frac{m!n!}{j!(m-j)!(n-j)!}(-1)^j S^{-j} $

```python
def factorial(i):
    if i in factorial_storage:
        return factorial_storage[i]
    if i <= 1:
        return 1.0
    else:
        out =  factorial(i - 1) * i
        factorial_storage[i] = out
        return out

def ndarray_factorial(i_array):
    return np.array([factorial(i) for i in i_array])
```

```python
def overlap_function(ground_quantum_number, excited_quantum_number):
    m = ground_quantum_number
    n = excited_quantum_number
    if (m,n) in overlap_storage:
        return overlap_storage[(m,n)]
    output = (-1)**n
    output *= math.sqrt(math.exp(-S) * S**(m + n) /(factorial(m) * factorial(n)) )
    j_indeces = np.array(range(0, min(m,n) + 1))
    j_summation = factorial(m) * factorial(n) * np.power(-1.0, j_indeces) * np.power(S, -j_indeces)
    j_summation = j_summation / (ndarray_factorial(j_indeces) * ndarray_factorial( m - j_indeces) * ndarray_factorial(n - j_indeces) )
    output *= np.sum(j_summation)
    overlap_storage[(m,n)] = output
    return output
```

# Solving the Differential Equation

\begin{align*}
	\left(\frac{d G_a(t)}{dt} + \frac{i}{\hbar}\Omega_{(a)}\right) &=-E(t)\frac{i}{\hbar}  \sum_{b} E_b(t) \mu_{a}^{b}\\
	\left(\frac{d E_b(t)}{dt} + \frac{i}{\hbar} \Omega^{(b)} \right) &=-E(t)\frac{i}{\hbar} \sum_{a} G_a(t) \mu_{a}^{b}
\end{align*}
Or in a more compact form:
\begin{align*}
	\frac{d}{dt}\begin{bmatrix}
		G_a(t) \\
		E_b(t)
	\end{bmatrix}
	= -\frac{i}{\hbar}
	\begin{bmatrix}
		\Omega_{(a)} & E(t) \mu_{a}^{b} \\
		E(t) \mu_{a}^{b} & \Omega^{b}
	\end{bmatrix}
	\cdot
	\begin{bmatrix}
		G_a(t) \\
		E_b(t)
	\end{bmatrix}
\end{align*}

```python
def ode_diagonal_matrix(number_ground_states, number_excited_states):
    #Define the Matrix on the RHS of the above equation
    ODE_DIAGONAL_MATRIX = np.zeros((number_ground_states + number_excited_states, number_ground_states + number_excited_states), dtype=np.complex)
    #set the diagonals
    for ground_i in range(number_ground_states):
        ODE_DIAGONAL_MATRIX[ground_i, ground_i] = -1.0j * (energy_g + energy_gamma * (ground_i + .5)) / hbar

    for excited_i in range(number_excited_states):
        excited_index = excited_i + number_ground_states #the offset since the excited states comes later
        ODE_DIAGONAL_MATRIX[excited_index, excited_index] = -1.0j * (energy_e + energy_epsilon * (excited_i + .5)) / hbar

    return ODE_DIAGONAL_MATRIX

#now for the off-diagonals
def mu_matrix(c, number_ground_states, number_excited_states):
    MU_MATRIX = np.zeros((number_ground_states, number_excited_states), dtype = np.complex)
    for ground_a in range(number_ground_states):
        for excited_b in range(number_excited_states):
            new_mu_entry = overlap_function(ground_a, excited_b)
            if ground_a >0:
                new_mu_entry += c * math.sqrt(ground_a) * overlap_function(ground_a - 1, excited_b)

            new_mu_entry += c * math.sqrt(ground_a+1) * overlap_function(ground_a + 1, excited_b)

            MU_MATRIX[ground_a, excited_b] = new_mu_entry
    return MU_MATRIX

def ode_off_diagonal_matrix(c_value, number_ground_states, number_excited_states):
    output  = np.zeros((number_ground_states + number_excited_states, number_ground_states + number_excited_states), dtype=np.complex)
    MU_MATRIX = mu_matrix(c_value, number_ground_states, number_excited_states)
    output[0:number_ground_states, number_ground_states:] = -1.0j * mu_0 * MU_MATRIX  / hbar
    output[number_ground_states:, 0:number_ground_states] = -1.0j * mu_0 * MU_MATRIX.T  / hbar
    return output

def IR_transition_dipoles(number_ground_states, number_excited_states):
    "outputs matrices to calculate ground and excited state IR emission spectra.  Can be combined for total"
    output_g = np.zeros((number_ground_states + number_excited_states, number_ground_states + number_excited_states), dtype=np.complex)
    output_e = np.zeros((number_ground_states + number_excited_states, number_ground_states + number_excited_states), dtype=np.complex)

    for ground_a in range(number_ground_states):
        try:
            output_g[ground_a, ground_a + 1] = math.sqrt(ground_a + 1)
            output_g[ground_a + 1, ground_a] = math.sqrt(ground_a + 1)
        except:
            pass

        try:
            output_g[ground_a, ground_a - 1] = math.sqrt(ground_a)
            output_g[ground_a - 1, ground_a] = math.sqrt(ground_a)
        except:
            pass

    for excited_a in range(number_excited_states):
        matrix_index_e = number_ground_states + excited_a -1 #because of how 'number_ground_states' is defined
        try:
            output_e[matrix_index_e, matrix_index_e + 1] = math.sqrt(excited_a + 1)
            output_e[matrix_index_e + 1, matrix_index_e] = math.sqrt(excited_a + 1)
        except:
            pass

        try:
            output_e[matrix_index_e, matrix_index_e - 1] = math.sqrt(excited_a)
            output_e[matrix_index_e - 1, matrix_index_e] = math.sqrt(excited_a)
        except:
            pass

    return output_g, output_e
```

\begin{align*}
	\mu(x) &= \mu_0 \left(1 + \lambda x \right) \\
	&= \mu_0 \left(1 + c\left(a + a^{\dagger} \right) \right) \\
	\mu_{a}^{b} &= \mu_0\left(O_{a}^{b}  + c\left(\sqrt{a}O_{a-1}^{b} + \sqrt{a+1}O_{a+1}^{b}\right) \right)
\end{align*}

```python

```

```python
class VibrationalStateOverFlowException(Exception):
    def __init__(self):
        pass
```

```python

```

```python
def propagate_amplitude_to_end_of_perturbation(c_value, ratio_01_00, kappa=1, starting_ground_states=STARTING_GROUND_STATES, starting_excited_states=STARTING_EXCITED_STATES):
    center_time, stdev, time_function = time_function_handle_from_tau_and_kappa(ratio_01_00, kappa)
    ending_time = center_time + 8.0 * stdev

    number_ground_states = starting_ground_states
    number_excited_states = starting_excited_states

    while number_excited_states + number_ground_states < MAX_VIBRATIONAL_STATES:

        #define time scales
        max_energy = energy_e + energy_epsilon * (.5 + number_excited_states) + kappa * energy_gamma * mu_0
        dt = 1.0 / (time_scale_set *  max_energy)

        ODE_DIAGONAL = ode_diagonal_matrix(number_ground_states, number_excited_states)
        ODE_OFF_DIAGONAL = ode_off_diagonal_matrix(c_value, number_ground_states, number_excited_states)

        def ODE_integrable_function(time, coefficient_vector):
            ODE_TOTAL_MATRIX = ODE_OFF_DIAGONAL * time_function(time) + ODE_DIAGONAL
            return np.dot(ODE_TOTAL_MATRIX, coefficient_vector)

        #define the starting wavefuntion
        initial_conditions = blank_wavefunction(number_ground_states, number_excited_states)
        initial_conditions[0] = 1

        #create ode solver
        current_time = 0.0
        ode_solver = scipy.integrate.complex_ode(ODE_integrable_function)
        ode_solver.set_initial_value(initial_conditions, current_time)

        #Run it
        results = []
        try:  #this block catches an overflow into the highest ground or excited vibrational state
            while current_time < ending_time:
#                 print(current_time, ZERO_TOLERANCE)
                #update time, perform solution
                current_time = ode_solver.t+dt
                new_result = ode_solver.integrate(current_time)
                results.append(new_result)
                #make sure solver was successful
                if not ode_solver.successful():
                    raise Exception("ODE Solve Failed!")
                #make sure that there hasn't been substantial leakage to the highest excited states
                re_start_calculation = False
                if abs(new_result[number_ground_states - 1])**2 >= ZERO_TOLERANCE:
                    number_ground_states +=1
#                     print("Increasing Number of Ground vibrational states to %i " % number_ground_states)
                    re_start_calculation = True
                if abs(new_result[-1])**2 >= ZERO_TOLERANCE:
                    number_excited_states +=1
#                     print("Increasing Number of excited vibrational states to %i " % number_excited_states)
                    re_start_calculation = True
                if re_start_calculation:
                    raise VibrationalStateOverFlowException()
        except VibrationalStateOverFlowException:
            #Move on and re-start the calculation
            continue

        #Finish calculating
        results = np.array(results)
        return results, number_ground_states, number_excited_states
    raise Exception("NEEDED TOO MANY VIBRATIONAL STATES!  RE-RUN WITH DIFFERENT PARAMETERS!")
```

```python
def get_average_quantum_number_time_series(c_value, ratio_01_00, kappa=1, starting_ground_states=STARTING_GROUND_STATES, starting_excited_states=STARTING_EXCITED_STATES):
    results, number_ground_states, number_excited_states = propagate_amplitude_to_end_of_perturbation(c_value, ratio_01_00, kappa, starting_ground_states, starting_excited_states)
    probabilities = np.abs(results)**2
    #calculate the average_vibrational_quantum_number series
    average_ground_quantum_number = probabilities[:,0:number_ground_states].dot(np.array(range(number_ground_states)) )
    average_excited_quantum_number = probabilities[:,number_ground_states:].dot(np.array(range(number_excited_states)))
    return average_ground_quantum_number, average_excited_quantum_number, results, number_ground_states, number_excited_states
```

```python
def IR_emission_spectrum_after_excitation(c_value, ratio_01_00, kappa=1, starting_ground_states=STARTING_GROUND_STATES, starting_excited_states=STARTING_EXCITED_STATES):
    center_time, stdev, time_function = time_function_handle_from_tau_and_kappa(ratio_01_00, kappa)

    perturbation_ending_time = center_time + 8.0 * stdev
    simulation_ending_time = perturbation_ending_time + low_frequency_cycles * hbar/min_energy

    number_ground_states = starting_ground_states
    number_excited_states = starting_excited_states

    while number_excited_states + number_ground_states < MAX_VIBRATIONAL_STATES:

        ir_transDipole_g, ir_transDipole_e = IR_transition_dipoles(number_ground_states, number_excited_states)

        time_emission_g = [0]
        time_emission_e = [0]

        #define time scales
        e = energy_e + energy_epsilon * (.5 + number_excited_states)
        g = energy_g + energy_gamma* (.5 + number_ground_states)
        plus = e + g
        minus = e - g
        J = kappa * energy_gamma * mu_0
        max_split_energy = plus + math.sqrt(minus**2 + 4 * J**2)
        max_energy = max_split_energy * .5
        dt = 1.0 / (time_scale_set *  max_energy)

        time_values = np.arange(0, simulation_ending_time, dt)

        ODE_DIAGONAL = ode_diagonal_matrix(number_ground_states, number_excited_states)
        ODE_OFF_DIAGONAL = ode_off_diagonal_matrix(c_value, number_ground_states, number_excited_states)

        def ODE_integrable_function(time, coefficient_vector):
            ODE_TOTAL_MATRIX = ODE_OFF_DIAGONAL * time_function(time) + ODE_DIAGONAL
            return np.dot(ODE_TOTAL_MATRIX, coefficient_vector)

        def ODE_jacobean(time, coefficient_vector):
            ODE_TOTAL_MATRIX = ODE_OFF_DIAGONAL * time_function(time) + ODE_DIAGONAL
            return ODE_TOTAL_MATRIX

        #define the starting wavefuntion
        initial_conditions = blank_wavefunction(number_ground_states, number_excited_states)
        initial_conditions[0] = 1

        #create ode solver
        current_time = 0.0
        try:
            del ode_solver
        except:
            pass
#         ode_solver = scipy.integrate.complex_ode(ODE_integrable_function)
        ode_solver = scipy.integrate.complex_ode(ODE_integrable_function, jac = ODE_jacobean)
#         ode_solver.set_integrator("lsoda")
        ode_solver.set_integrator("vode", with_jacobian=True)

        ode_solver.set_initial_value(initial_conditions, current_time)

        #Run it
        results = []
        try:  #this block catches an overflow into the highest ground or excited vibrational state
            while current_time < simulation_ending_time:
#                 print(current_time, ZERO_TOLERANCE)
                #update time, perform solution
                current_time = ode_solver.t+dt
                new_result = ode_solver.integrate(current_time)
                results.append(new_result)
                #make sure solver was successful
                if not ode_solver.successful():
                    raise Exception("ODE Solve Failed!")

                if current_time < perturbation_ending_time:
                    #make sure that there hasn't been substantial leakage to the highest excited states
                    re_start_calculation = False
                    if abs(new_result[number_ground_states - 1])**2 >= ZERO_TOLERANCE:
                        number_ground_states +=1
#                         print("Increasing Number of Ground vibrational states to %i " % number_ground_states)
                        re_start_calculation = True
                    if abs(new_result[-1])**2 >= ZERO_TOLERANCE:
                        number_excited_states +=1
#                         print("Increasing Number of excited vibrational states to %i " % number_excited_states)
                        re_start_calculation = True
                    if re_start_calculation:
                        raise VibrationalStateOverFlowException()
                #calculate IR emission
                time_emission_g.append(np.conj(new_result).T.dot(ir_transDipole_g.dot(new_result)))
                time_emission_e.append(np.conj(new_result).T.dot(ir_transDipole_e.dot(new_result)))
                #on to next time value...

        except VibrationalStateOverFlowException:
            #Move on and re-start the calculation
            continue

        #Finish calculating
        results = np.array(results)
        n_t = len(time_emission_e)

        time_emission_g = np.array(time_emission_g)
        time_emission_e = np.array(time_emission_e)

        filter_x = np.array(range(n_t))
        filter_center = n_t / 2.0
        filter_sigma = n_t / 10.0
        filter_values = np.exp(-(filter_x - filter_center)**2 / (2 * filter_sigma**2))

        frequencies = np.fft.fftshift(np.fft.fftfreq(time_emission_g.shape[0], d= dt))
        frequency_emission_g = dt * np.fft.fftshift(np.fft.fft(time_emission_g * filter_values))
        frequency_emission_e = dt * np.fft.fftshift(np.fft.fft(time_emission_e * filter_values))

        return results, frequencies, frequency_emission_g, frequency_emission_e, number_ground_states, number_excited_states
    raise Exception("NEEDED TOO MANY VIBRATIONAL STATES!  RE-RUN WITH DIFFERENT PARAMETERS!")
```

```python
c_values = np.logspace(-2.5, np.log10(1), 15)
tau_values = np.logspace(-3, np.log10(.9), 3)
kappa_values = np.logspace(-1, np.log10(5), 22)

number_calcs = c_values.shape[0] * tau_values.shape[0] * kappa_values.shape[0]

heating_results_ground = np.zeros((kappa_values.shape[0],  tau_values.shape[0], c_values.shape[0]))
ir_amplitudes = np.zeros(heating_results_ground.shape)
heating_results_excited = np.zeros(heating_results_ground.shape)

# Keep track of the IR Spectrum

n_g = STARTING_GROUND_STATES
n_e = STARTING_EXCITED_STATES

counter = 1
# we will use the value of c as a bellweather for how many starting states to work with.
c_to_ng = {}
c_to_ne = {}
for i_kappa, kappa in enumerate(kappa_values):
    # as we increase in both tau and
    for i_tau, tau in enumerate(tau_values):

        for i_c, c in enumerate(c_values):
            try:
                n_g = c_to_ng[c]
                n_e = c_to_ne[c]
            except:
                n_g = STARTING_GROUND_STATES
                n_e = STARTING_EXCITED_STATES
                c_to_ng[c] = n_g
                c_to_ne[c] = n_e

            sys.stdout.flush()
            sys.stdout.write("\r%i / %i Calculating kappa=%f, c=%f, tau=%f at n_g = %i and n_e=%i..." %(counter, number_calcs, kappa, c, tau, n_g, n_e))
#             print("\r%i / %i Calculating kappa=%f, c=%f, tau=%f at n_g = %i and n_e=%i..." %(counter, number_calcs, kappa, c, tau, n_g, n_e))
#             n_bar_g, n_bar_e, results, num_g, num_e = get_average_quantum_number_time_series(c,
#                                                                                tau,
#                                                                                kappa,
#                                                                                starting_ground_states = n_g,
#                                                                                starting_excited_states = n_e)

#             heating_results_ground[i_kappa, i_tau, i_c] = n_bar_g[-1]
#             heating_results_excited[i_kappa, i_tau, i_c] = n_bar_e[-1]

            _, frequencies, emission_g, emission_e, num_g, num_e = IR_emission_spectrum_after_excitation(c,
                                                                                tau,
                                                                                kappa,
                                                                                starting_ground_states = n_g,
                                                                                starting_excited_states = n_e)
            if num_g > c_to_ng[c]:
                c_to_ng[c] = num_g
            if num_e > c_to_ne[c]:
                c_to_ne[c] = num_e
            vibrational_frequency_index = np.argmin(np.abs(energy_gamma - frequencies))
            ir_power = np.abs(emission_g[vibrational_frequency_index])**2
            ir_amplitudes[i_kappa, i_tau, i_c] = ir_power
            counter +=1
#             plt.figure()
#             plt.title(r"$\kappa{}, \tau={}, c={}".format(kappa, tau, c))
#             plt.plot(frequencies, emission_g)
#             plt.xlim(0, 2)

```

    990 / 990 Calculating kappa=5.000000, c=1.000000, tau=0.900000 at n_g = 75 and n_e=60...

```python
print(c_to_ng)
print(c_to_ne)
```

    {0.056234132519034911: 11, 0.010857111194022039: 9, 0.1279802213997954: 14, 1.0: 96, 0.43939705607607904: 37, 0.004770582696143927: 9, 0.016378937069540647: 9, 0.024709112279856043: 10, 0.0031622776601683794: 9, 0.084834289824407216: 12, 0.0071968567300115215: 9, 0.29126326549087383: 24, 0.037275937203149402: 10, 0.19306977288832505: 18, 0.6628703161826448: 58}
    {0.056234132519034911: 9, 0.010857111194022039: 9, 0.1279802213997954: 11, 1.0: 79, 0.43939705607607904: 27, 0.004770582696143927: 9, 0.016378937069540647: 9, 0.024709112279856043: 9, 0.0031622776601683794: 9, 0.084834289824407216: 9, 0.0071968567300115215: 9, 0.29126326549087383: 18, 0.037275937203149402: 9, 0.19306977288832505: 13, 0.6628703161826448: 45}

```python
# decreased dt, does that keep these parameters from failing? last one is 600 / 800 Calculating kappa=8.858668, c=0.750000, tau=0.850000
for i_kappa, kappa in enumerate(kappa_values):
    for i_tau, tau in enumerate(tau_values):
        ir_power = ir_amplitudes[i_kappa, i_tau, :]
        plt.loglog(c_values, ir_power, "*-")
plt.xlabel(r"$c$")
plt.figure()
for i_kappa, kappa in enumerate(kappa_values):
    for i_c, c in enumerate(c_values):
        ir_power = ir_amplitudes[i_kappa, :, i_c]
        plt.loglog(tau_values, ir_power, "*-")
plt.xlabel(r"$\tau$")
plt.figure()
for i_tau, tau in enumerate(tau_values):
    for i_c, c in enumerate(c_values):
#     for i_c in [0,-1]:
        ir_power = ir_amplitudes[:, i_tau, i_c]
#         plt.loglog(kappa_values, ir_power, ["blue", "red"][i_c])
        plt.loglog(kappa_values, ir_power)
plt.xlabel(r"$\kappa$")
# plt.xlim(-.1, 1.1)
# plt.ylim(-10, 500)
```

```python
c_log = np.log10(c_values)
tau_log = np.log10(tau_values)
kappa_log = np.log10(kappa_values)
log_ir_amplitudes = np.log(ir_amplitudes)
num_levels = 100
contours = np.linspace(np.min(log_ir_amplitudes), np.max(log_ir_amplitudes), num_levels)
```

```python

# for i_kappa, kappa in enumerate(kappa_values):
#     ir_power = log_ir_amplitudes[i_kappa, :, :]
#     plt.figure()
#     plt.contourf(c_log, tau_log, ir_power, contours)
#     plt.title(r"$\kappa = {}$".format(kappa))
#     plt.ylabel(r"$c$")
#     plt.xlabel(r"$\tau$")
#     plt.colorbar()

for i_tau, tau in enumerate(tau_values):
    ir_power = log_ir_amplitudes[:, i_tau, :]
    plt.figure()
    plt.contourf(c_log, kappa_log, ir_power, contours)
    plt.title(r"$\tau = {}$".format(tau))
    plt.ylabel(r"$\kappa$")
    plt.xlabel(r"$c$")
    plt.colorbar()

```

```python
for i_c, c in enumerate(c_values):
    ir_power = log_ir_amplitudes[:, :, i_c]
    plt.figure()
    plt.contourf(tau_log, kappa_log, ir_power, contours)
    plt.title(r"$c = {}$".format(c))
    plt.ylabel(r"$\kappa$")
    plt.xlabel(r"$\tau$")
    plt.colorbar()
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```