     Histograms of Oriented Gradients (HOG)

Introduction

As we saw with the ORB algorithm, we can use keypoints in images to do keypoint-based matching to detect objects in images. These type of algorithms work great when you want to detect objects that have a lot of consistent internal features that are not affected by the background. For example, these algorithms work well for facial detection because faces have a lot of consistent internal features that don’t get affected by the image background, such as the eyes, nose, and mouth. However, these type of algorithms don’t work so well when attempting to do more general object recognition, say for example, pedestrian detection in images. The reason is that people don’t have consistent internal features, like faces do, because the body shape and style of every person is different (see Fig. 1). This means that every person is going to have a different set of internal features, and so we need something that can more generally describe a person.

Fig. 1. - Pedestrians.

One option is to try to detect pedestrians by their contours instead.  Detecting objects in images by their contours (boundaries) is very challenging because we have to deal with the difficulties brought about by the contrast between the background and the foreground. For example, suppose you wanted to detect a pedestrian in an image that is walking in front of a white building and she is wearing a white coat and black pants (see Fig. 2). We can see in Fig. 2, that since the background of the image is mostly white, the black pants are going to have a very high contrast, but the coat, since it is white as well, is going to have very low contrast. In this case, detecting the edges of pants is going to be easy but detecting the edges of the coat is going to be very difficult. This is where **HOG** comes in. HOG stands for **Histograms of Oriented Gradients** and it was first introduced by Navneet Dalal and Bill Triggs in 2005.

Fig. 2. - High and Low Contrast.

The HOG algorithm works by creating histograms of the distribution of gradient orientations in an image and then normalizing them in a very special way. This special normalization is what makes HOG so effective at detecting the edges of objects even in cases where the contrast is very low. These normalized histograms are put together into a feature vector, known as the HOG descriptor, that can be used to train a machine learning algorithm, such as a Support Vector Machine (SVM), to detect objects in images based on their boundaries (edges). Due to its great success and reliability, HOG has become one of the most widely used algorithms in computer vison for object detection.

In this notebook, you will learn:

* How the HOG algorithm works
* How to use OpenCV to create a HOG descriptor
* How to visualize the HOG descriptor.

# The HOG Algorithm

As its name suggests, the HOG algorithm, is based on creating histograms from the orientation of image gradients. The HOG algorithm is implemented in a series of steps:

1. Given the image of particular object, set a detection window (region of interest) that covers the entire object in the image (see Fig. 3).

2. Calculate the magnitude and direction of the gradient for each individual pixel in the detection window.

3. Divide the detection window into connected *cells* of pixels, with all cells being of the same size (see Fig. 3). The size of the cells is a free parameter and it is usually chosen so as to match the scale of the features that want to be detected. For example, in a 64 x 128 pixel detection window, square cells 6 to 8 pixels wide are suitable for detecting human limbs.

4. Create a Histogram for each cell, by first grouping the gradient directions of all pixels in each cell into a particular number of orientation (angular) bins; and then adding up the gradient magnitudes of the gradients in each angular bin (see Fig. 3). The number of bins in the histogram is a free parameter and it is usually set to 9 angular bins.

5. Group adjacent cells into *blocks* (see Fig. 3). The number of cells in each block is a free parameter and all blocks must be of the same size. The distance between each block (known as the stride) is a free parameter but it is usually set to half the block size, in which case you will get overlapping blocks (*see video below*).  The HOG algorithm has been shown empirically to work better with overlapping blocks.

6. Use the cells contained within each block to normalize the cell histograms in that block (see Fig. 3). If you have overlapping blocks this means that most cells will be normalized with respect to different blocks (*see video below*). Therefore, the same cell may have several different normalizations.

7. Collect all the normalized histograms from all the blocks into a single feature vector called the HOG descriptor.

8. Use the resulting HOG descriptors from many images of the same type of object to train a machine learning algorithm, such as an SVM, to detect those type of objects in images. For example, you could use the HOG descriptors from many images of pedestrians to train an SVM to detect pedestrians in images. The training is done with both positive a negative examples of the object you want detect in the image.

9. Once the SVM has been trained, a sliding window approach is used to try to detect and locate objects in images.  Detecting an object in the image entails finding the part of the image that looks similar to the HOG pattern learned by the SVM.

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# Why The HOG Algorithm Works

As we learned above, HOG creates histograms by adding the magnitude of the gradients in particular orientations in localized portions of the image called *cells*.  By doing this we guarantee that stronger gradients will contribute more to the magnitude of their respective angular bin, while the effects of weak and randomly oriented gradients resulting from noise are minimized. In this manner the histograms tell us the dominant gradient orientation of each cell.

### Dealing with contrast

Now, the magnitude of the dominant orientation can vary widely due to variations in local illumination and the contrast between the background and the foreground.

To account for the background-foreground contrast differences, the HOG algorithm tries to detect edges locally.  In order to do this, it defines groups of cells, called **blocks**, and normalizes the histograms using this local group of cells. By normalizing locally, the HOG algorithm can detect the edges in each block very reliably; this is called **block normalization**.

In addition to using block normalization, the HOG algorithm also uses overlapping blocks to increase its performance. By using overlapping blocks, each cell contributes several independent components to the final HOG descriptor, where each component corresponds to a cell being normalized with respect to a different block. This may seem redundant but, it has been shown empirically that by normalizing each cell several times with respect to different local blocks, the performance of the HOG algorithm increases dramatically.

### Loading Images and Importing Resources

The first step in building our HOG descriptor is to load the required packages into Python and to load our image.

We start by using OpenCV to load an image of a triangle tile. Since, the `cv2.imread()` function loads images as BGR we will convert our image to RGB so we can display it with the correct colors. As usual we will convert our BGR image to Gray Scale for analysis.

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

    The original image has shape:  (250, 250, 3)
    The gray scale image has shape:  (250, 250)

    <matplotlib.figure.Figure at 0x7f0a81ff4e48>

# Creating The HOG Descriptor

We will be using OpenCV’s `HOGDescriptor` class to create the HOG descriptor. The parameters of the HOG descriptor are setup using the `HOGDescriptor()` function. The parameters of the `HOGDescriptor()` function and their default values are given below:

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

Parameters:

* **win_size** – *Size*
Size of detection window in pixels (*width, height*). Defines the region of interest. Must be an integer multiple of cell size.

* **block_size** – *Size*
Block size in pixels (*width, height*). Defines how many cells are in each block. Must be an integer multiple of cell size and it must be smaller than the detection window. The smaller the block the finer detail you will get.

* **block_stride** – *Size*
Block stride in pixels (*horizontal, vertical*). It must be an integer multiple of cell size. The `block_stride` defines the distance between adjecent blocks, for example, 8 pixels horizontally and 8 pixels vertically. Longer `block_strides` makes the algorithm run faster (because less blocks are evaluated) but the algorithm may not perform as well.

* **cell_size** – *Size*
Cell size in pixels (*width, height*). Determines the size fo your cell. The smaller the cell the finer detail you will get.

* **nbins** – *int*
Number of bins for the histograms. Determines the number of angular bins used to make the histograms. With more bins you capture more gradient directions. HOG uses unsigned gradients, so the angular bins will have values between 0 and 180 degrees.

* **win_sigma** – *double*
Gaussian smoothing window parameter. The performance of the HOG algorithm can be improved by smoothing the pixels near the edges of the blocks by applying a Gaussian spatial window to each pixel before computing the histograms.

* **threshold_L2hys** – *double*
L2-Hys (Lowe-style clipped L2 norm) normalization method shrinkage. The L2-Hys method is used to normalize the blocks and it consists of an L2-norm followed by clipping and a renormalization. The clipping limits the maximum value of the descriptor vector for each block to have the value of the given threshold (0.2 by default). After the clipping the descriptor vector is renormalized as described in *IJCV*, 60(2):91-110, 2004.

* **gamma_correction** – *bool*
Flag to specify whether the gamma correction preprocessing is required or not. Performing gamma correction slightly increases the performance of the HOG algorithm.

* **nlevels** – *int*
Maximum number of detection window increases.

As we can see, the  `cv2.HOGDescriptor()`function supports a wide range of parameters. The first few arguments (`block_size, block_stride, cell_size`, and `nbins`) are probably the ones you are most likely to change. The other parameters can be safely left at their default values and you will get good results.

In the code below, we will use the `cv2.HOGDescriptor()`function to set the cell size, block size, block stride, and the number of bins for the histograms of the HOG descriptor. We will then use `.compute(image)`method to compute the HOG descriptor (feature vector) for the given `image`.

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

    The gray scale image has shape:  (250, 250)

    HOG Descriptor Parameters:

    Window Size: (246, 246)
    Cell Size: (6, 6)
    Block Size: (12, 12)
    Block Stride: (6, 6)
    Number of Bins: 9

# Number of Elements In The HOG Descriptor

The resulting HOG Descriptor (feature vector), contains the normalized histograms from all cells from all blocks in the detection window concatenated in one long vector. Therefore, the size of the HOG feature vector will be given by the total number of blocks in the detection window, multiplied by the number of cells per block, times the number of orientation bins:

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

If we don’t have overlapping blocks (*i.e.* the `block_stride`equals the `block_size`), the total number of blocks can be easily calculated by dividing the size of the detection window by the block size. However, in the general case we have to take into account the fact that we have overlapping blocks. To find the total number of blocks in the general case (*i.e.* for any `block_stride` and `block_size`), we can use the formula given below:

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

Where Total$_x$, is the total number of blocks along the width of the detection window, and Total$_y$, is the total number of blocks along the height of the detection window. This formula for Total$_x$ and Total$_y$, takes into account the extra blocks that result from overlapping. After calculating Total$_x$ and Total$_y$, we can get the total number of blocks in the detection window by multiplying Total$_x$  $\times$ Total$_y$. The above formula can be simplified considerably because the `block_size`, `block_stride`, and `window_size`are all defined in terms of the `cell_size`. By making all the appropriate substitutions and cancelations the above formula reduces to:

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

Where cells$_x$ is the total number of cells along the width of the detection window, and cells$_y$, is the total number of cells along the height of the detection window. And $N_x$ is the horizontal block stride in units of `cell_size` and $N_y$ is the vertical block stride in units of `cell_size`.

Let's calculate what the number of elements for the HOG feature vector should be and check that it matches the shape of the HOG Descriptor calculated above.

```python
# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

    The total number of elements in the HOG Feature Vector should be:  40 x 40 x 2 x 2 x 9 = 57600

    The HOG Descriptor has shape: (57600, 1)

# Visualizing The HOG Descriptor

We can visualize the HOG Descriptor by plotting the histogram associated with each cell as a collection of vectors. To do this, we will plot each bin in the histogram as a single vector whose magnitude is given by the height of the bin and its orientation is given by the angular bin that its associated with. Since any given cell might have multiple histograms associated with it, due to the overlapping blocks, we will choose to average all the histograms for each cell to produce a single histogram for each cell.

OpenCV has no easy way to visualize the HOG Descriptor, so we have to do some manipulation first in order to visualize it. We will start by reshaping the HOG Descriptor in order to make our calculations easier.  We will then compute the average histogram of each cell and finally we will convert the histogram bins into vectors. Once we have the vectors, we plot the corresponding vectors for each cell in an image.

The code below produces an interactive plot so that you can interact with the figure. The figure contains:
* the grayscale image,
* the HOG Descriptor (feature vector),
* a zoomed-in portion of the HOG Descriptor, and
* the histogram of the selected cell.

**You can click anywhere on the gray scale image or the HOG Descriptor image to select a particular cell**. Once you click on either image a *magenta* rectangle will appear showing the cell you selected. The Zoom Window will show you a zoomed in version of the HOG descriptor around the selected cell; and the histogram plot will show you the corresponding histogram for the selected cell. The interactive window also has buttons at the bottom that allow for other functionality, such as panning, and giving you the option to save the figure if desired. The home button returns the figure to its default value.

**NOTE**: If you are running this notebook in the Udacity workspace, there is around a 2 second lag in the interactive plot. This means that if you click in the image to zoom in, it will take about 2 seconds for the plot to refresh.

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

    The feature vector has shape: (57600, 1)
    The reshaped feature vector has shape: (40, 40, 2, 2, 9)
    The average gradient array has shape:  (41, 41, 9)

    <IPython.core.display.Javascript object>

# Understanding The Histograms

Let's take a look at a couple of snapshots of the above figure to see if the histograms for the selected cell make sense. Let's start looking at a cell that is inside a triangle and not near an edge:

Fig. 4. - Histograms Inside a Triangle.

In this case, since the triangle is nearly all of the same color there shouldn't be any dominant gradient in the selected cell. As we can clearly see in the Zoom Window and the histogram, this is indeed the case. We have many gradients but none of them clearly dominates over the other.

Now let’s take a look at a cell that is near a horizontal edge:

Fig. 5. - Histograms Near a Horizontal Edge.

Remember that edges are areas of an image where the intensity changes abruptly. In these cases, we will have a high intensity gradient in some particular direction. This is exactly what we see in the corresponding histogram and Zoom Window for the selected cell.  In the Zoom Window, we can see that the dominant gradient is pointing up, almost at 90 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 90-degree bin in the histogram to dominate strongly over the others. This is in fact what we see.

Now let’s take a look at a cell that is near a vertical edge:

Fig. 6. - Histograms Near a Vertical Edge.

In this case we expect the dominant gradient in the cell to be horizontal, close to 180 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 170-degree bin in the histogram to dominate strongly over the others. This is what we see in the histogram but we also see that there is another dominant gradient in the cell, namely the one in the 10-degree bin. The reason for this, is because the HOG algorithm is using unsigned gradients, which means 0 degrees and 180 degrees are considered the same. Therefore, when the histograms are being created, angles between 160 and 180 degrees, contribute proportionally to both the 10-degree bin and the 170-degree bin. This results in there being two dominant gradients in the cell near the vertical edge instead of just one.

To conclude let’s take a look at a cell that is near a diagonal edge.

Fig. 7. - Histograms Near a Diagonal Edge.

To understand what we are seeing, let’s first remember that gradients have an *x*-component, and a *y*-component, just like vectors. Therefore, the resulting orientation of a gradient is going to be given by the vector sum of its components. For this reason, on vertical edges the gradients are horizontal, because they only have an x-component, as we saw in Figure 4. While on horizontal edges the gradients are vertical, because they only have a y-component, as we saw in Figure 3. Consequently, on diagonal edges, the gradients are also going to be diagonal because both the *x* and *y* components are non-zero.  Since the diagonal edges in the image are close to 45 degrees, we should expect to see a dominant gradient orientation in the 50-degree bin. This is in fact what we see in the histogram but, just like in Figure 4., we see there are two dominant gradients instead of just one. The reason for this is that when the histograms are being created, angles that are near the boundaries of bins, contribute proportionally to the adjacent bins.  For example, a gradient with an angle of 40 degrees, is right in the middle of the 30-degree and 50-degree bin. Therefore, the magnitude of the gradient is split evenly into the 30-degree and 50-degree bin. This results in there being two dominant gradients in the cell near the diagonal edge instead of just one.

Now that you know how HOG is implemented, in the workspace you will find a notebook named *Examples*. In there, you will be able set your own paramters for the HOG descriptor for various images. Have fun!

```python

```


# Support Vector Machines

- [1, What are SVMs?](#what-are-svms)
- [2. Primal approach - Hard-Margin SVM](#hard-margin-svm)
- [(Optional) - Deriving the margin requirement](#margin-derivation)
- [3. Primal approach - Soft-Margin SVM](#soft-margin-svm)
- [4. Solving the primal optimization problem](#solving-primal-svm)
   - [4.1 Hinge loss function](#hinge-loss)
   - [4.2 Updated objective function](#updated-objective-function)
   - [(Optional) Three parts of the objective function](#two-parts)
   - [4.3 Subgradient descent](#subgradient-descent)
   - [(Optional) Subgradient descent vs. gradient descent](#subgradient-descent-vs-gradient-descent)
- [5. Implementation of primal approach](#primal-implementation)
   - [5.1 Toy dataset](#toy-dataset)
   - [5.2 SVM class](#primal-svm-class)
   - [5.3 Training and testing an SVM](#train-test-svm)
   - [5.4 Visualizing the decision boundary](#decision-boundary)
- [6. Dual approach](#dual-approach)
    - [6.1 Recap Lagrange multipliers](#recap-lagrange-multipliers)
    - [6.2 Recap Lagrangian](#recap-lagrangian)
    - [6.3 Dual optimization problem](#dual-optimization-problem)
- [7. Primal vs. dual approach](#primal-vs-dual)
- [8. Kernels / non-linear SMVs](#kernel-svms)
    - [8.1 What is a kernel?](#what-is-a-kernel)
    - [8.2 What are kernels good for?](#what-are-kernels-good-for)
    - [8.3 Example](#kernel-example)
    - [8.4 Can we also use kernels in the primal SVM?](#kernel-in-primal-svm)
- [9. Sources and further reading](#sources)

## Link to interactive demo

[Click here](https://mybinder.org/v2/gh/zotroneneis/machine_learning_basics/HEAD?filepath=support_vector_machines.ipynb) to run the notebook online (using Binder) without installing jupyter or downloading the code.

Sometimes, the GitHub version of the Jupyter notebook does not display the math formulas correctly. Please refer to the Binder version in case you think something might be off or missing.

I also wrote a [blog post containing the contents of the notebook](https://alpopkes.com/posts/machine_learning/support_vector_machines/).

## 1. What are support vector machines?

Support vector machines (short: SVMs) are supervised machine learning models. They are the most prominent member of the class of [*kernel methods*](https://en.wikipedia.org/wiki/Kernel_method). SVMs can be used both for classification and regression. The original SVM proposed in 1963 is a simple binary linear classifier. What does this mean?

Assume we are given a dataset $D = \big \{ \mathbf{x}_n, y_n \big \}_{n=1}^N$, where $\mathbf{x}_n \in \mathbb{R}^D$ and labels $y_n \in \{-1, +1 \}$. A linear (hard-margin) SVM separates the two classes using a ($D-1$ dimensional) hyperplane.

Special to SVMs is that they use not any hyperplane but the one that maximizes the distance between itself and the two sets of datapoints. Such a hyperplane is called *maximum-margin* hyperplane:

In case you have never heard the term margin: the margin describes the distance between the hyperplane and the closest examples in the dataset.

Two types of SVMs exist: primals SVMs and dual SVMs. Although most research in the past looked into dual SVMs both can be used to perform non-linear classification. Therefore, we will look at both approaches and compare them in the end.

## 2. Primal approach - Hard-margin SVM

When training an SVM our goal is to find the hyperplane that maximizes the margin between the two sets of points. This hyperplane is fully defined by the points closest to the margin, which are also called *support vectors*.

The equation of a hyperplane is given by $\langle \mathbf{w}, \mathbf{x} \rangle + b = 0$, where $\langle \cdot, \cdot \rangle$ denotes the inner product and $\mathbf {w}$ is the normal vector to the hyperplane. If an example $\mathbf{x}_i$ lies on the right side of the hyperplane (that is, it has a positive label) we have $\langle \mathbf{w}, \mathbf{x}_i \rangle + b \gt 0$. If instead $\mathbf{x}_i$ lies on the left side (= negative label) we have $\langle \mathbf{w}, \mathbf{x}_i \rangle + b \lt 0$.

The support vectors lie exactly on the margin and the optimal separating hyperplane should have the same distance from all support vectors. In this sense the maximum margin hyperplane lies between two separating hyperplanes that are determined by the support vectors:

### Goal 1:
When deriving a formal equation for the maximum margin hyperplane we assume that the two delimiting hyperplanes are given by:
$$\langle \mathbf{w}, \mathbf{x}_{+} \rangle + b = +1$$
$$\langle \mathbf{w}, \mathbf{x}_{-} \rangle + b = -1$$

In other words: we want our datapoints two lie at least a distance of 1 away from the decision hyperplane into both directions. To be more precise: for our positive examples (those with label $y_n = +1$) we want the following to hold: $\langle \mathbf{w}, \mathbf{x}_n \rangle + b \ge +1$.

For our negative examples (those with label $y_n = -1$) we want the opposite: $\langle \mathbf{w}, \mathbf{x}_n \rangle + b \le -1$. This can be combined into a single equation: $y_n(\langle \mathbf{w}, \mathbf{x}_n \rangle + b) \ge 1$. This is our first goal: **We want a decision boundary that classifies our training examples correctly.**

### Goal 2:
Our second goal is to maximize the margin of this decision boundary. The margin is given by $\frac{1}{\mathbf{w}}$. If you would like to understand where this value is coming from take a look at the section "*(Optional) Deriving the margin equation*" below.

Our goal to maximize the margin can be expressed as follows:
$$ \max_{\mathbf{w}, b} \frac{1}{\Vert \mathbf{w} \Vert}$$

Instead of maximizing $\frac{1}{\Vert \mathbf{w} \Vert}$ we can instead minimize $\frac{1}{2} \Vert \mathbf{w} \Vert^2$. This simplifies the computation of the gradient.

### Combined goal
Combining goal one and goal two yields the following objective function:
$$
\min_{\mathbf{w}, b} \frac{1}{2} \Vert \mathbf{w} \Vert^2
$$
$$
\text{subject to: } y_n(\langle \mathbf{w}, \mathbf{x}_n \rangle + b) \ge 1 \text{ for all } n = 1, ..., N
$$

In words: we want to find the values for $\mathbf{w}$ and $b$ that maximize the margin while classifying all training examples correctly. This approach is called the *hard-margin support vector machine*. "Hard" because it does not allow for violations of the margin requirement (= no points are allowed to be within the margin).

## (Optional) Deriving the margin equation

We can derive the width of the margin in several ways (see sections 12.2.1-12.2.2 of the [Mathematics for Machine Learning book](https://mml-book.com)). Personally, I found the explanation of [this MIT lecture on SVMs](https://www.youtube.com/watch?v=_PwhiWxHK8o) easiest to understand.

The derivation of the margin is based on the assumptions that we have already noted above:
$$\langle \mathbf{w}, \mathbf{x}_{+} \rangle + b = +1$$
$$\langle \mathbf{w}, \mathbf{x}_{-} \rangle + b = -1$$

Including the label of each example we can rewrite this as
$$y_i (\langle \mathbf{w}, \mathbf{x}_{i} \rangle + b) -1 = 0$$

Let's say we have a positive example $\mathbf{x}_{+}$ that lies on the right delimiting hyperplane and a negative example $\mathbf{x}_{-}$ that lies on the left delimiting hyperplane. The distance between these two vectors is given by ($\mathbf{x}_{+} - \mathbf{x}_{-})$. We want to compute the orthogonal projection of the vector onto the line that is perpendicular to the decision hyperplane. This would give us the width between the two delimiting hyperplanes. We can compute this by multiplying the vector ($\mathbf{x}_{+} - \mathbf{x}_{-})$ with a vector that is perpendicular to the hyperplane. We know that the vector $\mathbf{w}$ is perpendicular to the decision hyperplane. So we can compute the margin by multiplying $(\mathbf{x}_{+} - \mathbf{x}_{-})$ with the vector $\mathbf{w}$ where the latter is divided by the scale $||\mathbf{w}||$ to make it a unit vector.

$$
\begin{align*}
\text{width} &= (\mathbf{x}_{+} - \mathbf{x}_{-}) \cdot \frac{\mathbf{w}}{||\mathbf{w}||} \\
&= \frac{\mathbf{x}_{+} \cdot \mathbf{w}}{||\mathbf{w}||} - \frac{\mathbf{x}_{-} \cdot \mathbf{w}}{||\mathbf{w}||}
\end{align*}
$$

For the positive example $\mathbf{x}_{+}$ we have $y_+ = +1$ and therefore $(\langle \mathbf{w}, \mathbf{x}_{+} \rangle = 1 - b$. For the negative example $\mathbf{x}_{-}$ we have $y_- = -1$ and therefore $- (\langle \mathbf{w}, \mathbf{x}_{-} \rangle) = 1 + b$:

$$
\begin{align*}
\text{width} &= (\mathbf{x}_{+} - \mathbf{x}_{-}) \cdot \frac{\mathbf{w}}{||\mathbf{w}||} \\
&= \frac{\mathbf{x}_{+} \cdot \mathbf{w}}{||\mathbf{w}||} - \frac{\mathbf{x}_{-} \cdot \mathbf{w}}{||\mathbf{w}||} \\
&= \frac{(1 - b) + (1 + b)}{||\mathbf{w}||}\\
&= \frac{2}{||\mathbf{w}||}\\
\end{align*}
$$

We conclude that the width between the two delimiting hyperplanes equals $\frac{2}{\mathbf{w}}$. And therefore, that the distance between the decision hyperplane and each delimiting hyperplane is $\frac{1}{\mathbf{w}}$.

##  3. Primal approach - Soft-margin SVM

In most real-world situations the available data is not linearly separable. Even if it is, we might prefer a solution which separates the data well while ignoring some noisy examples and outliers. This motivated an extension of the original hard-margin SVM called *soft-margin SVM*.

A soft-margin SVM allows for violations of the margin requirement (= classification errors). In other words: not all training examples need to be perfectly classified. They might fall within the margin or even lie on the wrong side of the decision hyperplane. However, such violations are not for free. We pay a cost for each violation, where the value of the cost depends on how far the example is from meeting the margin requirement.

To implement this we introduce so called *slack variables* $\xi_n$. Each training example $(\mathbf{x}_n, y_n)$ is assigned a slack variable $\xi_n \ge 0$. The slack variable allows this example to be within the margin or even on the wrong side of the decision hyperplane:

- If $\xi_n = 0$ the training example $(\mathbf{x}_n, y_n)$ lies exactly on the margin
- $0 \lt \xi_n \lt 1$ the training example lies within the margin but on the correct side of the decision hyperplane
- $\xi_n \ge 1$ the training example lies on the wrong side of the decision hyperplane

We extend our objective function to include the slack variables as follows:
$$ \min_{\mathbf{w}, b, \mathbf{\xi}} \frac{1}{2} \Vert \mathbf{w} \Vert^2 + C \sum_{n=1}^N \xi_n $$

$$ \text{subject to:} $$

$$ \begin{equation}
y_n(\langle \mathbf{w}, \mathbf{x}_n \rangle + b) \ge 1 - \xi_n
\end{equation}$$

$$ \xi_i \ge 0 \text{ for all } n = 1, ..., N $$

Note: the objective function is somewhat not displayed correctly within the GitHub version of the notebook. It should look as follows:

The parameter $C$ is a regularization term that controls the trade-off between maximizing the margin and minimizing the training error (which in turn means classifying all training examples correctly). If the value of $C$ is small, we care more about maximizing the margin than classifying all points correctly. If the value of $C$ is large, we care more about classifying all points correctly than maximizing the margin.

## 4. Solving the primal optimization problem

Theoretically, the primal SVM can be solved in multiple ways. The most well known way is to use the [hinge loss function](https://en.wikipedia.org/wiki/Hinge_loss) together with [subgradient descent](https://en.wikipedia.org/wiki/Subgradient_method).

### 4.1 Hinge loss function
The hinge loss function given the true target $y \in \{-1, +1\}$ and the prediction $f(\mathbf{x}) = \langle\mathbf{w}, \mathbf{x}\rangle+b$ is computed as follows:

$$\ell(t)=\max \{0,1-t\} \quad \text{where} \quad t=y \cdot f(\mathbf{x})= y \cdot \big(\langle\mathbf{w}, \mathbf{x}\rangle+b\big)$$

Let's understand the output of this loss function with a few examples:
- If a training example has label $y = -1$ and the prediction is on the correct side of the marghin (that is, $f(\mathbf{x}) \le -1$), the value of $t$ is larger or equal to $+1$. Therefore, the hinge loss will be zero ($\ell(t) = 0$)
- The same holds if a training example has label $y = 1$ and the prediction is on the correct side of the margin (that is, $f(\mathbf{x}) \ge 1$)
- If a training example ($y = 1$) is on the correct side of the decision hyperplane but lies within the margin (that is, $0 \lt f(\mathbf{x}) \lt 1$) the hinge loss will output a positive value.
- If a training example ($y = 1$) is on the wrong side of the decision hyperplane (that is, $f(\mathbf{x}) \lt 0$), the hinge loss returns an even larger value. This value increases linearly with the distance from the decision hyperplane

### 4.2 Updated objective function

Using the hinge loss we can reformulate the optimization problem of the primal soft-margin SVM. Given a dataset $D = \big \{ \mathbf{x}_n, y_n \big \}_{n=1}^N$ we would like to minimize the total loss which is now given by:

$$
\min _{\mathbf{w}, b} \frac{1}{2}\|\mathbf{w}\|^{2} + C \sum_{n=1}^{N} \max \left\{0,1-y_{n}\left(\left\langle\mathbf{w}, \mathbf{x}_{n}\right\rangle+b\right)\right\}
$$

If you would like to understand why this is equivalent to our previous formulation of the soft-margin SVM please take a look at chapter 12.2.5 of the [Mathematics for Machine Learning book](https://mml-book.com).

### [Optional] Three parts of the objective function

Our objective function can be divided into three distinct parts:

Part 1: $\frac{1}{2}\|\mathbf{w}\|^{2}$

This part is also called the *regularization term*. It expresses a preference for solutions that separate the datapoints well, thereby maximizing the margin. In theory, we could replace this term by a different regularization term that expresses a different preference.

Part 2: $\sum_{n=1}^{N} \max \left\{0,1-y_{n}\left(\left\langle\mathbf{w}, \mathbf{x}_{n}\right\rangle+b\right)\right\}$

This part is also called the *empirical loss*. In our case it's the hinge loss which penalizes solutions that make mistakes when classifying the training examples. In theory, this term could be replaced with another loss function that expresses a different preference.

Part 3: The hyperparameter $C$ that controls the tradeoff between a large margin and a small hinge loss.

### 4.3 Sub-gradient descent

The hinge loss function is not differentiable (namely at the point $t=1$). Therefore, we cannot compute the gradient right away. However, we can use a method called [subgradient descent](https://en.wikipedia.org/wiki/Subgradient_method) to solve our optimization problem. To simplify the derivation we will adapt two things:
1. We assume that the bias $b$ is contained in our weight vector as the first entry $w_0$, that is $\mathbf{w} = [b, w_1, ..., w_D]$
2. We divide the hinge loss by the number of samples

Our cost function is then given by
$$
J(\mathbf{w}) = \frac{1}{2}\|\mathbf{w}\|^{2} + C \frac{1}{N} \sum_{n=1}^{N} \max \left\{0,1-y_{n}\left(\left\langle\mathbf{w}, \mathbf{x}_{n}\right\rangle\right)\right\}
$$

We will reformulate this to simplify computing the gradient:
$$
J(\mathbf{w}) = \frac{1}{N} \sum_{n=1}^{N} \Big[ \frac{1}{2}\|\mathbf{w}\|^{2} + C \max \left\{0,1-y_{n}\left(\left\langle\mathbf{w}, \mathbf{x}_{n}\right\rangle\right)\right\}\Big]
$$

The gradient is given by:
$$
\nabla_{w} J(\mathbf{w}) = \frac{1}{N} \sum_{n=1}^N \left\{\begin{array}{ll}
\mathbf{w} & \text{if} \max \left(0,1-y_{n} \left(\langle \mathbf{w}, \mathbf{x}_{n} \rangle \right)\right)=0 \\
\mathbf{w}-C  y_{n} \mathbf{x}_{n} & \text { otherwise }
\end{array}\right.
$$

With this formula we can apply stochastic gradient descent to solve the optimization problem.

### [Optional] Difference subgradient descent and gradient descent

The subgradient method allows us to minimize a non-differentiable convex function. Although looking similar to gradient descent the method has several important differences.

#### What is a subgradient?

A subgradient can be described as a generalization of gradients to non-differentiable functions. Informally, a sub-tangent at a point is any line that lies below the function at the point. The subgradient is the slope of this line. Formally, the subgradient of a convex function $f$ at $w_0$ is defined as all vectors $g$ such that for any other point $w$

$$ f(w) - f(w_0) \ge g \cdot (w - w_0) $$

If $f$ is differentiable at $w_0$, the subgradient contains only one vector which equals the gradient $\nabla f(w_0)$. If, however, $f$ is not differentiable, there may be several values for $g$ that satisfy this inequality. This is illustrated in the figure below.

#### Subgradient method

To minimize the objective function $f$ the subgradient method uses the following update formula for iteration $k+1$:

$$ w^{(k+1)} = w^{(k)} - \alpha_k g^{(k)}$$

Where $g^{(k)}$ is *any* subgradient of $f$ at $w^{(k)}$ and $\alpha_k$ is the $k$-th step size. Thus, at each iteration, we make a step into the direction of the negative subgradient. When $f$ is differentiable, $g^{(k)}$ equals the gradient $\nabla f(x^{(k)})$ and the method reduces to the standard gradient descent method.

More details on the subgradient method can be found [here](https://web.stanford.edu/class/ee392o/subgrad_method.pdf).

## 5. Implementation primal approach

### 5.1 Toy dataset
To implement what we have learned about primal SVMs we first have to generate a dataset. In the cell below we create a simple dataset with two features and labels +1 and -1. We further split the dataset into a test and train set.

```python
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

data_features, data_targets = make_blobs(n_samples=600, centers=2, n_features=2, random_state=42)

# The function outputs targets 0 and 1 so we need to convert targets 0 to -1
transformed_data_targets = [-1 if t == 0 else +1 for t in data_targets]

# Plot the dataset
plt.figure(figsize=(8, 6))
plt.scatter(data_features[:, 0], data_features[:, 1], c = transformed_data_targets)
plt.title("Toy dataset")
plt.ylabel("Feature 2")
plt.xlabel("Feature 1")
plt.show()

# Split data into training and test set
features_train, features_test, labels_train, labels_test = train_test_split(data_features,
                                                                              transformed_data_targets,
                                                                              test_size = 0.3)
```

### 5.2 SVM class definition

Next, we would like to implement an SVM class. We will use the knowledge we already acquired:

1. Our objective function using the hinge loss function is given by:
$$
J(\mathbf{w}) = \frac{1}{2}\|\mathbf{w}\|^{2} + C \frac{1}{N} \sum_{n=1}^{N} \max \left\{0,1-y_{n}\left(\left\langle\mathbf{w}, \mathbf{x}_{n}\right\rangle\right)\right\}
$$
2. We can minimize this function by computing the gradient:
$$
\nabla_{w} J(\mathbf{w}) = \frac{1}{N} \sum_{n=1}^N \left\{\begin{array}{ll}
\mathbf{w} & \text{if} \max \left(0,1-y_{n} \left(\langle \mathbf{w}, \mathbf{x}_{n} \rangle \right)\right)=0 \\
\mathbf{w}-C  y_{n} \mathbf{x}_{n} & \text { otherwise }
\end{array}\right.
$$
3. Given the gradient we use stochastic gradient descent to train our model
4. After training our model we can make predictions using the [sign function](https://en.wikipedia.org/wiki/Sign_function)

As mentioned previously, we will assume that the bias $b$ is contained in our weight vector as the first entry $w_0$, that is $\mathbf{w} = [b, w_1, ..., w_D] = \mathbf{w} = [w_0, w_1, ..., w_D]$.

```python
import numpy as np
from sklearn.utils import shuffle

class LinearSVM:

    def __init__(self, regularization_param):
        """
        Initialize the model by setting the regularization parameter
        and a boolean variable for our trained weights.
        """
        self.regularization_param = regularization_param
        self.trained_weights = None

    def add_bias_term(self, features):
        """
        Add intercept 1 to each training example for bias b
        """
        n_samples = features.shape[0]
        ones = np.ones((n_samples, 1))
        return np.concatenate((ones, features), axis=1)

    def compute_cost(self, weights, features, labels) -> float:
        """
        Compute the value of the cost function
        """
        n_samples = features.shape[0]

        # Compute hinge loss
        predictions = np.dot(features, weights).flatten()
        distances = 1 - labels * predictions
        hinge_losses = np.maximum(0, distances)

        # Compute sum of the individual hinge losses
        sum_hinge_loss = np.sum(hinge_losses) / n_samples

        # Compute entire cost
        cost = (1 / 2) * np.dot(weights.T, weights) + self.regularization_param * sum_hinge_loss

        return float(cost)

    def compute_gradient(self, weights, features, labels) -> np.ndarray:
        """
        Compute the gradient, needed for training
        """
        predictions = np.dot(features, weights)
        distances = 1 - labels * predictions
        n_samples, n_feat = features.shape
        sub_gradients = np.zeros((1, n_feat))

        for idx, dist in enumerate(distances):
            if max(0, dist) == 0:
                sub_gradients += weights.T
            else:
                sub_grad = weights.T - (self.regularization_param * features[idx] * labels[idx])
                sub_gradients += sub_grad

        # Sum up and divide by the number of samples
        avg_gradient = sum(sub_gradients) / len(labels)

        return avg_gradient

    def train(self, train_features, train_labels, n_epochs, learning_rate=0.01, batch_size=1):
        """
        Train the model with stochastic gradient descent using the
        specified number of epochs, learning rate and batch size.
        """
        # Add bias term to features
        train_features = self.add_bias_term(train_features)

        # Initalize weight vector
        n_samples, n_feat = train_features.shape
        weights = np.zeros(n_feat)[:, np.newaxis]

        # Train the model for a certain number of epochs
        for epoch in range(n_epochs):
            features, labels = shuffle(train_features, train_labels)
            features, labels = train_features, train_labels
            start, end = 0, batch_size
            while end <= len(labels): # Training loop over the dataset
                batch = features[start:end]
                batch_labels = labels[start:end]

                grad = self.compute_gradient(weights, batch, batch_labels)
                update = (learning_rate * grad)[:, np.newaxis]
                weights = weights - update
                start, end = end, end + batch_size

            current_cost = self.compute_cost(weights, features, labels)
            print(f"Epoch {epoch + 1}, cost: {current_cost}")

        # Set the trained weights to allow making predictions
        self.trained_weights = weights

    def predict(self, test_features) -> np.ndarray:
        """
        Predict labels for new test features.
        Raises ValueError if model has not been trained yet.
        """
        test_features = self.add_bias_term(test_features)
        if self.trained_weights is None:
            raise ValueError("You haven't trained the SVM yet!")

        predicted_labels = []
        n_samples = test_features.shape[0]
        for idx in range(n_samples):
            prediction = np.sign(np.dot(self.trained_weights.T, test_features[idx]))
            predicted_labels.append(prediction)

        return np.array(predicted_labels)
```

```python
# Compute some values to make sure the cost is computed correctly
# I calculated the values for this example by hand first
svm = LinearSVM(regularization_param=1)
weights = np.array([1, 2])[:, np.newaxis]
features = np.array([[0.5], [2.5]])
new_features = svm.add_bias_term(features)

labels = np.array([-1, +1])
assert svm.compute_cost(weights, new_features, labels) == 4.
gradient = svm.compute_gradient(weights, new_features, labels[:, np.newaxis])
```

### 5.3 Training and testing an SVM

After defining our SVM class we can train a model and test it on unseen examples.

```python
# Initialize a new SVM and train it on the given toy dataset
regularization_param = 100
lr = 0.000001
svm = LinearSVM(regularization_param)
trained_weights = svm.train(features_train, labels_train, n_epochs=10, learning_rate=lr)
```

    Epoch 1, cost: 27.249462152357992
    Epoch 2, cost: 7.348722759212155
    Epoch 3, cost: 3.3972581191442885
    Epoch 4, cost: 2.27870883462286
    Epoch 5, cost: 1.8875111235175854
    Epoch 6, cost: 1.6020090021589712
    Epoch 7, cost: 1.413202164641454
    Epoch 8, cost: 1.2452165334256207
    Epoch 9, cost: 1.1141080363838292
    Epoch 10, cost: 1.0118493571787424

```python
# Predict lables for unknown test samples
from sklearn.metrics import accuracy_score, recall_score, precision_score

predicted_labels = svm.predict(features_test)
predicted_labels = predicted_labels.flatten()

print("Accuracy on test dataset: {}".format(accuracy_score(labels_test, predicted_labels)))
print("Recall on test dataset: {}".format(recall_score(labels_test, predicted_labels)))
print("Precision on test dataset: {}".format(precision_score(labels_test, predicted_labels)))
```

    Accuracy on test dataset: 1.0
    Recall on test dataset: 1.0
    Precision on test dataset: 1.0

### 5.4 Visualizing the decision boundary

Given our trained model we can visualize the decision boundary, as done below.

```python
import numpy as np

# Create dataset for visualization
size=40000
feat_1 = np.random.uniform(low=-7, high=8, size=size)
feat_2 = np.random.uniform(low=-2, high=14, size=size)
features_vis = np.column_stack((feat_1, feat_2))

labels_vis = svm.predict(features_vis)

# Plot the decision boundary
plt.figure(figsize=(7, 5))
plt.scatter(features_vis[:, 0], features_vis[:, 1], c = labels_vis)
# Plot original dataset
positive_samples = [idx for idx in range(len(transformed_data_targets)) if transformed_data_targets[idx] == +1]
negative_samples = [idx for idx in range(len(transformed_data_targets)) if transformed_data_targets[idx] == -1]
plt.scatter(data_features[positive_samples, 0],
            data_features[positive_samples, 1],
            c="yellow", label="Original positive samples")
plt.scatter(data_features[negative_samples, 0],
            data_features[negative_samples, 1],
            c="purple", label="Original negative samples")
plt.title("SVM decision boundary")
plt.ylabel("Feature 2")
plt.xlabel("Feature 1")
plt.legend(loc=2)
plt.show()
```

## 6. Dual approach

In the previous sections we took a detailed look at the primal SVM. To solve a primal SVM, we need to find the best values for the weights and bias. Recall that our input examples $\mathbf{x} \in \mathbb{R}^D$ have $D$ features. Consequently, our weights $\mathbf{w}$ have $D$ features, too. This can become problematic if the number of features $D$ is large.

That's where the second way of formalizing SVMs (called *dual approach*) comes in handy. The optimization problem of the dual approach is independent of the number of features. Instead, the number of parameters increases with the number of examples in the training set.

The dual approach uses the method of Lagrange multipliers. Lagrange multipliers allow us to find the minimum or maximum of a function if there are one or more constraints on the input values we are allowed to used.

If you never heard of Lagrange multipliers I can recommend the [blog posts](https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/constrained-optimization/a/lagrange-multipliers-single-constraint) and [video tutorials](https://www.youtube.com/watch?v=yuqB-d5MjZA&list=PLCg2-CTYVrQvNGLbd-FN70UxWZSeKP4wV&index=1) on the topic from Khan Academy.

### 6.1 Recap Lagrange multipliers

Lagrange multipliers allow us to solve constrained optimization problems. Let's say we want to maximize the function $f(x, y) = 2x + y$ under the constraint that our values of $x$ and $y$ satisfy the following equation: $g(x, y) := x^2 + y^2 = 1$. This constraint equation describes a circle of radius 1.

The key insight behind the solution to this problem is that we need to find those values for $x$ and $y$ where the gradients of $f$ and $g$ are aligned. This can be expressed using a Lagrange multiplier (typically $\lambda$):

We want to find those values $x_m, y_m$ where $\nabla f(x_m, y_m) = \lambda \nabla g(x_m, y_m)$.

In our example the gradient vectors look as follows:
$$ \nabla f(x, y)=\left[\begin{array}{c}
\frac{\partial}{\partial x}(2 x+y) \\
\frac{\partial}{\partial y}(2 x+y)
\end{array}\right]=\left[\begin{array}{c}
2 \\ 1
\end{array}\right]$$

$$ \nabla g(x, y)=\left[\begin{array}{c}
\frac{\partial}{\partial x}\left(x^{2}+y^{2}-1\right) \\
\frac{\partial}{\partial y}\left(x^{2}+y^{2}-1\right)
\end{array}\right]=\left[\begin{array}{c}
2 x \\ 2 y
\end{array}\right]$$

Therefore, the tangency condition results in
$$ \left[\begin{array}{l}
2 \\ 1
\end{array}\right]=\lambda \left[\begin{array}{l}
2 x_{m} \\ 2 y_{m}
\end{array}\right] $$

We can rewrite the vector form into individual equations that can be solved by hand:
- $2 = \lambda 2 x_m $
- $1 = \lambda 2 y_m $
- $x_m^2 + y_m^2 = 1 $

Solving the equations yields
$$ \begin{aligned}
\left(x_{0}, y_{0}\right) &=\left(\frac{1}{\lambda_{0}}, \frac{1}{2 \lambda_{0}}\right) \\
&=\left(\frac{2}{\sqrt{5}}, \frac{1}{\sqrt{5}}\right) \quad \text { or } \quad\left(\frac{-2}{\sqrt{5}}, \frac{-1}{\sqrt{5}}\right)
\end{aligned} $$

where the first point denotes a maximum (what we wanted to find) and the second a minimum. This solves our constrained optimization problem. For more details and a full solution look at [this Khan academy post](https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/constrained-optimization/a/lagrange-multipliers-single-constraint).

### 6.2 Recap Lagrangian

The Lagrangian is a way to repackage the individual conditions of our constrained optimization problem into a single equation. In the example above we wanted to optimize some function $f(x, y)$ under the constraint that the inputs $x$ and $y$ satisfy the equation $g(x, y) = x^2 + y^2 = c$. In our case the constant $c$ was given by 1. We know that the solution is given by those points where the gradients of $f$ and $g$ align. The Lagrangian function puts all of this into a single equation:

$$ \mathcal{L}(x, y, \lambda)=f(x, y)-\lambda(g(x, y)-c) $$

When computing the partial derivatives of $\mathcal{L}$ with respect to $x, y$ and $\lambda$ and setting them to zero, we will find that they correspond exactly to the three constraints we looked at earlier. This can be summarized by simply setting the gradient of $\mathcal{L}$ equal to the zero vector: $\nabla \mathcal{L} = \mathbf{0}$. The compact Lagrangian form is often used when solving constrained optimization problem with computers because it summarizes the elaborate problem with multiple constraints into a single equation. For more details and a full solution look at [this Khan academy post](https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/constrained-optimization/a/lagrange-multipliers-single-constraint).

### 6.3 Dual optimization problem
For the primal soft-margin SVM we considered the following optimization problem:
$$ \min_{\mathbf{w}, b, \mathbf{\xi}} \frac{1}{2} \Vert \mathbf{w} \Vert^2 + C \sum_{n=1}^N \xi_n $$

$$ \text{subject to:} $$

$$ y_n(\langle \mathbf{w}, \mathbf{x}_n \rangle + b) \ge 1 - \xi_n $$

$$ \xi_i \ge 0 \text{ for all } n = 1, ..., N $$

Note: the optimization problem is somewhat not displayed correctly within the GitHub version of the notebook. It should look as follows:

To derive the corresponding Lagrangian we will introduce two Lagrange multipliers: $\alpha_n$ for the first constraint (that all examples are classified correctly) and $\lambda_n$ for the second constraint (non-negativity of the slack variables). The Lagrangian is then given by:

$$
\begin{aligned}
\mathfrak{L}(\boldsymbol{w}, b, \xi, \alpha, \gamma)=& \frac{1}{2}\|\boldsymbol{w}\|^{2}+C \sum_{n=1}^{N} \xi_{n} \\
& \underbrace{-\sum_{n=1}^{N} \alpha_{n}\left(y_{n}\left(\left\langle\boldsymbol{w}, \boldsymbol{x}_{n}\right\rangle+b\right)-1+\xi_{n}\right)}_{\text{first constraint}} \underbrace{-\sum_{n=1}^{N} \gamma_{n} \xi_{n}}_{\text{second constraint}}
\end{aligned}
$$

Next, we have to compute the partial derivatives of the Lagrangian with respect to the variables $\mathbf{w}, b$ and $\xi$: $\frac{\partial \mathfrak{L}}{\partial \mathbf{w}}, \frac{\partial \mathfrak{L}}{\partial b}, \frac{\partial \mathfrak{L}}{\partial \xi}$. When setting the first partial derivative to zero we obtain an important interim result:
$$ \mathbf{w} = \sum_{n=1}^N \alpha_n y_n \mathbf{x}_n $$

This equation states the the optimal solution for the weight vector is given by a linear combination of our training examples. After setting the other partial derivatives to zero, using the result and simplifying the equations we end up with the following optimization problem (for details see section 12.3.1 of the [Mathematics for Machine Learning book](https://mml-book.com)):

$$\min _{\boldsymbol{\alpha}} \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} y_{i} y_{j} \alpha_{i} \alpha_{j}\left\langle\mathbf{x}_{i}, \mathbf{x}_{j}\right\rangle-\sum_{i=1}^{N} \alpha_{i}$$
$$ \text{subject to:} $$
$$\sum_{i=1}^{N} y_{i} \alpha_{i}=0$$
$$0 \le \alpha_{i} \le C \text{ for all } i=1, \ldots, N$$

Note: the optimization problem is somewhat not displayed correctly within the GitHub version of the notebook. It should look as follows:

This constrained quadratic optimization problem can be solved very efficiently, for example with quadratic programming techniques. One popular library for solving dual SVMs is [libsvm](https://github.com/cjlin1/libsvm) which makes use of a decomposition method to solve the problem (see [this paper](https://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf) for more details). However, several other approaches exist.

## 7. Primal vs. dual approach

Most SVM research in the last decade has been about the dual formulation. Why this is the case is not clear. Both approaches have advantages and disadvantages. In the paper ["Training a Support Vector Machine in the Primal"](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.129.3368&rep=rep1&type=pdf) Chapelle et al. mention the following hypothesis:

> We believe that it is because SVMs were first introduced in their hard margin formulation (Boser et al., 1992), for which a dual optimization (because of the constraints) seems more natural. In general, however, soft margin SVMs should be preferred, even if the training data are separable: the decision boundary is more robust because more training points are taken into account (Chapelle et al., 2000). We do not pretend that primal optimization is better in general; our main motivation wasto point out that primal and dual are two sides of the same coin and that there is no reason to look always at the same side.

## 8. Kernels / non-linear SVM

### 8.1 What is a kernel?
If you take another look at the optimization equation of dual SVMs you will notice that it computes the inner product $\left\langle\mathbf{x}_{i}, \mathbf{x}_{j}\right\rangle$ between all datapoints $\mathbf{x}_{i}, \mathbf{x}_{j}$. A kernel is a way to compute this inner product implicitely in some (potentially very high dimensional) feature space. To be more precise: assume we have some mapping function $\varphi$ which maps an $n$ dimensional input vector to an $m$ dimensional output vector: $\varphi \, : \, \mathbb R^n \to \mathbb R^m$. Given this mapping function we can compute the dot product of two vectors $\mathbf x$ and $\mathbf y$ in this space as follows: $\varphi(\mathbf x)^T \varphi(\mathbf y)$.

A kernel is a function $k$ that gives the same result as this dot product: $k(\mathbf x, \mathbf y) = \varphi(\mathbf x)^T \varphi(\mathbf y)$. In other words: the kernel function is equivalent to the dot product of the mapping function.

### 8.2 What are kernels good for?

Until now (apart from the soft-margin SVM) our SVMs, both primal and dual, are only able to classify data that is [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). However, most datasets in practice won't be of this form. We need a way to classify data that is **not** linearly separable. This is where the so called **kernel trick** comes into play.

Because the objective function of the dual SVM contains inner products only between datapoints $\mathbf{x}_i, \mathbf{x}_j$, we can easily replace this inner product (that is, $\left\langle\mathbf{x}_{i}, \mathbf{x}_{j}\right\rangle$ ) with some mapping function $\varphi(\mathbf{x}_i)^T \varphi(\mathbf{x}_j)$. This mapping function can be non-linear, allowing us to compute an SVM that is non-linear with respect to the input examples. The mapping function takes our input data (which is not linearly separable) and transforms it into some higher-dimensional space where it becomes linearly separable. This is illustrated in the figure below.

In theory, we could use any mapping function we like. In practice, however, computing inner products is expensive. Therefore, we use mapping functions that have a corresponding kernel function. This will allow us to map the datapoints into a higher dimensional space without ever explicitely computing the (expensive) inner products.

Let's take a look at an example.

### 8.3 Example
Note: this example was taken from [this StackExchange post](https://stats.stackexchange.com/posts/153134).

We can create a simple polynomial kernel as follows: $k(\mathbf{x}, \mathbf{y}) = (1 + \mathbf x^T \mathbf y)^2$ with $\mathbf x, \mathbf y \in \mathbb R^2$. The kernel does not seem to correspond to any mapping function $\varphi$, it's just a function that returns a real number. Our input vectors $\mathbf{x}, \mathbf{y}$ are 2-dimensional: $\mathbf x = (x_1, x_2)$ and $\mathbf y = (y_1, y_2)$. With this knowledge we can expand the kernel computation:

$\begin{align*}
k(\mathbf x, \mathbf y) & = (1 + \mathbf x^T \mathbf y)^2 \\
&= (1 + x_1 \, y_1  + x_2 \, y_2)^2 \\
 & = 1 + x_1^2 y_1^2 + x_2^2 y_2^2 + 2 x_1 y_1 + 2 x_2 y_2 + 2 x_1 x_2 y_1 y_2
\end{align*}$

Note that this is nothing else but a dot product between two vectors $(1, x_1^2, x_2^2, \sqrt{2} x_1, \sqrt{2} x_2, \sqrt{2} x_1 x_2)$ and $(1, y_1^2, y_2^2, \sqrt{2} y_1, \sqrt{2} y_2, \sqrt{2} y_1 y_2)$. This can be expressed with the following mapping function:
$$\varphi(\mathbf x) = \varphi(x_1, x_2) = (1, x_1^2, x_2^2, \sqrt{2} x_1, \sqrt{2} x_2, \sqrt{2} x_1 x_2)$$

So the kernel $k(\mathbf x, \mathbf y) = (1 + \mathbf x^T \mathbf y)^2 = \varphi(\mathbf x)^T \varphi(\mathbf y)$ computes a dot product in 6-dimensional space without explicitly visiting this space. The generalization from an inner product to a kernel function is known as the **kernel trick**.

Several popular kernel functions exist. Popular ones are, for example, the [polyomial kernel](https://en.wikipedia.org/wiki/Polynomial_kernel) or [RBF kernel](https://en.wikipedia.org/wiki/Radial_basis_function_kernel).

### 8.4 Can we also use kernels in the primal SVM?

Yes, the kernel trick can be applied to primal SVM's, too. It's not as straightforward as with dual SVMs but still possible. Consider [this paper by Chapelle et al.](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.129.3368&rep=rep1&type=pdf) as an example.

### 8.5 How do I choose the right kernel for my problem?

The problem of choosing the right kernel has been answered in [this StackExchange post](
https://stats.stackexchange.com/questions/18030/how-to-select-kernel-for-svm).

Summary:
- Without expert knowledge, the Radial Basis Function kernel makes a good default kernel (in case you need a non-linear model)
- The choice of the kernel and parameters can be automated by optimising a cross-validation based model selection
- Choosing the kernel and parameters automatically is tricky, as it is very easy to overfit the model selection criterion

## 9. Sources and further reading

The basis for this notebook is chapter 12 of the book [Mathematics for Machine Learning](https://mml-book.github.io/). I can highly recommend to read through the entire chapter to get a deeper understanding of support vector machines.

Another source I liked very much is [this MIT lecture on SVMs](https://www.youtube.com/watch?v=_PwhiWxHK8o).

```python

```


# Support Vector Machines on the D-Wave Quantum Annealer
#### Created by Gabriele Cavallaro (g.cavallaro@fz-juelich.de)

### 0. Setting Up the Access to the D-Wave 2000Q quantum computer

- Make a free account to run on the D-Wave through [Leap](https://www.dwavesys.com/take-leap)

- Install Ocean Software with [pip install dwave-ocean-sdk](https://docs.ocean.dwavesys.com/en/latest/overview/install.html)

- Configuring the D-Wave System as a Solver with [dwave config create](https://docs.ocean.dwavesys.com/en/latest/overview/dwavesys.html#dwavesys)

### 1. Data Preparation

#### 1.1 Load of the Python Modules

```python
from utils import * # It contains functions for threat the data (I/O, encoding/decoding) and metrics for evaluations
```

#### 1.2 Select the Dataset

##### In this notebook we consider the datasets of [HyperLabelMe](http://hyperlabelme.uv.es/index.html) (i.e., a benchmark system for remote sensing image classification).

- It contains 43 image datasets, both multi- and hyperspectral
- For each one, training pairs (spectra and their labels) and test spectra are provided
- The test labels are not given. The predicted labels needs to be uploaded in HyperLabelMe which will return the accuracy

```python
# Load the data
# Ima40.txt can be downloaded after registering at [1]
id_dataset='Im40'
[X_train, Y_train, X_test]=dataread('input_datasets/hyperlabelme/'+id_dataset+'.txt')
```

#### 1.3 Background on Support Vector Machines (SVMs)

A SVM learns its parameters from a set of annotated training samples

- $D=\left\{\textbf{x}_{n}, y_{n}: n=0, \ldots, N-1\right\}$
- with $\textbf{x}_{n} \in \mathbb{R}^{d}$ being a feature vector and $y_n$ its label.

A SVM separates the samples of different classes in their feature space by tracing maximum margin hyperplanes.

The training consists of solving a [quadratic programming (QP)](https://www.cambridge.org/us/academic/subjects/mathematics/numerical-recipes/numerical-recipes-art-scientific-computing-3rd-edition?format=HB&utm_source=shortlink&utm_medium=shortlink&utm_campaign=numericalrecipes) problem.

\begin{equation}
\label{eq:qp_equation}
L=\frac{1}{2} \sum_{n m} \alpha_{n} \alpha_{m} y_{n} y_{m} k\left(\mathbf{x}_{n}, \mathbf{x}_{m}\right)-\sum_{n} \alpha_{n} \qquad \qquad \qquad \text { (1) }
\end{equation}

\begin{equation}
\label{eq:svm_constrains}
\text {subject to} \quad 0 \leq \alpha_{n} \leq C  \> \> \text { and } \> \> \sum_{n} \alpha_{n} y_{n}=0 \qquad  \qquad  \text { (2) }
\end{equation}

For $N$ coefficients $\alpha_{n} \in \mathbb{R}$, where $C$ is a regularization parameter and $k(.,.)$ is a kernel function that enables a SVM to compute non-linear decision functions (by means of the kernel trick
[kernel trick](https://dl.acm.org/citation.cfm?id=559923)).
The type of kernel function which is most commonly used is the RBF:
$\operatorname{rbf}\left(\mathbf{x}_{n}, \mathbf{x}_{m}\right)=e^{-\overline{\gamma}\left\|\mathbf{x}_{n}-\mathbf{x}_{m}\right\|^{2}}$.

The SVM decision boundary is based on the samples corresponding to $\alpha_{n} \neq 0$ (i.e., support vectors).
A typical solution often contains many $\alpha_{n}=0$.
The prediction for an arbitrary
sample $\mathbf{x} \in \mathbb{R}^{d}$ can be made by evaluating the decision function (i.e., signed distance between the sample $\mathbf{x}$ and the decision boundary)

\begin{equation}
\label{eq:decision_function}
f(\mathbf{x})=\sum_{n} \alpha_{n} y_{n} k\left(\mathbf{x}_{n}, \mathbf{x}\right)+b \qquad  \qquad  \text { (3) }
\end{equation}

where the bias $b$ can be computed by

\begin{equation}
b=\frac{\sum_{n} \alpha_{n}\left(C-\alpha_{n}\right)\left[y_{n}-\sum_{m} \alpha_{m} y_{m} k\left(\mathbf{x}_{m}, \mathbf{x}_{n}\right)\right]}{\sum_{n} \alpha_{n}\left(C-\alpha_{n}\right)} \qquad  \qquad  \text { (4) }
\end{equation}

The class label for $\mathbf{x}$ predicted  is $\widetilde{y}=\operatorname{sign}(f(\mathbf{x}))$.

#### 1.4 Quantum SVM

The DW2000Q QA requires the SVM training to be formulated as a [Quadratic Unconstrained Binary Optimization (QUBO)](https://docs.dwavesys.com/docs/latest/c_gs_3.html) problem which is defined as the minimization of the energy function:

\begin{equation}
E=\sum_{i \leq j} a_{i} Q_{i j} a_{j} \qquad  \qquad  \text { (5) }
\end{equation}

with $a_{i} \in\{0,1\}$ the binary variables of the optimization problem, and $Q$
the QUBO weight matrix (i.e., an upper-triangular matrix of real numbers).

Since the solution of Eqs. (1)-(2) consists of real numbers $\alpha_{n} \in \mathbb{R}$ and Eq.(4) can only computes discrete solutions, the following encoding is used:

\begin{equation}
\label{eq:encoding}
\alpha_{n}=\sum_{k=0}^{K-1} B^{k} a_{K n+k} \qquad  \qquad  \text { (6) }
\end{equation}

where $a_{K n+k} \in\{0,1\}$ are binary variables, $K$ is the number of
binary variables to encode $\alpha_{n}$, and $B$ is the base used for the
encoding.

The formulation of the QP of Eqs. (1)-(2) as QUBO is obtained through the encoding of Eq. (6) and the introduction of a multiplier $\xi$ to include the first constraint of Eq. (2) as a squared penalty term:

\begin{equation}
\label{eq:formulation_qp_quantum_1}
E=\frac{1}{2} \sum_{n m k j} a_{K n+k} a_{K m+j} B^{k+j} y_{n} y_{m} k\left(\mathbf{x}_{n}, \mathbf{x}_{m}\right)
-\sum_{n k} B^{k} a_{K n+k}+\xi\left(\sum_{n k} B^{k} a_{K n+k} y_{n}\right)^{2}  \qquad  \qquad  \text { (7) }
\end{equation}

\begin{equation}
\label{eq:formulation_qp_quantum_2}
 =\sum_{n, m=0}^{N-1} \sum_{k, j=0}^{K-1} a_{K n+k} \widetilde{Q}_{K n+k, K m+j} a_{K m+j}   \qquad  \qquad  \text { (8) }
\end{equation}

where $\widetilde{Q}$ is a matrix of size $K N \times K N$ given by

\begin{equation}
\label{eq:q_embedding}
\widetilde{Q}_{K n+k, K m+j} =\frac{1}{2} B^{k+j} y_{n} y_{m}\left(k\left(\mathbf{x}_{n}, \mathbf{x}_{m}\right)+\xi\right)  -\delta_{n m} \delta_{k j} B^{k}   \qquad  \qquad  \text { (9) }
\end{equation}

Since $\widetilde{Q}$ is symmetric, the upper-triangular \ac{QUBO} matrix $Q$ is defined by
$Q_{i j}=\widetilde{Q}_{i j}+\widetilde{Q}_{j i}$ for $i<j$ and $Q_{i i}=\widetilde{Q}_{i i}$. The second constraint of Eq. (2) is automatically included in Eq. (8) through the encoding given in Eq. (6), since the maximum for $\alpha_{n}$ is given by

\begin{equation}
C=\sum_{k=1}^{K} B^{k}  \qquad  \qquad  \text { (10) }
\end{equation}

The last step required to run the optimization on the DW2000Q QA is the embedding procedure.
This is necessary because the QUBO problem given in Eq. (5) includes some couplers $Q_{i,j}\neq0$ between qubit $i$ and qubit $j$ for which no physical connection exists on the chip (i.e., constraint of the Chimera topology of the DW2000Q quantum processor).
The embedding increases the number of logical connections between the qubits.
When no embedding can be found, the number of nonzero couplers $n_{cpl}$ is the parameter that can be reduced until an embedding is found.

The DW2000Q QA computes a variety of close-to-optimal solutions (i.e., different coefficients  $\{\alpha_{n}\}^{(i)}$ obtained from Eq. (6)). Many of these solutions may have a slightly higher
energy than the global minimum $\{\alpha_{n}\}^*$ that can be found by the classical SVM. However, these solutions can still solve the classification problem for the training data.
For each run on the DW2000Q QA, the 20 lowest energy samples from 10,000 reads are kept.

#### 1.5 Quantum SVM: Calibration Phase

The SVM on the QA depends on four hyperparameters:
the encoding base $B$, the number $K$ of qubits per coefficient $\alpha_{n}$, the multiplier $\xi$, and the kernel parameter $\gamma$. The parameter $n_{cpl}$ varies for each run  and is not a parameter of the SVM itself.

The hyperparameters are selected through a 10-fold cross-validation. Each training set includes only 30 samples (i.e., choice due to the limitations of the QA). The validation includes the remaining samples that are used for the evaluation of the performance.

For each dataset, the values are calibrated by evaluating the SVM for $B \in \{2, 3, 5, 10\}$, $K\in \{2, 3\}$, $\xi \in \{0, 1, 5\}$, and $\gamma\in \{ −1, 0.125, 0.25, 0.5, 1, 2, 4, 8 \}$.

```python
from sklearn.model_selection import train_test_split
from sklearn import preprocessing

# 10-fold Monte Carlo (or split-and-shuffle) cross-validation
fold=10

for i in range(0,fold):
    X_train_cal, X_val_cal, Y_train_cal, Y_val_cal = train_test_split(X_train,Y_train, test_size=0.94, random_state=i)

    # Pre-processing
    X_train_cal = preprocessing.scale(X_train_cal)
    X_val_cal = preprocessing.scale(X_val_cal)

    # Write the data
    write_samples(X_train_cal, Y_train_cal,'input_datasets/calibration/'+id_dataset+'/'+id_dataset+'calibtrain'+str(i))
    write_samples(X_val_cal, Y_val_cal,'input_datasets/calibration/'+id_dataset+'/'+id_dataset+'calibval'+str(i))

print('Each training set includes '+str(X_train_cal.shape[0])+ ' samples')
print('Each validation set includes '+str(X_val_cal.shape[0])+ ' samples')
```

```python
from quantum_SVM import *

# Hyperparameters
B=[2,3,5,10]
K=[2,3]
xi=[0,1,5]
gamma=[-1,0.125,0.25,0.5,1,2,4,8]
n_experiments=len(B)*len(K)*len(xi)*len(gamma)

hyperparameters=np.zeros([n_experiments,4], dtype=float)

path_data_key='input_datasets/calibration/'+id_dataset+'/'
data_key = id_dataset+'calibtrain'
path_out='outputs/calibration/'+id_dataset+'/'

trainacc=np.zeros([fold], dtype=float)
trainauroc=np.zeros([fold], dtype=float)
trainauprc=np.zeros([fold], dtype=float)

testacc=np.zeros([fold], dtype=float)
testauroc=np.zeros([fold], dtype=float)
testauprc=np.zeros([fold], dtype=float)

trainacc_all=np.zeros([n_experiments], dtype=float)
trainauroc_all=np.zeros([n_experiments], dtype=float)
trainauprc_all=np.zeros([n_experiments], dtype=float)

testacc_all=np.zeros([n_experiments], dtype=float)
testauroc_all=np.zeros([n_experiments], dtype=float)
testauprc_all=np.zeros([n_experiments], dtype=float)

f = open(path_out+'calibration_results.txt',"w")
f.write("B\t K\t xi\t   gamma\t trainacc\t trainauroc\t trainauprc\t testacc\t testauroc\t testauprc\n")

count=0
for x in range(0,len(B)):
    for y in range(0,len(K)):
        for z in range(0,len(xi)):
            for i in range(0,len(gamma)):
                for j in range(0,fold):
                    path=gen_svm_qubos(B[x],K[y],xi[z],gamma[i],path_data_key,data_key+str(j),path_out)
                    pathsub=dwave_run(path_data_key,path)
                    [trainacc[j],trainauroc[j],trainauprc[j],testacc[j],testauroc[j],testauprc[j]]=eval_run_rocpr_curves(path_data_key,pathsub,'noplotsave')

                hyperparameters[count,0]=B[x]
                hyperparameters[count,1]=K[y]
                hyperparameters[count,2]=xi[z]
                hyperparameters[count,3]=gamma[i]

                trainacc_all[count]=np.average(trainacc)
                trainauroc_all[count]=np.average(trainauroc)
                trainauprc_all[count]=np.average(trainauprc)

                testacc_all[count]=np.average(testacc)
                testauroc_all[count]=np.average(testauroc)
                testauprc_all[count]=np.average(testauprc)

                np.save(path_out+'hyperparameters', hyperparameters)
                np.save(path_out+'trainacc_all', trainacc_all)
                np.save(path_out+'trainauroc_all', trainauroc_all)
                np.save(path_out+'trainauprc_all', trainauprc_all)
                np.save(path_out+'testacc_all', testacc_all)
                np.save(path_out+'testauroc_all', testauroc_all)
                np.save(path_out+'testauprc_all', testauprc_all)

                f.write(f'{B[x]}\t {K[y]}\t {xi[z]}\t {gamma[i]:8.3f}\t {np.average(trainacc):8.4f}\t {np.average(trainauroc):8.4f}\t {np.average(trainauprc):8.4f}\t {np.average(testacc):8.4f}\t {np.average(testauroc):8.4f}\t {np.average(testauprc):8.4f}')
                f.write("\n")
                count=count+1

f.close()
```

#### 1.6 Quantum SVM: Training Phase

To overcome the problem of the limited connectivity of the Chimera graph of the DW2000Q QA the whole training set is split into small disjoint subsets $D^{(train,l)}$ of $~40$ samples, with $l=0,...,int(N/40)$.
The strategy is to build an ensemble of quantum weak SVMs (qeSVMs) where each classifier is trained on $D^{(train,l)}$.
This is achieved in two steps. First, for each subset $D^{(train,l)}$ the twenty best solutions from the DW2000Q QA (i.e., qSVM$(B, K, \xi , \gamma )\#i$ for $i =0, ... ,19$) are combined by averaging over the respective decision functions $f^{l,i}(\mathbf{x})$ (see Eq. (3)).

Since the decision function is linear in the coefficients
and the bias $b^{(l,i)}$ is computed from $\alpha_{n}^{(l,i)}$ via Eq. (4), this procedure effectively results in one classifier with an effective set of coefficients
$\alpha_{n}^{(l)}=\sum_{i} \alpha_{n}^{(l, i)} / 20$ and bias
$b^{l}=\sum_{i} b^{(l, i)} / 20$.
Second, an average is made over the $int(N/40)$ subsets.

Note, however, that the data points
$\left(\mathbf{x}_{n}^{(l)}, y_{n}^{(l)}\right) \in D^{(\text {train }, l)}$ are now different for each $l$. The full decision function is

\begin{equation}
F(\mathbf{x})=\frac{1}{L} \sum_{n l} \alpha_{n}^{(l)} y_{n}^{(l)} k\left(\mathbf{x}_{n}^{(l)}, \mathbf{x}\right)+b,
\end{equation}

where $b=\sum_{l} b^{(l)} / L$. As before, the decision for the class label of a point $\mathbf{x}$ is obtained through $\widetilde{t}=\operatorname{sign}(F(\mathbf{x}))$.

```python
from quantum_SVM import *
import numpy as np
from utils import *
from sklearn.model_selection import KFold
from sklearn import preprocessing

# Write the data
experiments=1
slice=40 # Number of samples to use for the training
fold=int(len(X_train)/40)

print(fold)

for i in range(0,experiments):
    cv = KFold(n_splits=fold, random_state=i, shuffle=True)
    count=0
    for test_index, train_index in cv.split(X_train):
        #print("Train Index: ", len(train_index), "\n")

        X_train_slice, y_train_slice = X_train[train_index], Y_train[train_index]
        X_train_slice = preprocessing.scale(X_train_slice)

        X_test_slice, y_test_slice = X_train[test_index], Y_train[test_index]
        X_test_slice = preprocessing.scale(X_test_slice)

        write_samples(X_train_slice, y_train_slice,f'input_datasets/train/'+id_dataset+'/'+id_dataset+'calibtrain'+str(i)+'_'+str(count))
        write_samples(X_test_slice, y_test_slice,f'input_datasets/train/'+id_dataset+'/'+id_dataset+'calibval'+str(i)+'_'+str(count))

        count=count+1

print("Each training set has", len(train_index), "samples\n")
```

```python
# Get the calibration results
path_out='outputs/calibration/'+id_dataset+'/'
hyperparameters=np.load(path_out+'hyperparameters.npy')
testauprc_all=np.load(path_out+'testauprc_all.npy')

# Select the best hyperparameter set for the max value of testauprc
idx_max = np.where(testauprc_all == np.amax(testauprc_all))
B=int(hyperparameters[int(idx_max[0]),0])
K=int(hyperparameters[int(idx_max[0]),1])
xi=int(hyperparameters[int(idx_max[0]),2])
gamma=hyperparameters[int(idx_max[0]),3]
print('The best hyperparameters are:\n'+'B = '+str(B)+' K = '+str(K)+' xi = '+str(xi)+' gamma = '+str(gamma))

path_data_key='input_datasets/train/'+id_dataset+'/'
data_key = id_dataset+'calibtrain'
path_out='outputs/train/'+id_dataset+'/'

trained_SVMs=[]

for j in range(0,experiments):
    for i in range(0,fold):
        path=gen_svm_qubos(B,K,xi,gamma,path_data_key,data_key+str(j)+'_'+str(i),path_out)
        trained_SVMs.append(dwave_run(path_data_key,path))
        np.save(path_out+'trained_SVMs',trained_SVMs)
```

```python
from quantum_SVM import *
import numpy as np
from utils import *
path_data_key='input_datasets/train/'+id_dataset+'/'
data_key = id_dataset+'calibtrain'

[trainacc[j],trainauroc[j],trainauprc[j],testacc[j],testauroc[j],testauprc[j]]=eval_run_rocpr_curves(path_data_key,'outputs/train/im16/runim16calibtrain0_0_B=2_K=3_xi=1_gamma=0.25/result_couplers=2000/','saveplot')
```

#### 1.7 Quantum SVM: Test Phase

The performance of the qeSVMs can be evaluated directly on [HyperLabelMe](http://hyperlabelme.uv.es/index.html) by uploading the predictions (i.e., output file of the next cell)

```python
from quantum_SVM import *
from sklearn import preprocessing

# Pre-processing the test spectra
X_test = preprocessing.scale(X_test)

path_data_key='input_datasets/train/'+id_dataset+'/'
data_key = id_dataset+'calibtrain'
path_train_out='outputs/train/'+id_dataset+'/'
path_test_out='outputs/test/'+id_dataset+'/'

path_files=np.load(path_train_out+'trained_SVMs.npy')

experiments=1
slices=10
scores=[]

for j in range(0,experiments):
    for i in range(0,slices):
        scores.append(predict(path_data_key,path_files[i],X_test))

avg_scores=np.zeros((scores[0].shape[0]))
Y_predicted=np.zeros((scores[0].shape[0]),int)

for i in range(0,scores[0].shape[0]):
    tmp=0
    for y in range(0,slices):
        tmp=tmp+scores[y][i]
    avg_scores[i]=tmp/slices

for i in range(0,scores[0].shape[0]):
    if(avg_scores[i]<0):
        Y_predicted[i]=1
    else:
        Y_predicted[i]=2

datawrite(path_test_out,'qeSVM', 'Im16', Y_predicted)
```