# Lab Assignment 1

## Important notes

**Submission deadline:**
* **Regular problems: last lab session before or on Monday, 21.10.19**
* **Bonus problems: deadline for Lab Assignment 2**

**Points: 11 + 5 bonus points**

Please note: some of the assignments are tedious or boring if you are already a NumPy ninja. The bonus problems were designed to give you a more satisfying alternative.

The assignment is in the form of a Jupyter notebook. We will be using [Google Colab](https://colab.research.google.com) to solve it. Below you will find a "Setup" section. Follow instructions from this paragraph to download the notebook and open it using [Google Colab](https://colab.research.google.com).

Your goal is to solve problems posted below. Whenever possible, add your solutions to the notebook.

Please email us about any problems with it - we will try to correct them quickly. Also, please do not hesitate to use GitHubâ€™s pull requests to send us corrections!

## Setup

### 1. Open the notebook using Google Colab

1. From Github: Click on "View in Colaboratory", then save to your Google Drive.
2. Alternatively upload manually to Drive:
  1. Download the notebook or clone https://github.com/janchorowski/ml_uwr.
  2. Go to  [Google Colab](https://colab.research.google.com).
  3. Go to "UPLOAD" tab and select a local copy of the notebook that you downloaded in point 1.

Colab Tips:
1. Set tab width to 4 spaces under `Tools` â†’ `Preferences`.

### 2. Open the notebook offline using Jupyter/IPython

This notebook can be opened using Jupyter notebook. Simply install a scientific Python distribution on your computer (e.g. [Anaconda](https://www.anaconda.com/) or [WinPython](http://winpython.github.io/)), clone the repository https://github.com/janchorowski/nn_assignments and run `jupyter notebook`.

###   3. Install required dependencies, download data and import packages

Run cells below. To run a cell either click it and click a run button or press "shift + enter"

```python
# Please note that this code needs only to be run in a fresh runtime.
# However, it can be rerun afterwards too.
!pip install -q gdown httpimport
![ -e mnist.npz ] || gdown 'https://drive.google.com/uc?id=1QPaC3IKB_5tX6yIZgRgkpcqFrfVqPTXU' -O mnist.npz
```

    Downloading...
    From: https://drive.google.com/uc?id=1QPaC3IKB_5tX6yIZgRgkpcqFrfVqPTXU
    To: /Users/michal/ML/Assignment1/mnist.npz
    55.4MB [01:38, 624kB/s]

```python
# Standard IPython notebook imports
%matplotlib inline

import os

import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook
import scipy.stats as sstats

import seaborn as sns
from sklearn import datasets

# In this way we can import functions straight from github
with httpimport.github_repo('janchorowski', 'nn_assignments',
                            module='common', branch='nn18'):
     from common.plotting import plot_mat

sns.set_style('whitegrid')
```

### 4. Follow the notebook and solve problems posted below

## Problems

### Problem 0 [0p]

1. To learn more about Jupyter,  read [Jupyter tutorial from Data Analysis in Biological Sciences course at Caltech](http://bebi103.caltech.edu/2015/tutorials/t0b_intro_to_jupyter_notebooks.html) (which itself can be downloaded as a Jupyter notebook). Feel free to skip the tutorial if you have some prior experience with Jupyter notebook.
2. To learn more about basic Google Colab features, go to [Google Colab](https://colab.research.google.com) and select "Overview of Colaboratory Features" in "EXAMPLES" tab. To learn more about / set up useful keyboard shortcuts (e.g. to add a new cell without clicking ""+ code"), go to "Tools --> Keyboard shortcuts"

### Problem 1: NumPy [2p]

First, get familiar with Python at https://docs.python.org/3/tutorial/. Then, get
to know the capabilities of NumPy, the prime numerical library of Python http://www.numpy.org/, for instance with the tutorial at http://wiki.scipy.org/Tentative_NumPy_Tutorial. Finally, look into Pandas at https://pandas.pydata.org/pandas-docs/stable/getting_started/tutorials.html.

You might also need:
  1. another intro to NumPy,
http://people.duke.edu/~ccc14/pcfb/numerics.html
  2. a better interactive shell for Python
http://ipython.org/
  3. a plotting library for Python
http://matplotlib.org/
  4. nice statistical plots for matplotlib https://seaborn.pydata.org/.

**a) Declare variables:**
1. $a=10$,
2. $b=2.5\times 10^{23}$,
3. $c=2+3i$, where $i$ is an imaginary unit,
4. $d=e^{i2\pi/3}$, where $i$ is an imaginary unit, $e$ is the Euler's number (use `exp`, `pi`).

```python
import cmath
a = 10
b = 2.5*pow(10.0,23.0)
c = complex(2,3)
d = cmath.exp(cmath.pi*2j/3)
a, b, c, d
```

    (10, 2.5e+23, (2+3j), (-0.4999999999999998+0.8660254037844388j))

**b) Declare vectors:**
1. $aVec=\begin{bmatrix} 3.14 & 15 & 9 & 26 \end{bmatrix}$,
2. $bVec=\begin{bmatrix} 5 & 4.8 & \cdots & -4.8 & -5 \end{bmatrix}$ (vector of numbers from $5$ to $-5$ decreasing by $0.2$),
3. $cVec=\begin{bmatrix} 10^0 & 10^{0.01} & \cdots & 10^{0.99} & 10^1 \end{bmatrix}$ (logarithmically spaced numbers from 1 to 10, use `logspace` and make sure, that the result has correct length!),
4. $dVec=Hello$ ($eVec$ is a string of characters, thus a vector).

```python
aVec = np.asmatrix((3.14,15,9,26))
bVec = np.transpose(np.array([[2.71,8,28,182]]))
cVec = np.array([x/5 for x in range(25,-26,-1)])
dVec = np.array([10**(x/100) for x in range(0,101)])
eVec = np.array(list('Hello'))
```

**c) Declare matrices:**
1. $aMat=\begin{bmatrix}
                    2      & \cdots & 2 \\
                    \vdots & \ddots & \vdots \\
                    2      & \cdots & 2
                \end{bmatrix}$,

matrix $9\times 9$ filled with 2s (use `ones` or `zeros`),
2. $bMat=\begin{bmatrix}
                    1      & 0      & \cdots &        & 0      \\
                    0      & \ddots & 0      &        & 0      \\
                    \vdots & 0      & 5      & 0      & \vdots \\
                           &        & 0      & \ddots & 0      \\
                    0      &        & \cdots & 0      & 1
                \end{bmatrix}$,

matrix $9\times 9$ filled with zeros, with $\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 4 & 3 & 2 & 1 \end{bmatrix}$ on its diagonal (use `zeros`, `diag`),
3. $cMat=\begin{bmatrix}
                    1      & 11     & \cdots & 91     \\
                    2      & 12     & \cdots & 92     \\
                    \vdots & \vdots & \ddots & \vdots \\
                    10     & 20     & \cdots & 100
                \end{bmatrix}$,

matrix $10\times 10$, columns of which form the vector $1:100$ (use `reshape`),
4. $dMat=\begin{bmatrix}
                    NaN & NaN & NaN & NaN \\
                    NaN & NaN & NaN & NaN \\
                    NaN & NaN & NaN & NaN
                \end{bmatrix}$,

matrix $3\times 4$ filled with `NaN`s (use... `NaN`),
5. $eMat=\begin{bmatrix}
                    13  & -1  & 5  \\
                    -22 & 10  & -87
                \end{bmatrix}$,
6. $fMat$ of shape $3\times 3$ filled with random natural numbers from $[-3,3]$ (use `rand` and `floor` or `ceil`).

```python
import random
aMat = np.ones((9,9))*2
bMat = np.diag(list(range(1,6)) + list(range(4,0,-1)))
cMat = np.arange(1,101).reshape(10, 10, order='F')
dMat = np.ones((3,4))*np.NaN
eMat = np.array([[13,-1,5],[-22,10,-87]])
fMat = np.array([random.randint(-3,3) for x in range(9)]).reshape(3,3)
```

**d) Declare a multiplication table**
as a $10\times 10$ matrix `mulMat`. Use matrix/vector multiplication.

```python
mulMat = (np.arange(1,11)).reshape(10,1)*(np.arange(1,11))
mulMat
```

    array([[  1,   2,   3,   4,   5,   6,   7,   8,   9,  10],
           [  2,   4,   6,   8,  10,  12,  14,  16,  18,  20],
           [  3,   6,   9,  12,  15,  18,  21,  24,  27,  30],
           [  4,   8,  12,  16,  20,  24,  28,  32,  36,  40],
           [  5,  10,  15,  20,  25,  30,  35,  40,  45,  50],
           [  6,  12,  18,  24,  30,  36,  42,  48,  54,  60],
           [  7,  14,  21,  28,  35,  42,  49,  56,  63,  70],
           [  8,  16,  24,  32,  40,  48,  56,  64,  72,  80],
           [  9,  18,  27,  36,  45,  54,  63,  72,  81,  90],
           [ 10,  20,  30,  40,  50,  60,  70,  80,  90, 100]])

**e) Compute element-wise using values from b).**

For instance, the first element of $xVec[0]$ should be equal to

\begin{equation}
1/(\sqrt{2\pi2.5^2}) e^{-cVec[0]^2 / (2\cdot\pi 2.5^2)}.
\end{equation}

1. $xVec=1/(\sqrt{2\pi2.5^2}) e^{-cVec^2 / (2\cdot\pi 2.5^2)}$
2. $yVec=\log_{10}(1/cVec)$, using `log10`

```python
import math
xVec = np.array(
    [
     1/(2*np.pi*2.5**2)**0.5*np.exp(-x**2/(2*np.pi*2.5**2))
     for x in cVec
    ]
)
yVec = np.array([math.log(1/x,10) for x in dVec])
```

**f) Compute with matrix/vector operations using values from c).**

**NOTE:** Every multiplication (and power) in this subtask is a [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication).
1. $xMat=[0,  1, ..., 6][10, 20, ..., 60]^T$,
2. $yMat=[10, 20, ..., 60]^T[0,  1, ..., 6]$

(remember, that matrix multiplication is not commutative).

```python
xMat = np.arange(0, 7, 1).reshape(-1,1) @ np.arange(10, 70, 10).reshape(1,-1)
yMat = np.arange(0, 70, 10).reshape(1,-1) @ np.arange(0, 7, 1).reshape(-1,1)

```

**g) Declare `ismagic(A)` function**
which checks if matrix $A$ is a [magic square](https://en.wikipedia.org/wiki/Magic_square) and returns a boolean.

```python
def ismagic(A):
    if(len(A)==0):
        return False
    if(np.unique(A).size!=A.size):
        return False

    d1_sum=np.trace(A)
    d2_sum=np.trace(A[:,::-1])

    if (d2_sum!=d1_sum):
        return False
    c_sum = np.sum(A,axis=0)

    if(np.unique(c_sum).size!=1 or c_sum.item(0)!=d1_sum):
        return False

    r_sum = np.sum(A,axis=1)
    if(np.unique(r_sum).size!=1 or r_sum.item(0)!=d1_sum):
        return False

    return True

assert not ismagic(np.array([[1,1], [2,2]]))
assert ismagic(np.array([[2,7,6],[9,5,1],[4,3,8]]))
```

### Problem 2: Pandas and Seaborn [2p]

1. Load the IRIS Data into a `DataFrame`

```python
iris_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'
# Use read_csv to load the data. Make sure you get 150 examples!
iris_df = pd.read_csv(iris_url, header=None)

# Set the column names to
# 'sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'target'
iris_df.columns = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'target']

# Print the first 10 entries
iris_df.head(10)
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      target

      0
      5.1
      3.5
      1.4
      0.2
      Iris-setosa

      1
      4.9
      3.0
      1.4
      0.2
      Iris-setosa

      2
      4.7
      3.2
      1.3
      0.2
      Iris-setosa

      3
      4.6
      3.1
      1.5
      0.2
      Iris-setosa

      4
      5.0
      3.6
      1.4
      0.2
      Iris-setosa

      5
      5.4
      3.9
      1.7
      0.4
      Iris-setosa

      6
      4.6
      3.4
      1.4
      0.3
      Iris-setosa

      7
      5.0
      3.4
      1.5
      0.2
      Iris-setosa

      8
      4.4
      2.9
      1.4
      0.2
      Iris-setosa

      9
      4.9
      3.1
      1.5
      0.1
      Iris-setosa

```python
# Show numerical summary of the data, using DataFrame.describe()
iris_df.describe()
```

      sepal_length
      sepal_width
      petal_length
      petal_width

      count
      150.000000
      150.000000
      150.000000
      150.000000

      mean
      5.843333
      3.054000
      3.758667
      1.198667

      std
      0.828066
      0.433594
      1.764420
      0.763161

      min
      4.300000
      2.000000
      1.000000
      0.100000

      25%
      5.100000
      2.800000
      1.600000
      0.300000

      50%
      5.800000
      3.000000
      4.350000
      1.300000

      75%
      6.400000
      3.300000
      5.100000
      1.800000

      max
      7.900000
      4.400000
      6.900000
      2.500000

```python
# Plot the data using seaborn's pairplot
sns.set(style="ticks")
sns.pairplot(iris_df, hue="target")
```

The Iris data is in a so-called 'wide' format, in which each column corresponds to a variable and each row of the DataFrame corresponds to one observation. Turn it into a 'long' format in which each row is a measurement.

Specifically, change the data layout of the IRIS dataFrame so that it has 3 columns:
- variable (one of sepal_length, sepal_width, petal_length, petal_width)
- value
- target

If you would like to learn more, [Tidy Data](https://www.jstatsoft.org/index.php/jss/article/view/v059i10/v59i10.pdf) by [Hadley Wickham](http://hadley.nz/) provides a very nice explanation of best practices for data formating.

Hint: look at reshaping functions in http://pandas.pydata.org/Pandas_Cheat_Sheet.pdf

```python
iris_df_long = pd.melt(iris_df, id_vars=['target'],var_name='variable', value_name='value')
iris_df_long.head()
```

      target
      variable
      value

      0
      Iris-setosa
      sepal_length
      5.1

      1
      Iris-setosa
      sepal_length
      4.9

      2
      Iris-setosa
      sepal_length
      4.7

      3
      Iris-setosa
      sepal_length
      4.6

      4
      Iris-setosa
      sepal_length
      5.0

Now create a box-plot of values that each variable takes, split by the target species.

```python
# Hint: use a `catplot`
sns.catplot(x='target', y='value', data=iris_df_long,  kind="box", col="variable" )

# TODO: create two more plots, using a boxenplot and a swarmplot.

sns.catplot(x='target', y='value', data=iris_df_long,  kind="boxen", col="variable" )
sns.catplot(x='target', y='value', data=iris_df_long,  kind="swarm", col="variable" )
```

### k-Nearest Neighbors

We will use the loaded Iris data describing iris flowers
and shows relations between their length and petal width for three
species (namely: setosa, versicolor, virginica).

For this exercise we will restrict our analysis to just two variables: **petal length** and **petal width**.

```python
unknown_df = pd.DataFrame(
    [[1.5, 0.3, 'unknown'],
     [4.5, 1.2, 'unknown'],
     [5.1, 1.7, 'unknown'],
     [5.5, 2.3, 'unknown']],
     columns=['petal_length', 'petal_width', 'target'])

sns.scatterplot(x='petal_length', y='petal_width', hue='target', data=iris_df)
sns.scatterplot(x='petal_length', y='petal_width', color='gray', marker='v',
                label='unknown', s=70, data=unknown_df)
```

Based on these two features, it is easy to distinguish iris setosa from the two remaining species. Yet iris versicolor and virginica remain mixed together.

Looking closely at the plot, we might estimate the species of the selected unknown irises (gray triangles). For three of them the answer seems obvious â€“ they belong in uniformly-colored areas covered by one species only. Yet unknown iris flower in (5.1, 1.7) is troublesome â€“ it lays on the boundary of versicolor and virginica clusters. We can assume, that its species is the one of the closest one to it, coming from the training set (and so having a label).

K-Nearest Neighbors method (http://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) solves the classification problem, i.e. sets class labels (species in case of irises) of a previously unseen sample by choosing the most common class among the top k neighbors of the sample in question (for instance according to the Euclidean distance). Thus, the k-Nearest Neighbors algorithm works as follows. For each unlabeled sample x:
1. Find k nearest neighbors among the labeled samples.
2. Set the most common label among them as label of x.

#### Problem 3 [3p]

##### Implement the k-Nearest Neighbors algorithm [1p].

Take advantage of matrix calculus rather than using for loops.

**Tip:** What is computed by \begin{equation} \sqrt{(X - Y)^T (X - Y)} \end{equation} when both X and Y are vectors?

**Tip:** Try to use broadcasting (NumPy: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) and built-ins sort, numpy.sort, numpy.argsort (sorting), scipy.stats.mode (choosing the most common element of the set).

```python
def KNN(train_X, train_Y, test_X, ks, verbose=False):
    """
    Compute predictions for various k
    Args:
        train_X: array of shape Ntrain x D
        train_Y: array of shape Ntrain
        test_X: array of shape Ntest x D
        ks: list of integers
    Returns:
        preds: dict k: predictions for k
    """
    # Cats data to float32
    train_X = train_X.astype(np.float32)
    test_X = test_X.astype(np.float32)

    # Alloc space for results
    preds = {}

    if verbose:
        print("Computing distances... ", end='')
    #
    # TODO: fill in an efficient distance matrix computation
    #
    distans = np.sqrt(
        -2*train_X.dot(test_X.T)
        + (train_X**2).sum(axis=1).reshape(-1,1)
        + (test_X**2).sum(axis=1).reshape(1,-1)
        )

    if verbose:
        print("Sorting... ", end='')

    # TODO: findes closest trainig points
    # Hint: use argsort
    closest = distans.argsort(axis=0)

    if verbose:
        print("Computing predictions...", end='')

    targets = train_Y[closest]

    for k in ks:
        predictions = sstats.mode(targets[:k], axis=0)[0]

        predictions = predictions.ravel()
        preds[k] = predictions
    if verbose:
        print("Done")
    return preds
```

```python
# Now classify the 4 unknown points
iris_x = np.array(iris_df[['petal_length', 'petal_width']])
iris_y = np.array(iris_df['target'])

unknown_x = np.array(unknown_df[['petal_length', 'petal_width']])

KNN(iris_x, iris_y, unknown_x, [1, 3, 5, 7])
```

    {1: array(['Iris-setosa', 'Iris-versicolor', 'Iris-versicolor',
            'Iris-virginica'], dtype=object),
     3: array(['Iris-setosa', 'Iris-versicolor', 'Iris-versicolor',
            'Iris-virginica'], dtype=object),
     5: array(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica',
            'Iris-virginica'], dtype=object),
     7: array(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica',
            'Iris-virginica'], dtype=object)}

##### Plot the Decision boundary [1p]

Use meshgrid to generate the points in the space spanned by data
Than map the classes to numbers 0, 1, 2 and make a contour plot with the
decision boundary

```python
iris_x = np.array(iris_df[['petal_length', 'petal_width']])
iris_y = np.array(iris_df['target'])
plot_step = 0.01

mesh_x, mesh_y = np.meshgrid(
    np.arange(np.min(iris_x[:,0]) - 1, np.max(iris_x[:,0]) + 1, plot_step),
    np.arange(np.min(iris_x[:,1]) - 1, np.max(iris_x[:,1]) + 1, plot_step)
)
# use np.unique with suitable options to map the class names to numbers
target_names, iris_y_ids = np.unique(iris_y, return_inverse=True)
mesh_data = np.hstack([mesh_x.reshape(-1, 1), mesh_y.reshape(-1, 1)])

preds = KNN(iris_x, iris_y_ids, mesh_data, [1, 3, 5, 7])
for k, preds_k in preds.items():
    plt.figure()
    plt.title(f"Decision boundary for k={k}")
    plt.contourf(mesh_x, mesh_y, preds_k.reshape(mesh_x.shape), cmap='spring')
    plt.scatter(iris_x[:,0], iris_x[:,1], c=iris_y_ids, cmap='spring', edgecolors='black')

```

##### Estimate performance for various ks [1p]
Consider the following experiment:
1. We scramble the data and split it into two parts - training set (66.6% of all samples) and test set (33.4%).
2. Based on the training set, we use the k-NN algorithm to predict the labels on the test set.
3. We then check the number of errors and write it down.

Do this 500 times for k âˆˆ {1, 3, 5, ..., 19}. Plot a function of the average number of errors as the function of k. It should be similar to the one below.

```python
#TODO: write a function to compute error rates
def err_rates(preds, test_Y):
    ret = {}
    for k, preds_k in  preds.items():
        # TODO: fill in error count computation
        ret[k] = np.sum(preds_k != test_Y)/len(preds_k)
    return ret
```

```python
iris_x = np.array(iris_df[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']])
iris_y = np.array(iris_df['target'])

ks = range(1, 20, 2)
results = []

for _rep in tqdm_notebook(range(1000)):
    #TODO
    # Use np.split and np.permutation to get training and testing indices
    train_idx, test_idx = np.split(
        np.random.permutation(len(iris_y)),
        [(len(iris_y) * 2) // 3]
    )

    #TODO: apply your kNN classifier to data subset
    preds = KNN(iris_x[train_idx], iris_y[train_idx], iris_x[test_idx], ks)
    errs = err_rates(preds, iris_y[test_idx])

    for k, errs_k in errs.items():
        results.append({'K':k, 'err_rate': errs_k})

# results_df will be a data_frame in long format
results_df = pd.DataFrame(results)

plt.figure()
sns.regplot("K", 'err_rate',  data=results_df, x_estimator=np.mean, order=2, truncate=True,)
plt.figure()
sns.regplot("K", 'err_rate',  data=results_df, x_estimator=np.mean, order=2, )
```

#### Problem 5 [2p + 2p bonus]

Download a categorical dataset from UCI and try to find the most predictive variables

```python
columns = [
 "target", "cap-shape", "cap-surface", "cap-color", "bruises?", "odor",
 "gill-attachment", "gill-spacing", "gill-size", "gill-color", "stalk-shape",
 "stalk-root", "stalk-surface-above-ring", "stalk-surface-below-ring",
 "stalk-color-above-ring", "stalk-color-below-ring", "veil-type", "veil-color",
 "ring-number", "ring-type", "spore-print-color", "population", "habitat", ]

# Use read_csv to load the data.
url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/agaricus-lepiota.data'
mushroom_df = pd.read_csv(url, header=None, names=columns)
mushroom_df.head()
```

      target
      cap-shape
      cap-surface
      cap-color
      bruises?
      odor
      gill-attachment
      gill-spacing
      gill-size
      gill-color
      ...
      stalk-surface-below-ring
      stalk-color-above-ring
      stalk-color-below-ring
      veil-type
      veil-color
      ring-number
      ring-type
      spore-print-color
      population
      habitat

      0
      p
      x
      s
      n
      t
      p
      f
      c
      n
      k
      ...
      s
      w
      w
      p
      w
      o
      p
      k
      s
      u

      1
      e
      x
      s
      y
      t
      a
      f
      c
      b
      k
      ...
      s
      w
      w
      p
      w
      o
      p
      n
      n
      g

      2
      e
      b
      s
      w
      t
      l
      f
      c
      b
      n
      ...
      s
      w
      w
      p
      w
      o
      p
      n
      n
      m

      3
      p
      x
      y
      w
      t
      p
      f
      c
      n
      n
      ...
      s
      w
      w
      p
      w
      o
      p
      k
      s
      u

      4
      e
      x
      s
      g
      f
      n
      f
      w
      b
      k
      ...
      s
      w
      w
      p
      w
      o
      e
      n
      a
      g

5 rows × 23 columns

Implement the function `entropy` to compute the entropy of a columnt of the dataset.

The [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)) of a discrete variable is defined to be:

$$H(X) = -\sum_x p_X(x) \log_2 p_X(x)$$.

A good in tutorial is given by Chris Olah: https://colah.github.io/posts/2015-09-Visual-Information/.

When $X$ is a discrete random variables, we can estimate the probabilities with counts:

$$p_X(x) = \frac{\text{number of instances where }X=x}{\text{total number of instances}}$$.

Hint: the following `pandas` functions may be useful:
- `count`
- `value_count`

Then use the datafranme's `apply` function to compute the entropy of all columns.

```python
def entropy(series):
    values_counts = series.value_counts()
    count = series.count()
    return - np.sum(values_counts * np.log2(values_counts / count)) / count

mushroom_df.apply(entropy)
```

    target                      0.999068
    cap-shape                   1.652889
    cap-surface                 1.575486
    cap-color                   2.510143
    bruises?                    0.979327
    odor                        2.319414
    gill-attachment             0.173129
    gill-spacing                0.637878
    gill-size                   0.892256
    gill-color                  3.030433
    stalk-shape                 0.986927
    stalk-root                  1.822922
    stalk-surface-above-ring    1.221348
    stalk-surface-below-ring    1.399135
    stalk-color-above-ring      1.936809
    stalk-color-below-ring      1.978163
    veil-type                  -0.000000
    veil-color                  0.196238
    ring-number                 0.420680
    ring-type                   1.535121
    spore-print-color           2.203227
    population                  2.003398
    habitat                     2.274747
    dtype: float64

Implement the conditional entropy computation

$$H(Y|X) = \sum_x p_X(x) H(Y|x) = \sum_x p_X(x) \sum_y p_Y(y|x) \log_2 p_Y(y|x)$$

Hint 1: the above formlua can be computed as follows:
1. split the data by the values of $X$
2. for each value $x$ that $X$ takes, compute the entropy of $Y$
3. average the entropies, weighting them by how frequent the $x$ value ocurred.

Hint 2: helpful pandas constructs are:
- `groupby` and `agg`
- you can aggregate a grouping using your own custom functions

```python
def cond_entropy(df, X, Y):
    """Compute the conditional H(X|Y) in dataframe df
    Args:
        df: a dataframe
        X: the name of the conditioning columt
        Y: the name of the column whose entropy we wish to compute
    """
    grouped = df.groupby(X)[Y]
    return grouped.agg(entropy).agg(np.average, weights=grouped.agg('count'))

```

```python
# Now for each column C compute the conditional entropy H(target|C)
# Knowing which variable tells us the most about the target?
for cname in mushroom_df.columns:
    print(f"{cname}: {cond_entropy(mushroom_df, cname, 'target')}")
```

    target: 0.0
    cap-shape: 0.9502711949370874
    cap-surface: 0.9704776640986875
    cap-color: 0.9630186138962564
    bruises?: 0.8066884111112408
    odor: 0.09299291948846057
    gill-attachment: 0.984902869621844
    gill-spacing: 0.8981847128758902
    gill-size: 0.7689135217244145
    gill-color: 0.5820903734563291
    stalk-shape: 0.9915511243027961
    stalk-root: 0.8642502592451845
    stalk-surface-above-ring: 0.7143422976539759
    stalk-surface-below-ring: 0.727173423479714
    stalk-color-above-ring: 0.7452227234102207
    stalk-color-below-ring: 0.7576523303448939
    veil-type: 0.9990678968724604
    veil-color: 0.9752508807515438
    ring-number: 0.96061522762937
    ring-type: 0.6810463860789229
    spore-print-color: 0.5183629791875449
    population: 0.7971098778056079
    habitat: 0.8422342922673683

Bonus questions:
- **[1p]** Implement computation of [Mutual Information ](https://en.wikipedia.org/wiki/Mutual_information)
- **[1p]** Add an ID column, that assigns a unique ID to each observation (row). Comompute the $H(target|ID)$ and the mutual information between target and ID. How to interpret the results? Do you think the ID is important in predicting the target?

```python
def mutual_information(df, x, y):
    if x == y:
        return 1
    X, Y = x, y
    new_df = df[[X,Y]]
    freq_x = new_df.groupby(x).agg('count')/len(df)
    freq_y = new_df.groupby(y).agg('count')/len(df)
    freq_x_y = df.groupby([x, y]).agg('count')/len(df)
    res = 0
    for a in freq_x_y.index:
        a_xy = freq_x_y.loc[a].agg(np.max).astype('float')
        a_x = np.max(freq_x.loc[a[0]].astype('float'))
        a_y = np.max(freq_y.loc[a[1]].astype('float'))
        res += a_xy * np.log2(a_xy / (a_x * a_y))

    return res

for cname in mushroom_df.columns:
    print(f"{cname}: {mutual_information(mushroom_df, cname, 'target')}")
```

    target: 1.0
    cap-shape: 0.9990678968724606
    cap-surface: 0.9990678968724602
    cap-color: 0.9990678968724603
    bruises?: 0.9990678968724607
    odor: 0.9990678968724606
    gill-attachment: 0.9990678968724603
    gill-spacing: 0.9990678968724606
    gill-size: 0.9990678968724607
    gill-color: 0.9990678968724604
    stalk-shape: 0.9990678968724603
    stalk-root: 0.9990678968724602
    stalk-surface-above-ring: 0.9990678968724606
    stalk-surface-below-ring: 0.9990678968724604
    stalk-color-above-ring: 0.9990678968724607
    stalk-color-below-ring: 0.9990678968724606
    veil-type: 0.9990678968724604
    veil-color: 0.9990678968724606
    ring-number: 0.9990678968724604
    ring-type: 0.9990678968724606
    spore-print-color: 0.9990678968724603
    population: 0.9990678968724604
    habitat: 0.9990678968724607

```python
id_mashroom_df = mushroom_df.copy()
id_mashroom_df['id'] = list(range(len(id_mashroom_df)))
print(f"id: {cond_entropy(id_mashroom_df, 'id', 'target')}")
```

    id: 0.0

#### Problem 6 [2p]

Apply the K-Nearest Neighbors (K-NN) algorithm to the MNIST dataset.

The MNIST (http://yann.lecun.com/exdb/mnist/) dataset consists of normalized (centered and stretched) scans of hand-written digits. Specifically, each element of the dataset is a 28 Ã— 28 grayscale image, thus having 764 8-bit pixels.

1. Display a few objects from each of the classes, paying attention to aesthetics and clarity of your presentation. **Note:** You already downloaded the dataset in "Setup" section. Please use the code below to get started.

2. **[2p]** Apply a k-NN classifier to the MNIST dataset. First, divide the training set into two parts, which we will call training and validation. On MNIST use the first 50000 samples for training and the last 10000 for validation. Then find the optimal number of neighbors by assessing the accuracy on the validation set. You do not need to repeat this experiment multiple times. Finally, compute the accuracy on the test set obtained with the best previously chosen number of neighbors. On MNIST you should get about 3% errors. Pick a few mislabeled samples from the test dataset and plot them along with the correct ones. **Note:**
  * MNIST is much larger than the Iris dataset. A good implementation may need a few minutes depending on your runtime type. Please optimize your algorithm:
  * Compute the distances only once, then test for different values of k.
  * Use vectorized expressions to compute the distance. It is possible to compute all distances between the training and testing points in one expression. Hint: think about the vectorized expression \begin{equation}(X - Y)^T (X - Y)\end{equation}
  * You can use single precision numbers in computation.
  * If your code is taking a long time to execute, please save its results before the lab session.

**Note:** in NumPy, matrices have its own data type (dtype), which is retained during
calculations. Please pay attention to it. I particular, do not subtract values of data types not
having the sign bit, do not divide integers, etc. Results of such operations will not be
automatically casted to types having the required precision.

```python
with np.load('mnist.npz') as data:
    mnist_full_train_data_uint8 = data['train_data']
    mnist_full_train_labels_int64 = data['train_labels']
    mnist_test_data_uint8 = data['test_data']
    mnist_test_labels_int64 = data['test_labels']

# Split train data into train and validation sets
mnist_train_data_uint8 = mnist_full_train_data_uint8[:50000]
mnist_train_labels_int64 = mnist_full_train_labels_int64[:50000]
mnist_valid_data_uint8 = mnist_full_train_data_uint8[50000:]
mnist_valid_labels_int64 = mnist_full_train_labels_int64[50000:]
```

```python
plot_mat(mnist_train_data_uint8[:20, None], cmap='gray')
```

```python
# MNIST is large.
# Implement a batched KNN classifier, which processes the test data in small batches
# and returns the error rates

# The code should not run for more than a couple of minutes on the Colab runtime,
# If it is slower, optimize the distance computation in KNN

def batched_KNN(train_X, train_Y, test_X, ks, verbose=False, batch_size=200):
    all_preds = {k: [] for k in ks}
    for i in range(0, test_X.shape[0], batch_size):
        batch_X = test_X[i:i + batch_size]
        if verbose:
            print(f"Processing batch {i}:{i + batch_X.shape[0]}... ", end='')

        # TODO: run KNN on the batch and save the predictions
            preds = KNN(train_X, train_Y, batch_X, ks, verbose)
        # TODO: combine predictions from batches
            for k in all_preds.keys():
                all_preds[k] = np.concatenate((all_preds[k], preds[k]))
    return all_preds
```

```python
# Now find the best k on the validation set
ks = [1, 3, 5, 7, 9]
mnist_validation_preds = batched_KNN(
    mnist_train_data_uint8.astype('float32').reshape(-1, 28*28), mnist_train_labels_int64,
    mnist_valid_data_uint8.astype('float32').reshape(-1, 28*28),
    ks, verbose=True)

mnist_validation_errs = err_rates(mnist_validation_preds, mnist_valid_labels_int64)
plt.plot(ks, [mnist_validation_errs[k] for k in ks])
```

```python
# Now use the best k to compute the test error

best_K = 3

mnist_test_preds = batched_KNN(
    mnist_full_train_data_uint8.astype('float32').reshape(-1, 28*28),
    mnist_full_train_labels_int64,
    mnist_test_data_uint8.astype('float32').reshape(-1, 28*28),
    [best_K], verbose=True)

mnist_test_errs = err_rates(mnist_test_preds, mnist_test_labels_int64)
print(f"\n\nWhen k={best_K} the test error rate is {mnist_test_errs[best_K] * 100.0:.1f}%%")
```

    Processing batch 0:200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 200:400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 400:600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 600:800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 800:1000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 1000:1200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 1200:1400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 1400:1600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 1600:1800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 1800:2000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 2000:2200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 2200:2400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 2400:2600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 2600:2800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 2800:3000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 3000:3200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 3200:3400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 3400:3600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 3600:3800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 3800:4000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 4000:4200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 4200:4400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 4400:4600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 4600:4800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 4800:5000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 5000:5200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 5200:5400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 5400:5600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 5600:5800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 5800:6000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 6000:6200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 6200:6400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 6400:6600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 6600:6800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 6800:7000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 7000:7200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 7200:7400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 7400:7600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 7600:7800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 7800:8000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 8000:8200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 8200:8400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 8400:8600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 8600:8800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 8800:9000... Computing distances... Sorting... Computing predictions...Done
    Processing batch 9000:9200... Computing distances... Sorting... Computing predictions...Done
    Processing batch 9200:9400... Computing distances... Sorting... Computing predictions...Done
    Processing batch 9400:9600... Computing distances... Sorting... Computing predictions...Done
    Processing batch 9600:9800... Computing distances... Sorting... Computing predictions...Done
    Processing batch 9800:10000... Computing distances... Sorting... Computing predictions...Done

    When k=3 the test error rate is 2.9%%

### Locality sensitive hashing

Problem 5 was about speeding up the inference using loops implicitly present in matrix multiplication instead of explicit loops in Python. In this problem, we will explore a strategy to truly reduce the total number of computations required to find nearest neighbors without sacrificing too much accuracy.

To speed up nearest neighbor search we will employ *Locality Sensitive Hashing (LSH)* functions. For a given distance metric, the locality sensitive hash should put items that are similar into the same bucket. Notice that this is essentially a design choice opposite to traditional cryptographic hash functions that should amplify the difference of similar inputs (typically we want that small perturbations of data result in large changes to the hash value).

One of the simplest implementations of LSH approximates the cosine distance. Let $x\in \mathbb{R}^N$ and $y\in \mathbb{R}^N$ be two vectors. Their cosine distance is defined as:

\begin{equation}
    d_\text{cos}(x,y) = \frac{x \cdot y}{\|x\| \|y\|} = \cos\left(\theta(x,y)\right),
\end{equation}
where $\theta(x,y)$ is the unsigned angle between $x$ and $y$.

We will construct a family $H$ of hash functions that are an LSH for angle distances (an approximation to cosine distance). Assume $p\in \mathbb{R}^N$ is a random vector (components are sampled from the normal distribution) of length 1. Then define the hash function $h(x) = \text{sgn}(x\cdot p)$, where $\text{sgn()}$ is the sign function. It can be proven that:

\begin{equation}
    p_{h\in H}[h(x)=h(y)] = 1 - \frac{\theta(x,y)}{\pi}.
\end{equation}

The equation means that the probability of a hash collision grows as the the angle between two vectors gets smaller. Therefore, vectors that are close according to the cosine distance will be put with high probability into the same bin (we use the fact that for small $\theta$ we can approximate $\cos(\theta) = 1 - \theta/\pi$.

We will say that a family of randomly chosen hash functions $H$ is $(d_1, d_2, p_1, p_2)$-sensitive with respect to a distance metric $d$ if for any $x$ and $y$:
1. If $d(x,y) \leq d_1$ then $p_{h\in H}[h(x)=h(y)] \geq p_1$.
2. If $d(x,y) \geq d_2$ then $p_{h\in H}[h(x)=h(y)] \leq p_2$.

For example, our family of randomly chosen hyperplanes is $(d_1, d_2, (1-d_1)/\pi, (1-d_2)/\pi)$-sensitive.

Ideally, vectors should be placed into the same bin with a high probability if their distance is smaller than a threshold, and with a low probability if their distance is larger that the threshold. By combining hashing functions we can get closer to this ideal sensitivity.

Given a family of hash functions $H$ with sensitivity $(d_1, d_2, p_1, p_2)$ we can construct a new family $H'$ by combining $r$ functions from $H$:
1. AND: let $h=[h_1, h_2, \ldots, h_r] \in H'$ and $h(x)=h(y)$ if and only if $\forall_i h_i(x)=h_i(y)$. Then $H'$ is $(d_1, d_2, (p_1)^r, (p_2)^r)$-sensitive.
2. OR: let $h=[h_1, h_2, \ldots, h_r] \in H'$ and $h(x)=h(y)$ if and only if $\exists_i h_i(x)=h_i(y)$. Then $H'$ is $(d_1, d_2, 1-(1-p_1)^r, 1-(1-p_2)^r)$-sensitive.

AND makes all probabilities shrink, but properly choosing $r$ we can make the lower probability approach 0 while the higher does not. Conversely, OR makes all probabilities grow, we can make the upper probability approach 1 while the lower does not.

#### Problem 7 [1-3p bonus]

1. **[1bp for exercises list]** **Note:** you can show sketches of proofs for this assignment.
    1. Show that angle between vectors is a metric (https://en.wikipedia.org/wiki/Metric_(mathematics)).

    2. Show that $p_{h\in H}[h(x)=h(y)] = 1 - \frac{\theta(x,y)}{\pi}$ for $h$ computed using a randomly chosen hyperplane.

    3. Show the properties of either AND or OR boosting of LSH.

  Please show the solution to this problem dirung the Session for Homework 1, the bonus point will also be added to the points from Homework 1.

3. **[1-3bp]** Reimplement k-Nearest Neighbors for MNIST classification using the cosine distance instead of the Euclidean distance. Choose a sensible value of $k$. Use Locality Sensitive Hashing to achieve an error rate no greater than $150\%$ of the original error rate with at least a $90\%$ speedup (i.e., by considering on average at most 5000 training samples per query image). For a few settings plot the speedup-vs-accuracy relation.

  **Note:** points will be awarded based on ingenuity of your solution. Feel free to explore your own ideas!

```python

```


**Submission deadline:**
* **Regular problems: last lab session before or on Monday, 13.01.2020**
* **Bonus problems: Last lab during semester**

**Points: 5 + 9 bonus points**

Please note: some of the assignments are tedious or boring if you are already a NumPy ninja. The bonus problems were designed to give you a more satisfying alternative.

## Heads Up!

This assignment comes with starter code, but you are not forced to use it, as long as you execute all analysis demanded in the problems.

## A note about plots!

Plots are a way of communication. Just lke text, they can be paraphrased. You do not have to exactly reproducy my plots, but you must try to make suer yourp plots tell a similar story:
- label axis
- add titles
- choose plot type properly
- choose a color scale, limits, ticks

so that you can describe what is happening!

## Bugs?!

Please submit Github PRs or email us about any problems with the notebook - we will try to correct them quickly.

```
# Standard IPython notebook imports
%matplotlib inline

import os

from io import StringIO
import itertools
import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook

import scipy.stats as sstats
import scipy.optimize as sopt

import seaborn as sns

import sklearn.datasets
import sklearn.ensemble
import sklearn.svm
import sklearn.tree
from sklearn.utils import resample
from sklearn.model_selection import train_test_split

import cvxopt
from sklearn.metrics.pairwise import rbf_kernel
from scipy.linalg import solve_triangular

# In this way we can import functions straight from github
with httpimport.github_repo('janchorowski', 'nn_assignments',
                            module='common', branch='nn18'):
     from common.plotting import plot_mat

sns.set_style('whitegrid')
```

# SVM Theory
A linear SVM assigns points $x^{(i)}\in\mathbb{R}^n$ to one of two
classes,  $y^{(i)}\in\{-1,1\}$ using the decision rule:

\begin{equation}
y = \text{signum}(w^T x + b).
\end{equation}

SVM training consists of finding weights $w\in\mathbb{R}^n$
and bias $b\in\mathbb{R}$ that maximize the separation margin. This
corresponds to solving the following quadratic optimization problem:

\begin{equation}
\begin{split}
  \min_{w,b,\xi}  &\frac{1}{2}w^Tw  + C\sum_{i=1}^m \xi_i  \\
  \text{s.t. } & y^{(i)}(w^T x^{(i)} + b) \geq 1- \xi_i\;\; \forall_i \\
  & \xi_i \geq 0 \;\; \forall_i.
\end{split}
\end{equation}

# Problem 1 [2p]

Load the iris dataset.

1. [1p] Using the `sklearn.svm.SVC` library train a linear SVM that
separates the Virginica from the Versicolor class using the
petal length and petal width features. Plot the obtained decision boundary and
the support vectors (their locations and weights - coefficients $\alpha$).

2. [.5p] Now train a nonlinear SVM using the Gaussian kernel. Tune the parameetrs `C` and `gamma` (for the kernel) to reach maximum training accurracy. Plot the decision boundary and supprt vectors.

3. [.5p] Answer the following questions:
    - When the SVM is forced to maximally accurate on the train set, roughly how many support vectors do we get ?
    - what is the relationship between the regularization constant `C` and the support vector weights `alpha`?

```
# load iris, extract petal_length and petal_width of versicolors and virginicas

iris = sklearn.datasets.load_iris()
print('Features: ', iris.feature_names)
print('Targets: ', iris.target_names)
petal_length = iris.data[:,iris.feature_names.index('petal length (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]

IrisX = np.array(iris.data.T)
IrisX = IrisX[:, iris.target!=0]

IrisX2F = np.vstack([petal_length, petal_width])
IrisX2F = IrisX2F[:, iris.target!=0]

# Set versicolor=0 and virginia=1
IrisY = (iris.target[iris.target!=0]-1).reshape(1,-1).astype(np.float64)

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
```

```
#
# Fit a linear SVM using libsvm
#

svm_model = sklearn.svm.SVC(gamma='auto', kernel='linear', C=60)
svm_model.fit(IrisX2F.T, IrisY.T)

print("libsvm error rate: %f" % (svm_model.predict(IrisX2F.T)!=IrisY).mean())
```

    libsvm error rate: 0.060000

    /usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)

```
#
# Plot the decision boundary
#

petal_lengths, petal_widths = np.meshgrid(np.linspace(IrisX2F[0,:].min(), IrisX2F[0,:].max(), 100),
                                          np.linspace(IrisX2F[1,:].min(), IrisX2F[1,:].max(), 100))

IrisXGrid = np.vstack([petal_lengths.ravel(), petal_widths.ravel()])
predictions_Grid = svm_model.predict(IrisXGrid.T)

plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

```
#
# Plot the decision boundary and the support vectors.
#
# You can extract the indices of support vectors and their weights from fields of the
# svm object. Display the location of support vectors and their weights (by changing the
# size in the scatterplot)
#
# TODO
#

support_vector_indices = svm_model.support_
support_vector_coefficients = svm_model.dual_coef_
preds = np.dot(svm_model.coef_, svm_model.support_vectors_.T) + support_vector_coefficients[0]

print(support_vector_coefficients)
print(preds)
sizes = np.abs(preds)

plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')
plt.scatter(IrisX2F[0,support_vector_indices], IrisX2F[1,support_vector_indices],
            c=IrisY.ravel()[support_vector_indices],
            s=sizes,
            cmap='spring',
            edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

```
#
# Fit a nonlinear SVM with a Gaussian kernel using libsvm.
# Optimize the SVM to make
#

svm_gauss_model = sklearn.svm.SVC(gamma=90, kernel='rbf', C=100)
svm_gauss_model.fit(IrisX2F.T, IrisY.T.ravel())

print("libsvm error rate: %f" % (svm_gauss_model.predict(IrisX2F.T)!=IrisY).mean())

predictions_Grid = svm_gauss_model.predict(IrisXGrid.T)

support_vector_indices = svm_gauss_model.support_
support_vector_coefficients = svm_gauss_model.dual_coef_

# TODO: plot the decision boundary with support vectors
plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')
plt.scatter(IrisX2F[0,support_vector_indices], IrisX2F[1,support_vector_indices],
            c=IrisY.ravel()[support_vector_indices],
            s=abs(support_vector_coefficients) * 2 + 40,
            cmap='spring',
            edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

# Problem 2 [1p]

Reimplement the linear SVM using the use `cvxopt.solvers.qp`
Quadratic Programming (QP) solver. You will need to define the matrices
that define the problem. Compare the obtained solutions. Extract the
support vectors from the LIBSVM solution and plot the support vectors.

The `cvxopt.solvers.qp` solves the following optimization problem:

\begin{align}
\text{minimize over } x \text{: }& \frac{1}{2} x^T P x + q^T x \\
\text{subject to: } & Gx \leq h \\
& Ax = b
\end{align}

SVM problem to compare:

\begin{equation}
\begin{split}
  \min_{w,b,\xi}  &\frac{1}{2}w^Tw  + C\sum_{i=1}^m \xi_i  \\
  \text{s.t. } & y^{(i)}(w^T x^{(i)} + b) \geq 1- \xi_i\;\; \forall_i \\
  & \xi_i \geq 0 \;\; \forall_i.
\end{split}
\end{equation}

To solve the SVM problem you need to encode the weights $W$, biases $b$, and slack variables $\xi$ as elements of the vector $x$, then properly fill the matrices and vectors $P$, $q$, $G$, $h$. We can ignore setting the $A$ and $b$ parametrs, since there are no linear constraints.

```
#
# Now solve the SVM using the QP solver
#

y = IrisY.T.ravel() * 2 - 1
n, m = IrisX2F.shape
C= 10.0

P = np.zeros((n+1+m, n+1+m)) #w, bias, xi
q = np.zeros((n+1+m,1))
G = np.zeros((2*m, n+1+m)) # we have two constrains for each data point:
                           # that the margin is equal to 1-xi
                           # and that xi is nonnegative
h = np.zeros((2*m,1))
#
# TODO: fill in P, q, G, h
#

for i in range(m):
    h[i] = -1
    q[n+1+i] = C
    G[i][n] = -y[i]
    G[i][n+1+i] = -1
    G[m+i][n+1+i] = -1
    for j in range(n):
        G[i][j] = -y[i] * IrisX2F[j][i]

for i in range(n):
    P[i][i] = 1

#
# Now run the solver
#
ret  = cvxopt.solvers.qp(cvxopt.matrix(P), cvxopt.matrix(q),
                         cvxopt.matrix(G), cvxopt.matrix(h))

ret = np.array(ret['x'])
support_vextors = IrisX2F
#
# extract the weights and biases
#
W = ret[:n].reshape(-1,1)
b = ret[n]

#
# Extract the weight and bias from libsvm for comparison
#

svm_model = sklearn.svm.SVC(gamma='auto', kernel='linear', C=C)
svm_model.fit(IrisX2F.T, y)

Wlibsvm = svm_model.coef_
blibsvm = svm_model.intercept_

print('W', W.T, 'Wlibsvm', Wlibsvm)
print('b', b, 'blibsvm', blibsvm)
```

         pcost       dcost       gap    pres   dres
     0: -8.2045e+03  4.6654e+03  2e+04  2e+01  3e+00
     1:  5.3426e+01 -5.8951e+02  3e+03  2e+00  4e-01
     2:  1.7058e+02 -1.9137e+01  4e+02  3e-01  5e-02
     3:  1.4063e+02  7.5166e+01  1e+02  7e-02  1e-02
     4:  1.3820e+02  9.8722e+01  6e+01  3e-02  5e-03
     5:  1.4016e+02  1.1207e+02  3e+01  4e-03  8e-04
     6:  1.3507e+02  1.1797e+02  2e+01  1e-03  2e-04
     7:  1.2551e+02  1.2356e+02  2e+00  2e-04  3e-05
     8:  1.2445e+02  1.2440e+02  5e-02  4e-06  6e-07
     9:  1.2442e+02  1.2442e+02  1e-03  9e-08  2e-08
    10:  1.2442e+02  1.2442e+02  2e-05  1e-09  1e-10
    Optimal solution found.
    W [[2.75844069 4.827271  ]] Wlibsvm [[2.75873306 4.82861713]]
    b [-21.2054472] blibsvm [-21.20914737]

# Problem 3 [2p]

Repeat 100 bootstrap experiments to establish the effect of constant $C$ on SVM.

For each experiment do the following:

1. Sample (with replacement) a bootstrap dataset equal in size to the
    training dataset. This will be this experiment's training dataset.
2. Prepare the experiment's testing dataset by using samples not
    inluded in the bootstrap dataset.
3. For all $C$ from the set $\{10^{-4}, 10^{-3.5}, 10^{-3.}, \ldots, 10^{6}\}$
    fit a nonlinear SVM (Gaussian kernel, called \texttt{rbf} in
    LIBSVM using the default $\gamma$) and record the training and
    testing errors.

Analyze a box plot of errors as a function of $C$. Can you see its
influence on the training and testing error, as well as on the
testing error variability?

**Indicate regions of overfitting and underfitting.**

```
res = []
iters = 100
for rep in range(iters):
    bootstrap_sel = resample(np.arange(len(IrisX.T)), n_samples = len(IrisX.T))
    test_sel = np.array([i for i in np.arange(len(IrisX.T)) if not i in bootstrap_sel])

    bootstrap_IrisX = IrisX2F[:,bootstrap_sel].T
    bootstrap_IrisY = IrisY[:,bootstrap_sel].T

    test_IrisX = IrisX2F[:,test_sel].T
    test_IrisY = IrisY[:,test_sel].T

    #
    # TODO: Loop over a list of exponents.
    #
    for Cexponent in np.arange(-4, 6.5, 0.5):
        C = 10.0**Cexponent
        svm_model = sklearn.svm.SVC(gamma='auto', kernel='rbf', C=C)
        svm_model.fit(bootstrap_IrisX, bootstrap_IrisY.ravel())
        train_acc = (svm_model.predict(bootstrap_IrisX) == bootstrap_IrisY.ravel()).mean()
        test_acc = (svm_model.predict(test_IrisX) == test_IrisY.ravel()).mean()

        # VISUALIZATION
        # predictions_Grid = svm_model.predict(IrisXGrid.T)
        # plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')
        # plt.scatter(bootstrap_IrisX[:, 0], bootstrap_IrisX[:, 1],
        #     c=svm_model.predict(bootstrap_IrisX),
        #     cmap='spring',
        #     edgecolors='black')
        # plt.show()
        # print(train_acc, test_acc)

        res.append(dict(Cexponent=Cexponent, err=1-test_acc, subset='test'))
        res.append(dict(Cexponent=Cexponent, err=1-train_acc, subset='train'))

res = pd.DataFrame(res)
chart = sns.catplot(kind='box', x='Cexponent', y='err', col='subset',
            color='blue', data=res)
chart.set_xticklabels(rotation=45)
```

# Problem 4 [3p bonus]

Implement a nonlinear SVM by solving the dual problem using the Quadratic Programming solver. Compare results with LIBSVM.

Please see [page 20 if CS229 lecture notes](http://cs229.stanford.edu/notes/cs229-notes3.pdf) for problem formulation.

Soft SVM, dual problem:
\begin{aligned}
    \min_{\alpha}  \frac{1}{2}  \alpha^T \mathbf{H}  \alpha - 1^T \alpha
    \\
     s.t.  - \alpha_i \leq 0
    \\
      \alpha_i \leq C
     \\
     \ y^T \alpha = 0
     \\
     \ H = y^TKy
     \\
     \ K_{i,j} = K(x_i,x_j)
\end{aligned}

Solver form:
\begin{align}
\text{minimize over } x \text{: }& \frac{1}{2} x^T P x + q^T x \\
\text{subject to: } & Gx \leq h \\
& Ax = b
\end{align}

```
def get_bias(X,y,alphas,S,gamma=1):
    support_vextors = X[S]
    sv_target = y[S]
    B = []
    for i,sv in enumerate(support_vextors):
        b = sv_target[i] - np.sum(rbf_kernel(support_vextors,sv.reshape(1,-1),gamma=gamma).ravel() * sv_target.ravel() * alphas[S].ravel())
        # b == np.array of length 1
        B.append(b[0])
    return sstats.mode(np.array(B))[0][0]

def predict(X,y,sample,alphas,b,S,gamma=1):
    SX = X[S]
    Sy = y[S]
    v = np.sum(rbf_kernel(SX,sample.reshape(1,-1),gamma=gamma).ravel() * Sy.ravel() * alphas[S].ravel()) + b
    return np.sign(v)

def predict_all(Y,gamma=90):
  preds = []
  for sample in Y:
      preds.append(predict(X,y,sample,alphas,b,S,gamma=gamma))
  return np.array(preds)
```

```
C = 1
gamma = 90
th = 1e-5

X = IrisX2F.T
y = (IrisY * 2 - 1).reshape(-1,1)
m,n = X.shape
K = rbf_kernel(X,X,gamma=gamma) # rbf kernel from sklearn

P = cvxopt.matrix(np.outer(y,y) * K)
q = cvxopt.matrix(-np.ones((m, 1)))
G = cvxopt.matrix(np.vstack((np.eye(m)*-1,np.eye(m))))
h = cvxopt.matrix(np.hstack((np.zeros(m), np.ones(m) * C)))
A = cvxopt.matrix(y.reshape(1, -1))
b = cvxopt.matrix(np.zeros(1))

#Run solver
sol = cvxopt.solvers.qp(P, q, G, h, A, b)
alphas = np.array(sol['x'])
```

         pcost       dcost       gap    pres   dres
     0: -2.7886e+01 -2.0457e+02  7e+02  2e+00  5e-16
     1: -2.6152e+01 -1.2082e+02  9e+01  8e-15  4e-16
     2: -2.8097e+01 -3.9348e+01  1e+01  8e-15  2e-16
     3: -2.9282e+01 -3.0253e+01  1e+00  3e-15  2e-16
     4: -2.9435e+01 -2.9508e+01  7e-02  4e-15  2e-16
     5: -2.9451e+01 -2.9454e+01  3e-03  2e-15  1e-16
     6: -2.9452e+01 -2.9452e+01  2e-04  7e-15  2e-16
     7: -2.9452e+01 -2.9452e+01  1e-05  3e-15  1e-16
    Optimal solution found.

```
# take alphas > 0
S = (alphas > th).reshape(-1, )
```

```
b = get_bias(X,y,alphas,S,gamma)
b
```

    0.2171403129519649

```
preds = []
for i in range(100):
    s = X[i]
    preds.append(predict(X,y,s,alphas,b,S,gamma=gamma))
preds = np.array(preds)
```

```
print(f'Accuracy: {np.mean(preds == y.ravel()) * 100}%')
```

    Accuracy: 99.0%

```
petal_lengths, petal_widths = np.meshgrid(np.linspace(IrisX2F[0,:].min(), IrisX2F[0,:].max(), 100),
                                          np.linspace(IrisX2F[1,:].min(), IrisX2F[1,:].max(), 100))

IrisXGrid = np.vstack([petal_lengths.ravel(), petal_widths.ravel()])
predictions_Grid = predict_all(IrisXGrid.T,gamma=gamma)

plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title(f'Decision boundary found by SVM, kernel = rbf, gamma = {gamma}, c = {C}')
```

# Problem 5 [2p bonus]
Compare two ways to implement a multi-class
SVM: by training ``1-vs-1`` classifier for each class combination,
and by training a ``1-vs-rest`` classifier for each clas. See
http://www.csie.ntu.edu.tw/\~cjlin/papers/multisvm.pdf for
details.

```
# load iris, extract septal_width and petal_width of all classes

iris = sklearn.datasets.load_iris()
print('Features: ', iris.feature_names)
print('Targets: ', iris.target_names)
petal_length = iris.data[:,iris.feature_names.index('sepal width (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]

IrisX = np.array(iris.data.T)

IrisX2F = np.vstack([petal_length, petal_width])

IrisY = iris.target.reshape(1,-1).astype(np.float64)

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')
plt.xlabel('sepal_width')
plt.ylabel('petal_width')
```

```
def get_models(X,y,C=10,gamma=1):
    classes = np.unique(y)
    models = []
    for c in classes:
        Y = ((y == c) * 2 - 1).ravel()
        svm_model = sklearn.svm.SVC(kernel='rbf',C=C,gamma=gamma)
        svm_model.fit(X.T,Y)
        models.append(svm_model)
    return models

def predict_one_vs_rest(models,X,classes):
    l = []
    for model in models:
        l.append(list(model.decision_function(X.T)))
    l = np.array(l)
    preds = np.argmax(l, axis=0)
    return classes[preds]
```

```
C=1
gamma=10

models = get_models(IrisX2F,IrisY,C,gamma)
classes = np.unique(IrisY)
```

## One vs rest
Also known as one-vs-all, this strategy consists in fitting one classifier per class. For each classifier, the class is fitted against all the other classes.

For m classes, we will have m probability estimates of positive class for one test sample. Compare them and whichever probability is maximum of m probabilities, it belongs to that particular class.
I.e. to determine the final class we take argmax of all predictions.

 This is the most commonly used strategy for multiclass classification and is a fair default choice.

```
sepal_width, petal_widths = np.meshgrid(np.linspace(IrisX2F[0,:].min(), IrisX2F[0,:].max(), 100),
                                          np.linspace(IrisX2F[1,:].min(), IrisX2F[1,:].max(), 100))

IrisXGrid = np.vstack([sepal_width.ravel(), petal_widths.ravel()])
predictions_Grid = predict_one_vs_rest(models, IrisXGrid, classes)

plt.contourf(sepal_width, petal_widths, predictions_Grid.reshape(sepal_width.shape), cmap='spring')
plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')

plt.xlabel('sepal_width')
plt.ylabel('petal_width')
plt.title(f'One vs rest, kernel = rbf, gamma = {gamma}, c = {C}')
```

## One vs one:

Two pairs of classes are selected at a time and a binary classifier trained for them. This is done for every possible pair of classes thus there are n(n-1)/2 of them where n is the total number of classes. During the classification phases, all the binary classifiers are tested. For each of them, a “win” for one class is a vote for that class. The class with the most votes wins.

```
def get_models_ovo(X,y,C=10,gamma=1):
    classes = np.unique(y)
    models = []
    for i in range(len(classes)):
        for j in range(i+1,len(classes)):
            c1,c2 = classes[i],classes[j]
            idx = np.logical_or(IrisY == c1, IrisY == c2).ravel()
            X_ = X.T[idx]
            Y_ = t = (IrisY == c1) * 1 + (IrisY == c2) * -1
            Y_ = Y_[Y_!=0]
            svm_model = sklearn.svm.SVC(kernel='rbf',C=C,gamma=gamma)
            svm_model.fit(X_,Y_)
            models.append((svm_model,c1,c2))
    return models

def predict_one_vs_one(models,X):
  l = []
  for model,c1,c2 in models:
      preds = model.decision_function(X.T)
      preds = [c1 if np.sign(p) == 1.0 else c2 for p in preds]
      l.append(preds)
  l = np.array(l)
  preds = sstats.mode(l,axis=0)
  return preds[0]
```

```
models = get_models_ovo(IrisX2F,IrisY,C,gamma)
```

```
sepal_width, petal_widths = np.meshgrid(np.linspace(IrisX2F[0,:].min(), IrisX2F[0,:].max(), 100),
                                          np.linspace(IrisX2F[1,:].min(), IrisX2F[1,:].max(), 100))

IrisXGrid = np.vstack([sepal_width.ravel(), petal_widths.ravel()])

predictions_Grid2 = predict_one_vs_one(models,IrisXGrid)

plt.contourf(sepal_width, petal_widths, predictions_Grid2.reshape(sepal_width.shape), cmap='spring')

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')

plt.xlabel('sepal_width')
plt.ylabel('petal_width')
plt.title(f'One vs one, kernel = rbf, gamma = {gamma}, c = {C}')
```

## Comparison between One vs All and One vs One

```
fig, ax = plt.subplots(1, 2, figsize=(18,6))
ax[0].contourf(sepal_width, petal_widths, predictions_Grid.reshape(sepal_width.shape), cmap='spring')
ax[0].scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),cmap='spring', edgecolors='k')
ax[0].set_title('One vs all')
ax[1].contourf(sepal_width, petal_widths, predictions_Grid2.reshape(sepal_width.shape), cmap='spring')
ax[1].scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),cmap='spring', edgecolors='k')
ax[1].set_title('One vs one')
plt.xlabel('sepal_width')
plt.ylabel('petal_width')
```

# Problem 6 [4p bonus]

Implement a Kernelized linear regression. Train it on Iris using a Gaussian kernel. Compare to the non-linear SVM.

We know that in linear regression optimal solution can be described as $\theta_0 = (X^T X + \lambda I)^{-1} X^T Y $.

We know that this problem has dual formulation, and we can say that: $ \theta_0 = X^T (X X^T + \lambda I)^{-1} Y $

Now we can define that $ \alpha = (K + \lambda I)^{-1} Y$, where $K_{i,j} = <x^{(i)},x^{(j)}>$, and when we have to predict new point $x$ we know that $
f(x) = \sum_{i=0}^{n} \alpha_i <x,x^{(i)}> $.

Now insted of using dot prodact we can use kernel trick.

So given new sample $x$ we can say that:

$f(x) = \sum_{i=0}^{n} \alpha_i K(x,x^{(i)})
\\ \alpha = (K + \lambda I)^{-1} Y
\\ K_{i,j} = K(x^{(i)},x^{(j)}) $

```
def qr_inv(a):
    q,r = np.linalg.qr(a, mode='complete')
    N = r.shape[0]
    inv_ = np.dot(solve_triangular(r,np.eye(N)),q.T)
    return inv_

def classify_LS_SVM(X,sample,alphas,gamma=1):
    return np.sign(np.sum(rbf_kernel(X,sample.reshape(1,-1),gamma=gamma).ravel() * alphas))

def predict_LS_SVM(X,Y,alphas,gamma=1):
    preds = []
    for sample in Y:
        preds.append(classify_LS_SVM(X,sample,alphas,gamma=gamma))
    return np.array(preds)
```

```
gamma = 90
lambda_ = 1.0

petal_length = iris.data[:,iris.feature_names.index('petal length (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]

IrisX = np.array(iris.data.T)
IrisX = IrisX[:, iris.target!=0]

IrisX2F = np.vstack([petal_length, petal_width])
IrisX2F = IrisX2F[:, iris.target!=0]

# Set versicolor=0 and virginia=1
IrisY = (iris.target[iris.target!=0]-1).reshape(1,-1).astype(np.float64)
m
X = IrisX2F.T
y = (IrisY * 2 - 1).reshape(-1,1)
Y = y.ravel()

n,m = X.shape
K = rbf_kernel(X,X,gamma=gamma)
alphas_ls = qr_inv(K + lambda_ * np.eye(n)).dot(Y)
```

```
np.mean(predict_LS_SVM(X,IrisX2F.T,alphas_ls) == Y)
```

    0.96

```
petal_lengths, petal_widths = np.meshgrid(np.linspace(IrisX2F[0,:].min(), IrisX2F[0,:].max(), 100),
                                          np.linspace(IrisX2F[1,:].min(), IrisX2F[1,:].max(), 100))

IrisXGrid = np.vstack([petal_lengths.ravel(), petal_widths.ravel()])
predictions_Grid_LS_SVM = predict_LS_SVM(X,IrisXGrid.T,alphas_ls,gamma=gamma)
predictions_Grid_NL_SVM = predict_all(IrisXGrid.T,gamma=90)

fig, ax = plt.subplots(1, 2, figsize=(18,6))

ax[0].contourf(petal_lengths, petal_widths, predictions_Grid_LS_SVM.reshape(petal_lengths.shape), cmap='spring')
ax[0].scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),cmap='spring', edgecolors='k')
ax[0].set_title(f'Decision boundary found by LS-SVM, gamma = {gamma}, lambda = {lambda_}')

ax[1].contourf(petal_lengths, petal_widths, predictions_Grid_NL_SVM.reshape(petal_lengths.shape), cmap='spring')
ax[1].scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),cmap='spring', edgecolors='k')
ax[1].set_title(f'Decision boundary found by SVM, kernel = rbf, gamma = {gamma}, c = {C}')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
```

```

```


     Histograms of Oriented Gradients (HOG)

Introduction

As we saw with the ORB algorithm, we can use keypoints in images to do keypoint-based matching to detect objects in images. These type of algorithms work great when you want to detect objects that have a lot of consistent internal features that are not affected by the background. For example, these algorithms work well for facial detection because faces have a lot of consistent internal features that don’t get affected by the image background, such as the eyes, nose, and mouth. However, these type of algorithms don’t work so well when attempting to do more general object recognition, say for example, pedestrian detection in images. The reason is that people don’t have consistent internal features, like faces do, because the body shape and style of every person is different (see Fig. 1). This means that every person is going to have a different set of internal features, and so we need something that can more generally describe a person.

Fig. 1. - Pedestrians.

One option is to try to detect pedestrians by their contours instead.  Detecting objects in images by their contours (boundaries) is very challenging because we have to deal with the difficulties brought about by the contrast between the background and the foreground. For example, suppose you wanted to detect a pedestrian in an image that is walking in front of a white building and she is wearing a white coat and black pants (see Fig. 2). We can see in Fig. 2, that since the background of the image is mostly white, the black pants are going to have a very high contrast, but the coat, since it is white as well, is going to have very low contrast. In this case, detecting the edges of pants is going to be easy but detecting the edges of the coat is going to be very difficult. This is where **HOG** comes in. HOG stands for **Histograms of Oriented Gradients** and it was first introduced by Navneet Dalal and Bill Triggs in 2005.

Fig. 2. - High and Low Contrast.

The HOG algorithm works by creating histograms of the distribution of gradient orientations in an image and then normalizing them in a very special way. This special normalization is what makes HOG so effective at detecting the edges of objects even in cases where the contrast is very low. These normalized histograms are put together into a feature vector, known as the HOG descriptor, that can be used to train a machine learning algorithm, such as a Support Vector Machine (SVM), to detect objects in images based on their boundaries (edges). Due to its great success and reliability, HOG has become one of the most widely used algorithms in computer vison for object detection.

In this notebook, you will learn:

* How the HOG algorithm works
* How to use OpenCV to create a HOG descriptor
* How to visualize the HOG descriptor.

# The HOG Algorithm

As its name suggests, the HOG algorithm, is based on creating histograms from the orientation of image gradients. The HOG algorithm is implemented in a series of steps:

1. Given the image of particular object, set a detection window (region of interest) that covers the entire object in the image (see Fig. 3).

2. Calculate the magnitude and direction of the gradient for each individual pixel in the detection window.

3. Divide the detection window into connected *cells* of pixels, with all cells being of the same size (see Fig. 3). The size of the cells is a free parameter and it is usually chosen so as to match the scale of the features that want to be detected. For example, in a 64 x 128 pixel detection window, square cells 6 to 8 pixels wide are suitable for detecting human limbs.

4. Create a Histogram for each cell, by first grouping the gradient directions of all pixels in each cell into a particular number of orientation (angular) bins; and then adding up the gradient magnitudes of the gradients in each angular bin (see Fig. 3). The number of bins in the histogram is a free parameter and it is usually set to 9 angular bins.

5. Group adjacent cells into *blocks* (see Fig. 3). The number of cells in each block is a free parameter and all blocks must be of the same size. The distance between each block (known as the stride) is a free parameter but it is usually set to half the block size, in which case you will get overlapping blocks (*see video below*).  The HOG algorithm has been shown empirically to work better with overlapping blocks.

6. Use the cells contained within each block to normalize the cell histograms in that block (see Fig. 3). If you have overlapping blocks this means that most cells will be normalized with respect to different blocks (*see video below*). Therefore, the same cell may have several different normalizations.

7. Collect all the normalized histograms from all the blocks into a single feature vector called the HOG descriptor.

8. Use the resulting HOG descriptors from many images of the same type of object to train a machine learning algorithm, such as an SVM, to detect those type of objects in images. For example, you could use the HOG descriptors from many images of pedestrians to train an SVM to detect pedestrians in images. The training is done with both positive a negative examples of the object you want detect in the image.

9. Once the SVM has been trained, a sliding window approach is used to try to detect and locate objects in images.  Detecting an object in the image entails finding the part of the image that looks similar to the HOG pattern learned by the SVM.

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# Why The HOG Algorithm Works

As we learned above, HOG creates histograms by adding the magnitude of the gradients in particular orientations in localized portions of the image called *cells*.  By doing this we guarantee that stronger gradients will contribute more to the magnitude of their respective angular bin, while the effects of weak and randomly oriented gradients resulting from noise are minimized. In this manner the histograms tell us the dominant gradient orientation of each cell.

### Dealing with contrast

Now, the magnitude of the dominant orientation can vary widely due to variations in local illumination and the contrast between the background and the foreground.

To account for the background-foreground contrast differences, the HOG algorithm tries to detect edges locally.  In order to do this, it defines groups of cells, called **blocks**, and normalizes the histograms using this local group of cells. By normalizing locally, the HOG algorithm can detect the edges in each block very reliably; this is called **block normalization**.

In addition to using block normalization, the HOG algorithm also uses overlapping blocks to increase its performance. By using overlapping blocks, each cell contributes several independent components to the final HOG descriptor, where each component corresponds to a cell being normalized with respect to a different block. This may seem redundant but, it has been shown empirically that by normalizing each cell several times with respect to different local blocks, the performance of the HOG algorithm increases dramatically.

### Loading Images and Importing Resources

The first step in building our HOG descriptor is to load the required packages into Python and to load our image.

We start by using OpenCV to load an image of a triangle tile. Since, the `cv2.imread()` function loads images as BGR we will convert our image to RGB so we can display it with the correct colors. As usual we will convert our BGR image to Gray Scale for analysis.

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

    The original image has shape:  (250, 250, 3)
    The gray scale image has shape:  (250, 250)

    <matplotlib.figure.Figure at 0x7f330704dd30>

# Creating The HOG Descriptor

We will be using OpenCV’s `HOGDescriptor` class to create the HOG descriptor. The parameters of the HOG descriptor are setup using the `HOGDescriptor()` function. The parameters of the `HOGDescriptor()` function and their default values are given below:

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

Parameters:

* **win_size** – *Size*
Size of detection window in pixels (*width, height*). Defines the region of interest. Must be an integer multiple of cell size.

* **block_size** – *Size*
Block size in pixels (*width, height*). Defines how many cells are in each block. Must be an integer multiple of cell size and it must be smaller than the detection window. The smaller the block the finer detail you will get.

* **block_stride** – *Size*
Block stride in pixels (*horizontal, vertical*). It must be an integer multiple of cell size. The `block_stride` defines the distance between adjecent blocks, for example, 8 pixels horizontally and 8 pixels vertically. Longer `block_strides` makes the algorithm run faster (because less blocks are evaluated) but the algorithm may not perform as well.

* **cell_size** – *Size*
Cell size in pixels (*width, height*). Determines the size fo your cell. The smaller the cell the finer detail you will get.

* **nbins** – *int*
Number of bins for the histograms. Determines the number of angular bins used to make the histograms. With more bins you capture more gradient directions. HOG uses unsigned gradients, so the angular bins will have values between 0 and 180 degrees.

* **win_sigma** – *double*
Gaussian smoothing window parameter. The performance of the HOG algorithm can be improved by smoothing the pixels near the edges of the blocks by applying a Gaussian spatial window to each pixel before computing the histograms.

* **threshold_L2hys** – *double*
L2-Hys (Lowe-style clipped L2 norm) normalization method shrinkage. The L2-Hys method is used to normalize the blocks and it consists of an L2-norm followed by clipping and a renormalization. The clipping limits the maximum value of the descriptor vector for each block to have the value of the given threshold (0.2 by default). After the clipping the descriptor vector is renormalized as described in *IJCV*, 60(2):91-110, 2004.

* **gamma_correction** – *bool*
Flag to specify whether the gamma correction preprocessing is required or not. Performing gamma correction slightly increases the performance of the HOG algorithm.

* **nlevels** – *int*
Maximum number of detection window increases.

As we can see, the  `cv2.HOGDescriptor()`function supports a wide range of parameters. The first few arguments (`block_size, block_stride, cell_size`, and `nbins`) are probably the ones you are most likely to change. The other parameters can be safely left at their default values and you will get good results.

In the code below, we will use the `cv2.HOGDescriptor()`function to set the cell size, block size, block stride, and the number of bins for the histograms of the HOG descriptor. We will then use `.compute(image)`method to compute the HOG descriptor (feature vector) for the given `image`.

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

    The gray scale image has shape:  (250, 250)

    HOG Descriptor Parameters:

    Window Size: (246, 246)
    Cell Size: (6, 6)
    Block Size: (12, 12)
    Block Stride: (6, 6)
    Number of Bins: 9

# Number of Elements In The HOG Descriptor

The resulting HOG Descriptor (feature vector), contains the normalized histograms from all cells from all blocks in the detection window concatenated in one long vector. Therefore, the size of the HOG feature vector will be given by the total number of blocks in the detection window, multiplied by the number of cells per block, times the number of orientation bins:

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

If we don’t have overlapping blocks (*i.e.* the `block_stride`equals the `block_size`), the total number of blocks can be easily calculated by dividing the size of the detection window by the block size. However, in the general case we have to take into account the fact that we have overlapping blocks. To find the total number of blocks in the general case (*i.e.* for any `block_stride` and `block_size`), we can use the formula given below:

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

Where Total$_x$, is the total number of blocks along the width of the detection window, and Total$_y$, is the total number of blocks along the height of the detection window. This formula for Total$_x$ and Total$_y$, takes into account the extra blocks that result from overlapping. After calculating Total$_x$ and Total$_y$, we can get the total number of blocks in the detection window by multiplying Total$_x$  $\times$ Total$_y$. The above formula can be simplified considerably because the `block_size`, `block_stride`, and `window_size`are all defined in terms of the `cell_size`. By making all the appropriate substitutions and cancelations the above formula reduces to:

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

Where cells$_x$ is the total number of cells along the width of the detection window, and cells$_y$, is the total number of cells along the height of the detection window. And $N_x$ is the horizontal block stride in units of `cell_size` and $N_y$ is the vertical block stride in units of `cell_size`.

Let's calculate what the number of elements for the HOG feature vector should be and check that it matches the shape of the HOG Descriptor calculated above.

```python
# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

    The total number of elements in the HOG Feature Vector should be:  40 x 40 x 2 x 2 x 9 = 57600

    The HOG Descriptor has shape: (57600, 1)

# Visualizing The HOG Descriptor

We can visualize the HOG Descriptor by plotting the histogram associated with each cell as a collection of vectors. To do this, we will plot each bin in the histogram as a single vector whose magnitude is given by the height of the bin and its orientation is given by the angular bin that its associated with. Since any given cell might have multiple histograms associated with it, due to the overlapping blocks, we will choose to average all the histograms for each cell to produce a single histogram for each cell.

OpenCV has no easy way to visualize the HOG Descriptor, so we have to do some manipulation first in order to visualize it. We will start by reshaping the HOG Descriptor in order to make our calculations easier.  We will then compute the average histogram of each cell and finally we will convert the histogram bins into vectors. Once we have the vectors, we plot the corresponding vectors for each cell in an image.

The code below produces an interactive plot so that you can interact with the figure. The figure contains:
* the grayscale image,
* the HOG Descriptor (feature vector),
* a zoomed-in portion of the HOG Descriptor, and
* the histogram of the selected cell.

**You can click anywhere on the gray scale image or the HOG Descriptor image to select a particular cell**. Once you click on either image a *magenta* rectangle will appear showing the cell you selected. The Zoom Window will show you a zoomed in version of the HOG descriptor around the selected cell; and the histogram plot will show you the corresponding histogram for the selected cell. The interactive window also has buttons at the bottom that allow for other functionality, such as panning, and giving you the option to save the figure if desired. The home button returns the figure to its default value.

**NOTE**: If you are running this notebook in the Udacity workspace, there is around a 2 second lag in the interactive plot. This means that if you click in the image to zoom in, it will take about 2 seconds for the plot to refresh.

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

    The feature vector has shape: (57600, 1)
    The reshaped feature vector has shape: (40, 40, 2, 2, 9)
    The average gradient array has shape:  (41, 41, 9)

    <IPython.core.display.Javascript object>

# Understanding The Histograms

Let's take a look at a couple of snapshots of the above figure to see if the histograms for the selected cell make sense. Let's start looking at a cell that is inside a triangle and not near an edge:

Fig. 4. - Histograms Inside a Triangle.

In this case, since the triangle is nearly all of the same color there shouldn't be any dominant gradient in the selected cell. As we can clearly see in the Zoom Window and the histogram, this is indeed the case. We have many gradients but none of them clearly dominates over the other.

Now let’s take a look at a cell that is near a horizontal edge:

Fig. 5. - Histograms Near a Horizontal Edge.

Remember that edges are areas of an image where the intensity changes abruptly. In these cases, we will have a high intensity gradient in some particular direction. This is exactly what we see in the corresponding histogram and Zoom Window for the selected cell.  In the Zoom Window, we can see that the dominant gradient is pointing up, almost at 90 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 90-degree bin in the histogram to dominate strongly over the others. This is in fact what we see.

Now let’s take a look at a cell that is near a vertical edge:

Fig. 6. - Histograms Near a Vertical Edge.

In this case we expect the dominant gradient in the cell to be horizontal, close to 180 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 170-degree bin in the histogram to dominate strongly over the others. This is what we see in the histogram but we also see that there is another dominant gradient in the cell, namely the one in the 10-degree bin. The reason for this, is because the HOG algorithm is using unsigned gradients, which means 0 degrees and 180 degrees are considered the same. Therefore, when the histograms are being created, angles between 160 and 180 degrees, contribute proportionally to both the 10-degree bin and the 170-degree bin. This results in there being two dominant gradients in the cell near the vertical edge instead of just one.

To conclude let’s take a look at a cell that is near a diagonal edge.

Fig. 7. - Histograms Near a Diagonal Edge.

To understand what we are seeing, let’s first remember that gradients have an *x*-component, and a *y*-component, just like vectors. Therefore, the resulting orientation of a gradient is going to be given by the vector sum of its components. For this reason, on vertical edges the gradients are horizontal, because they only have an x-component, as we saw in Figure 4. While on horizontal edges the gradients are vertical, because they only have a y-component, as we saw in Figure 3. Consequently, on diagonal edges, the gradients are also going to be diagonal because both the *x* and *y* components are non-zero.  Since the diagonal edges in the image are close to 45 degrees, we should expect to see a dominant gradient orientation in the 50-degree bin. This is in fact what we see in the histogram but, just like in Figure 4., we see there are two dominant gradients instead of just one. The reason for this is that when the histograms are being created, angles that are near the boundaries of bins, contribute proportionally to the adjacent bins.  For example, a gradient with an angle of 40 degrees, is right in the middle of the 30-degree and 50-degree bin. Therefore, the magnitude of the gradient is split evenly into the 30-degree and 50-degree bin. This results in there being two dominant gradients in the cell near the diagonal edge instead of just one.

Now that you know how HOG is implemented, in the workspace you will find a notebook named *Examples*. In there, you will be able set your own paramters for the HOG descriptor for various images. Have fun!

```python

```