## The mean shift algorithm

"Mean shift is a **non-parametric** feature-space analysis technique for locating the maxima of a density function, a so-called **mode-seeking algorithm**. Application domains include cluster analysis in computer vision and image processing." -- https://en.wikipedia.org/wiki/Mean_shift

Think of mean-shift clustering as k-means but you don't have to specify the number of clusters. (You have to specify the **bandwidth** but that can be automated.) *Core algorithmic task*: Efficiently find the **peaks** or **modes** of a distribution given a set of samples, unlike the means of a distribution as done by k-means or EM.

**Reference material**:
1. Section 5.3.2 of text book
2. D. Comaniciu and P. Meer, "[Mean shift: a robust approach toward feature space analysis](https://ieeexplore.ieee.org/document/1000236)," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002, doi: 10.1109/34.1000236.
3. https://en.wikipedia.org/wiki/Mean_shift

Let $\mathbf{x}_i$ represent the vector representing a pixel in terms of color or other attributes. We have seen how the probability distribution of this pixel feature vector can be represented using a Gaussian mixture model (GMM) that is specified by the paramters of the Gaussian components and the mixture probabilities. Here we will use a non-parameteric probability density function for the pixel values, called kernel distribution (or sometimes [Parzen](https://en.wikipedia.org/wiki/Emanuel_Parzen) distribution). The density function does not have any parameters (This is not quite right. We do need one parameter, as we shall see. However, this single parameter need is way less than needed for GMM, so close enough to zero.)

**Kernel distribution** is an estimate of a *continuous* probability density function from samples. It is achieved by replacing a continuous function, which we will call the kernel function $G(\mathbf{x})$ at **each** sample and adding them up.

It effectively gives us a continuous interpolation of the data between the samples.

The most used kernel function is a Gaussian function.
    \begin{equation}
        f(\mathbf{x}) = c \sum_{i=1}^{N} G (\mathbf{x} - \mathbf{x}_i) = c \sum_{i=1}^N \exp \left( -\frac{||\mathbf{x} - \mathbf{x}_i||^2}{2 h^2} \right )
    \end{equation}

The constant $c$ is a normalizing constant to make the function integrate out to one. For a vector valued sample, this is basically a sum of Gaussian balls centered on the samples.

```python
# Generate n_sample number of random 1D samples.
import numpy as np
import matplotlib.pyplot as plt

n_clusters = 2
n_samples = 10
m = np.random.uniform(-35, 35, (n_clusters, 1))

slices = [np.random.normal(m[i], 10, n_samples)
          for i in range(n_clusters)]
# slices is an array of array
data = np.concatenate(slices).astype(np.float32)

plt.bar(data, np.ones(data.shape),width=0.1)
plt.plot(data, np.zeros(data.shape), 'r*');
```

```python
# place a Gaussian function at each sample and add

def G(y, s) :
  z = (1/(np.sqrt(2*np.pi)*s))* np.exp(-(y**2)/(2*s*s))
  return(z)

h = 5
x = np.linspace(-3*h+np.min(data), 3*h+np.max(data),num=300)
p = np.zeros(x.shape)
for i in range(data.shape[0]) :
  G_i = G((x - data[i]), h)
  plt.plot(x, G_i, 'g')
  p = p+G_i

# plot the kernel density function
plt.plot(x, p, 'b')

# plot the initial data sample
plt.plot(data, np.zeros(data.shape), 'rx');

```

We have to find the peaks of this kernel density function **without** "computing" the kernel density function for all points as we did above.

**Multiple restart gradient ascent**: Our approach to finding the peaks in this continuous function would be to start from a random point and then to move it iteratively towards a peak. Ideally, we would like to start from many different points so that we are able to that converge around all the available peaks. How many samples should we start from? That is an issue that is needs to be looked at during implementation time. This is a kin to hill climbing but from many different points.

Another way to visualize this process is to imagine throwing a random number of particles on this continuous function and then moving these particles up the hill wherever they are placed. The idea is that overtime the particles will accumulate on the hilltops nearest to them. Let us first see how we can quantify the step we need to take at each iteration from each particle or starting point.

For each particle (or estimate of a possible peak, $\mathbf{y}_k$, we modify it by an uphill step, which is estimated by the gradient of the function at $\mathbf{y}_k$.

So, first we need to an expression of the gradient of the continuous function $f(\mathbf{x})$. Note that the function is a scalar value define over a vector field.

**The 1D case:** The expression of a 1D kernel density function is given by:
\begin{equation}
        f(x) = c \sum_{i=1}^N \exp \left( -\frac{(x-x_i)^2}{2 h^2} \right )
\end{equation}

The first derivative of this kernel function is given by
\begin{equation}
        f^{'}(x) = c \frac{d f(x)}{dx} = c\sum_{i=1}^N - \frac{x-x_i}{h^2} \exp \left( \frac{(x-x_i)^2}{2 h^2} \right )
\end{equation}

Note that this is equivalent to placing a first derivative of a Gaussian function at each sample point.

```python
# place a derivative of a Gaussian function at each sample and add

def G_prime(y, s) :
  z = -(1/(np.sqrt(2*np.pi)*s*s))* (y) * np.exp(-(y**2)/(2*s*s))
  return(z)

h = 5
x = np.linspace(-3*h+np.min(data), 3*h+np.max(data),num=300)
p = np.zeros(x.shape)
p_prime = np.zeros(x.shape)
for i in range(data.shape[0]) :
  G_i = G((x - data[i]), h)
  G_prime_i = G_prime((x - data[i]), h)
  p_prime = p_prime+G_prime_i
  p = p + G_i

# plot the kernel density function
plt.plot(x, p, 'b')

# plot the kernel density function
plt.plot(x, p_prime, 'g')

# plot the initial data sample
plt.plot(data, np.zeros(data.shape), 'rx')

```

```python
# 1 D illustration of the mean shift

# plot the kernel density function from the previous cell
plt.plot(x, p, 'b')

y_k = np.linspace(-3*h+np.min(data), 3*h+np.max(data),num=4)

# plot values of the initial estimate of peak locations
plt.plot(y_k, np.zeros(y_k.shape), 'gx')
plt.bar(y_k, 0.5*np.ones(y_k.shape), width=0.2)

for iteration in range (5) :
    numerator = np.zeros(y_k.shape)
    denominator = np.zeros(y_k.shape)
    for i in range(data.shape[0]) :
      G_i = G((y_k - data[i]), h)
      xG_i = data[i]*G_i
      numerator = numerator +  xG_i
      denominator = denominator + G_i
    y_k = numerator/denominator

    plt.plot(y_k, np.zeros(y_k.shape), 'ro')
    plt.bar(y_k, 0.5*np.ones(y_k.shape), width=0.3, color='r')

# plot the initial data sample
# plt.plot(data, np.zeros(data.shape), 'rx')

```

## Lecture 3 - HW or In Class Assignment # 1

There are various way to initialize the potential location of the peaks. In the code **above** random points are picked within a range. Change the code to randomly pick some points from the data itself. That is, some random points of from the data itself is taken as the potential peak locations. Show the code some output plots for different random choices.

# n dimensional mean shift derivation

\begin{eqnarray}
        \nabla f(\mathbf{x}) & =  & c \sum_{i=1}^N - \frac{\mathbf{x}-\mathbf{x}_i}{h^2} \exp \left( \frac{||\mathbf{x}-\mathbf{x}_i||^2}{2 h^2} \right ) \\
        & = & c \sum_{i=1}^N - \frac{\mathbf{x}-\mathbf{x}_i}{h^2} G (\mathbf{x}-\mathbf{x}_i)
\end{eqnarray}

\begin{eqnarray}
        \nabla f(\mathbf{x}) & =  & \frac{c}{h^2}\sum_{i=1}^N - (\mathbf{x}-\mathbf{x}_i)G (\mathbf{x}-\mathbf{x}_i)\\
        & = & \frac{c}{h^2}\sum_{i=1}^N
        - \mathbf{x}   G (\mathbf{x}-\mathbf{x}_i)
        + \mathbf{x}_i G (\mathbf{x}-\mathbf{x}_i) \\
        & = & \frac{c}{h^2} \left ( - \mathbf{x} \sum_{i=1}^N
           G (\mathbf{x}- \mathbf{x}_i)
        + \sum_{i=1}^N \mathbf{x}_i G (\mathbf{x}-\mathbf{x}_i) \right) \\
        & = & \frac{c}{h^2} \left(\sum_{i=1}^N G (\mathbf{x}-\mathbf{x}_i) \right )
        \left ( - \mathbf{x}  + \frac{ \sum_{i=1}^N  \mathbf{x}_i G (\mathbf{x}-\mathbf{x}_i)}{ \sum_{i=1}^N G (\mathbf{x}-\mathbf{x}_i) } \right ) \\
        & = & \left [ \frac{c}{h^2} \sum_{i=1}^N  G (\mathbf{x}-\mathbf{x}_i) \right ]
        \left ( \frac{\sum_{i=1}^N  \mathbf{x}_i G (\mathbf{x}-\mathbf{x}_i)}{
        \sum_{i=1}^N G (\mathbf{x}-\mathbf{x}_i) } - \mathbf{x} \right )
\end{eqnarray}

Note that in the last expression above, the term in the square bracket is a scalar quantity, while the term in parenthesis is a vector. The direction of the vector is the direction of the gradient. We will denote this vector by $\mathbf{m}(\mathbf{x})$.

\begin{equation}
        \mathbf{m}(\mathbf{x}) = \left ( \frac{\sum_{i=1}^N  \mathbf{x}_i G (\mathbf{x}-\mathbf{x}_i)}{
        \sum_{i=1}^N G (\mathbf{x}-\mathbf{x}_i) } - \mathbf{x} \right )
\end{equation}

Now that we have an expression for the gradient vector, we can use it to modify the current estimate of the peak denoted by $\mathbf{y}_k$.

\begin{equation}
        \mathbf{y}_{k+1} = \mathbf{y}_k + \mathbf{m}(\mathbf{y}_k) =  \frac{\sum_{i=1}^N  \mathbf{x}_i G (\mathbf{y}_k-\mathbf{x}_i)}{
        \sum_{i=1}^N G (\mathbf{y}_k-\mathbf{x}_i) }
\end{equation}

 The above question is basically saying that the new estimate of the peak location is the weighted mean of the sample, $\mathbf{x}_i$. The weights are local because the Gaussian function is locally weighted or has high values only with a small region around the center.

Note that the updated points is the weighted average of all the points. The weights of each point is inversely proportional to the distance of the point from the current estimate of the peak. This weighted average will be effected by points that are nearby. Intuitively, the above method works because by definition you have more samples around the peaks and a random starting point will be pulled towards denser set of points.

# Other possible kernel functions

Although the above is a derivation for the Gaussian kernel, Comaniciu and Meer (2002) has shown that this also works for other kernel functions as long as they are monotonically decreasing with distance from the center. Another kernel functions studied by them was the Epamechnikov kernel that is defined over the unit ball given by:

\begin{equation}
        E(\mathbf{x}-\mathbf{x}_i) = \max \left ( 0, 1 - \frac{||\mathbf{x}-\mathbf{x}_i||^2}{h^2} \right )
\end{equation}

```python
# place a Epamechnikov function at each sample and add

def Ep(y, s) :
  z = np.maximum(0, 1 - (y**2)/(h*h))
  return(z)

h = 5
x = np.linspace(-3*h+np.min(data), 3*h+np.max(data),num=300)
p = np.zeros(x.shape)
for i in range(data.shape[0]) :
  G_i = Ep((x - data[i]), h)
  plt.plot(x, G_i, 'g')
  p = p+G_i

# plot the kernel density function
plt.plot(x, p, 'b')

# plot the initial data sample
plt.plot(data, np.zeros(data.shape), 'rx')

```

# Lecture 3 - HW or In-class Assignment # 2

What is the $\mathbf{m}(\mathbf{x})$, the direction of the gradient at any point, for the Epamechnikov kernel? What is the update equation? Modify one of the code samples above to illustrate how the update would work for this kernel for an 1D examples.

# Another variation of the kernel (for images)
There are many methods that have been proposed in the literature to make this the update process more efficient and fast. The speed depends on the number of pixels, which can be very large for big images. Complexity is $\mathcal{O}(N^2)$, where $N$ is the number of pixels.

To prevent clustering of accidentally similar colored pixels from distant parts of the image, in practice, one can expand the pixel value vector to include the location $(x, y)$. We have to use different values for the radius $h$ (kernel width) for the different dimensions as they represent different things. We use $h_s$ for the spatial dimension and $h_r$ for the color dimension.

\begin{eqnarray}
         G(\mathbf{x}-\mathbf{x}_i) & = & \exp \left( -\frac{||\mathbf{x} - \mathbf{x}_i||^2}{2 h^2} \right ) \\
         & = & \exp \left( \frac{(x-x_i)^2 + (y-y_i)^2}{2 h_s^2} \right )  \\
         &   & + \exp \left( \frac{(r-r_i)^2 + (g-g_i)^2 + + (b-b_i)^2}{2 h_r^2} \right )
\end{eqnarray}

The choice of the parameters $h_s$ and $h_r$ is by trial and error. Literature survey will reveal several strategies.

# General Implementation - for n dimensional data
(Some of these cells are taken and adapted from  https://github.com/sotte/pytorch_tutorial/blob/master/notebooks/mean_shift_clustering.ipynb)

Algo:
```python
# PSEUDO CODE
while not_converged():
    for i, point in enumerate(points):
        # distance for the given point to all other points
        distances = calc_distances(point, points)

        # turn distance into weights using a gaussian
        weights = gaussian(dist, bandwidth=2.5)

        # update the weights by using the weights
        points[i] = (weights * points).sum(0) / weights.sum()

return points
```

# efficient distance computation

```python
def distance(x, X):
    # return np.linalg.norm(x - X, axis=1)
    return np.sqrt(((x - X)**2).sum(1))
```

Let's try it out. (More on how this function works shortly)

```python
a = np.array([1, 2])
b = np.array([[1, 2],
           [2, 3],
           [-1, -3]])

dist = distance(a, b)
dist
```

    array([0.        , 1.41421356, 5.38516481])

```python
def gaussian(dist, bandwidth):
    return np.exp(-0.5 * ((dist / bandwidth))**2) / (bandwidth * np.sqrt(2 * np.pi))
```

```python
gaussian(dist, 2.5)
```

    array([0.15957691, 0.13598247, 0.0156822 ])

# General n-dimensional mean shift

```python
def meanshift_step(Y_k, X, bandwidth=2.5):
  # Y_k is a M by 2 array of seeds at step k - could be all the points
  # X is the N by 2 array of all data points
 for i in range(Y_k.shape[0]): # for each seed in y_k
     norm_dist = distance(Y_k[i], X)
     #distance of each of the seeds to rest of the data
     G_kernel = gaussian(norm_dist, bandwidth)
     # update seed
     Y_k[i] = (G_kernel [:, None] * X).sum(0) / G_kernel.sum()
 return Y_k
```

# Test data: create some random data with known peaks

Create a synthetic dataset of `n_clusters` clusters, with `n_samples` random points from each cluster.

```python
n_clusters = 6
n_samples = 1000

# Generate random points (n_clusters of them) in a 2D plane with coordinates spread between -35 and 35
# These centroids will be the means of the clusters.
centroids = np.random.uniform(-35, 35, (n_clusters, 2))

# Generate Gaussian random numbers for each of the above centroid with a (diagonal) co-variance matrix of [[5 0][0 5]]
slices = [np.random.multivariate_normal(centroids[i], np.diag([10., 10.]), n_samples)
          for i in range(n_clusters)]
# slices is an array of array
data = np.concatenate(slices).astype(np.float32)
# data is a n_samples * n_clusters by 2 array of numbers

# plot the data and centroid
def plot_data(centroids, data, n_samples):
    colors = plt.cm.rainbow(np.linspace(0,1,len(centroids)))

    fig, ax = plt.subplots(figsize=(4, 4))
    #for i, centroid in enumerate(centroids):
    for i in range(0, n_clusters) :
        samples = data[i * n_samples : (i + 1) * n_samples] # data for the n-th cluster
        ax.scatter(samples[:, 0], samples[:, 1], color=colors[i], s=1)
        #ax.plot(centroids[i, 0], centroids[i, 1], markersize=10, marker="x", color='k', mew=5)
        ax.plot(centroids[i, 0], centroids[i, 1], markersize=5, marker="x", color='k', mew=2)
    plt.axis('equal')

plot_data(centroids, data, n_samples)
```

# One step of the mean shift for some random samples

```python
# assume each data point as a potential peak. Compare the spread with above
_X = meanshift_step(np.copy(data), np.copy(data))
plot_data(centroids, _X, n_samples)
```

Just repeat this/iterate a few times and we have the complete mean shift algorithm:

```python
def meanshift(X, n_iterations):
    X = np.copy(X)
    for _ in range(n_iterations):
        X = meanshift_step(X, X)
    return X
```

```python
%%time
X = meanshift(data, 5)
plot_data(centroids, X, n_samples)
```

```python
def meanshift_random_seeds(X):
    X = np.copy(X)
    index = np.random.choice(X.shape[0], 100*n_clusters, replace=False)
    cluster_seeds = X[index,:]
    for _ in range(5):
        cluster_seeds = meanshift_step(cluster_seeds, X)
    return cluster_seeds
```

# Mean shift in PyTorch (with GPU)

PyTorch is like numpy and the interface is very similar.

We actually don't have to adjust anything really to use torch instead of numpy.

```python
import torch
```

We have to copy the data into a PyTorch GPU tensor and use torch functions for exp, sqrt etc.

```python
def distance(x, X):
    # return np.linalg.norm(x - X, axis=1)
    return torch.sqrt(((x - X)**2).sum(1))

def gaussian(dist, bandwidth):
    return torch.exp(-0.5 * ((dist / bandwidth))**2) / (bandwidth * torch.sqrt(2 * torch.tensor(np.pi)))

def meanshift_torch(X):
    X = torch.from_numpy(np.copy(X)).cuda()
    for it in range(5):
        X = meanshift_step(X, X)
    return X
```

```python
%time X = meanshift_torch(data).cpu().numpy()
plot_data(centroids, X, n_samples)
```

Same results, but the implementation is about the same speed.

CUDA kernels have to be started for each calculation and the kernels don't have enough to do.
Let's not process individual points, but batches of points.

# Batch processing

```python
#`distance_batch` contains some broadcast magic that allows us to compute the distance from each point in a batch to all points in the data.
# See - https://numpy.org/doc/stable/user/basics.broadcasting.html
#  and https://scipy-lectures.org/intro/numpy/operations.html
def distance_batch(a, b):
    return torch.sqrt(((a[None,:] - b[:,None]) ** 2).sum(2))

# example.
a = torch.rand(2, 2)
b = torch.rand(3, 2)
print(" a=", a, "\n b=", b, "\n", a[None,:], "\n", b[:,None])
d = distance_batch(a, b)
print(d, d.shape)
```

     a= tensor([[0.9570, 0.7006],
            [0.5009, 0.9639]])
     b= tensor([[0.4996, 0.5102],
            [0.2298, 0.9907],
            [0.1620, 0.7188]])
     tensor([[[0.9570, 0.7006],
             [0.5009, 0.9639]]])
     tensor([[[0.4996, 0.5102]],

            [[0.2298, 0.9907]],

            [[0.1620, 0.7188]]])
    tensor([[0.4954, 0.4537],
            [0.7829, 0.2724],
            [0.7952, 0.4183]]) torch.Size([3, 2])

```python
def meanshift_torch2(data, batch_size=500, n_iterations = 5, kernel_width=2.5):
    n = len(data)
    X = torch.from_numpy(np.copy(data)).cuda()
    for _ in range(n_iterations):
        for i in range(0, n, batch_size):
            s = slice(i, min(n, i + batch_size))
            weight = gaussian(distance_batch(X, X[s]), kernel_width)
            num = (weight[:, :, None] * X).sum(dim=1)
            X[s] = num / weight.sum(1)[:, None]
    return X
```

```python
%time X = meanshift_torch2(data, batch_size=1).cpu().numpy()
```

    CPU times: user 8.33 s, sys: 3.32 ms, total: 8.33 s
    Wall time: 8.33 s

```python
%time X = meanshift_torch2(data, batch_size=10).cpu().numpy()
```

    CPU times: user 854 ms, sys: 0 ns, total: 854 ms
    Wall time: 857 ms

```python
%time X = meanshift_torch2(data, batch_size=100).cpu().numpy()
```

    CPU times: user 106 ms, sys: 10 ms, total: 116 ms
    Wall time: 117 ms

```python
%time X = meanshift_torch2(data, batch_size=1000).cpu().numpy()
```

    CPU times: user 54.3 ms, sys: 38.1 ms, total: 92.4 ms
    Wall time: 92.6 ms

```python
%time X = meanshift_torch2(data, batch_size=12000).cpu().numpy()
```

    CPU times: user 64.4 ms, sys: 46.1 ms, total: 111 ms
    Wall time: 112 ms

```python
plot_data(centroids+1, X, n_samples)
```

# Mean shift in scikit-learn

Of course, sklearn also offers `MeanShift`.
Let's see how it performs

```python
from sklearn.cluster import MeanShift
```

```python
%%time
model = MeanShift()
model.fit(data)
```

    CPU times: user 41.1 s, sys: 285 ms, total: 41.3 s
    Wall time: 41.3 s

This is a faster than our naive implementation, but much slower than the GPU version.

# Image Segmentation
We can use meanshift to segment images

```python
from google.colab import drive
drive.mount('/content/drive')
!ls /content/drive/MyDrive/Colab\ Notebooks/data/house_facade.png

```

    Mounted at /content/drive
    '/content/drive/MyDrive/Colab Notebooks/data/house_facade.png'

```python
def show_pixel_scatter (pixels, n_plot = 1000) : # 2D array of pixel values, one pixel per row
  fig = plt.figure()
  ax = fig.add_subplot(111, projection='3d')
  rand_pixels = pixels[np.random.choice(pixels.shape[0], n_plot, replace=False), :]
  ax.scatter(rand_pixels[:,0], rand_pixels[:,1], rand_pixels[:,2])
  ax.set_xlabel('Channel 1')
  ax.set_ylabel('Channel 2')
  ax.set_zlabel('Channel 3')
  plt.show()

def reduce_size (img, scale_percent = 20) :
  #calculate the scale_percent of original dimensions
  width = int(img.shape[1] * scale_percent / 100)
  height = int(img.shape[0] * scale_percent / 100)
  # resize image
  return (cv2.resize(img, (width, height)))

```

```python
import cv2

# Load an color image in grayscale
img = cv2.imread('/content/drive/MyDrive/Colab Notebooks/data/house_facade.png')

# Change color spaces. Can experiment with other spaces.
img = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
print("Input image size:", img.shape)

img = reduce_size(img, scale_percent=20)
plt.imshow(cv2.cvtColor(img, cv2.COLOR_LAB2BGR))
print("Reduced image size:", img.shape)

pixels = np.float32(img.reshape((-1,3))) # change the shape of the 2D image into a 1D array
show_pixel_scatter (pixels, n_plot = 1000)

##------------------
%time X = meanshift_torch2(pixels, batch_size=6000, n_iterations=20, kernel_width=10).cpu().numpy()
##------------------

show_pixel_scatter (X, n_plot = 1000)

seg_img = np.uint8(X.reshape(img.shape))
plt.imshow( cv2.cvtColor(seg_img, cv2.COLOR_LAB2BGR))

```

# Lecture 3 - HW or In-class Assignment #3

1. Conduct ablation study by trying out various combinations of color spaces (BGR, LAB, HCV), kernel_widths (5, 10, 15, 20), n_iterations (20, 100, 200, 500)

2. The above study should reveal that the choice of the kernel_width is critical. Research the literature to find out what strategies have been proposed to choose the kernel_width automatically or at least in an informed mannner.

3. Modify the distance and gaussian kernel computation function to take in a (diagonal) matrix of kernel_width values, not just one scalar number. That is, each dimension, e.g. L, A, and B, can be associated with a different kernel width.

4. Change the code to incorporate a Gaussian kernel over the spatial domain. Comment on the effect.


```python
%pylab inline
%load_ext autoreload
%autoreload 2
import matplotlib.pyplot as plt
import numpy as np
import numpy.random as npr
import sys
sys.path.append("../subjective-fits/")
import seaborn as sns
figsize(8,6)
```

```python
mpl.rcParams['xtick.labelsize'] = 22
mpl.rcParams['ytick.labelsize'] = 22
plt.rc('axes', labelsize=22)
plt.rc('legend', fontsize=22)
mpl.rcParams['ps.useafm'] = True
mpl.rcParams['pdf.use14corefonts'] = True
mpl.rcParams['text.usetex'] = True
```

# Comparing approaches to subjective probability and statistics

We illustrate various approaches to statistics based on subjective probability. Our test case is the estimation of the mean of a one-dimensional Gaussian. We will demonstrate each approach to subjective probability using one point estimate and one "uncertainty" interval. We will also comment on the mathematical derivation as well as the interpretation of these two outputs for each approach.

The setting is $x_1,\dots,x_N\sim\mathcal{N}(\theta,\sigma^2)$, with $\sigma^2$ known. The statistician's goal is to infer $\theta$, and state his uncertainty about his estimate.

```python
def generateData(N, mu, sigma2, tag="well-specified"):
    if tag=="well-specified":
        return mu+np.sqrt(sigma2)*npr.randn(N)
```

```python
#npr.seed(1)
thetaTrue = 1 # "true" value of the mean
sigma2True = 1
alpha = .05
gamma = 0.5 # importance of the length of the interval vs. the interval containing the true value
N = 10
x = generateData(N, thetaTrue, sigma2True)
```

```python
plt.hist(x)
plt.show()
```

The dataset is denoted by $\mathcal{D}= \left\lbrace x_1, ..., x_N \right\rbrace$. The corresonding likelihood function is
$$ \mathcal{L} \left( \theta; \mathcal{D} \right)  = \prod_{i=1}^N \frac{1}{\sigma \sqrt{2\pi}}\exp \left( - \frac{\left( \theta - x_i \right)^2}{2 \sigma^2} \right).$$

The maximum likelihood estimate (MLE) of $\theta$ is the sample mean $\bar{x}$.

```python
intervals = {}
```

## Conjugate Bayes

### The Bayesian paradigm
The Bayesian approach is based on the expected utility principle.
* First, define a cost function $L(\theta,a)$ that measures how much you suffer from picking action $a$ when the state of the world --here, the variable to infer-- is $\mu$. Actions should be functions from the set of states of the world to a space of "outcomes". For our application, an action should consist in picking an estimate $\hat{\theta}$ and a small enough interval $I$, and the outcomes that matter to know if we've suceeded are $a(\theta) = (\theta-\hat{\theta}, \vert I\vert, 1_{\theta\in I})$. So we need to define $L(\theta,a) = L(\theta,\hat{\theta},I)$.
* Second, simply choose a pdf $p(\theta)$ over states of the world, your prior, and choose the action $a$ that minimizes the posterior expected loss
$$
\mathbb{E}_\pi L(\theta, a) = \int L(\theta, a) \pi(\theta)d\theta,
$$
where we have denoted by $\pi(\theta)$ the probability distribution function (pdf) proportional to $p(x\vert\theta)p(\theta)$. By Bayes' theorem, this is the conditional pdf of $\theta$ given data $x$, also called the posterior distribution.

### Choosing a loss function
For the estimate $\hat{\theta}$, we follow here the long tradition of squared losses and we decide to suffer $(\theta-\hat{\theta})^2$ if we output $\hat{\theta}$ while the actual mean of the Gaussian is $\theta$. For the interval, we want to penalize intervals that do not contain the actual $\theta$, while also penalizing intervals that are too wide, to avoid vacuous statements like $I=\mathbb{R}$. This leads us to

$$ L \left (\theta, \hat{\theta}, I \right) = (\theta-\hat{\theta})^2 + \gamma \vert I\vert + 1_{\theta\notin I}. $$

The parameter $\gamma>0$ needs to be chosen by the user, and is interpreted as how much the user favors small intervals over intervals that contain the actual value. There is no need for an additional tradeoff parameter in front of the first term of the right-hand side, since
$$
\min_{\hat{\theta}, I} \mathbb{E}_\pi L(\theta, \hat{\theta}, I) = \min_{\hat{\theta}}\mathbb{E}_\pi (\theta-\hat{\theta})^2 + \min_I \left[\gamma \vert I \vert + \int 1_{\theta\notin I} \pi(\theta) d\theta \right].
$$
Actually this global optimization problem breaks into two simpler ones (wrt to $\theta$ and wrt $I$).

By a property of the mean, the Bayesian estimate $\hat{\theta}$ for our choice of loss is thus the mean of the posterior distribution $\pi$. The Bayesian estimate of $I$ is not obvious at this stage.

### Picking a prior
Now, if our prior $p(\theta)$ is a Gaussian $\mathcal{N}(\mu_\text{prior}, \sigma^2_\text{prior})$, and our likelihood is also Gaussian $\mathcal{N}(\theta,\sigma^2)$, you can check that the posterior $\pi$ is
\begin{equation}
\mathcal{N}\left( \frac{\frac{\mu_\text{prior}}{\sigma^2_\text{prior}}+\frac{\sum_{i=1}^N x_i}{\sigma^2}}{\frac{1}{\sigma^2_\text{prior}}+\frac{N}{\sigma^2}} ,  \frac{1}{\sigma^2_\text{prior}}+\frac{N}{\sigma^2} \right) .
\label{e:posterior}
\end{equation}
This setting where the posterior has the same functional form (here Gaussian) as the prior is called "conjugate", hence the title of this section. Note that we chose a conjugate prior as a mathematical convenience and because it allows easy interpretation, but being Bayesian does not require it. Note how the mean of the posterior is a convex combination of the prior mean and the average of the data points. `ConjugateBayes.estimate()` simply returns this convex combination.

Because the posterior is Gaussian, an interval of given width will always contain more posterior mass if it is centered around the posterior mean. Hence, we can limit our search for the best $I$ to intervals $I_\alpha$ centered around the mean, where $\alpha$ indicates that the posterior puts mass $1-\alpha$ on this interval. Computing the endpoints of $I_\alpha$ simply involves calling the `erf` function, and that is what `ConjugateBayes.interval(1-alpha)` does.

```python
import ConjugateBayes as cb
cb = cb.ConjugateBayes(data=x, sigma2Lhd=sigma2True**2, muPrior=10, sigma2Prior=100)
```

```python
# Each approach should be implemented in one class, in a separate python file. Each approach should have the following two methods:
print(cb.estimate()) # yields a point estimate, here the posterior mean
print(cb.interval(1-alpha)) # yields an uncertainty quantification, here a credible interval of posterior mass 1-alpha
```

Now recall that among centered intervals, I want to pick the $I_\alpha$ with the lowest integrated cost
$$\gamma \vert I_\alpha\vert +  \int \left(1_{\theta\notin I_\alpha}\right) \pi(\theta)d\theta = \gamma \vert I_\alpha\vert + \alpha$$

```python
alphas = np.linspace(1e-5,1-1e-5, 100)
costs_cb = []
for alpha_loc  in alphas:
    # Compute cost for various alphas
    lo, hi = cb.interval(1-alpha_loc)
    costs_cb.append( gamma*(hi-lo) + alpha_loc )

ind = np.argmin(costs_cb) # Find the alpha that minimizes the cost
alphaStar = alphas[ind]
intervals["conjugate Bayes"] = cb.interval(1-alphaStar)
print("The cost is minimized for alpha=", alphaStar)
print("The interval is then", intervals["conjugate Bayes"])
```

```python
# We can visualize the cost as a function of alpha
plt.plot(alphas, costs_cb, color='b')

# adjust the plotting window and show minimizer
m = np.min(costs_cb)
M = np.max(costs_cb)
delta = M-m
yMin = m-.5*delta
yMax = M+.5*delta
plt.ylim([yMin,yMax])
plt.vlines(alphaStar,yMin,yMax,linestyle='--',color='blue')

plt.xlabel(r"$\alpha$")
plt.ylabel("cost")
plt.grid()
plt.show()
```

The Bayes action is to report the credible interval with $\alpha$ set to the minizer of this curve! We can also visualize this interval.

```python
for k, res in enumerate(intervals.items()):
    method, interval = res
    plt.plot([k, k], [interval[0], interval[1]], label=method, linewidth=6)
    plt.grid()
    plt.ylabel(r"$\theta$")
    plt.legend()
```

## Finitely additive Bayes

The expected utility principle, along with the rule that says you should update your beliefs by conditioning on observed events, can be axiomatized. One of the most significant work in this direction is known as Savage's axioms, which is usually what statisticians refer to when they claim  that "being Bayesian is being a coherent rational agent". However, Savage's axioms actually only warrant using finitely additive probability measures, and these measures have to be defined for all subsets of the set of states of the world. This is incompatible with using a Gaussian prior over $\theta\in\mathbb{R}$, as there will be nonmeasurable sets for this Gaussian. TBC.

## Likelihood based Belief Function

### Definitions

In the belief function framework, as well as in other ill-known probabilities frameworks, we work with a pair of functions interpreted as lower and upper bounds of the odds of $\theta$ being $\mu$. The lower bound $bel$ is called **belief function** while the upper bound $pl$ is called **plausibility function**. These functions are set functions, which means that their domain is the set of subsets of $\Theta = \mathbb{R}$. For any subset $A\subset \Theta$, the two functions are related by

$$bel \left( A \right) = 1 - pl \left(A^c\right).$$

There are several ways to derive such function. In this notebook, we present belief functions *à la Dempster* which are induced by a random set. Let $(\Omega,\sigma_\Omega,\mu)$ denote a  probability space. Define a mapping $\Gamma$ from $\Omega$ to the set of subsets of $\Theta$. Under measurability conditions, $\Gamma$ is a **random set**. We can define the two following pseudo-inverse for $\Gamma$:
$$ \Gamma^{-1}_{\top}\left(B\right) = \left\lbrace \omega | \Gamma\left(\omega\right)\cap B \neq \emptyset \right\rbrace,\forall B\subseteq \Theta ,\\
\Gamma^{-1}_{\bot}\left(B\right) = \left\lbrace \omega | \Gamma\left(\omega\right)\subseteq B \right\rbrace,\forall B\subseteq \Theta .$$

Then, we have $pl = \mu \circ \Gamma^{-1}_{\top}$ and $bel = \mu \circ \Gamma^{-1}_{\bot} $.

**Example**:

Going back to our inference problem for $\theta$, we can use the likelihood based belief function. The corresponding plausibility function is defined as
$$ pl \left( \theta \right) = \frac{\mathcal{L} \left( \theta ; \mathcal{D} \right)}{\mathcal{L} \left( \bar{x} ; \mathcal{D} \right)}, \forall \theta \in \mathbb{R} \\
pl \left( H \right) = \underset{\theta \in H}{\sup} pl \left( \theta \right), \forall H \subseteq \mathbb{R}
$$

This model stems from the probability space $(U,\sigma_U,\lambda)$ where $U$ is the unit interval, $\sigma_U$ the Borel sigma-field on this interval and $\lambda$ is the uniform probability measure. Now, by mapping each $u \in U$ to the $\alpha$-cut of function $\frac{\mathcal{L} \left( \theta ; \mathcal{D} \right)}{\mathcal{L} \left( \bar{x} ; \mathcal{D} \right)}$, we span the above plausibility function.

**Interpretation**:
For any event $H$, we can understand the belief and plausibility values as:
* $bel \left( H\right)$ : probability to know that $H$ is true,
* $pl \left( H\right)$ : probability to know that $H^c$ is false,
* $pl \left( H\right) - bel \left( H\right)$ : probability not to know whether $H$ is true.

So we see that the probability mass $pl \left( H\right) - bel \left( H\right)$ is not assigned to any outcome $H$ or $H^c$ in light of the available data.

In the specific case of the Likelihood based belief function, we also have random interval interpretation. In this case, we build the belief function by choosing uniformly a confidence level $\alpha$ and propagate the correponding probability measure to the $\alpha$-cuts of the likelihood function. $bel\left(H\right)$ is then the probability that the random interval is included in $H$ while $pl\left(H\right)$ is the probability that the random interval intersects $H$.

### Belief functions and decision theory

Like the Bayesian paradigm, making decisions with belief functions can also be formalized using expected utility.
Yet belief functions are non-additive measures and we cannot integrate a cost against them in the sense of Lebesgue integral. For this purpose, we need to resort a non-additive integral now as Choquet integral. Let alone these technicalities, making decisions using a belief function shares the same philosophy as in the Bayesian case:

* First, also pick a loss function. In the sequel, we will go for a different loss function as before but our aim is still the same. We need to pick an estimate of $\theta$ and a small condidence interval which is believed to contain $\theta$. We will also penalize incorrect estimations, large intervals and intervals that miss $\theta$.
* Second, update your prior information with your data. The information drawn from the data is embodied by the likelihood based belief function. The prior information is also a belief function. The two functions are combined through an operation called Dempster's rule which we do not detail here. In the case your prior information is a probability distribution, then Dempster's rule yields a posterior distribution which conincides with the Bayesian approach. In the sequel, we choose a genuinely uninformative prior, i.e. a vacuous belief function ($bel = 1_\Theta$).
* Finally, choose the action that minimizes the integrated cost. However, we have two non additive measures against which it can be integrated :
  $$ (C)\int J\: dbel  \text{ (Conservative approach)},\\
  (C)\int J\: dpl  \text{ (Optimistic approach)},$$

  where $(C)\int$ is the Choquet integral symbol.

Just like Bayesians do with Savage's representation theorem, BF practionners invoke Gilboa's representation theorem to claim that the BF approach is compliant with rational decision making. As proved by Gilboa, appraising the cost of an action based on expected utility relying on non-additive measures translates into a preference relation among actions with axioms similar to Savage's. Two important restrictions appears in both Savage's and Gilboa's axiomatizations: the utility function must be bounded (we will comply to this requirement in the sequel) and the measure must have convex range.

### Choosing a loss function

The non-additivity of belief function leads to non-intuitive calculations for people who are familiar with standard probability calculus. In particular, we can use the $0-1$ loss $L_{0-1}$ for the estimation of $\theta$, while this would be pointless within Lebesgue integrals. In the likelihood belief function case, we have

# John, can you justify here separating the cost minimization into the point estimate and the interval estimate? I think you had an argument based on comonotonicity

$$ (C)\int L_{0-1}\: dbel = 1 - pl \left( \hat{\theta} \right) ,\\
  (C)\int L_{0-1}\: dpl = 1.$$

In the belief case, we obtain $\hat{\theta} = \bar{x}$, while plausibility cannot draw any conclusion. We thus use the belief function for the estimation of $\theta$, and the method  `BaselineBF.estimate` returns the MLE estimate.

For the interval estimation problem, we will use the same cost function as before, and we obtain
$$ (C)\int \vert I\vert + 1_{\theta\notin I} \: dbel = \gamma \vert I \vert ,\\
  (C)\int  \vert I\vert + 1_{\theta\notin I}\: dpl = \gamma \vert I \vert + \sup_{\theta\notin I}pl(\theta).$$

The integral wrt the belief function is always minimized when $\vert I\vert=0$, leaving $I$ undetermined. For the plausibility function, the likelihood being Gaussian, it is enough to minimize the expected cost among intervals that are upper level sets of the likelihood, that is
$$ I_\alpha = \{\theta/ pl(\theta)<=\alpha\}.$$
We minimize over $\alpha$ below.

To conclude, note that we cannot use the same non-additive measure for both estimation problems. This is in contrast with the Bayesian approach.

```python
import BaselineBF as bf
bf = bf.BaselineBF(data=x, sigma2Lhd=sigma2True)
```

```python
print(bf.estimate()) # yields a point estimate, here the MLE
```

  Now, we are still interested in the same interval cost function $J \left(\alpha \right)$

  The method `BaselineBF.interval` returns the estimated "confidence" interval.

```python
print(bf.interval(1-alpha))
```

```python
costs_cons = []
costs_optm = []
for alpha_loc  in alphas:
    lo, hi = bf.interval(1-alpha_loc)
    costs_cons.append( gamma*(hi-lo))
    costs_optm.append( gamma*(hi-lo) + alpha_loc )

yMin = 0
yMax = np.max(costs_optm)
plt.ylim([yMin,yMax])

# Conservative approach
ind = np.argmin(costs_cons) # Find the alpha that minimizes the cost
alphaStar = alphas[ind]
plt.vlines(alphaStar,yMin,yMax,linestyle='--',color='red')
print("The conservative cost is minimized for alpha=", alphaStar)
lo, hi = bf.interval(1-alphaStar)
intervals["conservative likelihood-based BF"] = [lo, hi]
print("The interval is then", intervals["conservative likelihood-based BF"])

# Optimistic approach
ind = np.argmin(costs_optm) # Find the alpha that minimizes the cost
alphaStar = alphas[ind]
plt.vlines(alphaStar,yMin,yMax,linestyle='--',color='green')
print("The optimistic cost is minimized for alpha=", alphaStar)
lo, hi = bf.interval(1-alphaStar)
intervals["optimistic likelihood-based BF"] = [lo, hi]
print("The interval is then", intervals["optimistic likelihood-based BF"])

# plot the costs
plt.plot(alphas, costs_cons, label="conservative", color='r')
plt.plot(alphas, costs_optm, label="optimistic", color='g')
plt.plot(alphas, costs_cb, label="conjugate Bayes", color='b')

# show the minimzers

plt.xlabel(r"$\alpha$")
plt.ylabel("cost")
plt.legend()
plt.show()
```

We focus on the optimistic cost as the conservative does not provide a sufficient level of information. In this experiment, the optimistic cost is minimized for the same value of $\alpha$ as in the conjugate Bayes case. This may be purely conincidental as other values of $\gamma$ may not lead to such a result. Obviously, when $\gamma$ is small, both costs become closer.

We also see that the optimistic cost curve is not convex while the conjugate Bayes curve seems to be.

## Comparing intervals

```python
for k, res in enumerate(intervals.items()):
    method, interval = res
    plt.plot([k, k], [interval[0], interval[1]], label=method, linewidth=6)
    plt.grid()
    plt.legend()
```

```python
print(intervals)
```

```python

```

```python

```


# Decision Tree's

A learning algorithm that uses _information gain_ to measure how well a feature of the data distinguishes between different classes. Information gain is measured in terms of _entropy_.

Decision Tree's can be thought of as a compact representation of a function. In a binary classification scenario, this would mean that a decision tree represents a truth table. Decision tree's attempt to predict a qualitative (categorical) response by stratifying the predictor space into simpler regions. Concretely, the prediction space is divided into regions representing classes. Decision tree's can also be used in regression scenarios.

## MONK Dataset

In this notebook we'll use the [MONK Dataset](https://archive.ics.uci.edu/ml/datasets/MONK's+Problems) from UC Irvine's Machine Learning Repository. The MONK dataset is a collection of artificial data which spans two classes and distributed by some underlying function.

### True Functions

* MONK-1: $(a_1 = a_2) \wedge (a_5 = 1)$
* MONK-2: $a_i = 1 \text{ for exactly two } i \in \{1,2,\ldots,6\}$
* MONK-3: $(a_5 = 1 \wedge a_4 = 1) \vee (a_5 \neq 4 \wedge a_2 \neq 3)$

MONK-3 contains misclassified data (5%). In the case of noisy data, the learning algorithm may generate only partially consistent rules.

### Loading Data

Data will be accessible via a dictionary, e.g. `data['monks-1']['train']`

```python
from pathlib import Path

data_path = Path(__name__).resolve().parents[1] / 'data' / 'MONK'
m1_train = data_path / 'monks-1.train'
print("A single example from MONK-1")
with m1_train.open() as f:
    print(f.readline())
```

    A single example from MONK-1
     1 1 1 1 1 3 1 data_5

Attribute information:
  1. class: 0, 1
  2. a1:    1, 2, 3
  3. a2:    1, 2, 3
  4. a3:    1, 2
  5. a4:    1, 2, 3
  6. a5:    1, 2, 3, 4
  7. a6:    1, 2
  8. Id:    (A unique symbol for each instance)

```python
from typing import NamedTuple

data = {
    'monks-1': {
        'train': [],
        'test': [],
    },
    'monks-2': {
        'train': [],
        'test': [],
    },
    'monks-3': {
        'train': [],
        'test': [],
    },
}

class MonkDatum(NamedTuple):
    a: list
    label: int
    symbol: str

for key, subkeys in data.items():
    for subkey in subkeys:
        data_file = data_path / f"{key}.{subkey}"
        with data_file.open() as f:
            data[key][subkey] = [MonkDatum(a=line[1:7], label=line[0], symbol=line[7]) for line in (line.split() for line in f.readlines())]
        print(f"Loaded {key} {subkey} dataset")
```

    Loaded monks-1 train dataset
    Loaded monks-1 test dataset
    Loaded monks-2 train dataset
    Loaded monks-2 test dataset
    Loaded monks-3 train dataset
    Loaded monks-3 test dataset

## Entropy

Entropy is a measure of information. In natural language processing, entropy can be a measure of how much information there is in a grammar or how well a grammar fits a language. To compute entropy, establish a random variable $S$ over the class labels. . The entropy of any $S$ is.

\begin{equation}
\text{Entropy}(S) = - \sum_ip_i\log_2p_i
\end{equation}

$S$: A collection of samples

$p_i$: Proportion of examples of class $i$ in $S$. This is a probability function $p(x)$

In a binary classification problem, $\{0, 1\} \in S$, and the entropy of $S$ would become:

\begin{equation}
\text{Entropy}(S) = - p_0 \log_2 p_0 - p_1 \log_2 p_1
\end{equation}

The log can be measured in any base. In base 2, the units of entropy are *bits*. The intuition behind entropy is to think of it as a lower bound on the number of bits needed for an optimal length encoding scheme for some piece of information.

```python

def entropy(S, p):
    """Compute entropy of S given probability function p"""
    pass
```


# Cálculo diferencial en varias variables (I)

Unha das funcionalindades máis importantes do módulo **Sympy** é as ferramentas das que dispón para o cálculo de derivadas, xa se traten de derivadas ordinarias (para expresións que dependen dunha única variable) como de derivadas parciais (para expresións definidas a partir de varias variables). Nesta práctica, estudaremos como calcular derivadas parciais de primeira orde, derivadas direccionais usando o vector gradiente e a relación entre o gradiente e as curvas de nivel dunha función de dúas variables.

### Obxectivos:

- Cálculo de derivadas ordinarias nunha variable
- Cálculo de derivadas parciais de primeira orde
- Cálculo do gradiente e de derivadas direccionais
- Representación gráfica do gradiente e das curvas de nivel

Para dispoñer do módulo **Sympy** e importalo para o resto do guión de prácticas, usaremos:

```python
import sympy as sp
```

## Cálculo de derivadas ordinarias nunha variable
Para calcular derivadas, o módulo **Sympy** ten a función `sp.diff` no que se debe especificar como argumentos de entrada tanto a expresión matemática como a variable con respecto á que se deriva. Por exemplo, para calcular $f'(x)$ sendo $f(x)=x^2\cos(2x)$ empregaríamos:

```python
x = sp.Symbol('x', real=True) # define a variable simbólica x
f = sp.Lambda((x), x**2*sp.cos(2*x)) # define a función a derivadar
df = sp.diff(f(x),x) # cálculo da derivada
display(df)
```

    -2*x**2*sin(2*x) + 2*x*cos(2*x)

Para calcular a segunda derivada ou a derivada $n$-ésima dunha expresión, unicamente hai que indicarlle o número de derivadas que se quere realizar engadindo este valor como argumento opcional. Ademais, as funcións dunha variable poder conter parámetros constantes. Por exemplo, para calcular as derivadas sucesivas da función $f(x)=x^2+ax+b$ sendo $a$, $b$ dous parámetros constantes:

```python
a, b = sp.symbols('a b', real=True) # define as variables simbólicas a, b
f = sp.Lambda((x), x**2+a*x+b) # define a función a derivadar
df = sp.diff(f(x),x,1) # cálculo da primeira derivada
display(df)
d2f = sp.diff(f(x),x,2) # cálculo da primeira derivada
display(d2f)
d3f = sp.diff(f(x),x,3) # cálculo da primeira derivada
display(d3f)
```

    a + 2*x

    2

    0

## Cálculo de derivadas parciais de primeira orde
O cálculo  de derivadas parciais de funcións de varias variables faise do mesmo xeito que no caso de derivadas ordinarias, tendo coidado de especificar a variable con respecto a que se está a derivar:

```python
x, y = sp.symbols('x y', real=True) # define as variables simbólicas x, y
f=sp.Lambda((x,y), x/((x+y)*y))
dfx = sp.diff(f(x,y),x)
display(sp.factor(dfx)) # Emprégase sp.factor para simplificar a expresión das derivadas
dfy = sp.diff(f(x,y),y)
display(sp.factor(dfy))
```

    (x + y)**(-2)

    -x*(x + 2*y)/(y**2*(x + y)**2)

Podemos comprobar que as expresións obtidas coinciden coa definición baseada no cálculo de límites de cocientes incrementais ao variar cada unha das variables $x$ e $y$:

```python
h = sp.Symbol('h', real=True) # define a variable simbólica h
dfx_def = sp.limit((f(x+h,y)-f(x,y))/h,h,0)
display(sp.factor(dfx_def))  # Emprégase sp.factor para simplificar a expresión das derivadas
dfy_def = sp.limit((f(x,y+h)-f(x,y))/h,h,0)
display(sp.factor(dfy_def))
```

    (x + y)**(-2)

    -x*(x + 2*y)/(y**2*(x + y)**2)

### **Exercicio 6.1**
Calcula todas as derivadas parciais de primeira orde da función de catro variables $f(x,y,z,u)=\displaystyle\frac{y+x-z}{u-x^3}$.

```python
# O TEU CÓDIGO AQUÍ
```

## Cálculo do vector gradiente
Posto que o módulo **Sympy** é capaz de calcular as derivadas parciais de primeira orde, tamén podemos calcular o vector gradiente. Para evitar complicacións na implementación (e evitar o uso das clases de obxectos e métodos que están implementados no submódulo `sp.vector`), imos a definir a noso propio procedemento de calcular gradientes (almacenando as compoñentes do vector gradiente nunha lista):

```python
def gradient(expr, varlist):
    return [sp.diff(expr,var) for var in varlist] # cálculo do gradiente almacenado nunha lista
```

Deste xeito, o vector gradiente calcúlase e almacénase nunha lista, onde cada derivada parcial se garda na lista seguindo a orde das variables con respecto á que se deriva. Por exemplo no caso de $f(x,y)=3x^2-5y^2$, obtemos que $$
\nabla f(x,y)=
\begin{pmatrix}
6x\\
-10y
\end{pmatrix}.
$$

```python
x, y = sp.symbols('x y', real=True) # define as variables simbólicas x, y
f = sp.Lambda((x,y), 3*x**2-5*y**2)
grad_f = gradient(f(x,y),(x,y)) # cálculo do gradiente de f usando as variables (x,y)
display(grad_f)
```

    [6*x, -10*y]

> **Nota**: Nesta implementación, estamos a almacenar cada unha das compoñentes do vector gradiente como os elementos dunha lista. Nótese que poderíamos empregar unha implementación alternativa usando os obxectos `sp.vector` que están dispoñibles en **Sympy** ou `numpy.array` que xa se empregaron no módulo **Numpy**.

### Cálculo de derivadas direccionais
Como xa revisamos nas clases de pizarra, a derivada direccional na dirección do vector unitario $\vec{u}$ pódese calcular mediante o produto escalar do gradiente polo vector unitario, isto é, $D_{\vec{u}}f=\nabla f\cdot\vec{u}$. Posto que nesta práctica estamos a usar listas para almacenar os vectores, debemos implementar tamén unha función **Python** para calcular o produto escalar de dúas listas (non se podería usar a función `np.dot` xa que non se está a traballar con obxectos da clase `np.array`):

```python
def dot(u, v):
    return sum([a*b for a,b in zip(u, v)]) # cálculo do produto interior de dous vectores almacenados en listas
```

Agora tendo en conta esta función `dot`, o cálculo de $D_{\vec{u}}f=\nabla f\cdot\vec{u}$ é inmediato. Véxamos cun exemplo onde se calcular a derivada direccional de $f(x,y)=3x+2y$ na dirección do vector unitario $\vec{u}=(-1,1)^{t}/\sqrt{2}$:

```python
f = sp.Lambda((x,y),3*x+2*y)
u = [-1/sp.sqrt(2), 1/sp.sqrt(2)] # vector unitario que define a dirección
Dfu = dot(gradient(f(x,y),(x,y)),u)
display(Dfu)
```

    -sqrt(2)/2

### **Exercicio 6.2**
Calcula a derivada direccional da función $f(x,y,z,u)=\displaystyle\frac{y+x-z}{u-x^3}$ con respecto á dirección indicada polo vector $\vec{v}=(1,0,-1,1)^t/\sqrt{3}$ no punto $(2,1,-2,1)$.

```python
# O TEU CÓDIGO AQUÍ
```

## Representación gráfica de gradientes e curvas de nivel
Cando temos en conta as curvas de nivel e a súa relación cos gradientes da función que se está a estudar, é doado comprobar (usando derivación implícita) que os vectores tanxentes á curva de nivel son perpediculares ao vector gradiente da funcion que define as curvas de nivel. Para ilustrar isto, consideremos a función de dúas variables:
$$
f(x, y) = \sin(x) \cos\left(\frac{xy^2}{10}\right),\qquad (x,y)\in[-\pi,\pi]\times[-\pi,\pi].
$$
Para iso, definimos a función co módulo **Sympy** e calculamos o seu gradiente:

```python
x, y = sp.symbols('x y', real=True)
f = sp.Lambda((x,y),sp.sin(x)*sp.cos(x*y**2/10))
grad_f = gradient(f(x,y),(x,y))
display(grad_f)
```

    [-y**2*sin(x)*sin(x*y**2/10)/10 + cos(x)*cos(x*y**2/10),
     -x*y*sin(x)*sin(x*y**2/10)/5]

Imos primeiro a representar as curvas de nivel, tal e como se fixo na práctica 4, primeiro importamos os módulos de **Numpy** e **Matplotlib**:

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

Nembargantes, antes de poder representar graficamente debemos ser capaces de traducir os obxectos que dependen do módulo **Sympy** a obxectos do módulo **Numpy**, que serán os que usaremos para a representación de curvas de nivel e dos vectores gradiente. Para isto, empregaremos o método `sp.lambdify`:

```python
F = sp.lambdify((x,y),f(x,y),"numpy") # función numpy coa expresión de f(x,y)
grad_F = sp.lambdify((x,y),grad_f,"numpy") # función numpy coa expresión vectorial de grad(f)
```

Para representar as curvas de nivel e o campo de vectores do gradiente é preciso crear unha nube de puntos sobre a que pintar os valores da función. Usaremos dúas nubes de puntos diferentes para cada caso:

```python
N = 50  # 50 puntos en cada dirección dos eixes cartesianos
x = np.linspace(-np.pi, np.pi, N)
y = np.linspace(-np.pi, np.pi, N)
X, Y = np.meshgrid(x, y)
Values = F(X,Y)

Ng = 10  # 10 puntos en cada dirección dos eixes cartesianos
xg = np.linspace(-np.pi, np.pi, Ng)
yg = np.linspace(-np.pi, np.pi, Ng)
Xg, Yg = np.meshgrid(xg, yg)
Values_grad=grad_F(Xg,Yg)
```

E usando a función `contour`  e `quiver` obtense a representación gráfica das curvas de nivel e o gradiente:

```python
p = plt.contour(X, Y, Values) # Representación gráfica das curvas de nivel
plt.clabel(p) # Engádese o valor numérico a cada curva de nivel
plt.quiver(Xg, Yg, Values_grad[0], Values_grad[1]) # Representación gráfica do campo de vectores gradiente
# Etiquetas dos eixes
plt.xlabel('x')
plt.ylabel('y')
plt.axis('square')
plt.show()
```

### **Exercicio 6.3**
Define unha función de dúas variables onde todas as súas curvas de nivel sexan rectas. Representa graficamente as curvas de nivel e comproba que o gradiente ven dado por un campo de vectores que posúen a mesma dirección. Verificanto as condicións anteriores, constrúe exemplos nos que:
- O gradiente é constante para todos os puntos
- O gradiente ten a mesma dirección en todos os puntos pero varía o módulo dos vectores en cada punto

```python
# O TEU CÓDIGO AQUÍ
```


# Prior distributions

One important aspect of Bayesian inference has not yet been discussed in this tutorial: [prior distributions](https://en.wikipedia.org/wiki/Prior_probability). In Bayesian statistics, one has to provide probability (density) values for every possible parameter value *before* taking into account the data at hand. This prior distribution thus reflects all *prior* knowledge of the system that is to be investigated. In the case that no prior knowledge is available, a *non-informative* prior in the form of the so-called [Jeffreys prior](https://en.wikipedia.org/wiki/Jeffreys_prior) allows to minimize the effect of the prior on the results. The next two sub-sections discuss how one can set custom prior distributions for the parameters of the observation model and for hyper-parameters in a hyper-study or change-point study.

```python
%matplotlib inline
import matplotlib.pyplot as plt # plotting
import seaborn as sns           # nicer plots
sns.set_style('whitegrid')      # plot styling

import numpy as np
import bayesloop as bl

# prepare study for coal mining data
S = bl.Study()
S.loadExampleData()
```

    + Created new study.
    + Successfully imported example data.

## Parameter prior

*bayesloop* employs a forward-backward algorithm that is based on [Hidden Markov models](http://www.cs.sjsu.edu/~stamp/RUA/HMM.pdf). This inference algorithm iteratively produces a parameter distribution for each time step, but it has to start these iterations from a specified probability distribution - the parameter prior. All built-in observation models already have a predefined prior, stored in the attribute `prior`. Here, the prior distribution is stored as a Python function that takes as many arguments as there are parameters in the observation model. The prior distributions can be looked up directly within `observationModels.py`. For the `Poisson` model discussed in this tutorial, the default prior distribution is defined in a method called `jeffreys` as
```
def jeffreys(x):
    return np.sqrt(1. / x)
```
corresponding to the non-informative Jeffreys prior, $p(\lambda) \propto 1/\sqrt{\lambda}$. This type of prior can also be determined automatically for arbitrary user-defined observation models, see [here](customobservationmodels.html#Sympy.stats-random-variables).

### Prior functions and arrays

To change the predefined prior of a given observation model, one can add the keyword argument `prior` when defining an observation model. There are different ways of defining a parameter prior in *bayesloop*: If `prior=None` is set, *bayesloop* will assign equal probability to all parameter values, resulting in a uniform prior distribution within the specified parameter boundaries. One can also directly supply a Numpy array with prior probability (density) values. The shape of the array must match the shape of the parameter grid! Another way to define a custom prior is to provide a function that takes exactly as many arguments as there are parameters in the defined observation model. *bayesloop* will then evaluate the function for all parameter values and assign the corresponding probability values.

**Note:** In all of the cases described above, *bayesloop* will re-normalize the provided prior values, so they do not need to be passed in a normalized form. Below, we describe the possibility of using probability distributions from the SymPy stats module as prior distributions, which are not re-normalized by *bayesloop*.

Next, we illustrate the difference between the Jeffreys prior and a flat, uniform prior with a very simple inference example: We fit the coal mining example data set using the `Poisson` observation model and further assume the rate parameter to be static:

```python
# we assume a static rate parameter for simplicity
S.set(bl.tm.Static())

print 'Fit with built-in Jeffreys prior:'
S.set(bl.om.Poisson('accident_rate', bl.oint(0, 6, 1000)))
S.fit()
jeffreys_mean = S.getParameterMeanValues('accident_rate')[0]
print('-----\n')

print 'Fit with custom flat prior:'
S.set(bl.om.Poisson('accident_rate', bl.oint(0, 6, 1000),
                                     prior=lambda x: 1.))
# alternatives: prior=None, prior=np.ones(1000)
S.fit()
flat_mean = S.getParameterMeanValues('accident_rate')[0]
```

    + Transition model: Static/constant parameter values. Hyper-Parameter(s): []
    Fit with built-in Jeffreys prior:
    + Observation model: Poisson. Parameter(s): ['accident_rate']
    + Started new fit:
        + Formatted data.
        + Set prior (function): jeffreys. Values have been re-normalized.

        + Finished forward pass.
        + Log10-evidence: -88.00564

        + Finished backward pass.
        + Computed mean parameter values.
    -----

    Fit with custom flat prior:
    + Observation model: Poisson. Parameter(s): ['accident_rate']
    + Started new fit:
        + Formatted data.
        + Set prior (function): <lambda>. Values have been re-normalized.

        + Finished forward pass.
        + Log10-evidence: -87.98915

        + Finished backward pass.
        + Computed mean parameter values.

First note that the model evidence indeed slightly changes due to the different choices of the parameter prior. Second, one may notice that the posterior mean value of the flat-prior-fit does not exactly match the arithmetic mean of the data. This small deviation shows that a flat/uniform prior is not completely non-informative for a Poisson model! The fit using the Jeffreys prior, however, succeeds in reproducing the *frequentist* estimate, i.e. the arithmetic mean:

```python
print('arithmetic mean     = {}'.format(np.mean(S.rawData)))
print('flat-prior mean     = {}'.format(flat_mean))
print('Jeffreys prior mean = {}'.format(jeffreys_mean))
```

    arithmetic mean     = 1.69090909091
    flat-prior mean     = 1.7
    Jeffreys prior mean = 1.69090909091

### SymPy prior

The second option is based on the [SymPy](http://www.sympy.org/en/index.html) module that introduces symbolic mathematics to Python. Its sub-module [sympy.stats](http://docs.sympy.org/dev/modules/stats.html) covers a wide range of discrete and continuous random variables. The keyword argument `prior` also accepts a list of `sympy.stats` random variables, one for each parameter (if there is only one parameter, the list can be omitted). The multiplicative joint probability density of these random variables is then used as the prior distribution. The following example defines an exponential prior for the `Poisson` model, favoring small values of the rate parameter:

```python
import sympy.stats
S.set(bl.om.Poisson('accident_rate', bl.oint(0, 6, 1000),
                                     prior=sympy.stats.Exponential('expon', 1)))
S.fit()
```

    + Observation model: Poisson. Parameter(s): ['accident_rate']
    + Started new fit:
        + Formatted data.
        + Set prior (sympy): exp(-x)

        + Finished forward pass.
        + Log10-evidence: -87.94640

        + Finished backward pass.
        + Computed mean parameter values.

Note that one needs to assign a name to each `sympy.stats` variable. In this case, the output of *bayesloop* shows the mathematical formula that defines the prior. This is possible because of the symbolic representation of the prior by `SymPy`.

**Note:** The support interval of a prior distribution defined via SymPy can deviate from the parameter interval specified in *bayesloop*. In the example above, we specified the parameter interval ]0, 6[, while the exponential prior has the support ]0, $\infty$[. SymPy priors are not re-normalized with respect to the specified parameter interval. Be aware that the resulting model evidence value will only be correct if no parameter values outside of the parameter boundaries gain significant probability values. In most cases, one can simply check whether the parameter distribution has sufficiently *fallen off* at the parameter boundaries.

## Hyper-parameter priors

As shown before, [hyper-studies](hyperstudy.html) and [change-point studies](changepointstudy.html) can be used to determine the full distribution of hyper-parameters (the parameters of the transition model). As for the time-varying parameters of the observation model, one might have prior knowledge about the values of certain hyper-parameters that can be included into the study to refine the resulting distribution of these hyper-parameters. Hyper-parameter priors can be defined just as regular priors, either by an arbitrary function or by a list of `sympy.stats` random variables.

In a first example, we return to the simple change-point model of the coal-mining data set and perform to fits of the change-point: first, we specify no hyper-prior for the time step of our change-point, assuming equal probability for each year in our data set. Second, we define a Normal distribution around the year 1920 with a (rather unrealistic) standard deviation of 5 years as the hyper-prior using a SymPy random variable. For both fits, we plot the change-point distribution to show the differences induced by the different priors:

```python
print 'Fit with flat hyper-prior:'
S = bl.ChangepointStudy()
S.loadExampleData()

L = bl.om.Poisson('accident_rate', bl.oint(0, 6, 1000))
T = bl.tm.ChangePoint('tChange', 'all')

S.set(L, T)
S.fit()

plt.figure(figsize=(8,4))
S.plot('tChange', facecolor='g', alpha=0.7)
plt.xlim([1870, 1930])
plt.show()
print('-----\n')

print 'Fit with custom normal prior:'
T = bl.tm.ChangePoint('tChange', 'all', prior=sympy.stats.Normal('norm', 1920, 5))
S.set(T)
S.fit()

plt.figure(figsize=(8,4))
S.plot('tChange', facecolor='g', alpha=0.7)
plt.xlim([1870, 1930]);
```

Since we used a quite narrow prior (containing a lot of information) in the second case, the resulting distribution is strongly shifted towards the prior. The following example revisits the two break-point-model from [here](changepointstudy.html#Analyzing-structural-breaks-in-time-series-models) and a linear decrease with a varying slope as a hyper-parameter. Here, we define a Gaussian prior for the slope hyper-parameter, which is centered around the value -0.2 with a standard deviation of 0.4, via a lambda-function. For simplification, we set the break-points to fixed years.

```python
S = bl.HyperStudy()
S.loadExampleData()

L = bl.om.Poisson('accident_rate', bl.oint(0, 6, 1000))
T = bl.tm.SerialTransitionModel(bl.tm.Static(),
                                bl.tm.BreakPoint('t_1', 1880),
                                bl.tm.Deterministic(lambda t, slope=np.linspace(-2.0, 0.0, 30): t*slope,
                                                    target='accident_rate',
                                                    prior=lambda slope: np.exp(-0.5*((slope + 0.2)/(2*0.4))**2)/0.4),
                                bl.tm.BreakPoint('t_2', 1900),
                                bl.tm.Static()
                                )

S.set(L, T)
S.fit()
```

    + Created new study.
      --> Hyper-study
    + Successfully imported example data.
    + Observation model: Poisson. Parameter(s): ['accident_rate']
    + Transition model: Serial transition model. Hyper-Parameter(s): ['slope', 't_1', 't_2']
    + Set hyper-prior(s): ['<lambda> (re-normalized)', 'uniform', 'uniform']
    + Started new fit.
        + 30 analyses to run.

        + Computed average posterior sequence
        + Computed hyper-parameter distribution
        + Log10-evidence of average model: -74.84129
        + Computed local evidence of average model
        + Computed mean parameter values.
    + Finished fit.

Finally, note that you can mix SymPy- and function-based hyper-priors for nested transition models.


# Gradient Descent
### A from-stratch implementation in Julia

I thought I understood backprop. I had read a number of explanations and had no trouble following the math or the code examples. Besides, it's just the chain rule, right?

Then I tried to implement backprop myself from the chain rule equations and all of I sudden I had no idea what I was doing and my code would either behave strangely or not work at all due to incompatibly shaped matrices.

It turns out that a lot of explanations out there of the math behind backprop seem precise but are actually pretty vaguely connected to 1) the network they are operating over and 2) the actual code. So I started this series of notebooks to explain to myself how to implement backprop at increasing levels of complexity, being as explicit as possible about how the math connects to the code. Julia was a natural choice for this, as it has built in support for TeX characters like $\partial$, $\nabla$, etc. which come up often.

We'll start with the simplest network I could think of: 2 input nodes connected directly to a single output node, no hidden layers, no bias, and a linearly separable problem.

# A very basic backprop

Backpropagation is often discussed in the context of *deep* neural networks, but it works just fine for linear models too, so let's start there. To do this, we'll need a linearly separable problem.

$X = \begin{pmatrix} 1 & 3 \\ 2 & 3 \\ 2 & 1 \\ 1 & 1 \end{pmatrix}$

$y = \begin{pmatrix} 1 \\ 1 \\ 0 \\ 0 \end{pmatrix}$

```julia
X = [1 3; 2 3; 2 1; 1 1]
y = [1, 1, 0, 0]
println(X)
println(y)
```

    [1 3; 2 3; 2 1; 1 1]
    [1, 1, 0, 0]

```julia
using Plots; pyplot()
scatter(X[:, 1], X[:, 2], color=[:blue, :blue, :red, :red], markersize=10, legend=false)
```

Immediately, we can see that it is possible to separate the blue from the red points by drawing a horizontal decision boundary at around $y=2$. Below, we'll define a linear model and use backprop to estimate this decision boundary.

## Defining the model

We'll define a linear model with two input nodes (one for each column, or *feature*, in $X$) and one output node using sigmoid activation, which is standard fare for binary prediction models. Such models generally include a bias term as well, but we'll leave it out for now to keep things simple.

Let's write out the equations for this network here.

$o = \sigma(XW)$  the model's predictions

where

$X \in \mathbb{R}^{4 \times 2}$ the input data

$W \in \mathbb{R}^{2 \times 1}$ the weights

$\sigma(z)= \frac{1}{1 + e^{-z}}$  the sigmoid activation function

Below, we initialze $W$ to some starting values and define a function for computing the sigmoid.

```julia
W = [0.1, 0.2]

function σ(z)
    # Sigmoid activation function
    return @. 1 / (1 + exp(-z))
end
```

    σ (generic function with 1 method)

## The forward pass

Now, let's compute our model's predictions $o$ by stepping through the equations above. Given our initialized $W$

$W = \begin{pmatrix} 0.1 \\ 0.2 \end{pmatrix}$

the weighted inputs, $z$, are computed

$z = XW = \begin{pmatrix} 1 & 3 \\ 2 & 3 \\ 2 & 1 \\ 1 & 1 \end{pmatrix} \begin{pmatrix} 0.1 \\ 0.2 \end{pmatrix} = \begin{pmatrix} 0.7 \\ 0.8 \\ 0.4 \\ 0.3 \end{pmatrix}$

Applying the sigmoid to these values we get

$o = \sigma(z) = \begin{pmatrix} 0.67 \\ 0.69 \\ 0.60 \\ 0.57 \end{pmatrix}$

```julia
z = X * W
println("z: $z")
o = σ(z)
println("o: $o")
```

    z: [0.7000000000000001, 0.8, 0.4, 0.30000000000000004]
    o: [0.6681877721681662, 0.6899744811276125, 0.598687660112452, 0.574442516811659]

### Evaluation

The model outputs real-valued scores, so we'll need a decision function to transform these into discrete binary values. Our decision function will assign a 1 to values in $o \ge 0.5$ and a 0 otherwise.

$\hat{y} = \begin{cases}
    1, & \text{if}~~~ o \ge 0.5 \\
    0, & \text{otherwise}
\end{cases}$

We can then use these decisions to compute the model's accuracy against the gold-standard labels $y$.

```julia
function decision(o)
    return Int.(o .>= 0.5)
end

function accuracy(y, y_hat)
    N = size(y, 1)
    return sum(y .== y_hat) / N
end

y_hat = decision(o)
acc = accuracy(y, y_hat)

println("̂y: $(y_hat)")
println("y: $(y)")
println("accuracy: $(acc)")
```

    ̂y: [1, 1, 1, 1]
    y: [1, 1, 0, 0]
    accuracy: 0.5

Not very good, but we haven't started training yet!

### Loss

In order to train with backprop, we'll need a way to compute the total loss of our model's real-valued predictions $o$ with respect to the gold-standard labels $y$. Since this is a binary classification problem, we can use binary cross-entropy loss.

$C = -\frac{1}{N} \sum_{i=1}^{N} y_i \times ln(o_i) + (1 - y_i) \times ln(1 - o_i)$

Where $N$ is the number of examples in $X$ (4 in our case), $y_i$ and $o_i$ are the true and predicted labels for example $i$, respectively.

$C = \frac{-1}{4} \times \begin{pmatrix}
    1 \times ln(0.67) + (1 - 1) \times ln(1 - 0.67) ~ + \\
    1 \times ln(0.69) + (1 - 1) \times ln(1 - 0.69) ~ + \\
    0 \times ln(0.60) + (1 - 0) \times ln(1 - 0.60) ~ + \\
    0 \times ln(0.57) + (1 - 0) \times ln(1 - 0.57) ~~~~
\end{pmatrix}
=
\frac{-1}{4} \times \begin{pmatrix}
    -0.40 ~ + \\
    -0.37 ~ + \\
    -0.92 ~ + \\
    -0.84 ~~~~
\end{pmatrix}
= \frac{2.54}{4} = 0.635
$

A function for computing the binary cross-entropy loss is below.

```julia
function clip(a, lo=1e-15, hi=1 - 1e-15)
    if a < lo
        return lo
    elseif a > hi
        return hi
    else
        return a
    end
end

function binary_crossentropy(y, o)
    N = size(y, 1)
    # We "clip" the predicted values to avoid
    # domain errors in the log() function if our
    # model happens to predict 0.0, and to prevent
    # log returning 0.0 if the model predicts 1.0
    o = clip.(o)
    ces = @. (y * log(o)) + ((1-y) * log(1-o))
    return (-1/N) * sum(ces)
end

loss = binary_crossentropy(y, o)
println("Loss: $(loss)")
```

    Loss: 0.6354143029254289

## Derivatives for the backward pass

The goal of the backward pass is to compute the updates for the model weights $W$. We do this by propagating the loss backwards through the model, from the loss $C$ to $W$. In other words, we want to know how the loss changes when we change each of the weights $w_i \in W$. This means we need to take the partial derivative of the loss with respect to each of the weights. Since the value of our loss function $C$ is not a direct function of $W$, but rather an indirect function of $W$ via $o = \sigma(z), z = XW$, we need the chain rule.

$$ \frac{\partial{C}}{\partial{W}} = \frac{\partial{C}}{\partial{o}} \frac{d{o}}{d{z}} \frac{\partial{z}}{\partial{W}} $$

## 1. $\frac{\partial{C}}{\partial{o}}$
Let's first compute the partial derivative of the cost with respect to the network output, $\frac{\partial{C}}{\partial{o}}$. We saw above that

$$ C = -\frac{1}{N} \sum_{i=1}^{N} y_i \times ln(o_i) + (1 - y_i) \times ln(1 - o_i) $$

We'll make this a little nicer to look at by working with only a single input example first.

$$ C = - \left[ y \times ln(o) + (1 - y) \times ln(1 - o) \right] $$
$$ \frac{\partial C}{\partial o} = - \frac{\partial}{\partial o} \left[ y \times ln(o) + (1 - y) \times ln(1 - o) \right] $$

We can further simplify things by splitting up the derivatives and distributing the $-$.

$$ \frac{\partial C}{\partial o} = - \left[ \frac{\partial}{\partial o} y \times ln(o) \right] - \left[ \frac{\partial}{\partial o} (1 - y) \times ln(1 - o) \right]$$

The first term is
$$ \frac{\partial}{\partial o} y \times ln(o) = \frac{y}{o} $$

The second term is
$$ \frac{\partial}{\partial o} (1 - y) \times ln(1 - o) = - \frac{1-y}{1-o} $$

Therefore,

$$ \frac{\partial C}{\partial o} = - \frac{y}{o} + \frac{1-y}{1-o} = \frac{o - y}{o(1-o)} $$

We derived this by looking at a single input example, but we can "vectorize" this last equation by simply treating the subtractions as element-wise over input examples, so the value of $\frac{\partial C}{\partial o}$ will be a vector in $\mathbb{R}^N$.

## 2. $\frac{d{o}}{d{z}} = \frac{d{\sigma}}{d{z}}$
Next up is the derivative of the output $o$ with respect to the weighted input $z$, which is really just the derivative of the sigmoid function $\sigma$.

$$ o = \sigma(z) = \frac{1}{1 + e^{-z}} $$

By applying the quotient rule you should get

$$ \frac{d\sigma}{dz} = \frac{e^{-z}}{(1+e^{-z})^2} $$

But we can simplify this further using some tricky algebra. First, splitting the square into a product we get

$$ \frac{d\sigma}{dz} = \frac{1}{1+e^{-z}} \frac{e^{-z}}{1+e^{-z}} = \sigma(z) \frac{e^{-z}}{1+e^{-z}}$$

Now the fun bit.

$$
\begin{align}
\frac{d\sigma}{dz} &= \sigma(z) \frac{e^{-z}}{1+e^{-z}} \\
                   &= \sigma(z) \frac{(1+e^{-z}) -1}{1+e^{-z}} \\
                   &= \sigma(z) \left( \frac{1+e^{-z}}{1+e^{-z}} - \frac{1}{1+e^{-z}}\right) \\
                   &= \sigma(z) (1 - \sigma(z))
\end{align}
$$

## 3. $ \frac{\partial{z}}{\partial{W}} $

This one's easy. Since $z = XW$,

$$ \frac{\partial{z}}{\partial{W}} = X $$

## The Backward Pass

Let's actually compute these then.

$\frac{\partial{C}}{\partial{o}} = \frac{o - y}{o(1-o)} =
    \begin{pmatrix} \frac{0.67 - 1}{0.67(1-0.67)} \\
                    \frac{0.69 - 1}{0.69(1-0.69)} \\
                    \frac{0.60 - 0}{0.60(1-0.60)} \\
                    \frac{0.57 - 0}{0.57(1-0.57)}
    \end{pmatrix}
    =
    \begin{pmatrix} -1.50 \\ -1.45 \\ 2.49 \\ 2.35 \end{pmatrix}
$

$\frac{d{o}}{d{z}} = \frac{d{\sigma}}{d{z}} = \sigma(z)(1-\sigma(z)) =
    \begin{pmatrix}
        0.67(1-0.67) \\ 0.69(1-0.69) \\ 0.60(1-0.60) \\ 0.57(1-0.57)
    \end{pmatrix}
    =
    \begin{pmatrix} 0.22 \\ 0.21 \\ 0.24 \\ 0.24 \end{pmatrix}
$

$\frac{\partial{z}}{\partial{W}} = X = \begin{pmatrix} 1 & 3 \\ 2 & 3 \\ 2 & 1 \\ 1 & 1 \end{pmatrix}$

```julia
function ∂C_∂o(y, o)
    num = o .- y
    denom = o .* (1 .- o)
    return num ./ denom
end

function dσ_dz(z)
    o = σ(z)
    return @. o * (1.0 - o)
end

println("∂C_∂o: $(∂C_∂o(y, o))")
println("dσ_dz: $(dσ_dz(z))")
println("∂z_∂W = X = $(X)")
```

    ∂C_∂o: [-1.4965853037914094, -1.4493289641172216, 2.4918246976412703, 2.349858807576003]
    dσ_dz: [0.22171287329310904, 0.2139096965202944, 0.24026074574152914, 0.24445831169074586]
    ∂z_∂W = X = [1 3; 2 3; 2 1; 1 1]

Now the actual computation of $\frac{\partial{C}}{\partial{W}} = \frac{\partial{C}}{\partial{o}} \frac{do}{dz} \frac{\partial{z}}{\partial{W}}$ can get a little tricky if we're not careful, so let's be more obvious with our notation.

$\frac{\partial{C}}{\partial{W}} = \Big ( \frac{\partial{C}}{\partial{o}} \odot \frac{do}{dz} \Big )^T X$

where $\odot$ is elementwise multiplication, otherwise known as the *Hadamard product*.

Looking at the shapes of what we're multiplying together also helps.

$\Big ( \mathbb{R}^{4 \times 1} \odot \mathbb{R}^{4 \times 1} \Big ) = \mathbb{R}^{4 \times 1}$

${\mathbb{R}^{4 \times 1}}^T \mathbb{R}^{4 \times 2} = \mathbb{R}^{1 \times 2}$

This *almost* checks out with what we expect. Remember that the result of this operation should tell us how to update $W$, but $W \in \mathbb{R}^{2 \times 1}$, so we'll need to transpose one or the other in order to get them in the same orientation.

$\Big ( \frac{\partial{C}}{\partial{o}} \odot \frac{do}{dz} \Big )^T = \Big (
    \begin{pmatrix} -1.50 \\ -1.45 \\ 2.49 \\ 2.35 \end{pmatrix}
        \odot
    \begin{pmatrix} 0.22 \\ 0.21 \\ 0.24 \\ 0.24 \end{pmatrix} \Big )^T
    =
    \begin{pmatrix} -0.33 & -0.30 & 0.60 & 0.56 \end{pmatrix}
$

$\frac{\partial{C}}{\partial{W}} = \Big ( \frac{\partial{C}}{\partial{o}} \odot \frac{do}{dz} \Big )^T X =
    \begin{pmatrix} -0.33 & -0.30 & 0.60 & 0.56 \end{pmatrix}
    \begin{pmatrix} 1 & 3 \\ 2 & 3 \\ 2 & 1 \\ 1 & 1 \end{pmatrix}
    =
    \begin{pmatrix} 0.82 & -0.75 \end{pmatrix}
$

```julia
∂C_∂W = transpose(∂C_∂o(y, o) .* dσ_dz(z)) * X
```

    1×2 LinearAlgebra.Transpose{Float64,Array{Float64,1}}:
     0.819955  -0.752383

## Weight updates

Since our binary cross entropy error function is 2 dimensional, we can plot it, with $y$ (the gold standard) and $o$ (the predicted scores) on the xy-plane and the value of the function on the z-axis.

```julia
using Distributions

a = range(0.1, stop=0.8, length=50)
b = range(0.2, stop=0.8, length=50)
f(a, b) = binary_crossentropy.(a, b)
plot(a, b, f, st=:surface, cbar=false, seriesalpha=0, camera=(10,25))
```

If you're familiar with gradients from calculus, you may already know that $\frac{\partial{C}}{\partial{W}}$ is a vector that points in the direction of *maximum increase* of the error function $C$ with respect to $W$.

Thinking about this with respect to the image above, the gradient of our initial guess will point *up* the slope, towards the yellow. Therefore, to minimize the error and go *down* the slope towards the purple, we'll want to move opposite the gradient (i.e. $-\frac{\partial{C}}{\partial{W}}$).

This insight leads to our first attempt at an update equation for $W$

$W^{(t+1)} \leftarrow W^{(t)} - \frac{\partial{C}}{\partial{W}}$

where $W^{(t)}$ represents the weight vector at training step $t$.

Iteratively computing the error of our model and then computing the weight updates given the error is the essence of gradient descent. Let's put this all together in a simple `for` loop to get started.

```julia
W_t = copy(W)
for t=1:10
    # Forward pass
    z = X * W_t
    o = σ(z)
    loss = binary_crossentropy(y, o)
    println("Loss at training step $t: $loss")
    # Backward pass and weight update
    ∂C_∂W = transpose(∂C_∂o(y, o) .* dσ_dz(z)) * X
    W_t = W_t .- transpose(∂C_∂W)
end
```

    Loss at training step 1: 0.6354143029254289
    Loss at training step 2: 0.4058656742028299
    Loss at training step 3: 0.41868591352928425
    Loss at training step 4: 1.0067750278046674
    Loss at training step 5: 0.9049607854375845
    Loss at training step 6: 1.811648377225643
    Loss at training step 7: 0.1410851517809659
    Loss at training step 8: 0.13829190891212276
    Loss at training step 9: 0.13561251612794425
    Loss at training step 10: 0.13306864098930912

### Debugging

What's wrong here? The learning rate initially decreases from step 1 to 2, but then oscillates a few times before settling down. If the negative gradient points towards the minimum of the loss function, why would the loss increase?

It turns out that if the magnitude of the gradient is too large, it will overshoot the minimum and shoot up the other side. A too-large gradient can be cause by (primarily) two things:

 1. The computation of $\frac{\partial C}{\partial W}$ accumulates the gradient of all the input examples. Therefore, the magnitude will increase as the number of training examples increases.

 2. The error surface changes rapidly around the current point. For example, perhaps the slope of the error surface is steep at the current point, levels off suddenly just below it at a local minimum, and then increases steeply again just past the minimum.

In both cases, simply subtracting the gradient vector from the weights could move the model parameters too far, entirely skipping over a minima, and can even result in gradient descent getting stuck oscillating between two non-optimal points.

### The Learning Rate $\eta$

To combat this, we can scale the gradient by a parameter $\eta \in \mathbb{R}$, commonly called the *learning rate*, as it's magnitude determines how slowly or quickly gradient descent travels across the error surface. Our new update is then,

$W^{(t+1)} \leftarrow W^{(t)} - \eta \frac{\partial{C}}{\partial{W}}$

In practice, $\eta$ is often set in the range $[1e^{-5}, 1e^{-1}]$. Here we'll use $\eta = 0.5$ just to get things moving on this toy problem.

```julia
η = 0.5
W_t = copy(W)
for t=1:10
    z = X * W_t
    o = σ(z)
    loss = binary_crossentropy(y, o)
    println("Loss at training step $t: $loss")
    ∂C_∂W = transpose(∂C_∂o(y, o) .* dσ_dz(z)) * X
    # Scale the gradient by the learning rate.
    W_t = W_t .- (η .* transpose(∂C_∂W))
end
```

    Loss at training step 1: 0.6354143029254289
    Loss at training step 2: 0.5021368690276575
    Loss at training step 3: 0.42489287091790723
    Loss at training step 4: 0.38001125080664333
    Loss at training step 5: 0.3484655288292048
    Loss at training step 6: 0.33106153286204654
    Loss at training step 7: 0.29877480501381526
    Loss at training step 8: 0.2826410701985972
    Loss at training step 9: 0.2593564746815564
    Loss at training step 10: 0.2467018688501049

That looks better! Let's run it for more iterations to see where it ends up.

```julia
η = 0.5
W_t = copy(W)
for t=1:50
    z = X * W_t
    o = σ(z)
    if t == 1 || t % 5 == 0
        loss = binary_crossentropy(y, o)
        acc = accuracy(y, decision(o))
        println("Training step $t: Loss: $loss, Accuracy: $acc")
    end
    ∂C_∂W = transpose(∂C_∂o(y, o) .* dσ_dz(z)) * X
    W_t = W_t .- (η .* transpose(∂C_∂W))
end
```

    Training step 1: Loss: 0.6354143029254289, Accuracy: 0.5
    Training step 5: Loss: 0.3484655288292048, Accuracy: 1.0
    Training step 10: Loss: 0.2467018688501049, Accuracy: 1.0
    Training step 15: Loss: 0.20842413699493623, Accuracy: 1.0
    Training step 20: Loss: 0.18733695410574, Accuracy: 1.0
    Training step 25: Loss: 0.17237567448371208, Accuracy: 1.0
    Training step 30: Loss: 0.16079325562841384, Accuracy: 1.0
    Training step 35: Loss: 0.15132582953245927, Accuracy: 1.0
    Training step 40: Loss: 0.14330143874177748, Accuracy: 1.0
    Training step 45: Loss: 0.13632537461110222, Accuracy: 1.0
    Training step 50: Loss: 0.13014797436551886, Accuracy: 1.0

I'd call that solved!

---

## A non-linear problem

So we've defined a linear model (i.e. a model with no hidden layers) to solve a very straightforward classification task. But what if our task isn't linearly separable? Let's take the *exclusive or (XOR)* function, for example.

$XOR(a, b) = \begin{cases}
    1, & \text{if}~~~ a \neq b \\
    0, & \text{otherwise}
\end{cases}$

```julia
X = [0 0; 1 1; 0 1; 1 0]
y = [0, 0, 1, 1]
using Plots; pyplot()
scatter(X[:, 1], X[:, 2], color=[:blue, :blue, :red, :red], markersize=10, legend=false)
```

There's no way to draw a line that perfectly separates the blue points from the red points, so we can be pretty sure that our linear model isn't going to do well here.

Let's try it anyway, just to see what happens.

```julia
η = 1
W_t = copy(W)
for t=1:100
    # Forward pass
    z = X * W_t
    o = σ(z)
    if t == 1 || t % 10 == 0
        loss = binary_crossentropy(y, o)
        acc = accuracy(y, decision(o))
        println("Training step $t: Loss: $loss, Accuracy: $acc")
    end
    # Backward pass and weight updates
    ∂C_∂W = transpose(∂C_∂o(y, o) .* dσ_dz(z)) * X
    W_t = W_t .- (η .* transpose(∂C_∂W))
end
```

    Training step 1: Loss: 0.6975094886209088, Accuracy: 0.5
    Training step 10: Loss: 0.6931480652512929, Accuracy: 0.25
    Training step 20: Loss: 0.6931471833654955, Accuracy: 0.25
    Training step 30: Loss: 0.6931471805688423, Accuracy: 0.25
    Training step 40: Loss: 0.6931471805599736, Accuracy: 0.25
    Training step 50: Loss: 0.6931471805599454, Accuracy: 0.25
    Training step 60: Loss: 0.6931471805599453, Accuracy: 0.25
    Training step 70: Loss: 0.6931471805599453, Accuracy: 0.25
    Training step 80: Loss: 0.6931471805599453, Accuracy: 0.25
    Training step 90: Loss: 0.6931471805599453, Accuracy: 0.25
    Training step 100: Loss: 0.6931471805599453, Accuracy: 0.25

The loss starts out decreasing (albeit slowly) but then eventually stagnates after training step 50, and accuracy never improves.

In order to solve this problem, we'll have to introduce *nonlinearities* into our model. Recall that a single matrix multiplication is equivalent to a [linear transformation](https://www.freetext.org/Introduction_to_Linear_Algebra/Linear_Transformations/Matrix_Multiplication/), so chaining multiple matrix multiplications together will comrpise a nonlinear transformation. The model we define here is a linear model because it only uses a single matrix multiplication ($XW$), so to make it nonlinear, we can just add another weight matrix, which is equivalent to adding another hidden layer. Backprop generalizes nicely to multiple hidden layers, but we'll need to make some changes to our training loop. See the next notebook for details.