# Tutorial 2: Principal Component Analysis
**Week 1, Day 5: Dimensionality Reduction**

**By Neuromatch Academy**

__Content creators:__ Alex Cayco Gajic, John Murray

__Content reviewers:__ Roozbeh Farhoudi, Matt Krause, Spiros Chavlis, Richard Gao, Michael Waskom, Siddharth Suresh, Natalie Schaworonkow, Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

*Estimated timing of tutorial: 45 minutes*

In this notebook we'll learn how to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

Overview:
- Calculate the eigenvectors of the sample covariance matrix.
- Perform PCA by projecting data onto the eigenvectors of the covariance matrix.
- Plot and explore the eigenvalues.

To quickly refresh your knowledge of eigenvalues and eigenvectors, you can watch this [short video](https://www.youtube.com/watch?v=kwA3qM0rm7c) (4 minutes) for a geometrical explanation. For a deeper understanding, this [in-depth video](https://www.youtube.com/watch?v=PFDu9oVAE-g&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab&index=14) (17 minutes) provides an excellent basis and is beautifully illustrated.

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/kaq2x/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

```python
# @title Video 1: PCA
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1hK411H7Zi", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="-f6T9--oM0E", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

---
# Setup

```python
# Imports

import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import ipywidgets as widgets  # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Plotting Functions

def plot_eigenvalues(evals):
  """
  Plots eigenvalues.

  Args:
      (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """
  plt.figure(figsize=(4, 4))
  plt.plot(np.arange(1, len(evals) + 1), evals, 'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
  plt.xticks(np.arange(1, len(evals) + 1))
  plt.ylim([0, 2.5])

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  scatter plot of their joint activity. The title indicates the sample
  correlation calculated from the data.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(X[:, 0], color='k')
  plt.ylabel('Neuron 1')
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(X[:, 1], color='k')
  plt.xlabel('Sample Number (sorted)')
  plt.ylabel('Neuron 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:, 0], X[:, 1], '.', markerfacecolor=[.5, .5, .5],
           markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:, 0], X[:, 1])[0, 1]))
  plt.show()

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases. Similar to plot_data
  but with colors corresponding to projections onto basis 1 (red) and
  basis 2 (blue).
  The title indicates the sample correlation calculated from the data.

  Note that samples are re-sorted in ascending order for the first random
  variable.

  Args:
    Y (numpy array of floats) : Data matrix in new basis each column
                                corresponds to a different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(Y[:, 0], 'r')
  plt.ylabel('Projection \n basis vector 1')
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(Y[:, 1], 'b')
  plt.xlabel('Sample number')
  plt.ylabel('Projection \n basis vector 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:, 0], Y[:, 1], '.', color=[.5, .5, .5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:, 0], Y[:, 1])[0, 1]))
  plt.show()

def plot_basis_vectors(X, W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable
    W (numpy array of floats) : Square matrix representing new orthonormal
                                basis each column represents a basis vector

  Returns:
    Nothing.
  """

  plt.figure(figsize=[4, 4])
  plt.plot(X[:, 0], X[:, 1], '.', color=[.5, .5, .5], label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
           label='Basis vector 1')
  plt.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
           label='Basis vector 2')
  plt.legend()
  plt.show()
```

```python
# @title Helper functions

def sort_evals_descending(evals, evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats)    : Vector of eigenvalues
    evectors (numpy array of floats) : Corresponding matrix of eigenvectors
                                        each column corresponds to a different
                                        eigenvalue

  Returns:
    (numpy array of floats)          : Vector of eigenvalues after sorting
    (numpy array of floats)          : Matrix of eigenvectors after sorting
  """

  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:, index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:, 0],
                           1 / np.sqrt(2) * np.array([1, 1]))) > np.pi / 2:
      evectors[:, 0] = -evectors[:, 0]
    if np.arccos(np.matmul(evectors[:, 1],
                           1 / np.sqrt(2) * np.array([-1, 1]))) > np.pi / 2:
      evectors[:, 1] = -evectors[:, 1]
  return evals, evectors

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

  Note that samples are sorted in ascending order for the first random
  variable.

  Args:
    var_1 (scalar)                     : variance of the first random variable
    var_2 (scalar)                     : variance of the second random variable
    cov_matrix (numpy array of floats) : desired covariance matrix

  Returns:
    (numpy array of floats)            : samples from the bivariate Gaussian,
                                          with each column corresponding to a
                                          different random variable
  """

  mean = np.array([0, 0])
  X = np.random.multivariate_normal(mean, cov_matrix, size=1000)
  indices_for_sorting = np.argsort(X[:, 0])
  X = X[indices_for_sorting, :]
  return X

def calculate_cov_matrix(var_1, var_2, corr_coef):
  """
  Calculates the covariance matrix based on the variances and
  correlation coefficient.

  Args:
    var_1 (scalar)         :  variance of the first random variable
    var_2 (scalar)         :  variance of the second random variable
    corr_coef (scalar)     :  correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  cov = corr_coef * np.sqrt(var_1 * var_2)
  cov_matrix = np.array([[var_1, cov], [cov, var_2]])
  return cov_matrix

def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats) : arbitrary 2D vector used for new basis

  Returns:
    (numpy array of floats)   : new orthonormal basis columns correspond to
                                basis vectors
  """

  u = u / np.sqrt(u[0] ** 2 + u[1] ** 2)
  w = np.array([-u[1], u[0]])
  W = np.column_stack((u, w))
  return W

def change_of_basis(X, W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)   : Data matrix expressed in new basis
  """

  Y = np.matmul(X, W)
  return Y
```

---
# Section 1: Calculate the eigenvectors of the the sample covariance matrix

As we saw in the lecture, PCA represents data in a new orthonormal basis defined by the eigenvectors of the covariance matrix. Remember that in the previous tutorial, we generated bivariate normal data with a specified covariance matrix $\bf \Sigma$, whose $(i,j)$th element is:
\begin{align}
\Sigma_{ij} = \mathbb{E}[ x_i x_j ] - \mathbb{E}[ x_i] \mathbb{E}[ x_j ] .
\end{align}
However, in real life we don't have access to this ground-truth covariance matrix. To get around this, we can use the sample covariance matrix, $\bf\hat\Sigma$, which is calculated directly from the data. The $(i,j)$th element of the sample covariance matrix is:
\begin{align}
 \hat \Sigma_{ij} =  \frac{1}{N_\text{samples}}{\bf x}_i^T {\bf x}_j - \bar {\bf x}_i \bar{\bf x}_j ,
\end{align}
where ${\bf x}_i = [ x_i(1), x_i(2), \dots,x_i(N_\text{samples})]^T$ is a column vector representing all measurements of neuron $i$, and  $\bar {\bf x}_i$ is the mean of neuron $i$ across samples:
\begin{align}
\bar {\bf x}_i = \frac{1}{N_\text{samples}} \sum_{k=1}^{N_\text{samples}} x_i(k).
\end{align}
If we assume that the data has already been mean-subtracted, then we can write the sample covariance matrix in a much simpler matrix form:
\begin{align}
{\bf \hat \Sigma}
&= \frac{1}{N_\text{samples}} {\bf X}^T {\bf X}.
\end{align}

where $\bf X$ is the full data matrix (each column of $\bf X$ is a different $\bf x_i$).

## Coding Exercise 1.1: Calculate the covariance matrix

Before calculating the eigenvectors, you must first calculate the sample covariance matrix.

**Steps**
* Complete the function `get_sample_cov_matrix` by first subtracting the sample mean of the data, then calculate $\bf \hat \Sigma$ using the equation above.
* Use `get_data` to generate bivariate normal data, and calculate the sample covariance matrix with your finished `get_sample_cov_matrix`. Compare this estimate to the true covariate matrix using `calculate_cov_matrix`. You've seen both `get_data` and `calculate_cov_matrix` in Tutorial 1.

```python
help(get_data)
help(calculate_cov_matrix)
```

    Help on function get_data in module __main__:

    get_data(cov_matrix)
        Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

        Note that samples are sorted in ascending order for the first random
        variable.

        Args:
          var_1 (scalar)                     : variance of the first random variable
          var_2 (scalar)                     : variance of the second random variable
          cov_matrix (numpy array of floats) : desired covariance matrix

        Returns:
          (numpy array of floats)            : samples from the bivariate Gaussian,
                                                with each column corresponding to a
                                                different random variable

    Help on function calculate_cov_matrix in module __main__:

    calculate_cov_matrix(var_1, var_2, corr_coef)
        Calculates the covariance matrix based on the variances and
        correlation coefficient.

        Args:
          var_1 (scalar)         :  variance of the first random variable
          var_2 (scalar)         :  variance of the second random variable
          corr_coef (scalar)     :  correlation coefficient

        Returns:
          (numpy array of floats) : covariance matrix

```python
def get_sample_cov_matrix(X):
  """
  Returns the sample covariance matrix of data X

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Covariance matrix
  """

  #################################################
  ## TODO for students: calculate the covariance matrix
  # Fill out function and remove
  #raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  #################################################

  # Subtract the mean of X
  X = X - np.mean(X, 0)

  # Calculate the covariance matrix (hint: use np.matmul)
  cov_matrix = (1 / X.shape[0]) * np.matmul(X.T, X)

  return cov_matrix

# Set parameters
np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

# Calculate covariance matrix
cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
print(cov_matrix)

# Generate data with that covariance matrix
X = get_data(cov_matrix)

# Get sample covariance matrix
sample_cov_matrix = get_sample_cov_matrix(X)
print(sample_cov_matrix)
```

    [[1.  0.8]
     [0.8 1. ]]
    [[0.99315313 0.82347589]
     [0.82347589 1.01281397]]

```python
# to_remove solution

def get_sample_cov_matrix(X):
  """
  Returns the sample covariance matrix of data X

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Covariance matrix
  """

  # Subtract the mean of X
  X = X - np.mean(X, 0)

  # Calculate the covariance matrix (hint: use np.matmul)
  cov_matrix = cov_matrix = 1 / X.shape[0] * np.matmul(X.T, X)

  return cov_matrix

# Set parameters
np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

# Calculate covariance matrix
cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
print(cov_matrix)

# Generate data with that covariance matrix
X = get_data(cov_matrix)

# Get sample covariance matrix
sample_cov_matrix = get_sample_cov_matrix(X)
print(sample_cov_matrix)
```

    [[1.  0.8]
     [0.8 1. ]]
    [[0.99315313 0.82347589]
     [0.82347589 1.01281397]]

SAMPLE OUTPUT
```
[[1.  0.8]
 [0.8 1. ]]
[[0.99315313 0.82347589]
 [0.82347589 1.01281397]]
```

We get a sample covariance matrix that doesn't look too far off from the true covariance matrix, so that's a good sign!

## Coding Exercise 1.2: Eigenvectors of the covariance matrix

Next you will calculate the eigenvectors of the covariance matrix. Plot them along with the data to check that they align with the geometry of the data.

**Steps:**
* Calculate the eigenvalues and eigenvectors of the sample covariance matrix. (**Hint:** use `np.linalg.eigh`, which finds the eigenvalues of a symmetric matrix).
* Use the provided code to sort the eigenvalues in descending order.
* Plot the eigenvectors on a scatter plot of the data, using the function `plot_basis_vectors`.

```python
help(sort_evals_descending)
```

    Help on function sort_evals_descending in module __main__:

    sort_evals_descending(evals, evectors)
        Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
        eigenvectors to be in first two quadrants (if 2D).

        Args:
          evals (numpy array of floats)    : Vector of eigenvalues
          evectors (numpy array of floats) : Corresponding matrix of eigenvectors
                                              each column corresponds to a different
                                              eigenvalue

        Returns:
          (numpy array of floats)          : Vector of eigenvalues after sorting
          (numpy array of floats)          : Matrix of eigenvectors after sorting

```python
#################################################
## TODO for students
# Fill out function and remove
#raise NotImplementedError("Student excercise: calculate and sort eigenvalues")
#################################################

# Calculate the eigenvalues and eigenvectors
evals, evectors = np.linalg.eigh(cov_matrix)

# Sort the eigenvalues in descending order
evals, evectors = sort_evals_descending(evals, evectors)

# Visualize
plot_basis_vectors(X, evectors)
```

```python
# to_remove solution

# Calculate the eigenvalues and eigenvectors
evals, evectors = np.linalg.eigh(cov_matrix)

# Sort the eigenvalues in descending order
evals, evectors = sort_evals_descending(evals, evectors)

# Visualize
with plt.xkcd():
  plot_basis_vectors(X, evectors)
```

---
# Section 2: Perform PCA by projecting data onto the eigenvectors

*Estimated timing to here from start of tutorial: 25 min*

To perform PCA, we will project the data onto the eigenvectors of the covariance matrix, i.e.:
\begin{align}
\bf S = X W
\end{align}
where $\bf S$ is an $N_\text{samples} \times N$ matrix representing the projected data (also called *scores*), and $\bf W$ is an $N\times N$ orthogonal matrix, each of whose columns represents the eigenvectors of the covariance matrix (also called *weights* or *loadings*).

## Coding Exercise 2: PCA implementation

You will now perform PCA on the data using the intuition and functions you have developed so far. Fill in the function below to carry out the steps to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

**Steps:**
* First subtract the mean and calculate the sample covariance matrix.
* Then find the eigenvalues and eigenvectors and sort them in descending order.
* Finally project the mean-centered data onto the eigenvectors.

```python
help(change_of_basis)
```

    Help on function change_of_basis in module __main__:

    change_of_basis(X, W)
        Projects data onto a new basis.

        Args:
          X (numpy array of floats) : Data matrix each column corresponding to a
                                      different random variable
          W (numpy array of floats) : new orthonormal basis columns correspond to
                                      basis vectors

        Returns:
          (numpy array of floats)   : Data matrix expressed in new basis

```python
def pca(X):
  """
  Sorts eigenvalues and eigenvectors in decreasing order.

  Args:
    X (numpy array of floats): Data matrix each column corresponds to a
                               different random variable

  Returns:
    (numpy array of floats)  : Data projected onto the new basis
    (numpy array of floats)  : Vector of eigenvalues
    (numpy array of floats)  : Corresponding matrix of eigenvectors

  """

  #################################################
  ## TODO for students: calculate the covariance matrix
  # Fill out function and remove
  #raise NotImplementedError("Student excercise: sort eigenvalues/eigenvectors!")
  #################################################

  # Calculate the sample covariance matrix
  cov_matrix = get_sample_cov_matrix(X)

  # Calculate the eigenvalues and eigenvectors
  evals, evectors = np.linalg.eigh(cov_matrix)

  # Sort the eigenvalues in descending order
  evals, evectors = sort_evals_descending(evals, evectors)

  # Project the data onto the new eigenvector basis
  score = change_of_basis(X, np.column_stack(evectors[::-1]))

  return score, evectors, evals

# Perform PCA on the data matrix X
score, evectors, evals = pca(X)

# Plot the data projected into the new basis
plot_data_new_basis(score)
```

```python
# to_remove solution
def pca(X):
  """
  Performs PCA on multivariate data.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Data projected onto the new basis
    (numpy array of floats)   : Vector of eigenvalues
    (numpy array of floats)   : Corresponding matrix of eigenvectors

  """

  # Calculate the sample covariance matrix
  cov_matrix = get_sample_cov_matrix(X)

  # Calculate the eigenvalues and eigenvectors
  evals, evectors = np.linalg.eigh(cov_matrix)

  # Sort the eigenvalues in descending order
  evals, evectors = sort_evals_descending(evals, evectors)

  # Project the data onto the new eigenvector basis
  score = change_of_basis(X, evectors)

  return score, evectors, evals

# Perform PCA on the data matrix X
score, evectors, evals = pca(X)

# Plot the data projected into the new basis
with plt.xkcd():
  plot_data_new_basis(score)
```

   Finally, we will examine the eigenvalues of the covariance matrix. Remember that each eigenvalue describes the variance of the data projected onto its corresponding eigenvector. This is an important concept because it allows us to rank the PCA basis vectors based on how much variance each one can capture. First run the code below to plot the eigenvalues (sometimes called the "scree plot"). Which eigenvalue is larger?

```python
plot_eigenvalues(evals)
```

## Interactive Demo 2: Exploration of the correlation coefficient

Run the following cell and use the slider to change the correlation coefficient in the data. You should see the scree plot and the plot of basis vectors update.

1.  What happens to the eigenvalues as you change the correlation coefficient?
2.  Can you find a value for which both eigenvalues are equal?
3.  Can you find a value for which only one eigenvalue is nonzero?

```python
# @markdown Make sure you execute this cell to enable the widget!

def refresh(corr_coef=.8):
  cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
  X = get_data(cov_matrix)
  score, evectors, evals = pca(X)
  plot_eigenvalues(evals)
  plot_basis_vectors(X, evectors)

_ = widgets.interact(refresh, corr_coef=(-1, 1, .1))
```

    interactive(children=(FloatSlider(value=0.8, description='corr_coef', max=1.0, min=-1.0), Output()), _dom_clas…

```python
# to_remove explanation

"""

1) As the absolute value of the correlation coefficient decreases, the two eigenvalues
become more similar (closer in value)

2) The two eigenvalues are extremely similar (basically equal) with a correlation coefficient of 0.

3) A correlation coefficient of 1 or -1 results in only one non-zero eigenvalues.

"""
```

---
# Summary

*Estimated timing of tutorial: 45 minutes*

- In this tutorial, we learned that goal of PCA is to find an orthonormal basis capturing the directions of maximum variance of the data. More precisely, the $i$th basis vector is the direction that maximizes the projected variance, while being orthogonal to all previous basis vectors. Mathematically, these basis vectors are the eigenvectors of the covariance matrix (also called *loadings*).
- PCA also has the useful property that the projected data (*scores*) are uncorrelated.
- The projected variance along each basis vector is given by its corresponding eigenvalue. This is important because it allows us rank the "importance" of each basis vector in terms of how much of the data variability it explains. An eigenvalue of zero means there is no variation along that direction so it can be dropped without losing any information about the original data.
- In the next tutorial, we will use this property to reduce the dimensionality of high-dimensional data.

---
# Notation

\begin{align}
\mathbf{x_i}  &\quad \text{all measurements of neuron } i\\
\bar{\bf x_i}  &\quad \text{mean across samples for neuron } i \\
\bf \Sigma &\quad \text{covariance matrix}\\
\bf \hat \Sigma &\quad \text{sample covariance matrix}\\
\bf W &\quad \text{weights, loadings matrix}\\
{\bf X} &\quad \text{original data matrix}\\
\bf S &\quad \text{projected matrix, scores}\\
N &\quad \text{data dimensionality}\\
N_\text{samples} &\quad \text{number of samples}\\
\end{align}

---
# Bonus

---
## Bonus Section 1: Mathematical basis of PCA properties

```python
# @title Video 2: Properties of PCA
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1xa4y1a77c", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="p56UrMRt6-U", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

```python

```


# Perceptron Multicouche

Auteur: Jérémie Decock

[](https://mybinder.org/v2/gh/jdhp-docs/notebooks/master?urlpath=apps%2Fnb_sci_ai%2Fai_ml_multilayer_perceptron_fr.ipynb)

```python
%matplotlib inline

#import nnfigs

# https://github.com/jeremiedecock/neural-network-figures.git
import nnfigs.core as nnfig
import matplotlib.pyplot as plt
import numpy as np

import ipywidgets
from ipywidgets import interact

# TODO

# Shallow et Deep learning à lire:
# - https://www.miximum.fr/blog/introduction-au-deep-learning-2/
# - https://sciencetonnante.wordpress.com/2016/04/08/le-deep-learning/
# - https://www.technologies-ebusiness.com/enjeux-et-tendances/le-deep-learning-pas-a-pas
# - http://scholar.google.fr/scholar_url?url=https://arxiv.org/pdf/1404.7828&hl=fr&sa=X&scisig=AAGBfm07Y2UDlPpbninerh4gxHUj2SJfDQ&nossl=1&oi=scholarr&sqi=2&ved=0ahUKEwjfxMu7jKnUAhUoCsAKHR_RDlkQgAMIKygAMAA
```

$
\newcommand{\cur}{i}
\newcommand{\prev}{j}
\newcommand{\prevcur}{{\cur\prev}}
\newcommand{\next}{k}
\newcommand{\curnext}{{\next\cur}}
\newcommand{\ex}{\eta}
\newcommand{\pot}{\rho}
\newcommand{\feature}{x}
\newcommand{\weight}{w}
\newcommand{\wcur}{{\weight_{\cur\prev}}}
\newcommand{\activthres}{\theta}
\newcommand{\activfunc}{f}
\newcommand{\errfunc}{E}
\newcommand{\learnrate}{\epsilon}
\newcommand{\learnit}{n}
\newcommand{\sigout}{y}
\newcommand{\sigoutdes}{d}
\newcommand{\weights}{\boldsymbol{W}}
\newcommand{\errsig}{\Delta}
$

```python
# Notations :
# - $\cur$: couche courante
# - $\prev$: couche immédiatement en amont de la courche courrante (i.e. vers la couche d'entrée du réseau)
# - $\next$: couche immédiatement en aval de la courche courrante (i.e. vers la couche de sortie du réseau)
# - $\ex$: exemple (*sample* ou *feature*) courant (i.e. le vecteur des entrées courantes du réseau)
# - $\pot_\cur$: *Potentiel d'activation* du neurone $i$ pour l'exemple courant
# - $\wcur$: Poids de la connexion entre le neurone $j$ et le neurone $i$
# - $\activthres_\cur$: *Seuil d'activation* du neurone $i$
# - $\activfunc_\cur$: *Fonction d'activation* du neurone $i$
# - $\errfunc$: *Fonction objectif* ou *fonction d'erreur*
# - $\learnrate$: *Pas d'apprentissage* ou *Taux d'apprentissage*
# - $\learnit$: Numéro d'itération (ou cycle ou époque) du processus d'apprentissage
# - $\sigout_\cur$: Signal de sortie du neurone $i$ pour l'exemple courant
# - $\sigoutdes_\cur$: Sortie désirée (*étiquette*) du neurone $i$ pour l'exemple courant
# - $\weights$: Matrice des poids du réseau (en réalité il y a une matrice de taille potentiellement différente par couche)
# - $\errsig_i$: *Signal d'erreur* du neurone $i$ pour l'exemple courant
```

## Introduction

### Qu'est-ce qu'un réseau de neurones ?

Une grosse fonction parametrique.
Pour peu qu'on donne suffisamment de paramètres à cette fonction, elle est capable d'approximer n'importe quelle fonction continue.

Représentation schématique d'une fonction paramètrique avec 3 paramètres avec une entrée en une sortie à 1 dimension

$$\mathbb{R} \rightarrow \mathbb{R}$$

$$x \mapsto g_{\boldsymbol{\omega}}(x)$$

TODO: image/schéma intuition : entrés -> fonction avec paramètres = table de mixage -> sortie

### À quoi ça sert ?

TODO : expliquer la régression et la classification

TODO : applications avec références...
Exemples d'application concrètes :
- Reconnaissance de texte manuscrit
- Reconnaissance de formes, d'objets, de visages, etc. dans des images
- Reconnaissance de la parole
- Prédiction de séries temporelles (cours de la bourse, etc.)
- etc.

## Définition du neurone "formel"

```python
STR_CUR = r"i"       # Couche courante
STR_PREV = r"j"      # Couche immédiatement en amont de la courche courrante (i.e. vers la couche d'entrée du réseau)
STR_NEXT = r"k"      # Couche immédiatement en aval de la courche courrante (i.e. vers la couche de sortie du réseau)
STR_EX = r"\eta"     # Exemple (*sample* ou *feature*) courant (i.e. le vecteur des entrées courantes du réseau)
STR_POT = r"x"       # *Potentiel d'activation* du neurone $i$ pour l'exemple $\ex$
STR_POT_CUR = r"x_i"       # *Potentiel d'activation* du neurone $i$ pour l'exemple $\ex$
STR_WEIGHT = r"w"
STR_WEIGHT_CUR = r"w_{ij}"  # Poids de la connexion entre le neurone $j$ et le neurone $i$
STR_ACTIVTHRES = r"\theta"  # *Seuil d'activation* du neurone $i$
STR_ACTIVFUNC = r"f"        # *Fonction d'activation* du neurone $i$
STR_ERRFUNC = r"E"          # *Fonction objectif* ou *fonction d'erreur*
STR_LEARNRATE = r"\epsilon" # *Pas d'apprentissage* ou *Taux d'apprentissage*
STR_LEARNIT = r"n"          # Numéro d'itération (ou cycle ou époque) du processus d'apprentissage
STR_SIGIN = r"x"            # Signal de sortie du neurone $i$ pour l'exemple $\ex$
STR_SIGOUT = r"y"           # Signal de sortie du neurone $i$ pour l'exemple $\ex$
STR_SIGOUT_CUR = r"y_i"
STR_SIGOUT_PREV = r"y_j"
STR_SIGOUT_DES = r"d"           # Sortie désirée (*étiquette*) du neurone $i$ pour l'exemple $\ex$
STR_SIGOUT_DES_CUR = r"d_i"
STR_WEIGHTS = r"W"              # Matrice des poids du réseau (en réalité il y a une matrice de taille potentiellement différente par couche)
STR_ERRSIG = r"\Delta"          # *Signal d'erreur* du neurone $i$ pour l'exemple $\ex$

def tex(tex_str):
    return r"$" + tex_str + r"$"
```

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=4)

nnfig.draw_synapse(ax, (0, -6), (10, 0))
nnfig.draw_synapse(ax, (0, -2), (10, 0))
nnfig.draw_synapse(ax, (0, 2),  (10, 0))
nnfig.draw_synapse(ax, (0, 6),  (10, 0), label=tex(STR_WEIGHT_CUR), label_position=0.5, fontsize=14)

nnfig.draw_synapse(ax, (10, 0), (12, 0))

nnfig.draw_neuron(ax, (0, -6), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)
nnfig.draw_neuron(ax, (0, 6),  0.5, empty=True)
plt.text(x=0,    y=7.5,  s=tex(STR_PREV),        fontsize=14)
plt.text(x=10,   y=1.5,  s=tex(STR_CUR),         fontsize=14)
plt.text(x=0,    y=0,    s=r"$\vdots$",          fontsize=14)
plt.text(x=-2.5, y=0,    s=tex(STR_SIGOUT_PREV), fontsize=14)
plt.text(x=13,   y=0,    s=tex(STR_SIGOUT_CUR),  fontsize=14)
plt.text(x=9.2,  y=-1.8, s=tex(STR_POT_CUR),     fontsize=14)

nnfig.draw_neuron(ax, (10, 0), 1, ag_func="sum", tr_func="sigmoid")

plt.show()
```

$$
\sigout = \activfunc \left( \sum_i \weight_i \feature_i \right)
$$

$$
\pot_\cur = \sum_\prev \wcur \sigout_{\prev}
$$

$$
\sigout_{\cur} = \activfunc(\pot_\cur)
$$

$$
\weights = \begin{pmatrix}
    \weight_{11} & \cdots & \weight_{1m} \\
    \vdots       & \ddots & \vdots       \\
    \weight_{n1} & \cdots & \weight_{nm}
\end{pmatrix}
$$

Avec :
- $\cur$: couche courante
- $\prev$: couche immédiatement en amont de la courche courrante (i.e. vers la couche d'entrée du réseau)
- $\next$: couche immédiatement en aval de la courche courrante (i.e. vers la couche de sortie du réseau)
- $\ex$: exemple (*sample* ou *feature*) courant (i.e. le vecteur des entrées courantes du réseau)
- $\pot_\cur$: *Potentiel d'activation* du neurone $i$ pour l'exemple courant
- $\wcur$: Poids de la connexion entre le neurone $j$ et le neurone $i$
- $\activthres_\cur$: *Seuil d'activation* du neurone $i$
- $\activfunc_\cur$: *Fonction d'activation* du neurone $i$
- $\errfunc$: *Fonction objectif* ou *fonction d'erreur*
- $\learnrate$: *Pas d'apprentissage* ou *Taux d'apprentissage*
- $\learnit$: Numéro d'itération (ou cycle ou époque) du processus d'apprentissage
- $\sigout_\cur$: Signal de sortie du neurone $i$ pour l'exemple courant
- $\sigoutdes_\cur$: Sortie désirée (*étiquette*) du neurone $i$ pour l'exemple courant
- $\weights$: Matrice des poids du réseau (en réalité il y a une matrice de taille potentiellement différente par couche)
- $\errsig_i$: *Signal d'erreur* du neurone $i$ pour l'exemple courant

### Fonction d'activation

#### Fonction sigmoïde

La fonction sigmoïde (en forme de "S") est définie par :

$$f(x) = \frac{1}{1 + e^{-x}}$$

pour tout réel $x$.

On peut la généraliser à toute fonction dont l'expression est :

$$f(x) = \frac{1}{1 + e^{-\lambda x}}$$

```python
def sigmoid(x, _lambda=1.):
    y = 1. / (1. + np.exp(-_lambda * x))
    return y
```

```python
%matplotlib inline

x = np.linspace(-5, 5, 300)

y1 = sigmoid(x, 1.)
y2 = sigmoid(x, 5.)
y3 = sigmoid(x, 0.5)

plt.plot(x, y1, label=r"$\lambda=1$")
plt.plot(x, y2, label=r"$\lambda=5$")
plt.plot(x, y3, label=r"$\lambda=0.5$")

plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')
plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')

plt.legend()

plt.title("Fonction sigmoïde")
plt.axis([-5, 5, -0.5, 2]);
```

#### Tangente hyperbolique

```python
def tanh(x):
    y = np.tanh(x)
    return y
```

```python
x = np.linspace(-5, 5, 300)
y = tanh(x)

plt.plot(x, y)

plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')
plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')

plt.title("Fonction tangente hyperbolique")
plt.axis([-5, 5, -2, 2]);
```

#### Fonction logistique

Fonctions ayant pour expression

$$
f(t) = K \frac{1}{1+ae^{-\lambda t}}
$$

où $K$ et $\lambda$ sont des réels positifs et $a$ un réel quelconque.

Les fonctions sigmoïdes sont un cas particulier de fonctions logistique avec $a > 0$.

```python
def logistique(x, a=1., k=1., _lambda=1.):
    y = k / (1. + a * np.exp(-_lambda * x))
    return y
```

```python
%matplotlib inline

x = np.linspace(-5, 5, 300)

y1 = logistique(x, a=1.)
y2 = logistique(x, a=2.)
y3 = logistique(x, a=0.5)

plt.plot(x, y1, label=r"$a=1$")
plt.plot(x, y2, label=r"$a=2$")
plt.plot(x, y3, label=r"$a=0.5$")

plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')
plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')

plt.legend()

plt.title("Fonction logistique")
plt.axis([-5, 5, -0.5, 2]);
```

### Le terme de biais

TODO

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=6)

HSPACE = 6
VSPACE = 4

# Synapse #####################################

#nnfig.draw_synapse(ax, (0,2*VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_0"), label_position=0.3)
nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_1"), label_position=0.3)
nnfig.draw_synapse(ax, (0,       0), (HSPACE, 0), label=tex(STR_WEIGHT + "_2"), label_position=0.3)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_3"), label_position=0.3, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (HSPACE, 0), (HSPACE + 2, 0))

# Neuron ######################################

# Layer 1 (input)
#nnfig.draw_neuron(ax, (0,2*VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0,  VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0,       0), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -VSPACE), 0.5, empty=True)

# Layer 2
nnfig.draw_neuron(ax, (HSPACE, 0), 1, ag_func="sum", tr_func="sigmoid")

# Text ########################################

# Layer 1 (input)
#plt.text(x=0.5, y=VSPACE+1, s=tex(STR_SIGOUT + "_i"), fontsize=12)

#plt.text(x=-1.7, y=2*VSPACE,    s=tex("1"), fontsize=12)
plt.text(x=-1.7, y=VSPACE,      s=tex(STR_SIGIN + "_1"), fontsize=12)
plt.text(x=-1.7, y=-0.2,        s=tex(STR_SIGIN + "_2"), fontsize=12)
plt.text(x=-1.7, y=-VSPACE-0.2, s=tex(STR_SIGIN + "_3"), fontsize=12)

# Layer 2
#plt.text(x=HSPACE-1.25, y=1.5, s=tex(STR_POT), fontsize=12)
#plt.text(x=2*HSPACE+0.4,  y=1.5, s=tex(STR_SIGOUT + "_o"), fontsize=12)

plt.text(x=HSPACE+2.5,  y=-0.3,
         s=tex(STR_SIGOUT),
         fontsize=12)

plt.show()
```

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=6)

HSPACE = 6
VSPACE = 4

# Synapse #####################################

nnfig.draw_synapse(ax, (0,2*VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_0"), label_position=0.3)
nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_1"), label_position=0.3)
nnfig.draw_synapse(ax, (0,       0), (HSPACE, 0), label=tex(STR_WEIGHT + "_2"), label_position=0.3)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_3"), label_position=0.3, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (HSPACE, 0), (HSPACE + 2, 0))

# Neuron ######################################

# Layer 1 (input)
nnfig.draw_neuron(ax, (0,2*VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0,  VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0,       0), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -VSPACE), 0.5, empty=True)

# Layer 2
nnfig.draw_neuron(ax, (HSPACE, 0), 1, ag_func="sum", tr_func="sigmoid")

# Text ########################################

# Layer 1 (input)
#plt.text(x=0.5, y=VSPACE+1, s=tex(STR_SIGOUT + "_i"), fontsize=12)

plt.text(x=-1.7, y=2*VSPACE,    s=tex("1"), fontsize=12)
plt.text(x=-1.7, y=VSPACE,      s=tex(STR_SIGIN + "_1"), fontsize=12)
plt.text(x=-1.7, y=-0.2,        s=tex(STR_SIGIN + "_2"), fontsize=12)
plt.text(x=-1.7, y=-VSPACE-0.2, s=tex(STR_SIGIN + "_3"), fontsize=12)

# Layer 2
#plt.text(x=HSPACE-1.25, y=1.5, s=tex(STR_POT), fontsize=12)
#plt.text(x=2*HSPACE+0.4,  y=1.5, s=tex(STR_SIGOUT + "_o"), fontsize=12)

plt.text(x=HSPACE+2.5,  y=-0.3,
         s=tex(STR_SIGOUT),
         fontsize=12)

plt.show()
```

### Exemple

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=6)

HSPACE = 6
VSPACE = 4

# Synapse #####################################

nnfig.draw_synapse(ax, (0,2*VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_0"), label_position=0.3)
nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_1"), label_position=0.3)
nnfig.draw_synapse(ax, (0,       0), (HSPACE, 0), label=tex(STR_WEIGHT + "_2"), label_position=0.3)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE, 0), label=tex(STR_WEIGHT + "_3"), label_position=0.3, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (HSPACE, 0), (HSPACE + 2, 0))

# Neuron ######################################

# Layer 1 (input)
nnfig.draw_neuron(ax, (0,2*VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0,  VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0,       0), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -VSPACE), 0.5, empty=True)

# Layer 2
nnfig.draw_neuron(ax, (HSPACE, 0), 1, ag_func="sum", tr_func="sigmoid")

# Text ########################################

# Layer 1 (input)
#plt.text(x=0.5, y=VSPACE+1, s=tex(STR_SIGOUT + "_i"), fontsize=12)

plt.text(x=-1.7, y=2*VSPACE,    s=tex("1"), fontsize=12)
plt.text(x=-1.7, y=VSPACE,      s=tex(STR_SIGIN + "_1"), fontsize=12)
plt.text(x=-1.7, y=-0.2,        s=tex(STR_SIGIN + "_2"), fontsize=12)
plt.text(x=-1.7, y=-VSPACE-0.2, s=tex(STR_SIGIN + "_3"), fontsize=12)

# Layer 2
#plt.text(x=HSPACE-1.25, y=1.5, s=tex(STR_POT), fontsize=12)
#plt.text(x=2*HSPACE+0.4,  y=1.5, s=tex(STR_SIGOUT + "_o"), fontsize=12)

plt.text(x=HSPACE+2.5,  y=-0.3,
         s=tex(STR_SIGOUT),
         fontsize=12)

plt.show()
```

Pour vecteur d'entrée = ... et un vecteur de poids arbitrairement fixé à ...
et un neurone défini avec la fonction sigmoïde,
on peut calculer la valeur de sortie du neurone :

On a:

$$
\sum_i \weight_i \feature_i = \dots
$$
donc
$$
y = \frac{1}{1 + e^{-\dots}}
$$

```python
@interact(w1=(-10., 10., 0.5), w2=(-10., 10., 0.5))
def nn1(wb1=0., w1=10.):
    x = np.linspace(-10., 10., 100)
    xb = np.ones(x.shape)

    s1 = wb1 * xb + w1 * x
    y = sigmoid(s1)

    plt.plot(x, y)
```

## Définition d'un réseau de neurones

### Disposition des neurones en couches et couches cachées

TODO

### Exemple : réseau de neurones à 1 couche "cachée"

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=4)

HSPACE = 6
VSPACE = 4

# Synapse #####################################

# Layer 1-2
nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_1"), label_position=0.4)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_3"), label_position=0.25, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE, -VSPACE), label=tex(STR_WEIGHT + "_2"), label_position=0.25)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE, -VSPACE), label=tex(STR_WEIGHT + "_4"), label_position=0.4, label_offset_y=-0.8)

# Layer 2-3
nnfig.draw_synapse(ax, (HSPACE,  VSPACE), (2*HSPACE, 0), label=tex(STR_WEIGHT + "_5"), label_position=0.4)
nnfig.draw_synapse(ax, (HSPACE, -VSPACE), (2*HSPACE, 0), label=tex(STR_WEIGHT + "_6"), label_position=0.4, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (2*HSPACE, 0), (2*HSPACE + 2, 0))

# Neuron ######################################

# Layer 1 (input)
nnfig.draw_neuron(ax, (0,  VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -VSPACE), 0.5, empty=True)

# Layer 2
nnfig.draw_neuron(ax, (HSPACE,  VSPACE), 1, ag_func="sum", tr_func="sigmoid")
nnfig.draw_neuron(ax, (HSPACE, -VSPACE), 1, ag_func="sum", tr_func="sigmoid")

# Layer 3
nnfig.draw_neuron(ax, (2*HSPACE, 0), 1, ag_func="sum", tr_func="sigmoid")

# Text ########################################

# Layer 1 (input)
#plt.text(x=0.5, y=VSPACE+1, s=tex(STR_SIGOUT + "_i"), fontsize=12)
plt.text(x=-1.7, y=VSPACE,      s=tex(STR_SIGIN + "_1"), fontsize=12)
plt.text(x=-1.7, y=-VSPACE-0.2, s=tex(STR_SIGIN + "_2"), fontsize=12)

# Layer 2
#plt.text(x=HSPACE-1.25, y=VSPACE+1.5, s=tex(STR_POT + "_1"), fontsize=12)
plt.text(x=HSPACE+0.4,  y=VSPACE+1.5, s=tex(STR_SIGOUT + "_1"), fontsize=12)

#plt.text(x=HSPACE-1.25, y=-VSPACE-1.8, s=tex(STR_POT + "_2"), fontsize=12)
plt.text(x=HSPACE+0.4,  y=-VSPACE-1.8, s=tex(STR_SIGOUT + "_2"), fontsize=12)

# Layer 3
#plt.text(x=2*HSPACE-1.25, y=1.5, s=tex(STR_POT + "_o"), fontsize=12)
#plt.text(x=2*HSPACE+0.4,  y=1.5, s=tex(STR_SIGOUT + "_o"), fontsize=12)

plt.text(x=2*HSPACE+2.5,  y=-0.3,
         s=tex(STR_SIGOUT),
         fontsize=12)

plt.show()
```

TODO: il manque les biais...

$$
\sigout =
\activfunc \left(
\weight_5 ~ \underbrace{\activfunc \left(\weight_1 \feature_1 + \weight_3 \feature_2 \right)}_{\sigout_1}
+
\weight_6 ~ \underbrace{\activfunc \left(\weight_2 \feature_1 + \weight_4 \feature_2 \right)}_{\sigout_2}
\right)
$$

```python
@interact(wb1=(-10., 10., 0.5), w1=(-10., 10., 0.5), wb2=(-10., 10., 0.5), w2=(-10., 10., 0.5))
def nn1(wb1=0.1, w1=0.1, wb2=0.1, w2=0.1):
    x = np.linspace(-10., 10., 100)
    xb = np.ones(x.shape)

    s1 = wb1 * xb + w1 * x
    y1 = sigmoid(s1)

    s2 = wb2 * xb + w2 * x
    y2 = sigmoid(s2)

    s = wb2 * xb + w2 * x
    y = sigmoid(s)

    plt.plot(x, y)
```

### Exemple : réseau de neurones à 2 couches "cachée"

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=4)

HSPACE = 6
VSPACE = 4

# Synapse #####################################

# Layer 1-2
nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_1"), label_position=0.4)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_3"), label_position=0.25, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE, -VSPACE), label=tex(STR_WEIGHT + "_2"), label_position=0.25)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE, -VSPACE), label=tex(STR_WEIGHT + "_4"), label_position=0.4, label_offset_y=-0.8)

# Layer 2-3
nnfig.draw_synapse(ax, (HSPACE,  VSPACE), (2*HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_5"), label_position=0.4)
nnfig.draw_synapse(ax, (HSPACE, -VSPACE), (2*HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_7"), label_position=0.25, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (HSPACE,  VSPACE), (2*HSPACE, -VSPACE), label=tex(STR_WEIGHT + "_6"), label_position=0.25)
nnfig.draw_synapse(ax, (HSPACE, -VSPACE), (2*HSPACE, -VSPACE), label=tex(STR_WEIGHT + "_8"), label_position=0.4, label_offset_y=-0.8)

# Layer 3-4
nnfig.draw_synapse(ax, (2*HSPACE,  VSPACE), (3*HSPACE, 0), label=tex(STR_WEIGHT + "_9"), label_position=0.4)
nnfig.draw_synapse(ax, (2*HSPACE, -VSPACE), (3*HSPACE, 0), label=tex(STR_WEIGHT + "_{10}"), label_position=0.4, label_offset_y=-0.8)

nnfig.draw_synapse(ax, (3*HSPACE, 0), (3*HSPACE + 2, 0))

# Neuron ######################################

# Layer 1 (input)
nnfig.draw_neuron(ax, (0,  VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -VSPACE), 0.5, empty=True)

# Layer 2
nnfig.draw_neuron(ax, (HSPACE,  VSPACE), 1, ag_func="sum", tr_func="sigmoid")
nnfig.draw_neuron(ax, (HSPACE, -VSPACE), 1, ag_func="sum", tr_func="sigmoid")

# Layer 3
nnfig.draw_neuron(ax, (2*HSPACE,  VSPACE), 1, ag_func="sum", tr_func="sigmoid")
nnfig.draw_neuron(ax, (2*HSPACE, -VSPACE), 1, ag_func="sum", tr_func="sigmoid")

# Layer 4
nnfig.draw_neuron(ax, (3*HSPACE, 0), 1, ag_func="sum", tr_func="sigmoid")

# Text ########################################

# Layer 1 (input)
#plt.text(x=0.5, y=VSPACE+1, s=tex(STR_SIGOUT + "_i"), fontsize=12)
plt.text(x=-1.7, y=VSPACE,      s=tex(STR_SIGIN + "_1"), fontsize=12)
plt.text(x=-1.7, y=-VSPACE-0.2, s=tex(STR_SIGIN + "_2"), fontsize=12)

# Layer 2
#plt.text(x=HSPACE-1.25, y=VSPACE+1.5, s=tex(STR_POT + "_1"), fontsize=12)
plt.text(x=HSPACE+0.4,  y=VSPACE+1.5, s=tex(STR_SIGOUT + "_1"), fontsize=12)

#plt.text(x=HSPACE-1.25, y=-VSPACE-1.8, s=tex(STR_POT + "_2"), fontsize=12)
plt.text(x=HSPACE+0.4,  y=-VSPACE-1.8, s=tex(STR_SIGOUT + "_2"), fontsize=12)

# Layer 3
#plt.text(x=2*HSPACE-1.25, y=VSPACE+1.5, s=tex(STR_POT + "_3"), fontsize=12)
plt.text(x=2*HSPACE+0.4,  y=VSPACE+1.5, s=tex(STR_SIGOUT + "_3"), fontsize=12)

#plt.text(x=2*HSPACE-1.25, y=-VSPACE-1.8, s=tex(STR_POT + "_4"), fontsize=12)
plt.text(x=2*HSPACE+0.4,  y=-VSPACE-1.8, s=tex(STR_SIGOUT + "_4"), fontsize=12)

# Layer 4
#plt.text(x=3*HSPACE-1.25, y=1.5, s=tex(STR_POT + "_o"), fontsize=12)
#plt.text(x=3*HSPACE+0.4,  y=1.5, s=tex(STR_SIGOUT + "_o"), fontsize=12)

plt.text(x=3*HSPACE+2.5,  y=-0.3,
         s=tex(STR_SIGOUT),
         fontsize=12)

plt.show()
```

TODO: il manque le biais...

$
\newcommand{\yone}{\underbrace{\activfunc \left(\weight_1 \feature_1 + \weight_3 \feature_2 \right)}_{\sigout_1}}
\newcommand{\ytwo}{\underbrace{\activfunc \left(\weight_2 \feature_1 + \weight_4 \feature_2 \right)}_{\sigout_2}}
\newcommand{\ythree}{\underbrace{\activfunc \left(\weight_5 \yone + \weight_7 \ytwo \right)}_{\sigout_3}}
\newcommand{\yfour}{\underbrace{\activfunc \left(\weight_6 \yone + \weight_8 \ytwo \right)}_{\sigout_4}}
$

$$
\sigout =
\activfunc \left(
\weight_9 ~ \ythree
+
\weight_{10} ~ \yfour
\right)
$$

### Pouvoir expressif d'un réseau de neurones

TODO

## Apprentissage

### Fonction objectif (ou fonction d'erreur)

Fonction objectif: $\errfunc \left( \weights \right)$

Typiquement, la fonction objectif (fonction d'erreur) est la somme du carré de l'erreur de chaque neurone de sortie.

$$
\errfunc = \frac12 \sum_{\cur \in \Omega} \left[ \sigout_\cur - \sigoutdes_\cur \right]^2
$$

$\Omega$: l'ensemble des neurones de sortie

Le $\frac12$, c'est juste pour simplifier les calculs de la dérivée.

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=4)

nnfig.draw_synapse(ax, (0, -6), (10, 0))
nnfig.draw_synapse(ax, (0, -2), (10, 0))
nnfig.draw_synapse(ax, (0, 2),  (10, 0))
nnfig.draw_synapse(ax, (0, 6),  (10, 0))

nnfig.draw_synapse(ax, (0, -6), (10, -4))
nnfig.draw_synapse(ax, (0, -2), (10, -4))
nnfig.draw_synapse(ax, (0, 2),  (10, -4))
nnfig.draw_synapse(ax, (0, 6),  (10, -4))

nnfig.draw_synapse(ax, (0, -6), (10, 4))
nnfig.draw_synapse(ax, (0, -2), (10, 4))
nnfig.draw_synapse(ax, (0, 2),  (10, 4))
nnfig.draw_synapse(ax, (0, 6),  (10, 4))

nnfig.draw_synapse(ax, (10, -4), (12, -4))
nnfig.draw_synapse(ax, (10, 0), (12, 0))
nnfig.draw_synapse(ax, (10, 4), (12, 4))

nnfig.draw_neuron(ax, (0, -6), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)
nnfig.draw_neuron(ax, (0, 6),  0.5, empty=True)

nnfig.draw_neuron(ax, (10, -4), 1, ag_func="sum", tr_func="sigmoid")
nnfig.draw_neuron(ax, (10, 0),  1, ag_func="sum", tr_func="sigmoid")
nnfig.draw_neuron(ax, (10, 4),  1, ag_func="sum", tr_func="sigmoid")

plt.text(x=0, y=7.5, s=tex(STR_PREV), fontsize=14)
plt.text(x=10, y=7.5, s=tex(STR_CUR), fontsize=14)

plt.text(x=0,   y=0,    s=r"$\vdots$", fontsize=14)
plt.text(x=9.7, y=-6.1, s=r"$\vdots$", fontsize=14)
plt.text(x=9.7, y=5.8,  s=r"$\vdots$", fontsize=14)

plt.text(x=12.5, y=4,  s=tex(STR_SIGOUT + "_1"), fontsize=14)
plt.text(x=12.5, y=0,  s=tex(STR_SIGOUT + "_2"), fontsize=14)
plt.text(x=12.5, y=-4, s=tex(STR_SIGOUT + "_3"), fontsize=14)

plt.text(x=16, y=4,  s=tex(STR_ERRFUNC + "_1 = " + STR_SIGOUT + "_1 - " + STR_SIGOUT_DES + "_1"), fontsize=14)
plt.text(x=16, y=0,  s=tex(STR_ERRFUNC + "_2 = " + STR_SIGOUT + "_2 - " + STR_SIGOUT_DES + "_2"), fontsize=14)
plt.text(x=16, y=-4, s=tex(STR_ERRFUNC + "_3 = " + STR_SIGOUT + "_3 - " + STR_SIGOUT_DES + "_3"), fontsize=14)

plt.text(x=16, y=-8, s=tex(STR_ERRFUNC + " = 1/2 ( " + STR_ERRFUNC + "^2_1 + " + STR_ERRFUNC + "^2_2 + " + STR_ERRFUNC + "^2_3 + \dots )"), fontsize=14)

plt.show()
```

#### Mise à jours des poids

$$
\weights_{\learnit + 1} = \weights_{\learnit} - \underbrace{\learnrate \nabla_{\weights} \errfunc \left( \weights_{\learnit} \right)}
$$

$- \learnrate \nabla_{\weights} \errfunc \left( \weights_{\learnit} \right)$: descend dans la direction opposée au gradient (plus forte pente)

avec $\nabla_{\weights} \errfunc \left( \weights_{\learnit} \right)$: gradient de la fonction objectif au point $\weights$

$\learnrate > 0$: pas (ou taux) d'apprentissage

$$
\begin{align}
\delta_{\wcur} & = \wcur_{\learnit + 1} - \wcur_{\learnit} \\
               & = - \learnrate \frac{\partial \errfunc}{\partial \wcur}
\end{align}
$$

$$
\Leftrightarrow \wcur_{\learnit + 1} = \wcur_{\learnit} - \learnrate \frac{\partial \errfunc}{\partial \wcur}
$$

Chaque présentation de l'ensemble des exemples = un *cycle* (ou une *époque*) d'apprentissage

Critère d'arrêt de l'apprentissage: quand la valeur de la fonction objectif se stabilise (ou que le problème est résolu avec la précision souhaitée)

#### Dérivée des principales fonctions d'activation

##### Fonction sigmoïde

Fonction dérivée :

$$
f'(x) = \frac{\lambda e^{-\lambda x}}{(1+e^{-\lambda x})^{2}}
$$

qui peut aussi être défini par

$$
\frac{\mathrm{d} y}{\mathrm{d} x} = \lambda y (1-y)
$$

où $y$ varie de 0 à 1.

```python
def d_sigmoid(x, _lambda=1.):
    e = np.exp(-_lambda * x)
    y = _lambda * e / np.power(1 + e, 2)
    return y
```

```python
%matplotlib inline

x = np.linspace(-5, 5, 300)

y1 = d_sigmoid(x, 1.)
y2 = d_sigmoid(x, 5.)
y3 = d_sigmoid(x, 0.5)

plt.plot(x, y1, label=r"$\lambda=1$")
plt.plot(x, y2, label=r"$\lambda=5$")
plt.plot(x, y3, label=r"$\lambda=0.5$")

plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')
plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')

plt.legend()

plt.title("Fonction dérivée de la sigmoïde")
plt.axis([-5, 5, -0.5, 2]);
```

##### Tangente hyperbolique

Dérivée :

$$
\tanh '= \frac{1}{\cosh^{2}} = 1-\tanh^{2}
$$

```python
def d_tanh(x):
    y = 1. - np.power(np.tanh(x), 2)
    return y
```

```python
x = np.linspace(-5, 5, 300)
y = d_tanh(x)

plt.plot(x, y)

plt.hlines(y=0, xmin=-5, xmax=5, color='gray', linestyles='dotted')
plt.vlines(x=0, ymin=-2, ymax=2, color='gray', linestyles='dotted')

plt.title("Fonction dérivée de la tangente hyperbolique")
plt.axis([-5, 5, -2, 2]);
```

```python
# TODO
# - "généralement le minimum local suffit" (preuve ???)
# - "dans le cas contraire, le plus simple est de recommencer plusieurs fois l'apprentissage avec des poids initiaux différents et de conserver la meilleure matrice $\weights$ (celle qui minimise $\errfunc$)"
```

```python
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(4, 4))

x = np.arange(10, 30, 0.1)
y = (x - 20)**2 + 2

ax.set_xlabel(r"Poids $" + STR_WEIGHTS + "$", fontsize=14)
ax.set_ylabel(r"Fonction objectif $" + STR_ERRFUNC + "$", fontsize=14)

# See http://matplotlib.org/api/axes_api.html#matplotlib.axes.Axes.tick_params
ax.tick_params(axis='both',       # changes apply to the x and y axis
               which='both',      # both major and minor ticks are affected
               bottom='on',       # ticks along the bottom edge are on
               top='off',         # ticks along the top edge are off
               left='on',         # ticks along the left edge are on
               right='off',       # ticks along the right edge are off
               labelbottom='off', # labels along the bottom edge are off
               labelleft='off')   # labels along the lefleft are off

ax.set_xlim(left=10, right=25)
ax.set_ylim(bottom=0, top=5)

ax.plot(x, y);
```

*Apprentissage incrémentiel* (ou *partiel*) (ang. *incremental learning*):
on ajuste les poids $\weights$ après la présentation d'un seul exemple
("ce n'est pas une véritable descente de gradient").
C'est mieux pour éviter les minimums locaux, surtout si les exemples sont
mélangés au début de chaque itération

```python
# *Apprentissage différé* (ang. *batch learning*):
# TODO
# Est-ce que la fonction objectif $\errfunc$ est une fonction multivariée
# ou est-ce une aggrégation des erreurs de chaque exemple ?
```

```python
# **TODO: règle du delta / règle du delta généralisée**
```

### Rétropropagation du gradient

*Rétropropagation du gradient*:
une méthode pour calculer efficacement le gradient de la fonction objectif $\errfunc$.

Intuition:
La *rétropropagation du gradient* n'est qu'une méthode parmis d'autre pour résoudre le probème d'optimisation des poids $\weight$. On pourrait très bien résoudre ce problème d'optimisation avec des algorithmes évolutionnistes par exemple.
En fait, l'intérêt de la méthode de la *rétropropagation du gradient* (et ce qui explique sa notoriété) est qu'elle formule le problème d'optimisation des poids avec une écriture analytique particulièrement efficace qui élimine astucieusement un grand nombre de calculs redondants (un peu à la manière de ce qui se fait en programmation dynamique): quand on decide d'optimiser les poids via une *descente de gradient*, certains termes (les *signaux d'erreurs* $\errsig$) apparaissent un grand nombre de fois dans l'écriture analytique complète du gradient. La méthode de la retropropagation du gradient fait en sorte que ces termes ne soient calculés qu'une seule fois.
À noter qu'on aurrait très bien pu résoudre le problème avec une descente de gradient oú le gradient $\frac{\partial \errfunc}{\partial\wcur_{\learnit}}$ serait calculé via une approximation numérique (*méthode des différences finies* par exemple) mais ce serait beaucoup plus lent et beaucoup moins efficace...

Principe:
on modifie les poids à l'aide des *signaux d'erreur* $\errsig$.

$$
\wcur_{\learnit + 1} = \wcur_{\learnit} \underbrace{- \learnrate \frac{\partial \errfunc}{\partial \wcur_{\learnit}}}_{\delta_\prevcur}
$$

$$
\begin{align}
\delta_\prevcur & = - \learnrate \frac{\partial \errfunc}{\partial \wcur(\learnit)} \\
                & = - \learnrate \errsig_\cur \sigout\prev
\end{align}
$$

- Dans le cas de *l'apprentissage différé* (*batch*), on calcule pour chaque exemple l'erreur correspondante. Leur contribution individuelle aux modifications des poids sont additionnées
- L'apprentissage suppervisé fonctionne mieux avec des neurones de sortie linéaires (fonction d'activation $\activfunc$ = fonction identitée) "car les signaux d'erreurs se transmettent mieux".
- Des données d'entrée binaires doivent être choisies dans $\{-1,1\}$ plutôt que $\{0,1\}$ car un signal nul ne contribu pas à l'apprentissage.

```python
# TODO
#Voc:
#- *erreur marginale*: **TODO**
```

Note intéressante de Jürgen Schmidhuber : http://people.idsia.ch/~juergen/who-invented-backpropagation.html

### Signaux d'erreur $\errsig_\cur$ pour les neurones de sortie $(\cur \in \Omega)$

$$
\errsig_\cur = \activfunc'(\pot_\cur)[\sigout_\cur - \sigoutdes_\cur]
$$

### Signaux d'erreur $\errsig_\cur$ pour les neurones cachés $(\cur \not\in \Omega)$

$$
\errsig_\cur = \activfunc'(\pot_\cur) \sum_\next \weight_\curnext \errsig_\next
$$

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=4)

nnfig.draw_synapse(ax, (0, -2), (10, 0))
nnfig.draw_synapse(ax, (0, 2),  (10, 0), label=tex(STR_WEIGHT + "_{" + STR_NEXT + STR_CUR + "}"), label_position=0.5, fontsize=14)

nnfig.draw_synapse(ax, (10, 0), (12, 0))

nnfig.draw_neuron(ax, (0, -2), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, 2),  0.5, empty=True)

plt.text(x=0, y=3.5, s=tex(STR_CUR), fontsize=14)
plt.text(x=10, y=3.5, s=tex(STR_NEXT), fontsize=14)
plt.text(x=0, y=-0.2, s=r"$\vdots$", fontsize=14)

nnfig.draw_neuron(ax, (10, 0), 1, ag_func="sum", tr_func="sigmoid")

plt.show()
```

### Plus de détail : calcul de $\errsig_\cur$

Dans l'exemple suivant on ne s'intéresse qu'aux poids $\weight_1$, $\weight_2$, $\weight_3$, $\weight_4$ et $\weight_5$ pour simplifier la demonstration.

```python
fig, ax = nnfig.init_figure(size_x=8, size_y=4)

HSPACE = 6
VSPACE = 4

# Synapse #####################################

# Layer 1-2
nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_1"), label_position=0.4)
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE,  VSPACE), color="lightgray")

nnfig.draw_synapse(ax, (0,  VSPACE), (HSPACE, -VSPACE), color="lightgray")
nnfig.draw_synapse(ax, (0, -VSPACE), (HSPACE, -VSPACE), color="lightgray")

# Layer 2-3
nnfig.draw_synapse(ax, (HSPACE,  VSPACE), (2*HSPACE,  VSPACE), label=tex(STR_WEIGHT + "_2"), label_position=0.4)
nnfig.draw_synapse(ax, (HSPACE, -VSPACE), (2*HSPACE,  VSPACE), color="lightgray")

nnfig.draw_synapse(ax, (HSPACE,  VSPACE), (2*HSPACE, -VSPACE), label=tex(STR_WEIGHT + "_3"), label_position=0.4)
nnfig.draw_synapse(ax, (HSPACE, -VSPACE), (2*HSPACE, -VSPACE), color="lightgray")

# Layer 3-4
nnfig.draw_synapse(ax, (2*HSPACE,  VSPACE), (3*HSPACE, 0), label=tex(STR_WEIGHT + "_4"), label_position=0.4)
nnfig.draw_synapse(ax, (2*HSPACE, -VSPACE), (3*HSPACE, 0), label=tex(STR_WEIGHT + "_5"), label_position=0.4, label_offset_y=-0.8)

# Neuron ######################################

# Layer 1 (input)
nnfig.draw_neuron(ax, (0,  VSPACE), 0.5, empty=True)
nnfig.draw_neuron(ax, (0, -VSPACE), 0.5, empty=True, line_color="lightgray")

# Layer 2
nnfig.draw_neuron(ax, (HSPACE,  VSPACE), 1, ag_func="sum", tr_func="sigmoid")
nnfig.draw_neuron(ax, (HSPACE, -VSPACE), 1, ag_func="sum", tr_func="sigmoid", line_color="lightgray")

# Layer 3
nnfig.draw_neuron(ax, (2*HSPACE,  VSPACE), 1, ag_func="sum", tr_func="sigmoid")
nnfig.draw_neuron(ax, (2*HSPACE, -VSPACE), 1, ag_func="sum", tr_func="sigmoid")

# Layer 4
nnfig.draw_neuron(ax, (3*HSPACE, 0), 1, ag_func="sum", tr_func="sigmoid")

# Text ########################################

# Layer 1 (input)
plt.text(x=0.5, y=VSPACE+1, s=tex(STR_SIGOUT + "_i"), fontsize=12)

# Layer 2
plt.text(x=HSPACE-1.25, y=VSPACE+1.5, s=tex(STR_POT + "_1"), fontsize=12)
plt.text(x=HSPACE+0.4,  y=VSPACE+1.5, s=tex(STR_SIGOUT + "_1"), fontsize=12)

# Layer 3
plt.text(x=2*HSPACE-1.25, y=VSPACE+1.5, s=tex(STR_POT + "_2"), fontsize=12)
plt.text(x=2*HSPACE+0.4,  y=VSPACE+1.5, s=tex(STR_SIGOUT + "_2"), fontsize=12)

plt.text(x=2*HSPACE-1.25, y=-VSPACE-1.8, s=tex(STR_POT + "_3"), fontsize=12)
plt.text(x=2*HSPACE+0.4,  y=-VSPACE-1.8, s=tex(STR_SIGOUT + "_3"), fontsize=12)

# Layer 4
plt.text(x=3*HSPACE-1.25, y=1.5, s=tex(STR_POT + "_o"), fontsize=12)
plt.text(x=3*HSPACE+0.4,  y=1.5, s=tex(STR_SIGOUT + "_o"), fontsize=12)

plt.text(x=3*HSPACE+2,  y=-0.3,
         s=tex(STR_ERRFUNC + " = (" + STR_SIGOUT + "_o - " + STR_SIGOUT_DES + "_o)^2/2"),
         fontsize=12)

plt.show()
```

**Attention**: $\weight_1$ influe $\pot_2$ **et** $\pot_3$ en plus de $\pot_1$ et $\pot_o$.

#### Calcul de la dérivée partielle de l'erreur par rapport au poid synaptique $\weight_4$

rappel:

$$
\begin{align}
\errfunc  &= \frac12 \left( \sigout_o - \sigoutdes_o \right)^2 \tag{1} \\
\sigout_o &= \activfunc(\pot_o)  \tag{2} \\
\pot_o    &= \sigout_2 \weight_4 + \sigout_3 \weight_5  \tag{3} \\
\end{align}
$$

c'est à dire:

$$
\errfunc = \frac12 \left( \activfunc \left( \sigout_2 \weight_4 + \sigout_3 \weight_5 \right) - \sigoutdes_o \right)^2
$$

donc, en appliquant les règles de derivation de fonctions composées, on a:

$$
\frac{\partial \errfunc}{\partial \weight_4} =
\frac{\partial \pot_o}{\partial \weight_4}
\underbrace{
\frac{\partial \sigout_o}{\partial \pot_o}
\frac{\partial \errfunc}{\partial \sigout_o}
}_{\errsig_o}
$$

Rappel: dérivation des fonctions composées (parfois appelé règle de dérivation en chaîne ou règle de la chaîne)

$$
\frac{\mathrm{d} y}{\mathrm{d} x} = \frac{\mathrm{d} y}{\mathrm{d} u} \cdot \frac{\mathrm{d} u}{\mathrm {d} x}
$$

de (1), (2) et (3) on déduit:

$$
\begin{align}
\frac{\partial \pot_o}{\partial \weight_4}   &= \sigout_2 \\
\frac{\partial \sigout_o}{\partial \pot_o}   &= \activfunc'(\pot_o) \\
\frac{\partial \errfunc}{\partial \sigout_o} &= \sigout_o - \sigoutdes_o \\
\end{align}
$$

le signal d'erreur s'écrit donc:

$$
\begin{align}
\errsig_o &=
\frac{\partial \sigout_o}{\partial \pot_o}
\frac{\partial \errfunc}{\partial \sigout_o} \\
&= \activfunc'(\pot_o) [\sigout_o - \sigoutdes_o]
\end{align}
$$

#### Calcul de la dérivée partielle de l'erreur par rapport au poid synaptique $\weight_5$

$$
\frac{\partial \errfunc}{\partial \weight_5} =
\frac{\partial \pot_o}{\partial \weight_5}
\underbrace{
\frac{\partial \sigout_o}{\partial \pot_o}
\frac{\partial \errfunc}{\partial \sigout_o}
}_{\errsig_o}
$$

avec:

$$
\begin{align}
\frac{\partial \pot_o}{\partial \weight_5}   &= \sigout_3 \\
\frac{\partial \sigout_o}{\partial \pot_o}   &= \activfunc'(\pot_o) \\
\frac{\partial \errfunc}{\partial \sigout_o} &= \sigout_o - \sigoutdes_o \\
\errsig_o &=
\frac{\partial \sigout_o}{\partial \pot_o}
\frac{\partial \errfunc}{\partial \sigout_o} \\
&= \activfunc'(\pot_o) [\sigout_o - \sigoutdes_o]
\end{align}
$$

#### Calcul de la dérivée partielle de l'erreur par rapport au poid synaptique $\weight_2$

$$
\frac{\partial \errfunc}{\partial \weight_2} =
\frac{\partial \pot_2}{\partial \weight_2}
%
\underbrace{
 \frac{\partial \sigout_2}{\partial \pot_2}
 \frac{\partial \pot_o}{\partial \sigout_2}
 \underbrace{
  \frac{\partial \sigout_o}{\partial \pot_o}
  \frac{\partial \errfunc}{\partial \sigout_o}
 }_{\errsig_o}
}_{\errsig_2}
$$

avec:

$$
\begin{align}
\frac{\partial \pot_2}{\partial \weight_2} &= \sigout_1 \\
\frac{\partial \sigout_2}{\partial \pot_2} &= \activfunc'(\pot_2) \\
\frac{\partial \pot_o}{\partial \sigout_2} &= \weight_4 \\
\errsig_2 &=
\frac{\partial \sigout_2}{\partial \pot_2}
\frac{\partial \pot_o}{\partial \sigout_2}
\errsig_o \\
&= \activfunc'(\pot_2) \weight_4 \errsig_o
\end{align}
$$

#### Calcul de la dérivée partielle de l'erreur par rapport au poid synaptique $\weight_3$

$$
\frac{\partial \errfunc}{\partial \weight_3} =
\frac{\partial \pot_3}{\partial \weight_3}
%
\underbrace{
 \frac{\partial \sigout_3}{\partial \pot_3}
 \frac{\partial \pot_o}{\partial \sigout_3}
 \underbrace{
  \frac{\partial \sigout_o}{\partial \pot_o}
  \frac{\partial \errfunc}{\partial \sigout_o}
 }_{\errsig_o}
}_{\errsig_3}
$$

avec:

$$
\begin{align}
\frac{\partial \pot_3}{\partial \weight_3} &= \sigout_1 \\
\frac{\partial \sigout_3}{\partial \pot_3} &= \activfunc'(\pot_3) \\
\frac{\partial \pot_o}{\partial \sigout_3} &= \weight_5 \\
\errsig_3 &=
\frac{\partial \sigout_3}{\partial \pot_3}
\frac{\partial \pot_o}{\partial \sigout_3}
\errsig_o \\
&= \activfunc'(\pot_3) \weight_5 \errsig_o
\end{align}
$$

#### Calcul de la dérivée partielle de l'erreur par rapport au poid synaptique $\weight_1$

$$
\frac{\partial \errfunc}{\partial \weight_1} =
\frac{\partial \pot_1}{\partial \weight_1}
%
\underbrace{
 \frac{\partial \sigout_1}{\partial \pot_1}
 \left(
  \frac{\partial \pot_2}{\partial \sigout_1} % err?
  \underbrace{
   \frac{\partial \sigout_2}{\partial \pot_2}
   \frac{\partial \pot_o}{\partial \sigout_2}
   \underbrace{
    \frac{\partial \sigout_o}{\partial \pot_o}
    \frac{\partial \errfunc}{\partial \sigout_o}
   }_{\errsig_o}
  }_{\errsig_2}
  +
  \frac{\partial \pot_3}{\partial \sigout_1} % err?
  \underbrace{
   \frac{\partial \sigout_3}{\partial \pot_3}
   \frac{\partial \pot_o}{\partial \sigout_3}
   \underbrace{
    \frac{\partial \sigout_o}{\partial \pot_o}
    \frac{\partial \errfunc}{\partial \sigout_o}
   }_{\errsig_o}
  }_{\errsig_3}
 \right)
}_{\errsig_1}
$$

avec:

$$
\begin{align}
\frac{\partial \pot_1}{\partial \weight_1} &= \sigout_i \\
\frac{\partial \sigout_1}{\partial \pot_1} &= \activfunc'(\pot_1) \\
\frac{\partial \pot_2}{\partial \sigout_1} &= \weight_2 \\
\frac{\partial \pot_3}{\partial \sigout_1} &= \weight_3 \\
\errsig_1 &=
\frac{\partial \sigout_1}{\partial \pot_1}
\left(
\frac{\partial \pot_2}{\partial \sigout_1}
\errsig_2
+
\frac{\partial \pot_3}{\partial \sigout_1}
\errsig_3
\right) \\
&=
\activfunc'(\pot_1) \left( \weight_2 \errsig_2 + \weight_3 \errsig_3 \right)
\end{align}
$$

## Python implementation

```python
# Define the activation function and its derivative
activation_function = tanh
d_activation_function = d_tanh
```

```python
def init_weights(num_input_cells, num_output_cells, num_cell_per_hidden_layer, num_hidden_layers=1):
    """
    The returned `weights` object is a list of weight matrices,
    where weight matrix at index $i$ represents the weights between
    layer $i$ and layer $i+1$.

    Numpy array shapes for e.g. num_input_cells=2, num_output_cells=2,
    num_cell_per_hidden_layer=3 (without taking account bias):
    - in:        (2,)
    - in+bias:   (3,)
    - w[0]:      (3,3)
    - w[0]+bias: (3,4)
    - w[1]:      (3,2)
    - w[1]+bias: (4,2)
    - out:       (2,)
    """

    # TODO:
    # - faut-il que wij soit positif ?
    # - loi normale plus appropriée que loi uniforme ?
    # - quel sigma conseillé ?

    W = []

    # Weights between the input layer and the first hidden layer
    W.append(np.random.uniform(low=0., high=1., size=(num_input_cells + 1, num_cell_per_hidden_layer + 1)))

    # Weights between hidden layers (if there are more than one hidden layer)
    for layer in range(num_hidden_layers - 1):
        W.append(np.random.uniform(low=0., high=1., size=(num_cell_per_hidden_layer + 1, num_cell_per_hidden_layer + 1)))

    # Weights between the last hidden layer and the output layer
    W.append(np.random.uniform(low=0., high=1., size=(num_cell_per_hidden_layer + 1, num_output_cells)))

    return W
```

```python
def evaluate_network(weights, input_signal):      # TODO: find a better name

    # Add the bias on the input layer
    input_signal = np.concatenate([input_signal, [-1]])

    assert input_signal.ndim == 1
    assert input_signal.shape[0] == weights[0].shape[0]

    # Compute the output of the first hidden layer
    p = np.dot(input_signal, weights[0])
    output_hidden_layer = activation_function(p)

    # Compute the output of the intermediate hidden layers
    # TODO: check this
    num_layers = len(weights)
    for n in range(num_layers - 2):
        p = np.dot(output_hidden_layer, weights[n + 1])
        output_hidden_layer = activation_function(p)

    # Compute the output of the output layer
    p = np.dot(output_hidden_layer, weights[-1])
    output_signal = activation_function(p)

    return output_signal
```

```python
def compute_gradient():
    # TODO
    pass
```

```python
weights = init_weights(num_input_cells=2, num_output_cells=2, num_cell_per_hidden_layer=3, num_hidden_layers=1)
print(weights)
#print(weights[0].shape)
#print(weights[1].shape)
```

```python
input_signal = np.array([.1, .2])
input_signal
```

```python
evaluate_network(weights, input_signal)
```

## Divers

Le PMC peut approximer n'importe quelle fonction **continue** avec une précision arbitraire suivant le nombre de neurones présents sur la couche cachée.

Initialisation des poids: généralement des petites valeurs aléatoires

```python
# TODO: la différence entre:
# * réseau bouclé
# * réseau récurent
```

## Notes de la documentation sklearn

- *features* : les données d'entrée du réseau (i.e. les entrées de la 1ere couche du réseau)
  - "nombre de features" =  taille du vecteur d'entrées
- *loss function*: fonction objectif (ou fonction d'erreur)
- *fitting*: processus d'apprentissage (training)
- *sample*: exemple

Les biais sont stockés dans une liste de vecteurs plutôt qu'une liste de scalaires... pourquoi ???

Avantages des PMC:
- capables d'apprendre des modèles non linéaires
- capables d'apprendre des modèles en temps réel (apprentissage *on-line*)

Inconvenients des PMC:
- les PMC avec une ou plusieurs couches cachées ont une fonction objectif non-convexe avec des minimas locaux. Par conséquent, le résultat du processus d'apprentissage peut varier d'une execution à l'autre suivant la valeur des poids initiaux et l'obtention d'un réseau optimal n'est pas garanti
- pour obtenir un résultat satisfaisant, il est souvant nécessaire de régler (plus ou moins empiriquement) de nombreux meta-paramètres (nombres de couches cachées, nombre de neurones sur les couches cachées, nombres d'itérations, ...)
- une  mauvaise normalisation des données d'entrée a un impact très négatif sur la qualité du résultat ("mal conditionné" ???)

*Cross-Entropy Loss Function*: ...

*Softmax*: ...

*Multi-label classification*: ... modèle de classifieur qui permet a un exemple d'appartenir à plusieurs classes


# Stochastic Variational Inference for scalability with SVGP

One of the main criticisms of Gaussian processes is their scalability to large datasets. In this notebook, we illustrate how to use the state-of-the-art Stochastic Variational Gaussian Process (SVGP) (*Hensman, et. al. 2013*) to overcome this problem.

```python
%matplotlib inline
import itertools
import numpy as np
import time
import gpflow
import tensorflow as tf
import matplotlib.pyplot as plt
from gpflow.ci_utils import ci_niter

plt.style.use("ggplot")

# for reproducibility of this notebook:
rng = np.random.RandomState(123)
tf.random.set_seed(42)
```

## Generating data
For this notebook example, we generate 10,000 noisy observations from a test function:
\begin{equation}
f(x) = \sin(3\pi x) + 0.3\cos(9\pi x) + \frac{\sin(7 \pi x)}{2}
\end{equation}

```python
def func(x):
    return np.sin(x * 3 * 3.14) + 0.3 * np.cos(x * 9 * 3.14) + 0.5 * np.sin(x * 7 * 3.14)

N = 10000  # Number of training observations

X = rng.rand(N, 1) * 2 - 1  # X values
Y = func(X) + 0.2 * rng.randn(N, 1)  # Noisy Y values
data = (X, Y)
```

We plot the data along with the noiseless generating function:

```python
plt.plot(X, Y, "x", alpha=0.2)
Xt = np.linspace(-1.1, 1.1, 1000)[:, None]
Yt = func(Xt)
_ = plt.plot(Xt, Yt, c="k")
```

## Building the model
The main idea behind SVGP is to approximate the true GP posterior with a GP conditioned on a small set of "inducing" values. This smaller set can be thought of as summarizing the larger dataset. For this example, we will select a set of 50 inducing locations that are initialized from the training dataset:

```python
M = 50  # Number of inducing locations

kernel = gpflow.kernels.SquaredExponential()
Z = X[:M, :].copy()  # Initialize inducing locations to the first M inputs in the dataset

m = gpflow.models.SVGP(kernel, gpflow.likelihoods.Gaussian(), Z, num_data=N)
```

## Likelihood computation: batch vs. minibatch
First we showcase the model's performance using the whole dataset to compute the ELBO.

```python
elbo = tf.function(m.elbo)
```

```python
# TensorFlow re-traces & compiles a `tf.function`-wrapped method at *every* call if the arguments are numpy arrays instead of tf.Tensors. Hence:
tensor_data = tuple(map(tf.convert_to_tensor, data))
elbo(tensor_data)  # run it once to trace & compile
```

    <tf.Tensor: shape=(), dtype=float64, numpy=-17730.676725310346>

```python
%%timeit
elbo(tensor_data)
```

    55.7 ms ± 7.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

We can speed up this calculation by using minibatches of the data. For this example, we use minibatches of size 100.

```python
minibatch_size = 100

train_dataset = tf.data.Dataset.from_tensor_slices((X, Y)).repeat().shuffle(N)

train_iter = iter(train_dataset.batch(minibatch_size))

ground_truth = elbo(tensor_data).numpy()
```

```python
%%timeit
elbo(next(train_iter))
```

    1.63 ms ± 262 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

### Stochastical estimation of ELBO
The minibatch estimate should be an unbiased estimator of the `ground_truth`. Here we show a histogram of the value from different evaluations, together with its mean and the ground truth. The small difference between the mean of the minibatch estimations and the ground truth shows that the minibatch estimator is working as expected.

```python
evals = [elbo(minibatch).numpy() for minibatch in itertools.islice(train_iter, 100)]
```

```python
plt.hist(evals, label="Minibatch estimations")
plt.axvline(ground_truth, c="k", label="Ground truth")
plt.axvline(np.mean(evals), c="g", ls="--", label="Minibatch mean")
plt.legend()
plt.title("Histogram of ELBO evaluations using minibatches")
print("Discrepancy between ground truth and minibatch estimate:", ground_truth - np.mean(evals))
```

### Minibatches speed up computation
The reason for using minibatches is that it decreases the time needed to make an optimization step, because estimating the objective is computationally cheaper with fewer data points. Here we plot the change in time required with the size of the minibatch. We see that smaller minibatches result in a cheaper estimate of the objective.

```python
# Evaluate objective for different minibatch sizes
minibatch_proportions = np.logspace(-2, 0, 10)
times = []
objs = []
for mbp in minibatch_proportions:
    batchsize = int(N * mbp)
    train_iter = iter(train_dataset.batch(batchsize))
    start_time = time.time()
    objs.append([elbo(minibatch) for minibatch in itertools.islice(train_iter, 20)])
    times.append(time.time() - start_time)
```

```python
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))
ax1.plot(minibatch_proportions, times, "x-")
ax1.set_xlabel("Minibatch proportion")
ax1.set_ylabel("Time taken")

ax2.plot(minibatch_proportions, np.array(objs), "kx")
ax2.set_xlabel("Minibatch proportion")
ax2.set_ylabel("ELBO estimates")
```

### Running stochastic optimization

First we create a utility function that plots the model's predictions:

```python
def plot(title=""):
    plt.figure(figsize=(12, 4))
    plt.title(title)
    pX = np.linspace(-1, 1, 100)[:, None]  # Test locations
    pY, pYv = m.predict_y(pX)  # Predict Y values at test locations
    plt.plot(X, Y, "x", label="Training points", alpha=0.2)
    (line,) = plt.plot(pX, pY, lw=1.5, label="Mean of predictive posterior")
    col = line.get_color()
    plt.fill_between(
        pX[:, 0],
        (pY - 2 * pYv ** 0.5)[:, 0],
        (pY + 2 * pYv ** 0.5)[:, 0],
        color=col,
        alpha=0.6,
        lw=1.5,
    )
    Z = m.inducing_variable.Z.numpy()
    plt.plot(Z, np.zeros_like(Z), "k|", mew=2, label="Inducing locations")
    plt.legend(loc="lower right")

plot(title="Predictions before training")
```

Now we can train our model. For optimizing the ELBO, we use the Adam Optimizer *(Kingma and Ba 2015)* which is designed for stochastic objective functions. We create a `run_adam` utility function  to perform the optimization.

```python
minibatch_size = 100

# We turn off training for inducing point locations
gpflow.set_trainable(m.inducing_variable, False)

def run_adam(model, iterations):
    """
    Utility function running the Adam optimizer

    :param model: GPflow model
    :param interations: number of iterations
    """
    # Create an Adam Optimizer action
    logf = []
    train_iter = iter(train_dataset.batch(minibatch_size))
    training_loss = model.training_loss_closure(train_iter, compile=True)
    optimizer = tf.optimizers.Adam()

    @tf.function
    def optimization_step():
        optimizer.minimize(training_loss, model.trainable_variables)

    for step in range(iterations):
        optimization_step()
        if step % 10 == 0:
            elbo = -training_loss().numpy()
            logf.append(elbo)
    return logf
```

Now we run the optimization loop for 20,000 iterations.

```python
maxiter = ci_niter(20000)

logf = run_adam(m, maxiter)
plt.plot(np.arange(maxiter)[::10], logf)
plt.xlabel("iteration")
_ = plt.ylabel("ELBO")
```

Finally, we plot the model's predictions.

```python
plot("Predictions after training")
```

## Further reading

Several notebooks expand on this one:

- [Advanced Sparse GP regression](../advanced/advanced_many_points.ipynb), which goes into deeper detail on sparse Gaussian process methods.
- [Optimization](../advanced/optimisation.ipynb) discussing optimizing GP models.
- [Natural gradients](../advanced/natural_gradients.ipynb) for optimizing SVGP models efficiently.

## References:
Hensman, James, Nicolo Fusi, and Neil D. Lawrence. "Gaussian processes for big data." Uncertainty in Artificial Intelligence (2013).

Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).