# Belief propagation

> In this class, we study an alternative class of algorithms to the Variable Elimination algorithm, which is the message passing class of algorithms.
>
> Although the Message Passing algorithms are closely related to Variable Elimination in some ways, they also offer us additional flexibility in how we do the summation and factor product steps. This means that we may (potentially) come up with a lower complexity in some cases.

> **Objetives:**
> - To study the cluster graphs and how to pass messages between them.
> - To describe the Belief Propagation algorithm algorithm.
> - To analyze the properties of the Belief Propagation algorithm.
> - To study the Belief Propagation algorihm in clique trees.

> **References:**
> - Probabilistic Graphical Models: Principles and Techniques, By Daphne Koller and Nir Friedman. Ch. 10.
> - Mastering Probabilistic Graphical Models Using Python, By Ankur Ankan and Abinash Panda. Ch. 3.
> - Probabilistic Graphical Models Specialization, offered through Coursera. Prof. Daphne Koller.

 Imagen recuperada de: https://static.thenounproject.com/png/542457-200.png.

___

> In the last class we studied how we can exploit the graphical structure to come up with an (Variable Elimination) algorithm that performs exact inference in an effective way.
>
> The main insight behind the VE algorithm is that the factorization of the distribution allows us to perform local operations on the factors.
>
> In this class, we exploit the same insight with a different class of algorithms.

As in the previous class, we assume that we are dealing with a set of factors we have a set of factors $\bar{\Phi}=\{\phi_1(\bar{X}_1), \dots, \phi_m(\bar{X}_m)\}$ over $\bar{X}$, that define an (unnormalized) measure:

$$\tilde{P}_{\bar{\Phi}}(\bar{X}) = \prod_{\phi_i\in\bar{\Phi}} \phi_i(\bar{X}_i).$$

# 1. Cluster Graph

## 1. Cluster graphs

We begin by defining the *cluster graphs*:

> *Definition.* A **cluster graph** $\mathcal{U}$ for a set of factors $\bar{\Phi}$ over $\bar{X}$ is an undirected graph, whose nodes, say $i$, are associated with a subset of variables, say $\bar{C}_i\subset\bar{X}$, which we call clusters.
>
> **Family preservation property:** Moreover, each factor $\phi_k\in\bar{\Phi}$ must be associated one and only one cluster $\bar{C}_{\alpha(k)}$ such that $\mathrm{scope}[\phi_k]\subseteq \bar{C}_{\alpha(k)}$.
>
> Each edge between a pair of clusters $\bar{C}_i$ and $\bar{C}_j$ is associated with a **sepset** $\bar{S}_{ij}\subseteq \bar{C}_i\cap\bar{C}_j$.

**Example.** Let's consider the pairwise MN:

```python
from IPython.display import Image
```

```python
Image("figures/clusterGraph.png")
```

**Example.** Consider following set of factors:

$$\bar{\Phi}=\{\phi_1(A,B,C), \phi_2(B,C), \phi_3(B,D), \phi_4(D,E), \phi_5(B,E), \phi_6(B,D), \phi_7(B,D,F)\}.$$

Then, possible cluster graphs are ($\alpha(1)=1$, $\alpha(2)=2$, $\alpha(3)=2$, $\alpha(4)=5$, $\alpha(5)=4$, $\alpha(6)=3$, $\alpha(7)=3$):

```python
Image("figures/clusterGraph2.png")
```

```python
Image("figures/clusterGraph3.png")
```

- The idea behind the belief propagation algorithm is that clusters will "talk" to each other;
- They are going to try to convince each other that what they think about a variable is correct.

## 1.2. Belief propagation algorithm

Now that we know what a cluster graph is, we will describe the Belief Propagation algorithm:

1. Assign each factor $\phi_k \in \bar{\Phi}$ to a cluster $C_{\alpha(k)}$.

2. Construct initial potentials $\psi_i(\bar{C_i}) = \prod_{k: \alpha(k)=i} \phi_k(\bar{X}_k)$.

3. Initialize all messages $\delta_{ij}(\bar{S}_{ij})=1$.

4. Repeat:
   - Select edge (i,j) and pass message
   $$\delta_{ij}(\bar{S}_{ij}) = \sum_{\bar{C}_i\setminus\bar{S}_{ij}} \psi_i(\bar{C}_i) \prod_{k\in\mathcal{N}_i \setminus \{j\}} \delta_{ki}(\bar{S}_{ki}),$$
   where $\mathcal{N}_i$ are the set of neighboring nodes of node $i$.

   Explain

5. Compute beliefs:

   $$\beta_i(\bar{C}_i) = \psi_i(\bar{C}_i) \prod_{k\in\mathcal{N}_i} \delta_{ki}(\bar{S}_{ki})$$

### Important

- This algorithm doesn't converge always.
- When it does, it's only approximate, in the general case.

## 1.3. Properties of cluster graphs

Until now, we know what the cluster graphs are, and how to pass messages between clusters to run the belief propagation algorithm.

However, we don't know how to construct cluster graphs given a PGM yet. To this end, let's examine some properties that cluster graphs need to satisfy in order to support message passing:

**1. Family preservation**

Given a set of factors $\bar{\Phi}$, we assign each factor $\phi_k\in\bar{\Phi}$ to one and only one cluster $\bar{C}_{\alpha(k)}$ such that $\mathrm{scope}[\phi_k]\subseteq \bar{C}_{\alpha(k)}$.

- *All the information is used only once.*

**2. Running intersection property**

For each pair of clusters $\bar{C}_i, \bar{C}_j$, and a variable $X \in \bar{C}_i \cap \bar{C}_j$, there exists a unique path between $\bar{C}_i$ and $\bar{C}_j$ for which all clusters and sepsets contain $X$.

- *The existence of such a path porevents the isolation of information about a variable*.
- *The uniqueness prevents self-reinforcement feedback loops*.

Equivalently: For any $X$, the set of clusters and sepsets that contain $X$ form a **tree**.

**Example.** Consider again the set of factors:

$$\bar{\Phi}=\{\phi_1(A,B,C), \phi_2(B,C), \phi_3(B,D), \phi_4(D,E), \phi_5(B,E), \phi_6(B,D), \phi_7(B,D,F)\}.$$

$\alpha(1)=1$, $\alpha(2)=2$, $\alpha(3)=2$, $\alpha(4)=5$, $\alpha(5)=4$, $\alpha(6)=3$, $\alpha(7)=3$.

Then, the following cluster graph satisfies both, the family preservation and the running intersection properties.

```python
Image("figures/clusterGraph2.png")
```

What about this one?

```python
Image("figures/clusterGraph4.png")
```

Is it a valid cluster graph?

- No, because the running intersection property does not hold. For instance, taking clusters 1 and 2, and $B\in \bar{C}_1\cap \bar{C}_2$, there is not a unique path between clusters 1 and 2 such that all clusters and sepsets contain $B$ (paths: 1-2, 1-4-2).

And this one?

```python
Image("figures/clusterGraph5.png")
```

Is it a valid cluster graph?

- No. Again, the running intersection property does not hold. This time, there does not exist a path between $\bar{C}_1$ and $\bar{C}_2$ such that all clusters and sepsets contain $B$.

**Bethe cluster graph**

This type of cluster graphs always satisfy both properties.

1. For each factor $\phi_k\in\bar{\Phi}$, assign a cluster $\bar{C}_k$ such that $\mathrm{scope}[\phi_k]\subseteq \bar{C}_k$.
2. For each variable $X_i$ assign a singleton cluster $\{X_i\}$.
3. Add an edge $\bar{C}_k - X_i$ if $X_i\in\bar{C}_k$.

For the example we were analyzing before (in the whiteboard, then show):

```python
# clusterGraph6
Image("figures/clusterGraph6.png")
```

Although the correlations between factors is lost, the Bethe cluster graph provides an easy way to construct cluster graphs satisfying the properties of family preservation and running intersection.

## 1.4. Properties of belief propagation

Now, we turn our attention to some properties that will allow to study convergence of the belief propagation algorithm.

**1. Calibration**

Recall the pairwise MN:

```python
Image("figures/clusterGraph.png")
```

The cluster beliefs are defined as:

$$\beta_i(\bar{C}_i) = \psi_i(\bar{C}_i) \prod_{k\in\mathcal{N}_i}\delta_{ki}(\bar{S}_{ki})$$

For instance,

$$\beta_1(A,B) = \psi_1(A,B) \delta_{41}(A) \delta_{21}(B)$$

> *Definition.* A cluster graph is **calibrated** if every pair of adjacent clusters $\bar{C}_i, \bar{C}_j$ agree on their sepset $\bar{S}_{ij}$:
>
> $$\sum_{\bar{C_i}\setminus\bar{S}_{ij}}\beta_i(\bar{C}_i) = \sum_{\bar{C_j}\setminus\bar{S}_{ij}}\beta_j(\bar{C}_j).$$

This is, the cluster graph is calibrated if the marginal beliefs agree.

**2. Convergence**

The convergence property refers to an *steady state achievement*.

> *Definition.* A cluster graph has attained **convergence** if all the messages at the next time step equal the messages at the previous time step:
>
> $$\delta_{ij}(\bar{S}_{ij}) = \delta_{ij}'(\bar{S}_{ij})$$

One important result is that **convergence implies calibration** (see in the whiteboard, then show)

**3. Reparametrization**

This property state that there is no loss of information in the message passing process.

To see this, recall that:

$$\beta_i(\bar{C}_i) = \psi_i(\bar{C}_i) \prod_{k\in\mathcal{N}_i}\delta_{ki}(\bar{S}_{ki}),$$

and

$$\mu_{ij}(\bar{S}_{ij}) = \delta_{ij}(\bar{S}_{ij}) \delta_{ji}(\bar{S}_{ji}).$$

Then,

$$\frac{\prod_{i}\beta_i(\bar{C}_i)}{\prod_{i,j} \mu_{ij}(\bar{S}_{ij})} = \frac{\prod_{i}\psi_i(\bar{C}_i) \prod_{j\in\mathcal{N}_i}\delta_{ji}(\bar{S}_{ji})}{\prod_{i,j} \delta_{ij}(\bar{S}_{ij})} = \prod_{i}\psi_i(\bar{C}_i) = \tilde{P}_{\bar{\Phi}}(\bar{X}).$$

___
# 2. Belief Propagation in Clique Trees

We know now that in the general case, the belief propagation algorithm don't work always.

However, when applied over *clique trees*, the belief propagation algorithm has stronger performance guarantees:
- Faster convergence.
- Convergence to exact inference.

## 2.1. Message passing in trees

Consider the following chain:

```python
Image("figures/chain.png")
```

The following is a valid tree cluster graph for the above chain:

```python
Image("figures/clusterGraphChain.png")
```

**How is the form of all the messages that will be passed over this tree cluster graph?**

See in the whiteboard, then show.

```python
# clusterGraphChain2
Image("figures/clusterGraphChain2.png")
```

Thus, it is not hard to realize that after a single run of the belief propagation algorithm in both directions, the beliefs converge to the correct marginals.

For instance, consider cluster 3:

$$\begin{align}
\beta_3(C,D) & = \psi_3(C,D) \delta_{23}(C) \delta_{43}(D) \\
             & = \psi_3(C,D) \sum_{B} \psi_2(B,C) \delta_{12}(B) \sum_{E} \psi_4(D,E)\\
             & = \psi_3(C,D) \sum_{B} \psi_2(B,C) \sum_{A} \psi_1(A,B) \sum_{E} \psi_4(D,E)\\
             & = \tilde{P}_{\bar{\Phi}}(C,D)
\end{align}$$

## 2.2. Clique trees

The behavior in the chain cluster graph is also attained by *clique trees* structures.

> *Definition.* A **clique tree** is an undirected tree such that:
>
> 1. Its nodes are clusters $\bar{C}_i \subset \bar{X}$, which correspond to *cliques* in the original graphical model.
> 2. The edges between $\bar{C}_i$ and $\bar{C}_j$ are associated with the sepset $\bar{S}_{ij}=\bar{C}_i \cap \bar{C}_j$.

What is a *clique*?

> *Definition.* A **clique** is a complete subgraph. This is, a portion of the complete graph where every two nodes are connected by some edge.

When talking about cliques, we always consider the **induced undirected network** (Section 9.4.2.3, page 308).

Of course, since clique trees are a special case of cluster graphs, they should satisfy both, the family preservation and the running intersection properties.

The **running intersection property for clique trees** can be rewritten as:

- For each pair of clusters $\bar{C}_i, \bar{C}_j$ and a variable $X\in\bar{C}_i \cap \bar{C}_j$, in the unique path between $\bar{C}_i$ and $\bar{C}_j$, all clusters and sepsets contain $X$.

**Example.**

Consider the extended student network:

```python
Image("figures/extendedStudent.png")
```

The induced graph for the bove Bayesian network is:

```python
Image("figures/extendedStudentInduced.png")
```

Hence, a valid clique tree for this model is:

```python
Image("figures/extendedStudentCliqueTree.png")
```

What is the form of the messages?

### Running intersection property implies correctness

- If a variable $X$ is marginalized (eliminated) when we pass the message $\delta_{ij}$, then $X$ does not appear in the $\bar{C}_j$ side of the clique tree.

- So, in this case, the computation of the **belief propagation algorithm is a variant of the variable elimination algorithm**, and the resulting beliefs are guaranteed to be correct marginals of $\tilde{P}_{\bar{\Phi}}$.

## 2.3. Clique tree algorithm computation

Let's consider the chain example:

```python
Image("figures/chain.png")
```

```python
Image("figures/clusterGraphChain2.png")
```

One can note some things:
1. Once you compute $\delta_{12}$, it never changes (instant convergence).

2. For $\delta_{23}$, wait for $\delta_{12}$, then calculate and you get convergence.

3. For $\delta_{34}$, wait for $\delta_{23}$, then calculate and you get convergence.

4. The same happens in the opposite direction.

That is why this is known as **forward-backward algorithm**:

- Once $\bar{C}_i$ receives a final message from all neighbors, except $\bar{C}_j$, then $\delta_{ij}(\bar{S}_{ij})$ is also final.

- Messages from leaves of the tree are immediately final.

- If messages are passed in the right order, you only need to pass $2(k-1)$ messages, where $k$ is the number of cluster.

**Thus, you can compute the marginals over all variables at only twice the cost of one variable elimination over all variables.**

___

How to use belief propagation algorithm in pgmpy?

Restaurant example

```python
Image("figures/restaurant.png")
```

```python
# Import pgmpy.models.BayesianModel

```

```python
# Import pgmpy.factors.discrete.TabularCPD

```

```python
# Define model skeleton

```

```python
# Define CPDs

```

```python
# Attach CPDs to the model

```

```python
# Check if the model is correctly defined

```

```python
# Get a valid clique tree

```

```python
# Type

```

```python
# Nodes

```

```python
# Edges

```

```python
# Import pgmpy.inference.BeliefPropagation

```

```python
# Create inference object

```

```python
# Calibrate

```

```python
# Get beliefs

```

```python

```

```python

```

```python
# Perform query

```

___

## 2.4. Clique trees and variable elimination

In the variable elimination algorithm we have:

1. Each steps creates a factor $\lambda_k$ through factor product.

2. A variable is eliminated in $\lambda_k$ to generate the new factor $\tau_k$.

3. $\tau_k$ is used in the computation of other factors $\lambda_j$.

From a clique tree perspective:

1. The intermediate factors $\lambda_k$ are the "cliques".

2. $\tau_k$ are "messages" generated by the clique $\lambda_k$ and transmitted to another clique $\lambda_j$.

Thus, the variable elimination algorithm defines the clique tree!

- Define a cluster $\bar{C}_i$ for each factor $\lambda_i$ used in the computation.

- Draw an edge $\bar{C}_i - \bar{C}_j$ if the factor generated from $\lambda_i$ is used in the computation of $\lambda_j$.

```python
Image("figures/cliqueTreeVE.png")
```

This clique tree satisfies both, the family preserving property and the running intersection property.

**Example**

For the restaurant example of the last class:

$$P(L,Q,C,N)= \phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C).$$

1. Eliminate $Q$: $\tau_1(C,L) = \sum_{Q} \underbrace{\phi_Q(Q) \phi_C(C,L,Q)}_{\lambda_1(C,L,Q)}$.

2. Eliminate $C$: $\tau_2(N,L) = \sum_{C} \underbrace{\phi_N(N,L,C)\tau_1(C,L)}_{\lambda_2(N,L,C)}$.

3. Eliminate $L$: $\tau_3(N) = \sum_{L} \underbrace{\phi_L(L)\tau_2(N,L)}_{\lambda_3(N,L)}$.

4. Eliminate $N$: $\tau_4() = \sum_{N} \tau_3(N)$.

```python
Image("figures/cliqueTreeVE2.png")
```

Remove redundant cliques: variables that are subsets of adjacent cliques.

# Announcements

## Quiz.

Created with Jupyter by Esteban Jiménez Rodríguez.


[Sebastian Raschka](http://sebastianraschka.com)

- [Open in IPython nbviewer](http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/tutorials/python_data_entry_point.ipynb?create=1)

- [Link to this IPython notebook on Github](https://github.com/rasbt/python_reference/blob/master/tutorials/python_data_entry_point.ipynb)

- [Link to the GitHub Repository pattern_classification](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/python_howtos/)

```
%load_ext watermark
```

```
%watermark -a 'Sebastian Raschka' -v -d -p numpy,scipy,matplotlib,scikit-learn
```

    Sebastian Raschka 10/07/2014

    CPython 3.4.1
    IPython 2.1.0

    numpy 1.8.1
    scipy 0.14.0
    matplotlib 1.3.1
    scikit-learn 0.15.0b1

[More information](http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/ipython_magic/watermark.ipynb) about the `watermark` magic command extension.

I would be happy to hear your comments and suggestions.
Please feel free to drop me a note via
[twitter](https://twitter.com/rasbt), [email](mailto:bluewoodtree@gmail.com), or [google+](https://plus.google.com/+SebastianRaschka).

# Entry point: Data

### - Using Python's sci-packages to prepare data for Machine Learning tasks and other data analyses

In this short tutorial I want to provide a short overview of some of my favorite Python tools for common procedures as entry points for general pattern classification and machine learning tasks, and various other data analyses.

# Sections

- [Installing Python packages](#Installing-Python-packages)

- [About the dataset](#About-the-dataset)

- [Downloading and saving CSV data files from the web](#Downloading-and-savin-CSV-data-files-from-the-web)

- [Reading in a dataset from a CSV file](#Reading-in-a-dataset-from-a-CSV-file)

- [Visualizating of a dataset](#Visualizating-of-a-data)

    - [Histograms](#Histograms)

    - [Scatterplots](#Scatterplots)

- [Splitting into training and test dataset](#Splitting-into-training-and-test-dataset)

- [Feature Scaling](#Feature-Scaling)

    - [Standardization](#Standardization)

    - [Min-Max scaling (Normalization)](#Min-Max-scaling-Normalization)

- [Linear Transformation: Principal Component Analysis (PCA)](#PCA)

- [Linear Transformation: Linear Discrciminant Analysis (LDA)](#MDA)

- [Simple Supervised Classification](#Simple-Supervised-Classification)

    - [Linear Discriminant Analysis as simple linear classifier](#Linear-Discriminant-Analysis-as-simple-linear-classifier)

    - [Classification Stochastic Gradient Descent (SGD)](#SGD)

- [Saving the processed datasets](#Saving-the-processed-datasets)

    - [Pickle](#Pickle)

    - [Comma Separated Values (CSV)](#Comma-Separated-Values)

## Installing Python packages

[[back to top]](#Sections)

**In this section want to recommend a way for installing the required Python-packages packages if you have not done so, yet. Otherwise you can skip this part.**

The packages we will be using in this tutorial are:

- [NumPy](http://www.numpy.org)
- [SciPy](http://www.scipy.org)
- [matplotlib](http://matplotlib.org)
- [scikit-learn](http://scikit-learn.org/stable/)

Although they can be installed step-by-step "manually", but I highly recommend you to take a look at the [Anaconda](https://store.continuum.io/cshop/anaconda/) Python distribution for scientific computing.

Anaconda is distributed by Continuum Analytics, but it is completely free and includes more than 195+ packages for science and data analysis as of today.
The installation procedure is nicely summarized here: http://docs.continuum.io/anaconda/install.html

If this is too much, the [Miniconda](http://conda.pydata.org/miniconda.html) might be right for you. Miniconda is basically just a Python distribution with the Conda package manager, which let's us install a list of Python packages into a specified `conda` environment from the Shell terminal, e.g.,

$[bash]> conda create -n myenv python=3
$[bash]> source activate myenv
$[bash]> conda install -n myenv numpy scipy matplotlib scikit-learn

When we start "python" in your current shell session now, it will use the Python distribution in the virtual environment "myenv" that we have just created. To un-attach the virtual environment, you can just use

$[bash]> source deactivate myenv

**Note:** environments will be created in ROOT_DIR/envs by default, you can use the `-p` instead of the `-n` flag in the conda commands above in order to specify a custom path.

**I find this procedure very convenient, especially if you are working with different distributions and versions of Python with different modules and packages installed and it is extremely useful for testing your own modules.**

## About the dataset

[[back to top]](#Sections)

For the following tutorial, we will be working with the free "Wine" Dataset that is deposited on the UCI machine learning repository
(http://archive.ics.uci.edu/ml/datasets/Wine).

**Reference:**
Forina, M. et al, PARVUS - An Extendible Package for Data
Exploration, Classification and Correlation. Institute of Pharmaceutical
and Food Analysis and Technologies, Via Brigata Salerno,
16147 Genoa, Italy.

Bache, K. & Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

The Wine dataset consists of 3 different classes where each row correspond to a particular wine sample.

The class labels (1, 2, 3) are listed in the first column, and the columns 2-14 correspond to the following 13 attributes (features):

1) Alcohol
2) Malic acid
3) Ash
4) Alcalinity of ash
5) Magnesium
6) Total phenols
7) Flavanoids
8) Nonflavanoid phenols
9) Proanthocyanins
10) Color intensity
11) Hue
12) OD280/OD315 of diluted wines
13) Proline

An excerpt from the wine_data.csv dataset:

1,14.23,1.71,2.43,15.6,127,2.8,3.06,.28,2.29,5.64,1.04,3.92,1065
1,13.2,1.78,2.14,11.2,100,2.65,2.76,.26,1.28,4.38,1.05,3.4,1050
[...]
2,12.37,.94,1.36,10.6,88,1.98,.57,.28,.42,1.95,1.05,1.82,520
2,12.33,1.1,2.28,16,101,2.05,1.09,.63,.41,3.27,1.25,1.67,680
[...]
3,12.86,1.35,2.32,18,122,1.51,1.25,.21,.94,4.1,.76,1.29,630
3,12.88,2.99,2.4,20,104,1.3,1.22,.24,.83,5.4,.74,1.42,530

## Downloading and saving CSV data files from the web

[[back to top]](#Sections)

Usually, we have our data stored locally on our disk in as a common text (or CSV) file with comma-, tab-, or whitespace-separated rows. Below is just an example for how you can CSV datafile from a HTML website directly into Python and optionally save it locally.

```
import csv
import urllib

url = 'https://raw.githubusercontent.com/rasbt/pattern_classification/master/data/wine_data.csv'
csv_cont = urllib.request.urlopen(url)
csv_cont = csv_cont.read() #.decode('utf-8')

# Optional: saving the data to your local drive
with open('./wine_data.csv', 'wb') as out:
    out.write(csv_cont)
```

**Note:** If you'd rather like to work with the data directly in `str`ing format, you could just apply the `.decode('utf-8')` method to the data that was read in byte-format by default.

## Reading in a dataset from a CSV file

[[back to top]](#Sections)

Since it is quite typical to have the input data stored locally, as mentioned above, we will use the [`numpy.loadtxt`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html) function  now to read in the data from the CSV file.
(alternatively [`np.genfromtxt()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html) could be used in similar way, it provides some additional options)

```
import numpy as np

# reading in all data into a NumPy array
all_data = np.loadtxt(open("./wine_data.csv","r"),
        delimiter=",",
        skiprows=0,
        dtype=np.float64
        )

# load class labels from column 1
y_wine = all_data[:,0]

# conversion of the class labels to integer-type array
y_wine = y_wine.astype(np.int64, copy=False)

# load the 14 features
X_wine = all_data[:,1:]

# printing some general information about the data
print('\ntotal number of samples (rows):', X_wine.shape[0])
print('total number of features (columns):', X_wine.shape[1])

# printing the 1st wine sample
float_formatter = lambda x: '{:.2f}'.format(x)
np.set_printoptions(formatter={'float_kind':float_formatter})
print('\n1st sample (i.e., 1st row):\nClass label: {:d}\n{:}\n'
          .format(int(y_wine[0]), X_wine[0]))

# printing the rel.frequency of the class labels
print('Class label frequencies')
print('Class 1 samples: {:.2%}'.format(list(y_wine).count(1)/y_wine.shape[0]))
print('Class 2 samples: {:.2%}'.format(list(y_wine).count(2)/y_wine.shape[0]))
print('Class 3 samples: {:.2%}'.format(list(y_wine).count(3)/y_wine.shape[0]))
```

    total number of samples (rows): 178
    total number of features (columns): 13

    1st sample (i.e., 1st row):
    Class label: 1
    [14.23 1.71 2.43 15.60 127.00 2.80 3.06 0.28 2.29 5.64 1.04 3.92 1065.00]

    Class label frequencies
    Class 1 samples: 33.15%
    Class 2 samples: 39.89%
    Class 3 samples: 26.97%

## Visualizating of a dataset

[[back to top]](#Sections)

There are endless way to visualize datasets for get an initial idea of how the data looks like. The most common ones are probably histograms and scatter plots.

### Histograms

[[back to top]](#Sections)

Histograms are a useful data to explore the distribution of each feature across the different classes. This could provide us with intuitive insights which features have a good and not-so-good inter-class separation. Below, we will plot a sample histogram for the "Alcohol content" feature for the three wine classes.

```
%matplotlib inline
```

```
from matplotlib import pyplot as plt
from math import floor, ceil # for rounding up and down

plt.figure(figsize=(10,8))

# bin width of the histogram in steps of 0.15
bins = np.arange(floor(min(X_wine[:,0])), ceil(max(X_wine[:,0])), 0.15)

# get the max count for a particular bin for all classes combined
max_bin = max(np.histogram(X_wine[:,0], bins=bins)[0])

# the order of the colors for each histogram
colors = ('blue', 'red', 'green')

for label,color in zip(
        range(1,4), colors):

    mean = np.mean(X_wine[:,0][y_wine == label]) # class sample mean
    stdev = np.std(X_wine[:,0][y_wine == label]) # class standard deviation
    plt.hist(X_wine[:,0][y_wine == label],
             bins=bins,
             alpha=0.3, # opacity level
             label='class {} ($\mu={:.2f}$, $\sigma={:.2f}$)'.format(label, mean, stdev),
             color=color)

plt.ylim([0, max_bin*1.3])
plt.title('Wine data set - Distribution of alocohol contents')
plt.xlabel('alcohol by volume', fontsize=14)
plt.ylabel('count', fontsize=14)
plt.legend(loc='upper right')

plt.show()
```

### Scatterplots

[[back to top]](#Sections)

Scatter plots are useful for visualizing features in more than just one dimension, for example to get a feeling for the correlation between particular features.
Unfortunately, we can't plot all 13 features here at once, since the visual cortex of us humans is limited to a maximum of three dimensions.

Below, we will create an example 2D-Scatter plot from the features "Alcohol content" and "Malic acid content".
Additionally, we will use the [`scipy.stats.pearsonr`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html) function to calculate a Pearson correlation coefficient between these two features.

```
from scipy.stats import pearsonr

plt.figure(figsize=(10,8))

for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue', 'red', 'green')):

    # Calculate Pearson correlation coefficient
    R = pearsonr(X_wine[:,0][y_wine == label], X_wine[:,1][y_wine == label])
    plt.scatter(x=X_wine[:,0][y_wine == label], # x-axis: feat. from col. 1
                y=X_wine[:,1][y_wine == label], # y-axis: feat. from col. 2
                marker=marker, # data point symbol for the scatter plot
                color=color,
                alpha=0.7,
                label='class {:}, R={:.2f}'.format(label, R[0]) # label for the legend
                )

plt.title('Wine Dataset')
plt.xlabel('alcohol by volume in percent')
plt.ylabel('malic acid in g/l')
plt.legend(loc='upper right')

plt.show()
```

If we want to pack 3 different features into one scatter plot at once, we can also do the same thing in 3D:

```
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111, projection='3d')

for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue','red','green')):

    ax.scatter(X_wine[:,0][y_wine == label],
               X_wine[:,1][y_wine == label],
               X_wine[:,2][y_wine == label],
               marker=marker,
               color=color,
               s=40,
               alpha=0.7,
               label='class {}'.format(label))

ax.set_xlabel('alcohol by volume in percent')
ax.set_ylabel('malic acid in g/l')
ax.set_zlabel('ash content in g/l')

plt.title('Wine dataset')

plt.show()
```

## Splitting into training and test dataset

[[back to top]](#Sections)

It is a typical procedure for machine learning and pattern classification tasks to split one dataset into two: a training dataset and a test dataset.
The training dataset is henceforth used to train our algorithms or classifier, and the test dataset is a way to validate the outcome quite objectively before we apply it to "new, real world data".

Here, we will split the dataset randomly so that 70% of the total dataset will become our training dataset, and 30% will become our test dataset, respectively.

```
from sklearn.cross_validation import train_test_split
from sklearn import preprocessing

X_train, X_test, y_train, y_test = train_test_split(X_wine, y_wine,
     test_size=0.30, random_state=123)
```

Note that since this a random assignment, the original relative frequencies for each class label are not maintained.

```
print('Class label frequencies')

print('\nTraining Dataset:')
for l in range(1,4):
    print('Class {:} samples: {:.2%}'.format(l, list(y_train).count(l)/y_train.shape[0]))

print('\nTest Dataset:')
for l in range(1,4):
    print('Class {:} samples: {:.2%}'.format(l, list(y_test).count(l)/y_test.shape[0]))
```

    Class label frequencies

    Training Dataset:
    Class 1 samples: 36.29%
    Class 2 samples: 42.74%
    Class 3 samples: 20.97%

    Test Dataset:
    Class 1 samples: 25.93%
    Class 2 samples: 33.33%
    Class 3 samples: 40.74%

## Feature Scaling

[[back to top]](#Sections)

### Standardization

Another important procedure is to standardize the data prior to fitting the model and other analyses so that the features will have the properties of a standard normal distribution with

$\mu = 0$ and $\sigma = 1$

where $\mu$ is the mean (average) and $\sigma$ is the standard deviation from the mean; standard scores (also called ***z*** scores) of the samples are calculated as follows:

\begin{equation} z = \frac{x - \mu}{\sigma}\end{equation}

Standardizing the features so that they are centered around 0 with a standard deviation of 1 is especially important if we are comparing measurements that have different units, e.g., in our "wine data" example, where the alcohol content is measured in volume percent, and the malic acid content in g/l.

```
std_scale = preprocessing.StandardScaler().fit(X_train)
X_train = std_scale.transform(X_train)
X_test = std_scale.transform(X_test)
```

```
f, ax = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(10,5))

for a,x_dat, y_lab in zip(ax, (X_train, X_test), (y_train, y_test)):

    for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue','red','green')):

        a.scatter(x=x_dat[:,0][y_lab == label],
                y=x_dat[:,1][y_lab == label],
                marker=marker,
                color=color,
                alpha=0.7,
                label='class {}'.format(label)
                )

    a.legend(loc='upper left')

ax[0].set_title('Training Dataset')
ax[1].set_title('Test Dataset')
f.text(0.5, 0.04, 'malic acid (standardized)', ha='center', va='center')
f.text(0.08, 0.5, 'alcohol (standardized)', ha='center', va='center', rotation='vertical')

plt.show()
```

### Min-Max scaling (Normalization)

[[back to top]](#Sections)

An alternative approach to standardization is the so-called Min-Max scaling (sometimes also referred to as "normalization").
In this approach, the data is scaled to a fixed range - usually 0 to 1.
The cost of having this bounded range - in contrast to standardization - is that we will end up with small standard deviations, for example in the case where outliers are present.

The equation to calculate the "normalized" score is:

\begin{equation} X' = \frac{X - X_{min}}{X_{max}-X_{min}} \end{equation}

```
minmax_scale = preprocessing.MinMaxScaler(feature_range=(0, 1)).fit(X_train)
X_train_minmax = minmax_scale.transform(X_train)
X_test_minmax = minmax_scale.transform(X_test)
```

```
f, ax = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(10,5))

for a,x_dat, y_lab in zip(ax, (X_train_minmax, X_test_minmax), (y_train, y_test)):

    for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue','red','green')):

        a.scatter(x=x_dat[:,0][y_lab == label],
                y=x_dat[:,1][y_lab == label],
                marker=marker,
                color=color,
                alpha=0.7,
                label='class {}'.format(label)
                )

    a.legend(loc='upper left')

ax[0].set_title('Training Dataset')
ax[1].set_title('Test Dataset')
f.text(0.5, 0.04, 'malic acid (normalized)', ha='center', va='center')
f.text(0.08, 0.5, 'alcohol (normalized)', ha='center', va='center', rotation='vertical')

plt.show()
```

## Linear Transformation: Principal Component Analysis (PCA)

[[back to top]](#Sections)

The main purposes of a principal component analysis are the analysis of data to identify patterns and finding patterns to reduce the dimensions of the dataset with minimal loss of information.

Here, our desired outcome of the principal component analysis is to project a feature space (our dataset consisting of n x d-dimensional samples) onto a smaller subspace that represents our data "well". A possible application would be a pattern classification task, where we want to reduce the computational costs and the error of parameter estimation by reducing the number of dimensions of our feature space by extracting a subspace that describes our data "best".

If you are interested in the Principal Component Analysis in more detail, I have outlined the procedure in a separate article
["Implementing a Principal Component Analysis (PCA) in Python step by step](http://sebastianraschka.com/Articles/2014_pca_step_by_step.html).

Here, we will use the [`sklearn.decomposition.PCA`](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) to transform our training data onto 2 dimensional subspace:

```
from sklearn.decomposition import PCA
sklearn_pca = PCA(n_components=2) # number of components to keep
sklearn_transf = sklearn_pca.fit_transform(X_train)

plt.figure(figsize=(10,8))

for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue', 'red', 'green')):

    plt.scatter(x=sklearn_transf[:,0][y_train == label],
                y=sklearn_transf[:,1][y_train == label],
                marker=marker,
                color=color,
                alpha=0.7,
                label='class {}'.format(label)
                )

plt.xlabel('vector 1')
plt.ylabel('vector 2')

plt.legend()
plt.title('Most significant singular vectors after linear transformation via PCA')

plt.show()
```

#### PCA for feature extraction

As mentioned in the short introduction above (and in more detail in my separate [PCA article](http://sebastianraschka.com/Articles/2014_pca_step_by_step.html)), PCA is commonly used in the field of pattern classification for feature selection (or dimensionality reduction).
By default, the transformed data will be ordered by the components with the maximum variance (in descending order).

In the example above, I only kept the top 2 components (the 2 components with the maximum variance along the axes): The sample space of projected onto a 2-dimensional subspace, which was basically sufficient for plotting the data onto a 2D scatter plot.

However, if we want to use PCA for feature selection, we probably don't want to reduce the dimensionality that drastically. By default, the `PCA` function (`PCA(n_components=None)`) keeps all the components in ranked order. So we could basically either set the number `n_components` to a smaller size then the input dataset, or we could extract the top **n** components later from the returned NumPy array.

To get an idea about how well each components (relatively) "explains" the variance, we can use `explained_variance_ratio_` instant method, which also confirms that the components are ordered from most explanatory to least explanatory (the ratios sum up to 1.0).

```
sklearn_pca = PCA(n_components=None)
sklearn_transf = sklearn_pca.fit_transform(X_train)
sklearn_pca.explained_variance_ratio_
```

    array([0.36, 0.21, 0.10, 0.08, 0.06, 0.05, 0.04, 0.03, 0.02, 0.02, 0.01,
           0.01, 0.01])

## Linear Transformation: Linear Discriminant Analysis (MDA)

[[back to top]](#Sections)

The main purposes of a Linear Discriminant Analysis (LDA) is to analyze the data to identify patterns to project it onto a subspace that yields a better separation of the classes. Also, the dimensionality of the dataset shall be reduced with minimal loss of information.

**The approach is very similar to a Principal Component Analysis (PCA), but in addition to finding the component axes that maximize the variance of our data, we are additionally interested in the axes that maximize the separation of our classes (e.g., in a supervised pattern classification problem)**

Here, our desired outcome of the Linear discriminant analysis is to project a feature space (our dataset consisting of n d-dimensional samples) onto a smaller subspace that represents our data "well" and has a good class separation. A possible application would be a pattern classification task, where we want to reduce the computational costs and the error of parameter estimation by reducing the number of dimensions of our feature space by extracting a subspace that describes our data "best".

#### Principal Component Analysis (PCA) Vs. Linear Discriminant Analysis (LDA)

Both Linear Discriminant Analysis and Principal Component Analysis are linear transformation methods and closely related to each other. In PCA, we are interested to find the directions (components) that maximize the variance in our dataset, where in LDA, we are additionally interested to find the directions that maximize the separation (or discrimination) between different classes (for example, in pattern classification problems where our dataset consists of multiple classes. In contrast two PCA, which ignores the class labels).

**In other words, via PCA, we are projecting the entire set of data (without class labels) onto a different subspace, and in LDA, we are trying to determine a suitable subspace to distinguish between patterns that belong to different classes. Or, roughly speaking in PCA we are trying to find the axes with maximum variances where the data is most spread (within a class, since PCA treats the whole data set as one class), and in LDA we are additionally maximizing the spread between classes.**

In typical pattern recognition problems, a PCA is often followed by an LDA.

If you are interested, you can find more information about the LDA in my IPython notebook
[Stepping through a Linear Discriminant Analysis - using Python's NumPy and matplotlib](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/minear_discriminant_analysis.ipynb?create=1).

Like we did in the PCA section above, we will use a `scikit-learn` funcion, [`sklearn.lda.LDA`](http://scikit-learn.org/stable/modules/generated/sklearn.lda.LDA.html) in order to transform our training data onto 2 dimensional subspace:

```
from sklearn.lda import LDA
sklearn_lda = LDA(n_components=2)
transf_lda = sklearn_lda.fit_transform(X_train, y_train)

plt.figure(figsize=(10,8))

for label,marker,color in zip(
        range(1,4),('x', 'o', '^'),('blue', 'red', 'green')):

    plt.scatter(x=transf_lda[:,0][y_train == label],
                y=transf_lda[:,1][y_train == label],
                marker=marker,
                color=color,
                alpha=0.7,
                label='class {}'.format(label)
                )

plt.xlabel('vector 1')
plt.ylabel('vector 2')

plt.legend()
plt.title('Most significant singular vectors after linear transformation via LDA')

plt.show()
```

#### LDA for feature extraction

If we want to use LDA for projecting our data onto a smaller subspace (i.e., for dimensionality reduction), we can directly set the number of components to keep via `LDA(n_components=...)`; this is analogous to the [PCA function](#PCA-for-feature-extraction), which we have seen above.

## Simple Supervised Classification

[[back to top]](#Sections)

### Linear Discriminant Analysis as simple linear classifier

[[back to top]](#Sections)

The LDA that we've just used in the section above can also be used as a simple linear classifier.

```
# fit model
lda_clf = LDA()
lda_clf.fit(X_train, y_train)
LDA(n_components=None, priors=None)

# prediction
print('1st sample from test dataset classified as:', lda_clf.predict(X_test[0,:]))
print('actual class label:', y_test[0])
```

    1st sample from test dataset classified as: [3]
    actual class label: 3

Another handy subpackage of sklearn is `metrics`. The [`metrics.accuracy_score`](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html), for example, is quite useful to evaluate how many samples can be classified correctly:

```
from sklearn import metrics
pred_train_lda = lda_clf.predict(X_train)

print('Prediction accuracy for the training dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_train, pred_train_lda)))
```

    Prediction accuracy for the training dataset
    100.00%

To verify that over model was not overfitted to the training dataset, let us evaluate the classifier's accuracy on the test dataset:

```
pred_test_lda = lda_clf.predict(X_test)

print('Prediction accuracy for the test dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_test, pred_test_lda)))
```

    Prediction accuracy for the test dataset
    98.15%

**Confusion Matrix**
As we can see above, there was a very low misclassification rate when we'd apply the classifier on the test data set. A confusion matrix can tell us in more detail which particular classes could not classified correctly.

	<colgroup width="60"></colgroup>
	<colgroup span="4" width="82"></colgroup>

		predicted class

		class 1
		class 2
		class 3

		actual class
		class 1
		True positives

		class 2

		True positives

		class 3

		True positives

```
print('Confusion Matrix of the LDA-classifier')
print(metrics.confusion_matrix(y_test, lda_clf.predict(X_test)))
```

    Confusion Matrix of the LDA-classifier
    [[14  0  0]
     [ 1 17  0]
     [ 0  0 22]]

As we can see, one sample from class 2 was incorrectly labeled as class 1, from the perspective of class 1, this would be 1 "False Negative" or a "False Postive" from the perspective of class 2, respectively

### Classification Stochastic Gradient Descent (SGD)

[[back to top]](#Sections)

Let us now compare the classification accuracy of the LDA classifier with a simple classification (we also use the probably not ideal default settings here) via stochastic gradient descent, an algorithm that minimizes a linear objective function.
More information about the `sklearn.linear_model.SGDClassifier` can be found [here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html).

```
from sklearn.linear_model import SGDClassifier

sgd_clf = SGDClassifier()
sgd_clf.fit(X_train, y_train)

pred_train_sgd = sgd_clf.predict(X_train)
pred_test_sgd = sgd_clf.predict(X_test)

print('\nPrediction accuracy for the training dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_train, pred_train_sgd)))

print('Prediction accuracy for the test dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_test, pred_test_sgd)))

print('Confusion Matrix of the SGD-classifier')
print(metrics.confusion_matrix(y_test, sgd_clf.predict(X_test)))
```

    Prediction accuracy for the training dataset
    99.19%

    Prediction accuracy for the test dataset
    100.00%

    Confusion Matrix of the SGD-classifier
    [[14  0  0]
     [ 0 18  0]
     [ 0  0 22]]

Quite impressively, we achieved a 100% prediction accuracy on the test dataset without any additional efforts of tweaking any parameters and settings.

### Decision Regions

```
sgd_clf2 = SGDClassifier()
sgd_clf2.fit(X_train[:, :2], y_train)

x_min = X_test[:, 0].min()
x_max = X_test[:, 0].max()
y_min = X_test[:, 1].min()
y_max = X_test[:, 1].max()

step = 0.01
X, Y = np.meshgrid(np.arange(x_min, x_max, step), np.arange(y_min, y_max, step))

Z = sgd_clf2.predict(np.c_[X.ravel(), Y.ravel()])
Z = Z.reshape(X.shape)

# Plots decision regions
plt.contourf(X, Y, Z)

# Plots samples from training data set
plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train)
plt.show()
```

## Saving the processed datasets

[[back to top]](#Sections)

### Pickle

[[back to top]](#Sections)

The in-built [`pickle`](https://docs.python.org/3.4/library/pickle.html) module is a convenient tool in Python's standard library to save Python objects in byte format. This allows us, for example, to save our NumPy arrays and classifiers so that we can load them in a later or different Python session to continue working with our data, e.g., to train a classifier.

```
# export objects via pickle

import pickle

pickle_out = open('standardized_data.pkl', 'wb')
pickle.dump([X_train, X_test, y_train, y_test], pickle_out)
pickle_out.close()

pickle_out = open('classifiers.pkl', 'wb')
pickle.dump([lda_clf, sgd_clf], pickle_out)
pickle_out.close()
```

```
# import objects via pickle

my_object_file = open('standardized_data.pkl', 'rb')
X_train, X_test, y_train, y_test = pickle.load(my_object_file)
my_object_file.close()

my_object_file = open('classifiers.pkl', 'rb')
lda_clf, sgd_clf = pickle.load(my_object_file)
my_object_file.close()

print('Confusion Matrix of the SGD-classifier')
print(metrics.confusion_matrix(y_test, sgd_clf.predict(X_test)))
```

    Confusion Matrix of the SGD-classifier
    [[14  0  0]
     [ 0 18  0]
     [ 0  0 22]]

### Comma-Separated-Values

[[back to top]](#Sections)

And it is also always a good idea to save your data in common text formats, such as the CSV format that we started with. But first, let us add back the class labels to the front column of the test and training data sets.

```
training_data = np.hstack((y_train.reshape(y_train.shape[0], 1), X_train))
test_data = np.hstack((y_test.reshape(y_test.shape[0], 1), X_test))
```

Now, we can save our test and training datasets as 2 separate CSV files using the [`numpy.savetxt`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.savetxt.html) function.

```
np.savetxt('./training_set.csv', training_data, delimiter=',')
np.savetxt('./test_set.csv', test_data, delimiter=',')
```


# Tutorial 2: Matrices
**Week 0, Day 3: Linear Algebra**

**By Neuromatch Academy**

__Content creators:__ Ella Batty

__Content reviewers:__ Keith van Antwerp, Aderogba Bayo, Anoop Kulkarni, Pooya Pakarian

__Production editors:__ Siddharth Suresh, Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

*Estimated timing of tutorial: 1 hour, 35 minutes*

During today, we will learn the basics of linear algebra, focusing on the topics that underlie the material on future days in the NMA Computational Neuroscience course. In this tutorial, we focus on matrices: their definition, their properties & operations, and especially on a geometrical intuition of them.

By the end of this tutorial, you will be able to :
* Explain matrices as a linear transformation and relate matrix properties to properties of that linear transformation
*  Perform matrix multiplication by hand
*  Define what eigenvalues/eigenvectors are

**Code Credit:**

Some elements of this problem set are from or inspired by https://openedx.seas.gwu.edu/courses/course-v1:GW+EngComp4+2019/about. In particular, we are using their `plot_linear_transformation` and `plot_linear_transformations` functions.

 Code under BSD 3-Clause License © 2019 Lorena A. Barba, Tingyu Wang. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

---
# Setup

```python
# Imports

import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure settings
import ipywidgets as widgets# interactive display
from ipywidgets import fixed
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Plotting functions
import numpy
from numpy.linalg import inv, eig
from math import ceil
from matplotlib import pyplot, ticker, get_backend, rc
from mpl_toolkits.mplot3d import Axes3D
from itertools import cycle

_int_backends = ['GTK3Agg', 'GTK3Cairo', 'MacOSX', 'nbAgg',
                 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo',
                 'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo']
_backend = get_backend()   # get current backend name

# shrink figsize and fontsize when using %matplotlib notebook
if _backend in _int_backends:
  fontsize = 4
  fig_scale = 0.75
else:
  fontsize = 5
  fig_scale = 1

grey = '#808080'
gold = '#cab18c'   # x-axis grid
lightblue = '#0096d6'  # y-axis grid
green = '#008367'  # x-axis basis vector
red = '#E31937'    # y-axis basis vector
darkblue = '#004065'

pink, yellow, orange, purple, brown = '#ef7b9d', '#fbd349', '#ffa500', '#a35cff', '#731d1d'

quiver_params = {'angles': 'xy',
                 'scale_units': 'xy',
                 'scale': 1,
                 'width': 0.012}

grid_params = {'linewidth': 0.5,
               'alpha': 0.8}

def set_rc(func):
  def wrapper(*args, **kwargs):
    rc('font', family='serif', size=fontsize)
    rc('figure', dpi=200)
    rc('axes', axisbelow=True, titlesize=5)
    rc('lines', linewidth=1)
    func(*args, **kwargs)
  return wrapper

@set_rc
def plot_vector(vectors, tails=None):
  ''' Draw 2d vectors based on the values of the vectors and the position of their tails.

  Parameters
  ----------
  vectors : list.
    List of 2-element array-like structures, each represents a 2d vector.

  tails : list, optional.
    List of 2-element array-like structures, each represents the coordinates of the tail
    of the corresponding vector in vectors. If None (default), all tails are set at the
    origin (0,0). If len(tails) is 1, all tails are set at the same position. Otherwise,
    vectors and tails must have the same length.

  Examples
  --------
  >>> v = [(1, 3), (3, 3), (4, 6)]
  >>> plot_vector(v)      # draw 3 vectors with their tails at origin
  >>> t = [numpy.array((2, 2))]
  >>> plot_vector(v, t)   # draw 3 vectors with their tails at (2,2)
  >>> t = [[3, 2], [-1, -2], [3, 5]]
  >>> plot_vector(v, t)   # draw 3 vectors with 3 different tails

  '''
  vectors = numpy.array(vectors)
  assert vectors.shape[1] == 2, "Each vector should have 2 elements."
  if tails is not None:
    tails = numpy.array(tails)
    assert tails.shape[1] == 2, "Each tail should have 2 elements."
  else:
    tails = numpy.zeros_like(vectors)

  # tile vectors or tails array if needed
  nvectors = vectors.shape[0]
  ntails = tails.shape[0]
  if nvectors == 1 and ntails > 1:
    vectors = numpy.tile(vectors, (ntails, 1))
  elif ntails == 1 and nvectors > 1:
    tails = numpy.tile(tails, (nvectors, 1))
  else:
    assert tails.shape == vectors.shape, "vectors and tail must have a same shape"

  # calculate xlimit & ylimit
  heads = tails + vectors
  limit = numpy.max(numpy.abs(numpy.hstack((tails, heads))))
  limit = numpy.ceil(limit * 1.2)   # add some margins

  figsize = numpy.array([2,2]) * fig_scale
  figure, axis = pyplot.subplots(figsize=figsize)
  axis.quiver(tails[:,0], tails[:,1], vectors[:,0], vectors[:,1], color=darkblue,
                  angles='xy', scale_units='xy', scale=1)
  axis.set_xlim([-limit, limit])
  axis.set_ylim([-limit, limit])
  axis.set_aspect('equal')

  # if xticks and yticks of grid do not match, choose the finer one
  xticks = axis.get_xticks()
  yticks = axis.get_yticks()
  dx = xticks[1] - xticks[0]
  dy = yticks[1] - yticks[0]
  base = max(int(min(dx, dy)), 1)   # grid interval is always an integer
  loc = ticker.MultipleLocator(base=base)
  axis.xaxis.set_major_locator(loc)
  axis.yaxis.set_major_locator(loc)
  axis.grid(True, **grid_params)

  # show x-y axis in the center, hide frames
  axis.spines['left'].set_position('center')
  axis.spines['bottom'].set_position('center')
  axis.spines['right'].set_color('none')
  axis.spines['top'].set_color('none')

@set_rc
def plot_transformation_helper(axis, matrix, *vectors, unit_vector=True, unit_circle=False, title=None):
  """ A helper function to plot the linear transformation defined by a 2x2 matrix.

  Parameters
  ----------
  axis : class matplotlib.axes.Axes.
    The axes to plot on.

  matrix : class numpy.ndarray.
    The 2x2 matrix to visualize.

  *vectors : class numpy.ndarray.
    The vector(s) to plot along with the linear transformation. Each array denotes a vector's
    coordinates before the transformation and must have a shape of (2,). Accept any number of vectors.

  unit_vector : bool, optional.
    Whether to plot unit vectors of the standard basis, default to True.

  unit_circle: bool, optional.
    Whether to plot unit circle, default to False.

  title: str, optional.
    Title of the plot.

  """
  assert matrix.shape == (2,2), "the input matrix must have a shape of (2,2)"
  grid_range = 20
  x = numpy.arange(-grid_range, grid_range+1)
  X_, Y_ = numpy.meshgrid(x,x)
  I = matrix[:,0]
  J = matrix[:,1]
  X = I[0]*X_ + J[0]*Y_
  Y = I[1]*X_ + J[1]*Y_
  origin = numpy.zeros(1)

  # draw grid lines
  for i in range(x.size):
    axis.plot(X[i,:], Y[i,:], c=gold, **grid_params)
    axis.plot(X[:,i], Y[:,i], c=lightblue, **grid_params)

  # draw (transformed) unit vectors
  if unit_vector:
    axis.quiver(origin, origin, [I[0]], [I[1]], color=green, **quiver_params)
    axis.quiver(origin, origin, [J[0]], [J[1]], color=red, **quiver_params)

  # draw optional vectors
  color_cycle = cycle([pink, darkblue, orange, purple, brown])
  if vectors:
    for vector in vectors:
      color = next(color_cycle)
      vector_ = matrix @ vector.reshape(-1,1)
      axis.quiver(origin, origin, [vector_[0]], [vector_[1]], color=color, **quiver_params)

  # draw optional unit circle
  if unit_circle:
    alpha =  numpy.linspace(0, 2*numpy.pi, 41)
    circle = numpy.vstack((numpy.cos(alpha), numpy.sin(alpha)))
    circle_trans = matrix @ circle
    axis.plot(circle_trans[0], circle_trans[1], color=red, lw=0.8)

  # hide frames, set xlimit & ylimit, set title
  limit = 4
  axis.spines['left'].set_position('center')
  axis.spines['bottom'].set_position('center')
  axis.spines['left'].set_linewidth(0.3)
  axis.spines['bottom'].set_linewidth(0.3)
  axis.spines['right'].set_color('none')
  axis.spines['top'].set_color('none')
  axis.set_xlim([-limit, limit])
  axis.set_ylim([-limit, limit])
  if title is not None:
    axis.set_title(title)

@set_rc
def plot_linear_transformation(matrix, *vectors, name = None, unit_vector=True, unit_circle=False):
  """ Plot the linear transformation defined by a 2x2 matrix using the helper
  function plot_transformation_helper(). It will create 2 subplots to visualize some
  vectors before and after the transformation.

  Parameters
  ----------
  matrix : class numpy.ndarray.
    The 2x2 matrix to visualize.

  *vectors : class numpy.ndarray.
    The vector(s) to plot along with the linear transformation. Each array denotes a vector's
    coordinates before the transformation and must have a shape of (2,). Accept any number of vectors.

  unit_vector : bool, optional.
    Whether to plot unit vectors of the standard basis, default to True.

  unit_circle: bool, optional.
    Whether to plot unit circle, default to False.

  """
  figsize = numpy.array([4,2]) * fig_scale
  figure, (axis1, axis2) = pyplot.subplots(1, 2, figsize=figsize)
  plot_transformation_helper(axis1, numpy.identity(2), *vectors, unit_vector=unit_vector, unit_circle=unit_circle, title='Before transformation')
  plot_transformation_helper(axis2, matrix, *vectors, unit_vector=unit_vector, unit_circle=unit_circle, title='After transformation')
  if name is not None:
    figure.suptitle(f'Population {name}')

def plot_eig_vec_transform(W):
  classic = 'k'
  vec_names = ['a', 'b','c','d','e','f','g', 'h']

  _, vecs = np.linalg.eig(W)
  vecs = vecs.T

  fig, axes = plt.subplots(1, 2, figsize=(2, 1))
  colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

  for i in range(2):
    axes[i].set(xlim=[-3.5, 3.5], ylim=[-3.5,3.5])
    axes[i].axis('Off')
    axes[i].plot([0, 0], [-3.5, 3.5], classic, alpha=.4)
    axes[i].plot([-3.5, 3.5], [0, 0], classic, alpha=.4)

  for i_vec, vec in enumerate(vecs):
    axes[0].arrow(0, 0, vec[0], vec[1], head_width=.2, facecolor=colors[i_vec], edgecolor=colors[i_vec], length_includes_head=True)
    axes[0].annotate(vec_names[i_vec], xy=(vec[0]+np.sign(vec[0])*.15, vec[1]+np.sign(vec[1])*.15), color=colors[i_vec])

    transformed_vec = np.matmul(W, vec)
    axes[1].arrow(0, 0, transformed_vec[0], transformed_vec[1], head_width=.2, facecolor=colors[i_vec], edgecolor=colors[i_vec], length_includes_head=True)
    axes[1].annotate(vec_names[i_vec], xy=(transformed_vec[0]+np.sign(transformed_vec[0])*.15, transformed_vec[1]+np.sign(transformed_vec[1])*.15), color=colors[i_vec])

  axes[0].set_title('Before')
  axes[1].set_title('After')
```

---

# Section 1: Intro to matrices

## Section 1.1: Matrices to solve systems of equations

```python
# @title Video 1: Systems of Equations
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Aq4y1x7hf", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="3ecnOrMEh00", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers using a matrix to solve a linear system of equations.

 Click here for text recap of video

In a variety of contexts, we may encounter systems of linear equations like this one:

$$
\begin{align}
3x_1 + 2x_2 + x_3 &= y_1 \\
7x_1 + x_2 + 2x_3 &= y_2 \\
x_1 - x_2  - 2x_3 &= y_3 \\
\end{align}$$
We may know all the x's and want to solve for y's, or we may know the y's and want to solve for the x's. We can solve this in several different ways but one especially appealing way is to cast it as a matrix-vector equation:

$$\mathbf{W}\mathbf{x} = \mathbf{y}$$
where
$$\begin{align}
\mathbf{W} &= \begin{bmatrix} 3 & 2 & 1 \\ 7 & 1 & 2 \\ 1 &-1 &-2 \end{bmatrix}, \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}, \mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix}\\
\end{align}$$

If we know $\mathbf{W}$ and $\mathbf{x}$, we can solve for $\mathbf{y}$ using matrix-vector multiplication. Each row of $\mathbf{y}$ is computed as the dot product of the equivalent row of $\mathbf{W}$ and $\mathbf{x}$.

If we know $\mathbf{W}$ and $\mathbf{y}$, we can sometimes solve for $\mathbf{x}$ by using the inverse of $\mathbf{W}$:
$$ \mathbf{x} = W^{-1}\mathbf{y} $$.
The reason this only sometimes works will be dived into later in this tutorial!

### Coding Exercise 1.1: Understanding neural transformations

We will look at a group of 2 LGN neurons which get input from 2 retinal neurons: we will call the population of LGN neurons population p. Below, we have the system of linear equations that dictates the neuron models for each population. $r_1$ and $r_2$ correspond to the retinal neural activities (of neuron 1 and 2). $g_{p_1}$ and  $g_{p_2}$ correspond to the responses of the LGN neurons 1 and 2 in population p.

$$\begin{align}
r_1 + 3r_2 &= g_{p_1} \\
2r_1 + r_2 &= g_{p_2} \\
\end{align}$$

1) Cast each this as a matrix-vector multiplication:

$$ \mathbf{g}_p = \mathbf{P}\mathbf{r} $$
where P is the weight matrix to population p.

2) Let's say we only recorded from the LGN cells (and know the weight matrix) and are trying to figure out how the retinal cells responded. Solve the matrix equation for the given LGN activities:

$$\mathbf{g}_p = \begin{bmatrix}
16 \\
7 \\
\end{bmatrix}$$

```python
# Create P (using np array)
P = np.array([[1, -4],
              [-3, 2]])

# Create g_p (using np array)
#g_p = np.array([16, 7])

# Solve for r (using np.linalg.inv)
#r = np.linalg.inv(P) @ g_p
r = np.array([[1, 0],
              [0, 1]])
g_p = P @ r

# Print r
#print(r)
print(g_p)
```

    [[ 1 -4]
     [-3  2]]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_2fa7cf85.py)

You should see the output [1, 5]

You can recover how the retinal neurons respond given the weight matrix and LGN responses! You have solved the system of equations using matrices. We can't always do this though: let's say we have a different group of 2 LGN neurons -  population q - with the following weight matrix from the retinal neurons.

$$Q = \begin{bmatrix}
4 & 1 \\
8 & 2 \\
\end{bmatrix}, $$

As you can see if you uncomment and run the next code cell, we get an error if we try to invert this matrix to solve the equation. We'll find out more about this in the next sections.

```python
g_q = np.array([16, 7])
Q = np.array([[4, 1], [8, 2]])

print(np.linalg.inv(Q) @ g_q)
#error due to Q inverse not existing.
```

## Section 1.2: Matrices as linear transformations

*Estimated timing to here from start of tutorial: 20 min*

```python
# @title Video 2: Linear Transformations
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1KB4y1T7zM", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="N6UUV9tVIr8", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Matrices can be thought of as enacting linear transformations. When multiplied with a vector, they transform it into another vector. In fact, they are transforming a grid of space in a linear manner: the origin stays in place and grid lines remain straight, parallel, and evenly spaced.

### Coding Exercise 1.2: Creating matrices for transformations

Come up with a matrix $A$ for which the corresponding linear transformation is reflection through the $y$ axis (flipping across the $y$ axis). For example, $\mathbf{x} = \begin{bmatrix}
2 \\
6  \\
\end{bmatrix}$ should become $\mathbf{b} = \begin{bmatrix}
-2 \\
6  \\
\end{bmatrix}$ when multiplied with $A$.

**Remember to think about where your basis vectors should end up! Then your matrix consists of the transformed basis vectors. Drawing out what you want to happen can help**

```python
A = np.array([[-1, 0], # matrix columns = transform basis vectors
              [0, 1]])

# Uncomment to visualize transformation
plot_linear_transformation(A)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_a86560cc.py)

*Example output:*

Come up with a matrix $A$ for which the corresponding linear transformation is projecting onto the $x$ axis. For example, $\bar{x} = \begin{bmatrix}
2 \\
3  \\
\end{bmatrix}$ should become $\bar{b} = \begin{bmatrix}
2 \\
0  \\
\end{bmatrix}$ when multiplied with $A$.

```python
A = np.array([[1, 0],
              [0, 0]])

# Uncomment to visualize transformation
plot_linear_transformation(A)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_8d909295.py)

*Example output:*

## Section 1.3: Rank & Null Space

*Estimated timing to here from start of tutorial: 35 min*

```python
# @title Video 3: Rank & Null Space
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1vw411R7eA", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="ay2p5rgkMcY", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers properties of matrices, such as range, rank, and null space, in the context of matrices enacting a linear transformation.

 Click here for text recap of video

Square matrices always result in vectors with the same dimensions (number of components) but can alter the dimensionality of the transformed space. Let's say you have a 3 x 3 matrix. You will transform from 3-dimensional vectors to 3-dimensional vectors. You will often be transforming from all of 3D space to all of 3D space. However, this isn't always the case! You could have a 3 x 3 matrix that always results in a vector that lies along a 2D plane through 3D space. This matrix would be transforming from a 3 dimensional vector space (all of R3) to a 2 dimensional vector space (the 2D plane).

Matrices that aren't square are enacting transformations that change the dimensionality of the vectors. If you have a 4 x 5 matrix, you are transforming 5-dimensional vectors to 4-dimensional vectors. Similarly, if you have a 4 x 2 matrix, you are transforming from 4-dimensional vectors to 2-dimensional vectors.

The **range of a matrix** is the set of all possible vectors it can lead to after a transformation. The **rank of a matrix** is the dimensionality of the range.

Sometimes, a matrix will transform a non-zero vector into a zero vector (the origin). The **null space** of a matrix is the set of all vectors that will be transformed into the origin.

### Think! 1.3: Neural coding

Let's return to the setup of the previous coding exercise: we have two populations of LGN neurons, p and q, responding to retinal neurons. Visualize the linear transformations of these matrices by running the next code cell. Then discuss the following questions:

1) What are the ranks of weight matrix P and Q?

2) What does the null space of these matrices correspond to in our neuroscience setting? **Advanced:** What do you think the dimensionality of the null space is for P and Q?

3) What is the intrinsic dimensionality of the population of neural responses in population p? How about in q? The intrinsic dimensionality is the minimal number of dimensions required for a complete representation of the data.

4) If we wanted to decode retinal neural activity from the LGN activities, would we always be able to completely recover the retinal activity when looking at population p? How about population q? What does this tell us about the information loss of the neural processing?

```python
# @markdown Execute to visualize linear transformations
P = np.array([[1, 3],
              [2, 1]])
plot_linear_transformation(P, name = 'p')

Q = np.array([[4, 1],
              [8, 2]]) # because (8,2) are just the product by 2 of 4, 1 we now the result is a straight line
plot_linear_transformation(Q, name = 'q')
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_95edac8e.py)

---
# Section 2: Eigenvalues & Eigenvectors

*Estimated timing to here from start of tutorial: 65 min*

```python
# @title Video 4: Eigenstuff
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1KK4y1M7Ez", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="l-c7ptT7znM", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers eigenvalues and eigenvectors.

Eigenvectors, $\mathbf{v}$ of a matrix $\mathbf{W}$ are vectors that, when multipled by the matrix, equal a scalar multiple of themselves. That scalar multiple is the corresponding eigenvalue, $\lambda$.

$$\mathbf{W}\mathbf{v} = \lambda\mathbf{v} $$

If we have one eigenvector for a matrix, we technically have an infinite amount: every vector along the span of that eigenvector is also an eigenvector. So, we often use the unit vector in that direction to summarize all the eigenvectors along that line.

We can find the eigenvalues and eigenvectors of a matrix in numpy using `np.linalg.eig`.

## Think! 2: Identifying transformations from eigenvectors

Earlier, we learned how to think about linear transformations in terms of where the standard basis vectors end up. We can also think about them in terms of eigenvectors.

Just by looking at eigenvectors before and after a transformation, can you describe what the transformation is in words? Try for each of the two plots below.

Note that I show an eigenvector for every eigenvalue. The x/y limits do not change in before vs after (so eigenvectors are showed scaled by the eigenvalues).

Here are some transformation words to jog your memory and guide discussion: contraction, expansion, horizontal vs vertical, projection onto an axis, reflection, and rotation.

```python
# @title
# @markdown Execute this cell to visualize vectors

W = np.array([[3, 0], [0, 1]])
plot_eig_vec_transform(W)
```

```python
# @title
# @markdown Execute this cell to visualize vectors

W = np.array([[0, 1], [1, 0]])
plot_eig_vec_transform(W)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_80704181.py)

As we saw above, looking at how just the eigenvectors change after a transformation can be very informative about what that transformation was.

---
# Section 3: Matrix multiplication

*Estimated timing to here from start of tutorial: 80 min*

```python
# @title Video 5: Matrix Multiplication
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Rb4y1C7cE", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="OFyWfegC9Cs", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

We sometimes want to multiple two matrices together, instead of a matrix with a vector. Let's say we're multiplying matrices $\mathbf{A}$ and $\mathbf{B}$ to get $\mathbf{C}$:
$$ \mathbf{C} = \mathbf{A}\mathbf{B}$$.

We take the dot product of each row of A with each column of B. The resulting scalar is placed in the element of $\mathbf{C}$ that is the same row (as the row in A) and column (as the column in B). So the element of $\mathbf{C}$ at row 4 and column 2 is the dot product of the 4th row of $\mathbf{A}$ and the 2nd column of $\mathbf{B}$. We can write this in a formula as:

$$\mathbf{C}_{\text{row i, column j}} = \mathbf{A}_{\text{row i}} \cdot \mathbf{B}_{\text{column j}}$$

## Exercise 2: Computation corner

Break out the pen and paper - it's critical to implement matrix multiplication yourself to fully understand how it works.

Let's say we have 3 retina neurons and 2 LGN neurons. The weight matrix, $W$, between the retina and LGN neurons is:

$$W = \begin{bmatrix}
3 &  2 & 1 \\
1 & 2 & 7 \\
\end{bmatrix}, $$

We are going to look at the activity at two time steps (each time step is a column). Our retina activity matrix, $R$, is:

$$ R= \begin{bmatrix}
0 &  1  \\
2 & 4 \\
5 & 1 \\
\end{bmatrix} $$

Please compute the LGN neural activity, $G$, according to our linear model:

$$G = WR $$.

Please calculate it 1) by-hand and then 2) using code. Check that the answers match!

```python
# Compute by hand first!
```

```python
# Define R
R = np.array([[0,1],[2, 4],[5, 1]])

# Define W
W = np.array([[3, 2, 1], [1, 2, 7]])

# Compute G
# in Python, we can use @ for matrix multiplication: matrix1 @ matrix2
G = W @ R

# Print values of G
print(G)
```

    [[ 9 12]
     [39 16]]

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_4b71ea28.py)

---
# Summary

*Estimated timing of tutorial: 1 hour, 35 minutes*

In this tutorial, you have learned how to think about matrices from the perspective of solving a system of equations and as a linear transformation of space. You have learned:
- Properties of a matrix, such as rank & null space
- How the invertibility of matrices relates to the linear transform they enact
- What eigenvalues/eigenvectors are and why they might be useful

We will be using this knowledge in many of the days in the NMA computational neuroscience course.


```python
# Import the packages for experiment
import warnings
warnings.simplefilter("ignore")

import pandas as pd
import numpy as np
import tensorflow as tf

import time
import random
import keras

from keras import layers
from itertools import product
from math import log2, ceil
from joblib import Parallel, delayed
from multiprocessing import Pool
from sklearn.model_selection import StratifiedKFold, train_test_split
```

# Random Classification Experiment

This experiment will use images from the **CIFAR-100** database (https://www.cs.toronto.edu/~kriz/cifar.html) and showcase the classification efficiency of algorithms in the **Progressive Learning** project (https://github.com/neurodata/progressive-learning).

## Progressive Learning

The Progressive Learning project aims to improve program performance on sequentially learned tasks, proposing a lifelong learning approach. It contains two different algorithms: **Lifelong Learning Forests** (L2F) and **Lifelong Learning Network** (L2N). L2F uses Uncertainy Forest as transformers, while L2N uses deep networks. These two algorithms achieve both forward knowledge transfer and backward knowledge transfer, and this experiment is designed to cover both models.

```python
# Import the progressive learning packages
from proglearn.progressive_learner import ProgressiveLearner
from proglearn.deciders import SimpleArgmaxAverage
from proglearn.transformers import TreeClassificationTransformer, NeuralClassificationTransformer
from proglearn.voters import TreeClassificationVoter, KNNClassificationVoter
```

### Separating data sets

The method will separate the data into training and testing subsets. The whole process is randomized to ensure the tests' validity.

```python
# The method randomly selects training and testing subsets from the original datasets,
# making cross-validation on program results possible.
def cross_val_data(
    data_x, data_y, num_points_per_task, total_task=10, shift=1, slot=0, task=0
):
    skf = StratifiedKFold(n_splits=6)
    for _ in range(shift + 1):
        train_idx, test_idx = next(skf.split(data_x, data_y))

    data_x_train, data_y_train = data_x[train_idx], data_y[train_idx]
    data_x_test, data_y_test = data_x[test_idx], data_y[test_idx]

    selected_classes = np.random.choice(range(0, 100), total_task)
    train_idxs_of_selected_class = np.array(
        [np.where(data_y_train == y_val)[0] for y_val in selected_classes]
    )
    num_points_per_class_per_slot = [
        int(len(train_idxs_of_selected_class[class_idx]) // total_task)
        for class_idx in range(len(selected_classes))
    ]
    selected_idxs = np.concatenate(
        [
            np.random.choice(
                train_idxs_of_selected_class[class_idx],
                num_points_per_class_per_slot[class_idx],
            )
            for class_idx in range(len(selected_classes))
        ]
    )
    train_idxs = np.random.choice(selected_idxs, num_points_per_task)
    data_x_train = data_x_train[train_idxs]
    data_y_train = data_y_train[train_idxs]

    test_idxs_of_selected_class = np.concatenate(
        [np.where(data_y_test == y_val)[0] for y_val in selected_classes]
    )
    data_x_test = data_x_test[test_idxs_of_selected_class]
    data_y_test = data_y_test[test_idxs_of_selected_class]

    return data_x_train, data_y_train, data_x_test, data_y_test
```

### Constructing experiments

This main method integrates both algorithms: L2F and L2N. When executing the experiment, users can choose between L2F and L2N models to generate desired results.

```python
# The method runs the lifelong learning experiments
def L2_experiment(
    data_x, data_y, ntrees, shift, slot, model, num_points_per_task, acorn=None
):

    # construct dataframes
    df = pd.DataFrame()
    shifts = []
    slots = []
    accuracies_across_tasks = []
    train_times_across_tasks = []
    inference_times_across_tasks = []

    # choose Uncertainty Forest as transformer
    if model == "uf":
        default_transformer_class = TreeClassificationTransformer
        default_transformer_kwargs = {"kwargs": {"max_depth": 30}}

        default_voter_class = TreeClassificationVoter
        default_voter_kwargs = {}

        default_decider_class = SimpleArgmaxAverage

    # choose deep neural network as transformer
    elif model == "dnn":
        default_transformer_class = NeuralClassificationTransformer

        network = keras.Sequential()
        network.add(
            layers.Conv2D(
                filters=16,
                kernel_size=(3, 3),
                activation="relu",
                input_shape=np.shape(train_x_task0)[1:],
            )
        )
        network.add(
            layers.Conv2D(
                filters=32,
                kernel_size=(3, 3),
                strides=2,
                padding="same",
                activation="relu",
            )
        )
        network.add(
            layers.Conv2D(
                filters=64,
                kernel_size=(3, 3),
                strides=2,
                padding="same",
                activation="relu",
            )
        )
        network.add(
            layers.Conv2D(
                filters=128,
                kernel_size=(3, 3),
                strides=2,
                padding="same",
                activation="relu",
            )
        )
        network.add(
            layers.Conv2D(
                filters=254,
                kernel_size=(3, 3),
                strides=2,
                padding="same",
                activation="relu",
            )
        )

        network.add(layers.Flatten())
        network.add(layers.Dense(2000, activation="relu"))
        network.add(layers.Dense(2000, activation="relu"))
        network.add(layers.Dense(units=10, activation="softmax"))

        default_transformer_kwargs = {
            "network": network,
            "euclidean_layer_idx": -2,
            "num_classes": 10,
            "optimizer": keras.optimizers.Adam(3e-4),
        }

        default_voter_class = KNNClassificationVoter
        default_voter_kwargs = {"k": int(np.log2(num_points_per_task * 0.33))}

        default_decider_class = SimpleArgmaxAverage

    # construct the learner with model selected
    progressive_learner = ProgressiveLearner(
        default_transformer_class=default_transformer_class,
        default_transformer_kwargs=default_transformer_kwargs,
        default_voter_class=default_voter_class,
        default_voter_kwargs=default_voter_kwargs,
        default_decider_class=default_decider_class,
    )

    # randomly separate the training and testing subsets
    train_x_task0, train_y_task0, test_x_task0, test_y_task0 = cross_val_data(
        data_x, data_y, num_points_per_task, total_task=10, shift=shift, slot=slot
    )

    # training process
    train_start_time = time.time()
    progressive_learner.add_task(
        X=train_x_task0,
        y=train_y_task0,
        num_transformers=1 if model == "dnn" else ntrees,
        transformer_voter_decider_split=[0.67, 0.33, 0],
        decider_kwargs={"classes": np.unique(train_y_task0)},
    )
    train_end_time = time.time()

    # testing process
    inference_start_time = time.time()
    task_0_predictions = progressive_learner.predict(test_x_task0, task_id=0)
    inference_end_time = time.time()

    # record results
    shifts.append(shift)
    slots.append(slot)
    accuracies_across_tasks.append(np.mean(task_0_predictions == test_y_task0))
    train_times_across_tasks.append(train_end_time - train_start_time)
    inference_times_across_tasks.append(inference_end_time - inference_start_time)

    # repeating the tasks for 20 times
    for task_ii in range(1, 20):

        # randomly separate the training and testing subsets
        train_x, train_y, _, _ = cross_val_data(
            data_x,
            data_y,
            num_points_per_task,
            total_task=10,
            shift=shift,
            slot=slot,
            task=task_ii,
        )

        # training process
        train_start_time = time.time()
        progressive_learner.add_transformer(
            X=train_x,
            y=train_y,
            transformer_data_proportion=1,
            num_transformers=1 if model == "dnn" else ntrees,
            backward_task_ids=[0],
        )
        train_end_time = time.time()

        # testing process
        inference_start_time = time.time()
        task_0_predictions = progressive_learner.predict(test_x_task0, task_id=0)
        inference_end_time = time.time()

        # record results
        shifts.append(shift)
        slots.append(slot)
        accuracies_across_tasks.append(np.mean(task_0_predictions == test_y_task0))
        train_times_across_tasks.append(train_end_time - train_start_time)
        inference_times_across_tasks.append(inference_end_time - inference_start_time)

    # finalize dataframes
    df["data_fold"] = shifts
    df["slot"] = slots
    df["accuracy"] = accuracies_across_tasks
    df["train_times"] = train_times_across_tasks
    df["inference_times"] = inference_times_across_tasks

    # save results
    return df
```

### Processing parallel tasks
To ensure the experiment's efficiency, this method would make running parallel processes possible.

```python
# The method allows multiple lifelong forest experiments to be run at the same time
def run_parallel_exp(
    data_x, data_y, n_trees, model, num_points_per_task, slot=0, shift=1
):
    if model == "dnn":
        with tf.device("/gpu:" + str(shift % 4)):
            return L2_experiment(
                data_x,
                data_y,
                n_trees,
                shift,
                slot,
                model,
                num_points_per_task,
                acorn=12345,
            )
    else:
        return L2_experiment(
            data_x,
            data_y,
            n_trees,
            shift,
            slot,
            model,
            num_points_per_task,
            acorn=12345,
        )
```

### Choosing hyperparameters

The hyperparameters here, especially the `model`, are used for determining how the experiment will run. `uf` will use the L2F algorithm, while `dnn` will use the L2N algorithm.

```python
### MAIN HYPERPARAMS ###
model = "uf" # choose "uf" or "dnn"
num_points_per_task = 500
slot_num = 10
shift_num = 6
alg_num = 1
task_num = 20
tree_num = 10
########################
```

### Loading datasets

The CITAR-100 database contains 100 classes of 600 images, each separating into 500 training images and 100 testing images. If the L2F model is chosen, the data will be fitted to the proper shape.

```python
# load image datasets from the CIFAR-100 database
(X_train, y_train), (X_test, y_test) = keras.datasets.cifar100.load_data()

# modify data shapes for specific model
data_x = np.concatenate([X_train, X_test])
if model == "uf":
    data_x = data_x.reshape(
        (data_x.shape[0], data_x.shape[1] * data_x.shape[2] * data_x.shape[3])
    )
data_y = np.concatenate([y_train, y_test])
data_y = data_y[:, 0]
```

### Running experiment

Depending on the chosen model, the experiment will feed the data into proper parallel processes.

```python
slot_fold = range(10)

# run the L2F model if selected
if model == "uf":
    shift_fold = range(1, 7, 1)
    n_trees = [10]
    iterable = product(n_trees, shift_fold, slot_fold)
    df_results = Parallel(n_jobs=-1, verbose=1)(
        delayed(run_parallel_exp)(
            data_x, data_y, ntree, model, num_points_per_task, slot=slot, shift=shift
        )
        for ntree, shift, slot in iterable
    )

# run the L2N model if selected
elif model == "dnn":

    df_results = pd.DataFrame()

    print("Performing Stage 1 Shifts")
    for slot in slot_fold:

        def perform_shift(shift):
            return run_parallel_exp(
                data_x, data_y, 0, model, num_points_per_task, slot=slot, shift=shift
            )

        stage_1_shifts = range(1, 5)
        with Pool(4) as p:
            df_results = df_results.append(p.map(perform_shift, stage_1_shifts))

    print("Performing Stage 2 Shifts")
    for slot in slot_fold:

        def perform_shift(shift):
            return run_parallel_exp(
                data_x, data_y, 0, model, num_points_per_task, slot=slot, shift=shift
            )

        stage_2_shifts = range(5, 7)
        with Pool(4) as p:
            df_results = df_results.p.map(perform_shift, stage_2_shifts)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.

### Analyzing the results

After the experiment processes finish, the codes below will illustrate the training efficiency and time taken for both training and testing. Though individual computing time might vary, the whole process should be efficient enough to run on any common device.

```python
# Import the packages for plotting
import matplotlib.pyplot as plt
from matplotlib import rcParams
rcParams.update({'figure.autolayout': True})
```

```python
reps = slot_num * shift_num

btes = []
train_times = []
inference_times = []

# read the experiment results
for slot in range(slot_num):
    for shift in range(shift_num):

        # load the dataframes
        df = df_results[shift * slot_num + slot]

        # calculate the backward transfer efficiency
        err = 1 - df["accuracy"]
        bte = err[0] / err

        # record information
        btes.append(bte)
        train_times.append(df["train_times"])
        inference_times.append(df["inference_times"])

# calculate the average numbers
bte = np.mean(btes, axis=0)
train_time_across_tasks = np.mean(train_times, axis=0)
inference_time_across_tasks = np.mean(inference_times, axis=0)

# setting plot parameters
fontsize = 22
ticksize = 20
```

### Plotting backward transfer efficiency

Backward transfer efficiency (BTE) measures the relative effect of future task data on the performance on a certain task. It is the expected ratio of two risk functions of the learned hypothesis, one with access to the data up to and including the last observation from task t, and the other with access to the entire data sequence.

\begin{align}
BTE^t(f_n):= E[R^t(f_n^{<t}) / R^t(f_n)]
\end{align}

```python
# plot the BTE
ax = plt.subplot(111)

# hide the right and top spines
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)

# only show ticks on the left and bottom spines
ax.yaxis.set_ticks_position("left")
ax.xaxis.set_ticks_position("bottom")

ax.plot(
    range(len(bte)),
    bte,
    c="blue" if model == "dnn" else "red",
    linewidth=3,
    linestyle="solid",
    label="L2N" if model == "dnn" else "L2F",
)

ax.set_xlabel("Number of Tasks Seen", fontsize=fontsize)
ax.set_ylabel("BTE (Task 1)", fontsize=fontsize)
ax.set_yticks([1.0, 1.1, 1.2])
ax.tick_params(labelsize=ticksize)
ax.legend(fontsize=22)

# show the BTE plot
plt.tight_layout()
plt.show()
```

### Plotting time

The graph will show how training time and testing time change during the experiment.

```python
# plot the recorded times
ax = plt.subplot(111)

# Hide the right and top spines
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)

# Only show ticks on the left and bottom spines
ax.yaxis.set_ticks_position("left")
ax.xaxis.set_ticks_position("bottom")

ax.plot(
    range(len(train_time_across_tasks)),
    train_time_across_tasks,
    linewidth=3,
    linestyle="solid",
    label="Train Time",
)
ax.plot(
    range(len(inference_time_across_tasks)),
    inference_time_across_tasks,
    linewidth=3,
    linestyle="solid",
    label="Inference Time",
)

ax.set_xlabel("Number of Tasks Seen", fontsize=fontsize)
ax.set_ylabel("Time (seconds)", fontsize=fontsize)
ax.tick_params(labelsize=ticksize)
ax.legend(fontsize=22)

# show the time plot
plt.tight_layout()
plt.show()
```