```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import matplotlib

# for 3d plotting
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.patches import FancyArrowPatch
from mpl_toolkits.mplot3d import proj3d

%matplotlib inline
#%matplotlib widget
#%matplotlib notebook
```

```python
matplotlib.rcParams['figure.figsize'] = (12, 10) # set default figure size, 8in by 6in
```

In this module, we introduce Principal Components Analysis, and show how it can be used for data compression to speed up learning algorithms as well as for visualizations of complex datasets.

# Video W8 06: Motivation I Data Compression

[YouTube Video Link](https://www.youtube.com/watch?v=h8ri2gbBjIc&index=82&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)

This video gives a good intuitive idea of how data compression by reducing dimensions works.  As shown in the video, a 2
dimensional highly redundante (linear) feature can be basically reduced to 1 dimension with almost no loss of information.
This type of reduction of dimensions is very important in practice.  Less dimensions means less features we have for our learning
algorithms.  This reduces the time and effort needed to build models using the features.

It might seem a bit contrived, but it is really quite common, especially when working with big data, that a lot
of features will have a large bit of overlap.  Such redundant features are often not going to add any predictive
power to a classifier, and are thus best eliminated.

For example, from the video we imagine a data set that has features both measuring the length of some item,
but one was measured in cm and one in inches.

```python
# make some cm measurements at random, from 10 to 30 cm
m = 14
x1 = np.random.choice(np.arange(10, 31), m, replace=False)

# convert to inches and round to nearest inch
x2 = np.round(x1 / 2.54)

#print(x1)
#print(x2)

# plot the points
plt.plot(x1, x2, 'kx', markersize=10)
plt.xlabel('$x_1$ (cm)', fontsize=16)
plt.ylabel('$x_2$ (in)', fontsize=16);

# plot a fitted line to the points
theta = np.polyfit(x1, x2, 1)
h = np.poly1d(theta)
x = np.linspace(10,30,100)
plt.plot(x, h(x), 'b-');
```

We approximate the original data set by projecting each original point onto the line.

Reduce data from 2D to 1D

\begin{equation}
\begin{split}
x^{(1)} \in \mathbb{R}^2 &\to z^{(1)} \in \mathbb{R} \\
x^{(2)} \in \mathbb{R}^2 &\to z^{(2)} \in \mathbb{R} \\
\vdots \\
x^{(m)} \in \mathbb{R}^2 &\to z^{(m)} \in \mathbb{R} \\
\end{split}
\end{equation}

## Reduce data from 3D to 2D Example

In a typical data reduction/compression we might be doing something more like reducing $1000D \to 100D$
for example.

When you visualize a projection from 3D to 2D, what will happen is that you will project the data
to a 2-dimensional hyperplane.

```python
# generate some 3d data by hand in 2 classes, just for this example
mu_vec1 = np.array([0,0,0])
cov_mat1 = np.array([[1,0.66,0],[0.66,1,0.5],[0,0.5,1]])
X = np.random.multivariate_normal(mu_vec1, cov_mat1, 100)
print(X.shape)
```

    (100, 3)

```python
fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111, projection='3d')
plt.rcParams['legend.fontsize'] = 10
ax.plot(X[:, 0], X[:, 1], X[:, 2], 'o', markersize=8, color='blue', alpha=0.25)

plt.title('Random samples with correlation/covariation');

```

The following example is from [here](https://stackoverflow.com/questions/22867620/putting-arrowheads-on-vectors-in-matplotlibs-3d-plot) of plotting arrows using eigenvectors calculated from the data.

```python
# matplotlib still doesn't have an arrow patch for drawing vectors, so we add it
class Arrow3D(FancyArrowPatch):
    def __init__(self, xs, ys, zs, *args, **kwargs):
        FancyArrowPatch.__init__(self, (0,0), (0,0), *args, **kwargs)
        self._verts3d = xs, ys, zs

    def draw(self, renderer):
        xs3d, ys3d, zs3d = self._verts3d
        xs, ys, zs = proj3d.proj_transform(xs3d, ys3d, zs3d, renderer.M)
        self.set_positions((xs[0],ys[0]),(xs[1],ys[1]))
        FancyArrowPatch.draw(self, renderer)

```

```python
#%matplotlib notebook

# mean values
mean_x = np.mean(X[:,0])
mean_y = np.mean(X[:,1])
mean_z = np.mean(X[:,2])

#eigenvectors and eigenvalues
eig_val, eig_vec = np.linalg.eig(cov_mat1)

################################
#plotting eigenvectors
################################

fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111, projection='3d')

ax.plot(X[:,0], X[:,1], X[:,2], 'o', markersize=10, color='blue', alpha=0.25)
ax.plot([mean_x], [mean_y], [mean_z], 'o', markersize=10, color='green', alpha=0.4)
for d,v in enumerate(eig_vec[:3]):
    #ax.plot([mean_x,v[0]], [mean_y,v[1]], [mean_z,v[2]], color='red', alpha=0.8, lw=3)
    #I will replace this line with:
    v = v * eig_val[d]
    a = Arrow3D([mean_x, v[0]], [mean_y, v[1]],
                [mean_z, v[2]], mutation_scale=20,
                lw=3, arrowstyle="-|>", color="green")
    ax.add_artist(a)
ax.set_xlabel('x_values')
ax.set_ylabel('y_values')
ax.set_zlabel('z_values')

plt.title('Random samples with correlation/covariation and first 2 Eigenvectors')

```

```python
# use PCA to project from 3D to 2D
from sklearn.decomposition import PCA
pca = PCA(n_components=2, whiten=True)
pca.fit(X)
print(pca.components_)
print(pca.explained_variance_)

X_pca = pca.transform(X)
print("original shape:   ", X.shape)
print("transformed shape:", X_pca.shape)

# plot the 2D projection
plt.figure(figsize=(8,8))
plt.scatter(X[:, 0], X[:, 1], alpha=0.25)
plt.xlabel('$z_0$', fontsize=16)
plt.ylabel('$z_1$', fontsize=16);
```

# Video W8 07: Motivation II Data Visualization

[YouTube Video Link](https://www.youtube.com/watch?v=18xeFJZAdMw&index=83&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)

Another main use of dimensionality reduction is in order to visualize very high dimensional data in 2
or 3 dimensional space and plots.

The previous plot and subsequent reduction from 3D to 2D was an example.  But for visualization, usually what we
mean is we have a huge set of features, and we want to project down to the most important 2 or 3 features and plot
those features.

Using dimensionality reduction, we can project high dimensional data into a 3D or 2D set of data that we can
then successfully visualize in 3 or 2 dimensions.

When you use PCA, the new features do not directly correspond to any of the original features, so you will need
to infer what they might most directly relate to.

# Video W8 08: Principal Component Analysis

[YouTube Video Link](https://www.youtube.com/watch?v=aSl7tPE1Pyo&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=84)

The formal method for accomplishing dimensionality reduction through the technique known as Principle Component Analysis (PCA) has
some surface similarity to performing a linear regression.  Intuitively, we are trying to find a lower dimensional hyperplane for our data
to which when we project our original data we minimize the sum squared error of the distances we had to project the data in order
to reduce the dimensionality (the projection error).

Principal Component Analysis (PCA) attempts to find a projection to a lower dimensional hyperplane
that  minimizes the sum squared projection error of the projection of the points onto the new lower
dimensional hyperplane.  As shown in the example in the video, if our original data is
$\mathbb{R}^2$ (2 dimensional), the lower dimensional hyperplane is in this case a 1 dimensional line.

PCA is not linear regression.  The cost function is similar, but in PCA we are minimizing the projection
distance to the line/plane, instead of minimizing the distance from the point to the dependent variable we
are trying to predict.

Lets look at a quick example of using PCA.  The following example is taken directly from the
`scikit-learn` library [example on using PCA](http://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_iris.html).

First of all, lets look at a rather famous data set, the Iris dataset.  This data set consists of
only $m = 150$ data points.  The data points are labeled between 3 different species of Iris
flowers (species Virginica, Versicolor and Setosa).  PCA is an unsupervised learning technique, so
we don't need the labels.  Each of the 150 data points has $\mathbb{R}^4$ 4 dimensional measurements associated with it (petal length and width, and sepal length and width).  We of course can't visualze
all 4 dimensions simultaneously, so lets visualize 3 of the 4 data points:

```python
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
y = iris.target

print(iris.DESCR)
```

    .. _iris_dataset:

    Iris plants dataset
    --------------------

    **Data Set Characteristics:**

        :Number of Instances: 150 (50 in each of three classes)
        :Number of Attributes: 4 numeric, predictive attributes and the class
        :Attribute Information:
            - sepal length in cm
            - sepal width in cm
            - petal length in cm
            - petal width in cm
            - class:
                    - Iris-Setosa
                    - Iris-Versicolour
                    - Iris-Virginica

        :Summary Statistics:

        ============== ==== ==== ======= ===== ====================
                        Min  Max   Mean    SD   Class Correlation
        ============== ==== ==== ======= ===== ====================
        sepal length:   4.3  7.9   5.84   0.83    0.7826
        sepal width:    2.0  4.4   3.05   0.43   -0.4194
        petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
        petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
        ============== ==== ==== ======= ===== ====================

        :Missing Attribute Values: None
        :Class Distribution: 33.3% for each of 3 classes.
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
    from Fisher's paper. Note that it's the same as in R, but not as in the UCI
    Machine Learning Repository, which has two wrong data points.

    This is perhaps the best known database to be found in the
    pattern recognition literature.  Fisher's paper is a classic in the field and
    is referenced frequently to this day.  (See Duda & Hart, for example.)  The
    data set contains 3 classes of 50 instances each, where each class refers to a
    type of iris plant.  One class is linearly separable from the other 2; the
    latter are NOT linearly separable from each other.

    .. topic:: References

       - Fisher, R.A. "The use of multiple measurements in taxonomic problems"
         Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
         Mathematical Statistics" (John Wiley, NY, 1950).
       - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.
         (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
       - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
         Structure and Classification Rule for Recognition in Partially Exposed
         Environments".  IEEE Transactions on Pattern Analysis and Machine
         Intelligence, Vol. PAMI-2, No. 1, 67-71.
       - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
         on Information Theory, May 1972, 431-433.
       - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
         conceptual clustering system finds 3 classes in the data.
       - Many, many more ...

```python
# plot 3 of the 4 dimnsions
fig = plt.figure()
plt.clf()
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)
ax.scatter(X[:, 0], X[:, 1], X[:, 2], c=y, cmap=plt.cm.jet)

# The data in the 4 dimensions are:
# X[:, 0] = 'sepal length'
# X[:, 1] = 'sepal width'
# X[:, 2] = 'petal length'
# X[:, 3] = 'petal width'
# so if you want the labels on the axis to be correct if you plot different
# dimensions, need to also change these labels.
dims = ['Sepal length', 'Sepal width', 'Petal length', 'Petal width']
ax.xaxis.set_label_text(dims[0])
ax.yaxis.set_label_text(dims[1])
ax.zaxis.set_label_text(dims[2])

# likewise, we are labeling the type of each of the 3 species clases
# by putting a text label on the feature at the mean.  You would have to
# change the code here ias well to specify which dimensions of X[,?] you are plotting
for name, label in [('Setosa', 0), ('Versicolour', 1), ('Virginica', 2)]:
    ax.text3D(X[y == label, 0].mean(),
              X[y == label, 1].mean(),
              X[y == label, 2].mean(), name,
              horizontalalignment='center',
              bbox=dict(alpha=.5, edgecolor='w', facecolor='w'))

```

I encourage you to try plotting some other combinations of dimensions.  Try using the `%matplotlib notebook`
notebook directive to allow for interactive plots (you may need to install additional libraries/components
to get this to work.

Lets compress the data from 4 dimensions to its 2 principal component dimensions and see the result.

```python
from sklearn import decomposition

pca = decomposition.PCA(n_components=2)
pca.fit(X)
X = pca.transform(X)

plt.figure()
plt.scatter(X[y==0, 0], X[y==0, 1], c='b', label='Setosa')
plt.scatter(X[y==1, 0], X[y==1, 1], c='g', label='Versicolor')
plt.scatter(X[y==2, 0], X[y==2, 1], c='r', label='Virginica')

plt.xlabel('z1')
plt.ylabel('z2')
plt.legend()
```

```python
print(pca.components_)
print(pca.explained_variance_ratio_)
print(pca.mean_)
```

    [[ 0.36138659 -0.08452251  0.85667061  0.3582892 ]
     [ 0.65658877  0.73016143 -0.17337266 -0.07548102]]
    [0.92461872 0.05306648]
    [5.84333333 3.05733333 3.758      1.19933333]

So if PCA is working correctly, this represents the 2 dimensions from the original 4 with the least amount of projection error of the original 4 dimensional data.  As you can see, while Setosa is well separated
from the other 2 classes, there is some overlaop in Versicolor and Virginica classes.

# Video W8 09: Principal Component Analysis Algorithm

[YouTube Video Link](https://www.youtube.com/watch?v=G7HJWzjQNSQ&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=85)

This video starts out by noting that you should always perform mean feature scaling and mean normalization
of the features.  You will notice that in the example I gave above I did not do this by hand.  This is
because the `scikit-learn` PCA library automatically performs the normalization on the
data you give it before running PCA.  You will notice, for example, that the 2 dimensional principal
components are centered at the origin $0, 0$

The video at this point goes into the details of how the PCA algorithm is computed.  As stated in the
video, we won't look at any proofs that these procedures correctly comptue the hyperplanes that
minimize the projection error (it is beyond the scope of our course).  But lets just compute
the PCA by hand using Python functions, and compare our results to the PCA computation obtained
through the `scikit-learn` library.

As in the previous example, lets continue using the iris data set with 4 dimensions to perform
our example of computing the PCA algorithm:

```python
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
y = iris.target

m, n = X.shape # m training examples, each n dimensional
print(m, "training examples")
print(n, "features/dimensions")
```

    150 training examples
    4 features/dimensions

Before when we used PCA from the `scikit-learn` library, it perform feature normalization for us.  But
if we want to do PCA ourself by hand, we need to perform normalization:

$$
\mu_j = \frac{1}{m} \sum_{i=1}^{m} x_j^{(i)}
$$

Where the $\mu_j$ is really the mean for each of the $n$ features of our data.  We replace each
feature $x_j^{(i)}$ with $x_j - \mu_j$ (e.g. we are simply subtracting the mean from each feature).
We can do feature normalization by hand like this:

```python
# we could use loops to calculate the mean of each of the 4 features, but we can
# use a simply numpy operation to get the mean along the appropriate axis
mu = np.mean(X, axis=0)

# we should end up with the mean value of each of our 4 dimensions, compare this to the means
# in the iris dataset documentation to verify
print(mu.shape)
print(mu)
```

    (4,)
    [5.84333333 3.05733333 3.758      1.19933333]

Once we have the means of each dimension, we can perform mean normalization.  Again we'll use numpy
operations to subtract the mean for each dimension from each data set item

```python
X_norm = X - mu
print(X_norm.shape)

 # notice that the means are all now basically 0 in our mean normalized data
print(np.mean(X_norm, axis=0))
```

    (150, 4)
    [-1.12502600e-15 -7.60872846e-16 -2.55203266e-15 -4.48530102e-16]

To also do the feature scaling, one simple way is to calculate the standard deviation of each of
our dimensions in the data, then divide the values by this standard deviation.  This has the effect of
scaling the standard deviation of the dimension to be 1.0.  For example, we can do a very similar
procedure to calculate the sd of each dimension, the perform both normalization and scaling

$$
X_j^{(i)} \leftarrow \frac{X_j^{(i)} - \mu_j}{s_j}
$$

```python
sd =  np.std(X, axis=0)

# we should end up with the standard deviation value of each of our 4 dimensions, again you
# can compare this to the reported sd for each dimension
print(sd.shape)
print(sd)
```

    (4,)
    [0.82530129 0.43441097 1.75940407 0.75969263]

```python
X_norm_scaled = (X - mu) / sd
print(X_norm_scaled.shape)

 # notice that the means are all now basically 0 in our mean normalized data
print(np.mean(X_norm_scaled, axis=0))
print(np.std(X_norm_scaled, axis=0))
```

    (150, 4)
    [-1.69031455e-15 -1.84297022e-15 -1.69864123e-15 -1.40924309e-15]
    [1. 1. 1. 1.]

So as shown, the final result of normalization and scaling is that all features now have a mean of 0 and
a standard deviation of 1.0.  It appears that the `scikit-learn` PCA only performs mean normalization
for you on the data you fit.  Thus, in the above cells I created arrays `X_norm` and `X_nrom_scaled`,
but we will only use `X_norm` in the next cells when we recreate the PCA algorithm by hand.

## PCA Algorithm (computing by hand)

**Step 1**: The first step shown in the video to perform the **PCA algorithm** is to compute a "covariance
matrix", that we will name `Sigma`.  The formula for computing the covariance matrix is:

$$
\Sigma = \frac{1}{m} \sum_{i=1}^{m} (x^{(i)}) (x^{(i)})^T
$$

**NOTE**: There is a typo first time this equation is shown in video around time 8:00.  The sum goes over
the $m$ training input patterns.

We can compute that using Python for a single dimension, like dimension 0, with the following code:

```python
# In python, our dimensions are 0, 1, 2 and 3, so the above formula sums from i = 0 to 3.
# For example, if we start with dimension 0, and we multiple x^(0) x^(0)^T we get
c = np.dot(X_norm[0, :].reshape( (n,1) ), X_norm[0, :].reshape( (n,1) ).T)
print(c.shape)
print(c)
```

    (4, 4)
    [[ 0.55254444 -0.32904889  1.75278     0.74283778]
     [-0.32904889  0.19595378 -1.043808   -0.44237156]
     [ 1.75278    -1.043808    5.560164    2.356428  ]
     [ 0.74283778 -0.44237156  2.356428    0.99866711]]

The result of doing this for dimension 0 is a 4 x 4 covariance matrix for that sample $i$.  This basically measures the amount of variance between each dimension for input pattern 0.  To get the full covariance
matrix, we need to do this for all 150 input patterns, and sum up and average the result

```python
Sigma1 = np.empty( (n, n) )
for i in range(m):
    Sigma1 += np.dot( X_norm[i,:].reshape( (n, 1) ), X_norm[i, :].reshape(n, 1).T )
Sigma1 = Sigma1 / m # finally take average by dividing by m

print(Sigma1.shape)
print(Sigma1)
```

    (4, 4)
    [[ 0.68480585 -0.03995745  1.2775052   0.51778114]
     [-0.03995745  0.19001925 -0.32049995 -0.1178793 ]
     [ 1.2775052  -0.32049995  3.13257043  1.30268152]
     [ 0.51778114 -0.1178793   1.30268152  0.58379067]]

```python
# as shown later in the video, an equivalent way of computing
# sigma using linear algebra matrix multiplication exclusively is
Sigma2 = np.dot(X_norm.T, X_norm) / m
print(Sigma2.shape)
print(Sigma2)
```

    (4, 4)
    [[ 0.68112222 -0.04215111  1.26582     0.51282889]
     [-0.04215111  0.18871289 -0.32745867 -0.12082844]
     [ 1.26582    -0.32745867  3.09550267  1.286972  ]
     [ 0.51282889 -0.12082844  1.286972    0.57713289]]

```python
# I was expecting maybe some rounding error, but probably closer than this.  Not sure
# if we have a slight bug/problem in the above example or not.
Sigma1 - Sigma2
```

    array([[0.00368363, 0.00219366, 0.0116852 , 0.00495225],
           [0.00219366, 0.00130636, 0.00695872, 0.00294914],
           [0.0116852 , 0.00695872, 0.03706776, 0.01570952],
           [0.00495225, 0.00294914, 0.01570952, 0.00665778]])

**Step 2**: Once we have the `Sigma` covariance matrix, the next step shown was to calculated the "eigenvectors"
of the matrix.  We will also use the singular value decomposition of the matrix from the numpy
linalg library to do this computation.

SVD computes the eigenvectors of the Sigma covariance matrix.  You can also use `np.linalg.eig()` to get the
same eigenvectors.

What SVD outputs is 3 matrices, $U$, $S$ and $V$.  What we really need for dimensionality reduction is
the $U$ matrix. The $U$ matrix will be an $n \times n$ shaped matrix, where $n$ is the
number of features or dimensions of the data set we are using.
The columns of the $U$ matrix will be exactly the principle component vectors for the
set of data we gave that generates the covariance matrix Sigma.  On in other words, $U$ will have the
principle component vectors of the new space we want to project our data onto.

So for data reduction, if we want to reduce from $n$ to $k$ dimensions, we simply want to take the first $k$
columns of the $U$ matrix to form the new basis for our projected space.

```python
U, S, V = np.linalg.svd(Sigma2)
print(U.shape)
print(S.shape)
print(V.shape)
```

    (4, 4)
    (4,)
    (4, 4)

If you look at the U matrix, these are the $u$ vectors that define the PCA space, and if we want only
the first $k$ dimensions, we take the first $k$ column vectors of the U matrix

```python
print(U)
# if we wanted only 2 dimensions, we could take the first 2 columns
U_reduce =  U[:,:2]
print(U_reduce)
```

    [[-0.36138659 -0.65658877  0.58202985  0.31548719]
     [ 0.08452251 -0.73016143 -0.59791083 -0.3197231 ]
     [-0.85667061  0.17337266 -0.07623608 -0.47983899]
     [-0.3582892   0.07548102 -0.54583143  0.75365743]]
    [[-0.36138659 -0.65658877]
     [ 0.08452251 -0.73016143]
     [-0.85667061  0.17337266]
     [-0.3582892   0.07548102]]

**Step 3**: Finally to perform our reduction/projection, we want to use the $U_{reduce}$ matrix.  So to get the new vector Z, we multiple the transpose of U_reduce times the original input data X

For the iris data set, this should give us the same results as we got from calling PCA from the
`scikit-learn` library.  Lets calculate our z values and plot them to compare to our previous figure:

```python
Z = np.dot(X_norm, U_reduce)
print(X_norm.shape)
print(Z.shape)
```

    (150, 4)
    (150, 2)

```python
plt.figure()
plt.scatter(Z[y==0, 0], Z[y==0, 1], c='b', label='Setosa')
plt.scatter(Z[y==1, 0], Z[y==1, 1], c='g', label='Versicolor')
plt.scatter(Z[y==2, 0], Z[y==2, 1], c='r', label='Virginica')

plt.xlabel('$z_0$')
plt.ylabel('$z_1$')
plt.legend();
```

You may have noticed that the $z_0$ dimension (which is our dimension 0 using python 0 based indexing) is
flipped in this figure.  As mentioned in the figures, PCA can return z or the -z of a vector.  Thus, to
get exactly the same figure as we had before, we just need to negate our z vector for the first dimension.

```python
plt.figure()
plt.scatter(-Z[y==0, 0], Z[y==0, 1], c='b', label='Setosa')
plt.scatter(-Z[y==1, 0], Z[y==1, 1], c='g', label='Versicolor')
plt.scatter(-Z[y==2, 0], Z[y==2, 1], c='r', label='Virginica')

plt.xlabel('$z_0$')
plt.ylabel('$z_1$')
plt.legend();
```

If you look closely at this figure and compare it to the figure produced by using the `scikit-learn`
implementation of PCA, you should see that we got exactly the same result.

Also as we will later discuss, the other matrices from the SVD contain useful information.  For example, previously
we printed out the `components_` and the `explained_variance_ratio_` returned from PCA.  The `components` should
be equivalent to our $U_{reduced}$ reduced principal components matrix.  Likewise the `explained_variance_ratio_`
is the same as the $S$ vector from the SVD (normalized to sum up to 1):

```python
>>> print(pca.components_)
[[ 0.36138659 -0.08452251  0.85667061  0.3582892 ]
 [ 0.65658877  0.73016143 -0.17337266 -0.07548102]]
>>> print(pca.explained_variance_ratio_)
[0.92461872 0.05306648]
```

```python
print(U_reduce)
print(S / np.sum(S))
```

    [[-0.36138659 -0.65658877]
     [ 0.08452251 -0.73016143]
     [-0.85667061  0.17337266]
     [-0.3582892   0.07548102]]
    [0.92461872 0.05306648 0.01710261 0.00521218]

# Video W8 10: Choosing the Number of Principal Components

[YouTube Video Link](https://www.youtube.com/watch?v=qqK-8HZjwTo&index=86&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)

As shown in this video, we can use the S matrix from the singular value decomposition to see how much
variance each of the new dimensions discovered by the PCA explains.  I didn't mention it above, but we
can get the S matrix that `scikit-learn` calculates, as well as the ratio of the variance explained
by each of the Z dimensions, like this:

```python
print(pca.explained_variance_)
print(pca.explained_variance_ratio_)
```

    [4.22824171 0.24267075]
    [0.92461872 0.05306648]

The explained variance is basically the S matrix from the SVD:

```python
print(S)
```

    [4.20005343 0.24105294 0.0776881  0.02367619]

The variance ratio is simply obtained by dividing each variance measure by the total sum of the variance
measures.

```python
print(S / np.sum(S))
```

    [0.92461872 0.05306648 0.01710261 0.00521218]

As discussed in this video, the ratio of the variances in PCA can be interpreted as the amount of
information each dimension holds.  In the case of the iris data set PCA, the first dimension explains
92.5% of the variance of the data, and the second dimension another 5.3%, for a total of almost 98%
of the variance represented by the first 2 dimensions.  Another way that I think of these measures,
the first 2 dimensions of our PCA actually only introduce about 2% of projection error when projecting
the original 4 dimensional data onto the discovered 2 principle component dimensions.

Sometimes you will see people plot the variance ratios.  This isn't too useful for our iris data with
only 4 dimensions total, but when you have a large number of dimensions, it can visually let you
know which dimensions are important, and which add little and mostly represent redundant information
or noise in the data.

```python
#%matplotlib notebook
plt.plot(range(4), S/np.sum(S), 'bo')
plt.plot(range(4), S/np.sum(S), 'k--')
plt.axis([-0.5, 3.5, 0.0, 1.0])
plt.xlabel('Dimension')
plt.ylabel('Ratio of Variance Explained');
```

As discussed in the video, typically we choose the number of dimensions $k$ when doing reduction or compression
to retain some percentage of the variance.  Typically we might choose to retain $0.99$ or $0.95$ ratio of the
variance.  This means we need to look at the explained variance, and choose the dimension $k$ that is at or
exceeds the target.  So instead of plotting the ratio like we did before, it is often more informative to
plot the cumulative sum of the ratio explained.

```python
#%matplotlib notebook

plt.plot(range(4), np.cumsum(S/np.sum(S)), 'k-')
plt.xlabel('Dimension')
plt.ylabel('Cumulative Sum of Variance Explained');
```

Another think to note, in practice it is often the case that a lot of dimensions are only capturing noise or are
highly correlated with other dimensions.  So often it can be surprising how many dimensions will not be needed
to retain 95% or even 99% of the variance.  Often a large number of dimensions are possible to be dropped while still
retaining a high percentage of the variance of the original data.

# Video W8 11: Reconstruction from Compressed Representation

[YouTube Video Link](https://www.youtube.com/watch?v=ATeoRX1GKVE&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=87)

So we can project back to the original high dimensional space using the information from our PCA.

\begin{equation}
X_{approx} = U_{reduce} \times Z
\end{equation}

Where $Z$ is the projected $k$ dimensional space of the samples of the data.

So we could reconstruct the 4D iris data set back from the 2D representation we showed above, though of course we
wouldn't really be able to visualize what this does.

# Video W8 12: Advice for Applying PCA

[YouTube Video Link](https://www.youtube.com/watch?v=QitS8I_Thd8&index=88&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)

PCA can be used to speed up the learning time of an algorithm.  For a supervised learning problem with $m$
samples we might often have very high dimensional (big) data.  So the features of each sample could be in the 10s
or 100s of thousands of features (or even bigger).  For example when doing processing on images, each pixel
will be 1 feature (or even 3 features if doing color images).

For big data, it is often useful to use dimensionality reduction to reduce the representation to a much
lower number of dimensions.  This helps both because a) many of the low variance dimensions are just capturing
noise, and are thus not useful, if not even actively harmful, to a supervised learner in modeling the data.
Also when the data is reduced down in dimension significantly, this significantly simplifies the task that
the supervied learning mechanism must perform.

**NOTE**: The mapping of $x^{(i)} \to z^{(i)}$ should be defined by running PCA **only** on the training set.
This mapping can be applied as well to the examples in the cross-validation and test sets when needed.  But the
mapping should never be learned from the test or cv data, because if you do this you are leaking information
from your test data potentially back to your training information.

## Application of PCA

- Compression
  - Reduce memory/disk needed to store data
  - Speed up learning algorithm
- Visualization

For compression we will choose $k$ to retain some percentage of the variation in the data, typically 99% or 95%.
But for visualization, of course, the goal is to better understand the data.  So in that case we must choose to
reduce to $k=2$ or $k=3$ so we can plot a visualization of the data.

```python
import sys
sys.path.append("../../src") # add our class modules to the system PYTHON_PATH

from ml_python_class.custom_funcs import version_information
version_information()
```

                  Module   Versions
    --------------------   ------------------------------------------------------------
             matplotlib:   ['3.3.0']
                  numpy:   ['1.18.5']
                 pandas:   ['1.0.5']


# QPC nanowire spectrum

```python
from functools import partial

import adaptive
import kwant
import numpy as np
import sympy

import common
import funcs

adaptive.notebook_extension()
sympy.init_printing()
```

# Hamiltonian

```python
funcs.get_sympy_hamiltonian({}, 2)
```

# Band structure example

```python
import matplotlib.pyplot
syst = funcs.make_wire(a=10, r=50, shape='hexagon', dim=3)
```

```python
params = dict(alpha=0, mu=0, g=50, B_x=0, B_y=0, B_z=0, **funcs.constants)
# params['exp'] = lambda x: 1  # Uncomment this to disable the orbital effect
ham = syst.hamiltonian_submatrix(params=params)
```

```python
kwant.plotter.bands(syst, params=params, show=False, fig_size=(12, 6));
matplotlib.pyplot.ylim(-.1, 100);
```

# Calculate spectrum

```python
def minimize_wrap(x, key, lead_pars, params, band_num):
    import funcs
    from scipy.optimize import minimize_scalar
    lead = funcs.make_wire(**lead_pars)
    params[key] = x
    params = {**params, **funcs.constants}
    bands = lambda k, band_num: kwant.physics.Bands(lead, params=params)(k=k)[band_num]
    return minimize_scalar(bands, tol=1e-3, args=(band_num)).fun
```

```python
lead_pars = dict(a=10, r=50, shape='hexagon', dim=3)
params = dict(alpha=0, mu=0, g=50, B_x=0, B_y=0, B_z=0)
learners = []
for band_num in range(8):
    learner = adaptive.Learner1D(partial(minimize_wrap, key='B_x', lead_pars=lead_pars, params=params, band_num=band_num), (0, 2))
    learners.append(learner)
learner = adaptive.BalancingLearner(learners)
runner = adaptive.Runner(learner, goal=lambda l: l.loss() < 0.01)
```

```python
runner.live_info()
```

```python
plotter = lambda learner: learner.plot().overlay().redim(x='B_x', y='E')
runner.live_plot(plotter=plotter, update_interval=10)
```

# Wavefunction

```python
import holoviews as hv
def lat_from_syst(syst):
    lats = set(s.family for s in syst.sites)
    if len(lats) > 1:
        raise Exception('No unique lattice in the system.')
    return list(lats)[0]

def get_cross_section(syst, pos, direction):
    coord = np.array([s.pos for s in syst.sites if s.pos[direction] == pos])
    cross_section = np.delete(coord, direction, 1)
    return cross_section

def get_densities(lead, k, params):
    xy = get_cross_section(lead, pos=0, direction=0)
    h, t = lead.cell_hamiltonian(params=params), lead.inter_cell_hopping(params=params)
    h_k = h + t * np.exp(1j * k) + t.T.conj() * np.exp(-1j * k)

    vals, vecs = np.linalg.eigh(h_k)
    indxs = np.argsort(np.abs(vals))
    vecs = vecs[:, indxs]
    vals = vals[indxs]

    norbs = lat_from_syst(lead).norbs
    densities = np.linalg.norm(vecs.reshape(-1, norbs, len(vecs)), axis=1)**2
    return xy, vals, densities.T

def plot_wfs_in_cross_section(lead, params, k, num_bands=40):
    xy, energies, densities = get_densities(lead, k, params)
    wfs = [kwant.plotter.mask_interpolate(xy, density, oversampling=1)[0]
           for density in densities[:num_bands]]
    ims = {E: hv.Image(wf) for E, wf in zip(energies, wfs)}
    return hv.HoloMap(ims, kdims=[hv.Dimension('E', unit='meV')])

plot_wfs_in_cross_section(syst, params, k=0)
```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license © 2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.  Based on [CFD Python](https://github.com/barbagroup/CFDPython), © 2013 L.A. Barba, also under CC-BY license.

# Relax and hold steady

Welcome to the second notebook of *"Relax and hold steady: elliptic problems"*, **Module 5** of the course [**"Practical Numerical Methods with Python"**](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/about). Are you relaxed yet?

In the [previous notebook](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_01_2D.Laplace.Equation.ipynb), you learned to use Jacobi iterations to solve Laplace's equation. The iterations *relax* the solution from an initial guess to the final, steady-state solution. You also saw again that the way we treat boundary conditions can influence our solution. Using a first-order approximation of the Neumann boundary messed up our spatial convergence in the whole domain! (We expected second-order spatial convergence from the central difference scheme, but we got closer to first order.) This was easily fixed by using a second-order scheme for the Neumann boundary. *It's always good to check that you get the expected order of convergence.*

A word of warning: in this course module, we will introduce a different use of the word *"convergence"*. Before, we used it to refer to the decay of the truncation errors (in space and time) with a decrease in the grid spacing ($\Delta x$ and $\Delta t$). Now, we also have a relaxation scheme, and we use the word convergence to mean that the iterative solution approaches the exact solution of the linear system. Sometimes, this is called *algebraic convergence*. We'll concern ourselves with this in the next lesson. But first, let's play with Poisson.

## Poisson equation

The **Poisson equation** has a forcing function that drives the solution to its steady state. Unlike the Laplace equation, Poisson's equation involves imposed values inside the field (a.k.a., sources):

$$
\frac{\partial ^2 p}{\partial x^2} + \frac{\partial ^2 p}{\partial y^2} = b
$$

In discretized form, this looks almost the same as [the Laplace Equation](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_01_2D.Laplace.Equation.ipynb), except for the source term on the right-hand side:

$$
\frac{p_{i+1,j}^{k}-2p_{i,j}^{k}+p_{i-1,j}^{k}}{\Delta x^2}+\frac{p_{i,j+1}^{k}-2 p_{i,j}^{k}+p_{i,j-1}^{k}}{\Delta y^2}=b_{i,j}^{k}
$$

As before, we rearrange this to obtain an equation for $p$ at point $i,j$, based on its neighbors:

$$
p_{i,j}^{k+1}=\frac{(p_{i+1,j}^{k}+p_{i-1,j}^{k})\Delta y^2+(p_{i,j+1}^{k}+p_{i,j-1}^{k})\Delta x^2-b_{i,j}^{k}\Delta x^2\Delta y^2}{2(\Delta x^2+\Delta y^2)}
$$

It's slightly more complicated than the Laplace equation, but nothing we can't handle.

### An example problem

Let's consider the following Poisson equation:

$$
\begin{equation}
\nabla^2 p = -2\left(\frac{\pi}{2}\right)^2\sin\left( \frac{\pi x}{L_x} \right) \cos\left(\frac{\pi y}{L_y}\right)
\end{equation}
$$

in the domain

$$
\left\lbrace \begin{align*}
0 &\leq x\leq 1  \\
-0.5 &\leq y \leq 0.5
\end{align*} \right.
$$

where $L_x = L_y = 1$ and with Dirichlet boundary conditions

$$p=0 \text{ at } \left\lbrace
\begin{align*}
x&=0\\
y&=0\\
y&=-0.5\\
y&=0.5
\end{align*} \right.$$

To solve this equation, we assume an initial state of $p=0$ everywhere, apply the boundary conditions and then iteratively relax the system until we converge on a solution.

To start, let's import the libraries and set up our spatial mesh.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

```python
# Set parameters.
nx = 41  # number of points in the x direction
ny = 41  # number of points in the y direction
xmin, xmax = 0.0, 1.0  # domain limits in the x direction
ymin, ymax = -0.5, 0.5  # domain limits in the y direction
Lx = (xmax - xmin)  # domain length in the x direction
Ly = (ymax - ymin)  # domain length in the y direction
dx = Lx / (nx - 1)  # grid spacing in the x direction
dy = Ly / (ny - 1)  # grid spacing in the y direction

# Create the gridline locations.
x = numpy.linspace(xmin, xmax, num=nx)
y = numpy.linspace(ymin, ymax, num=ny)
```

```python
def poisson_source(x, y, Lx, Ly):
    """
    Computes and returns the source term (right-hand side)
    of the Poisson equation.

    Parameters
    ----------
    x : numpy.ndarray
        The gridline locations in the x direction
        as a 1D array of floats.
    y : numpy.ndarray
        The gridline locations in the y direction
        as a 1D array of floats.
    Lx : float
        Domain length in the x direction.
    Ly : float
        Domain length in the y direction.

    Returns
    -------
    b : numpy.ndarray of floats
        The forcing function as a 2D array.
    """
    X, Y = numpy.meshgrid(x, y)
    b = (-2.0 * numpy.pi / Lx * numpy.pi / Ly *
         numpy.sin(numpy.pi * X / Lx) *
         numpy.cos(numpy.pi * Y / Ly))
    return b
```

The Jacobi iterations need an exit condition, based on some norm of the difference between two consecutive iterations. We can use the same relative L2-norm that we wrote for the Laplace exit condition, so we saved the function into a helper Python file (`helper.py`) for easy importing.

```python
from helper import l2_norm
```

Now, what value to choose for the exit condition? We saw in the previous notebook that with an exit tolerance of $10^{-8}$, we could converge well for the different grids we tried, and observe second-order spatial convergence (with the second-order Neumann BC). We speculated in the end that we might be able to use a less stringent exit tolerance, since the spatial error was a lot larger (around $0.0002$ for the finer grid). Here, we'll try with $2\times 10^{-7}$. Go ahead and try with different values and see what you get!

It's time to write the function to solve the Poisson equation.  Notice that all of the boundaries in this problem are Dirichlet boundaries, so no BC updates required!

There's also one extra piece we're adding in here.  To later examine the convergence of the iterative process, we will save the L2-norm of the difference between successive solutions. A plot of this quantity with respect to the iteration number will be an indication of how fast the relaxation scheme is converging.

```python
def poisson_2d_jacobi(p0, b, dx, dy, maxiter=20000, rtol=1e-6):
    """
    Solves the 2D Poisson equation for a given forcing term
    using Jacobi relaxation method.

    The function assumes Dirichlet boundary conditions with value zero.
    The exit criterion of the solver is based on the relative L2-norm
    of the solution difference between two consecutive iterations.

    Parameters
    ----------
    p0 : numpy.ndarray
        The initial solution as a 2D array of floats.
    b : numpy.ndarray
        The forcing term as a 2D array of floats.
    dx : float
        Grid spacing in the x direction.
    dy : float
        Grid spacing in the y direction.
    maxiter : integer, optional
        Maximum number of iterations to perform;
        default: 20000.
    rtol : float, optional
        Relative tolerance for convergence;
        default: 1e-6.

    Returns
    -------
    p : numpy.ndarray
        The solution after relaxation as a 2D array of floats.
    ite : integer
        The number of iterations performed.
    conv : list
        The convergence history as a list of floats.
    """
    p = p0.copy()
    conv = []  # convergence history
    diff = rtol + 1.0  # initial difference
    ite = 0  # iteration index
    while diff > rtol and ite < maxiter:
        pn = p.copy()
        p[1:-1, 1:-1] = (((pn[1:-1, :-2] + pn[1:-1, 2:]) * dy**2 +
                          (pn[:-2, 1:-1] + pn[2:, 1:-1]) * dx**2 -
                          b[1:-1, 1:-1] * dx**2 * dy**2) /
                         (2.0 * (dx**2 + dy**2)))
        # Dirichlet boundary conditions at automatically enforced.
        # Compute and record the relative L2-norm of the difference.
        diff = l2_norm(p, pn)
        conv.append(diff)
        ite += 1
    return p, ite, conv
```

We can use the `plot_3d` function we wrote in the previous notebook to explore the field $p$, before and after the relaxation. We saved this plotting function into the helper Python file, so we can re-use it here.

```python
from helper import plot_3d
```

Now we initialize all of the problem variables and plot!

```python
# Set the initial conditions.
p0 = numpy.zeros((ny, nx))

# Compute the source term.
b = poisson_source(x, y, Lx, Ly)

# Plot the initial scalar field.
plot_3d(x, y, p0)
```

That looks suitably boring. Zeros everywhere and boundaries held at zero. If this were a Laplace problem we would already be done!

But the Poisson problem has a source term that will evolve this zero initial guess to something different. Let's run our relaxation scheme and see what effect the forcing function has on `p`.

```python
# Compute the solution using Jacobi relaxation method.
p, ites, conv = poisson_2d_jacobi(p0, b, dx, dy, rtol=2e-7)
print('Jacobi relaxation: {} iterations '.format(ites) +
      'to reach a relative difference of {}'.format(conv[-1]))
```

    Jacobi relaxation: 3125 iterations to reach a relative difference of 1.9958631078740742e-07

It took 3,125 iterations to converge to the exit criterion (that's quite a lot, don't you think? Let's now take a look at a plot of the final field:

```python
# Plot the solution.
plot_3d(x, y, p)
```

Something has definitely happened. That looks good, but what about the error? This problem has the following analytical solution:

$$
\begin{equation}
p(x,y) = \sin{\left(\frac{x\pi}{L_x} \right)}\cos{\left(\frac{y\pi}{L_y} \right)}
\end{equation}
$$

Time to compare the calculated solution to the analytical one. Let's do that.

```python
def poisson_solution(x, y, Lx, Ly):
    """
    Computes and returns the analytical solution of the Poisson equation
    on a given two-dimensional Cartesian grid.

    Parameters
    ----------
    x : numpy.ndarray
        The gridline locations in the x direction
        as a 1D array of floats.
    y : numpy.ndarray
        The gridline locations in the y direction
        as a 1D array of floats.
    Lx : float
        Length of the domain in the x direction.
    Ly : float
        Length of the domain in the y direction.

    Returns
    -------
    p : numpy.ndarray
        The analytical solution as a 2D array of floats.
    """
    X, Y = numpy.meshgrid(x, y)
    p = numpy.sin(numpy.pi * X / Lx) * numpy.cos(numpy.pi * Y / Ly)
    return p
```

```python
# Compute the analytical solution.
p_exact = poisson_solution(x, y, Lx, Ly)
```

```python
# Compute the relative L2-norm of the error.
l2_norm(p, p_exact)
```

    0.00044962635351970283

That seems small enough. Of course, each application problem can have different accuracy requirements.

### Algebraic convergence

Remember that we saved the L2-norm of the difference between two consecutive iterations. The purpose of that was to look at how the relaxation scheme *converges*, in algebraic sense: with consecutive solutions getting closer and closer to each other. Let's use a line plot for this.

```python
# Plot the convergence history.
pyplot.figure(figsize=(9.0, 4.0))
pyplot.xlabel('Iterations')
pyplot.ylabel('Relative $L_2$-norm\nof the difference')
pyplot.grid()
pyplot.semilogy(conv, color='C0', linestyle='-', linewidth=2)
pyplot.xlim(0, len(conv));
```

It looks like in the beginning, iterations started converging pretty fast, but they quickly adopted a slower rate. As we saw before, it took more than 3,000 iterations to get to our target difference between two consecutive solutions (in L2-norm). That is a *lot* of iterations, and we would really like to relax faster! No worries, we'll learn to do that in the next notebook.

### Spatial convergence

For a sanity check, let's make sure the solution is achieving the expected second-order convergence in space.

```python
# List of the grid sizes to investigate.
nx_values = [11, 21, 41, 81]

# Create an empty list to record the error on each grid.
errors = []

# Compute the solution and error for each grid size.
for nx in nx_values:
    ny = nx  # same number of points in all directions
    dx = Lx / (nx - 1)  # grid spacing in the x direction
    dy = Ly / (ny - 1)  # grid spacing in the y direction
    # Create the gridline locations.
    x = numpy.linspace(xmin, xmax, num=nx)
    y = numpy.linspace(ymin, ymax, num=ny)
    # Set the initial conditions.
    p0 = numpy.zeros((ny, nx))
    # Compute the source term.
    b = poisson_source(x, y, Lx, Ly)
    # Relax the solution.
    # We do not return number of iterations
    # or the convergence history.
    p, ites, _ = poisson_2d_jacobi(p0, b, dx, dy, rtol=2e-7)
    print('[nx = {}] Number of Jacobi iterations: {}'.format(nx, ites))
    # Compute the analytical solution.
    p_exact = poisson_solution(x, y, Lx, Ly)
    # Compute and record the relative L2-norm of the error.
    errors.append(l2_norm(p, p_exact))
```

    [nx = 11] Number of Jacobi iterations: 249
    [nx = 21] Number of Jacobi iterations: 892
    [nx = 41] Number of Jacobi iterations: 3125
    [nx = 81] Number of Jacobi iterations: 10708

```python
# Plot the error versus the grid-spacing size.
pyplot.figure(figsize=(6.0, 6.0))
pyplot.xlabel(r'$\Delta x$')
pyplot.ylabel('Relative $L_2$-norm\nof the error')
pyplot.grid()
dx_values = Lx / (numpy.array(nx_values) - 1)
pyplot.loglog(dx_values, errors,
              color='black', linestyle='--', linewidth=2, marker='o')
pyplot.axis('equal');
```

That looks pretty much second order! Remember that the boundary conditions can adversely affect convergence, but Dirichlet boundaries are "exact" and will never impact your convergence.

## Final word

We have used the difference between two consecutive solutions in the iterative process as a way to indicate convergence. However, this is *not* in general the best idea. For some problems and some iterative methods, you could experience iterates *stagnating* but the solution *not converging*.

Convergence of an iterative solution of a system $A \mathbf{x} = \mathbf{b}$ means that:

$$
\begin{equation}
\lim_{k \rightarrow \infty} \mathbf{x}^k = \mathbf{x}
\end{equation}
$$

The error in the solution is actually $\mathbf{x}-\mathbf{x}^k$, but we're looking at $\mathbf{x}^{k+1}-\mathbf{x}^k$ for our exit criterion. They are not the same thing and the second could tend to zero (or machine precision) without the first being comparably small.

A discussion of better ways to apply stopping criteria for iterative methods is a more advanced topic than we want cover in this course module. Just keep this in mind as you continue your exploration of numerical methods in the future!

---
###### The cell below loads the style of the notebook

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


# Water heating

An insulated, rigid tank contains 4 kg of water at 100 kPa, where initially 0.25 of the mass is liquid. An electric heater turns on and operates until all of the liquid has vaporized. (Neglect the heat capacity of the tank and heater.)

**Problem:**
- Determine the final temperature and pressure of the water.
- Determine the electrical work required by this process.
- Determine the total change in entropy associated with this process.
- Plot the state points for the water on a temperature-specific entropy diagram.

First, load the necessary modules and specify the known/initial conditions.

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import cantera as ct

import matplotlib_inline.backend_inline
matplotlib_inline.backend_inline.set_matplotlib_formats('pdf', 'png')
plt.rcParams['figure.dpi']= 150
plt.rcParams['savefig.dpi'] = 150

from pint import UnitRegistry
ureg = UnitRegistry()
Q_ = ureg.Quantity
```

```python
mass = Q_(4, 'kg')
pressure_initial = Q_(100, 'kPa')
quality_initial = 0.25

quality_final = 1.0

# specify the initial state using pressure and quality
state_initial = ct.Water()
state_initial.PQ = pressure_initial.to('Pa').magnitude, quality_initial
state_initial()
```

      water:

           temperature   372.81 K
              pressure   1e+05 Pa
               density   2.3567 kg/m^3
      mean mol. weight   18.016 kg/kmol
        vapor fraction   0.25
       phase of matter   liquid-gas-mix

                              1 kg             1 kmol
                         ---------------   ---------------
              enthalpy       -1.4989e+07       -2.7004e+08  J
       internal energy       -1.5031e+07        -2.708e+08  J
               entropy            6337.1        1.1417e+05  J/K
        Gibbs function       -1.7351e+07        -3.126e+08  J
     heat capacity c_p   inf               inf              J/K
     heat capacity c_v   nan               nan              J/K

## Find final temperature and pressure

Due to conservation of mass, since the mass and volume of the system are fixed, the specific volume and density must be constant:

$$
v_2 = v_1 \\
\rho_2 = \rho_1
$$

Therefore the final state is fixed by the density and quality, where $x_2 = 1$:

```python
state_final = ct.Water()
state_final.DQ = state_initial.density, quality_final
```

Hmm, what happened here? It looks like Cantera unfortunately does not support specifying the thermodynamic state using density and quality. (With quality as one property, it only supports temperature or pressure as the other property.)

Fortunately, CoolProp *does* support specifying the state the way we need to solve this problem, so let's use that for the final state:

```python
from CoolProp.CoolProp import PropsSI

temp_final = PropsSI(
    'T', 'D', state_initial.density, 'Q', quality_final, 'water'
    ) * ureg.kelvin
pres_final = PropsSI(
    'P', 'D', state_initial.density, 'Q', quality_final, 'water'
    ) * ureg.pascal

print(f'Final temperature: {temp_final: .2f}')
print(f'Final pressure: {pres_final: .2f}')

# We can then set the final state using the Cantera object,
# now that we know temperature
state_final = ct.Water()
state_final.TQ = temp_final.magnitude, quality_final
```

    Final temperature: 420.08 kelvin
    Final pressure: 438257.38 pascal

## Find electrical work required

To find the work required, we can do an energy balance on the (closed) system:
\begin{equation}
W_{\text{in}} = m (u_2 - u_1)
\end{equation}

```python
work = mass * (Q_(state_final.u, 'J/kg') - Q_(state_initial.u, 'J/kg'))
print(f'Electrical work required: {work.to(ureg.megajoule): .2f}')
```

    Electrical work required: 6.47 megajoule

## Find entropy change

The total entropy change is the change in entropy of the system plus that of the surroundings:

$$
\Delta S_{\text{total}} = \Delta S_{\text{system}} + \Delta S_{\text{surr}} \\
\Delta S_{\text{total}} = \Delta S_{\text{system}} = m (s_2 - s_1)
$$

since the entropy change of the surroundings is zero.

```python
entropy_change = mass * (Q_(state_final.s, 'J/kg') - Q_(state_initial.s, 'J/kg'))
print(f'Entropy change: {entropy_change: .2f}')
```

    Entropy change: 16195.98 joule

This process is irreversible, associated with a positive increase in total entropy.

## Plot the state points for water

We can construct the saturated liquid and saturated vapor lines in a temperature–specific entropy diagram (T–s diagram), and then plot the initial and final states locations along with the process line (of constant density):

```python
f = ct.Water()

# Array of temperatures from fluid minimum temperature to critical temperature
temps = np.arange(np.ceil(f.min_temp) + 0.15, f.critical_temperature, 1.0)

def get_sat_entropy_fluid(T):
    '''Gets entropy for temperature along saturated liquid line'''
    f = ct.Water()
    f.TQ = T, 0.0
    return f.s
def get_sat_entropy_gas(T):
    '''Gets entropy for temperature along saturated vapor line'''
    f = ct.Water()
    f.TQ = T, 1.0
    return f.s

# calculate entropy values associated with temperatures along
# saturation lines
entropies_f = np.array([get_sat_entropy_fluid(T) for T in temps])
entropies_g = np.array([get_sat_entropy_gas(T) for T in temps])

# critical point
f.TP = f.critical_temperature, f.critical_pressure

fig, ax = plt.subplots(figsize=(5, 3))

# Plot the saturated liquid line, critical point,
# and saturated vapor line
ax.plot(entropies_f, temps)
ax.plot(entropies_g, temps)
ax.plot(f.s, f.T, 'o')
plt.xlabel('Specific entropy (J/kg⋅K)')
plt.ylabel('Temperature (K)')

# Plot the initial and final states, and label them
ax.plot(state_initial.s, state_initial.T, 's')
ax.annotate('(1)', xy=(state_initial.s, state_initial.T),
            xytext=(0, -20), textcoords='offset points',
            ha='right', va='bottom'
            )

ax.plot(state_final.s, state_final.T, 's')
ax.annotate('(2)', xy=(state_final.s, state_final.T),
            xytext=(20, 0), textcoords='offset points',
            ha='right', va='bottom'
            )

# show process line of constant density
temps = np.arange(state_initial.T, state_final.T, 1.0)
def get_entrophy(T, density):
    f = ct.Water()
    f.TD = T, density
    return f.s
entropies = np.array([get_entrophy(T, state_initial.density) for T in temps])
ax.plot(entropies, temps, '--')

plt.grid(True)
fig.tight_layout()
plt.show()
```


```python
# This cell is added by sphinx-gallery
# It can be customized to whatever you like
%matplotlib inline
```

Noisy circuits
==============

.. meta::
    :property="og:description": Learn how to simulate noisy quantum circuits
    :property="og:image": https://pennylane.ai/qml/_images/N-Nisq.png

.. related::

    tutorial_noisy_circuit_optimization Optimizing noisy circuits with Cirq
    pytorch_noise PyTorch and noisy devices

In this demonstration, you'll learn how to simulate noisy circuits using built-in functionality in
PennyLane. We'll cover the basics of noisy channels and density matrices, then use example code to
simulate noisy circuits. PennyLane, the library for differentiable quantum computations, has
unique features that enable us to compute gradients of noisy channels. We'll also explore how
to employ channel gradients to optimize noise parameters in a circuit.

We're putting the N in NISQ.

.. figure:: ../demonstrations/noisy_circuits/N-Nisq.png
    :align: center
    :width: 20%

    ..

Noisy operations
----------------
Noise is any unwanted transformation that corrupts the intended
output of a quantum computation. It can be separated into two categories.

* **Coherent noise** is described by unitary operations that maintain the purity of the
  output quantum state. A common source are systematic errors originating from
  imperfectly-calibrated devices that do not exactly apply the desired gates, e.g., applying
  a rotation by an angle $\phi+\epsilon$ instead of $\phi$.

* **Incoherent noise** is more problematic: it originates from a quantum computer
  becoming entangled with the environment, resulting in mixed states --- probability
  distributions over different pure states. Incoherent noise thus leads to outputs that are
  always random, regardless of what basis we measure in.

Mixed states are described by `density matrices
<https://en.wikipedia.org/wiki/Density_matrices>`__.
They provide a more general method of describing quantum states that elegantly
encodes a distribution over pure states in a single mathematical object.
Mixed states are the most general description of a quantum state, of which pure
states are a special case.

The purpose of PennyLane's ``default.mixed`` device is to provide native
support for mixed states and for simulating noisy computations. Let's use ``default.mixed`` to
simulate a simple circuit for preparing the
Bell state $|\psi\rangle=\frac{1}{\sqrt{2}}(|00\rangle+|11\rangle)$. We ask the QNode to
return the expectation value of $Z_0\otimes Z_1$:

```python
import pennylane as qml
from pennylane import numpy as np

dev = qml.device('default.mixed', wires=2)

@qml.qnode(dev)
def circuit():
    qml.Hadamard(wires=0)
    qml.CNOT(wires=[0, 1])
    return qml.expval(qml.PauliZ(0) @ qml.PauliZ(1))

print(f"QNode output = {circuit():.4f}")
```

The device stores the output state as a density matrix. In this case, the density matrix is
equal to $|\psi\rangle\langle\psi|$,
where $|\psi\rangle=\frac{1}{\sqrt{2}}(|00\rangle + |11\rangle)$.

```python
print(f"Output state is = \n{np.real(dev.state)}")
```

Incoherent noise is modelled by
quantum channels. Mathematically, a quantum channel is a linear, completely positive,
and trace-preserving (`CPTP
<https://www.quantiki.org/wiki/channel-cp-map>`__) map. A convenient strategy for representing
quantum channels is to employ `Kraus operators
<https://en.wikipedia.org/wiki/Quantum_operation#Kraus_operators>`__
$\{K_i\}$ satisfying the condition
$\sum_i K_{i}^{\dagger} K_i = I$. For an initial state $\rho$, the output
state after the action of a channel $\Phi$ is:

\begin{align}\Phi(\rho) = \sum_i K_i \rho K_{i}^{\dagger}.\end{align}

Just like pure states are special cases of mixed states, unitary
transformations are special cases of quantum channels. Unitary transformations are represented
by a single Kraus operator,
the unitary $U$, and they transform a state as
$U\rho U^\dagger$.

More generally, the action of a quantum channel can be interpreted as applying a
transformation corresponding to the Kraus operator $K_i$ with some associated
probability. More precisely, the channel applies the
transformation
$\frac{1}{p_i}K_i\rho K_i^\dagger$ with probability $p_i = \text{Tr}[K_i \rho K_{i}^{
\dagger}]$. Quantum
channels therefore represent a probability distribution over different possible
transformations on a quantum state. For
example, consider the bit flip channel. It describes a transformation that flips the state of
a qubit (applies an X gate) with probability $p$ and leaves it unchanged with probability
$1-p$. Its Kraus operators are

\begin{align}K_0 &= \sqrt{1-p}\begin{pmatrix}1 & 0\\ 0 & 1\end{pmatrix}, \\
    K_1 &= \sqrt{p}\begin{pmatrix}0 & 1\\ 1 & 0\end{pmatrix}.\end{align}

This channel can be implemented in PennyLane using the :class:`qml.BitFlip <pennylane.BitFlip>`
operation.

Let's see what happens when we simulate this type of noise acting on
both qubits in the circuit. We'll evaluate the QNode for different bit flip probabilities.

```python
@qml.qnode(dev)
def bitflip_circuit(p):
    qml.Hadamard(wires=0)
    qml.CNOT(wires=[0, 1])
    qml.BitFlip(p, wires=0)
    qml.BitFlip(p, wires=1)
    return qml.expval(qml.PauliZ(0) @ qml.PauliZ(1))

ps = [0.001, 0.01, 0.1, 0.2]
for p in ps:
    print(f"QNode output for bit flip probability {p} is {bitflip_circuit(p):.4f}")
```

The circuit behaves quite differently in the presence of noise! This will be familiar to anyone
that has run an algorithm on quantum hardware. It is also highlights why error
mitigation and error correction are so important. We can use PennyLane to look under the hood and
see the output state of the circuit for the largest noise parameter

```python
print(f"Output state for bit flip probability {p} is \n{np.real(dev.state)}")
```

Besides the bit flip channel, PennyLane supports several other noisy channels that are commonly
used to describe experimental imperfections: :class:`~.pennylane.PhaseFlip`,
:class:`~.pennylane.AmplitudeDamping`, :class:`~.pennylane.GeneralizedAmplitudeDamping`,
:class:`~.pennylane.PhaseDamping`, and the :class:`~.pennylane.DepolarizingChannel`. You can also
build your own custom channel using the operation :class:`~.pennylane.QubitChannel` by
specifying its Kraus operators, or even submit a `pull request
<https://pennylane.readthedocs.io/en/stable/development/guide.html>`__ introducing a new channel.

Let's take a look at another example. The depolarizing channel is a
generalization of
the bit flip and phase flip channels, where each of the three possible Pauli errors can be
applied to a single qubit. Its Kraus operators are given by

\begin{align}K_0 &= \sqrt{1-p}\begin{pmatrix}1 & 0\\ 0 & 1\end{pmatrix}, \\
    K_1 &= \sqrt{p/3}\begin{pmatrix}0 & 1\\ 1 & 0\end{pmatrix}, \\
    K_2 &= \sqrt{p/3}\begin{pmatrix}0 & -i\\ i & 0\end{pmatrix}, \\
    K_3 &= \sqrt{p/3}\begin{pmatrix}1 & 0\\ 0 & -1\end{pmatrix}.\end{align}

A circuit modelling the effect of depolarizing noise in preparing a Bell state is implemented
below.

```python
@qml.qnode(dev)
def depolarizing_circuit(p):
    qml.Hadamard(wires=0)
    qml.CNOT(wires=[0, 1])
    qml.DepolarizingChannel(p, wires=0)
    qml.DepolarizingChannel(p, wires=1)
    return qml.expval(qml.PauliZ(0) @ qml.PauliZ(1))

ps = [0.001, 0.01, 0.1, 0.2]
for p in ps:
    print(f"QNode output for depolarizing probability {p} is {depolarizing_circuit(p):.4f}")
```

As before, the output deviates from the desired value as the amount of
noise increases.
Modelling the noise that occurs in real experiments requires careful consideration.
PennyLane
offers the flexibility to experiment with different combinations of noisy channels to either mimic
the performance of quantum algorithms when deployed on real devices, or to explore the effect
of more general quantum transformations.

Channel gradients
-----------------

The ability to compute gradients of any operation is an essential ingredient of
:doc:`quantum differentiable programming </glossary/quantum_differentiable_programming>`.
In PennyLane, it is possible to
compute gradients of noisy channels and optimize them inside variational circuits.
PennyLane supports analytical
gradients for channels whose Kraus operators are proportional to unitary
matrices [#johannes]_. In other cases, gradients are evaluated using finite differences.

To illustrate this property, we'll consider an elementary example. We aim to learn the noise
parameters of a circuit in order to reproduce an observed expectation value. So suppose that we
run the circuit to prepare a Bell state
on a hardware device and observe that the expectation value of $Z_0\otimes Z_1$ is
not equal to 1 (as would occur with an ideal device), but instead has the value 0.7781. In the
experiment, it is known that the
major source of noise is amplitude damping, for example as a result of photon loss.
Amplitude damping projects a state to $|0\rangle$ with probability $p$ and
otherwise leaves it unchanged. It is
described by the Kraus operators

\begin{align}K_0 = \begin{pmatrix}1 & 0\\ 0 & \sqrt{1-p}\end{pmatrix}, \quad
    K_1 = \begin{pmatrix}0 & \sqrt{p}\\ 0 & 0\end{pmatrix}.\end{align}

What damping parameter ($p$) explains the experimental outcome? We can answer this question
by optimizing the channel parameters to reproduce the experimental
observation! 💪 Since the parameter $p$ is a probability, we use a sigmoid function to
ensure that the trainable parameters give rise to a valid channel parameter, i.e., a number
between 0 and 1.

```python
ev = np.tensor([0.7781], requires_grad=False)  # observed expectation value

def sigmoid(x):
    return 1/(1+np.exp(-x))

@qml.qnode(dev)
def damping_circuit(x):
    qml.Hadamard(wires=0)
    qml.CNOT(wires=[0, 1])
    qml.AmplitudeDamping(sigmoid(x), wires=0)  # p = sigmoid(x)
    qml.AmplitudeDamping(sigmoid(x), wires=1)
    return qml.expval(qml.PauliZ(0) @ qml.PauliZ(1))
```

We optimize the circuit with respect to a simple cost function that attains its minimum when
the output of the QNode is equal to the experimental value:

```python
def cost(x, target):
    return (damping_circuit(x) - target[0])**2
```

All that remains is to optimize the parameter. We use a straightforward gradient descent
method.

```python
opt = qml.GradientDescentOptimizer(stepsize=10)
steps = 35
x = np.tensor([0.0], requires_grad=True)

for i in range(steps):
    (x, ev), cost_val = opt.step_and_cost(cost, x, ev)
    if i % 5 == 0 or i == steps - 1:
        print(f"Step: {i}    Cost: {cost_val}")

print(f"QNode output after optimization = {damping_circuit(x):.4f}")
print(f"Experimental expectation value = {ev[0]}")
print(f"Optimized noise parameter p = {sigmoid(x.take(0)):.4f}")
```

Voilà! We've trained the noisy channel to reproduce the experimental observation. 😎

Developing quantum algorithms that leverage the power of NISQ devices requires serious
consideration of the effects of noise. With PennyLane, you have access to tools that can
help you design, simulate, and optimize noisy quantum circuits. We look forward to seeing what
the quantum community can achieve with them! 🚀 🎉 😸

References
----------

.. [#johannes]

    Johannes Jakob Meyer, Johannes Borregaard, and Jens Eisert, "A variational toolbox for quantum
    multi-parameter estimation." `arXiv:2006.06303 (2020) <https://arxiv.org/abs/2006.06303>`__.


# Neuromatch Academy: Week 1, Day 5, Tutorial 1
# Dimensionality Reduction: Geometric view of data

---

Tutorial objectives
In this notebook we'll explore how multivariate data can be represented in different orthonormal bases. This will help us build intuition that will be helpful in understanding PCA in the following tutorial.

Steps:
 1. Generate correlated multivariate data.
 2. Define an arbitrary orthonormal basis.
 3. Project data onto new basis.

---

```python
#@title Video: Geometric view of data
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="emLW0F-VUag", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=emLW0F-VUag

# Setup
Run these cells to get the tutorial started.

```python
#library imports
import time                        # import time
import numpy as np                 # import numpy
import scipy as sp                 # import scipy
import math                        # import basic math functions
import random                      # import basic random number generator functions

import matplotlib.pyplot as plt    # import matplotlib
from IPython import display

```

```python
#@title Figure Settings
%matplotlib inline
fig_w, fig_h = (8, 8)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
plt.style.use('ggplot')
%config InlineBackend.figure_format = 'retina'
```

```python
#@title Helper functions

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

  Note that samples are sorted in ascending order for the first random variable.

  Args:
    cov_matrix (numpy array of floats): desired covariance matrix

  Returns:
    (numpy array of floats) : samples from the bivariate Gaussian, with
                              each column corresponding to a different random variable
  """
  mean = np.array([0,0])
  X = np.random.multivariate_normal(mean,cov_matrix,size = 1000)
  indices_for_sorting = np.argsort(X[:,0])
  X = X[indices_for_sorting,:]
  return X

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  plot of their joint activity. The title indicates the sample correlation
  calculated from the data.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(X[:,0],color='k')
  plt.ylabel('Neuron 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(X[:,0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(X[:,1],color='k')
  plt.xlabel('Sample Number')
  plt.ylabel('Neuron 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(X[:,1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:,0],X[:,1],'.',markerfacecolor=[.5,.5,.5], markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:,0],X[:,1])[0,1]))

def plot_basis_vectors(X,W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable
    W (numpy array of floats):   Square matrix representing new orthonormal basis
                                  each column represents a basis vector

  Returns:
    Nothing.
  """
  plt.figure(figsize=[4,4])
  plt.plot(X[:,0],X[:,1],'.',color=[.5,.5,.5],label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0,W[0,0]],[0,W[1,0]],color='r',linewidth=3,label = 'Basis vector 1')
  plt.plot([0,W[0,1]],[0,W[1,1]],color='b',linewidth=3,label = 'Basis vector 2')
  plt.legend()

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases. Similar to plot_data but
  with colors corresponding to projections onto basis 1 (red) and basis 2 (blue).
  The title indicates the sample correlation calculated from the data.

  Note that samples are re-sorted in ascending order for the first random variable.

  Args:
    Y (numpy array of floats):   Data matrix in new basis
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(Y[:,0],'r')
  plt.xlabel
  plt.ylabel('Projection \n basis vector 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(Y[:,0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(Y[:,1],'b')
  plt.xlabel('Sample number')
  plt.ylabel('Projection \n basis vector 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(Y[:,1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:,0],Y[:,1],'.',color=[.5,.5,.5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:,0],Y[:,1])[0,1]))

```

# Generate correlated multivariate data

```python
#@title Video: Multivariate data
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="YOan2BQVzTQ", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=YOan2BQVzTQ

To study multivariate data, first we generate it. In this exercise we generate data from a *bivariate normal distribution*. This is an extension of the one-dimensional normal distribution to two dimensions, in which each $x_i$ is marginally normal with mean $\mu_i$ and variance $\sigma_i^2$:

\begin{align}
x_i \sim \mathcal{N}(\mu_i,\sigma_i^2)
\end{align}

Additionally, the joint distribution for $x_1$ and $x_2$ has a specified correlation coefficient $\rho$. Recall that the correlation coefficient is a normalized version of the covariance, and ranges between -1 and +1.

\begin{align}
\rho = \frac{\text{cov}(x_1,x_2)}{\sqrt{\sigma_1^2 \sigma_2^2}}
\end{align}

For simplicity, we will assume that the mean of each variable has already been subtracted, so that $\mu_i=0$. The remaining parameters can be summarized in the covariance matrix:
\begin{equation*}
{\bf \Sigma} =
\begin{pmatrix}
 \text{var}(x_1) & \text{cov}(x_1,x_2) \\
 \text{cov}(x_1,x_2) &\text{var}(x_2)
\end{pmatrix}
\end{equation*}
Note that this is a symmetric matrix with the variances $\text{var}(x_i) = \sigma_i^2$ on the diagonal, and the covariance on the off-diagonal.

### Exercise
We have provided code to draw random samples from a zero-mean bivariate normal distribution. These samples could be used to simulate changes in firing rates for two neurons. Fill in the function below to calculate the covariance matrix given the desired variances and correlation coefficient. The covariance can be found by rearranging the equation above:

\begin{align}
\text{cov}(x_1,x_2) = \rho \sqrt{\sigma_1^2 \sigma_2^2}
\end{align}

Use these functions to generate and plot data while varying the parameters. You should get a feel for how changing the correlation coefficient affects the geometry of the simulated data.

**Suggestions**
* Fill in the function `calculate_cov_matrix` to calculate the covariance.
* Generate and plot the data for $\sigma_1^2 =1$, $\sigma_1^2 =1$, and $\rho = .8$. Try plotting the data for different values of the correlation coefficent: $\rho = -1, -.5, 0, .5, 1$.

```python
help(plot_data)
help(get_data)
```

    Help on function plot_data in module __main__:

    plot_data(X)
        Plots bivariate data. Includes a plot of each random variable, and a scatter
        plot of their joint activity. The title indicates the sample correlation
        calculated from the data.

        Args:
          X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable

        Returns:
          Nothing.

    Help on function get_data in module __main__:

    get_data(cov_matrix)
        Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

        Note that samples are sorted in ascending order for the first random variable.

        Args:
          cov_matrix (numpy array of floats): desired covariance matrix

        Returns:
          (numpy array of floats) : samples from the bivariate Gaussian, with
                                    each column corresponding to a different random variable

```python

def calculate_cov_matrix(var_1,var_2,corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation coefficient.

  Args:
    var_1 (scalar):    variance of the first random variable
    var_2 (scalar):    variance of the second random variable
    corr_coef (scalar):      correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  ###################################################################
  ## Insert your code here to:
  ##                calculate the covariance from the variances and correlation
  # cov = ...
  cov_matrix = np.array([[var_1,cov],[cov,var_2]])

  #uncomment once you've filled in the function
  raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  ###################################################################
  return cov

###################################################################
## Insert your code here to:
##                generate and plot bivariate Gaussian data with variances of 1
##                      and a correlation coefficients of: 0.8
##                repeat while varying the correlation coefficient from -1 to 1
###################################################################
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

#uncomment to test your code and plot
#cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
#X = get_data(cov_matrix)
#plot_data(X)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial1_Solution_62df7ae6.py)

*Example output:*

# Define a new orthonormal basis

```python
#@title Video: Orthonormal bases
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="dK526Nbn2Xo", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=dK526Nbn2Xo

Next, we will define a new orthonormal basis of vectors ${\bf u} = [u_1,u_2]$ and ${\bf w} = [w_1,w_2]$. As we learned in the video, two vectors are orthonormal if:

1.   They are orthogonal (i.e., their dot product is zero):
\begin{equation}
{\bf u\cdot w} = u_1 w_1 + u_2 w_2 = 0
\end{equation}
2.   They have unit length:
\begin{equation}
||{\bf u} || = ||{\bf w} || = 1
\end{equation}

In two dimensions, it is easy to make an arbitrary orthonormal basis. All we need is a random vector ${\bf u}$, which we have normalized. If we now define the second basis vector to be ${\bf w} = [-u_2,u_1]$, we can check that both conditions are satisfied:
\begin{equation}
{\bf u\cdot w} = - u_1 u_2 + u_2 u_1 = 0
\end{equation}
and
\begin{equation}
{|| {\bf w} ||} = \sqrt{(-u_2)^2 + u_1^2} = \sqrt{u_1^2 + u_2^2} = 1,
\end{equation}
where we used the fact that ${\bf u}$ is normalized. So, with an arbitrary input vector, we can define an orthonormal basis, which we will write in matrix by stacking the basis vectors horizontally:

\begin{equation}
{{\bf W} } =
\begin{pmatrix}
 u_1 & w_1 \\
 u_2 & w_2
\end{pmatrix}.
\end{equation}

### Exercise

In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary 2-dimensional vector as an input.

**Suggestions**
* Modify the function `define_orthonormal_basis` to first normalize the first basis vector $\bf u$.
* Then complete the function by finding a basis vector $\bf w$ that is orthogonal to $\bf u$.
* Test the function using initial basis vector ${\bf u} = [3,1]$. Plot the resulting basis vectors on top of the data scatter plot using the function `plot_basis_vectors`. (For the data, use  $\sigma_1^2 =1$, $\sigma_1^2 =1$, and $\rho = .8$).

```python
help(plot_basis_vectors)
```

    Help on function plot_basis_vectors in module __main__:

    plot_basis_vectors(X, W)
        Plots bivariate data as well as new basis vectors.

        Args:
          X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable
          W (numpy array of floats):   Square matrix representing new orthonormal basis
                                        each column represents a basis vector

        Returns:
          Nothing.

```python
def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats):    arbitrary 2-dimensional vector used for new basis

  Returns:
    (numpy array of floats) : new orthonormal basis
                                columns correspond to basis vectors
  """
  ###################################################################
  ## Insert your code here to:
  ##                normalize vector u
  ##                calculate vector w that is orthogonal to w

  #u = ....
  #w = ...
  #W = np.column_stack((u,w))

  #comment this once you've filled the function
  raise NotImplementedError("Student excercise: implement the orthonormal basis function")
  ###################################################################
  return W

variance_1 = 1
variance_2 = 1
corr_coef = 0.8

cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
X = get_data(cov_matrix)
u = np.array([3,1])

#uncomment and run below to plot the basis vectors
##define_orthonormal_basis(u)
#plot_basis_vectors(X,W)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial1_Solution_c9ca4afa.py)

*Example output:*

# Project data onto new basis

```python
#@title Video: Change of basis
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="5MWSUtpbSt0", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=5MWSUtpbSt0

Finally, we will express our data in the new basis that we have just found. Since $\bf W$ is orthonormal, we can project the data into our new basis using simple matrix multiplication :

\begin{equation}
{\bf Y = X W}.
\end{equation}

We will explore the geometry of the transformed data $\bf Y$ as we vary the choice of basis.

#### Exercise
In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary vector as an input.

**Suggestions**
* Complete the function `change_of_basis` to project the data onto the new basis.
* Plot the projected data using the function `plot_data_new_basis`.
* What happens to the correlation coefficient in the new basis? Does it increase or decrease?
* What happens to variance?

```python
def change_of_basis(X,W):
  """
  Projects data onto new basis W.

  Args:
    X (numpy array of floats) : Data matrix
                                each column corresponding to a different random variable
    W (numpy array of floats):  new orthonormal basis
                                columns correspond to basis vectors

  Returns:
    (numpy array of floats) : Data matrix expressed in new basis
  """
  ###################################################################
  ## Insert your code here to:
  ##                project data onto new basis described by W

  #Y = ...
  #comment this once you've filled the function
  raise NotImplementedError("Student excercise: implement change of basis")
  ###################################################################
  return Y

## Unomment below to transform the data by projecting it into the new basis
## Plot the projected data
# Y = change_of_basis(X,W)
# plot_data_new_basis(Y)
# disp(...)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial1_Solution_b434bc0d.py)

*Example output:*

#### Exercise
To see what happens to the correlation as we change the basis vectors, run the cell below. The parameter $\theta$ controls the angle of $\bf u$ in degrees. Use the slider to rotate the basis vectors.

**Questions**
* What happens to the projected data as you rotate the basis?
* How does the correlation coefficient change? How does the variance of the projection onto each basis vector change?
* Are you able to find a basis in which the projected data is uncorrelated?

```python
###### MAKE SURE TO RUN THIS CELL VIA THE PLAY BUTTON TO ENABLE SLIDERS ########

import ipywidgets as widgets

def refresh(theta = 0):
  u = [1,np.tan(theta * np.pi/180.)]
  W = define_orthonormal_basis(u)
  Y = change_of_basis(X,W)
  plot_basis_vectors(X,W)
  plot_data_new_basis(Y)

_ = widgets.interact(refresh,
  theta = (0, 90, 5))
```