Lambda School Data Science

*Unit 4, Sprint 2, Module 1*

---

# Neural Networks (Prepare)

## Learning Objectives
* Part 1: Describe the foundational components of a neural network
* Part 2: Implement a Perceptron from scratch in Python

Neural Networks are a whole new area of study and application that can be intimidating, but which represents some of the most powerful tools and techniques that we possess in machine learning today. In spite of the hype surrounding these topics I hope that you will come to see them as just another tool in your toolbag with their own strengths and weaknesses. They are useful, but they are not a silver bullet, and they are not always preferable to other -perhaps more simple- machine learning methods.

The goal of this week is to familiarize you with the fundamental theory, terminology and libraries that will enable you to approach different neural network architectures (called topologies) in a sophisticated manner. This week will not be a run-through of the history of Neural Networks and each of the individual advancements leading up to current technologies -we don't have time for that. We will spend some time on some older methods, but only to the degree that they will help introduce us to relevant terminology and understand more complex versions of these technologies.

# Foundational Components of Neural Networks (Learn)

## Overview

Neural Networks aren't exactly a new technology, but recent breakthroughs have revitalized the area. For example the "Perceptron" -one of the basic building blocks of the technology- was invented in 1957.

Artificial Neural Networks are a computational model that was inspired by how neural networks in the brain process information. In the brain electrochemical signals flow from earlier neurons through the dendrites of the cell toward the cell body. If the received signals surpass a certain threshold with a given timing then the neuron fires sending a large spike of energy down the axon and through the axon terminals to other neurons down the line.

In Artificial Neural Networks the neurons or "nodes" are similar in that they receive inputs and pass on their signal to the next layer of nodes if a certain threshold is reached, but that's about where the similarities end. Remember that ANNs are not brains. Don't fall into the common trap of assuming that if an Artificial Neural Network has as many nodes as the human brain that it will be just as powerful or just as capable. The goal with ANNs is not to create a realistic model of the brain but to craft robust algorithms and data structures that can model the complex relationships found in data.

## Follow Along

Say you want to buy a house, you're quoted $400,000 for a 2000 square foot house. Is that good? Is that bad? How can we know? Well, we can gather up some more prices of other homes in the area and their square footages to compare.

| Square Footage | Price  |
|----------------|--------|
| 2104           | 399900 |
| 1600           | 329900 |
| 2400           | 369000 |

Ok, so now we have a little bit more data to compare to, how can we compare the price that we're getting to the prices of these houses? We can't compare price directly because all of these homes are different sizes, so we decide to calculate an average price per square foot for these three homes and we'll compare that to the average square footage price for the house that we have been looking at.

It turns out that these houses cost about $180 per square foot. Now, assuming that information we gathered about houses in the area is representative of housing prices in the area we can create a little diagram for ourselves to help us evaluate the price of any home:

What does our neural network predict should be the price of the house that we are looking at?

```python
price = 2000 * 180
print(price)
```

    360000

### Node Maps

We've created our first *extremely* basic neural network, it takes an input, modifies it by a weight (180) and reports an output. Please, instead of thinking about neural networks as if they were brains think about them as a function or a "mapping" from inputs to outputs just like we have established many times before.

What we have here above is what's known as a "Node Map" it's a visual diagram of the architecture or "topology" of our neural network. It's kind of like a flow chart in that it shows the path from inputs to outputs. They are usually color coded and help us understand at a very high level, some of the differences in architecture between kinds of neural networks. Just like with all the areas of machine learning that we have studied before there is a "[zoo](http://www.asimovinstitute.org/neural-network-zoo/)" of neural network architectures:

### Types of Layers:

There are three main types of neuron layers in a typical NN topology:

#### Input or Visible Layers

The Input Layer is what receives input from our dataset. Sometimes it is called the visible layer because it's the only part that is exposed to our data and that our data interacts with directly. Typically node maps are drawn with one input node for each of the different inputs/features/columns of our dataset that will be passed to the network.

### Back to our House Problem

 Is a neural network that predicts home prices based on the average square footage of homes in the are a good one? How can we evaluate the quality of our predictions?

We evaluate the quality of our predictions just like we would any machine learning algorithm. We look at what the algorithm predicted and compare it to the actual price to see how far off we are. We know that our predictions are getting better if our predictions get closer to the actual prices in our training data / testing data.

So how is our Neural Network doing?

```python
import pandas as pd
import matplotlib.pyplot as plt

x = [2104, 1600, 2400]
y = [399.9, 329.9, 369]
predictions = [379, 288, 432]
difference = [21, 42, -63]
difference_squared = [449, 1756, 3969]

df = pd.DataFrame({'sqft': x, "price_in_thous": y, "predictions": predictions,
                   "difference": difference, "diff_squared": difference_squared})
df.head()
```

      sqft
      price_in_thous
      predictions
      difference
      diff_squared

      0
      2104
      399.9
      379
      21
      449

      1
      1600
      329.9
      288
      42
      1756

      2
      2400
      369.0
      432
      -63
      3969

```python
plt.scatter(df.sqft, df.price_in_thous)
plt.xlabel("Square Feet")
plt.ylabel("Price in Thousands of Dollars")
plt.title("Home Price Estimates")

# Plot the line
price_per_sqft = .180

y_hat = [price_per_sqft*x for x in df.sqft]
plt.plot(df.sqft, y_hat, color="red")

plt.show()
```

### Look Familiar?

Well it should because we've just plotted a regression line hypothesis based on the average square foot price of the three homes that we looked at. Our average error for this line is:

```python
print("Mean Squared Error:", df.diff_squared.mean())
```

    Mean Squared Error: 2058.0

Now, we didn't fit this line using any regression techniques so it's likely that it's **not** the line of best fit. But we could try out other lines and see if the error gets bigger or smaller. Well it just so happens that in this two dimensional world that if our line is stuck to the origin on one end then we don't have all of the freedom that we want to control our predictions. We need some way to slide it up and down the y axis so that we can make any line that we want in this two-dimensional world. This y-intercept is a constant value, or in other words it's always weighted by 1. We're going to call this y-intercept our "bias" term. We'll define this a little bit better in a minute.

Now I don't love this diagram because it shows our "bias" term as if it was a new horizontal layer, in reality the bias term interacts with a single layer to affect all layers after the hidden layer that it is associated with. This diagram will give you a better idea:

```python
import numpy as np

weight = .100
bias = 160

X = np.array([2104, 1600, 2400])
y = np.array([399.9, 329.9, 369])
predictions = [weight*x + bias for x in X]
print("Predictions:", predictions)
difference = y-predictions
print("Difference:", difference)
difference_squared = difference**2
print("Difference Squared:", difference_squared)
MSE = difference_squared.sum()/len(difference_squared)
print("Mean Squared Error", MSE)
```

    Predictions: [370.4, 320.0, 400.0]
    Difference: [ 29.5   9.9 -31. ]
    Difference Squared: [870.25  98.01 961.  ]
    Mean Squared Error 643.0866666666665

```python
plt.scatter(X, y)
plt.xlabel("Square Feet")
plt.ylabel("Price in Thousands of Dollars")
plt.title("Home Price Estimates")

# Plot the line
price_per_sqft = .180

y_hat = [weight*x + bias for x in X]
plt.plot(X, y_hat, color="green")

y_hat = [price_per_sqft*x for x in df.sqft]
plt.plot(df.sqft, y_hat, color="red")

plt.show()
```

The bias value in coordination with our weight gives our network the full range of motion that it needs to find the best way to explain the patterns in the data.

Well, what if instead of only looking at square footage we collected a second data point like number of bathrooms there is in each house. Lets also get a few more data points while we're at it:

```python
sqft = [2104, 1600, 2400, 1416, 3000, 1985, 1534, 1427, 1380, 1419]
bathrooms = [3, 3, 3, 2, 4, 4, 3, 3, 3, 3]
price = [399900, 329900, 369000, 232000, 539000, 299900, 314900, 198999,
        212000, 242500]

df = pd.DataFrame({'sqft': sqft, 'bathrooms': bathrooms, 'price': price})
df.head(10)
```

      sqft
      bathrooms
      price

      0
      2104
      3
      399900

      1
      1600
      3
      329900

      2
      2400
      3
      369000

      3
      1416
      2
      232000

      4
      3000
      4
      539000

      5
      1985
      4
      299900

      6
      1534
      3
      314900

      7
      1427
      3
      198999

      8
      1380
      3
      212000

      9
      1419
      3
      242500

### Two Input Nodes

Now that we have two inputs or two x variables how might we modify the architecture of our network to accept these inputs?

You'll notice that now we need a weight for each one of our input nodes, or for each feature of our dataset. If we were to write out the calculation that is happening as our data flows through this network it might look something like this:

### I hope this looks extremely familiar.

Now we have a new network that takes in two inputs. But now instead of guessing at slope and coefficient values and evaluating them. How can we explore more combinations of weights and bias until we find the values that minimize our Mean-Squared Error?

[Lets Try It!](http://jalammar.github.io/visual-interactive-guide-basics-neural-networks/)

### ANNs are not Linear Regression

It just so happens that the one that we have defined in this way acts the same way. We've defined a neural network that does the same thing as Linear Regression for a couple of different reasons

1) To familiarize you with node maps (even if I don't love the ones in the article).

2) To introduce the concepts of "weights" and "biases" in a context that you're already familiar with.

3) Each layer in a feed-forward neural network affects the next layer by a weighted sum of inputs plus a bias term.

4) To make it plain to you that we can search out the optimal weights and biases of a neural network through gradient descent if we have a loss function that evaluates the quality of our predictions compared to the y values in our training data.

### Why are Neural Networks so Powerful?

So if we can use a very simple neural network to represent a linear regression problem, what do neural networks with lots of additional features/layers represent? Why can't we just use linear regression for everything?

#### Nonlinearities

Linear regression is built to fit more or less linear models in n-dimensional space. While it is true that we can fit non-linear features using linear regression as we include polynomial features, the reality is that in order to fit really curvy nonlinear patterns in data in really complex high dimensional features spaces, the number of polynomial terms that we would have to include in a linear or logistic regression model faces a problem of combinatorial explosion in terms of the number of features that would be required.

Well what kinds of data exhibit these really strong nonlinearities? Well, how about images for example:

Think about all of the different forms a car could take in an image yet our human brains recognize them as cars flawlessly. Think about all of the complexity that we would have to take into account. How could you even begin to feature engineer a dataset of pixel values in order to give a regression algorithm something that it could cue on as being a car or not a car?

[Andrew Ng can probably explain this better than I can](https://www.youtube.com/watch?v=1ZhtwInuOD0)

The interactions between layers of neurons in neural networks in a way accounts for that combinatorial explosion within the structure of the algorithm as needed instead of us having to provide it beforehand.

## Challenge

In the corresponding module project, you will be asked to summarize the different components of a neural network.

# The Perceptron (Learn)

## Overview
The first and simplest kind of neural network that we could talk about is the perceptron. A perceptron is just a single node or neuron of a neural network with nothing else. It can take any number of inputs and spit out an output. What a neuron does is it takes each of the input values, multplies each of them by a weight, sums all of these products up, and then passes the sum through what is called an "activation function" the result of which is the final value.

I really like figure 2.1 found in this [pdf](http://www.uta.fi/sis/tie/neuro/index/Neurocomputing2.pdf) even though it doesn't have bias term represented there.

If we were to write what is happening in some verbose mathematical notation, it might look something like this:

\begin{align}
 y = sigmoid(\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)
\end{align}

Understanding what happens with a single neuron is important because this is the same pattern that will take place for all of our networks.

When imagining a neural network I like to think about the arrows as representing the weights, like a wire that has a certain amount of resistance and only lets a certain amount of current through. And I like to think about the node itselef as containing the prescribed activation function that neuron will use to decide how much signal to pass onto the next layer.

### Activation Functions (transfer functions)

In Neural Networks, each node has an activation function. Each node in a given layer typically has the same activation function. These activation functions are the biggest piece of neural networks that have been inspired by actual biology. The activation function decides whether a cell "fires" or not. Sometimes it is said that the cell is "activated" or not. In Artificial Neural Networks activation functions decide how much signal to pass onto the next layer. This is why they are sometimes referred to as transfer functions because they determine how much signal is transferred to the next layer.

##### Common Activation Functions:

## Follow Along
### Implementing a Perceptron from scratch in Python

### Establish training data

```python

```

### Sigmoid activation function and its derivative for updating weights

```python

```

### Updating weights with derivative of sigmoid function:

### Initialize random weights for our three inputs

```python

```

    array([[0.5049808 ],
           [0.60592761],
           [0.45748719]])

```python

```

    array([[ 0.91571635],
           [-1.69922707],
           [-0.84540448]])

```python

```

    array([[0.91106099],
           [0.91779574],
           [0.47330774]])

### Calculate weighted sum of inputs and weights

```python

```

    array([[0.47330774],
           [2.30216446],
           [1.38436872],
           [1.39110348]])

### Output the activated value for the end of 1 training epoch

```python

```

    array([[0.61616635],
           [0.90905614],
           [0.79969172],
           [0.80076835]])

### take difference of output and true values to calculate error

```python

```

    array([[-0.61616635],
           [ 0.09094386],
           [ 0.20030828],
           [-0.80076835]])

### Gradient descent/backprop - magic!

```python

```

```python

```

    array([[-0.14029845],
           [ 0.01861741],
           [ 0.0428529 ],
           [-0.17124209]])

```python

```

```python

```

    array([[0, 1, 1, 0],
           [0, 1, 0, 1],
           [1, 1, 1, 1]])

```python

```

    array([[-1.80396920e-04],
           [ 1.20253345e-04],
           [ 8.01214633e-05],
           [-1.20209213e-04]])

```python

```

    array([[0.97253129],
           [0.76517105],
           [0.22323751]])

### Put it all together

```python
# Steps we've already done:
# 1. Randomly Initialized Weights already. Those are in memory as `weights`
# 2. We've already got input data & correct_outputs

# Update our weights 10,000 times - (fingers crossed that this process reduces error)
for iteration in range(10000):

    # Weighted sum of inputs / weights
    weighted_sum = np.dot(inputs, weights)

    # Activate!
    activated_output = sigmoid(weighted_sum)

    # Cac error
    error = correct_outputs - activated_output

    adjustments = error * sigmoid_derivative(activated_output)

    # Update the Weights
    weights += np.dot(inputs.T, adjustments)

print("Weights after training")
print(weights)

print("Output after training")
print(activated_output)
```

    Weights after training
    [[15.0387592 ]
     [-0.40616679]
     [-7.23343492]]
    Output after training
    [[7.21587773e-04]
     [9.99388545e-01]
     [9.99592566e-01]
     [4.80836879e-04]]

### Perceptrons can classify linearly separable classes

<https://www.bogotobogo.com/python/scikit-learn/Perceptron_Model_with_Iris_DataSet.php>

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', header=None)

# Grab 50 versicolor and 50 virginica
df.iloc[145:150, 0:5]
```

      0
      1
      2
      3
      4

      145
      6.7
      3.0
      5.2
      2.3
      Iris-virginica

      146
      6.3
      2.5
      5.0
      1.9
      Iris-virginica

      147
      6.5
      3.0
      5.2
      2.0
      Iris-virginica

      148
      6.2
      3.4
      5.4
      2.3
      Iris-virginica

      149
      5.9
      3.0
      5.1
      1.8
      Iris-virginica

```python
y = df.iloc[0:100, 4].values
y
```

    array(['Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
           'Iris-setosa', 'Iris-setosa', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor'],
          dtype=object)

```python
y = np.where(y == 'Iris-setosa', -1, 1)
y
```

    array([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
           -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
           -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1,
            1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
            1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
            1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1])

```python
y
```

    array([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
           -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
           -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1,
            1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
            1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
            1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1])

```python
X = df.iloc[0:100, [0, 2]].values
X
```

    array([[5.1, 1.4],
           [4.9, 1.4],
           [4.7, 1.3],
           [4.6, 1.5],
           [5. , 1.4],
           [5.4, 1.7],
           [4.6, 1.4],
           [5. , 1.5],
           [4.4, 1.4],
           [4.9, 1.5],
           [5.4, 1.5],
           [4.8, 1.6],
           [4.8, 1.4],
           [4.3, 1.1],
           [5.8, 1.2],
           [5.7, 1.5],
           [5.4, 1.3],
           [5.1, 1.4],
           [5.7, 1.7],
           [5.1, 1.5],
           [5.4, 1.7],
           [5.1, 1.5],
           [4.6, 1. ],
           [5.1, 1.7],
           [4.8, 1.9],
           [5. , 1.6],
           [5. , 1.6],
           [5.2, 1.5],
           [5.2, 1.4],
           [4.7, 1.6],
           [4.8, 1.6],
           [5.4, 1.5],
           [5.2, 1.5],
           [5.5, 1.4],
           [4.9, 1.5],
           [5. , 1.2],
           [5.5, 1.3],
           [4.9, 1.5],
           [4.4, 1.3],
           [5.1, 1.5],
           [5. , 1.3],
           [4.5, 1.3],
           [4.4, 1.3],
           [5. , 1.6],
           [5.1, 1.9],
           [4.8, 1.4],
           [5.1, 1.6],
           [4.6, 1.4],
           [5.3, 1.5],
           [5. , 1.4],
           [7. , 4.7],
           [6.4, 4.5],
           [6.9, 4.9],
           [5.5, 4. ],
           [6.5, 4.6],
           [5.7, 4.5],
           [6.3, 4.7],
           [4.9, 3.3],
           [6.6, 4.6],
           [5.2, 3.9],
           [5. , 3.5],
           [5.9, 4.2],
           [6. , 4. ],
           [6.1, 4.7],
           [5.6, 3.6],
           [6.7, 4.4],
           [5.6, 4.5],
           [5.8, 4.1],
           [6.2, 4.5],
           [5.6, 3.9],
           [5.9, 4.8],
           [6.1, 4. ],
           [6.3, 4.9],
           [6.1, 4.7],
           [6.4, 4.3],
           [6.6, 4.4],
           [6.8, 4.8],
           [6.7, 5. ],
           [6. , 4.5],
           [5.7, 3.5],
           [5.5, 3.8],
           [5.5, 3.7],
           [5.8, 3.9],
           [6. , 5.1],
           [5.4, 4.5],
           [6. , 4.5],
           [6.7, 4.7],
           [6.3, 4.4],
           [5.6, 4.1],
           [5.5, 4. ],
           [5.5, 4.4],
           [6.1, 4.6],
           [5.8, 4. ],
           [5. , 3.3],
           [5.6, 4.2],
           [5.7, 4.2],
           [5.7, 4.2],
           [6.2, 4.3],
           [5.1, 3. ],
           [5.7, 4.1]])

```python
plt.scatter(X[:50, 0], X[:50, 1], color='red', marker='o', label='setosa')
plt.scatter(X[50:100, 0], X[50:100, 1], color='blue', marker='x', label='versicolor')
plt.xlabel('petal length')
plt.ylabel('sepal length')
plt.legend(loc='upper left')
plt.show()
```

```python
class Perceptron(object):
  def __init__(self, rate = 0.01, niter = 10):
    self.rate = rate
    self.niter = niter

  def fit(self, X, y):
    """Fit training data
    X : Training vectors, X.shape : [#samples, #features]
    y : Target values, y.shape : [#samples]
    """

    # weights
    self.weight = np.zeros(1 + X.shape[1])

    # Number of misclassifications
    self.errors = []  # Number of misclassifications

    for i in range(self.niter):
      err = 0
      for xi, target in zip(X, y):
        delta_w = self.rate * (target - self.predict(xi))
        self.weight[1:] += delta_w * xi
        self.weight[0] += delta_w
        err += int(delta_w != 0.0)
      self.errors.append(err)
    return self

  def net_input(self, X):
    """Calculate net input"""
    return np.dot(X, self.weight[1:]) + self.weight[0]

  def predict(self, X):
    """Return class label after unit step"""
    return np.where(self.net_input(X) >= 0.0, 1, -1)
```

```python
pn = Perceptron(0.1, 10)
pn.fit(X, y)
plt.plot(range(1, len(pn.errors) + 1), pn.errors, marker='o')
plt.xlabel('Epochs')
plt.ylabel('Number of misclassifications')
plt.show()
```

```python
from matplotlib.colors import ListedColormap
def plot_decision_regions(X, y, classifier, resolution=0.02):
  # setup marker generator and color map
  markers = ('s', 'x', 'o', '^', 'v')
  colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
  cmap = ListedColormap(colors[:len(np.unique(y))])

  # plot the decision surface
  x1_min, x1_max = X[:,  0].min() - 1, X[:, 0].max() + 1
  x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
  xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
  np.arange(x2_min, x2_max, resolution))
  Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
  Z = Z.reshape(xx1.shape)
  plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
  plt.xlim(xx1.min(), xx1.max())
  plt.ylim(xx2.min(), xx2.max())

  # plot class samples
  for idx, cl in enumerate(np.unique(y)):
    plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
    alpha=0.8, c=cmap(idx),
    marker=markers[idx], label=cl)
```

```python
>>> plot_decision_regions(X, y, classifier=pn)
>>> plt.xlabel('sepal length [cm]')
>>> plt.ylabel('petal length [cm]')
>>> plt.legend(loc='upper left')
plt.show()
```

## Challenge

You will apply the perceptron to a dataset during today's assignment.

# Sources:

- 3 Blue 1 Brown Neural Network Video: <https://youtu.be/aircAruvnKk>
- Andrew Ng Neural Network Introduction Videos: <https://www.youtube.com/watch?v=1ZhtwInuOD0&list=PLLssT5z_DsK-h9vYZkQkYNWcItqhlRJLN&index=43>


We now extend SEIR model to:

\begin{align}
\dot{S_{i}}	&=-\lambda_{i}(t)S_{i}+\sigma_{i}, \\
\dot{E}_{i}	&=\lambda_{i}(t)S_{i}-\gamma_{E}E_{i},\\
\dot{I}_{i}^{a}	&=\alpha\gamma_{E}E_{i}-\gamma_{I^{a}}I_{i}^{a},\\
\dot{I}_{i}^{s}	&=\bar{\alpha}\gamma_{E}E_{i}-\gamma_{I^{s}}I_{i}^{s},\\
\dot{I}_{i}^{h}	&=h_{i}\gamma_{I^{s}}I_{i}^{s}-\gamma_{I^{h}}I_{i}^{h},\\
\dot{I}_{i}^{c}	&=c_{i}\gamma_{I^{h}}I_{i}^{h}-\gamma_{I^{c}}I_{i}^{c},\\
\dot{I}_{i}^{m}	&=m_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{R}_{i} &=\gamma_{I^{a}}I_{i}^{a}+\bar{h}_{i}\gamma_{I^{s}}I_{i}^{s}+\bar{c}_{i}\gamma_{I^{h}}I_{i}^{h}+\bar{m}_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{N}_{i}	&=\sigma_{i}-I_{i}^{m}.
\end{align}

$\lambda_{i}(t)=\beta\sum_{j=1}^{M}\left(C_{ij}^{a}\frac{I_{j}^{a}}{N_{j}}+C_{ij}^{s}\frac{I_{j}^{s}}{N_{j}}+C_{ij}^{h}\frac{I_{j}^{h}}{N_{j}}\right)$.

Here
* $I^{a}$ : asymptomatic infectives
* $I^{s}$ : symptomatic infectives
* $I^{h}$ : hospitalized infectives
* $I^{c}$ : ICU cases
* $I^{m}$ : mortality
* ${h}_{i}=1-\bar h_{i}$ is the fraction of syptomatics who are hospitalized
* $c_{i}=1-\bar{c}_{i}$, is the fraction of hospitalizeds, who are in ICU
* $m_{i}=1-\bar{m}_{i}$, is the fraction of ICU which leads to mortality
* $C_{ij}^{s}=f^{s}C_{ij}^{a}\equiv f^{s}C_{ij}$
* $C_{ij}^{h}=f^{h}C_{ij}^{a}\equiv f^{h}C_{ij}$.

S ---> E ---> Ia, Is

Ia ---> R

Is ---> Ih, R

Ih ---> Ic, R

Ic ---> Im, R

```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
```

```python
%matplotlib inline
import numpy as np
import pyross
import matplotlib.pyplot as plt
```

```python
M=16  # number of age groups

# load age structure data
my_data = np.genfromtxt('../data/age_structures/UK.csv', delimiter=',', skip_header=1)
aM, aF = my_data[:, 1], my_data[:, 2]

# set age groups
Ni=aM+aF;   Ni=Ni[0:M];  N=np.sum(Ni)
```

```python
# contact matrices
CH, CW, CS, CO = pyross.contactMatrix.UK()

## matrix of total contacts
C=CH+CW+CS+CO

fig,aCF =  plt.subplots(2,2);
aCF[0][0].pcolor(CH, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[0][1].pcolor(CW, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][0].pcolor(CS, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][1].pcolor(CO, cmap=plt.cm.get_cmap('GnBu', 10));
```

## Fraction of asymptomatic infectives is constant

```python
beta  = 0.036692            # infection rate

gE    = 1/5
gIa   = 1./7                # recovery rate of asymptomatic infectives
gIs   = 1./7                # recovery rate of symptomatic infectives
alpha = 0.3                 # fraction of asymptomatic infectives
fsa   = 0.2                 # the self-isolation parameter
fh    = 0
gIh   = 1/14
gIc   = 1/14

sa    = 100*np.ones(M)      #  rate of additional/removal of population by birth etc
sa[0]     = 1500            # birth
sa[12:16] = -300            # mortality

hh   = 0.1*np.ones(M)       # fraction which goes from Is to hospital
cc   = 0.05*np.ones(M)      # fraction which goes from  hospital to ICU
mm   = 0.4*np.ones(M)       # mortality from IC

# initial conditions
Is_0 = np.zeros((M));  #Is_0[6:13]=8;  Is_0[2:6]=4; Is_0[13:16]=4

Ia_0 = 1000*np.ones((M));
R_0  = np.zeros((M))
E_0  = np.zeros((M))
Ih_0  = np.zeros((M))
Ic_0  = np.zeros((M))
Im_0  = np.zeros((M))

S_0  = Ni - (Ia_0 + Is_0 + R_0 + Ih_0 + Ic_0)

# matrix for linearised dynamics
L0 = np.zeros((M, M))
L  = np.zeros((2*M, 2*M))

for i in range(M):
    for j in range(M):
        L0[i,j]=C[i,j]*Ni[i]/Ni[j]

L[0:M, 0:M]     =    alpha*beta/gIs*L0
L[0:M, M:2*M]   = fsa*alpha*beta/gIs*L0
L[M:2*M, 0:M]   =    ((1-alpha)*beta/gIs)*L0
L[M:2*M, M:2*M] = fsa*((1-alpha)*beta/gIs)*L0

r0 = np.max(np.linalg.eigvals(L))
print("The basic reproductive ratio for these parameters is", r0)
```

    The basic reproductive ratio for these parameters is (1.3199080176950944+0j)

```python
# duration of simulation and data file
Tf=480;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC  = np.zeros((Nf))
for i in range(1*M):
        IC += data['X'][:,2*M+i]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC, '-', lw=4, color='#A60628', label='forecast', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```

## Fraction of asymptomatic infectives is age-dependent

```python
alpha = 0.5*np.ones(M)      #  rate of additional/removal of population by birth etc
alpha[6:M] = 0.1              # everyone is symtomatic in large age-group

Ia_0 = 1000*np.ones((M));
```

```python

# duration of simulation and data file
Tf=480;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC1  =  data['X'][:,2*M+1];
IC2  =  data['X'][:,2*M+8]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC1, '-', lw=4, color='#A60628', label='M=1', alpha=0.8)
plt.plot(t, IC2, '-', lw=4, color='green', label='M=16', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```


# Electronic Structure - Basic Structure/Procedure

#### Born-Oppenheimer approximation

In molecules, nuclei are much heavier than electrons, thus they don't move on same time scale; therefore the behaviour of nuclei and electrons can be decoupled.

Therefore, one can first tackle the electronic problem with nuclear coordinates entering only as parameters. The energy levels of the electrons in the molecule can be found by solving the non-relativistic time-independent Schroedinger equation,

$
\begin{align}
\mathcal{H}_{el} |\Psi_n \rangle = E_n |\Psi_n \rangle
\end{align}
$

The ground state energy is given by:

$
\begin{align}
E_0 = \frac{ \langle\Psi_0|H_{el}|\Psi_0\rangle } { \langle\Psi_0|\Psi_0\rangle }
\end{align}
$

We would like to prepare the ground state, $\Psi_0$ on a quantum computer and measure the Hamiltonian expectation value, $E_0$ directly.

### The Hartree-Fock initial state

Good starting point to this problem is the Hartree-Fock method. This method approximates a N-body problem into N one-body problems where each electron evolves in the mean-field of other.

The Hamiltonian can then be expressed in the basis of the solutions of the HF method, also called Molecular Orbitals (MOs).

MOs ($\phi_\mu$) can be occupied or virtual (unoccupied). One MO can contain 2 electrons. For now we work with Spin Orbitals which are associated with a spin up ($\alpha$) or spin down ($\beta$) electron. Thus spin orbitals can contain one electron or be un-occupied.

There are different codes able to find HF solutions: **Gaussian, Psi4, PyQuante, PySCF**

Below we set up a PySCF driver for H2 molecule at equilibrium bond length 0.735 Angstrom, in the singlet state and with no charge.

```python
from qiskit_nature.drivers import UnitsType, Molecule
from qiskit_nature.drivers.second_quantization import (
    ElectronicStructureDriverType,
    ElectronicStructureMoleculeDriver,
)

molecule = Molecule(
    geometry = [
        ["H", [0.0, 0.0, 0.0]],
        ["H", [0.0, 0.0, 0.735]]],
    charge = 0,
    multiplicity = 1
)

driver = ElectronicStructureMoleculeDriver(
    molecule,
    basis = "sto3g",
    driver_type = ElectronicStructureDriverType.PYSCF
)
```

    <frozen importlib._bootstrap>:219: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject

Molecular Hamiltonian is expressed in terms of fermionic operators and these operators mmust be mapped to spin operators to be Qubit Hamiltonian.

Different mapping types: Jordan-Wigner mapping, Parity mapping, Bravyi-Kitaev mapping.

JW mapping usu maps each Spin Orbital to a qubit.

Below, we set up Electronic Structure Problem to generate the Second quantized operator and a qubit converter that will map it to a qubit operator.

```python
from qiskit_nature.problems.second_quantization import ElectronicStructureProblem
from qiskit_nature.converters.second_quantization import QubitConverter
from qiskit_nature.mappers.second_quantization import JordanWignerMapper, ParityMapper
```

```python
es_problem = ElectronicStructureProblem(driver)
second_q_op = es_problem.second_q_ops()
print(second_q_op[0])
```

    Fermionic Operator
    register length=4, number terms=14
      (0.18093119978423106+0j) * ( +_0 -_1 +_2 -_3 )
    + (-0.18093119978423128+0j) * ( +_0 -_1 -_2 +_3 )
    + (-0.18093119978423128+0j) * ( -_0 +_1 +_2 -_3 )
    + (0.18093119978423144+0j) * ( -_0 +_1 -_2 +_3  ...

Transform this Hamiltonian for given driver defined above we get our qubit operator:

```python
qubit_converter = QubitConverter(mapper=JordanWignerMapper())
qubit_op = qubit_converter.convert(second_q_op[0])
print(qubit_op)
```

    0.04523279994605781 * YYYY
    + 0.04523279994605781 * XXYY
    + 0.04523279994605781 * YYXX
    + 0.04523279994605781 * XXXX
    - 0.8105479805373281 * IIII
    - 0.22575349222402358 * ZIII
    + 0.17218393261915543 * IZII
    + 0.12091263261776633 * ZZII
    - 0.22575349222402358 * IIZI
    + 0.17464343068300436 * ZIZI
    + 0.16614543256382414 * IZZI
    + 0.17218393261915543 * IIIZ
    + 0.16614543256382414 * ZIIZ
    + 0.16892753870087904 * IZIZ
    + 0.12091263261776633 * IIZZ

In the minimal (STO-3G) basis set 4 qubits are required. We can reduce number of qubits by using Parity mapping, which allows for removal of 2 qubits by exploiting known symmetries arising from the mapping.

```python
qubit_converter = QubitConverter(mapper=ParityMapper(), two_qubit_reduction=True)
qubit_op = qubit_converter.convert(second_q_op[0], num_particles=es_problem.num_particles)
print(qubit_op)
```

    0.18093119978423114 * XX
    - 1.0523732457728605 * II
    - 0.39793742484317884 * ZI
    + 0.39793742484317884 * IZ
    - 0.011280104256235171 * ZZ

This time only two qubits are required.

The Hamiltonian is ready and can be used in a quantum algorithm to find into about electronic structure of corresponding molecule.


```python
import numpy
from matplotlib import pyplot
%matplotlib inline
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16

import math
```

# Table of Contents

  [1 Euler Method](#no_1)

  [2 Plot](#no_2)

      [2.1 Regular 2D plot](#no_21)

      [2.2 Loglog Plot (2D)](#no_22)

  [3 Norm](#no_3)

      [3.1 L1 Norm](#no_31)

  [4 Order of Convergence](#no_4)

  [5 Second Order / RK / Modified Euler Method](#no_5)

  [6 Tricks](#no_6)

      [6.1 Mask](#no_61)

      [6.2 Animation](#no_62)

  [7 CFL number](#no_7)

  [8 Convection Schemes](#no_8)

      [8.1 Lax-Friedrichs scheme](#no_81)

      [8.2 Lax-Wendroff scheme](#no_82)

      [8.3 MacCormack scheme](#no_83)

      [8.4 The Richtmyer method](#no_84)

# 1 Euler Method

$$
\begin{equation}
u_{n+1} = u_n + \Delta t \,f(u_n)
\end{equation}
$$

```python
def euler_step(u, f, dt, *args):
    """
    Returns the solution at the next time step using Euler's method.

    Parameters
    ----------
    u : numpy.ndarray
        Solution at the previous time step
        as a 1D array of floats.
    f : function
        Function to compute the right-hand side of the system.
    dt : float
        Time-step size.
    args : tuple, optional
        Positional arguments to pass to the function f.

    Returns
    -------
    u_new : numpy.ndarray
        The solution at the next time step
        as a 1D array of floats.
    """
    u_new = u + dt * f(u, *args)
    return u_new

```

[Return to Top](#contents)

# 2 Plot

## 2.1 Regular 2D plot

```python
# dummy x,y,z
dummy_x = numpy.linspace(0.0, 10.0, num=201)
dummy_y = numpy.sin(2.0 * dummy_x)
dummy_z = numpy.cos(2.0 * dummy_x)

custom_title = 'Title'
custom_xlabel = 'xlabel'
custom_ylabel = 'ylabel'

pyplot.figure(figsize=(9.0, 4.0))  # set the size of the figure
pyplot.title(custom_title)  # set the title
pyplot.xlabel(custom_xlabel)  # set the x-axis label
pyplot.ylabel(custom_ylabel)  # set the y-axis label
pyplot.xlim(dummy_x[0], dummy_x[-1])  # set the x-axis limits
pyplot.ylim(numpy.min(dummy_y), numpy.max(dummy_y))  # set the y-axis limits
pyplot.grid()  # set a background grid to improve readability
pyplot.plot(dummy_x, dummy_y, color='C0', label='dummy_y', linestyle='-', linewidth=2)
pyplot.plot(dummy_x, dummy_z, color='C1', label='dummy_z', linestyle='--', linewidth=2)
pyplot.legend();
```

[Return to Top](#contents)

## 2.2 Loglog Plot (2D)

```python
# dummy x,y,z
dummy_x = numpy.array([0.1, 0.05, 0.01, 0.005, 0.001])
dummy_y = dummy_x * 2.0
dummy_z = dummy_x * 3.0

custom_title = 'Title'
custom_xlabel = 'xlabel'
custom_ylabel = 'ylabel'

pyplot.figure(figsize=(9.0, 4.0))  # set the size of the figure
pyplot.title(custom_title)  # set the title
pyplot.xlabel(custom_xlabel)  # set the x-axis label
pyplot.ylabel(custom_ylabel)  # set the y-axis label
pyplot.xlim(dummy_x[0], dummy_x[-1])  # set the x-axis limits
# pyplot.ylim(numpy.min(dummy_y), numpy.max(dummy_y))  # set the y-axis limits
pyplot.grid()  # set a background grid to improve readability
pyplot.loglog(dummy_x, dummy_y, color='C0', label='dummy_y', linestyle='-', linewidth=2, marker='o')
pyplot.loglog(dummy_x, dummy_z, color='C1', label='dummy_z', linestyle='--', linewidth=2, marker='o')
pyplot.legend()
pyplot.axis('equal');
```

[Return to Top](#contents)

# 3 Norm

## 3.1 L1 Norm

```python
def l1_error(z, z_exact, dt):
    error = dt * numpy.sum(numpy.abs(z - z_exact))
    return error
```

```python
def l1_diff(u_coarse, u_fine, dt):
    """
    Returns the difference in the L1-norm between the solution on
    a coarse grid and the solution on a fine grid.

    Parameters
    ----------
    u_coarse : numpy.ndarray
        Solution on the coarse grid as an array of floats.
    u_fine : numpy.ndarray
        Solution on the fine grid as an array of floats.
    dt : float
        Time-step size.

    Returns
    -------
    diff : float
        The difference between the two solutions in the L1-norm
        scaled by the time-step size.
    """
    N_coarse = len(u_coarse)
    N_fine = len(u_fine)
    ratio = math.ceil(N_fine / N_coarse)
    diff = dt * numpy.sum(numpy.abs(u_coarse - u_fine[::ratio]))
    return diff
```

[Return to Top](#contents)

# 4 Order of Convergence

The order of convergence is the rate at which the numerical solution approaches the exact one as the mesh is refined. Considering that we're not comparing with an exact solution, we use 3 grid resolutions that are refined at a constant ratio $r$ to find the *observed order of convergence* ($p$), which is given by:

$$
\begin{equation}
p = \frac{\log \left(\frac{f_3-f_2}{f_2-f_1} \right) }{\log(r)}
\end{equation}
$$

where $f_1$ is the finest mesh solution, and $f_3$ the coarsest.

Please see [Module 1 Lesson 3](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb#Order-of-convergence) for codes in details.

[Return to Top](#contents)

# 5 Second Order / RK / Modified Euler Method

$$
\begin{align}
u_{n+1/2}   & = u_n + \frac{\Delta t}{2} f(u_n) \\
u_{n+1} & = u_n + \Delta t \,\, f(u_{n+1/2})
\end{align}
$$

```python
def rk2_step(u, f, dt, *args):
    """
    Returns the solution at the next time step using 2nd-order
    Runge-Kutta method.

    Parameters
    ----------
    u : numpy.ndarray
        Solution at the previous time step
        as a 1D array of floats.
    f : function
        Function to compute the right-hand side of the system.
    dt : float
        Time-step size.
    args : tuple, optional
        Positional arguments to pass to the function f.

    Returns
    -------
    u_new : numpy.ndarray
        The solution at the next time step
        as a 1D array of floats.
    """
    u_star = u + 0.5 * dt * f(u, *args)
    u_new = u + dt * f(u_star, *args)
    return u_new
```

```python
def leapfrog_step(u_prev, u, f, dt, *args):
    """
    Returns the solution at the next time step using
    the leapfrog method.

    Parameters
    ----------
    u_prev : numpy.ndarray
        Solution at the time step n-1
        as a 1D array of floats.
    u : numpy.ndarray
        Solution at the previous time step
        as a 1D array of floats.
    f : function
        Function to compute the right-hand side of the system.
    dt : float
        Time-step size.
    args : tuple, optional
        Positional arguments to pass to the function f.

    Returns
    -------
    u_new : numpy.ndarray
        The solution at the next time step
        as a 1D array of floats.
    """
    u_new = u_prev + 2.0 * dt * f(u, *args)
    return u_new
```

[Return to Top](#contents)

# 6 Tricks

## 6.1 Mask

```python
# Define the grid point coordinates.
x = numpy.linspace(0.0, 2.0, num=41)
# Set initial conditions with 1.0 everywhere (for now).
u0 = numpy.ones(41)
# Get a list of indices where 0.5 <= x <= 1.0.
mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
print(mask)

# Set initial condition u = 2.0 where 0.5 <= x <= 1.0.
u0[mask] = 2.0
print(u0)
```

    (array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]),)
    [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 1. 1. 1.
     1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

[Return to Top](#contents)

## 6.2 Animation

```python
from matplotlib import animation
from IPython.display import HTML
```

```python
def update_plot(n, u_hist):
    """
    Update the line y-data of the Matplotlib figure.

    Parameters
    ----------
    n : integer
        The time-step index.
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    fig.suptitle('Time step {:0>2}'.format(n))
    line.set_ydata(u_hist[n])
```
fig = pyplot.figure(figsize=(6.0, 4.0))

# Create an animation.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(u_hist,),
                               interval=100)

# Display the video.
HTML(anim.to_html5_video())
### Create interatctive figure with `ipywidgets`
python
import ipywidgets

# create a function to plot your solution
def plot_density(n, rho_hist):
    pyplot.figure(figsize=(8.0,4.0))
    pyplot.xlabel('Distance [km]')
    pyplot.ylabel('Density [cars / km]')
    pyplot.grid()
    pyplot.plot(x, rho_hist[0], label='Initial')
    pyplot.plot(x, rho_hist[n], label='Time step {}'.format(n))
    pyplot.ylim(rho_hist[0].min() - 1.0, rho_hist[0].max() + 1.0)
    pyplot.legend()

# create a ipywidgets slider for the time levels
nt_slider = ipywidgets.IntSlider(value=0, min=0, max=nt, step=1,
                                 description='time step')

# create the ipywidgets interactive figure
w = ipywidgets.interactive(plot_density,
                               n=nt_slider,
                               rho_hist=ipywidgets.fixed(rho_hist))

# display the interactive figure
display(w)
[Return to Top](#contents)

# 7 CFL number

### 1D Convection

$$
\begin{equation}
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0
\end{equation}
$$

$$
\begin{equation}
\sigma = \frac{c \Delta t}{\Delta x} \leq 1
\end{equation}
$$
dt = sigma * dx / c
### 1D Diffusion

$$
\begin{equation}
\frac{\partial u}{\partial t}= \nu \frac{\partial^2 u}{\partial x^2}
\end{equation}
$$

$$
\begin{equation}
\nu \frac{\Delta t}{\Delta x^2} \leq \frac{1}{2}
\end{equation}
$$
dt = sigma * dx**2 / nu  # time-step size
[Return to Top](#contents)

# 8 Convection Schemes

## 8.1 Lax-Friedrichs scheme

$$
\begin{equation}
\rho_i^{n+1} = \frac{1}{2}(\rho^n_{i+1}+\rho^n_{i-1}) - \frac{\Delta t}{2 \Delta x}(F^n_{i+1}-F^n_{i-1})
\end{equation}
$$

```python
def lax_friedrichs(rho0, nt, dt, dx, bc_values, *args):
    """
    Computes the traffic density on the road
    at a certain time given the initial traffic density.
    Integration using Lax-Friedrichs scheme.

    Parameters
    ----------
    rho0 : numpy.ndarray
        The initial traffic density along the road
        as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    dt : float
        The time-step size to integrate.
    dx : float
        The distance between two consecutive locations.
    bc_values : 2-tuple of floats
        The value of the density at the first and last locations.
    args : list or tuple
        Positional arguments to be passed to the flux function.

    Returns
    -------
    rho_hist : list of numpy.ndarray objects
        The history of the car density along the road.
    """
    rho_hist = [rho0.copy()]
    rho = rho0.copy()
    for n in range(nt):
        # Compute the flux.
        F = flux(rho, *args)
        # Advance in time using Lax-Friedrichs scheme.
        rho[1:-1] = (0.5 * (rho[:-2] + rho[2:]) -
                     dt / (2.0 * dx) * (F[2:] - F[:-2]))
        # Set the value at the first location.
        rho[0] = bc_values[0]
        # Set the value at the last location.
        rho[-1] = bc_values[1]
        # Record the time-step solution.
        rho_hist.append(rho.copy())
    return rho_hist
```

[Return to Top](#contents)

## 8.2 Lax-Wendroff scheme

$$
\begin{equation}
J = \frac{\partial F}{\partial \rho} = u _{\rm max} \left(1-2\frac{\rho}{\rho_{\rm max}} \right)
\end{equation}
$$

\begin{equation}J^n_i = \frac{\partial F}{\partial \rho} = u_{\rm max} \left(1-2\frac{\rho^n_i}{\rho_{\rm max}} \right)\end{equation}

```python
def jacobian(rho, u_max, rho_max):
    """
    Computes the Jacobian for our traffic model.

    Parameters
    ----------
    rho : numpy.ndarray
       Traffic density along the road as a 1D array of floats.
    u_max : float
        Maximum speed allowed on the road.
    rho_max : float
        Maximum car density allowed on the road.

    Returns
    -------
    J : numpy.ndarray
        The Jacobian as a 1D array of floats.
    """
    J = u_max * (1.0 - 2.0 * rho / rho_max)
    return J
```

```python
def lax_wendroff(rho0, nt, dt, dx, bc_values, *args):
    """
    Computes the traffic density on the road
    at a certain time given the initial traffic density.
    Integration using Lax-Wendroff scheme.

    Parameters
    ----------
    rho0 : numpy.ndarray
        The initial traffic density along the road
        as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    dt : float
        The time-step size to integrate.
    dx : float
        The distance between two consecutive locations.
    bc_values : 2-tuple of floats
        The value of the density at the first and last locations.
    args : list or tuple
        Positional arguments to be passed to the
        flux and Jacobien functions.

    Returns
    -------
    rho_hist : list of numpy.ndarray objects
        The history of the car density along the road.
    """
    rho_hist = [rho0.copy()]
    rho = rho0.copy()
    for n in range(nt):
        # Compute the flux.
        F = flux(rho, *args)
        # Compute the Jacobian.
        J = jacobian(rho, *args)
        # Advance in time using Lax-Wendroff scheme.
        rho[1:-1] = (rho[1:-1] -
                     dt / (2.0 * dx) * (F[2:] - F[:-2]) +
                     dt**2 / (4.0 * dx**2) *
                     ((J[1:-1] + J[2:]) * (F[2:] - F[1:-1]) -
                      (J[:-2] + J[1:-1]) * (F[1:-1] - F[:-2])))
        # Set the value at the first location.
        rho[0] = bc_values[0]
        # Set the value at the last location.
        rho[-1] = bc_values[1]
        # Record the time-step solution.
        rho_hist.append(rho.copy())
    return rho_hist
```

[Return to Top](#contents)

## 8.3 MacCormack scheme

$$
\begin{equation}
\rho^*_i = \rho^n_i - \frac{\Delta t}{\Delta x} (F^n_{i+1}-F^n_{i}) \ \ \ \ \ \ \text{(predictor)}
\end{equation}
$$

$$
\begin{equation}
\rho^{n+1}_i = \frac{1}{2} (\rho^n_i + \rho^*_i - \frac{\Delta t}{\Delta x} (F^*_i - F^{*}_{i-1})) \ \ \ \ \ \ \text{(corrector)}
\end{equation}
$$

```python
def maccormack(rho0, nt, dt, dx, bc_values, *args):
    """
    Computes the traffic density on the road
    at a certain time given the initial traffic density.
    Integration using MacCormack scheme.

    Parameters
    ----------
    rho0 : numpy.ndarray
        The initial traffic density along the road
        as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    dt : float
        The time-step size to integrate.
    dx : float
        The distance between two consecutive locations.
    bc_values : 2-tuple of floats
        The value of the density at the first and last locations.
    args : list or tuple
        Positional arguments to be passed to the flux function.

    Returns
    -------
    rho_hist : list of numpy.ndarray objects
        The history of the car density along the road.
    """
    rho_hist = [rho0.copy()]
    rho = rho0.copy()
    rho_star = rho.copy()
    for n in range(nt):
        # Compute the flux.
        F = flux(rho, *args)
        # Predictor step of the MacCormack scheme.
        rho_star[1:-1] = (rho[1:-1] -
                          dt / dx * (F[2:] - F[1:-1]))
        # Compute the flux.
        F = flux(rho_star, *args)
        # Corrector step of the MacCormack scheme.
        rho[1:-1] = 0.5 * (rho[1:-1] + rho_star[1:-1] -
                           dt / dx * (F[1:-1] - F[:-2]))
        # Set the value at the first location.
        rho[0] = bc_values[0]
        # Set the value at the last location.
        rho[-1] = bc_values[1]
        # Record the time-step solution.
        rho_hist.append(rho.copy())
    return rho_hist
```

[Return to Top](#contents)

## 8.4 The Richtmyer method

Richtmyer is a *two-step method*, given by:

$$
\begin{align}
\underline{\mathbf{u}}^{n+\frac{1}{2}}_{i+\frac{1}{2}} &= \frac{1}{2} \left( \underline{\mathbf{u}}^n_{i+1} + \underline{\mathbf{u}}^n_i \right) -
\frac{\Delta t}{2 \Delta x} \left( \underline{\mathbf{f}}^n_{i+1} - \underline{\mathbf{f}}^n_i\right) \\
\underline{\mathbf{u}}^{n+1}_i &= \underline{\mathbf{u}}^n_i - \frac{\Delta t}{\Delta x} \left(\underline{\mathbf{f}}^{n+\frac{1}{2}}_{i+\frac{1}{2}} - \underline{\mathbf{f}}^{n+\frac{1}{2}}_{i-\frac{1}{2}} \right)
\end{align}
$$

The flux vectors used in the second step are obtained by evaluating the flux functions on the output of the first step:

$$
\underline{\mathbf{f}}^{n+\frac{1}{2}}_{i+\frac{1}{2}} = \underline{\mathbf{f}}\left(\underline{\mathbf{u}}^{n+\frac{1}{2}}_{i+\frac{1}{2}}\right)
$$

The first step is like a *predictor* of the solution: if you look closely, you'll see that we are applying a Lax-Friedrichs scheme here. The second step is a *corrector* that applies a leapfrog update. Figure 3 gives a sketch of the stencil for Richtmyer method, where the "intermediate time" $n+1/2$ will require a temporary variable in your code, just like we had in the MacCormack scheme.

[Return to Top](#contents)

```python

```


# Neuromatch Academy: Week 1, Day 5, Tutorial 3
# Dimensionality Reduction and reconstruction

__Content creators:__ Alex Cayco Gajic, John Murray

__Content reviewers:__ Roozbeh Farhoudi, Matt Krause, Spiros Chavlis, Richard Gao, Michael Waskom

---
# Tutorial Objectives

In this notebook we'll learn to apply PCA for dimensionality reduction, using a classic dataset that is often used to benchmark machine learning algorithms: MNIST. We'll also learn how to use PCA for reconstruction and denoising.

Overview:
- Perform PCA on MNIST
- Calculate the variance explained
- Reconstruct data with different numbers of PCs
- (Bonus) Examine denoising using PCA

You can learn more about MNIST dataset [here](https://en.wikipedia.org/wiki/MNIST_database).

```python
# @title Video 1: PCA for dimensionality reduction
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="oO0bbInoO_0", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=oO0bbInoO_0

---
# Setup
Run these cells to get the tutorial started.

```python
# Imports
import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import ipywidgets as widgets   # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper Functions

def plot_variance_explained(variance_explained):
  """
  Plots eigenvalues.

  Args:
    variance_explained (numpy array of floats) : Vector of variance explained
                                                 for each PC

  Returns:
    Nothing.

  """

  plt.figure()
  plt.plot(np.arange(1, len(variance_explained) + 1), variance_explained,
           '--k')
  plt.xlabel('Number of components')
  plt.ylabel('Variance explained')
  plt.show()

def plot_MNIST_reconstruction(X, X_reconstructed):
  """
  Plots 9 images in the MNIST dataset side-by-side with the reconstructed
  images.

  Args:
    X (numpy array of floats)               : Data matrix each column
                                              corresponds to a different
                                              random variable
    X_reconstructed (numpy array of floats) : Data matrix each column
                                              corresponds to a different
                                              random variable

  Returns:
    Nothing.
  """

  plt.figure()
  ax = plt.subplot(121)
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(X[k, :], (28, 28)),
                 extent=[(k1 + 1) * 28, k1 * 28, (k2 + 1) * 28, k2 * 28],
                 vmin=0, vmax=255)
  plt.xlim((3 * 28, 0))
  plt.ylim((3 * 28, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.title('Data')
  plt.clim([0, 250])
  ax = plt.subplot(122)
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(np.real(X_reconstructed[k, :]), (28, 28)),
                 extent=[(k1 + 1) * 28, k1 * 28, (k2 + 1) * 28, k2 * 28],
                 vmin=0, vmax=255)
  plt.xlim((3 * 28, 0))
  plt.ylim((3 * 28, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.clim([0, 250])
  plt.title('Reconstructed')
  plt.tight_layout()

def plot_MNIST_sample(X):
  """
  Plots 9 images in the MNIST dataset.

  Args:
     X (numpy array of floats) : Data matrix each column corresponds to a
                                 different random variable

  Returns:
    Nothing.

  """

  fig, ax = plt.subplots()
  k = 0
  for k1 in range(3):
    for k2 in range(3):
      k = k + 1
      plt.imshow(np.reshape(X[k, :], (28, 28)),
                 extent=[(k1 + 1) * 28, k1 * 28, (k2+1) * 28, k2 * 28],
                 vmin=0, vmax=255)
  plt.xlim((3 * 28, 0))
  plt.ylim((3 * 28, 0))
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  plt.clim([0, 250])
  ax.set_xticks([])
  ax.set_yticks([])
  plt.show()

def plot_MNIST_weights(weights):
  """
  Visualize PCA basis vector weights for MNIST. Red = positive weights,
  blue = negative weights, white = zero weight.

  Args:
     weights (numpy array of floats) : PCA basis vector

  Returns:
     Nothing.
  """

  fig, ax = plt.subplots()
  cmap = plt.cm.get_cmap('seismic')
  plt.imshow(np.real(np.reshape(weights, (28, 28))), cmap=cmap)
  plt.tick_params(axis='both', which='both', bottom=False, top=False,
                  labelbottom=False)
  plt.clim(-.15, .15)
  plt.colorbar(ticks=[-.15, -.1, -.05, 0, .05, .1, .15])
  ax.set_xticks([])
  ax.set_yticks([])
  plt.show()

def add_noise(X, frac_noisy_pixels):
  """
  Randomly corrupts a fraction of the pixels by setting them to random values.

  Args:
     X (numpy array of floats)  : Data matrix
     frac_noisy_pixels (scalar) : Fraction of noisy pixels

  Returns:
     (numpy array of floats)    : Data matrix + noise

  """

  X_noisy = np.reshape(X, (X.shape[0] * X.shape[1]))
  N_noise_ixs = int(X_noisy.shape[0] * frac_noisy_pixels)
  noise_ixs = np.random.choice(X_noisy.shape[0], size=N_noise_ixs,
                               replace=False)
  X_noisy[noise_ixs] = np.random.uniform(0, 255, noise_ixs.shape)
  X_noisy = np.reshape(X_noisy, (X.shape[0], X.shape[1]))

  return X_noisy

def change_of_basis(X, W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)   : Data matrix expressed in new basis
  """

  Y = np.matmul(X, W)

  return Y

def get_sample_cov_matrix(X):
  """
  Returns the sample covariance matrix of data X.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Covariance matrix
"""

  X = X - np.mean(X, 0)
  cov_matrix = 1 / X.shape[0] * np.matmul(X.T, X)
  return cov_matrix

def sort_evals_descending(evals, evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats)    :   Vector of eigenvalues
    evectors (numpy array of floats) :   Corresponding matrix of eigenvectors
                                         each column corresponds to a different
                                         eigenvalue

  Returns:
    (numpy array of floats)          : Vector of eigenvalues after sorting
    (numpy array of floats)          : Matrix of eigenvectors after sorting
  """

  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:, index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:, 0],
                           1 / np.sqrt(2) * np.array([1, 1]))) > np.pi / 2:
      evectors[:, 0] = -evectors[:, 0]
    if np.arccos(np.matmul(evectors[:, 1],
                           1 / np.sqrt(2)*np.array([-1, 1]))) > np.pi / 2:
      evectors[:, 1] = -evectors[:, 1]

  return evals, evectors

def pca(X):
  """
  Performs PCA on multivariate data. Eigenvalues are sorted in decreasing order

  Args:
     X (numpy array of floats) :   Data matrix each column corresponds to a
                                   different random variable

  Returns:
    (numpy array of floats)    : Data projected onto the new basis
    (numpy array of floats)    : Vector of eigenvalues
    (numpy array of floats)    : Corresponding matrix of eigenvectors

  """

  X = X - np.mean(X, 0)
  cov_matrix = get_sample_cov_matrix(X)
  evals, evectors = np.linalg.eigh(cov_matrix)
  evals, evectors = sort_evals_descending(evals, evectors)
  score = change_of_basis(X, evectors)

  return score, evectors, evals

def plot_eigenvalues(evals, limit=True):
  """
  Plots eigenvalues.

  Args:
     (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """

  plt.figure()
  plt.plot(np.arange(1, len(evals) + 1), evals, 'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
  if limit:
    plt.show()
```

---
# Section 1: Perform PCA on MNIST

The MNIST dataset consists of a 70,000 images of individual handwritten digits. Each image is a 28x28 pixel grayscale image. For convenience, each 28x28 pixel image is often unravelled into a single 784 (=28*28) element vector, so that the whole dataset is represented as a 70,000 x 784 matrix. Each row represents a different image, and each column represents a different pixel.

Enter the following cell to load the MNIST dataset and plot the first nine images.

```python
from sklearn.datasets import fetch_openml
mnist = fetch_openml(name='mnist_784')
X = mnist.data
plot_MNIST_sample(X)
```

The MNIST dataset has an extrinsic dimensionality of 784, much higher than the 2-dimensional examples used in the previous tutorials! To make sense of this data, we'll use dimensionality reduction. But first, we need to determine the intrinsic dimensionality $K$ of the data. One way to do this is to look for an "elbow" in the scree plot, to determine which eigenvalues are signficant.

## Exercise 1: Scree plot of MNIST

In this exercise you will examine the scree plot in the MNIST dataset.

**Steps:**
- Perform PCA on the dataset and examine the scree plot.
- When do the eigenvalues appear (by eye) to reach zero? (**Hint:** use `plt.xlim` to zoom into a section of the plot).

```python
help(pca)
help(plot_eigenvalues)
```

    Help on function pca in module __main__:

    pca(X)
        Performs PCA on multivariate data. Eigenvalues are sorted in decreasing order

        Args:
           X (numpy array of floats) :   Data matrix each column corresponds to a
                                         different random variable

        Returns:
          (numpy array of floats)    : Data projected onto the new basis
          (numpy array of floats)    : Vector of eigenvalues
          (numpy array of floats)    : Corresponding matrix of eigenvectors

    Help on function plot_eigenvalues in module __main__:

    plot_eigenvalues(evals, limit=True)
        Plots eigenvalues.

        Args:
           (numpy array of floats) : Vector of eigenvalues

        Returns:
          Nothing.

```python
#################################################
## TO DO for students: perform PCA and plot the eigenvalues
#################################################

# perform PCA
# score, evectors, evals = ...
# plot the eigenvalues
# plot_eigenvalues(evals, limit=False)
# plt.xlim(...)  # limit x-axis up to 100 for zooming
```

```python
# to_remove solution
# perform PCA
score, evectors, evals = pca(X)

# plot the eigenvalues
with plt.xkcd():
  plot_eigenvalues(evals, limit=False)
  plt.xlim([0, 100])  # limit x-axis up to 100 for zooming
```

---
# Section 2: Calculate the variance explained

The scree plot suggests that most of the eigenvalues are near zero, with fewer than 100 having large values. Another common way to determine the intrinsic dimensionality is by considering the variance explained. This can be examined with a cumulative plot of the fraction of the total variance explained by the top $K$ components, i.e.,

\begin{equation}
\text{var explained} = \frac{\sum_{i=1}^K \lambda_i}{\sum_{i=1}^N \lambda_i}
\end{equation}

The intrinsic dimensionality is often quantified by the $K$ necessary to explain a large proportion of the total variance of the data (often a defined threshold, e.g., 90%).

## Exercise 2: Plot the explained variance

In this exercise you will plot the explained variance.

**Steps:**
- Fill in the function below to calculate the fraction variance explained as a function of the number of principal componenets. **Hint:** use `np.cumsum`.
- Plot the variance explained using `plot_variance_explained`.

**Questions:**
- How many principal components are required to explain 90% of the variance?
- How does the intrinsic dimensionality of this dataset compare to its extrinsic dimensionality?

```python
help(plot_variance_explained)
```

    Help on function plot_variance_explained in module __main__:

    plot_variance_explained(variance_explained)
        Plots eigenvalues.

        Args:
          variance_explained (numpy array of floats) : Vector of variance explained
                                                       for each PC

        Returns:
          Nothing.

```python
def get_variance_explained(evals):
  """
  Calculates variance explained from the eigenvalues.

  Args:
    evals (numpy array of floats) : Vector of eigenvalues

  Returns:
    (numpy array of floats)       : Vector of variance explained

  """

  #################################################
  ## TO DO for students: calculate the explained variance using the equation
  ## from Section 2.
  # Comment once you've filled in the function
  raise NotImplementedError("Student excercise: calculate explaine variance!")
  #################################################

  # cumulatively sum the eigenvalues
  csum = ...
  # normalize by the sum of eigenvalues
  variance_explained = ...

  return variance_explained

#################################################
## TO DO for students: call the function and plot the variance explained
#################################################

# calculate the variance explained
variance_explained = ...

# Uncomment to plot the variance explained
# plot_variance_explained(variance_explained)
```

```python
# to_remove solution
def get_variance_explained(evals):
  """
  Plots eigenvalues.

  Args:
    (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """

  # cumulatively sum the eigenvalues
  csum = np.cumsum(evals)
  # normalize by the sum of eigenvalues
  variance_explained = csum / np.sum(evals)

  return variance_explained

# calculate the variance explained
variance_explained = get_variance_explained(evals)
with plt.xkcd():
  plot_variance_explained(variance_explained)
```

---
# Section 3: Reconstruct data with different numbers of PCs

```python
# @title Video 2: Data Reconstruction
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="ZCUhW26AdBQ", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=ZCUhW26AdBQ

Now we have seen that the top 100 or so principal components of the data can explain most of the variance. We can use this fact to perform *dimensionality reduction*, i.e., by storing the data using only 100 components rather than the samples of all 784 pixels. Remarkably, we will be able to reconstruct much of the structure of the data using only the top 100 components. To see this, recall that to perform PCA we projected the data $\bf X$ onto the eigenvectors of the covariance matrix:
\begin{equation}
\bf S = X W
\end{equation}
Since $\bf W$ is an orthogonal matrix, ${\bf W}^{-1} = {\bf W}^T$. So by multiplying by ${\bf W}^T$ on each side we can rewrite this equation as
\begin{equation}
{\bf X = S W}^T.
\end{equation}
This now gives us a way to reconstruct the data matrix from the scores and loadings. To reconstruct the data from a low-dimensional approximation, we just have to truncate these matrices.  Let's call ${\bf S}_{1:K}$ and ${\bf W}_{1:K}$ as keeping only the first $K$ columns of this matrix. Then our reconstruction is:
\begin{equation}
{\bf \hat X = S}_{1:K} ({\bf W}_{1:K})^T.
\end{equation}

## Exercise 3: Data reconstruction

Fill in the function below to reconstruct the data using different numbers of principal components.

**Steps:**

* Fill in the following function to reconstruct the data based on the weights and scores. Don't forget to add the mean!
* Make sure your function works by reconstructing the data with all $K=784$ components. The two images should look identical.

```python
help(plot_MNIST_reconstruction)
```

    Help on function plot_MNIST_reconstruction in module __main__:

    plot_MNIST_reconstruction(X, X_reconstructed)
        Plots 9 images in the MNIST dataset side-by-side with the reconstructed
        images.

        Args:
          X (numpy array of floats)               : Data matrix each column
                                                    corresponds to a different
                                                    random variable
          X_reconstructed (numpy array of floats) : Data matrix each column
                                                    corresponds to a different
                                                    random variable

        Returns:
          Nothing.

```python
def reconstruct_data(score, evectors, X_mean, K):
  """
  Reconstruct the data based on the top K components.

  Args:
    score (numpy array of floats)    : Score matrix
    evectors (numpy array of floats) : Matrix of eigenvectors
    X_mean (numpy array of floats)   : Vector corresponding to data mean
    K (scalar)                       : Number of components to include

  Returns:
    (numpy array of floats)          : Matrix of reconstructed data

  """

  #################################################
  ## TO DO for students: Reconstruct the original data in X_reconstructed
  # Comment once you've filled in the function
  raise NotImplementedError("Student excercise: reconstructing data function!")
  #################################################

  # Reconstruct the data from the score and eigenvectors
  # Don't forget to add the mean!!
  X_reconstructed =  ...

  return X_reconstructed

K = 784

#################################################
## TO DO for students: Calculate the mean and call the function, then plot
## the original and the recostructed data
#################################################

# Reconstruct the data based on all components
X_mean = ...
X_reconstructed = ...

# Plot the data and reconstruction
# plot_MNIST_reconstruction(X, X_reconstructed)
```

```python
# to_remove solution
def reconstruct_data(score, evectors, X_mean, K):
  """
  Reconstruct the data based on the top K components.

  Args:
    score (numpy array of floats)    : Score matrix
    evectors (numpy array of floats) : Matrix of eigenvectors
    X_mean (numpy array of floats)   : Vector corresponding to data mean
    K (scalar)                       : Number of components to include

  Returns:
    (numpy array of floats)          : Matrix of reconstructed data

  """

  # Reconstruct the data from the score and eigenvectors
  # Don't forget to add the mean!!
  X_reconstructed =  np.matmul(score[:, :K], evectors[:, :K].T) + X_mean

  return X_reconstructed

K = 784

# Reconstruct the data based on all components
X_mean = np.mean(X, 0)
X_reconstructed = reconstruct_data(score, evectors, X_mean, K)

# Plot the data and reconstruction
with plt.xkcd():
  plot_MNIST_reconstruction(X, X_reconstructed)
```

## Interactive Demo: Reconstruct the data matrix using different numbers of PCs

Now run the code below and experiment with the slider to reconstruct the data matrix using different numbers of principal components.

**Steps**
* How many principal components are necessary to reconstruct the numbers (by eye)? How does this relate to the intrinsic dimensionality of the data?
* Do you see any information in the data with only a single principal component?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def refresh(K=100):
  X_reconstructed = reconstruct_data(score, evectors, X_mean, K)
  plot_MNIST_reconstruction(X, X_reconstructed)
  plt.title('Reconstructed, K={}'.format(K))

_ = widgets.interact(refresh, K=(1, 784, 10))
```

## Exercise 4: Visualization of the weights

Next, let's take a closer look at the first principal component by visualizing its corresponding weights.

**Steps:**

* Enter `plot_MNIST_weights` to visualize the weights of the first basis vector.
* What structure do you see? Which pixels have a strong positive weighting? Which have a strong negative weighting? What kinds of images would this basis vector differentiate?
* Try visualizing the second and third basis vectors. Do you see any structure? What about the 100th basis vector? 500th? 700th?

```python
help(plot_MNIST_weights)
```

    Help on function plot_MNIST_weights in module __main__:

    plot_MNIST_weights(weights)
        Visualize PCA basis vector weights for MNIST. Red = positive weights,
        blue = negative weights, white = zero weight.

        Args:
           weights (numpy array of floats) : PCA basis vector

        Returns:
           Nothing.

```python
#################################################
## TO DO for students: plot the weights calling the plot_MNIST_weights function
#################################################

# Plot the weights of the first principal component
# plot_MNIST_weights(...)
```

```python
# to_remove solution
# Plot the weights of the first principal component
with plt.xkcd():
  plot_MNIST_weights(evectors[:, 0])
```

---
# Summary
* In this tutorial, we learned how to use PCA for dimensionality reduction by selecting the top principal components. This can be useful as the intrinsic dimensionality ($K$) is often less than the extrinsic dimensionality ($N$) in neural data. $K$ can be inferred by choosing the number of eigenvalues necessary to capture some fraction of the variance.
* We also learned how to reconstruct an approximation of the original data using the top $K$ principal components. In fact, an alternate formulation of PCA is to find the $K$ dimensional space that minimizes the reconstruction error.
* Noise tends to inflate the apparent intrinsic dimensionality, however the higher components reflect noise rather than new structure in the data. PCA can be used for denoising data by removing noisy higher components.
* In MNIST, the weights corresponding to the first principal component appear to discriminate between a 0 and 1. We will discuss the implications of this for data visualization in the following tutorial.

---
# Bonus: Examine denoising using PCA

In this lecture, we saw that PCA finds an optimal low-dimensional basis to minimize the reconstruction error. Because of this property, PCA can be useful for denoising corrupted samples of the data.

## Exercise 5: Add noise to the data
In this exercise you will add salt-and-pepper noise to the original data and see how that affects the eigenvalues.

**Steps:**
- Use the function `add_noise` to add noise to 20% of the pixels.
- Then, perform PCA and plot the variance explained. How many principal components are required to explain 90% of the variance? How does this compare to the original data?

```python
help(add_noise)
```

    Help on function add_noise in module __main__:

    add_noise(X, frac_noisy_pixels)
        Randomly corrupts a fraction of the pixels by setting them to random values.

        Args:
           X (numpy array of floats)  : Data matrix
           frac_noisy_pixels (scalar) : Fraction of noisy pixels

        Returns:
           (numpy array of floats)    : Data matrix + noise

```python
###################################################################
# Insert your code here to:
# Add noise to the data
# Plot noise-corrupted data
# Perform PCA on the noisy data
# Calculate and plot the variance explained
###################################################################
np.random.seed(2020)  # set random seed
X_noisy = ...
# score_noisy, evectors_noisy, evals_noisy = ...
# variance_explained_noisy = ...
# plot_MNIST_sample(X_noisy)
# plot_variance_explained(variance_explained_noisy)
```

```python
# to_remove solution

np.random.seed(2020)  # set random seed
X_noisy = add_noise(X, .2)
score_noisy, evectors_noisy, evals_noisy = pca(X_noisy)
variance_explained_noisy = get_variance_explained(evals_noisy)

with plt.xkcd():
  plot_MNIST_sample(X_noisy)
  plot_variance_explained(variance_explained_noisy)
```

## Exercise 6: Denoising

Next, use PCA to perform denoising by projecting the noise-corrupted data onto the basis vectors found from the original dataset. By taking the top K components of this projection, we can reduce noise in dimensions orthogonal to the K-dimensional latent space.

**Steps:**
- Subtract the mean of the noise-corrupted data.
- Project the data onto the basis found with the original dataset (`evectors`, not `evectors_noisy`) and take the top $K$ components.
- Reconstruct the data as normal, using the top 50 components.
- Play around with the amount of noise and K to build intuition.

```python
###################################################################
# Insert your code here to:
# Subtract the mean of the noise-corrupted data
# Project onto the original basis vectors evectors
# Reconstruct the data using the top 50 components
# Plot the result
###################################################################

X_noisy_mean = ...
projX_noisy = ...
X_reconstructed = ...
# plot_MNIST_reconstruction(X_noisy, X_reconstructed)
```

```python
# to_remove solution

X_noisy_mean = np.mean(X_noisy, 0)
projX_noisy = np.matmul(X_noisy - X_noisy_mean, evectors)
X_reconstructed = reconstruct_data(projX_noisy, evectors, X_noisy_mean, 50)

with plt.xkcd():
  plot_MNIST_reconstruction(X_noisy, X_reconstructed)
```