# Lecture 7: Bias-Variance Tradeoff, Regularization
## 11/14/2020

### Table Of Contents
* [Recap](#recap)
* [Bias-Variance Tradeoff](#bv-tradeoff)
    * [Bias](#bias)
    * [Variance](#variance)
    * [The Tradeoff](#the-tradeoff)
    * [Polynomial Regression](#polynomial-regression)
* [Regularization](#regularization)
    * [Ridge](#ridge)
    * [LASSO](#lasso)
    * [Visualizing Ridge and Lasso](#visualizing-ridge-and-lasso)
    * [Regularization and Bias Variance](#regularization-and-bias-variance)
    * [Lambda](#lambda)
    * [Validation on Lambda](#validation-on-lambda)
* [Exercises](#exercises)

> Bias/Variance is one way to understand Overfitting and Underfitting

If we look at the first model, the points seem to follow some sort of curve, but our predictor is linear (not very complicated) and *unable to fit all the points* no matter how we wiggle it. We have chosen a model which is not complex enough to accurately capture all the information from our data set.

In the last model, the predictor is super complex and adjusts based on every point. In this case, the model changes too much based on the data and isn't considering the 'big picture'. This model is fitting too much to the minor jitters.

**High bias/Low Variance ~ underfitting**

**Low bias/High Variance ~ overfitting**

**Goldilocks - porridge**

Summary: We try to fiddle with bias/variance to get better models. Variance is important! because otherwise we only are building a model for our train not our normal dataset.

 Bias-Variance error decomposition only works for convex loss functions (especially L2): https://www.textbook.ds100.org/ch/11/gradient_convexity.html

# Thinking Experiment

Lets say we were building a model of a dog. On the left we would have a model that would classify everything as a dog (including a hippo) versus on the far right

    Underfitting
    Just Right
    Overfitting

Related kaggle project on dog breed classification.

# Why do we care?

Bias-Variance is *directly related* to what a model predicts and algorithmic bias. (Articles [here](https://www.zdnet.com/article/what-is-bias-in-ai-really-and-why-cant-ai-neutralize-it/), [here](https://academicworks.cuny.edu/cgi/viewcontent.cgi?article=1042&context=jj_etds) and [here](https://www.cio.com/article/3385123/bias-in-ai-creating-artificial-intelligence-that-is-less-like-humans.html))

Bias-Variance is the main idea behind the other methods we will cover here. i.e. **"What are some ways to try and make appropriate bias/variance tradeoffs"**

--

Frequently people feel they should minimize bias even at the expense of variance. They think that presence of bias indicates fundamental issues with their model and algorithm. Yes, variance is also bad but a model with high variance could at least predict well on average, at least it is not fundamentally wrong.

This is mistaken. A high variance and low bias model can preform well in some sort of long-run average sense. However, in practice modelers are always dealing with a concrete actual data set in the context of some business problem. In these cases, long run averages are irrelevant, what is important is the performance of the model on the data you actually have and in this case bias and variance are equally important and one should not be improved at an excessive expense to the other. [Source](http://scott.fortmann-roe.com/docs/BiasVariance.html)

# Bias-Variance Tradeoff

When making this tradeoff we do it by proxy -- using various metrics:

Today we'll perform **model evaluation**, where we'll judge how our linear regression models actually perform. Last week, we talked about **loss functions**, a function that describes a numerical value for how far your model is from the true values.

Today we will use:
$$\text{Mean Squared Error: } \frac{1}{n}\sum_{i=1}^n \left(y_i - \hat{f}(x_i)\right)^2$$

In this loss function, $y_i$ is a scalar, and $x_i$ is a $p$ dimensional vector, because there are $p$ features. This loss is called **mean squared error**, or **MSE**.

(i.e. if we have x=1, y=2, but our model says $\hat{f}(x_{1})=3$. Then the MSE is $(2-3)^2$.

## Model Evaluation

## Bias

When evaluating a model, the most intuitive first step is to look at how well the model performs. For classification, this may be the percentage of data points correctly classified, or for regression it may be how close the predicted values are to actual. The **bias** of a model is *a measure of how close our prediction is to the actual value on average from an average model*.

Note that bias is **not** a measure of a single model, it encapsulates the scenario in which we collect many datasets, create models for each dataset, and average the error over all of models. Bias is not a measure of error for a single model, but a more abstract concept describing the average error over all errors. A low value for the bias of a model describes that on average, our predictions are similar to the actual values.

 Super Mathy: Squared difference between best possible prediction for x, f(x), and our “long-term” expectation for what the learner will do if we averaged over many datasets D: $E_D[h_D(x)]$

## Variance
The **variance** of  a model relates to the variance of the distribution of all models. In the previous section about bias, we envisoned the scenario of collecting many datasets, creating models for each dataset, and averaging the error overall the datasets. Instead, the variance of a model describes the variance in prediction. While we might be able to predict a value very well on average, if the variance of predictions is very high this may not be very helpful, as when we train a model we only have one such instance, and a high model variance tells us little about the true nature of the predictions. A low variance describes that our model will not predict very different values for different datasets.

 Super Mathy: Squared difference btwn our longterm expectation for the learners performance, $E_D[h_D(x)]$, and what we expect in a representative run on a dataset D ($\hat{y}$)

**We can take a look at how bias and variance differ and can be explained in a dataset with the following diagram:**

source:http://scott.fortmann-roe.com/docs/BiasVariance.html

The image describes what bias and variance are in a more simplified example. Consider that we would like to create a model that selects a point close to the center. The models on the top row have low bias, meaning the center of the cluster is close to the red dot on the target. The models on the left column have low variance, the clusters are quite tight, meaning our predictions are close together.

**Question: What do the blue dots represent? What about the bullseye?**

**Question: Which is the of best scenario? Which is the worst?**

## The Tradeoff

We are trying to minimize **expected error**, or the average **MSE** over all datasets. It turns out (with some advanced probability gymnastics), that:

$$\text{Mean Squared Error} = \text{Noise Variance} + \text{Bias}^2 + \text{Variance}$$

Note that $\text{Noise Variance}$ is constant: we assume there is some noise, and $\text{moise variance}$ is simply a value that describes how noisy your dataset will be on average. This is often also called "irreducible noise", as it is literally irreducible, we cannot avoid this.

Furthermore, notice that the equation above is the sum of (squared) bias and variance. Thus there is a literal tradeoff between these two, since decreasing one increases the other. This defines what is known as the **bias variance tradeoff**.

source:http://scott.fortmann-roe.com/docs/BiasVariance.html

**Why does this happen?**

At some point, as we decrease **bias**, instead of getting closer to the **true model** $h$, we go past and try to fit to the $\epsilon$ (noise) that is part of our current dataset. This is equivalent to making our model more noisy, or **overfit** on our dataset, which means that over all datasets, it has more **variance**.

**Questions for understanding**:
> 1. Where does underfitting and overfitting lie in the graph above? How do they relate to bias and variance?
> 2. Why can't we usually just make a bunch of models with low bias and high variance and average them?
> 3. Why is low variance important in models?

## Polynomial Regression

Let's revisit the polynomial problem that we have discussed.

In this case, if our model has degree $d$, we have $d + 1$ features: $x = [x^0, x^1, ..., x^d]$. Now, we have a linear model with $d + 1$ features:

$$\hat{f}(x) = \sum_{i=0}^{d} a_i x^i$$

Model complexity in this case is the degree of the polynomial. As we saw last week, as $d$ increases, model complexity increases. The model gets better, but then gets erratic. This directly corresponds to the bias-variance graph above.

```python
import matplotlib.pyplot as plt
import random
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from plotting import overfittingDemo, plot_multiple_linear_regression, ridgeRegularizationDemo, lassoRegularizationDemo
from scipy.optimize import curve_fit
from sklearn.metrics import mean_squared_error
import warnings
warnings.filterwarnings('ignore')

%matplotlib inline
```

```python
overfittingDemo()
```

As we saw from last time, the best model was a degree 3 model.

# Regularization

We talked about **validation** as a means of combating overfitting. However, this is not the only method to combat overfitting. Another method to do so is to add *regularization* terms to our loss function. **Regularization** basically penalizes complexity in our models. This allows us to add explanatory variables to our model without worrying as much about overfitting. Here's what our ordinary least squares model looks like with a regularization term:

$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - f_\boldsymbol{\theta}(x_i))^2 + \lambda R(\boldsymbol{\theta})$$

We've written the model a little differently here, but the first term is the same as the ordinary least squares regression model you learned last week. This time it's just generalized to any function of $x$ where $\theta$ is a list of parameters, or weights on our explanatory variables, such as coefficients to a polynomial. We're minimizing a loss function to find the best coefficients for our model.

The second term is the **regularization** term. The $\lambda$ parameter in front of it dictates how much we care about our regularization term – the higher $\lambda$ is, the more we penalize large weights, and the more the regularization makes our weights deviate from OLS.

**Question**: What happens when $\lambda = 0$?

So, what is $R(\theta)$ in the equation? There are a variety of different regularization functions that could take its place, but today, we'll just talk about the two most common types of functions: **ridge regression** and **LASSO regression**.

$$\begin{align}\text{ Ridge (L2 Norm)}: &\  R(\boldsymbol{\theta}) = \|\theta\|_2^2 = \sum_{i=1}^p \theta_i^2\\
\text{ LASSO (L1 Norm)}: &\ R(\boldsymbol{\theta}) = \|\theta\|_1=\sum_{i=1}^p \lvert \theta_i\rvert\end{align}$$

## Ridge
$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - \hat{f_\theta}(x_i))^2 + \lambda \|\theta\|_2^2$$
In **ridge** regression, the regularization function is the sum of squared weights. One nice thing about ridge regression is that there is always a unique, mathematical solution to minimizing the loss of that term. The solution involves some linear algebra, which we won't get into in this notebook, but the existence of this formula makes this minimization computationally easy to solve!

$$\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T \boldsymbol{X} + \lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{Y}$$

If you recall, the solution to linear regression was of the form:

$$\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T \boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{Y}$$

And we said that the $\boldsymbol{X}^T\boldsymbol{X}$ isn't always invertible. **What about $\boldsymbol{X}^T \boldsymbol{X} + \lambda\boldsymbol{I}$?**

Turns out, this is always invertible! If you are familiar with linear algebra, this is equivalent to adding $\lambda$ to all the eigenvalues of $X^TX$.

---

L2 regularization. Regularization trades a bit of model accuracy for improved generalization and works by constraining the size of model parameters to "reasonable" values. L2 regularization constrains coefficients to a spherical hyper volume by adding an L2 norm penalty term to the linear model loss function. The term Ridge from the original Hoerl and Kennard Ridge paper was taken from the "ridge traces" on their coefficient plots. Unfortunately, the ridge constraint region looks like a lasso and the lasso constraint region looks like a ridge. [Source](https://explained.ai/statspeak/index.html)

**To see this in practice**, we'll first create a regular linear regression model, and compare how it does against models using regularization on the `mpg` dataset we used from last week! We'll be constructing models of `displacement` vs. `mpg`, and seeing the difference from there!

First, let's construct the `mpg_train` dataset!

```python
mpg = pd.read_csv("mpg.csv", index_col='name')# load mpg dataset
mpg = mpg.loc[mpg["horsepower"] != '?'].astype(float) # remove columns with missing horsepower values
mpg_train, mpg_test = train_test_split(mpg, test_size = .2, random_state = 0) # split into training set and test set
mpg_train, mpg_validation = train_test_split(mpg_train, test_size = .5, random_state = 0)
mpg_train.head()
```

      mpg
      cylinders
      displacement
      horsepower
      weight
      acceleration
      model_year
      origin

      name

      toyota corolla
      34.0
      4.0
      108.0
      70.0
      2245.0
      16.9
      82.0
      3.0

      buick century
      17.0
      6.0
      231.0
      110.0
      3907.0
      21.0
      75.0
      1.0

      cadillac eldorado
      23.0
      8.0
      350.0
      125.0
      3900.0
      17.4
      79.0
      1.0

      bmw 320i
      21.5
      4.0
      121.0
      110.0
      2600.0
      12.8
      77.0
      2.0

      ford fairmont futura
      24.0
      4.0
      140.0
      92.0
      2865.0
      16.4
      82.0
      1.0

**Exercise:** Now, let's create a regular linear regression model using the same process we've learned before (fitting, predicting, finding the loss)!

```python
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

x_train = np.vander(mpg_train["displacement"], 13)
y_train = mpg_train[["mpg"]]

x_validation = np.vander(mpg_validation["displacement"], 13)
y_validation = mpg_validation[["mpg"]]

# instantiate your model
linear_model = LinearRegression()

# fit the model
linear_model.fit(x_train, y_train)
# make predictions on validation set
linear_prediction = linear_model.predict(x_validation)
# find mean squared error
linear_loss = mean_squared_error(y_validation, linear_prediction)
lin_r2 = r2_score(y_validation, linear_prediction)

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("R2 of Linear Model: {:.2f}".format(lin_r2))
```

    Root Mean Squared Error of Linear Model: 25.09
    R2 of Linear Model: 0.60

**Exercise:** Using what you did above as reference, do the same using a Ridge regression model!

```python
from sklearn.linear_model import Ridge

linear_model = Ridge()
linear_model.fit(x_train, y_train)
linear_prediction = linear_model.predict(x_validation)
ridge_loss = mean_squared_error(y_validation, linear_prediction)
ridge_r2 = r2_score(y_validation, linear_prediction)

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("Root Mean Squared Error of Ridge Model: {:.2f}".format(ridge_loss))
print("R2 of Linear Model: {:.2f}".format(lin_r2))
print("R2 of Ridge Model: {:.2f}".format(ridge_r2))
```

    Root Mean Squared Error of Linear Model: 25.09
    Root Mean Squared Error of Ridge Model: 20.11
    R2 of Linear Model: 0.60
    R2 of Ridge Model: 0.68

A final note on ridge is that ridge regression helps with *robustness*. (Don't worry if you don't understand this part, it's not necessary to understand the rest of the lecture.

Consider the above diagram.

It turns out we can write the weights of least squares regression as a linear combination of the inverses of the singular values. When a matrix is non-invertible, these singular values are zero. However, when a matrix is almost non-invertible, these singular values are very small, leading their inverses to be very large. This leas them to dominate the least squares regression.
Ridge regression fixes this problem by downweighting very small singular values into something more reasonable. In the least squares case, tiny changes to the input data could have large effects on the model if it's along small singular values. Ridge regression is more *robust* to these changes.

## LASSO
$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - f_\boldsymbol{\theta}(x_i))^2 + \lambda \|\theta\|_1$$
In **LASSO** regression, the regularization function is **the sum of absolute values of the weights**. One key thing to note about **LASSO** is that it is **sparsity inducing**, meaning it forces weights to be zero values rather than really small values (which can happen in **Ridge Regression**), leaving you with fewer explanatory variables in the resulting model! Unlike Ridge Regression, LASSO doesn't necessarily have a unique solution that can be solved for with linear algebra, so there's no formula that determines what the optimal weights should be.

---

L1 regularization. Regularization trades a bit of model accuracy for improved generalization and works by constraining the size of model parameters to "reasonable" values. L1 regularization constrains coefficients to a diamond shaped hyper volume by adding an L1 norm penalty term to the linear model loss function. The term LASSO means "Least absolute Shrinkage and Selection Operator" from the original Tibshirani paper. The LASSO name is perfectly fine except for the fact that the constraint region has lots of pointy discontinuities and looks like it should be called "ridge regularization" from the shape.
[Source](https://explained.ai/statspeak/index.html)

```python
from sklearn.linear_model import Lasso

linear_model = Lasso()
linear_model.fit(x_train, y_train)
y_val_pred = linear_model.predict(x_validation)

lasso_loss = mean_squared_error(y_validation, linear_prediction)
lasso_r2 = r2_score(y_validation, y_val_pred)

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("Root Mean Squared Error of LASSO Model: {:.2f}".format(lasso_loss))
print("Root Mean Squared Error of Ridge Model: {:.2f}".format(ridge_loss))
print("R2 of Linear Model: {:.2f}".format(lin_r2))
print("R2 of Ridge Model: {:.2f}".format(ridge_r2))
print("R2 of Lasso Model: {:.2f}".format(lasso_r2))
```

    Root Mean Squared Error of Linear Model: 25.09
    Root Mean Squared Error of LASSO Model: 21.44
    Root Mean Squared Error of Ridge Model: 20.11
    R2 of Linear Model: 0.60
    R2 of Ridge Model: 0.68
    R2 of Lasso Model: 0.66

As we can see, both **Ridge Regression and LASSO Regression** minimized the loss of our linear regression models, so maybe penalizing some features allowed us to prevent overfitting on our dataset!

#####
### Derivatives and Sparsity

Let's take at another visualization that may help build some intuition behind how both of these regularization methods work!

The above plot is of the ridge regression penalty in blue and the LASSO in red. Note that when the values are small, the LASSO penalty is actually higher than the ridge penalty.

**Question**: Based on these visualizations, could you explain why LASSO is sparsity-inducing?

For a more mathematical motivation, consider the derivative of the LASSO and ridge penalties.

$$\frac{\partial}{\partial w} w^2 = 2w$$
$$\frac{\partial}{\partial w} |w| = sign(w)$$

Recall that gradient descent algorithms go along the derivative of the loss. When we train a neural network to minimize these penalties, a network trained on LASSO will reduce the penalty to 0 (when possible) as the derivative is always 1. However, with the ridge penalty, the loss will essentailly "flatten out" when w approaches zero, and the network will pick a value for w that is close to zero, but not quite zero.

Results from an emperical statement.

## Regularization and Bias Variance
As we mentioned earlier, **bias** is the average linear squares loss term across multiple models of the same family (e.g. same degree polynomial) trained on separate datasets. **Variance** is the average variance of the weight vectors (coefficients) on your features.

Without the regularization term, we’re just minimizing bias; the regularization term means we won’t get the lowest possible bias, but we’re exchanging that for some lower variance so that our model does better at generalizing to data points outside of our training data.

## Lambda

We said that $\lambda$ is how much we care about the regularization term, but what does that look like? Let's return to the polynomial example from last week, and see what the resulting models look like with different values of $\lambda$ given a degree 8 polynomial.

```python
ridgeRegularizationDemo([0, 0.5, 1.0, 5.0, 10.0], 8)
```

From the diagram above, it's difficult to determine which lambda value help fit our model the closest to the true data distribution. So, **how do we know what to use for $\lambda$ (or `alpha` in the `sklearn.linear_model` constructors)?**

That's right, let's use the process of **validation** here! In this case, we'd be finding the value for lambda that **minimizes the loss for ridge regression, and then the one that minimizes the loss for LASSO regression**!

## Validation on Lambda
Let's try to find the best $\lambda$ for the degree 20 polynomial on `displacement` from above.

```python
lambdas = np.arange(0, 200) # create a list of potential lambda values

# create a list containing the corresponding mean_squared_error for each lambda usinb both ridge and lasso regression
ridge_errors = []
lasso_errors = []

for l in lambdas:

    ridge_errors.append(...)

    lasso_errors.append(...)

# finds the index of the minimum value in each list
answer = ridge_errors.index(min(ridge_errors)), lasso_errors.index(min(lasso_errors))
answer
```

As we can see from above, we've been able to determine which lambdas minimizes our ridge regression model and our LASSO regression model through validation by iterating through potential lambda values and finding the ones that minimize our loss for each model!

# Validation

When we train machine learning models, we use three types of data, used for different purposes.

### Training data
The data we use to train the model (typically around 80% of the data)

### Validation data
The data we use to fine tune the model and adjust hyperparameters (typically around 15% of the data)

### Test data
The data we use to evaluate the model's performance (only used once, typically around 5% of the data)

*tl;dr you train your dog classifier on 3 of your dogs and then test it out on your friend's 4 dogs. Don't ask why we have so many dogs*

Cross-validation is a technique for evaluating ML models by training ML models on different subsets of the available input data and evaluating them on the complementary subset of the data. Use cross-validation to detect overfitting, ie, failing to generalize a pattern.  [Source](https://docs.aws.amazon.com/machine-learning/latest/dg/cross-validation.html)

---

Cross-validation is primarily a way of measuring the predictive performance of a statistical model. Every statistician knows that the model fit statistics are not a good guide to how well a model will predict: high
$R^2$ does not necessarily mean a good model. It is easy to over-fit the data by including too many degrees of freedom and so inflate $R^2$ and other fit statistics. For example, in a simple polynomial regression I can just keep adding higher order terms and so get better and better fits to the data. But the predictions from the model on new data will usually get worse as higher order terms are added. [Source](https://robjhyndman.com/hyndsight/crossvalidation/)

When we run cross validation we tend to see stagnant test error but consistently better train error, telling us that our model is really good at fitting our training dataset but not good at modeling the overall dataset.

**Question**
What could you say is happening in each of these scenarios? {overfitting, underfitting, good model}
#### high training error, high validation error
#### low training error, high validation error
#### low training error, low validation error

# Conclusion

Through the course of the notebook, we introduced two main concepts, **bias** and **variance**, and how the two relate with one another when it comes to finding the best model for our dataset! We also went into different methods we can use to minimize overfitting our model, and in turn lower variance, by taking look at a process called **regularization**. We saw that the two main regression models, **ridge regression** and **LASSO regression**, and saw the difference between the two (ridge -> penalize large weights, LASSO -> make weights sparse). We also took a look at different visualizations between the two to build up some more intuition behind how they work, through **graphs** and **derivatives**. Finally, we went through a familiar process (**validation**) to determine what the best values of lambda were for our models, officially ending our journey of **bias** and **variance**, and how we can minimze both in our models!

# Congratulations! You are now a Bias + Variance Ninja!

## Exercises

1. What happens as $\lambda$ increases?
    1. bias increases, variance increases
    2. bias increases, variance decreases
    3. bias decreases, variance increases
    4. bias decreases, variance decreases

**Insert answer here**:

2. **True** or **False**? Bias is how much error your model makes.

**Insert answer here:**

3. What is **sparsity**?

**Insert answer here:**

4. For each of the following, choose **ridge**, **lasso**, **both**, or **neither**:
    1. L1-norm
    2. L2-norm
    3. Induces sparsity
    4. Has analytic (mathematical) solution
    5. Increases bias
    6. Increases variance

**Insert answer here:**

5. Which one is better to use: Ridge Regression or LASSO Regression?

**Insert answer here:**

### Congrats! You've finished our few conceptual questions and learned about bias, variance, and regularization!


# Chapter 2 Gotchas

Chapter1で簡単に述べたように、`SymPy`は`Python`のライブラリとして提供されている。そのため**`Python`の構文規則は全く同様に使える**。

例えば、四則演算`+`, `-`, `*`, `/`, `//`, `%` は`SymPy`でも同様に使える。

**コメント**:
チュートリアル原文では、SymPyで扱う変数を*SymPyシンボル*と呼んでいます。Pythonではシンボルに対応する概念がないのですが、チュートリアルに従っています。*SymPyオブジェクト*と呼んだときは、*SymPy*で定義されている関数・メソッドや、*SymPy*シンボル等を含めて総称として呼ぶことにします。

### 2.1 Symbols

```python
from sympy import * #関数やクラスをすべてインポートする.
init_printing() #表示をキレイにするため、
```

```python
x + 1
```

---> Python(オブジェクト)では「変数」は意味を持たない. 値を代入して初めて意味を持つ. その意味で
```
NameError: name 'x' is not defined
```
が出ている。

**注意** `sympy`で変数として扱いたい文字は、`SymPy`オブジェクトに変換するために`symbols`で定義しないといけない:

```python
x = symbols('x') #引数は文字列(str)
```

```python
type(x)
```

    sympy.core.symbol.Symbol

*コメント*: 一つの変数を定義するときも、`symbols()`が使える。1変数用に`Symbols`が定義されているが、必要ないかも?

##### 複数の変数を定義する

```python
x, y, z = symbols('x, y, z') #引数はスペースだけでも良い.
```

このようにして定義された変数は、(Pythonの変数とは違い)変数それ自体として意味を持つ. すなわち

```python
a, b = symbols('a, b')
```

```python
a
```

```python
type(a) #aはSymPyオブジェクト
```

    sympy.core.symbol.Symbol

```python
b
```

となって、出力結果が文字`a`, `b`で表現されている。

ところで上の定義の`a`, `b`を入れ替えて

```python
a, b = symbols('b a') #左辺: Sympyシンボル 右辺:pythonオブジェクト
```

とすると、

```python
a
```

```python
b
```

と出力される。

これは何を意味するか? `a`と名付けられたPython変数(左辺)は、`Sympy`シンボルの`b`(右辺)に対応している(残りも同様). これは非常に混乱しやすいので、通常は名前をそれぞれ一致させるようにする.

**例外**:

- Sympyのシンボル名がPythonの予約語を含んでいるとき
- シンボル名(=Sympyの変数)がとても長く、短いPython変数で表したいとき

##### Python変数とSympyシンボルの違い

次に代入操作の違いについて考える.

```python
x = symbols('x')
```

```python
expr = x + 1 #xはSympyシンボル
```

このSymPyシンボル`x`に対して、`Python`でよく知られた以下の代入操作を行うとどうなるだろうか？

```python
x = 2
```

```python
print(expr)
```

    x + 1

`expr = 3`が出力されない！しかしながら`x`は値`2`が代入されている.

```python
x
```

*コメント* `expr`は`SymPy`シンボル`x`で定義されているが、代入操作`x = 2`はそれの定義**後**に行われている. そのため`expr`は変更を受けない。

もしこの代入操作を反映した`expr`がほしいなら、以下のように**`expr`を定義する前に**代入操作を行う必要がある.

```python
x = symbols('x')
```

```python
x = 2
```

```python
expr = x + 1
```

```python
expr
```

```python
x
```

以上の操作は文字列に対しても同様に考えることができる.

**例**

```python
x = 'abc'
```

```python
expr = x + 'def'
```

```python
expr
```

    'abcdef'

```python
x = 'ABC' #exprの定義後に代入
```

```python
expr #'ABCdef'にならない.
```

    'abcdef'

**Q.** では、プログラムの中で`x`が変更を受けた後、再度exprの値を評価するにはどうすればよいか？

```python
x = symbols('x')
```

```python
expr = x + 1
```

```python
expr.subs({x:2}) #代入操作(多変数への代入があるとき)
```

```python
expr.subs(x,2) #代入操作(一変数への代入のとき)
```

### 2.4 等号

等号`=`, 真偽判定`==` もPythonと同様にして使えるが、数式評価には注意が必要

```python
x + 1 == 4
```

    False

--->日常でよく用いられる数式としてのイコール(`=`)とは違う!? (`Python`では論理判定として使われている)

つまり、*通常私たちはa=bか判定するのは、Pythonで`a == b`を行っているのと同じ.*

等式を表すには、`Eq`関数を使う

```python
Eq(x + 1, 4)
```

[等式判定] **Q.** たとえば $$(x+1)^2=x^2+2x+1$$が正しいかどうか知りたいとき、上で用いた==は使えるだろうか？

```python
(x + 1)**2 == x**2 + 2*x + 1 #使えない！
```

    False

ここで、==は両辺の表式が完全に等しいか聞いている。**記号として等しくないと**`True`判定にならない。 ---> 括弧の位置、変数の位置諸々異なる。

「`a == b` を調べること「 (同じ) 「`a = b` を調べること」 (同じ) 「`a - b = 0`を調べること」

===> つまり、数式評価を行うには`a - b = 0`の形で行うのがベスト。

その上で、`Eq`関数を使うときは、`(右辺)=0`となるような式の左辺を第一引数に入れるようにする.

式変形は`simplify()`関数で実行可能(詳しくは次章以降)

```python
a = (x + 1)**2
```

```python
b = x**2 + 2*x + 1
```

```python
simplify(a - b) #a - b が 0 に等しくなることをチェック.
```

```python
c = x**2 - 2*x + 1
```

```python
simplify(a - c)
```

##### 任意の点で等しいかどうか調べる方法

```python
a = cos(x)**2 - sin(x)**2
```

```python
b = cos(2*x)
```

```python
a.equals(b) #equalsメソッドを使う.
```

    True

### 2.5 ^ と /

`^` : or, `/` : and (論理判定)

```python
True ^ False
```

    True

```python
True ^ True
```

    False

```python
x ^ y
```

**注意**: べき乗として`^`を使ってはいけない.

##### コメント

- `Sympy`オブジェクト + `Sympy`オブジェクト
- `Sympy`オブジェクト + `Python`オブジェクト

--> これらすべてSympyオブジェクトを与える.

```python
type(Integer(1) + 1) #Sympyオブジェクト+ Pythonオブジェクト
```

    sympy.core.numbers.Integer

```python
type(1 + 1) #Pythonオブジェクト + Pythonオブジェクト = Pythonオブジェクト
```

    int

##### 注意 : 割り算

**注意**: SymPyオブジェクトとしての整数型`Integer()`同士の割り算は有理数を与える。
(浮動小数点数でない。)

```python
Integer(1) / Integer (3) #SymPy: 有理数
```

```python
type(Integer(1) / Integer (3))
```

    sympy.core.numbers.Rational

```python
1 / 3 #Python: 浮動小数点数
```

**注意**

- `Python2`では、整数型同士の割り算は小数点切り捨てになる

```
>> 1 / 3 = 0
```
- また以下を付けたときは、`Python2`でも浮動小数点数になる.

```
from __future__ import division
```

`Python`での計算なら、上のように浮動小数点数で得られる.

---> 上でまとめたように、`Python`のバージョンで `/`の役割が異なるので、`SymPy`で割り算を表したいときは`Rational`関数を用いると良い。

```python
Rational(1,3)
```

```python
type(Rational(1,3))
```

    sympy.core.numbers.Rational

のように定義する。このようにすれば

```python
x + Rational(1,3)
```

```python
type(x + Rational(1,3))
```

    sympy.core.add.Add

とおける。

このあたりの注意については
http://docs.sympy.org/latest/gotchas.html#gotchas
を読む。

次は[Chapter3 Basic Operations](https://hiroyuki827.github.io/SymPy_tutorial/Chapter3_Basic_Operations.html)へ!


# Local Interaction

**Tomohiro Kusano**
*Graduate School of Economics, University of Tokyo*

This notebook demonstrates how to study local interaction model using the **`localint`** Python library.

```python
import numpy as np
import networkx as nx
import matplotlib.pyplot as plt
import random
from matplotlib.animation import FuncAnimation
from __future__ import division
from localint import LocalInteraction
from IPython.display import Image
import io
import base64
from IPython.display import HTML
```

**Note:** We don't use `%matplotlib inline` here because if use it, `animation`, which is a function defined in this notebook, doesn't work in ordinary environment.

## Local Interaction Game

Let $\chi$ be a finite set of players and $P:\chi \times \chi \to \mathbb{R}_+$ be a function such that

* $P(x,x) = 0$ for all $x \in \chi$
* $P(x,y) = P(y,x)$ for all $x,y \in \chi$.

A *local interaction system* is the undirected graph induced by $(\chi, P)$. Note that $P$ can be represented by a matrix, which will be introduced as "adjacency matrix" in the next section, since $\chi$ is finite here.

For example, $(\chi, P)$, where $\chi = {0,1,2}$ and

\begin{equation*}
P =
\begin{bmatrix}
0 & 1 & 0 \\
0 & 0 & 2 \\
3 & 0 & 0
\end{bmatrix}
\end{equation*}

represents the following local interaction system.

```python
Image(filename='./localint_materials/figure_1.png')
```

The integer on each edge denote the corresponding weight on the edge.

In each period, given the local interaction system, each player plays a game constructing his/her belief, which is a distribution on the action space, according to the weights on the edges and what other players are taking.

For example, let's consider the above system. Suppose that each player has two actions (0 and 1), and Player 1, 2 are taking action 0, 1 respectively. Given the system and other players' action, Player 0 constructs a belief $(1, 3)$, which means the ratio of the probability that Player 0 meets a player taking action 0 to the probability that Player 0 meets a player taking action 1 is 1:3.

## The `LocalInteraction` class

The **`LocalInteraction`** class requires two parameters, **payoff matrix** and **adjacency matrix**.

### Payoff Matrix

Payoff matrix must be 2-dimensional square numpy array.   In a game-theoretic model, it means that both the set of actions and the payoff function are the same across all players.

For instance, consider a coordination game where the payoff table is given by the following:

1$\backslash$2 | $A$           | $B$
 ------------- |---------------| ---------
 $A$           | 4, 4          |    0, 2
 $B$           | 2, 0          |   3, 3

Note that this payoff table implies that the game is symmetric. Because of the symmetricity, it suffices to record the only one of the player's payoffs like the following:

```python
payoff_matrix = np.asarray([[4, 0],
                            [2, 3]])
print payoff_matrix
```

    [[4 0]
     [2 3]]

### Adjacency Matrix

Adjacency matrix represents how the nodes in the system are connected. In particular, in the context of the local interaction model, it represents whether each pair of players interacts and how strong the interaction of them is if they are connected.

Let's consider an adjacency matrix given by the following:
\begin{equation}
[a_{ij}] =
\begin{bmatrix}
0 &1 &3\\
2 &0 &1\\
3 &2 &0
\end{bmatrix}
\end{equation}

```python
adj_matrix = np.asarray([[0, 1, 3],
                         [2, 0, 1],
                         [3, 2, 0]])
print adj_matrix
```

    [[0 1 3]
     [2 0 1]
     [3 2 0]]

For example, $a_{12}(=1)$ denotes the weight on player 2's action to player 1. Note that the weight on player 1's action  player 2 ($a_{21}=2$) is different. That is, the **`LocalInteraction`** class allow adjacency matrix to be asymmetric.

### Creating a `LocalInteraction`

Now that we have two parameters, `payoff_matrix` and `adj_matrix`, we can create a `LocalInteraction`:

```python
li = LocalInteraction(payoff_matrix, adj_matrix)
```

```python
li.players[0]
```

    Player in a 2-player normal form game with payoff matrix:
    [[4, 0], [2, 3]]

The adjacency matrix is saved in the form of [`csr_matrix`](https://docs.scipy.org/doc/scipy-0.15.1/reference/sparse.html).

```python
li.adj_matrix
```

    <3x3 sparse matrix of type '<type 'numpy.int32'>'
    	with 6 stored elements in Compressed Sparse Row format>

### Initializing current actions

Originally, current actions are $N$-dimensional zero vector, where $N =$ "the number of players".

```python
li.N, li.current_actions
```

    (3, array([0, 0, 0]))

To initialize `current_actions`, we can use `set_init_actions`:

```python
init_actions = [1, 0, 1]
li.set_init_actions(init_actions)
```

```python
li.current_actions
```

    array([1, 0, 1])

If we don't specify the list of the players' actions, `set_init_actions` randomly set `current_actions`.

```python
li.set_init_actions()
```

```python
li.current_actions
```

    array([1, 1, 0])

## Examples

In this section, we give you a couple of examples for typical graphs, and analyze the local interaction models corresponding to those graphs.

In order to show those results graphically, we have to define functions to draw a graph and generate an animation.

```python
def draw_graph(graph_dict, figsize=(16,10), node_size=200, linewidth=2):
    fig = plt.figure(figsize=figsize, facecolor='w')
    nx.draw_networkx_nodes(graph_dict['G'], graph_dict['pos'],
                           node_size=node_size, node_color='w')
    nx.draw_networkx_edges(graph_dict['G'], graph_dict['pos'],
                           alpha=0.5, width=linewidth, arrows=False)
    plt.axis('off')
    plt.show()
```

```python
def animation(li, init_actions=None, pos='circular', node_size=200,
              node_colors=None, linewidth=2, interval=200, figsize=(16,10)):
    num_actions = li.num_actions

    if node_colors is None:
        node_colors = mpl.rcParams['axes.color_cycle']
    num_colors = len(node_colors)
    if num_colors < num_actions:
        raise ValueError('{0} colors required '.format(num_actions) +
                         '(only {0} provided)'.format(num_colors))

    G = nx.DiGraph(li.adj_matrix)

    if isinstance(pos, dict):
        pos = pos
    else:
        try:
            layout_func = getattr(nx, '{0}_layout'.format(pos))
            pos = layout_func(G)
        except:
            raise ValueError(
                "pos must be a dictionary of node-position pairs, or one of " +
                "{'circular', 'random', 'shell', 'spring', 'spectral'}")

    def get_fig(n):
        for i in range(num_actions):
            nodelist = np.where(li.current_actions == i)[0].tolist()
            nx.draw_networkx_nodes(G, pos, node_size=node_size,
                                   nodelist=nodelist,
                                   node_color=node_colors[i])
        li.play()
        return fig

    li.set_init_actions(init_actions)

    fig = plt.figure(figsize=figsize, facecolor='w')
    nx.draw_networkx_edges(G, pos, alpha=0.5, width=linewidth, arrows=False)
    anim = FuncAnimation(fig, get_fig, interval=interval)
    plt.axis('off')
    plt.show()
    plt.close()
```

### 2-actions case

For convenience, we focus on a coordination game, which is given by the following:

```python
coordination_game = np.array([[11, 0],
                              [9, 8]])
```

Also, let `node_colors_2` be a list whose $i$-th ($i = 0, 1$) element denotes a color of players taking action $i$:

```python
node_colors_2 = ['b', 'y']
```

Actually, in this case, the action 1, which leads to the risk-dominant but inefficient outcome if both players take it, is *contageous* in some sense although we don't formally define it. You would see what it means in the following section before long.

#### Circle

We first examine one of the simplest graph, called "circle graph".

```python
N = 100
circle = {}
G = nx.cycle_graph(n=N)
circle['G'] = G
circle['adj_matrix'] = nx.adjacency_matrix(G)
circle['pos'] = nx.circular_layout(G)
```

Note that we have to specify not only the graph and the adjacency matrix but also positions of nodes since `draw_graph` and `animation` require it.

```python
draw_graph(circle)
```

```python
li_coor = LocalInteraction(coordination_game, circle['adj_matrix'])
```

```python
init_actions = np.zeros(li_coor.N, dtype=int)
init_actions[[0, -1]] = 1
animation(li_coor, init_actions=init_actions, pos=circle['pos'],
          node_colors=node_colors_2, interval=100)
```

You can see that the distribution of the players taking action 1 is spreaded across all nodes as time goes on.

#### Two-dimensional lattice

We next examine another simple graph, called "Two-dimensional lattice". Actually, Its procedure for simulation is the same as the circle graph, except for that it is tedious to specify the positions of nodes in this case.

```python
N = 100
lattice2d = {}
m, n = 10, 10
G = nx.grid_2d_graph(m, n)
lattice2d['adj_matrix'] = nx.adjacency_matrix(G)
lattice2d['G'] = nx.Graph(lattice2d['adj_matrix'])
lattice2d['pos'] = {}
for i, (x, y) in enumerate(G.nodes_iter()):
    lattice2d[(x, y)] = i
    lattice2d['pos'][i] = (x/(m-1), y/(n-1))
```

```python
draw_graph(lattice2d)
```

```python
li_coor = LocalInteraction(coordination_game, lattice2d['adj_matrix'])
```

```python
# m, n = 10, 10
init_actions = np.zeros(li_coor.N, dtype=int)
for node in [(m//2-i, n//2-j) for i in range(2) for j in range(2)]:
    init_actions[lattice2d[node]] = 1
animation(li_coor, init_actions=init_actions, pos=lattice2d['pos'],
          node_colors=node_colors_2, figsize=(14,8), interval=500)
```

### 3-actions case

The `localint` module works even in 3-actions case. Let's consider the following game, which is called "Bilingual Game":

```python
def bilingual_game(e, a=11, b=0, c=9, d=8):
    A = np.array([[a  , a  , b],
                  [a-e, a-e, d-e],
                  [c  , d  , d]])
    return A
```

```python
bg = bilingual_game(e=0.1)
bg
```

    array([[ 11. ,  11. ,   0. ],
           [ 10.9,  10.9,   7.9],
           [  9. ,   8. ,   8. ]])

```python
node_colors_3 = ['b', 'r', 'y']
```

We show that even the action 0, which leads to Pareto efficient outcome, can be contagious in this case.

#### Circle

```python
li_bg = LocalInteraction(bg, circle['adj_matrix'])
```

```python
init_actions = np.ones(li_bg.N, dtype=int) * 2
init_actions[[0, 1, -2, -1]] = 0
animation(li_bg, init_actions=init_actions, pos=circle['pos'],
          node_colors=node_colors_3, interval=100)
```

#### Two-dimensional lattice

```python
li_bg = LocalInteraction(bg, lattice2d['adj_matrix'])
```

```python
# m, n = 10, 10
init_actions = np.ones(li_bg.N, dtype=int) * 2
for node in [(m//2-i, n//2-j) for i in range(2) for j in range(2)]:
    init_actions[lattice2d[node]] = 0
animation(li_bg, init_actions=init_actions, pos=lattice2d['pos'],
          node_colors=node_colors_3, interval=500)
```


# Understanding Learning by Error Propagation

This notebook is a designed in a way such that it serves as a walkthrough for the paper [Learning Internal Representations by Error Propagation](https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf) (1985) by Rumelhart D., Hinton G. and Williams R. The notebook aims to build a neural network that is learning pattern through propagating error on input/output pairs.

Table of Contents

1  Deep Neural Network

1.1  Hidden Units

1.2  Generalized Delta Rule

2  Implementation

2.0.1  Define Error Measure

2.0.2  Initializing Weights

2.0.3  Forward Propagation

2.0.4  Backward Propagation

2.0.5  Updating weights

2.0.6  Repeating the Procedure

2.0.7  Step - 7 Inference

3  3. Sample Model

3.0.1  Learning

3.0.2  Other Datasets

**Notes**

1. To keep things simple, we will use the planar dataset for illustration and application purposes. At the end of the notebook, we will also build a network with *breast cancer* dataset.
2. To be able to make the shift from reading to understanding the idea smoother, we try to preserve the original notation from the paper which is different than the commonly used notations/names. However, we include a map of names from original notation to common notation.
3. For whom taken the course of deeplearning.ai's [Deep Learning Specialization](https://www.coursera.org/learn/neural-networks-deep-learning), the structure of the notebook should be similar with the programming assignments with little changes in implementation.

## Deep Neural Network

Deep Neural Network term is used for neural networks with one or more hidden layers. This implementation is not based on some predefined architecture, but rather it's built for any type of architectures. The paper presented the architecture as inputs starting from bottom and fed to top, see the figure below.

Figure 1: Bottom-Up Architecture

Although this is the original presentation from the paper, here we use left-to-right presentation rather than bottom-to-top in illustrations as it's slightly easier to apperceive, see the figure below. Also, we will make an implementation where the layer size, and number of units are configurable so the image below represents fixed layer sizes only for illustration purposes.

Figure 2: Left-to-Right Architecture

### Hidden Units

Hidden unit is a network unit which is a player in generating an internal representation of input pattern. Actually, if we are speaking layer-by-layer, a layer of hidden units of size `n_h` just outputs a representation from previous layer which is input pattern if its the first layer we are speaking of. Directly quoting from the paper (Figure 1 is Figure 1 in the notebook), see the statement below about internal representation and hidden units,

> ..., if there is a layer of simple perceptron-like hidden units, as shown in Figure 1, with which the original input pattern can be augmented, there is always a recoding (i.e., an internal representation) of the input patterns in the hidden units in which the similarity of the patterns among the hidden units can support any required mapping from the input to the output units. Thus, if we have the right connections from the input units to a large enough set of hidden units, we can always find a representation that will perform any mapping from input to output through these hidden units. ...

Forward propagation by feeding hidden units with the previous layer's output $o^{l-1}$ the equations are shown below:

$$net_{pj} = \sum_{i} w_{ji} o_{pi} + \theta_j$$
where $o_{pi} = i_{pi}$ if $u_j$ is an input unit, and
$$o_{pj} = f_j(net_{pj})$$

In a vectorized form using superscript for layers the equations is as follows:

$$net^{l} = o^{l-1} w^{l} + \theta^{l}$$
where $o^l = X$ if $l = 0$ is the input layer, and
$$o^{l} = f_l(net^{l})$$

### Generalized Delta Rule

At the end of the chapter "The Problem" of the paper the authors pointed out previous work and approaches for learning in networks with hidden units.

In the paper they introduced error function as a squared difference of target values and output values of the network. Find the section below in the paper:

>To be more specific, then, let
$E_p = \frac{1}{2} \sum_j (t_{pj} - o_{pj})^2$
be our measure of the error on input/output pattern $p$ and let $E = \sum E_p$ be our
overall measure of the error.

The proposed rule for changing the weights is as follows:

\begin{equation}
  \begin{aligned}
    \Delta_p w_{ji} &= \eta \delta_{pj} o_{pi}\\
    \Delta_p t_{j} &= \eta \sum_k \delta_{pk}
  \end{aligned}
\end{equation}

where $\Delta_p w_{ji}$ is the change to be made, and, the part where $\delta_{pj}$ is explained is quoted below.

>The trick is to figure out what $\delta_{pj}$ should be for
each unit $u_j$ in the network. The interesting result, which we now derive, is that
there is a simple recursive computation of these $\delta$'s which can be implemented by
propagating error signals backward through the network.
>
>To compute $\delta_{pj} = -\frac{\partial E_p}{\partial {net}_{pj}}$, we apply the chain rule to write this partial derivative as the product of two factors, one factor reflecting the change in error as a function of the output of the unit and one reflecting the change in the output as a function of changes in the input. Thus, we have

>$$\delta_{pj} = -\frac{\partial E_p}{\partial {net}_{pj}} = -\frac{\partial E_p}{\partial {o}_{pj}}\frac{\partial o_{pj}}{\partial {net}_{pj}}$$

>, and $$\frac{\partial o_{pj}}{\partial {net}_{pj}} = f'_j({net}_{pj})$$

, with using $E_p$ introduced in the paper, the rule for $\delta_{pj}$ is as follows

$$\begin{equation}
\delta_{pj} = \begin{cases} \label{eq:delta_rule} \tag{1}
      (t_{pj} - o_{pj})f'_j({net}_{pj}) & \textit{if $u_j$ is an output unit} \\
      f'_j({net}_{pj})\sum_{k} \delta_{pk} w_{kj} & \textit{otherwise}
   \end{cases}
\end{equation}$$

If you have read the derivation part and followed the equations, you probably already saw that the subscripts are everywhere, that is the paper equations proceed for one unit at a time. We make a slight change here, and rather than applying loops and compute the weights, outputs, and delta item by item, we use vectorization. In vectorized form, the change of the update rule ($\Delta_p w_{ji}$) could be expressed as follows:

$$\begin{equation}
  \begin{aligned}
    \nabla w^{l} &= \eta \: \frac{1}{m} {o^{l}}^T \delta^l\\
    \nabla \theta^{l} &= \eta \: \frac{1}{m} \: \sum_{i} \delta^{l}_{ij}
  \end{aligned}
\end{equation}$$

, and

$$\begin{equation}
\delta^{l} = \begin{cases} \label{eq:vectorized_delta_rule} \tag{2}
      (t - o^{l}) f'_l({net}^{l}) & \textit{if $l=L$} \\
      f'_l({net}^{l}) \delta^{l+1} {w^{l}}^T & \textit{otherwise}
   \end{cases}
\end{equation}$$

where $L = \textit{# of layers} - 1$ in a way that excluding the input layer. So consider L is the number of hidden layers plus the output layer. Also, note that we are going in a backward direction, i.e computing from $\delta^{4}$ for towards $\delta^{0}$ hence $\delta^{l+1}$ is the previous $\delta$ computed. It's important to note that we do not need $\delta^{0}$ as it is not used in any computation of weight changes.

, and
$\Delta w^{l}$ is the change in the weight matrix of layer $l$,
$\delta^{l}$ is the delta matrix computed at layer $l$,
$o^{l-1}$ is the output matrix of the layer $l-1$

**Notes**

- It's important to note that we use $L = \textit{# of layers} - 1$ in order to attain the index of the last layer as the $L^{th}$ layer.
- Also note that in the delta rule we used $o^{l}$ to compute $\delta^{l}$ that's because we will include input X (i.e $i^{l}$) in both the cache of $net$ and $o$ as the starting values. Thus, we have ${net}^{0} = o^{0} = X$. You can consider it as if input data is our first $net$ values, and it's activated by a linear activation, $f(x) = x$.

## Implementation

Let's start implementing components of a network. Before jumping into the code here, in the notebook, I wanted to explicitly construct the components with functions, and avoided using a class on purpose. The main reason is for one to see how each function works, and what inputs are needed for each function. At the end, you can see the full-implementation from `./nnet.py`. After making the point, let's start building.

**Architecture**

We will build two hidden layer neural network for planar data classification. This data is also used in "week 3 programming assignment" of [Neural Networks and Deep Learning](https://www.coursera.org/learn/neural-networks-deep-learning?specialization=deep-learning). We also want to use that dataset as well as we mentioned it at the beginning of this notebook, it makes it easier to reader to follow up what is going on and also, the network architecture will be simple. We make, however, a little change and build a two hidden layer network. The network will be as follows, where we have 2 hidden layers of sizes 6 and 3 respectively using tanh(x) activation function in both, and 1 unit in output layer we will use sigmoid as an activation function. The shape of the matrices or vectors are written in brackets, where m is the number of observations and *name_i* indicating the $i^{th}$ component of that variable.

Figure 3: Network for Implementation

The main steps to perform gradient descent using back propagation is as follows:

1. Define Error measure
2. Initialize weights
3. Forward Propagation
4. Backward Propagation (Find Gradients)
5. Update Weights.
6. Repeat step 3-5 for `# of epochs` times.
7. Make prediction with final weights.

**Notes**

- We used subscript p in variable names even though we are using vectorized form to make easier readability with paper.

First, import the required libraries.

```python
# Scientific libraries
import numpy as np
from scipy.misc import derivative

# Loading Plotting Modules
import matplotlib.pyplot as plt
```

#### Define Error Measure

We will use [MSE](https://en.wikipedia.org/wiki/Mean_squared_error) loss to measure the error between label/output pair, and we will have `error()` function to evaluate overall error. For simplicity, our error function use MSE as a loss function only, but other loss functions can be implemented for ones who'd like to try different losses. A list of some loss functions can be found [here](https://en.wikipedia.org/wiki/Category:Loss_functions).

```python
def mse_loss(t_p, o_p):
    """
    Computes Mean Squared Error given target and network output.
    Vector E_p in short.

    Parameters
    ----------
    t_p : np.ndarray
        Target matrix/labels containing ground truth values.

    Returns
    -------
    loss : float
        Return all loss values for each label/output pair.
    """
    m = t_p.shape[0]
    return 1/(2*m) * (t_p - o_p)**2

def error(t_p, o_p):
    """
    Commonly known as the cost function.
    Computing the total cost by MSE given target and output by network.

    Parameters
    ----------
    t_p : np.ndarray
        Target vector/labels containing ground truth values.
    o_p : np.ndarray
        Output vector generated by the network.

    Returns
    -------
    total_cost : float
        Overall measure of error. Sum of all losses.
    """
    # Mean Sqaured Error (MSE)
    E_p = mse_loss(t_p, o_p)
    return np.sum(E_p)

```

#### Initializing Weights

We will initialize weight matrices and bias vectors given layer sizes. Referring to Figure 3, you can see that our layer sizes are 2 for the input, 6 and 3 for hidden layers, 1 for the output layer (2,6,3,1), and weight shapes are (2,6), (6,3) and (3,1), and for bias vectors (1,6), (1,3) and (1,1). The pattern is obvious for shapes. So let's implement this.

For convenience we will name list of weight matrices as `W`, and later in passes single weight matrix associated with a layer as `w`. Also for list of bias vectors we use the name `T` for variable, and for single bias vector as `t`. Please do not confuse this lower case `t` with `t_p` (target variable/labels).

```python
def initialize_weights(layers: list):
    """
    Randomly initialize weight matrices and bias vectors by given layer sizes.

    Parameters
    ----------
    layers : list
        A Python list containing layer sizes.

    Returns
    -------
    weights : tuple
        A tuple of size 2 containing weight matrices and bias vectors respectively.
    """
    W = [np.random.randn(layers[i], s) * 0.01 for i, s in enumerate(layers[1:])]
    T = [np.zeros((1, s)) for s in layers[1:]]

    return W, T
```

#### Forward Propagation

Basic forward propagation procedure, starting from the input layer feed the input through the network, and get the cache and the output of last layer. We then use it for error propagation. See the equations in chapter 1.1. for forward passes.

```python
# Forward Propagation

def forward_pass(o_p, w, t, activation_func):
    """
    Performs one step of forward propagation.

    Parameters
    ----------
    o_p : np.ndarray
        An array containing the outputs of the layer l.
    w : np.ndarray
        Weight matrix to compute layer l+1.
    t : np.ndarray
        Bias vector to comput layer l+1.

    Returns
    -------
    cache : tuple
        A tuple of size 2 containing output and net for layer l+1 respectively.
    """
    net_p = np.matmul(o_p, w) + t
    o_p = activation_func(net_p)
    return o_p, net_p

def forward_propagate(input_data, W, T, activation_func, task):
    """
    Performs L step forward propagation.

    Parameters
    ----------
    input_data : np.ndarray
        A matrix of input data of size m,n where m and n are # of observations and # of features respectively.
    W : np.ndarray
        List of weight matrices for L layers.
    T : np.ndarray
        List of bias vectors for L layers.
    activation_func: function
        Activation function to be used to compute outputs of layers.
    task: str
        String indicating the learning task.

    Returns
    -------
    cache : tuple
        A tuple of size 2 containing outputs and nets for L layers.
    """
    L = len(W)
    o_ps, net_ps = [input_data], [input_data] # See Notes under 1.2.

    o_p = input_data

    for layer in range(L):
        w, t = W[layer], T[layer]
        if layer == L-1 and task == 'binary_classification':
            o_p, net_p = forward_pass(o_p, w, t, activation_func=sigmoid)
        elif layer == L-1 and task == 'regression':
            o_p, net_p = forward_pass(o_p, w, t, activation_func=linear)
        else:
            o_p, net_p = forward_pass(o_p, w, t, activation_func=activation_func)

        o_ps.append(o_p); net_ps.append(net_p)

    return o_ps, net_ps
```

#### Backward Propagation

Back propagation procedure as its name suggests proceeds starting from the last layer (output of last layer) in a backward (right-to-left) direction. This step is needed to compute the gradients which are later used to update the weights. See equations in chapter 1.2.

```python
def backward_pass(f, net_p, delta_prev, w):
    """
    Performs one step of backward propagation.

    Parameters
    ----------
    f : function
        Activation function used to compute layer l.
    net_p : np.ndarray
        A net matrix computed at layer l.
    delta_prev : np.ndarray
        Delta matrix computed at layer l+1.
    w : np.ndarray
        Weight matrix at layer l.

    Returns
    -------
    delta_p : np.ndarray
        Matrix of delta at layer l.
    """
    dnet_p = derivative(f, net_p)
    delta_w = np.matmul(delta_prev, w.T)
    return dnet_p * delta_w

def backward_propagate(o_ps, net_ps, t_p, W, activation_func, task):
    """
    Performs L step backward propagation.

    Parameters
    ----------
    o_ps : list
        Outputs of layers returned by cache at forward propagation.
    net_ps : list
        Pre-outputs (before activations), nets, of layers returned by cache at forward propagation.
    t_p : np.ndarray
        Target matrix/labels containing ground truth values.
    W : np.ndarray
        List of weight matrices for L layers.
    activation_func: function
        Activation function to be used to compute outputs of layers.
    task: str
        String indicating the learning task.

    Returns
    -------
    grads : tuple
        Gradients for weight matrices and bias vectors respectively.
    """
    L = len(W)
    m = t_p.shape[0]
    dW, dT = [], []

    # Initialize the recursion with the output layer
    do_p = derivative(lambda x: mse_loss(t_p, x), o_ps[-1])
    dnet_p = derivative(sigmoid, net_ps[-1])
    delta_p = do_p * dnet_p

    for layer in reversed(range(L)):
        dw = np.matmul(o_ps[layer].T, delta_p)
        dt = np.sum(delta_p, axis=0, keepdims=True)

        if layer != 0: # No need to compute delta_0 (skipped for performance)
            delta_p = backward_pass(activation_func, net_p=net_ps[layer], delta_prev=delta_p, w=W[layer])

        dW.insert(0, dw); dT.insert(0, dt)

    return dW, dT
```

#### Updating weights

Now we are here to update the weights to reduce the error between target and output of the network (output of output layer). We need an error metric by the way.

```python
def update_weights(W, T, dW, dT, eta):
    """
    Update weights given gradients and learning rate.

    Parameters
    ----------
    W : np.ndarray
        List of weight matrices for L layers.
    T : np.ndarray
        List of bias vectors for L layers.
    W : np.ndarray
        List of gradients for weight matrices for L layers.
    T : np.ndarray
        List of gradients for bias vectors for L layers.
    eta : float
        Learning rate.

    Returns
    -------
    weights : tuple
        List of updated weight matrices and bias vectors.
    """
    i = 0
    while i < len(W):
        W[i] = W[i] - eta * dW[i]
        T[i] = T[i] - eta * dT[i]
        i += 1

    return W, T
```

#### Repeating the Procedure

As of now, we completed the one step of forward and backward propagation process. Now, we will gather what we did in the previous steps in a function called `fit()`. The name *fit* is a widely common and accepted choice for starting the training process in most of the well-known libraries and frameworks, i.e fitting X to y.

```python
def fit(X, y, layers, epochs: int = 50, learning_rate: float = 0.1, activation_func=None, task='binary_classification',
       print_cost_each=0, return_cost=False):
    """
    Build a network map from the given input data X to target y with given size of layers and parameters.

    Parameters
    ----------
    X : np.ndarray
        Input data to be fed into the network.
    y : np.ndarray
        Target variable/Labels for the input data.
    layers : list
        List of layer sizes of each layer in a left-to-right architecture.
    epochs : int
        Number of epochs.
    learning_rate : float
        Learning rate.
    activation_func : function
        An activation function to be used in layers except the output layer.
    task : str
        A string indicating the current task. Values `binary_classification` or `regression`.
    print_cost_each : int, default=0
        An integer to print cost value for every number specified.
    return_cost : bool, default=False
        A boolean value for whether to return the cost as a 3rd return value.

    Returns
    -------
    weights : tuple
        Tuple of computed final/learned weight matrices and bias vectors respectively. If return_cost is True
        the return tuple have 3 elements with the last element as the cost after the last epoch.
    """
    L = len(layers)
    layers_cp = layers.copy()
    input_size = X.shape[1]
    layers_cp.insert(0, input_size)
    W, T = initialize_weights(layers_cp)

    y = y.reshape(len(y), 1)

    if activation_func is None:
        # default to linear activation
        activation_func = linear
    elif isinstance(activation_func, list):
        assert len(activation_func) == L

    for epoch in range(epochs):
        o_ps, net_ps = forward_propagate(X, W, T, activation_func=activation_func, task=task)
        E = error(t_p=y, o_p=o_ps[-1])
        dW, dT = backward_propagate(o_ps, net_ps, y, W, activation_func=activation_func, task=task)
        W, T = update_weights(W, T, dW, dT, learning_rate)

        if print_cost_each != 0 and epoch % print_cost_each == 0:
            print(f'Epoch: {epoch} \t Cost: {E}')

    print(f'Epoch: {epochs} \t Cost: {E}')
    if return_cost:
        return W, T, E

    return W, T
```

#### Inference

Finally, after learning the parameters, we can make predictions given input data. The prediciton process is just one step of forward propagation with the weights and biases that are learned in the training process. See the predict function below

```python
def predict(X, W, T, activation_func=None, task='binary_classification', return_classes=False):
    """
    Make predictions given input data X using the learned weights from fit().

    Parameters
    ----------
    X : np.ndarray
        Input data to be fed into the network.
    W : list
        List of learned weight matrices returned from fit().
    T : list
        List of learned bias vectors returned from fit().
    activation_func : function, default=None
        An activation function to be used in layers except the output layer. If None,
        linear activation will be used.
    task : str, default='binary_classification'
        A string indicating the current task to apply which activation function at the output layer.
        Values `binary_classification` or `regression`.
    return_classes : bool, default=False
        A boolean indicating whether to return predicted classes. If False, probabilities are returned.

    Returns
    -------
    y_pred : np.ndarray
        An array of predicted values for input X. If return classes binary values are returned.
    """
    ops, _ = forward_propagate(X, W, T, activation_func=activation_func, task=task)

    if return_classes:
        return np.vectorize(lambda x: 1 if x>0.5 else 0)(ops[-1])

    return ops[-1]
```

## 3. Sample Model

We will build a model on a planar dataset which has two features, and has a fancy shape on a scatter plot by these two features. See the plot below.

Figure 3: Scatter plot of features of X by classes 0 and 1

```python
from nb_utils import load_planar_dataset, plot_decision_boundary, sigmoid, tanh
from sklearn.metrics import accuracy_score
```

```python
X, y = load_planar_dataset()

layers_ = [6,3,1]
task_ = 'binary_classification'
activation_func_ = tanh
learning_rate_ = 1.2
epochs_ = 10000

W, T = fit(X, y, layers=layers_, epochs=epochs_, learning_rate=learning_rate_, activation_func=activation_func_, task=task_,
          print_cost_each=1000)

y_pred = predict(X, W, T, activation_func_, task=task_, return_classes=True)

accuracy = accuracy_score(y, y_pred)
print(f'Accuracy: {accuracy*100: .4f}')

plot_decision_boundary(lambda x: predict(x, W, T, activation_func_, task=task_, return_classes=True), X, y)
```

#### Learning

Observe the plots below to see how the model learned the weights by 1000 epochs up to 10000 epochs.

```python
plt.figure(figsize=(16, 32))

for i in range(1,11):
    plt.subplot(5, 2, i)
    epoch = i*1000
    W, T, cost = fit(X, y, layers=layers_, epochs=epoch, learning_rate=learning_rate_, activation_func=activation_func_,
               task=task_, print_cost_each=0, return_cost=True)

    y_pred = predict(X, W, T, activation_func_, task=task_, return_classes=True)

    accuracy = accuracy_score(y, y_pred)

    plot_decision_boundary(lambda x: predict(x, W, T, activation_func_, task=task_, return_classes=True), X, y)
    plt.title(f'Epoch {epoch}, Accuracy: {accuracy*100: .3f}%, Cost: {cost: .3f}')
```

#### Other Datasets

Let's build a network on another dataset, and as we mentioned from the beginning of the notebook, we will use the [breast cancer dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) from sklearn for simplicity as a classification task, and [boston dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html) as a regression task. For the implementations of neural nets for these datasets, see [this notebook](./example_datasets.ipynb) You can try the network on different datasets, and make predictions. We again do not make predictions with unseen data, one can do it if she/he wishes to do with using the simple predict function. Also, we are going to use a bigger network now, with 2 hidden layers with 40 and 20 units respectively. Also, notice that we use different *learning rate*, *activation function* and *number of epochs*.


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib widget
```

# shortcuts
> a : to create new cell above

> b : to create new cell below

> dd : delete cell

> z : to undo

> shit + enter : run cell

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib widget
x = np.linspace(0,20,100)
y = np.exp(-x)
y1 = np.exp(-x/10)
fig = plt.figure()
ax = fig.add_subplot()
ax.plot(y)
ax.plot(y1)
plt.show()
```

        Figure

---
# [Line integral](https://www.youtube.com/watch?v=yyqOvA8dlvk&list=PL4ElPQw0dPdlhkDrkVLsktCZWkILzKhvM&index=3)
> Filed F by line r
    $$ \int_{c} \vec{F} \cdot d\vec{r} \\
        \int_{c} \vec{F}(\vec{r}(t)) \cdot \frac{dr}{dt}dt
    $$

# Problem

> vector field
> - $\vec u \,= \,(xy, z^2, x) $

> curve
> - $x = 1 + t , y = 0, z = t^2, \\
    0 \leq x \leq 3 $

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp
%matplotlib widget

t = np.linspace(0,3,100)

rx = 1 + t
ry = np.zeros_like(rx)
rz = t**2
x = np.linspace(0,5,10)
x,y,z = np.meshgrid(x,x,x)
ux = x*y
uy = z**2
uz = x

fig = plt.figure()
ax = fig.add_subplot(projection='3d')
ax.set_title(r'$\vec{u} = < xy,z^2,x>, curve = <1+t,0,t2>, 0 \leq x \leq 3$')
ax.quiver(x,y,z,ux,uy,uz,length=0.1,normalize=True)
ax.plot(rx,ry,rz,'r')
plt.show()

```

        Figure

# solve

$$ \int_{c} \vec{u}(\vec{r(t)}) \cdot \frac{dr}{dt}dt \\
    \vec{u} = < xy, z^2, x >, \\
    \vec{u} = < (1+t)* 0, (t^2)^2, 1+t > \\
    curve = \vec {r} = < 1+t, 0, t^2 > \to \vec{r}dt = < 1,0,2t> \\
    < 0,t^4,1+t> \cdot <1,0,2t>dt \to < 0,0,2t + t^2>dt \\
    \int_{t_0}^{t_3} <0,0,2t+t^2>dt \\
        = \int_{0}^{3} 2t+t^2 dt \\
        = \left . t^2 + \frac{1}{3}t^3 \right|_{0}^{3}
$$

```python

```