Copyright (c) 2015, 2016
[Sebastian Raschka](http://sebastianraschka.com/)
[Li-Yi Wei](http://liyiwei.org/)

https://github.com/1iyiwei/pyml

[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)

# Python Machine Learning - Code Examples

# Chapter 3 - A Tour of Machine Learning Classifiers

* Logistic regression
 * Binary and multiple classes
* Support vector machine
 * Kernel trick
* Decision tree
 * Random forest for ensemble learning
* K nearest neighbors

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,pandas,matplotlib,sklearn
```

    last updated: 2016-11-07

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.1
    pandas 0.18.1
    matplotlib 1.5.1
    sklearn 0.18

*The use of `watermark` is optional. You can install this IPython extension via "`pip install watermark`". For more information, please see: https://github.com/rasbt/watermark.*

### Overview

- [Choosing a classification algorithm](#Choosing-a-classification-algorithm)
- [First steps with scikit-learn](#First-steps-with-scikit-learn)
    - [Training a perceptron via scikit-learn](#Training-a-perceptron-via-scikit-learn)
- [Modeling class probabilities via logistic regression](#Modeling-class-probabilities-via-logistic-regression)
    - [Logistic regression intuition and conditional probabilities](#Logistic-regression-intuition-and-conditional-probabilities)
    - [Learning the weights of the logistic cost function](#Learning-the-weights-of-the-logistic-cost-function)
    - [Handling multiple classes](#Handling-multiple-classes)
    - [Training a logistic regression model with scikit-learn](#Training-a-logistic-regression-model-with-scikit-learn)
    - [Tackling overfitting via regularization](#Tackling-overfitting-via-regularization)
- [Maximum margin classification with support vector machines](#Maximum-margin-classification-with-support-vector-machines)
    - [Maximum margin intuition](#Maximum-margin-intuition)
    - [Dealing with the nonlinearly separable case using slack variables](#Dealing-with-the-nonlinearly-separable-case-using-slack-variables)
    - [Alternative implementations in scikit-learn](#Alternative-implementations-in-scikit-learn)
- [Solving nonlinear problems using a kernel SVM](#Solving-nonlinear-problems-using-a-kernel-SVM)
    - [Using the kernel trick to find separating hyperplanes in higher dimensional space](#Using-the-kernel-trick-to-find-separating-hyperplanes-in-higher-dimensional-space)
- [Decision tree learning](#Decision-tree-learning)
    - [Maximizing information gain – getting the most bang for the buck](#Maximizing-information-gain-–-getting-the-most-bang-for-the-buck)
    - [Building a decision tree](#Building-a-decision-tree)
    - [Combining weak to strong learners via random forests](#Combining-weak-to-strong-learners-via-random-forests)
- [K-nearest neighbors – a lazy learning algorithm](#K-nearest-neighbors-–-a-lazy-learning-algorithm)
- [Summary](#Summary)

```python
from IPython.display import Image
%matplotlib inline
```

```python
# Added version check for recent scikit-learn 0.18 checks
from distutils.version import LooseVersion as Version
from sklearn import __version__ as sklearn_version
```

# Choosing a classification algorithm

There is no free lunch; different algorithms are suitable for different data and applications.

# First steps with scikit-learn

In the linear perceptron part, we wrote the models from ground up.
* too much coding

Existing machine learning libraries
* scikit-learn
* torch7, caffe, tensor-flow, theano, etc.

Scikit-learn
* will use for this course
* not as powerful as other deep learning libraries
* easier to use/install
* many library routines and data-sets to use, as exemplified below for main steps for a machine learning pipeline.

## Iris dataset

Let's use this dataset for comparing machine learning methods

Setosa

Versicolor

Virginica

Loading the Iris dataset from scikit-learn.

```python
from sklearn import datasets
import numpy as np

iris = datasets.load_iris()

print('Data set size: ' + str(iris.data.shape))

X = iris.data[:, [2, 3]]
y = iris.target

print('Class labels:', np.unique(y))
```

    Data set size: (150, 4)
    Class labels: [0 1 2]

```python
import pandas as pd
df = pd.DataFrame(iris.data)
df.tail()
```

      0
      1
      2
      3

      145
      6.7
      3.0
      5.2
      2.3

      146
      6.3
      2.5
      5.0
      1.9

      147
      6.5
      3.0
      5.2
      2.0

      148
      6.2
      3.4
      5.4
      2.3

      149
      5.9
      3.0
      5.1
      1.8

Here, the third column represents the petal length, and the fourth column the petal width of the flower samples.

The classes are already converted to integer labels where 0=Iris-Setosa, 1=Iris-Versicolor, 2=Iris-Virginica.

## Data sets: training versus test

Use different data sets for training and testing a model (generalization)

```python
if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split

# splitting data into 70% training and 30% test data:
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0)
```

```python
num_training = y_train.shape[0]
num_test = y_test.shape[0]
print('training: ' + str(num_training) + ', test: ' + str(num_test))
```

    training: 105, test: 45

## Data scaling

It is better to scale the data so that different features/channels have similar mean/std.

```python
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
sc.fit(X_train)
X_train_std = sc.transform(X_train)
X_test_std = sc.transform(X_test)
```

## Training a perceptron via scikit-learn

We learned and coded perceptron in chapter 2.
Here we use the scikit-learn library version.

The perceptron only handles 2 classes for now.
We will discuss how to handle $N > 2$ classes.

```python
from sklearn.linear_model import Perceptron

ppn = Perceptron(n_iter=40, eta0=0.1, random_state=0)
_ = ppn.fit(X_train_std, y_train)
```

```python
y_pred = ppn.predict(X_test_std)
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
```

    Misclassified samples: 4 out of 45

```python
from sklearn.metrics import accuracy_score

print('Accuracy: %.2f' % accuracy_score(y_test, y_pred))
```

    Accuracy: 0.91

```python
from matplotlib.colors import ListedColormap
import matplotlib.pyplot as plt
import warnings

def versiontuple(v):
    return tuple(map(int, (v.split("."))))

def plot_decision_regions(X, y, classifier, test_idx=None,
                          resolution=0.02, xlabel='', ylabel='', title=''):

    # setup marker generator and color map
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot the decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                           np.arange(x2_min, x2_max, resolution))
    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)
    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    for idx, cl in enumerate(np.unique(y)):
        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
                    alpha=0.8, c=cmap(idx),
                    marker=markers[idx], label=cl)

    # highlight test samples
    if test_idx:
        # plot all samples
        if not versiontuple(np.__version__) >= versiontuple('1.9.0'):
            X_test, y_test = X[list(test_idx), :], y[list(test_idx)]
            warnings.warn('Please update to NumPy 1.9.0 or newer')
        else:
            X_test, y_test = X[test_idx, :], y[test_idx]

        plt.scatter(X_test[:, 0],
                    X_test[:, 1],
                    c='',
                    alpha=1.0,
                    linewidths=1,
                    marker='o',
                    s=55, label='test set')

    plt.title(title)
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.legend(loc='upper left')

    plt.tight_layout()
    plt.show()
```

Training a perceptron model using the standardized training data:

```python
X_combined_std = np.vstack((X_train_std, X_test_std))
y_combined = np.hstack((y_train, y_test))

test_idx = range(X_train_std.shape[0], X_combined_std.shape[0])

plot_decision_regions(X=X_combined_std, y=y_combined,
                      classifier=ppn, test_idx=test_idx,
                      xlabel='petal length [standardized]',
                      ylabel='petal width [standardized]')
```

# Modeling class probabilities via logistic regression

* $\mathbf{x}$: input
* $\mathbf{w}$: weights
* $z = \mathbf{w}^T \mathbf{x}$
* $\phi(z)$: transfer function
* $y$: predicted class

## Perceptron
$
y = \phi(z) =
\begin{cases}
1 \; z \geq 0 \\
-1 \; z < 0
\end{cases}
$

## Adaline
$
\begin{align}
\phi(z) &= z \\
y &=
\begin{cases}
1 \; \phi(z) \geq 0 \\
-1 \; \phi(z) < 0
\end{cases}
\end{align}
$

## Logistic regression
$
\begin{align}
\phi(z) &= \frac{1}{1 + e^{-z}} \\
y &=
\begin{cases}
1 \; \phi(z) \geq 0.5 \\
0 \; \phi(z) < 0.5
\end{cases}
\end{align}
$

Note: this is actually classification (discrete output) not regression (continuous output); the naming is historical.

```python
import matplotlib.pyplot as plt
import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

z = np.arange(-7, 7, 0.1)
phi_z = sigmoid(z)

plt.plot(z, phi_z)
plt.axvline(0.0, color='k')
plt.ylim(-0.1, 1.1)
plt.xlabel('z')
plt.ylabel('$\phi (z)$')

# y axis ticks and gridline
plt.yticks([0.0, 0.5, 1.0])
ax = plt.gca()
ax.yaxis.grid(True)

plt.tight_layout()
# plt.savefig('./figures/sigmoid.png', dpi=300)
plt.show()
```

### Logistic regression intuition and conditional probabilities

$\phi(z) = \frac{1}{1 + e^{-z}}$: sigmoid function

$\phi(z) \in [0, 1]$, so can be interpreted as probability: $P(y = 1 \; | \; \mathbf{x} ; \mathbf{w}) = \phi(\mathbf{w}^T \mathbf{x})$

We can then choose class by interpreting the probability:
$
\begin{align}
y &=
\begin{cases}
1 \; \phi(z) \geq 0.5 \\
0 \; \phi(z) < 0.5
\end{cases}
\end{align}
$

The probability information can be very useful for many applications
* knowing the confidence of a prediction in addition to the prediction itself
* e.g. weather forecast: tomorrow might rain versus tomorrow might rain with 70% chance

Perceptron:

Adaline:

Logistic regression:

### Learning the weights of the logistic cost function

$J(\mathbf{w})$: cost function to minimize with parameters $\mathbf{w}$

$z = \mathbf{w}^T \mathbf{x}$

For Adaline, we minimize sum-of-squared-error:
$$
J(\mathbf{w}) = \frac{1}{2} \sum_i \left( y^{(i)} - t^{(i)}\right)^2
= \frac{1}{2} \sum_i \left( \phi\left(z^{(i)}\right) - t^{(i)}\right)^2
$$

#### Maximum likelihood estimation (MLE)
For logistic regression, we take advantage of the probability interpretation to maximize the likelihood:
$$
L(\mathbf{w}) = P(t \; | \; \mathbf{x}; \mathbf{w}) = \prod_i P\left( t^{(i)} \; | \; \mathbf{x}^{(i)} ; \mathbf{w} \right) = \prod_i \phi\left(z^{(i)}\right)^{t^{(i)}} \left(1 - \phi\left(z^{(i)}\right)\right)^{1-t^{(i)}}
$$

Why?
$$
\begin{align}
\phi\left(z^{(i)}\right)^{t^{(i)}} \left(1 - \phi\left(z^{(i)}\right)\right)^{1-t^{(i)}} =
\begin{cases}
\phi\left(z^{(i)} \right) & \; if \; t^{(i)} = 1 \\
1 - \phi\left(z^{(i)}\right) & \; if \; t^{(i)} = 0
\end{cases}
\end{align}
$$

This is equivalent to minimize the negative log likelihood:
$$
J(\mathbf{w})
= -\log L(\mathbf{w})
= \sum_i -t^{(i)}\log\left(\phi\left(z^{(i)}\right)\right) - \left(1 - t^{(i)}\right) \log\left(1 - \phi\left(z^{(i)}\right) \right)
$$

Converting prod to sum via log() is a common math trick for easier computation.

```python
def cost_1(z):
    return - np.log(sigmoid(z))

def cost_0(z):
    return - np.log(1 - sigmoid(z))

z = np.arange(-10, 10, 0.1)
phi_z = sigmoid(z)

c1 = [cost_1(x) for x in z]
plt.plot(phi_z, c1, label='J(w) if t=1')

c0 = [cost_0(x) for x in z]
plt.plot(phi_z, c0, linestyle='--', label='J(w) if t=0')

plt.ylim(0.0, 5.1)
plt.xlim([0, 1])
plt.xlabel('$\phi$(z)')
plt.ylabel('J(w)')
plt.legend(loc='best')
plt.tight_layout()
# plt.savefig('./figures/log_cost.png', dpi=300)
plt.show()
```

### Relationship to cross entropy

### Optimizing for logistic regression

From
$$
\frac{\partial \log\left(x\right)}{\partial x} = \frac{1}{x}
$$
and chain rule:
$$
\frac{\partial f\left(y\left(x\right)\right)}{\partial x} = \frac{\partial f(y)}{\partial y} \frac{\partial y}{\partial x}
$$

We know
$$
J(\mathbf{w})
= \sum_i -t^{(i)}\log\left(\phi\left(z^{(i)}\right)\right) - \left(1 - t^{(i)}\right) \log\left(1 - \phi\left(z^{(i)}\right) \right)
$$

$$
\frac{\partial J(\mathbf{w})}{\partial \mathbf{w}} =
\sum_i \left(\frac{-t^{(i)}}{\phi\left(z^{(i)}\right)} + \frac{1- t^{(i)}}{1 - \phi\left(z^{(i)}\right)} \right) \frac{\partial \phi \left(z^{(i)}\right)}{\partial \mathbf{w}}
$$

For sigmoid
$
\frac{\partial \phi(z)}{\partial z} = \phi(z)\left(1-\phi(z)\right)
$

Thus
$$
\begin{align}
\delta J =
\frac{\partial J(\mathbf{w})}{\partial \mathbf{w}} &=
\sum_i \left(\frac{-t^{(i)}}{\phi\left(z^{(i)}\right)} + \frac{1- t^{(i)}}{1 - \phi\left(z^{(i)}\right)} \right)
\phi\left(z^{(i)}\right)\left(1 - \phi\left(z^{(i)}\right) \right)
\frac{\partial z^{(i)}}{\partial \mathbf{w}} \\
&=
\sum_i \left( -t^{(i)}\left(1 - \phi\left(z^{(i)}\right)\right) + \left(1-t^{(i)}\right)\phi\left(z^{(i)}\right) \right) \mathbf{x}^{(i)} \\
&=
\sum_i \left( -t^{(i)} + \phi\left( z^{(i)} \right) \right) \mathbf{x}^{(i)}
\end{align}
$$

For gradient descent
$$
\begin{align}
\delta \mathbf{w} &= -\eta \delta J = \eta \sum_i \left( t^{(i)} - \phi\left( z^{(i)} \right) \right) \mathbf{x}^{(i)} \\
\mathbf{w} & \leftarrow \mathbf{w} + \delta \mathbf{w}
\end{align}
$$
as related to what we did for optimizing in chapter 2.

### Handling multiple classes

So far we have discussed only binary classifiers for 2 classes.

How about $K > 2$ classes, e.g. $K=3$ for the Iris dataset?

#### Multiple binary classifiers

##### One versus one

Build $\frac{K(K-1)}{2}$ classifiers,
each separating a different pair of classes $C_i$ and $C_j$

The green region is ambiguous: $C_1$, $C_2$, $C_3$

##### One versus rest (aka one versus all)

Build $K$ binary classifiers,
each separating class $C_k$ from the rest

The green region is ambiguous: $C_1$, $C_2$

#### Ambiguity

Both one-versus-one and one-versus-all have ambiguous regions and may incur more complexity/computation.

Ambiguity can be resolved via tie breakers, e.g.:
* activation values
* majority voting
* more details: http://scikit-learn.org/stable/modules/multiclass.html

#### One multi-class classifier

Multiple activation functions $\phi_k, k=1, 2, ... K$ each with different parameters
$$
\phi_k\left(\mathbf{x}\right) = \phi\left(\mathbf{x}, \mathbf{w_k}\right) = \phi\left(\mathbf{w}_k^T \mathbf{x} \right)
$$

We can then choose the class based on maximum activation:
$$
y = argmax_k \; \phi_k\left( \mathbf{x} \right)
$$

#### Can also apply the above for multiple binary classifiers

Caveat
* need to assume the individual classifiers have compatible activations
* https://en.wikipedia.org/wiki/Multiclass_classification

Binary logistic regression:

$$
J(\mathbf{w})
= \sum_{i \in samples} -t^{(i)}\log\left(\phi\left(z^{(i)}\right)\right) - \left(1 - t^{(i)}\right) \log\left(1 - \phi\left(z^{(i)}\right) \right)
$$

Multi-class logistic regression:
$$
J(\mathbf{w})
= \sum_{i \in samples} \sum_{j \in classes} -t^{(i, j)}\log\left(\phi\left(z^{(i, j)}\right)\right) - \left(1 - t^{(i, j)}\right) \log\left(1 - \phi\left(z^{(i, j)}\right) \right)
$$

For $\phi \geq 0$, we can normalize for probabilistic interpretation:
$$
P\left(k \; | \; \mathbf{x} ; \{\mathbf{w}_k\} \right) =
\frac{\phi_k\left(\mathbf{x}\right)}{\sum_{m=1}^K \phi_m\left(\mathbf{x}\right) }
$$

Or use softmax (normalized exponential) for any activation $\phi$:
$$
P\left(k \; | \; \mathbf{x} ; \{\mathbf{w}_k\} \right) =
\frac{e^{\phi_k\left(\mathbf{x}\right)}}{\sum_{m=1}^K e^{\phi_m\left(\mathbf{x}\right)} }
$$

For example, if $\phi(z) = z$:
$$
P\left(k \; | \; \mathbf{x} ; \{\mathbf{w}_k\} \right) =
\frac{e^{\mathbf{w}_k^T\mathbf{x}}}{\sum_{m=1}^K e^{\mathbf{w}_m^T \mathbf{x}} }
$$

For training, the model can be optimized via gradient descent.

The likelihood function (to maximize):
$$
L(\mathbf{w})
= P(t \; | \; \mathbf{x}; \mathbf{w})
= \prod_i P\left( t^{(i)} \; | \; \mathbf{x}^{(i)} ; \mathbf{w} \right)
$$

The loss function (to minimize):
$$
J(\mathbf{w})
= -\log{L(\mathbf{w})}
= -\sum_i \log{P\left( t^{(i)} \; | \; \mathbf{x}^{(i)} ; \mathbf{w} \right)}
$$

### Training a logistic regression model with scikit-learn

The code is quite simple.

```python
from sklearn.linear_model import LogisticRegression

lr = LogisticRegression(C=1000.0, random_state=0)
lr.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=lr, test_idx=test_idx,
                      xlabel = 'petal length [standardized]',
                      ylabel='petal width [standardized]')
```

```python
if Version(sklearn_version) < '0.17':
    print(lr.predict_proba(X_test_std[0, :]))
else:
    print(lr.predict_proba(X_test_std[0, :].reshape(1, -1)))

```

    [[  2.05743774e-11   6.31620264e-02   9.36837974e-01]]

### Tackling overfitting via regularization

Recall our general representation of our modeling objective:
$$\Phi(\mathbf{X}, \mathbf{T}, \Theta) = L\left(\mathbf{X}, \mathbf{T}, \mathbf{Y}=f(\mathbf{X}, \Theta)\right) + P(\Theta)$$
* $L$ - loss/objective for data fitting
* $P$ - regularization to favor simple model

Need to balance between accuracy/bias (L) and complexity/variance (P)

* If the model is too simple, it might be inaccurate (high bias)
* If the model is too complex, it might over-fit and over-sensitive to training data (high variance)

A well-trained model should
* fit the training data well (low bias)
* remain stable with different training data for good generalization (to unseen future data; low variance)

The following illustrates bias and variance for a potentially non-linear model

$L_2$ norm is a common form for regularization, e.g.
$
P = \lambda ||\mathbf{w}||^2
$
for the linear weights $\mathbf{w}$

$\lambda$ is a parameter to weigh between bias and variance

$C = \frac{1}{\lambda}$ for scikit-learn

```python
weights, params = [], []
for c in np.arange(-5, 5):
    lr = LogisticRegression(C=10**c, random_state=0)
    lr.fit(X_train_std, y_train)
    # coef_ has shape (n_classes, n_features)
    # we visualize only class 1
    weights.append(lr.coef_[1])
    params.append(10**c)

weights = np.array(weights)
plt.plot(params, weights[:, 0],
         label='petal length')
plt.plot(params, weights[:, 1], linestyle='--',
         label='petal width')
plt.ylabel('weight coefficient')
plt.xlabel('C')
plt.legend(loc='upper left')
plt.xscale('log')
# plt.savefig('./figures/regression_path.png', dpi=300)
plt.show()
```

# Reading
* PML Chapter 3

# Maximum margin classification with support vector machines

Another popular type of machine learning algorithm
* basic version for linear classification
* kernel version for non-linear classification

Linear classification

* decision boundary
$
\mathbf{w}^T \mathbf{x}
\begin{cases}
\geq 0 \; class +1 \\
< 0 \; class -1
\end{cases}
$
* similar to perceptron
* based on different criteria

Perceptron
* minimize misclassification error
* more sensitive to outliers
* incremental learning (via SGD)

SVM
* maximize margins to nearest samples (called support vectors)
* more robust against outliers
* batch learning

## Maximum margin intuition

Maximize the margins of support vectors to the decision plane $\rightarrow$ more robust classification for future samples (that may lie close to the decision plane)

Let us start with the simple case of two classes with labels +1 and -1.
(We choose this particular combination of labeling for numerical simplicity, as follows.)
Let the training dataset be $\{\mathbf{x}^{(i)}, y^{(i)}\}$, $i=1$ to $N$.

The goal is to find hyper-plane parameters $\mathbf{w}$ and $w_0$ so that
$$y^{(i)} \left( \mathbf{w}^T\mathbf{x}^{(i)} + w_0\right) \geq 1, \; \forall i$$.

Note that $y^{(i)} = \pm1$ above.

 We use t or y for target labels depending on the context

 We separate out $w_0$ from the rest of
$
\mathbf{w} =
\begin{bmatrix}
w_1 \\
w_2 \\
\vdots \\
w_n
\end{bmatrix}
$ for math derivation below

### Geometry perspective
For the purpose of optimization, we can cast the problem as maximize $\rho$ for:
$$\frac{y^{(i)} \left( \mathbf{w}^T\mathbf{x}^{(i)} + w_0\right)}{||\mathbf{w}||} \geq \rho, \; \forall i$$
; note that the left-hand side can be interpreted as the distance from $\mathbf{x}^{(i)}$ to the hyper-plane.

### Scaling
Note that the above equation remains invariant if we multiply $||\mathbf{w}||$ and $w_0$ by any non-zero scalar.

To eliminate this ambiguity, we can fix $\rho ||\mathbf{w}|| = 1$ and minimize $||\mathbf{w}||$, i.e.:
min $\frac{1}{2} ||\mathbf{w}||^2$ subject to $y^{(i)}\left( \mathbf{w}^T \mathbf{x}^{(i)} + w_0\right) \geq 1, \; \forall i$

### Optimization
We can use Lagrangian multipliers $\alpha^{(i)}$ for this constrained optimization problem:
$$
\begin{align}
L(\mathbf{w}, w_0, \alpha)
&=
\frac{1}{2} ||\mathbf{w}||^2 - \sum_i \alpha^{(i)} \left( y^{(i)} \left( \mathbf{w}^T \mathbf{x}^{(i)} + w_0\right) -1 \right)
\\
&=
\frac{1}{2} ||\mathbf{w}||^2 - \sum_i \alpha^{(i)} y^{(i)} \left( \mathbf{w}^T \mathbf{x}^{(i)} + w_0\right) + \sum_i \alpha^{(i)}
\end{align}
$$

With some calculus/algebraic manipulations:
$$\frac{\partial L}{\partial \mathbf{w}} = 0 \Rightarrow \mathbf{w} = \sum_i \alpha^{(i)} y^{(i)} \mathbf{x}^{(i)}$$
$$\frac{\partial L}{\partial w_0} = 0 \Rightarrow \sum_i \alpha^{(i)} y^{(i)} = 0$$

Plug the above two into $L$ above, we have:
$$
\begin{align}
L(\mathbf{w}, w_0, \alpha) &= \frac{1}{2} \mathbf{w}^T \mathbf{w} - \mathbf{w}^T \sum_i \alpha^{(i)}y^{(i)}\mathbf{x}^{(i)} - w_0 \sum_i \alpha^{(i)} y^{(i)} + \sum_i \alpha^{(i)} \\
&= -\frac{1}{2} \mathbf{w}^T \mathbf{w} + \sum_i \alpha^{(i)} \\
&= -\frac{1}{2} \sum_i \sum_j \alpha^{(i)} \alpha^{(j)} y^{(i)} y^{(j)} \left( \mathbf{x}^{(i)}\right)^T \mathbf{x}^{(j)} + \sum_i \alpha^{(i)}
\end{align}
$$
, which can be maximized, via quadratic optimization with $\alpha^{(i)}$ only, subject to the constraints: $\sum_i \alpha^{(i)} y^{(i)} = 0$ and $\alpha^{(i)} \geq 0, \; \forall i$

Note that $y^{(i)} = \pm 1$.

Once we solve $\{ \alpha^{(i)} \}$ we will see that most of them are $0$ with a few $> 0$.

The $>0$ ones correspond to lie on the decision boundaries and thus called support vectors:
$$y^{(i)} \left( \mathbf{w}^T \mathbf{x}^{(i)} + w_0\right) = 1$$
from which we can calculate $w_0$.

## Dealing with the nonlinearly separable case using slack variables

Soft margin classification

Some datasets are not linearly separable

Avoid thin margins for linearly separable cases
* bias variance tradeoff

For datasets that are not linearly separable, we can introduce slack variables $\{\xi^{(i)}\}$ as follows:
$$y^{(i)} \left( \mathbf{w}^T \mathbf{x}^{(i)} + w_0\right) \geq 1 - \xi^{(i)}, \; \forall i$$

* If $\xi^{(i)} = 0$, it is just like the original case without slack variables.
* If $0 < \xi^{(i)} <1$, $\mathbf{x}^{(i)}$ is correctly classified but lies within the margin.
* If $\xi^{(i)} \geq 1$, $\mathbf{x}^{(i)}$ is mis-classified.

For optimization, the goal is to minimize
$$\frac{1}{2} ||\mathbf{w}||^2 + C \sum_i \xi^{(i)}$$
, where $C$ is the strength of the penalty factor (like in regularization).

Using the Lagrangian multipliers $\{\alpha^{(i)}, \mu^{(i)} \}$ with constraints we have:
$$L = \frac{1}{2} ||\mathbf{w}||^2 + C \sum_i \xi^{(i)} - \sum_i \alpha^{(i)} \left( y^{(i)} \left( \mathbf{w}^{(i)}\mathbf{x}^{(i)} + w_0\right) - 1 + \xi^{(i)}\right) - \sum_i \mu^{(i)} \xi^{(i)}$$
, which can be solved via a similar process as in the original case without slack variables.

## Coding with SVM via scikit learn is simple

```python
from sklearn.svm import SVC

svm = SVC(kernel='linear', C=1.0, random_state=0)
svm.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=svm, test_idx=test_idx,
                      xlabel='petal length [standardized]', ylabel='petal width [standardized]')
```

## Alternative implementations in scikit-learn

# Solving non-linear problems using a kernel SVM

SVM can be extended for non-linear classification

This is called kernel SVM
* will explain what kernel means
* and introduce kernel tricks :-)

##  Intuition

The following 2D circularly distributed data sets are not linearly separable.

However, we can elevate them to a higher dimensional space for linear separable:
$
\phi(x_1, x_2) = (x_1, x_2, x_1^2 + x_2^2)
$
,
where $\phi$ is the mapping function.

## Animation visualization

https://youtu.be/3liCbRZPrZA

https://youtu.be/9NrALgHFwTo

```python
from IPython.display import YouTubeVideo
YouTubeVideo('3liCbRZPrZA')
```

```python
YouTubeVideo('9NrALgHFwTo')
```

## Using the kernel trick to find separating hyperplanes in higher dimensional space

For datasets that are not linearly separable, we can map them into a higher dimensional space and make them linearly separable.
Let $\phi$ be this mapping:
$\mathbf{z} = \phi(\mathbf{x})$

And we perform the linear decision in the $\mathbf{z}$ instead of the original $\mathbf{x}$ space:
$$y^{(i)} \left( \mathbf{w}^{(i)} \mathbf{z}^{(i)} + w_0\right) \geq 1 - \xi^{(i)}$$

Following similar Lagrangian multiplier optimization as above, we eventually want to optimize:
$$
\begin{align}
L &= \frac{1}{2} \sum_i \sum_j \alpha^{(i)} \alpha^{(j)} y^{(i)} y^{(j)} \left(\mathbf{z}^{(i)}\right)^T \mathbf{z}^{(j)} + \sum_i \alpha^{(i)} \\
&= \frac{1}{2} \sum_i \sum_j \alpha^{(i)} \alpha^{(j)} y^{(i)} y^{(j)} \phi\left(\mathbf{x}^{(i)}\right)^T \phi\left(\mathbf{x}^{(j)}\right) + \sum_i \alpha^{(i)}
\end{align}
$$

The key idea behind kernel trick, and kernel machines in general, is to represent the high dimensional dot product by a kernel function:
$$K\left(\mathbf{x}^{(i)}, \mathbf{x}^{(j)}\right) = \phi\left(\mathbf{x}^{(i)}\right)^T \phi\left(\mathbf{x}^{(j)}\right)$$

Intuitively, the data points become more likely to be linearly separable in a higher dimensional space.

## Kernel trick for evaluation

Recall from part of our derivation above:
$$
\frac{\partial L}{\partial \mathbf{w}} = 0 \Rightarrow \mathbf{w} = \sum_i \alpha^{(i)} y^{(i)} \mathbf{z}^{(i)}
$$

Which allows us to compute the discriminant via kernel trick as well:
$$
\begin{align}
\mathbf{w}^T \mathbf{z}
&=
\sum_i \alpha^{(i)} y^{(i)} \left(\mathbf{z}^{(i)}\right)^T \mathbf{z}
\\
&=
\sum_i \alpha^{(i)} y^{(i)} \phi\left(\mathbf{x}^{(i)}\right)^T \phi(\mathbf{x})
\\
&=
\sum_i \alpha^{(i)} y^{(i)} K\left(\mathbf{x}^{(i)}, \mathbf{x}\right)
\end{align}
$$

## Non-linear classification example

 x  y xor(x, y)

  0  0  0

  0  1  1

  1  0  1

  1  1  0

Xor is not linearly separable
* math proof left as exercise

Random point sets classified via XOR based on the signs of 2D coordinates:

```python
import matplotlib.pyplot as plt
import numpy as np

np.random.seed(0)
X_xor = np.random.randn(200, 2)
y_xor = np.logical_xor(X_xor[:, 0] > 0,
                       X_xor[:, 1] > 0)
y_xor = np.where(y_xor, 1, -1)

plt.scatter(X_xor[y_xor == 1, 0],
            X_xor[y_xor == 1, 1],
            c='b', marker='x',
            label='1')
plt.scatter(X_xor[y_xor == -1, 0],
            X_xor[y_xor == -1, 1],
            c='r',
            marker='s',
            label='-1')

plt.xlim([-3, 3])
plt.ylim([-3, 3])
plt.legend(loc='best')
plt.tight_layout()
# plt.savefig('./figures/xor.png', dpi=300)
plt.show()
```

This is the classification result using a rbf (radial basis function) kernel

Notice the non-linear decision boundaries

```python
svm = SVC(kernel='rbf', random_state=0, gamma=0.10, C=10.0)
svm.fit(X_xor, y_xor)
plot_decision_regions(X_xor, y_xor,
                      classifier=svm)
```

## Types of kernels

A variety of kernel functions can be used.
The only requirement is that the kernel function behaves like inner product;
larger $K(\mathbf{x}, \mathbf{y})$ for more similar $\mathbf{x}$ and $\mathbf{y}$

### Linear
$
K\left(\mathbf{x}, \mathbf{y}\right) = \mathbf{x}^T \mathbf{y}
$

### Polynomials of degree $q$
$
K\left(\mathbf{x}, \mathbf{y}\right) =
(\mathbf{x}^T\mathbf{y} + 1)^q
$

Example for $d=2$ and $q=2$
$$
\begin{align}
K\left(\mathbf{x}, \mathbf{y}\right) &= \left( x_1y_1 + x_2y_2 + 1 \right)^2 \\
&= 1 + 2x_1y_1 + 2x_2y_2 + 2x_1x_2y_1y_2 + x_1^2y_1^2 + x_2^2y_2^2
\end{align}
$$
, which corresponds to the following kernel function:
$$
\phi(x, y) = \left[1, \sqrt{2}x, \sqrt{2}y, \sqrt{2}xy, x^2, y^2 \right]^T
$$

### Radial basis function (RBF)

Scalar variance:
$$
K\left(\mathbf{x}, \mathbf{y} \right) = e^{-\frac{\left|\mathbf{x} - \mathbf{y}\right|^2}{2s^2}}
$$

General co-variance matrix:
$$
K\left(\mathbf{x}, \mathbf{y} \right) = e^{-\frac{1}{2} \left(\mathbf{x}-\mathbf{y}\right)^T \mathbf{S}^{-1} \left(\mathbf{x} - \mathbf{y}\right)}
$$

General distance function $D\left(\mathbf{x}, \mathbf{y}\right)$:
$$
K\left(\mathbf{x}, \mathbf{y} \right) = e^{-\frac{D\left(\mathbf{x}, \mathbf{y} \right)}{2s^2}}
$$

RBF essentially projects to an infinite dimensional space.

### Sigmoid
$$
K\left(\mathbf{x}, \mathbf{y} \right) = \tanh\left(2\mathbf{x}^T\mathbf{y} + 1\right)
$$

## Kernel SVM for the Iris dataset

Let's apply RBF kernel

The kernel width is controlled by a gamma $\gamma$ parameter for kernel influence
$
K\left(\mathbf{x}, \mathbf{y} \right) =
e^{-\gamma D\left(\mathbf{x}, \mathbf{y} \right)}
$
and $C$ for regularization

```python
from sklearn.svm import SVC

svm = SVC(kernel='rbf', random_state=0, gamma=0.2, C=1.0)
svm.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=svm, test_idx=test_idx,
                      xlabel = 'petal length [standardized]',
                      ylabel = 'petal width [standardized]')
```

```python
svm = SVC(kernel='rbf', random_state=0, gamma=100.0, C=1.0)
svm.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=svm, test_idx=test_idx,
                      xlabel = 'petal length [standardized]',
                      ylabel='petal width [standardized]')
```

# Reading
* PML Chapter 3
* IML Chapter 13-1 to 13.7
* [The kernel trick](http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html)

# Decision tree learning

Machine learning can be like black box/magic; the model/method works after tuning parameters and such, but how and why?

Decision tree shows you how it makes decision, e.g. classification.

## Example decision tree

* analogous to flow charts for designing algorithms
* every internal node can be based on some if-statement
* automatically learned from data, not manually programmed by human

## Decision tree learning

1. Start with a single node that contains all data
 1. Select a node and split it via some criterion to optimize some objective, usually information/impurity $I$
 2. Repeat until convergence:
  good enough classification measured by $I$;
  complex enough model (overfitting);
2. Each leaf node belongs to one class
 * Multiple leaf nodes can be of the same class
 * Each leaf node can have misclassified samples - majority voting

* usually split along one dimension/feature
* a finite number of choices from the boundaries of sample classes

## Maximizing information gain - getting the most bang for the buck

$I(D)$ information/impurity for a tree node with dataset $D$

Maximize information gain $IG$ for splitting each (parent) node $D_p$ into $m$ child nodes $j$:
$$
IG = I(D_p) - \sum_{j=1}^m \frac{N_j}{N_p} I(D_j)
$$
Usually $m=2$ for simplicity (binary split)

Commonly used impurity measures $I$

$p(i|t)$ - probability/proportion of dataset in node $t$ belongs to class $i$

### Entropy

$$
I_H(t) = - \sum_{i=1}^c p(i|t) \log_2 p(i|t)
$$

* $0$ if all samples belong to the same class
* $1$ if uniform distribution
$
0.5 = p(0|t) = p(1|t)
$

Entropy (information theory)

Random variable $X$ with probability mass/density function $P(X)$

Information content
$
I(X) = -\log_b\left(P(X)\right)
$

Entropy is the expectation of information
$$
H(X) = E(I(X)) = E(-\log_b(P(X)))
$$

log base $b$ can be $2$, $e$, $10$

Continuous $X$:
$$
H(X) = \int P(x) I(x) \; dx = -\int P(x) \log_b P(x) \;dx
$$
Discrete $X$:
$$
H(X) = \sum_i P(x_i) I(x_i) = -\sum_i P(x_i) \log_b P(x_i)
$$

$-\log_b P(x)$ - number of bits needed to represent $P(x)$
* the rarer the event $\rightarrow$ the less $P(x)$ $\rightarrow$ the more bits

### Gini index

Minimize expected value of misclassification

$$
I_G(t) = \sum_{i=1}^c p(i|t) \left( 1 - p(i|t) \right) = 1 - \sum_{i=1}^c p(i|t)^2
$$

* $p(i|t)$ - probability of class $i$
* $1-p(i|t)$ - probability of misclassification, i.e. $t$ is not class $i$

Similar to entropy
* expected value of information: $-\log_2 p(i|t)$
* information and mis-classification probability: both larger for lower $p(i|t)$

### Classification error

$$
I_e(t) = 1 - \max_i p(i|t)
$$

$
argmax_i \; p(i|t)
$
as the class label for node $t$

## Compare different information measures

Entropy and Gini are probabilisitic
* not assuming the label of the node (decided later after more splitting)

Classification error is deterministic
* assumes the majority class would be the label

Entropy and Gini index are similar, and tend to behave better than classification error
* curves below via a 2-class case
* example in the PML textbook

```python
import matplotlib.pyplot as plt
import numpy as np

def gini(p):
    return p * (1 - p) + (1 - p) * (1 - (1 - p))

def entropy(p):
    return - p * np.log2(p) - (1 - p) * np.log2((1 - p))

def error(p):
    return 1 - np.max([p, 1 - p])

x = np.arange(0.0, 1.0, 0.01)

ent = [entropy(p) if p != 0 else None for p in x]
sc_ent = [e * 0.5 if e else None for e in ent]
err = [error(i) for i in x]

fig = plt.figure()
ax = plt.subplot(111)
for i, lab, ls, c, in zip([ent, sc_ent, gini(x), err],
                          ['Entropy', 'Entropy (scaled)',
                           'Gini Impurity', 'Misclassification Error'],
                          ['-', '-', '--', '-.'],
                          ['black', 'lightgray', 'red', 'green', 'cyan']):
    line = ax.plot(x, i, label=lab, linestyle=ls, lw=2, color=c)

ax.legend(loc='upper center', bbox_to_anchor=(0.5, 1.15),
          ncol=3, fancybox=True, shadow=False)

ax.axhline(y=0.5, linewidth=1, color='k', linestyle='--')
ax.axhline(y=1.0, linewidth=1, color='k', linestyle='--')
plt.ylim([0, 1.1])
plt.xlabel('p(i=1)')
plt.ylabel('Impurity Index')
plt.tight_layout()
#plt.savefig('./figures/impurity.png', dpi=300, bbox_inches='tight')
plt.show()
```

## Building a decision tree

A finite number of choices for split

Split only alone boundaries of different classes

Exactly where? Maxmize margins

```python
from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(criterion='entropy', max_depth=3, random_state=0)
tree.fit(X_train, y_train)

X_combined = np.vstack((X_train, X_test))
y_combined = np.hstack((y_train, y_test))
plot_decision_regions(X_combined, y_combined,
                      classifier=tree, test_idx=test_idx,
                      xlabel='petal length [cm]',ylabel='petal width [cm]')
```

## Visualize the decision tree

```python
from sklearn.tree import export_graphviz

export_graphviz(tree,
                out_file='tree.dot',
                feature_names=['petal length', 'petal width'])
```

Install [Graphviz](http://www.graphviz.org/)

dot -Tsvg tree.dot -o tree.svg

**Note**

If you have scikit-learn 0.18 and pydotplus installed (e.g., you can install it via `pip install pydotplus`), you can also show the decision tree directly without creating a separate dot file as shown below. Also note that `sklearn 0.18` offers a few additional options to make the decision tree visually more appealing.

```python
#import pydotplus

from IPython.display import Image
from IPython.display import display

if False and Version(sklearn_version) >= '0.18':

    try:

        import pydotplus

        dot_data = export_graphviz(
        tree,
        out_file=None,
        # the parameters below are new in sklearn 0.18
        feature_names=['petal length', 'petal width'],
        class_names=['setosa', 'versicolor', 'virginica'],
        filled=True,
        rounded=True)

        graph = pydotplus.graph_from_dot_data(dot_data)
        display(Image(graph.create_png()))

    except ImportError:
        print('pydotplus is not installed.')
```

# Decisions trees and SVM

* SVM considers only margins to nearest samples to the decision boundary
* Decision tree considers all samples

Case studies

## Pruning a decision tree

Split until all leaf nodes are pure?

* not always a good idea due to potential over-fitting

Simplify the tree via pruning

Pre-pruning
* stop splitting a node if the contained data size is below some threshold (e.g. 5% of all data)

Post-pruning
* build a tree first, and remove excessive branches
* reserve a pruning subset separate from the training data
* for each sub-tree (top-down or bottom-up), replace it with a leaf node labeled with the majority vote if not worsen performance for the pruning subset

Pre-pruning is simpler, post-pruning works better

## Combining weak to strong learners via random forests

Forest = collection of trees

An example of ensemble learning (more about this later)
* combine multiple weak learners to build a strong learner
* better generalization, less overfitting

Less interpretable than a single tree

### Random forest algorithm

Decide how many trees to build

To train each tree:
* Draw a random subset of samples (e.g. random sample with replacement of all samples)
* Split each node via a random subset of features (e.g. $d = \sqrt{m}$ of the original dimensionality)

(randomization is a key)

Majority vote from all trees

### Code example

```python
from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(criterion='entropy',
                                n_estimators=10,
                                random_state=1,
                                n_jobs=2)
forest.fit(X_train, y_train)

plot_decision_regions(X_combined, y_combined,
                      classifier=forest, test_idx=test_idx,
                      xlabel = 'petal length [cm]', ylabel = 'petal width [cm]')
```

# Reading
* PML Chapter 3
* IML Chapter 9

# Parametric versus non-parametric models

* (fixed) number of parameters trained and retained
* amount of data retained
* trade-off between training and evaluation time

## Example

Linear classifiers (SVM, perceptron)
* parameters: $\mathbf{w}$
* data throw away after training
* extreme end of parametric

Kernel SVM
* depends on the type of kernel used (exercise)

Decision tree
* parameters: decision boundaries at all nodes
* number of parameters vary depending on the training data
* data throw away after training
* less parametric than SVM

Personal take:
Parametric versus non-parametric is more of a continuous spectrum than a binary decision.
Many algorithms lie somewhere in between.

# K-nearest neighbors - a lazy learning algorithm

KNN keeps all data and has no trained parameters
* extreme end of non-parametric

How it works:
* Choose the number $k$ of neighbors and a distance measure
* For each sample to classify, find the $k$ nearest neighbors in the dataset
* Assign class label via majority vote

$k$ is a hyper-parameter (picked by human), not a (ordinary) parameter (trained from data by machines)

Pro:
* zero training time
* very simple

Con:
* need to keep all data
* evaluation time linearly proportional to data size (acceleration possible though, e.g. kd-tree)
* vulnerable to curse of dimensionality

## Practical usage

[Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance) of order $p$:
$
d(\mathbf{x}, \mathbf{y}) = \sqrt[p]{\sum_k |\mathbf{x}_k - \mathbf{y}_k|^p}
$
* $p = 2$, Euclidean distance
* $p = 1$, Manhattan distance

Number of neighbors $k$ trade-off between bias and variance
* too small $k$ - low bias, high variance

```python
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5, p=2, metric='minkowski')
knn.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=knn, test_idx=test_idx,
                      xlabel='petal length [standardized]', ylabel='petal width [standardized]')
```

```python
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=1, p=2, metric='minkowski')
knn.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=knn, test_idx=test_idx,
                      xlabel='petal length [standardized]', ylabel='petal width [standardized]')
```

Too small k can cause overfitting (high variance).

```python
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=100, p=2, metric='minkowski')
knn.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=knn, test_idx=test_idx,
                      xlabel='petal length [standardized]', ylabel='petal width [standardized]')
```

Too large k can cause under-fitting (high bias).

How about using different $p$ values for Minkowski distance?

```python
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5, p=1, metric='minkowski')
knn.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=knn, test_idx=test_idx,
                      xlabel='petal length [standardized]', ylabel='petal width [standardized]')
```

```python
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5, p=10, metric='minkowski')
knn.fit(X_train_std, y_train)

plot_decision_regions(X_combined_std, y_combined,
                      classifier=knn, test_idx=test_idx,
                      xlabel='petal length [standardized]', ylabel='petal width [standardized]')
```

# Reading
* PML Chapter 3


# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.
    	There will be an option for Colab users and another for Jupyter (local) users.

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [4.66293670e-17 1.66533454e-17 1.85962357e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029261167605239e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 4.05e-10
    b1 relative error: 4.44e-08
    b2 relative error: 2.22e-08
    b3 relative error: 1.01e-10
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 6.96e-09
    gamma2 relative error: 1.96e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.29e-06
    W3 relative error: 1.11e-08
    b1 relative error: 2.78e-09
    b2 relative error: 2.22e-08
    b3 relative error: 2.10e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 4.23e-09
    gamma1 relative error: 6.27e-09
    gamma2 relative error: 5.28e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.314000; val_acc: 0.266000
    (Iteration 21 / 200) loss: 2.039365
    (Epoch 2 / 10) train acc: 0.389000; val_acc: 0.278000
    (Iteration 41 / 200) loss: 2.036704
    (Epoch 3 / 10) train acc: 0.502000; val_acc: 0.322000
    (Iteration 61 / 200) loss: 1.779836
    (Epoch 4 / 10) train acc: 0.527000; val_acc: 0.324000
    (Iteration 81 / 200) loss: 1.291448
    (Epoch 5 / 10) train acc: 0.572000; val_acc: 0.313000
    (Iteration 101 / 200) loss: 1.245071
    (Epoch 6 / 10) train acc: 0.677000; val_acc: 0.327000
    (Iteration 121 / 200) loss: 1.099809
    (Epoch 7 / 10) train acc: 0.696000; val_acc: 0.334000
    (Iteration 141 / 200) loss: 1.150739
    (Epoch 8 / 10) train acc: 0.725000; val_acc: 0.330000
    (Iteration 161 / 200) loss: 0.759309
    (Epoch 9 / 10) train acc: 0.753000; val_acc: 0.341000
    (Iteration 181 / 200) loss: 0.961651
    (Epoch 10 / 10) train acc: 0.759000; val_acc: 0.324000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662213
    (Epoch 5 / 10) train acc: 0.431000; val_acc: 0.298000
    (Iteration 101 / 200) loss: 1.703401
    (Epoch 6 / 10) train acc: 0.528000; val_acc: 0.347000
    (Iteration 121 / 200) loss: 1.570579
    (Epoch 7 / 10) train acc: 0.542000; val_acc: 0.316000
    (Iteration 141 / 200) loss: 1.417201
    (Epoch 8 / 10) train acc: 0.624000; val_acc: 0.330000
    (Iteration 161 / 200) loss: 1.059592
    (Epoch 9 / 10) train acc: 0.617000; val_acc: 0.328000
    (Iteration 181 / 200) loss: 0.959507
    (Epoch 10 / 10) train acc: 0.719000; val_acc: 0.338000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
When the scale of weight initialization is less than 10e-1, the model with batch normalization performs better than the another; however, the performance of the one with batch normalization go down as the weight scale continue to increase.

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
It is obvious that the performance of btach normalization-equipped model is positively related to the batch size, what is to say, the growth in batch size comes with the increase in accuracy.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
> The difference between LN and BN is that under layer normalization, all the hidden units in a layer share the same normalization terms $\mu$ and $\sigma$, but different training cases have different normalization terms. Unlike batch normalization, layer normaliztion does not impose any constraint on the size of a mini-batch and it can be used in the pure online regime with batch size 1.

Batch Normalization: 3, 4 \[BN processes data with info from all images\]<br>
Layer Normalization: 1, 2 \[LN only processes images respectively with info from corresponding image\]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.433615657860454e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
2, if the dimension of features is too small, the mean and deviation can be unstable and random, causing bad performance.

```python

```


```python
import numpy as np
from pydgrid.pydgrid import grid
from pydgrid.plot_bokeh import plot_results
import sympy as sym
import pydae.build as db
from pydae.grid_urisi import unb_ri_si
from pydae.models import dcgrid2dae,dcrail2dae
from pydae.tools import get_v,get_i,get_s,save,load
from pydae.train_tools import trains_update,trains2params
import json
import matplotlib.pyplot as plt
```

## Grid building

```python
grid_data = {
        "buses":[
                 {"bus": "B1",  "pos_x":   0, "pos_y":  0, "units": "m", "U_kV":20.0},
                 {"bus": "B2",  "pos_x":  50, "pos_y":  0, "units": "m", "U_kV":20.0},
                 {"bus": "B3",  "pos_x": 100, "pos_y":  0, "units": "m", "U_kV":20.0},
                 {"bus": "B4",  "pos_x": 150, "pos_y":  0, "units": "m", "U_kV":20.0},
                 {"bus": "B5",  "pos_x": 200, "pos_y":  0, "units": "m", "U_kV":20.0},
                 {"bus": "B6",  "pos_x": 250, "pos_y":  0, "units": "m", "U_kV":20.0},
                 {"bus": "B7",  "pos_x": 350, "pos_y":  0, "units": "m", "U_kV":20.0},
                 {"bus": "B2lv",  "pos_x": 50, "pos_y":  -10, "units": "m", "U_kV":1.5},
                 {"bus": "B3lv",  "pos_x": 100, "pos_y": -10, "units": "m", "U_kV":1.5},
                 {"bus": "B4lv",  "pos_x": 150, "pos_y": -10, "units": "m", "U_kV":1.5},
                 {"bus": "B5lv",  "pos_x": 200, "pos_y": -10, "units": "m", "U_kV":1.5},
                 {"bus": "B6lv",  "pos_x": 250, "pos_y": -10, "units": "m", "U_kV":1.5},
                ],
        "grid_formers":[
                        {"bus": "B1",
                        "bus_nodes": [1, 2, 3], "deg": [0, -120, -240],
                        "kV": [20.0/np.sqrt(3), 20.0/np.sqrt(3), 20.0/np.sqrt(3)]},
                        {"bus": "B7",
                        "bus_nodes": [1, 2, 3], "deg": [0, -120, -240],
                        "kV": [20.0/np.sqrt(3), 20.0/np.sqrt(3), 20.0/np.sqrt(3)]},
                       ],
        "lines":[
                 {"bus_j": "B1",  "bus_k": "B2",  "code": "mv_al_185", "m": 2600,'monitor':True},
                 {"bus_j": "B2",  "bus_k": "B3",  "code": "mv_al_185", "m": 2600,'monitor':True},
                 {"bus_j": "B3",  "bus_k": "B4",  "code": "mv_al_185", "m": 2600,'monitor':True},
                # {"bus_j": "B4",  "bus_k": "B5",  "code": "mv_al_185", "m": 2600,'monitor':True},
                 {"bus_j": "B5",  "bus_k": "B6",  "code": "mv_al_185", "m": 2600,'monitor':True},
                 {"bus_j": "B6",  "bus_k": "B7",  "code": "mv_al_185", "m": 2600,'monitor':True},
                ],
"transformers":[
        {"bus_j": "B2",  "bus_k": "B2lv",
        "S_n_kVA": 2200.0, "U_j_kV":20.0, "U_k_kV":1.5,
        "R_cc_pu": 0.01, "X_cc_pu":0.04, "connection": "Dyn11",
  "conductors_j": 3, "conductors_k": 4, "monitor":True},
        {"bus_j": "B3",  "bus_k": "B3lv",
        "S_n_kVA": 2200.0, "U_j_kV":20.0, "U_k_kV":1.5,
        "R_cc_pu": 0.01, "X_cc_pu":0.04, "connection": "Dyn11",
  "conductors_j": 3, "conductors_k": 4, "monitor":True},
        {"bus_j": "B4",  "bus_k": "B4lv",
        "S_n_kVA": 2200.0, "U_j_kV":20.0, "U_k_kV":1.5,
        "R_cc_pu": 0.01, "X_cc_pu":0.04, "connection": "Dyn11",
  "conductors_j": 3, "conductors_k": 4, "monitor":True},
        {"bus_j": "B5",  "bus_k": "B5lv",
        "S_n_kVA": 2200.0, "U_j_kV":20.0, "U_k_kV":1.5,
        "R_cc_pu": 0.01, "X_cc_pu":0.04, "connection": "Dyn11",
  "conductors_j": 3, "conductors_k": 4, "monitor":True},
        {"bus_j": "B6",  "bus_k": "B6lv",
        "S_n_kVA": 2200.0, "U_j_kV":20.0, "U_k_kV":1.5,
        "R_cc_pu": 0.01, "X_cc_pu":0.04, "connection": "Dyn11",
         "conductors_j": 3, "conductors_k": 4, "monitor":True}
        ],
        "loads":[
                 {"bus": "B2lv" , "kVA": [740.0,740.0,740.0], "pf":[0.9]*3,"type":"3P+N"},
                 {"bus": "B3lv" , "kVA": [740.0,740.0,740.0], "pf":[0.9]*3,"type":"3P+N"},
                 {"bus": "B4lv" , "kVA": [740.0,740.0,740.0], "pf":[0.9]*3,"type":"3P+N"},
                 {"bus": "B5lv" , "kVA": [740.0,740.0,740.0], "pf":[0.9]*3,"type":"3P+N"},
                 {"bus": "B6lv" , "kVA": [740.0,740.0,740.0], "pf":[0.9]*3,"type":"3P+N"}
                ],
        "shunts":[
                 {"bus": "B2lv" , "R": 0.001, "X": 0.0, "bus_nodes": [4,0]},
                 {"bus": "B3lv" , "R": 0.001, "X": 0.0, "bus_nodes": [4,0]},
                 {"bus": "B4lv" , "R": 0.001, "X": 0.0, "bus_nodes": [4,0]},
                 {"bus": "B5lv" , "R": 0.001, "X": 0.0, "bus_nodes": [4,0]},
                 {"bus": "B6lv" , "R": 0.001, "X": 0.0, "bus_nodes": [4,0]}
                 ],
        "line_codes":
            {
                "lv_cu_150":  {"Rph":0.167,"Xph":0.08, "Rn":0.167, "Xn": 0.08},
                "mv_al_150":   {"R1":0.262,  "X1":0.118, "C_1_muF":0.250 },
                "mv_al_185":   {"R1":0.209,  "X1":0.113, "C_1_muF":0.281 },
                "mv_al_240":   {"R1":0.161,  "X1":0.109, "C_1_muF":0.301 },
                "mv_al_300":   {"R1":0.128,  "X1":0.105, "C_1_muF":0.340 }
            }
       }
```

```python
grid_1 = grid()
grid_1.read(grid_data)  # Load data
grid_1.pf()  # solve power flow
p=plot_results(grid_1)

```

```python
grid_mv = unb_ri_si(grid_data)

params_dict  = grid_mv.dae['params']
f_list = grid_mv.dae['f']
x_list = grid_mv.dae['x']
g_list = grid_mv.dae['g']
y_list = grid_mv.dae['y']
u_dict = grid_mv.dae['u']

h_dict = grid_mv.dae['h_v_m_dict']

sys_dict = {'name':'grid_mv',
           'params_dict':params_dict,
           'f_list':f_list,
           'g_list':g_list,
           'x_list':x_list,
           'y_ini_list':y_list,
           'y_run_list':y_list,
           'u_run_dict':u_dict,
           'u_ini_dict':u_dict,
           'h_dict':h_dict
           }

db.system(sys_dict)
db.sys2num(sys_dict)

data = json.dumps(grid_mv.dae['xy_0_dict'], indent=4)
fobj = open("grid_mv_xy_0_dict.json","w")
fobj.write(data)
fobj.close()
```

    system without dynamic equations, adding dummy dynamic equation

```python
Y_ii = grid_1.Y_ii.toarray()
Y_vv = grid_1.Y_vv
Y_vi = grid_1.Y_vi
inv_Y_ii = np.linalg.inv(Y_ii)
N_nz_nodes = grid_1.params_pf[0].N_nz_nodes
N_v = grid_1.params_pf[0].N_nodes_v
nodes_list = grid_1.nodes
Y_primitive = grid_1.Y_primitive_sp.toarray()
A_conect = grid_1.A_sp.toarray()
node_sorter  = grid_1.node_sorter
N_v = grid_1.N_nodes_v

np.savez('matrices',Y_primitive=Y_primitive,A_conect=A_conect,nodes_list=nodes_list,
         node_sorter=node_sorter,N_v=N_v, Y_vv=Y_vv, Y_vi=Y_vi)

with open("grid_data.json", "w") as fobj:
    json.dump(grid_1.data, fobj, indent=4, sort_keys=True)

```

## Railway building

```python
data_input = load('trains_lv_5vsc_params.json')   # See at the end of the notebook for .json creation
```

```python
dcgrid_dae = dcgrid2dae(data_input)  # Creates the DC grid from data_input
dcrail2dae(data_input,dcgrid_dae)    # Modify the DC grid with the railway particularities
```

```python
rho = 1.72e-8    # copper conductivity
s = 300.0/1.0e6  # conductor section m^2
r_m = rho/s  # Ohm/m

sections = data_input['sections']
trains = data_input['trains']
trips = data_input['trips']

for trip in trips:    # converts json list to np.array
    trips[trip]['times'] = np.array(trips[trip]['times'])
    trips[trip]['positions'] = np.array(trips[trip]['positions'])
    trips[trip]['powers'] = np.array(trips[trip]['powers'])

t=60*2
train_positions_list,train_powers_list = trains_update(t,trains,trips)           # for the given t this creates a list
                                                                                # of train positions and their respective
                                                                                # powers
params_dict_trains,nodes_dict = trains2params(sections,train_positions_list,train_powers_list,r_m)   # for the given trains the topology in defined

```

```python
f_list = dcgrid_dae['f']
g_list = dcgrid_dae['g']
y_list = dcgrid_dae['y']
x_list = dcgrid_dae['x']
u_run_dict = {'Dv_r_13':0.0,'Dv_r_23':0.0,'Dv_r_33':0.0,'Dv_r_43':0.0,'Dv_r_53':0.0}
u_run_dict.update({'v_nom':3e3,'T_v':0.02,'K_r':0.05*3e3/500e3})
h_dict = dcgrid_dae['h_dict']

sys = {'name':f"railway_5vsc",
       'params_dict':params_dict_trains,
       'f_list':f_list,
       'g_list':g_list,
       'x_list':x_list,
       'y_ini_list':y_list,
       'y_run_list':y_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_run_dict,
       'h_dict':h_dict
      }

sys = db.system(sys)
db.sys2num(sys)
```

## Grid and railway interaction

```python
params_dict  = grid_mv.dae['params']
params_dict.update(params_dict_trains)
f_list = grid_mv.dae['f'] + dcgrid_dae['f']
x_list = grid_mv.dae['x'] + dcgrid_dae['x']
g_list = grid_mv.dae['g'] + dcgrid_dae['g']
y_list = grid_mv.dae['y'] + dcgrid_dae['y']
u_dict = grid_mv.dae['u']
u_dict.update({'Dv_r_13':0.0,'Dv_r_23':0.0,'Dv_r_33':0.0,'Dv_r_43':0.0,'Dv_r_53':0.0})
u_dict.update({'v_nom':3e3,'T_v':0.02,'K_r':0.05*3e3/500e3})
h_dict = dcgrid_dae['h_dict']

ac_dc_buses = [('B2lv','13'),('B3lv','23'),('B4lv','33'),('B5lv','43'),('B6lv','53')]

for bus_ac,bus_dc in ac_dc_buses:

    p_a,p_b,p_c = sym.symbols(f'p_{bus_ac}_a,p_{bus_ac}_b,p_{bus_ac}_c', real=True)
    i_dc,v_dc = sym.symbols(f'i_{bus_dc},v_{bus_dc}', real=True)
    i_a_r,i_a_i = sym.symbols(f'i_load_{bus_ac}_a_r,i_load_{bus_ac}_a_i', real=True)
    a,b,c = sym.symbols(f'a_{bus_ac},b_{bus_ac},c_{bus_ac}', real=True)

    i_rms = sym.sqrt(i_a_r**2+i_a_i**2+1)

    p_simple = a + b*i_rms + c*i_rms*i_rms

    p_vsc_loss = p_simple*6

    p_d = i_dc*v_dc

    p_dc = p_d + sym.Piecewise((p_vsc_loss, p_d < 1), (p_vsc_loss, p_d > 0),(0.0, True))
    p_ac = p_dc

    g_list += [p_a + p_ac/3]
    g_list += [p_b + p_ac/3]
    g_list += [p_c + p_ac/3]

    y_list += [p_a]
    y_list += [p_b]
    y_list += [p_c]

    u_dict.update({f'a_{bus_ac}':2.92075867})
    u_dict.update({f'b_{bus_ac}':0.44961794})
    u_dict.update({f'c_{bus_ac}':0.02717091})

    u_dict.pop(str(p_a))
    u_dict.pop(str(p_b))
    u_dict.pop(str(p_c))
```

```python

sys = {'name':f"grid_railway_5vsc",
       'params_dict':params_dict,
       'f_list':f_list,
       'g_list':g_list,
       'x_list':x_list,
       'y_ini_list':y_list,
       'y_run_list':y_list,
       'u_run_dict':u_dict,
       'u_ini_dict':u_dict,
       'h_dict':h_dict
      }

sys = db.system(sys)
db.sys2num(sys)
```

```python
xy_0_dict = grid_mv.dae['xy_0_dict']
xy_0_dict.update(dcgrid_dae['xy_0_dict'])
```

```python
data = json.dumps(xy_0_dict, indent=4)
fobj = open("xy_0_dict.json","w")
fobj.write(data)
fobj.close()
```

```python
u_dict
```

    {'v_B1_a_r': 11547.005383792517,
     'v_B1_a_i': 0.0,
     'v_B1_b_r': -5773.502691896256,
     'v_B1_b_i': -10000.000000000002,
     'v_B1_c_r': -5773.502691896264,
     'v_B1_c_i': 10000.0,
     'v_B7_a_r': 11547.005383792517,
     'v_B7_a_i': 0.0,
     'v_B7_b_r': -5773.502691896256,
     'v_B7_b_i': -10000.000000000002,
     'v_B7_c_r': -5773.502691896264,
     'v_B7_c_i': 10000.0,
     'i_B2lv_n_r': 5.684341886080802e-14,
     'i_B2lv_n_i': 1.2505552149377763e-12,
     'i_B3lv_n_r': -3.979039320256561e-13,
     'i_B3lv_n_i': 1.2505552149377763e-12,
     'i_B4lv_n_r': -3.979039320256561e-13,
     'i_B4lv_n_i': 1.1368683772161603e-12,
     'i_B5lv_n_r': -8.526512829121202e-13,
     'i_B5lv_n_i': 3.410605131648481e-13,
     'i_B6lv_n_r': -2.8421709430404007e-13,
     'i_B6lv_n_i': 4.547473508864641e-13,
     'i_B2_a_r': 0.0,
     'i_B2_a_i': 0.0,
     'i_B2_b_r': 0.0,
     'i_B2_b_i': 0.0,
     'i_B2_c_r': 0.0,
     'i_B2_c_i': 0.0,
     'i_B3_a_r': 0.0,
     'i_B3_a_i': 0.0,
     'i_B3_b_r': 0.0,
     'i_B3_b_i': 0.0,
     'i_B3_c_r': 0.0,
     'i_B3_c_i': 0.0,
     'i_B4_a_r': 0.0,
     'i_B4_a_i': 0.0,
     'i_B4_b_r': 0.0,
     'i_B4_b_i': 0.0,
     'i_B4_c_r': 0.0,
     'i_B4_c_i': 0.0,
     'i_B5_a_r': 0.0,
     'i_B5_a_i': 0.0,
     'i_B5_b_r': 0.0,
     'i_B5_b_i': 0.0,
     'i_B5_c_r': 0.0,
     'i_B5_c_i': 0.0,
     'i_B6_a_r': 0.0,
     'i_B6_a_i': 0.0,
     'i_B6_b_r': 0.0,
     'i_B6_b_i': 0.0,
     'i_B6_c_r': 0.0,
     'i_B6_c_i': 0.0,
     'q_B2lv_a': -322558.5103381794,
     'q_B2lv_b': -322558.5103381794,
     'q_B2lv_c': -322558.5103381795,
     'q_B3lv_a': -322558.50336306763,
     'q_B3lv_b': -322558.5033630677,
     'q_B3lv_c': -322558.5033630677,
     'q_B4lv_a': -322558.4994660576,
     'q_B4lv_b': -322558.49946605746,
     'q_B4lv_c': -322558.4994660576,
     'q_B5lv_a': -322558.51338830066,
     'q_B5lv_b': -322558.51338830066,
     'q_B5lv_c': -322558.51338830055,
     'q_B6lv_a': -322558.51542340475,
     'q_B6lv_b': -322558.51542340487,
     'q_B6lv_c': -322558.5154234048,
     'u_dummy': 1.0,
     'Dv_r_13': 0.0,
     'Dv_r_23': 0.0,
     'Dv_r_33': 0.0,
     'Dv_r_43': 0.0,
     'Dv_r_53': 0.0,
     'v_nom': 3000.0,
     'T_v': 0.02,
     'K_r': 0.0003,
     'a_B2lv': 2.92075867,
     'b_B2lv': 0.44961794,
     'c_B2lv': 0.02717091,
     'a_B3lv': 2.92075867,
     'b_B3lv': 0.44961794,
     'c_B3lv': 0.02717091,
     'a_B4lv': 2.92075867,
     'b_B4lv': 0.44961794,
     'c_B4lv': 0.02717091,
     'a_B5lv': 2.92075867,
     'b_B5lv': 0.44961794,
     'c_B5lv': 0.02717091,
     'a_B6lv': 2.92075867,
     'b_B6lv': 0.44961794,
     'c_B6lv': 0.02717091}

### Railway data building

```python
import numpy as np
railway_pos_ini = 0.0e3
railway_pos_end = 13.0e3
xend = railway_pos_end
data_input = {
    "system":{"f_hz":50,"model_type":"ae"},
    "buses":[{'bus':'11'},{'bus':'12'},{'bus':'13'},{'bus':'14'},{'bus':'15'},
             {'bus':'21'},{'bus':'22'},{'bus':'23'},{'bus':'24'},{'bus':'25'},
             {'bus':'31'},{'bus':'32'},{'bus':'33'},{'bus':'34'},{'bus':'35'},
             {'bus':'41'},{'bus':'42'},{'bus':'43'},{'bus':'44'},{'bus':'45'},
             {'bus':'51'},{'bus':'52'},{'bus':'53'},{'bus':'54'},{'bus':'55'},],
    "lines":    [
                {"bus_j":"11", "bus_k":"12", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":2.82},
                {"bus_j":"12", "bus_k":"13", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":4.42},
                {"bus_j":"13", "bus_k":"14", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"14", "bus_k":"15", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"21", "bus_k":"22", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":2.82},
                {"bus_j":"22", "bus_k":"23", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":4.42},
                {"bus_j":"23", "bus_k":"24", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"24", "bus_k":"25", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"31", "bus_k":"32", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":2.82},
                {"bus_j":"32", "bus_k":"33", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":4.42},
                {"bus_j":"33", "bus_k":"34", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"34", "bus_k":"35", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"41", "bus_k":"42", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":2.82},
                {"bus_j":"42", "bus_k":"43", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":4.42},
                {"bus_j":"43", "bus_k":"44", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"44", "bus_k":"45", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"51", "bus_k":"52", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":2.82},
                {"bus_j":"52", "bus_k":"53", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":4.42},
                {"bus_j":"53", "bus_k":"54", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"54", "bus_k":"55", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61},
                {"bus_j":"15", "bus_k":"21", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":2.82}, # module link
                {"bus_j":"25", "bus_k":"31", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":4.42}, # module link
                {"bus_j":"35", "bus_k":"41", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61}, # module link
                {"bus_j":"45", "bus_k":"51", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":0.61}, # module link
             ],
"loads" : [
        #{"bus": "12", "kVA":0e3, "pf": 0.95, "T_i":0.01,"I_max":1200},
        #{"bus": "13", "kVA":0e3, "pf": 0.95, "T_i":0.01,"I_max":1200},
        #{"bus": "14", "kVA":0e3, "pf": 0.95, "T_i":0.01,"I_max":1200}
        ],
"trains":[{'t_ini':0, 'direction':'right', 'trip':'right'},   # T1 llega a B 130
          {'t_ini':2, 'direction':'right', 'trip':'right'},  # T3 llega a B 160
          {'t_ini':10, 'direction':'right', 'trip':'right'},  # T5 llega a B 220
          {'t_ini':15, 'direction':'right', 'trip':'right'}, # T2
          {'t_ini':20, 'direction':'right', 'trip':'right'}, # T4
          {'t_ini':0, 'direction':'left', 'trip':'left'},   # T2
          {'t_ini':5, 'direction':'left', 'trip':'left'},   # T4
          {'t_ini':10, 'direction':'left', 'trip':'left'},  # T6
          {'t_ini':15, 'direction':'left', 'trip':'left'},  # T1
          {'t_ini':20, 'direction':'left', 'trip':'left'}
         ],  # T3
"grid_formers" : [
                  {"bus": "13","V_phph":3e3, "deg":0.0, 'position':2.6e3},
                  {"bus": "23","V_phph":3e3, "deg":0.0, 'position':4.2e3},
                  {"bus": "33","V_phph":3e3, "deg":0.0, 'position':5.8e3},
                  {"bus": "43","V_phph":3e3, "deg":0.0, 'position':8.4e3},
                  {"bus": "53","V_phph":3e3, "deg":0.0, 'position':12.8e3},
                 ],
"sections" : [
    {"name":"ini_vsc13",  "pos_ini":    0.0,"pos_end": 2.2e3,'N_tnodes':2,'N_trains':0,'T_power':[],'T_pos':[],'nodes_i':['11','12'],'nodes_v':['13']},
    {"name":"vsc13_vsc23","pos_ini":  2.2e3,"pos_end": 4.2e3,'N_tnodes':4,'N_trains':0,'T_power':[],'T_pos':[],'nodes_i':['14','15','21','22'],'nodes_v':['13','23']},
    {"name":"vsc23_vsc33","pos_ini":  4.2e3,"pos_end": 5.8e3,'N_tnodes':4,'N_trains':0,'T_power':[],'T_pos':[],'nodes_i':['24','25','31','32'],'nodes_v':['23','33']},
    {"name":"vsc33_vsc43","pos_ini":  5.8e3,"pos_end": 8.4e3,'N_tnodes':4,'N_trains':0,'T_power':[],'T_pos':[],'nodes_i':['34','35','41','42'],'nodes_v':['33','43']},
    {"name":"vsc43_vsc53","pos_ini":  8.4e3,"pos_end":11.0e3,'N_tnodes':4,'N_trains':0,'T_power':[],'T_pos':[],'nodes_i':['44','45','51','52'],'nodes_v':['43','53']},
    {"name":"vsc53_end",  "pos_ini": 11.0e3,"pos_end": 13e3,'N_tnodes':2,'N_trains':0,'T_power':[],'T_pos':[],'nodes_i':['54','55'],'nodes_v':['53']},
]
}

l_trip = np.genfromtxt('./data/izquierda_csv.csv',delimiter=';',skip_header=2)
x_left = l_trip[:,1]
p_left = l_trip[:,3]
t_left = l_trip[:,0]
v_left = l_trip[:,2]

mask = np.abs(np.diff(x_left))>1
mask[0]=True
mask[-1]=True
x_left_r =x_left[0:-1][mask]
p_left_r =p_left[0:-1][mask]

r_trip = np.genfromtxt('./data/derecha_csv.csv',delimiter=';',skip_header=2)
x_right = r_trip[:,1]
p_right = r_trip[:,3]
t_right = r_trip[:,0]
v_right = r_trip[:,2]

mask = np.abs(np.diff(x_right))>1
mask[0]=True
mask[-1]=True
x_right_r =x_right[0:-1][mask]
p_right_r =p_right[0:-1][mask]

trips = {
    'right':{'times':list(t_right),'positions':list(x_right),'powers':list(p_right),'speed':list(v_right),'slope':list(v_right*0)},
    'left': {'times':list( t_left),'positions':list( x_left),'powers':list( p_left),'speed':list( v_left),'slope':list( v_left*0)},
     }

data_input.update({'trips':trips})
save('trains_lv_5vsc_params.json',data_input)
```

## Losses

```python
# https://www.hitachiabb-powergrids.com/es/es/offering/product-and-system/semiconductors/semis-models/semis4

# m=0.85, v_dc=3000, pf=0.85 for all de simulations

# 5SNA 1000G650300, 1000A, 150ºC, 900Hz, 0.005K/W
# 20A  1.11kW 2.36%
#200A  6.11kW 1.31%
#500A 16.52kW 1.42%
#800A 30.41kW 1.63%

# 5SNA 0600G650100 600A, 125ºC,900Hz, 0.005K/W
#  1A  1.30kW 36.11%
# 50A  2.53kW  2.15%
#200A  7.36kW  1.58%
#500A 21.25kW  1.82%
#600A 27.69kW  1.97%

# 5SNA 0400J650100 400 A, 125ºC,900Hz, 0.005K/W
#  1A  0.47kW
#100A  3.15kW
#200A  6.82kW
#300A 11.32kW
#400A 17.03kW

V_ac = 0.85*3000/np.sqrt(2)
S_ac = np.sqrt(3)*V_ac*400
S_ac
```

    1249239.7688194206

```python
IGBT_5SNA_0400J650100 = np.array([[1,470],[100,3150],[200,6820],[300,11320],[400.0,17030]])

from scipy.optimize import curve_fit
def func(x, a, b, c):
    return a*x**2 + b*x + c

popt, pcov = curve_fit(func, IGBT_5SNA_0400J650100[:,0],IGBT_5SNA_0400J650100[:,1])
a,b,c = popt
```

```python
fig, axes = plt.subplots(nrows=1,ncols=1, figsize=(6, 5), dpi=100)

I_rms = np.linspace(0,500,100)
axes.plot(np.sqrt(3)*V_ac*IGBT_5SNA_0400J650100[:,0]/1e6,IGBT_5SNA_0400J650100[:,1],'.')
axes.plot(np.sqrt(3)*V_ac*I_rms/1e6,4.89073990e-02*I_rms**2+2.17705101e+01*I_rms+ 4.65090148e+02)
```

```python
    axes[0].plot(grid_railway_droop.node_pos[it]/1e3,grid_railway_droop.voltages_dc[it], label = 'V', lw=0.08, color='#888888' )
    axes[1].plot(grid_railway_droop.node_pos[it]/1e3,grid_railway_droop.voltages_dc[it], label = 'V')

for ax in axes:
    #ax.set_ylim((-1,4))
    ax.grid()
    ax.set_ylabel('Voltages (V)')

line_trains = axes[1].plot([],[], 'o', label = 'trains')
axes[1].set_xlabel('Position (km)')
fig.tight_layout()
fig
```

```python
params_30A = {
 'a_i': 1.95773625,
 'b_i': 0.32430738,
 'c_i':-0.22694456,
 'd_i': 0.0155737 ,
 'e_i':-0.01093976,
 'a_d': 0.96302242,
 'b_d': 0.02653386,
 'c_d': 0.05055759,
 'd_d': 0.01328264,
 'e_d': 0.01394945,
 'R_th_igbt_sink': 0.5167999063238943,
 'R_th_diode_sink': 0.8394969099630814,
 'R_th_sink_a': 0.6619445743152772}

params = params_30A

a_i = params['a_i']
b_i = params['b_i']
c_i = params['c_i']
d_i = params['d_i']
e_i = params['e_i']
a_d = params['a_d']
b_d = params['b_d']
c_d = params['c_d']
d_d = params['d_d']
e_d = params['e_d']

m = 0.95
fp = 0.8
i_rms = np.linspace(0,30,600)
p_igbt  = a_i + (b_i - c_i*m*fp)*i_rms + (d_i - e_i*m*fp)*i_rms*i_rms
p_diode = a_d + (b_d - c_d*m*fp)*i_rms + (d_d - e_d*m*fp)*i_rms*i_rms

p_switch = p_igbt + p_diode
#print(f'p_switch = {p_switch} W')
```