P1 discontinuous SWIPDG, stationary linear elliptic ESV2007 problem
==================================

This example is about approximating the solution $u$ of the elliptic problem

$$\begin{align}
  -\nabla\cdot( \kappa \nabla u ) &= f   &&\text{in } \Omega\\
                                u &= g_D &&\text{on }\partial\Omega
\end{align}$$

with datafunction as defined in `dune/gdt/test/linearelliptic/problems/ESV2007.hh` (see below) using a piecewise linear SWIPDG scheme, as in `dune/gdt/test/linearelliptic/discretizers/ipdg.hh`.

Note that the discretization below contains handling of arbitrary Dirichlet and Neumann boundary data, although the problem at hand contains only trivial Dirichlet data.

```python
from dune.xt import common, grid, functions, la
from dune import gdt
common.init_mpi()
```

$$\begin{align}
  \Omega &= [-1, 1]^2\\
  \Gamma_D &= \partial\Omega\\
  \Gamma_N &= \emptyset
\end{align}$$

```python
g = grid.make_cube_grid__2d_simplex_aluconform(lower_left=[-1, -1],
                                               upper_right=[1, 1],
                                               num_elements=[4, 4],
                                               num_refinements=2,
                                               overlap_size=[0, 0])
#g.visualize('../swipdg_esv2007_grid')
boundary_info = grid.make_boundary_info_on_leaf_layer(g, 'xt.grid.boundaryinfo.alldirichlet')
apply_on_neumann_boundary = grid.make_apply_on_neumann_intersections_leaf_part(boundary_info)
apply_on_dirichlet_boundary = grid.make_apply_on_dirichlet_intersections_leaf_part(boundary_info)
```

$$\begin{align}\kappa(x) &:= 1\\
f(x) &:= \tfrac{1}{2} \pi^2 \cos(\tfrac{1}{2} \pi x_0) \cos(\tfrac{1}{2} \pi x_1)\\
g_D(x) &:= 0\end{align}$$

Note that the grid `g` is only provided to select the correct _type_ of function. These functions do not rely on the actual grid which is why we need to later on provide the grid again, i.e., for `visualize(g)`.

```python
kappa = functions.make_constant_function_1x1(g, 1.0, name='diffusion')
f = functions.make_expression_function_1x1(g,
                                           'x',
                                           '0.5*pi*pi*cos(0.5*pi*x[0])*cos(0.5*pi*x[1])',
                                           order=3,
                                           name='force')
g_D = functions.make_constant_function_1x1(g, 0.0, name='dirichlet')
g_N = functions.make_constant_function_1x1(g, 0.0, name='neumann')
#kappa.visualize(g, '../swipdg_esv2007_diffusion')
#f.visualize(g, '../swipdg_esv2007_force')
```

```python
space = gdt.make_dg_leaf_part_to_1x1_fem_p1_space(g)
#space.visualize('../swipdg_esv2007_dg_space')

# There are two ways to create containers:
# * manually create them and given them to the operators/functionals
# * let those create appropriate ones

# in the CG example we chose the latter, so here we do the former
system_matrix = la.IstlRowMajorSparseMatrixDouble(space.size(), space.size(), space.compute_pattern('face_and_volume'))
swipdg_operator = gdt.make_elliptic_swipdg_matrix_operator(kappa, boundary_info, system_matrix, space)

rhs_vector = la.IstlDenseVectorDouble(space.size(), 0.0)
l2_force_functional = gdt.make_l2_volume_vector_functional(f, rhs_vector, space)

# there are two equivalent ways to restrict the integration domain of the face functional:
# * provide an apply_on_... tag on construction
# * provide an apply_on_... tag when appending the functional to the system assembler

l2_neumann_functional = gdt.make_l2_face_vector_functional(g_N, rhs_vector, space, apply_on_neumann_boundary)

swipdg_dirichlet_functional = gdt.make_elliptic_swipdg_dirichlet_vector_functional(g_D, kappa, boundary_info, rhs_vector, space)

# compute everything in one grid walk
system_assembler = gdt.make_system_assembler(space)
system_assembler.append(swipdg_operator)
system_assembler.append(l2_force_functional)
system_assembler.append(l2_neumann_functional)
system_assembler.append(swipdg_dirichlet_functional)
system_assembler.assemble()

# solve the linear system
u_h = la.IstlDenseVectorDouble(space.size(), 0.0)
la.make_solver(system_matrix).apply(rhs_vector, u_h)

# visualize (this will write swipdg_esv2007_solution.vtu)
gdt.make_discrete_function(space, u_h, 'solution').visualize('../swipdg_esv2007_solution')
```


__Chapter 12 - Implementing a Multilayer Artificial Neural Network from Scratch__

1. [Import](#Import)
1. [Modeling complex functions with artificial neural networks](#Modeling-complex-functions-with-artificial-neural-networks)
    1. [Activating a neural network via forward propagation](#Activating-a-neural-network-via-forward-propagation)
1. [Classifying handwritten digits](#Classifying-handwritten-digits)
1. [Implementing a multilayer perceptron](#Implementing-a-multilayer-perceptron)
    1. [Homegrown implementation](#Homegrown-implementation)
1. [Training an artificial neural network](#Training-an-artificial-neural-network)
    1. [Logistic cost function refresher](#Logistic-cost-function-refresher)
    1. [Computing the logistic cost function](#Computing-the-logistic-cost-function)
    1. [Developing intuition for backpropagation](#Developing-your-intuition-for-backpropagation)
    1. [Training neural networks via backpropagation](#Training-neural-networks-via-backpropagation)

# Import

```python
# standard libary and settings
import os
import sys
import importlib
import itertools
from io import StringIO
import warnings

warnings.simplefilter("ignore")
from IPython.core.display import display, HTML

display(HTML("<style>.container { width:95% !important; }</style>"))

# data extensions and settings
import numpy as np

np.set_printoptions(threshold=np.inf, suppress=True)
import pandas as pd

pd.set_option("display.max_rows", 500)
pd.options.display.float_format = "{:,.6f}".format

# modeling extensions
from sklearn.base import TransformerMixin, BaseEstimator
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering
from sklearn.datasets import load_boston, load_wine, load_iris, load_breast_cancer, make_blobs, make_moons
from sklearn.decomposition import PCA, LatentDirichletAllocation
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier, ExtraTreesClassifier, IsolationForest
from sklearn.feature_extraction.text import CounterVectorizer, TfidfTransformer, TfidfVectorizer, HashingVectorizer
from sklearn.feature_selection import f_classif, f_regression, VarianceThreshold, SelectFromModel, SelectKBest
from sklearn.linear_model import Lasso, Ridge, ElasticNet, LinearRegression, LogisticRegression, SGDRegressor
from sklearn.metrics import precision_score, recall_score, f1_score, explained_variance_score, mean_squared_log_error, mean_absolute_error, median_absolute_error, mean_squared_error, r2_score, confusion_matrix, roc_curve, accuracy_score, roc_auc_score, homogeneity_score, completeness_score, classification_report, silhouette_samples
from sklearn.model_selection import KFold, train_test_split, GridSearchCV, StratifiedKFold, cross_val_score, RandomizedSearchCV
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.pipeline import make_pipeline, Pipeline, FeatureUnion
from sklearn.preprocessing import StandardScaler, RobustScaler, PolynomialFeatures, OrdinalEncoder, LabelEncoder, OneHotEncoder, KBinsDiscretizer, QuantileTransformer, PowerTransformer, MinMaxScaler
from sklearn.svm import SVC, SVR
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import sklearn.utils as utils

# visualization extensions and settings
import seaborn as sns
import matplotlib.pyplot as plt

# custom extensions and settings
sys.path.append("/home/mlmachine") if "/home/mlmachine" not in sys.path else None
sys.path.append("/home/prettierplot") if "/home/prettierplot" not in sys.path else None

import mlmachine as mlm
from prettierplot.plotter import PrettierPlot
import prettierplot.style as style

# magic functions
%matplotlib inline
```

# Modeling complex functions with artificial neural networks

A fully connected network, also known as a multilayer perceptron (MLP), has one input layer of neurons, one hidden layer and one output layer. The units in the hidden layer are fully connected to the input layer, and the output layer is fully connected to the hidden layer. If more than one hidden layer is present then the MLP is considered to be a deep artificial neural network.

Each neuron, or activation unit, can be identified by its position amongst the other activation neurons and the layer in which it appear - $a_i^l$ is the $i$th neuron in the $l$th layer. For simplicity, this walkthrough will use the $l$ values of $in, h, out$ to describe the input, hidden and output layer. So $a_i^{out}$ is the $i$th activation unit of the outer layer. The input and hidden layers each have bias units, $a_0^{in}$ and $a_0^{out}$ and these are set to one. This means the input layer is just the input values plus the bias unit:

$$
a^{in}
=
\begin{bmatrix} a_0^{in} \\ a_1^{in} \\ \vdots \\  a_m^{in} \end{bmatrix}
=
\begin{bmatrix} 1 \\ x_1^{in} \\ \vdots \\  x_m^{in} \end{bmatrix}
$$

Each activation unit in layer $l$ is connected to all of the units in layer $l$ + 1 by a weight coefficient. As an example, the connection between the $k$th unit layer $l$ to the $j$th in layer $l$ + 1 is written as $w_{k,j}^l$. So the weight matrix that connects the input layer to the hidden layer is $\mathbf{W}^{h}$, the weight matrix that connects the hidden layer to the output layer is $\mathbf{W}^{out}$. The weight matrix that connects, for example, the input and hidden layers is $\mathbf{W}^h \in \mathbb{R}^{m \times d}$, where $d$ is the number of hidden units and $m$ is the numnber of input units (including bias).

Having one unit in the output layer is sufficient for a binary classificaiton task, but having more than one enables multiclass classification through one-hot vector representation of the multiclass labels.:
$$
0
=
\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}
1
=
\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}
2
=
\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}
$$

## Activating a neural network via forward propagation

The MLP learning procedure in three steps:

1. Starting at the input layer, forward propagate the patterns of the training data through the network to generate an output
2. Using the output, calculate the error to be minimized using a cost function
3. Backpropagate the error, find its derivative with respect to each weight in the network, then update the model

In feedforward networks, each layer serves as the input to the next layer without any loops. This contrasts with recurrent neural networks. The three steps above are repeated for multiple epochs to learn the best weights, and then forward propagation is used to the calculate the network output and apply a threshold function to obtain the predicted class labels represented in the one-hot format above. Describing each step in more detail:

The first activation unit in the hidden layer $a_1^{h}$ is connected to all units in the input layer, and is calculated by:

$$
z_1^h = a_0^{in}w_{0,1}^h + a_1^{in}w_{1,1}^h + ... a_m^{in}w_{m,1}^h
$$
$$
a_1^h = \phi\big(z_1^h\big)
$$
$z_1^h$ is the net input and $\phi(\cdot)$ is the activation function that acts on $z_1^h$. This activation function need to be differentiable to learn the weights that connect the neurons using a gradient-based approach. Non-linear activation function are also possible and are used to solve complex problems like image classification. One familiar non-linear activation function is the sigmoid function, which arose in the context of logistic regression:

$$
\phi(z) = \frac{1}{1 + e^{-z}}
$$

This is an S-shaped curve that maps the input $z$ onto a logistic distribution that ranges from 0 to 1 and cross the y-axis at $z$ = 0. Given this, we can think of each neuron as logistic regression units that return values in the number range of 0 to 1. To describe this activation function in linear algebra notation:

$$
\begin{equation}
\textbf{z}^{h} = \textbf{a}^{in}\textbf{W}^h
\\
\textbf{a}^h = \phi\big(\textbf{z}^h\big)
\end{equation}
$$

$\textbf{a}^{in}$ is the 1 x $m$ dimensional feature vector for a sample $\textbf{x}^{in}$, plus the bias unit. $\textbf{W}^{h}$ is the $m$ x $d$ dimensional weight matrix where $d$ is the number of units in the hidden layer. Through matrix-vector multiplication, we obtain a 1 x $d$ dimensional net input vector $\textbf{z}^h$ to be used to calculate the activation $\textbf{a}^{h}$ ($\textbf{a}^{h} \in \mathbb{R}^{1 \times d}$). This computation can be generalized to all $n$ samples in the training set by:

$$
\textbf{Z}^{h} = \textbf{A}^{in}\textbf{W}^{h}
$$

In this representation, $\textbf{A}^{in}$ is an $n$ x $m$ matrix, and the matrix-matrix multiplication results in an $n$ x $d$ dimensional net input matrix $\textbf{Z}^{h}$. Lastly, apply the activation function $\phi(\cdot)$ to each value in the net input matrix to get the $n$ x $d$ dimensional matrix $\textbf{A}^{h}$ for the next layer, which in this case is the output layer:

$$
A^h = \phi\big(\textbf{Z}^{h}\big)
$$

Just as above, we can write the activation function of the output layer in vectorized form for multiple samples:
$$
\textbf{Z}^{out} = \textbf{A}^{h}\textbf{W}^{out}
$$

In this last step, we multiply the $d$ x $t$ matrix $\textbf{W}^{out}$ (where $t$ is the number of output units) by the $n$ by $d$ dimensional matrix $\textbf{A}^{h}$ to obtain the $n$ by $t$ dimensional matrix $\textbf{Z}^{out}$, where the columns in this matrix represent the outputs for each sample. The last step is to apply the sgmoid activation function to obtain the number valued ouput of the network:

$$
\textbf{A}^{out} = \phi\big(\textbf{Z}^{out}\big), \textbf{A}^{out} \in \mathbb{R}^{n \times t}
$$

# Classifying handwritten digits

Implement and train our first MLP to classify handwritten digits from the Mixed National Institute of Standards and Techngology (MNIST). It consists of handwritten digits from 250 people - half high school students and half Census Bureau employees.

```python
# Load data and print dimensions
df_train = pd.read_csv("s3://tdp-ml-datasets/kaggle-mnist//train.csv", sep=",")
df_test = pd.read_csv("s3://tdp-ml-datasets/kaggle-mnist//test.csv", sep=",")

print("Training data dimensions: {}".format(df_train.shape))
print("Test data dimensions: {}".format(df_test.shape))

# separate
df_train_label = df_train["label"]
df_train = df_train.drop(labels="label", axis=1)
```

    Training data dimensions: (42000, 785)
    Test data dimensions: (28000, 784)

```python
# train/test split
X_train, X_test, y_train, y_test = train_test_split(
    df_train, df_train_label, test_size=0.2
)
```

```python
# visualize samples digits
fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True)

ax = ax.flatten()
for i in range(10):
    img = np.array(X_train[y_train == i].iloc[0]).reshape(28, 28)
    ax[i].imshow(img, cmap="Greys")

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
plt.show()
```

```python
# visualize multiple samples of the same digit
fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True)

ax = ax.flatten()
for i in range(25):
    img = np.array(X_train[y_train == 7].iloc[i]).reshape(28, 28)
    ax[i].imshow(img, cmap="Greys")

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
plt.show()
```

# Implementing a multilayer perceptron

## Homegrown implementation

```python
# multilayer perceptron custom class
class NeuralNetMLP:
    """
    Info:
        Description:
            Feedforward neural network / Multilayer perceptron classifier
        Parameters:
            n_hidden : int, default=30
                Number of hidden units
            l2 :  float, default=0.
                Lambda value for L2-regularization
                No regularization if l2 = 0.
            epochs : int (default : 100)
                Number of passes through training set
            eta : float, default=0.001
                Learning rate
            shuffle : bool, default=True
                Shuffle data after every epoch if True to prevent circles
            minibatch_size : int, default=1
                # of training samples per mini batch
            seed : int, default=None
                Random seed for weight initialization and shuffling
        Attributes:
            eval_ : dict
                Dictionary that collects cost, training accuracy and validation
                accuracy for each epoch during training
    """

    def __init__(
        self,
        n_hidden=30,
        l2=0.0,
        epochs=100,
        eta=0.001,
        shuffle=True,
        minibatch_size=1,
        seed=None,
    ):
        self.random = np.random.RandomState(seed)
        self.n_hidden = n_hidden
        self.l2 = l2
        self.epochs = epochs
        self.eta = eta
        self.shuffle = shuffle
        self.minibatch_size = minibatch_size

    def _onehot(self, y, n_classes):
        """
        Info:
            Description:
                Encode labels into one-hot representation
            Parameters:
                y : numpy array
                    Target values
                n_classes : int
                    Number of unique classes
            Returns:
                onehot : array, shape = (n_samples, n_labels)
        """
        onehot = np.zeros((n_classes, y.shape[0]))
        for idx, val in enumerate(y.astype(int)):
            onehot[val, idx] = 1.0
        return onehot.T

    def _sigmoid(self, z):
        """
        Info:
            Description:
                Compute logistic function (sigmoid)
            Parameters:
                z : float
                    net input
        """
        return 1.0 / (1.0 + np.exp(-np.clip(z, -250, 250)))

    def _forward(self, X):
        """
        Info:
            Description:
                Compute forward propagation step
            Parameters:
                X : array
                    Input data
        """
        # step 1: net input of hidden layer
        # [n_samples, n_features] dot [n_features, n_hidden] -> [n_samples, n_hidden]
        z_h = np.dot(X, self.w_h) + self.b_h

        # step 2: activation of hidden layer
        a_h = self._sigmoid(z_h)

        # step 3: net input of output layer
        # [n_samples, n_hidden] dot [n_hidden, n_classlabels] -> [n_samples, n_classlabels]
        z_out = np.dot(a_h, self.w_out) + self.b_out

        # step 4: activation of output layer
        a_out = self._sigmoid(z_out)

        return z_h, a_h, z_out, a_out

    def _compute_cost(self, y_enc, output):
        """
        Info:
            Description:
                Compute cost function
            Parameters:
                y_enc : array, shape = (n_samples, n_labes)
                    one-hot encoded class labels
                output : array, shape = (n_samples, n_output_units)
                    Activation of the output layer (forward propagation)
            Returns:
                cost : float
                    Regularized cost of epoch
        """
        L2_term = self.l2 * (np.sum(self.w_h ** 2.0) + np.sum(self.w_out ** 2.0))
        term1 = -y_enc * (np.log(output))
        term2 = (1.0 - y_enc) * np.log(1.0 - output)
        cost = np.sum(term1 - term2) + L2_term
        return cost

    def predict(self, X):
        """
        Info:
            Description:
                Predict class labels
            Parameters:
                X : array, shape = [n_samples, n_features]
            Returns:
                y_pred : array, shape = [n_samples]
                    Predicted class labels
        """
        z_h, a_h, z_out, a_out = self._forward(X)
        y_pred = np.argmax(z_out, axis=1)
        return y_pred

    def fit(self, X_train, y_train, X_valid, y_valid):
        """
        Info:
            Description:
                Learn weights from training data
            Parameters:
                X_train : array, shape = [n_samples, n_features]
                    Input layer with original features
                y_train : array, shape = [n_samples]
                    Correct target labels
                X_valid : array, shape = [n_samples, n_features]
                    Sample features for validation during training
                y_valid : array, shape = [n_samples]
                    Sample labels
        """
        n_output = np.unique(y_train).shape[0]  # number of class labels
        n_features = X_train.shape[1]

        #######################
        # weight initialization

        # weights for input to hidden
        self.b_h = np.zeros(self.n_hidden)
        self.w_h = self.random.normal(
            loc=0.0, scale=0.1, size=(n_features, self.n_hidden)
        )

        # weights for hidden to output
        self.b_out = np.zeros(n_output)
        self.w_out = self.random.normal(
            loc=0.0, scale=0.1, size=(self.n_hidden, n_output)
        )

        epoch_strlen = len(str(self.epochs))  # for progr. format
        self.eval_ = {"cost": [], "train_acc": [], "valid_acc": []}

        y_train_enc = self._onehot(y_train, n_output)

        # iterate over training epochs
        for i in range(self.epochs):
            # iterate over mini batches
            indices = np.arange(X_train.shape[0])
            if self.shuffle:
                self.random.shuffle(indices)
            for start_idx in range(
                0, indices.shape[0] - self.minibatch_size + 1, self.minibatch_size
            ):
                batch_idx = indices[start_idx : start_idx + self.minibatch_size]

                # forward propagation
                z_h, a_h, z_out, a_out = self._forward(X_train[batch_idx])

                #######################
                # backpropagation

                # [n_samples, n_classlabels]
                sigma_out = a_out - y_train_enc[batch_idx]

                # [n_samples, n_hidden]
                sigmoid_derivative_h = a_h * (1.0 - a_h)

                # [n_samples, n_classlabels] dot [n_classlabels, n_hidden] -> [n_samples, n_hidden]
                sigma_h = np.dot(sigma_out, self.w_out.T) * sigmoid_derivative_h

                # [n_features, n_samples] dot [n_samples, n_hidden] -> [n_features, n_hidden]
                grad_w_h = np.dot(X_train[batch_idx].T, sigma_h)
                grad_b_h = np.sum(sigma_h, axis=0)

                # [n_hidden, n_samples] dot [n_samples, n_classlabels] -> [n_hidden, n_classlabels]
                grad_w_out = np.dot(a_h.T, sigma_out)
                grad_b_out = np.sum(sigma_out, axis=0)

                # regularization and weight updates
                delta_w_h = grad_w_h + self.l2 * self.w_h
                delta_b_h = grad_b_h  # bias is not regularized
                self.w_h -= self.eta * delta_w_h
                self.b_h -= self.eta * delta_b_h

                delta_w_out = grad_w_out + self.l2 * self.w_out
                delta_b_out = grad_b_out  # bias is not regularized
                self.w_out -= self.eta * delta_w_out
                self.b_out -= self.eta * delta_b_out

            #######################
            # evaluation
            z_h, a_h, z_out, a_out = self._forward(X_train)

            cost = self._compute_cost(y_enc=y_train_enc, output=a_out)

            y_train_pred = self.predict(X_train)
            y_valid_pred = self.predict(X_valid)

            train_acc = (np.sum(y_train == y_train_pred)).astype(
                np.float
            ) / X_train.shape[0]
            valid_acc = (np.sum(y_valid == y_valid_pred)).astype(
                np.float
            ) / X_valid.shape[0]

            sys.stderr.write(
                "\r%0*d/%d | Cost: %.2f | Train/Valid Accuracy: %.2f%%/%.2f%%"
                % (
                    epoch_strlen,
                    i + 1,
                    self.epochs,
                    cost,
                    train_acc * 100,
                    valid_acc * 100,
                )
            )
            sys.stderr.flush()

            self.eval_["cost"].append(cost)
            self.eval_["train_acc"].append(train_acc)
            self.eval_["valid_acc"].append(valid_acc)

        return self
```

```python
# initialize a 784-100-10 MLP - a neural network with 784 samples,
# 100 hidden units and 10 output units
nn = NeuralNetMLP(
    n_hidden=100,
    l2=0.01,
    epochs=50,
    eta=0.0005,
    minibatch_size=100,
    shuffle=True,
    seed=1,
)

# train neural network using first 30,000 training samples
# and use the remainder for validation
nn.fit(
    X_train=X_train.values[:30000],
    y_train=y_train.values[:30000],
    X_valid=X_train.values[30000:],
    y_valid=y_train.values[30000:],
)
```

    50/50 | Cost: 12305.40 | Train/Valid Accuracy: 94.21%/92.83%

    <__main__.NeuralNetMLP at 0x7f1c865875c0>

```python
# plot cost over the 200 epochs
p = PrettierPlot()
ax = p.make_canvas(
    title="Cost reduction over time", x_label="Epochs", y_label="Cost", y_shift=0.8
)
p.line(
    x=np.arange(nn.epochs),
    y=np.array(nn.eval_["cost"]),
    linecolor=style.style_hex_mid[0],
    x_ticks=np.arange(0, 51, 5),
    bbox=(1.2, 0.9),
    marker_on=False,
    ax=ax,
)
```

> Remarks - The cost decreased substantially over the first 100 epochs before slowing afterwards. Even after the 100th epoch, it seems there is gradual improvement and would likely be more if this were allowed to run for more epochs.

```python
# plot cost over the 200 epochs
p = PrettierPlot()
ax = p.make_canvas(
    title="Accuracy improvement over time - train vs. test",
    x_label="Epochs",
    y_label="Accuracy",
    y_shift=0.7,
)
p.multi_line(
    x=np.arange(nn.epochs),
    y=np.c_[nn.eval_["train_acc"], nn.eval_["valid_acc"]],
    label=["Training", "Validation"],
    x_units="f",
    y_units="fff",
    bbox=(1.2, 0.9),
    marker_on=False,
    ax=ax,
)
```

> Remarks - The plot shows an increasing gap between the training and validation increases.

```python
# calculate the prediction accuracy on test
y_testPred = nn.predict(X_test)
acc = np.sum(y_test == y_testPred).astype(np.float) / X_test.shape[0]
print("Test accuracy: {:.2f}%".format(acc * 100))
```

    Test accuracy: 93.17%

> Remarks - Despite overfitting on the training data, this model performed pretty well by achieving 97.54 percent accuracy on the unseen test data. To further refine the model, we can adjust the number of hidden units, the regularization parameter, the learning rate, among other things.

```python
# review some of the misclassified images
miscl_img = X_test[y_test != y_testPred][:25]
actual_label = y_test[y_test != y_testPred][:25]
predicted_label = y_testPred[y_test != y_testPred][:25]

fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True)

ax = ax.flatten()
for i in range(25):
    img = np.array(miscl_img.iloc[i]).reshape(28, 28)
    ax[i].imshow(img, cmap="Greys", interpolation="nearest")
    ax[i].set_title(
        "%d) t: %d p: %d" % (i + 1, actual_label.iloc[i], predicted_label[i]),
        fontsize=10,
    )

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
plt.show()
```

# Training an artificial neural network

## Logistic cost function refresher

With linear regression, we seek to minimize the sum of squares to learn the weights of the models. We did this by minimizing this cost function:

$$
J(\textbf{w}) = \sum_i\frac{1}{2}\big(\phi(z^i) - y^i\big)^2
$$

With the logistic cost function, we want to maximize likelihood. Assuming the samples in our dataset are independent of one another, likelihood $L$ is defined as:

$$
L(\textbf{x}) = P(\textbf{y}\vert\textbf{x};\textbf{w}) = \prod^n_{i=1}P(y^i\vert\textbf{x}^i;\textbf{w}) = \prod^n_{i=1}\big(\phi\big(z^i\big)\big)^{y^i}\big(1-\phi\big(z^i\big)\big)^{1-y^i}
$$

The crux of this function is the logistic sigmoid function:

$$
\phi(z) = \frac{1}{1+e^{-z}}
$$

Where z is the net input, which is the linear combination of weights and sample feature values: $z = w_0x_0 + w_1x_1 + ... + w_mx_m$

And in practice, it is more straightforward to maximize the (natural) log of the cost function, which is called the log likelihood function:

$$
l(\textbf{w}) = \mbox{log}L(\textbf{w}) = \sum^n_{i=1}\bigg[y^i\mbox{log}\big(\phi\big(z^i\big)\big) + \big(1-y^i\big)\mbox{log}\big(1-\phi(z^i)\big)\bigg]
$$

Applying the log function reduces the potential for numberal underflow, which can occur if the likelihoods are very small. Further, converting the product of factors into a summation of factors makes it easier to obtain the derivative of this function. With this cost function, we can either maximize it using gradient ascent or rewrite it as a cost function to be minimized. This function can be rewritten as:

$$
J(\textbf{w}) = \sum^n_{i=1}\bigg[-y^i\mbox{log}\big(\phi\big(z^i\big)\big) - \big(1-y^i\big)\mbox{log}\big(1-\phi(z^i)\big)\bigg]
$$

To better understand this function, we can look at the cost that we calculate for one training sample:

$$
J(\phi(z),y;\textbf{w}) = -y\mbox{log}\big(\phi\big(z\big)\big) - \big(1-y\big)\mbox{log}\big(1-\phi(z)\big)
$$

The first term becomes zero if $y$ = 0 and the second term becomes zero if $y$ = 1, so this cost function can be reshaped as:

$$
J(\phi(z),y;\textbf{w}) =
\left\{
    \begin{array}{ll}
        -y\mbox{log}\big(\phi\big(z\big)\big)  & \mbox{if } y = 1 \\
        -\mbox{log}\big(1-\phi(z)\big)  &\mbox{if } y = 0
    \end{array}
\right.
$$

```python
# plot cost function for y = 0 and y = 1
def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def cost_1(z):
    return -np.log(sigmoid(z))

def cost_0(z):
    return -np.log(1 - sigmoid(z))

z = np.arange(-10, 10, 0.1)
phi_z = sigmoid(z)

c1 = [cost_1(x) for x in z]
c0 = [cost_0(x) for x in z]
c = np.c_[c1, c0]

# plot
p = PrettierPlot()
ax = p.make_canvas(title="", x_label="$\phi(z)$", y_label="J(x)", y_shift=0.9)
p.multi_line(
    x=phi_z,
    y=c,
    label=["J(w) if y = 1", "J(w) if y = 0"],
    x_units="fff",
    y_units="f",
    bbox=(1.2, 0.9),
    marker_on=False,
    ax=ax,
)
plt.ylim(0, 5)
```

> Remarks - The key takeaway is seeing how the cost J(w) drastically increases for each curve as the prediction becomes further away from the truth.

## Computing the logistic cost function

$$
J(\textbf{w}) = -\sum^n_{i=1}y^i\mbox{log}\big(a^i\big) + \big(1-y^i\big)\mbox{log}\big(1-a^i\big)
$$

where $a^i$ is the sigmoid activation function of the $i$th sample in the dataset, which is computed in the forward propagation step: $a^i = \phi(z^i)$. The $i$ superscript refers to a specific training sample index, not a neural net layer.

We can add a regularization term to reduce the degree of overfitting:

$$
J(\textbf{w}) = -\bigg[\sum^n_{i=1}y^i\mbox{log}\big(a^i\big) + \big(1-y^i\big)\mbox{log}\big(1-a^i\big)\bigg] + \frac{\lambda}{2}\Vert\textbf{w}\Vert^2_2
$$

This MLP is built to perform multiclass classification, and returns an output vector of $t$ elements that gets compared to the 1 x $t$ dimensional target vector in the one-hot encoding representation. As an example, the activation of the third layer and the target class 2 for a particular sample may look like:

$$
a^{out}
=
\begin{bmatrix} 0.1 \\ 0.9 \\ \vdots \\  0.3 \end{bmatrix}
, y =
\begin{bmatrix} 0 \\ 1 \\ \vdots \\  0 \end{bmatrix}
$$

Because of this, we need to generalize the logistic cost function to all $t$ activation units in our network. So it becomes the following:

$$
J(\textbf{W}) = -\sum^n_{i=1}\sum^t_{j=1}y^i_j\mbox{log}\big(a^i_j\big) + \big(1-y^i_j\big)\mbox{log}\big(1-a^i_j\big)
$$

The $i$ again refers to the index of a specific training sample. Here is the full function with the regularization term added:

$$
J(\textbf{W}) = -\bigg[\sum^n_{i=1}\sum^t_{j=1}y^i_j\mbox{log}\big(a^i_j\big) + \big(1-y^i_j\big)\mbox{log}\big(1-a^i_j\big)\bigg] + \frac{\lambda}{2}\sum_{l=1}^{L-1}\sum_{i=1}^{\mu_l}\sum_{j=1}^{\mu_{l+1}}(w^l_{j,i})^2
$$

This generalized regularization term calculates the sum of all weights of an $l$ layer. $\mu_1$ refers to the number of units in a given layer $l$. To minimize this cost function, we need to calculate the partial derivative of the parameters $\textbf{W}$ with respect to each weight in every layer of the network:

$$
\frac{\partial}{\partial(w^l_{j,i})}J(\textbf{W})
$$

It's important to note that $\textbf{W}$ consists of multiple matrices. In an MLP with one hidden unit, we have the weight matrix $\textbf{W}^h$ that connects the input layer to the hidden layer, and $\textbf{W}^{out}$ that connects the hidden layer to the output layer. This is a three dimensional tensor.

For $\textbf{W}^h$, the rows represent the features, and the columns represent the hidden units. One particular value represents the weight that a certain hidden unit has for a certain feature. For $\textbf{W}^{out}$, the rows represent the hidden units, and the columns represent the output units.  The matrices in this tensor do not necessarily have the same shape. This only occurs if the MLP is initialized with the samenumber of hidden units, output units and input feautres.

## Developing intuition for backpropagation

Backpropagation is one of the most widely use algorithms for efficiently training neural networks. It is a very computationally efficient approach to compute the partial derivatives of a complex cost function in MLPs. The goal is to use the derivatives to learn the weight coefficients. The challenging aspect of weight determination in an MLP is that we are typically dealing with a very large number of weight coefficients in a high-dimensional feature space. Further, the error surface of a neural network cost function is not convex or smooth with respect to the parameters, unlike single layer networks such as Adaline or logistic regression. There will be many local minima that need to be overcome in order to find teh global minimum of the cost function.

This calls forth the calculcus concept of the chain rule, which is a useful trick for finding the derivative of a complex, nested function like $f(g(x))$

$$
\frac{d}{dx}\bigg(f\big(g(x)\big)\bigg) = \frac{df}{dg} \cdot \frac{dg}{dx}
$$

The chain rule can be used for an arbitrarily long function. Even if we have five different functions $f(x)$, $g(x)$, $h(x)$, $u(x)$ and $v(x)$, where $F$ is the function composition $F(x) = f(g(h(u(v(x)))))$. The derivative of this function is:

$$
\frac{dF}{dx} = \frac{d}{dx}F(x) = \frac{d}{dx}f(g(h(u(v(x))))) = \frac{df}{dg} \cdot \frac{dg}{dh} \cdot \frac{dh}{du} \cdot \frac{du}{dv} \cdot \frac{dv}{dx}
$$

Automatic differentiation is a technique for solving these kinds of problems efficiently. It comes in two modes, forward and reverse. Backpropagation is a special case of rever mode automatic differentiation. The key point is that applying the chain rule in the forward mode would be expensive since we need to multiply large matrices for each layer (Jacobians) that are eventually multiplied by a vector to obtain the output. The trick within the reverse mode is that we start from right to left: we multiply a matrix by a vector, which yields another vector that is multiplied by the next matrix, and so on. Matrix-vector multiplication is much cheaper computationally that matrix-matrix multiplication, which makes backpropagation very popular in this space.

## Training neural networks via backpropagation

Earlier we showed how to calculate the cost as the difference between the activation of the last layer and the actual target class label. To get here, we need to apply forward propagation in order to obtain the activaiton of the output layer:

$$
\textbf{Z}^{h} = \textbf{A}^{in}\textbf{W}^{h} \ \mbox{(net input of the hidden layer)}
\\
\textbf{A}^{h} = \phi(\textbf{Z}^{h}) \ \mbox{(activation of the hidden layer)}
\\
\textbf{Z}^{out} = \textbf{A}^{h}\textbf{W}^{out} \ \mbox{(net input of the output layer)}
\\
\textbf{A}^{out} = \phi(\textbf{Z}^{out}) \ \mbox{(activation of the output layer)}
$$

Following that calculation, backpropagation is used to update the weights in the MLP model. The error is propagated from the right to the left. This start by first calculating the error vector of the output layer:

$$
\mathbf{\delta^{out}} = \mathbf{a^{out}} - \textbf{y}
$$

$\textbf{y}$ is the vector of the true class labels. In the homegrown implementation, this is the 'sigma_out variale. Next, calculate the rror term of the hidden layer:

$$
\mathbf{\delta^{h}} = \mathbf{\delta^{out}}\big(\mathbf{W^{out}}\big) \odot \frac{\partial\phi(z^h)}{\partial z^h}
$$

The right most term is just the derivative of the sigmoid activation function, which is:
$$
\frac{\partial\phi(z^h)}{\partial z^h} = \big(a^h \odot (1-a^h)\big)
$$

The symbol $\odot$ stands for element-wise multiplication, which is simply the multiplication of elements in separate matrices that share the same position (when using two equally sized matrices). Element-wise multiplication can also occur between a vector and a matrix as the the rows are equal.

After that, compute the $\delta^h$ layer error matrix (value 'sigma_h' in the homegrowm implementation):

$$
\delta^h = \delta^{out}\big(\textbf{W}^{out}\big)^T \odot \bigg(a^h \odot \big(1 - a^h\big)\bigg)
$$

The $\delta^h$ term was calculated by taking the transpose $\big(\textbf{W}^{out}\big)^T$ of the $h$ x $t$ dimensional matrix $\textbf{W}^{out}$. The value $t$ is the number of output class labels and $h$ is the number of hidden units. Multiplying $\big(\textbf{W}^{out}\big)^T$ by the $n$ x $t$ dimensional $\mathbf{\delta}^{out}$ results in an $n$ x $t$ dimensional matrix that is multiplied elementwise by the sigmoid function derivative that is also of $n$ x $t$ dimension to ultimately obtain $\delta^h$.

Now that we have all of the $\delta$ terms, we can write the derivation of the cost function as follows:

$$
\frac{\partial}{\partial w_{i,y}^{out}}\mathbf{J(W)} = a_j^h\delta_i^{out}
\\
\frac{\partial}{\partial w_{i,y}^{h}}\mathbf{J(W)} = a_j^{in}\delta_i^{h}
$$

Next, we need to accumulate the partial derivative of every node in each layer and the error of the node in the next layer. This requires that we compute $\Delta_{i,j}^l$ for every sample in the training set. This vectorized implementation of this is as follows:

$$
\Delta^h = \Delta^h + \big(\textbf{A}^{in}\big)^T \delta^h
\\
\Delta^{out} = \Delta^{out} + \big(\textbf{A}^{h}\big)^T \delta^{out}
$$

And after accumulating the partial derivatives, we can dd the regularization term:

$$
\Delta^t := \Delta^t + \lambda^l \ \mbox{(except for the bias term)}
$$

The two immediate mathematical expression above correspond to the variables 'delta_w_h',  'delta_b_h', 'delta_w_out' and 'delta_b_out'. Lastly, after computing the gradients, we can now updates the weights by taking an opposite step towards the gradient for each layer $l$:

$$
\textbf{W}^t := \textbf{W}^t - \eta\Delta^t
$$


```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 3

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Jan. 31 at **12:30pm**.

## Exercise: Income and Growth Across Countries

The data file `cross_country_gdp_pc.csv` contains annual data on GDP per capita for over 100 countries and is available here: https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/cross_country_gdp_pc.csv. The data are was constructed from the Penn World Table (https://www.rug.nl/ggdc/productivity/pwt/).

In this exercise, you will:

1. Analyze some basic facts about GDP per capita across the world.
2. Compute the average annual growth rate of GDP per capita for each country in the data.
3. Compute a linear regression of average GDP per capita growth on GDP per capita in the initial year.

Follow the instructions in the following cells.

### Part (a)

```python
# Create a variable called 'data' that stores the data in the file 'cross_country_gdp_pc.csv' in a DataFrame
# Make sure that 'Year', the left-most column of the data, is set to be the index column
data = pd.read_csv('https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/cross_country_gdp_pc.csv',index_col=0)

# Print the first 5 rows of data
data.head()
```

      Argentina - ARG
      Australia - AUS
      Austria - AUT
      Burundi - BDI
      Belgium - BEL
      Benin - BEN
      Burkina Faso - BFA
      Bangladesh - BGD
      Bolivia (Plurinational State of) - BOL
      Brazil - BRA
      ...
      Turkey - TUR
      Taiwan - TWN
      U.R. of Tanzania: Mainland - TZA
      Uganda - UGA
      Uruguay - URY
      United States - USA
      Venezuela (Bolivarian Republic of) - VEN
      South Africa - ZAF
      Zambia - ZMB
      Zimbabwe - ZWE

      Year

      1960
      2851.49
      14223.37
      8984.20
      676.57
      9100.09
      1616.84
      724.34
      1429.30
      1771.04
      2114.20
      ...
      4691.59
      2376.92
      1173.90
      789.13
      6731.78
      17462.63
      6749.85
      5913.81
      1665.98
      1990.92

      1961
      2897.14
      14303.40
      9427.85
      570.27
      9550.57
      1601.02
      681.06
      1391.08
      1785.90
      2338.23
      ...
      4649.28
      2442.46
      1187.66
      782.75
      6739.92
      17616.14
      6721.12
      6008.94
      1670.34
      2040.24

      1962
      2903.85
      14820.87
      9630.59
      607.52
      10008.56
      1522.49
      724.86
      1445.78
      1816.33
      2442.88
      ...
      4820.57
      2550.53
      1267.04
      778.84
      6725.06
      18395.60
      6972.68
      6119.67
      1581.47
      1960.41

      1963
      2758.01
      15573.81
      9960.43
      623.15
      10322.03
      1572.27
      698.74
      1505.08
      1889.88
      2533.68
      ...
      5163.09
      2725.46
      1330.11
      774.15
      6514.72
      18943.34
      7222.31
      6466.65
      1614.93
      1944.13

      1964
      2853.73
      16042.42
      10375.80
      611.82
      10947.25
      1622.50
      718.58
      1466.38
      1961.70
      2572.07
      ...
      5211.92
      2947.73
      1492.33
      816.41
      6683.54
      19808.61
      7757.37
      6738.99
      1792.59
      1836.15

5 rows × 111 columns

```python
# Each column contains data for a specific country. Print the number of countries (e.g., the length of data.columns)
print("Number of countries: ",len(data.columns))
```

    Number of countries:  111

```python
# Each column contains data for a specific year. Print the number of countries (e.g., the length of data.index)
print("Number of years of observations: ",len(data.index))
```

    Number of years of observations:  58

```python
# Print the values of the initial year and the final year in the data
print("Initial year: ",data.index[0])
print("Final year:   ",data.index[-1])
```

    Initial year:  1960
    Final year:    2017

```python
# Create a variable called 'data_initial' that is equal to the first row of the data DataFrame
data_initial = data.iloc[0]

# Create a variable called 'data_final' that is equal to the last row of the data DataFrame
data_final = data.iloc[-1]
```

```python
# Use the sort_values() method of data_initial to sort the Series in ascending order.
data_initial=data_initial.sort_values()

# Print the name and GDP per capita of the countries with the 10 lowest GDPs per capita in the initial year of the data
print('Countries with lowest GDP per capita in '+str(data.index[0])+':\n')
print(data_initial.head(10))
```

    Countries with lowest GDP per capita in 1960:

    Botswana - BWA         404.68
    Mozambique - MOZ       493.70
    Ethiopia - ETH         522.52
    Burundi - BDI          676.57
    Egypt - EGY            685.77
    Nepal - NPL            701.33
    Mali - MLI             703.97
    Malta - MLT            718.99
    Burkina Faso - BFA     724.34
    Guinea-Bissau - GNB    763.46
    Name: 1960, dtype: float64

```python
# Use the sort_values() method of data_final to sort the Series in ascending order.
data_final=data_final.sort_values()

# Print the name and GDP per capita of the countries with the 10 lowest GDPs per capita in the final year of the data
print('Countries with lowest GDP per capita in '+str(data.index[-1])+':\n')
print(data_final.head(10))
```

    Countries with lowest GDP per capita in 2017:

    Central African Republic - CAF     727.29
    D.R. of the Congo - COD            760.47
    Burundi - BDI                      799.61
    Niger - NER                        908.65
    Malawi - MWI                      1038.32
    Chad - TCD                        1307.97
    Mozambique - MOZ                  1376.55
    Togo - TGO                        1566.15
    Ethiopia - ETH                    1591.49
    Madagascar - MDG                  1591.55
    Name: 2017, dtype: float64

```python
# Print the names of the countries that are among the 10 with lowest GDPs per capita in BOTH the initial year and
# the final year.
print('Countries among lowest 10 GDPs per capita in '+str(data.index[0])+' and '+str(data.index[-1])+':\n')
for country in data_initial.head(10).index:
    if country in data_final.head(10).index:
        print(country.split("-")[0])
```

    Countries among lowest 10 GDPs per capita in 1960 and 2017:

    Mozambique
    Ethiopia
    Burundi

```python
# Print the name and GDP per capita of the countries with the 10 highest GDPs per capita in the initial year of the data
print('Countries with highest GDP per capita in both '+str(data.index[0])+':\n')
print(data_initial.tail(10))
```

    Countries with highest GDP per capita in both 1960:

    Netherlands - NLD       11461.35
    Sweden - SWE            11766.14
    United Kingdom - GBR    11909.77
    Denmark - DNK           12026.71
    New Zealand - NZL       12835.86
    Canada - CAN            13360.91
    Australia - AUS         14223.37
    Luxembourg - LUX        15757.45
    United States - USA     17462.63
    Switzerland - CHE       20086.09
    Name: 1960, dtype: float64

```python
# Print the name and GDP per capita of the countries with the 10 highest GDPs per capita in the final year of the data
print('Countries with highest GDP per capita in '+str(data.index[-1])+':\n')
print(data_final.tail(10))
```

    Countries with highest GDP per capita in 2017:

    Austria - AUT          46103.27
    Netherlands - NLD      46601.56
    Australia - AUS        47392.62
    Germany - DEU          47556.00
    United States - USA    54794.76
    Norway - NOR           60590.25
    Switzerland - CHE      64219.30
    Singapore - SGP        67138.11
    Luxembourg - LUX       74605.23
    Ireland - IRL          75916.08
    Name: 2017, dtype: float64

```python
# Print the names of the countries that are among the 10 with highest GDPs per capita in BOTH the initial year
# and the final year
print('Countries among highest 10 GDPs per capita in both '+str(data.index[0])+' and '+str(data.index[-1])+':\n')
for country in data_initial.tail(10).index:
    if country in data_final.tail(10).index:
        print(country.split("-")[0])
```

    Countries among highest 10 GDPs per capita in both 1960 and 2017:

    Netherlands
    Australia
    Luxembourg
    United States
    Switzerland

### Part (b)

Let $y_t$ denotes GDP per capita for some country in some year $t$ and let $g$ denotes the average annual growth in GDP per capita between years 0 and $T$. $g$ is defined by:
\begin{align}
y_T & = (1+g)^T y_0
\end{align}
which implies:
\begin{align}
g & = \left(\frac{y_T}{y_0}\right)^{1/T} - 1
\end{align}
We can use this equation to compute the aveage growth rate of GDP per capita for each country in our data. Note that for our data, $T$ is equal to `len(data.index)-1`.

```python
# Create a variable called 'growth_rates' that contains the average growth rate of each country in the data.
# NOTE: You do not need to re-sort the variables data_initial and data_final. If their indices don't align,
# Pandas will automatically sort their indices alphabetically.
T = len(data.index)-1
growth_rates = (data_final/data_initial)**(1/T)-1

# Use the sort_values() method of growth_rates to sort the Series in ascending order.
growth_rates=growth_rates.sort_values()
```

```python
# Print the average growth rate of GDP per capita for all countries rounded to 4 decimal places
print('Average growth rate of GDP per capita for all countries: ',round(growth_rates.mean(),4))
```

    Average growth rate of GDP per capita for all countries:  0.0224

```python
# Print the standard deviation of the growth rate of GDP per capita for all countries rounded to 4 decimal places
print('Standard deviation of the growth rate of GDP per capita for all countries: ',round(growth_rates.std(),4))
```

    Standard deviation of the growth rate of GDP per capita for all countries:  0.0159

```python
# Print the name and growth rates of the countries with the 10 lowest growth rates of GDP per capita
print('Countries with lowest growth rates of GDP per capita:\n')
print(growth_rates.head(10))
```

    Countries with lowest growth rates of GDP per capita:

    D.R. of the Congo - COD          -0.022039
    Central African Republic - CAF   -0.011405
    Niger - NER                      -0.007151
    Guinea - GIN                     -0.004706
    Zimbabwe - ZWE                   -0.001082
    Chad - TCD                       -0.000086
    Gambia - GMB                      0.000916
    Nigeria - NGA                     0.001056
    Madagascar - MDG                  0.001781
    Barbados - BRB                    0.001924
    dtype: float64

```python
# Print the name and growth rates of the countries with the 10 highest growth rates of GDP per capita
print('Countries with highest growth rates of GDP per capita:\n')
print(growth_rates.tail(10))
```

    Countries with highest growth rates of GDP per capita:

    China - CHN                0.045830
    Thailand - THA             0.048224
    Equatorial Guinea - GNQ    0.048863
    Egypt - EGY                0.050227
    Romania - ROU              0.051858
    Taiwan - TWN               0.052199
    Singapore - SGP            0.058379
    Republic of Korea - KOR    0.063021
    Botswana - BWA             0.065429
    Malta - MLT                0.068160
    dtype: float64

```python
# Print the names of the countries that were BOTH among the 10 with lowest GDPs per capita in the initial year and
# among the 10 with the highest growth of GDP per capita.
print('Countries among highest 10 GDPs per capita in both '+str(data.index[0])+' and '+str(data.index[-1])+':\n')
for country in growth_rates.tail(10).index:
    if country in data_initial.head(10).index:
        print(country.split("-")[0])
```

    Countries among highest 10 GDPs per capita in both 1960 and 2017:

    Egypt
    Botswana
    Malta

```python
# Use the sort_index() method of data_initial to alphabetize the index [e.g., data_initial = data_initial.sort_index()]
data_initial = data_initial.sort_index()

# Use the sort_index() method of growth_rates to alphabetize the index
growth_rates = growth_rates.sort_index()

# Construct a scatter plot with:
#     1. GDP per capita in the initial year on the horizontal axis
#     2. Average growth rate of GDP per capita on the vertical axis
#     3. Size of scatter plot markers at least 50
#     4. Opacity of scatter plot markers no greater than 0.5
#     5. x-axis limit: [0,20000]
#     6. Suitable title and labels for the axes

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(data_initial,growth_rates,s=100,alpha=0.2)
ax.set_title('Income versus Growth for '+str(len(data.columns))+' Countries')
ax.set_xlabel('GDP per capita in '+str(data.index[0])+'\n(Thousands of $)')
ax.set_ylabel('Growth rate of GDP per capita\nfrom '+str(data.index[0])+' to '+str(data.index[-1]))
ax.set_xlim([0,20000])
ax.grid()
```

**Question**

1. Describe in words the relationship between GDP per capita in the initial year and average growth of GDP per capita. How is the relationship different for low income countries and high income countries?

**Answer**

1. There is a right-pointing triangle-shaped relationship between GDP per capita in the initial year and average growth of GDP per capita. Countries with high GDP per capita in the initial year had average growth rates that were almost uniformly close to the world average of about 2 percent. Countries with lower GDP per capita in the initial year had much more variation across the average growth rates of their GDP per capita. The countries with the highest growth rates of GDP per capita were among the lowest income initially and the countries with the lowest growth rates of GDP per capita were also among the lowest income initially.

### Part (c)

In this part you will use OLS to estimate the relationship between GDP per capita in the initial year and average growth of GDP per capita.

\begin{align}
g_i & = \beta_0 + \beta_{\text{1}}y_i + \epsilon_i,
\end{align}

where $g_i$ is the average annual growth rate of country $i$, $y_i$ is country $i$'s initial GDP per capita, and $\epsilon_i$ is the residual of the regression.

```python
growth_rates = growth_rates.sort_index()
data_initial = data_initial.sort_index()

# Create a variable called 'y' that is equal to the average growth rates of the countries
y = growth_rates

# Create a variable called x' that is equal to the GDP per capita of the countries in the initial year
x = data_initial

# Use the Statsmodels function add_constant() to add a constant column to the variable x
x = sm.add_constant(data_initial)

# Create a variable called 'model' equal to the output of the Statsmodels OLS function regressing y upon x
model = sm.OLS(y,x)

# Create a variable called 'results' that stores the fitted model (i.e., the output of the fit() method of model)
results = model.fit()

# Print the summary of results generated by the summary() method of results
print(results.summary2())
```

                      Results: Ordinary least squares
    ===================================================================
    Model:              OLS              Adj. R-squared:     -0.009
    Dependent Variable: y                AIC:                -601.5469
    Date:               2020-01-24 10:32 BIC:                -596.1278
    No. Observations:   111              Log-Likelihood:     302.77
    Df Model:           1                F-statistic:        0.01452
    Df Residuals:       109              Prob (F-statistic): 0.904
    R-squared:          0.000            Scale:              0.00025479
    ----------------------------------------------------------------------
              Coef.     Std.Err.       t       P>|t|      [0.025    0.975]
    ----------------------------------------------------------------------
    const     0.0226      0.0022    10.3406    0.0000     0.0182    0.0269
    1960     -0.0000      0.0000    -0.1205    0.9043    -0.0000    0.0000
    -------------------------------------------------------------------
    Omnibus:                4.381        Durbin-Watson:           1.666
    Prob(Omnibus):          0.112        Jarque-Bera (JB):        3.995
    Skew:                   0.318        Prob(JB):                0.136
    Kurtosis:               3.678        Condition No.:           8515
    ===================================================================
    * The condition number is large (9e+03). This might indicate
    strong multicollinearity or other numerical problems.

    /Users/bcjenkin/opt/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2495: FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
      return ptp(axis=axis, out=out, **kwargs)

**Questions**

1. Interpret the estimate for the coefficient on $y_i$.
2. Interpret the estimate for the constant. Where did we see this number earlier?
3. Interpret the R-squared of the regression.

**Answers**

1. The estimated coefficient on $y_i$ is both very close to zero and is not statistically different from zero.
2. The estimated constant is equal to the average growth rate of GDP per capita across all countries and we computed that in the second code cell of Part (b).
3. The R-squared implies that the linear regression model explains essentially zero percent of the variation in average GDP per capita growth across countries.

## Exercise: Income and Growth Across US States

The data file `state_income_data.csv` contains annual data on income per capita for the 48 continental states, Washington D.C., and the United States as a whole and is available here: https://raw.githubusercontent.com/letsgoexploring/economic-data/master/us-convergence/csv/state_income_data.csv.

In this exercise, you will:

1. Analyze some basic facts about income per capita among US states.
2. Compute the average annual growth rate of income per capita for each state in the data.
3. Compute a linear regression of average income per capita growth on income per capita in the initial year.

Follow the instructions in the following cells.

### Part (a)

```python
# Create a variable called 'state_df' that stores the data in the file 'cross_country_gdp_pc.csv' in a DataFrame
# Make sure that 'Year', the left-most column of the data, is set to be the index column
state_df= pd.read_csv('https://raw.githubusercontent.com/letsgoexploring/economic-data/master/us-convergence/csv/state_income_data.csv',index_col=0)

# Print the first 5 rows of data
print(state_df.head())
```

                   AL           AR           AZ            CA           CO  \
    1840   915.824576  1175.020210          NaN           NaN          NaN
    1880  1075.181892  1190.379951  4550.323363   3724.737268  4243.128537
    1900  1285.610348  1306.345998  4147.130154   3753.152789  3628.738884
    1929  3223.520000  3051.740000  6042.850000  10034.360000  6366.210000
    1930  2758.260000  2349.240000  5432.620000   9323.540000  6019.930000

                    CT        DC            DE           FL           GA  ...  \
    1840   1572.453517       NaN   1175.020210  1192.299919   984.943411  ...
    1880   2803.152789       NaN   2073.565077   921.584479  1075.181892  ...
    1900   3234.761520       NaN   2591.956346  1389.288601  1161.196443  ...
    1929  10398.140000  13369.04  10418.350000  5244.540000  3466.050000  ...
    1930   9711.590000  13780.81   8998.430000  4897.750000  3177.770000  ...

                   TN           TX           UT  United States           VA  \
    1840   812.146322          NaN          NaN    1123.181083   933.104285
    1880   998.383185  1151.980598  1555.173808    1823.969281   979.183508
    1900  1264.874697  1741.794665  2363.864188    2343.128537  1368.552951
    1929  3769.200000  4779.710000  5537.590000    7063.460000  4395.720000
    1930  3366.540000  4268.480000  5180.910000    6512.850000  4058.730000

                   VT           WA           WI           WV           WY
    1840  1123.181083          NaN  1382.376718   933.104285          NaN
    1880  1996.766370  2687.954729  1804.769604  1036.782538  2956.750202
    1900  2446.806791  3753.152789  2218.714632  1638.116411  4914.349232
    1929  6366.210000  7477.770000  6770.410000  4628.130000  6790.620000
    1930  5998.950000  6890.400000  6124.800000  4268.480000  6103.830000

    [5 rows x 50 columns]

The DataFrame method `dropna()` returns a DataFrame that omits rows that contain missing (NaN means *not a number*) values. The following command will remove the rows with missing values from the DataFrame `df`:

    df =  df.dropna()

The DataFrame method `drop()` returns a DataFrame that omits rows or columns with a given label. The following command will remove the column named `'LABEL'` from the DataFrame `df`:

    df = df.drop('LABEL',axis=1)

The argument `axis=1` means to look among the columns for `'LABEL'`. If we had wanted to remove a row with the index value `'LABEL'`, then the axis argument would have been: `axis=0`.

```python
# Remove the rows with missing data from state_df
state_df = state_df.dropna()

# Remove the column named 'United States' from state_df
state_df = state_df.drop('United States',axis=1)

# Set state_df equal to itself divided by 1000 so that its units are in thousands of dollars
state_df = state_df/1000

# Print the first 5 rows of data
print(state_df.head())
```

               AL       AR       AZ        CA       CO        CT        DC  \
    1929  3.22352  3.05174  6.04285  10.03436  6.36621  10.39814  13.36904
    1930  2.75826  2.34924  5.43262   9.32354  6.01993   9.71159  13.78081
    1931  2.57160  2.41964  5.02630   8.80187  5.51724   9.42139  14.64641
    1932  2.10512  2.01245  4.24997   7.73203  4.71336   8.30134  14.65643
    1933  2.23281  2.08305  4.20694   7.47447  4.80599   8.03268  12.86590

                DE       FL       GA  ...       SD       TN       TX       UT  \
    1929  10.41835  5.24454  3.46605  ...  4.19361  3.76920  4.77971  5.53759
    1930   8.99843  4.89775  3.17777  ...  3.74410  3.36654  4.26848  5.18091
    1931   9.05903  4.64056  2.96902  ...  2.82876  3.19112  4.03273  4.34833
    1932   7.82471  4.19701  2.60824  ...  2.50232  2.56852  3.46882  4.02489
    1933   7.70592  3.93465  2.75017  ...  1.85160  2.73656  3.45813  4.04357

               VA       VT       WA       WI       WV       WY
    1929  4.39572  6.36621  7.47777  6.77041  4.62813  6.79062
    1930  4.05873  5.99895  6.89040  6.12480  4.26848  6.10383
    1931  4.33665  5.50555  6.24196  5.48217  4.13793  5.57569
    1932  3.78657  4.79280  5.32239  4.79280  3.38938  4.96491
    1933  3.89381  4.58816  5.11913  4.53370  3.51259  5.06467

    [5 rows x 49 columns]

```python
# Create a variable called 'state_growth_rates' that contains the average growth rate of each state in the data.
T = len(state_df)-1
state_growth_rates = (state_df.iloc[-1]/state_df.iloc[0])**(1/T)-1
```

```python
# Construct a scatter plot with:
#     1. GDP per capita in the initial year on the horizontal axis
#     2. Average growth rate of GDP per capita on the vertical axis
#     3. Size of scatter plot markers at least 50
#     4. Opacity of scatter plot markers no greater than 0.5
#     5. y-axis limit: [0.015,0.035]
#     6. Suitable title and labels for the axes

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(state_df.iloc[0],state_growth_rates,s=100,alpha=0.2)
ax.set_title('Income versus Growth for '+str(len(state_df.columns))+' States incl. DC')
ax.set_xlabel('Income per capita in '+str(state_df.index[0])+'\n(Thousands of $)')
ax.set_ylabel('Growth rate of income per capita\nfrom '+str(state_df.index[0])+' to '+str(state_df.index[-1]))
ax.set_ylim([0.015,0.035])
ax.grid()
```

### Part (b)

Like you did for the country data above, use OLS to estimate the relationship between income per capita in the initial year and average growth of income per capita.

\begin{align}
g_i & = \beta_0 + \beta_{\text{1}}y_i + \epsilon_i,
\end{align}

where $g_i$ is the average annual growth rate of country $i$, $y_i$ is state $i$'s initial income per capita, and $\epsilon_i$ is the residual of the regression.

```python
# Create a variable called 'y' that is equal to the average growth rates of the states
y = state_growth_rates

# Create a variable called x' that is equal to the income per capita of the states in the initial year
x = state_df.iloc[0]

# Use the Statsmodels function add_constant() to add a constant column to the variable x
x = sm.add_constant(x)

# Create a variable called 'model' equal to the output of the Statsmodels OLS function regressing y upon x
model = sm.OLS(y,x)

# Create a variable called 'results' that stores the fitted model (i.e., the output of the fit() method of model)
results = model.fit()

# Print the summary of results generated by the summary() method of results
print(results.summary2())
```

                      Results: Ordinary least squares
    ===================================================================
    Model:              OLS              Adj. R-squared:     0.757
    Dependent Variable: y                AIC:                -483.2704
    Date:               2020-01-24 10:32 BIC:                -479.4867
    No. Observations:   49               Log-Likelihood:     243.64
    Df Model:           1                F-statistic:        150.9
    Df Residuals:       47               Prob (F-statistic): 2.82e-16
    R-squared:          0.762            Scale:              2.9300e-06
    ---------------------------------------------------------------------
                Coef.    Std.Err.      t       P>|t|     [0.025    0.975]
    ---------------------------------------------------------------------
    const       0.0309     0.0007    45.3635   0.0000    0.0295    0.0323
    1929       -0.0012     0.0001   -12.2826   0.0000   -0.0014   -0.0010
    -------------------------------------------------------------------
    Omnibus:                1.395        Durbin-Watson:           1.945
    Prob(Omnibus):          0.498        Jarque-Bera (JB):        1.323
    Skew:                   0.377        Prob(JB):                0.516
    Kurtosis:               2.720        Condition No.:           19
    ===================================================================

**Questions**

1. Interpret the estimate for the coefficient on $y_i$.
2. Interpret the R-squared of the regression.
3. Why do you think that initial income and average growth are so strongly correlated in the state data but not so in the country data? (If you're not sure, referesh your knowledge about the difference between *conditional* and *unconditional* convergence: https://en.wikipedia.org/wiki/Convergence_(economics)#Types_of_Convergence)

**Answers**

1. The estimated coefficient on $y_i$ implies that a \$1,000 increase in income per captia reduces a states average growth rate of income per capita by 1.2 percent. I.e., states with lower initial income had greater income growth than higher income states.
2. The R-squared implies that the linear regression model explains about 76 percent of the variation in average income per capita growth across states.
3. Lower-income US states are "economically similar" to higher-income states in that they have the same currency, federal regulation, legal traditions, and language, and they have similar laws, poltical institutions. Therefore, it's not surprising to observe *catch-up growth* among the lower-income states. By contrast, lower-income countries are a diverse group with broad differences in policies and institutions and so it is not at all guaranteed that low income countries will enjoy catch-up growth simply by virtue of their low income.


# NODDI Bingham

The recently proposed NODDI-Bingham model *(Tariq et al. 2016)* extends NODDI to using a Bingham distribution instead of a Watson and describes the signal as

\begin{align}
 E^{\textrm{NODDI}}_{\textrm{Bingham}}= \underbrace{f_{\textrm{CSF}}\overbrace{E_{\textrm{iso}}(\cdot|D_{\textrm{CSF}})}^{\textrm{Ball}}}_{\textrm{CSF}}
                             +\overbrace{B(\kappa_1,\kappa_2,\boldsymbol{\mu}_i)}^{\textrm{Bingham}}\,*_{\mathbb{S}^2}\, \left[\underbrace{f_h\overbrace{E_{\textrm{h}}(\cdot|\lambda_\perp^{\textrm{tort}},\lambda_\parallel^{\textrm{tort}})}^{\textrm{Zeppelin}}}_{\textrm{Hindered Extra-Axonal}}+\underbrace{f_r\overbrace{E_r(\cdot|\lambda_\parallel)}^{\textrm{Stick}}}_{\textrm{Intra-Axonal}}\right].
\end{align}

The added value of the Bingham distribution in NODDI has been shown in an ex-vivo study on bending cortical fibers *(Tariq et al. 2015)*.
However, it does not address any of the other limitations of NODDI. It is also interesting to see that this model is equivalent to the BAR model if we set $N=1$ and add a tortuous Zeppelin.

## Generate NODDI-Bingham model

Setting up the NODDI-Bingham model in Dmipy is straight-forward, and only differs one line from generating NODDI-Watson (calling a Bingham instead of a Watson distribution).

First we call the basic Ball, Stick and Zeppelin components:

```python
from dmipy.signal_models import cylinder_models, gaussian_models
ball = gaussian_models.G1Ball()
stick = cylinder_models.C1Stick()
zeppelin = gaussian_models.G2Zeppelin()
```

Then we call the BinghamDistributed instead of the WatsonDistributed model:

```python
from dmipy.distributions.distribute_models import SD2BinghamDistributed
bingham_dispersed_bundle = SD2BinghamDistributed(models=[stick, zeppelin])
bingham_dispersed_bundle.parameter_names
```

    ['SD2Bingham_1_odi',
     'SD2Bingham_1_beta_fraction',
     'G2Zeppelin_1_lambda_perp',
     'SD2Bingham_1_psi',
     'SD2Bingham_1_mu',
     'G2Zeppelin_1_lambda_par',
     'C1Stick_1_lambda_par',
     'partial_volume_0']

We set NODDI-Bingham's parameter assumptions: The tortuosity constraint, equal $\lambda_\parallel$ between the Stick and Zeppelin, and fixing the diffusivity to 1.7e-9 $m^2/s$.

```python
bingham_dispersed_bundle.set_tortuous_parameter('G2Zeppelin_1_lambda_perp','C1Stick_1_lambda_par','partial_volume_0')
bingham_dispersed_bundle.set_equal_parameter('G2Zeppelin_1_lambda_par', 'C1Stick_1_lambda_par')
bingham_dispersed_bundle.set_fixed_parameter('G2Zeppelin_1_lambda_par', 1.7e-9)
```

The Bingham-dispersed Stick and Ball and then combined in a multi-compartment model:

```python
from dmipy.core.modeling_framework import MultiCompartmentModel
NODDI_bingham_mod = MultiCompartmentModel(models=[ball, bingham_dispersed_bundle])
```

```python
NODDI_bingham_mod.parameter_names
```

    ['G1Ball_1_lambda_iso',
     'SD2BinghamDistributed_1_SD2Bingham_1_odi',
     'SD2BinghamDistributed_1_SD2Bingham_1_beta_fraction',
     'SD2BinghamDistributed_1_SD2Bingham_1_mu',
     'SD2BinghamDistributed_1_partial_volume_0',
     'SD2BinghamDistributed_1_SD2Bingham_1_psi',
     'partial_volume_0',
     'partial_volume_1']

And we finally the set isotropic diffusivity of the ball to free water diffusivity:

```python
NODDI_bingham_mod.set_fixed_parameter('G1Ball_1_lambda_iso', 3e-9)
```

Visualize the model:

```python
from IPython.display import Image
NODDI_bingham_mod.visualize_model_setup(view=False, cleanup=False)
Image('Model Setup.png')
```

# Human Connectome Project Example

```python
from dmipy.data import saved_data
scheme_hcp, data_hcp = saved_data.wu_minn_hcp_coronal_slice()
```

    This data slice originates from Subject 100307 of the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.

```python
import matplotlib.pyplot as plt
import matplotlib.patches as patches
%matplotlib inline

fig, ax = plt.subplots(1)
ax.imshow(data_hcp[:, 0, :, 0].T, origin=True)
rect = patches.Rectangle((70,70),20,20,linewidth=1,edgecolor='r',facecolor='none')
ax.add_patch(rect)
ax.set_axis_off()
ax.set_title('HCP coronal slice B0 with ROI');
```

## Fit NODDI to HCP data
Having pathos and numba installed, fitting the model to the coronal slice takes about 40 minutes.

```python
NODDI_fit_hcp = NODDI_bingham_mod.fit(scheme_hcp, data_hcp, mask=data_hcp[..., 0]>0)
```

    Using parallel processing with 8 workers.
    Setup brute2fine optimizer in 61.3360249996 seconds
    Fitting of 8181 voxels complete in 2200.6279161 seconds.
    Average of 0.268992533443 seconds per voxel.

## Visualize results

```python
fitted_parameters = NODDI_fit_hcp.fitted_parameters

fig, axs = plt.subplots(2, 3, figsize=[15, 10])
axs = axs.ravel()

counter = 0
for name, values in fitted_parameters.items():
    if values.squeeze().ndim != 2:
        continue
    cf = axs[counter].imshow(values.squeeze().T, origin=True, interpolation='nearest')
    axs[counter].set_title(name)
    fig.colorbar(cf, ax=axs[counter], shrink=0.5)
    counter += 1
```

## Estimating Fiber Orientation Distributions (FODs)

We estimate FODs on the right side of the corpus callosum and use the estimated ODI and beta_fraction as background.

```python
from dipy.data import get_sphere
from dipy.viz.actor import slicer
sphere = get_sphere(name='symmetric724')
fods = NODDI_fit_hcp.fod(sphere.vertices)
```

```python
import numpy as np
affine = np.eye(4)
affine[0,3] = -10
affine[1,3] = -10

odi_res = fitted_parameters['SD2BinghamDistributed_1_SD2Bingham_1_odi']
odi_im = slicer(odi_res[70:90,0, 70:90, None],interpolation='nearest', affine=affine, opacity=0.7)

beta_res = fitted_parameters['SD2BinghamDistributed_1_SD2Bingham_1_beta_fraction']
beta_im = slicer(beta_res[70:90,0, 70:90],interpolation='nearest', affine=affine, opacity=0.7)
```

```python
from dipy.viz import fvtk
ren = fvtk.ren()
fod_spheres = fvtk.sphere_funcs(fods[70:90,:, 70:90], sphere, scale=1., norm=False)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
fvtk.add(ren, fod_spheres)
fvtk.add(ren, odi_im)
fvtk.record(ren=ren, size=[700, 700], out_path='dipy_odi.png')

ren = fvtk.ren()
fod_spheres = fvtk.sphere_funcs(fods[70:90,:, 70:90], sphere, scale=1., norm=False)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
fvtk.add(ren, fod_spheres)
fvtk.add(ren, beta_im)
fvtk.record(ren=ren, size=[700, 700], out_path='dipy_beta.png')
```

```python
import matplotlib.image as mpimg
img1 = mpimg.imread('dipy_odi.png')
img2 = mpimg.imread('dipy_beta.png')

fig, axs = plt.subplots(1, 2, figsize=[20, 70])
axs[0].set_title('FODs with ODI background', fontsize=20)
axs[0].imshow(img1[100:-97, 100:-85])
axs[0].axis('off')
axs[1].set_title('FODs with beta_fraction background', fontsize=20)
axs[1].imshow(img2[100:-97, 100:-85])
axs[1].axis('off');
```

You can see the corpus callosum coming in the from the left and joining the centrum semiovale on the right. On the left, the intensity of the background shows the ODI of the Bingam distribution, inversely related to only $\kappa_1$. Notice it is low in the corpus callosum and higher in non-white matter areas. Also notice that NODDI-Bingham still cannot estimate crossings, which are expected in the centrum semiovale on the right. This biases any comparison/estimation of volume fractions and concentrations in these areas.

On the right, we show the FODs with beta_fraction as the background, which we related to $\kappa_1$ and $\kappa_2$ as $\kappa_2$=beta_fraction$*\kappa_1$. Since we limit beta_fraction between 0 and 1, this means that $\kappa_2$ is always limited between 0 and $\kappa_1$. Bright intensities indicate voxels where $\kappa_2$ is close to $\kappa_1$, meaning the axon dispersion is significantly anisotropic.

```python
from os import remove
remove('dipy_odi.png')
remove('dipy_beta.png')
```

## Estimating Error Metrics: MSE and $R^2$

Whenever you fit a model, it's always wise to check out the error of the model fit. Here we estimate the Mean Squared Error (MSE) and the $R^2$ coefficient of determination. In MSE the lower is better, while $R^2$ has an upper bound of one, which represents a perfect model fit.

```python
mse = NODDI_fit_hcp.mean_squared_error(data_hcp)
R2 = NODDI_fit_hcp.R2_coefficient_of_determination(data_hcp)

fig, axs = plt.subplots(1, 2, figsize=[15, 15])
cf = axs[0].imshow(mse.squeeze().T, origin=True, vmax=0.02, cmap='Greys_r')
fig.colorbar(cf, ax=axs[0], shrink=0.33)
axs[0].set_title('Mean Squared Error', fontsize=20)
cf = axs[1].imshow(R2.squeeze().T, origin=True, vmin=0.6, cmap='Greys_r')
fig.colorbar(cf, ax=axs[1], shrink=0.33)
axs[1].set_title('R2 - Coefficient of Determination', fontsize=20);
```

Ideally, you want the MSE and $R^2$ to show no structure of the underlying data - which would mean the model can at least fit the data equally well everywhere. We can still see the structure of different brain structure in NODDI-Bingham, but they are becoming less obvious to see than in Ball and Stick or NODDI-Watson.

When doing these studies, ALWAYS verify whether differences in estimated parameters reflect actual tissue changes, or only reflect the limitations of the model itself.

## Comparison MIX versus brute2fine optimization for FOD example

Given that NODDI-Bingham is starting to have many parameters, it is interesting to see if the Microstructure in Crossings (MIX) optimization algorithms will find the same result as our relatively straightforwardly implemented Brute2Fine algorithm. Since MIX takes considerably longer to fit, we will only study the are where we illustrated the FODs.

```python
data_hcp_small = data_hcp[70:90,: , 70:90]
NODDI_fit_hcp_small = NODDI_bingham_mod.fit(scheme_hcp, data_hcp_small, solver='mix')
```

    Using parallel processing with 8 workers.
    Setup MIX optimizer in 5.96046447754e-06 seconds
    Fitting of 400 voxels complete in 1390.47426891 seconds.
    Average of 3.47618567228 seconds per voxel.

```python
fods_mix = NODDI_fit_hcp_small.fod(sphere.vertices)
odi_res = NODDI_fit_hcp_small.fitted_parameters['SD2BinghamDistributed_1_SD2Bingham_1_odi']
odi_im = slicer(odi_res[:,0],interpolation='nearest', affine=affine, opacity=0.7)

beta_res = NODDI_fit_hcp_small.fitted_parameters['SD2BinghamDistributed_1_SD2Bingham_1_beta_fraction']
beta_im = slicer(beta_res[:,0],interpolation='nearest', affine=affine, opacity=0.7)
```

```python
from dipy.viz import fvtk
ren = fvtk.ren()
fod_spheres = fvtk.sphere_funcs(fods_mix, sphere, scale=1., norm=False)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
fvtk.add(ren, fod_spheres)
fvtk.add(ren, odi_im)
fvtk.record(ren=ren, size=[700, 700], out_path='dipy_odi.png')

ren = fvtk.ren()
fod_spheres = fvtk.sphere_funcs(fods_mix, sphere, scale=1., norm=False)
fod_spheres.RotateX(90)
fod_spheres.RotateZ(180)
fod_spheres.RotateY(180)
fvtk.add(ren, fod_spheres)
fvtk.add(ren, beta_im)
fvtk.record(ren=ren, size=[700, 700], out_path='dipy_beta.png')
```

```python
import matplotlib.image as mpimg
img1 = mpimg.imread('dipy_odi.png')
img2 = mpimg.imread('dipy_beta.png')

fig, axs = plt.subplots(1, 2, figsize=[20, 70])
axs[0].set_title('MIX - FODs with ODI background', fontsize=20)
axs[0].imshow(img1[100:-97, 100:-85])
axs[0].axis('off')
axs[1].set_title('MIX - FODs with beta_fraction background', fontsize=20)
axs[1].imshow(img2[100:-97, 100:-85])
axs[1].axis('off');
```

```python
from os import remove
remove('dipy_odi.png')
remove('dipy_beta.png')
```

Comparing this image and the one above we can see that the ODI image is very similar, but MIX seems to do a better job at obtaining a smoother beta_fraction map. This is likely due to that Brute2Fine currently insufficiently samples the psi parameter of the Bingham distribution in the Brute step, meaning it will sometimes fall in a local minimum, while MIX uses stochastic sampling over the entire parameter space. Increasing the sampling "Ns" parameter will likely improve results.

## References
- Tariq, Maira, et al. "Bingham–NODDI: mapping anisotropic orientation dispersion of neurites using diffusion MRI." NeuroImage 133 (2016): 207-223.
- Tariq, M., et al. "Validation of NODDI estimation of dispersion anisotropy in V1 of the human neocortex." International Society for Magnetic Resonance in Medicine (ISMRM), 2015.


```python
%matplotlib inline
```

# Scaling the regularization parameter for SVCs

The following example illustrates the effect of scaling the
regularization parameter when using `svm` for
`classification <svm_classification>`.
For SVC classification, we are interested in a risk minimization for the
equation:

\begin{align}C \sum_{i=1, n} \mathcal{L} (f(x_i), y_i) + \Omega (w)\end{align}

where

    - $C$ is used to set the amount of regularization
    - $\mathcal{L}$ is a `loss` function of our samples
      and our model parameters.
    - $\Omega$ is a `penalty` function of our model parameters

If we consider the loss function to be the individual error per
sample, then the data-fit term, or the sum of the error for each sample, will
increase as we add more samples. The penalization term, however, will not
increase.

When using, for example, `cross validation <cross_validation>`, to
set the amount of regularization with `C`, there will be a
different amount of samples between the main problem and the smaller problems
within the folds of the cross validation.

Since our loss function is dependent on the amount of samples, the latter
will influence the selected value of `C`.
The question that arises is `How do we optimally adjust C to
account for the different amount of training samples?`

The figures below are used to illustrate the effect of scaling our
`C` to compensate for the change in the number of samples, in the
case of using an `l1` penalty, as well as the `l2` penalty.

l1-penalty case
-----------------
In the `l1` case, theory says that prediction consistency
(i.e. that under given hypothesis, the estimator
learned predicts as well as a model knowing the true distribution)
is not possible because of the bias of the `l1`. It does say, however,
that model consistency, in terms of finding the right set of non-zero
parameters as well as their signs, can be achieved by scaling
`C1`.

l2-penalty case
-----------------
The theory says that in order to achieve prediction consistency, the
penalty parameter should be kept constant
as the number of samples grow.

Simulations
------------

The two figures below plot the values of `C` on the `x-axis` and the
corresponding cross-validation scores on the `y-axis`, for several different
fractions of a generated data-set.

In the `l1` penalty case, the cross-validation-error correlates best with
the test-error, when scaling our `C` with the number of samples, `n`,
which can be seen in the first figure.

For the `l2` penalty case, the best result comes from the case where `C`
is not scaled.

.. topic:: Note:

    Two separate datasets are used for the two different plots. The reason
    behind this is the `l1` case works better on sparse data, while `l2`
    is better suited to the non-sparse case.

```python
print(__doc__)

# Author: Andreas Mueller <amueller@ais.uni-bonn.de>
#         Jaques Grobler <jaques.grobler@inria.fr>
# License: BSD 3 clause

import numpy as np
import matplotlib.pyplot as plt

from sklearn.svm import LinearSVC
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import GridSearchCV
from sklearn.utils import check_random_state
from sklearn import datasets

rnd = check_random_state(1)

# set up dataset
n_samples = 100
n_features = 300

# l1 data (only 5 informative features)
X_1, y_1 = datasets.make_classification(n_samples=n_samples,
                                        n_features=n_features, n_informative=5,
                                        random_state=1)

# l2 data: non sparse, but less features
y_2 = np.sign(.5 - rnd.rand(n_samples))
X_2 = rnd.randn(n_samples, n_features // 5) + y_2[:, np.newaxis]
X_2 += 5 * rnd.randn(n_samples, n_features // 5)

clf_sets = [(LinearSVC(penalty='l1', loss='squared_hinge', dual=False,
                       tol=1e-3),
             np.logspace(-2.3, -1.3, 10), X_1, y_1),
            (LinearSVC(penalty='l2', loss='squared_hinge', dual=True,
                       tol=1e-4),
             np.logspace(-4.5, -2, 10), X_2, y_2)]

colors = ['navy', 'cyan', 'darkorange']
lw = 2

for fignum, (clf, cs, X, y) in enumerate(clf_sets):
    # set up the plot for each regressor
    plt.figure(fignum, figsize=(9, 10))

    for k, train_size in enumerate(np.linspace(0.3, 0.7, 3)[::-1]):
        param_grid = dict(C=cs)
        # To get nice curve, we need a large number of iterations to
        # reduce the variance
        grid = GridSearchCV(clf, refit=False, param_grid=param_grid,
                            cv=ShuffleSplit(train_size=train_size,
                                            n_splits=250, random_state=1))
        grid.fit(X, y)
        scores = grid.cv_results_['mean_test_score']

        scales = [(1, 'No scaling'),
                  ((n_samples * train_size), '1/n_samples'),
                  ]

        for subplotnum, (scaler, name) in enumerate(scales):
            plt.subplot(2, 1, subplotnum + 1)
            plt.xlabel('C')
            plt.ylabel('CV Score')
            grid_cs = cs * float(scaler)  # scale the C's
            plt.semilogx(grid_cs, scores, label="fraction %.2f" %
                         train_size, color=colors[k], lw=lw)
            plt.title('scaling=%s, penalty=%s, loss=%s' %
                      (name, clf.penalty, clf.loss))

    plt.legend(loc="best")
plt.show()
```