$$
\newcommand{\mat}[1]{\boldsymbol {#1}}
\newcommand{\mattr}[1]{\boldsymbol {#1}^\top}
\newcommand{\matinv}[1]{\boldsymbol {#1}^{-1}}
\newcommand{\vec}[1]{\boldsymbol {#1}}
\newcommand{\vectr}[1]{\boldsymbol {#1}^\top}
\newcommand{\rvar}[1]{\mathrm {#1}}
\newcommand{\rvec}[1]{\boldsymbol{\mathrm{#1}}}
\newcommand{\diag}{\mathop{\mathrm {diag}}}
\newcommand{\set}[1]{\mathbb {#1}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\pderiv}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\bb}[1]{\boldsymbol{#1}}
$$

# Tutorial 2: Multilayer Perceptron

## Todays Agenda
---
* recap on linear models
* Linear (fully connected) layers
* Activation functions
* 2-Layer MLP implementation from scratch (last time we use numpy for networks)
* N-layer MLP with PyTorch's `nn`, `autograd` and `optim` modules
* Weights initilization

## imports
---

```python
# Setup
%matplotlib inline
import os
import numpy as np
import sklearn
import torch
import matplotlib.pyplot as plt
from typing import Sequence
```

```python
plt.rcParams['font.size'] = 20
```

## Reminders and recaps
---
### Perceptrons and linear models

The following hypothesis class
$$
\mathcal{H} =
\left\{ h: \mathcal{X}\rightarrow\mathcal{Y}
~\vert~
h(\vec{x}) = \varphi(\vectr{w}\vec{x}+b); \vec{w}\in\set{R}^D,~b\in\set{R}\right\}
$$
where $\varphi(\cdot)$ is some nonlinear function, is composed of functions representing the **perceptron** model.

Schematic of a single perceptron, and it's inspiration, a biological neuron.

### logistic regression
we train a model using the logistic function (aka sigmoid):
$$\varphi(\vec{z})=\sigma(\vec{z})=\frac{1}{1+\exp(-\vec{z})}\in[0,1].$$

**Limitation**: logistic regression is still a linear classifier. In what sense is it linear if we use a non-linear function?

$$\hat{y} = \sigma(\vectr{w}\vec{x}+b)$$

**Linear** in the sense that output depends only on a linear combination of weights and inputs.
we can also call it- first moment of the input.

Decision boundaries are therefore straight lines, or can be transformed to linear via link function.

If we define $y(\vec{x})=\vectr{w}\vec{x}+b$, then the **decision surface** of the classifier is $y(\vec{x})=\text{const}$ (usually zero).

.

What if our data is not linearly separable?
in most cases in the real world, we want to learn a non-linear combination of the input, such as the examples below:

or even something more abstract, like a corrolation between 2 or more inputs

Can we still use e.g. logistic regression?

What if we first apply a **fixed** nonlinear transformation to the data?

For some $\psi: \set{R}^d \rightarrow \set{R}^D$, what does the following classifier do?

$$\hat{y} = \varphi(\vectr{w}\psi(\vec{x})+b).$$

Now we can get nonlinear decision boundaries with respect to $\vec{x}$.

But how can we choose a nonlinear transformation?
in **Machine Learning** you learn to look at the data and manually craft those kernel functions.

* what if the input is from high dimention and we can't look at it?
* waht if we look at some graphs of diffrent inputs, but we're not sure witch ones are impotant? (x1 vs x2? x3 corrolation with x5?)
* what do we do when we look but we don't know how to model the relation? we choose a set of functions and do hypothesis testing?
* what if we want to **learn** the non-linearity?

## Multilayer Perceptron (MLP)
---

### Model
the MLP

Composed of $L$ **layers**, each layer $l$ with $n_l$ **perceptron** ("neuron") units.

Each layer $l$ operates on the output of the previous layer ($\vec{y}_{l-1}$) and calculates:

$$
\vec{y}_l = \varphi\left( \mat{W}_l \vec{y}_{l-1} + \vec{b}_l \right),~
\mat{W}_l\in\set{R}^{n_{l}\times n_{l-1}},~ \vec{b}_l\in\set{R}^{n_l},~ l \in \{1,2,\dots,L\}.
$$

- Note that both input and output are **vectors**. We can think of the above equation as describing a layer of **multiple perceptrons**.
- We'll henceforth refer to such layers as **fully-connected** or FC layers.
- The first layer accepts the input of the model, i.e. $\vec{y}_0=\vec{x}\in\set{R}^d$.
- The last layer, $L$, is the output layer, so $y_L$ is the output of the model.
- The layers $1, 2, \dots, L-1$ are called hidden layers.

How powerful is an MLP model? I.e. which functions can it approximate?

**Universal approximator theorem**

Given enough parameters, an MLP with $L>1$ and any non-linear activation function, can approximate any continuous function up to any specified precision (Cybenko, 1989).

The MLP is therefore a **potent hypothesis class!** (recall approximation error).

See [here](http://neuralnetworksanddeeplearning.com/chap4.html) for an intuitive explanation of the UAT.

Personally, i prefer to look at it as aproximation we do using non-linearities.
sort of in the notion of Taylor series expantion:

in taylor expantion we use all moments, but the idea remain the same, aproximation of a function when reducing the error.

Given an input sample $\vec{x}^i$, the computed function of an $L$-layer MLP is:
$$
\vec{y}_L^i= \varphi \left(
\mat{W}_L \varphi \left( \cdots
\varphi \left( \mat{W}_1 \vec{x}^i + \vec{b}_1 \right)
\cdots \right)
+ \vec{b}_L \right)
$$

This expression is fully differentiable w.r.t. parameters using the **Chain Rule**.

And notice that $\vec{y}_L^i = \varphi(\mat{W}_L \vec{y}_{L-1}^i + \vec{b}_L)$ is just the linear model we started with.

So, intuitively, we can think of $\vec{y}_{L-1}$ as **learned** non-linear features of the input! In other words, $\vec{y}_{L-1} = \psi_{\vec{\Theta}}(\vec{x})$.

Since an MLP it has a non-linear dependency on it's inputs, through a learned transformation, non-linear decision boundaries are possible.

Cool visualization of the effect of MLP layers: http://playground.tensorflow.org/

## Activation functions
---

An **activation function** is the non-linear elementwise function $\varphi(\cdot)$ which operates on the affine part of the perceptron model.

Why do we even need non-linearities in the first place? Isn't the depth enough?

Without them, the MLP model would be equivalent to a single affine transform.

Common choices for the activation functions are:

- The logistic function (sigmoid)
    $$ \varphi(t) = \sigma(t) = \frac{1}{1+e^{-t}} \in  [0,1] $$
- The hyperbolic tangent (a shifted and scaled sigmoid)
    $$ \varphi(t) = \mathrm{tanh}(t) = \frac{e^t - e^{-t}}{e^t +e^{-t}} \in [-1,1]$$

- ReLU, rectified linear unit
    $$ \varphi(t) = \max\{t,0\} $$
Note that ReLU is not strictly differentiable. However, sub-gradients exist. We will define its gradient as:
    $$ \pderiv{\varphi}{t} = \begin{cases} 1, & t\geq0 \\ 0, & t<0 \end{cases} $$

```python
# activation functions
def sigmoid(z, deriv=False):
    output = 1 / (1 + np.exp(-1.0 * z))
    if deriv:
        return output * (1 - output)
    return output

def tanh(z, deriv=False):
    output = np.tanh(z)
    if deriv:
        return 1 - np.square(output)
    return output

def relu(z, deriv=False):
    output = z if z > 0 else 0
    if deriv:
        return 1 if z > 0 else 0
    return output
```

```python
def plot_activations():
    x = np.linspace(-5, 5, 1000)
    y_sig = sigmoid(x)
    y_tanh = tanh(x)
    y_relu = list(map(lambda z: relu(z), x))
    fig = plt.figure(figsize=(8, 5))
    ax1 = fig.add_subplot(2,1,1)
    ax1.plot(x, y_sig, label='sigmoid')
    ax1.plot(x, y_tanh, label='tanh')
    ax1.plot(x, y_relu, label='relu')
    ax1.grid()
    ax1.legend()
    ax1.set_xlabel('x')
    ax1.set_ylabel('y')
    ax1.set_ylim([-2, 2])
    ax1.set_title('Activation Functions')

    y_sig_derv = sigmoid(x, deriv=True)
    y_tanh_derv = tanh(x, deriv=True)
    y_relu_derv = list(map(lambda z: relu(z, deriv=True), x))
    ax2 = fig.add_subplot(2,1,2)
    ax2.plot(x, y_sig_derv, label='sigmoid')
    ax2.plot(x, y_tanh_derv, label='tanh')
    ax2.plot(x, y_relu_derv, label='relu')
    ax2.grid()
    ax2.legend()
    ax2.set_xlabel('x')
    ax2.set_ylabel('y')
    # ax2.set_ylim([-2, 2])
    ax2.set_title('Activation Functions Derivatives')
    plt.tight_layout()
```

```python
plot_activations()
```

Some reasons people cite for using ReLU are:

- Does not suffer from vanishing gradient when $x$ is far from zero (though gradient can still be zero).

- Much faster to compute than sigmoid and tanh.

- Promotes sparse weight vectors: "dead neurons" arguably cause sparsity in the next layer.

## ceate a function with MLP
---

here we notate $\vec{y}_i$ as $\vec{z}_i$

one fun way of thinking would be, why do we use the activision function in each layer?

well, if we wouldn't, we could do something very fun..

le'ts look at MLP with one hidden layer:

$$
\hat{y} =\mat{W}_2~ \underbrace{\mathrm{relu}(\overbrace{\mat{W}_1 \vec{x}+ \vec{b}_1}^{\vec{z}})}_{\vec{a}} + \vec{b}_2,
$$

if we didn't use activision function, we had

$$
\hat{y} =\mat{W}_2~ \underbrace{(\overbrace{\mat{W}_1 \vec{x}+ \vec{b}_1}^{\vec{z}})}_{\vec{a}} + \vec{b}_2,
$$

remember that the matrix dims are
$\mat{W}_l\in\set{R}^{n_{l}\times n_{l-1}},~ \vec{b}_l\in\set{R}^{n_l},~ l \in \{1,2,\dots,L\}.$

so instead of a complex model with 2 matrices of W and 2 biases,

we could have:
$$
\hat{y} =\mat{W}_2~ (\mat{W}_1 \vec{x}+ \vec{b}_1) + \vec{b}_2 = \mat{W}_3 \vec{x}+ \vec{b}_3 + \vec{b}_2 = \mat{W}_3 \vec{x}+ \vec{b}_4
$$
where
$$
\mat{W}_3 = \mat{W}_2 \mat{W}_1 \\
\vec{b}_3 = \mat{W}_2 \vec{b}_1 \\
\vec{b}_4 = \vec{b}_3 + \vec{b}_2
$$

that could be great for us, as we can quntize all layers without non-linear activision to be a one layer. but then we can't express non-linear functions.

more then that, if we did so, we could say that after the multiplications, we have a more compact set of weights that we could optimize them from the begining...

essentially, what we say is that for each learnable layer that you use another layer after and no-activisions, you're just making the optimization process harder, as you could not use this layer and have the same model.

a word about `activision` and `bias`

Now, why is it so important to use those activisions? we said that we could represent any function in the world... we already know from furierr theory and laplace teory that with exponents we can represent any function.

moreover, bias sound like a small correction, but it's a learnable vector of how to modify our input, so let's demonstrate with some example:

assume you want to learn this function (of a singular input) with MLP:

this is not homework in ML course, we we will give the solution

and in math notation:

$h_{1,1} = relu(x+1)$

$h_{1,2} = relu(x+1)$

$h_{1,3} = relu(x-1)$

$h_{2} = z_1 - 2z_2 + 2z_3$

## Two-layer MLP implementation
---

Let's solve a simple **regression** problem with a 2-layer MLP (one hidden layer, one output layer).

We're trying to learn a continuous and perhaps non-deterministic function $y=f(\vec{x})$.

- Domain: $\vec{x}^i \in \set{R}^{D_{\text{in}}}$

- Target: $y^i \in \set{R}^{D_{\text{out}}}$

- Model: $\hat{y} =\mat{W}_2~ \varphi(\mat{W}_1 \vec{x}+ \vec{b}_1) + \vec{b}_2$

  i.e. a 2-layer MLP, where:
    - $\vec{x}\in\set{R}^{D_{\text{in}}}$ sample (feature vector)
    - $\mat{W}_1\in\set{R}^{H\times D_{\text{in}}},\ \vec{b}_1\in\set{R}^{H}$
    - $\mat{W}_2\in\set{R}^{D_{\text{out}}\times H},\ \vec{b}_2\in\set{R}^{D_{\text{out}}}$
    - $\varphi(\cdot) = \mathrm{ReLU}(\cdot) = \max\{\cdot,0\}$
    - $H$ is the hidden dimension
    - We'll set $D_{\text{out}}=1$ so output is a scalar

- MSE loss with L2 regularization:
    $$
    \begin{align}
    \ell(\vec{y},\vec{\hat y}) &= \frac{1}{2}\norm{\vec{\hat y} - \vec{y}}^2 \\
    L_{\mathcal{S}} &= \frac{1}{N}\sum_{i=1}^{N}\ell(\vec{y},\vec{\hat y}) + \frac{\lambda}{2}\left(\norm{\mat{W}_1}_F^2 + \norm{\mat{W}_2}_F^2 \right)
    \end{align}
    $$
- Optimization scheme: Vanilla SGD

### Computing the loss gradients with backpropagation

Let's write our model as
$$
\hat{y} =\mat{W}_2~ \underbrace{\mathrm{relu}(\overbrace{\mat{W}_1 \vec{x}+ \vec{b}_1}^{\vec{z}})}_{\vec{a}} + \vec{b}_2,
$$

and manually derive the gradient of the point-wise loss $\ell(\vec{y},\vec{\hat y})$ using the **chain rule**.
Remember that to use SGD, we need the gradient of the loss w.r.t. our parameter tensors.

$$
\begin{align}
&\pderiv{\ell}{\vec{\hat y}}=2\cdot\frac{1}{2}(\vec{\hat y}-\vec{y}) = (\vec{\hat y}-\vec{y})\\
(\ast)~~&\pderiv{\ell}{\mat{W}_2}= \pderiv{\ell}{\vec{\hat y}}\pderiv{\vec{\hat y}}{\mat{W}_2}
=(\vec{\hat y}-\vec{y})\vectr{a}
\end{align}
$$

How do we continue into the nonlinearity? Recall that we defined

$$ \pderiv{\mathrm{relu}(x)}{x} = \begin{cases} 1, & x\geq0 \\ 0, & x<0 \end{cases} $$

and that we apply the non-linearity elementwise on input tensors.

Also remember that the gradient of a vector w.r.t. another vector is the Jacobian, a matrix of mixed derivatives:
$\pderiv{a_i}{z_j}$.

We have $\vec{a}=\mathrm{relu}(\vec{z})$. Thus,

$$
\pderiv{\vec{a}}{\vec{z}}=\mathrm{diag}(\mathbb{1}[\vec{z}>0]).
$$

And so,
$$
\begin{align}
&\pderiv{\ell}{\vec{a}}=\pderiv{\vec{\hat y}}{\vec{a}}\pderiv{\ell}{\vec{\hat y}}
=\mattr{W}_2(\vec{\hat y}-\vec{y})\\
&\pderiv{\ell}{\vec{z}}=\pderiv{\vec{a}}{\vec{z}}\pderiv{\ell}{\vec{a}}
=\mathrm{diag}(\mathbb{1}[\vec{z}>0])\mattr{W}_2(\vec{\hat y}-\vec{y})\\
(\ast)~~&\pderiv{\ell}{\mat{W}_1}=\pderiv{\ell}{\vec{z}}\pderiv{\vec{z}}{\mat{W}_1}
=\mathrm{diag}(\mathbb{1}[\vec{z}>0])\mattr{W}_2(\vec{\hat y}-\vec{y})\vectr{x}\\
\end{align}
$$

For the biases, we can easily see that:
$$
\begin{align}
(\ast)~~&\pderiv{\ell}{\vec{b}_2}=\pderiv{\vec{\hat y}}{\vec{b}_2}\pderiv{\ell}{\vec{\hat y}}
=I_{D_{\text{out}}} \pderiv{\ell}{\vec{\hat y}} \\
(\ast)~~&\pderiv{\ell}{\vec{b}_1}=\pderiv{\vec{z}}{\vec{b}_1}\pderiv{\ell}{\vec{z}}
=I_{H}\pderiv{\ell}{\vec{z}}\\
\end{align}
$$

The final gradients for weight update, including regularization will be

$$
\nabla_{\mat{W}_j}L_{\mathcal{S}}=\frac{1}{N}\sum_{i=1}^{N} \pderiv{\ell_i}{\mat{W}_j} + \lambda\mat{W}_j, \ j=1,2.
$$

Let's implement it from scratch using just `numpy`.

```python
# A simple MLP with one hidden layer

# N: batch size
# D_in: number of features
N, D_in =  64, 10
# H: hidden-layer
# D_out: output dimension
H, D_out =  100, 1

# Random input data
X = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Model weights and biases
wstd = 0.01
W1 = np.random.randn(H, D_in)*wstd
b1 = np.random.randn(H,)*wstd + 0.1
W2 = np.random.randn(D_out, H)*wstd
b2 = np.random.randn(D_out,)*wstd + 0.1

reg_lambda = 0.5
learning_rate = 1e-3
```

```python
losses = []
for epoch in range(250):
    # Forward pass, hidden layer: A = relu(X W1 + b1), Shape: (N, H)
    Z = X.dot(W1.T) + b1
    A = np.maximum(Z, 0)

    # Forward pass, output layer: Y_hat = A W2 + b2, Shape: (N, D_out)
    Y_hat = A.dot(W2.T) + b2

    # Loss calculation (MSE)
    loss = np.mean((Y_hat - y) ** 2); losses.append(loss) # (N, D_out)

    # Backward pass:  Output layer
    d_Y_hat = (1./N) * (Y_hat - y)     # (N, D_out)
    d_W2 = d_Y_hat.T.dot(A)            # (D_out, H)
    d_A = d_Y_hat.dot(W2)              # (N, H)
    d_b2 = np.sum(d_Y_hat, axis=0)     # (D_out,)

    # Backward pass: Hidden layer
    d_Z = d_A * np.array(Z > 0, dtype=float)  # (N, H)
    d_W1 = d_Z.T.dot(X)                          # (H, D_in)
    d_b1 = np.sum(d_Z, axis=0)                   # (H,)

    # Backward pass: Regularization term
    d_W2 += reg_lambda * W2
    d_W1 += reg_lambda * W1

    # Gradient descent step
    W2 -= d_W2 * learning_rate; b2 -= d_b2 * learning_rate
    W1 -= d_W1 * learning_rate; b1 -= d_b1 * learning_rate
    print(epoch+1, end=' ')
```

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250

```python
# Plot losses
_, ax = plt.subplots(figsize=(10,5))
ax.plot(losses)
ax.set_ylabel('MSE loss'); ax.set_xlabel('Epoch');
```

Note that this implementation is not ideal, as it's:

- Non modular (hard to switch components)
- Hard to extend (e.g. to add layers)
- Error prone (hard-coded manual calculations)

And now, we'll see how to address these issues using PyTorch's API.

##  N-Layer MLP using PyTorch
---

Let's create all our usual components:
- Dataset
- Model
- Loss function
- Optimizer

But this time we'll create a modular implementation where each of these components is separate and can be changed independently of the others.

### Dataset

As in the first tutorial we're goint to use  MNIST database of handwritten digits.

```python
import torch
import torch.utils.data
import torchvision #for dataset
import torchvision.transforms as tv_transforms

root_dir = os.path.expanduser('~/.pytorch-datasets/')
```

```python
tf_tensor = tv_transforms.ToTensor()

batch_size = 512
train_size = batch_size * 10
test_size = batch_size * 2

# Datasets and loaders
ds_train = torchvision.datasets.MNIST(root=root_dir, download=True, train=True, transform=tf_tensor)
ds_test =  torchvision.datasets.MNIST(root=root_dir, download=True, train=False, transform=tf_tensor)

dl_train = torch.utils.data.DataLoader(ds_train, batch_size,
                                       sampler=torch.utils.data.SubsetRandomSampler(range(0,train_size)))
dl_test = torch.utils.data.DataLoader(ds_test, batch_size,
                                       sampler=torch.utils.data.SubsetRandomSampler(range(0,test_size)))

x0, y0 = ds_train[0]
n_features = torch.numel(x0)
n_classes = 10

print(f'x0: {x0.shape}, y0: {y0}')
```

    x0: torch.Size([1, 28, 28]), y0: 5

```python
import sys
sys.path.append('..')
import T1.plot_utils as plot_utils
# Show first few samples
print(f'x0: {x0.shape}, y0: {y0}')
plot_utils.dataset_first_n(ds_train, 10, cmap='gray', show_classes=True);
```

### Model Implementation

- The `torch.nn` module contains building blocks such as neural network layers,
  loss functions, activations and more.
- In this section, we'll see various parts of the `torch.nn` API.
- We'll use `nn.Linear` which implements a single MLP layer.
- We'll implement our model as a subclass of `nn.Module`, which means:
    - Any tensors we set as properties will be registered as model parameters.
    - We can nest `nn.Modules` and get all model parameters from the top-level `nn.Module`.
    - Can be used as a function if we implement the `forward()` method.

To understand `nn.Module`, lets look at a very basic one: the [**fully-connected**](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear) layer.

```python
import torch.nn as nn

fc = nn.Linear(in_features=3, out_features=5, bias=True)

# 10 samples of 3 features
t = torch.randn(10, 3)

print(f'w: {fc.weight.shape} \nb: {fc.bias.shape}')

# Forward pass, notice that grad_fn exists
print(f'output {fc(t).shape}')
fc(t)
```

    w: torch.Size([5, 3])
    b: torch.Size([5])
    output torch.Size([10, 5])

    tensor([[-0.2845,  0.2501, -0.1356,  0.1289, -0.0985],
            [ 0.0745,  0.6346, -1.2990, -0.2640, -0.7732],
            [ 0.2629,  0.3942, -0.8706, -0.7141, -0.9265],
            [ 0.2376,  0.0940, -0.1527, -0.5201, -0.4389],
            [ 0.7591, -0.2891,  0.3530, -0.6938, -0.0777],
            [-0.4175,  0.4670, -0.5351, -0.2827, -0.7649],
            [ 0.2646,  0.2808, -0.6120, -0.4198, -0.5187],
            [ 0.3480, -0.0863,  0.1714,  0.2539,  0.5456],
            [-0.2826,  0.0870,  0.2471, -0.0189, -0.0816],
            [-0.9074,  0.4458, -0.1236,  0.1463, -0.4566]],
           grad_fn=<AddmmBackward>)

`nn.Modules` have registered **parameters**, which are tensors which `require_grad`.

```python
# Note parameter dimenstions
list(fc.parameters())
```

    [Parameter containing:
     tensor([[ 0.2730,  0.0484, -0.4255],
             [-0.1319, -0.1516, -0.0656],
             [ 0.1011,  0.3209,  0.4693],
             [ 0.0722, -0.1560,  0.2997],
             [ 0.2828,  0.0105,  0.2425]], requires_grad=True),
     Parameter containing:
     tensor([ 0.1362,  0.2407, -0.4235, -0.3026, -0.3985], requires_grad=True)]

We can create custom `nn.Module`s with arbitrary logic.

Let's recreate a fully-connected layer ourselves:

```python
class MyFullyConnectedLayer(nn.Module):

    def __init__(self, in_features, out_features,bias=True):
        super().__init__() # don't forget this to inherent from nn.Module
        self.bias = bias
        # nn.Parameter just marks W,b for inclusion in list of parameters
        self.W = nn.Parameter(torch.randn(out_features, in_features, requires_grad=True))
        if self.bias:
            self.b = nn.Parameter(torch.randn(out_features, requires_grad=True))

    def forward(self, x):
        # x assumed to be (N, in_features)
        z = torch.matmul(x, self.W.T)
        if self.bias:
            z+= self.b
        return z
```

```python
myfc = MyFullyConnectedLayer(in_features=3, out_features=5)
```

```python
myfc(t)
```

    tensor([[ 0.3321,  1.3101, -2.6998, -2.4024,  3.5277],
            [ 2.1238,  1.2073, -3.4486, -2.1502, -1.8029],
            [ 1.8333,  0.6354, -1.4190,  1.1150, -4.3340],
            [ 0.6007,  0.7535, -0.1058,  2.0324, -0.9300],
            [-0.4182,  0.7467,  3.2832,  6.1788,  0.1734],
            [ 1.4934,  0.7424, -3.8291, -2.6588, -1.5390],
            [ 1.1283,  0.9889, -1.0591,  0.7976, -1.0397],
            [-0.8361,  1.8123,  0.5110,  0.8824,  7.1833],
            [-0.0582,  1.0418, -1.7586, -1.0589,  3.1049],
            [ 0.8704,  0.9493, -5.2797, -5.3564,  1.9541]], grad_fn=<AddBackward0>)

```python
list(myfc.parameters())
```

    [Parameter containing:
     tensor([[-0.4792, -0.3346, -0.6215],
             [ 0.2471, -0.2397,  0.0607],
             [ 1.3051,  1.0067, -0.9659],
             [ 1.2895,  1.4464, -1.8601],
             [ 1.6681, -0.5136,  2.6390]], requires_grad=True),
     Parameter containing:
     tensor([ 0.8429,  1.0350, -1.2680,  0.2599,  0.1117], requires_grad=True)]

Quick visualization of our custom module's **computation graph**:

```python
import torchviz
torchviz.make_dot(myfc(t), params=dict(W=myfc.W, b=myfc.b))
```

Now that we know about `nn.Module`s, lets create an fairly-general MLP for multiclass classification.

```python
class MLP(nn.Module):
    NLS = {'relu': torch.nn.ReLU, 'tanh': nn.Tanh, 'sigmoid': nn.Sigmoid, 'softmax': nn.Softmax, 'logsoftmax': nn.LogSoftmax}

    def __init__(self, D_in: int, hidden_dims: Sequence[int], D_out: int, nonlin='relu'):
        super().__init__()

        all_dims = [D_in, *hidden_dims, D_out]
        non_linearity = MLP.NLS[nonlin]
        layers = []

        for in_dim, out_dim in zip(all_dims[:-1], all_dims[1:]):
            layers += [
                nn.Linear(in_dim, out_dim, bias=True),
                non_linearity()
            ]

        self.fc_layers = nn.Sequential(*layers[:-1])
        self.log_softmax = nn.LogSoftmax(dim=1)

    def forward(self, x):
        x = torch.reshape(x, (x.shape[0], -1)) #flatten x
        z = self.fc_layers(x)
        y_pred = self.log_softmax(z)
        # Output is always log-probability
        return y_pred
```

```python
# Create an instance of the model: 5-layer MLP
mlp5 = MLP(D_in=n_features, hidden_dims=[32, 64, 128, 64], D_out=n_classes, nonlin='tanh')

print(mlp5)
```

    MLP(
      (fc_layers): Sequential(
        (0): Linear(in_features=784, out_features=32, bias=True)
        (1): Tanh()
        (2): Linear(in_features=32, out_features=64, bias=True)
        (3): Tanh()
        (4): Linear(in_features=64, out_features=128, bias=True)
        (5): Tanh()
        (6): Linear(in_features=128, out_features=64, bias=True)
        (7): Tanh()
        (8): Linear(in_features=64, out_features=10, bias=True)
      )
      (log_softmax): LogSoftmax(dim=1)
    )

```python
print(f'number of parameter tensors: {len(list(mlp5.parameters()))}')
```

    number of parameter tensors: 10

```python
print(f'number of parameters: {np.sum([torch.numel(p) for p in mlp5.parameters()])}')
```

    number of parameters: 44458

```python
# Test a forward pass
y_hat0 = mlp5(x0)

print(f'{x0.shape=}\n')
print(f'{y_hat0.shape=}\n')
print(f'{y_hat0=}')
```

    x0.shape=torch.Size([1, 28, 28])

    y_hat0.shape=torch.Size([1, 10])

    y_hat0=tensor([[-2.2445, -2.3043, -2.1575, -2.3291, -2.4427, -2.4495, -2.3029, -2.1782,
             -2.4060, -2.2581]], grad_fn=<LogSoftmaxBackward>)

Quick visualization of our full MLP's **computation graph**:

```python
torchviz.make_dot(mlp5(x0), params=dict(mlp5.named_parameters()))
```

### Loss and Optimizer

For the loss function, we'll use PyTorch's built in negative log-likelihood loss since our model outputs probabilities.

Log Softmax is advantageous over softmax for numerical stability, optimisation and heavy penalisation for highly incorrect class

softmax for first class:
$$
\frac{e^{w_1x_1}}{\sum_i{e^{w_ix_i}}}
$$
log softmax for first class:
$$
\frac{log(e^{w_1x_1})}{log(\sum_i{e^{w_ix_i}})} = \frac{log(e^{w_1x_1})}{\sum_i{log(e^{w_ix_i}})} =\frac{log(e^{w_1x_1})}{\sum_i{w_ix_i}}
$$

```python
Probability =  torch.tensor([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
Log_Probability = torch.log(Probability)
print(f'{Probability=}\n {Log_Probability=}')
```

    Probability=tensor([0.1000, 0.2000, 0.3000, 0.4000, 0.5000, 0.6000, 0.7000, 0.8000, 0.9000,
            1.0000])
     Log_Probability=tensor([-2.3026, -1.6094, -1.2040, -0.9163, -0.6931, -0.5108, -0.3567, -0.2231,
            -0.1054,  0.0000])

```python
import torch.optim

# Loss:
# Note: NLLLoss assumes *log*-probabilities (given by our LogSoftmax layer)
loss_fn = nn.NLLLoss()

# Fake ground-truth labels
# Notice that we don't need to 1-hot encode them!
yt = torch.randint(low=0, high=n_classes, size=(y_hat0.shape[0],))

# Try out the loss
loss_fn(y_hat0, yt)
```

    tensor(2.3029, grad_fn=<NllLossBackward>)

As for the optimization scheme, we'll use a built in SGD optimizer from the `torch.optim` module.

We won't need to calculate the loss gradient this time, as we'll use `autograd` for automatic differentiation.

```python
torch.manual_seed(42)

# Model for training
model = MLP(D_in=n_features, hidden_dims=[32, 32, 32], D_out=n_classes, nonlin='relu')

# Optimizer over our model's parameters
optimizer = torch.optim.SGD(params=model.parameters(), lr=5e-2, weight_decay=0.01, momentum=0.9)
```

### Training loop

This time we'll train over lazy-loaded batches from our data loader.

Notice that except from our model's `__init__()` and `__forward()__`, we're using PyTorch facilities for the entire training implementation.

```python
num_epochs = 20
losses_mlp = []
for epoch_idx in range(num_epochs):
    total_loss = 0

    for batch_idx, (X, y) in enumerate(dl_train):
        # Forward pass
        y_pred = model(X) #btz x channels x input_w x input_h --> btz x num_classes

        # Compute loss
        loss = loss_fn(y_pred, y)
        total_loss += loss.item()
        losses_mlp.append(loss.item())

        # Backward pass
        optimizer.zero_grad() # Zero gradients of all parameters
        loss.backward()       # Run backprop algorithms to calculate gradients

        # Optimization step
        optimizer.step()      # Use gradients to update model parameters

    print(f'Epoch #{epoch_idx+1}: Avg. loss={total_loss/len(dl_train)}')
```

    Epoch #1: Avg. loss=2.3051342964172363
    Epoch #2: Avg. loss=2.2864507913589476
    Epoch #3: Avg. loss=2.256749725341797
    Epoch #4: Avg. loss=2.176988196372986
    Epoch #5: Avg. loss=1.9311089277267457
    Epoch #6: Avg. loss=1.416105616092682
    Epoch #7: Avg. loss=0.9191413521766663
    Epoch #8: Avg. loss=0.6763408541679382
    Epoch #9: Avg. loss=0.561395674943924
    Epoch #10: Avg. loss=0.5014575719833374
    Epoch #11: Avg. loss=0.4618980139493942
    Epoch #12: Avg. loss=0.40606015622615815
    Epoch #13: Avg. loss=0.3734510004520416
    Epoch #14: Avg. loss=0.36528579890727997
    Epoch #15: Avg. loss=0.33605656325817107
    Epoch #16: Avg. loss=0.32324054539203645
    Epoch #17: Avg. loss=0.3065242201089859
    Epoch #18: Avg. loss=0.30083635449409485
    Epoch #19: Avg. loss=0.28492291569709777
    Epoch #20: Avg. loss=0.2703426152467728

```python
# Plot losses
_, ax = plt.subplots(figsize=(10,5))
ax.plot(losses_mlp)
ax.set_ylabel('NLL loss'); ax.set_xlabel('Epoch');
```

Using the basic PyTorch building blocks we have arrived at a much more robust implementation:
- Easy to change architecture: layers and activation functions
- Easy to change loss
- Easy to change optimization method
- No need for manual gradient derivations

## Further reading
---

### initialization

We saw today a FC layer (or MLP or Linear layer), Neural networks has other type of layers that we're going to see in the next couple of tutorials.

when we implemented it ourself, we used torch.randn, that widraw from the standart normal distribution $\mathcal{N}(0,1)$.

But why? or more important, since we optimize it, **why does it matter**?

The answer to the second questioin is quite simple. just like K-Means, Neural network is not a stable algorithem, since diffrent initilizations will bring a diffrent outcome.

why is that? remember that the loss function is not a simple convex function!
 if we start with diffrent initilizations we're probably going to end up at diffrent local minima.

**note** that i'm lying to you a little bit, since we usually use Batch optimization algorithms and diffrent batches already contain a little randomness so even if the initial point is the same, we're not going to get the same trajectory from them on the loss function, and therfore this problem might not exist...

The honest answer is that this is an active research topic and the jastifications adressing some isues that we will state here, but it's not a rigorous definition.

Intuitivly, first thing we would think of a constant value, like 0 for all weights.

The idea of no constant is to:
* accelerate training by using small values
* keep neumerical stability
* brake symmetry

Other problem that we did not adress yet, is that with long networks, the chain rule is pruned make the gradients too small or too big (**vanishing gradients** and **exploding gradients** respectively).

So what can we do if we want to sqeeze the best results we can?

#### Xavier
Briefly, Normal distribution is always a good start, and to avoid edges effect normelize with the input size would help.
however, activision functions turn to non-linear shift in distibusion in neural networks.

that's where **Xavier initilization** came to life in [Understanding the difficulty of training deep feedforward neural networks](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)

basically they did a uniform distibution in [-1,1] and then scaling by $\frac{1}{\sqrt{n}}$.

:

here you can see the gradients without xavier (left) and with xavier (right)

note that we can use this normalization with $~\mathcal(N)$ or $~\mathcal(U)$

* **Xavier Uniform** - Initialize with values using uniform distribution. The resulting tensor will have values sampled from $\mathcal{U}(-a, a)$ where $$ a = \text{gain} \times \sqrt{\frac{6}{\text{fan}_{in} + \text{fan}_{out}}} $$

* **Xavier ** - Initialize with values using a normal distribution. The resulting tensor will have values sampled from $\mathcal{N}(0,\text{std}^2)$ where $$ \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan}_{in} + \text{fan}_{out}}} $$

* `fan_in` is the number of input units in the weight tensor and `fan_out` is the number of output units in the weight tensor

```python
w = torch.empty(2, 1000)
xavier_u = torch.nn.init.xavier_uniform_(w, gain=1.0)
plt.hist(xavier_u);
```

```python
w = torch.empty(1, 2000)
xavier_n = torch.nn.init.xavier_normal_(w, gain=1.0)
plt.hist(xavier_n);
```

One more improvment is **Kaiming Initialization** from [Delving Deep into Rectifiers:
Surpassing Human-Level Performance on ImageNet Classification](https://arxiv.org/pdf/1502.01852.pdf).

The only diffrence is that modern neural networks usually use Relu, or other type of non-symetric activision functions.

we won't go into the math of the paper, but the only noticeble diffrence is that we scale by  $\frac{\sqrt{2}}{\sqrt{n}}$.

* **Kaiming  Uniform** - The resulting tensor will have values sampled from $\mathcal{U}(-\text{bound}, \text{bound})$ where $$ \text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan_mode}}} $$

* **Kaiming (He) Normal** - Initialize with values sampled from normal distribution. The resulting tensor will have values sampled from $\mathcal{N}(0,\text{std}^2)$ where $$ \text{std} = \frac{\text{gain}}{\sqrt{\text{fan_mode}}} $$

    * `fan_mode` – either `fan_in` (default) or `fan_out`. Choosing `fan_in` preserves the magnitude of the variance of the weights in the forward pass. Choosing `fan_out`

and ever since, this is part of the Pytorch NN package.

```python
#`nonlinearity` – the non-linear function (`nn.functional` name), recommended to use only with `relu` or `leaky_relu` (default).
w = torch.empty(2, 1000)
k_u = torch.nn.init.kaiming_uniform_(w, a=0, mode='fan_in', nonlinearity='relu')
plt.hist(k_u);
```

```python
w = torch.empty(3, 1000)
k_n = torch.nn.init.kaiming_normal_(w, a=0, mode='fan_in', nonlinearity='relu')
plt.hist(k_n);
```

Note that initilization work for each channel seperatly

If we would like to change back to Xavier from kaiming (mostly, since there is no noticable diffrence for layers without ReLU)

```python
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Linear') != -1:
        torch.nn.init.xavier_normal_(m.weight, gain=1.0)
model.apply(weights_init)
```

    MLP(
      (fc_layers): Sequential(
        (0): Linear(in_features=784, out_features=32, bias=True)
        (1): ReLU()
        (2): Linear(in_features=32, out_features=32, bias=True)
        (3): ReLU()
        (4): Linear(in_features=32, out_features=32, bias=True)
        (5): ReLU()
        (6): Linear(in_features=32, out_features=10, bias=True)
      )
      (log_softmax): LogSoftmax(dim=1)
    )

when we use diffrent activisions, or when we do non-standart work, we might consider Xavier or something else...

thank you

## Links
---
[3b1b backprop video](https://www.youtube.com/watch?v=Ilg3gGewQ5U&ab_channel=3Blue1Brown)

[advanced loss functions](https://www.kdnuggets.com/2019/11/research-guide-advanced-loss-functions-machine-learning-models.html)

[how to choose loss functions](https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/)

[Weight Initialization in Neural Networks: A Journey From the Basics to Kaiming](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79)

## Credits
---

This tutorial heavly based on [Technion IIT cs-236781 tutorial](https://github.com/vistalab-technion/cs236781-tutorials/blob/master/t03/tutorial3-MLP.ipynb) that written by [Aviv Rosenberg](https://avivr.net) and re-weitten by [Moshe Kimhi](https://www.linkedin.com/in/moshekimhi/)

To re-use, please provide attribution and link to the original.

Some images in this tutorial were taken and/or adapted from the following sources:

- MartinThoma [CC0], via Wikimedia Commons https://commons.wikimedia.org/wiki/File:Perceptron-unit.svg
- Pattern Recognition and Machine Learning, C. M. Bishop, Springer, 2006
- Sebastian Raschka https://sebastianraschka.com/Articles/2015_singlelayer_neurons.html
- Favio Vázquez https://towardsdatascience.com/a-conversation-about-deep-learning-9a915983107
- Fundamentals of Deep Learning, Nikhil Buduma, Oreilly 2017


# Bayesian Model Example - The Deregulation Act

Basic idea: have taxi- and non-taxi-related crime (i.e. what Jacob is doing) diverged after the deregulation act?

Method:
 - Index both data sets to 1.0
 - Model the correlation between the two data sets, using something similar to the text messaging example from Probabilistic Programming book. I.e.:
   - Model Pearson correlation, assuming that it is drawn from a normal distribution (with parametrs $\mu$ and $\tau$).
   - Assume that two correlations exist: one before the Act, and one after.
   - Define $z$ as the point at which they switch (i.e. when the Act came into force).
   - Create two sets of $\mu$ and $\tau$ parameters - one before $z$ and one after.
   - See what the posterior distributions of the variables are. Do they correspond with the introduction of the Act?

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats as stats
from IPython.core.pylabtools import figsize

import pymc3 as pm
import theano.tensor as tt
```

    /Users/nick/anaconda3/envs/py36/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

## Data

Prepare the data

```python
data = pd.read_csv("./data/taxi_crime.csv")

date = np.array(data.loc[:,"Month"])
taxi_crime = np.array(data.loc[:,"Taxi Crime"])
all_crime = np.array(data.loc[:,"All Crime"])
```

Plot them

```python
figsize(12.5, 4)
plt.plot(np.arange(len(date)), taxi_crime*100, color="#348ABD", label="Taxi crime (*100)")
plt.plot(np.arange(len(date)), all_crime, color="#00AA00", label="All crime")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.title("Crime volumes")
plt.legend()
```

As _all crime_ has much larger volumes than _taxi crime_, index to the first point and then calculate the difference. This will give them comparable sizes.

```python
taxi_i = np.array( [ x /  taxi_crime[0]  for x in taxi_crime ] )
all_i  = np.array( [ x /  all_crime[0]  for x in all_crime ] )

CUTOFF = 30 # Apprixmate time of the act
```

```python
figsize(12.5, 4)
plt.plot(np.arange(len(date)), taxi_i, color="#00AA00", label="Taxi indexed")
plt.plot(np.arange(len(date)), all_i, color="#348ABD", label="All indexed")
plt.xticks(np.arange(len(date)), date, rotation=70)
plt.axvline(x=CUTOFF, color="gray", ls="dashed")
plt.text(CUTOFF, 0.50, "Cutoff (approximate date of the Act)")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.legend()
```

Have a look at some Pearson correlations

```python

p_all = np.corrcoef(taxi_i, all_i)
p_before = np.corrcoef(taxi_i[CUTOFF:], all_i[CUTOFF:])
p_after = np.corrcoef(taxi_i[:CUTOFF], all_i[:CUTOFF])

print("Correlations:\n\tAll:{}\n\tBefore:{}\n\tAfter:{}".
      format(p_all[0][1], p_before[0][1], p_after[0][1]))
```

    Correlations:
    	All:0.2880288515969878
    	Before:-0.05846675286697224
    	After:0.23114534880058518

It doesn't work very well using these noisy data. So try calculating the difference between the two proportions as a 12-month rolling average

```python
CUTOFF_AVG = CUTOFF - 12 # Because data have been averaged now, so we lost the first 12 time points

avg_taxi = []
avg_all = []
avg_date = []
for i in range(12, len(taxi_i)):
    taxi_mean =  np.mean(taxi_i[i-12:i])
    all_mean =  np.mean(all_i[i-12:i])
    avg_taxi.append(taxi_mean)
    avg_all.append(all_mean)
    avg_date.append(date[i])

avg_taxi = np.array(avg_taxi)
avg_all  = np.array(avg_all)
```

```python
figsize(12.5, 4)
plt.plot(np.arange(len(avg_diff)), avg_taxi, color="#00AA00", label="Taxi indexed 12-month average")
plt.plot(np.arange(len(avg_diff)), avg_all, color="#348ABD", label="All indexed 12-month average")
plt.xticks(np.arange(len(avg_date)), avg_date, rotation=70)
plt.axvline(x=CUTOFF_AVG, color="gray", ls="dashed")
plt.text(CUTOFF_AVG, 0.95, "Cutoff\n(approximate date of the Act)")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.title("12-month rolling average differences")
plt.legend()
```

```python

p_all = np.corrcoef(avg_taxi, avg_all)
p_before = np.corrcoef(avg_taxi[CUTOFF_AVG:], avg_all[CUTOFF_AVG:])
p_after = np.corrcoef(avg_taxi[:CUTOFF_AVG], avg_all[:CUTOFF_AVG])

print("Correlations:\n\tAll:{}\n\tBefore:{}\n\tAfter:{}".
      format(p_all[0][1], p_before[0][1], p_after[0][1]))
```

    Correlations:
    	All:0.9144747968488516
    	Before:0.6257686280703504
    	After:0.9579472316594856

## Model Conception

We want to model the correlation between the two indexed crime volumes. The correlation is continuous in the range [-1,1]. Therefore lets assume that the difference can be modelled using a normal distribution:

$$X \sim N(\mu, 1/\tau)$$

The values of $\mu$ and $\tau$ determine the shape of the distribution. The $\tau$ will give us information about the spread of the difference. $\mu$ is especially useful: a _larger_ value for $\mu$ will higher probability of _larger_ variable values being chosen. Therefore if a model suggests that $\mu$ increases after some point (i.e. the introduction of the Deregulation Act) then it would appear that the correlation has increased.

```python

nor = stats.norm
x = np.linspace(-8, 7, 150)
mu = (-2, 0, 3)
tau = (.7, 1, 2.8)
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip(mu, tau, colors)

for _mu, _tau, _color in parameters:
    plt.plot(x, nor.pdf(x, _mu, scale=1./_tau),
             label="$\mu = %d,\;\\tau = %.1f$" % (_mu, _tau), color=_color)
    plt.fill_between(x, nor.pdf(x, _mu, scale=1./_tau), color=_color,
                     alpha=.33)

plt.legend(loc="upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title("Probability distribution of three different Normal random \
variables");
```

We want to test whether at some point (e.g. at the introduction of the Deregulation Act) the difference in crime volumes changed. Call this point $z$. After $z$, the correlation suddenly becomes larger; $\mu$ increases. Therefore there are two $\mu$ parameters, one before $z$ and one after (and each will have its own $\tau$ parameter):

$$
X \sim N
\begin{cases}
    (\mu_1, 1/\tau_1 ) & \text{if } t \lt z \cr
    (\mu_2, 1/\tau_2 ) & \text{if } t \ge z
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\mu_1 = \mu_2$, then the $\mu$s posterior distributions should look about equal.

We are interested in inferring the unknown $\mu$s and $\tau$s. To use Bayesian inference, we need to assign prior probabilities to their different possible values. $\mu$ is continuous and might be negative, so use a normal distrubtion again. $\tau$ is also continuous, but cannot be negative, so use an exponential distribution.  Each of those distributions has their own parameters, so we now have six _hyper-parameters_ (two for each of the $\mu$s and one for each $\tau$). Name them $\alpha$, $\beta$ and $\gamma$:

$$
\begin{align}
& \mu_1 \sim \text{N} (\alpha_1, 1/\beta_1) \text{ , } \tau_1 \sim \text{Exp} (\gamma_1) \\\
& \mu_2 \sim \text{N} (\alpha_2, 1/\beta_2) \text{ , } \tau_2 \sim \text{Exp} (\gamma_2)
\end{align}
$$

The $\alpha$, $\beta$ and $\gamma$ parameters influence other parameters, so our initial guesses do not influence the model too strongly, so we have some flexibility in our choice.

We also need a distribution for the point at which the difference changes ($z$). Assign a *uniform prior belief* to every possible time point. This will reduce any personal bias.

\begin{align}
& z \sim \text{DiscreteUniform(1,N) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{N}
\end{align}

where $N$ is the number of time points in the data.

## Model Definition

Now define the model using PYMC. (_Note that we use the same values for the hyperparameters, although above I talked about them as separate variables for each of the two Normal distributions_).

```python

with pm.Model() as model:
    # The hyperparameters
    alpha = 1.0
    beta = 1.0
    gamma = 1.0/avg_all.mean()

    # The two mean and sd variables parameters.
    # (We want to see if their posterior distributions change)
    mu_1 = pm.Normal("mu_1", mu = alpha, tau = beta)
    mu_2 = pm.Normal("mu_2", mu = alpha, tau = beta)

    tau_1 = pm.Exponential("tau_1", gamma)
    tau_2 = pm.Exponential("tau_2", gamma)
    #taus = pm.Exponential("taus", gamma, shape=2)

    # Z is the point at which the texting behaviour changed.
    # Don't know anything about this so assign a uniform prior belief
    z = pm.DiscreteUniform("z", lower=0, upper=len(avg_diff) - 1)

    # Creates new functions using switch() to assign the correct mu or tau variables
    # These can be thought of as the random variables mu and tau definied initially.
    idx = np.arange(len(avg_diff)) # Index
    mu_ =  pm.math.switch(z > idx, mu_1, mu_2)
    tau_ = pm.math.switch(z > idx, tau_1, tau_2)

    # Combine the observation data (we assume that it has been generated by
    # a normal distribution with our mu and tau parameters).
    XXXX HOW TO INCORPORATE OBSERVED DATA HERE??
    obs = pm.Normal("obs", mu=mu_, tau=tau_, observed=avg_diff)
```

The model has been defined, now we can perform MCMC to sample from the posterior distribution

```python
N = 5000 # Number of samples
with model:
    step = pm.Metropolis()
    start = pm.find_MAP() # Help it to start from a good place
    trace = pm.sample(N, tune=int(N*2), step=step, start=start, njobs=4)
```

    logp = -38.639, ||grad|| = 0.11181: 100%|██████████| 15/15 [00:00<00:00, 1004.34it/s]
    100%|██████████| 30000/30000 [00:29<00:00, 1012.12it/s]

The model has finished. Now example our samples

```python
mu_1_samples = trace['mu_1']
mu_2_samples = trace['mu_1']
tau_1_samples = trace['tau_1']
tau_2_samples = trace['tau_2']
z_samples = trace['z']
```

```python
figsize(12.5, 10)
#histogram of the samples:

# MUs

ax = plt.subplot(321) # rows, columns, index
ax.set_autoscaley_on(False)
plt.hist(mu_1_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\mu_1$", color="#A60628", normed=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables""")
#plt.xlim([15, 30])
#plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(322) # rows, columns, index
ax.set_autoscaley_on(False)
plt.hist(mu_2_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\mu_2$", color="#A60628", normed=True)
plt.legend(loc="upper left")
#plt.xlim([15, 30])
#plt.xlabel("$\lambda_1$ value")

# TAUs

ax = plt.subplot(323)
ax.set_autoscaley_on(False)
plt.hist(tau_1_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\tau_1$", color="#7A68A6", normed=True)
plt.legend(loc="upper left")
#plt.xlim([15, 30])
plt.xlabel("$\tau_1$ value")

ax = plt.subplot(324)
ax.set_autoscaley_on(False)
plt.hist(tau_2_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\tau_2$", color="#7A68A6", normed=True)
plt.legend(loc="upper left")
#plt.xlim([15, 30])
plt.xlabel("$\tau_2$ value")

plt.subplot(325)
w = 1.0 / z_samples.shape[0] * np.ones_like(z_samples)
plt.hist(z_samples, bins=len(diff), alpha=1,
         label=r"posterior of $z$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(len(diff)))

plt.legend(loc="upper left")
#plt.ylim([0, .75])
#plt.xlim([35, len(dif)-20])
plt.xlabel(r"$z$ (in months)")
plt.ylabel("probability");
```

Some more diagnostic visualisations

```python
pm.plots.traceplot(trace=trace, varnames=["mu_1", "mu_2", "tau_1", "tau_2"])
```

```python
pm.plots.plot_posterior(trace=trace, varnames=["mu_1", "mu_2", "tau_1", "tau_2", "z"])
#pm.plots.plot_posterior(trace=trace["centers"][:,1])
#pm.plots.autocorrplot(trace=trace, varnames=["centers"]);
```

```python

```

```python

```

## Next steps:

Think about How to detect structural change in a timeseries.

E.g.: [https://stats.stackexchange.com/questions/16953/how-to-detect-structural-change-in-a-timeseries]

    Can involve Jose too.

```python

```


# An "Affine" Mathematical Model for Wealth Inequality
In an article[[1]](https://www.scientificamerican.com/article/is-inequality-inevitable/) published in Scientific American (which is itself based on these papers[[2]](https://arxiv.org/abs/1511.00770)[[3]](https://arxiv.org/abs/1604.02370)), a model is presented that parametrizes wealth inequality based on a simple "garage sale" exchange model. This Jupyter notebook contains my attempt to reproduce and build upon the results presented in the aforementioned article.

```python
import numpy as np
import time
from matplotlib import pyplot as plt
from scipy import stats
from random import randint, gammavariate
from tqdm.notebook import tqdm
from multiprocessing import Pool
```

```python
class Analysis:
    def __init__(self, N_epochs=2500, N_checkpoints=0, N_actors=1000,
                 starting_wealth=1000, transaction_size=0.1):
        # Seeding parameters
        self.N_actors = N_actors
        self.starting_wealth = starting_wealth
        self.transaction_size = transaction_size
        self.N_epochs = N_epochs
        self.N_checkpoints = N_checkpoints
        # Steady-state tracking information
        self.gini_coefficients = []
        # Bank
        self.reset_bank()
        # Seed RNG
        np.random.seed(int(time.time()))

    def reset_bank(self, show_init_dist=True):
        """Reset bank information"""
        self.bank = np.ones(self.N_actors)*self.starting_wealth
        self.sorted_bank = None
        self.sorted_actors = None
        self.total_wealth = np.sum(self.bank)
        if show_init_dist:
            # Show that wealth is initially evenly distributed
            self.plot_wealth_dist()
            plt.title("Initial Wealth Distribution")
            plt.ylim(0, self.starting_wealth+100)

        return

    def update_bank(self, transactions, cur_epoch):
        """Update bank information"""
        self.bank += transactions
        if self.N_checkpoints > 0:
            epoch_step = int(1.0*self.N_epochs/self.N_checkpoints)
            if cur_epoch%epoch_step == 0 or self.N_epochs <= self.N_checkpoints:
                self.sorted_bank = np.sort(self.bank)[::-1]
                self.sorted_actors = np.argsort(self.bank)[::-1]
                self.total_wealth = np.sum(self.bank)
                gini = self.plot_lorenz_curve(no_plot=True, N_points=10000)
                self.gini_coefficients.append(gini)
        elif cur_epoch == self.N_epochs-1:
            self.sorted_bank = np.sort(self.bank)[::-1]
            self.sorted_actors = np.argsort(self.bank)[::-1]
            self.total_wealth = np.sum(self.bank)
        return

    def plot_wealth_dist(self, show_oligarch=False):
        """Plot a basic wealth distribution"""
        plt.scatter(range(self.N_actors), self.bank, s=10);
        if show_oligarch:
            plt.scatter(self.bank.argmax(), self.bank.max(), marker="*",
                        s=50, label="Oligarch")
        plt.xlabel("Actor");
        plt.ylabel("Wealth");
        plt.title("Wealth Distribution");
        plt.axhline(np.average(self.bank), color="r", label="Average Wealth");
        plt.legend();

        to_print = float(np.sum(self.bank > np.average(self.bank))/self.N_actors*100)
        print("{:0.2f}% of actors have more wealth than average.".format(to_print))
        return

    def plot_wealth_ineq(self, N_points=50):
        """p% of actors vs. w% of total wealth held by P% of actors"""
        vals = []
        percentages = np.linspace(0, 1, N_points)
        for p in percentages:
            # Wealth held by p% of actors
            wealth_held = np.sum(self.sorted_bank[:int(self.N_actors*p)])
            vals.append(wealth_held/self.total_wealth)

        # Wealth inequality plot
        plt.scatter(percentages, vals, s=10);
        plt.xlabel("% of Actors");
        plt.ylabel("% of Total Wealth Held");
        plt.title("Basic Wealth Inequality Plot");
        return

    def gini_coeff(self, x, y):
        """Calculate Gini coefficient using trapezoidal integral approximation"""
        return (np.trapz(x,y)-0.5)/0.5

    def plot_lorenz_curve(self, N_points=500, no_plot=False):
        """Fraction of wealth held by actors with wealth < w vs. fraction of actors that
           hold wealth < w, where w is some percentage of the total wealth
        """
        # Calculate values for Lorenz curve
        cumulative_wealth = []
        cumulative_actors = []
        for p in np.linspace(0, 1, N_points):
            actors_with_lt_w = self.sorted_bank < self.total_wealth*p
            # Fraction of wealth held by actors with wealth < w
            cumulative_wealth.append(
                np.sum(self.sorted_bank[actors_with_lt_w])/self.total_wealth
            )
            # Fraction of actors that hold wealth < w
            cumulative_actors.append(np.sum(actors_with_lt_w)/self.N_actors)
        # Cast to numpy arrays
        cumulative_wealth = np.array(cumulative_wealth)
        cumulative_actors = np.array(cumulative_actors)
        # Calculate Gini coefficient
        gini = self.gini_coeff(cumulative_actors, cumulative_wealth)
        if no_plot:
            return gini
        else:
            # Set up plot
            fig = plt.figure(figsize=(5,5))
            axes = plt.subplot(111)
            axes.set_aspect(1.0)
            # Fill plot
            plt.scatter(cumulative_actors, cumulative_wealth, s=10);
            plt.xlabel("Cumulative Population");
            plt.ylabel("Cumulative Wealth");
            plt.title("Lorenz Curve (Gini = {:.3f})".format(gini));
            return

    def plot_gini_evolution(self):
        """Plot Gini coefficients stored during run of simulation"""
        # Fill plot
        plt.scatter(
            np.arange(0, self.N_epochs, 1.0*self.N_epochs/self.N_checkpoints),
            self.gini_coefficients
        )
        # Format plot
        plt.xlabel("Epoch");
        plt.ylabel("Gini Coefficient");
        plt.title("Evolution of Gini Coefficients");
```

## 1. This for That: The Garage Sale Model
We begin with a simple exchange between two actors: the garage sale model. Specifically, it models a bargain, hence its namesake, between two parties wherein the purchaser may win or lose some percentage of their wealth $w$ (which we will henceforth call $\Delta w$) by chance - a coin flip in this case. This is intended to provide a skeletal description of free market capitalism as a whole, as such economies are essentially driven by such exchanges. To model a coin flip, we will use a pseudo-random number $R\in\{0,1\}$ such that we may generate a value $\mathcal{R}\in\{-1,1\}$ given by

$$
\mathcal{R} = (-1)^{R}
$$
where we can calculate the partner result in parallel using $\mathcal{R}' = (-1)^{R+1}$. This means that the expectation values of $\mathcal{R}$ and $\mathcal{R}^2$ are

$$
\langle\mathcal{R}\rangle = 0\\
\langle\mathcal{R}^2\rangle = 1
$$

This coin toss determines the transaction between an actor with initial wealth $w_i$ and their partner with initial wealth $w_i'$ as follows:

$$
w_f = w_i + \mathcal{R}\Delta w
\\
w_f' = w_i' + \mathcal{R}'\Delta w
$$

### 1.1 Simulation

```python
class GarageSale(Analysis):
    def __init__(self, N_exchanges=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if not N_exchanges:
            self.N_exchanges = self.N_actors
        else:
            self.N_exchanges = N_exchanges

    def run_round(self):
        """Run one round of the Garage Sale Model"""
        N = self.N_actors
        N_exchanges = self.N_exchanges
        ts = self.transaction_size
        bank = self.bank
        # Set up transaction information
        wealth_exchanged = np.zeros(N_exchanges)
        actor_exchanges = np.zeros(N_exchanges)
        partner_exchanges = np.zeros(N_exchanges)
        transactions = np.zeros(N)
        # Get actor/partner wealth
        actors = np.random.permutation(N)[:N_exchanges]
        partners = np.random.permutation(N)[:N_exchanges]
        actor_wealth = bank[actors]
        partner_wealth = bank[partners]
        # Determine amount of wealth exchanged
        actor_is_wealthier = (actor_wealth > partner_wealth)
        wealth_exchanged[actor_is_wealthier] = ts*partner_wealth[actor_is_wealthier]
        wealth_exchanged[~actor_is_wealthier] = ts*actor_wealth[~actor_is_wealthier]
        # Determine actor/partner exchanges
        actor_won_toss = (np.random.rand(N_exchanges) > 0.5)
        actor_exchanges = wealth_exchanged*(-1)**(~actor_won_toss)
        partner_exchanges = (-1)*actor_exchanges
        # Complete transactions
        transactions[actors] += actor_exchanges
        transactions[partners] += partner_exchanges

        return transactions

    def run(self):
        for e in tqdm(range(self.N_epochs)):
            transactions = self.run_round()
            self.update_bank(transactions, e)

affine0 = GarageSale()
```

```python
affine0.run()
```

    HBox(children=(FloatProgress(value=0.0, max=2500.0), HTML(value='')))

### 1.2 Plots

```python
affine0.plot_wealth_dist(show_oligarch=True)
```

```python
affine0.plot_wealth_ineq()
```

```python
affine0.plot_lorenz_curve(N_points=1000)
```

## 2. To Each According to his Needs: $\chi$
We now introduce a common feature of modern economies: wealth redistribution. After each round of exchanges, we shift each actor's wealth by the following amount:

$$
W = \chi(\bar{w}-w)
$$

where $\bar{w}$ is the mean wealth of the entire population, $w$ is the individual's personal wealth, and $\chi\in[0,1]$. Thus, we notice that $W < 0$ if that particular actor holds more wealth than average and $W > 0$ otherwise. Overall, this results in all actors taking a step towards the mean wealth of the entire population.

### 2.1 Simulation

```python
class WealthTax(GarageSale):
    def __init__(self, chi=0.0, *args, **kwargs):
        self.chi = chi
        super().__init__(*args, **kwargs)

    def redistribute(self, transactions):
        N_actors = self.N_actors
        bank = self.bank
        # Redistribute wealth
        actor_wealth = bank+transactions
        mean_wealth = np.ones(N_actors)*np.mean(actor_wealth)
        transactions += (mean_wealth-(actor_wealth))*self.chi

        return transactions

    def run(self):
        for e in tqdm(range(self.N_epochs)):
            transactions = self.run_round()
            redistributed = self.redistribute(transactions)
            self.update_bank(redistributed, e)

affine1 = WealthTax(chi=0.05, N_epochs=3000, transaction_size=0.1)
```

```python
affine1.run()
```

    HBox(children=(FloatProgress(value=0.0, max=3000.0), HTML(value='')))

### 2.2 Plots

```python
affine1.plot_wealth_dist()
```

```python
affine1.plot_wealth_ineq()
```

```python
affine1.plot_lorenz_curve(N_points=10000)
```

## 3. The Rich Get Richer: $\zeta$
So far, the exchange in the Garage Sale Model has been agnostically governed by the flip of a pseudo-fair coin. Now, in an attempt to model the real bias of everyday exchanges, we allow the coin flip to be biased towards the wealthier actor. To do this, we must modify the coin flip such that our outcome $\mathcal{R}$ has an expectation value given by

$$
\langle\mathcal{R}\rangle = \zeta\bigg(\frac{w-w'}{\bar{w}}\bigg)
$$

where $\bar{w}$ is again the mean wealth of the entire population, $w$ is the wealth of the actor, $w'$ is the wealth of their partner, and $\zeta\in[0,1]$. We notice that if $w > w'$, if the actor is wealthier than their partner, then $\langle\mathcal{R}\rangle > 0$ which implies that the actor will _win_ the coin toss on average. Conversely, if the actor is poorer than their parter ($w < w'$), then $\langle\mathcal{R}\rangle < 0$ so the actor will _lose_ the coin toss on average. Refer back to Section 1 for how the coin toss determines the outcome of the transaction in our garage sale model.

By the definition of $\langle\mathcal{R}\rangle$, then, we may simply find $p(\mathcal{R}=+1)$ and $p(\mathcal{R}=-1)$:

$$
\begin{align}
\langle\mathcal{R}\rangle &\equiv (1)p(\mathcal{R}=+1) + (-1)p(\mathcal{R}=-1)
\\
\zeta\bigg(\frac{w-w'}{\bar{w}}\bigg) &= p(\mathcal{R}=+1) - p(\mathcal{R}=-1)
\end{align}
$$

However, we have the fundamental constraint that $p(\mathcal{R}=+1) + p(\mathcal{R}=-1) = 1$. Adding this equation to the above, we obtain

$$
\begin{align}
\zeta\frac{w}{\bar{w}}-\zeta\frac{w'}{\bar{w}}+1 &= 2p(\mathcal{R}=+1)
\\
p(\mathcal{R}=+1) &= \frac{\zeta}{2}\bigg(\frac{w}{\bar{w}}-\frac{w'}{\bar{w}}\bigg)+\frac{1}{2}
\\
\implies p(\mathcal{R}=-1) &= \frac{1}{2}-\frac{\zeta}{2}\bigg(\frac{w}{\bar{w}}-\frac{w'}{\bar{w}}\bigg)
\end{align}
$$

We can now use either of the above as a working point for biasing the coin toss. At the same time, we see that this leaves $\langle\mathcal{R}^2\rangle$ unchanged, that is $\langle\mathcal{R}^2\rangle = 1$ since we still have $\mathcal{R}\in\{-1,1\}$. From these, of course, we may calculate the standard deviation $\sigma = \sqrt{\langle\mathcal{R}^2\rangle-\langle\mathcal{R}\rangle^2}$. Explicitly, we have

$$
\sigma = \sqrt{1-\zeta^2\bigg(\frac{w-w'}{\bar{w}}\bigg)^2}
$$

The mean $\mu \equiv \langle\mathcal{R}\rangle$ and standard deviation $\sigma$ define a Gaussian distribution which we could sample for our biased coin toss.

Alternatively, and perhaps more accurately, we could define a binomial distribution:

$$
\begin{align}
\mu &= np\\
\sigma^2 &= np(1-p)\\
\implies p &= 1-\frac{\sigma^2}{\mu}, n=\frac{\mu}{1-\frac{\sigma^2}{\mu}}
\end{align}
$$

However, we must be careful of how we would define $\mu$ and $\sigma$, since we are counting _successes_ after $n$ tosses with a binomial distribution, whereas here we simply care about the outcome of a single coin toss per transaction.

### 3.1 Simulation

```python
class BiasedGarageSale(WealthTax):
    def __init__(self, chi=0.0, zeta=0.0, *args, **kwargs):
        self.zeta = zeta
        super().__init__(chi=chi, *args, **kwargs)

    def run_round(self):
        """Run one round of the Garage Sale Model biased towards the
           wealthier party
        """
        N = self.N_actors
        N_exchanges = self.N_exchanges
        ts = self.transaction_size
        bank = self.bank
        total_wealth = np.sum(bank)
        # Set up transaction information
        wealth_exchanged = np.zeros(N_exchanges)
        actor_exchanges = np.zeros(N_exchanges)
        partner_exchanges = np.zeros(N_exchanges)
        transactions = np.zeros(N)
        # Get actor/partner wealth
        actors = np.random.permutation(N)[:N_exchanges]
        partners = np.random.permutation(N)[:N_exchanges]
        actor_wealth = bank[actors]
        partner_wealth = bank[partners]
        # Determine amount of wealth exchanged
        actor_is_wealthier = (actor_wealth > partner_wealth)
        wealth_exchanged[actor_is_wealthier] = ts*partner_wealth[actor_is_wealthier]
        wealth_exchanged[~actor_is_wealthier] = ts*actor_wealth[~actor_is_wealthier]
        # Determine actor/partner exchanges
        prob_minus1 = 0.5*(1-self.zeta*(actor_wealth-partner_wealth)/(total_wealth/N))
        actor_won_toss = (np.random.rand(N_exchanges) > (prob_minus1))
        actor_exchanges = wealth_exchanged*(-1)**(~actor_won_toss)
        partner_exchanges = (-1)*actor_exchanges
        # Complete transactions
        transactions[actors] += actor_exchanges
        transactions[partners] += partner_exchanges

        return transactions

affine2 = BiasedGarageSale(chi=0.036, zeta=0.05)
```

```python
affine2.run()
```

    HBox(children=(FloatProgress(value=0.0, max=2500.0), HTML(value='')))

### 3.2 Plots

```python
affine2.plot_wealth_dist(show_oligarch=True)
```

```python
affine2.plot_wealth_ineq()
```

```python
affine2.plot_lorenz_curve(N_points=10000)
```

## 4. Negative Wealth: $\kappa$
For all of the plots produced above, no actor can have wealth less than zero. This is because the actor can only lose some percentage of their personal wealth after any exchange, meaning they can actually only _approach_ zero in theory. This is not, of course, realistic. It is a simple fact that many people are in debt, so the poorest should actually have _negative_ wealth. To address this, we require that an actor can have wealth no less than $-\Lambda$ at any time, which is defined as follows (from here[3], Eq. (24) taking $\Delta \rightarrow \Lambda$ for personal preference):

$$
\Lambda = \bigg(\frac{\kappa}{1-\kappa}\bigg)\bar{w}
$$

where $\kappa\in[0,1]$. Then, before each transaction, we loan every actor $\Lambda$ and require that they pay back their debt after the transaction, effectively shifting the wealth distribution to allow for negative wealth.

### 4.1 Simulation

```python
class LoanShark(BiasedGarageSale):
    def __init__(self, chi=0.0, zeta=0.0, kappa=0.0, *args, **kwargs):
        self.kappa = kappa
        super().__init__(chi=chi, zeta=zeta, *args, **kwargs)

    def run(self):
        for e in tqdm(range(self.N_epochs)):
            bank = self.bank
            kappa = self.kappa
            # Apply loans
            loans = np.ones(self.N_actors)*(kappa/(1-kappa))*np.mean(bank)
            self.bank += loans
            # Run transactions
            transactions = self.run_round()
            redistributed = self.redistribute(transactions)
            # Update bank with remainder after loan collection
            self.update_bank(redistributed-loans, e)

affine3 = LoanShark(N_epochs=5000, chi=0.05, kappa=0.1, zeta=0.05)
```

```python
affine3.run()
```

    HBox(children=(FloatProgress(value=0.0, max=5000.0), HTML(value='')))

### 4.2 Plots

```python
affine3.plot_wealth_dist()
```

```python
affine3.plot_wealth_ineq()
```

```python
affine3.plot_lorenz_curve(N_points=10000)
```

## 5. Consolidating the Affine Model
So far, the different models have been iteratively built upon by a somewhat convoluted chain of inheritance. For convenience, we will consolidate this into a single class.

```python
class Affine(Analysis):
    def __init__(self, chi=0.0, zeta=0.0, kappa=0.0, N_exchanges=None,
                 *args, **kwargs):
        self.chi = chi
        self.zeta = zeta
        self.kappa = kappa
        super().__init__(*args, **kwargs)
        if not N_exchanges:
            self.N_exchanges = self.N_actors
        else:
            self.N_exchanges = N_exchanges

    def run_round(self):
        """Run one round of the Garage Sale Model biased towards the
           wealthier party
        """
        N = self.N_actors
        N_exchanges = self.N_exchanges
        ts = self.transaction_size
        bank = self.bank
        total_wealth = np.sum(bank)
        # Set up transaction information
        wealth_exchanged = np.zeros(N_exchanges)
        actor_exchanges = np.zeros(N_exchanges)
        partner_exchanges = np.zeros(N_exchanges)
        transactions = np.zeros(N)
        # Get actor/partner wealth
        actors = np.random.permutation(N)[:N_exchanges]
        partners = np.random.permutation(N)[:N_exchanges]
        actor_wealth = bank[actors]
        partner_wealth = bank[partners]
        # Determine amount of wealth exchanged
        actor_is_wealthier = (actor_wealth > partner_wealth)
        wealth_exchanged[actor_is_wealthier] = ts*partner_wealth[actor_is_wealthier]
        wealth_exchanged[~actor_is_wealthier] = ts*actor_wealth[~actor_is_wealthier]
        # Determine actor/partner exchanges
        prob_minus1 = 0.5*(1-self.zeta*(actor_wealth-partner_wealth)/(total_wealth/N))
        actor_won_toss = (np.random.rand(N_exchanges) > (prob_minus1))
        actor_exchanges = wealth_exchanged*(-1)**(~actor_won_toss)
        partner_exchanges = (-1)*actor_exchanges
        # Complete transactions
        transactions[actors] += actor_exchanges
        transactions[partners] += partner_exchanges

        return transactions

    def redistribute(self, transactions):
        """Apply wealth tax after running base model"""
        N_actors = self.N_actors
        bank = self.bank
        # Redistribute wealth
        actor_wealth = bank+transactions
        mean_wealth = np.ones(N_actors)*np.mean(actor_wealth)
        transactions += (mean_wealth-(actor_wealth))*self.chi

        return transactions

    def run(self):
        """Run simulation"""
        for e in tqdm(range(self.N_epochs)):
            bank = self.bank
            kappa = self.kappa
            # Apply loans
            loans = np.ones(self.N_actors)*(kappa/(1-kappa))*np.mean(bank)
            self.bank += loans
            # Run transactions
            transactions = self.run_round()
            redistributed = self.redistribute(transactions)
            # Update bank with remainder after loan collection
            self.update_bank(redistributed-loans, e)
```

## 6. Simulating Real-world Economies

### 6.1 The United States
According to the article, the 2016 U.S. household wealth distribution is best described as having  $\chi = 0.036$, $\zeta = 0.050$ and $\kappa = 0.058$.

#### 6.1.1 Simulation

```python
united_states = Affine(
    N_epochs=2000,
    N_checkpoints=200,
    transaction_size=0.25,
    starting_wealth=100,
    N_actors=5000,
    chi=0.036,
    kappa=0.058,
    zeta=0.05
)
```

```python
united_states.run()
```

    HBox(children=(FloatProgress(value=0.0, max=2000.0), HTML(value='')))

#### 6.1.2 Plots

```python
united_states.plot_wealth_dist()
```

```python
united_states.plot_wealth_ineq()
```

```python
united_states.plot_lorenz_curve(N_points=10000)
```

```python
united_states.plot_gini_evolution()
```

## Bibliography
1. https://www.scientificamerican.com/article/is-inequality-inevitable/
2. https://arxiv.org/abs/1511.00770
3. https://arxiv.org/abs/1604.02370
4. https://arxiv.org/abs/1212.6300.pdf
5. https://en.wikipedia.org/wiki/Gini_coefficient
6. https://en.wikipedia.org/wiki/List_of_countries_by_wealth_equality


Contenido bajo licencia Creative Commons BY 4.0 y código bajo licencia MIT. © Juan Gómez y Nicolás Guarín-Zapata 2020. Este material es parte del curso Modelación Computacional en el programa de Ingeniería Civil de la Universidad EAFIT.

# Lectura y procesamiento de mallas de Gmsh

## Introducción

Discutiremos algunos aspectos sobre la creación de mallas de Gmsh,
cómo importarlas y analizar datos asociados a grupos físicos.

**Al completar este notebook usted debería estar en la capacidad de:**

* Entender el concepto de grupo físico en Gmsh.

* Recopilar información asociada a un grupo físico específico.

* Entender el uso de parámetros de mallado en Gmsh.

## Ejemplo: Visualización del esfuerzos para un problema de elasticidad plana

Queremos visualizar el campo de esfuerzos para una placa (infinitamente) grande con un agujero de radio $R$ y sometida a un esfuerzo $p$ remotamente, como se muestra a continuación.

Los esfuerzos en este problema están dados por

\begin{align*}
 \sigma_{rr} &= -\left(1 - \frac{R_1^2}{r^2}\right) p\, ,\\
 \sigma_{\theta\theta} &= -\left(1 + \frac{R_1^2}{r^2} \right) p\, ,\\
 \tau_{r\theta} &= 0\, .
\end{align*}

### Geometría para este problema

Estos son los puntos de nuestro modelo

Los puntos 2, 3, 4, y 5 tienen las siguientes coordenadas

$$
x = \pm R \cos\left(\frac{\pi}{4}\right)\, ,\quad
y = \pm R \sin\left(\frac{\pi}{4}\right)\, .
$$

Si conectamos los puntos, obtenemos las siguientes líneas.

Los contornos (y superficies) los definimos usando la regla de la mano derecha.

Adicionalmente de la información geométrica,
podemos crear grupos (de puntos, líneas, superficies) que
luego podemos usar en nuestras tareas.

## Visualización

Inicialmente, veremos cómo visualizar este campo usando únicamente Python. Luego, usaremos Gmsh
para la creación de la geometría y el paquete [meshio](https://github.com/nschloe/meshio) para
importar la malla Python.

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.tri import Triangulation
import meshio
```

La siguiente celda configura algunos aspectos de las gráficas en este Notebook.

```python
%matplotlib notebook
gris = '#757575'
plt.rcParams["mathtext.fontset"] = "cm"
plt.rcParams["xtick.color"] = gris
plt.rcParams["ytick.color"] = gris
plt.rcParams["axes.labelcolor"] = gris
plt.rcParams["axes.edgecolor"] = gris
plt.rcParams["axes.spines.right"] = False
plt.rcParams["axes.spines.top"] = False
```

Definimos una función que me calcule el campo de esfuerzos

```python
def esfuerzos_placa(r, R, p):
    """Campo de esfuerzos para placa con agujero y carga remota

    Los parámetros de entrada son:
      r : coordenada del punto.
      R : radio interno de la placa.
      p : valor de la carga remota.
    """
    return -(1 - R**2/r**2)*p,  -(1 + R**2/r**2)*p
```

## Visualización usando Python únicamente

Si consideramos el dominio de interés como un disco, esta visualización puede ser hecha completamente
en Python.

Inicialmente, definimos las coordenadas de los puntos. Por las características del problema, usaremos
coordenadas polares.

```python
R_in = 1.0
R_out = 10.0
```

```python
r = np.linspace(R_in, R_out, 20)
theta = np.linspace(0, 2*np.pi, 20)
```

Ahora, para poder visualizar la solución en 2D necesitamos una rejilla con las coordenadas
$(r, \theta)$. Por suerte, la función ``meshgrid`` de NumPy nos permite crearla.

```python
r, theta = np.meshgrid(r, theta)
```

Si preferimos trabajar en coordenadas cartesianas podemos usar

\begin{align}
x &= r\cos\theta\\
y &= r\sin\theta
\end{align}

```python
x = r*np.cos(theta)
y = r*np.sin(theta)
```

Y si hacemos un gráfico de los puntos obtenemos lo siguiente.

```python
plt.figure()
plt.plot(x, y, ".", color="gray")
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.axis("image");
```

    <IPython.core.display.Javascript object>

Calculamos los esfuerzos para los puntos

```python
sigma_rr, sigma_tt = esfuerzos_placa(r, R_in, 1.0)
```

```python
plt.figure(figsize=(6, 3))

# sigma_rr
plt.subplot(1, 2, 1)
plt.contourf(x, y, sigma_rr)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.colorbar(shrink=0.5)
plt.axis("image")

#sigma_tt
plt.subplot(1, 2, 2)
plt.contourf(x, y, sigma_tt)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.colorbar(shrink=0.5)
plt.axis("image")
plt.tight_layout();
```

    <IPython.core.display.Javascript object>

## Visualización creando la geometría en Gmsh

Ahora, si la geometría es más complicada no tendremos una parametrización sencilla
de las coordenadas y no será práctico hacerlo completamente en Python.

En esta sección vamos a leer un archivo de malla creado en Gmsh y luego vamos a
visualizar el mismo campo de esfuerzos.

Podemos leer los archivos de malla generados con Gmsh usando meshio, como se muestra a continuación

```python
mesh = meshio.read("files/cuadrado_hueco.msh")
points = mesh.points
cells = mesh.cells
point_data = mesh.point_data
cell_data = mesh.cell_data
```

```python
x = points[:, 0]
y = points[:, 1]
r = np.sqrt(x**2 + y**2)
theta = np.arctan2(y, x)
triangulos = cells["triangle"]
```

Calculamos los esfuerzos para cada punto en la malla

```python
R = 1.0
p = 1.0
sigma_rr, sigma_tt = esfuerzos_placa(r, R, p)
```

Y graficamos los esfuerzos

```python
plt.figure()
plt.tricontourf(x, y, triangulos, sigma_rr)
plt.colorbar(shrink=0.8)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.axis("image");
```

    <IPython.core.display.Javascript object>

```python
plt.figure()
plt.tricontourf(x, y, triangulos, sigma_tt)
plt.colorbar(shrink=0.8)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.axis("image");
```

    <IPython.core.display.Javascript object>

### Visualización de los esfuerzos en la superficie libre y radialmente

```python
grupos_lineas = cell_data["line"]["gmsh:physical"]
lineas = cells["line"]
```

Para determinar las lineas que corresponden al círculo interior podemos
comparar cuáles líneas tienen el identificador 1. De la siguiente
manera podemos hacerlo

```python
circulo = np.array(lineas[grupos_lineas == 1])
print(circulo)
```

    [[  0 480]
     [480 481]
     [481 482]
     [482 483]
     [483 484]
     [484 485]
     [485 486]
     [486 487]
     [487 488]
     [488 489]
     [489 490]
     [490 491]
     [491 492]
     [492 493]
     [493 494]
     [494 495]
     [495 496]
     [496 497]
     [497 498]
     [498 499]
     [499 500]
     [500 501]
     [501 502]
     [502 503]
     [503 504]
     [504 505]
     [505 506]
     [506 507]
     [507 508]
     [508 509]
     [509 510]
     [510 511]
     [511 512]
     [512 513]
     [513 514]
     [514 515]
     [515 516]
     [516 517]
     [517 518]
     [518   1]
     [  1 519]
     [519 520]
     [520 521]
     [521 522]
     [522 523]
     [523 524]
     [524 525]
     [525 526]
     [526 527]
     [527 528]
     [528 529]
     [529 530]
     [530 531]
     [531 532]
     [532 533]
     [533 534]
     [534 535]
     [535 536]
     [536 537]
     [537 538]
     [538 539]
     [539 540]
     [540 541]
     [541 542]
     [542 543]
     [543 544]
     [544 545]
     [545 546]
     [546 547]
     [547 548]
     [548 549]
     [549 550]
     [550 551]
     [551 552]
     [552 553]
     [553 554]
     [554 555]
     [555 556]
     [556 557]
     [557   2]
     [  2 558]
     [558 559]
     [559 560]
     [560 561]
     [561 562]
     [562 563]
     [563 564]
     [564 565]
     [565 566]
     [566 567]
     [567 568]
     [568 569]
     [569 570]
     [570 571]
     [571 572]
     [572 573]
     [573 574]
     [574 575]
     [575 576]
     [576 577]
     [577 578]
     [578 579]
     [579 580]
     [580 581]
     [581 582]
     [582 583]
     [583 584]
     [584 585]
     [585 586]
     [586 587]
     [587 588]
     [588 589]
     [589 590]
     [590 591]
     [591 592]
     [592 593]
     [593 594]
     [594 595]
     [595 596]
     [596   3]
     [  3 597]
     [597 598]
     [598 599]
     [599 600]
     [600 601]
     [601 602]
     [602 603]
     [603 604]
     [604 605]
     [605 606]
     [606 607]
     [607 608]
     [608 609]
     [609 610]
     [610 611]
     [611 612]
     [612 613]
     [613 614]
     [614 615]
     [615 616]
     [616 617]
     [617 618]
     [618 619]
     [619 620]
     [620 621]
     [621 622]
     [622 623]
     [623 624]
     [624 625]
     [625 626]
     [626 627]
     [627 628]
     [628 629]
     [629 630]
     [630 631]
     [631 632]
     [632 633]
     [633 634]
     [634 635]
     [635   0]]

Podemos ver que esta lista nos muestra los nodos que forman cada línea en el
círculo interior, sin embargo, algunos puntos aparecen más de una vez.

El siguiente código nos permite tener un arreglo con los nodos sin repeticiones.

```python
circulo = circulo.flatten()
circulo = list(set(circulo))
print(circulo)
```

    [0, 512, 513, 514, 515, 516, 517, 518, 1, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 2, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 3, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511]

```python
theta_circ = theta[circulo]
sigma_rr_circ = sigma_rr[circulo]
sigma_tt_circ = sigma_tt[circulo]
```

Ahora, necesitamos ordenar estos valores de acuerdo a theta.

```python
orden_theta = np.argsort(theta_circ)
```

```python
plt.figure()
plt.plot(theta_circ[orden_theta], sigma_rr_circ[orden_theta])
plt.plot(theta_circ[orden_theta], sigma_tt_circ[orden_theta])
plt.xlabel(r"$\theta$")
plt.ylabel("Esfuerzo sobre la superficie libre")
plt.legend([r"$\sigma_{rr}$", r"$\sigma_{\theta\theta}$"])
plt.ylim(-3, 3);
```

    <IPython.core.display.Javascript object>

Podemos repetir este proceso para la el grupo asociado al identificador 2.

```python
radio = lineas[grupos_lineas == 2]
radio = radio.flatten()
radio = list(set(radio))
rad = r[radio]
sigma_rr_rad = sigma_rr[radio]
sigma_tt_rad = sigma_tt[radio]
orden_r = np.argsort(rad)
```

Y podemos ver cómo varía el esfuerzo conforme nos alejamos del origen.

```python
plt.figure()
plt.plot(rad[orden_r], sigma_rr_rad[orden_r])
plt.plot(rad[orden_r], sigma_tt_rad[orden_r])
plt.xlabel(r"$r$")
plt.ylabel("Esfuerzo")
plt.legend([r"$\sigma_{rr}$", r"$\sigma_{\theta\theta}$"]);
```

    <IPython.core.display.Javascript object>

## Actividad para la clase

Dada una presa triangular como la que se muestra a continuación

analice los esfuerzos en un notebook nuevo.

Para este problema los esfuerzos están dados por:

\begin{align}
&\sigma_{xx} = -\gamma(H - y)\, ,\\
&\sigma_{yy} = \gamma(H - y) - 2\gamma x\, ,\\
&\tau_{xy} = -\gamma x\, .
\end{align}

La geometría se encuentra en el archivo `presa_triangular.geo` y
la malla en el archivo `presa_triangular.msh`.

## Referencias bibliográficas

1. Gómez, Juan., Sierra, César., Vergara, Juan., Sáenz, Mario., and Guarín-Zapata, Nicolás. (2018) [Notas de clase: Mecánica del medio continuo](https://github.com/AppliedMechanics-EAFIT/Notas-MMC/raw/master/notas_de_clase/notas_medios.pdf). Universidad EAFIT.

## Formato del notebook

La siguiente celda cambia el formato del Notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>