# **Выводы в конце ноутбука:** Пишем свой фреймворк

Материалы:

* [Andrew Karpahy: yes, you should understand backprop](https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b)
* [Stanford CS231n](http://cs231n.stanford.edu/)
* [Deep Learning](http://sereja.me/f/deep_learning_goodfellow.pdf) — с 204 страницы и до прозрения
* [Xavier, Bengio](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)

```python
# только numpy, только хардкор
import numpy as np

# !! для проверки слоев и фунций, не более
import torch
import torch.nn as nn

```

Мотивация: конечным пользователям вашего фреймворка не хочется думать, как они работают слои внутри. Им просто хочется объявить последовательность элементарных операций над входными данными, а о градиентах и прочем матане пусть позаботится сам фреймворк.

**Module** — это абстрактный класс, от которого будут наследоваться слои нашей нейронной сети. Абстрактные классы нужны, чтобы можно было реализовывать не все методы, а только переопределить некоторые. Все в лучших традициях ООП.

Модуль — это такая чёрная коробка, которая
1. Умеет принимать какие-то входные данные $X$ и возращать какие-то выходные данные $Y$ (`forward`)
2. Возможно, имеет какие-то параметры, которые можно изменять, (`parameters`, `grad_parameters`)
3. Будучи встроенной в вычислительный граф, умеет по градиенту относительно своих выходных значений вычислять градиент относительно входных данных, а также собственных параметров (`backward`)
4. Умеет переключаться в режимы обучения и инференса, если они отличаются (`train`, `eval`)

Теперь поподробнее.

## Входные данные

Современные нейросети оптимизируют различными вариантами стохастического градиентного спуска, и мы тоже будем его использовать. Его отличие от обычного в том, что на каждом шаге мы не считаем градиент на всем датасете (это было бы слишком долго), а оцениваем его, усреднив градиенты на его случайно выбранной малой части, которую называют батчем (`batch`). Если батч формировать случайно, и если его размер достаточно большой, то мы можем быстро получить немного шумную, но приемлемую для нас оценку градиента, и не прогонять через сеть все миллионы примеров ради одного маленького шага. Эта интуиция ограничивает размер батча сверху.

Математик бы принял время прогона одного примера по всей сети за константу и пришел бы к выводу, что нужно считать по одному примеру и делать каждый раз один шаг, но маленький. Это верное заключение, но в реальности, если увеличить размер батча в $k$ раз, то он будет работать не в $k$ раз дольше, а намного меньше.

Самая долгая операция в большинстве нейросетей — это перемножение матриц. Начиная с каких-то размеров для их перемножения имеет использовать алгоритм Штрассена, который работает уже быстрее, чем линейно. Проведём небольшой вычислительный эксперимент:

```python
A = np.random.randn(256, 2000)
B = np.random.randn(2000, 800)

# помножить каждый вектор-строку на B и сконкатенировать
%time C = np.stack(np.dot(A[i].T, B) for i in range(256))

# это то же самое, что использовать одно-большое матричное умножение
%time C = np.dot(A, B)
```

    E:\Users\vsmel\AppData\Local\Programs\Python\Python39\lib\site-packages\IPython\core\magic.py:187: FutureWarning: arrays to stack must be passed as a "sequence" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.
      call = lambda f, *a, **k: f(*a, **k)

    Wall time: 271 ms
    Wall time: 22 ms

Такая чисто вычислительная причина ограничивает размер батча снизу. На практике, в большинстве случаев оптимальный размер батча — несколько сотен. В случае с CPU это несколько десятков, потому что выгода от распараллеливания вычислений не такая сильная.

Вообще, почти все наши слои будут работать с векторами независимо, но из-за вычилсительных причин мы будем объединять их в матрицы. Вообще, более сложные нейросети работают с тензорами. «Тензор» это, вообще говоря, сложный математический объект, но в DL этот термин используется просто в занчении «многомерный массив». Например, картинки — это четырехмерный тензор: `[batch, channel, x, y]`.

### Forward

Эта функция просто принимает тензор (`numpy.ndarray`) и возвращает какой-то другой, над которым применили соответствующие операции.

Важный нюанс: нам позже для реализации `backward` почти всегда будет нужно сохранять где-нибудь выход `forward` (это создает очень большую нагрузку на память при обучении; [в принципе это можно и не делать](https://arxiv.org/pdf/1604.06174.pdf), но так проще). Условимся сохранять его в `self.output`, сразу после того, как посчитали.

### Параметры

Параметр модели — это что-то, что можно поодгонять, чтобы функция потерь стала меньше. Он должен быть доступен оптимизатору, а оптимизатору не обязательно знать, как всё у слоя все внутри работает. Ему нужны просто градиенты — насколько ему нужно подвинуть параметры сети, чтобы стало лучше.

Общаться с ним мы будем посредством двух функций: `params` и `grad_params`. Обе возвращают списки из тензоров — значения параметров и их посчитанных градиентов (см. `backward`) соответственно. Питон делает shallow copy, поэтому у оптимизатора так есть доступ на их изменение.

### Backward

После того, как мы в вычислительном графе все последовательно посчитали и дошли до функции потерь, нам надо подогнать параметры так, чтобы на тех же данных при повторном прогоне она стала меньше — иными словами, нам надо сделать шаг против градиента функции потерь относительно параметров сети.

Посчитать эти градиенты — нетривиальная задача. Мы могли бы рассмотреть каждый параметр по отдельности и как-нибудь посчитать градиент для него. Но это очень долго — параметров в современных сетях бывает по несколько миллионов.

Вместо этого мы применим трюк, основанный на формуле для производной сложной функции:

$$ f(g(x))' = f'(g(x)) \cdot g'(x) $$

Представьте, что часть сети от параметра до выхода — это всего две последовательно выполненные функции: $g$ и $f$. Тогда, согласно формуле, нам для этого параметра достаточно посчитать и перемножить две величины — $g'(x)$ (производная текущего слоя) и f'(g(x)) (производная относительно выхода текущего слоя).

Какие-то другие параметры могли тоже зависеть от производной относительно выхода.  и мы получаем выигрыш за счет того, что считаем её только один раз и запоминаем. Можно сказать, что мы применяем таким образом динамическое программирование на вычислительном графе, чтобы посчитать градиенты относительно всех его параметров.

Обратный прогон (`backward`) определяется для каждого слоя и нужен как раз для подсчета градиентов, имея градиент относительно своих выходных значений (аналог $f'(g(x)))$.

Он должен делать две вещи:

1. Посчитать градиент относительно собственных параметров.
2. Посчитать и вернуть градиент относительно своих входных данных.

Для лучшего понимания рассмотрите пример с `Linear` и `ReLU`.

### train / eval

Некоторые слои ведут себя по-разному во время обучечния и предсказания (`inference`). Обычно, это связано с разного вида регуляризацией — например, так ведут себя `BatchNorm` и `Dropout`.

По сути, для таких слоев нужно просто написать два разных `forward`-а для обучения и инференса.

```python
class Module():
    _MAX_VALUE_E = 200.

    def __init__(self):
        self._train = True

    def forward(self, input):
        raise NotImplementedError

    def backward(self,input, grad_output):
        raise NotImplementedError

    def parameters(self):
        'Возвращает список собственных параметров.'
        return [] # возврат по ссылке, чтобы оптимизатор их изменял,
                  # но обычно они по ссылке

    def grad_parameters(self):
        'Возвращает список тензоров-градиентов для своих параметров.'
        return []

    def train(self):
        self._train = True

    def eval(self):
        self._train = False
```

Это **абстрактный класс** — от него наследуются другие слои, в которых эти функции будут реализованы.

# Sequential

**Sequential** будет оборачивать список модулей и выполнять их последовательно.

Это своего рода контейнер, внутри которого есть какой-то пайплайн.

Можно даже засовывать один Sequential внутри другого.

Многие не знают, но в питоне почти всегда для итерирования используется не **deep copy**, а **shallow copy**. Это делается для экономии памяти.

```python
class Sequential(Module):
    def __init__ (self, *layers):
        super().__init__()
        self.layers = layers

    def forward(self, input):
        """
        Прогоните данные последовательно по всем слоям:

            y[0] = layers[0].forward(input)
            y[1] = layers[1].forward(y_0)
            ...
            output = module[n-1].forward(y[n-2])

        Это должен быть просто небольшой цикл: for layer in layers...

        Хранить выводы ещё раз не надо: они сохраняются внутри слоев после forward.
        """

        for layer in self.layers:
            input = layer.forward(input)

        self.output = input
        return self.output

    def backward(self, input, grad_output):
        """
        Backward -- это как forward, только наоборот. (с)

        Предназначение backward:
        1. посчитать градиенты для собственных параметров
        2. передать градиент относительно своего входа

        О своих параметрах модули сами позаботятся. Нам же нужно позаботиться о передачи градиента.

            g[n-1] = layers[n-1].backward(y[n-2], grad_output)
            g[n-2] = layers[n-2].backward(y[n-3], g[n-1])
            ...
            g[1] = layers[1].backward(y[0], g[2])
            grad_input = layers[0].backward(input, g[1])

        Тут цикл будет уже чуть посложнее.
        """

        for i in range(len(self.layers)-1, 0, -1):
            grad_output = self.layers[i].backward(self.layers[i-1].output, grad_output)

        grad_input = self.layers[0].backward(input, grad_output)

        return grad_input

    def parameters(self):
        'Можно просто сконкатенировать все параметры в один список.'
        res = []
        for l in self.layers:
            res += l.parameters()
        return res # возвращаем список nampy arrays, чтобы работало для слоев без параметров

    def grad_parameters(self):
        'Можно просто сконкатенировать все градиенты в один список.'
        res = []
        for l in self.layers:
            res += l.grad_parameters()
        return res

    def train(self):
        for layer in layers:
            layer.train()

    def eval(self):
        for layer in layers:
            layer.eval()
```

# Слои

Приступим к реализации содержательной части — самих слоев.

На вход всех слоев будет подаваться матрица размера `batch_size` $\times$ `n_features` (см. описание `forward`).

Начнем с основного: линейный слой, он же fully-conected.

$$ Y = X W + b $$

Правильнее его называть афинным: после матричного умножения добавляется вектор $b$.

`forward` у него трививальный, а `backward` уже сложнее: нужно посчитать градиенты относительно трёх вещей:
1. Входных данных. Автор добродушен и спалит вам ответ, а вам нужно его доказать:

$$\nabla X = \nabla Y \cdot W^T$$

2. Матрица весов $W$. Тут нужно подумать, как каждый вес влияет на каждое выходное значение, и выразить ваши мысли линейной алгеброй.
3. Вектор $b$. С ним всё будет просто.

Не забудьте, что `grad_params` должен иметь такие же размерности, как и соответствующие параметры.

```python
class Linear(Module):
    def __init__(self, dim_in, dim_out):
        super().__init__()

        # Xavier initialization: инциализируем так,
        # что если на вход идет N(0, 1)
        # то и на выходе должно идти N(0, 1)
        stdv = 1./np.sqrt(dim_in)
        self.W = np.random.uniform(-stdv, stdv, size=(dim_in, dim_out))
        self.b = np.random.uniform(-stdv, stdv, size=dim_out)

    def forward(self, input):
        self.output = np.dot(input, self.W) + self.b
        return self.output

    def backward(self, input, grad_output):
        self.grad_b = np.sum(grad_output, axis=0) # !! mean?

        #     in_dim x batch_size
        self.grad_W = np.dot(input.T, grad_output)
        #                 batch_size x out_dim

        grad_input = np.dot(grad_output, self.W.T)

        return grad_input

    def parameters(self):
        return [self.W, self.b]

    def grad_parameters(self):
        return [self.grad_W, self.grad_b]
```

## Функции активации

**ReLU** — одна из самых простых функций активации:

$$
ReLU(x)=
\begin{cases}
x, & x > 0\\
0, & x \leq 0\\
\end{cases}
$$

`ReLU` это очень простой слой, поэтому автору не жалко её реализовать его за вас:

```python
class ReLU(Module):
    def __init__(self):
         super().__init__()

    def forward(self, input):
        self.output = np.maximum(input, 0)
        return self.output

    def backward(self, input, grad_output):
        grad_input = np.multiply(grad_output, input > 0)
        return grad_input
```

У ReLU есть проблема — у него бесполезная нулевая производная при $x < 0$.

[**Leaky Rectified Linear Unit**](http://en.wikipedia.org/wiki%2FRectifier_%28neural_networks%29%23Leaky_ReLUs) — это его модифицированная версия, имеющая в отрицательных координатах не нулевой градиент, а просто помноженный на маленькую константу `slope`.

$$
LeakyReLU_k(x)=
\begin{cases}
x, & x > 0\\
kx, & x \leq 0\\
\end{cases}
$$

При `slope` = 0 он превращается в обычный `ReLU`.

```python
class LeakyReLU(Module):
    def __init__(self, slope=0.03):
        super().__init__()

        self.slope = slope

    def forward(self, input):
        #mask = (input > 0)
        #coef = mask + (1-mask)*self.slope
        #self.output = np.multiply(input, coef)
        self.output = np.maximum(input, self.slope*input)
        return self.output

    def backward(self, input, grad_output):
        mask = (input > 0)
        coef = mask + (1-mask)*self.slope
        grad_input = np.multiply(grad_output, coef)
        return grad_input
```

```python
# Проверка кода
layer = LeakyReLU(0.05)
x = np.array([1., 2., -3., 4., -7.])

print(layer.forward(x))

print(layer.backward(x, np.ones(5)))

torch_layer = nn.LeakyReLU(0.05)
torch_layer.requires_grad_ = True

x_tensor = torch.tensor(x)

print(torch_layer.forward(x_tensor))

# print(torch_layer.)
# !! получить backward от torch

```

    [ 1.    2.   -0.15  4.   -0.35]
    [1.   1.   0.05 1.   0.05]
    tensor([ 1.0000,  2.0000, -0.1500,  4.0000, -0.3500], dtype=torch.float64)

**Сигмоида** определяется формулой $\sigma(x) = \frac{1}{1+e^{-x}}$.

Когда-то она была самой часто используемой функции активации, потому что имела логичную вероятностную интерпретацию (вероятность наличия какой-то фичи), но потом перестали, потому что на очень больших или маленьких значениях её производные почти нулевые (см. проблема затухающего градиента).

Также используют [гипреболический тангенс](https://ru.wikipedia.org/wiki/%D0%93%D0%B8%D0%BF%D0%B5%D1%80%D0%B1%D0%BE%D0%BB%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%B8%D0%B5_%D1%84%D1%83%D0%BD%D0%BA%D1%86%D0%B8%D0%B8), который на самом деле просто сигмоида, отнормированная так, чтобы значения были в $[-1, 1]$: $tanh(x) = 2 \sigma(x) - 1$. Мы его отдельно реализовывать не будем.

Давайте посчитаем её производную:

$$
\begin{align}
\sigma'(x) &= (\frac{1}{1+e^{-x}})'
\\         &= \frac{e^{-x}}{(1+e^{-x})^2}
\\         &= \frac{1+e^{-x}-1}{(1+e^{-x})^2}
\\         &= \frac{1+e^{-x}}{(1+e^{-x})^2} - \frac{1}{(1+e^{-x})^2}
\\         &= \frac{1}{1+e^{-x}} - \frac{1}{(1+e^{-x})^2}
\\         &= \sigma(x) - \sigma(x)^2
\\         &= \sigma(x)(1 - \sigma(x))
\end{align}
$$

```python
class Sigmoid(Module):
    def __init__(self):
        super().__init__()

    def forward(self, input):
        input_clamp = np.clip(input, -self._MAX_VALUE_E, self._MAX_VALUE_E)
        self.output = 1 / (1 + np.exp(-input_clamp))
        return self.output

    def backward(self, input, grad_output):
        input_clamp = np.clip(input, -self._MAX_VALUE_E, self._MAX_VALUE_E)
        sigm = 1 / (1 + np.exp(-input_clamp))
        grad_input = grad_output * sigm * (1 - sigm)
        return grad_input
```

```python
# Проверка кода
layer = Sigmoid()
x = np.array([1., 2., -3., 4., -7.])

print("My forward:\n", layer.forward(x))

print("My backward:\n", layer.backward(x, np.ones(5)))

torch_layer = nn.Sigmoid()
torch_layer.requires_grad_ = True

x_tensor = torch.tensor(x)

print("Torch forward:\n", torch_layer.forward(x_tensor))

# print(torch_layer.)
# !! получить backward от torch
```

    My forward:
     [7.31058579e-01 8.80797078e-01 4.74258732e-02 9.82013790e-01
     9.11051194e-04]
    My backward:
     [0.19661193 0.10499359 0.04517666 0.01766271 0.00091022]
    Torch forward:
     tensor([7.3106e-01, 8.8080e-01, 4.7426e-02, 9.8201e-01, 9.1105e-04],
           dtype=torch.float64)

**Софтмакс** определяется так:

$$ \sigma(x)_k = \frac{e^{x_k}}{\sum_{i=1}^n e^{x_i} }$$

Можно заметить, что сигмоида — это частный случай софтмакса. Его можно интерпретировать как вероятностное распределение: его выходы положительны и суммируются в единицу. Поэтому его используют как последний слой для классификации.

Софтмакс — самый сложный с точки зрения написания `backward`. Как и все остальное, оно считается в 5 строчек кода, но [вывести их трудно](https://deepnotes.io/softmax-crossentropy).

```python
class SoftMax(Module):
    def __init__(self):
         super().__init__()

    def forward(self, input):
        batch_size = input.shape[0]
        input_clamp = np.subtract(input, input.max(axis=1, keepdims=True))

        # важная деталь: если входы большие,
        # то экспоненты будут ещё больше
        # дополнительно обрежем маленькие inputы, хотя эту строчку можно и убрать
        input_clamp = np.clip(input_clamp, -self._MAX_VALUE_E, 10e-10)
        exps = np.exp(input_clamp)
        # print("exps:", exps)
        # grad[range(batch_size), input] -= 1
        sums = np.sum(exps, axis=1, keepdims=True)
        # print("sums:", sums)
        self.output = exps / sums

        # self.output = np.true_divide(exps, np.sum(exps, axis=1)[:, None])

        return self.output

    def backward(self, input, grad_output):
        """batch_size = input.shape[0]
        samples = input.shape[1]
        input_clamp = np.subtract(input, input.max(axis=1, keepdims=True))
        input_clamp = np.clip(input_clamp, -self._MAX_VALUE_E, 10e-10)
        exps = np.exp(input_clamp)

        sums = np.sum(exps, axis=1, keepdims=True)
        soft_a = exps / sums"""

        # a_row = soft_a
        # a_col = soft_a.reshape(batch_size, 1, soft_a.shape[1])

        # b = a_row*a_col

        # mask = 1

        # grad_input = np.array(map(lambda x: np.sum(map(lambda y: (x*(x==y) - y), soft_a)), soft_a, range(10)))

        # C = [[0 for row in range(len(B[0]))] for col in range(len(A))]

        # grad_input = [[[grad_output[k][i]*sum(soft_a[k][i]*((i == j) - soft_a[k][j])) for j in range(
        #    samples)] for i in range(samples)] for k in range(batch_size)]

        # grad_input = np.zeros((batch_size, samples))
        # grad_input = list(grad_input)
        # soft_a = list(soft_a)
        # grad_output = list(grad_output)
        # for k in range(batch_size):
        #     for i in range(samples):
        #         for j in range(samples):
        #             grad_input[k][i] += grad_output[k][i]*(soft_a[k][i]*((i == j) - soft_a[k][j]))

        # s = grad_output.reshape(-1, 1)
        # grad_input = np.diagflat(s) - np.dot(s, s.T)

        batch_size = input.shape[0]
        grad_input = []
        for i in range(batch_size):
            s = input[i, :].reshape(-1, 1)
            j = np.diagflat(s) - np.dot(s, s.T)

            grad_input.append(
                (np.dot(j, grad_output[i, :].reshape(-1, 1))).reshape(input.shape[1], ))

        grad_input = np.array(grad_input)  # * (-1)

        # batch_size = input.shape[0]
        # s = self.output
        # grad_local = -s[..., None] * s[:, None, :]
        # iy, ix = np.diag_indices_from(grad_local[0])
        # grad_local[:, iy, ix] = s * (1 - s)

        # for i in range(grad_local.shape[0]):
        #     grad_local[i] = grad_local[i] * grad_output[i]

        # grad_input = np.sum(grad_local, 2).reshape(batch_size, -1)
        # grad_input = grad_input * (-1)

        # grad_input[k][i] = grad_output[k][i]*sum(  soft_a[i]*((i==j) - soft_a[j])  )
        return np.array(grad_input)
```

```python
x = np.array([[1., 2., 3., 4., 5.], [1., 2., 3., 4., 5.]])
layer = SoftMax()

input = torch.tensor(x, requires_grad=True)
# можно рандомный тензор ещё ^

my_loss = layer.forward(x)
my_grad = layer.backward(x, my_loss)

# torch_loss = nn.Softmax(input)
# torch_loss.backward()

# assert my_loss == torch_loss.item()  # +- eps?
# assert np.equal(my_grad, input.grad.numpy())

```

```python
#input = np.array([[1, 2, 3, 1, -1], [1, 0, 3, 1, -1], [1, 2, 0, 1, -1]])

# Проверка кода
layer = SoftMax()
x = np.array([[1., 2., 3., 4., 5.], [1., 2., 3., 4., 5.]])
#x = np.array([[1., 2., 3., 4., 5.]])

output = layer.forward(x)
print("My forward:\n", output)
output_sum = sum(output)
# print(output_sum)
# print(sum(output_sum))

print("My backward:\n", layer.backward(x, output))

torch_layer = nn.Softmax()
x_tensor = torch.tensor(x, requires_grad=True)
print("Torch forward:\n", torch_layer.forward(x_tensor))

torch_ans = torch_layer(x_tensor)
t = torch.sum(torch_ans)
print(t.grad_fn)
t.backward()
output_back_torch = x_tensor.grad

print("Torch backword:\n", output_back_torch)

# print(torch_layer.)
# !! получить backward от torch

# -------------------- backward

```

## Регуляризация

Самый популярный регуляризатор в нейросетях — [**дропаут**](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf). Идея простая: просто помножим поэлементно входные данные на случайную бинарную маску того же размера, как и сами данные. Сгенерировать маску можно через `np.random.binomial`.

Дропаута обычно хватает как единственного регуляризатора. Если вы заметите, что сеть оверфитится — просто добавьте его побольше.

**У дропаута разное поведение в режимах `train` и `eval`**. При `eval` он не должен делать ничего, а в `train` вместо применения маски нужно ещё домножить вход на $p$, чтобы скомпенсировать дропаут при обучении (так математическое ожидание значений будет такое же, как на трейне).

```python
class Dropout(Module):
    def __init__(self, p=0.5):
        super().__init__()

        self.p = p
        self.mask = None
        self.all = None # матожидание
        self.count = None
        self.first = False

    def forward(self, input):
        if self._train:
            if (not self.first):
                self.all = np.zeros(input.shape)
                self.count = np.zeros(input.shape)
                self.count += 1e-9
                self.first = True

            self.mask = np.random.binomial(1, self.p, size=input.shape)
            self.output = input * self.mask

            self.all += self.output
            self.count += self.mask
        else:
            mean = self.all/self.count
            self.output = input*mean
        return self.output

    def backward(self, input, grad_output):
        if self._train:
            grad_input = grad_output * self.mask
        else:
            grad_input = grad_output
        return grad_input
```

```python
layer = Dropout()
```

```python
#layer.eval()
x = np.array([[1., 2., 3., 4., 5.], [1., 2., 3., 4., 5.]])
y = np.ones((2, 5))
#x = np.array([[1., 2., 3., 4., 5.]])

output = layer.forward(x)
print("My forward:\n", output)
output_sum = sum(output)
# print(output_sum)
# print(sum(output_sum))

print("My backward:\n", layer.backward(x, output))

# Dropout() in train state
```

    My forward:
     [[1. 0. 3. 4. 0.]
     [1. 2. 3. 4. 5.]]
    My backward:
     [[1. 0. 3. 4. 0.]
     [1. 2. 3. 4. 5.]]

```python
output = layer.forward(y)
print("My forward:\n", output)
output_sum = sum(output)
# print(output_sum)
# print(sum(output_sum))

print("My backward:\n", layer.backward(y, output))
```

    My forward:
     [[1. 1. 0. 0. 0.]
     [0. 1. 1. 1. 1.]]
    My backward:
     [[1. 1. 0. 0. 0.]
     [0. 1. 1. 1. 1.]]

```python
layer.eval()
x = np.array([[1., 2., 3., 4., 5.], [1., 2., 3., 4., 5.]])
#x = np.array([[1., 2., 3., 4., 5.]])

output = layer.forward(y)
print("My forward:\n", output)
# Dropout() in eval state
```

    My forward:
     [[1.  1.  3.  4.  0. ]
     [1.  1.5 2.  2.5 3. ]]

`BatchNorm` -- относительно современный слой, сильно улучшающий сходимость. Всё, что он делает -- это нормирует свои входные значения так, что на выходе получаются значения со средним 0 и дисперсией 1.

Почитать про вывод градиента для него можно тут: https://wiseodd.github.io/techblog/2016/07/04/batchnorm/

BatchNorm тоже по-разному ведёт себя при обучении и инференсе. Во время инференса он использует в качестве оценки среднего и дисперсии свои экспоненциально усреднённые исторические значения. Это связано с тем, что батч может быть маленьким, и оценки среднего и дисперсии будут неточными (при батче размера 1 дисперсия вообще будет нулевая, и нам в алгоритме нужно будет делить на ноль).

```python
class BatchNorm(Module):
    def __init__(self, num_features, gamma):
        self.gamma = gamma
        self.beta = np.zeros((1, int(np.prod(num_features))))
        self.params = [self.gamma, self.beta]

    def forward(self, input):
        self.n_X = input.shape[0]
        self.X_shape = input.shape

        self.X_flat = input.ravel().reshape(self.n_X, -1)
        self.mu = np.mean(self.X_flat, axis=0)
        self.sigma = np.var(self.X_flat, axis=0)
        self.X_norm = (self.X_flat - self.mu) / np.sqrt(self.sigma + 1e-8)
        out = self.gamma * self.X_norm + self.beta

        self.output = out.reshape(self.X_shape)
        return self.output

    def backward(self, input, grad_output):

        grad_output = grad_output.ravel().reshape(grad_output.shape[0], -1)
        X_mu = input - self.mu
        sigma_inv = 1. / np.sqrt(self.sigma + 1e-8)

        dbeta = np.sum(grad_output, axis=0)
        dgamma = np.sum(grad_output * self.X_norm, axis=0)

        dX_norm = grad_output * self.gamma
        dsigma = np.sum(dX_norm * X_mu, axis=0) * - \
            0.5 * (self.sigma + 1e-8)**(-3 / 2)
        dmu = np.sum(dX_norm * -sigma_inv, axis=0) + dsigma * \
            1 / self.n_X * np.sum(-2. * X_mu, axis=0)
        dX = (dX_norm * sigma_inv) + (dmu / self.n_X) + \
            (dsigma * 2 / self.n_X * X_mu)

        grad_input = dX.reshape(self.X_shape)
        return  grad_input

```

```python
layer = BatchNorm(3, 0.5)
x = np.array([[1., 2., 3., 4., 5.], [1., 2., 3., 4., 5.]])
#x = np.array([[1., 2., 3., 4., 5.]])

#output = layer.forward(x)
#print("My forward:\n", output)
#output_sum = sum(output)
# print(output_sum)
# print(sum(output_sum))

#print("My backward:\n", layer.backward(x, output))

```

## Критерии

Критерии — это специальные функции, которые меряют качество, имея реальные данные и предсказанные. Все критерии возвращают скаляр — одно число, усреднённое значение метрики по всему батчу.

По сути это тоже модули, но мы всё равно создадим для них отдельный класс, потому что у них нет `train` / `eval`, а `backward` не требует `grad_output` — эта вершина и так конечная в вычислительном графе. Также нам не понадобится сохранять для них `output`.

```python
class Criterion():
    _MAX_VALUE_E = 200.
    EPS = 1e-9

    def forward(self, input, target):
        raise NotImplementedError

    def backward(self, input, target):
        raise NotImplementedError
```

В качестве примера реализуем среднюю квадратичную ошибку (`MSE`).

Обратите внимание, что в критериях мы делим итоговое число на размер батча — мы не хотим, чтобы функция потерь зависела от количества примеров.

```python
class MSE(Criterion):
    def forward(self, input, target):
        batch_size = input.shape[0]
        self.output = np.sum(np.power(input - target, 2)) / batch_size
        return self.output

    def backward(self, input, target):
        batch_size = input.shape[0]
        self.grad_output = (input - target) * 2 / batch_size
        return self.grad_output
```

Ваша задача посложнее: вам нужно реализовать кроссэнтропию — это стандартная функция потерь для классификации. Тут можно почитать про вывод её градиентов, а также софтмакса: https://deepnotes.io/softmax-crossentropy

Напоминаем интуицию за принципом максимального правдоподобия: мы максимизируем произведение предсказанных вероятностей реально случившихся событий $ L = \prod p_i $.

Произведение оптимизировать не очень удобно, и поэтому мы возьмём логарифм (любой, ведь все логарифмы отличаются в константу раз) и будем вместо него максимизировать сумму:

$$ \log L = \log \prod p_i = \sum \log p_i $$

Эту штуку называют кроссэнтропией. Такое название пошло из теории информации, но нам пока знать это не надо.

Для удобноства вместо чисел — от 0 до 9 — будем использовать вектора размера 10, где будет стоять единица в нужном месте (такое кодирование называется one-hot).

```python
"""
class CrossEntropy(Criterion):
    def __init__(self):
        super().__init__()

    def forward(self, input, target):
        # чтобы нигде не было взятий логарифма от нуля:
        input = SoftMax().forward(input)

        batch_size = input.shape[0]
        input_clamp = np.subtract(input, input.max(axis=1, keepdims=True))
        input_clamp = np.clip(input_clamp, -self._MAX_VALUE_E, 10e-10)
        exps = np.exp(input_clamp)
        sums = np.sum(exps, axis=1, keepdims=True)
        input = exps / sums

        input_clamp = np.clip(input, self.EPS, 1 - self.EPS)

        if len(target.shape) == 1:
            correct_confidences = input_clamp[range(batch_size), target]

        if len(target.shape) == 2:
            correct_confidences = np.sum(input_clamp*target, axis=1)

        # кроссэнтропия работает для всего, тк ohe
        # logs = (target == 1) * np.log(input_clamp) + \
        #     (target == 0) * np.log(1-input_clamp)
        # sums = np.sum(logs, axis=1, keepdims=True)

        # negative_log_likelihoods
        self.output = sum(-np.log(correct_confidences)) / \
            batch_size  # sum тк были батчи (найдем среднее)
        # print(logs)
        # loss(x, class) = -log(exp(x[class]) / (\sum_j exp(x[j]))) = -x[class] + log(\sum_j exp(x[j]))

        return self.output

    def backward(self, input, target):
        eps = 1e-9
        input_clamp = np.clip(input, eps, 1-eps)

        self.grad_output = target * (1 / input_clamp) + \
            (1-target) * (1/(1-input_clamp))

        return self.grad_output

"""
# кроссэнтропия работает для всего, тк ohe
class CrossEntropy(Criterion):
    def __init__(self):
        super().__init__()

    def forward(self, input, target):
        # чтобы нигде не было взятий логарифма от нуля:
        input = SoftMax().forward(input)

        batch_size = input.shape[0]
        input_clamp = np.subtract(input, input.max(axis=1, keepdims=True))
        input_clamp = np.clip(input_clamp, -self._MAX_VALUE_E, 10e-10)
        exps = np.exp(input_clamp)
        sums = np.sum(exps, axis=1, keepdims=True)
        input = exps / sums

        input_clamp = np.clip(input, self.EPS, 1 - self.EPS)

        if len(target.shape) == 1:
            correct_confidences = input_clamp[range(batch_size), target]

        if len(target.shape) == 2:
            correct_confidences = np.sum(input_clamp*target, axis=1)

        # кроссэнтропия работает для всего, тк ohe
        # logs = (target == 1) * np.log(input_clamp) + \
        #     (target == 0) * np.log(1-input_clamp)
        # sums = np.sum(logs, axis=1, keepdims=True)

        # negative_log_likelihoods
        self.output = sum(-np.log(correct_confidences))/batch_size # sum тк были батчи (найдем среднее)
        # print(logs)
        # loss(x, class) = -log(exp(x[class]) / (\sum_j exp(x[j]))) = -x[class] + log(\sum_j exp(x[j]))

        return self.output

    def backward(self, input, target):
        #input = SoftMax().forward(input)
        #input_clamp = np.clip(input, self.EPS, 1 - self.EPS)
        batch_size = input.shape[0]
        samples = input.shape[1]

        # input_clamp[range(batch_size), target] -= 1
        # input_clamp /= samples

        delta_w = input - target

        delta_w = list(delta_w)
        for i in range (batch_size):
            for j in range(samples):
                temp = delta_w[i][j]
                if (abs(temp) < self.EPS) or (abs(1-temp) < self.EPS):
                    delta_w[i][j] += 2*self.EPS

        delta_w = np.array(delta_w)
        # delta_w = ((delta_w < self.EPS) or (-self.EPS < delta_w))*(delta_w+2*self.EPS) + \
        #     (1-(delta_w < self.EPS) or (-self.EPS < delta_w))*delta_w

        self.grad_output = target * (1 / delta_w) + \
            (1-target) * (1/(1-delta_w))

        return self.grad_output

```

```python
#input = np.array([[1, 2, 3, 1, -1], [1, 0, 3, 1, -1], [1, 2, 0, 1, -1]])

# Проверка кода, проверял сразу в сети hw_backprop ноутбука
layer = CrossEntropy()
#x = np.array([[0.5, 0.5, 0., 0., 0.]])
x = np.array([[0.5, 0.5, 0., 0., 0.], [0.5, 0.5, 0.0, 0., 0.]])
#y_true = np.array([[1., 0., 0., 0., 0.]])
y_true = np.array([[1., 0., 0., 0., 0.], [0., 0., 1., 0., 0.]])
# y_true = np.array([0, 1])

output = layer.forward(x, y_true)
print("My forward:\n", output)
# print(output_sum)
# print(sum(output_sum))

print("My backward:\n", layer.backward(x, y_true))

torch_layer = nn.CrossEntropyLoss()
torch_layer.requires_grad_ = True

x_tensor = torch.tensor(x, requires_grad=True)
y_tensor = torch.tensor(y_true, requires_grad=True)

print("Torch forward:\n", torch_layer.forward(x_tensor, y_tensor))

# a = torch.tensor(0.1, requires_grad=True)
# b = torch.tensor(1.0, requires_grad=True)
# c = torch.tensor(0.1, requires_grad=True)
# y = 3*a + 2*b*b + torch.log(c)

# y.backward()

# print(a.grad)  # tensor(3.)
# print(b.grad)  # tensor(4.)
# print(c.grad)  # tensor(10.)

# print("Torch forward:\n", torch_layer.forward(x_tensor, y_tensor))

# print(torch_layer.)
# !! получить backward от torch

```

    My forward:
     1.6004182214058682
    My backward:
     [[-2.  2.  1.  1.  1.]
     [ 2.  2. -1.  1.  1.]]
    Torch forward:
     tensor(1.5901, dtype=torch.float64, grad_fn=<DivBackward1>)


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 10

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Mar. 16 at **2pm.**

## Exercise: The New-Keynesian Model with Optimal Monetary Policy

Consider the New-Keynesian model opptimal monetary policy:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
\pi_t & = \beta E_t\pi_{t+1}  + \kappa y_t+ u_t\\
\kappa\pi & = -\lambda y
\end{align}

where $\bar{y}$ and $\pi^T$ have been set to zero. We can interpret $y$ as the output gap (log difference of output from steady state). The system above has four equations and four endogenous variables: $y_t$, $\pi_t$, $i_t$, and $r_t$ ($u_t$ and $g_t$ are exogenous to the system).

### Exogenous Shocks and Parameters

As usual, the variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}.
\end{align}

Use the following parameterization in the simulations below:

| $$\beta$$ | $$\bar{r}$$    | $$\kappa$$ | $$\rho_g$$ | $$\rho_u$$ | $$\lambda$$ |
|-----------|----------------|------------|------------|------------|-------------|
| 0.995     | $$-\log\beta$$ | 0.25       | 0.5        | 0.5        | 0.1         |

The shock covariance matrix for the model is:

\begin{align}
\text{Covariance matrix} & = \left[\begin{array}{cc}\sigma_g^2 & 0\\ 0 & \sigma_u^2\end{array}\right]
\end{align}

Use the following values for $\sigma_g$ and $\sigma_u$ in the simulations:

| $$\sigma_g$$ | $$\sigma_u$$ |
|--------------|--------------|
| 0.015        | 0.003        |

```python
# Create a variable called 'parameters' that stores the model parameter values (except lambda) in a Pandas Series. PROVIDED
parameters = pd.Series({'beta':0.995,'r_bar':-np.log(0.995),'kappa':0.25,'rho_g':0.5,'rho_u':0.5})

# Add the value for lambda as 'lam' to 'parameters'
parameters['lam'] = 0.1

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first. PROVIDED
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable. PROVIDED
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # IS equation.
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation.
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy. Variable name: 'monetary_policy'
    monetary_policy = p.lam*cur.y  + p.kappa*cur.pi

    # Phillips curve.
    phillips_curve = p.beta*fwd.pi + p.kappa*cur.y + cur.u - cur.pi

    # Demand process.
    demand_process = p.rho_g*cur.g - fwd.g

    # Inflation process.
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array. PROVIDED
    return np.array([is_equation,fisher_equation,monetary_policy,phillips_curve,demand_process,inflation_process])

# Initialize the model into a variable named 'model'. PROVIDED
nk_model = ls.model(equations = equilibrium_equations,n_states=2,var_names=var_names,shock_names=shock_names,parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model. PROVIDED
nk_model.compute_ss([0,0,0,0.01,0.01,0.01])
```

### Simulation with $\lambda = 0$

You will:

1. Compute a 401 period simulation of the model.
2. Construct a plot of simulated output and inflation.
3. Compute standard deviations of simulated output and inflation.

```python
# Set the value for 'lam' in nk_model.parameters to 0
nk_model.parameters['lam'] = 0

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False because the model is already log-linear.
nk_model.approximate_and_solve(log_linear = False)

# Compute simulation with seed equal to 126 and T equal to 401
nk_model.stoch_sim(cov_mat=[[0.015**2,0],[0,0.003**2]],T=401,seed=126)
```

```python
# Create a figure with dimensions 12x8. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output (times 100)
ax.plot(nk_model.simulated['y']*100,label='output',lw=2)

# Plot inflation (times 400)
ax.plot(nk_model.simulated['pi']*400,label='inflation',lw=2)

# Set axis title
ax.set_title('Simulated NK model')

# Set y-axis label to "Percent"
ax.set_ylabel('Percent')

# Construct legends PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

```python
# Print standard deviation of output (times 100)
nk_model.simulated['y'].std()*100
```

    1.4151683664708032

```python
# print standard deviation of inflation (times 400)
nk_model.simulated['pi'].std()*400
```

    0.0

### Simulation with $\lambda = 1$

You will:

1. Compute a 401 period simulation of the model.
2. Construct a plot of simulated output and inflation.
3. Compute standard deviations of simulated output and inflation.

```python
# Set the value for 'lam' in nk_model.parameters to 1
nk_model.parameters['lam'] = 1

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False because the model is already log-linear.
nk_model.approximate_and_solve(log_linear = False)

# Compute simulation with seed equal to 126 and T equal to 401
nk_model.stoch_sim(cov_mat=[[0.015**2,0],[0,0.003**2]],T=401,seed=126)
```

```python
# Create a figure with dimensions 12x8. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output (times 100)
ax.plot(nk_model.simulated['y']*100,label='output',lw=2)

# Plot inflation (times 400)
ax.plot(nk_model.simulated['pi']*400,label='inflation',lw=2)

# Set axis title
ax.set_title('Simulated NK model')

# Set y-axis label to "Percent"
ax.set_ylabel('Percent')

# Construct legends PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

```python
# Print standard deviation of output (times 100)
nk_model.simulated['y'].std()*100
```

    0.1565451732821685

```python
# print standard deviation of inflation (times 400)
nk_model.simulated['pi'].std()*400
```

    2.504722772514695

### Simulation with $\lambda = 10$

You will:

1. Compute a 401 period simulation of the model.
2. Construct a plot of simulated output and inflation.
3. Compute standard deviations of simulated output and inflation.

```python
# Set the value for 'lam' in nk_model.parameters to 10
nk_model.parameters['lam'] = 10

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False because the model is already log-linear.
nk_model.approximate_and_solve(log_linear = False)

# Compute simulation with seed equal to 126 and T equal to 401
nk_model.stoch_sim(cov_mat=[[0.015**2,0],[0,0.003**2]],T=401,seed=126)
```

```python
# Create a figure with dimensions 12x8. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output (times 100)
ax.plot(nk_model.simulated['y']*100,label='output',lw=2)

# Plot inflation (times 400)
ax.plot(nk_model.simulated['pi']*400,label='inflation',lw=2)

# Set axis title
ax.set_title('Simulated NK model')

# Set y-axis label to "Percent"
ax.set_ylabel('Percent')

# Construct legends PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

```python
# Print standard deviation of output (times 100)
nk_model.simulated['y'].std()*100
```

    0.01738536076745458

```python
# print standard deviation of inflation (times 400)
nk_model.simulated['pi'].std()*400
```

    2.7816577227927337

**Questions**

1. In terms of a central bank's preference for stabilizing output and inflation, explain in words what it means for $\lambda$ to increase.
2. Which value for $\lambda$ produced the smallest fluctuations in output?
3. Which value for $\lambda$ produced the smallest fluctuations in inflation?

**Answers**

1. As $\lambda$ is increasing, the central bank is placing more weight on output fluctuations in its loss function. That is, the central bank is caring more about stabilizing output relative to stabilizing inflation.  <!-- answer -->

2. $\lambda = 10$  <!-- answer -->

3. $\lambda = 0$  <!-- answer -->

## Exercise: Optimal Monetary Policy v. Taylor's Rule

In this exercise, you will compute and compare impulse responses to supply and demand shocks for the new Keynesian model under two assumptions about monetary policy:

1. The central bank follows optimal monetary policy.
2. The central bank follows Taylor's simple rule.

### NK Model with Optimal Policy

The New-Keynesian model opptimal monetary policy:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
\pi_t & = \beta E_t\pi_{t+1}  + \kappa y_t+ u_t\\
\kappa\pi & = -\lambda y
\end{align}

The system above has four equations and four endogenous variables: $y_t$, $\pi_t$, $i_t$, and $r_t$ ($u_t$ and $g_t$ are exogenous to the system).

### Exogenous Shocks and Parameters

As usual, the variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}.
\end{align}

Use the following parameterization in the simulations:

| $$\beta$$ | $$\bar{r}$$    | $$\kappa$$ | $$\rho_g$$ | $$\rho_u$$ | $$\lambda$$ |
|-----------|----------------|------------|------------|------------|-------------|
| 0.995     | $$-\log\beta$$ | 0.25       | 0.5        | 0.5        | 0.1         |

```python
# Create a variable called 'parameters' that stores the model parameter values (except lambda) in a Pandas Series. PROVIDED
parameters = pd.Series({'beta':0.995,'r_bar':-np.log(0.995),'kappa':0.25,'rho_g':0.5,'rho_u':0.5})

# Add the value for lambda as 'lam' to 'parameters'
parameters['lam'] = 0.1

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first. PROVIDED
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable. PROVIDED
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # IS equation. PROVIDED
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation. PROVIDED
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy. Variable name: 'monetary_policy'
    monetary_policy = p.lam*cur.y  + p.kappa*cur.pi

    # Phillips curve. PROVIDED
    phillips_curve = p.beta*fwd.pi + p.kappa*cur.y + cur.u - cur.pi

    # Demand process. PROVIDED
    demand_process = p.rho_g*cur.g - fwd.g

    # Inflation process. PROVIDED
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array. PROVIDED
    return np.array([is_equation,fisher_equation,monetary_policy,phillips_curve,demand_process,inflation_process])

# Initialize the model into a variable named 'nk_model_optimal'. PROVIDED
nk_model_optimal = ls.model(equations = equilibrium_equations,n_states=2,var_names=var_names,shock_names=shock_names,parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model_optimal. PROVIDED
nk_model_optimal.compute_ss([0,0,0,0.01,0.01,0.01])

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model_optimal
# set argumement 'log_linear' to False because the model is already log-linear. PROVIDED
nk_model_optimal.approximate_and_solve(log_linear = False)
```

### NK Model with Taylor's Rule

The New-Keynesian model opptimal monetary policy:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
\pi_t & = \beta E_t\pi_{t+1}  + \kappa y_t+ u_t\\
i_{t} & = \bar{r} + \pi^T + \phi_{\pi}\big(\pi_t - \pi^T\big) + \phi_{y}\big(y_t - \bar{y}\big)
\end{align}

The system above has four equations and four endogenous variables: $y_t$, $\pi_t$, $i_t$, and $r_t$ ($u_t$ and $g_t$ are exogenous to the system).

### Exogenous Shocks and Parameters

As usual, the variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}.
\end{align}

Use the following parameterization in the simulations:

| $$\beta$$ | $$\bar{r}$$    | $$\kappa$$ | $$\rho_g$$ | $$\rho_u$$ | $$\pi^T$$ | $$\phi_{\pi}$$ | $\phi_{y}$ | $$\bar{y}$$ |
|-----------|----------------|------------|------------|------------|-----------|----------------|------------|-------------------|
| 0.995     | $$-\log\beta$$ | 0.25       | 0.5        | 0.5        | 0.02/4    | 1.5            | 0.5/4      | 0                 |

```python
# Create a variable called 'parameters' that stores the model parameter values (except lambda) in a Pandas Series. PROVIDED
parameters = pd.Series({'beta':0.995,'r_bar':-np.log(0.995),'kappa':0.25,'rho_g':0.5,'rho_u':0.5})

# Add the value for 'pi_T' to 'parameters'
parameters['pi_T'] = 0.02/4

# Add the value for 'phi_pi' to 'parameters'
parameters['phi_pi'] = 1.5

# Add the value for 'phi_y' to 'parameters'
parameters['phi_y'] = 0.5/4

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first. PROVIDED
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable. PROVIDED
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # IS equation. . PROVIDED
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation. . PROVIDED
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy. Variable name: 'monetary_policy'
    monetary_policy = p.r_bar + p.pi_T + p.phi_pi*(cur.pi - p.pi_T) + p.phi_y*cur.y - cur.i

    # Phillips curve. . PROVIDED
    phillips_curve = p.beta*fwd.pi + p.kappa*cur.y + cur.u - cur.pi

    # Demand process. . PROVIDED
    demand_process = p.rho_g*cur.g - fwd.g

    # Inflation process. . PROVIDED
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array. PROVIDED
    return np.array([is_equation,fisher_equation,monetary_policy,phillips_curve,demand_process,inflation_process])

# Initialize the model into a variable named 'nk_model_taylor'. PROVIDED
nk_model_taylor = ls.model(equations = equilibrium_equations,n_states=2,var_names=var_names,shock_names=shock_names,parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model_taylor. PROVIDED
nk_model_taylor.compute_ss([0,0,0,0.01,0.01,0.01])

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model_taylor
# set argumement 'log_linear' to False because the model is already log-linear. PROVIDED
nk_model_taylor.approximate_and_solve(log_linear = False)
```

### Impulse Responses

Compute a 21 period impulse responses of the each model's variables to a 0.01 unit shock to the exogenous component of  demand ($g_t$) and a 0.01/4 unit shock to the exogenous component of inflation ($u_t$) in period 5.

```python
# Compute the desired impulse responses for the model with optimal monetary policy
nk_model_optimal.impulse(shocks=[0.01,0.01/4],T=21,t0=5)

# Compute the desired impulse responses for the model with Taylor's rule
nk_model_taylor.impulse(shocks=[0.01,0.01/4],T=21,t0=5)
```

#### Demand Shock

Plot the impulse responses of output and inflation to the demand shock under each monetary policy specification. Multiply output by 100 and multiply inflation by 400 (to annualize the values).

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create left subplot. PROVIDED
ax1 = fig.add_subplot(1,2,1)
# Create right subplot. PROVIDED
ax2 = fig.add_subplot(1,2,2)

# Left subplot title. PROVIDED
ax1.set_title('Output $y_t$')
# Right subplot title. PROVIDED
ax2.set_title('Inflation $\pi_t$')

# Left subplot y-axis label. PROVIDED
ax1.set_ylabel('Percent')
# Right subplot y-axis label. PROVIDED
ax2.set_ylabel('Percent')

# Plot output (multiplied by 100) under both monetary policies on right axis.
ax1.plot(nk_model_optimal.irs['e_g']['y']*100,lw=4)
ax1.plot(nk_model_taylor.irs['e_g']['y']*100,lw=4)

# Plot inflation (multiplied by 400) under both monetary policies on right axis. Add labels for the lines.
ax2.plot(nk_model_optimal.irs['e_g']['pi']*400,lw=4,label='Optimal')
ax2.plot(nk_model_taylor.irs['e_g']['pi']*400,lw=4,label='Taylor')

# Construct legends PROVIDED
ax2.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

#### Inflation Shock

Plot the impulse responses of output and inflation to the inflation shock under each monetary policy specification. Multiply output by 100 and multiply inflation by 400 (to annualize the values).

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create left subplot. PROVIDED
ax1 = fig.add_subplot(1,2,1)
# Create right subplot. PROVIDED
ax2 = fig.add_subplot(1,2,2)

# Left subplot title. PROVIDED
ax1.set_title('Output $y_t$')
# Right subplot title. PROVIDED
ax2.set_title('Inflation $\pi_t$')

# Left subplot y-axis label. PROVIDED
ax1.set_ylabel('Percent')
# Right subplot y-axis label. PROVIDED
ax2.set_ylabel('Percent')

# Plot output (multiplied by 100) under both monetary policies on left axis.
ax1.plot(nk_model_optimal.irs['e_u']['y']*100,lw=4)
ax1.plot(nk_model_taylor.irs['e_u']['y']*100,lw=4)

# Plot inflation (multiplied by 400) under both monetary policies on right axis. Add labels for the lines.
ax2.plot(nk_model_optimal.irs['e_u']['pi']*400,lw=4,label='Optimal')
ax2.plot(nk_model_taylor.irs['e_u']['pi']*400,lw=4,label='Taylor')

# Construct legends PROVIDED
ax2.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

**Questions**

1. Does Taylor's rule appear to do a better job matching the optimal response to demand or inflation shocks? Explain

**Answers**

1.  Taylor's rule does a better job matching the optimal response to inflation shocks. The impulse responses of output and inflation to the inflation shock are very close under the two policies. But the optimal response to demand shocks entails perfect stabilization of output and inflation, an outcome not remotely achieved under Taylor's rule.  <!-- answer -->


```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.optimize import fmin

np.random.seed(1789)

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

---

# Regression modeling

A general, primary goal of many statistical data analysis tasks is to relate the influence of one variable on another.

For example:

- how different medical interventions influence the incidence or duration of disease
- how baseball player's performance varies as a function of age.
- [how test scores are correlated with tissue LSD concentration](http://www.lycaeum.org/research/researchpdfs/1748.pdf)

```python
from io import StringIO

data_string = """
Drugs	Score
0	1.17	78.93
1	2.97	58.20
2	3.26	67.47
3	4.69	37.47
4	5.83	45.65
5	6.00	32.92
6	6.41	29.97
"""

lsd_and_math = pd.read_table(StringIO(data_string), sep='\t', index_col=0)
lsd_and_math
```

      Drugs
      Score

      0
      1.17
      78.93

      1
      2.97
      58.20

      2
      3.26
      67.47

      3
      4.69
      37.47

      4
      5.83
      45.65

      5
      6.00
      32.92

      6
      6.41
      29.97

> Taking LSD was a profound experience, one of the most important things in my life --Steve Jobs

```python
lsd_and_math.plot(x='Drugs', y='Score', style='ro', legend=False, xlim=(0,8))
```

We can build a model to characterize the relationship between $X$ and $Y$, recognizing that additional factors other than $X$ (the ones we have measured or are interested in) may influence the response variable $Y$.

- $M(Y|X) = E(Y|X)$
- $M(Y|X) = Pr(Y=1|X)$

In general,

$$M(Y|X) = f(X)$$

for linear regression

$$M(Y|X) = f(X\beta)$$

where $f$ is some function, for example a linear function:

$y_i = \beta_0 + \beta_1 x_{1i} + \ldots + \beta_k x_{ki} + \epsilon_i$

Regression is a **weighted sum** of independent predictors

and $\epsilon_i$ accounts for the difference between the observed response $y_i$ and its prediction from the model $\hat{y_i} = \beta_0 + \beta_1 x_i$. This is sometimes referred to as **process uncertainty**.

Interpretation: coefficients represent the change in Y for a unit increment of the predictor X.

Two important regression **assumptions**:

1. normal errors
2. homoscedasticity

## Parameter estimation

We would like to select $\beta_0, \beta_1$ so that the difference between the predictions and the observations is zero, but this is not usually possible. Instead, we choose a reasonable criterion: ***the smallest sum of the squared differences between $\hat{y}$ and $y$***.

$$R^2 = \sum_i (y_i - [\beta_0 + \beta_1 x_i])^2 = \sum_i \epsilon_i^2 $$

Squaring serves two purposes:

1. to prevent positive and negative values from cancelling each other out
2. to strongly penalize large deviations.

Whether or not the latter is a desired depends on the goals of the analysis.

In other words, we will select the parameters that minimize the squared error of the model.

```python
sum_of_squares = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x) ** 2)
```

```python
sum_of_squares([0,1], lsd_and_math.Drugs, lsd_and_math.Score)
```

    17159.8154

```python
x, y = lsd_and_math.T.values
b0, b1 = fmin(sum_of_squares, [0,1], args=(x,y))
b0, b1
```

    Optimization terminated successfully.
             Current function value: 253.881329
             Iterations: 97
             Function evaluations: 179

    (89.123909209804239, -9.0094696583309499)

```python
ax = lsd_and_math.plot(x='Drugs', y='Score', style='ro', legend=False, xlim=(0,8))
ax.plot([0,10], [b0, b0+b1*10])
```

```python
ax = lsd_and_math.plot(x='Drugs', y='Score', style='ro', legend=False, xlim=(0,8), ylim=(20, 90))
ax.plot([0,10], [b0, b0+b1*10])
for xi, yi in zip(x,y):
    ax.plot([xi]*2, [yi, b0+b1*xi], 'k:')
```

## Alternative loss functions

Minimizing the sum of squares is not the only criterion we can use; it is just a very popular (and successful) one. For example, we can try to minimize the sum of absolute differences:

```python
sum_of_absval = lambda theta, x, y: np.sum(np.abs(y - theta[0] - theta[1]*x))
```

```python
b0, b1 = fmin(sum_of_absval, [0,0], args=(x,y))
print('\nintercept: {0:.2}, slope: {1:.2}'.format(b0,b1))
ax = lsd_and_math.plot(x='Drugs', y='Score', style='ro', legend=False, xlim=(0,8))
ax.plot([0,10], [b0, b0+b1*10])
```

We are not restricted to a straight-line regression model; we can represent a curved relationship between our variables by introducing **polynomial** terms. For example, a cubic model:

$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \epsilon_i$

```python
sum_squares_quad = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x - theta[2]*(x**2)) ** 2)
```

```python
b0,b1,b2 = fmin(sum_squares_quad, [1,1,-1], args=(x,y))
print('\nintercept: {0:.2}, x: {1:.2}, x2: {2:.2}'.format(b0,b1,b2))
ax = lsd_and_math.plot(x='Drugs', y='Score', style='ro', legend=False, xlim=(0,8))
xvals = np.linspace(0, 8, 100)
ax.plot(xvals, b0 + b1*xvals + b2*(xvals**2))
```

Although a polynomial model characterizes a nonlinear relationship, it is a linear problem in terms of estimation. That is, the regression model $f(y | x)$ is **linear in the parameters**.

For some data, it may be reasonable to consider polynomials of order>2. For example, consider the relationship between the number of spawning salmon and the number of juveniles recruited into the population the following year; one would expect the relationship to be positive, but not necessarily linear.

```python
sum_squares_cubic = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x - theta[2]*(x**2)
                                  - theta[3]*(x**3)) ** 2)
```

```python
salmon = pd.read_table("../data/salmon.dat", delim_whitespace=True, index_col=0)
plt.plot(salmon.spawners, salmon.recruits, 'r.')
b0,b1,b2,b3 = fmin(sum_squares_cubic, [0,1,-1,0], args=(salmon.spawners, salmon.recruits))
xvals = np.arange(500)
plt.plot(xvals, b0 + b1*xvals + b2*(xvals**2) + b3*(xvals**3))
```

## Linear Regression with `scikit-learn`

In practice, we need not fit least squares models by hand because they are implemented generally in packages such as [`scikit-learn`](http://scikit-learn.org/) and [`statsmodels`](https://github.com/statsmodels/statsmodels/). For example, `scikit-learn` package implements least squares models in its `LinearRegression` class:

```python
from sklearn import linear_model

straight_line = linear_model.LinearRegression()
straight_line.fit(x[:, np.newaxis], y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

```python
straight_line.coef_
```

    array([-9.00946642])

```python
straight_line.intercept_
```

    89.123873767993061

```python
plt.plot(x, y, 'ro')
plt.plot(x, straight_line.predict(x[:, np.newaxis]), color='blue',
         linewidth=3)
```

For more general regression model building, its helpful to use a tool for describing statistical models, called `patsy`. With `patsy`, it is easy to specify the desired combinations of variables for any particular analysis, using an "R-like" syntax. `patsy` parses the formula string, and uses it to construct the approriate *design matrix* for the model.

For example, the quadratic model specified by hand above can be coded as:

```python
from patsy import dmatrix

X = dmatrix('salmon.spawners + I(salmon.spawners**2)')
```

The `dmatrix` function returns the design matrix, which can be passed directly to the `LinearRegression` fitting method.

```python
poly_line = linear_model.LinearRegression(fit_intercept=False)
poly_line.fit(X, salmon.recruits)
```

    LinearRegression(copy_X=True, fit_intercept=False, n_jobs=1, normalize=False)

```python
poly_line.coef_
```

    array([  2.13819247e+01,   9.59677283e-01,  -8.39694795e-04])

```python
plt.plot(salmon.spawners, salmon.recruits, 'ro')
frye_range = np.arange(500)
plt.plot(frye_range, poly_line.predict(dmatrix('frye_range + I(frye_range**2)')), color='blue')
```

## Generalized linear models

Often our data violates one or more of the linear regression assumptions:

- non-linear
- non-normal error distribution
- heteroskedasticity

this forces us to **generalize** the regression model in order to account for these characteristics.

As a motivating example, consider the Olympic medals data that we compiled earlier in the tutorial.

```python
medals = pd.read_csv('../data/medals.csv')
medals.head()
```

      medals
      population
      oecd
      log_population

      0
      1
      96165
      0
      11.473821

      1
      1
      281584
      0
      12.548186

      2
      6
      2589043
      0
      14.766799

      3
      25
      10952046
      0
      16.209037

      4
      41
      18348078
      1
      16.725035

We expect a positive relationship between population and awarded medals, but the data in their raw form are clearly not amenable to linear regression.

```python
medals.plot(x='population', y='medals', kind='scatter')
```

Part of the issue is the scale of the variables. For example, countries' populations span several orders of magnitude. We can correct this by using the logarithm of population, which we have already calculated.

```python
medals.plot(x='log_population', y='medals', kind='scatter')
```

This is an improvement, but the relationship is still not adequately modeled by least-squares regression.

```python
linear_medals = linear_model.LinearRegression()
X = medals.log_population[:, np.newaxis]
linear_medals.fit(X, medals.medals)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

```python
ax = medals.plot(x='log_population', y='medals', kind='scatter')
ax.plot(medals.log_population, linear_medals.predict(X), color='red',
         linewidth=2)
```

This is due to the fact that the response data are **counts**. As a result, they tend to have characteristic properties.

- discrete
- positive
- variance grows with mean

to account for this, we can do two things: (1) model the medal count on the **log scale** and (2) assume **Poisson errors**, rather than normal.

Recall the Poisson distribution from the previous section:

$$p(y)=\frac{e^{-\lambda}\lambda^y}{y!}$$

* $Y=\{0,1,2,\ldots\}$
* $\lambda > 0$

$$E(Y) = \text{Var}(Y) = \lambda$$

So, we will model the logarithm of the expected value as a linear function of our predictors:

$$\log(\lambda) = X\beta$$

In this context, the log function is called a **link function**. This transformation implies the mean of the Poisson is:

$$\lambda = \exp(X\beta)$$

We can plug this into the Poisson likelihood and use maximum likelihood to estimate the regression covariates $\beta$.

$$\log L = \sum_{i=1}^n -\exp(X_i\beta) + Y_i (X_i \beta)- \log(Y_i!)$$

As we have already done, we just need to code the kernel of this likelihood, and optimize!

```python
# Poisson negative log-likelhood
poisson_loglike = lambda beta, X, y: -(-np.exp(X.dot(beta)) + y*X.dot(beta)).sum()
```

Let's use the `assign` method to add a column of ones to the design matrix.

```python
poisson_loglike([0,1], medals[['log_population']].assign(intercept=1), medals.medals)
```

    -627.255735551737

We will use Nelder-Mead to minimize the negtive log-likelhood.

```python
b1, b0 = fmin(poisson_loglike, [0,1], args=(medals[['log_population']].assign(intercept=1).values,
                                            medals.medals.values))
```

    Optimization terminated successfully.
             Current function value: -1381.299433
             Iterations: 68
             Function evaluations: 131

```python
b0, b1
```

    (-5.2973029170604393, 0.44873025169011005)

The resulting fit looks reasonable.

```python
ax = medals.plot(x='log_population', y='medals', kind='scatter')
xvals = np.arange(12, 22)
ax.plot(xvals, np.exp(b0 + b1*xvals), 'r--')
```

### Exercise: Multivariate GLM

Add the OECD indicator variable to the model, and estimate the model coefficients.

```python
# Write your answer here
```

### Interactions among variables

Interactions imply that the effect of one covariate $X_1$ on $Y$ depends on the value of another covariate $X_2$.

$$M(Y|X) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 +\beta_3 X_1 X_2$$

the effect of a unit increase in $X_1$:

$$M(Y|X_1+1, X_2) - M(Y|X_1, X_2)$$

$$\begin{align}
&= \beta_0 + \beta_1 (X_1 + 1) + \beta_2 X_2 +\beta_3 (X_1 + 1) X_2
- [\beta_0 + \beta_1 X_1 + \beta_2 X_2 +\beta_3 X_1 X_2] \\
&= \beta_1 + \beta_3 X_2
\end{align}$$

```python
ax = medals[medals.oecd==1].plot(x='log_population', y='medals', kind='scatter', alpha=0.8)
medals[medals.oecd==0].plot(x='log_population', y='medals', kind='scatter', color='red', alpha=0.5, ax=ax)
```

Interaction can be interpreted as:

- $X_1$ interacts with $X_2$
- $X_1$ modifies the effect of $X_2$
- $X_2$ modifies the effect of $X_1$
- $X_1$ and $X_2$ are non-additive or synergistic

Let's construct a model that predicts medal count based on population size and OECD status, as well as the interaction. We can use patsy to set up the design matrix.

```python
y = medals.medals
X = dmatrix('log_population * oecd', data=medals)
X
```

    DesignMatrix with shape (79, 4)
      Intercept  log_population  oecd  log_population:oecd
              1        11.47382     0              0.00000
              1        12.54819     0              0.00000
              1        14.76680     0              0.00000
              1        16.20904     0              0.00000
              1        16.72504     1             16.72504
              1        16.14509     1             16.14509
              1        15.91733     0              0.00000
              1        13.99525     0              0.00000
              1        15.10232     1             15.10232
              1        15.29285     1             15.29285
              1        16.15816     0              0.00000
              1        16.55847     1             16.55847
              1        14.31108     0              0.00000
              1        15.47603     1             15.47603
              1        15.10551     1             15.10551
              1        16.14891     1             16.14891
              1        14.46491     0              0.00000
              1        15.78927     1             15.78927
              1        15.99697     1             15.99697
              1        16.93468     0              0.00000
              1        18.22083     1             18.22083
              1        15.44896     1             15.44896
              1        16.16795     1             16.16795
              1        17.21615     1             17.21615
              1        16.01637     0              0.00000
              1        16.64175     0              0.00000
              1        17.88266     1             17.88266
              1        17.86445     1             17.86445
              1        17.63646     0              0.00000
              1        16.13364     1             16.13364
      [49 rows omitted]
      Terms:
        'Intercept' (column 0)
        'log_population' (column 1)
        'oecd' (column 2)
        'log_population:oecd' (column 3)
      (to view full data, use np.asarray(this_obj))

Now, fit the model.

```python
interaction_params = fmin(poisson_loglike, [0,1,1,0], args=(X, y))
```

    Warning: Maximum number of function evaluations has been exceeded.

```python
interaction_params
```

    array([-5.2958393 ,  0.42441607, -2.16727213,  0.17978487])

Notice anything odd about these estimates?

The main effect of the OECD effect is negative, which seems counter-intuitive. This is because the variable is interpreted as the OECD effect when the **log-population is zero**. This is not particularly meaningful.

We can improve the interpretability of this parameter by **centering** the log-population variable prior to entering it into the model. This will result in the OECD main effect being interpreted as the marginal effect of being an OECD country for an **average-sized** country.

```python
y = medals.medals
X = dmatrix('center(log_population) * oecd', data=medals)
X
```

    DesignMatrix with shape (79, 4)
      Intercept  center(log_population)  oecd  center(log_population):oecd
              1                -5.03303     0                     -0.00000
              1                -3.95866     0                     -0.00000
              1                -1.74005     0                     -0.00000
              1                -0.29781     0                     -0.00000
              1                 0.21819     1                      0.21819
              1                -0.36176     1                     -0.36176
              1                -0.58952     0                     -0.00000
              1                -2.51159     0                     -0.00000
              1                -1.40453     1                     -1.40453
              1                -1.21400     1                     -1.21400
              1                -0.34869     0                     -0.00000
              1                 0.05163     1                      0.05163
              1                -2.19577     0                     -0.00000
              1                -1.03081     1                     -1.03081
              1                -1.40133     1                     -1.40133
              1                -0.35793     1                     -0.35793
              1                -2.04194     0                     -0.00000
              1                -0.71758     1                     -0.71758
              1                -0.50988     1                     -0.50988
              1                 0.42783     0                      0.00000
              1                 1.71398     1                      1.71398
              1                -1.05789     1                     -1.05789
              1                -0.33889     1                     -0.33889
              1                 0.70930     1                      0.70930
              1                -0.49047     0                     -0.00000
              1                 0.13490     0                      0.00000
              1                 1.37582     1                      1.37582
              1                 1.35760     1                      1.35760
              1                 1.12961     0                      0.00000
              1                -0.37321     1                     -0.37321
      [49 rows omitted]
      Terms:
        'Intercept' (column 0)
        'center(log_population)' (column 1)
        'oecd' (column 2)
        'center(log_population):oecd' (column 3)
      (to view full data, use np.asarray(this_obj))

```python
fmin(poisson_loglike, [0,1,1,0], args=(X, y))
```

    Optimization terminated successfully.
             Current function value: -1492.092233
             Iterations: 303
             Function evaluations: 513

    array([ 1.71003145,  0.42443035,  0.80033982,  0.17976563])

## Model Selection

How do we choose among competing models for a given dataset? More parameters are not necessarily better, from the standpoint of model fit. For example, fitting a 6th order polynomial to the LSD example certainly results in an overfit.

```python
def calc_poly(params, data):
        x = np.c_[[data**i for i in range(len(params))]]
        return np.dot(params, x)

x, y = lsd_and_math.T.values

sum_squares_poly = lambda theta, x, y: np.sum((y - calc_poly(theta, x)) ** 2)
betas = fmin(sum_squares_poly, np.zeros(7), args=(x,y), maxiter=1e6)
plt.plot(x, y, 'ro')
xvals = np.linspace(0, max(x), 100)
plt.plot(xvals, calc_poly(betas, xvals))
```

One approach is to use an information-theoretic criterion to select the most appropriate model. For example **Akaike's Information Criterion (AIC)** balances the fit of the model (in terms of the likelihood) with the number of parameters required to achieve that fit. We can easily calculate AIC as:

$$AIC = n \log(\hat{\sigma}^2) + 2p$$

where $p$ is the number of parameters in the model and $\hat{\sigma}^2 = RSS/(n-p-1)$.

Notice that as the number of parameters increase, the residual sum of squares goes down, but the second term (a penalty) increases.

AIC is a metric of **information distance** between a given model and a notional "true" model. Since we don't know the true model, the AIC value itself is not meaningful in an absolute sense, but is useful as a relative measure of model quality.

To apply AIC to model selection, we choose the model that has the **lowest** AIC value.

```python
n = len(x)

aic = lambda rss, p, n: n * np.log(rss/(n-p-1)) + 2*p

RSS1 = sum_of_squares(fmin(sum_of_squares, [0,1], args=(x,y)), x, y)
RSS2 = sum_squares_quad(fmin(sum_squares_quad, [1,1,-1], args=(x,y)), x, y)

print('\nModel 1: {0}\nModel 2: {1}'.format(aic(RSS1, 2, n), aic(RSS2, 3, n)))
```

    Optimization terminated successfully.
             Current function value: 253.881329
             Iterations: 97
             Function evaluations: 179
    Optimization terminated successfully.
             Current function value: 251.093792
             Iterations: 177
             Function evaluations: 319

    Model 1: 33.05400811127588
    Model 2: 36.99049978705

Hence, on the basis of "information distance", we would select the 2-parameter (linear) model.

### Exercise: Olympic medals model selection

Use AIC to select the best model from the following set of Olympic medal prediction models:

- population only
- population and OECD
- interaction model

For these models, use the alternative form of AIC, which uses the log-likelhood rather than the residual sums-of-squares:

$$AIC = -2 \log(L) + 2p$$

```python
# Write your answer here
```

## Logistic Regression

Fitting a line to the relationship between two variables using the least squares approach is sensible when the variable we are trying to predict is continuous, but what about when the data are dichotomous?

- male/female
- pass/fail
- died/survived

Let's consider the problem of predicting survival in the Titanic disaster, based on our available information. For example, lets say that we want to predict survival as a function of the fare paid for the journey.

```python
titanic = pd.read_excel("../data/titanic.xls", "titanic")
titanic.name
```

    0                           Allen, Miss. Elisabeth Walton
    1                          Allison, Master. Hudson Trevor
    2                            Allison, Miss. Helen Loraine
    3                    Allison, Mr. Hudson Joshua Creighton
    4         Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
    5                                     Anderson, Mr. Harry
    6                       Andrews, Miss. Kornelia Theodosia
    7                                  Andrews, Mr. Thomas Jr
    8           Appleton, Mrs. Edward Dale (Charlotte Lamson)
    9                                 Artagaveytia, Mr. Ramon
    10                                 Astor, Col. John Jacob
    11      Astor, Mrs. John Jacob (Madeleine Talmadge Force)
    12                          Aubart, Mme. Leontine Pauline
    13                           Barber, Miss. Ellen "Nellie"
    14                   Barkworth, Mr. Algernon Henry Wilson
    15                                    Baumann, Mr. John D
    16                               Baxter, Mr. Quigg Edmond
    17        Baxter, Mrs. James (Helene DeLaudeniere Chaput)
    18                                  Bazzani, Miss. Albina
    19                                   Beattie, Mr. Thomson
    20                          Beckwith, Mr. Richard Leonard
    21       Beckwith, Mrs. Richard Leonard (Sallie Monypeny)
    22                                  Behr, Mr. Karl Howell
    23                                  Bidois, Miss. Rosalie
    24                                      Bird, Miss. Ellen
    25                                    Birnbaum, Mr. Jakob
    26                                Bishop, Mr. Dickinson H
    27                Bishop, Mrs. Dickinson H (Helen Walton)
    28                                 Bissette, Miss. Amelia
    29              Bjornstrom-Steffansson, Mr. Mauritz Hakan
                                  ...
    1279                 Vestrom, Miss. Hulda Amanda Adolfina
    1280                                      Vovk, Mr. Janko
    1281                                 Waelens, Mr. Achille
    1282                                  Ware, Mr. Frederick
    1283                          Warren, Mr. Charles William
    1284                                    Webber, Mr. James
    1285                                  Wenzel, Mr. Linhart
    1286      Whabee, Mrs. George Joseph (Shawneene Abi-Saab)
    1287                     Widegren, Mr. Carl/Charles Peter
    1288                            Wiklund, Mr. Jakob Alfred
    1289                              Wiklund, Mr. Karl Johan
    1290                     Wilkes, Mrs. James (Ellen Needs)
    1291                     Willer, Mr. Aaron ("Abi Weller")
    1292                                   Willey, Mr. Edward
    1293                    Williams, Mr. Howard Hugh "Harry"
    1294                                 Williams, Mr. Leslie
    1295                                  Windelov, Mr. Einar
    1296                                     Wirz, Mr. Albert
    1297                               Wiseman, Mr. Phillippe
    1298                            Wittevrongel, Mr. Camille
    1299                                  Yasbeck, Mr. Antoni
    1300              Yasbeck, Mrs. Antoni (Selini Alexander)
    1301                                 Youseff, Mr. Gerious
    1302                                    Yousif, Mr. Wazli
    1303                                Yousseff, Mr. Gerious
    1304                                 Zabour, Miss. Hileni
    1305                                Zabour, Miss. Thamine
    1306                            Zakarian, Mr. Mapriededer
    1307                                  Zakarian, Mr. Ortin
    1308                                   Zimmerman, Mr. Leo
    Name: name, dtype: object

```python
jitter = np.random.normal(scale=0.02, size=len(titanic))

plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
plt.yticks([0,1])
plt.ylabel("survived")
plt.xlabel("log(fare)")
```

I have added random jitter on the y-axis to help visualize the density of the points, and have plotted fare on the log scale.

Clearly, fitting a line through this data makes little sense, for several reasons. First, for most values of the predictor variable, the line would predict values that are not zero or one. Second, it would seem odd to choose least squares (or similar) as a criterion for selecting the best line.

```python
x = np.log(titanic.fare[titanic.fare>0])
y = titanic.survived[titanic.fare>0]
betas_titanic = fmin(sum_of_squares, [1,1], args=(x,y))
```

    Optimization terminated successfully.
             Current function value: 277.621917
             Iterations: 55
             Function evaluations: 103

```python
jitter = np.random.normal(scale=0.02, size=len(titanic))
plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
plt.yticks([0,1])
plt.ylabel("survived")
plt.xlabel("log(fare)")
plt.plot([0,7], [betas_titanic[0], betas_titanic[0] + betas_titanic[1]*7.])
```

If we look at this data, we can see that for most values of `fare`, there are some individuals that survived and some that did not. However, notice that the cloud of points is denser on the "survived" (`y=1`) side for larger values of fare than on the "died" (`y=0`) side.

### Stochastic model

Rather than model the binary outcome explicitly, it makes sense instead to model the *probability* of death or survival in a **stochastic** model. Probabilities are measured on a continuous [0,1] scale, which may be more amenable for prediction using a regression line. We need to consider a different probability model for this exerciese however; let's consider the **Bernoulli** distribution as a generative model for our data:

$$f(y|p) = p^{y} (1-p)^{1-y}$$

where $y = \{0,1\}$ and $p \in [0,1]$. So, this model predicts whether $y$ is zero or one as a function of the probability $p$. Notice that when $y=1$, the $1-p$ term disappears, and when $y=0$, the $p$ term disappears.

So, the model we want to fit should look something like this:

$$p_i = \beta_0 + \beta_1 x_i + \epsilon_i$$

However, since $p$ is constrained to be between zero and one, it is easy to see where a linear (or polynomial) model might predict values outside of this range. As with the Poisson regression, we can modify this model slightly by using a link function to transform the probability to have an unbounded range on a new scale. Specifically, we can use a **logit transformation** as our link function:

$$\text{logit}(p) = \log\left[\frac{p}{1-p}\right] = x$$

Here's a plot of $p/(1-p)$

```python
logit = lambda p: np.log(p/(1.-p))
unit_interval = np.linspace(0,1)
plt.plot(unit_interval/(1-unit_interval), unit_interval)
plt.xlabel(r'$p/(1-p)$')
plt.ylabel('p');
```

And here's the logit function:

```python
plt.plot(logit(unit_interval), unit_interval)
plt.xlabel('logit(p)')
plt.ylabel('p');
```

The inverse of the logit transformation is:

$$p = \frac{1}{1 + \exp(-x)}$$

```python
invlogit = lambda x: 1. / (1 + np.exp(-x))
```

So, now our model is:

$$\text{logit}(p_i) = \beta_0 + \beta_1 x_i + \epsilon_i$$

We can fit this model using maximum likelihood. Our likelihood, again based on the Bernoulli model is:

$$L(y|p) = \prod_{i=1}^n p_i^{y_i} (1-p_i)^{1-y_i}$$

which, on the log scale is:

$$l(y|p) = \sum_{i=1}^n y_i \log(p_i) + (1-y_i)\log(1-p_i)$$

We can easily implement this in Python, keeping in mind that `fmin` minimizes, rather than maximizes functions:

```python
def logistic_like(theta, x, y):

    p = invlogit(theta[0] + theta[1] * x)

    # Return negative of log-likelihood
    return -np.sum(y * np.log(p) + (1-y) * np.log(1 - p))
```

Remove null values from variables (a *bad idea*, which we will show later) ...

```python
x, y = titanic[titanic.fare.notnull()][['fare', 'survived']].values.T
```

... and fit the model.

```python
b0, b1 = fmin(logistic_like, [0.5,0], args=(x,y))
b0, b1
```

    Optimization terminated successfully.
             Current function value: 827.015955
             Iterations: 47
             Function evaluations: 93

    (-0.88238984528338194, 0.012452067664164127)

```python
jitter = np.random.normal(scale=0.01, size=len(x))
plt.plot(x, y+jitter, 'r.', alpha=0.3)
plt.yticks([0,.25,.5,.75,1])
xvals = np.linspace(0, 600)
plt.plot(xvals, invlogit(b0+b1*xvals))
```

As with our least squares model, we can easily fit logistic regression models in `scikit-learn`, in this case using the `LogisticRegression`.

```python
logistic = linear_model.LogisticRegression()
logistic.fit(x[:, np.newaxis], y)
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr',
              penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0)

```python
logistic.coef_
```

    array([[ 0.0123847]])

### Exercise: multivariate logistic regression

Which other variables might be relevant for predicting the probability of surviving the Titanic? Generalize the model likelihood to include 2 or 3 other covariates from the dataset.

```python
# Write your answer here
```

notebooks/3. Fitting Regression Models.ipynb


# Enforce conformal 3-metric $\det{\bar{\gamma}_{ij}}=\det{\hat{\gamma}_{ij}}$ constraint (Eq. 53 of [Ruchlin, Etienne, and Baumgarte (2018)](https://arxiv.org/abs/1712.07658))

## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

[comment]: <> (Abstract: TODO)

**Module Status:**  Validated

**Validation Notes:** This tutorial module has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). In addition, its output has been

### NRPy+ Source Code for this module: [BSSN/Enforce_Detgammabar_Constraint.py](../edit/BSSN/Enforce_Detgammabar_Constraint.py)

## Introduction:
[Brown](https://arxiv.org/abs/0902.3652)'s covariant Lagrangian formulation of BSSN, which we adopt, requires that $\partial_t \bar{\gamma} = 0$, where $\bar{\gamma}=\det \bar{\gamma}_{ij}$. Further, all initial data we choose satisfies $\bar{\gamma}=\hat{\gamma}$.

However, numerical errors will cause $\bar{\gamma}$ to deviate from a constant in time. This actually disrupts the hyperbolicity of the PDEs, so to cure this, we adjust $\bar{\gamma}_{ij}$ at the end of each Runge-Kutta timestep, so that its determinant satisfies $\bar{\gamma}=\hat{\gamma}$ at all times. We adopt the following, rather standard prescription (Eq. 53 of [Ruchlin, Etienne, and Baumgarte (2018)](https://arxiv.org/abs/1712.07658)):

$$
\bar{\gamma}_{ij} \to \left(\frac{\hat{\gamma}}{\bar{\gamma}}\right)^{1/3} \bar{\gamma}_{ij}.
$$

# Table of Contents
$$\label{toc}$$

This module is organized as follows:

1. [Step 1](#initializenrpy): Initialize needed NRPy+ modules
1. [Step 2](#enforcegammaconstraint): Enforce the $\det{\bar{\gamma}_{ij}}=\det{\hat{\gamma}_{ij}}$ constraint
1. [Step 3](#code_validation): Code Validation against `BSSN.Enforce_Detgammabar_Constraint` NRPy+ module
1. [Step 4](#latex_pdf_output): Output this module to $\LaTeX$-formatted PDF

# Step 1: Initialize needed NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

```python
# Step P1: import all needed modules from NRPy+:
from outputC import *
import NRPy_param_funcs as par
import grid as gri
import loop as lp
import indexedexp as ixp
import finite_difference as fin
import reference_metric as rfm
import BSSN.BSSN_quantities as Bq

# Set spatial dimension (must be 3 for BSSN)
DIM = 3
par.set_parval_from_str("grid::DIM",DIM)

# Then we set the coordinate system for the numerical grid
par.set_parval_from_str("reference_metric::CoordSystem","SinhSpherical")
rfm.reference_metric() # Create ReU, ReDD needed for rescaling B-L initial data, generating BSSN RHSs, etc.
```

# Step 2: Enforce the $\det{\bar{\gamma}_{ij}}=\det{\hat{\gamma}_{ij}}$ constraint \[Back to [top](#toc)\]
$$\label{enforcegammaconstraint}$$

Recall that we wish to make the replacement:
$$
\bar{\gamma}_{ij} \to \left(\frac{\hat{\gamma}}{\bar{\gamma}}\right)^{1/3} \bar{\gamma}_{ij}.
$$
Notice the expression on the right is guaranteed to have determinant equal to $\hat{\gamma}$.

$\bar{\gamma}_{ij}$ is not a gridfunction, so we must rewrite the above in terms of $h_{ij}$:
\begin{align}
\left(\frac{\hat{\gamma}}{\bar{\gamma}}\right)^{1/3} \bar{\gamma}_{ij} &= \bar{\gamma}'_{ij}  \\
&= \hat{\gamma}_{ij} + \varepsilon'_{ij} \\
&= \hat{\gamma}_{ij} + \text{Re[i][j]} h'_{ij} \\
\implies h'_{ij} &= \left[\left(\frac{\hat{\gamma}}{\bar{\gamma}}\right)^{1/3} \bar{\gamma}_{ij} - \hat{\gamma}_{ij}\right] / \text{Re[i][j]} \\
&= \left(\frac{\hat{\gamma}}{\bar{\gamma}}\right)^{1/3} \frac{\bar{\gamma}_{ij}}{\text{Re[i][j]}} - \delta_{ij}\\
&= \left(\frac{\hat{\gamma}}{\bar{\gamma}}\right)^{1/3} \frac{\hat{\gamma}_{ij} + \text{Re[i][j]} h_{ij}}{\text{Re[i][j]}} - \delta_{ij}\\
&= \left(\frac{\hat{\gamma}}{\bar{\gamma}}\right)^{1/3} \left(\delta_{ij} + h_{ij}\right) - \delta_{ij}
\end{align}

Upon inspection, when expressing $\hat{\gamma}$ SymPy generates expressions like `(xx0)^{4/3} = pow(xx0, 4./3.)`, which can yield $\text{NaN}$s when `xx0 < 0` (i.e., in the `xx0` ghost zones). To prevent this, we know that $\hat{\gamma}\ge 0$ for all reasonable coordinate systems, so we make the replacement $\hat{\gamma}\to |\hat{\gamma}|$ below:

```python
# We will need the h_{ij} quantities defined within BSSN_RHSs
#    below when we enforce the gammahat=gammabar constraint
# Step 1: All barred quantities are defined in terms of BSSN rescaled gridfunctions,
#         which we declare here in case they haven't yet been declared elsewhere.

Bq.declare_BSSN_gridfunctions_if_not_declared_already()
hDD = Bq.hDD
Bq.BSSN_basic_tensors()
gammabarDD = Bq.gammabarDD

# First define the Kronecker delta:
KroneckerDeltaDD = ixp.zerorank2()
for i in range(DIM):
    KroneckerDeltaDD[i][i] = sp.sympify(1)

# The detgammabar in BSSN_RHSs is set to detgammahat when BSSN_RHSs::detgbarOverdetghat_equals_one=True (default),
#    so we manually compute it here:
dummygammabarUU, detgammabar = ixp.symm_matrix_inverter3x3(gammabarDD)

# Next apply the constraint enforcement equation above.
hprimeDD = ixp.zerorank2()
for i in range(DIM):
    for j in range(DIM):
        hprimeDD[i][j] = \
        (sp.Abs(rfm.detgammahat)/detgammabar)**(sp.Rational(1,3)) * (KroneckerDeltaDD[i][j] + hDD[i][j]) \
        - KroneckerDeltaDD[i][j]

enforce_detg_constraint_vars = [ \
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD00"),rhs=hprimeDD[0][0]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD01"),rhs=hprimeDD[0][1]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD02"),rhs=hprimeDD[0][2]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD11"),rhs=hprimeDD[1][1]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD12"),rhs=hprimeDD[1][2]),
                                lhrh(lhs=gri.gfaccess("in_gfs","hDD22"),rhs=hprimeDD[2][2]) ]

enforce_gammadet_string = fin.FD_outputC("returnstring",enforce_detg_constraint_vars,
                                         params="outCverbose=False,preindent=0,includebraces=False")

with open("BSSN/enforce_detgammabar_constraint.h", "w") as file:
    indent = "   "
    file.write("void enforce_detgammabar_constraint(const int Nxx_plus_2NGHOSTS[3],REAL *xx[3], REAL *in_gfs) {\n\n")
    file.write(lp.loop(["i2","i1","i0"],["0","0","0"],
                       ["Nxx_plus_2NGHOSTS[2]","Nxx_plus_2NGHOSTS[1]","Nxx_plus_2NGHOSTS[0]"],
                       ["1","1","1"],["#pragma omp parallel for",
                                      "    const REAL xx2 = xx[2][i2];",
                                      "        const REAL xx1 = xx[1][i1];"],"",
                                     "const REAL xx0 = xx[0][i0];\n"+enforce_gammadet_string))
    file.write("}\n")
print("Output C implementation of det(gammabar) constraint to file BSSN/enforce_detgammabar_constraint.h")
```

    Output C implementation of det(gammabar) constraint to file BSSN/enforce_detgammabar_constraint.h

# Step 3: Code Validation against `BSSN.Enforce_Detgammabar_Constraint` NRPy+ module \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the C code output between

1. this tutorial and
2. the NRPy+ [BSSN.Enforce_Detgammabar_Constraint](../edit/BSSN/Enforce_Detgammabar_Constraint.py) module.

```python
!mv BSSN/enforce_detgammabar_constraint.h BSSN/enforce_detgammabar_constraint.h-validation

gri.glb_gridfcs_list = []

import BSSN.Enforce_Detgammabar_Constraint as EGC
EGC.output_Enforce_Detgammabar_Constraint_Ccode()

import filecmp
for file in ["BSSN/enforce_detgammabar_constraint.h"]:
    if filecmp.cmp(file,file+"-validation") == False:
        print("VALIDATION TEST FAILED on file: "+file+".")
        exit(1)
    else:
        print("Validation test PASSED on file: "+file)
```

    Output C implementation of det(gammabar) constraint to file BSSN/enforce_detgammabar_constraint.h
    Validation test PASSED on file: BSSN/enforce_detgammabar_constraint.h

# Step 4: Output this module to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.pdf](Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.ipynb
!pdflatex -interaction=batchmode Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.tex
!pdflatex -interaction=batchmode Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.tex
!pdflatex -interaction=batchmode Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    [NbConvertApp] Converting notebook Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.ipynb to latex
    [NbConvertApp] Writing 41262 bytes to Tutorial-BSSN-Enforcing_Determinant_gammabar_equals_gammahat_Constraint.tex
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode