# Regressão linear 2:  gradiente descendente

### Aprendizado e otimização

Dada uma função *target* desconhecida $f:\mathbb{R}^{d} \rightarrow \mathbb{R}$ e uma hipótese $h_{\mathbf{w}}:\mathbb{R}^{d} \rightarrow \mathbb{R}$ o erro de $h_{\mathbf{w}}$ é definido por:

\begin{equation}
E_{out}(h_{\mathbf{w}}) = \mathbb{E}_{\mathbf{x}\sim p_{data}}L(h(\mathbf{x}; \mathbf{w}), \; f(\mathbf{x}))
\end{equation}

em que $p_{data}$ é a distribuição geradora dos dados, $L$  é alguma função de perda (por exemplo, o quadrado da diferença) e $h(\mathbf{x}; \mathbf{w}) = h_{\mathbf{w}}(\mathbf{x})$. Se tivéssemos acesso a $p_{data}$ poderíamos calcular a função acima para qualquer $h_{\mathbf{w}}$ e escolher aquele com erro mínimo.

Como não temos acesso a $p_{data}$, define-se

\begin{equation}
E_{in}(h_{\mathbf{w}}) = J(\mathbf{w}) = \frac{1}{N}\sum_{i=1}^{N} L(h(\mathbf{x}_{i}; \mathbf{w}), \; y_{i})
\end{equation}

em que $N$ é o tamanho do dataset de treinamento e $y_{i} = f(\mathbf{x}_{i})$.

Como visto em aula, uma relação entre $E_{in}$ e $E_{out}$ pode ser estabelecida pela **Inequação de Hoeffding**.

Aqui estamos interessados em encontrar $h_{\mathbf{w}}$ com erro $J(\mathbf{w})$ mínimo. Isto corresponde a determinar o ponto mínimo da função $J(\mathbf{w})$. Por isso, vamos nos concentrar apenas em uma técnica de otimização para minimizar $E_{in}$.

### Gradiente ascendente e gradiente descendente

Dados $\mathbf{w}, \mathbf{u} \in \mathbb{R}^{d}$ e $J:\mathbb{R}^{d} \rightarrow \mathbb{R}$ tal que $||\mathbf{u}||_{2} = 1$ a taxa de variação de $J$ no ponto $\mathbf{w}$ em direção a $\mathbf{u}$ é chamada de **derivada direcional**, $D_{\mathbf{u}}J(\mathbf{w})$. Definindo $g(h) = J(\mathbf{w} + h\mathbf{u})$, podemos usar a regra da cadeia e a definição de produto escalar para mostrar que

\begin{equation}
D_{\mathbf{u}}J(\mathbf{w}) \;\;=\;\; \left. \frac{dg}{dh} \right|_{h=0} \;\;=\;\; \mathbf{u}^{T}\nabla_{\mathbf{w}}J(\mathbf{w})
    \;\;=\;\; ||\mathbf{u}||_{2}||\nabla_{\mathbf{w}}J(\mathbf{w})||_{2}cos\theta \;\;=\;\; ||\nabla_{\mathbf{w}}J(\mathbf{w})||_{2}cos\theta
\end{equation}

em que $\theta$ é o ângulo entre $\mathbf{u}$ e $\nabla_{\mathbf{w}}J(\mathbf{w})$. Assim, pensando em $D_{\mathbf{u}}J(\mathbf{w})$ em termos do vetor $\mathbf{u}$ temos que: $D_{\mathbf{u}}J(\mathbf{w})$ tem o valor máximo quando $\mathbf{u}$ tem a mesma direção que $\nabla_{\mathbf{w}}J(\mathbf{w})$ ($\theta=0$); similarmente $D_{\mathbf{u}}J(\mathbf{w})$ tem o valor mínimo quando $\mathbf{u}$ tem a direção oposta a $\nabla_{\mathbf{w}}J(\mathbf{w})$ ($\theta=\pi$).

Desse modo podemos maximizar $J$ alterando $\mathbf{w}$ na direção de $\nabla_{\mathbf{w}}J(\mathbf{w})$ e podemos minimizar $J$ alterando $\mathbf{w}$ na direção de $-\nabla_{\mathbf{w}}J(\mathbf{w})$. Isso permite dois métodos bem simples de otimização:

**Gradiente Ascendente**

- $\mathbf{w}(0) = \mathbf{w}$
- for $t = 0, 1, 2, \dots$ do
    * $\mathbf{w}(t+1) = \mathbf{w}(t) + \eta \nabla_{\mathbf{w}(t)}J(\mathbf{w}(t))$

**Gradiente Descendente**

- $\mathbf{w}(0) = \mathbf{w}$
- for $t = 0, 1, 2, \dots$ do
    * $\mathbf{w}(t+1) = \mathbf{w}(t) - \eta \nabla_{\mathbf{w}(t)}J(\mathbf{w}(t))$

Em ambos os casos o parâmetro $\eta \in \mathbb{R}_{\geq}$ é chamado de **taxa de aprendizado** (*learning rate*); ele pondera o tamanho de cada atualização.

#####   A seguir são apresentados uma sequência de exercícios que ilustram diferentes aspectos práticos na implementação do algoritmo *gradient descent*
Iremos considerar o problema de regressão linear.

```python
# all imports
import numpy as np
import time
from util import r_squared, randomize_in_place, get_housing_prices_data
from plots import simple_step_plot, plot_points_regression, plot_cost_function_curve

%matplotlib inline
```

#####  Vamos usar o mesmo dataset de antes, mas agora vamos dividir os dados em treinamento, validação e teste.
Essa divisão é comumente realizada na prática: os dados de treinamento são usados na otimização da função custo $J$, os dados de validação são usados para aferir a qualidade da otimização, e os dados de teste são usados para aferir a qualidade da predição.

```python
X, y = get_housing_prices_data(N=350, verbose=False)
randomize_in_place(X, y)

train_X = X[0:250]
train_y = y[0:250]
valid_X = X[250:300]
valid_y = y[250:300]
test_X = X[300:]
test_y = y[300:]

print("\nDividindo os dados em treinamento, validação e teste")
print("\ntrain_X shape = {}".format(train_X.shape))
print("\ntrain_y shape = {}".format(train_y.shape))
print("\nvalid_X shape = {}".format(valid_X.shape))
print("\nvalid_y shape = {}".format(valid_y.shape))
print("\ntest_X shape = {}".format(test_X.shape))
print("\ntest_y shape = {}".format(test_y.shape))
```

```python
plot_points_regression(train_X,
                       train_y,
                       title='Training data',
                       xlabel="m\u00b2",
                       ylabel='$')
```

### Para minimizar a função de custo vamos ter que colocar os dados em uma outra escala

Um modo de fazer isso é o chamado [standard/z score](https://en.wikipedia.org/wiki/Standard_score).
Aplicamos a seguinte transformação:

\begin{equation}
\mathbf{X}^{\top}_{i} \leftarrow \frac{\mathbf{X}^{\top}_{i} - \mu_{i}}{\sigma_{i}}
\end{equation}

em que $\mathbf{X}^{T}_{i} \in \mathbb{R}^{N}$ ($i = 1, \dots, d$) é um vetor de features da design matrix $\mathbf{X}$, $\mu_{i}$ é a média de tal vetor, e $\sigma_{i}$ seu desvio padrão.

A importância de se fazer essa transformação é discutida mais ao final deste notebook.

#####  **Exercício 1)**
Use a biblioteca numpy para implementar a função que altera os dados conforme a equação acima (essa função deve funcionar para uma design matrix $\mathbf{X}$ com um número arbitrário de features).

```python
def standardize(X):
    """
    Returns standardized version of the ndarray 'X'.

    :param X: input array
    :type X: np.ndarray(shape=(N, d))
    :return: standardized array
    :rtype: np.ndarray(shape=(N, d))
    """

    # YOUR CODE HERE:
    raise NotImplementedError("falta completar a função standardize")
    # END YOUR CODE

    return X_out
```

**Teste exercício 1)**

```python
toy_X = np.array([[1100.3, 2.4, 34.34],
                  [2300.3, 1.4, 442.23]])
toy_y = np.array([[1000.2], [2000.5]])
toy_X_norm = standardize(toy_X)
toy_y_norm = standardize(toy_y)
xmean, xstd = np.mean(toy_X_norm), np.std(toy_X_norm)
ymean, ystd = np.mean(toy_y_norm), np.std(toy_y_norm)
assert -1 <= xmean < 0
assert 0 <= ymean < 1
assert 0.9 <= xstd <= 1
assert 0.9 <= ystd <= 1
```

```python
train_X_norm = standardize(train_X)
train_y_norm = standardize(train_y)

xmean, xstd = np.mean(train_X), np.std(train_X)
xmax, xmin = np.max(train_X), np.min(train_X)
ymean, ystd = np.mean(train_y), np.std(train_y)
ymax, ymin = np.max(train_y), np.min(train_y)

print("Dados originais\n")
print("X:\nmean {}, std {:.2f}, max {}, min {}".format(xmean,
                                                       xstd,
                                                       xmax,
                                                       xmin))
print("\ny:\nmean {}, std {:.2f}, max {}, min {}\n".format(ymean,
                                                         ystd,
                                                         ymax,
                                                         ymin))

xmean, xstd = np.mean(train_X_norm), np.std(train_X_norm)
xmax, xmin = np.max(train_X_norm), np.min(train_X_norm)
ymean, ystd = np.mean(train_y_norm), np.std(train_y_norm)
ymax, ymin = np.max(train_y_norm), np.min(train_y_norm)

print("Dados normalizados\n")
print("X:\nmean {}, std {:.2f}, max {}, min {}".format(xmean,
                                                       xstd,
                                                       xmax,
                                                       xmin))
print("\ny:\nmean {}, std {:.2f}, max {}, min {}\n".format(ymean,
                                                         ystd,
                                                         ymax,
                                                         ymin))
plot_points_regression(train_X_norm,
                       train_y_norm,
                       title='Training data (normalized)',
                       xlabel="m\u00b2",
                       ylabel='$')
```

#####  Adicionando uma componente com apenas 1s como uma nova feature
Conforme já vimos, adicionar uma componente (coordenada artificial) constante 1 é conveniente. Isto é, em vez de $\mathbf{x} \in \mathbb{R}^d$ é conveniente considerarmos $(1,\mathbf{x}) \in \mathbb{R}^{d+1}$.

```python
def add_feature_ones(X):
    """
    Returns the ndarray 'X' with the extra
    feature column containing only 1s.

    :param X: input array
    :type X: np.ndarray(shape=(N, d))
    :return: output array
    :rtype: np.ndarray(shape=(N, d+1))
    """
    return np.concatenate((np.ones((X.shape[0], 1)), X), axis=1)

train_X_1 = add_feature_ones(train_X_norm)
print("\ntrain_X shape = {}".format(train_X_1.shape))
```

#####  Criando a predição da regressão linear e plotando uma predição arbitrária

```python
def linear_regression_prediction(X, w):
    """
    Calculates the linear regression prediction.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param w: weights
    :type w: np.array(shape=(d, 1))
    :return: prediction
    :rtype: np.array(shape=(N, 1))
    """

    return X.dot(w)

w = np.array([[1.2], [2.3]])
prediction = linear_regression_prediction(train_X_1, w)

plot_points_regression(train_X_norm,
                       train_y_norm,
                       title='Training data (normalized)',
                       xlabel="m\u00b2",
                       ylabel='$',
                       prediction=prediction,
                       legend=True)
```

### Computando a função de custo

Usando o erro quadrárico médio, a função de custo $J(\mathbf{w})$ para a tarefa de regressão linear pode ser escrita de dois modos. A forma iterativa:

\begin{equation}
J(\mathbf{w}) = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_{i} - y_{i})^{2}
\end{equation}

e a forma vetorial:

\begin{equation}
J(\mathbf{w}) = \frac{1}{N}(\mathbf{X}\mathbf{w} - \mathbf{y})^{T}(\mathbf{X}\mathbf{w} - \mathbf{y})
\end{equation}

#####  **Exercício 2)**
Use a biblioteca numpy para implementar a função de custo.

```python
def compute_cost(X, y, w):
    """
    Calculates  mean square error cost.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param y: regression targets
    :type y: np.ndarray(shape=(N, 1))
    :param w: weights
    :type w: np.array(shape=(d, 1))
    :return: cost
    :rtype: float
    """

    # YOUR CODE HERE:
    raise NotImplementedError("falta completar a função compute_cost")
    # END YOUR CODE

    return J
```

**Teste exercício 2)**

```python
toy_w = np.array([[1], [1], [2]])
toy_X = np.array([[2, 3, 1],
                  [5, 1, 2]])
toy_y = np.array([[1], [1]])
assert compute_cost(toy_X, toy_y, toy_w) == 58.5
```

Podemos olhar a superficie de custo e ver onde se situa um valor $J(\mathbf{w})$.

```python
initial_w = np.array([[15], [-35.3]])
initial_J = compute_cost(train_X_1, train_y_norm, initial_w)

plot_cost_function_curve(train_X_1,
                         train_y_norm,
                         compute_cost,
                         title="Optimization landscape",
                         weights_list=[initial_w.flatten()],
                         cost_list=[initial_J])
```

### Calculando os gradientes

É fácil calcular a derivada parcial de $J(\mathbf{w})$ com relação a cada entrada $j$ de $\mathbf{w}$:

\begin{equation}
\frac{\partial J(\mathbf{w})}{\partial \mathbf{w}_{j}} = \frac{2}{N}\sum_{i=1}^{N} (\hat{y}_i - y_i) \mathbf{x}_{ij}
\end{equation}

Lembre que o gradiente de $J(\mathbf{w})$ com relação a $\mathbf{w}$ é:

\begin{equation}
\nabla_{\mathbf{w}}J(\mathbf{w}) = \begin{bmatrix}\frac{\partial J(\mathbf{w})}{\partial \mathbf{w}_{1}} \dots \frac{\partial J(\mathbf{w})}{\partial \mathbf{w}_{m}} \end{bmatrix}
\end{equation}

#####  **Exercício 3)**
Use a biblioteca numpy para calcular $\nabla_{\mathbf{w}}J(\mathbf{w})$.

```python
def compute_wgrad(X, y, w):
    """
    Calculates gradient of J(w) with respect to w.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param y: regression targets
    :type y: np.ndarray(shape=(N, 1))
    :param w: weights
    :type w: np.array(shape=(d, 1))
    :return: gradient
    :rtype: np.array(shape=(d, 1))
    """

    # YOUR CODE HERE:
    raise NotImplementedError("falta completar a função compute_wgrad")
    # END YOUR CODE

    return grad
```

**Teste exercício 3)**

```python
def grad_check(X, y, w, h=1e-4):
    """
    Check gradients for linear regression.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param y: regression targets
    :type y: np.ndarray(shape=(N, 1))
    :param w: weights
    :type w: np.array(shape=(d, 1))
    :param h: small variation
    :type h: float
    :return: gradient test
    :rtype: boolean
    """
    Jw = compute_cost(X, y, w)
    grad = compute_wgrad(X, y, w)
    passing = True
    d = w.shape[0]
    for i in range(d):
        w_plus_h = np.array(w, copy=True)
        w_plus_h[i] = w_plus_h[i] + h
        Jw_plus_h = compute_cost(X, y, w_plus_h)
        w_minus_h = np.array(w, copy=True)
        w_minus_h[i] = w_minus_h[i] - h
        Jw_minus_h = compute_cost(X, y, w_minus_h)
        numgrad_i = (Jw_plus_h - Jw_minus_h) / (2 * h)
        reldiff = abs(numgrad_i - grad[i]) / max(1, abs(numgrad_i), abs(grad[i]))
        if reldiff > 1e-5:
            passing = False
            msg = """
            Seu gradiente = {0}
            Gradiente numérico = {1}""".format(grad[i], numgrad_i)
            print("            " + str(i) + ": " + msg)
            print("            Jw = {}".format(Jw))
            print("            Jw_plus_h = {}".format(Jw_plus_h))
            print("            Jw_minus_h = {}\n".format(Jw_minus_h))

    if passing:
        print("Gradiente passando!")

    return passing

toy_w1 = np.array([[1.], [2.], [1.], [2.]])
toy_X1 = np.array([[2., 3., 1., 2.],
                  [5., 1., 1., 2.]])
toy_y1 = np.array([[1.], [-1.]])
toy_w2 = np.array([[-100.22], [20002.1], [102.5]])
toy_X2 = np.array([[2111.3, -2223., 404.0],
                  [5222., -22221., 3.3]])
toy_y2 = np.array([[122.], [221.]])
toy_w3 = np.array([[-10.22], [-3.1]])
toy_X3 = np.array([[1.3, -1.2],
                  [2.2, -2.1],
                  [-2.3, -5.5],
                  [3.2, 8.1],
                  [3.3, -1.1],
                  [-3.4, -2.22],
                  [2.23, -4.4],
                  [5.2, -2.3]])
toy_y3 = np.array([[10.3],
                   [23.3],
                   [10.1],
                   [-20.2],
                   [-10.2],
                   [20.2],
                   [-14.4],
                   [-30.3]])

assert grad_check(toy_X1, toy_y1, toy_w1)
assert grad_check(toy_X2, toy_y2, toy_w2)
assert grad_check(toy_X3, toy_y3, toy_w3)
```

### Batch gradient descent

A versão mais simples do algoritmo *gradient descent* faz uso de todas as observações do dataset de treinamento (esse algoritmo também é conhecido como *batch gradient descent* ou *vanilla gradient descent*).

**Batch gradient descent**

- $\mathbf{w}(0) = \mathbf{w}$
- for $t = 0, 1, 2, \dots$ do
    * Compute the gradient $\nabla_{\mathbf{w}(t)}J(\mathbf{w}(t))$
    * Apply update : $\mathbf{w}(t+1) = \mathbf{w}(t) - \eta \nabla_{\mathbf{w}(t)}J(\mathbf{w}(t))$

#####  **Exercício 4)**
Implemente o algoritmo batch gradient descent com a taxa de apreendizado fixa para a regressão linear. A função abaixo deve retornar três coisas: o vetor de pesos $\mathbf{w}$, uma lista com cada peso obtido ao longo do treinamento, e uma lista com o custo de cada peso.

```python
def batch_gradient_descent(X, y, w, learning_rate, num_iters):
    """
     Performs batch gradient descent optimization.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param y: regression targets
    :type y: np.ndarray(shape=(N, 1))
    :param w: weights
    :type w: np.array(shape=(d, 1))
    :param learning_rate: learning rate
    :type learning_rate: float
    :param num_iters: number of iterations
    :type num_iters: int
    :return: weights, weights history, cost history
    :rtype: np.array(shape=(d, 1)), list, list
    """

    weights_history = [w.flatten()]
    cost_history = [compute_cost(X, y, w)]

    # YOUR CODE HERE:
    raise NotImplementedError("falta completar a função batch_gradient_descent")
    # END YOUR CODE

    return w, weights_history, cost_history
```

**Teste exercício 4)**

```python
learning_rate = 0.8
iterations = 20000
init = time.time()
w, weights_history, cost_history = batch_gradient_descent(train_X_1,
                                                          train_y_norm,
                                                          initial_w,
                                                          learning_rate,
                                                          iterations)
assert cost_history[-1] < cost_history[0]
assert type(w) == np.ndarray
assert len(weights_history) == len(cost_history)
init = time.time() - init
print("Tempo de treinamento = {:.8f}(s)".format(init))
print("Tem que ser em menos de 1 segundo ")
```

#####  Agora podemos treinar o modelo

```python
learning_rate = 0.03
iterations = 400
w, weights_history, cost_history = batch_gradient_descent(train_X_1,
                                                          train_y_norm,
                                                          initial_w,
                                                          learning_rate,
                                                          iterations)
title = "Optimization landscape\nlearning rate = {} | iterations = {}".format(learning_rate,
                                                                              iterations)
plot_cost_function_curve(train_X_1,
                         train_y_norm,
                         compute_cost,
                         title=title,
                         weights_list=weights_history,
                         cost_list=cost_history)
simple_step_plot([cost_history],
             "loss",
             'Training loss\nlearning rate = {} | iterations = {}'.format(learning_rate,
                                                                          iterations))
```

### Hiper parâmetros (*hyperparameters*)

Hiper parâmetros são parâmetros que controlam o comportamento do algoritmo. Eles não são modificados pelo algoritmo de aprendizado. Escolhemos os hiper parâmetros de acordo com a performance deles no dataset de treinamento. Para evitar que o modelo decore o dataset de treinamento, pegamos uma parte desse dataset só para achar os melhores hiper parâmetros. Essa parte é chamada de **dataset de validação** (*validation set*).

```python

valid_X_norm = standardize(valid_X)
valid_y_norm = standardize(valid_y)
valid_X_1 = add_feature_ones(valid_X_norm)

hyper_params = [(0.001, 200),
                (0.1, 10),
                (0.9, 8),
                (0.02, 600)]

all_costs = []
all_w = []

for param in hyper_params:
    learning_rate = param[0]
    iterations = param[1]
    w, weights_history, cost_history = batch_gradient_descent(train_X_1,
                                                              train_y_norm,
                                                              initial_w,
                                                              learning_rate,
                                                              iterations)
    all_costs.append(compute_cost(valid_X_1, valid_y_norm, w))
    all_w.append(w)
    title = "Optimization landscape\n"
    title += "learning rate = {} | iterations = {}".format(learning_rate,
                                                           iterations)

    plot_cost_function_curve(train_X_1,
                             train_y_norm,
                             compute_cost,
                             title=title,
                             weights_list=weights_history,
                             cost_list=cost_history)

best_result_i = np.argmin(all_costs)
best_w = all_w[best_result_i]
lowest_cost = all_costs[best_result_i]
best_params = hyper_params[best_result_i]

result_str = "Best hyperparameters\n"
result_str += "learning rate = {}".format(best_params[0])
result_str += " | iterations = {}\n".format(best_params[1])
result_str += "w = {}\n".format(best_w.flatten())
result_str += "lowest validation set cost = {}\n".format(lowest_cost)

print(result_str)
```

##### Com o modelo treinado e escolhidos os melhores hiper parâmetros, podemos avaliá-lo sobre o dataset de teste.

```python
test_X_norm = standardize(test_X)
test_y_norm = standardize(test_y)
test_X_1 = add_feature_ones(test_X_norm)

prediction = linear_regression_prediction(test_X_1, best_w)
prediction = (prediction * np.std(train_y)) + np.mean(train_y)
r_2 = r_squared(test_y, prediction)

plot_points_regression(test_X,
                       test_y,
                       title='Test data',
                       xlabel="m\u00b2",
                       ylabel='$',
                       prediction=prediction,
                       r_squared=r_2,
                       legend=True)
```

### Gradiente descendente estocástico

Nos casos em que $N$ é um número grande, computar $\nabla_{\mathbf{w}}J(\mathbf{w})$ a cada iteração se torna algo muito custoso. Uma estratégia para lidar com isso é **aproximar** $\nabla_{\mathbf{w}}J(\mathbf{w})$ usando o gradiente:

\begin{equation}
\hat{\nabla_{\mathbf{w}}J(\mathbf{w})} = \nabla_{\mathbf{w}}\frac{1}{m}\sum_{i=1}^{m} L(h(\mathbf{x}_{i}; \mathbf{w}), \; y_{i})
\end{equation}

em que $(\mathbf{x}_{1}, y_{1}), \dots ,(\mathbf{x}_{m}, y_{m})$ é uma amostragem aleatória dos dados de treinamento. A  estocasticidade surge da escolha desses $m$ dados (para que $\hat{\nabla_{\mathbf{w}}J(\mathbf{w})}$ seja um estimador não enviesado de $\nabla_{\mathbf{w}}J(\mathbf{w})$ nós amostramos os $m$ dados a cada iteração). Normalmente usamos o nome **gradiente descendente estocástico** (*stochastic gradient descent* ou *online gradient descent*) quando $m=1$, e usamos o nome **minibatch stochastic gradient descent** quando $1 < m <N$ (nesse caso estamos usando apenas um pequeno lote dos dados, um *minibatch*). Usamos *batch* para referir a um *minibatch*, não confunda isso com *batch gradient descent*.

**Stochastic gradient descent (SGD)**

- $\mathbf{w}(0) = \mathbf{w}$
- for $t = 0, 1, 2, \dots$ do
    * Sample a minibatch of $m$ examples from the training data.
    * Compute the gradient estimate $\hat{\nabla_{\mathbf{w}(t)}J(\mathbf{w}(t))}$
    * Apply update : $\mathbf{w}(t+1) = \mathbf{w}(t) - \eta \hat{\nabla_{\mathbf{w}(t)}J(\mathbf{w}(t))}$

#####  **Exercício 5)**
Implemente o algoritmo stochastic gradient descent para a regressão linear com a taxa de apreendizado fixa. A saída da função é a mesma da função do exercício 4.

```python
def stochastic_gradient_descent(X, y, w, learning_rate, num_iters, batch_size):
    """
     Performs stochastic gradient descent optimization

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param y: regression targets
    :type y: np.ndarray(shape=(N, 1))
    :param w: weights
    :type w: np.array(shape=(d, 1))
    :param learning_rate: learning rate
    :type learning_rate: float
    :param num_iters: number of iterations
    :type num_iters: int
    :param batch_size: size of the minibatch
    :type batch_size: int
    :return: weights, weights history, cost history
    :rtype: np.array(shape=(d, 1)), list, list
    """
    # YOUR CODE HERE:
    raise NotImplementedError("falta completar a função stochastic_gradient_descent")
    # END YOUR CODE

    return w, weights_history, cost_history
```

**Teste exercício 5)**

```python
init = time.time()
learning_rate = 0.8
iterations = 2000
batch_size = 36
w, weights_history, cost_history = stochastic_gradient_descent(train_X_1,
                                                               train_y_norm,
                                                               initial_w,
                                                               learning_rate,
                                                               iterations,
                                                               batch_size)
assert cost_history[-1] < cost_history[0]
assert type(w) == np.ndarray
assert len(weights_history) == len(cost_history)
init = time.time() - init
print("Tempo de treinamento = {:.8f}(s)".format(init))
print("Tem que ser em menos de 1.2 segundos")
```

###### Podemos experimentar com diferentes tamanhos de batch para ver que quanto maior o tamanho do batch (mais próximo de $N$) menor a variância.

```python
hyper_params = [(0.001, 1000, 1),
                (0.001, 1000, 10),
                (0.001, 1000, 36)]
all_costs = []

for param in hyper_params:
    learning_rate = param[0]
    iterations = param[1]
    batch_size = param[2]
    _, weights_history, cost_history = stochastic_gradient_descent(train_X_1,
                                                                   train_y_norm,
                                                                   initial_w,
                                                                   learning_rate,
                                                                   iterations,
                                                                   batch_size)
    all_costs.append(cost_history)
    title = "Optimization landscape\n"
    title += "learning rate = {}".format(learning_rate)
    title += " | iterations = {}".format(iterations)
    title += " | batch size = {}".format(batch_size)
    plot_cost_function_curve(train_X_1,
                             train_y_norm,
                             compute_cost,
                             title=title,
                             weights_list=weights_history,
                             cost_list=cost_history)

_, _, cost_history_full = batch_gradient_descent(train_X_1,
                                                 train_y_norm,
                                                 initial_w,
                                                 learning_rate=0.001,
                                                 num_iters=1000)

all_costs.append(cost_history_full)
labels_size = ["batch size = " + str(param[2]) for param in hyper_params]
labels_size += ["batch size = " + str(train_X_1.shape[0])]

simple_step_plot(all_costs,
                 "loss",
                 'Training loss',
                  figsize=(8, 8),
                  labels=labels_size)
```

### Por que normalizar?

O primeiro motivo para se normalizar os dados é para evitar *overflow*. Também é verdade que quando não normalizamos os dados as *features* podem apresentar diferentes escalas -- note que esse é o caso nesse dataset em que uma *feature* só tem $1$s e a outra ($m^{2}$) apresenta bastante variação. Isso influencia no gradiente de modo que a cada atualização os valores dos pesos vão mudar de modo diferente mesmo usando o mesmo *learning rate*.

Isso pode ser visto quando acompanhamos a mudança nos pesos ao longo do treinamento no dataset original e no normalizado. Note como o parâmetro $\mathbf{w}[1]$ (que pondera a feature ($m^{2}$)) muda bem mais que o parâmetro $\mathbf{w}[0]$ quando usamos o dataset não normalizado.

```python
_, weights_history_norm, cost_history_norm = batch_gradient_descent(train_X_1,
                                                                    train_y_norm,
                                                                    initial_w,
                                                                    learning_rate,
                                                                    10)

train_X_1_non_norm = add_feature_ones(train_X)

w, weights_history, cost_history = batch_gradient_descent(train_X_1_non_norm,
                                                          train_y,
                                                          initial_w,
                                                          0.000002,
                                                          10)

w0_hist_norm = [w[0] for w in weights_history_norm]
w1_hist_norm = [w[1] for w in weights_history_norm]

w0mean, w0sdt, w0max, w0min = np.mean(w0_hist_norm), np.std(w0_hist_norm), np.max(w0_hist_norm), np.min(w0_hist_norm)
w1mean, w1sdt, w1max, w1min = np.mean(w0_hist_norm), np.std(w1_hist_norm), np.max(w1_hist_norm), np.min(w1_hist_norm)

print("\nVariação dos pesos com o dataset normalizado\n")
print("w[0]:\nmean {}, std {:.2f}, max {}, min {}".format(w0mean, w0sdt, w0max, w0min))
print("w[1]:\nmean {}, *std {:.2f}*, max {}, min {}".format(w1mean, w1sdt, w1max, w1min))

w0_hist = [w[0] for w in weights_history]
w1_hist = [w[1] for w in weights_history]

w0mean, w0sdt, w0max, w0min = np.mean(w0_hist), np.std(w0_hist), np.max(w0_hist), np.min(w0_hist)
w1mean, w1sdt, w1max, w1min = np.mean(w1_hist), np.std(w1_hist), np.max(w1_hist), np.min(w1_hist)

print("\nVariação dos pesos com o dataset não normalizado\n")
print("w[0]:\nmean {}, std {:.2f}, max {}, min {}".format(w0mean, w0sdt, w0max, w0min))
print("w[1]:\nmean {}, *std {:.2f}*, max {}, min {}".format(w1mean, w1sdt, w1max, w1min))
```

#### Um dos resultados dessa atualização em scala diferente para cada feature é a não convergência do algoritmo

```python
simple_step_plot([cost_history_norm],
                 "loss",
                 'Training loss (normalized)')

simple_step_plot([cost_history],
                 "loss",
                 'Training loss (non normalized)')

plot_cost_function_curve(train_X_1_non_norm,
                         train_y,
                         compute_cost,
                         title="Optimization landscape\n(non normalized data)",
                         weights_list=weights_history,
                         cost_list=cost_history,
                         range_points=(100, 100))
```

### Mais otimização!

Há muitos outros algoritmos de otimização construídos em cima da ideia de gradiente descendente. Um bom resumo de alguns desses algoritimos pode ser encontrado [aqui](http://ruder.io/optimizing-gradient-descent/).


# Derivatives of the $\Lambda$ function

#### Validation of the derivatives of the $\Lambda$ function for linearly limb-darkened transits

Let's import some stuff.

```python
import numpy as np
import matplotlib.pyplot as pl
%matplotlib notebook
from IPython.display import display, Math
import sympy
from sympy import *
from sympy.functions.special.delta_functions import Heaviside
from sympy.functions.special.elliptic_integrals import elliptic_k, elliptic_e, elliptic_pi

# Initialize the session
init_session(quiet=True)

# Let's report what version of sympy this is
print("Using sympy version", sympy.__version__)
```

    Using sympy version 1.3

We're going to take all the derivatives analytically using `sympy`.

```python
Lambda, r, b, ksq = symbols("\Lambda r b, k^2")
ksq = (1 - r ** 2 - b ** 2 + 2 * b * r) / (4 * b * r)
```

## $k^2$ < 1

Here's the expression from the paper, in terms of the standard elliptic integrals $E$, $K$, and $\Pi$ (which `sympy` knows how to compute):

```python
f1 = 1 / (9 * pi * sqrt(b * r))
f2 = ((r + b) ** 2 - 1) / (r + b)
f3 = -2 * r * (2 * (r + b) ** 2 + (r + b) * (r - b) - 3)
f4 = 3 * (b - r)
f5 = -4 * b * r * (4 - 7 * r ** 2 - b ** 2)
K = elliptic_k(ksq)
E = elliptic_e(ksq)
PI = elliptic_pi(ksq * (b + r) ** 2, ksq)
Lambda = f1 * (f2 * (f3 * K + f4 * PI) + f5 * E)
Lambda
```

$$\frac{- 4 b r \left(- b^{2} - 7 r^{2} + 4\right) E\left(\frac{- b^{2} + 2 b r - r^{2} + 1}{4 b r}\right) + \frac{\left(- 2 r \left(\left(- b + r\right) \left(b + r\right) + 2 \left(b + r\right)^{2} - 3\right) K\left(\frac{- b^{2} + 2 b r - r^{2} + 1}{4 b r}\right) + \left(3 b - 3 r\right) \Pi\left(\frac{\left(b + r\right)^{2} \left(- b^{2} + 2 b r - r^{2} + 1\right)}{4 b r}\middle| \frac{- b^{2} + 2 b r - r^{2} + 1}{4 b r}\right)\right) \left(\left(b + r\right)^{2} - 1\right)}{b + r}}{9 \pi \sqrt{b r}}$$

### Derivative with respect to $r$

The derivative of this expression with respect to $r$ is

```python
dLdr = simplify(diff(Lambda, r))
dLdr
```

We claim this simplifies to:

```python
dLdr_paper = (8 * b * r ** 2 * elliptic_e(ksq) + 2 * r * (1 - (b + r) ** 2) * elliptic_k(ksq)) / (pi * sqrt(b * r))
dLdr_paper
```

Do the expressions agree?

```python
simplify(dLdr_paper - dLdr) == 0
```

    True

**QED**

Now, using the relation

$\ \ \ \ \ \ \ \ \ \lambda K(k^2) + q E(k^2) = \mathrm{cel}(k_c, 1, \lambda + q, \lambda + q k_c^2)$

with

$\ \ \ \ \ \ \ \ \ \lambda = 2 r (1 - (b + r)^2)$

and

$\ \ \ \ \ \ \ \ \ q = 8 b r^2$,

we have

$\ \ \ \ \ \ \ \ \ \lambda + q =$

```python
lam = 2 * r * (1 - (b + r) ** 2)
q = 8 * b * r ** 2
factor(simplify(lam + q))
```

$=2 r (1 − (b − r)^2 )$

and

$\ \ \ \ \ \ \ \ \ \lambda + q k_c^2 =$

```python
simplify(lam + q * (1 - ksq))
```

We can then re-write the derivative as

$\frac{1}{\pi \sqrt{b r}} \mathrm{cel}(k_c, 1, 2 r (1 − (b − r)^2 ), 0)$

**QED**

### Derivative with respect to $b$

The derivative of this expression with respect to $b$ is

```python
dLdb = simplify(diff(Lambda, b))
dLdb
```

We claim this simplifies to:

```python
dLdb_paper = (4 * r * (r ** 2 + b ** 2 - 1) * elliptic_e(ksq) + 2 * r * (1 - (b + r) ** 2) * elliptic_k(ksq)) / (3 * pi * sqrt(b * r))
dLdb_paper
```

Do the expressions agree?

```python
simplify(dLdb_paper - dLdb) == 0
```

    True

**QED**

Now, using the relation

$\ \ \ \ \ \ \ \ \ \lambda K(k^2) + q E(k^2) = \mathrm{cel}(k_c, 1, \lambda + q, \lambda + q k_c^2)$

with

$\ \ \ \ \ \ \ \ \ \lambda = \frac{2r(1 - (b + r)^2)}{1 - (b - r)^2}$

and

$\ \ \ \ \ \ \ \ \ q = \frac{4r(r^2 + b^2 - 1)}{1 - (b - r)^2}$

we have

$\ \ \ \ \ \ \ \ \ \lambda + q =$

```python
lam = 2 * r * (1 - (b + r) ** 2) / (1 - (b - r) ** 2)
q = 4 * r * (r ** 2 + b ** 2 - 1) / (1 - (b - r) ** 2)
factor(simplify(lam + q))
```

and

$\ \ \ \ \ \ \ \ \ \lambda + q k_c^2 =$

```python
factor(simplify(lam + q * (1 - ksq)))
```

$=\frac{1 - (b + r)^2}{b}$

We can then re-write the derivative as

$\frac{1 - (b - r)^2}{3\pi \sqrt{b r}} \mathrm{cel}(k_c, 1, -2r, \frac{1 - (b + r)^2}{b})$

**QED**

## $k^2$ > 1

Here's the expression from the paper, in terms of the standard elliptic integrals $E$, $K$, and $\Pi$ (which `sympy` knows how to compute):

```python
f1 = 2 / (9 * pi)
f2 = 1 - (r + b) ** 2
f3 = sqrt(1 - (b - r) ** 2)
f4 = 3 * (b - r) / ((b + r) * sqrt(1 - (b - r) ** 2))
f5 = -sqrt(1 - (b - r) ** 2) * (4 - 7 * r ** 2 - b ** 2)
K = elliptic_k(1 / ksq)
E = elliptic_e(1 / ksq)
PI = elliptic_pi(1 / (ksq * (b + r) ** 2), 1 / ksq)
Lambda = f1 * (f2 * (f3 * K + f4 * PI) + f5 * E)
Lambda
```

$$\frac{2 \left(\left(\sqrt{- \left(b - r\right)^{2} + 1} K\left(\frac{4 b r}{- b^{2} + 2 b r - r^{2} + 1}\right) + \frac{\left(3 b - 3 r\right) \Pi\left(\frac{4 b r}{\left(b + r\right)^{2} \left(- b^{2} + 2 b r - r^{2} + 1\right)}\middle| \frac{4 b r}{- b^{2} + 2 b r - r^{2} + 1}\right)}{\left(b + r\right) \sqrt{- \left(b - r\right)^{2} + 1}}\right) \left(- \left(b + r\right)^{2} + 1\right) - \sqrt{- \left(b - r\right)^{2} + 1} \left(- b^{2} - 7 r^{2} + 4\right) E\left(\frac{4 b r}{- b^{2} + 2 b r - r^{2} + 1}\right)\right)}{9 \pi}$$

### Derivative with respect to $r$

The derivative of this expression with respect to $r$ is

```python
dLdr = simplify(diff(Lambda, r))
dLdr
```

We claim this simplifies to:

```python
dLdr_paper = 4 * r / pi * sqrt(1 - (b - r) ** 2) * elliptic_e(1 / ksq)
dLdr_paper
```

Do the expressions agree?

```python
simplify(dLdr_paper - dLdr) == 0
```

    True

**QED**

Now, using the relation

$\ \ \ \ \ \ \ \ \ E(m_k) = \mathrm{cel}(k_c,1,1,1 − m_k)$

the derivative may be written as

$\ \ \ \ \ \ \ \ \ \frac{4 r}{\pi} \sqrt{1 - (b - r)^2} \mathrm{cel}(k_c,1,1,k_c^2)$

**QED**

### Derivative with respect to $b$

The derivative of this expression with respect to $b$ is

```python
dLdb = simplify(diff(Lambda, b))
dLdb
```

We claim this simplifies to:

```python
dLdb_paper = 2 / (3 * b * pi) * sqrt(1 - (b - r) ** 2) * ((r ** 2 + b ** 2 - 1) * elliptic_e(1 / ksq) + (1 - (b + r) ** 2) * elliptic_k(1 / ksq))
dLdb_paper
```

Do the expressions agree?

```python
simplify(dLdb_paper - dLdb) == 0
```

    True

**QED**

Now, using the relation

$\ \ \ \ \ \ \ \ \ \lambda K(k^2) + q E(k^2) = \mathrm{cel}(k_c, 1, \lambda + q, \lambda + q k_c^2)$

with

$\ \ \ \ \ \ \ \ \ \lambda = \frac{1 - (b + r)^2}{2 b r}$

and

$\ \ \ \ \ \ \ \ \ q = \frac{r^2 + b^2 - 1}{2 b r}$

we have

$\ \ \ \ \ \ \ \ \ \lambda + q =$

```python
lam = (1 - (b + r) ** 2) / (2 * b * r)
q = (r ** 2 + b ** 2 - 1) / (2 * b * r)
factor(simplify(lam + q))
```

and

$\ \ \ \ \ \ \ \ \ \lambda + q k_c^2 =$

```python
kc2 = (1 - 1 / ksq)
factor(simplify(lam + q * kc2))
```

$=$

```python
factor(kc2)
```

$=k_c^2$

We can then re-write the derivative as

$\frac{4 r}{3\pi} \sqrt{1 - (b - r)^2} \mathrm{cel}(k_c, 1, -1, k_c^2)$

**QED**

## r = 0

In this case, $\Lambda(r, b) = 0$, so the derivatives are trivially equal to zero.

**QED**

## |r - b| $\ge$ 1

Same here. The derivatives are just zero.

**QED**

## b = 0

```python
Lambda = -Rational(2, 3) * (1 - r ** 2) ** Rational(3, 2)
```

The expression for lambda is

```python
Lambda
```

The derivative with respect to $r$ is

```python
diff(Lambda, r)
```

The expression for $\Lambda$ does not depend on $b$, so the derivative with respect to $b$ is zero.

**QED**

## b = r < 1/2

Let's differentiate the expression for $k^2 > 1$ and substitute $b = r$:

```python
dLdr_paper = 4 * r / pi * sqrt(1 - (b - r) ** 2) * elliptic_e(1 / ksq)
simplify(dLdr_paper.subs({"b": r}))
```

**QED**

Let's do the same for the $b$ derivative:

```python
dLdb_paper = 2 / (3 * b * pi) * sqrt(1 - (b - r) ** 2) * ((r ** 2 + b ** 2 - 1) * elliptic_e(1 / ksq) + (1 - (b + r) ** 2) * elliptic_k(1 / ksq))
simplify(dLdb_paper.subs({"b": r}))
```

Now, using the relation

$\ \ \ \ \ \ \ \ \ \lambda K(k^2) + q E(k^2) = \mathrm{cel}(k_c, 1, \lambda + q, \lambda + q k_c^2)$

with

$\ \ \ \ \ \ \ \ \ \lambda = -2 + \frac{1}{2 r ^ 2}$

and

$\ \ \ \ \ \ \ \ \ q = 1 -  \frac{1}{2 r ^ 2}$,

we have

$\ \ \ \ \ \ \ \ \ \lambda + q =$

```python
lam = -2 + 1 / (2 * r ** 2)
q = 1 - 1 / (2 * r ** 2)
factor(simplify(lam + q))
```

and

$\ \ \ \ \ \ \ \ \ \lambda + q k_c^2 =$

```python
kc2 = (1 - 1 / ksq).subs({"b": r})
simplify(factor(simplify(lam + q * kc2)))
```

$=$

```python
kc2
```

We can then re-write the derivative as

$\frac{4r}{3\pi} \mathrm{cel}(k_c, 1, -1, k_c^2)$

**QED**

## b = r > 1/2

Let's differentiate the expression for $k^2 < 1$ and substitute $b = r$:

```python
dLdr_paper = (8 * b * r ** 2 * elliptic_e(ksq) + 2 * r * (1 - (b + r) ** 2) * elliptic_k(ksq)) / (pi * sqrt(b * r))
simplify(dLdr_paper.subs({"b": r}))
```

Now, using the relation

$\ \ \ \ \ \ \ \ \ \lambda K(k^2) + q E(k^2) = \mathrm{cel}(k_c, 1, \lambda + q, \lambda + q k_c^2)$

with

$\ \ \ \ \ \ \ \ \ \lambda = -(4 r ^ 2 - 1)$

and

$\ \ \ \ \ \ \ \ \ q = 4 r ^ 2$,

we have

$\ \ \ \ \ \ \ \ \ \lambda + q =$

```python
lam = -(4 * r ** 2 - 1)
q = 4 * r ** 2
factor(simplify(lam + q))
```

and

$\ \ \ \ \ \ \ \ \ \lambda + q k_c^2 =$

```python
kc2 = (1 - ksq).subs({"b": r})
simplify(factor(simplify(lam + q * kc2)))
```

We can then re-write the derivative as

$\frac{2}{\pi} \mathrm{cel}(k_c, 1, 1, 0)$

**QED**

Let's do the same for the $b$ derivative:

```python
dLdb_paper = (4 * r * (r ** 2 + b ** 2 - 1) * elliptic_e(ksq) + 2 * r * (1 - (b + r) ** 2) * elliptic_k(ksq)) / (3 * pi * sqrt(b * r))
simplify(dLdb_paper.subs({"b": r}))
```

Now, using the relation

$\ \ \ \ \ \ \ \ \ \lambda K(k^2) + q E(k^2) = \mathrm{cel}(k_c, 1, \lambda + q, \lambda + q k_c^2)$

with

$\ \ \ \ \ \ \ \ \ \lambda = 4r^2 - 1$

and

$\ \ \ \ \ \ \ \ \ q = -2(2r^2 - 1)$,

we have

$\ \ \ \ \ \ \ \ \ \lambda + q =$

```python
lam = (4 * r ** 2 - 1)
q = -2 * (2 * r ** 2 - 1)
factor(simplify(lam + q))
```

and

```python
kc2 = (1 - ksq).subs({"b": r})
simplify(factor(simplify(lam + q * kc2)))
```

$=$

```python
2 * kc2
```

We can then re-write the derivative as

$-\frac{2}{3\pi} \mathrm{cel}(k_c, 1, 1, 2k_c^2)$

**QED**

## b + r = 1

Let's differentiate the expression for $k^2 > 1$ and substitute $b = 1 - r$:

```python
dLdr_paper = 4 * r / pi * sqrt(1 - (b - r) ** 2) * elliptic_e(1 / ksq)
```

```python
simplify(dLdr_paper.subs({"b": 1 - r}))
```

**QED**

Let's do the same for the $b$ derivative:

```python
dLdb_paper = 2 / (3 * b * pi) * sqrt(1 - (b - r) ** 2) * ((r ** 2 + b ** 2 - 1) * elliptic_e(1 / ksq) + (1 - (b + r) ** 2) * elliptic_k(1 / ksq))
```

```python
simplify(dLdb_paper.subs({"b": 1 - r}))
```

**QED**

## b = r = 1/2

This is a special case of $b + r = 1$. The derivative with respect to $r$ is thus

```python
simplify((8 * r * sqrt(r * (-r + 1)) / pi).subs({"r": Rational(1, 2)}))
```

**QED**

The derivative with respect to $b$ is

```python
simplify((-8 * r * sqrt(r * (-r + 1)) / (3 * pi)).subs({"r": Rational(1, 2)}))
```

**QED**


# Solution to Exercise 6.1

## Monte-Carlo Error

This is the error in the estimate at each episode when the values are updated at the end of the episode after the reward is known.

The Monte-Carlo Error is given by $G_t - V(S_t)$ which simplifies to:
$$ \begin{align*} G_t - V(S_t) &= R_{t+1} + \gamma G_{t+1} - V(S_t)\\
&= R_{t+1} + \gamma G_{t+1} - V(S_t) + \gamma V(S_{t+1}) -  \gamma V(S_{t+1}) \end{align*}$$

Given, $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$

$$ \begin{align*} G_t - V(S_t) &= \delta_t + \gamma (G_{t+1} - V(S_{t+1})) \\
&= \delta_t + \gamma \delta_{t+1} + \gamma^2 (G_{t+2} - V(S_{t+2})) \\
&= \delta_t + \gamma \delta_{t+1} + \ldots + \gamma^{T-t} (G_T - V(S_T)) \end{align*}$$

Since $G_T = V(S_T)$, we have the final simplified term for MC error as:

$$ G_t - V(S_t) = \sum_{k=t}^{T} \gamma^{k-t} \delta_k $$

## Bootstrap Error
This is the error in the estimate at each episode when the values are updated at each step of the episode before the (final/terminal) reward is known.

The update at time $t$ for a $TD(0)$ update is given as:
$ V_{t}(S_t) = V_{t-1}(S_t) + \alpha (R_{t+1} + \gamma V_{t-1}(S_{t+1}) - V_{t-1}(S_t))$

The Bootstrap Error is given by $G_t - V(S_t)$ which simplifies to:
$$ \begin{align*} G_{t} - V_{t}(S_t) &= R_{t+1} + \gamma G_{t+1} - V_t(S_t)\\
&= R_{t+1} + \gamma G_{t+1} - V_{t-1}(S_t) - \alpha (R_{t+1} + \gamma V_{t-1}(S_{t+1}) - V_{t-1}(S_t)) \\
&= R_{t+1} + \gamma G_{t+1} - V_{t-1}(S_t) - \alpha (R_{t+1} + \gamma V_{t-1}(S_{t+1}) - V_{t-1}(S_t)) + \gamma V_{t-1}(S_{t+1}) -  \gamma V_{t-1}(S_{t+1}) \\
&= (1-\alpha)R_{t+1} - (1-\alpha)V_{t-1}(S_t) + \gamma (G_{t+1} - V_{t-1}(S_{t+1})) + (1-\alpha) \gamma V_{t-1}(S_{t+1})  + \gamma V_{t}(S_{t+1}) -  \gamma V_{t}(S_{t+1})\\
&= (1-\alpha)\delta_t + \gamma (G_{t+1} - V_{t}(S_{t+1})) + \gamma (V_{t}(S_{t+1}) - V_{t-1}(S_{t+1}))\\
&= (1-\alpha)\delta_t + \gamma ((1-\alpha)\delta_{t+1} + \gamma (G_{t+2} - V_{t}(S_{t+2})) + \gamma \alpha \delta_{t+2}) + \gamma \alpha \delta_{t+1}\\
&= (1-\alpha) \delta_t + \gamma \delta_{t+1} + \ldots + \gamma^{T-t} (G_T - V(S_T))\\
&= (1-\alpha) \delta_t + \sum_{k=t+1}^{T} \gamma^{k-t} \delta_k \\
&= \sum_{k=t}^{T} \gamma^{k-t} \delta_k - \alpha \delta_t \end{align*}$$

# Solution to Exercice 7.1

## $n$-step TD Bootstrap Error

The above 2 derivations show the expression for the bootstrapping error for MonteCarlo and $1$-step TD methods. The following derivation is a more general derivation for the $n$-step TD Bootstrap error with the $1$-step method ($n=0$) & the Monte-Carlo method ($n=\infty$) in opposite extremes.

This is the error in the estimate at each episode when the values are updated at each step of the episode before the (final/terminal) reward is known.

The update at time $t$ for an $n$-step update is given by:
$$ V_{t}(S_t) = V_{t-1}(S_t) + \alpha \left(\sum_{k=t+1}^{t+n} R_{k} + \gamma^n V_{t+n-1}(S_{t+n}) - V_{t-1}(S_t) \right)$$

The Bootstrap Error is given by $G_t - V(S_t)$ is given by:
$$ \begin{align} G_t - V_t(S_t) &= R_{t+1} + \gamma G_{t+1} - V_{t-1}(S_t)-\alpha \left (\sum_{k=t+1}^{t+n} \gamma^{k-t-1} R_k + \gamma^n V_{t+n-1}(S_{t+n}) - V_{t-1}(S_t) \right) \\
&= (1-\alpha) \left(\sum_{k=t+1}^{t+n} \gamma^{k-t-1} R_k - V_{t-1}(S_t) \right) + \left(\sum_{k=t+n+1}^{T} \gamma^{k-t-1} R_k - \alpha \gamma^n V_{t+n-1}(S_{t+n}) \right) + \gamma^n V_{t+n-1}(S_{t+n}) - \gamma^n V_{t+n-1}(S_{t+n}) \\
&= (1-\alpha) \delta_t + \left(\sum_{k=t+n+1}^{T} \gamma^{k-t-1} R_k - \gamma^n V_{t+n-1}(S_{t+n}) \right) + \gamma^n V_{t+n}(S_{t+n}) - \gamma^n V_{t+n}(S_{t+n})\\
&= (1-\alpha) \delta_t + \left(\sum_{k=t+n+1}^{T} \gamma^{k-t-1} R_k - \gamma^n V_{t+n}(S_{t+n})\right) + \alpha \gamma^n \delta_{t+n} \\
&= (1-\alpha) \delta_t + \gamma^n (G_{t+n} - V_{t+n}(S_{t+n})) + \alpha \gamma^n \delta_{t+n} \\
&= (1-\alpha) \delta_t + \gamma^n  \left( (1-\alpha) \delta_{t+n} + \gamma^n (G_{t+2n} - V_{t+2n}(S_{t+2n}) + \alpha \gamma^n \delta_{t+2n}) \right) + \alpha \gamma^n \delta_{t+n} \\
&= (1-\alpha) \delta_t + \gamma^n \delta_{t+n} + \gamma^{2n} \left( (1-\alpha) \delta_{t+2n} + \gamma^n (G_{t+3n} - V_{t+3n}(S_{t+3n}) + \alpha \gamma^n \delta_{t+3n}) \right) + \alpha \gamma^{2n} \delta_{t+2n} \\
&= (1-\alpha) \delta_t + \sum_{k=1}^{T-n} \gamma^{kn} \delta_{t+kn}  \end{align}$$

# Solution to Exercice 7.4

Novel TD-Error with $n$-step SARSA

The $n$-step returns with SARSA is given by,
$$ \begin{align*} G_{t:t+n} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{n-1} R_{t+n} + \gamma^n Q_{t+n-1}(S_{t+n},A_{t+n}) \\
&= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{n-1} R_{t+n} + Q_{t-1}(S_t,A_t) + \sum_{1}^{n}\gamma^{k-1} (\gamma Q_{t+k-1}(S_{t+k},A_{t+k}) - Q_{t+k-2}(S_{t+k-1},A_{t+k-1}))  \\
&= Q_{t-1}(S_t,A_t) + \sum_{k=t}^{t+n-1} \gamma^{k-t} (R_k + \gamma Q_{k}(S_{k+1},A_{k+1})-Q_{k-1}(S_k,A_k))\end{align*} $$


# Indoor Positioning

## Theory

MAC + RSSI(Received Signal Strength Indicator)

+ RSSI: $[0, -100]$. The larger RSSI is, the more intense a signal is.

### Idea
1. Divide the area into multiple grids
2. Build the fingerprint database: $(X, Y, [(\text{MAC}_1, \text{RSSI}_1), (\text{MAC}_2, \text{RSSI}_2, \cdots, (\text{MAC}_n, \text{RSSI}_n))])$
    + When we compute RSSIs for one position, it might fluctuate, so we need to preprocess these signals
    + Both the mobile device and server can save the library
3. Obtain the vector of $([(\text{MAC}_1, \text{RSSI}_1), (\text{MAC}_2, \text{RSSI}_2, \cdots, (\text{MAC}_n, \text{RSSI}_n))])$ for user
    + Filter out improper data

4. Match the user vector in the fingerprint library data, and predict the user's location

### 1. WKNN (Weighted KNN)
#### Steps
1. Filter out unnecessary signals based on MAC. <!-- What is an unnecessary signal? How can we filter out them? -->
2. Find out the $k$ fingerprint points that is closet to the user by measuring RSSI. <!-- Is the distance measured by the RSSI, i.e., the larger RSSI is, the close the signal is to the user? -->
3. Get $(x, y)$ from $k$ fingerprint points and compute their weighted average as follows:
$$
(\hat{x}, \hat{y}) = \sum_{i=1}^K \frac{\frac{1}{D_i + \epsilon}}{\sum_{j=1}^K{\frac{1}{D_j + \epsilon}}}(x_i, y_i)
$$

#### Distance
$$
RSSI=\begin{bmatrix}
(x_1, y_1) & rssi_{11} & rssi_{12} & \dots & rssi_{1n} \\
(x_2, y_2) & rssi_{21} & rssi_{22} & \dots & rssi_{2n} \\
\vdots & \vdots & \vdots & & \vdots \\
(x_p, y_p) & rssi_{p1} & rssi_{p2} & \dots & rssi_{pn} \\
\end{bmatrix}
$$
\begin{align}
D_m = (\sum_{j=1}^m \vert \text{rssi}_j - \text{RSSI}_{pm}  \rvert^q )^{\frac{1}{q}}, j = 1, 2, \cdots, m, 1 \leq m \leq n \label{eq:distance}\tag{1}
\end{align}

#### Weight in Averaging
The closer a signal is to the user, the weight is larger. The distance is computed as (\ref{eq:distance}) and we simply choose its reverse as its weight.
+ $\epsilon$: Avoid division by zero. Example is $0.000001$

+ $p$: Number of fingerprint points in database
+ $n$: Number of signals.
+ $m$: Number of rssi. The user collects a vector of rssi, i.e., $[\text{rssi}_1, \text{rssi}_2, \cdots, \text{rssi}_m]$. The reason that $m \leq n$ is we filter out unncessary signals in Step 1.
+ $q$: If $q$ is selected as $2$, we compute the Euclidean distance

## Signal Preprocessing
+ filter
+ cluster

### Common Approach in fingerprint filtering
#### 1. deterministic denoising
+ Channel Propagation Model Denoising
+ Median Filter: Select the median number in the window. See details and examples in [Median_filter Wiki](https://en.wikipedia.org/wiki/Median_filter)
+ Neighborhood Mean Filtering

#### 2. probabilistic denoising


# Binary classification from 2 features using Tensor Flow 2.0

Based on the same data model as in the **Binary classification notebook** ([HTML](ClassificationContinuous2Features.html) / [Jupyter](ClassificationContinuous2Features.ipynb)).

The classification is on a single boundary defined by a continuous function and added white noise.

Learning goals:
- Learn about TensorFlow as a deep learning computation engine
    - Symbolic description of a pipeline (versus value based)
    - Understand the Tensors
    - Powerful gradient optimizers
    - The concepts of layers and metrics

This tutorial is an intermediate step between the home made binary regression, and the fully packaged regression implemented with the Keras API of TensorFlow in **Binary classification from 2 features with Keras** ([HTML](ClassificationContinuous2Features-Keras.html) / [Jupyter](ClassificationContinuous2Features-Keras.ipynb))

```python
import tensorflow as tf # Requires TF v2.0+
import numpy as np
from numpy import random
import matplotlib.pyplot as plt
import matplotlib.colors as pltcolors
from sklearn import metrics
import pandas
import seaborn as sns
```

## Model

Quadratic function as a boundary between positive and negative values

Adding some unknown as a Gaussian noise

The values of X are uniformly distributed and independent

```python
# Two features, Gaussian noise
nFeatures = 2
def generateBatch(N):
    #
    xMin = 0
    xMax = 1
    b = 0.1
    std = 0.1
    #
    x = random.uniform(xMin, xMax, (N, nFeatures))
    # 4th degree relation to shape the boundary
    boundary = 2*(x[:,0]**4 + (x[:,0]-0.3)**3 + b)
    # Adding some gaussian noise
    labels = boundary + random.normal(0, std, N) > x[:,1]
    return (x, labels)
```

### Training data

```python
N = 2000
# x has 1 dim in R, label has 1 dim in B
xTrain, labelTrain = generateBatch(N)

colors = ['blue','red']

fig = plt.figure(figsize=(12,4))
plt.subplot(1,3,1)
plt.scatter(xTrain[:,0], xTrain[:,1], c=labelTrain, cmap=pltcolors.ListedColormap(colors), marker=',', alpha=0.1)
plt.xlabel('x0')
plt.ylabel('x1')
plt.title('Generated train data')
plt.grid()
cb = plt.colorbar()
loc = np.arange(0,1,1./len(colors))
cb.set_ticks(loc)
cb.set_ticklabels([0,1])
plt.subplot(1,3,2)
plt.scatter(xTrain[:,0], labelTrain, marker=',', alpha=0.01)
plt.xlabel('x0')
plt.ylabel('label')
plt.grid()
plt.subplot(1,3,3)
plt.scatter(xTrain[:,1], labelTrain, marker=',', alpha=0.01)
plt.xlabel('x1')
plt.ylabel('label')
plt.grid()
```

```python
count, bins, ignored = plt.hist(labelTrain*1.0, 10, density=True, alpha=0.5)
p = np.mean(labelTrain)
print('Bernouilli parameter of the distribution:', p)
```

### Test data for verification of the model

```python
xTest, labelTest = generateBatch(N)
testColors = ['navy', 'orangered']
```

# Helpers

```python
def plotHeatMap(X, classes, title=None, fmt='.2g', ax=None, xlabel=None, ylabel=None):
    """ Fix heatmap plot from Seaborn with pyplot 3.1.0, 3.1.1
        https://stackoverflow.com/questions/56942670/matplotlib-seaborn-first-and-last-row-cut-in-half-of-heatmap-plot
    """
    ax = sns.heatmap(X, xticklabels=classes, yticklabels=classes, annot=True, fmt=fmt, cmap=plt.cm.Blues, ax=ax) #notation: "annot" not "annote"
    bottom, top = ax.get_ylim()
    ax.set_ylim(bottom + 0.5, top - 0.5)
    if title:
        ax.set_title(title)
    if xlabel:
        ax.set_xlabel(xlabel)
    if ylabel:
        ax.set_ylabel(ylabel)

def plotConfusionMatrix(yTrue, yEst, classes, title=None, fmt='.2g', ax=None):
    plotHeatMap(metrics.confusion_matrix(yTrue, yEst), classes, title, fmt, ax, \
                xlabel='Estimations', ylabel='True values');
```

# Using TensorFlow 2.0

```python
# Labels as float {0., 1.}
labelTrainF = np.multiply(labelTrain*1.0, 1.0)
labelTrainF.dtype
```

    dtype('float64')

```python
# (Mini) Batch size
nBatch = 128
# Number of batches per Epoch
nBatchPerEpoch = 10
# Safe guard to stop on number of epochs
nEpochMax = 2000
```

```python
# Simple custom layer exposing the logistic regression model
class MyLogisticRegressionLayer(tf.keras.layers.Layer):
    def __init__(self, *args, **kwargs):
        super(MyLogisticRegressionLayer, self).__init__(*args, **kwargs)

    def build(self, input_shape):
        print('Logistic layer input_shape', input_shape)
        self.w = self.add_weight(
            shape=[input_shape[1], 1],
            dtype=self.dtype,
            initializer=tf.keras.initializers.ones(),
            #regularizer=tf.keras.regularizers.l2(0.02),
            trainable=True)
        self.b = self.add_weight(
            shape=1,
            dtype=self.dtype,
            initializer=tf.keras.initializers.ones(),
            #regularizer=tf.keras.regularizers.l2(0.02),
            trainable=True)

    @tf.function
    def call(self, x, training=None):
        logits = tf.matmul(x, self.w) + self.b
        predictions = tf.math.sigmoid(logits)
        return predictions
```

**Accuracy** is defined as:
$$ \mathcal{Accuracy} = \frac 1n \sum_{i=1}^n \mathbb{1}_{\hat{y}_i = y_i}$$

With :
- $\hat{y_i} \in \{0, 1\}$ the estimated label
- $y_i \in \{0, 1\}$ the actual label
- $\mathbb{1}$ the characteristic function wich is 1 when argument is true and 0 otherwise

```python
# Using TensorFlow 2.0 style of metrics to implement accuracy
class MyBinaryAccuracy(tf.keras.metrics.Metric):

    def __init__(self, name='my_accuracy', **kwargs):
        super(MyBinaryAccuracy, self).__init__(name=name, **kwargs)
        self.accuracySum = self.add_weight(name='accuracySum',
                                           initializer='zeros')
        self.accuracyCount = self.add_weight(name='accuracyCount',
                                             initializer='zeros')

    def update_state(self, labels, yEst):
        labels = tf.cast(labels, tf.bool)
        labelEst = tf.greater(yEst, 0.5)

        values = tf.cast(tf.equal(labels, labelEst), self.dtype)
        self.accuracySum.assign_add(tf.reduce_sum(values))
        self.accuracyCount.assign_add(values.get_shape()[0])

    def result(self):
        return self.accuracySum / self.accuracyCount
```

The loss is computed as binary cross-entropy on the predictions:

As per the [binary classification](ClassificationContinuous2Features.html) notebook, the formula is :

$$\begin{align}
J{_\theta} & = - \sum_{i=0}^N{log \bigl(y_i * sig(w,b, x_i) + (1-y_i) * \left(1-sig(w,b, x_i)\right) \bigr)}
\end{align}$$

With :
$$\begin{align}
sig(w,b, x_i) &= \frac{1}{1+e^{-(w x_i + b)}} \\
y_i &\in \{0, 1\}
\end{align}$$

```python
# Model 1, instantiate the custom layer
model1 = tf.keras.Sequential([MyLogisticRegressionLayer(input_shape=[nFeatures], dtype=tf.float64)])

# Stochastic Gradient Descent Optimizer
optim1 = tf.keras.optimizers.SGD(0.01, momentum=0.01)

# Accuracy metric instance
accuracy = MyBinaryAccuracy()

# Perform a train step on a mini-batch
#  This function's code is rewritten by TensorFlow 2.0 and shall be compiled at every execution of the optimizer
@tf.function
def trainStep1(x, labels):
    with tf.GradientTape() as tape:
        predictions = model1(x, training=True)
        # Sigmoid cross-entropy from predictions
        loss = -tf.reduce_sum(tf.math.log((labels * predictions) + ((1 - labels) * (1 - predictions))))

    gradients = tape.gradient(loss, model1.trainable_variables)
    optim1.apply_gradients(zip(gradients, model1.trainable_variables))
    return loss, predictions

# Loop on epochs and mini batch
hist = []
for epoch in range(nEpochMax+1):
    cost_cumul = 0
    accuracy.reset_states()
    for b in range(0, nBatchPerEpoch):
        (xTrain, labelTrain) = generateBatch(nBatch)
        labelTrainF = (labelTrain*1.0).reshape(-1, 1)
        cost, predictions = trainStep1(xTrain, labelTrainF)
        cost_cumul += cost
        accuracy.update_state(labelTrainF, predictions)

    cost_epoch = cost_cumul / nBatchPerEpoch
    W = model1.get_weights()
    hist.append((cost_epoch.numpy(), accuracy.result().numpy(), W[1][0], W[0][0][0], W[0][1][0]))
    # Displaying result on current Epoch
    if (epoch) % 1000 == 0:
        hNow = hist[-1]
        print("Epoch %4d, Cost: %.3f, Accuracy: %.3f, b: %.3f, W: %.3f, %.3f" \
               % (epoch, hNow[0], hNow[1], hNow[2], hNow[3], hNow[4]))

# Save history as a Panda Data Frame
df = pandas.DataFrame(hist, columns = ('cost', 'accuracy', 'b', 'w0', 'w1'))
```

    Logistic layer input_shape (None, 2)
    Epoch    0, Cost: 90.565, Accuracy: 0.652, b: -0.328, W: 1.278, -0.531
    Epoch 1000, Cost: 31.143, Accuracy: 0.895, b: -1.205, W: 13.092, -10.422
    Epoch 2000, Cost: 27.600, Accuracy: 0.911, b: -1.233, W: 13.124, -10.436

```python
bEst = df['b'][len(df)-1]
wEst = np.array([df['w0'][len(df)-1], df['w1'][len(df)-1]]).reshape(-1)
print('Estimated b =', bEst, ', w =', wEst)
```

    Estimated b = -1.2326976552033388 , w = [ 13.12365251 -10.43582348]

The accuracy is quickly reaching a good level!

But weights' convergence is very noisy showing numerical instability.

```python
plt.figure(figsize=(15,9))
plt.subplot(2,3,1)
plt.plot(df['b'], marker=',');
plt.title('b')
plt.grid()
plt.subplot(2,3,2)
plt.plot(df['w0'], marker=',');
plt.title('w0')
plt.grid()
plt.subplot(2,3,3)
plt.plot(df['w1'], marker=',');
plt.title('w1')
plt.grid()
plt.subplot(2,3,4)
plt.plot(df['cost'])
plt.grid()
plt.title('Cost')
plt.subplot(2,3,5)
plt.plot(df['accuracy'])
plt.grid()
plt.title('Accuracy');
```

## Testing the model

```python
yEst = model1(xTest).numpy()
labelEst = (yEst > 0.5).reshape(-1)
```

```python
plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plt.scatter(xTest[:,0], xTest[:,1], c=labelEst, cmap=pltcolors.ListedColormap(testColors), marker='x', alpha=0.2);
plt.xlabel('x0')
plt.ylabel('x1')
plt.grid()
plt.title('Estimated')
cb = plt.colorbar()
loc = np.arange(0,1,1./len(testColors))
cb.set_ticks(loc)
cb.set_ticklabels([0,1]);
plt.subplot(1,2,2)
plt.scatter(xTest[:,0], xTest[:,1], c=labelTest, cmap=pltcolors.ListedColormap(colors), marker='x', alpha=0.1);
plt.xlabel('x0')
plt.ylabel('x1')
plt.grid()
plt.title('Generator')
cb = plt.colorbar()
loc = np.arange(0,1,1./len(colors))
cb.set_ticks(loc)
cb.set_ticklabels([0,1]);
```

```python
plt.hist(labelEst*1.0, 10, density=True)
print('Bernouilli parameter =', np.mean(labelEst))
```

```python
plotConfusionMatrix(labelTest, labelEst, np.array(['Blue', 'Red']));
```

```python
print(metrics.classification_report(labelTest, labelEst))
```

                  precision    recall  f1-score   support

           False       0.87      0.89      0.88       928
            True       0.90      0.88      0.89      1072

        accuracy                           0.89      2000
       macro avg       0.88      0.89      0.88      2000
    weighted avg       0.89      0.89      0.89      2000

# Conclusion

As we can reads in this turorial, designing a simple machine learning pipeline starting from math functions requires a lot of learning and details. Much time is spent digging into the details of the Tensors to fix dimensions.

Fortunatly, TensorFlow is also providing a higher level API, **Keras**. Same problem is solved in the **Binary classification from 2 features with Keras** ([HTML](ClassificationContinuous2Features-Keras.html) / [Jupyter](ClassificationContinuous2Features-Keras.ipynb)).

# Where to go from here ?

__Other linear implementations and simple neural nets__ [using "raw" Python or SciKit Learn](ClassificationContinuous2Features.html) ([Notebook](ClassificationContinuous2Features.ipynb)), [using Keras](ClassificationContinuous2Features-Keras.html) ([Notebook](ClassificationContinuous2Features-Keras.ipynb)) up to a simple neural net to achieve better fit, or the [__K Nearest Neighbors classifier__](ClassificationContinuous2Features-KNN.html) ([Notebook](ClassificationContinuous2Features-KNN.ipynb))

__More complex multi-class models__ on the Czech and Norways flags [using Keras](ClassificationMulti2Features-Keras.html) ([Notebook](ClassificationMulti2Features-Keras.ipynb)), showing one of the main motivations to neural networks.

__Compare with the two feature linear regression__ [using simple algorithms](../linear/LinearRegressionBivariate.html) ([Notebook](LinearRegressionBivariate.ipynb])), [or using Keras](LinearRegressionBivariate-Keras.html) ([Notebook](LinearRegressionUnivariate-Keras.ipynb))


## 3 Oct 2019

RULES

Date: Level 2 heading ##

Example Heading: Level 3 heading ###

Method Heading: Level 4 heading ####

### References

1. [Taan_S_Elali]_Discrete_Systems_and_Digital_Signal(z-lib.org)

### Imports

```python
import numpy as np
from sympy import oo
import math
import sympy as sp
import matplotlib.pyplot as plt
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
from IPython.display import display
from IPython.display import display_latex
from sympy import latex
import math
from scipy import signal
```

### Setup

```python
sp.init_printing(use_latex = True)

z, f, i = sp.symbols('z f i')
x, k = sp.symbols('x k')
theta = sp.Symbol('theta')
```

```python
# Usage: display_equation('u_x', x)
def display_equation(idx, symObj):
    if(isinstance(idx, str)):
        eqn = '\\[' + idx + ' = ' + latex(symObj) + '\\]'
        display_latex(eqn, raw=True)
    else:
        eqn = '\\[' + latex(idx) + ' = ' + latex(symObj) + '\\]'
        display_latex(eqn, raw=True)
    return
```

```python
# Usage: display_full_latex('u_x')
def display_full_latex(idx):
    if(isinstance(idx, str)):
        eqn = '\\[' + idx + '\\]'
        display_latex(eqn, raw=True)
    else:
        eqn = '\\[' + latex(idx) + '\\]'
        display_latex(eqn, raw=True)
    return
```

```python
def sum_of_GP(a, r):
    return sp.simplify(a/(1-r))
```

```python
def display_ztrans(f, k, limits = (-4, 4)):
    F = sp.summation(f/z**k, ( k, -oo, oo ))
    display_equation('f(k)', f)
    display_equation('F(k)_{\infty}', F)

    F = sp.summation(f/z**k, (k, limits[0], limits[1]))
    display_equation('F(k)_{'+ str(limits[0]) + ',' + str(limits[1]) + '}', F)
    return
```

```python
# Credit: https://www.dsprelated.com/showcode/244.php
import numpy as np
import matplotlib.pyplot as plt
from  matplotlib import patches
from matplotlib.figure import Figure
from matplotlib import rcParams

def zplane(b,a,filename=None):
    """Plot the complex z-plane given a transfer function.
    """

    # get a figure/plot
    ax = plt.subplot(111)

    # create the unit circle
    uc = patches.Circle((0,0), radius=1, fill=False,
                        color='black', ls='dashed')
    ax.add_patch(uc)

    # The coefficients are less than 1, normalize the coeficients
    if np.max(b) > 1:
        kn = np.max(b)
        b = b/float(kn)
    else:
        kn = 1

    if np.max(a) > 1:
        kd = np.max(a)
        a = a/float(kd)
    else:
        kd = 1

    # Get the poles and zeros
    p = np.roots(a)
    z = np.roots(b)
    k = kn/float(kd)

    # Plot the zeros and set marker properties
    t1 = plt.plot(z.real, z.imag, 'go', ms=10)
    plt.setp( t1, markersize=10.0, markeredgewidth=1.0,
              markeredgecolor='k', markerfacecolor='g')

    # Plot the poles and set marker properties
    t2 = plt.plot(p.real, p.imag, 'rx', ms=10)
    plt.setp( t2, markersize=12.0, markeredgewidth=3.0,
              markeredgecolor='b', markerfacecolor='b')

    ax.spines['left'].set_position('center')
    ax.spines['bottom'].set_position('center')
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)

    # set the ticks
    r = 1.5; plt.axis('scaled'); plt.axis([-r, r, -r, r])
    ticks = [-1, -.5, .5, 1]; plt.xticks(ticks); plt.yticks(ticks)

    if filename is None:
        plt.show()
    else:
        plt.savefig(filename)

    return z, p, k
```

### Discrete Fourier Transform

```python
display_full_latex('X(\\theta) = \sum_{-\infty}^{\infty} x[n]e^{j(\\theta)n}')
```

\[X(\theta) = \sum_{-\infty}^{\infty} x[n]e^{j(\theta)n}\]

### [1] Example 3.8

```python
# Symbolic
n = sp.Symbol('n')
fs = (0.5 ** n) * sp.Heaviside(n)
# or
#f = sp.Piecewise((0, n < 0), (f1, n >= 0))

display_equation('x(n)', fs)
q = sum_of_GP(1, 0.5 * sp.exp(-theta))
display_equation('X(\\theta)', q)

# Numeric
max = 20
t = np.arange(0, max, 1)
f = np.power(0.5, t)
plt.stem(t, f)
plt.xlabel('t')
plt.ylabel('Magnitude')
plt.title('Time Domain')

# Symbolic
sp.plot(fs, (n, -10, 10))
```

```python

```

```python

```


# Evaluating a Training Algorithm

Suppose we are trying to improve the performance of a machine learning system. What should we try next?

## 1. Deciding what to try next

Suppose that we've implemented the regularized linear regression algorithm to predict the house prices.

\begin{align}
h_{\theta}(\boldsymbol{x}) & = \theta_0 x_0 + \theta_1 x_1 + \dots + \theta_n x_n \\
                           & = \boldsymbol{x}^T \boldsymbol{\theta} \\
                           & = \boldsymbol{\theta}^T \boldsymbol{x}
\end{align}

$$
\min_{\boldsymbol{\theta}} \frac{1}{2m}\left[\sum_{i=1}^{m}(h_{\theta}(\boldsymbol{x}^{(i)}) - y^{i})^2 + \lambda \sum_{j=1}^{n}\theta_j^2\right].
$$

However, when we test our trained hypothesis on new examples, we find out that the errors in the predictions are unacceptably large. What should we do?

- Get more training examples.
- Try smaller sets of features.
- Try getting additional features.
- Try adding polynomial features.
- Try increasing/decreasing $\lambda$

### Machine learning diagnostic

> A **diagnostic** is a test that one can run to gain insight of what is/isn't working with a learning algorithm, and gain guidance as how to best to improve its performance.

## 2. Evaluating a hypothesis

When we learn the parameters of our learning algorithm, we think about choosing the parameters to minimize the training cost function. Hence, one may think that a very low value of the cost function on the training examples may be a good thing.

However, recall the overfitting example:

```python
# Import libraries
from matplotlib import pyplot as plt
import numpy as np
%matplotlib inline
```

```python
# Data
x = np.array([1, 1.5, 2, 3, 5])
y = np.array([1, 2, 3, 3.5, 4])
```

```python
# Second-order degree polynomial
X2 = np.concatenate([np.array([x**i]).T for i in range(3)], axis=1)
theta2 = np.linalg.solve(X2.T.dot(X2), X2.T.dot(y))
theta2
```

    array([-1.07295285,  2.48362283, -0.29528536])

```python
# Fourth-order degree polynomial
X4 = np.concatenate([np.array([x**i]).T for i in range(5)], axis=1)
theta4 = np.linalg.solve(X4.T.dot(X4), X4.T.dot(y))
theta4
```

    array([ 2.21428572, -5.53571429,  6.05952381, -1.92857143,  0.19047619])

```python
# Draw some data
x_fit = np.linspace(0, 7)
X2_fit = np.concatenate([np.array([x_fit**i]).T for i in range(3)], axis=1)
X4_fit = np.concatenate([np.array([x_fit**i]).T for i in range(5)], axis=1)
y2 = X2_fit.dot(theta2)
y4 = X4_fit.dot(theta4)

plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.plot(x, y, 'rx', ms=10, label='data')
plt.plot(x_fit, y2, label='Polynomial degree 2')
plt.tick_params(
    axis='both',       # changes apply to the x-axis
    which='both',      # both major and minor ticks are affected
    bottom=False,      # ticks along the bottom edge are off
    left=False,
    labelleft=False,
    labelbottom=False) # labels along the bottom edge are off
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.axis([0, 6, 0, 5])
plt.legend(loc='best')
plt.subplot(1, 2, 2)
plt.plot(x, y, 'rx', ms=10, label='data')
plt.plot(x_fit, y4, label='Polynomial degree 4')
plt.tick_params(
    axis='both',       # changes apply to the x-axis
    which='both',      # both major and minor ticks are affected
    bottom=False,      # ticks along the bottom edge are off
    left=False,
    labelleft=False,
    labelbottom=False) # labels along the bottom edge are off
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.axis([0, 6, 0, 5])
plt.legend(loc='best')
```

... A perfect fit may not be a good hypothesis, actually.

This type of overfit hypothesis will fail to generalize over new examples.

The standard way to evaluate a learned hypothesis is as follows:

1. Suppose we have the following dataset, with 10 training examples:

| Size | Price |
| ---- | ----- |
| 2104 | 400   |
| 1600 | 330   |
| 2400 | 369   |
| 1416 | 232   |
| 3000 | 540   |
| 1985 | 300   |
| 1534 | 315   |
| 1427 | 199   |
| 1380 | 212   |
| 1494 | 243   |

2. We split this dataset as follows:
   - We assume the dataset is shuffled (if not, shuffle it first).
   - 70% (80%) is left as **training set** $\{(x^{(1)}, y^{(1)}), \dots, (x^{(m)}, y^{(m)})\}$.

| Size | Price |
| ---- | ----- |
| 2104 | 400   |
| 1600 | 330   |
| 2400 | 369   |
| 1416 | 232   |
| 3000 | 540   |
| 1985 | 300   |
| 1534 | 315   |

   - The rest 30% (20%) is the **test set** $\{(x_{test}^{(1)}, y_{test}^{(1)}), \dots, (x_{test}^{(m_{test})}, y^{(m_{test})})\}$.

| Size | Price |
| ---- | ----- |
| 1427 | 199   |
| 1380 | 212   |
| 1494 | 243   |

3. With this:
   - We learn the parameters $\theta$ minimizing the cost function over the trainig set.
   - We compute the cost function over the test set:

     $$
     J_{test}(\theta) = \frac{1}{2m_{test}} \sum_{i=1}^{m_{test}}(h_{\theta}(x_{test}^{(i)}) - y_{test}^{(i)}),
     $$

     for the linear regression, or,

     $$
     J_{test}(\theta) = -\frac{1}{m_{test}} \sum_{i=1}^{m_{test}}y_{test}^{(i)}\log h_{\theta}(x_{test}^{(i)}) + (1 - y_{test}^{(i)})\log (1 - h_{\theta}(x_{test}^{(i)})),
     $$

     for the logistic regression.

Another metric we could use to evaluate the logistic regression is me misclassification error:

$$
err(h_{\theta}(x), y) = \left\{\begin{array}{ccc}
1 & \text{if} & h_{\theta}(x)\geq 0.5 \text{ and } y=0 \text{, or } h_{\theta}(x)< 0.5 \text{ and } y=1 \\
0 & \text{otherwise} &
\end{array}\right.
$$

$$
error = \frac{1}{m_{test}} \sum_{i=1}^{m_{test}} err(h_{\theta}(x_{test}^{(i)}), y_{test}^{(i)})
$$

## 3. Model Selection: Train/Test/Validation Sets

As we saw in the overfit example, since the parameters are fit w.r.t. the training dataset, the cost function evaluated over the training dataset is likely to be lower than the cost function evaluated on unseen data instances.

In this sense, the cost function evaluated on the training set is not a good evaluation metric.

### Model selection

Let's assume that we have fitted a polynomials of order 1, 2, ..., 10 to some training dataset:

1. $h_{\theta}(x) = \theta_0 + \theta_1 x$
2. $h_{\theta}(x) = \theta_0 + \theta_1 x + \theta_2 x^2$
3. $h_{\theta}(x) = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3$

...

10. $h_{\theta}(x) = \theta_0 + \theta_1 x + \dots + \theta_{10} x^{10}$

Then, one think we could do in order to decide which degree $d$ is the best, is to evaluate the cost function over the test dataset for each hypothesis:

1. $J_{test}(\Theta^{(1)})$
2. $J_{test}(\Theta^{(2)})$
3. $J_{test}(\Theta^{(3)})$

...

10. $J_{test}(\Theta^{(10)})$

Thus, if we select the degree of the polynomial $d$ according to these values, then our **hyperparameter** $d$ will fit to the test set. This is, it will be an optimistic estimate of the generalization error.

### Evaluating your hypothesis

In the sense of the above problem, what one can do is to split the example data into three sets:

- The training set: This is the set which will be used to learn the model parameters $\Theta$. $\{(x^{(1)}, y^{(1)}), \dots, (x^{(m)}, y^{(m)})\}$
- The validation set: This is the set which will be used to learn the model hyperparameters (such as $d$, in the above example). $\{(x_{cv}^{(1)}, y_{cv}^{(1)}), \dots, (x_{cv}^{(m_{cv})}, y_{cv}^{(m_{cv})})\}$
- The test set: This set is to be used to evaluate the model only. $\{(x_{test}^{(1)}, y_{test}^{(1)}), \dots, (x_{test}^{(m_{test})}, y_{test}^{(m_{test})})\}$

Thus, instead of using the test set to select the hyperparameter(s) of a model, we should use the validation dataset.

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez. Based on the content of the Machine Learning course offered through coursera by Prof. Andrew Ng.
</footer>


# 2.4 Deep Taylor Decomposition Part 1.

## Introduction

**This section corresponds to Section 5.2 of the original paper.**

I recommend reviewing Section 2.2 before reading this section.

In the previous section, we induced the LRP-$\alpha \beta$ rule through a series of constraints. Deep Taylor Decomposition is also a variant of LRP that is based upon Taylor Decomposition. In Simple Taylor Decomposition (Section 2.2), we applied Taylor Expansion to the entire network without considering the structure of DNN. However, in Deep Taylor Decomposition, we will apply Taylor Expansion in a *layer-wise* manner, much like how we propagated relevance layer by layer in the LRP-$\alpha \beta$ tutorial. I will show the application of Taylor Expansion to a simplified one-layer network. *Deep Taylor Decomposition is simple once the one-layer version is derived, since all we have to do is to apply the one-layer version iteratively from output layer to input layer.*

I will restate the three constraints below so we can refer back to it whenever necessary.

##### Constraint 1. $R(x_i) > 0$ should indicate positive contribution and $R(x_i) < 0$ negative contribution. If the range of $R$ is limited to nonnegative numbers, $R(x_i) = 0$ indicates no contribution and $R(x_i) > 0$ indicates positive contribution (Section 2.1).

##### Constraint 2. Relevance Conservation Constraint (Section 2.2):

\begin{equation}
f(x) = \sum^V_{i=1} R(x_i)
\end{equation}

##### Constraint 3. Layer-wise Relevance Conservation Constraint: total relevance must be preserved throughout layers (Section 3.1).

\begin{equation}
f(x) = \cdots = \sum_{d = 1}^{V(l+1)} R^{(l+1)}_d = \sum_{d = 1}^{V(l)} R^{(l)}_d = \cdots = \sum_{i = 1}^{V(1)} R^{(1)}_d
\end{equation}

## Notations

Let us review the notations before moving onto the algorithm.

\begin{equation*}
R_{i \leftarrow j}^{(l,l+1)}
\end{equation*}

The above notation denotes the portion of relevance distributed from neuron $j$ of layer $l+1$ to neuron $i$ of layer $l$.

\begin{equation*}
R_{i}^{(l)}
\end{equation*}

This denotes the total relevance allocated to neuron $i$ of layer $l$. $R_{i \leftarrow j}^{(l,l+1)}$, $R_{i}^{(l)}$ and $R_{j}^{(l+1)}$ are related by the *Layer-wise Relevance Conservation Constraint*:

\begin{align*}
\sum_i R_{i \leftarrow j}^{(l,l+1)} & = R_{j}^{(l+1)} \\
\sum_j R_{i \leftarrow j}^{(l,l+1)} & = R_{i}^{(l)}
\end{align*}

## Application to One-layer Networks

Consider a simple three-layer ReLU network. The original paper says it's an one-layer network, but I will treat it as a three-layer network throughout this tutorial for clarity and coherence of notation. Then, we can define the network as:

\begin{equation}
x_j = \max \left( 0, \sum_i x_i w_{ij} + b_j \right)
\end{equation}
and
\begin{equation}
x_k = \sum_j x_j
\end{equation}
where $\{x_i\}$ is a $d$-dimensional input, $\{x_j\}$ is a detection layer, $x_k$ is the output and $w_{ij}$ and $b_j$ are the weight and bias parameters of the network. The mapping $\{x_i\} \rightarrow x_k$ then defines a function $f$. We now perform the Taylor Decomposition of this network. We will set an additional constraint on biases, where we force $b_j \leq 0$ for all $j$. This constraint ensures the existance of a root point $\{\tilde{x}_i\}$ of the function $f$ and therefore allows us to apply Taylor Decomposition which requires a root point.

### Relevance distribution from output to detection layer

Since we must satisfy the *Layer-wise Relevance Conservation Constraint*, we start by equating the predicted output to the amount of total relevance that must be backpropagated, i.e., $R_{k}^{(3)} = x_k$. The relevance for the top layer can now be expressed in terms of lower-layer neurons as:

\begin{equation}
R_{k}^{(3)} = \sum_j x_j
\end{equation}

Having established the mapping between $\{x_j\}$ and $R_{k}^{(3)}$, we would like to redistribution $R_{k}^{(3)}$ onto neurons $\{x_j\}$. Instead of using the LRP-$\alpha \beta$ rule as we did in the previous section, we apply Taylor Decomposition. Then the redistributed relevances $R_j$ can be written as:

\begin{equation}
R_{j \leftarrow k}^{(2,3)} = \frac{\partial R_{k}^{(3)}}{\partial x_j} |_{\{\tilde{x}_j\}} \cdot (x_j - \tilde{x}_j)
\end{equation}
where $\{\tilde{x}_j\}$ is the root point, i.e., $\sum_j \tilde{x}_j = 0$. Note that since there is only one output neuron, $R_{j}^{(2)} = \sum_k R_{j \leftarrow k}^{(2,3)} = R_{j \leftarrow k}^{(2,3)}$.

Finding the root point is simple. Because $x_j = \max \left( 0, \sum_i x_i w_{ij} + b_j \right)$, $x_j$ must be nonnegative. The only point that is both a root ($\sum_j \tilde{x}_j = 0$) and nonnegative ($\forall \ j : \tilde{x}_j \geq 0$) is $\{\tilde{x}_j\} = 0$. Plugging this root into *Eq. (6)* and using the fact that $\partial R_{k}^{(3)} / \partial x_j = 1$, we obtain the first rule for relevance distribution:

\begin{equation}
R_{j}^{(2)} = x_j
\end{equation}

In other words, the relevance must be redistributed on the neurons of the detection layer in the same proportion as their activation value. Trivially, we can also verify that the relevance is conserved during the redistribution process ($\sum_j R_{j}^{(2)} = \sum_j x_j = R_{k}^{(3)}$).

### Relevance distribution from detection to input layer

Let us now express the relevance $R_{j}^{(2)}$ as a function of the input neurons $\{x_i\}$. Since we have $R_{j}^{(2)} = x_j$ by *Eq. (7)*, we can write

\begin{equation}
R_{j}^{(2)} = \max \left( 0, \sum_i x_i w_{ij} + b_j \right)
\end{equation}

This establishes a mapping between $\{x_i\}$ and $R_{j}^{(2)}$. We will again apply Taylor Decomposition to derive $R_{i}^{(1)}$ as follows

\begin{equation}
R_{i \leftarrow j}^{(1,2)} = \frac{\partial R_{j}^{(2)}}{\partial x_i} |_{\{\tilde{x}_i\}^{(j)}} \cdot (x_i - \tilde{x}_i^{(j)})
\end{equation}
where $\{\tilde{x}_i\}^{(j)}$ are the set of root points for which $\max \left( 0, \sum_i \tilde{x}_i^{(j)} w_{ij} + b_j \right) = x_j = 0$. By the *Layer-wise Relevance Conservation Constraint*

\begin{equation}
R_{i}^{(1)} = \sum_j \frac{\partial R_{j}^{(2)}}{\partial x_i} |_{\{\tilde{x}_i\}^{(j)}} \cdot (x_i - \tilde{x}_i^{(j)})
\end{equation}

Relevances $\{R_i\}$ can therefore be obtained by performing as many Taylor Decompositions as there are neurons in the hidden layer.

## Various Propagation Rules

We will now consider various methods for choosing a root $\{\tilde{x}_i\}^{(j)}$ that takes the diversity of possible input domain $\mathcal{X} \subseteq \mathbb{R}^d$ to which the data belongs. Different choice of input domain leads to a different rule for propagating relevance from $\{R_{j}^{(2)}\}$ to $\{R_{i}^{(1)}\}$. For pooling layers (e.g., max-pooling, sum-pooling), we use the same propagation rules as those of LRP-$\alpha \beta$. *As I said in the introduction, once we derive the propagation rules, we only have to apply them from the output layer to the input layer in order to achieve Deep Taylor Decomposition.*

### Unconstrained input space $\mathcal{X} = \mathbb{R}^d$ and the $w^2$-rule

We first consider the simplest case where any real-valued input is valid. In that case, we can always choose the root point $\{\tilde{x}_i\}^{(j)}$ that is nearest to the actual data point $\{x_i\}$ under the Euclidean norm (since we are using first-order Taylor Expansion, we must choose the closest root in order to minimize the residual error: review Section 2.2, footnote 1). We know that the root must satisfy the plane equation $\sum_i \tilde{x}_i^{(j)} w_{ij} + b_j = 0 $. Since $\mathbf{w}_j$ is the normal vector of the plane, we can derive the line of maximum descent $\{\tilde{x}_i\}^{(j)} = \{x_i\} + t \cdot \mathbf{w}_j$ where $t \in \mathbb{R}$. Because the line intersects perpendicularly with the plane, the intersection of those two subspaces gives us the nearest root point. With that root point, we can express $R_{i \leftarrow j}^{(1,2)}$ in terms of $w_{ij}$ and $R_{j}^{(2)}$. Calculation process is given below.

First we derive $\tilde{x}_i^{(j)}$ in terms of $x_i$, $w_{ij}$ and $b_j$.
\begin{align}
\sum_i \tilde{x}_i^{(j)} w_{ij} + b_j = 0 \\
\sum_i (x_i + t w_{ij}) w_{ij} + b_j = 0 \\
\sum_i (x_i w_{ij} + t w_{ij}^2) + b_j = 0 \\
\sum_i x_i w_{ij} + t \sum_i w_{ij}^2 + b_j = 0 \\
t = \frac{-1}{\sum_i w_{ij}^2} \left( \sum_i x_i w_{ij} + b_j \right) \\
\tilde{x}_{i}^{(j)} = x_i - \frac{w_{ij}}{\sum_i w_{ij}^2} \left( \sum_i x_i w_{ij} + b_j \right)
\end{align}

Now we calculate the partial derivative of $R_{j}^{(2)}$ with respect to $x_i$.
\begin{align}
R_{j}^{(2)} = \max \left( 0, \sum_i x_i w_{ij} + b_j \right) \\
\frac{\partial R_{j}^{(2)}}{\partial x_i} = w_{ij}
\end{align}

Now we are ready to calculate $R_{i \leftarrow j}^{(1,2)}$.
\begin{align}
R_{i \leftarrow j}^{(1,2)} & = \frac{\partial R_{j}^{(2)}}{\partial x_i} |_{\{\tilde{x}_i\}^{(j)}} \cdot (x_i - \tilde{x}_i^{(j)}) \\
& = \frac{w_{ij}^2}{\sum_{i} w_{ij}^2} \left( \sum_i x_i w_{ij} + b_j \right) \\
& = \frac{w_{ij}^2}{\sum_{i} w_{ij}^2} R_{j}^{(2)}
\end{align}

Total relevance can be calculated by the *Layer-wise Relevance Conservation Constraint*.
\begin{equation}
R_{i}^{(1)} = \sum_j \frac{w_{ij}^2}{\sum_{i'} w_{i'j}^2} R_{j}^{(2)}
\end{equation}

We call this propagation method the $w^2$-rule.

### Constrained input space $\mathcal{X = \mathbb{R}_{+}^{d}}$ and the $z^+$-rule

We now consider the case $\mathcal{X = \mathbb{R}_{+}^{d}}$ where the input values are constrained to be positive, e.g., the activations of ReLU neurons. In that case, we restrict the search domain to the segment $(\{x_i 1_{w_{ij} < 0}\}, \{x_i\}) \subset \mathbb{R}_+^d$ that is guaranteed to have at least one root $\{\tilde{x}_i\}^{(j)} = \{x_i 1_{w_{ij} < 0}\}$ (you can verify this fact by checking that $\sum_i (x_i 1_{w_{ij} < 0}) w_{ij} \leq 0$ and therefore the ReLU activation value is $0$.)

We can derive the propagation rule through the following series of calculations:

Since the search domain is restricted to $(\{x_i 1_{w_{ij} < 0}\}, \{x_i\})$, we need a different expression for $\tilde{x}_i^{(j)}$:

\begin{equation}
\tilde{x}_i^{(j)} = \left( 1 + t \frac{w_{ij}^+}{w_{ij}} \right) x_i = x_i + t x_i \frac{w_{ij}^+}{w_{ij}}
\end{equation}
where $t \in (-1,0)$ and $w_{ij}^+$ denotes the positive part of $w_{ij}$.

We can now express $\tilde{x}_i^{(j)}$ in terms of $x_i$, $w_{ij}$ and $b_j$.
\begin{align}
\sum_i \tilde{x}_i^{(j)} w_{ij} + b_j = 0 \\
\sum_i \left( x_i + t x_i \frac{w_{ij}^+}{w_{ij}} \right) w_{ij} + b_j = 0 \\
\sum_i (x_i w_{ij} + t x_i w_{ij}^+) + b_j = 0 \\
\sum_i x_i w_{ij} + t \sum_i x_i w_{ij}^+ + b_j = 0 \\
t = \frac{-1}{\sum_i x_i w_{ij}^+} \left( \sum_i x_i w_{ij} + b_j \right) \\
\tilde{x}_{i}^{(j)} = x_i - \frac{x_i w_{ij}^+}{w_{ij} \sum_i x_i w_{ij}^+} \left( \sum_i x_i w_{ij} + b_j \right)
\end{align}

We reuse the results of *Eq. (18)* for the partial derivative calculation. Now we are ready to calculate $R_{i \leftarrow j}^{(1,2)}$.

\begin{align}
R_{i \leftarrow j}^{(1,2)} & = \frac{\partial R_{j}^{(2)}}{\partial x_i} |_{\{\tilde{x}_i\}^{(j)}} \cdot (x_i - \tilde{x}_i^{(j)}) \\
& = w_{ij} \cdot \frac{x_i w_{ij}^+}{w_{ij} \sum_i x_i w_{ij}^+} \left( \sum_i x_i w_{ij} + b_j \right) \\
& = \frac{x_i w_{ij}^+}{\sum_i x_i w_{ij}^+} R_{j}^{(2)} \\
& = \frac{z_{ij}^+}{\sum_i z_{ij}^+} R_{j}^{(2)}
\end{align}
where $z_{ij}^+ = x_i w_{ij}^+$.

Total relevance can be calculated by the *Layer-wise Relevance Conservation Constraint*.
\begin{equation}
R_{i}^{(1)} = \sum_j \frac{z_{ij}^+}{\sum_{i'} z_{i'j}^+} R_{j}^{(2)}
\end{equation}

We call this propagation method the $z^+$-rule. Note that this method is exactly the LRP-$\alpha_1 \beta_0$ rule we derived in Section 3.1.

#### $z^+$-rule implementation for dense and convolution layers

```python
def backprop_dense(self, activation, kernel, relevance):
    W_p = tf.maximum(0., kernel)
    z = tf.matmul(activation, W_p) + 1e-10
    s = relevance / z
    c = tf.matmul(s, tf.transpose(W_p))
    return activation * c

def backprop_conv(self, activation, kernel, relevance, strides, padding='SAME'):
    W_p = tf.maximum(0., kernel)
    z = nn_ops.conv2d(activation, W_p, strides, padding) + 1e-10
    s = relevance / z
    c = nn_ops.conv2d_backprop_input(tf.shape(activation), W_p, s, strides, padding)
    return activation * c
```

### Constrained input space $\mathcal{B} = \{\{x_i\}: \forall_{i=1}^d l_i \geq x_i \geq h_i\}$ and the $z^{\mathcal{B}}$-rule

For image classification tasks, pixel values are typically bounded above and below. That is, they exist within a box $\mathcal{B} = \{\{x_i\}: \forall_{i=1}^d l_i \geq x_i \geq h_i\}$ where $l_i \leq 0$ and $h_i \geq 0$ are the smallest and largest allowed pixel values for each dimension. In that case, we can restrict the search domain to the segment $(\{l_i 1_{w_{ij} > 0} + h_i 1_{w_{ij} < 0}\}, \{x_i\}) \subset \mathcal{B}$ that is also guaranteed to have at least one root $\{l_i 1_{w_{ij} > 0} + h_i 1_{w_{ij} < 0}\}$ ($\sum_i (l_i 1_{w_{ij} > 0} + h_i 1_{w_{ij} < 0}) w_{ij} \leq 0$ and therefore the ReLU activation value is $0$).

We can derive the propagation rule through the following series of calculations:

Since the search domain is restricted to $(\{l_i 1_{w_{ij} > 0} + h_i 1_{w_{ij} < 0}\})$, we need a different expression for $\tilde{x}_i^{(j)}$:

\begin{equation}
\tilde{x}_i^{(j)} = x_i + t \left( x_i - \frac{w_{ij}^+}{w_{ij}} l_i - \frac{w_{ij}^-}{w_{ij}} h_i \right)
\end{equation}
where $t \in (-1,0)$ and $w_{ij}^+$ and $w_{ij}^-$ denote the positive and negative parts of $w_{ij}$.

We can now express $\tilde{x}_i^{(j)}$ in terms of $x_i$, $w_{ij}$ and $b_j$.
\begin{align}
\sum_i \tilde{x}_i^{(j)} w_{ij} + b_j = 0 \\
\sum_i \left( x_i + t \left( x_i - \frac{w_{ij}^+}{w_{ij}} l_i - \frac{w_{ij}^-}{w_{ij}} h_i \right) \right) w_{ij} + b_j = 0 \\
\sum_i (x_i w_{ij} + t (x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-)) + b_j = 0 \\
\sum_i x_i w_{ij} + t \sum_i (x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-) + b_j = 0 \\
t = \frac{-1}{\sum_i (x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-)} \left( \sum_i x_i w_{ij} + b_j \right) \\
\tilde{x}_{i}^{(j)} = x_i - \frac{x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-}{w_{ij} \sum_i (x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-)} \left( \sum_i x_i w_{ij} + b_j \right)
\end{align}

We reuse the results of *Eq. (18)* for the partial derivative calculation. Now we are ready to calculate $R_{i \leftarrow j}^{(1,2)}$.

\begin{align}
R_{i \leftarrow j}^{(1,2)} & = \frac{\partial R_{j}^{(2)}}{\partial x_i} |_{\{\tilde{x}_i\}^{(j)}} \cdot (x_i - \tilde{x}_i^{(j)}) \\
& = w_{ij} \cdot \frac{x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-}{w_{ij} \sum_i (x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-)} \left( \sum_i x_i w_{ij} + b_j \right) \\
& = \frac{x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-}{\sum_i x_i w_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-} R_{j}^{(2)} \\
& = \frac{z_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-}{\sum_i z_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-} R_{j}^{(2)}
\end{align}
where $z_{ij} = x_i w_{ij}$.

Total relevance can be calculated by the *Layer-wise Relevance Conservation Constraint*.

\begin{equation}
R_{i}^{(1)} = \sum_j \frac{z_{ij} - l_i w_{ij}^+ - h_i w_{ij}^-}{\sum_{i'} z_{i'j} - l_i w_{i'j}^+ - h_i w_{i'j}^-} R_{j}^{(2)}
\end{equation}

We call this propagation method the $z^{\mathcal{B}}$-rule.

#### $z^{\mathcal{B}}$-rule implementation for dense and convolution layers

```python
def backprop_dense_input(self, X, kernel, relevance, lowest=0., highest=1.):
    W_p = tf.maximum(0., kernel)
    W_n = tf.minimum(0., kernel)

    L = tf.ones_like(X, tf.float32) * lowest
    H = tf.ones_like(X, tf.float32) * highest

    z_o = tf.matmul(X, kernel)
    z_p = tf.matmul(L, W_p)
    z_n = tf.matmul(H, W_n)

    z = z_o - z_p - z_n + 1e-10
    s = relevance / z

    c_o = tf.matmul(s, tf.transpose(kernel))
    c_p = tf.matmul(s, tf.transpose(W_p))
    c_n = tf.matmul(s, tf.transpose(W_n))

    return X * c_o - L * c_p - H * c_n

def backprop_conv_input(self, X, kernel, relevance, strides, padding='SAME', lowest=0., highest=1.):
    W_p = tf.maximum(0., kernel)
    W_n = tf.minimum(0., kernel)

    L = tf.ones_like(X, tf.float32) * lowest
    H = tf.ones_like(X, tf.float32) * highest

    z_o = nn_ops.conv2d(X, kernel, strides, padding)
    z_p = nn_ops.conv2d(L, W_p, strides, padding)
    z_n = nn_ops.conv2d(H, W_n, strides, padding)

    z = z_o - z_p - z_n + 1e-10
    s = relevance / z

    c_o = nn_ops.conv2d_backprop_input(tf.shape(X), kernel, s, strides, padding)
    c_p = nn_ops.conv2d_backprop_input(tf.shape(X), W_p, s, strides, padding)
    c_n = nn_ops.conv2d_backprop_input(tf.shape(X), W_n, s, strides, padding)

    return X * c_o - L * c_p - H * c_n
```


```python
import numpy as np
from sympy import *
init_printing(use_latex='mathjax')
```

```python
def get_jacobian(expressions, symbols):
    rows = len(expressions)
    columns = len(symbols)

#     print("Expressions:")
#     for expression in expressions:
#         display(expression)

    results = [[0 for x in range(columns)] for y in range(rows)]
    for row, expression in enumerate(expressions):
        for column, symbol in enumerate(symbols):
#             print('Row %d, column %d, expression: %s, symbol: %s' % (row, column, expression, symbol))
            df = diff(expression, symbol)
#             print("DF: %s" % df)
            results[row][column] = df
    return results
```

```python
def get_hessians(jacobian, symbols):
    jacobian = np.array(jacobian)
    rows, columns = jacobian.shape
    hessians = [[[None for z in range(len(symbols))] for x in range(columns)] for y in range(rows)]
    for row in range(rows):
        for column in range(columns):
            for index, symbol in enumerate(symbols):
                df = diff(jacobian[row, column], symbol)
                hessians[row][column][index] = df

    return hessians
```

```python
def evaluate_at(exprs, symbols):
    results = []
    def process_list(exprs, symbols):
        r = []
        for expr in exprs:
            if isinstance(expr, list):
                r.append(process_list(expr, symbols))
            else:
                r.append(expr.subs(symbols))
        return r

    if isinstance(exprs, list):
        results = process_list(exprs, symbols)
    else:
        results = []
        results.append(exprs.subs(symbols))
    return results
```

```python
x, y = symbols('x y')
f = sin(pi * x - y * x ** 2 )
j = get_jacobian([f], [x,y])
h = get_hessians(j, [x,y])
evaluate_at(h, {x: 1, y: pi})
```

$$\left [ \left [ \left [ - 2 \pi, \quad -2\right ], \quad \left [ -2, \quad 0\right ]\right ]\right ]$$

```python

```

Certification 2/Week4.2 - 2D polynomial estimation - final.ipynb


```python
import sympy as sm
```

```python
x = sm.symbols('x')
```

```python
y1 = sm.Pow(sm.E, x)
```

```python
y1.diff(x)
```

$\displaystyle e^{x}$

```python
y1.integrate(x)
```

$\displaystyle e^{x}$

```python
%matplotlib inline
fig = sm.plotting.plot(y1, x)
```

```python

```


# Formula kuadrat

Formula kuadrat adalah rumus yang digunakan untuk mencari akar-akar dari suatu persamaan kuadrat.

## Persamaan kuadrat
Terdapat suatu persamaan kuadrat dalam bentuk

\begin{equation}\label{eq1}\tag{1}
y = ax^2 + bx + c
\end{equation}

yang akar-akarnya dapat dicari melalui formula kuadrat.

## Formula kuadrat
Suatu persamaan memiliki dua akar, di mana

\begin{equation}\label{eq2}\tag{2}
x_1 = \frac{-b - \sqrt{b^2 - 4ac}}{2a}
\end{equation}

akan memberikan akar pertama dan

\begin{equation}\label{eq3}\tag{3}
x_2 = \frac{-b + \sqrt{b^2 - 4ac}}{2a}
\end{equation}

akan memberikan akar kedua dengan $x_1 < x_2$. Penerapan Persamaan \eqref{eq2} dan  \eqref{eq3} akan diberikan pada bagian berikutnya.

## Contoh
Berikut ini adalah contoh sederhana penerapan dua persamaan sebelumnya dengan persamaannya adalah

$$
y = x^2 - 5x + 6
$$

sehingga $a = 1$, $b = -5$, dan $c = 6$.

```python
import math

a = 1
b = -5
c = 6

D = b * b - 4 * a * c

x1 = (-b - math.sqrt(D)) / (2*a)
x2 = (-b + math.sqrt(D)) / (2*a)

print("x1 = ", x1)
print("x2 = ", x2)
```

    x1 =  2.0
    x2 =  3.0

Dengan demikian diperoleh bahwa akar-akar dari persamaan $y = x^2 - 5x + 6$ adalah $x_1 = 2.0$ dan $x_2 = 3.0$.

## Grafik
Untuk persamaan $y = x^2 - 5x + 6$ dapat dibuat grafiknya sebagai berikut.

```python
import matplotlib.pyplot as plt

def poly(x):
    a = 1
    b = -5
    c = 6
    y = a * x * x + b * x + c
    return y

x = []
y = []
for i in range(0, 21):
    x.append(i * 0.25)
    y.append(poly(i * 0.25))

print(x)
print(y)
```

    [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.25, 3.5, 3.75, 4.0, 4.25, 4.5, 4.75, 5.0]
    [6.0, 4.8125, 3.75, 2.8125, 2.0, 1.3125, 0.75, 0.3125, 0.0, -0.1875, -0.25, -0.1875, 0.0, 0.3125, 0.75, 1.3125, 2.0, 2.8125, 3.75, 4.8125, 6.0]

```python
plt.plot(x,y)

plt.show()
```

```python

```