---
layout: page
title: Gradiente Descendente
nav_order: 17
---

[](https://colab.research.google.com/github/icd-ufmg/icd-ufmg.github.io/blob/master/_lessons/17-gradiente.ipynb)

# Gradiente Descendente

{: .no_toc .mb-2 }

Entendimento de relação entre dados.
{: .fs-6 .fw-300 }

{: .no_toc .text-delta }
Resultados Esperados

1. Entender como o computador otimiza funções
1. Entender o algoritmo de gradiente descendente e ascendente
1. Implementar os dois

---
**Sumário**
1. TOC
{:toc}
---

```python
# -*- coding: utf8

from scipy import stats as ss

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
```

```python
plt.style.use('seaborn-colorblind')
plt.rcParams['figure.figsize']  = (16, 10)
plt.rcParams['axes.labelsize']  = 20
plt.rcParams['axes.titlesize']  = 20
plt.rcParams['legend.fontsize'] = 20
plt.rcParams['xtick.labelsize'] = 20
plt.rcParams['ytick.labelsize'] = 20
plt.rcParams['lines.linewidth'] = 4
```

```python
plt.ion()
```

```python
def despine(ax=None):
    if ax is None:
        ax = plt.gca()
    # Hide the right and top spines
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)

    # Only show ticks on the left and bottom spines
    ax.yaxis.set_ticks_position('left')
    ax.xaxis.set_ticks_position('bottom')
```

# Aula 18 - Gradiente Descendente (*Gradient Descent*)

**Material adaptado do Professor Pedro Olmo**

Ao fazer ciência de dados é comum se deparar com casos onde buscamos um melhor modelo. Diferente dos testes de hipóteses, onde buscamos a chance dos nossos dados em um modelo. Aqui, estamos tentando encontrar o melhor modelo para uma determinada situação. Geralmente "melhor" significará algo como "minimiza o erro do modelo" ou "maximiza a verossimilhaça dos dados".

Para casos simples, como na regressão linear, é possível achar o melhor modelo usando cálculo. Porém, em casos particulares, precisamos achar tal modelo na marra. Nossa abordagem será uma técnica chamada gradiente descendente, que funciona muito bem para diferentes cenário. Gradiente descendente é a base da maioria dos __frameworks__ de aprendizado de máquina. Você pode não achar isso super excitante por si só, mas ao combinar gradiente descendente com modelos complexos temos a fórmula do sucesso de aprendizando hoje em dia.

## A ideia por trás do gradiente descendente

Vamos focar no caso de regressão linear com dados z-normalizados. Aqui, temos um modelo muito simples:

$$y_i = \beta x_i + \epsilon_i$$

Sabemos tratar com o mesmo de um ponto de vista de minimos quadrados como também de máxima verossimilhança. Vamos explorar uma base onde queremos prever o comprimento dos narizes de Kangaroos. Para isto vamos usar uma feature fantástica, a largura do nariz do Kangaroo.

```python
df = pd.read_csv('https://media.githubusercontent.com/media/icd-ufmg/material/master/aulas/18-GradienteDescendente/gray-kangaroos.csv', names=['comprimento', 'largura'], header=0)
df.head()
```

      comprimento
      largura

      0
      609
      241

      1
      629
      222

      2
      620
      233

      3
      564
      207

      4
      645
      247

Abaixo temos o modelo.

```python
sns.regplot(x='largura', y='comprimento', data=df, n_boot=10000,
            line_kws={'color':'magenta', 'lw':4},
            scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
despine()
```

z-normalizando

```python
df -= df.mean()
df /= df.std(ddof=1)
sns.regplot(x='largura', y='comprimento', data=df, n_boot=10000,
            line_kws={'color':'magenta', 'lw':4},
            scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
despine()
```

A regressão. Note o intercepto = 0.

```python
y = df['comprimento']
x = df['largura']
ss.linregress(x, y)
```

    LinregressResult(slope=0.8814144365382701, intercept=5.560934949938166e-16, rvalue=0.8814144365382703, pvalue=1.3423645343535288e-15, stderr=0.07203174220550969)

Abaixo temos um plot com a soma dos erros ao quadrado que é utilizado para chegar na reta. Lembre-se das aulas anteriores. Chegamos no resultado acima usando cálculo, isto é, setamos as derivadas para zero. Uma outra abordagem para minimizar o erro é escolher um ponto de partida aleatório. Deste, calculamos o gradiente (derivadas). Depois, damos um pequeno passo na oposta do gradiente.

Inicialmente, observe como o erro é para um ponto. Lembrando que estamos trabalhando com uma função de erros quadrado. Diferente das aulas anteriores, ao invés de somar a mesma vamos tirar a média. Como o número de pontos, $n$, é uma constante, não afeta nossos algoritmos. Ao setar a derivada para zero o n some. De qualquer forma, pode ajudar os nossos algoritmos (evitar overflow).

$L(\beta) = n^{-1} \sum_i (y_i - \beta x_i)^2$

```python
betas = np.linspace(0.1, 16, 100)
errs = np.zeros_like(betas)
for j in range(len(betas)):
    err = ((y[0] - betas[j] * x[0]) ** 2).mean()
    errs[j] = err
plt.plot(betas, errs, label='L(beta)')
plt.xlabel(r'$\beta$')
plt.title(r'Erro para um ponto: x = {:.2f}, y = {:.2f}'.format(x[2], y[2]))
plt.ylabel(r'$(y_i - \beta x_i)^2$')
plt.grid()
despine()
```

A soma de duas funções convexas é convexa. Então, a soma do erro quadrado nos pontos tamém é convexa. (prova nos slides)

```python
betas = np.linspace(0.1, 1.7, 100)
errs = np.zeros_like(betas)
grads = np.zeros_like(betas)
for j in range(len(betas)):
    err = ((y - betas[j] * x) ** 2).mean()
    grads[j] = (-2 * (y - betas[j] * x) * x).mean()
    errs[j] = err
plt.plot(betas, errs, label='L(beta)')
#plt.plot(betas, grads, label='L\'(beta)')
plt.xlabel(r'$\beta$')
plt.title(r'Erro na soma para todos os pontos')
plt.ylabel(r'$\sum_i(y_i - \beta x_i)^2$')
plt.legend()
plt.grid()
despine()
```

Observe que essa função possui um ponto de mínimo nas quando $\beta \approx 0.9$.  Observe que o gradiente, ou a derivada, é negativa quando estamos do lado esquero do mínimo. É positiva do lado direto e zero no valor ótimo. Então, a partir de um local inicial o sinal da derivada, na verdade o oposto do sinal como estamos minimizando, indica a direção que devemos ir.

```python
plt.plot(betas, errs, label='L(beta)')
plt.plot(betas, grads, label='L\'(beta)')
plt.xlabel(r'$\beta$')
plt.title(r'Erro na soma para todos os pontos')
plt.ylabel(r'$\sum_i(y_i - \beta x_i)^2$')
plt.legend()
plt.grid()
despine()
```

Caso você tenha problemas com derivadas, use o módulo sympy. Abaixo temos a derivadas para um ponto apenas.

```python
import sympy as sy

x_i , y_i, beta = sy.symbols('x_i y_i beta')
fun = (y_i - beta * x_i)**2
sy.diff(fun, beta)
```

$\displaystyle - 2 x_{i} \left(- \beta x_{i} + y_{i}\right)$

## Gradiente Descendente

Com o conhecimento acima temos a base do Gradiente Descendente. Iniciando de um local qualquer, $\beta=2$. Vamos:

1. Computar a derivada $\frac{dL(\beta)}{d\beta}$
1. Setar $\beta = \beta - \lambda \frac{dL(\beta)}{d\beta}$
1. Repetir

O valor de $\lambda$ define nossa taxa de aprendizado.

```python
def ssq_znormed(x, y, beta):
    return ((y - beta * x) ** 2).mean()
```

```python
def grad_znormed(x, y, beta):
    return (-2 * x * (y - beta * x)).mean()
```

```python
beta = 2
beta_passo = [beta]
erros = []
for _ in range(500):
    err_sq = ssq_znormed(x, y, beta)
    grad = grad_znormed(x, y, beta)
    beta = beta - 0.01 * grad
    beta_passo.append(beta)
    erros.append(err_sq)
plt.plot(beta_passo)
plt.xlabel('Passo')
plt.ylabel('Valor de Beta')
despine()
```

Observe os erros

```python
plt.plot(erros)
plt.xlabel('Passo')
plt.ylabel('Erro Sq')
despine()
```

## Gradiente Descendente Estocástico

Muitas vezes usaremos o gradiente descendente para escolher os parâmetros de um modelo de maneira a minimizar alguma noção de erro. Usando a abordagem em lote anterior, cada etapa do método exige que façamos uma previsão e calculemos o gradiente para todo o conjunto de dados, o que faz com que cada etapa demore muito.

Geralmente, essas funções de erro são *aditivas*, o que significa que o erro preditivo em todo o conjunto de dados é simplesmente a soma dos erros preditivos para cada ponto de dados.

Quando este é o caso, podemos, em vez disso, aplicar uma técnica chamada gradiente descendente estocástico (ou *stochastic gradient descent*), que calcula o gradiente (e dá um passo) para apenas um ponto por vez. Ele passa sobre os dados repetidamente até atingir um ponto de parada.

Durante cada ciclo, queremos iterar nossos dados em uma ordem aleatória:

```python
beta = 2
beta_passo = [beta]
erros = []
for _ in range(1000):
    i = np.random.randint(0, len(x))
    x_i, y_i = x[i], y[i]
    err_sq = ssq_znormed(x, y, beta) # esta linha ainda precisa ser em todos os pontos
    grad = grad_znormed(x_i, y_i, beta)
    beta = beta - 0.01 * grad
    beta_passo.append(beta)
    erros.append(err_sq)
plt.plot(beta_passo)
plt.xlabel('Passo')
plt.ylabel('Valor de Beta')
despine()
```

```python
plt.plot(erros)
plt.xlabel('Passo')
plt.ylabel('Erro Sq')
despine()
```

## Escolhendo o tamanho certo do passo

Embora o raciocínio para se mover na direção oposta ao gradiente seja claro, até que ponto mover-se não é. De fato, escolher o tamanho certo do passo que devemos nos mover é mais uma arte do que uma ciência. Opções populares incluem:

* Usar um tamanho de passo fixo;

* Diminuir gradualmente o tamanho do passo ao longo do tempo;

* Em cada etapa, escolha o tamanho do passo que minimiza o valor da função objetivo.

O último parece ótimo, mas é, na prática, um cálculo caro. Podemos aproximá-lo tentando uma variedade de tamanhos de passo e escolhendo aquele que resulta no menor valor da função objetivo:

## Caso Multivariado

Até agora estamos roubando um pouco ao normalizar os dados. Isto é, estamos forçando $\alpha=0$ na equação abaixo:

$$y_i = \beta x_i + \alpha$$

Vamos voltar para os dados originais. Note que o resultado da regressão é outro. Em particular, o intercepto é 22. Queremos estimar o mesmo também.

```python
x = np.random.normal(size=1000)
y = 2 * x + np.random.normal(loc=3, size=1000)
ss.linregress(x, y)
```

    LinregressResult(slope=1.9416732636887266, intercept=3.033966007834007, rvalue=0.8897448608393281, pvalue=0.0, stderr=0.031531628347458034)

```python
plt.scatter(x, y, alpha=0.8, s=80, edgecolors='k')
despine()
```

Abaixo tenho as derivadas:

$$\frac{dL}{d\alpha} = n^{-1}\sum_i 2 (-y_i + \beta x_i + \alpha)$$

$$\frac{d L}{d\beta} =n^{-1} \sum_i 2 (-y_i + \beta x_i + \alpha) x_i$$

```python
def derivadas_regressao(parametros, x, y):
    alpha, beta = parametros
    da = (-2 * y) + (2 * beta * x) + (2 * alpha)
    db = (-2 * y * x) + (2 * beta * (x ** 2)) + (2 * alpha * x)
    return np.array([da.mean(), db.mean()])
```

Um gradient descent simples. O mesmo vai parar de executar quando os parâmetros convergem. Aqui temos uma função mais organizada. A mesma recebe uma outra função de derivada.

```python
def gd(parametros, d_fun, x, y, lambda_=0.01):
    parametros = parametros.copy()
    for i in range(1000):
        grad = d_fun(parametros, x, y)
        parametros_novos = parametros - lambda_ * grad
        if np.abs(parametros - parametros_novos).mean() <= 0.00001:
            break
        parametros = parametros_novos
        print('Iter {}; alpha, beta = '.format(i), parametros)
    return parametros
```

Resposta!

```python
parametros_iniciais = np.array([1, 1])
gd(parametros_iniciais, derivadas_regressao, x, y)
```

    Iter 0; alpha, beta =  [1.04079108 1.01956402]
    Iter 1; alpha, beta =  [1.08076401 1.03872175]
    Iter 2; alpha, beta =  [1.11993521 1.05748163]
    Iter 3; alpha, beta =  [1.15832076 1.07585192]
    Iter 4; alpha, beta =  [1.19593642 1.09384071]
    Iter 5; alpha, beta =  [1.23279763 1.11145591]
    Iter 6; alpha, beta =  [1.26891952 1.12870528]
    Iter 7; alpha, beta =  [1.30431693 1.14559642]
    Iter 8; alpha, beta =  [1.33900439 1.16213676]
    Iter 9; alpha, beta =  [1.37299614 1.17833358]
    Iter 10; alpha, beta =  [1.40630612 1.19419402]
    Iter 11; alpha, beta =  [1.43894803 1.20972506]
    Iter 12; alpha, beta =  [1.47093526 1.22493354]
    Iter 13; alpha, beta =  [1.50228093 1.23982615]
    Iter 14; alpha, beta =  [1.53299793 1.25440945]
    Iter 15; alpha, beta =  [1.56309885 1.26868987]
    Iter 16; alpha, beta =  [1.59259606 1.28267368]
    Iter 17; alpha, beta =  [1.62150167 1.29636706]
    Iter 18; alpha, beta =  [1.64982754 1.30977602]
    Iter 19; alpha, beta =  [1.6775853  1.32290648]
    Iter 20; alpha, beta =  [1.70478635 1.33576422]
    Iter 21; alpha, beta =  [1.73144185 1.34835489]
    Iter 22; alpha, beta =  [1.75756275 1.36068404]
    Iter 23; alpha, beta =  [1.78315976 1.37275711]
    Iter 24; alpha, beta =  [1.8082434  1.38457941]
    Iter 25; alpha, beta =  [1.83282397 1.39615614]
    Iter 26; alpha, beta =  [1.85691155 1.40749241]
    Iter 27; alpha, beta =  [1.88051604 1.4185932 ]
    Iter 28; alpha, beta =  [1.90364711 1.42946341]
    Iter 29; alpha, beta =  [1.92631428 1.44010783]
    Iter 30; alpha, beta =  [1.94852684 1.45053113]
    Iter 31; alpha, beta =  [1.97029391 1.46073792]
    Iter 32; alpha, beta =  [1.99162443 1.47073269]
    Iter 33; alpha, beta =  [2.01252715 1.48051984]
    Iter 34; alpha, beta =  [2.03301065 1.49010368]
    Iter 35; alpha, beta =  [2.05308335 1.49948844]
    Iter 36; alpha, beta =  [2.07275348 1.50867824]
    Iter 37; alpha, beta =  [2.09202912 1.51767714]
    Iter 38; alpha, beta =  [2.11091818 1.5264891 ]
    Iter 39; alpha, beta =  [2.12942841 1.535118  ]
    Iter 40; alpha, beta =  [2.14756741 1.54356765]
    Iter 41; alpha, beta =  [2.16534262 1.55184176]
    Iter 42; alpha, beta =  [2.18276136 1.55994398]
    Iter 43; alpha, beta =  [2.19983075 1.56787788]
    Iter 44; alpha, beta =  [2.21655782 1.57564695]
    Iter 45; alpha, beta =  [2.23294942 1.58325462]
    Iter 46; alpha, beta =  [2.24901229 1.59070424]
    Iter 47; alpha, beta =  [2.26475301 1.59799909]
    Iter 48; alpha, beta =  [2.28017806 1.60514239]
    Iter 49; alpha, beta =  [2.29529376 1.61213728]
    Iter 50; alpha, beta =  [2.31010631 1.61898685]
    Iter 51; alpha, beta =  [2.3246218  1.62569411]
    Iter 52; alpha, beta =  [2.33884619 1.63226201]
    Iter 53; alpha, beta =  [2.3527853  1.63869346]
    Iter 54; alpha, beta =  [2.36644487 1.64499128]
    Iter 55; alpha, beta =  [2.3798305  1.65115826]
    Iter 56; alpha, beta =  [2.39294769 1.6571971 ]
    Iter 57; alpha, beta =  [2.40580182 1.66311047]
    Iter 58; alpha, beta =  [2.41839816 1.66890098]
    Iter 59; alpha, beta =  [2.43074189 1.67457118]
    Iter 60; alpha, beta =  [2.44283807 1.68012356]
    Iter 61; alpha, beta =  [2.45469167 1.68556057]
    Iter 62; alpha, beta =  [2.46630755 1.69088461]
    Iter 63; alpha, beta =  [2.47769048 1.69609803]
    Iter 64; alpha, beta =  [2.48884514 1.70120312]
    Iter 65; alpha, beta =  [2.4997761  1.70620213]
    Iter 66; alpha, beta =  [2.51048784 1.71109727]
    Iter 67; alpha, beta =  [2.52098477 1.71589069]
    Iter 68; alpha, beta =  [2.53127119 1.7205845 ]
    Iter 69; alpha, beta =  [2.54135132 1.72518079]
    Iter 70; alpha, beta =  [2.55122931 1.72968156]
    Iter 71; alpha, beta =  [2.5609092 1.7340888]
    Iter 72; alpha, beta =  [2.57039497 1.73840447]
    Iter 73; alpha, beta =  [2.57969052 1.74263045]
    Iter 74; alpha, beta =  [2.58879965 1.74676861]
    Iter 75; alpha, beta =  [2.59772611 1.75082078]
    Iter 76; alpha, beta =  [2.60647355 1.75478874]
    Iter 77; alpha, beta =  [2.61504558 1.75867425]
    Iter 78; alpha, beta =  [2.62344571 1.76247901]
    Iter 79; alpha, beta =  [2.63167738 1.7662047 ]
    Iter 80; alpha, beta =  [2.63974398 1.76985296]
    Iter 81; alpha, beta =  [2.64764881 1.7734254 ]
    Iter 82; alpha, beta =  [2.65539512 1.77692361]
    Iter 83; alpha, beta =  [2.66298609 1.78034911]
    Iter 84; alpha, beta =  [2.67042483 1.78370342]
    Iter 85; alpha, beta =  [2.6777144  1.78698802]
    Iter 86; alpha, beta =  [2.68485779 1.79020436]
    Iter 87; alpha, beta =  [2.69185793 1.79335385]
    Iter 88; alpha, beta =  [2.6987177  1.79643789]
    Iter 89; alpha, beta =  [2.7054399  1.79945783]
    Iter 90; alpha, beta =  [2.7120273 1.802415 ]
    Iter 91; alpha, beta =  [2.7184826  1.80531071]
    Iter 92; alpha, beta =  [2.72480845 1.80814624]
    Iter 93; alpha, beta =  [2.73100745 1.81092283]
    Iter 94; alpha, beta =  [2.73708214 1.81364171]
    Iter 95; alpha, beta =  [2.74303501 1.81630409]
    Iter 96; alpha, beta =  [2.74886851 1.81891112]
    Iter 97; alpha, beta =  [2.75458503 1.82146397]
    Iter 98; alpha, beta =  [2.76018691 1.82396375]
    Iter 99; alpha, beta =  [2.76567646 1.82641158]
    Iter 100; alpha, beta =  [2.77105593 1.82880853]
    Iter 101; alpha, beta =  [2.77632753 1.83115566]
    Iter 102; alpha, beta =  [2.78149342 1.83345399]
    Iter 103; alpha, beta =  [2.78655571 1.83570456]
    Iter 104; alpha, beta =  [2.79151649 1.83790834]
    Iter 105; alpha, beta =  [2.7963778  1.84006631]
    Iter 106; alpha, beta =  [2.80114162 1.84217943]
    Iter 107; alpha, beta =  [2.80580992 1.84424862]
    Iter 108; alpha, beta =  [2.8103846 1.8462748]
    Iter 109; alpha, beta =  [2.81486755 1.84825886]
    Iter 110; alpha, beta =  [2.8192606  1.85020167]
    Iter 111; alpha, beta =  [2.82356557 1.8521041 ]
    Iter 112; alpha, beta =  [2.82778421 1.85396698]
    Iter 113; alpha, beta =  [2.83191825 1.85579114]
    Iter 114; alpha, beta =  [2.8359694  1.85757737]
    Iter 115; alpha, beta =  [2.83993931 1.85932647]
    Iter 116; alpha, beta =  [2.84382962 1.86103921]
    Iter 117; alpha, beta =  [2.84764192 1.86271634]
    Iter 118; alpha, beta =  [2.85137777 1.86435861]
    Iter 119; alpha, beta =  [2.85503871 1.86596673]
    Iter 120; alpha, beta =  [2.85862624 1.86754143]
    Iter 121; alpha, beta =  [2.86214183 1.86908338]
    Iter 122; alpha, beta =  [2.86558693 1.87059328]
    Iter 123; alpha, beta =  [2.86896295 1.87207178]
    Iter 124; alpha, beta =  [2.87227127 1.87351955]
    Iter 125; alpha, beta =  [2.87551325 1.87493722]
    Iter 126; alpha, beta =  [2.87869023 1.87632541]
    Iter 127; alpha, beta =  [2.8818035  1.87768474]
    Iter 128; alpha, beta =  [2.88485434 1.87901581]
    Iter 129; alpha, beta =  [2.88784401 1.8803192 ]
    Iter 130; alpha, beta =  [2.89077373 1.8815955 ]
    Iter 131; alpha, beta =  [2.89364471 1.88284525]
    Iter 132; alpha, beta =  [2.89645812 1.88406903]
    Iter 133; alpha, beta =  [2.89921511 1.88526735]
    Iter 134; alpha, beta =  [2.90191682 1.88644076]
    Iter 135; alpha, beta =  [2.90456436 1.88758978]
    Iter 136; alpha, beta =  [2.90715881 1.8887149 ]
    Iter 137; alpha, beta =  [2.90970124 1.88981662]
    Iter 138; alpha, beta =  [2.91219269 1.89089544]
    Iter 139; alpha, beta =  [2.91463418 1.89195183]
    Iter 140; alpha, beta =  [2.91702672 1.89298625]
    Iter 141; alpha, beta =  [2.91937128 1.89399916]
    Iter 142; alpha, beta =  [2.92166884 1.894991  ]
    Iter 143; alpha, beta =  [2.92392032 1.89596223]
    Iter 144; alpha, beta =  [2.92612666 1.89691325]
    Iter 145; alpha, beta =  [2.92828876 1.8978445 ]
    Iter 146; alpha, beta =  [2.9304075  1.89875638]
    Iter 147; alpha, beta =  [2.93248377 1.89964931]
    Iter 148; alpha, beta =  [2.9345184  1.90052366]
    Iter 149; alpha, beta =  [2.93651223 1.90137983]
    Iter 150; alpha, beta =  [2.93846609 1.90221819]
    Iter 151; alpha, beta =  [2.94038077 1.90303912]
    Iter 152; alpha, beta =  [2.94225706 1.90384298]
    Iter 153; alpha, beta =  [2.94409573 1.90463012]
    Iter 154; alpha, beta =  [2.94589753 1.90540089]
    Iter 155; alpha, beta =  [2.94766321 1.90615564]
    Iter 156; alpha, beta =  [2.94939348 1.90689468]
    Iter 157; alpha, beta =  [2.95108906 1.90761836]
    Iter 158; alpha, beta =  [2.95275064 1.90832698]
    Iter 159; alpha, beta =  [2.9543789  1.90902087]
    Iter 160; alpha, beta =  [2.95597452 1.90970032]
    Iter 161; alpha, beta =  [2.95753814 1.91036564]
    Iter 162; alpha, beta =  [2.95907042 1.91101713]
    Iter 163; alpha, beta =  [2.96057197 1.91165507]
    Iter 164; alpha, beta =  [2.96204341 1.91227973]
    Iter 165; alpha, beta =  [2.96348535 1.91289141]
    Iter 166; alpha, beta =  [2.96489838 1.91349036]
    Iter 167; alpha, beta =  [2.96628308 1.91407686]
    Iter 168; alpha, beta =  [2.96764001 1.91465115]
    Iter 169; alpha, beta =  [2.96896974 1.9152135 ]
    Iter 170; alpha, beta =  [2.9702728  1.91576416]
    Iter 171; alpha, beta =  [2.97154974 1.91630336]
    Iter 172; alpha, beta =  [2.97280108 1.91683134]
    Iter 173; alpha, beta =  [2.97402732 1.91734834]
    Iter 174; alpha, beta =  [2.97522898 1.91785459]
    Iter 175; alpha, beta =  [2.97640655 1.91835031]
    Iter 176; alpha, beta =  [2.97756051 1.91883571]
    Iter 177; alpha, beta =  [2.97869133 1.91931102]
    Iter 178; alpha, beta =  [2.97979948 1.91977644]
    Iter 179; alpha, beta =  [2.98088541 1.92023218]
    Iter 180; alpha, beta =  [2.98194956 1.92067844]
    Iter 181; alpha, beta =  [2.98299238 1.92111542]
    Iter 182; alpha, beta =  [2.9840143 1.9215433]
    Iter 183; alpha, beta =  [2.98501572 1.92196229]
    Iter 184; alpha, beta =  [2.98599706 1.92237255]
    Iter 185; alpha, beta =  [2.98695873 1.92277429]
    Iter 186; alpha, beta =  [2.98790112 1.92316766]
    Iter 187; alpha, beta =  [2.98882462 1.92355285]
    Iter 188; alpha, beta =  [2.98972959 1.92393003]
    Iter 189; alpha, beta =  [2.99061643 1.92429936]
    Iter 190; alpha, beta =  [2.99148548 1.92466101]
    Iter 191; alpha, beta =  [2.99233711 1.92501514]
    Iter 192; alpha, beta =  [2.99317167 1.92536189]
    Iter 193; alpha, beta =  [2.99398949 1.92570143]
    Iter 194; alpha, beta =  [2.99479091 1.92603391]
    Iter 195; alpha, beta =  [2.99557627 1.92635947]
    Iter 196; alpha, beta =  [2.99634588 1.92667826]
    Iter 197; alpha, beta =  [2.99710007 1.92699041]
    Iter 198; alpha, beta =  [2.99783913 1.92729607]
    Iter 199; alpha, beta =  [2.99856337 1.92759537]
    Iter 200; alpha, beta =  [2.99927309 1.92788844]
    Iter 201; alpha, beta =  [2.99996859 1.92817542]
    Iter 202; alpha, beta =  [3.00065014 1.92845642]
    Iter 203; alpha, beta =  [3.00131803 1.92873158]
    Iter 204; alpha, beta =  [3.00197252 1.92900101]
    Iter 205; alpha, beta =  [3.00261389 1.92926483]
    Iter 206; alpha, beta =  [3.00324241 1.92952317]
    Iter 207; alpha, beta =  [3.00385832 1.92977613]
    Iter 208; alpha, beta =  [3.00446189 1.93002382]
    Iter 209; alpha, beta =  [3.00505335 1.93026637]
    Iter 210; alpha, beta =  [3.00563296 1.93050386]
    Iter 211; alpha, beta =  [3.00620095 1.93073641]
    Iter 212; alpha, beta =  [3.00675755 1.93096413]
    Iter 213; alpha, beta =  [3.00730299 1.9311871 ]
    Iter 214; alpha, beta =  [3.00783749 1.93140544]
    Iter 215; alpha, beta =  [3.00836128 1.93161923]
    Iter 216; alpha, beta =  [3.00887457 1.93182857]
    Iter 217; alpha, beta =  [3.00937756 1.93203356]
    Iter 218; alpha, beta =  [3.00987048 1.93223427]
    Iter 219; alpha, beta =  [3.01035351 1.93243082]
    Iter 220; alpha, beta =  [3.01082686 1.93262327]
    Iter 221; alpha, beta =  [3.01129071 1.93281172]
    Iter 222; alpha, beta =  [3.01174527 1.93299624]
    Iter 223; alpha, beta =  [3.01219071 1.93317693]
    Iter 224; alpha, beta =  [3.01262723 1.93335385]
    Iter 225; alpha, beta =  [3.01305499 1.93352709]
    Iter 226; alpha, beta =  [3.01347418 1.93369673]
    Iter 227; alpha, beta =  [3.01388496 1.93386284]
    Iter 228; alpha, beta =  [3.01428751 1.93402549]
    Iter 229; alpha, beta =  [3.01468199 1.93418475]
    Iter 230; alpha, beta =  [3.01506856 1.9343407 ]
    Iter 231; alpha, beta =  [3.01544738 1.9344934 ]
    Iter 232; alpha, beta =  [3.0158186  1.93464293]
    Iter 233; alpha, beta =  [3.01618238 1.93478934]
    Iter 234; alpha, beta =  [3.01653887 1.9349327 ]
    Iter 235; alpha, beta =  [3.01688822 1.93507308]
    Iter 236; alpha, beta =  [3.01723055 1.93521054]
    Iter 237; alpha, beta =  [3.01756603 1.93534514]
    Iter 238; alpha, beta =  [3.01789478 1.93547694]
    Iter 239; alpha, beta =  [3.01821694 1.93560599]
    Iter 240; alpha, beta =  [3.01853264 1.93573236]
    Iter 241; alpha, beta =  [3.01884201 1.93585609]
    Iter 242; alpha, beta =  [3.01914518 1.93597725]
    Iter 243; alpha, beta =  [3.01944228 1.93609589]
    Iter 244; alpha, beta =  [3.01973341 1.93621206]
    Iter 245; alpha, beta =  [3.02001871 1.93632581]
    Iter 246; alpha, beta =  [3.02029829 1.93643719]
    Iter 247; alpha, beta =  [3.02057227 1.93654625]
    Iter 248; alpha, beta =  [3.02084075 1.93665305]
    Iter 249; alpha, beta =  [3.02110385 1.93675762]
    Iter 250; alpha, beta =  [3.02136168 1.93686001]
    Iter 251; alpha, beta =  [3.02161434 1.93696027]
    Iter 252; alpha, beta =  [3.02186193 1.93705844]
    Iter 253; alpha, beta =  [3.02210456 1.93715457]
    Iter 254; alpha, beta =  [3.02234233 1.9372487 ]
    Iter 255; alpha, beta =  [3.02257532 1.93734087]
    Iter 256; alpha, beta =  [3.02280365 1.93743112]
    Iter 257; alpha, beta =  [3.0230274  1.93751949]
    Iter 258; alpha, beta =  [3.02324667 1.93760603]
    Iter 259; alpha, beta =  [3.02346154 1.93769075]
    Iter 260; alpha, beta =  [3.0236721  1.93777372]
    Iter 261; alpha, beta =  [3.02387844 1.93785496]
    Iter 262; alpha, beta =  [3.02408064 1.93793451]
    Iter 263; alpha, beta =  [3.0242788 1.9380124]
    Iter 264; alpha, beta =  [3.02447297 1.93808866]
    Iter 265; alpha, beta =  [3.02466326 1.93816335]
    Iter 266; alpha, beta =  [3.02484973 1.93823647]
    Iter 267; alpha, beta =  [3.02503247 1.93830807]
    Iter 268; alpha, beta =  [3.02521154 1.93837819]
    Iter 269; alpha, beta =  [3.02538702 1.93844684]
    Iter 270; alpha, beta =  [3.02555898 1.93851406]
    Iter 271; alpha, beta =  [3.02572749 1.93857988]
    Iter 272; alpha, beta =  [3.02589263 1.93864434]
    Iter 273; alpha, beta =  [3.02605446 1.93870745]
    Iter 274; alpha, beta =  [3.02621304 1.93876924]
    Iter 275; alpha, beta =  [3.02636845 1.93882975]
    Iter 276; alpha, beta =  [3.02652073 1.938889  ]
    Iter 277; alpha, beta =  [3.02666997 1.93894702]
    Iter 278; alpha, beta =  [3.02681621 1.93900382]
    Iter 279; alpha, beta =  [3.02695953 1.93905945]
    Iter 280; alpha, beta =  [3.02709997 1.93911391]
    Iter 281; alpha, beta =  [3.02723759 1.93916724]
    Iter 282; alpha, beta =  [3.02737246 1.93921946]
    Iter 283; alpha, beta =  [3.02750462 1.9392706 ]
    Iter 284; alpha, beta =  [3.02763413 1.93932067]
    Iter 285; alpha, beta =  [3.02776105 1.93936969]
    Iter 286; alpha, beta =  [3.02788542 1.9394177 ]
    Iter 287; alpha, beta =  [3.0280073 1.9394647]
    Iter 288; alpha, beta =  [3.02812674 1.93951073]
    Iter 289; alpha, beta =  [3.02824378 1.93955579]
    Iter 290; alpha, beta =  [3.02835848 1.93959992]
    Iter 291; alpha, beta =  [3.02847087 1.93964313]
    Iter 292; alpha, beta =  [3.02858102 1.93968544]
    Iter 293; alpha, beta =  [3.02868895 1.93972687]
    Iter 294; alpha, beta =  [3.02879472 1.93976743]
    Iter 295; alpha, beta =  [3.02889838 1.93980715]
    Iter 296; alpha, beta =  [3.02899995 1.93984605]
    Iter 297; alpha, beta =  [3.02909949 1.93988413]
    Iter 298; alpha, beta =  [3.02919703 1.93992142]
    Iter 299; alpha, beta =  [3.02929262 1.93995793]
    Iter 300; alpha, beta =  [3.02938629 1.93999368]
    Iter 301; alpha, beta =  [3.02947808 1.94002869]
    Iter 302; alpha, beta =  [3.02956804 1.94006297]
    Iter 303; alpha, beta =  [3.02965619 1.94009653]
    Iter 304; alpha, beta =  [3.02974257 1.9401294 ]
    Iter 305; alpha, beta =  [3.02982722 1.94016158]
    Iter 306; alpha, beta =  [3.02991018 1.94019309]
    Iter 307; alpha, beta =  [3.02999147 1.94022394]
    Iter 308; alpha, beta =  [3.03007113 1.94025416]
    Iter 309; alpha, beta =  [3.0301492  1.94028374]
    Iter 310; alpha, beta =  [3.0302257 1.9403127]
    Iter 311; alpha, beta =  [3.03030067 1.94034106]
    Iter 312; alpha, beta =  [3.03037413 1.94036883]
    Iter 313; alpha, beta =  [3.03044612 1.94039603]
    Iter 314; alpha, beta =  [3.03051667 1.94042265]
    Iter 315; alpha, beta =  [3.03058581 1.94044872]
    Iter 316; alpha, beta =  [3.03065356 1.94047425]
    Iter 317; alpha, beta =  [3.03071995 1.94049925]
    Iter 318; alpha, beta =  [3.03078501 1.94052372]
    Iter 319; alpha, beta =  [3.03084877 1.94054769]
    Iter 320; alpha, beta =  [3.03091124 1.94057116]
    Iter 321; alpha, beta =  [3.03097247 1.94059413]
    Iter 322; alpha, beta =  [3.03103247 1.94061663]
    Iter 323; alpha, beta =  [3.03109127 1.94063866]
    Iter 324; alpha, beta =  [3.03114888 1.94066023]
    Iter 325; alpha, beta =  [3.03120535 1.94068135]
    Iter 326; alpha, beta =  [3.03126068 1.94070203]
    Iter 327; alpha, beta =  [3.0313149  1.94072228]
    Iter 328; alpha, beta =  [3.03136803 1.94074211]
    Iter 329; alpha, beta =  [3.0314201  1.94076153]
    Iter 330; alpha, beta =  [3.03147113 1.94078054]
    Iter 331; alpha, beta =  [3.03152113 1.94079915]
    Iter 332; alpha, beta =  [3.03157013 1.94081738]
    Iter 333; alpha, beta =  [3.03161815 1.94083523]
    Iter 334; alpha, beta =  [3.03166521 1.9408527 ]
    Iter 335; alpha, beta =  [3.03171132 1.94086981]
    Iter 336; alpha, beta =  [3.03175651 1.94088656]
    Iter 337; alpha, beta =  [3.0318008  1.94090297]
    Iter 338; alpha, beta =  [3.03184419 1.94091903]
    Iter 339; alpha, beta =  [3.03188672 1.94093476]
    Iter 340; alpha, beta =  [3.03192839 1.94095016]
    Iter 341; alpha, beta =  [3.03196923 1.94096524]
    Iter 342; alpha, beta =  [3.03200925 1.94098001]
    Iter 343; alpha, beta =  [3.03204847 1.94099446]
    Iter 344; alpha, beta =  [3.0320869  1.94100862]
    Iter 345; alpha, beta =  [3.03212456 1.94102248]
    Iter 346; alpha, beta =  [3.03216146 1.94103605]
    Iter 347; alpha, beta =  [3.03219763 1.94104934]
    Iter 348; alpha, beta =  [3.03223307 1.94106235]
    Iter 349; alpha, beta =  [3.0322678 1.9410751]
    Iter 350; alpha, beta =  [3.03230184 1.94108757]
    Iter 351; alpha, beta =  [3.03233519 1.94109979]
    Iter 352; alpha, beta =  [3.03236788 1.94111175]
    Iter 353; alpha, beta =  [3.03239991 1.94112346]
    Iter 354; alpha, beta =  [3.03243129 1.94113493]
    Iter 355; alpha, beta =  [3.03246205 1.94114616]
    Iter 356; alpha, beta =  [3.03249219 1.94115715]
    Iter 357; alpha, beta =  [3.03252173 1.94116792]
    Iter 358; alpha, beta =  [3.03255068 1.94117846]
    Iter 359; alpha, beta =  [3.03257904 1.94118878]
    Iter 360; alpha, beta =  [3.03260684 1.94119888]
    Iter 361; alpha, beta =  [3.03263408 1.94120878]
    Iter 362; alpha, beta =  [3.03266077 1.94121847]
    Iter 363; alpha, beta =  [3.03268693 1.94122795]
    Iter 364; alpha, beta =  [3.03271256 1.94123724]
    Iter 365; alpha, beta =  [3.03273769 1.94124634]
    Iter 366; alpha, beta =  [3.0327623  1.94125525]
    Iter 367; alpha, beta =  [3.03278643 1.94126397]
    Iter 368; alpha, beta =  [3.03281007 1.9412725 ]
    Iter 369; alpha, beta =  [3.03283323 1.94128087]
    Iter 370; alpha, beta =  [3.03285593 1.94128905]
    Iter 371; alpha, beta =  [3.03287818 1.94129707]
    Iter 372; alpha, beta =  [3.03289998 1.94130492]
    Iter 373; alpha, beta =  [3.03292135 1.9413126 ]
    Iter 374; alpha, beta =  [3.03294228 1.94132013]
    Iter 375; alpha, beta =  [3.0329628  1.94132749]
    Iter 376; alpha, beta =  [3.0329829  1.94133471]
    Iter 377; alpha, beta =  [3.03300261 1.94134177]
    Iter 378; alpha, beta =  [3.03302191 1.94134869]
    Iter 379; alpha, beta =  [3.03304083 1.94135546]
    Iter 380; alpha, beta =  [3.03305938 1.94136209]
    Iter 381; alpha, beta =  [3.03307755 1.94136858]
    Iter 382; alpha, beta =  [3.03309535 1.94137494]
    Iter 383; alpha, beta =  [3.0331128  1.94138116]
    Iter 384; alpha, beta =  [3.0331299  1.94138726]
    Iter 385; alpha, beta =  [3.03314665 1.94139323]
    Iter 386; alpha, beta =  [3.03316307 1.94139907]
    Iter 387; alpha, beta =  [3.03317917 1.94140479]
    Iter 388; alpha, beta =  [3.03319493 1.94141039]
    Iter 389; alpha, beta =  [3.03321039 1.94141588]
    Iter 390; alpha, beta =  [3.03322553 1.94142125]
    Iter 391; alpha, beta =  [3.03324037 1.94142651]

    array([3.03324037, 1.94142651])

Observe como alpha é o intercept e beta o slope da regressão com scipy

```python
ss.linregress(x, y)
```

    LinregressResult(slope=1.9416732636887266, intercept=3.033966007834007, rvalue=0.8897448608393281, pvalue=0.0, stderr=0.031531628347458034)

## Gradiente Ascendente

Vamos agora chegar no mesmo resultado usando Gradiente Ascendente. Em particular, ainda vamos nos manter em uma regressão linear. Porém, queremos maximizar a log-verossimilhança. Da aula anterior, nosso modelo é:

$$y_i = \beta x_i + \alpha + \epsilon_i $$
$$\epsilon_i = y_i - \beta x_i - \alpha $$
$$\epsilon_i \sim Normal(0, \sigma) $$

Partindo do mesmo chegamos em:

$$p(\epsilon_i\mid \sigma^2, \alpha, \beta) = \frac{1}{\sqrt {\sigma ^22\pi } }
e^{- ({y_i - \beta x_i - \alpha })^2 / 2\sigma^2} $$

Assumindo que cada erro é independente, a verossimilhança de todos os erros é:

$$p_{\Theta}(E) = \prod_{i} p_{\Theta}(\epsilon_i) \\
\log(p_{\Theta}(E)) = \mathcal{l}_{\Theta}(E) =  \sum_{i} \log(p_{\Theta}(\epsilon_i))$$

O log da mesma é:

$$\mathcal{l}_{\Theta}(E) =  -n \log(\sqrt{2\pi}) - n{\log(\sigma) } - { \sum^n_{i=1} ({y_i - \beta x_i - \alpha })^2 \over 2\sigma^2} $$

A derivada em $\sigma$ é:

$${d \mathcal{l}_{\Theta} \over d\sigma} =  -{n\over \sigma} + { 2 \sum^n_{i=1} ({y_i - \beta x_i - \alpha })^2 \over 2\sigma^3}$$

A derivada em $\alpha$ e $\beta$ é a mesma de antes, apenas invertemos o sinal:

$$\frac{d\mathcal{l}_{\Theta}}{d\alpha} = -n^{-1} \sum_i 2 (-y_i + \beta x_i + \alpha)$$

$$\frac{d\mathcal{l}_{\Theta}}{d\beta} = -n^{-1} \sum_i 2 (-y_i + \beta x_i + \alpha) x_i$$

Em outras palavras, vamos chegar no mesmo resultado.

```python
def derivadas_vero(parametros, x, y):
    alpha, beta = parametros
    da = -1 * ((-2 * y) + (2 * beta * x) + (2 * alpha))
    db = -1 * ((-2 * y * x) + (2 * beta * (x ** 2)) + (2 * alpha * x))
    return np.array([da.mean(), db.mean()])
```

```python
def ga(parametros, d_fun, x, y, lambda_=0.01):
    parametros = parametros.copy()
    print('Iter {}; alpha, beta = '.format(-1), parametros)
    for i in range(1000):
        grad = d_fun(parametros, x, y)
        parametros_novos = parametros + lambda_ * grad
        if np.abs(parametros - parametros_novos).mean() <= 0.00001:
            break
        parametros = parametros_novos
        print('Iter {}; alpha, beta = '.format(i), parametros)
    return parametros
```

```python
parametros_iniciais = np.array([0.1, 0.1])
parametros = ga(parametros_iniciais, derivadas_vero, x, y)
```

    Iter -1; alpha, beta =  [0.1 0.1]
    Iter 0; alpha, beta =  [0.15889788 0.13813834]
    Iter 1; alpha, beta =  [0.21661329 0.17548712]
    Iter 2; alpha, beta =  [0.27316995 0.21206267]
    Iter 3; alpha, beta =  [0.32859113 0.24788099]
    Iter 4; alpha, beta =  [0.38289965 0.28295777]
    Iter 5; alpha, beta =  [0.43611782 0.31730834]
    Iter 6; alpha, beta =  [0.48826756 0.35094774]
    Iter 7; alpha, beta =  [0.53937032 0.38389069]
    Iter 8; alpha, beta =  [0.5894471  0.41615161]
    Iter 9; alpha, beta =  [0.63851853 0.4477446 ]
    Iter 10; alpha, beta =  [0.68660477 0.4786835 ]
    Iter 11; alpha, beta =  [0.73372562 0.50898185]
    Iter 12; alpha, beta =  [0.77990046 0.53865289]
    Iter 13; alpha, beta =  [0.82514827 0.56770962]
    Iter 14; alpha, beta =  [0.86948769 0.59616476]
    Iter 15; alpha, beta =  [0.91293693 0.62403074]
    Iter 16; alpha, beta =  [0.95551389 0.65131978]
    Iter 17; alpha, beta =  [0.99723607 0.67804381]
    Iter 18; alpha, beta =  [1.03812063 0.70421452]
    Iter 19; alpha, beta =  [1.0781844  0.72984337]
    Iter 20; alpha, beta =  [1.11744385 0.75494158]
    Iter 21; alpha, beta =  [1.15591513 0.77952012]
    Iter 22; alpha, beta =  [1.19361407 0.80358976]
    Iter 23; alpha, beta =  [1.23055617 0.82716103]
    Iter 24; alpha, beta =  [1.26675663 0.85024425]
    Iter 25; alpha, beta =  [1.30223035 0.87284952]
    Iter 26; alpha, beta =  [1.33699191 0.89498672]
    Iter 27; alpha, beta =  [1.37105561 0.91666556]
    Iter 28; alpha, beta =  [1.40443546 0.93789552]
    Iter 29; alpha, beta =  [1.4371452  0.95868589]
    Iter 30; alpha, beta =  [1.46919827 0.97904577]
    Iter 31; alpha, beta =  [1.50060787 0.99898407]
    Iter 32; alpha, beta =  [1.53138691 1.01850952]
    Iter 33; alpha, beta =  [1.56154805 1.03763067]
    Iter 34; alpha, beta =  [1.5911037  1.05635588]
    Iter 35; alpha, beta =  [1.62006601 1.07469335]
    Iter 36; alpha, beta =  [1.6484469  1.09265111]
    Iter 37; alpha, beta =  [1.67625804 1.11023702]
    Iter 38; alpha, beta =  [1.70351087 1.12745878]
    Iter 39; alpha, beta =  [1.73021661 1.14432392]
    Iter 40; alpha, beta =  [1.75638622 1.16083982]
    Iter 41; alpha, beta =  [1.78203048 1.17701373]
    Iter 42; alpha, beta =  [1.80715994 1.19285271]
    Iter 43; alpha, beta =  [1.83178493 1.2083637 ]
    Iter 44; alpha, beta =  [1.85591558 1.22355349]
    Iter 45; alpha, beta =  [1.87956181 1.23842873]
    Iter 46; alpha, beta =  [1.90273336 1.25299593]
    Iter 47; alpha, beta =  [1.92543974 1.26726147]
    Iter 48; alpha, beta =  [1.9476903 1.2812316]
    Iter 49; alpha, beta =  [1.9694942  1.29491242]
    Iter 50; alpha, beta =  [1.99086039 1.30830993]
    Iter 51; alpha, beta =  [2.01179767 1.32143   ]
    Iter 52; alpha, beta =  [2.03231464 1.33427837]
    Iter 53; alpha, beta =  [2.05241975 1.34686066]
    Iter 54; alpha, beta =  [2.07212127 1.35918238]
    Iter 55; alpha, beta =  [2.09142729 1.37124893]
    Iter 56; alpha, beta =  [2.11034576 1.38306559]
    Iter 57; alpha, beta =  [2.12888446 1.39463754]
    Iter 58; alpha, beta =  [2.14705101 1.40596983]
    Iter 59; alpha, beta =  [2.16485289 1.41706743]
    Iter 60; alpha, beta =  [2.18229741 1.42793521]
    Iter 61; alpha, beta =  [2.19939175 1.43857791]
    Iter 62; alpha, beta =  [2.21614294 1.4490002 ]
    Iter 63; alpha, beta =  [2.23255787 1.45920664]
    Iter 64; alpha, beta =  [2.24864329 1.46920171]
    Iter 65; alpha, beta =  [2.26440582 1.47898977]
    Iter 66; alpha, beta =  [2.27985193 1.48857512]
    Iter 67; alpha, beta =  [2.29498799 1.49796195]
    Iter 68; alpha, beta =  [2.30982021 1.50715436]
    Iter 69; alpha, beta =  [2.32435469 1.5161564 ]
    Iter 70; alpha, beta =  [2.33859742 1.52497199]
    Iter 71; alpha, beta =  [2.35255424 1.533605  ]
    Iter 72; alpha, beta =  [2.3662309  1.54205921]
    Iter 73; alpha, beta =  [2.37963303 1.55033832]
    Iter 74; alpha, beta =  [2.39276613 1.55844596]
    Iter 75; alpha, beta =  [2.40563561 1.56638568]
    Iter 76; alpha, beta =  [2.41824676 1.57416094]
    Iter 77; alpha, beta =  [2.43060476 1.58177517]
    Iter 78; alpha, beta =  [2.44271469 1.58923169]
    Iter 79; alpha, beta =  [2.45458155 1.59653377]
    Iter 80; alpha, beta =  [2.4662102  1.60368461]
    Iter 81; alpha, beta =  [2.47760542 1.61068733]
    Iter 82; alpha, beta =  [2.48877192 1.61754502]
    Iter 83; alpha, beta =  [2.49971427 1.62426066]
    Iter 84; alpha, beta =  [2.51043697 1.6308372 ]
    Iter 85; alpha, beta =  [2.52094444 1.63727752]
    Iter 86; alpha, beta =  [2.531241   1.64358444]
    Iter 87; alpha, beta =  [2.54133087 1.64976072]
    Iter 88; alpha, beta =  [2.55121822 1.65580908]
    Iter 89; alpha, beta =  [2.5609071  1.66173215]
    Iter 90; alpha, beta =  [2.5704015  1.66753253]
    Iter 91; alpha, beta =  [2.57970533 1.67321277]
    Iter 92; alpha, beta =  [2.5888224  1.67877534]
    Iter 93; alpha, beta =  [2.59775647 1.68422269]
    Iter 94; alpha, beta =  [2.60651122 1.68955721]
    Iter 95; alpha, beta =  [2.61509023 1.69478122]
    Iter 96; alpha, beta =  [2.62349705 1.69989703]
    Iter 97; alpha, beta =  [2.63173512 1.70490686]
    Iter 98; alpha, beta =  [2.63980784 1.70981292]
    Iter 99; alpha, beta =  [2.64771852 1.71461734]
    Iter 100; alpha, beta =  [2.65547041 1.71932225]
    Iter 101; alpha, beta =  [2.66306671 1.72392969]
    Iter 102; alpha, beta =  [2.67051054 1.72844169]
    Iter 103; alpha, beta =  [2.67780496 1.73286022]
    Iter 104; alpha, beta =  [2.68495296 1.73718722]
    Iter 105; alpha, beta =  [2.69195749 1.74142458]
    Iter 106; alpha, beta =  [2.69882142 1.74557417]
    Iter 107; alpha, beta =  [2.70554759 1.74963779]
    Iter 108; alpha, beta =  [2.71213874 1.75361723]
    Iter 109; alpha, beta =  [2.71859761 1.75751423]
    Iter 110; alpha, beta =  [2.72492683 1.76133051]
    Iter 111; alpha, beta =  [2.73112902 1.76506772]
    Iter 112; alpha, beta =  [2.73720672 1.76872751]
    Iter 113; alpha, beta =  [2.74316243 1.77231148]
    Iter 114; alpha, beta =  [2.7489986  1.77582121]
    Iter 115; alpha, beta =  [2.75471763 1.77925822]
    Iter 116; alpha, beta =  [2.76032187 1.78262403]
    Iter 117; alpha, beta =  [2.76581363 1.78592011]
    Iter 118; alpha, beta =  [2.77119516 1.78914791]
    Iter 119; alpha, beta =  [2.77646868 1.79230883]
    Iter 120; alpha, beta =  [2.78163635 1.79540427]
    Iter 121; alpha, beta =  [2.7867003  1.79843557]
    Iter 122; alpha, beta =  [2.79166262 1.80140408]
    Iter 123; alpha, beta =  [2.79652533 1.80431108]
    Iter 124; alpha, beta =  [2.80129045 1.80715786]
    Iter 125; alpha, beta =  [2.80595992 1.80994566]
    Iter 126; alpha, beta =  [2.81053568 1.8126757 ]
    Iter 127; alpha, beta =  [2.81501959 1.81534917]
    Iter 128; alpha, beta =  [2.81941351 1.81796726]
    Iter 129; alpha, beta =  [2.82371924 1.8205311 ]
    Iter 130; alpha, beta =  [2.82793856 1.82304183]
    Iter 131; alpha, beta =  [2.83207318 1.82550053]
    Iter 132; alpha, beta =  [2.83612483 1.82790829]
    Iter 133; alpha, beta =  [2.84009515 1.83026617]
    Iter 134; alpha, beta =  [2.84398579 1.83257519]
    Iter 135; alpha, beta =  [2.84779834 1.83483637]
    Iter 136; alpha, beta =  [2.85153438 1.8370507 ]
    Iter 137; alpha, beta =  [2.85519542 1.83921915]
    Iter 138; alpha, beta =  [2.858783   1.84134267]
    Iter 139; alpha, beta =  [2.86229856 1.84342219]
    Iter 140; alpha, beta =  [2.86574357 1.84545862]
    Iter 141; alpha, beta =  [2.86911944 1.84745286]
    Iter 142; alpha, beta =  [2.87242755 1.84940577]
    Iter 143; alpha, beta =  [2.87566927 1.85131822]
    Iter 144; alpha, beta =  [2.87884593 1.85319105]
    Iter 145; alpha, beta =  [2.88195883 1.85502506]
    Iter 146; alpha, beta =  [2.88500926 1.85682108]
    Iter 147; alpha, beta =  [2.88799846 1.85857988]
    Iter 148; alpha, beta =  [2.89092768 1.86030223]
    Iter 149; alpha, beta =  [2.8937981 1.8619889]
    Iter 150; alpha, beta =  [2.89661091 1.86364061]
    Iter 151; alpha, beta =  [2.89936728 1.8652581 ]
    Iter 152; alpha, beta =  [2.90206832 1.86684207]
    Iter 153; alpha, beta =  [2.90471515 1.86839322]
    Iter 154; alpha, beta =  [2.90730887 1.86991223]
    Iter 155; alpha, beta =  [2.90985053 1.87139976]
    Iter 156; alpha, beta =  [2.91234118 1.87285646]
    Iter 157; alpha, beta =  [2.91478184 1.87428298]
    Iter 158; alpha, beta =  [2.91717352 1.87567994]
    Iter 159; alpha, beta =  [2.9195172  1.87704795]
    Iter 160; alpha, beta =  [2.92181385 1.87838761]
    Iter 161; alpha, beta =  [2.9240644 1.8796995]
    Iter 162; alpha, beta =  [2.92626979 1.88098421]
    Iter 163; alpha, beta =  [2.92843092 1.8822423 ]
    Iter 164; alpha, beta =  [2.93054867 1.88347432]
    Iter 165; alpha, beta =  [2.93262392 1.8846808 ]
    Iter 166; alpha, beta =  [2.93465753 1.88586228]
    Iter 167; alpha, beta =  [2.93665032 1.88701928]
    Iter 168; alpha, beta =  [2.93860312 1.8881523 ]
    Iter 169; alpha, beta =  [2.94051673 1.88926184]
    Iter 170; alpha, beta =  [2.94239194 1.89034838]
    Iter 171; alpha, beta =  [2.94422951 1.89141241]
    Iter 172; alpha, beta =  [2.94603021 1.89245439]
    Iter 173; alpha, beta =  [2.94779476 1.89347477]
    Iter 174; alpha, beta =  [2.94952391 1.894474  ]
    Iter 175; alpha, beta =  [2.95121835 1.89545253]
    Iter 176; alpha, beta =  [2.95287879 1.89641077]
    Iter 177; alpha, beta =  [2.95450591 1.89734916]
    Iter 178; alpha, beta =  [2.95610037 1.89826809]
    Iter 179; alpha, beta =  [2.95766283 1.89916799]
    Iter 180; alpha, beta =  [2.95919394 1.90004923]
    Iter 181; alpha, beta =  [2.96069432 1.9009122 ]
    Iter 182; alpha, beta =  [2.96216459 1.90175729]
    Iter 183; alpha, beta =  [2.96360536 1.90258487]
    Iter 184; alpha, beta =  [2.96501721 1.90339529]
    Iter 185; alpha, beta =  [2.96640073 1.90418892]
    Iter 186; alpha, beta =  [2.96775648 1.9049661 ]
    Iter 187; alpha, beta =  [2.96908503 1.90572716]
    Iter 188; alpha, beta =  [2.97038691 1.90647246]
    Iter 189; alpha, beta =  [2.97166267 1.90720231]
    Iter 190; alpha, beta =  [2.97291283 1.90791703]
    Iter 191; alpha, beta =  [2.9741379  1.90861693]
    Iter 192; alpha, beta =  [2.97533839 1.90930233]
    Iter 193; alpha, beta =  [2.97651478 1.90997352]
    Iter 194; alpha, beta =  [2.97766757 1.9106308 ]
    Iter 195; alpha, beta =  [2.97879722 1.91127446]
    Iter 196; alpha, beta =  [2.9799042  1.91190477]
    Iter 197; alpha, beta =  [2.98098897 1.91252202]
    Iter 198; alpha, beta =  [2.98205197 1.91312648]
    Iter 199; alpha, beta =  [2.98309364 1.9137184 ]
    Iter 200; alpha, beta =  [2.98411441 1.91429806]
    Iter 201; alpha, beta =  [2.98511469 1.9148657 ]
    Iter 202; alpha, beta =  [2.98609489 1.91542157]
    Iter 203; alpha, beta =  [2.98705543 1.91596593]
    Iter 204; alpha, beta =  [2.98799669 1.91649899]
    Iter 205; alpha, beta =  [2.98891907 1.91702101]
    Iter 206; alpha, beta =  [2.98982293 1.91753221]
    Iter 207; alpha, beta =  [2.99070866 1.91803281]
    Iter 208; alpha, beta =  [2.99157661 1.91852303]
    Iter 209; alpha, beta =  [2.99242715 1.91900309]
    Iter 210; alpha, beta =  [2.99326062 1.9194732 ]
    Iter 211; alpha, beta =  [2.99407736 1.91993357]
    Iter 212; alpha, beta =  [2.99487771 1.92038439]
    Iter 213; alpha, beta =  [2.995662   1.92082586]
    Iter 214; alpha, beta =  [2.99643056 1.92125819]
    Iter 215; alpha, beta =  [2.99718369 1.92168155]
    Iter 216; alpha, beta =  [2.99792171 1.92209613]
    Iter 217; alpha, beta =  [2.99864492 1.92250212]
    Iter 218; alpha, beta =  [2.99935361 1.92289969]
    Iter 219; alpha, beta =  [3.00004809 1.92328903]
    Iter 220; alpha, beta =  [3.00072863 1.92367029]
    Iter 221; alpha, beta =  [3.00139551 1.92404364]
    Iter 222; alpha, beta =  [3.00204902 1.92440926]
    Iter 223; alpha, beta =  [3.00268941 1.92476729]
    Iter 224; alpha, beta =  [3.00331694 1.92511791]
    Iter 225; alpha, beta =  [3.00393189 1.92546125]
    Iter 226; alpha, beta =  [3.0045345  1.92579748]
    Iter 227; alpha, beta =  [3.00512501 1.92612673]
    Iter 228; alpha, beta =  [3.00570368 1.92644916]
    Iter 229; alpha, beta =  [3.00627073 1.92676491]
    Iter 230; alpha, beta =  [3.0068264 1.9270741]
    Iter 231; alpha, beta =  [3.00737093 1.92737689]
    Iter 232; alpha, beta =  [3.00790453 1.9276734 ]
    Iter 233; alpha, beta =  [3.00842742 1.92796377]
    Iter 234; alpha, beta =  [3.00893982 1.92824811]
    Iter 235; alpha, beta =  [3.00944193 1.92852655]
    Iter 236; alpha, beta =  [3.00993398 1.92879923]
    Iter 237; alpha, beta =  [3.01041614 1.92906625]
    Iter 238; alpha, beta =  [3.01088864 1.92932773]
    Iter 239; alpha, beta =  [3.01135165 1.9295838 ]
    Iter 240; alpha, beta =  [3.01180537 1.92983455]
    Iter 241; alpha, beta =  [3.01224999 1.9300801 ]
    Iter 242; alpha, beta =  [3.01268569 1.93032057]
    Iter 243; alpha, beta =  [3.01311264 1.93055604]
    Iter 244; alpha, beta =  [3.01353103 1.93078664]
    Iter 245; alpha, beta =  [3.01394102 1.93101245]
    Iter 246; alpha, beta =  [3.01434278 1.93123358]
    Iter 247; alpha, beta =  [3.01473649 1.93145013]
    Iter 248; alpha, beta =  [3.01512229 1.93166218]
    Iter 249; alpha, beta =  [3.01550035 1.93186984]
    Iter 250; alpha, beta =  [3.01587083 1.93207319]
    Iter 251; alpha, beta =  [3.01623387 1.93227233]
    Iter 252; alpha, beta =  [3.01658963 1.93246733]
    Iter 253; alpha, beta =  [3.01693825 1.9326583 ]
    Iter 254; alpha, beta =  [3.01727988 1.9328453 ]
    Iter 255; alpha, beta =  [3.01761465 1.93302843]
    Iter 256; alpha, beta =  [3.0179427  1.93320775]
    Iter 257; alpha, beta =  [3.01826417 1.93338336]
    Iter 258; alpha, beta =  [3.01857919 1.93355533]
    Iter 259; alpha, beta =  [3.01888789 1.93372373]
    Iter 260; alpha, beta =  [3.0191904  1.93388864]
    Iter 261; alpha, beta =  [3.01948683 1.93405013]
    Iter 262; alpha, beta =  [3.01977732 1.93420827]
    Iter 263; alpha, beta =  [3.02006198 1.93436313]
    Iter 264; alpha, beta =  [3.02034093 1.93451478]
    Iter 265; alpha, beta =  [3.02061428 1.93466329]
    Iter 266; alpha, beta =  [3.02088215 1.93480871]
    Iter 267; alpha, beta =  [3.02114464 1.93495112]
    Iter 268; alpha, beta =  [3.02140186 1.93509058]
    Iter 269; alpha, beta =  [3.02165393 1.93522714]
    Iter 270; alpha, beta =  [3.02190093 1.93536088]
    Iter 271; alpha, beta =  [3.02214298 1.93549184]
    Iter 272; alpha, beta =  [3.02238018 1.93562008]
    Iter 273; alpha, beta =  [3.02261261 1.93574566]
    Iter 274; alpha, beta =  [3.02284038 1.93586864]
    Iter 275; alpha, beta =  [3.02306359 1.93598907]
    Iter 276; alpha, beta =  [3.02328231 1.936107  ]
    Iter 277; alpha, beta =  [3.02349664 1.93622248]
    Iter 278; alpha, beta =  [3.02370668 1.93633557]
    Iter 279; alpha, beta =  [3.0239125  1.93644632]
    Iter 280; alpha, beta =  [3.02411419 1.93655476]
    Iter 281; alpha, beta =  [3.02431183 1.93666096]
    Iter 282; alpha, beta =  [3.02450551 1.93676496]
    Iter 283; alpha, beta =  [3.0246953 1.9368668]
    Iter 284; alpha, beta =  [3.02488129 1.93696652]
    Iter 285; alpha, beta =  [3.02506354 1.93706418]
    Iter 286; alpha, beta =  [3.02524214 1.93715981]
    Iter 287; alpha, beta =  [3.02541715 1.93725346]
    Iter 288; alpha, beta =  [3.02558865 1.93734517]
    Iter 289; alpha, beta =  [3.02575671 1.93743497]
    Iter 290; alpha, beta =  [3.0259214  1.93752291]
    Iter 291; alpha, beta =  [3.02608279 1.93760903]
    Iter 292; alpha, beta =  [3.02624093 1.93769336]
    Iter 293; alpha, beta =  [3.02639591 1.93777595]
    Iter 294; alpha, beta =  [3.02654777 1.93785682]
    Iter 295; alpha, beta =  [3.02669659 1.93793601]
    Iter 296; alpha, beta =  [3.02684242 1.93801356]
    Iter 297; alpha, beta =  [3.02698533 1.9380895 ]
    Iter 298; alpha, beta =  [3.02712537 1.93816386]
    Iter 299; alpha, beta =  [3.02726259 1.93823669]
    Iter 300; alpha, beta =  [3.02739707 1.938308  ]
    Iter 301; alpha, beta =  [3.02752885 1.93837783]
    Iter 302; alpha, beta =  [3.02765798 1.93844621]
    Iter 303; alpha, beta =  [3.02778453 1.93851318]
    Iter 304; alpha, beta =  [3.02790853 1.93857876]
    Iter 305; alpha, beta =  [3.02803005 1.93864297]
    Iter 306; alpha, beta =  [3.02814913 1.93870586]
    Iter 307; alpha, beta =  [3.02826582 1.93876744]
    Iter 308; alpha, beta =  [3.02838017 1.93882774]
    Iter 309; alpha, beta =  [3.02849222 1.93888679]
    Iter 310; alpha, beta =  [3.02860203 1.93894462]
    Iter 311; alpha, beta =  [3.02870963 1.93900125]
    Iter 312; alpha, beta =  [3.02881507 1.9390567 ]
    Iter 313; alpha, beta =  [3.0289184 1.939111 ]
    Iter 314; alpha, beta =  [3.02901966 1.93916418]
    Iter 315; alpha, beta =  [3.02911888 1.93921625]
    Iter 316; alpha, beta =  [3.02921612 1.93926724]
    Iter 317; alpha, beta =  [3.0293114  1.93931717]
    Iter 318; alpha, beta =  [3.02940477 1.93936607]
    Iter 319; alpha, beta =  [3.02949627 1.93941395]
    Iter 320; alpha, beta =  [3.02958594 1.93946085]
    Iter 321; alpha, beta =  [3.0296738  1.93950676]
    Iter 322; alpha, beta =  [3.0297599  1.93955173]
    Iter 323; alpha, beta =  [3.02984427 1.93959576]
    Iter 324; alpha, beta =  [3.02992696 1.93963888]
    Iter 325; alpha, beta =  [3.03000798 1.9396811 ]
    Iter 326; alpha, beta =  [3.03008738 1.93972245]
    Iter 327; alpha, beta =  [3.03016518 1.93976294]
    Iter 328; alpha, beta =  [3.03024142 1.93980259]
    Iter 329; alpha, beta =  [3.03031614 1.93984142]
    Iter 330; alpha, beta =  [3.03038935 1.93987944]
    Iter 331; alpha, beta =  [3.0304611  1.93991667]
    Iter 332; alpha, beta =  [3.0305314  1.93995313]
    Iter 333; alpha, beta =  [3.0306003  1.93998884]
    Iter 334; alpha, beta =  [3.03066781 1.9400238 ]
    Iter 335; alpha, beta =  [3.03073397 1.94005804]
    Iter 336; alpha, beta =  [3.03079881 1.94009156]
    Iter 337; alpha, beta =  [3.03086234 1.94012439]
    Iter 338; alpha, beta =  [3.0309246  1.94015654]
    Iter 339; alpha, beta =  [3.0309856  1.94018803]
    Iter 340; alpha, beta =  [3.03104539 1.94021886]
    Iter 341; alpha, beta =  [3.03110397 1.94024905]
    Iter 342; alpha, beta =  [3.03116138 1.94027861]
    Iter 343; alpha, beta =  [3.03121764 1.94030756]
    Iter 344; alpha, beta =  [3.03127277 1.94033591]
    Iter 345; alpha, beta =  [3.03132679 1.94036367]
    Iter 346; alpha, beta =  [3.03137973 1.94039086]
    Iter 347; alpha, beta =  [3.03143161 1.94041748]
    Iter 348; alpha, beta =  [3.03148245 1.94044355]
    Iter 349; alpha, beta =  [3.03153226 1.94046908]
    Iter 350; alpha, beta =  [3.03158108 1.94049407]
    Iter 351; alpha, beta =  [3.03162892 1.94051855]
    Iter 352; alpha, beta =  [3.0316758  1.94054253]
    Iter 353; alpha, beta =  [3.03172174 1.940566  ]
    Iter 354; alpha, beta =  [3.03176675 1.94058899]
    Iter 355; alpha, beta =  [3.03181087 1.9406115 ]
    Iter 356; alpha, beta =  [3.0318541  1.94063354]
    Iter 357; alpha, beta =  [3.03189646 1.94065512]
    Iter 358; alpha, beta =  [3.03193797 1.94067626]
    Iter 359; alpha, beta =  [3.03197865 1.94069696]
    Iter 360; alpha, beta =  [3.03201851 1.94071723]
    Iter 361; alpha, beta =  [3.03205758 1.94073708]
    Iter 362; alpha, beta =  [3.03209586 1.94075651]
    Iter 363; alpha, beta =  [3.03213337 1.94077555]
    Iter 364; alpha, beta =  [3.03217013 1.94079419]
    Iter 365; alpha, beta =  [3.03220615 1.94081244]
    Iter 366; alpha, beta =  [3.03224145 1.94083031]
    Iter 367; alpha, beta =  [3.03227604 1.94084781]
    Iter 368; alpha, beta =  [3.03230994 1.94086495]
    Iter 369; alpha, beta =  [3.03234315 1.94088173]
    Iter 370; alpha, beta =  [3.03237571 1.94089817]
    Iter 371; alpha, beta =  [3.0324076  1.94091426]
    Iter 372; alpha, beta =  [3.03243886 1.94093002]
    Iter 373; alpha, beta =  [3.03246949 1.94094545]
    Iter 374; alpha, beta =  [3.03249951 1.94096056]
    Iter 375; alpha, beta =  [3.03252892 1.94097536]
    Iter 376; alpha, beta =  [3.03255775 1.94098985]
    Iter 377; alpha, beta =  [3.03258599 1.94100404]
    Iter 378; alpha, beta =  [3.03261367 1.94101794]
    Iter 379; alpha, beta =  [3.0326408  1.94103155]
    Iter 380; alpha, beta =  [3.03266738 1.94104487]
    Iter 381; alpha, beta =  [3.03269343 1.94105792]
    Iter 382; alpha, beta =  [3.03271895 1.9410707 ]
    Iter 383; alpha, beta =  [3.03274396 1.94108321]
    Iter 384; alpha, beta =  [3.03276847 1.94109546]
    Iter 385; alpha, beta =  [3.03279249 1.94110746]
    Iter 386; alpha, beta =  [3.03281603 1.94111921]
    Iter 387; alpha, beta =  [3.0328391  1.94113071]
    Iter 388; alpha, beta =  [3.0328617  1.94114198]
    Iter 389; alpha, beta =  [3.03288385 1.94115301]
    Iter 390; alpha, beta =  [3.03290555 1.94116382]
    Iter 391; alpha, beta =  [3.03292682 1.9411744 ]
    Iter 392; alpha, beta =  [3.03294767 1.94118476]
    Iter 393; alpha, beta =  [3.03296809 1.9411949 ]
    Iter 394; alpha, beta =  [3.03298811 1.94120484]
    Iter 395; alpha, beta =  [3.03300772 1.94121456]
    Iter 396; alpha, beta =  [3.03302694 1.94122409]
    Iter 397; alpha, beta =  [3.03304577 1.94123342]
    Iter 398; alpha, beta =  [3.03306423 1.94124255]
    Iter 399; alpha, beta =  [3.03308232 1.9412515 ]
    Iter 400; alpha, beta =  [3.03310004 1.94126026]
    Iter 401; alpha, beta =  [3.03311741 1.94126883]
    Iter 402; alpha, beta =  [3.03313443 1.94127723]
    Iter 403; alpha, beta =  [3.03315111 1.94128546]
    Iter 404; alpha, beta =  [3.03316745 1.94129351]
    Iter 405; alpha, beta =  [3.03318347 1.9413014 ]
    Iter 406; alpha, beta =  [3.03319916 1.94130912]
    Iter 407; alpha, beta =  [3.03321454 1.94131669]
    Iter 408; alpha, beta =  [3.03322962 1.94132409]
    Iter 409; alpha, beta =  [3.03324438 1.94133134]
    Iter 410; alpha, beta =  [3.03325886 1.94133845]
    Iter 411; alpha, beta =  [3.03327304 1.9413454 ]
    Iter 412; alpha, beta =  [3.03328694 1.94135221]
    Iter 413; alpha, beta =  [3.03330056 1.94135888]

```python
ss.linregress(x, y)
```

    LinregressResult(slope=1.9416732636887266, intercept=3.033966007834007, rvalue=0.8897448608393281, pvalue=0.0, stderr=0.031531628347458034)

## MLE

Abaixo tenho os plots para MLE. O material está nos slides.

```python
x = np.array([1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0])
x
```

    array([1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0])

```python
thetas = np.linspace(0.01, 0.99, 1000)
```

```python
l = []
for theta in thetas:
    model = (theta ** x) * ((1 - theta) ** (1-x))
    l.append(model.prod())
```

```python
plt.plot(thetas, l)
plt.xlabel(r'$\theta$')
plt.ylabel('Verossimilhança')
despine()
```

```python
ll = []
for theta in thetas:
    model = np.log((theta ** x) * ((1 - theta) ** (1-x)))
    ll.append(model.sum())
```

```python
plt.plot(thetas, ll)
plt.xlabel(r'$\theta$')
plt.ylabel('Log-Verossimilhança')
despine()
```

```python

```


```python
import matplotlib.pyplot as plt
from scipy import optimize
import numpy as np
import sympy as sm
import func
```

```python
sm.init_printing(use_unicode=True)
```

```python
#valgmuligheder:
# andelen af mulig arbejdstid som bruges på forskning:
sr = sm.symbols('s_Rt')
# I starten antages kapitalopspsparingsraten at være konstant
sk = sm.symbols('s_Kt')

# Ud fra de valg nogle udfald
# Først defineres variable
l = sm.symbols('L_{t}')
alpha = sm.symbols('alpha')
k =sm.symbols('K_{t}')
k1 = sm.symbols('K_{t+1}')
y = sm.symbols('Y_{t}')
c = sm.symbols('C')
ct = sm.symbols('C_{t}')
a = sm.symbols('A_{t}')
a1 = sm.symbols('A_{t+1}')
r = sm.symbols('r')
l1  = sm.symbols('L_{t+1}')
n = sm.symbols('n')

#Greek:
rho = sm.symbols('rho')
beta = sm.symbols('beta')
alpha = sm.symbols('alpha')
phi = sm.symbols('phi')
delta = sm.symbols('delta')
lambd = sm.symbols('lambda')
theta = sm.symbols('theta')

t = sm.symbols('t')

# Functions:
uc = sm.Function('u')(ct)
u = sm.Function('U')(c)
```

```python
prod = sm.Eq(y,a*k**alpha*(l*(1-sr))**(1-alpha))
```

```python
capakku = sm.Eq(k1,sk*y+(1-delta)*k)
```

```python
research = sm.Eq(a1,a+rho*a**phi*(l*sr)**lambd)
```

```python
popgrowth = sm.Eq(l1,(1+n)*l)
```

Model:

```python
prod
```

```python
capakku
```

```python
research
```

```python
popgrowth
```

also:

solving:

```python
sm.init_printing(use_unicode=False)
```

# Nummerisk

```python
# funktiions
def prod(a,k,l,sr,alpha):
    return k**alpha*(a*(1-sr)*l)**(1-alpha)
def capitalakku(a,k,l,sr,sk,alpha,delta):
    return prod(a,k,l,sr,alpha)*sk+(1-delta)*k
def research(a,sr,l,rho,phi):
    return a+rho*a**phi*(l*sr)**lambd
```

```python
def solowwalk(k0,a0, l0, n, sr, sk, alpha, delta, rho, phi, t):
    k_path = np.array([k0])
    a_path = np.array([a0])
    l_path = [l0*(1+n)**i for i in list(range(t))]
    y_path = np.array([prod(a_path[0],k_path[0],l_path[0],sr,alpha)])

    for i in range(1,t):
        k_plus = capitalakku(a_path[i-1],k_path[i-1],l_path[i-1],sr,sk,alpha,delta)
        a_plus = research(a_path[i-1],sr,l_path[i-1],rho,phi)
        y_plus = prod(a_plus,k_plus,l_path[i-1],sr,alpha)

        k_path = np.append(k_path, k_plus)
        a_path = np.append(a_path, a_plus)
        y_path = np.append(y_path, y_plus)

    k_tilde_path = k_path/(a_path*l_path)
    y_tilde_path = y_path/(a_path*l_path)
    return y_path, a_path, k_path, k_tilde_path, y_tilde_path
```

*** semi-endogen ***

```python
# parameters:
alpha = 1/3
delta = 0.05
rho = 0.9
phi = 0.1
lambd = 0.1
n=0.01

# fixed parameters
sr = 0.05
sk = 0.2

# starting values:
l0 = 100
k0 = 1000
a0 = 1000
```

```python
t = 1000
y_path, a_path, k_path, k_tilde_path, y_tilde_path = solowwalk(k0,a0, l0, n, sr, sk, alpha, delta, rho, phi, t)
```

```python
func.plot(range(t),k_tilde_path)
```

```python
gse = (1+n)**(lambd/(1-phi))-1
```

```python
# Predikteret k_tilde_ss:
(sk/(n+gse+delta+n*gse))**(1/(1-alpha))*(1-sr)
```

```python
k_tilde_path[-1]
```

*** fuldt endogen ***

```python
# parameters:
alpha = 1/3
delta = 0.05
rho = 0.05
phi = 1 #here
lambd = 0.1
n = 0.00 # here

# fixed parameters
sr = 0.005
sk = 0.2

# starting values:
l0 = 10
k0 = 150
a0 = 10
```

```python
t = 100
y_path, a_path, k_path, k_tilde_path,y_tilde_path = solowwalk(k0,a0, l0, n, sr, sk, alpha, delta, rho, phi, t)
```

```python
func.plot(range(t),y_tilde_path)
```

analytisk

```python
ge = rho*(sr*l0)**lambd
```

```python
k_tilde_path[-1]
```

```python
# k_tilde
(sk/(ge+delta))**(1/(1-alpha))*(1-sr)
```

```python
y_tilde_path[-1]
```

```python
# y_tilde
(sk/(ge+delta))**(alpha/(1-alpha))*(1-sr)
```


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *; x,h,t,y,z   = symbols("x h t y z", real=True)
Image(url= "https://i.imgur.com/bZFVJ2p.png")
```

```python
f  = 3*x**2 + 4*x + x*y -4
f
```

$\displaystyle 3 x^{2} + x y + 4 x - 4$

```python
solve(f,y)
```

    [{y: -3*x - 4 + 4/x}]

```python
diff(-3*x - 4 + 4/x).subs(x,4)
```

$\displaystyle - \frac{13}{4}$

```python
Image(url= "https://i.imgur.com/sAS2idS.png")
```


# SymPy를 사용한 함수 미분

## 데이터 분석에서 미분의 필요성

그다지 관련이 없어 보이지만 사실 데이터 분석에도 미분(differentiation)이 필요하다. 데이터 분석의 목표 중 하나는 확률 모형의 모수(parameter)나 상태 변수(state)를 추정(estimation)하는 작업이다. 이러한 작업은 근본적으로 함수의 최소점 혹은 최대점을 찾는 최적화(optimization) 작업이며 미분 혹은 편미분을 사용한 도함수를 필요로 한다. 따라서 함수 미분의 지식은 데이터 분석 및 머신 러닝의 각종 내부 구조를 이해하는데 필수적인다.

다행스러운 점은 데이터 분석자 입장에서 필요한 미분의 수준은 그다지 높지 않다는 점이다. 보통은 선형 다항식이나 지수함수의 편미분 정도의 개념만 알고 있으면 되고 대부분의 경우 최적화 라이브러리를 사용하거나 theano, tensorflow 등의 라이브러리에서 도함수나 미분값을 계산해 주기 때문에 실제로 도함수를 구할 일도 그다지 많지는 않다.

## 함수와 변수

프로그래밍을 익힌 사람에게는 변수(variable)와 함수(function)의 개념이 낯설지 않다. 변수란 실제 값을 대표하는 기호이며 함수는 이러한 변수를 기반으로 만들어진 수식으로 변수값이 어떤 수치로 결정되면 함수 값도 수식에 의해 결정된다.

변수는 보통 $x$, $y$, $z$ 등 알파벳 소문자로 표시하며 함수는 $f(x)$, $g(x,y)$ 와 같이 사용할 입력 변수를 괄호안에 넣어 표시한다. 함수의 결과를 다른 변수에 넣어 다시 사용하는 경우도 있다.

$$ y = f(x) $$
$$ z = g(y) = g(f(x)) $$

 파이썬의 함수는 이러한 함수의 개념을 그대로 구현한 것이다.

```python
def f(x):
    return 2*x

x = 10
y = f(x)
print(x, y)
```

    10 20

역함수(inverse function)는 함수의 입력과 출력을 반대로 한 것이며 다음과 같은 기호로 표시한다.

$$ y = f(x), \;\;\; \rightarrow \;\;\; x = f^{-1}(y) $$

## 예측 문제와 함수

예측(prediction) 문제는 독립 변수, 혹은 feature $x$를 입력으로 하여 원하는 종속 변수 혹은 targer $y$와 가능한한 비슷한 값을 만드는 함수 $f$를 찾는 문제라고 할 수 있다.

$$ y \approx \hat{y} = f(x) $$

## 데이터 분석에서 많이 사용되는 함수들

데이터 분석에서 많이 사용되는 함수의 형태는 다항식(polynomial) 함수, 지수(exponential) 함수, 로그(log) 함수 등이다.

### 다항식 함수

다항식 함수는 상수항 $c_0$, 일차항 $c_1x$, 이차항 $c_2x^2$, $\cdots$ 등의 거듭제곱 항의 선형 조합으로 이루어진 함수이다. 다음은 단변수(uni-variate) 다항식 함수의 전형적인 형태이다.

$$ f(x) = c_0 + c_1 x + c_2 x^2 + \cdots + c_n x^n $$

### 지수 함수와 로그 함수

밑(base)를 오일러 수 $e$로 하는 지수함수는 다음과 같이 표시한다. 이는 $e$라는 숫자를 $x$번 거듭제곱한 것이라 생각하면 된다.

$$ y = e^x $$
또는
$$ y = \exp x $$

지수 함수의 역함수는 자연로그 함수이다.

$$ y = \log x $$

만약 밑이 $e$가 아닌 경우에는 다음과 같이 변형하여 사용한다.

$$ y = a^x = e^{\log a \cdot x} $$

## 함수의 그래프와 기울기

함수의 형상을 직관적으로 파악하기 위해 그래프(graph)를 사용하기도 한다. 파이썬에서는 matplotlib의 라인 플롯을 사용하여 그래프를 만들 수 있다.

다만 matplotlib에서는 구체적인 위치가 있어야지만 플롯을 만들 수 있기 때문에 그래프를 작성할 $x$ 영역을 작은 구간으로 나눈 벡터를 생성하고 이 벡터 값에 대한 함수값을 계산하여 그래프를 작성한다. 구간의 간격이 너무 크면 그래프가 부정확해지고 구간의 간격이 너무 작으면 쓸데없이 세부적인 그림을 그리게 되므로 계산 시간이 증가하고 메모리 등의 리소스가 낭비된다.

```python
x = np.linspace(-0.9, 2.9, 100)
y = x**3 - 3*x**2 + x
plt.plot(x, y);
```

함수의 그래프는 앞에서 그린 것처럼 부드러운 곡선(curve)의 형태로 나타나는 경우가 많다. 이 곡선에 대해 한 점만 공통으로 가지는 접선(tangent)를 그릴 수 있는데 이 접선이 수평선과 이루는 각도를 기울기(slope)라고 한다.

```python
x = np.linspace(-0.9, 2.9, 100)
y = x**3-3*x**2+x
plt.plot(x, y)
plt.plot(0, 0, 'ro'); plt.plot(x, x, 'r:');
plt.plot(1, -1, 'go'); plt.plot(x, (3*1**2-6*1+1)*(x-1)-1, 'g:');
```

## 미분

미분(differenciation)이란 이러한 함수로부터 새로운 함수를 도출하는 변환의 일종이다. 미분을 통해 만들어진 새로운 함수는 원래 함수의 기울기(slope)를 나타낸다. 미분으로 만들어진 함수를 원래 함수의 도함수(derivative)라고 한다. 실제로는 극한과 수렴이라는 복잡한 개념을 사용하여  미분을 정의하지만 최적화(optimization)를 위해서는 단순히 기울기를 뜻한다고만 알아도 충분하다.

도함수는 함수 기호에 뒤에 prime 윗첨자를 붙이거나 함수 기호의 앞에 $\dfrac{d}{dx}$, $\dfrac{\partial}{\partial x}$ 등을 붙여서 표시한다. 분수처럼 표기하기도 하는데 분모의 위치에는 미분하고자 하는 변수가 오고 분자의 위치에는 미분하는 함수 자체의 기호나 혹은 함수 계산의 결과로 얻어지는 변수를 넣는다.

예를 들어 $y = f(x)$라는 함수를 미분하면 다음과 같다.

$$ f'(x) = \dfrac{d}{dx}(f) = \dfrac{df}{dx}  = \dfrac{d}{dx}(y) = \dfrac{dy}{dx} $$

## 미분 공식

현실적으로 미분은 다음에 설명할 몇가지 공식(formula)를 조합하여 원래 함수에서 도함수를 도출하는 과정이다. 함수가 복잡해지면 몇 페이지에 달아는 공식집이 필요할 정도이지만 여기에서는 가장 핵심적인 몇가지 공식만을 소개한다. 다양한 미분 공식에 대해 알고 싶다면 다음 웹사이트들을 참조한다.

* https://en.wikipedia.org/wiki/Derivative#Rules_of_computation
* https://en.wikipedia.org/wiki/Differentiation_rules

### 기본 미분 공식

* 상수

$$ \dfrac{d}{dx}(c) = 0 $$

$$ \dfrac{d}{dx}(cf) = c \cdot \dfrac{df}{dx} $$

* 거듭제곱

$$ \dfrac{d}{dx}(x^n) = n x^{n-1} $$

* 로그

$$ \dfrac{d}{dx}(\log x) = \dfrac{1}{x} $$

* 지수

$$ \dfrac{d}{dx}(e^x) = e^x $$

* 선형 조합

$$ \dfrac{d}{dx}\left(c_1 f_1 + c_2 f_2 \right) = c_1 \dfrac{df_1}{dx} + c_2 \dfrac{df_2}{dx}$$

이러한 기본 공식을 사용하여 다음 함수를 미분하면,

$$ y = 1 + 2x + 3x^2 + 4\exp(x) + 5\log(x) $$

답은 다음과 같다.

$$ \dfrac{dy}{dx} = 2 + 6x + 4\exp(x) + \dfrac{5}{x} $$

### 곱셈 법칙

어떤 함수의 형태가 두 개의 함수를 곱한 것과 같을 때는 다음과 같이 각 개별 함수의 도함수를 사용하여 원래의 함수의 도함수를 구한다.

$$  \dfrac{d}{dx}\left( f  \cdot g \right) =  \dfrac{df}{dx} \cdot g + f \cdot  \dfrac{dg}{dx} $$

곱셈 법칙을 사용하면 다음과 같은 함수를 미분하여,

$$ f = x \cdot \exp(x) $$

다음과 같은 도함수를 구한다.

$$ \dfrac{df}{dx} = \exp(x) + x \exp(x) $$

## 연쇄 법칙

연쇄 법칙(chain rule)은 미분하고자 하는 함수가 어떤 두 함수의 nested form 인 경우 적용할 수 있다.

$$ f(x) = h(g(x)) $$

인 경우 도함수는 다음과 같이 구한다.

$$ \dfrac{df}{dx} = \dfrac{df}{dg} \cdot \dfrac{dg}{dx} $$

예를 들어 정규 분포의 확률 밀도 함수는 기본적으로 다음과 같은 형태라고 볼 수 있다.

$$ f = \exp \dfrac{(x-\mu)^2}{\sigma^2} $$

이 함수의 도함수는 다음과 같이 구할 수 있다.

$$  f = exp(z) \;,\;\;\;\; z = \dfrac{y^2}{\sigma^2}   \;,\;\;\;\;   y = x-\mu   $$

$$ \dfrac{df}{dx} = \dfrac{df}{dz} \cdot \dfrac{dz}{dy} \cdot \dfrac{dy}{dx} $$

$$ \dfrac{df}{dz} = \exp(z) = \exp \dfrac{(x-\mu)^2}{\sigma^2} $$

$$ \dfrac{dz}{dy} =  \dfrac{2y}{\sigma^2} = \dfrac{2(x-\mu)}{\sigma^2} $$

$$ \dfrac{dy}{dx} =  1 $$

$$ \dfrac{df}{dx} =  \dfrac{2(x-\mu)}{\sigma^2} \exp \dfrac{(x-\mu)^2}{\sigma^2}$$

## 로그함수의 미분

로그 함수에 연쇄 법칙을 적용하면 다음과 같은 규칙을 얻을 수 있다.

$$ \dfrac{d}{dx} \log f(x) = \dfrac{f'(x)}{f(x)} $$

## 편미분

만약 함수가 두 개 이상의 독립변수를 가지는 다변수 함수인 경우에도 미분 즉, 기울기는 하나의 변수에 대해서만 구할 수 있다. 이를 편미분(partial differentiation)이라고 한다. 따라서 편미분의 결과로 하나의 함수에 대해 여러개의 도함수가 나올 수 있다.

다음은 편미분의 간단한 예이다.

$$ f(x,y) = x^2 + xy + y^2 $$

$$ f_x(x,y) = \dfrac{\partial f}{\partial x} = 2x + y $$

$$ f_y(x,y) = \dfrac{\partial f}{\partial y} = x + 2y $$

## SymPy

SymPy는 심볼릭 연산(symbolic operation)을 지원하기 위한 파이썬 패키지이다. 심볼릭 연산이란 사람이 연필로 계산하는 미분/적분과 동일한 형태의 연산을 말한다. 즉, $x^2$의 미분 연산을 수행하면 그 결과가 $2x$란 형태로 출력된다.

딥 러닝(deep learning) 등에 많이 사용되는 파이썬의 theano 패키지나 tensorflow 패키지도 뉴럴 네트워크 트레이닝시에 필요한 기울기 함수 계산을 위해 이러한 심볼릭 연산 기능을 갖추고 있다.

이를 위해서는 SymPy의 `symbols` 명령을 사용하여 $x$라는 기호가 단순한 숫자나 벡터 변수가 아닌 기호에 해당하는 것임을 알려주어야 한다.

```python
import sympy
sympy.init_printing(use_latex='mathjax')  # Juypter 노트북에서 수학식의 LaTeX 표현을 위해 필요함
```

```python
x = sympy.symbols('x')
x
```

$$x$$

```python
type(x)
```

    sympy.core.symbol.Symbol

일단 심볼 변수를 정의하면 이를 사용하여 다음과 같이 함수를 정의한다. 이 때 수학 함수는 SymPy 전용 함수를 사용해야 한다.

```python
f = x * sympy.exp(x)
f
```

$$x e^{x}$$

함수가 정의되면 `diff` 명령으로 미분을 할 수 있다. 또한 `simplify` 명령으로 소인수분해 등을 통한 수식 정리가 가능하다.

```python
sympy.diff(f)
```

$$x e^{x} + e^{x}$$

```python
sympy.simplify(sympy.diff(f))
```

$$\left(x + 1\right) e^{x}$$

편미분을 하는 경우에는 어떤 변수로 미분하는지를 명시해야 한다.

```python
x, y = sympy.symbols('x y')
f = x**2 + x*y + y**2
f
```

$$x^{2} + x y + y^{2}$$

```python
sympy.diff(f, x)
```

$$2 x + y$$

```python
sympy.diff(f, y)
```

$$x + 2 y$$

복수의 기호를 사용하는 경우에도 편미분을 해야 한다.

```python
x, mu, sigma = sympy.symbols('x mu sigma')
f = sympy.exp((x-mu)**2)/sigma**2
f
```

$$\frac{1}{\sigma^{2}} e^{\left(- \mu + x\right)^{2}}$$

```python
sympy.diff(f, x)
```

$$\frac{1}{\sigma^{2}} \left(- 2 \mu + 2 x\right) e^{\left(- \mu + x\right)^{2}}$$

```python
sympy.simplify(sympy.diff(f, x))
```

$$\frac{2}{\sigma^{2}} \left(- \mu + x\right) e^{\left(\mu - x\right)^{2}}$$


# The Discrete-Time Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Summary of Properties, Theorems and Transforms

The [properties](properties.ipynb), [theorems](theorems.ipynb) and transforms of the discrete-time Fourier transform (DTFT) as derived in the previous sections are summarized in the following. The corresponding tables serve as a reference for the application of the DTFT in the theory of signals and systems. Please refer to the respective sections for details.

### Definition

The DTFT and its inverse are defined as

\begin{align}
X(e^{j \Omega}) &= \sum_{k = -\infty}^{\infty} x[k] \, e^{- j \Omega k} \\
x[k] &= \frac{1}{2 \pi} \int_{-\pi}^{\pi} X(e^{j \Omega}) \, e^{j \Omega k} \; d \Omega
\end{align}

for $k \in \mathbb{Z}$ and $\Omega \in \mathbb{R}$.

### Properties and Theorems

The properties and theorems of the DTFT are given as

|  | $x[k] \qquad \qquad \qquad \qquad$ | $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \} \qquad \qquad \qquad$ |
|:---|:---:|:---:|:---|
| [Linearity](properties.ipynb#Linearity) | $A \, x_1[k] + B \, x_2[k]$ | $A \, X_1(e^{j \Omega}) + B \, X_2(e^{j \Omega})$ |
| [Real-valued signal](properties.ipynb#Real-valued-signals) | $x^*[k]$ | $X^*(e^{-j \Omega})$ |
| [Convolution](theorems.ipynb#Convolution-Theorem) | $x[k] * h[k]$ | $X(e^{j \Omega}) \cdot H(e^{j \Omega})$ |
| [Shift](theorems.ipynb#Shift-Theorem) | $x[k - \kappa]$ | $e^{-j \Omega \kappa} \cdot X(e^{j \Omega})$ |
| [Multiplication](theorems.ipynb#Multiplication-Theorem) | $x[k] \cdot h[k]$ | $\frac{1}{2 \pi} X(e^{j \Omega}) \circledast_{2 \pi} H(e^{j \Omega})$ |
| [Modulation](theorems.ipynb#Modulation-Theorem) | $e^{j \Omega_0 k} \cdot x[k]$ | $X(e^{j (\Omega- \Omega_0)})$ |
| [Parseval's Theorem](theorems.ipynb#Parseval's-Theorem) | $\sum_{k = -\infty}^{\infty} \lvert x[k] \rvert^2$ | $\frac{1}{2 \pi} \int_{-\pi}^{\pi} \lvert X(e^{j \Omega}) \rvert^2 \; d\Omega$ |

where $A, B \in \mathbb{C}$, $\Omega_0 \in \mathbb{R}$ and $\kappa \in \mathbb{Z}$

### Selected Transforms

DTFTs which are frequently used are given as

| $x[k] \qquad \qquad \qquad \qquad$ | $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \} \qquad \qquad \qquad \qquad \qquad \qquad$ |
|:---:|:---:|:---|
| $\delta[k]$ | $1$ |
| $1$ | ${\bot \!\! \bot \!\! \bot}\left( \frac{\Omega}{2 \pi} \right)$ |
| $\epsilon[k]$ | $\frac{1}{1 - e^{-j \Omega}} + \frac{1}{2} {\bot \!\! \bot \!\! \bot}\left( \frac{\Omega}{2 \pi} \right)$ |
| $\text{rect}_N[k]$ | $e^{-j \Omega \frac{N-1}{2}} \cdot \frac{\sin \left(\frac{N \Omega}{2} \right)}{\sin \left( \frac{\Omega}{2} \right)}$ |
| $e^{j \Omega_0 k} $ | ${\bot \!\! \bot \!\! \bot}\left( \frac{\Omega-\Omega_0}{2 \pi} \right)$ |
| $\sin(\Omega_0 t)$ | $\frac{1}{2} \left[ {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega + \Omega_0}{2 \pi} \right) + {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega - \Omega_0}{2 \pi} \right)  \right]$ |
| $\cos(\Omega_0 t)$ | $\frac{j}{2} \left[ {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega + \Omega_0}{2 \pi} \right) - {\bot \!\! \bot \!\! \bot} \left( \frac{\Omega - \Omega_0}{2 \pi} \right)  \right]$ |
| $a^k \epsilon[k]$ | $\frac{1}{1-a e^{-j \Omega}}$ |

where $\Omega_0 \in \mathbb{R}$ and $|a| < 1$. More transforms may be found in the literature or [online](https://en.wikipedia.org/wiki/Discrete-time_Fourier_transform#Table_of_discrete-time_Fourier_transforms).

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# 単回帰分析と重回帰分析

本章では、基礎的な機械学習手法として代表的な**単回帰分析**と**重回帰分析**の仕組みを、数式を用いて説明します。
また次章では、本章で紹介した数式を Python によるプログラミングで実装する例も紹介します。本章と次章を通じて、数学とプログラミングの結びつきを体験して理解することができます。

本チュートリアルの主題であるディープラーニングの前に、単回帰分析と重回帰分析を紹介することには 2 つの理由があります。
1 つ目は、単回帰分析と重回帰分析の数学がニューラルネットワーク含めたディープラーニングの数学の基礎となるためです。
2 つ目は、単回帰分析のアルゴリズムを通して微分、重回帰分析のアルゴリズムを通して線形代数に関する理解を深めることができるためです。

機械学習手法は、**教師あり学習 (supervised learning)** 、**教師なし学習 (unsupervised learning)** 、**強化学習 (reinforcement learning)** に大別され、単回帰分析は教師あり学習に含まれます。
本チュートリアルで扱う多くの手法が教師あり学習です。

教師あり学習の中でも典型的な問題設定は 2 つに大別されます。
与えられた入力変数から、$10$ や $0.1$ といった実数値を予測する **回帰 (regression)** と、「赤ワイン」、「白ワイン」といったカテゴリを予測する**分類 (classification)** の 2 つです。

単回帰分析は回帰を行うための手法であり、1 つの入力変数から 1 つの出力変数を予測します。
それに対し、重回帰分析は、複数の入力変数から 1 つの出力変数を予測します。
この両手法は教師あり学習であるため、訓練の際には、入力変数 $x$ と目的変数 $t$ がペアで準備されている必要があります。

回帰分析を行うアルゴリズムでは、以下の 3 ステップを順番に考えていきます。

- Step 1 : モデルを決める
- Step 2 : 目的関数を決める
- Step 3 : 最適なパラメータを求める

## 単回帰分析

### 問題設定（単回帰分析）

単回帰分析では、 1 つの入力変数から 1 つの出力変数を予測します。
今回は身近な例として、部屋の広さ $x$ から家賃 $y$ を予測する問題を考えてみます。

### Step 1：モデルを決める（単回帰分析）

まずはじめに、入力変数 $x$ と出力変数 $y$ との関係をどのように定式化するかを決定します。
この定式化したものを **モデル** もしくは **数理モデル** と呼びます。

単回帰分析におけるモデルを具体的に考えていきましょう。
例えば、家賃と部屋の広さの組で表されるデータを 3 つ集め、「家賃」を  $y$ 軸に、「部屋の広さ」を  $x$ 軸にとってそれらをプロットしたとき、次のようになっていたとします。

この場合、部屋が広くなるほど、家賃が高くなるという関係が予想されます。
また、この 2 変数間の関係性は直線によって表現を行うことができそうだと考えられます。
そこで、2 つのパラメータ $w$ と $b$ によって特徴づけられる直線の方程式

$$
y = wx + b
$$

によって、部屋の広さと家賃の関係を表すことを考えます。
ここで、$w$ は **重み (weight)** 、$b$ は **バイアス (bias)** の頭文字を採用しています。

単回帰分析では、このようにモデルとして直線 $y = wx + b$ を用います。
そして、2 つのパラメータ $w$ と $b$ を、直線がデータによくフィットするように調整します。

パラメータで特徴づけられたモデルを用いる場合、与えられた **データセット** に適合するように最適なパラメータを求めることが目標となります。
今回はデータセットとして部屋の広さ $x$ と家賃 $t$ の組からなるデータの集合を用います。
全部で $N$ 個のデータがあり、$n$ 番目のデータが $(x_n, t_n)$ と表されるとき、データセットは

$$
\begin{align}
\mathcal{D}
&= \{(x_1, t_1), (x_2, t_2), \dots, (x_N, t_N)\} \\
&= \{(x_n, t_n)\}_{n=1}^{N}
\end{align}
$$

と表すことができます（[注釈1](#note1)）。
これを用いて、新しい $x$ を入力すると、それに対応する $t$ を予測するモデルを訓練します。

### 前処理

次のステップに進む前に、データの**前処理 (preprocessing)** をひとつ紹介します。
データの **中心化 (centering)** は、データの平均値が 0 になるように全てのデータを平行移動する処理を言います。
下図は、データ集合 $(x_n, y_n) \ (n=1,\dots,11)$ を、平均が $(0, 0)$ になるように平行移動する例です。

中心化によるデータ前処理の利点の一つとして、調整すべきパラメータを削減できることが挙げられます。中心化処理を行うことで切片を考慮する必要がなくなるため、データ間の関係性を表現する直線の方程式を、 $y_c = wx_c$ のように、簡潔に表現可能となります。

データセット内の入力変数と目的変数の平均をそれぞれ $\bar{x}$, $\bar{t}$ としたとき、中心化後の入力変数と目的変数は、

$$
\begin{aligned}
x_{c} &= x - \bar{x} \\
t_{c} &= t - \bar{t}
\end{aligned}
$$

となります。

以降は記述を簡単にするため、$_c$ という添え字を省略し、事前に中心化を行っている前提でデータを扱います。
また、そのデータにフィットさせたいモデルは、

$$
y = wx
$$

と、こちらも添え字 $_c$ を省略して説明を行います。

### Step 2：目的関数を決める（単回帰分析）

ここでの目標は、部屋の広さと家賃の関係を直線の方程式によってモデル化することです。
このために、予め収集されたいくつかのデータセットを使って、モデルが部屋の広さの値から予測した家賃（予測値）と、その部屋の広さに対応する実際の家賃（目標値）の差が小さくなるように、モデルのパラメータを決定します。

今回は、目的関数として [こちらの章](https://tutorials.chainer.org/ja/src/03_Basic_Math_for_Machine_Learning_ja.html#note2) ですでに紹介した予測値と目標値との**二乗和誤差 (sum-of-squares error)** を用います。
二乗和誤差が $0$ のとき、またその時のみ予測値 $y$ は目標値 $t$ と完全に一致（$t = y$）しています。
$n$ 個目のデータの部屋の広さ $x_n$ が与えられたときのモデルの予測値 $y_n$ と、対応する目標値 $t_n$ との間の二乗誤差は、

$$
(t_{n} - y_{n})^{2}
$$

となります。これを全データに渡って合計したものが以下の二乗和誤差です。

$$
\begin{aligned}
L
&= (t_1 - y_1)^2
+ (t_2 - y_2)^2
+ \cdots
+ (t_N - y_N)^2 \\
&= \sum_{n=1}^N (t_n - y_n )^2 \\
\end{aligned}
$$

今回用いるモデルは

$$
y_{n} = wx_{n}
$$

であるため、目的関数は

$$
L = \sum_{n=1}^N (t_n - wx_n)^2
$$

と書くこともできます。

### Step 3：最適なパラメータを求める（単回帰分析）

この目的関数を最小化するようなパラメータを求めます。
ここで、目的関数は差の二乗和であり、常に正の値または $0$ を取るような、下に凸な二次関数となっています。
（一般的には多くの場合において、最適なパラメータを用いてもモデルがすべてのデータを完全に表現できず、目的関数の値は $0$ にはなりません。）

目的関数の値が最小となる点を求める際には、微分の知識が有用です。
微分では、対象とする関数の接線の傾きを求めることができます。凸関数では、この接線の傾きが 0 である点において、関数の最小値、もしくは最大値が得られます。
今回は、目的関数が $x$ に関する二次関数となっているため、下図のように重み $w$ に関する接線の傾きが $0$ であるときに、目的関数の値が最小となります。

それでは、具体的に今回定めた目的関数 $L$ をパラメータである $w$ で微分してみましょう。
微分に関する基本的な計算や性質は [こちらの章](https://tutorials.chainer.org/ja/src/04_Basics_of_Differential_ja.html)で紹介しました。

$$
\frac{\partial}{\partial w} L
= \frac{\partial}{\partial w} \sum_{n=1}^N (t_n - wx_n)^2
$$

ここで、微分の**線形性**から、和の微分は、微分の和となるため、

$$
\frac{\partial}{\partial w} L
= \sum_{n=1}^N \frac{\partial}{\partial w} (t_n - wx_n)^2
$$

と変形できます。

次に、総和（$\sum$）の中の各項に着目すると、

$$
\frac{\partial}{\partial w} (t_n - wx_n)^2
$$

となっており、この部分は $t_n - wx_n$ と $(\cdot)^2$ という関数の**合成関数**になっています。
そこで、$u_n = t_n - wx_n$、$f(u_n) = u_n^2$ とおいて計算すると、

$$
\begin{aligned}
\frac{\partial}{\partial w}(t_n - wx_n)^2
&= \frac{\partial}{\partial w} f(u_n) \\
&= \frac{\partial u_n}{\partial w}\frac{\partial f(u_n)}{\partial u_n} \\
&= -x_n (2 u_n) \\
&= -2x_n(t_n - wx_n)
\end{aligned}
$$

が得られます。

これを $\partial L / \partial w$ の式に戻すと、

$$
\begin{aligned}
\frac{\partial}{\partial w}
L
&= \sum_{n=1}^N \frac{\partial}{\partial w} (t_n - wx_n)^2 \\
&= -\sum_{n=1}^N 2x_n(t_n - wx_n)
\end{aligned}
$$

となります。

この導関数の値が $0$ となるときの $w$ が、目的関数を最小にするパラメータです。
そこで、$\frac{\partial}{\partial w} L = 0$ とおいてこれを $w$ について解きます。

$$
\begin{aligned}
\frac{\partial}{\partial w} L &= 0 \\
-2 \sum_{n=1}^N x_n (t_n - wx_n) &= 0 \\
-2 \sum_{n=1}^N x_n t_n + 2 \sum_{n=1}^N wx^2_n &= 0 \\
-2 \sum_{n=1}^N x_n t_n + 2 w \sum_{n=1}^N x^2_n &= 0 \\
w \sum_{n=1}^N x^2_n &= \sum_{n=1}^N x_n t_n \\
\end{aligned}
$$

以上より、

$$
w = \frac{\sum_{n=1}^N x_n t_n}{\sum_{n=1}^N x^2_n}
$$

と求まりました。これを最適な $w$ と呼びます。
この値は、与えられたデータセット $\mathcal{D} = \{x_n, t_n\}_{n=1}^{N}$ のみから決定されています。

### 数値例

例題にあげていた数値でパラメータ $w$ を求めてみましょう。

まずはデータの中心化を行うために、平均の値を事前に算出します。

$$
\begin{aligned}
\bar{x} &= \frac{1}{3} (20 + 40 + 60) = 40 \\
\bar{t} &= \frac{1}{3} (60000 + 115000 + 155000) = 110000
\end{aligned}
$$

この平均の値を使い、全変数に対して中心化を行うと、

$$
\begin{aligned}
x_{1} &= 20 - 40 = -20 \\
x_{2} &= 40 -40 = 0 \\
x_{3} &= 60- 40- = 20 \\
t_{1} &= 60000 - 110000 = -50000 \\
t_{2} &= 115000- 110000 = 5000 \\
t_{3} &= 155000 - 110000 = 45000
\end{aligned}
$$

となります。

これらの中心化後の値を用いて、最適なパラメータ $w$ を計算すると、

$$
\begin{aligned}
w
&= \frac{\sum_{n=1}^N x_n t_n}{\sum_{n=1}^N x_n^2} \\
&= \frac{x_1 t_1 + x_2 t_2 + x_3 t_3}{x_1^2 + x_2^2 + x_3^2} \\
&= \frac{-20 \times (-50000) + 0 \times 5000 + 20 \times 45000}{(-20)^2 + 0^2 + 20^2} \\
&= 2375
\end{aligned}
$$

と求まります。
したがって、家賃が　$1$ m$^{2}$ 増えるごとに、$2375$ 円家賃が上昇しているとわかります。

この $w$ を用いて決定される直線 $y = 2375 x$ と、学習データとして用いた 3 つの点をプロットした図が以下です。

この直線上の点の $y$ の値が、対応する $x$ の値に対するここで訓練したモデルによる予測値です。
ここで、$x$ 軸で負の値をとっていますが、これは中心化後であることに注意してください。

訓練済みのモデルを使って新しいサンプル $x_{q}$ に対する家賃の予測を行います。
例えば、部屋の広さが 50 m$^2$ の場合の家賃に対する予測値を計算する推論を行いましょう。

$$
\begin{aligned}
y_c &= wx_c \\
y_q - \bar{t} &= w(x_q - \bar{x}) \\
\Rightarrow y_q &= w(x_q - \bar{x}) + \bar{t} \\
&= 2375 (50 - 40) + 110000 \\
&= 133750
\end{aligned}
$$

このように、部屋の広さが　50 m$^{2}$ の場合の家賃が 133,750 円であると予測することができました。
上記のように、モデル化の際に中心化を行っていた処理を推論の際には元に戻して計算しましょう。

## 重回帰分析

### 問題設定（重回帰分析）

重回帰分析も単回帰分析の場合と同様に家賃を予測する問題を題材にして説明します。
単回帰分析の場合と異なり、入力変数として「部屋の広さ」だけでなく、「駅からの距離」や「犯罪発生率」などの変数を合わせて考慮することにします。
部屋の広さを $x_{1}$、駅からの距離を $x_{2}$、…、犯罪発生率を $x_{M}$ といった形で表し、$M$ 個の入力変数を扱うことを考えてみましょう。

### Step 1：モデルを決める（重回帰分析）

単回帰分析では、

$$
y = wx + b
$$

のように直線の方程式をモデルとして用いていました。重回帰分析でも、

$$
y = w_{1}x_{1} + w_{2}x_{2} + \cdots + w_{M}x_{M} + b
$$

のように、単回帰分析と似た形のモデルを定義します。
単回帰分析の際は二次元平面を考え、その平面上に存在するデータに最もよくフィットする直線を考えましたが、今回は $M$ 次元空間に存在するデータに最もよくフィットする 直線を考えることになります。

重回帰分析のモデルは総和の記号を使って表記すると、

$$
y = \sum_{m=1}^{M} w_{m} x_{m} + b
$$

と書くことができます。

ここでバイアス $b$ の扱い方を改めて考えます。
単回帰分析では、中心化を前処理として施し、バイアス $b$ を省略することで、簡潔に定式化することができました。

重回帰分析では、 $M$ 個の重み $w_{1}, w_{2}, \dots, w_{M}$ と 1 個のバイアス $b$ があり、合わせて $M + 1$ 個のパラメータが存在します。これらのパラメータをうまく定式化することを考えます。
そこで、今回は $x_0 = 1$、$w_0 = b$ とおくことで、

$$
\begin{aligned}
y
&= w_{1}x_{1} + w_{2}x_{2} + \cdots + w_{M}x_{M} + b \\
&= w_{1}x_{1} + w_{2}x_{2} + \cdots + w_{M}x_{M} + w_{0}x_{0} \\
&= w_{0}x_{0} + w_{1}x_{1} + \cdots + w_{M}x_{M} \\
&= \sum_{m=0}^M w_{m} x_{m}
\end{aligned}
$$

のように $b$ を総和の内側の項に含めて、簡潔に表記できるようにします。
（ここで、 $\sum$ 記号の下部が $m=1$ ではなく $m=0$ となっていることに注意してください。）

さらに、ここから線形代数で学んだ知識を活かして、数式を整理していきます。
上式をベクトルの内積を用いて表記しなおすと、

$$
\begin{aligned}
y
&= w_{0}x_{0} + w_{1}x_{1} + \cdots + w_{M}x_{M} \\
&=
\begin{bmatrix}
w_{0} & w_{1} & \cdots  & w_{M}
\end{bmatrix}
\begin{bmatrix}
x_{0} \\
x_{1} \\
\vdots \\
x_{M}
\end{bmatrix} \\
&= {\bf w}^{\rm T}{\bf x}
\end{aligned}
$$

のように、シンプルな形式で表現することができます。
このモデルが持つパラメータは前述の通り $M + 1$ 個あり、$M + 1$ 次元のベクトル ${\bf w}$ で表されています。
重回帰分析では、この ${\bf w}$ のすべての要素について最適な値を求めます。

### Step 2：目的関数を決める（重回帰分析）

単回帰分析の例と比べると、入力変数は増えましたが、家賃を目標値としている点は変わっていません。
そこで、単回帰分析と同じ目的関数

$$
L = (t_1 - y_1)^2 + (t_2 - y_2)^2 + \cdots + (t_N - y_N)^2
$$

を用います。
この目的関数は、ベクトルの内積を用いて表記し直すと、

$$
\begin{aligned}
L
&= (t_1 - y_1)^2 + (t_2 - y_2)^2 + \cdots + (t_N - y_N)^2 \\
&= \begin{bmatrix}
t_1 - y_1 & t_2 - y_2 & \cdots & t_N - y_N
\end{bmatrix}
\begin{bmatrix}
t_1 - y_1 \\
t_2 - y_2 \\
\vdots \\
t_N - y_N
\end{bmatrix} \\
&= ({\bf t} - {\bf y})^{\rm T}({\bf t} - {\bf y})
\end{aligned}
$$

と書くことができます。

ここで、内積には交換法則が成り立つため、${\bf w}^{\rm T}{\bf x}$ は ${\bf x}^{\rm T}{\bf w}$ と書くこともできます。これを利用して、モデルの方程式 ${\bf y} = {\bf w}^{\rm T}{\bf x}$ を、以下のように変形します。

$$
\begin{aligned}
{\bf y} =
\begin{bmatrix}
y_1 \\
y_2 \\
\vdots \\
y_N
\end{bmatrix} =
\begin{bmatrix}
{\bf x}_1^{\rm T}{\bf w} \\
{\bf x}_2^{\rm T}{\bf w} \\
\vdots \\
{\bf x}_N^{\rm T}{\bf w}
\end{bmatrix} =
\begin{bmatrix}
{\bf x}_1^{\rm T} \\
{\bf x}_2^{\rm T} \\
\vdots \\
{\bf x}_N^{\rm T}
\end{bmatrix}
{\bf w}
\end{aligned}
$$

さらに、${\bf x}_n^{\rm T} = \bigl[ x_{n0},\ x_{n1},\  x_{n2},\ \dots,\  x_{nM} \bigr]$ $(n = 1, \dots, N)$ と展開すると、

$$
\begin{aligned}
{\bf y}
&= \begin{bmatrix}
x_{10} & x_{11} & x_{12} & \cdots & x_{1M} \\
x_{20} & x_{21} & x_{22} & \cdots & x_{2M} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
x_{N0} & x_{N1} & x_{N2} & \cdots & x_{NM}
\end{bmatrix}
\begin{bmatrix}
w_{0} \\
w_{1} \\
w_{2} \\
\vdots \\
w_{M}
\end{bmatrix} \\
&= {\bf X}{\bf w}
\end{aligned}
$$

と表記できます。
ここで、$N \times M$ 行列 ${\bf X}$ は、各行が各データを表しており、各列が各入力変数を表しています。
このような行列を、**デザイン行列 (design matrix)** と呼びます。
各列はそれぞれ入力変数の種類に対応しており、例えば、部屋の広さや、駅からの距離などです。

各行が表すデータ点がどのように表されているかを説明するため、具体的な数値例を挙げます。
部屋の広さ $= 50{\rm m}^2$ 、駅からの距離 $= 600 {\rm m}$ 、犯罪発生率 $= 2\%$ という 3 つの入力変数を考える場合、$M = 3$ であり、これが $n$ 個目のデータのとき、${\bf x}_n^{\rm T}$ は、

$$
{\bf x}_n^{\rm T} =
\begin{bmatrix}
1 & 50 & 600 & 0.02
\end{bmatrix}
$$

となります。先頭に $1$ があるのは、Step 1 で $x_0 = 1$ と定めたためです。

### Step 3：パラメータを最適化する（重回帰分析）

それでは、目的関数 $L$ を最小化するモデルのパラメータベクトル ${\bf w}$ を求めましょう。
単回帰分析と同様に、目的関数をパラメータで微分して 0 とおき、${\bf w}$ について解きます。

まずは目的関数に登場している予測値 ${\bf y}$ を、パラメータ ${\bf w}$ を用いた表記に置き換えます。

$$
\begin{aligned}
L
&= ({\bf t} - {\bf y})^{\rm T} ({\bf t} - {\bf y}) \\
&= ({\bf t} - {\bf X}{\bf w})^{\rm T} ({\bf t} - {\bf X}{\bf w}) \\
&= \{ {\bf t}^{\rm T} - ({\bf X}{\bf w})^{\rm T} \} ({\bf t} - {\bf X}{\bf w}) \\
&= ({\bf t}^{\rm T} - {\bf w}^{\rm T}{\bf X}^{\rm T}) ({\bf t} - {\bf X}{\bf w})
\end{aligned}
$$

ここで、転置の公式 $({\bf A}{\bf B})^{\rm T} = {\bf B}^{\rm T}{\bf A}^{\rm T}$ を用いました。

さらに分配法則を使って展開すると、

$$
L
= {\bf t}^{\rm T}{\bf t}
- {\bf t}^{\rm T}{\bf X}{\bf w}
- {\bf w}^{\rm T}{\bf X}^{\rm T}{\bf t}
+ {\bf w}^{\rm T}{\bf X}^{\rm T}{\bf X}{\bf w}
$$

となります。この目的関数に対しパラメータの ${\bf w}$ についての偏微分を計算します。

その前に、この式をもう少し整理します。
まず、$(1)^{\rm T} = 1$ のように、スカラは転置しても変化しません。
また、${\bf w} \in \mathbb{R}^{M+1}$、${\bf X} \in \mathbb{R}^{N \times (M+1)}$ であり、${\bf X}{\bf w} \in \mathbb{R}^{N}$ となることから、${\bf t} \in \mathbb{R}^{N}$ との間の内積 ${\bf t}^{\rm T}{\bf X}{\bf w} \in \mathbb{R}$ は、スカラになります。
したがって、

$$
({\bf t}^{\rm T}{\bf X}{\bf w})^{\rm T} = {\bf t}^{\rm T}{\bf X}{\bf w}
$$

が成り立ちます。

さらに、転置の公式 $({\bf A}{\bf B}{\bf C})^{\rm T} = {\bf C}^{\rm T}{\bf B}^{\rm T}{\bf A}^{\rm T}$ より、

$$
({\bf t}^{\rm T}{\bf X}{\bf w})^{\rm T} = {\bf w}^{\rm T} {\bf X}^{\rm T} {\bf t}
$$

も成り立ちます。以上から、

$$({\bf t}^{T}{\bf X}{\bf w})^{T} = {\bf t}^{T}{\bf X}{\bf w} = {\bf w}^{T} {\bf X}^{T} {\bf t}$$

が導かれます。目的関数 $L$ は、この式を利用して、

$$
L = {\bf t}^{\rm T}{\bf t} - 2{\bf t}^{\rm T}{\bf X}{\bf w}+ {\bf w}^{\rm T}{\bf X}^{\rm T}{\bf X}{\bf w}
$$

と変形できます。

ここで、${\bf w}$ に関する偏微分を行いやすくするため、${\bf w}$ 以外の定数項を一つにまとめます。

$$
\begin{aligned}
L
&= {\bf t}^{\rm T}{\bf t}
- 2{\bf t}^{\rm T}{\bf X}{\bf w}
+ {\bf w}^{\rm T}{\bf X}^{\rm T}{\bf X}{\bf w} \\
&= {\bf t}^{\rm T}{\bf t}
- 2({\bf X}^{\rm T}{\bf t})^{\rm T} {\bf w}
+ {\bf w}^{\rm T}{\bf X}^{\rm T}{\bf X}{\bf w} \\
&= c + {\bf b}^{\rm T}{\bf w} + {\bf w}^{\rm T}{\bf A}{\bf w}
\end{aligned}
$$

すると、${\bf w}$ に関する二次形式で表現できました。
ここで、

$$
\begin{align}
{\bf A} &= {\bf X}^{\rm T}{\bf X} \\
{\bf b} &= -2 {\bf X}^{\rm T}{\bf t} \\
c &= {\bf t}^{\rm T}{\bf t}
\end{align}
$$

とおいていることに注意してください。

それでは、目的関数を最小にするパラメータ ${\bf w}$ の求め方を考えます。
目的関数はパラメータ ${\bf w}$ に関して二次関数になっています。
まずは、${\bf w}$ 以外のベクトルや行列に、具体的な数値を当てはめて考えてみましょう。
例えば、

$$
{\bf w} =
\begin{bmatrix}
w_1 \\ w_2
\end{bmatrix},
{\bf A} =
\begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix},
{\bf b} =
\begin{bmatrix}
1 \\
2
\end{bmatrix},
c = 1
$$

とおきます。すると、目的関数の値は

$$
\begin{aligned}
L
&= {\bf w}^{\rm T}{\bf A}{\bf w} +{\bf b}^{\rm T}{\bf w} + c \\
&=
\begin{bmatrix}
w_1 & w_2
\end{bmatrix}
\begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix}
\begin{bmatrix}
w_1 \\
w_2
\end{bmatrix}
+
\begin{bmatrix}
1 & 2
\end{bmatrix}
\begin{bmatrix}
w_1 \\
w_2
\end{bmatrix}
+
1 \\
&=
\begin{bmatrix}
w_1 & w_2
\end{bmatrix}
\begin{bmatrix}
w_1 + 2w_2 \\
3w_1 + 4w_2
\end{bmatrix}
+ w_1 + 2w_2 + 1 \\
&= w_1 (w_1 + 2w_2) + w_2 (3w_1 + 4w_2) + w_1 + 2w_2 + 1 \\
&= w_1^2 + 5 w_1 w_2 + 4 w_2^2 + w_1 + 2 w_2 + 1 \\
\end{aligned}
$$

となります。これを $w_1, w_2$ に関して整理すると、

$$
\begin{aligned}
L
&= w_1^2 + (5 w_2 + 1) w_1 + (4 w_2^2 + 2 w_2 + 1) \\
&= 4 w_2^2 + (5 w_1 + 2) w_2 + (w_1^2 + w_1 + 1)
\end{aligned}
$$

となり、$w_1, w_2$ それぞれについて二次関数になっていることが分かります。

目的関数 $L$ を $w_1$ の二次関数、$w_2$ の二次関数と見たとき、$L$ は、下図のような概形となっています。

さらに、各次元が $w_1, w_2, L$ を表す 3 次元空間上においては、 $L$ の概形は下図のようになっています。

ここでは 2 つのパラメータ $w_1$ と $w_2$ について図示していますが、目的関数が 任意の $M$ 個の変数 $w_0, w_1, w_2, \dots, w_M$ によって特徴づけられている場合でも、目的関数がそれぞれのパラメータについて二次形式になっている限り、同様のことが言えます。
すなわち、$M + 1$ 個の連立方程式、

$$
\begin{cases}
\frac {\partial }{\partial w_0}L = 0 \\
\frac {\partial }{\partial w_1}L = 0 \\
\ \ \ \ \ \vdots \\
\frac {\partial }{\partial w_M}L = 0 \\
\end{cases}
$$

を解けば良いということになります。
これはベクトルによる微分を用いて表記すると、以下のようになります。

$$
\begin{aligned}
\begin{bmatrix}
\frac {\partial}{\partial w_0} L \\
\frac {\partial}{\partial w_1} L \\
\vdots  \\
\frac {\partial}{\partial w_M} L \\
\end{bmatrix}
&=
\begin{bmatrix}
0 \\
0 \\
\vdots  \\
0 \\
\end{bmatrix} \\
\Rightarrow \frac {\partial}{\partial {\bf w}} L
&= {\bf 0} \\
\end{aligned}
$$

上式を ${\bf w}$ について解くために、以下のような式変形を行います。
式変形の途中で理解できない部分があった場合は、[こちらの章](https://tutorials.chainer.org/ja/src/05_Basics_of_Linear_Algebra_ja.html) を読み返してみてください。
まずは、左辺について整理を行います。
$$
\begin{aligned}
\frac{\partial}{\partial {\bf w}} L
&= \frac{\partial}{\partial {\bf w}} (c + {\bf b}^{\rm T}{\bf w} + {\bf w}^{\rm T}{\bf A}{\bf w}) \\
&=\frac{\partial}{\partial {\bf w}} (c) + \frac{\partial}{\partial {\bf w}} ({\bf b}^{\rm T}{\bf w}) + \frac{\partial}{\partial {\bf w}} ({\bf w}^{\rm T}{\bf A}{\bf w}) \\
&={\bf 0} + {\bf b} + ({\bf A} + {\bf A}^{\rm T}) {\bf w}
\end{aligned}
$$

これを $0$ とおき、${\bf A}$ 、${\bf b}$ を展開すると

$$
\begin{aligned}
-2{\bf X}^{\rm T}{\bf t} + \{ {\bf X}^{\rm T}{\bf X} + ({\bf X}^{\rm T}{\bf X})^{\rm T} \} {\bf w}
&= {\bf 0} \\
-2{\bf X}^{\rm T}{\bf t} + 2{\bf X}^{\rm T}{\bf X}{\bf w}
&= {\bf 0} \\
{\bf X}^{\rm T}{\bf X}{\bf w}& = {\bf X}^{\rm T}{\bf t} \\
\end{aligned}
$$

のように式変形できます。

ここで、${\bf X}^{\rm T}{\bf X}$に**逆行列が存在すると仮定**して、両辺に左側から $({\bf X}^{\rm T}{\bf X})^{-1}$ を掛けると、

$$
\begin{aligned}
({\bf X}^{\rm T}{\bf X})^{-1} {\bf X}^{\rm T}{\bf X} {\bf w} &= ({\bf X}^{\rm T}{\bf X})^{-1} {\bf X}^{\rm T}{\bf t} \\
{\bf I}{\bf w} &= ({\bf X}^{\rm T}{\bf X})^{-1} {\bf X}^{\rm T}{\bf t} \\
{\bf w} &= ({\bf X}^{\rm T}{\bf X})^{-1}{\bf X}^{\rm T}{\bf t}
\end{aligned}
$$

が導かれます。特に、この最後の式を**正規方程式 (normal equation)** と呼びます。
上式は、与えられたデータを並べたデザイン行列 ${\bf X}$ と、各データの目標値を並べたベクトル ${\bf t}$ から、最適なパラメータ ${\bf w}$ を計算しています。
${\bf I}$ は単位行列を表します。

${\bf w}$ を求める際に気をつけたいこととして、以下の誤った式変形をしてしまう例が挙げられます。

$$
\begin{aligned}
{\bf X}^{\rm T}{\bf X}{\bf w} &= {\bf X}^{\rm T}{\bf t} \\
({\bf X}^{\rm T})^{-1} {\bf X}^{\rm T}{\bf X}{\bf w} &= ({\bf X}^{\rm T})^{-1} {\bf X}^{\rm T}{\bf t} \\
{\bf X}{\bf w} &= {\bf t} \\
{\bf X}^{-1}{\bf X}{\bf w} &= {\bf X}^{-1}{\bf t} \\
{\bf w} &= {\bf X}^{-1}{\bf t}
\end{aligned}
$$

上記の式変形は一般には成立しません。
この式変形が可能かどうかは、${\bf X}^{-1}$ が存在するかどうか、に関わっています。
サンプル数 $N$ と独立変数の数 $M + 1$ が等しくない場合、${\bf X} \in \mathbb{R}^{N \times (M + 1)}$ は**正方行列ではない**ため、逆行列 ${\bf X}^{-1}$ を持ちません。
従って、上式の 2 行目の変形を行うことはできません（逆行列が求まるためのより厳密な条件についてはここでは省略します）。

一方、 ${\bf X}^{\rm T}{\bf X}$ は $(M + 1) \times (M + 1)$ 行列であり、その形はサンプル数 $N$ に依存することなく、常に正方行列となるため、これを利用して式変形を行います。

新しい入力変数の値 ${\bf x}_q = [x_1, \dots, x_M]^{\rm T}$ に対して、対応する目標値 $y_q$ を予測するためには、訓練により決定されたパラメータ ${\bf w}$ を用いて、

$$
y_q = {\bf w}^{\rm T}{\bf x}_q
$$

のように計算します。

**注釈 1**

データセット中の各データのことをデータ点（datum）ということがあります。具体的には、上の説明で同上した $\mathcal{D}$ 中の各 $(x_1, t_1)$ などのことです。

[▲上へ戻る](#ref_note1)


# Formulation of Maxwell's Equations in Cartesian Coordinates and Flat Spacetime - An Overview

## Author: Terrence Pierre Jacques

## This tutorial notebook outlines two formulations of Maxwell's equations.

### This module serves as a stepping stone towards solving Einstein's equations using NRPy+. As such, the [tutorial notebook on the scalar wave equation in curvilinear coordinates](Tutorial-ScalarWaveCurvilinear.ipynb) is *required* reading before beginning this module. That module, as well as its own prerequisite [module on reference metrics within NRPy+](Tutorial-Reference_Metric.ipynb) provides the needed overview of how NRPy+ handles reference metrics.

# Introduction

To numerically solve Einstein's equations, a system of coupled, non-linear equations, the spatial coordinates may be decoupled from the time coordinate. In doing so, numerical solutions may be marched forward in time, using the [Method of Lines](https://reference.wolfram.com/language/tutorial/NDSolveMethodOfLines.html) ([NRPy+ tutorial here](Tutorial-Method_of_Lines-C_Code_Generation.ipynb)). This is done using the 3 + 1 decomposition, originally done using the ADM formalism. However, it was found that using this formulation of Einstein's equations resulted in unstable simulations, due to its weakly hyperbolic system of equations. Hyperbolicity is the notion of the wave-like behavior of the numerical solution for a given system of PDE's. The ADM formulation of Einstein's equations involved mixed second derivatives, which give rise to a weakly hyperbolic system.

In contrast, the BSSN formalism removes these mixed second derivatives, resulting in a strongly hyperbolic system of equations. In this module, we follow the work of [Knapp, Walker & Baumgarte (2002)](https://arxiv.org/abs/gr-qc/0201051), showcasing the difference between using strongly versus weakly hyperbolic systems, when solving a physical problem with Maxwell's equations in vacuum.

 ### A Note on Notation:

As is standard in NRPy+,

* Greek indices refer to four-dimensional quantities where the zeroth component indicates temporal (time) component.
* Latin indices refer to three-dimensional quantities. This is somewhat counterintuitive since Python always indexes its lists starting from 0. As a result, the zeroth component of three-dimensional quantities will necessarily indicate the first *spatial* direction.

$$\label{top}$$

# Table of Contents:

1. [Step 1](#step1): Maxwell's Equations in Vacuum - System I (ADM-like)
1. [Step 2](#step2): Maxwell's Equations in Vacuum - System II (BSSN-like)
1. [Step 3](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Maxwell's Equations in Vacuum - System I (ADM-like) \[Back to [top](#top)\]
$$\label{step1}$$

We begin with Maxwell's equations in vacuum, i.e. no source terms, in flat space and in [Gaussian](https://en.wikipedia.org/wiki/Gaussian_units) and $c = 1$ units, given by

\begin{align}
&\vec{\nabla} \cdot \vec{E} = 0, \\
&\vec{\nabla} \cdot \vec{B} = 0, \\
&\frac{\partial \vec{B}}{\partial t} = -\vec{\nabla} \times \vec{E}, \\
&\frac{\partial \vec{E}}{\partial t} = \vec{\nabla} \times \vec{B},
\end{align}

and we have the associated potentials,

\begin{align}
\frac{\partial \vec{A}}{\partial t} &= -\vec{E} -\vec{\nabla}\varphi , \\
\vec{B} &= \vec{\nabla} \times \vec{A}.
\end{align}

Now, replacing $\vec{B}$ with $\vec{\nabla} \times \vec{A}$ in Ampere's law, and using the standard identity

$$
\vec{\nabla} \times \left( \vec{\nabla} \times \vec{A} \right) = \vec{\nabla} \left( \vec{\nabla} \cdot \vec{A} \right) - \nabla^2 \vec{A},
$$

we may write

$$
\vec{\nabla} \times \vec{B} = \vec{\nabla} \times \left( \vec{\nabla} \times \vec{A} \right) = \vec{\nabla} \left( \vec{\nabla} \cdot \vec{A} \right) - \nabla^2 \vec{A}.
$$

Thus, our time evolution equations become

\begin{align}
\frac{\partial \vec{A}}{\partial t} &= -\vec{E} -\vec{\nabla}\varphi , \\
\frac{\partial \vec{E}}{\partial t} &= \vec{\nabla} \left( \vec{\nabla} \cdot \vec{A} \right) - \nabla^2 \vec{A}.
\end{align}

Using index notation, in Cartesian coordinates we have

\begin{align}
\partial_t A^i &= -E^i - \partial^i \varphi, \\
\partial_t E^i &= \partial^i \partial_j A^j - \partial_j \partial^j A^i.
\end{align}

Note the presence of the mixed second derivative above, and that in our system of equations we have 6 equations (since $\vec{A}$ and $\vec{E}$ each have 3 components), but 7 unknowns (components of $\vec{A}$ and $\vec{E}$, and the scalar potential $\varphi$). Thus, we’ll add a time evolution equation to $\varphi$ as well, which amounts to choosing a gauge. In particular we’ll choose the [Lorenz gauge](https://en.wikipedia.org/wiki/Lorenz_gauge_condition):

$$
\partial_t \varphi = -\vec{\nabla} \cdot \vec{A},
$$

and using index notation, in Cartesian coordinates,

$$
\partial_t \varphi = -\partial_i A^i.
$$

Furthermore, note that because we are working in vacuum, in Cartesian coordinates we have the constraints

\begin{align}
\partial_i E^i &= 0, \\
\partial_i B^i &= 0,
\end{align}

But since $\vec{B} = \vec{\nabla} \times \vec{A}$, the divergence of $\vec{B}$ is automatically satisfied (excercise for the reader).

The right hand sides (RHSs) of our evolution equations are thus

\begin{align}
\partial_t A^i &= -E^i - \partial^i \varphi, \\
\partial_t E^i &= \partial^i \partial_j A^j - \partial_j \partial^j A^i, \\
\partial_t \varphi &= -\partial_i A^i,
\end{align}

subject to the constraint

$$
\mathcal{C} \equiv \partial_i E^i = 0.
$$

Tracking the departure of our numerical results from this constraint helps us keep track of the numerical error. We refer to this system of equations as System I.

In [this tutorial](Tutorial-VacuumMaxwell_Cartesian_RHSs.ipynb) the RHSs and constraint equations are implemented in NRPy+. The system is then evolved in time within a [start-to-finish notebook](Tutorial-Start_to_Finish-Solving_Maxwells_Equations_in_Vacuum-Cartesian.ipynb), using initial data defined in [this tutorial](Tutorial-VacuumMaxwell_InitialData.ipynb), showcasing the **instability** of the system as a result of the presence of the mixed second derivative for $E^i$.

# Step 2: Maxwell's Equations in Vacuum - System II (BSSN - like) \[Back to [top](#top)\]
$$\label{step2}$$

As dicussed in the introduction and following the previous section, to maintain numerical stability and accuracy we remove the mixed 2nd derivative $\partial^i \partial_j A^j$ in the equation for $E^i$ by introducing

\begin{align}
\Gamma &\equiv \partial_i A^i, \\
\partial_t \Gamma &= \partial_i \partial_t A^i = -\partial_i E^i - \partial_i \partial^i \varphi \\
 &= - \partial_i \partial^i \varphi.
\end{align}

Thus, our evolution equations are

\begin{align}
\partial_t A^i &= -E^i - \partial^i \varphi, \\
\partial_t E^i &= \partial^i \Gamma - \partial_j \partial^j A^i, \\
\partial_t \Gamma &= - \partial_i \partial^i \varphi, \\
\partial_t \varphi &= -\Gamma,
\end{align}

subject to the constraints

\begin{align}
\mathcal{G} &\equiv \Gamma - \partial_i A^i &= 0, \\
\mathcal{C} &\equiv \partial_i E^i &= 0.
\end{align}

We refer to this system of equations as System II.

In [this tutorial](Tutorial-VacuumMaxwell_Cartesian_RHSs.ipynb) the RHSs and constraints are implemented in NRPy+. The system is then evolved in time within a [start-to-finish notebook](Tutorial-Start_to_Finish-Solving_Maxwells_Equations_in_Vacuum-Cartesian.ipynb), using initial data defined in [this tutorial](Tutorial-VacuumMaxwell_InitialData.ipynb), showcasing the **stability** of the system as a result of the absence of the mixed second derivative for $E^i$.

# Step 3: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#top)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-VacuumMaxwell_formulation_Cartesian.pdf](Tutorial-VacuumMaxwell_formulation_Cartesian.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-VacuumMaxwell_formulation_Cartesian")
```

    Created Tutorial-VacuumMaxwell_formulation_Cartesian.tex, and compiled
        LaTeX file to PDF file Tutorial-VacuumMaxwell_formulation_Cartesian.pdf