# Lista 8 - Verossimilhança

## Introdução

```python
# -*- coding: utf8

from scipy import stats as ss

import matplotlib.pyplot as plt
from numpy.testing import assert_equal
import numpy as np
import pandas as pd
import seaborn as sns

plt.rcParams['figure.figsize']  = (18, 10)
plt.rcParams['axes.labelsize']  = 20
plt.rcParams['axes.titlesize']  = 20
plt.rcParams['legend.fontsize'] = 20
plt.rcParams['xtick.labelsize'] = 20
plt.rcParams['ytick.labelsize'] = 20
plt.rcParams['lines.linewidth'] = 4
```

```python
plt.ion()
plt.style.use('seaborn-colorblind')
plt.rcParams['figure.figsize']  = (12, 8)
```

```python
def despine(ax=None):
    if ax is None:
        ax = plt.gca()
    # Hide the right and top spines
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)

    # Only show ticks on the left and bottom spines
    ax.yaxis.set_ticks_position('left')
    ax.xaxis.set_ticks_position('bottom')
```

Continuando da aula passada. Vamos ver mais uma forma de entender um modelo de regressão linear. Lembre-se até agora falamos de correlação e covariância cobrindo os seguintes tópicos:

1. Covariância
1. Coeficiente de Pearson (Covariância Normalizada)
1. Coeficiente de Pearson como sendo a fração do desvio de y capturado por x
1. Mínimos Quadrados

Todos os passos acima chegam no mesmo local de traçar a "melhor" reta no gráfico de dispersão. Melhor aqui significa a reta que que minimiza o erro abaixo:

$$\Theta = [\alpha, \beta]$$
$$L(\Theta) = \sum_i (y_i - \hat{y}_i)^2$$
$$L(\Theta) = \sum_i (y_i - (\beta x_i + \alpha))^2$$

Chegamos em:

\begin{align}
 \alpha & = \bar{y} - \beta\,\bar{x}, \\[5pt]
  \beta &= \frac{ \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^n (x_i - \bar{x})^2 } \\[6pt]
            &= \frac{ \operatorname{Cov}(x, y) }{ \operatorname{Var}(x) } \\[5pt]
            &= r_{xy} \frac{s_y}{s_x}. \\[6pt]
\end{align}

## Visão probabílistica

Vamos aprender uma última forma de pensar na regressão. Em particular, vamos fazer uso de uma visão probabílistica. Para tal, exploraremos o caso dos apartamentos de BH abaixo.

Inicialmente, vamos observar os dados além do resultado da melhor regressão.

```python
df = pd.read_csv('https://raw.githubusercontent.com/pedroharaujo/ICD_Docencia/master/aptosBH.txt', index_col=0)
df['preco'] = df['preco'] / 1000

plt.scatter(df['area'], df['preco'], edgecolors='k', s=80, alpha=0.6)

plt.title('Preço de Apartamentos em BH')
plt.ylabel(r'Preço * $10^3$ (R\$)')
plt.xlabel(r'Área ($M^2$)')
despine()
```

O seaborn tem uma função regplot que plota a melhor reta além de um intervalo de confiança.

```python
sns.regplot(x='area', y='preco', data=df, n_boot=10000,
            line_kws={'color':'magenta', 'lw':4},
            scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
plt.title('Preco de Apartamentos em BH')
plt.ylabel(r'Preço * $10^3$ (R\$)')
plt.xlabel(r'Área ($M^2$)')
despine()
```

A reta pode ser recuperada usando scipy.

```python
model = ss.linregress(df['area'], df['preco'])
model
```

    LinregressResult(slope=3.5357191563336516, intercept=200.5236136898945, rvalue=0.694605637960645, pvalue=1.917920339304322e-32, stderr=0.2503210673009473)

Usando esta reta podemos prever o preço de um apartamento usando apenas a área do mesmo.

```python
beta = model.slope
alpha = model.intercept
novo_apt_area = 225
preco = beta * novo_apt_area + alpha
preco
```

    996.0604238649662

Ou seja, quando um apartamento de 225$m^2$ entra no mercado o mesmo custa em torno de 1M de reais.

## Erros Normais

Agora, será que conseguimos chegar no mesmo pensando na regressão como um modelo probabilístico?

Ao invés de focar no valor quadrático do erro, vamos pensar no caso ideal. Qual seria o valor ideal do erro da minha regressão? ZERO! Porém, isso é possível somente quando temos relações lineares perfeitas (ex: x = altura em cm, y = altura em polegadas), o que não é o caso geral.

Nesse caso, vamos assumir que:
$$\epsilon_i \sim Normal(0, \sigma^2)$$

Isto é, cada um dos erros vem de uma distribuição normal com média 0. Ou seja, a média dos erros ainda vai ser uma normal centrada em 0. Além disso, o desvio padrão dos erros poderão ser estimados através dos dados após a regressão.

## Verossimilhança

Na regressão, queremos prever o valor de $y$ em função de $x$. No exemplo mostrado acima, queremos prever o preço de compra de um apartamento em BH em função da área.

Estamos assumindo que:
$$\epsilon_i = x_i - \beta x_i - \alpha$$

Ou seja, um erro na estimativa de um ponto não depende de outros.

Temos uma função de densidade de probabilidade para os erros:
$$\epsilon_i \sim Normal(0, \sigma^2)$$

Observe a função de densidade da Normal:
$$p(x|\mu, \sigma^2) = \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(x-\mu)^2}{2 \sigma^2}}$$

Logo:
$$p(\epsilon_i|\mu, \sigma^2) = \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(\epsilon_i-\mu)^2}{2 \sigma^2}}$$

Como no nosso modelo a média é zero, ficamos com:
$$p(\epsilon_i|\mu=0, \sigma^2) = \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{\epsilon_i^2}{2 \sigma^2}}$$
$$p(\epsilon_i|\sigma^2, \alpha, \beta) = \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}}$$

A probabilidade mostrada acima é a probabilidade de uma observação (observar o erro $\epsilon_i$) dado os parâmetros $\sigma^2$, $\alpha$ e $\beta$!
Isto é uma **verossimilhança**!

O gráfico abaixo mostra a distribuição de probabilidade para erros vindos de uma distribuição normal com média igual a zero e desvio padrão igual a um.

```python
x = np.linspace(-5, 5, 100)
plt.plot(x, ss.distributions.norm.pdf(x, scale=1))
plt.xlabel(r'$\epsilon_i$')
plt.ylabel(r'$p(\epsilon_i \mid \mu=0, \sigma=1)$')

despine()
```

Um modelo estatístico na mais é do que um conjunto de hipóteses que se supõem válidas para a distribuição de probabilidades das variáveis aleatórias medidas na amostra.

Em particular, aqui estamos falando de modelos paramétricos, onde assumimos que o erro é uma normal e assumimos uma função de densidade (como mostrado no gráfico acima).

Vamos focar numa base de 3 pontos. Além do mais, vamos assumir uma reta qualquer, dessa forma, o desvio padrão dos erros podem ser estimados através dos dados após a regressão. Se os erros vem de uma distribuição normal com função de densidade já conhecida, qual dos três erros abaixo é o mais verossímil?

Observe:
$$p(\epsilon_i|\sigma^2, \alpha, \beta) = \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}}$$

Quanto menor o erro, maior a probabilidade, sendo assim, o segundo erro é o mais verossímil.

```python
beta = 1
alpha = 1

fig = plt.figure(figsize=(36, 10))

x = np.array([2, 8, 5])
y = np.array([0, 1, 3])

plt.subplot(121)
plt.scatter(x, y, edgecolors='k', s=80, alpha=0.6)
plt.title('3 Pontinhos')
plt.ylabel(r'Y')
plt.xlabel(r'X')

y_bar = x * beta + alpha
plt.plot(x, y_bar, color='magenta')

y_min = [min(y_i, y_bar_i) for y_i, y_bar_i in zip(y, y_bar)]
y_max = [max(y_i, y_bar_i) for y_i, y_bar_i in zip(y, y_bar)]
plt.vlines(x, ymin=y_min, ymax=y_max, color='magenta', lw=1)

despine()

plt.subplot(122)
plt.title('PDF da Normal')
ei_x = np.linspace(-10, 10, 100)
sigma = (y - y_bar).std(ddof=1)
plt.plot(ei_x, ss.distributions.norm.pdf(ei_x, scale=sigma))
plt.xlabel(r'$\epsilon_i$')
plt.ylabel(r'$p(\epsilon_i \mid \mu=0, \sigma={})$'.format(np.round(sigma, 2)))
despine()
```

Observe o exemplo considerando agora todos os pontos do exemplo de predição de preço de apartamentos em BH.

```python
beta = 3.535719156333653
alpha = 200.52361368989432

fig = plt.figure(figsize=(36, 10))

x = df['area']
y = df['preco']

plt.subplot(121)
plt.scatter(x, y, edgecolors='k', s=80, alpha=0.6)
plt.title('Preco de Apartamentos em BH')
plt.ylabel(r'Preço * $10^3$ (R\$)')
plt.xlabel(r'Área ($M^2$)')

y_bar = x * beta + alpha
plt.plot(x, y_bar, color='magenta')

y_min = [min(y_i, y_bar_i) for y_i, y_bar_i in zip(y, y_bar)]
y_max = [max(y_i, y_bar_i) for y_i, y_bar_i in zip(y, y_bar)]
plt.vlines(x, ymin=y_min, ymax=y_max, color='magenta', lw=1)

despine()

plt.subplot(122)
plt.title('PDF da Normal')
ei_x = np.linspace(-1000, 1000, 100)
sigma = (y - y_bar).std(ddof=1)
plt.plot(ei_x, ss.distributions.norm.pdf(ei_x, scale=sigma))
plt.xlabel(r'$\epsilon_i$')
plt.ylabel(r'$p(\epsilon_i \mid \mu=0, \sigma={})$'.format(np.round(sigma, 2)))
despine()
```

## Erros Independentes

Nos exemplos acima discutimos sobre qual erro é o mais verossímil... Mas como podemos calcular a verossimilhança de todos os erros?

Ao assumir **independência**, estamos dizendo que:
$$p(E|\sigma^2, \alpha, \beta) = \prod_i p(\epsilon_i|\sigma^2, \alpha, \beta) = \prod_i p(\epsilon_i|\theta)$$
onde $E = \{e_1, e_2, ..., e_n\}$ e $\theta = \{\sigma^2, \alpha, \beta\}$

Acabamos de definir nossa primeira função de verossimilhança. Como podemos proceder agora? Maximizando a mesma! Ou seja, queremos os parâmetros ($\theta$) que melhor se ajustam os nossos dados.
Porém, maximizar produtórios é chato... O mundo das somas é mais bacana do que o mundo da multiplicação.

## Log-verossimilhança

Como o log é uma função monotonicamente crescente, maximizar o log(f(x)) é equivalente a maximizar f(x). Portanto vamos trabalhar no log.

Ao invés de maximizar a função de **verossimilhança** $L(\theta)$, vamos maximizar a função de **log-verossimilhança** $l(\theta)$:

$$L(\theta) = \prod_i p(\epsilon_i|\theta)$$
$$log(L(E|\theta)) = log(\prod_i p(\epsilon_i|\theta))$$
$$l(\epsilon_i|\theta) = \sum_i log(p(\epsilon_i|\theta))$$
uma vez que $log(A*B) = log(A) + log(B)$.

Substituindo os parâmetros $\sigma^2$, $\alpha$, $\beta$ nas funções e fazendo algumas manipulações, temos que a verossimilhança e a log-verossimilhança são dadas por:

$$L(E|\theta) = \prod_i \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}}$$

$$l(E|\theta) = -nlog(\sqrt{2\pi}) -nlog(\sigma) - \frac{\sum_i (y_i - \beta x_i - \alpha)^2}{2 \sigma^2}$$

* A partir da fórmula da verossimilhança, dada por:
$$L(E|\theta) = \prod_i \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}}$$
verifique que a expressão da log-verossimilhança é dada por:
$$l(E|\theta) = -nlog(\sqrt{2\pi}) -nlog(\sigma) - \frac{\sum_i (y_i - \beta x_i - \alpha)^2}{2 \sigma^2}$$

$$L(E|\theta) = \prod_i \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}}$$
$$log(L(E|\theta)) = log(\prod_i \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}})$$
$$l(\epsilon_i|\theta) = \sum_i log(\frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}})$$
uma vez que $log(A*B) = log(A) + log(B)$.
$$l(\epsilon_i|\theta) = \sum_i log(\frac{1}{\sqrt{\sigma^2 2 \pi}}) + \sum_i log(e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}})$$
$$l(\epsilon_i|\theta) = n log(\frac{1}{\sqrt{\sigma^2 2 \pi}}) + \sum_i log(e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}})$$
$$l(\epsilon_i|\theta) = n log(\frac{1}{\sqrt{\sigma^2 2 \pi}}) - \frac{\sum_i (y_i - \beta x_i - \alpha)^2}{2 \sigma^2}$$
uma vez que $log(A^x) = x log(A)$ e $log(e) = 1$.
$$l(\epsilon_i|\theta) = - n log(\sqrt{\sigma^2 2 \pi}) - \frac{\sum_i (y_i - \beta x_i - \alpha)^2}{2 \sigma^2}$$
uma vez que $log(\frac{1}{x}) = - log(x)$.
$$l(\epsilon_i|\theta) = - n log(\sqrt{2 \pi}) - n log(\sqrt{\sigma}) - \frac{\sum_i (y_i - \beta x_i - \alpha)^2}{2 \sigma^2}$$

Maximizar a log verossimilhança implica em achar os parâmetros que melhor se ajustam aos dados. Além disso, lembre-se que maximizar é achar o local onde a derivada é zero.

Para isso, devemos resolver as derivadas em relação a cada um dos parâmetros: $\sigma$, $\alpha$ e $\beta$.

$$\alpha = \bar{y} - \beta \bar{x}$$
$$\beta = \frac{ \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^n (x_i - \bar{x})^2 }$$
$$\sigma^2 = \frac{\sum_i (y_i - \beta x_i - \alpha)^2}{n}$$

Observe que ao maximizar $\sigma$, chegamos no estimador da variância. Além disso, o $\sigma$ não impacta os valores de $\alpha$ e $\beta$, mas é útil para entendermos os erros do estimador (é a variância dos erros afinal).

Por fim, note que maximizar a verossimilhança da regressão é o mesmo que minimizar os erros quadrados.

## Estimação da Máxima Verossimilhança e Mínimos Quadrados

Por que escolher os mínimos quadrados? Uma justificativa envolve a estimativa de máxima verossimilhança.

Imagine que temos uma amostra de dados $v_1, \cdots, v_n$ que vem de uma distribuição que depende de algum parâmetro desconhecido $\theta$:

$$p(v_1, \cdots, v_n~|~\theta)$$

Se não conhecêssemos theta, poderíamos nos virar e pensar nessa quantidade como a probabilidade de $\theta$ dada a amostra:

$$L(\theta~|~v_1, \cdots, v_n)$$

Sob essa abordagem, o mais provável $\theta$ é o valor que maximiza essa função de verossimilhança; isto é, o valor que torna os dados observados os mais prováveis. No caso de uma distribuição contínua, na qual temos uma função de distribuição de probabilidade e não uma função de massa de probabilidade, podemos fazer a mesma coisa.

De volta à regressão. Uma suposição que muitas vezes é feita sobre o modelo de regressão simples é que os erros de regressão são normalmente distribuídos com média $0$ e algum desvio padrão (conhecido) $\sigma$. Se esse for o caso, a probabilidade baseada em ver um par $(x_i, y_i)$ é:

$$L(\alpha, \beta~|~x_i, y_i, \sigma) = \frac{1}{\sqrt{2\pi\sigma}}\exp{\big(\frac{-(y_i-\alpha-\beta x_i)^2}{2\sigma^2}\big)}$$

A probabilidade baseada em todo o conjunto de dados é o produto das probabilidades individuais, que é maior precisamente quando alfa e beta são escolhidos para minimizar a soma dos erros quadrados. Ou seja, nesse caso (e com essas suposições), minimizar a soma dos erros quadrados é equivalente a maximizar a probabilidade dos dados observados.

## Qualidade da regressão

Após encontrar os parâmetros do modelo que melhor se ajustam aos dados (minimizam os erros quadrados, maximizam a verossimilhança), devemos avaliar a qualidade do ajuste. A qualidade da regressão pode ser analisada através de gráficos ou métricas, como o R-quadrado, estudado anteriormente.

## Gráfico residual

O gráfico residual é uma forma de visualizar os erros versus a variável preditora para verificar a pressuposição de que os erros são aleatoriamente distribuídos e têm variância constante. De maneira ideal, os pontos devem cair aleatoriamente em ambos os lados de 0, sem padrões reconhecíveis nos pontos. Como os erros são centrados em zero, ao plotar valor de x pelo erro queremos pontos igualmente dispersados  positivos e negativos.
Lembre-se da PDF da distribuição normal. A mediana e a média são iguais, ou seja, 50% dos erros se concentram acima de 0 e 50% abaixo.

Observe o gráfico residual para o exemplo de prever o preço de um apartamento usando apenas a área do mesmo. No eixo-x temos a área do aparatamento e no eixo-y o erro. O gráfico não parece ter um padrão (é o que queremos) já que os erros estão espalhados em ambos os lados da linha tracejada que marca o erro igual a 0. Entretanto, os erros aparentam ser piores quando a área dos apartamentos (variável x) aumenta.

```python
sns.residplot(x='area', y='preco', data=df,
              line_kws={'color':'magenta', 'lw':4},
              scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
plt.ylabel(r'$\epsilon_i$')
plt.xlabel(r'Área ($M^2$)')
despine()
```

## QQ Plot

Por fim, outra forma de verificar se o modelo é bom se chama QQ Plot.
Esse gráfico é útil para checar a adequação da distribuição de frequência dos dados à uma distribuição de probabilidades. No caso dos modelos de regressão, o QQ Plot é usado para verificar se os erros apresentam distribuição normal.

Uma forma de construir o QQ Plot consiste em ordenar os erros (eixo-y), comparando com o local esperado do mesmo no modelo (distribuição normal). Nesse caso, a mediana fica no centro do plot. O erro mediano é zero caso a normal seja verdadeira!

Outra forma de construir o QQ Plot consiste em pegar o valor z-norm que leva para a probabilidade de cada um dos erros ordenados no modelo e plotar os erros ordenados no eixo-y e os valores z-normalizados no eixo-x. O gráfico abaixo apresenta o QQ Plot do exemplo de prever o preço de um apartamento usando apenas a área do mesmo.

Idealmente quanto mais próximo de uma reta melhor, uma vez que a reta quer dizer que os erros são perfeitamente normais. Sendo assim, na regressão perfeita observamos uma linha reta no QQ plot.

```python
ss.probplot(y - y_bar, plot=plt.gca());
```

## Exercícios - Outros datasets

### Enunciado

Abaixo estão listados algumas conjuntos de dados contendo relações entre variáveis:

**Supernovas**: Atualmente, uma das teorias mais aceitas sobre a formação do universo, diz que o universo está em constante expansão. Supernovas são estrelas que explodiram e morreram recentemente. Os dados contêm registros dessas supernovas. Cada linha na tabela corresponde a uma supernova próxima da Terra observada por astrônomos, indicando o quão longe da Terra a supernova estava e o quão rápido ela se afastava.

**Stocks and Unemployment**: A Bolsa está diretamente ligada à economia do país e do mundo. Quando ela desaba, isso pode ter consequências negativas até no dia a dia de quem nem sabe como a Bolsa funciona. Por exemplo, o desemprego pode aumentar e a inflação, acelerar, tornando os produtos no supermercado mais caros. Com base nisso, os dados apresentam a evolução no preço de uma ação e a taxa de desemprego em determinado período.

**Stocks and Interest**: Além de relacionar o valor de ações na Bolsa com o índice de desemprego, é possível correlacionar o valor das ações e outros indicadores. Os dados apresentam a evolução no preço de uma ação e a taxa de juros em determinado período.

**Dugongs**: Os dados incluem informações de peixe-bois. Cada linha contém o comprimento e a idade de indivíduos. A base de dados é muito utilizada para fazer predições do comprimento desses animais de acordo com a idade.

- **Exercício:** Para cada um dos datasets você deverá:

    1. Carregar os dados do arquivo.
    2. Fazer uma regressão linear.
    3. Avaliar a qualidade do modelo através do cálculo do $R^2$ e dos gráficos de erros: Gráfico residual e QQ Plot.
    4. Verificar se os erros são independentes. Para isso, conte quantos erros são maiores e quantos são menores do que zero.

Antes de realizar o exercício, é útil você definir as seguintes funções com base nos parâmetros citados em cada uma:

Note que após definir as funções pedidas corretamente até o fim do primeiro dataset, os demais irão executar sem erros, não precisando ser alterados.

```python
def error(alpha, beta, x, y):
    return

def sum_of_squared_errors(alpha, beta, x, y):
    return

def total_sum_of_squares(y):
    return

def r_squared(alpha, beta, x, y):
    return
```

### 1. Supernovas

#### 1.1 Carregar os dados

```python
df = pd.read_csv('https://raw.githubusercontent.com/pedroharaujo/ICD_Docencia/master/close_novas.csv')
x = df['Distance (million parsecs)']
y = df['Speed (parsecs/year)']
```

#### 1.2 Regressão Linear

```python
def linear_regression():
  # YOUR CODE HERE
  # deve retornar uma tupla contendo os paramentros
  # (alpha, beta)
  return
```

```python
(a1, b1) = linear_regression(x,y)
assert_equal(round(a1, 6), 0.000167)
assert_equal(round(b1, 6), 0.000068)
```

#### 1.3 Avaliação do Modelo via R²

```python
# avaliacao do Rsquared
r2 = r_squared(a1, b1, x, y)
assert_equal(round(r2, 4), 0.9645)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir

```

```python
# GABARITO GRAFICO
# sns.residplot(x='Distance (million parsecs)', y='Speed (parsecs/year)', data=df,
#               line_kws={'color':'magenta', 'lw':4},
#               scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
# plt.ylabel(r'$\epsilon_i$')
# despine()
# plt.show()
```

#### 1.4 Verificando se os erros são independentes

```python
def error_verification(alpha, beta, x, y):
  # YOUR CODE HERE
  # deve retornar uma tupla com as contagens de erros maiores e menores que 0
  # (maiores, menores)
  return
```

```python
(maiores, menores) = error_verification(a1, b1, x, y)
assert_equal(maiores, 78)
assert_equal(menores, 78)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir
```

```python
# ss.probplot(y - y.mean(), plot=plt)
# despine()
# plt.show()
```

### 2. Stocks anad Unemployment

#### 2.1 Carregar os dados

```python
df = pd.read_csv('https://raw.githubusercontent.com/pedroharaujo/ICD_Docencia/master/stocks_unemployment.csv')
x = df['Unemployment_Rate']
y = df['Stock_Index_Price']
```

#### 2.2 Regressão Linear

```python
(a2, b2) = linear_regression(x,y)
assert_equal(round(a2, 6), 4471.339321)
assert_equal(round(b2, 6), -588.962076)
```

#### 2.3 Avaliação do Modelo via R²

```python
# avaliacao do Rsquared
r2 = r_squared(a2, b2, x, y)
assert_equal(round(r2, 4), 0.8507)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir

```

```python
# GABARITO GRAFICO
# sns.residplot(x='Unemployment_Rate', y='Stock_Index_Price', data=df,
#               line_kws={'color':'magenta', 'lw':4},
#               scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
# plt.ylabel(r'$\epsilon_i$')
# despine()
# plt.show()
```

#### 2.4 Verificando se os erros são independentes

```python
(maiores, menores) = error_verification(a2, b2, x, y)
assert_equal(maiores, 13)
assert_equal(menores, 11)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir
```

```python
# ss.probplot(y - y.mean(), plot=plt)
# despine()
# plt.show()
```

### 3. Stocks and Interest

#### 3.1 Carregar os dados

```python
df = pd.read_csv('https://raw.githubusercontent.com/pedroharaujo/ICD_Docencia/master/stocks_interest.csv')
x = df['Interest_Rate']
y = df['Stock_Index_Price']
```

#### 3.2 Regressão Linear

```python
(a3, b3) = linear_regression(x,y)
assert_equal(round(a3, 6), -99.464319)
assert_equal(round(b3, 6), 564.203892)
```

#### 3.3 Avaliação do Modelo via R²

```python
# avaliacao do Rsquared
r2 = r_squared(a3, b3, x, y)
assert_equal(round(r2, 4), 0.8757)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir

```

```python
# GABARITO GRAFICO
# sns.residplot(x='Interest_Rate', y='Stock_Index_Price', data=df,
#               line_kws={'color':'magenta', 'lw':4},
#               scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
# plt.ylabel(r'$\epsilon_i$')
# despine()
# plt.show()
```

#### 3.4 Verificando se os erros são independentes

```python
(maiores, menores) = error_verification(a3, b3, x, y)
assert_equal(maiores, 12)
assert_equal(menores, 12)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir
```

```python
# ss.probplot(y - y.mean(), plot=plt)
# despine()
# plt.show()
```

### 4. Dugongs

#### 4.1 Carregar os dados

```python
df = pd.read_csv('https://raw.githubusercontent.com/pedroharaujo/ICD_Docencia/master/dugongs.csv')
x = df['Age']
y = df['Length']
```

#### 4.2 Regressão Linear

```python
(a4, b4) = linear_regression(x,y)
assert_equal(round(a4, 6), 2.018286)
assert_equal(round(b4, 6), 0.028955)
```

#### 4.3 Avaliação do Modelo via R²

```python
# avaliacao do Rsquared
r2 = r_squared(a4, b4, x, y)
assert_equal(round(r2, 4), 0.6883)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir

```

```python
# GABARITO GRAFICO
# sns.residplot(x='Age', y='Length', data=df,
#               line_kws={'color':'magenta', 'lw':4},
#               scatter_kws={'edgecolor':'k', 's':80, 'alpha':0.8})
# plt.ylabel(r'$\epsilon_i$')
# despine()
# plt.show()
```

#### 4.4 Verificando se os erros são independentes

```python
(maiores, menores) = error_verification(a4, b4, x, y)
assert_equal(maiores, 16)
assert_equal(menores, 11)
```

```python
# avaliacao via grafico dos residuos (sem correcao automatica)

# seu codigo aqui
# gabarito a seguir
```

```python
# ss.probplot(y - y.mean(), plot=plt)
# despine()
# plt.show()
```

## Estimador de Máxima Verossimilhança

Conforme vimos, a verossimilhança representa a probabilidade dos nossos dados $X$ (ou erros $E$) condicionados em um modelo. O modelo aqui são os parâmetros $\theta$ e a função de probabilidade $p$.

$$X = \{x_1, x_2, ..., x_n\}$$

$$p(X|\theta) = \prod_i p(x_i|\theta)$$

No modelo que estudamos:

$\epsilon_i \sim Normal(0, \sigma^2)$

$p(\epsilon_i|\sigma^2, \alpha, \beta) = \frac{1}{\sqrt{\sigma^2 2 \pi}} e^{-\frac{(y_i - \beta x_i - \alpha)^2}{2 \sigma^2}}$

$E = \{e_1, e_2, ..., e_n\}$ e $\theta = \{\sigma^2, \alpha, \beta\}$

$p(E|\theta) = \prod_i p(\epsilon_i|\theta)$

Entretanto, podemos estimar os parâmetros de qualquer modelo probabilístico.

Imagine o seguinte caso, em que observamos variáveis categóricas $X$. $X$ corresponde ao efeito do remédio em um grupo de pessoas, onde cada $x_i$ assume valo igual a 1, caso o remédio tenha funcionado e 0 caso contrário. Por exemplo: $X = \{1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0\}$

Podemos modelar estes dados a partir de uma distribuição [Bernoulli](https://en.wikipedia.org/wiki/Bernoulli_distribution). Ou seja,  $x_i \sim Bernoulli(\theta)$ onde $\theta$ representa a probabilidade do remédio funcionar para o paciente (probabilidade de sucesso).

* Derive o estimador de máxima verossimilhança para a distribuição Bernoulli.

$$x_i \sim Bernoulli(\theta)$$

$$p(x_i|\theta) = \theta^{x_i} (1-\theta)^{(1 - x_i)}$$

Estimador de máxima verossimilhança:

$$L(X|\theta) = \prod_i p(x_i|\theta) = \prod_i \theta^{x_i} (1-\theta)^{(1 - x_i)} = \theta^{\sum_i x_i} (1-\theta)^{\sum_i (1 - x_i)}$$
lembrando que $a^x b^y * a^z b^w = a^{x+z} b^{y+w}$

Estimador de máxima log-verossimilhança:

$$l(X|\theta) = log(\prod_i p(x_i|\theta)) = \sum_i log(\theta^{x_i} (1-\theta)^{(1 - x_i)}) = \sum_i x_i log(\theta) + \sum_i (1-x_i) log(1-\theta)$$
lembrando que $log(ab) = log(a) + log(b)$ e $log(a^x) = xlog(a)$

* Após derivar o estimador de máxima verossimilhança para a distribuição Bernoulli encontre o valor de $\theta$ que maximiza a verossimilhança.

Assumimos que as observações são independentes, maximizar a log da verossimilhança corresponde a derivar essa função, encontrando o seu ponto de máximo. Igualando a derivada da log-verossimilhança encontramos os parâmetros ótimos:

$$\frac{\partial l(X|\theta)}{\theta} = \frac{\partial (\sum_i x_i log(\theta) + \sum_i (1-x_i) log(1-\theta))}{\partial \theta} = \frac{\sum_i x_i}{\theta} - \frac{\sum_i (1-x_i)}{(1-\theta)}$$
Lembrando que $\frac{\partial log(a)}{\partial a} = \frac{1}{a}$ e $\frac{\partial log(1-a)}{\partial a} = \frac{1}{(1-a)}$

Igualando a derivada a zero, encontramos o valor de $\theta$ que maximiza a log-verossimilhança:

$$\frac{\sum_i x_i}{\theta} - \frac{\sum_i (1-x_i)}{(1-\theta)} = 0$$
$$(1-\theta) \sum_i x_i = \theta \sum_i (1-x_i)$$
$$\sum_i x_i - \theta \sum_i x_i = n\theta - \theta \sum_i x_i$$
$$\sum_i x_i = n\theta$$
$$\theta = \frac{\sum_i x_i}{n}$$


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Delay Line
***

Referring to Figure 4.2, the frequency response of a delay line canceller is given by (Equation 11.5)

\begin{equation}
    \left| H(f)\right|^2 = 4\,\sin^2\left( \pi f \right),
\end{equation}
***

Begin by getting the library path

```python
import lib_path
```

Set up the normalized frequency space using the `linspace` routine from `scipy`

```python
from numpy import linspace

frequency = linspace(0, 4, 1000)
```

Calculate the frequency response for a single delay line canceller using the `delay_line` routine from `countermeasures`

```python
from Libs.ecm.countermeasures import delay_line

response = delay_line(frequency) / 4.0
```

Import the `matplotlib` routines to display the frequency response of the single delay line canceller

```python
from matplotlib import pyplot as plt
```

Import the `log10` and `finfo` routines for plotting the frequency response in deciBels

```python
from numpy import log10, finfo
```

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(frequency, 10 * log10(response + finfo(float).eps), '')

# Set the plot title and labels

plt.title('Delay Line Response', size=14)

plt.xlabel('Normalized Frequency (f / PRF)', size=12)

plt.ylabel('Amplitude (dB)', size=12)

# Set the y-axis lim

plt.ylim([-30, 1])

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the tick label size

plt.tick_params(labelsize=12)
```

Set the pulse repetition stagger for a stagger type delay line canceller

```python
prf_stagger = 5.0/4.0
```

Calculate the reponse for each PRF separately and then combine to get the final result

```python
response_prf1 = delay_line(frequency) / 4.0

response_prf2 = delay_line(prf_stagger * frequency) / 4.0

response = 0.5 * (response_prf1 + response_prf2)
```

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(frequency, 10 * log10(response_prf1 + finfo(float).eps), '', label='PRF 1')

plt.plot(frequency, 10 * log10(response_prf2 + finfo(float).eps), '--', label='PRF 2')

plt.plot(frequency, 10 * log10(response + finfo(float).eps), ':', label='PRF Staggered')

# Place the legend

plt.legend(loc='lower left', prop={'size': 10})

# Set the plot title and labels

plt.title('Delay Line Response', size=14)

plt.xlabel('Normalized Frequency (f / PRF)', size=12)

plt.ylabel('Amplitude (dB)', size=12)

# Set the y-axis lim

plt.ylim([-30, 1])

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the tick label size

plt.tick_params(labelsize=12)
```


```python
%matplotlib inline
%run ../../setup/nb_setup
```

# Orbits 2: Orbits in Axisymmetric Potentials

## Solutions to Exercises

```python
from astropy.constants import G
import astropy.units as u

import matplotlib as mpl
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import numpy as np

import gala.dynamics as gd
import gala.integrate as gi
import gala.potential as gp
from gala.units import galactic
```

```python
mn_disk = gp.MiyamotoNagaiPotential(
    m=6.98e10 * u.Msun, a=3 * u.kpc, b=0.28 * u.kpc, units=galactic
)
```

### Exercise: Defining an comparison Plummer model

As mentioned above, in a particular limit, the MN potential becomes the Plummer potential. What should we set the Plummer scale length to so that, in the midplane (z=0) it has the same profile as our MN potential? I.e. what combination of the MN scale parameters $a_\textrm{MN}$ and $b_\textrm{MN}$ should we set the Plummer scale length to?

Write the answer here:

Define the comparison Plummer model using the mass from the MN potential, and the correct combination of the MN scale parameters (you can use `mn_disk.parameters['m']`, `mn_disk.parameters['a']`, and `mn_disk.parameters['b']` to retrieve the parameter values so you don't have to re-define them).

*Solution*:

A Plummer model that is equivalent to the MN potential at its midplane (z=0) will have a Plummer scale length that is equal to the sum of the MN parameters:

$$
b_\textrm{Plummer} = a_\textrm{MN} + b_\textrm{MN}
$$

```python
plummer = gp.PlummerPotential(
    m=mn_disk.parameters["m"],
    b=mn_disk.parameters["a"] + mn_disk.parameters["b"],
    units=galactic,
)

xyz = [8, 0, 0] * u.kpc
Menc_MN = mn_disk.mass_enclosed(xyz)
Menc_plummer = plummer.mass_enclosed(xyz)
print(Menc_MN, Menc_plummer)
```

### Exercise: Why do the orbits in the Miyamoto–Nagai potential look different from the Plummer orbits?

Compute and plot the three components of the angular momentum for all orbits in both potentials. Do you see any differences? Given what we discussed about integrals of motion in the last tutorial, what do you think the connection is between the angular momentum components and the phase-space properties of the orbit?

*Solution*:

```python
mn_dt = 1.0 * u.Myr
mn_steps = 4000

mn_xyz = ([[8, 0, 0.0], [8, 0, 1.0], [8, 0, 10.0]] * u.kpc).T
mn_vcirc = mn_disk.circular_velocity(mn_xyz)
mn_vxyz = mn_vcirc[np.newaxis] * np.array([0.1, 1, 0])[:, np.newaxis]
mn_w0 = gd.PhaseSpacePosition(pos=mn_xyz, vel=mn_vxyz)

mn_orbits = mn_disk.integrate_orbit(mn_w0, dt=mn_dt, n_steps=mn_steps)
plummer_orbits = plummer.integrate_orbit(mn_w0, dt=mn_dt, n_steps=mn_steps)
```

```python
for orbits, name in zip([mn_orbits, plummer_orbits], ["Miyamoto-Nagai", "Plummer"]):
    L = orbits.angular_momentum()

    fig, axes = plt.subplots(1, 3, figsize=(15, 5), constrained_layout=True)
    for i, (ax, lbl) in enumerate(zip(axes, ["$L_x$", "$L_y$", "$L_z$"])):
        ax.plot(orbits.t, L[i])
        ax.set_ylabel(lbl)
        ax.set_xlabel(f"time [{orbits.t.unit:latex_inline}]")
    fig.suptitle(f"{name} potential", fontsize=22)
```

Comparing these two potentials: In the Miyamoto–Nagai potential, generic orbits out of symmetry plane do not conserve the $x$ or $y$ components of the angular momentum, but the $z$ component of the angular momentum is still conserved. Of the integrals of motion we have discussed so far, orbits in axisymmetric potentials therefore only conserve energy $E$ and the $z$-component of the angular momentum $L_z$. However, if we integrate these orbits for a long time, we would find that they fill a 3D subspace of the full 6D phase-space. This implies that there must be a third isolating integral of motion that reduces the dimensionality of the orbit. In the context of "classical" integrals of motion (like $E$ and $L_z$), this third integral of motion is often called...*the third integral*. There is no general closed-form expression for the third integral as a function of the phase-space coordinates.

### Exercise: Estimate the guiding center radius for an orbit in the Miyamoto–Nagai potential

Estimate the guiding center radius of a planar orbit in the MN potential (the `mn_disk` we defined above) with the initial conditions:

$$
(x,y,z) = (8.5, 0, 0.02)~\textrm{kpc}\\
(v_x,v_y,v_z) = (0, 168, 0)~\textrm{km}~\textrm{s}^{-1}\\
$$

Hint: you might find the root finder `scipy.optimize.root` useful!

Compute an orbit from these initial conditions, plot it in the meridional plane, and draw a vertical line on the plot at the location of the guiding center radius.

*Solution:*

```python
from scipy.optimize import root
```

```python
w0 = gd.PhaseSpacePosition(pos=[8.5, 0, 0.02] * u.kpc, vel=[0, 168, 0] * u.km / u.s)
```

```python
trial_L = w0.angular_momentum()
trial_Lz = trial_L[2]
```

```python
def root_func(Rg, pot, Lz):
    dPhi_dR = pot.gradient([Rg[0], 0, 0]).to_value(Lz.unit ** 2 / u.kpc ** 3)[0, 0]
    return dPhi_dR - Lz.value ** 2 / Rg[0] ** 3
```

```python
result = root(root_func, [8.0], args=(mn_disk, trial_Lz))
trial_Rg = result.x[0] * u.kpc
print(f"Guiding center radius: {trial_Rg.value:.2f} kpc")
```

```python
trial_orbit = mn_disk.integrate_orbit(
    w0, dt=0.1 * u.Myr, t1=0, t2=2 * u.Gyr, Integrator=gi.DOPRI853Integrator
)
```

```python
fig = trial_orbit.cylindrical.plot(["rho", "z"], labels=["$R$", "$z$"])
fig.axes[0].axvline(trial_Rg.value, color="tab:red")
```

### Exercise: Estimate (analytically) the radial, vertical, and azimuthal frequencies for an orbit at the Solar radius

The Sun is approximately at a radius of $R_\odot \approx 8.1~\textrm{kpc}$ in the Milky Way's disk. This region of the disk is still dominated (in mass) by the gravitational potential of the stars and gas, so we can neglect the dark matter halo for first approximation.

Assuming $R_g = R_\odot$, estimate the azimuthal frequency of a circular orbit at the Solar circle using the `mn_disk` potential we defined above.

*Solution:*

```python
sun_Rg = 8.1 * u.kpc
sun_vcirc = mn_disk.circular_velocity([1, 0, 0] * sun_Rg)
sun_Lz = sun_Rg * sun_vcirc[0]
```

```python
sun_Omega_phi = sun_Lz / sun_Rg ** 2
sun_Omega_phi = sun_Omega_phi.to(u.rad / u.Gyr, u.dimensionless_angles())
sun_Omega_phi
```

Recall that the expression for the effective potential for a Miyamoto–Nagai disk is:

$$
\Phi_\textrm{eff}(R, z) = - \frac{G \, M}{\sqrt{R^2 + (a + \sqrt{b^2 + z^2})^2}} + \frac{L_z^2}{2\,R^2}
$$

Estimate the radial and vertical frequencies for an orbit near the solar circle. How do the frequency and period values compare (radial to azimuthal to vertical)?

*Hint: you either want to take second derivatives of the expression above and evaluate this manually, or you can use the `.hessian()` method on any Gala potential object (but note that this computes the Cartesian 2nd derivative matrix at a specified position)*

*Solution:*

One way to solve for the other frequencies is to take the derivatives of the effective potential expression above using Wolfram Alpha, or Mathematica, or Sympy, and evaluate them at $(R, z) = (R_g, 0)$:

```python
import sympy as sy

_G, _M, _R, _z, _a, _b, _Lz, _Rg = sy.symbols(
    "G, M, R, z, a, b, L_z, R_g", real=True, positive=True
)

MN_Phi_eff = -_G * _M / sy.sqrt(
    _R ** 2 + (_a + sy.sqrt(_z ** 2 + _b ** 2)) ** 2
) + _Lz ** 2 / (2 * _R ** 2)
```

```python
sy.simplify(sy.diff(MN_Phi_eff, _R, 2).subs({_R: _Rg, _z: 0}))
```

```python
sy.simplify(sy.diff(MN_Phi_eff, _z, 2).subs({_R: _Rg, _z: 0}))
```

However, Gala Potential objects can also compute the matrix of 2nd derivatives (the Hessian) using the `.hessian()` method. If we evaluate this at a position along the $x$ axis, the $\partial^2/\partial^2 x$ term is equivalent to the $\partial^2/\partial^2 R$ term (but we have to remember to add the term to make it the derivative of the *effective* potential):

```python
mn_Hess = mn_disk.hessian([1, 0, 0] * sun_Rg)[..., 0]
mn_d2Phi_dR2 = mn_Hess[0, 0]
mn_d2Phi_dz2 = mn_Hess[2, 2]

mn_d2Phi_eff_dR2 = mn_d2Phi_dR2 + 3 * sun_Lz ** 2 / sun_Rg ** 4
```

```python
sun_Omega_R = np.sqrt(mn_d2Phi_eff_dR2).to(u.rad / u.Gyr, u.dimensionless_angles())
sun_Omega_z = np.sqrt(mn_d2Phi_dz2).to(u.rad / u.Gyr, u.dimensionless_angles())
```

```python
print(f"Radial frequency: {sun_Omega_R:.1f}")
print(f"Azimuthal frequency: {sun_Omega_phi:.1f}")
print(f"Vertical frequency: {sun_Omega_z:.1f}")
```

```python
print(f"Radial period: {(2*np.pi*u.rad / sun_Omega_R).to(u.Myr):.1f}")
print(f"Azimuthal period: {(2*np.pi*u.rad / sun_Omega_phi).to(u.Myr):.1f}")
print(f"Vertical period: {(2*np.pi*u.rad / sun_Omega_z).to(u.Myr):.1f}")
```

These values are approximate and neglect the mass from the dark matter, but the ranking is typical for a typical star in the Galactic disk: The vertical frequency is much faster than the radial and azimuthal frequencies, which can have comparable values.

### Exercise: Are there values for the parameters (q, r_c) that lead to unphysical density values?

*Solution:*

Yes! Consider positions along $R=0$: The density is proportional to

$$
\begin{align}
\rho_L(0, z) &\propto (2\,q^2 + 1)r_c^2 + (2 - \frac{1}{q^2})\,z^2\\
    &\propto (2\,q^2 + 1)r_c^2 + 2\,z^2 - \frac{1}{q^2}\,z^2
\end{align}
$$

so, when

$$
\frac{1}{q^2}\,z^2 > (2\,q^2 + 1)r_c^2 + 2\,z^2
$$

the mass density will be negative! Assuming $r_c = 1$,

```python
z = np.linspace(0, 10, 256)
for q in np.linspace(0.5, 1, 8):
    plt.plot(
        z,
        2 * q ** 2 + 1 + 2 * z ** 2 - z ** 2 / q ** 2,
        marker="",
        label=f"$q={q:.2f}$",
    )
plt.axhline(0, color="#aaaaaa")
plt.ylim(-10, 10)
plt.legend()
plt.xlabel("$z$")
plt.ylabel(r"$2\,q^2 + 1 + 2\,z^2 - \frac{z^2}{q^2}$")
```


```python
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 3: NumPy

NumPy is a powerful Python module for scientific computing. Among other things, NumPy defines an N-dimensional array object that is especially convenient to use for plotting functions and for simulating and storing time series data. NumPy also defines many useful mathematical functions like, for example, the sine, cosine, and exponential functions and has excellent functions for probability and statistics including random number generators, and many cumulative density functions and probability density functions.

## Importing NumPy

The standard way to import NumPy so that the namespace is `np`. This is for the sake of brevity.

```python

```

## NumPy arrays

A NumPy `ndarray` is a homogeneous multidimensional array. Here, *homogeneous* means that all of the elements of the array have the same type. An `nadrray` is a table of numbers (like a matrix but with possibly more dimensions) indexed by a tuple of positive integers. The dimensions of NumPy arrays are called axes and the number of axes is called the rank. For this course, we will work almost exclusively with 1-dimensional arrays that are effectively vectors. Occasionally, we might run into a 2-dimensional array.

### Basics

The most straightforward way to create a NumPy array is to call the `array()` function which takes as an argument a `list`. For example:

```python
# Create a variable called a1 equal to a numpy array containing the numbers 1 through 5

# Find the type of a1

# Find the shape of a1

# Use ndim to find the rank or number of dimensions of a1

```

```python
# Create a variable called a2 equal to a 2-dimensionl numpy array containing the numbers 1 through 4

# Use ndim to find the rank or number of dimensions of a2

# Find the shape of a2

```

```python
# Create a variable called c an empty numpy array

# Use ndim to find the rank or number of dimensions of a3

# Find the shape of a3

```

### Special functions for creating arrays

Numpy has several built-in functions that can assist you in creating certain types of arrays: `arange()`, `zeros()`, and `ones()`. Of these, `arrange()` is probably the most useful because it allows you a create an array of numbers by specifying the initial value in the array, the maximum value in the array, and a step size between elements. `arrange()` has three arguments: `start`, `stop`, and `step`:

    arange([start,] stop[, step,])

The `stop` argument is required. The default for `start` is 0 and the default for `step` is 1. Note that the values in the created array will stop one increment *below* stop. That is, if `arrange()` is called with `stop` equal to 9 and `step` equal to 0.5, then the last value in the returned array will be 8.5.

```python
# Create a variable called b that is equal to a numpy array containing the numbers 1 through 5

```

```python
# Create a variable called c that is equal to a numpy array containing the numbers 0 through 10

```

The `zeros()` and `ones()` take as arguments the desired shape of the array to be returned and fill that array with either zeros or ones.

```python
# Construct a 1x5 array of zeros

```

```python
# Construct a 2x2 array of ones

```

### Math with NumPy arrays

A nice aspect of NumPy arrays is that they are optimized for mathematical operations. The following standard Python arithemtic operators `+`, `-`, `*`, `/`, and `**` operate element-wise on NumPy arrays as the following examples indicate.

```python
# Define three 1-dimensional arrays. PROVIDED
A = np.array([2,4,6])

```

```python
# Multiply A by a constant

```

```python
# Exponentiate A

```

```python
# Add  A and B together

```

```python
# Exponentiate A with B

```

```python
# Add A and C together

```

The error in the preceding example arises because addition is element-wise and `A` and `C` don't have the same shape.

```python
# Compute the sine of the values in A

```

### Iterating through Numpy arrays

NumPy arrays are iterable objects just like lists, strings, tuples, and dictionaries which means that you can use `for` loops to iterate through the elements of them.

```python
# Use a for loop with a NumPy array to print the numbers 0 through 4

```

### Example: Basel problem

One of my favorite math equations is:

\begin{align}
\sum_{n=1}^{\infty} \frac{1}{n^2} & = \frac{\pi^2}{6}
\end{align}

We can use an iteration through a NumPy array to approximate the lefthand-side and verify the validity of the expression.

```python
# Set N equal to the number of terms to sum

# Initialize a variable called summation equal to 0

# loop over the numbers 1 through N

# Print the approximation and the exact solution

```

## Random numbers and statistics

NumPy has many useful routines for generating draws from probability distributions. Random number generation is useful for computing dynamic simulations of stochastic economies. NumPy also has some routines for computing basic summary statistics (min, max, mean, median, variance) for data in an array.

For more advanced applications involving probability distributions, use SciPy's `scipy.stats` module (https://docs.scipy.org/doc/scipy-0.16.1/reference/stats.html). And for estimating statisticsl models (e.g., OLS models) use StatsModels (https://www.statsmodels.org/stable/index.html).

```python
# Use the help function to view documentation on the np.random.uniform function

```

```python
# Print a random draw from the uniform(0,1) probability distribution.

```

In the previous example, everyone will obtain different output for `np.random.uniform()`. However, we can set the *seed* for NumPy's PRNG (pseudorandom number generator) with the function `np.random.seed()`. Setting the seed before generating a random sequence can ensure replicability of results.

```python
# Set the seed for the NumPy random number generator to 271828

# Print a random draw from the uniform(0,1) probability distribution.

```

### Example: Random Draws from $\text{uniform}(-1,1)$ Distribution

Create a sample of 200 draws from the $\text{uniform}(-1,1)$ distribution. Compute some summary statistics. Plot the sample.

```python
# Set the seed for the NumPy random number generator to 314159

# Create a variable called 'uniform_data' containing 200 random draw from the uniform(-1,1) probability distribution.

# Print the variable data

```

```python
# Print the mean of the values in variable uniform_data

# Print the median of the values in variable uniform_data

# Print the standard deviation of the values in variable uniform_data

# Print the maximum of the values in variable uniform_data

# Print the index value of the arg max of the values in variable uniform_data

```

```python
# Create a variable called index_max equal to the argmax of data. Print the corresponding element of uniform_data

```

```python
# Plot random sample

```

### Example: Random Draws from $\text{normal}\left(0,\frac{1}{3}\right)$ Distribution

Create a sample of 200 draws from a normal distribution with mean 0 and variance $\frac{1}{3}$. Note that the variance of the $\text{uniform}(-1,1)$ distribution from the previous example has the same variance See: https://en.wikipedia.org/wiki/Uniform_distribution_(continuous).

Compute some summary statistics. Plot the sample.

```python
# Use the help function to view documentation on the np.random.normal function

```

```python
# Set the seed for the NumPy random number generator to 314159

# Create a variable called 'normal_data' containing 200 random draw from the normal(0,1/3) probability distribution.

# Print the variable normal_data

```

```python
# Print the mean of the values in variable normal_data

# Print the median of variable the values in normal_data

# Print the standard deviation of the values in variable normal_data

# Print the maximum of the values in variable normal_data

# Print the index value of the arg max of the values in variable normal_data

```

```python
# Create a variable called 'index_max' equal to the argmax of data. Print corresponding element of normal_data

```

```python
# Plot the random sample

```

```python
# Plot the uniform and normal samples together on the same axes

```


# Linear transformations with images
In this module we will work with linearly aligning images.  We'll apply transformations to images, and estimate optimal transformations.  Here problems will be nonlinear, and we'll need to use iterative methods.

```python
import numpy as np
%matplotlib notebook
import matplotlib as mpl
import matplotlib.pyplot as plt
import nibabel as nib
import os
import summerschool as ss
import scipy.interpolate as spi
```

## Load some data

```python
fname = os.path.join('mouse_images','PMD2052_orig_target_STS_clean.img')
img = nib.load(fname)
# load the image
I = img.get_data()[:,:,:,0] # note last axes is time, we'd like to remove it
# downsample so it can run on people's laptops
down = 3
I = ss.downsample_image(I,down)
# get voxeldx = img.header['pixdim'][1:4]*down voxel size and number of voxels
# note all the images in this dataset are in MNI space and are the same size and shape
nx = I.shape
dx = img.header['pixdim'][1:4]*down
# set up a domain
x0 = np.arange(nx[0])*dx[0]
x1 = np.arange(nx[1])*dx[1]
x2 = np.arange(nx[2])*dx[2]
# lets define the origin to be in the middle of our image
# linear transformations will be more reasonable this way
x0 = x0 - np.mean(x0)
x1 = x1 - np.mean(x1)
x2 = x2 - np.mean(x2)
X0,X1,X2 = np.meshgrid(x0,x1,x2,indexing='ij')
```

```python
ss.imshow_slices(x0,x1,x2,I)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.image.AxesImage at 0x7f2653a43518>,
     <matplotlib.image.AxesImage at 0x7f2653a43860>,
     <matplotlib.image.AxesImage at 0x7f2653a43b38>]

## Transformations act on images through their inverse

## Images are deformed by sampling them at new points: interpolation

```python
interp_args = {
    'method':'linear', # this says how we'd like to do interpolation, linear is fast and smooth
    'bounds_error':False, # if we try to sample outside the image, do not raise an error
    'fill_value':0 # if we try to sample outside the image, we'll return the value 0
}
```

```python
# A 2D example image
I2d = np.array(I[:,:,I.shape[-1]//2])
X02d = X0[:,:,0]
X12d = X1[:,:,0]
```

```python
# make a figure
nplots = 5
f,ax = plt.subplots(2, nplots, sharex=True, sharey=True)
for i in range(nplots):
    ax_ = ax[:,i]
    # choose a transformation
    if i == 0:
        A = np.eye(3)
        titlestring = 'identity'
    elif i == 1:
        T = np.random.randn(2)*2.0
        A = np.array([[1,0,T[0]],[0,1,T[1]],[0,0,1]])
        titlestring = 'translation'
    elif i == 2:
        theta = np.random.rand()*2.0*np.pi/6.0
        A = np.eye(3)
        A[:2,:2]= [[np.cos(theta),-np.sin(theta)],[np.sin(theta),np.cos(theta)]]
        titlestring = 'rotation'
    elif i==3:
        s = np.random.randn()*0.1 + 1.0
        A = np.eye(3)
        A[:2,:2] = np.eye(2)*s
        titlestring = 'scale'
    elif i == 4:
        A = np.eye(3)
        A[:2,:2] += np.random.randn(2,2)*0.2
        titlestring = 'linear'

    # make a transformed grid
    AX0 = A[0,0]*X02d + A[0,1]*X12d + A[0,2]
    AX1 = A[1,0]*X02d + A[1,1]*X12d + A[1,2]
    stride = 5
    ss.plot_grid(AX1, AX0, ax=ax_[0],
                 rstride=stride, cstride=stride,
                 color='r', linewidth=0.5, alpha=0.5)

    # make a transformed image
    B = np.linalg.inv(A)
    X0s = B[0,0]*X02d + B[0,1]*X12d + B[0,2]
    X1s = B[1,0]*X02d + B[1,1]*X12d + B[1,2]
    AI = spi.interpn([x0,x1],I2d,np.stack([X0s,X1s],axis=-1),**interp_args)
    ax_[0].imshow(AI, cmap='gray',
                  extent=[x1[1],x1[-1],x0[0],x0[-1]], origin='lower') # make sure you set extent and origin!

    ax_[0].set_aspect('equal')
    ax_[0].set_title(titlestring)

    # let's show the vector field
    ax_[1].quiver(X12d[::stride,::stride], X02d[::stride,::stride],
                  AX1[::stride,::stride]-X12d[::stride,::stride],
                  AX0[::stride,::stride]-X02d[::stride,::stride])
    ax_[1].set_aspect('equal')
    ax_[1].set_title('vector field')
f.suptitle('Illustration of several matrix transformations in 2D')
```

    <IPython.core.display.Javascript object>

    Text(0.5,0.98,'Illustration of several matrix transformations in 2D')

```python
# construct a transformation
A = np.eye(4)
A[:3,:3] += np.random.randn(3,3)*0.05 # a random deformation
A[:3,-1] += np.random.randn(3)*1.0 # a random shift
def sample_points_from_affine(X0,X1,X2,A):
    # find its inverse, using homogeneous coordinates
    B = np.linalg.inv(A)
    # get the sample points by matrix multiplication
    X0s = B[0,0]*X0 + B[0,1]*X1 + B[0,2]*X2 + B[0,3]
    X1s = B[1,0]*X0 + B[1,1]*X1 + B[1,2]*X2 + B[1,3]
    X2s = B[2,0]*X0 + B[2,1]*X1 + B[2,2]*X2 + B[2,3]
    return X0s,X1s,X2s
# this is defined in the summer school module
X0s,X1s,X2s = ss.sample_points_from_affine(X0,X1,X2,A)
AI = spi.interpn(
    [x0,x1,x2], # this labels where in space each voxel is
    I, # this says the intensity of our image at each voxel
    np.stack([X0s,X1s,X2s],axis=-1), # this says where we'd like to evaluate the image
    **interp_args # some more arguments that we defined above
)
f,ax=plt.subplots(2,3)
plt.subplots_adjust(wspace=0.45,hspace=0.35)
ss.imshow_slices(x0,x1,x2,I,ax[0])
for a in ax[0]:
    a.set_title('before')
    a.set_xlabel('')
ss.imshow_slices(x0,x1,x2,AI,ax[1])
for a in ax[1]: a.set_title('random transform')
Irandom = AI
```

    <IPython.core.display.Javascript object>

## Computing optimal transformations
As with points, we will write down a sum of square error cost
\begin{align*}
E(A) = \int_X \frac{1}{2}(I(A^{-1}x) - J(x))^2 dx
\end{align*}

And as before we'll take it's gradient with respect to $A$.  Since $A$ is a matrix, we'll consider an arbitrary perturbation to $A$, $A\mapsto A + \epsilon \delta A$, and take the gradient with respect to $\epsilon$ for any perturbation.
\begin{align*}
E(\epsilon) = \int_X \frac{1}{2}(I( (A + \epsilon \delta A)^{-1}x) - J(x))^2 dx
\end{align*}
We will use the fact that
\begin{align}
\frac{d}{d\epsilon}E(\epsilon) \bigg|_{\epsilon = 0} = \text{trace} \nabla E^T \delta A
\end{align}
This is just the definition of the directional derivative, as the gradient dot the directoin.

We are interested in
\begin{align*}
\frac{d}{d\epsilon}E(\epsilon) \bigg|_{\epsilon = 0}
\end{align*}

Let's attack this expression using the chain rule, first attack the square term
\begin{align*}
&= \int_X (I((A + \epsilon \delta A)^{-1}x) - J(x)) \frac{d}{d\epsilon} I((A + \epsilon \delta A)^{-1}x)dx \bigg|_{\epsilon = 0}\\
&=\int_X (I(A^{-1}x) - J(x)) \frac{d}{d\epsilon} I((A + \epsilon \delta A)^{-1}x)dx \bigg|_{\epsilon = 0}\\
\end{align*}
Now attack the image
\begin{align*}
&= \int_X (I(A^{-1}x) - J(x))  DI((A + \epsilon \delta A)^{-1}x) \frac{d}{d\epsilon} (A + \epsilon \delta A)^{-1}x dx \bigg|_{\epsilon = 0}\\
&= \int_X (I(A^{-1}x) - J(x))  DI(A^{-1}x) \frac{d}{d\epsilon} (A + \epsilon \delta A)^{-1}\bigg|_{\epsilon = 0} x dx \\
\end{align*}
Now the final term depends on taking the derivative of the inverse of a matrix.  Consider an arbitrary matrix $M$ that is a function of a parameter $t$.  We can work out the derivative of the inverse by
\begin{align*}
\frac{d}{dt}M^{-1}(t) &=\frac{d}{dt}\left( M^{-1}(t)M(t)M^{-1}(t) \right)\\
&= \frac{d}{dt}M^{-1}(t) M(t) M^{-1}(t) + M^{-1}(t) \frac{d}{dt}M(t) M^{-1}(t) + M^{-1}(t)M(t) \frac{d}{dt}M^{-1}(t)\\
 &=\frac{d}{dt}\left( M^{-1}(t)M(t)M^{-1}(t) \right)\\
&= \frac{d}{dt}M^{-1}(t) + M^{-1}(t) \frac{d}{dt}M(t) M^{-1}(t) + \frac{d}{dt}M^{-1}(t)\\
\end{align*}
Rearranging gives
\begin{align*}
\frac{d}{dt}M^{-1}(t) = - M^{-1}(t) \frac{d}{dt}M(t) M^{-1}(t)
\end{align*}
So in our problem we have
\begin{align*}
\frac{d}{dt}(A + \epsilon \delta A)^{-1} &= -(A + \epsilon \delta A)^{-1} \delta A (A + \epsilon \delta A)^{-1}
\end{align*}
Plugging this in with $\epsilon = 0$ gives the result
\begin{align*}
\frac{d}{d\epsilon}E(\epsilon) \bigg|_{\epsilon = 0} = -\int_X (I(A^{-1}x) - J(x))  DI(A^{-1}x) A^{-1}\delta A A^{-1} x dx
\end{align*}
Finally we can simplify this by recalling that $D[I(A^{-1}x)] = DI(A^{-1}x)A^{-1}$, giving
\begin{align*}
= -\int_X (I(A^{-1}x) - J(x))  D[I(A^{-1}x)]\delta A A^{-1} x dx
\end{align*}
Finally to work out the gradient with respect to $A$, we have to write this expression as a gradient, dot $\delta A$.  Taking the dot product on matrices using the trace gives
\begin{align*}
&= -\int_X \text{trace}(I(A^{-1}x) - J(x))  D[I(A^{-1}x)]\delta A A^{-1} x dx \\
&= -\int_X \text{trace}A^{-1} x (I(A^{-1}x) - J(x))  D[I(A^{-1}x)] dx \delta A
\end{align*}
which gives a gradient of
\begin{align*}
-\int_X (I(A^{-1}x)-J) \nabla[I(A^{-1}x)](A^{-1}x)^Tdx
\end{align*}

We will use this to build a gradient descent algorithm.  This is an iterative method, where we update date our current guess of $A$ by taking a "small step downhill", i.e. a small step in the direction opposite to the gradient.

```python
# load a second image to map to
fname = os.path.join('mouse_images','PMD3097_orig_target_STS_clean.img')
img = nib.load(fname)
J = img.get_data()[:,:,:,0] # note last axes is time, we'd like to remove it
J = ss.downsample_image(J,down)
f,ax = plt.subplots(2,3)
ss.imshow_slices(x0,x1,x2,J,ax[0])
'''
# note that these human MRI images have already been affine aligned into MNI space
# let's add a small affine transformation and then try to recover it
A = np.eye(4)
A[:3,:3] += np.random.randn(3,3)*0.05 # a random deformation
A[:3,-1] += np.random.randn(3)*2.0 # a random shift
X0s,X1s,X2s = ss.sample_points_from_affine(X0,X1,X2,A)
J = spi.interpn([x0,x1,x2],J,np.stack([X0s,X1s,X2s],axis=-1),**interp_args)
ss.imshow_slices(x0,x1,x2,J,ax[1])
'''
```

    <IPython.core.display.Javascript object>

    "\n# note that these human MRI images have already been affine aligned into MNI space\n# let's add a small affine transformation and then try to recover it\nA = np.eye(4)\nA[:3,:3] += np.random.randn(3,3)*0.05 # a random deformation\nA[:3,-1] += np.random.randn(3)*2.0 # a random shift\nX0s,X1s,X2s = ss.sample_points_from_affine(X0,X1,X2,A)\nJ = spi.interpn([x0,x1,x2],J,np.stack([X0s,X1s,X2s],axis=-1),**interp_args)\nss.imshow_slices(x0,x1,x2,J,ax[1])\n"

```python
# initialize a transformation
A = np.eye(4)
# choose a gradient descent step size
# note that the linear part and the translation part often need to be on very different scales
# for human
epsilonL = 1.0e-12
epsilonT = 1.0e-9
# for mouse
epsilonL = 1.0e-8
epsilonT = 1.0e-7

niter = 50
EAll = []
f,ax = plt.subplots(3,3)
plt.subplots_adjust(wspace=0.45,hspace=0.45)
for it in range(niter):
    # find the deformed image
    X0s,X1s,X2s = ss.sample_points_from_affine(X0,X1,X2,A)
    AI = spi.interpn([x0,x1,x2],I,np.stack([X0s,X1s,X2s],axis=-1),**interp_args)
    ss.imshow_slices(x0,x1,x2,AI,ax[0])

    # get the error
    err = AI-J
    ss.imshow_slices(x0,x1,x2,err,ax[1])
    for a in ax[1]: a.set_xlabel('')

    # find the gradient
    AI_0,AI_1,AI_2 = np.gradient(AI,dx[0],dx[1],dx[2])

    # calculate energy
    E = np.sum(err**2*0.5)*np.prod(dx)
    EAll.append(E)
    ax[2,0].cla()
    ax[2,0].plot(EAll)
    ax[2,0].set_title('cost')
    ax[2,0].set_xlabel('iteration')
    # find the gradient of the linear part
    gradL = np.empty((3,3))
    for i,AI_i in enumerate([AI_0,AI_1,AI_2]):
        for j,AX_j in enumerate([X0s,X1s,X2s]):
            gradL[i,j] = -np.sum(err*AI_i*AX_j)*np.prod(dx)
    # find the gradient of the translation part
    gradT = np.empty(3)
    for i,AI_i in enumerate([AI_0,AI_1,AI_2]):
        gradT[i] = -np.sum(err*AI_i)*np.prod(dx)
    # update
    A[:3,:3] -= epsilonL*gradL
    A[:3,-1] -= epsilonT*gradT

    f.canvas.draw()

# let's write out the transformation for the next tutorial
with open('affine.txt','wt') as f:
    for i in range(4):
        for j in range(4):
            f.write('{} '.format(A[i,j]))
        f.write('\n')

```

    <IPython.core.display.Javascript object>

## Rigid transformations
Above we considered perturbations of the form $A \mapsto \epsilon \delta A$.

With rigid transforms we cannot add like this.

But we know that small perturbations to the rotation group correspond to antisymmetric matrices.  We can write
\begin{align*}
A \mapsto \exp(\epsilon \delta A) A \simeq (Id + \epsilon \delta A)A
\end{align*}
where $\exp$ is the matrix exponential.  Repeating the above derivations with this analysis can tell us how to optimize over the rotation group.


Pipeline Collapse thickness calculation (APIRP1111).

The collapse pressure of the pipe shall exceed the net external pressure everywhere along the pipeline

```python
from pipedata import*  # imports all input data from the module pipedata.py
import sympy as sp
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Range of water depth
WD = np.linspace(WD_min,WD_max,10)

# Derated yield stress as a function of design temperature
def fy(Td):
    if Td < 121:
        fy = 1
    elif 121 < Td <= 149:
        fy = (Td -121)*(-0.033/28) + 1
    elif 149 < Td <= 177:
        fy = (Td -121)*(-0.033/28) + 0.967
    elif 177 < Td <= 204:
        fy = (Td -177)*(-0.033/27) + 0.933
    else:
        fy = (Td -204)*(-0.033/28) + 0.9
    return fy*SMYS

# Collapse thickness calculation
def coll(WD):
    tc = sp.symbols('tc',real=True)
    Pe = (rho_sw*g*WD)                         # external pressure calculation
    Pc = (Pe - Pmin)/f0                        # collapse pressure calculation
    Pel = 2*Es*(tc/D)**3/(1-v**2)              # elastic collapse pressure calculation
    Py = 2*fy(Td)*(tc/D)                       # yield collapse pressure calculation

    col = ((Pel*Py)/sp.sqrt(Pel**2+Py**2)) - Pc
    sol = sp.solve(col,check=False)[2]            # solution for thickness
    return sol

# get the thickness for the range of water depths,add CA(corrosion allowance) for shutdown case
tc = list(round((coll(WD)+CA)*1000,2) for WD in np.linspace(WD_min,WD_max,10))

D_t = list(round(D/(coll(WD)+CA),3) for WD in np.linspace(WD_min,WD_max,10)) # D/t ratio
print('max collapse thickness=',round(max(tc),2),'mm')
print('min collapse thickness=',round(min(tc),3),'mm')

#Write result to file
def write_to_file():
    pd.set_option('display.max_rows', 200) # increase no.of rows to a maximum of 200
    data = pd.DataFrame({'Water depth[m]':np.round(WD,decimals=2),'Thickness[mm]':tc,'D/tc ratio':D_t})
    print(data)
    file = open('Collapse_result.txt','w')
    file.write(str(data))
    return file.close()
write_to_file()

# Plots
def plots():
    #plot wall thickness vs. WD
    plt.plot(WD,tc,label='wall thickness',color='g')
    plt.title('WATER DEPTH vs.WALL THICKNESS')
    plt.xlabel('WD [m]')
    plt.ylabel('tc [m]')
    plt.savefig('fig A.water depth vs.Collapse thickness.png',dpi=100)
    plt.legend()
    plt.show()

    #plot D/t ratio vs collapse pressure vs. D/tc
    plt.plot(D_t,((rho_sw*g*WD) - Pmin)*10**-6/f0,label='wall thickness',color='r')
    plt.title('D/t vs.COLLAPSE PRESSURE' )
    plt.xlabel('D/t [m]')
    plt.ylabel('Pc [MPa]')
    plt.savefig('fig B.png',dpi=100)
    plt.legend()
    plt.show()
plots()
```


# Variational Autoencoder for the not-MNIST dataset

We will use a probabilisitc non-linear generative model for the not-MNIST dataset. Unlike the localization example, we will train both the **generative model parameters** and the parameters of the **amortized variaitonal family**.

----

---

----

---

## ELBO lower-bound to $p(\mathbf{X})$

Where $\text{KL}(q_{\eta,\mathbf{x}}(\mathbf{z})||p(\mathbf{z}))$ is known in closed form since it is the KL divergence between two Gaussian pdfs:

\begin{align}
\text{KL}(q_{\eta,\mathbf{x}}(\mathbf{z})||p(\mathbf{z})) = \frac{1}{2} \left[\text{tr}\left(\text{diag}(\sigma_{\eta}(\mathbf{x}))\right)+\left(\mu_{\eta}(\mathbf{x})^T\mu_{\eta}(\mathbf{x})\right)-2-\log\det \left(\text{diag}(\sigma_{\eta}(\mathbf{x}))\right) \right]
\end{align}

## SGD optimization

- Sample a M-size minibatch of images.
- Sample $\mathbf{\epsilon}^{(i)}$ from $\mathcal{N}(\mathbf{0},\mathbf{I})$, $i=1,\ldots,M$.
- For $i=1,\ldots,M$, compute
\begin{align}
\mathbf{z}^{(i)} = \mu_\eta(\mathbf{x}^{(i)}) + \sqrt{\text{diag}(\sigma_\eta(\mathbf{x}^{(i)}))} \circ \mathbf{\epsilon}^{(i)}
\end{align}
- Compute gradients of
\begin{align}
\hat{\mathcal{L}}(\mathbf{X},\theta,\eta) =\sum_{i=1}^M \Big(\log  p_\theta(\mathbf{x}^{(i)}|\mathbf{z}^{(i)}) - \text{KL}(q_{\eta,\mathbf{x}^{(i)}}(\mathbf{z})||p(\mathbf{z}))\Big)
\end{align}
w.r.t. $\theta,\eta$
- Perform SGD update

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import tensorflow as tf
from matplotlib.offsetbox import OffsetImage, AnnotationBbox

# use seaborn plotting defaults
import seaborn as sns; sns.set()

%matplotlib inline
```

    /Users/olmos/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

## Loading MNIST database

```python
from sklearn.datasets import fetch_mldata
from sklearn.model_selection import train_test_split
mnist = fetch_mldata('MNIST original')
```

```python
images = mnist.data.astype(np.float32)
labels = mnist.target

train_dataset,test_dataset,train_labels,test_labels = train_test_split(images, labels, test_size=0.33, random_state=42)

train_dataset,valid_dataset,train_labels,valid_labels = train_test_split(train_dataset, train_labels, test_size=0.33, random_state=42)

#Generate a random index
num_figs=3
img_size = 28

index=np.random.randint(train_dataset.shape[0], size=num_figs)

for i in range(num_figs):
    plt.figure()
    plt.imshow(train_dataset[index[i],:].reshape([28,28]),cmap=plt.cm.gray)
    plt.title('Label:' + str(train_labels[index[i]]))
```

### TensorFlow Computation Graph and Loss Function

```python
z_dim = 2      #Latent Space

model_name = 'model1'   #In 'model1.py' we define the variational family

learning_rate = 1e-3
num_imgs = 50        #Number of samples generated from the generative model (for testing)
num_iter = 20000      #SGD iterations
period_plot = 1000
sigma_reconstruction = 0.01 #Reconstruction variance
batch_size = 200
dims = [batch_size,784]
```

```python
sess_VAE = tf.Graph()

with sess_VAE.as_default():

    print('[*] Importing model: ' + model_name)
    model = __import__(model_name)

    print('[*] Defining placeholders')

    inputX = tf.placeholder(tf.float32, shape=dims, name='x-input')

    print('[*] Defining the encoder')
    log_var, mean, sample_z, KL = model.encoder(inputX,z_dim,batch_size)

    print('[*] Defining the decoder')

    loglik,img_reconstruction = model.decoder(inputX,sample_z,sigma_reconstruction,dims[1])

    loss = -tf.reduce_mean(loglik - KL)

    optim = tf.train.AdamOptimizer(learning_rate).minimize(loss)

    print('[*] Defining Sample operation...')
    samples = model.new_samples(num_imgs, z_dim, dims[1])

    # Output dictionary -> Useful if computation graph is defined in a separate .py file

    tf_nodes = {}

    tf_nodes['X'] = inputX

    tf_nodes['mean'] = mean

    tf_nodes['logvar'] = log_var

    tf_nodes['KL'] = tf.reduce_mean(KL)

    tf_nodes['loglik'] = tf.reduce_mean(loglik)

    tf_nodes['img_reconst'] = img_reconstruction

    tf_nodes['optim'] = optim

    tf_nodes['samples'] = samples

```

    [*] Importing model: model1
    [*] Defining placeholders
    [*] Defining the encoder
    [*] Defining the decoder
    [*] Defining Sample operation...

## SGD optimization

```python

############ SGD Inference #####################################

with tf.Session(graph=sess_VAE) as session:

    # Add ops to save and restore all the variables.
    saver = tf.train.Saver()

    tf.global_variables_initializer().run()

    print('Training the VAE ...')

    for it in range(num_iter):

        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.

        offset = (it * batch_size) % (train_labels.shape[0] - batch_size)

        # Generate a minibatch.

        batch_data = train_dataset[offset:(offset + batch_size), :]

        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.

        feedDict = {tf_nodes['X'] : batch_data.reshape([-1,784])}

        _,loglik,KL= session.run([tf_nodes['optim'],tf_nodes['loglik'],tf_nodes['KL']],feedDict)

        if(it % period_plot ==0):

            print("It = %d, loglik = %.5f, KL = %.5f" %(it,loglik,KL))

    it = 0
    offset = (it * batch_size) % (train_labels.shape[0] - batch_size)

    batch_data_train = train_dataset[offset:(offset + batch_size), :]
    batch_data_test = test_dataset[offset:(offset + batch_size), :]

    #We compute the latent representation of batch images and their reconstruction
    feedDict = {tf_nodes['X'] : batch_data_train.reshape([-1,784])}
    z,reconstructions_train = session.run([tf_nodes['mean'],tf_nodes['img_reconst']], feed_dict=feedDict)

    feedDict = {tf_nodes['X'] : batch_data_test.reshape([-1,784])}
    z_test,reconstructions_test = session.run([tf_nodes['mean'],tf_nodes['img_reconst']], feed_dict=feedDict)

    samples = session.run(tf_nodes['samples'])

```

    Training the VAE ...
    It = 0, loglik = -294680416.00000, KL = 2.37981
    It = 1000, loglik = -170337856.00000, KL = 14.90301
    It = 2000, loglik = -162173904.00000, KL = 17.67475
    It = 3000, loglik = -157364464.00000, KL = 17.43681
    It = 4000, loglik = -151082656.00000, KL = 19.02109
    It = 5000, loglik = -146750832.00000, KL = 19.01055
    It = 6000, loglik = -144242784.00000, KL = 17.92510
    It = 7000, loglik = -141732912.00000, KL = 18.24413
    It = 8000, loglik = -138781984.00000, KL = 19.72985
    It = 9000, loglik = -139642464.00000, KL = 19.51037
    It = 10000, loglik = -140557952.00000, KL = 19.72429
    It = 11000, loglik = -132590232.00000, KL = 18.34506
    It = 12000, loglik = -133731248.00000, KL = 20.11160
    It = 13000, loglik = -124924768.00000, KL = 20.37013
    It = 14000, loglik = -124596776.00000, KL = 20.34818
    It = 15000, loglik = -134286560.00000, KL = 20.94070
    It = 16000, loglik = -127114336.00000, KL = 20.84480
    It = 17000, loglik = -121211256.00000, KL = 19.79773
    It = 18000, loglik = -121058880.00000, KL = 19.89827
    It = 19000, loglik = -124864248.00000, KL = 21.91816

### Lets plot some train images and their reconstruction

```python
#Lets plot train images and their reconstruction

n_plots_axis=5

f2, axarr2 = plt.subplots(n_plots_axis,2)

for i in range(n_plots_axis):

    axarr2[i,0].imshow(batch_data_train[i,:].reshape([28,28]),cmap='gray');
    axarr2[i,1].imshow(reconstructions_train[i,:].reshape([28,28]),cmap='gray')

```

### Lets plot latent representation of train images

```python
def plot_latent_space_with_images(ax,recons_images,z_samples):

    ax.clear()
    for i in range(len(recons_images)):
        im = OffsetImage(recons_images[i].reshape([28,28]), zoom=1,cmap='gray')
        ab = AnnotationBbox(im, z_samples[i],frameon=True)
        ax.add_artist(ab)
    ax.set_xlim(np.min(z_samples[:,0])-1,np.max(z_samples[:,0])+1)
    ax.set_ylim(np.min(z_samples[:,1])-1,np.max(z_samples[:,1])+1)
    ax.set_title('Latent space Z with Images')

plt.figure()
f_latent, ax_latent = plt.subplots(1,1,figsize=(8, 8))
plot_latent_space_with_images(ax_latent,reconstructions_train,z)
plt.rcParams["figure.figsize"] = [14,14]
```

### Lets plot latent representation of test images

```python
plt.figure()
f_latent, ax_latent = plt.subplots(1,1,figsize=(8, 8))
plot_latent_space_with_images(ax_latent,reconstructions_test,z_test)
plt.rcParams["figure.figsize"] = [14,14]
```

```python
# Lets plot latent representation of test images

f3, axarr3 = plt.subplots(n_plots_axis,2)

for i in range(n_plots_axis):

    axarr3[i,0].imshow(batch_data_test[i,:].reshape([28,28]),cmap='gray');
    axarr3[i,1].imshow(reconstructions_test[i,:].reshape([28,28]),cmap='gray')

```

```python
# Lets plot samples of the model

n_plots_axis=5

f4, axarr4 = plt.subplots(n_plots_axis,2)

for i in range(n_plots_axis):

    axarr4[i,0].imshow(samples[i,:].reshape([28,28]),cmap='gray');
    axarr4[i,1].imshow(samples[i+20,:].reshape([28,28]),cmap='gray')

```

```python

```


# Свойства эллипсоидов

**Определение:** Множество $E \subseteq ℝ^n$ — **эллипсоид**, если существуют $a \in ℝ^n$ и положительно определенная матрица $A \in ℝ^{n \times n}$, такие что

$$E = E(A, a) := \{x \in ℝ^n \mid (x - a)^T A^{-1} (x - a) \le 1\} \tag 1$$

$$E(A, a) := \{x \in ℝ^n \mid \|x - a\|_A \le 1\}, \tag 2$$

то есть $E(A, a)$ — это единичный шар с центром в $a$ в линейном пространстве $ℝ^n$ с нормой $\|\cdot\|_A$. В частности, шар единичного радиуса $S(0, 1)$ с центром в нуле — это эллипсоид $E(I, 0)$. Эллипсоид $E$ одназначно определяет $a$ (центр) и $A$.

**Факт:** Для любой положительно определенной матрицы $A$ существует единственная положительно определенная матрица $A^{1/2}$, такая что $A = A^{1/2} A^{1/2} = (A^{1/2})^T A^{1/2}$.

**Следствие:**

$$E(A, a) = A^{1/2} S(0, 1) + a \tag 3$$

**Доказательство:**

$$A^{1/2} S(0, 1) + a = \{A^{1/2}x + a \mid x \in S(0, 1)\}\\
= \{x \mid A^{-1/2} (x - a) \in S(0, 1)\}\\
= \{x \mid (A^{-1/2} (x - a))^T A^{-1/2} (x - a) \le 1\}\\
= \{x \mid (x - a)^T (A^{-1/2})^T A^{-1/2} (x - a) \le 1\}\\
= \{x \mid (x - a)^T A^{-1} (x - a) \le 1\} = E(A, a)$$

**Замечание:** Каждый эллипсоид — это образ единичного шара при биективном афинном преобразовании.

**Факт:** Объем эллипсоида $vol\big(E(A, a)\big)$ зависит только от матрицы $A$ и размерности пространства:

$$vol\big(E(A, a)\big) = \sqrt {\det A} \cdot V_n, \tag 4$$

где $V_n$ — объем единичного шара $S(0, 1)$ в $ℝ^n$:

$$V_n = \frac{\pi^{n/2}}{\Gamma(n/2 + 1)} \sim \frac{1}{\sqrt {\pi n}} \bigg(\frac{2e\pi}{n} \bigg)^{n/2}. \tag 5$$

**Замечание:** Пусть $T : x \longmapsto Dx + d$ — биективное афинное преобразование, тогда $vol\bigg(T\big(E(A, a)\big)\bigg) = \det D \sqrt{\det A} \cdot V_n$. В частности,

$$\frac{vol\big(E(A, a)\big)}{vol\big(E(B, b)\big)} = \frac{vol\bigg(T\big(E(A, a)\big)\bigg)}{vol\bigg(T\big(E(B, b)\big)\bigg)},$$

то есть отношение объемов эллипсоидов инвариантно относительно биективных афинных преобразований.

**Наблюдение:** Для $c \neq 0, \max {c^T x}, x \in S(a, 1)$ достигается при $x^* = a + \frac{c}{\|c\|}$.

**Лемма:** Пусть $E(A, a) \subseteq ℝ^n$ — эллипсоид, $c \in ℝ^n \setminus \{0\}$,

$$b := \frac{1}{\sqrt {c^T A c}} A c, \tag 7\\
z_{max} := a + b,\\
z_{min} := a - b,$$

тогда $z_{max}$ максимизирует, а $z_{min}$ минимизирует $c^T x$ на $E$.

**Доказательство:** Пусть $Q := A^{1/2}$. Из $(3)$:

$$Q^{-1} E(A, a) = S(0, 1) + Q^{-1} a = S(Q^{-1} a, 1),$$

cледовательно

$$\max\{c^T x \mid x \in E(A, a)\} = \max\{c^T Q Q^{-1} x \mid Q^{-1} x \in Q^{-1} E(A, a)\}\\
= \max\{c^T Q y \mid y \in S(Q^{-1} a, 1)\}\\
= c^T Q \bigg( Q^{-1} a + \frac{(c^T Q)^T}{\|c^T Q\|}\bigg)\\
= c^T a + c^T Q \frac{1}{\|Q c\|} Q c\\
= c^T a + c^T \frac{1}{\sqrt{c^T A c}} A c\\
= c^T a + \sqrt{c^T A c}.$$

Подставляем $(7)$ и получаем требуемое:

$$c^T z_{max} = c^T \bigg(a + \frac{1}{\sqrt{c^T A c}}A c\bigg) = c^T a + \sqrt{c^T A c} = c^T a + \|c\|_{A^{-1}} = \max\{c^T x \mid x \in E(A, a)\}, \tag 8\\
c^T z_{min} = c^T \bigg(a - \frac{1}{\sqrt{c^T A c}} A c\bigg) = c^T a - \sqrt{c^T A c} = c^T a - \|c\|_{A^{-1}} = \min\{c^T x \mid x \in E(A, a)\}.$$

**Замечание:** Прямая, проходящая через $z_{max}$ и $z_{min}$ проходит через центр $a$ эллипса $E(A, a)$ и имеет направление $b$.

**Теорема:** Для любого выпуклого тела $K \subseteq ℝ^n$ существует единственный эллипсоид $E = E(A, a)$ минимального объема, содержащий $K$. Более того, $K$ содержит эллипсоид $E(n^{-2}A, a)$, получаемый из $E$ гомотетией из центра $E$ с коэффициентом $1/n$.

**Определение:** Эллипсоид минимального объема содержащий выпуклое тело $K$ называется эллипсоидом **Лёвнера-Джона** — $LJ(K)$.

**Факт:** Пусть $E(A, a) \subseteq ℝ^n$ — эллипсоид, $c \in ℝ^n \setminus \{0\}$,

$$E'(A, a, c) := E(A, a) \cap \{x \in ℝ^n \mid c^T x \le c^T a\}, \tag 9$$

то есть $E'(A, a, c)$ — половина эллипсоида $E(A, a)$, полученная разрезанием $E(A, a)$ через центр $a$ гиперплоскостью $\{x \in ℝ^n \mid c^T x = c^T a\}$.

Тогда $LJ\big(E'(A, a, c)\big)$ — эллипсоид $E(A', a')$, заданный формулами:

$$a' := a - \frac{1}{n + 1} b, \tag {10}$$

$$A' := \frac{n^2}{n^2 - 1}\left(A - \frac{2}{n + 1} b b^T \right), \tag {11}$$

где $b$ — вектор, определенный в $(7)$.

**Замечание:** Центр $a'$ эллипсоида $E(A', a') = LJ\big(E'(A, a, c)\big)$ лежит на прямой, проходящей через $z_{max}$ и $z_{min}$ из $(7)$. В частности, из $a$ можно получить $a'$, сделав шаг к $z_{min}$ длины $\frac{1}{n + 1}\|z_{min} - a\|$. Более того, граница $E(A', a')$ касается $E'(A, a, c)$ в точке $z_{min}$ и во множестве $\{x \mid \|x - a\|_A = 1\} \cap \{x \mid c^T x = c^T a\}$. В $ℝ^2$ последнее множество состоит из двух точек, а в $ℝ^3$ является эллипсом.

**Следствие:** Пусть $E(A, a) \subseteq ℝ^n$ — эллипсоид, $c \in ℝ^n \setminus \{0\}, \gamma \in ℝ$. Из $(8)$ следует, что гиперплоскость

$$H := \{x \in ℝ^n \mid c^T x = \gamma\}$$

имеет непустое пересечение с $E(A, a)$ тогда и только тогда, когда $c^T z_{min} \le \gamma \le c^T z_{max}$, что равносильно

$$|c^T a - \gamma| \le \sqrt {c^T A c}.$$

Для удобства обозначим

$$\alpha := \frac{c^T a - \gamma}{\sqrt {c^T A c}}. \tag {12}$$

Тогда $H$ пересекает $E(A, a)$ тогда и только тогда, когда $-1 \le \alpha \le 1$. Число $\alpha$ можно интерпретировать как расстояние со знаком от центра $a$ до границы полупространства $\{x \in ℝ^n \mid c^T x \le \gamma\}$ в пространстве $ℝ^n$ с нормой $\|\cdot\|_{A^{-1}}$ (расстояние неположительно, если $a$ содержится в полупространстве).

**Факт:** Пусть $E(A, a) \subseteq ℝ^n$ — эллипсоид, $c \in ℝ^n \setminus \{0\}, \gamma \in ℝ$,

$$E'(A, a, c, \gamma) := E(A, a) \cap \{x \in ℝ^n \mid c^T x \le \gamma\}, \tag {13}$$

то есть $E'(A, a, c, \gamma)$ — часть эллипсоида $E(A, a)$, полученная разрезанием $E(A, a)$ гиперплоскостью $H = \{x \in ℝ^n \mid c^T x = \gamma\}$.

Тогда $LJ\big(E'(A, a, c, \gamma)\big)$ — эллипсоид $E(A', a')$, заданный таким образом:

\begin{equation}
\begin{cases}
E(A, a),\ -1 \le \alpha \le -1/n\\
\begin{cases}
a' := a - \frac{1 + n \alpha}{n + 1} b, \tag {14}\\
A' := \frac{n^2}{n^2 - 1}(1 - \alpha^2)\big(A - \frac{2 (1 + n \alpha)}{(n + 1) (1 + \alpha)} b b^T \big)
\end{cases}, -1/n \le \alpha < 1
\end{cases},
\end{equation}

где $b$ — вектор, определенный в $(7)$.

**Замечание:** Если $\gamma = c^T a$, то $E'(A, a, c, \gamma) = E'(A, a, c)$ и формулы $(14)$ сводятся к $(10), (11)$.

**Определение:** Разрез через центр $a$ эллипсоида $E(A, a)$ гиперплоскостью $H$ называется **центральным**. Если $0 < \alpha < 1$, то $E'(A, a, c, \gamma)$ строго содержится в $E'(A, a, c)$, то есть от $E(A, a)$ отрезается больший кусок, поэтому разрез $c^T x \le \gamma$ называется **глубоким**. Если $-1/n < \alpha < 0$, то от $E(A, a)$ остается больше половины $E'(A, a, c)$, поэтому разрез $c^T x \le \gamma$ называется **мелким**, но даже в этом случае

$$vol\bigg(LJ\big(E'(A, a, c, \gamma)\big)\bigg) < vol\big(E(A, a)\big).$$

# Метод эллипсоидов

**Задача:** Дана система неравенств $c_i^T x \le \gamma_i, i = 1, \dots, m$, с $n$ переменными. Требуется определить пустое ли множество

$$P := \{x \in ℝ^n \mid c_i^T x \le \gamma_i, i = 1, \dots, m\} = \{x \mid C x \le d\}, \tag {15}$$

и если $P \neq \emptyset$, то найти точку, содержащуюся в $P$.

Известно, что либо $P = \emptyset$, либо $P$ — ограниченное полноразмерное множество.

**Следствие:** Существует $R > 0$, такое что $P \subseteq S(0, R)$.

**Следствие:** Существует $r > 0$, такое что либо $P = \emptyset$, либо $P \supseteq S(c, r)$ для некоторой точки $c$.

**Определение:** $P \subseteq S(0, R) = E(R^2 I, 0)$. Определим $E_0 = E(A_0, a_0)$, где $A_0 = R^2 I, a_0 = 0$. Пусть на текущем шаге есть эллипсоид $E_k := E(A_k, a_k) \supseteq P$. Можно подставить $a_k$ в систему неравенств $(15)$ и проверить, все ли из них выполняются. Если да, то решение задачи найдено. В противном случае хотя бы одно из неравенств нарушается, допустим $c^T x \le \gamma$. Значит, $c^T a_k > \gamma$. Гиперплоскость $\{x \mid c^T x = c^T a_k\}$, проходящая через центр $a_k$ эллипсоида $E_k$, делит $E_k$ на две половины. Из построения известно, что $P$ содержится в половине

$$E'(A_k, a_k, c) := \{x \in E(A_k, a_k) \mid c^T x \le c^T a_k\}. \tag {16}$$

Тогда определим следующий эллипсоид $E_{k+1}$ в последовательности так:

$$E_{k+1} = LJ\big(E'(A_k, a_k, c)\big) \supseteq E'(A_k, a_k, c) \supseteq P, $$

задающийся формулами $(10)$ и $(11)$.

**Лемма:**

$$\frac{vol(E_{k+1})}{vol(E_{k})} = \bigg(\big(\frac{n}{n+1}\big)^{n+1}\big(\frac{n}{n-1}\big)^{n-1}\bigg)^{1/2} < e^{-1/(2n)} < 1$$

**Доказательство:** Пусть $E = E_k$ — эллипсоид, а $E' = E_{k+1}$ — эллипсоид полученный из $E$ по формулам $(7), (10), (11)$. Чтобы оценить отношение объемов, предположим, что исходный эллипсоид $F := E(I, 0)$, то есть единичный шар с центром в нуле, а вектор $c$, который используется для вычисления $b$ из $(7)$, равен $(-1, 0, \dots, 0)^T$. В таком случае, по формулам $(7), (10), (11)$ получаем:

$$b = (-1, 0, \dots, 0)^T,\\
a' = a - \frac{1}{n + 1} b = \big(\frac{1}{n+1}, 0, \dots, 0\big)^T,\\
A' = \frac{n^2}{n^2 - 1} \big(I - \frac{2}{n + 1}(-1, 0, \dots, 0)^T(-1, 0, \dots, 0)\big)\\
= diag\bigg(\big(\frac{n^2}{(n + 1)^2}, \frac{n^2}{n^2 - 1}, \dots, \frac{n^2}{n^2 - 1}\big)^T\bigg).$$

Из этого и $(4)$ для объемов $F$ и $F' := E(A', a')$:

$$\frac{vol(F')}{vol(F)} = \frac{\sqrt {\det A'} V_n}{\sqrt {\det A} V_n} = \sqrt {\det A'} = \big(\frac{n^{2n}}{(n+1)^n (n-1)^n} \frac{n-1}{n+1}\big)^{1/2}\\
= \bigg(\big(\frac{n}{n+1}\big)^{n+1}\big(\frac{n}{n-1}\big)^{n-1}\bigg)^{1/2}.$$

Логарифмируя, получаем:

$$\frac{vol(F')}{vol(F)} < e^{-1/(2n)}  \Leftrightarrow e^{1/n} < \big(\frac{n + 1}{n}\big)^{n+1}\big(\frac{n-1}{n}\big)^{n-1}\\
\Leftrightarrow \frac{1}{n} < (n+1) \ln {\big(1 + \frac{1}{n}\big)} + (n-1) \ln {\big(1 - \frac{1}{n}\big)}.$$

Раскладывая логарифм в ряд $\ln x = \sum_{k=1}^\infty (-1)^{k+1} (x-1)^k / k, 0 < x \le 2$, получаем:

$$(n+1) \ln {\big(1 + \frac{1}{n}\big)} + (n-1) \ln {\big(1 - \frac{1}{n}\big)}=\\
= \sum_{k=1}^\infty (-1)^{k+1} \frac{n+1}{k n^k} - \sum_{k=1}^\infty \frac{n-1}{k n^k}\\
= \sum_{k=1}^\infty (-1)^{k+1} \frac{2}{k n^k} - \sum_{k=1}^\infty \frac{2(n-1)}{2k n^{2k}}\\
= \sum_{k=1}^\infty \frac{2}{(2k-1) n^{2k-1}} - \sum_{k=1}^\infty \frac{2}{2k n^{2k}} - \sum_{k=1}^\infty \frac{1}{k n^{2k-1}} + \sum_{k=1}^\infty \frac{1}{k n^{2k}}\\
= \sum_{k=1}^\infty \frac{1}{(2k-1) k} \frac{1}{n^{2k-1}}\\
> \frac{1}{n},$$

что доказывает требуемое для нашего конкретного выбора $F$ и $c$.

Теперь пусть $c \in ℝ^n \setminus \{0\}$ — произвольный вектор. Из $(3)$:

$$E = A^{1/2}E(I, 0) + a = A^{1/2}F + a.$$

Ясно, что существует ортогональная матрица $Q$, переводящая $A^{1/2} c$ в $(-\|Q A^{1/2} c\|, 0, \dots, 0)$, то есть

$$(-1, 0, \dots, 0)^T = \frac{1}{\|Q A^{1/2} c\|}Q A^{1/2} c.$$

Тогда

$$T(x) := A^{1/2} Q^T x + a$$

— биективное афинное преобразование, причем $T^{-1}(x) = QA^{-1/2}(x-a)$.

Теперь

$$T(F) = \{T(y) \mid y^T y \le 1\}\\
= \{x \mid (T^{-1}x)^T (T^{-1}x) \le 1\}\\
= \{x \mid (x - a)^T A^{-1/2} Q^T Q A^{-1/2} (x-a) \le 1\}\\
= \{x \mid (x - a)^T A^{-1} (x-a) \le 1\}\\
= E,$$

Аналогично для эллипсоида $F'$ получается $T(F') = E'$.

Пользуясь инвариантностью отношения объемов эллипсоидов к биективным афинным преобразованиям, имеем:

$$\frac{vol(E')}{vol(E)} = \frac{vol\big(T(E')\big)}{vol\big(T(E)\big)} = \frac{vol(F')}{vol(F)} = \bigg(\big(\frac{n}{n+1}\big)^{n+1}\big(\frac{n}{n-1}\big)^{n-1}\bigg)^{1/2} < e^{-1/(2n)}.$$

**Алгоритм:**

Инициализация: $k := 0, N := \min \{x \mid x > 2n^2 \ln {\frac{R}{r}}, x \in ℤ \}, A_0 := R^2 I, a_0 = 0, E_0 = E(A_0, a_0)$.

Шаг алгоритма:

* Если $k = N$, СТОП! (Объявить $P$ пустым)
* Если $a_k \in P$, СТОП! (Найдено возможное решение)
* Если $a_k \notin P$, выбрать неравенство $c^T x \le \gamma$ системы $Cx \le d$, которое нарушается при подстановке $a_k$, обновить значения $b, a_{k+1}, A_{k+1}$ через $a_k, A_k$ по формулам $(7), (10), (11)$.

**Теорема:** Алгоритм решает поставленную задачу.

**Доказательство:** После $2n$ шагов алгоритма объем эллипсоида уменьшается в $e$ раз.

$$vol(E_N) = e^{-\frac{N}{2n}} vol(E_0) < e^{-n \ln {\frac{R}{r}}} vol\big(E(R^2 I)\big)\\
= \frac{r^n}{R^n} R^n vol\big(E(I, 0)\big) = vol\big(E(r^2 I, 0)\big) = vol\big(S(c, r)\big),$$

то есть у $P \subseteq E_N$ объем строго меньше, чем у $S(c, r)$, но это противоречит второму следствию из задачи, если $P \neq \emptyset$. Значит, $P = \emptyset$.

```
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from IPython.display import clear_output, display
from scipy.linalg import svd
from matplotlib.patches import Ellipse
import time
import math

EPS = 1e-7

def ellipsoid_method(A, b, r=EPS, R=15):
    clear_output(wait=True)
    fig = plt.figure(figsize=(12, 12))
    ax = fig.add_subplot(1, 1, 1)
    plt.ion()
    rng = np.linspace(-10, 10, 1001)
    pts = np.array([[x, y]
                    for x in rng
                    for y in rng])
    mask = np.array([[x >= 0.0 and y >= 0.0 for x in rng] for y in rng])
    for a_, b_ in zip(A, b):
        mask &= (np.squeeze(a_.reshape((1, -1)) @ pts.T) < b_).reshape(len(rng), len(rng))

    ax.imshow(mask.T.astype(int), extent=(pts[:, 0].min(), pts[:, 0].max(),
                                        pts[:, 1].min(), pts[:, 1].max()),
              origin='lower', cmap='Greys', alpha=0.3)
    time.sleep(2)
    display(fig)

    n = A.shape[1]
    c = np.zeros((n,))
    Q = R * R * np.eye(n, dtype=float)
    for iter in range(2 * n * (n + 1) * round(math.log(R / r) + 1)):
        clear_output(wait=True)
        S, V, _ = np.linalg.svd(Q.copy())
        angle = np.degrees(np.arctan2(S[1, 0], S[0, 0]))
        width, height = 2 * np.sqrt(V)
        ax.add_patch(Ellipse(c.copy(), width, height, angle, facecolor='none', edgecolor='b'))
        ax.plot(c[0], c[1], 'bo')
        time.sleep(2)
        display(fig)
        mask = np.concatenate((np.squeeze(A @ c) <= b, c >= 0), axis=0)
        if np.count_nonzero(np.logical_not(mask)) == 0:
            plt.close()
            return c
        a = A[np.argmax(np.logical_not(mask)), :]
        a = a.reshape((-1, 1))
        if a.T @ c >= 0:
            a = -a

        a_ = np.squeeze(np.array([-a.copy()[1], a.copy()[0]]))
        l, r = 0, 30
        Q_inv = np.linalg.inv(Q)
        while r - l > 1e-7:
            m = (l + r) / 2
            x = (m * a_).reshape((-1, 1))
            if x.T @ Q_inv @ x <= 1.0:
                l = m
            else:
                r = m
        p1, p2 = c.copy() - l * a_, c.copy() + l * a_
        clear_output(wait=True)
        ax.plot([p1[0], p2[0]].copy(), [p1[1], p2[1]].copy(), 'r')
        time.sleep(2)
        display(fig)
        c += np.squeeze((Q @ a) / (math.sqrt(a.T @ Q @ a) * (n + 1)))
        Q = (n ** 2 / (n ** 2 - 1)) * (Q - (2 / ((n + 1) * a.T @ Q @ a)) * (Q @ a @ a.T @ Q))
    return None

ellipsoid_method(np.array([[1, 0.8],
                            [0, 1],
                            [-1.2, 0.1],
                            [0.1, -1],
                            [-1, -2.5]]),
                  np.array([7, 3, -0.2, -0.3, 3]))
```

# Литература

* https://www.mpi-inf.mpg.de/fileadmin/inf/d1/ellipsoid-lovasz.pdf
* https://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15859-f11/www/notes/lecture08.pdf


# Baseball problem: Solution
Copyright © 2020 Oliver Beckstein.

This notebook is based on `12-ODE-applications.ipynb`. The whole problem was live-coded from scratch during class and this notebook is a cleaned-up and commented solution. The lesson showed how
1. to define the physical problem (obtain trajectory of a baseball with spin and air resistance)
2. to derive the underlying equations (Newton's equations of motions and the forces acting on the ball)
3. to adapt an algorithm to solve the equations (use RK4 for integrating the ODEs and express the problem in ODE standard form so that one can use `ode.rk4()`)
4. to visualize and discuss the results (plot trajectories and assess the influence of spin)

## Packages
We use numpy and our ODE solvers (`ode.py`):

```python
import numpy as np
import ode
```

For 3D graphs you can use
```
%matplotlib ipympl
```
but you have to do this as the very first thing (before loading `matplotlib`). You will also have to "close" active figures before you can plot into a new one.

```python
%matplotlib inline
#%matplotlib ipympl
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

## Simple Baseball physics

- quadratic air resistance (with velocity-independent drag coefficient)
- Magnus force due to spin

### Quadratic air resistance
Occurs at high Reynolds numbers, i.e., turbulent flow. Only approximate:

$$
\mathbf{F}_2 = -b_2 v \mathbf{v}
$$

with
$$
b_2 = \frac{1}{2} C_D \rho A
$$
where $C_D$ is the drag coefficient, $\rho$ the density of air, and $A = r^2 \pi$ the cross sectional area of the baseball.

### Magnus effect

**Magnus effect**: airflow is changed around a spinning object. The Magnus force is

$$
\mathbf{F}_M = \alpha \boldsymbol{\omega} \times \mathbf{v}
$$

where $\boldsymbol{\omega}$ is the ball's angular velocity in rad/s (e.g., 200/s for a baseball).

For a sphere the proportionality constant $\alpha$ can be written

$$
\mathbf{F}_M = \frac{1}{2} C_L \rho A \frac{v}{\omega} \boldsymbol{\omega} \times \mathbf{v}
$$

where $C_L$ is the lift coefficient, $\rho$ the air density, $A$ the ball's cross section. (Advantage of defining $C_L$ this way: when spin and velocity are perpendicular, the Magnus force is simply $F_M = \frac{1}{2} C_L \rho A v^2$.)

$C_L$ is mainly a function of the *spin parameter*

$$
S = \frac{r\omega}{v}
$$

with the radius $r$ of the ball. In general we write

$$
\mathbf{F}_M = \frac{1}{2} C_L(S)  \frac{\rho A r}{S} \boldsymbol{\omega} \times \mathbf{v}
$$

For a baseball, experimental data show approximately a power law dependence of $C_L$ on $S$

$$
C_L(S) = 0.62 \times S^{0.7}
$$

All together:

\begin{align}
\mathbf{F}_M &= \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
v &= \sqrt{\mathbf{v}\cdot\mathbf{v}}\\
S &= \frac{r\omega}{v}\\
C_L &= 0.62 \times S^{0.7}\\
\alpha &= \frac{1}{2} C_L  \frac{\rho A r}{S}
\end{align}

### Equations of motion

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= -g \hat{\mathbf{e}}_y -\frac{b_2}{m} v \mathbf{v} + \alpha\ \boldsymbol{\omega} \times \mathbf{v}
\end{align}

(quadratic drag $-\frac{b_2}{m} v \mathbf{v}$ included.)

### Baseball simulation

Implement the full baseball equations of motions:
- gravity $a_\text{gravity}$
- quadratic drag $a_\text{drag}$
- Magnus effect $a_\text{Magnus}$

For the cross product you can look at [numpy.cross()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cross.html).

```python
def C_L(S):
    return 0.62 * S**0.7

def simulate_baseball(v0, omega, r0=None,
                      h=0.01, C_D=0.40, g=9.81, rho=1.225,
                      r=0.07468/2, m=0.14883,
                      R_homeplate=18.4):
    """simulate baseball pitch

    Parameters
    ----------
    v0 : array
         initial velocity (vx, vy, vz) in m/s
    omega : array
         angular velocity vector of the ball ("spin"), in rad/s
    r0 : array, optional
         initial position of the ball (in m) when it leaves the pitcher's hand
         as (x, y, z); the default is (0, 2, 0)
    h : float, optional
         integration time step in s, default is 0.01 s
    C_D : float, optional
         drag coefficient, default is 0.40
    g : float, optional
         acceleration due to gravity, default 9.81 kg/(m*s^2)
    rho : float, optional
         density of air, default 1.225 kg/m^3
    r : float, optional
         radius of the baseball
    m : float, optional
         mass of the baseball
    R_homeplate : float, optional
         distance of the catcher from the pitcher

    Returns
    -------

    positions : array
         The array contains an entry (time, x, y, z) for each time step.
    """
    # all SI units (kg, m)
    if r0 is None:
        r0 = np.array([0, 2, 0])  # pitching at 2m height

    omega = np.asarray(omega)

    domega = np.linalg.norm(omega)
    A = np.pi*r**2
    rhoArm = rho * A * r / m
    b2 = 0.5 * C_D * rho * A

    a_gravity = np.array([0, -g, 0])

    def f(t, y):
        # y = [x, y, z, vx, vy, vz]
        v = y[3:]
        dv = np.linalg.norm(v)
        S = r*domega/dv
        a_magnus = 0.5 * C_L(S) * rhoArm / S * np.cross(omega, v)
        a_drag = -b2/m * dv * v
        a = a_gravity + a_drag + a_magnus
        return np.array([y[3], y[4], y[5],
                         a[0], a[1], a[2]])

    t = 0
    # initialize 3D!
    y = np.array([r0[0], r0[1], r0[2], v0[0], v0[1], v0[2]], dtype=np.float64)
    positions = [[t, y[0], y[1], y[2]]] # record t, x and y, z

    while y[0] < R_homeplate and y[1] >= 0.2:
        y[:] = ode.rk4(y, f, t, h)
        positions.append([t, y[0], y[1], y[2]])  # record t, x and y, z
        t += h

    return np.array(positions)

```

#### Simulate throws

Simulate baseball throw for initial velocity $\mathbf{v} = (30\,\text{m/s}, 0)$.

Plot x vs y and x vs z (to see curving).

Try out different spins; a good value is $\boldsymbol{\omega} = 200\,\text{rad/s} \times (0, 1, 1)$.

200 rad/s is 32 revolutions per second or almost 2000 rpm.

```python
print(200/(2*np.pi), "revolutions per second")
print(200/(2*np.pi) * 60, "rpm")
```

    31.830988618379067 revolutions per second
    1909.859317102744 rpm

Simulate the baseball throw with
- almost no spin: $\omega = 0.001 \times (0, 0, 1)$  (our code does not handle $\omega = 0$ gracefully...)
- full upward spin: $\omega = 200 \times (0, 0, 1)$
- sideways spin: $\omega = 200 \times (0, 1, 1)$

```python
r = simulate_baseball([30, 0, 0], omega=0.001 * np.array([0,0,1]))
rz = simulate_baseball([30, 0, 0], omega=200. * np.array([0,0,1]))
rzy = simulate_baseball([30, 0, 0], omega=200. * 2**(-0.5) * np.array([0,1,1]))
```

#### Plotting

Plot the three scenarios in 2D planes: x-y (side view) and x-z (top view).

```python
idx = 2  # y
plt.plot(r[:,1], r[:,idx], label="no spin")
plt.plot(rz[:,1], rz[:,idx], label="upward spin")
plt.plot(rzy[:,1], rzy[:,idx], label="sideways spin")
plt.xlabel("$x$ (m)")
plt.ylabel("$y$ (m)")
plt.legend(loc="best")
plt.title("Side view");
```

Note how the balls with spin fly higher: the Magnus force lifts them up (due to the component of the spin along $z$).

```python
idx = 3  # z
plt.plot(r[:,1], r[:,idx], label="no spin")
plt.plot(rz[:,1], rz[:,idx], label="upward spin")
plt.plot(rzy[:,1], rzy[:,idx], label="sideways spin")
plt.xlabel("$x$ (m)")
plt.ylabel("$z$ (m)")
plt.legend(loc="best")
plt.title("Top-down view");
```

The ball with a component of the spin along the $y$ direction ($\boldsymbol{\omega} = \omega_0 \, \left(\begin{array}{c} 0\\1\\1\end{array}\right)$) has a 0.6 m deflection in the $-z$ direction.

#### 3D plot
Use simple `matplotlib` 3D plot. (BONUS: Make it work with vpython)

If we use the [`notebook` backend for matplotlib](http://ipython.readthedocs.io/en/stable/interactive/plotting.html) then we will be able to interactively rotate our [matplotlib 3D graphics](http://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html). (Note: If this does not seem to work, disable adblockers and allow javascript on the page.)

```python
%matplotlib ipympl
```

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(r[:,1], r[:,3], r[:,2], '.', label="no spin")
ax.plot(rz[:,1], rz[:,3], rz[:,2], '.', label="up spin")
ax.plot(rzy[:,1], rzy[:,3], rzy[:,2], '.', label="sideways spin")

# hand of the catcher, 0.2m above homeplate
ax.plot([18.4, 18.4], [0, 0], [0, 0.2], color="black", lw=6)

# drop line for last point
x,y,z = rzy[-1, 1:]
ax.plot([x,x], [z,z], [y,0], color="red")

ax.set_xlabel("$x$ (m)")
ax.set_ylabel("$z$ (m)")
ax.set_zlabel("$y$ (m)")
ax.legend(loc="upper left", numpoints=1)
ax.figure.tight_layout()
fig.savefig("baseball.png")
```

In the plot above we also added the hand of the catcher as a heavy black bar at 0.2 m above the home plate, and a drop-down bar in red to indicate the height of the ball thrown with *sideways spin* to make cleared that it also arrives at almost the same height as the ball with *up spin*.

```python

```