Guia básico para experimental no Python

 # 1. Bibliotecas

Primeiramente é importante verificar corretamente quais bibliotecas importar, em geral importamos:

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.optimize as optimization
import scipy.odr.odrpack as odrpack
```

### Numpy
$*$ Numpy é uma biblioteca usada para administrar arrays, sendo esses "vetores" que podem ser interpretados como as coordenadas no $\mathbb{R}^{n}$. Então, você pode colocar seus dados em arrays para fazer operações matemáticas com eles, por exemplo, podemos criar um array com incertezas de dados e operar com eles.

$*$ Também podemos usar ferramentas matemáticas com muita precisão, por exemplo, Cos(x), Sin(x), Sinh(x) e MUITOS outros.

$*$ Cálculo numérico ( integração, solução de sistemas lineares e não lineares, solução de EDO,etc)

$*$ Todas as informações podem ser encontradas na referência da biblioteca: https://numpy.org/doc/stable/reference/ ,
ou também nos StackOverflow da vida

```python
incertezas = np.array([0.5,0.6,0.4,0.3,0.6])
incertezas = incertezas*2
incertezas
```

Podemos mudar apenas uma componente no array, ou então, podemos mudar conjuntos dentro do array.

(Perceba que os valores estão mudados na célula de baixo devido as operações acima)

(Perceba também que o python não conta a terceira coordenda do vetor na separação, ou seja, o intervalo é aberto no final)

```python
incertezas[1:3] = incertezas[1:3]*np.cos(2)
incertezas
```

Podemos separar o array nas componentes que nos interessam. Isso se chama "Slicing".
###### Quando declarar uma variável como um slicing de um array temos um cópia, ou seja, no nosso caso, mudar o qualquer elemento do "pedaco" mudará o mesmo elemento na variável "incerteza". Para evitar isso, basta fazer uma cópia com np.copy(array)

```python
pedaco = incertezas[:4]
pedaco
```

Vou usar numpy para criar vetores com valores para embarcar no ajuste e no gráfico, então, vou definir os valores de x, y e sigmaX.

```python
x = np.linspace(0,100,10)

```

### Modelo

```python
def f(x, b, a):
    return a*x + b

# Criação dos valores de y
y = f(x,0.5,1)

#criando valores aleatórios entre o range [0.5,0.6]
sigma_y = np.random.uniform(low=2, high=5, size=(10,))
```

## Scipy

$*$ Essa biblioteca é um grupo de várias bibliotecas, nela temos quase todas as ferramentas numéricas para integração, Álgebra Linear, Analise de Fourier, estatística, etc.

$*$ A referência https://docs.scipy.org/doc/scipy/reference/index.html

### Scipy.optimization

$*$ Scipy.optimization é uma sub-biblioteca do scipy para optimização de parâmetros, no caso, usaremos o curve_fit, que serve para ajuste de funções lineares e não lineares usando MMQ, da mesma forma que o webROOT(para apenas incertezas em y), além de fornecer a matriz de covariância.
###### $*$ O método curve_fit só considera o $\sigma_y$, uma vez que utiliza o MMQ ordinário. Para incerteza em ambas as variáveis temos o MMQ total, que é feito com o método ODR do Scipy. Isso será abordado na última parte do notebook! ( parte prática).

$*$ A referência https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html

```python
#xo é o chute dos parâmetros
xo = [0,0]
fit = optimization.curve_fit(f, x, y, p0=xo, sigma=sigma_y,absolute_sigma=True)
fit
```

Agora perceba que o "fit" devolvou um dois arrays, como se fosse uma lista contendo dois arrays. O primeiro array é o chute do parâmetros e o segundo array é a matriz de covariância( note que esse array tem 2 colunas e 2 linhas). Podemos apresentar tudo isso de maneira mais agradável, como será feito na próxima sessão com a biblioteca Pandas.

(Você pode calcular o $\chi^2$ sem biblioteca, visto que é bem simples). Equação em baixo só para mostrar como usa latex aqui.

$$
\begin{align}
{\chi}^2=\sum_{k=1}^{n} \frac{(y_i - f(x_i))^2}{\sigma_y^2}
\end{align}
$$

```python
a = fit[0][1]
b = fit[0][0]
#np.diag pega a diagonal da matriz de covariância, outro artifício do numpy
stdevs = np.sqrt(np.diag(fit[1]))
#stdevs é um array contendo duas componentes stdevs[0] é a incerteza do parâmetro (b) e stdevs[1] do parâmetro (a)
Chi2 = chi = sum((y - f(x, b, a))**2/(sigma_y**2))
Chi2
```

## Pandas

$*$ Essa biblioteca se preocupa em manipular dados de forma estrutural, por exemplo, criando e operando "tabelas" que nós chamamos de DATA FRAME. Mas ela é muito mais complexa do que isso, tendo muita aplicação em machine learning( numpy também tem bastante), então, só vou abordar de forma bem superfícial, mas encorajo você a pesquisar e aprender a mexer em DATA FRAMES.

$*$ Vou usa-la para criar uma tabela dos coeficientes, uma aplicação bem porca para a biblioteca rs. Em geral eu monto os dados numa planilha do Google Sheets, baixo em csv e passo para o python em forma de Data Frame, assim fica muito fácil trabalhar. O pandas faz com que as colunas de Data Frames se comportem como arrays, dessa forma, posso fazer operações matemáticas com colunas e linhas.

$*$ O guia/referência é : https://pandas.pydata.org/docs/user_guide/index.html

```python
df = pd.DataFrame({'Parâmetro(a)' : [a,stdevs[1]], 'Parâmetro(b)' : [b,stdevs[0]]}, index=['Valor', 'Incerteza'])
df
```

### Matplotlib

$*$ O nome da biblioteca fala por ela. Para plotar todo tipo de coisa e ainda dá configurar ao máximo os gráficos.

$*$ Tem muito conteúdo na internet e a referencia dela é bacana e recheada de exemplo https://matplotlib.org/tutorials/index.html#introductory, novamente o StackOverflow é uma ótima fonte de pesquisa.

```python
h = plt.figure(1)
h = plt.plot(x,f(x,b,a), label="model", color='black')
h = plt.errorbar(x,y,sigma_y,fmt='.',color='red')
h = plt.grid()
h = plt.ylabel(r'$f(x)$')
h = plt.xlabel(r'$x$')
#para salvar o gráfico no mesmo diretório
plt.savefig('teste_1.png', format='png', dpi=150,bbox_inches = "tight")
plt.show(h)
```

Agora o gráfico de resíduos. (Lembrando, que estes são os gráficos defaults mais simples possíveis, caso queira sofisticar é só pesquisar, tem muito conteúdo!)

```python
g = plt.figure(2)
g.set_size_inches(6, 1)
g =plt.errorbar(x,y-f(x,b,a),sigma_y,fmt='.',color='red')
g = plt.hlines(0,x.min(),x.max(),color='black')
g = plt.grid()
g = plt.ylabel("f (x)")
g = plt.xlabel('x')
g = plt.savefig('teste_1_resid.png', format='png', dpi=150,bbox_inches = "tight")
g = plt.show(g)
```

## Dicas adicionais

Quero relembrar que todas as bibliotecas são vastas e os conteúdos sobre elas não se acabam, então, sempre que tiver dúvida você provavelmente vai achar com bastante facilidade na internet, principalmente, em referências e no StackOverflow.

$(*)$ Ótimo vídeo sobre o básico de curve fitting no python:
https://www.youtube.com/watch?v=Jl-Ye38qkRc&t=810s

$(*)$ Página com os códigos e noções básicas sobre o curve fitting no python:
https://towardsdatascience.com/basic-curve-fitting-of-scientific-data-with-python-9592244a2509

$(*)$ Eu usual faço tabelas no Google Sheets e import para cá com o pandas. Não é difícil fazer e nem mexer, mas exige um pouco para se acostumar e tals.

# 2. Colocando na prática

#### Vamos usar dados com incerteza em x e y ( como geralmente ocorre na disciplina de experimental)

$*$ É importante lembrar que a função curve_fit vista acima é apenas para o caso com incerteza em x

$*$ Para simular uma atividade foram gerados dados que estão em dois formatos, csv e txt.

#### Importando txt com Pandas

```python
data = pd.read_csv('exemplo.txt',sep="\t")

# No caso do separador com vírgulas é preciso fazer
# data_1 = pd.read_csv('exemplo.txt', sep="\t", decimal=",")

data.head()
```

#### Importando csv com Pandas

```python
data = pd.read_csv('exemplo.csv')
data.head()
```

### Análise dos dados

Vamos começar pelo modelo ( os dados criados são feitos para serem ajustados por uma parábola)

#### Modelo

```python
# Importante que notar que agora nossos parâmetros são representados por array
# Outro fator a se notar é a ordem na a f, ou seja, f(p,x) "p" vem primeiro

def f(p,x):
    return p[0]*x**2 + p[1]*x + p[2]
```

##### Usando os valores de $x$, $y$, $\sigma_x$ e $\sigma_y$

```python
x = data['x (cm)']
y = data['y (cm)']
sigma_x = data['sigma_x (cm)']
sigma_y = data['sigma_y (cm)']
# O x é uma "Série", seria algo como um array com indexes, mas se comporta de maneira
# bastante similar

x.head()
```

### O ajuste

Como agora nossa incerteza está nas duas variáveis, precisamos utiliar um outro método da mesma biblioteca( Scipy ). O método é o ODR.

```python
# objeto modelo ( isso é um termo técnico para dizer que o modelo está guardado em um "objeto")
model_object = odrpack.Model(f)
# Objeto data ( termo técnico para dizer que dizer que o modelo está guardado em um "objeto")
data_object = odrpack.RealData(x,y,sx=sigma_x, sy=sigma_y)
# Estabelece a relação entre data e modelo, e em geral necessita de das condições iniciais
# No caso do nosso modelo temos 3 parâmetros, ou seja beta0 = lista de 3 elementos
odr = odrpack.ODR(data_object, model_object, beta0=[1,1,1])
# Roda a regressão
odr.set_job(fit_type=0)  # fit_type = 0 executa a regressão por ODR e fit_type = 2 por MMQ
# O tópico do fit_type é um complicado e vai além do proposito dessa introdução
out = odr.run()
out.pprint()
```

Agora só falta implementar o $\chi^2$

$$
\begin{align}
{\chi}^2=\sum_{k=1}^{n} \frac{(y_i - f(x_i))^2}{\sigma_i^2 + \big(\frac{\partial f}{\partial x}\big)^2 \sigma_{x_i}^2 }
\end{align}
$$

Não impletarei, mas você pode fazer essa linha de código fácilmente sabendo a derivada analítica

### Gráficos

Vale ressaltar que o método plt.savefig() salvará os gráficos no diretório do .ipynb ou do .py

```python

ax2 = plt.figure(1)
ax2 = plt.errorbar(x, y, xerr=sigma_x, yerr=sigma_y, linestyle='None', marker='+',color='black')
#ax2 = plt.ylim(-1,2)
#ax2 = plt.xlim(0, 20000)
ax2 = plt.plot(x, f([2.01584576e-01, -3.87102908e+01, 1.85196706e+03],x), label='model',color='red')
# nomeando os eixos
ax2 = plt.xlabel(r'$x (cm)$')
ax2 = plt.ylabel(r'$y (cm)$')
ax2 = plt.grid()
#Salvando a figura
ax2 = plt.savefig('exemplo.png',dpi=150,bbox_inches = "tight")

ax3 = plt.figure(2)
ax3 = plt.errorbar(x, abs(y - f([2.01584576e-01, -3.87102908e+01, 1.85196706e+03],x))
                   ,xerr=sigma_x, yerr=sigma_y, linestyle='None', marker='+',color='black')

ax3 = plt.hlines(0,x.min(),x.max(),color='red')
ax3 = plt.grid()
# Nomeando os eixos
ax2 = plt.xlabel(r'$x (cm)$')
ax2 = plt.ylabel(r'$y(cm) - modelo$')
ax3 = plt.gcf()
ax3.set_size_inches(6, 1)
#Salvando a figura
ax3 = plt.savefig('exemplo_resid.png',dpi=150,bbox_inches = "tight")
```


```python
import numpy as np
import matplotlib.pyplot as plt
```

$$P_{(dx, dy)}=\|P(i, j)\|_{256 \times 256}$$

\begin{equation}
P(i, j)=\sum_{x=1}^{N} \sum_{y=1}^{M}\left\{\begin{array}{ll}
1, & \text { якщо } I(x, y)=i \text { та } I(x+d x, y + d y)=j ; \\
0, & \text { в інших випадках }
\end{array}\right.
\end{equation}

```python
def p(i,j, matr, d):
    n_rows, n_cols = matr.shape
    dx, dy = d

    res = 0
    for x in range(n_rows):
        for y in range(n_cols):
            # check for being in image's bounds
            props1 = [x + dx  < n_rows, y + dy < n_cols]

            if all(props1):
                if matr[x][y] == i and matr[x + dx][y + dy] == j:
                    res += 1

    return res

def coincidence_matr(image, d):
    """
    d -- (dx, dy) vector
    image -- N x M matrix (built by image)
    """

    res_matr = np.zeros((256, 256))
    vmin, vmax = image.min(), image.max()

    # it actually makes sense to look only at
    # rectangle (vmnin x vmax) and make the least
    # equals zero
    for i in range(vmin, vmax):
        for j in range(vmin, vmax):
            res_matr[i, j] = p(i, j, image, d)

    return res_matr

```

\begin{aligned}
\hat{P}=\frac{1}{8}\left(P_{(0, dy)}+P_{(0,-dy)}\right.&+P_{(-dx, dy)}+P_{(dx,-dy)}+\\
&\left.+P_{(-dx, 0)}+P_{(dx, 0)}+P_{(-dx,-dy)}+P_{(dx, dy)}\right)
\end{aligned}

```python
def mean_coincidence_matr(image, d):
    """
    image is given matrix
    d is (dx, dy) 2-dim vector
    """
    dx, dy = d

    D = [(0, dy), (0, -dy), (-dx, dy), (dx, -dy), (-dx,0),
    (dx, 0), (-dx, -dy), (dx, dy)]

    res_matr = np.zeros((256, 256))
    if d != (0, 0):
        for d_i in D:
            res_matr += coincidence_matr(ex1, d_i)

        return 1 / 8 * (res_matr)
    else:
        return coincidence_matr(ex1, d)
```

```python
matrices = []

for i in range(6):
    matrices.append([])
    with open(fr"D:\Documents\Курсова файли\needed_files\404\1_{i+1}_Gray.txt") as f:
        lines = f.readlines()
        for line in lines:
            t = [float(x) for x in line.split()]
            matrices[i].append(t)
```

```python
ex1 = np.array(matrices[0][:-1], dtype=int)
```

```python
plt.imshow(ex1, cmap='gray')
```

```python
plt.imshow(ex1, cmap='gray_r')
```

```python
%%time
d = (0, 1)

ex1_res_ = coincidence_matr(ex1, d)
```

    Wall time: 26.6 s

```python
plt.figure(dpi=100)
plt.imshow(ex1_res_, cmap='gray_r')
```

```python
%%time
ex1_res_mean = mean_coincidence_matr(ex1, d)
```

    Wall time: 3min 24s

```python
plt.figure(dpi=100)
plt.imshow(ex1_res_mean, cmap='gray_r')
```

```python

```


```python
import numpy as np
import random
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import LogisticRegression
from sympy.stats import Bernoulli, sample_iter
```

11. We defined a new variable based on weight (in pounds):

```
heavy = weight > 200 # 1 or 0
```

and then ran a logistic regression, predicting “heavy” from height (in inches):

```sh
glm(formula = heavy ~ height, family = binomial(link = "logit"))
            coef.est coef.se
(Intercept) -21.51     1.60
height        0.28     0.02
---
  n = 1984, k = 2
```

(a) Graph the logistic regression curve (the probability that someone is heavy) over the approximate range of the data. Be clear where the line goes through the 50% probability point.

(b) Fill in the blank: near the 50% point, comparing two people who differ by one inch in height, you’ll expect a difference of ____ in the probability of being heavy.

#### Responses

```python
toprob = lambda m: np.exp(m) / (1 + np.exp(m))

x = np.linspace(48,84, 84-47, endpoint=True)
y = np.array([toprob(random.normalvariate(-21.51, 1.6) + random.normalvariate(0.28, 0.02)*z) for z in x] )
```

```python

sns.set_style("dark")
ax = sns.scatterplot(x, y, cmap=cmap, facecolors="#9b59b6")
ax.set(xlabel='Height (inches)', ylabel='Probability heavy', ylim=[0,1])
plt.show()
```

#### Simulated values + logistic curve + 50% probaility line

```python
x = np.linspace(48,84, 84-47, endpoint=True)
sns.set_palette("Purples")
for i in range(500):
    y = [toprob(random.normalvariate(-21.51, 1.6) + random.normalvariate(0.28, 0.02)*z) for z in x]
    ax = sns.scatterplot(x,y, alpha=0.05, linewidth=0)

sns.set_style("dark")
yhat = [toprob(-21.51 + (0.28 * x)) for x in range(48,85)]
plt.plot(x, yhat, linewidth=5, c='green', alpha=1)
plt.plot([21.51/0.28, 21.51/0.28], [0,1], linewidth=1, c='red', alpha=1)
ax.set(xlabel='Height (inches)', ylabel='Probability heavy', ylim=[0,1])
plt.show()
```

#### Boostrap confidence interval

```python
x = np.linspace(48,84, 84-47, endpoint=True)
y = np.array([True if list(sample_iter(Bernoulli('X', v), numsamples=1))[0] else False for v in [toprob(random.normalvariate(-21.51, 1.6) + random.normalvariate(0.28, 0.02)*z) for z in x]])

X = np.vstack([x, np.ones(len(x))]).T

plt.figure(figsize=(12,8))

for i in range(0, 1000):
    sample_index = np.random.choice(range(0, len(y)), len(y))

    X_samples = X[sample_index]
    y_samples = y[sample_index]

    lr = LogisticRegression()
    lr.fit(X_samples, y_samples)
    yh = lr.predict_proba(X)[:,1]
    plt.plot(x, yh, color='grey', alpha=0.1, zorder=1)

plt.scatter(x,y, marker='o', color='orange', zorder=4)

lr = LogisticRegression()
lr.fit(X, y)
plt.plot(x, lr.predict_proba(X)[:,1], color='green', linewidth=3, zorder=5)
plt.show()
```

```python

```


# Bayesian classifier
In statistical classification, the Bayes classifier minimizes the probability of misclassification.

```python
import random
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
random.seed(42) # define the seed (important to reproduce the results)
```

```python
#data = pd.read_csv('data/vertebralcolumn-3C.csv', header=(0))
#data = pd.read_csv('data/BreastCancer.csv', header=(0))
data = pd.read_csv('data/Iris.csv', header=(0))
# data = pd.read_csv('data/Vehicle.csv', header=(0))
# data = pd.read_csv(r'data\pima-indians-diabetes.csv', index_col = 0)

data = data.dropna(axis='rows') #remove NaN
# armazena os nomes das classes
classes = np.array(pd.unique(data[data.columns[-1]]), dtype=str)

nrow, ncol = data.shape
print("Matriz de atributos: Número de linhas:", nrow, " colunas: ", ncol)
attributes = list(data.columns)
data.head(10)
```

    Matriz de atributos: Número de linhas: 150  colunas:  5

      sepal_length
      sepal_width
      petal_length
      petal_width
      species

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

      3
      4.6
      3.1
      1.5
      0.2
      setosa

      4
      5.0
      3.6
      1.4
      0.2
      setosa

      5
      5.4
      3.9
      1.7
      0.4
      setosa

      6
      4.6
      3.4
      1.4
      0.3
      setosa

      7
      5.0
      3.4
      1.5
      0.2
      setosa

      8
      4.4
      2.9
      1.4
      0.2
      setosa

      9
      4.9
      3.1
      1.5
      0.1
      setosa

Vamos construir as variáveis $X$ e $y$, sendo que o processo classificação se resume em estimar a função $f$ na relação $y = f(X) + \epsilon$, onde $\epsilon$ é o erro, que tem distribuição normal com média igual a zero e variância $\sigma^2$.

Convertemos os dados para o formato Numpy para facilitar a sua manipulação.

```python
data = data.to_numpy()
nrow,ncol = data.shape
y = data[:,-1]
X = data[:,0:ncol-1]
```

Vamos normalizar os dados, de modo a evitar o efeito da escala dos atributos.

```python
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X) # z = (x-u)/s
X = scaler.transform(X)

print('Dados transformados:')
print('Media: ', np.mean(X, axis = 0))
print('Desvio Padrao:', np.std(X, axis = 0))
```

    Dados transformados:
    Media:  [-4.73695157e-16 -6.63173220e-16  3.31586610e-16 -2.84217094e-16]
    Desvio Padrao: [1. 1. 1. 1.]

## Para treinar o classificador, precisamos definir o conjunto de teste e treinamento.

```python
from sklearn.model_selection import train_test_split
p = 0.8 # fracao de elementos no conjunto de treinamento
x_train, x_test, y_train, y_test = train_test_split(X, y, train_size = p, random_state = 42)
```

A partir desse conjunto de dados, podemos realizar a classificação.

## Classificador Bayesiano

Vamos considerar o caso paramétrico, assumindo que cada variável está distribuída de acordo com uma distribuição Normal. Outras distribuições também podem ser utilizadas.

Já selecionamos os conjuntos de treinamento e teste anteriormente. No conjunto de treinamento, vamos calcular a média e desvio padrão de cada atributo para cada classe. A seguir, realizamos a classificação, dos dados usando a teoria da decisão Bayesiana, isto é: $X \in C_i$ se, e somente se, $P(C_i|X) = \max P(C_j|X)$ para todo $j$.

```python
from scipy.stats import multivariate_normal

print('\n\ncódigo original\n')
#matrix to store the probabilities
P = pd.DataFrame(data=np.zeros((x_test.shape[0], len(classes))), columns = classes) # probability of each class
Pc = np.zeros(len(classes)) #fraction of elements in each class

    # para cada classe i
for i in np.arange(0, len(classes)):
    # tupla da posição em que o elemento é da classe i
    elements = tuple(np.where(y_train == classes[i]))

    #OBSERVAR AQUI
    Pc[i] = len(elements)/len(y_train)

    print(f'Tipo Elements:\t{type(elements)}')
    print(f'Tamanho Elements:{len(elements)}\nElements:{elements}\n')
    # Z tem as linhas com os elementos de cada classe e as colunas os atributos mesmo
    Z = x_train[elements,:][0]
    # média de elementos da classe para cada atributo da classe
    m = np.mean(Z, axis = 0)
    # covariancia entre os atributos de cada classe
    cv = np.cov(np.transpose(Z))

    ## calculo da probabilidade
    for j in np.arange(0,x_test.shape[0]):
        x = x_test[j,:]
        # probabilidade considerando a verossimilhança
        pj = multivariate_normal.pdf(x, mean=m, cov=cv, allow_singular=True)
        # prioris
        prior = Pc[i]
        # funciona assim porque o número de classes é discreto
        # probabilidade é a priori vezes a posteriori
        P[classes[i]][j] = pj*prior
        # eu não preciso dividir pela a informação porque é proporcional
```

    código original

    Tipo Elements:	<class 'tuple'>
    Tamanho Elements:1
    Elements:(array([  0,   1,   3,   4,   7,   8,   9,  13,  14,  23,  26,  27,  28,
            31,  32,  33,  35,  38,  41,  48,  51,  52,  55,  57,  58,  66,
            67,  70,  71,  72,  75,  78,  84,  91,  94,  98, 102, 104, 114,
           117], dtype=int64),)

    Tipo Elements:	<class 'tuple'>
    Tamanho Elements:1
    Elements:(array([  2,   6,  11,  12,  15,  18,  20,  22,  25,  29,  34,  36,  39,
            44,  45,  47,  49,  53,  54,  59,  60,  62,  65,  73,  79,  80,
            82,  86,  88,  89,  90,  92,  93,  95,  99, 105, 108, 110, 111,
           115, 118], dtype=int64),)

    Tipo Elements:	<class 'tuple'>
    Tamanho Elements:1
    Elements:(array([  5,  10,  16,  17,  19,  21,  24,  30,  37,  40,  42,  43,  46,
            50,  56,  61,  63,  64,  68,  69,  74,  76,  77,  81,  83,  85,
            87,  96,  97, 100, 101, 103, 106, 107, 109, 112, 113, 116, 119],
          dtype=int64),)

```python
from scipy.stats import multivariate_normal

print('código modificado\n')
#matrix to store the probabilities
P = pd.DataFrame(data=np.zeros((x_test.shape[0], len(classes))), columns = classes) # probability of each class
Pc = np.zeros(len(classes)) #fraction of elements in each class

    # para cada classe i
for i in np.arange(0, len(classes)):
    # tupla da posição em que o elemento é da classe i
    elements = tuple(np.where(y_train == classes[i]))

    #OBSERVAR AQUI
    Pc[i] = len(elements[0])/len(y_train)

    print(f'Tipo Elements modificado:\t{type(elements[0])}')
    print(f'Tamanho Elements modificado:{len(elements[0])}\nElements:{elements[0]}\n')

    # Z tem as linhas com os elementos de cada classe e as colunas os atributos mesmo
    Z = x_train[elements,:][0]
    # média de elementos da classe para cada atributo da classe
    m = np.mean(Z, axis = 0)
    # covariancia entre os atributos de cada classe
    cv = np.cov(np.transpose(Z))

    ## calculo da probabilidade
    for j in np.arange(0,x_test.shape[0]):
        x = x_test[j,:]
        # probabilidade considerando a verossimilhança
        pj = multivariate_normal.pdf(x, mean=m, cov=cv, allow_singular=True)
        # priori
        prior = Pc[i]
        # funciona assim porque o número de classes é discreto
        # probabilidade é a priori vezes a posteriori
        P[classes[i]][j] = pj*prior
        # eu não preciso dividir pela a informação porque é proporcional
print(f'Proporção das classes {Pc}\t Soma = {Pc.sum()}')
```

    código modificado

    Tipo Elements modificado:	<class 'numpy.ndarray'>
    Tamanho Elements modificado:40
    Elements:[  0   1   3   4   7   8   9  13  14  23  26  27  28  31  32  33  35  38
      41  48  51  52  55  57  58  66  67  70  71  72  75  78  84  91  94  98
     102 104 114 117]

    Tipo Elements modificado:	<class 'numpy.ndarray'>
    Tamanho Elements modificado:41
    Elements:[  2   6  11  12  15  18  20  22  25  29  34  36  39  44  45  47  49  53
      54  59  60  62  65  73  79  80  82  86  88  89  90  92  93  95  99 105
     108 110 111 115 118]

    Tipo Elements modificado:	<class 'numpy.ndarray'>
    Tamanho Elements modificado:39
    Elements:[  5  10  16  17  19  21  24  30  37  40  42  43  46  50  56  61  63  64
      68  69  74  76  77  81  83  85  87  96  97 100 101 103 106 107 109 112
     113 116 119]

    Proporção das classes [0.33333333 0.34166667 0.325     ]	 Soma = 1.0

## Predição nos dados de teste

```python
y_pred = []
#np.array(test_x.shape[0], dtype=str)
for i in np.arange(0, x_test.shape[0]):
    # c é a posição do maior
    c = np.argmax(np.array(P.iloc[[i]]))
    # aqui é a predição mesmo
    y_pred.append(classes[c])
y_pred = np.array(y_pred, dtype=str)
print(y_pred)
```

    ['versicolor' 'setosa' 'virginica' 'versicolor' 'versicolor' 'setosa'
     'versicolor' 'virginica' 'virginica' 'versicolor' 'virginica' 'setosa'
     'setosa' 'setosa' 'setosa' 'versicolor' 'virginica' 'versicolor'
     'versicolor' 'virginica' 'setosa' 'virginica' 'setosa' 'virginica'
     'virginica' 'virginica' 'virginica' 'virginica' 'setosa' 'setosa']

```python
from sklearn.metrics import accuracy_score
score = accuracy_score(y_pred, y_test)
print('Accuracy:', score)
```

    Accuracy: 0.9666666666666667

Código completo.

```python
import random
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from scipy.stats import multivariate_normal
from sklearn.metrics import accuracy_score

random.seed(42)

data = pd.read_csv('data/Vehicle.csv', header=(0))

classes = np.array(pd.unique(data[data.columns[-1]]), dtype=str)

# Converte para matriz e vetor do numpy
data = data.to_numpy()
nrow,ncol = data.shape
y = data[:,-1]
X = data[:,0:ncol-1]

# Transforma os dados para terem media igual a zero e variancia igual a 1
scaler = StandardScaler().fit(X)
X = scaler.transform(X)

# Seleciona os conjuntos de treinamento e teste
p = 0.8 # fraction of elements in the test set
x_train, x_test, y_train, y_test = train_test_split(X, y, train_size = p, random_state = 42)

####  Realiza a classificacao ####
# Matriz que armazena as probabilidades para cada classe
P = pd.DataFrame(data=np.zeros((x_train.shape[0], len(classes))), columns = classes)

Pc = np.zeros(len(classes)) # Armaze a fracao de elementos em cada classe
for i in np.arange(0, len(classes)): # Para cada classe
    elements = tuple(np.where(y_train == classes[i])) # elmentos na classe i
    Pc[i] = len(elements[0])/len(y_train) # Probabilidade pertencer a classe i
    Z = x_train[elements,:][0] # Elementos no conjunto de treinamento
    m = np.mean(Z, axis = 0) # Vetor media
    cv = np.cov(np.transpose(Z)) # Matriz de covariancia
    for j in np.arange(0,x_test.shape[0]): # para cada observacao no conjunto de teste
        x = x_test[j,:]
        # calcula a probabilidade pertencer a cada classe
        pj = multivariate_normal.pdf(x, mean=m, cov=cv, allow_singular=True)
        P[classes[i]][j] = pj*Pc[i]

y_pred = [] # Vetor com as classes preditas
for i in np.arange(0, x_test.shape[0]):
    c = np.argmax(np.array(P.iloc[[i]]))
    y_pred.append(classes[c])
y_pred = np.array(y_pred, dtype=str)
# calcula a acuracia
score = accuracy_score(y_pred, y_test)
print('Acuracia:', score)
```

    Acuracia: 0.8823529411764706

## Caso não paramétrico

Para o caso unidimensional, seja $(X_1,X_2, \ldots, X_n)$ uma amostra aleatória unidimensional identicamente distribuída de acordo com alguma função de distribuição $f$ não conhecida. Para estimarmos o formato de $f$, usamos um estimador (kernel density estimator):
\begin{equation}
\widehat{f}_{h}(x)={\frac {1}{n}}\sum _{i=1}^{n}K_{h}(x-x_{i})={\frac {1}{nh}}\sum _{i=1}^{n}K{\Big (}{\frac {x-x_{i}}{h}}{\Big )},
\end{equation}
onde $K$ é a função kernel.
A estimação depende do parâmetro $h$, que é um parâmetro livre e controla a abertura da função.

```python
import numpy as np
import matplotlib.pyplot as plt

N = 20
# gera os dados
X = np.array([1, 2, 3, 4, 12, 20,21,22,23,24,40,41, 50])
X = X.reshape((len(X), 1))
# mostra os dados
plt.figure(figsize=(12,4))
plt.plot(X[:, 0], 0.001*np.ones(X.shape[0]), 'ok')

# valores x para serem usados nas densidades
X_plot = np.linspace(np.min(X)-5, np.max(X)+5, 1000)[:, np.newaxis]

h=2
fhat = 0 # estimacao obtida
for x in X:
    # distribuição normal centrada em x
    # kernel é gaussiano, dá para usar kerneis diferentes
    f = (1/np.sqrt(2*np.pi*h))*np.exp(-((X_plot - x)**2)/(2*h**2))
    fhat = fhat + f # acumula as distribuições
    plt.plot(X_plot,f, '--', color = 'blue', linewidth=1)

# mostra a distribuição estimada
plt.plot(X_plot,fhat/(len(X)*np.sqrt(h)), color = 'green', linewidth=2)
plt.xlabel('x', fontsize = 20)
plt.ylabel('P(x)', fontsize = 20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)
plt.ylim((0, 0.3))
plt.savefig('kernel-ex.eps')
plt.show(True)
```

Esse resultado pode ser obtido usando-se a função KernelDensity scikit-learn.

```python
import numpy as np
from matplotlib.pyplot import cm
from sklearn.neighbors import KernelDensity

color=['red', 'blue', 'magenta', 'gray', 'green']

N = 20
X = np.array([1, 2, 3, 4, 12, 20,21,22,23,24,40,41, 50])
X = X.reshape((len(X), 1))

plt.figure(figsize=(12,4))
plt.plot(X[:, 0], 0.001*np.ones(X.shape[0]), 'ok')

X_plot = np.linspace(np.min(X)-5, np.max(X)+5, 1000)[:, np.newaxis]

h=2
fhat = 0
for x in X:
    f = (1/np.sqrt(2*np.pi*h))*np.exp(-((X_plot - x)**2)/(2*h**2))
    fhat = fhat + f
    plt.plot(X_plot,f, '--', color = 'blue', linewidth=1)

kde = KernelDensity(kernel='gaussian', bandwidth=h).fit(X)
log_dens = np.exp(kde.score_samples(X_plot)) # score_samples() returns the log density.
plt.plot(X_plot,log_dens, color = 'red', linewidth=2, label = 'h='+str(h))
plt.xlabel('x', fontsize = 20)
plt.ylabel('P(x)', fontsize = 20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)
plt.ylim((0, 0.3))

plt.show(True)
```

Notem que o formato da estimação depende do parâmetro livre $h$.

```python
import numpy as np
from matplotlib.pyplot import cm

color=['red', 'blue', 'gray', 'black', 'green', 'lightblue']

N = 20
X = np.array([1, 2, 3, 4, 12, 20,21,22,23,24,40,41, 50])
X = X.reshape((len(X), 1))

X_plot = np.linspace(np.min(X)-5, np.max(X)+5, 1000)[:, np.newaxis]

plt.figure(figsize=(12,4))
plt.plot(X[:, 0], 0.001*np.ones(X.shape[0]), 'ok')
c = 0
vh = [0.1, 0.5, 1, 2, 5, 10]
for h in vh:
    kde = KernelDensity(kernel='gaussian', bandwidth=h).fit(X)
    log_dens = np.exp(kde.score_samples(X_plot)) # score_samples() returns the log density.
    plt.plot(X_plot,log_dens, color = color[c], linewidth=2, label = 'h='+str(h))
    c = c + 1
plt.ylabel('P(x)', fontsize = 20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)
#plt.ylim((0, 0.2))
plt.legend(fontsize = 10)
#plt.savefig('kernel.eps')
plt.show(True)
```

Notem que a estimação é relacionada com a estimação usando-se histogramas.

```python
import numpy as np

N = 20
X = np.array([1, 2, 3, 4, 12, 20,21,22,23,24,40,41, 50])
X = X.reshape((len(X), 1))

plt.figure(figsize=(10,5))
# Histogram
nbins = 10
plt.hist(X,bins = nbins, density = True, color='gray',alpha=0.7, rwidth=0.95)

#Kernel density estimation
X_plot = np.linspace(np.min(X)-5, np.max(X)+5, 1000)[:, np.newaxis]
kde = KernelDensity(kernel='gaussian', bandwidth=2).fit(X)
log_dens = np.exp(kde.score_samples(X_plot)) # score_samples() returns the log density.
plt.plot(X_plot,log_dens, color = 'blue', linewidth=2)
plt.plot(X[:, 0], 0.001*np.ones(X.shape[0]), 'ok')
plt.xlabel('x', fontsize = 15)
plt.ylabel('P(x)', fontsize = 15)

plt.show(True)
```

Usando o método *kernel density estimation*, podemos realizar a classificação.

```python
import random
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KernelDensity
from sklearn.metrics import accuracy_score

random.seed(42)
data = pd.read_csv('data/Iris.csv', header=(0))
# data = pd.read_csv('data/Vehicle.csv', header=(0))

classes = np.array(pd.unique(data[data.columns[-1]]), dtype=str)

# Converte para matriz e vetor do numpy
data = data.to_numpy()
nrow,ncol = data.shape
y = data[:,-1]
X = data[:,0:ncol-1]

# Transforma os dados para terem media igual a zero e variancia igual a 1
scaler = StandardScaler().fit(X)
X = scaler.transform(X)

# Seleciona os conjuntos de treinamento e teste
p = 0.8 # fraction of elements in the training set
x_train, x_test, y_train, y_test = train_test_split(X, y, train_size = p, random_state = 42)

# Matriz que armazena as probabilidades para cada classe
P = pd.DataFrame(data=np.zeros((x_test.shape[0], len(classes))), columns = classes)
Pc = np.zeros(len(classes)) # Armaze a fracao de elementos em cada classe
h = 2
for i in np.arange(0, len(classes)): # Para cada classe
    elements = tuple(np.where(y_train == classes[i])) # elmentos na classe i
    Pc[i] = len(elements[0])/len(y_train) # Probabilidade pertencer a classe i
    Z = x_train[elements,:][0] # Elementos no conjunto de treinamento
    kde = KernelDensity(kernel='gaussian', bandwidth=h).fit(Z)
    for j in np.arange(0,x_test.shape[0]): # para cada observacao no conjunto de teste
        x = x_test[j,:]
        x = x.reshape((1,len(x)))
        # calcula a probabilidade pertencer a cada classe
        pj = np.exp(kde.score_samples(x))
        P[classes[i]][j] = pj*Pc[i]

y_pred = [] # Vetor com as classes preditas
for i in np.arange(0, x_test.shape[0]):
    c = np.argmax(np.array(P.iloc[[i]]))
    y_pred.append(classes[c])
y_pred = np.array(y_pred, dtype=str)
# calcula a acuracia
score = accuracy_score(y_pred, y_test)
print('Acuracia:', score)
```

    Acuracia: 0.9666666666666667


# Decision Trees and Random Forest

## Decision Trees

### Motivating Decision Trees

Decision trees are extremely intuitive ways of classifying data: you simply ask a series of questions in order to close in on the class label. For example, if you are into value investing, you might think of the following scheme to decide whether you invest into a certain stock or not:

(P/B = Price/Book ratio, P/E = Price/Earnings ratio, PEG = Price/Earnings to growth ratio, E = Equity, D = Debt).

Sources tell me this scheme is not Warren Buffet's key to success - it is obviously an overly simplified illustration. But it serves well to demonstrate the idea behind decision trees in ML. The binary splits narrow down the options. The big question here though is of course what questions we ought to ask to derive the desired answer and in what sequence. This we will discuss in the next sections - first intuitively and then in more rigorous terms.

### Simple Decision Trees Illustrated

To draw up a simple decision tree, we follow these two steps (James et al. (2013)):
* Divide the predictor space (set of possible values for $X_1, X_2, \ldots, X_p$) into $M$ distinct and non-overlapping regions $R_1, R_2, \ldots, R_M$.
* For every observation in $R_m$ we model the response as a constant. This constant is based on a majority vote among the observation in $R_m$.

Let us illustrate this on a two dimensional data set with features $X_1$ and $X_2$. The color of the dots indicate the true class label.

Without any split, we have but one region. A decision tree now splits the regions iteratively into predictor spaces $R_m$. This is shown below. The first figure on the left has two spaces, $R_1, R_2$, The second already has four ($R_1, R_2, R_3, R_4$) etc.. The background color indicates the label that the model would assign to a new data point in the respective area. Note that (as in the introductory decision tree figure) each branch can have a sperate number of splits. Some nodes are purer (get split more) than others.

The `depth=n` argument in the figure title refers to the depth of the tree. Nodes that contain only a single class (color) are not further split. All others will be further split until a stopping criterion is reached, e.g.
* all predictor spaces are pure (contain only one class),
* all predictor spaces contain a limited number of data points,
* we limit the number of splits through the `depth` argument etc.

### Mathematical Description
#### Maximizing Information Gain

How do we construct the regions $R_1, \ldots, R_M$? Or put in other words: How do we decide on the splitting variables (i.e. $X_1, X_2, \ldots, X_p$), split points and what topology (shape) the tree should have? To explain this we focus on the CART (classification and regression tree) approach of decision trees as implemented in Scikit-learn.

In order to describe the mathematics let us assume we have a data set with $p$ inputs for each of the $N$ observations: $(x_i, y_i)$ for $i = 1, 2, \ldots, N$, with $x_i = (x_{i1}, x_{i2}, \ldots, x_{ip})$.

Most libraries (including Scikit-learn) have implemented binary decision trees, meaning that each node is split into two child nodes. Hence for binary splits at node $m$ we take initial region $R_m$ and select $j$ (of feature $X_j$) and threshold $t_m$ such that the resulting two half-planes

$$\begin{equation}
R_{\text{left}}(R_m; j, t_m) = \{(X, y) \, | \, X_j < t_m \} \qquad \text{and} \qquad R_{\text{right}}(R_m; j, t_m) = \{(X, y) \, | \, X_j \geq t_m \}
\end{equation}$$

maximize the **information gain ($G$)** for any value of $j$ and $t_m$.

Let us define $\theta = (j, t_m)$ to simplify expressions. Given $R_m$ with $N_m$ being the total number of samples in $R_m$ and $n_{\text{left}}$, $n_{\text{right}}$ the number samples in the left ($R_{\text{left}}$) and right ($R_{\text{right}}$) child nodes, respectively, the information gain function $G$ is defined as

$$\begin{equation}
G(R_m; \theta) = H(R_m) - \left( \frac{n_{\text{left}}}{N_m} H(R_{\text{left}}) + \frac{n_{\text{right}}}{N_m} H(R_{\text{right}})\right)
\end{equation}$$

Here, $H()$ is simply a measure of impurity which we will get to shortly. With that, the information gain $G$ is simply the difference between the impurity of the parent node and the sum of the child node impurities. The lower the impurity of the child nodes, the larger information gain we get (Raschka (2015)).

Our optimization task can therefore be formulated as to find parameter $\theta$ that maximizes the following expression at every node $m$:

$$\begin{equation}
\theta^* = \arg \max_{\theta} G(R_m; \theta)
\end{equation}$$

This is done recursively for every node until the stopping criteria is reached (max. depth, max. number of samples in region etc.; see above).

#### Impurity Measures for Classification Trees

There are three common impurity measures (or splitting criteria), of which only the latter two are recommended for growing decision trees: Classification error rate ($H_E$), Gini index ($H_G$), cross-entropy ($H_H$). To discuss them, let us first define the proportion of class $k$ observations in node $m$ (Friedman et al. (2001)):

$$\begin{equation}
\hat{p}_{m,k} = \frac{1}{N_m} \sum_{x_i \in R_m} I(y_i = k)
\end{equation}$$

Earlier we mentioned that all observations in region $R_j$ are assigned to the same class following a majority vote. In more formal terms this means that observations in node $m$ are classified to class $k$ for which $k(m) = \arg \max_k \hat{p}_{m,k}$.

Now, the impurity measures are defined as follows:

$$\begin{align}
&\text{Classification Error rate: } & H_E(R_m) &= 1 - \max_k (p_{m,k}) \\
&\text{Gini Index: } & H_G(R_m) &= \sum_k p_{m,k} (1 - p_{m,k}) \\
&\text{Cross-Entropy: } & H_E(R_m) &= - \sum_k p_{m,k} \, \log(p_{m,k})
\end{align}$$

Note that for binary classification tasks (e.g. $y \in \{0, 1\}$), $\log$ in the cross-entropy is usually the [logarithm to the base 2.](https://stackoverflow.com/questions/1859554/what-is-entropy-and-information-gain)

Below figure graphs the three impurity measures with respect to $p$. As mentioned, the classification error rate should not be used as an impurity measure. In practice it is primarily used to prune a tree - a concept we will not discuss here. Cross-entropy is minimal if all samples at a node belong to the same class and maximal if we have a homogeneous distribution among the classes. Therefore, cross-entropy can be understood as a criterion that attempts to maximize the mutual information in the tree. The Gini-index on the other hand works towards minimizing the probability of misclassification. Similar to entropy it is maximal if classes are evenly distributed and minimal if the vast majority of samples belong to the same class.

In practice both Gini index as well as cross-entropy usually produce similar results and thus it is not advisable to spend much time on evaluating trees using different impurity criteria (Raschka (2015)).

### Advantages and Disadvantages of Decision Trees

Decision trees have several advantages over the other classification approaches discussed so far:

* Intuition is easy to explain
* Trees can be displayed graphically and are easily interpreted even by a layperson
* Trees handle quantitative as well as qualitative features; there's no need to scale the values
* Some people argue that decision trees mirror human thinking very closely, moreso than other approaches.

However, simple decision trees have also disadvantages, some of them are significant:
* Instability of trees/high variance: Small changes in the data often result in very different series of splits
* Decision trees tend to build complex decision boundaries, which often results in overfitting the data
* Low level of predictive accuracy compared to other classification approaches discussed in this course.

### Decision Trees in Python

To show how decision trees are applied in Python we once again rely on functions implemented in the Scikit-learn package. This time we will use the `Carseats` data set. It is again a data set that corresponds to James et al. (2013)'s book and contains information on child carseat sales at 400 different stores. `Sales` is the response variable with number of sold units in thousands. All other values are used as features. A detailed description of the data set can be found [here](https://cran.r-project.org/web/packages/ISLR/ISLR.pdf).

```python
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv('Data/Carseats.csv')
df.head()
```

      Sales
      CompPrice
      Income
      Advertising
      Population
      Price
      ShelveLoc
      Age
      Education
      Urban
      US

      0
      9.50
      138
      73
      11
      276
      120
      Bad
      42
      17
      Yes
      Yes

      1
      11.22
      111
      48
      16
      260
      83
      Good
      65
      10
      Yes
      Yes

      2
      10.06
      113
      35
      10
      269
      80
      Medium
      59
      12
      Yes
      Yes

      3
      7.40
      117
      100
      4
      466
      97
      Medium
      55
      14
      Yes
      Yes

      4
      4.15
      141
      64
      3
      340
      128
      Bad
      38
      13
      Yes
      No

Let us assume you are a financial consultant and work on a market study that discusses appropriate sales channels for child car seats. You know that your client's strategy is to grow the business. Furthermore you learned that to run sales at break even at a sales location the company needs to sell approximately 4'000 units. What feature drives sales and in which stores would you advise your client to offer their product? Here a decision tree helps very much in visualizing the sales driver.

Before we implement the model we prepare the data. Notice that decision trees are invariant to scaling meaning we could, but don't have to scale values. This is true for both quantitative as well as categorical variables. What we need to do though, is transforming categorical values `ShelveLoc, Urban` and `US` into numeric values. We will use pandas `map` method to (a) show an alternative to `pd.factorize` introduced in previous chapters and (b) ensure a mapping that does not confuse (`pd.factorize()` maps a column's first entry to value 0, second to 1 etc. This would mean that `pd.factorize()` would label `Yes` in columns `Urban` or `US` as 0. Yet `Yes` is predominantly represented by a 1. Therefore, with the `map` function we preclude any confusion).

```python
# Create 'BreakEven' column with 1 if Sales >= 4k, else 0
df['BreakEven'] = df.Sales.map(lambda x: 1 if x>=4 else 0)

# Replace category names with numbers
df.ShelveLoc = df.ShelveLoc.map({'Bad':0, 'Medium': 1, 'Good': 2})
df.Urban     = df.Urban.map({'No':0, 'Yes':1})
df.US        = df.US.map({'No':0, 'Yes':1})
print(df.head(3))
```

       Sales  CompPrice  Income  Advertising  Population  Price  ShelveLoc  Age  \
    0   9.50        138      73           11         276    120          0   42
    1  11.22        111      48           16         260     83          2   65
    2  10.06        113      35           10         269     80          1   59

       Education  Urban  US  BreakEven
    0         17      1   1          1
    1         10      1   1          1
    2         12      1   1          1

```python
# Assign features & response to X and y, respectively
X = df.drop(['Sales', 'BreakEven'], axis=1)
y = df.BreakEven

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
```

Now we are in a position to run the `DecisionTreeClassifier`.

```python
from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=4)
tree.fit(X_train, y_train)
```

    DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                           max_depth=4, max_features=None, max_leaf_nodes=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, presort='deprecated',
                           random_state=None, splitter='best')

Creating a decision tree is as simple as that. From the above output we see that per default the function will use the Gini index as criterion. The only argument we set is the maximal depth. Alternatively we could define the maximum tree nodes, the minimum number of samples required to split an internal node or the minimum number of samples required to be at a leaf node. There are more options and it is best to check the [documentation page](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier) to get a thorough understanding of the available options.

As with all other Scikit-learn functions, we can again call the usual performance metrics. The interpretation is as discussed in chapter 8.

```python
# Load metrics sublibrary
from sklearn import metrics

# Print performance metrics
print('True proportion of sales >= 4k: ', y.sum() / y.shape[0])
print('Train score: ', tree.score(X_train, y_train))
print('Test score: ', tree.score(X_test, y_test))
print(37*'-')

# Confusion matrix
y_pred = tree.predict(X_test)
print('Confusion matrix: \n',
      metrics.confusion_matrix(y_test, y_pred))

# Manual confusion matrix as pandas DataFrame
confm = pd.DataFrame({'Predicted sales >=4k': y_pred,
                      'True sales >=4k': y_test})
confm.replace(to_replace={0:'No', 1:'Yes'}, inplace=True)
print(confm.groupby(['True sales >=4k','Predicted sales >=4k']).size().unstack('Predicted sales >=4k'))
```

    True proportion of sales >= 4k:  0.91
    Train score:  0.94375
    Test score:  0.8875
    -------------------------------------
    Confusion matrix:
     [[ 0  4]
     [ 5 71]]
    Predicted sales >=4k   No   Yes
    True sales >=4k
    No                    NaN   4.0
    Yes                   5.0  71.0

### Visualizing Decision Trees

Visualizing decision trees was a horrendously cumbersome task in the past. However, starting Scikit-learn version 0.21 there is now a function that plots the decision tree very easily. The following Code snippet displays the necessary steps. For further details - as usual - visit the [Scikit-learn website](https://scikit-learn.org/stable/modules/generated/sklearn.tree.plot_tree.html). Be sure to have the list of class names for parameter `class_names` in the correct order.

```python
import matplotlib.pyplot as plt
from sklearn.tree import plot_tree

# Plot tree
plt.figure(figsize=(14, 6))
plot_tree(tree, class_names=['loss', 'profit'],
          filled=True, rounded=True);
```

### Decision Trees and Cross Validation

We have mentioned above that one of the major disadvantages of decision trees is the risk of overfitting. Having discussed cross validation in the previous chapter, we ought to ask the question how cross validation can be used to prevent overfitting. The answer to this seemingly trivial question might be more complex than initially thought. If we apply a $k$-fold cross validation and generate $k$ decision tree for every fold in the training set, then these $k$ tree will most certainly look different from fold to fold. Each of the $k$ decision trees will still suffer from overfitting. Furthermore, CV does not tell us anything about which of the $k$ tree we are supposed to select for prediction, and ultimately, this ought to be our goal. If CV produces 10 different trees but doesn't tell us which one of these to choose, we have gained nothing. Choosing the one tree (among the $k$) with (for example) the lowest error rate will not do it as this approach is most probably flawed: such a model is based on even less information than what would be available through the full training set and in general models with more information beat those with less. So CV will not help us on this end.

However, CV is still of use. To recapitulate the idea behind CV: Fundamentally, the purpose of cross validation is not to help select a particular instance of the $k$ decision trees but rather to qualify the model, i.e. to provide metrics such as error rate etc. which in turn can be useful in asserting the level of precision one can expect from the model. Therefore, CV comes into play when we are tuning the model to find the optimal hyperparameter. As an example: usually we do not know the optimal tree form. Does it generalize best when it has depth 2, 5, 10? Or is a stopping criterion of no more than 5, 10, 20 observation per region $R_j$ best? Here we can run different parameter in combination with CV and this, thus, will provide an answer on how well the model will generalize to new data - given the set of hyperparameter.

#### Cross Validation
Below we show two setups to do this: with loops or grid search. The first approach follows what we learned in the previous chapter on cross validation.

```python
# Max depth
maxDepth = np.array([1, 2, 5, 10])

# Minimum number of samples required to split any internal node
minSamplesNode = np.array([2, 5, 10, 20])

# The minimum number of samples required to be at a leaf/terminal node
minSamplesLeaf = np.array([2, 5, 10, 20])
```

```python
# Import necessary functions
from sklearn.model_selection import StratifiedKFold, cross_val_score

# Create k-Fold CV object
kFold = StratifiedKFold(n_splits=10)

# Loop through maxDept values, run CV and print results
for i in maxDepth:
    tree = DecisionTreeClassifier(max_depth=i, random_state=0)
    scrs = cross_val_score(tree, X_train, y_train, cv=kFold)
    print('Score (depth ={0: 3.0f}): {1: .3f} +/- {2: .3f}'.format(i, np.mean(scrs), np.std(scrs)))

print(50*'-')

# Loop through minSamplesNode values, run CV and print results
for i in minSamplesNode:
    tree = DecisionTreeClassifier(min_samples_split=i, random_state=0)
    scrs = cross_val_score(tree, X_train, y_train, cv=kFold)
    print('Score (min sample at node ={0: 3.0f}): {1: .3f} +/- {2: .3f}'.format(i, np.mean(scrs), np.std(scrs)))

print(50*'-')

# Loop through minSamplesNode values, run CV and print results
for i in minSamplesLeaf:
    tree = DecisionTreeClassifier(min_samples_leaf=i, random_state=0)
    scrs = cross_val_score(tree, X_train, y_train, cv=kFold)
    print('Score (min sample at leaf ={0: 3.0f}): {1: .3f} +/- {2: .3f}'.format(i, np.mean(scrs), np.std(scrs)))
```

    Score (depth =  1):  0.891 +/-  0.029
    Score (depth =  2):  0.866 +/-  0.044
    Score (depth =  5):  0.850 +/-  0.031
    Score (depth = 10):  0.856 +/-  0.035
    --------------------------------------------------
    Score (min sample at node =  2):  0.856 +/-  0.035
    Score (min sample at node =  5):  0.847 +/-  0.041
    Score (min sample at node = 10):  0.856 +/-  0.047
    Score (min sample at node = 20):  0.844 +/-  0.066
    --------------------------------------------------
    Score (min sample at leaf =  2):  0.841 +/-  0.045
    Score (min sample at leaf =  5):  0.869 +/-  0.041
    Score (min sample at leaf = 10):  0.866 +/-  0.037
    Score (min sample at leaf = 20):  0.863 +/-  0.049

Based on the output we can conclude that we get better scores with fewer nodes. As for the min sample at a node/leaf, the differences are too small to judge. However, what we can also conclude is that this is a fairly cumbersome process. Three separate loops to find the optimal hyperparameter. Furthermore, these three loops just check for one criterion, but what if we were interested in **all possible combinations**? Maybe we get better results if we combine them - we only know if we check. And, as you should be expecting by now, there's a convenient way of doing this: via grid search.

#### Grid Search
The approach of grid search is fairly simple: it's a brute-force search paradigm where we specify a list of values for different hyperparameters. The algorithm evaluates the model performance for each combination of hyperparameter to obtain the optimal combination of values from this set (Raschka (2015)). As usual we use a code example to show how this works.

```python
from sklearn.model_selection import GridSearchCV

# Define the hyperparameter values to be tested
param_grid = {'criterion': ['gini', 'entropy'],
              'max_depth': maxDepth,
              'min_samples_split': minSamplesNode,
              'min_samples_leaf': minSamplesLeaf}

# Run brute-force grid search
gs = GridSearchCV(estimator=DecisionTreeClassifier(random_state=0),
                  param_grid=param_grid,
                  scoring='accuracy',
                  cv=kFold, n_jobs=-1)
gs = gs.fit(X_train, y_train)
print(gs.best_score_)
print(gs.best_params_)
```

    0.9
    {'criterion': 'gini', 'max_depth': 1, 'min_samples_leaf': 20, 'min_samples_split': 2}

Using the preceding code, we train and tune a `DecisionTreeClassifier` on the given parameter grid. For this we define a dictionary called `param_grid` and apply this to the `GridSearchCV`. Using the training data we obtain the score of the best-performing model via the `best_score_` attribute (here based on the accuracy measure) and the corresponding parameter via the `best_params_`.

As it turns out, we are unable to increase the accuracy score by combining stopping criterion outside of the `min_samples_leaf` hyperparameter. The best result is indeed when we use a max. depth of 1 and `min_samples_split` criterion of 2. If a combination of the three criterion (or the entropy criterion) were to yield better results, it would mean that the latter two hyperparameter would have values $\geq$ than the minimum value of 1 and 2, respectively.

It is important to note that this result should not deceive you to believe that overfitting with decision trees is a myth. Here we seem to come across a rare exception where a prune tree of depth 1 yields the best performance. In general, decision trees have much better training results on deeper grown trees. Hence the risk of overfitting.

> Note that grid search might be a convenient and powerful way of tuning hyperparameter but because it is a brute-force approach it is computationally very expensive. Depending on the number of processors you run your script on and the task at hand this might take substantial time. If, for whatever reasons, this is not feasible, the `RandomizedSearchCV` class might be a feasible alternative. This class draws random parameter from sampling distributions with a specified budget. See [the documentation for more details](http://scikit-learn.org/stable/modules/grid_search.html#randomized-parameter-optimization).

Finally, to estimate the performance of these parameter on the independent test dataset, we can run these three lines:

```python
# Extract best parameter
clf = gs.best_estimator_

# Fit model given best parameter
clf.fit(X_train, y_train)

# Print out score on Test dataset
print('Test accuracy: {0: .4f}'.format(clf.score(X_test, y_test)))
```

    Test accuracy:  0.9500

## Random Forest
### Turning Weaknesses into Strengths

CV is so valuable because it provides reliable information on how well a model generalizes. Recall that given a set of $n$ independent observations $Z_1, Z_2, \ldots, Z_n$, each with variance $\sigma^2$, the variance of the mean $\bar{Z}$ of the observations is given by $\sigma^2/n$ (see appendix of the script). This shows that if we average a set of (independent) performance metrics (e.g. classification error), we actually reduce the variance of this error. And in doing so, we increase the validity of said metric.

If we now extend this idea to not only assessing performance metrics but also predicting outcomes, we enter the field of **ensemble models**. These models produce $k$ independent predictions (e.g. trees) and assign the class label based on a majority vote of the $k$ outcomes. In doing so we lower the variance without compromising on the low bias of decision trees. Therefore it is easy to see that ensemble methods have proven to be extremely valuable, especially in the field of decision trees. One of the more prominent ensemble algorithm with respect to decision trees is called 'Random Forest'.

Let us first define the steps of a random forest model (Raschka (2015, p. 90)):

1. Draw a random bootstrap sample of size $n$ (randomly choose $n$ samples from the training set with replacement)
2. Grow a decision tree from the bootstrap sample. At each node:
    1. Randomly select $m$ features without replacement (with $m < p$)
    2. Split the node using the feature that provides the best split according to the objective function (e.g. by maximizing the information gain)
3. Repeat steps 1. and 2. $B$ times
4. Aggregate the $B$ predictions (of each tree) and assign the class label by majority vote

Step two above might seem odd at first: Why would we restrict the model to only choose from a subset $m$ of features (instead of selecting from the complete set $p$)? This is best explained with James et al. (2013, p. 320):

> "Suppose that there is one very strong predictor in the data set, along with a number of other moderately strong predictors. Then in the collection of bagged trees, most or all of the trees will use this strong predictor in the top split. Consequently, all of the bagged trees will look quite similar to each other. Hence the predictions from the bagged trees will be highly correlated. Unfortunately, averaging many highly correlated quantities does not lead to as large of a reduction in variance as averaging many uncorrelated quantities. [...] Random forests overcome this problem by forcing each split to consider only a subset of the predictors. Therefore, on average $(p - m)/p$ of the splits will not even consider the strong predictor, and so other predictors will have more of a chance. We can think of this process as *decorrelating* the trees, thereby making the average of the resulting trees less variable and hence more reliable."

On a side note: There exists a predecessor algorithm that works similar to random forests except that it sets $m=p$ in step 2.1 per default. Python has it implement as `BaggingClassifier()`. Since random forests improves on the problem of correlated features, it is today clearly the preferred approach. For this reason we will only discuss the random forest implementation.

### Selecting Random Forest's Hyperparameter

Though the interpretability of a random forest does not meet the simplicity of a simple decision tree, a big advantage is that we do not have to worry that much about choosing good hyperparameter values. We have three primary values to set:
* The size $n$ of the bootstrap (step 1)
* The subset $m$ of possible features (step 2)
* The number of iterations $B$ (step 3)

Typically, the larger number of trees $B$, the better the performance of our random forest classifier. But this of course comes at the expense of (potentially significant) increased computational costs. Additionally, the marginal improvement decreases as the number of trees is increased, i.e. at a certain point the cost in computation time will outgrow the benefit in prediction accuracy from more trees. In the Scikit-learn implementation, this hyperparameter is steered through the `n_estimators` argument.

The feature subset size ($m$) to consider at each node is typically set to $m = \sqrt{p}$, that is, the number of predictors considered at each split is approximately equal to the square root of the total number of predictors $p$. Scikit-learn uses the `max_feature` argument to control for it.

Finally, via the size $n$ of the bootstrap we control the bias-variance trade-off. A large value for $n$ will decrease randomness and thus such a model is more likely to overfit. On the other hand, preventing overfitting by selecting smaller values come at the expense of the model predictive performance. And since predictive accuracy is what we are most interested in, the vast majority of random forest implementations, including the `RandomForestClassifier` implementation in Scikit-learn, have set the bootstrap sample size $n$ per default to the number of samples in the original training set. This provides a good bias-variance trade-off (Raschka (2015)).

### Random Forest in Scikit-Learn

There is an easily accessible implementation of a random forest classifier in Scikit-learn that we can use. For [a description of the available hyperparameter please check again the function's documentation](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html). It is also left to the reader to investigate how CV and/or grid-search can improve the performance. For this the same steps as explained for the `DecisionTreeClasifier` function can be applied.

```python
from sklearn.ensemble import RandomForestClassifier

# Create classifier object and fit it to data
forest = RandomForestClassifier(criterion='gini', random_state=0, n_jobs=-1)
forest.fit(X_train, y_train)
```

    RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                           criterion='gini', max_depth=None, max_features='auto',
                           max_leaf_nodes=None, max_samples=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, n_estimators=100,
                           n_jobs=-1, oob_score=False, random_state=0, verbose=0,
                           warm_start=False)

```python
# Print test score
print('Test accuracy: {0: .4f}'.format(clf.score(X_test, y_test)))
```

    Test accuracy:  0.9500

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Friedman, Jerome, Trevor Hastie, and Robert Tibshirani, 2001, *The Elements of Statistical Learning* (Springer, New York, NY).
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Raschka, Sebastian, 2015, *Python Machine Learning* (Packt Publishing Ltd., Birmingham, UK)
* Vanderplas, Jake, 2016, *Python Data Science Handbook* (O'Reilly Media, Sebastopol, CA)


```python
import numpy as np
```

<div style="width:image width px; font-size:100%; text-align:center;"> Figure 1 </div>

# ReLU
\begin{align}
 \mathbf{z_1} &= \mathbf{W_1}\mathbf{x} + \mathbf{b_1}  \tag{1} \\
 \mathbf{h} &= \mathrm{ReLU}(\mathbf{z_1})  \tag{2} \\
\end{align}

```python
np.random.seed(10)
```

```python
z1= 10 * np.random.rand(5,1) - 5
```

```python
z1
```

    array([[ 2.71320643],
           [-4.79248051],
           [ 1.33648235],
           [ 2.48803883],
           [-0.01492988]])

```python
h = z1.copy()
```

```python
h<0
```

    array([[False],
           [ True],
           [False],
           [False],
           [ True]])

```python
h[h<0] = 0
```

```python
h
```

    array([[2.71320643],
           [0.        ],
           [1.33648235],
           [2.48803883],
           [0.        ]])

```python
np.max(h,0)
```

    array([2.71320643])

```python
np.array([max(i,0) for i in h])
```

    array([[2.71320643],
           [0.        ],
           [1.33648235],
           [2.48803883],
           [0.        ]])

```python
for i in h:
    if i<0:
        h[i] = 0
h

```

    array([[2.71320643],
           [0.        ],
           [1.33648235],
           [2.48803883],
           [0.        ]])

```python
def relu(z):
    result = z.copy()
    result[result<0]=0
    return result

```

```python
z1
```

    array([[ 2.71320643],
           [-4.79248051],
           [ 1.33648235],
           [ 2.48803883],
           [-0.01492988]])

```python
relu(z1)
```

    array([[2.71320643],
           [0.        ],
           [1.33648235],
           [2.48803883],
           [0.        ]])

# Softmax

\begin{align}
 \mathbf{z_2} &= \mathbf{W_2}\mathbf{h} + \mathbf{b_2}   \tag{3} \\
 \mathbf{\hat y} &= \mathrm{softmax}(\mathbf{z_2})   \tag{4} \\
\end{align}

$$ \textrm{softmax}(\textbf{z})_i = \frac{e^{z_i} }{\sum\limits_{j=1}^{V} e^{z_j} }  \tag{5} $$

```python
z=np.array([9,8,11,10,8.5])
```

```python
z
```

    array([ 9. ,  8. , 11. , 10. ,  8.5])

```python
np.exp(z)
```

    array([ 8103.08392758,  2980.95798704, 59874.1417152 , 22026.46579481,
            4914.7688403 ])

```python
np.sum(np.exp(z))
```

    97899.41826492078

```python
np.exp(z)/np.sum(np.exp(z))
```

    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])

```python
def softmax(z):
    e_z= np.exp(z)
    sum_ez = np.sum(e_z)
    return e_z / sum_ez
```

```python
softmax(z)
```

    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])

```python

```


# Class 4: Plotting with Matplotlib

Matplotlib is a powerful plotting module that is part of Python's standard library. The website for matplotlib is at http://matplotlib.org/. And you can find a bunch of examples at the following two locations: http://matplotlib.org/examples/index.html and http://matplotlib.org/gallery.html.

Matplotlib contains a module called pyplot that was written to provide a Matlab-style plotting interface.

```python
# Import matplotlib.pyplot

```

Next, we want to make sure that the plots that we create are displayed in this notebook. To achieve this we have to issue a command to be interpretted by Jupyter -- called a *magic* command. A magic command is preceded by a `%` character. Magics are *not* Python and will create errors if used in a Python program outside of the Jupyter notebook (e.g., if you are running Python in IDLE or executing a Python script form the command line).

```python
# Magic command for the Jupyter Notebook to display plots in the notebook

```

## An Introductory Example

Create a plot of the sine function for x values between -6 and 6. Add axis labels and a title.

```python
# Import numpy as np

# Create an array of x values from -6 to 6

# Create a variable y equal to the sine of x

# Use the plot function to plot x on the horizontal axis and y on the vertical axis

# Add a title and axis labels

```

## Style Sheets

Plots made using Matplotlib are highly customizable. Users can issue commands to control the limits of the axes, location of axis ticks, labels associated with axis ticks, and the color, thickness, opacity, and style of plotted data points.

But as a starting-point, Matplotlib uses a *style sheet* to establish initial plot settings until the user specifies different alternatives. The release of Matplotlib 2.0 included a major change in the default syle and the new default doesn't appeal to me.

```python
# Set the Matplotlib style sheet to 'classic'

# Re-run the magic command for the Jupyter Notebook to display plots in the notebook

# Use the plot function to plot the sine function

# Add a title and axis labels

```

The matplotlib attribute `plt.style.available` stores as a `list` the default sstyle sheets that you can choose from. Here's a gallery of the different sytles compared with each other. Aside from `classic`, other styles that I think look nice include: `bmh`, `ggplot`, and `seaborn`. But try them all out and see what you like!

```python
#Print the available matplotlib style sheets. PROVIDED
print(plt.style.available)
```

## The `plot` Function

The `plot` function creates a two-dimensional plot of one variable against another.

```python
# Use the help function to see the documentation for plt.plot()

```

### Example

Create a plot of $f(x) = x^2$ with $x$ between -2 and 2.
* Set the linewidth to 3 points
* Set the line opacity (alpha) to 0.6
* Set axis labels and title
* Add a grid to the plot

```python
# Create an array of x values from -2 to 2

# Create a variable y equal to x squared

# Use the plot function to plot the line

# Add a title and axis labels

# Add grid

```

### Example

Create plots of the functions $f(x) = \log x$ (natural log) and $g(x) = 1/x$ between 0.01 and 5
* Make the line for $log(x)$ solid blue
* Make the line for $1/x$ dashed magenta
* Set the linewidth of each line to 3 points
* Set the line transparency (alpha) for each line to 0.6
* Set the limits for the $x$-axis to (0,5)
* Set the limits for the $y$-axis to (-2,5)
* Set axis labels and title
* Add a legend
* Add a grid to the plot

```python
# Create an array of x values from 0.01 to 5

# Create y variables

# Use the plot function to plot the lines

# Add a title and axis labels

# Set axis limits

# Make legend

# Add grid

```

### Example

Consider the linear regression model:
\begin{align}
y_i = \beta_0 + \beta_1 x_i + \epsilon_i
\end{align}
where $x_i$ is the independent variable, $\epsilon_i$ is a random regression error term, $y_i$ is the dependent variable and $\beta_0$ and $\beta_1$ are constants.

Let's simulate the model
* Set $\beta_0=1$ and $\beta_1=-0.5$
* Create an array of $x_i$ values from -5 to 5
* Create an array of $\epsilon_i$ values from the standard normal distribution equal in length to the array of $x_i$ values
* Create an array of $y_i$ values
* Plot y against x with either a circle ('o'), triangle ('^'), or square ('s') marker and transparency (alpha) to 0.5
* Add axis lables, a title, and a grid to the plot

```python
# Set betas

# Create x values

# Create epsilon values from the standard normal distribution

# Create y

# Plot

# Add a title and axis labels

# Set axis limits

# Add grid

```

### Example

Create plots of the functions $f(x) = x$, $g(x) = x^2$, and $h(x) = x^3$ for $x$ between -2 and 2
* Use the optional string format argument to format the lines:
  - $x$: solid blue line
  - $x^2$: dashed green line
  - $x^3$: dash-dot magenta line
* Set the linewidth of each line to 3 points
* Set transparency (alpha) for each line to 0.7
* Set axis labels and title
* Add a legend to lower right with 3 columns
* Add a grid to the plot

```python
# Create an array of x values from -2 to 2

# Create y variables

# Use the plot function to plot the lines

# Add a title and axis labels

# Make legend

# Add grid

```

## Figures, Axes, and Subplots

Often we want to create plots with multiple axes or we want to modify the size and shape of the plot areas. To be able to do these things, we need to explicity create a figure and then create axes within the figure. The best way to see how this works is by example.

### Example: A Single Plot with Double Width

The default dimensions of a matplotlib figure are 6 inches by 4 inches. As we saw above, this leaves some whitespace on the right side of the figure in the Notebook. Suppose we want to remove that by making the plot area twice as wide.

Plot the sine function on -6 to 6 using a figure with dimensions 12 inches by 4 inches

```python
# Create an array of x values from -6 to 6

# Create y variables

# Create a new figure that is twice as wide as the default

# Create axis within the figure

# Plot

# Add grid

```

In the previous example the `fig = plt.figure()` function creates a new figure called `fig` and `fig.add_subplot()` puts a new axis on the figure. The command `ax1 = fig.add_subplot(1,1,1)` means divide the figure `fig` into a 1 by 1 grid and assign the first component of that grid to the variable `ax1`.

### Example: Two Plots Stacked

Create a new figure with two axes in a column and plot the sine function on -6 to 6 on the top axis and the cosine function on -6 to 6 on the bottom axis.

```python
# Create an array of x values from -6 to 6

# Create y variables

# Create a new figure that is twice as tall as the default

# Create axis 1 and plot with title and axis labels

# Create axis 2 and plot with title and axis labels

# Improve the spacing between the plots

```

### Example: Block of Four Plots

The default dimensions of a matplotlib figure are 6 inches by 4 inches. As we saw above, this leaves some whitespace on the right side of the figure. Suppose we want to remove that by making the plot area twice as wide.

Create a new figure with four axes in a two-by-two grid. Plot the following functions on the interval -2 to 2:
* $y = x$
* $y = x^2$
* $y = x^3$
* $y = x^4$

Set the figure size to 12in. by 8in. Run the command `plt.tight_layout()` to adust the figure's margins after creating your figure, axes, and plots.

```python
# Create data for plot

# Create a new figure that is twice as wide and twice as tall as the default

# Create axis 1 and plot with title

# Create axis 2 and plot with title

# Create axis 3 and plot with title

# Create axis 4 and plot with title

# Improve the spacing between the plots

```

## Exporting Figures to Image Files

Use the `plt.savefig()` function to save figures to images.

```python
# Create data to plot sin(x)/x on [-20,20]

# Plot y against x

# Save figure as 'fig_econ126_class04_sine_over_x.png' at 120 DPI resolution

```

In the previous example, the image is saved as a PNG file with 120 dots per inch. This resolution is high enough to look good even when projected on a large screen. The image format is inferred by the extension on the filename.


# Introduction to [JuMP](https://github.com/JuliaOpt/JuMP.jl/blob/master/README.md)

February 27, 2020

*Uses JuMP version 0.21*

### What is JuMP?
JuMP stands for **Ju**lia for **M**athematical Programming. It is a "domain-specific modeling language for mathematical optimization embedded in Julia."

What does "domain-specific" mean? It means that some algorithms are useful in some knowledge domains (e.g. supply chain management) that may not be useful in other knowledge domains (e.g. estimating statistical model parameters).

What is great about JuMP is that it allows the user to change algorithms with little or no change to the modeling code. This means that someone without domain-specific knowledge can easily utilize JuMP to accomplish a task that would otherwise be impossible.

### What goes into JuMP?
To use JuMP, a user must specify the following components:

- An objective function (you gotta solve something!)
- Variables
- Constraints

JuMP does the rest! (I'll show you examples shortly of how you can do this.)

### What kinds of problems can JuMP handle?
JuMP provides the ability to solve the following types of objective functions:

- Linear
- Convex Quadratic
- Nonlinear (convex and nonconvex)

It also provides the ability to use constraints of the following types:

- Linear
- Convex Quadratic
- Second-order Conic
- Semidefinite
- Nonlinear (convex and nonconvex)

Finally, it provides support for any of the following types of variables:

- Continuous
- Integer-valued
- Semicontinuous
- Semi-integer

### What examples will we go through today?
Today I'll go through four examples:

1. Solve a very simple constrained system of equations (adapted from JuMP's Github repository [here](https://github.com/JuliaOpt/JuMP.jl/blob/master/examples/basic.jl))
2. Solve a Sudoku puzzle (adapted from JuMP's Github repository [here](https://github.com/JuliaOpt/JuMP.jl/blob/master/examples/sudoku.jl))
3. Solve a professional sports team's salary budget problem
4. Estimate the parameters of a linear regression model

---

## 1. A simple constrained system of equations
Suppose you want to solve the following (constrained) system of equations:

\begin{equation*}
\max\,\, 5x + 3y
\end{equation*}

subject to the following constraints:

\begin{align}
x + 5y & \leq 3 \\
x & \geq 0 \\
y & \geq 0 \\
x & \leq 2 \\
y & \leq 30 \\
\end{align}

In economics, this problem may represent a utility maximization problem, where utility is linear in $x$ and $y$, the individul has an income of 3, and the price of good $x$ is 1, with the price of good $y$ equal to 5. (And there are supply constraints on $x$ and $y$ governed by the market in general.)

If we wanted to, we could solve this using a Lagrangian (although in this case with linear utility, we know we will have a corner solution, so no need to bother with calculus).

But we want the computer to do this, so let's see how it's done.

### Specifying the model components
As explained above, JuMP needs an objective function, variables, and constraints. Additionally, we need to tell JuMP which optimization algorithm to use.

#### Optimizer
The first step is to declare a model and attach an optimizer. We are going to use the GLPK optimizer:
```julia
model = Model(GLPK.Optimizer)
```

#### Variables
Next, we need to tell it what the variables are, and any constraints on those variables:
```julia
@variable(model, 0 <= x <= 2)
@variable(model, 0 <= y <= 30)
```

#### Objective function
Next, we tell it the objective function (and whether we want to maximize or minimize):
```julia
@objective(model, Max, 5x + 3y)
```

#### Constraints
Finally, we give it the constraints. Note that the single constraints on each variable were incorporated when we declared the variables themselves, but we have one additional constraint that is a function of both variables. We could also impose a constraint that the optimal values of $x$ and $y$ be integers (if these represented indivisible objects, for example).
```julia
@constraint(model, 1x + 5y <= 3)
```

### Optimizing the model
Once we have all of the components declared, we can optimize the model:
```julia
JuMP.optimize!(model)
```

We can then look at the output of the optimization as follows:
```julia
obj_value = JuMP.objective_value(model)
x_value   = JuMP.value(x)
y_value   = JuMP.value(y)
```
We'll put all of the code together below so it can be run on your machine.

Following Julia protocol, we will also wrap everything in a function and then call that function.

### All of the code together

```julia
using JuMP, GLPK

# wrap all of our code inside a function (for better performance)
function example_basic()

    # define model and optimizer
    model = Model(GLPK.Optimizer)

    # define variables
    @variable(model, 0 <= x <= 2)
    @variable(model, 0 <= y <= 30)

    # define objective function
    @objective(model, Max, 5x + 3y)

    # add additional constraints
    @constraint(model, 1x + 5y <= 3.0)

    # display the model
    print(model)

    # optimize the model
    JuMP.optimize!(model)

    # return and print objective function and optimal values of variables
    obj_value = JuMP.objective_value(model)
    x_value = JuMP.value(x)
    y_value = JuMP.value(y)
    println("Objective value: ", obj_value)
    println("x = ", x_value)
    println("y = ", y_value)
end

# call the function defined above
example_basic()
```

    Max 5 x + 3 y
    Subject to
     x + 5 y <= 3.0
     x >= 0.0
     y >= 0.0
     x <= 2.0
     y <= 30.0
    Objective value: 10.6
    x = 2.0
    y = 0.2

So our optimal values are:

\begin{align}
x^* & = 2 \\
y^* & = 0.2 \\
\text{Objective} & = 10.6
\end{align}

As I mentioned before we did any programming, this problem would yield a corner solution (where, for one of the goods, the optimal value hits one of the constraints). In this case, it was $x^* = 2$.

#### Other types of constraints
As I mentioned above, we can also add other constraints, for example, that the optimal values be integers (e.g. if $x$ and $y$ are indivisible). In this case, we would add `@constraint(model, x in MOI.Integer())` and similar for $y$.

We could also put a constraint on the objective function itself (e.g. that the objective value be no larger than 10). To do this we would add the following:
```julia
@expression(model, objval, 5x + 3y)
@constraint(model, objval <= 10)
```
---

## 2. Solving a Sudoku puzzle
Now that we are more comfortable with JuMP, we can solve a Sudoku puzzle. All we need to do is appropriately tell JuMP how to understand the puzzle board.

### Sudoku objective
In this case, the objective function is to fill in the puzzle board given a starting grid that has some numbers filled in. We won't have a formal objective function for this; we will just give JuMP a starting grid and tell it to satisfy all of the constraints, where one of the constraints is that the board gets filled. (A blank board has an infinite number of solutions, but a partially completed board should have just one unique solution.)

### Sudoku variables
We will communicate the state of the puzzle board with an array of variables. The variables will be put in a 3-dimensional array, where the first two dimensions tell the "latitude" and "longitude" of the cell on the puzzle board, and the third dimension keeps track of which of the numbers 1-9 will fill that cell.

Mathematically, we have
```julia
x[i, j, k]
```
which, if equal to 1, indicates that cell $(i,j)$ should contain the number $k$. The indices $(i,j,k)$ each must take on integer values from 1 to 9, since the puzzle board has 81 squares. So our `x` array is a 9 x 9 x 9 cube.

### Sudoku constraints
The constraints of Sudoku are as follows:

1. Each cell can only contain one number (duh, but we have to explain this to the computer!)
2. Each row contains each number exactly once
3. Each column contains each number exactly once
4. Each 3x3 subgrid contains each number exactly once

### Code

```julia
using JuMP, GLPK
function example_sudoku()

    # input the initial puzzle board (0s mean blanks)
    initial_grid = [
                    3 1 0 0 5 8 0 0 4;
                    0 0 9 3 2 0 0 0 0;
                    0 2 5 1 0 4 0 9 0;
                    0 0 0 0 0 0 3 8 9;
                    0 0 8 0 0 0 5 0 0;
                    5 4 6 0 0 0 0 0 0;
                    0 8 0 2 0 3 6 5 0;
                    0 0 0 0 7 1 4 0 0;
                    7 0 0 4 8 0 0 2 1
                    ]

    # use GLPK Optimizer
    model = Model(GLPK.Optimizer)

    # Set up the variables: each one can only take on binary values, so we add "Bin" to the end as a constraint
    @variable(model, x[1:9, 1:9, 1:9], Bin)

    # Add the constraints
    @constraints(model, begin
                     # Constraint 1 - Only one value appears in each cell
                     cell[i in 1:9, j in 1:9], sum(x[i, j, :]) == 1
                     # Constraint 2 - Each value appears in each row once only
                     row[i in 1:9, k in 1:9], sum(x[i, :, k]) == 1
                     # Constraint 3 - Each value appears in each column once only
                     col[j in 1:9, k in 1:9], sum(x[:, j, k]) == 1
                     # Constraint 4 - Each value appears in each 3x3 subgrid once only
                     subgrid[i=1:3:7, j=1:3:7, val=1:9], sum(x[i:i + 2, j:j + 2, val]) == 1
                 end)

    # Add additional constraints that reflect the starting point of the puzzle board
    # (i.e. don't attempt to update the numbers that were given as part of the puzzle)
    for row in 1:9, col in 1:9
        if initial_grid[row, col] != 0
            @constraint(model, x[row, col, initial_grid[row, col]] == 1)
        end
    end

    # Solve it
    JuMP.optimize!(model)

    term_status = JuMP.termination_status(model)
    primal_status = JuMP.primal_status(model)
    is_optimal = term_status == MOI.OPTIMAL

    # Check solution
    if is_optimal
        mip_solution = JuMP.value.(x)
        sol = zeros(Int, 9, 9)
        for row in 1:9, col in 1:9, val in 1:9
            if mip_solution[row, col, val] >= 0.9
                sol[row, col] = val
            end
        end
        return sol
    else
        error("The solver did not find an optimal solution.")
    end
end

function print_sudoku_solution(solution)
    println("Solution:")
    println("[-----------------------]")
    for row in 1:9
        print("[ ")
        for col in 1:9
            print(solution[row, col], " ")
            if col % 3 == 0 && col < 9
                print("| ")
            end
        end
        println("]")
        if row % 3 == 0
            println("[-----------------------]")
        end
    end
end

sol = example_sudoku()
print_sudoku_solution(sol)
```

    Solution:
    [-----------------------]
    [ 3 1 7 | 9 5 8 | 2 6 4 ]
    [ 4 6 9 | 3 2 7 | 8 1 5 ]
    [ 8 2 5 | 1 6 4 | 7 9 3 ]
    [-----------------------]
    [ 2 7 1 | 6 4 5 | 3 8 9 ]
    [ 9 3 8 | 7 1 2 | 5 4 6 ]
    [ 5 4 6 | 8 3 9 | 1 7 2 ]
    [-----------------------]
    [ 1 8 4 | 2 9 3 | 6 5 7 ]
    [ 6 9 2 | 5 7 1 | 4 3 8 ]
    [ 7 5 3 | 4 8 6 | 9 2 1 ]
    [-----------------------]

---
# 3. Solving a salary cap problem
NBA general managers want to build a championship team. What is their objective function? To win a championship. Well, that's difficult to exactly write down, but we can look at other more easily measurable outputs (like points scored, points allowed, etc.)

Suppose we want to find the team that will have the best statistics, but at the cheapest price.

### Data
I obtained the data using [this R script](https://github.com/tyleransom/DScourseS20/blob/master/WebData/getNBAplayerStats.R), which makes use of the `nbastatR` package. The data is in CSV format, which we will directly read into Julia using the `HTTP` and `JuliaDB` packages.

### Objective function
It's not clear what objective function we should use, but let's start with a simple one: maximize points scored.

### Constraints
We have at least two constraints: our team can only have 15 players on it, and the total team salary must be below the luxury tax threshold (\\$132.6m). We will talk about other important constraints after trying things out with this most basic constraint.

### Variables
In this case, the variables are the players that we pick.

### Code

```julia
using HTTP, JuliaDB, JuMP, GLPK

# first function: read in the player data from the class GitHub repository
function read_in_data(url)
    newtable  = csvread(IOBuffer(HTTP.get(url).body), skiplines_begin=0, header_exists=true)
    players   = newtable[1][2]
    salaries  = newtable[1][4]./1000000
    ppg       = newtable[1][11]
    return players,salaries,ppg
end

function SolveModel(players,salary,points)
    N = length(salary)

    m = Model(GLPK.Optimizer)

    # define the variables: they are 0 if the player did not make the team, 1 if the player did make the team
    @variable(m, picked[1:N], Bin)

    # categories:
    @objective(m, Max, sum( points[i] * picked[i] for i in 1:N))

    @constraints m begin
        # Constraint 1 - payroll <= 132.6m
        sum(salary[i] * picked[i] for i in 1:N) <= 132.6
        # Constraint 2 - must have exactly 15 players on roster
        sum(picked[i] for i in 1:N) == 15
    end

    # Solve it
    JuMP.optimize!(m);
    pck     = convert(BitArray,JuMP.value.(picked))
    lineup  = players[pck]
    points  = JuMP.objective_value(m)
    payroll = sum(salary[pck])
    return lineup,points,payroll
end

# call first function (to import data)
players,salaries,pts = read_in_data("https://raw.githubusercontent.com/tyleransom/DScourseS20/master/WebData/playerSalaryStats.csv")

# pass data into second function to get optimal lineup
lineup,total_points,payroll = SolveModel(players,salaries,pts)
println("team: ",lineup)
println("total points scored per game: ",total_points)
println("payroll: ",payroll)
```

    team: ["Trae Young", "Jayson Tatum", "Jaylen Brown", "Zach LaVine", "Collin Sexton", "Domantas Sabonis", "Kendrick Nunn", "Giannis Antetokounmpo", "Brandon Ingram", "Shai Gilgeous-Alexander", "Buddy Hield", "Pascal Siakam", "Donovan Mitchell", "Luka Doncic", "Bradley Beal"]
    total points scored per game: 354.3
    payroll: 132.53548

The output of our optimization tells us that with the team listed above, we should expect them to score more than 354 points per game! Unfortunately, that answer makes no sense. What went wrong?

Two major things that went wrong:

1. We didn't account for the fact that there are only 240 minutes in an NBA game (48 minutes times 5 players on the floor)
2. We didn't account for the fact that teams can typically attempt no more than 80 field goals in an NBA game

Let's adjust our code so that we account for these important constraints and see if we get anything more reasonable. We will need to add minutes and field goal attampts into our data import function, and we will need to add a constraint on total minutes and total field goal attempts.

```julia
# first function: read in the player data from the class GitHub repository
function read_in_data(url)
    newtable = csvread(IOBuffer(HTTP.get(url).body), skiplines_begin=0, header_exists=true)
    players  = newtable[1][2]
    salaries = newtable[1][4]./1000000
    mpg      = newtable[1][5]
    fgaG5    = 1.0.*((newtable[1][9]).>5)
    fga      = newtable[1][9]
    ppg      = newtable[1][11]
    return players,salaries,mpg,fgaG5,fga,ppg
end

function SolveModel(players,salary,minutes,field_goals_over5,field_goals,points)
    N = length(salary)

    m = Model(GLPK.Optimizer)

    # define the variables: they are 0 if the player did not make the team, 1 if the player did make the team
    @variable(m, picked[1:N], Bin)

    # categories:
    @objective(m, Max, sum( points[i] * picked[i] for i in 1:N))

    @constraints m begin
        # Constraint 1 - payroll <= 132.6m
        sum(salary[i] * picked[i] for i in 1:N) <= 132.6
        # Constraint 2 - must have exactly 15 players on roster
        sum(picked[i] for i in 1:N) == 15
        # Constraint 3 - total minutes must not exceed 240
        sum(minutes[i] * picked[i] for i in 1:N) <= 240
        # Constraint 4 - total shot attempts must be lower than 80
        sum(field_goals[i] * picked[i] for i in 1:N) <= 80
    end

    # Solve it
    JuMP.optimize!(m);
    pck      = convert(BitArray,JuMP.value.(picked))
    lineup   = players[pck]
    totmin   = sum(minutes[pck])
    payroll  = sum(salary[pck])
    totshots = sum(field_goals[pck])
    points   = JuMP.objective_value(m)
    return lineup,points,totmin,payroll,totshots
end

# call first function (to import data)
players,salaries,minutes,over5fg,fga,pts = read_in_data("https://raw.githubusercontent.com/tyleransom/DScourseS20/master/WebData/playerSalaryStats.csv")

# pass data into second function to get optimal lineup
lineup,total_points,total_minutes,payroll,totshots = SolveModel(players,salaries,minutes,over5fg,fga,pts)
println("team: ",lineup)
println("total points scored per game: ",total_points)
println("payroll: ",payroll)
println("total shots per game: ",totshots)
println("total minutes per game: ",total_minutes)
```

    team: ["Damian Jones", "Javonte Green", "JaVale McGee", "Dwight Howard", "Goga Bitadze", "Jaxson Hayes", "Nerlens Noel", "Patrick Patterson", "Yogi Ferrell", "Chris Boucher", "Tony Bradley", "Mason Plumlee", "Thon Maker", "Christian Wood", "James Harden"]
    total points scored per game: 121.0
    payroll: 84.22254099999999
    total shots per game: 79.89999999999999
    total minutes per game: 239.90000000000003

The results gives us a much mroe reasonable number of 121 points, 80 field goal attempts, and a super-cheap payroll of \\$84m, which is \\$25m lower than the lowest in the NBA right now.

---

# 4. Estimating Linear Regression Coefficients
We can also use JuMP to estimate linear regression coefficients. In this case, we must use the `Ipopt` (pronounced eye-PEE-opt) optimizer. Why? Because our objective function is nonlinear (we are minimizing the sum of the squared residuals) and the optimizers we have used above are only valid for linear objective functions.

### Objective function
The objective function for OLS is

\begin{equation}
\min_{\beta} \sum_{i} (y_i - \beta_0 - \beta_1 x_1 - \beta_2 x_2 - \cdots - \beta_k x_k)^2
\end{equation}

### Variables
The variables in this case are the parameters we want to estimate---the $\beta$'s.

### Code

```julia
using HTTP, JuliaDB, JuMP, Ipopt
function import_auto(url)
    newtable  = csvread(IOBuffer(HTTP.get(url).body), skiplines_begin=0, header_exists=true)
    depvar    = log.(newtable[1][2]) # log price
    indepvars = cat(ones(size(depvar)),newtable[1][3],newtable[1][5],newtable[1][6]; dims=2) # constant, mpg, headroom, trunk
    return depvar,indepvars
end

Y,X = import_auto("https://tyleransom.github.io/teaching/MetricsLabs/auto.csv")

function jumpOLS(Y,X,startval=zeros(size(X,2),1))
    OLS = Model(Ipopt.Optimizer)

    # Declare the variables you are optimizing over
    @variable(OLS, b[i=1:size(X,2)], start = startval[i])

    # Write your objective function
    @NLobjective(OLS, Min, sum( (Y[i]-sum( X[i,k]*b[k] for k in 1:size(X,2) ))^2 for i in 1:size(X,1) ) )

    # Solve the objective function
    JuMP.optimize!(OLS)

    SSR = JuMP.objective_value(OLS)
    b_value = JuMP.value.(b)
    println("Objective value: ", SSR)
    println("beta hat = ", b_value)
    println("RMSE = ", sqrt(SSR/(size(X,1)-size(X,2))))
end

jumpOLS(Y,X)

```

    ******************************************************************************
    This program contains Ipopt, a library for large-scale nonlinear optimization.
     Ipopt is released as open source code under the Eclipse Public License (EPL).
             For more information visit http://projects.coin-or.org/Ipopt
    ******************************************************************************

    This is Ipopt version 3.12.10, running with linear solver mumps.
    NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

    Number of nonzeros in equality constraint Jacobian...:        0
    Number of nonzeros in inequality constraint Jacobian.:        0
    Number of nonzeros in Lagrangian Hessian.............:       10

    Total number of variables............................:        4
                         variables with only lower bounds:        0
                    variables with lower and upper bounds:        0
                         variables with only upper bounds:        0
    Total number of equality constraints.................:        0
    Total number of inequality constraints...............:        0
            inequality constraints with only lower bounds:        0
       inequality constraints with lower and upper bounds:        0
            inequality constraints with only upper bounds:        0

    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
       0  5.5361028e+03 0.00e+00 1.00e+02  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
       1  8.0341708e+00 0.00e+00 2.37e-14  -1.0 9.28e+00    -  1.00e+00 1.00e+00f  1

    Number of Iterations....: 1

                                       (scaled)                 (unscaled)
    Objective...............:   2.9676334739952230e-02    8.0341707763946211e+00
    Dual infeasibility......:   2.3739303738078894e-14    6.4268590449501054e-12
    Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
    Complementarity.........:   0.0000000000000000e+00    0.0000000000000000e+00
    Overall NLP error.......:   2.3739303738078894e-14    6.4268590449501054e-12

    Number of objective function evaluations             = 2
    Number of objective gradient evaluations             = 2
    Number of equality constraint evaluations            = 0
    Number of inequality constraint evaluations          = 0
    Number of equality constraint Jacobian evaluations   = 0
    Number of inequality constraint Jacobian evaluations = 0
    Number of Lagrangian Hessian evaluations             = 1
    Total CPU secs in IPOPT (w/o function evaluations)   =      7.216
    Total CPU secs in NLP function evaluations           =      3.174

    EXIT: Optimal Solution Found.
    Objective value: 8.034170776394621
    beta hat = [9.27849, -0.0296502, -0.115783, 0.0247281]
    RMSE = 0.33878292199313836

We can check our answer in R with the following code:
```r
df <- read.csv("https://tyleransom.github.io/teaching/MetricsLabs/auto.csv") %>% as_tibble %>%
      mutate(logprice = log(price)) %>%
      drop_na(foreign)
summary(lm(logprice ~ mpg + headroom + trunk, data=df))
```

which gives us

```
Call:
lm(formula = log(price) ~ mpg + headroom + trunk, data = df)

Residuals:
    Min      1Q  Median      3Q     Max
-0.6017 -0.2521 -0.1082  0.2104  1.0445

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept)  9.278489   0.314134  29.537  < 2e-16 ***
mpg         -0.029650   0.008434  -3.515 0.000775 ***
headroom    -0.115783   0.062605  -1.849 0.068619 .
trunk        0.024728   0.013857   1.785 0.078667 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.3388 on 70 degrees of freedom
Multiple R-squared:  0.2842,	Adjusted R-squared:  0.2535
F-statistic: 9.263 on 3 and 70 DF,  p-value: 3.082e-05
```