# VI Seminário de Avaliação de Pesquisa Científica e Tecnológica (SAPCT) e V Workshop de Integração e Capacitação em Processamento de Alto Desempenho (ICPAD)
## Geração Automática de Código Através de Computação Simbólica em Python: Introdução à LDE Devito
**Prof. Dr. Oscar Mojica (https://oscar-mojica.netlify.app/)
Centro de Supercomputação para Inovação Industrial
Senai Cimatec**

# Computação simbólica

Denomina-se computacão algébrica (CA) a manipulação de símbolos matemáticos - feita no computador - de acordo com as regras abstratas da matemática simbólica. Entre os mais conhecidos softwares que
permitem este tipo de manipulação estão o [Maple](https://www.maplesoft.com/) e o [Mathematica](https://www.wolfram.com/mathematica/). Nestes sistemas de CA encontramse implementadas, por exemplo, regras de álgebra, trigonometria e cálculo, de modo que pesquisadores e estudantes podem derivar, integrar, obter a solução de equações diferenciais, simplificar expressões, representar graficamente campos escalares, vetoriais e curvas paramêtricas. Além dos comandos usuais, esses sistemas de CA permitem um fácil e rápido desenvolvimento de programas matemáticos, através de um repertório vasto de funções e operações.

Na computação simbólica é possível manipular símbolos (não obrigatoriamente expressões decimais), representando números inteiros, racionais, irracionais, imaginários, ou, ainda, polinômios, funções, matrizes, sequências, sistemas de equações e até mesmo estruturas mais abstratas, como equações diferenciais. Enquanto em Computação Numérica se opera apenas com aproximações de decimais finitas para números reais, em Computação Algébrica é possível operar com símbolos que representam objetos matemáticos mais gerais (além de números).

Por exemplo, ao se calcular $\sqrt{2}\sqrt{3}$ usando uma calculadora ou um programa de Computação Numérica, o resultado será 1,4142135 × 1,7320508 = 2,4494897.

Por outro lado, um programa de computação algébrica mostrará resultado como sendo $\sqrt{2}\sqrt{3}=\sqrt{6}$, algo bastante próximo do que se faz em sala de aula, ou o que pode-se encontrar num artigo científico. Assim, um programa desse tipo efetua cálculos com radicais sem a necessidade de representá-los em forma decimal aproximada, pois “conhece” as regras e propriedades algébricas dos objetos envolvidos.

## Computação Simbólica em Python com SimPy

Em primeiro lugar, o que é SimPy ? Na documentação, eles o definem como: “um framework baseado em processos com simulação de eventos discretos utilizando as bibliotecas padrão do Python”. Você pode ver a documentação completa [aqui](https://simpy-portuguese-br.readthedocs.io/pt_BR/latest/), que eu recomendo fortemente. Lá você pode encontrar não apenas as coisas que você pode fazer, mas também muitos tutoriais simples, mas muito úteis. Simpy é de fácil uso e disponibilizado como software livre, sendo portanto amplamente usado na academias e indústria.

## Execução de código Python
Existem algumas maneiras de executar o Python:

1. Escreva seu código em um arquivo de texto com a extensão **`py`**. Em seguida, no terminal, digite **`python myPythonCode.py`**
2. Use um dos editores que acompanham o python. O Anaconda é fornecido com o `Spyder`. Outro editor famoso é o  `PyCharm` . Usando esses editores (na verdade, Integrated Development Environments, IDEs), você pode escrever código e executá-lo no local.
3. Use o Jupyter Notebook. Esta é minha abordagem favorita, pois permite que você escreva código Python de dentro de seu navegador e o misture com texto e equações. Basta abrir um terminal (prompt de comando no Windows) e digitar **`jupyter notebook`**.

### Operações básicas e considerações iniciais

Como mencionado anteriormente, a computação simbólica envolve a computação de objetos matemáticos simbolicamente. Isso significa que os objetos matemáticos são representados exatamente, não aproximadamente, e expressões matemáticas com variáveis não avaliadas são deixadas em forma simbólica, ou algébrica.
Vamos começar com o exemplo introdutório:

```python
import sympy as sym # importa as funções do sympy em um contexto diferenciado (no caso sym)
from sympy import *
```

```python
import math
print('A raiz de 9 é:', math.sqrt(9))
print('A raiz de 8 é:', math.sqrt(8))
```

    A raiz de 9 é: 3.0
    A raiz de 8 é: 2.8284271247461903

```python
print('A raiz de 9 é:', sym.sqrt(9))
print('A raiz de 8 é:', sym.sqrt(8))
```

    A raiz de 9 é: 3
    A raiz de 8 é: 2*sqrt(2)

É importante dizer para o sympy quais objetos são SÍMBOLOS;

```python
x, y, z, t = symbols("x y z t") # Define símbolos para variáveis genericas
```

```python
k, m, n = symbols("k m n", integer=True)  # Define símbolos para variáveis inteiras
```

os identificadores (variáveis) “x, y, z, t” são variáveis simbólicas e serão tratadas como objetos sympy, e não mais como identificadores usuais do Python. Podemos, então definir funções com essas variáveis e usá-las em expressões algébricas.

```python
3*pi/2 + sym.exp(I*x) / (x**2 + y)
```

$\displaystyle \frac{3 \pi}{2} + \frac{e^{i x}}{x^{2} + y}$

Algumas ações que podem ser feitas com expressões simbólicas: expandir (expand), factorizar(factor), simplificar(simplify) e resolver(solve)

```python
eq = ((x+y)**2 * (x+1))
eq
```

$\displaystyle \left(x + 1\right) \left(x + y\right)^{2}$

```python
expanded_eq = expand(eq)
display(expanded_eq)
```

$\displaystyle x^{3} + 2 x^{2} y + x^{2} + x y^{2} + 2 x y + y^{2}$

```python
factor(expanded_eq)
```

$\displaystyle \left(x + 1\right) \left(x + y\right)^{2}$

```python
a = 1/x + (x*sin(x) - 1)/x
print(a) #assim fica feio
display(a) #assim fica bonito
```

    (x*sin(x) - 1)/x + 1/x

$\displaystyle \frac{x \sin{\left(x \right)} - 1}{x} + \frac{1}{x}$

```python
simplify(a)
```

$\displaystyle \sin{\left(x \right)}$

```python
eq2 = Eq(x**3 - 3*x**2 + 4*x + 8., 0)
display(eq2)
```

$\displaystyle x^{3} - 3 x^{2} + 4 x + 8.0 = 0$

```python
solve(eq2,x)
```

    [-1.00000000000000, 2.0 - 2.0*I, 2.0 + 2.0*I]

Podemos também calcular derivadas. O exemplo abaixo, calcula a derivada de $e^{−2x^{2}}$

```python
sym.diff(sym.exp(-2*x**2))
```

$\displaystyle - 4 x e^{- 2 x^{2}}$

## Funções e gráficos
A forma mais simples de definir uma função no SymPy é simplesmente definir uma nova variável atribuindo uma expressão com variáveis simbólicas

```python
expr = x**3 -5*x**2 + 3*x + 2
expr
```

$\displaystyle x^{3} - 5 x^{2} + 3 x + 2$

```python
expr.subs(x, 2)
```

$\displaystyle -4$

Tendo o SymPy em funcionamento, é possivel fazer gráficos nele usando um comando tipo plot, porém o sympy.plot é menos rico que o plt.plot (do Matplotlib)

```python
sym.plot(expr,(x,-1.5,5)); # é fácil plotar o gráfico de uma função
```

## Mais operações

Vejamos alguns exemplos de como usar sympy em Cálculo...

```python
z = symbols("z")
expr = x**3 + 4*x*y - z
expr.subs([(x, 2), (y, 4), (z, 0)])
```

$\displaystyle 40$

```python
expr = sqrt(8)
```

```python
expr
```

$\displaystyle 2 \sqrt{2}$

```python
expr.evalf()
```

$\displaystyle 2.82842712474619$

```python
pi.evalf(100)
```

$\displaystyle 3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117068$

```python
expr = cos(2*x)
expr.evalf(subs={x: 2.4})
```

$\displaystyle 0.0874989834394464$

Já vimos que com o comando “diff” podemos realizar uma derivação

```python
diff(exp(2*x),x,3) # calcula a terceira derivada de exp(2*x)
```

$\displaystyle 8 e^{2 x}$

Podemos aplicar derivadas em objetos mais complexos, como matrizes, por exemplo. Primeiro, definimos uma matriz de variáveis e funções simbólicas com o comando Matrix()

```python
G = Matrix([[x**2,y+x],[2*x**3*y, 5*y**3]])
G
```

$\displaystyle \left[\begin{matrix}x^{2} & x + y\\2 x^{3} y & 5 y^{3}\end{matrix}\right]$

```python
diff(G,x) # a matriz G foi definida acima
```

$\displaystyle \left[\begin{matrix}2 x & 1\\6 x^{2} y & 0\end{matrix}\right]$

Para calcular derivadas parciais, como $\dfrac{\partial x}{\partial x \partial y}e^{(x^{2}+y^{3})}$

```python
diff(exp(x**2+y**3),x,y) # derivadas parciais com relação a x e y
```

$\displaystyle 6 x y^{2} e^{x^{2} + y^{3}}$

```python
f, g = symbols('f g', cls=Function)
f(x)
```

$\displaystyle f{\left(x \right)}$

```python
f(x).diff()
```

$\displaystyle \frac{d}{d x} f{\left(x \right)}$

```python
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x))
diffeq
```

$\displaystyle f{\left(x \right)} - 2 \frac{d}{d x} f{\left(x \right)} + \frac{d^{2}}{d x^{2}} f{\left(x \right)} = \sin{\left(x \right)}$

```python
dsolve(diffeq, f(x))
```

$\displaystyle f{\left(x \right)} = \left(C_{1} + C_{2} x\right) e^{x} + \frac{\cos{\left(x \right)}}{2}$

SimPy pode entender LaTeX e logo manipula-lo, por exemplo para computar uma derivada. Isto pode poupar muito tempo!

Formula:

\begin{equation}
y = log(\frac{1-\alpha}{\alpha\times\beta}x)
\end{equation}

Codigo:

`y = log(\frac{1-\alpha}{\alpha\times\beta}x)`

```python
from sympy.parsing.latex import parse_latex
x = symbols('x')
g = sym.log(x)
y = parse_latex(r'g(\frac{1-\alpha}{\alpha\times\beta}x)')

display(y)

sym.diff(y,x)
```

    ANTLR runtime and generated code versions disagree: 4.9.2!=4.7.2
    ANTLR runtime and generated code versions disagree: 4.9.2!=4.7.2

$\displaystyle g{\left(x \frac{1 - \alpha}{\alpha \beta} \right)}$

$\displaystyle \frac{1}{\alpha \beta} \left(1 - \alpha\right) \left. \frac{d}{d \xi_{1}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x \frac{1 - \alpha}{\alpha \beta} }}$

### Geração de código LaTeX com SymPy
Também pode ser gerado código LaTeX a partir de expressões simbólicas

```python
x = list(sym.symbols('x:2'))  # Vector (list of symbols)
x
```

    [x0, x1]

```python
H = x[0]**2/2 + sym.cosh(10*x[1]/sym.pi)-1
H
```

$\displaystyle \frac{x_{0}^{2}}{2} + \cosh{\left(\frac{10 x_{1}}{\pi} \right)} - 1$

```python
from sympy.printing import latex
```

```python
print(latex(H))    # Print latex code
```

    \frac{x_{0}^{2}}{2} + \cosh{\left(\frac{10 x_{1}}{\pi} \right)} - 1

$$\frac{x_{0}^{2}}{2} + \cosh{\left(\frac{10 x_{1}}{\pi} \right)} - 1$$

### Geração de código em linguagens de baixo nível (C, Fortran) with SimPy

```python
from sympy.printing import ccode, fcode
```

```python
H
```

$\displaystyle \frac{x_{0}^{2}}{2} + \cosh{\left(\frac{10 x_{1}}{\pi} \right)} - 1$

```python
ccode(H)    # Print C code
```

    '(1.0/2.0)*pow(x0, 2) + cosh(10*x1/M_PI) - 1'

```python
x,tau = symbols("x tau")
expr = Piecewise((x + 1, x > 0), (x, True)) # from https://docs.sympy.org/latest/modules/codegen.html
print(fcode(expr, tau))
```

          if (x > 0) then
             tau = x + 1
          else
             tau = x
          end if

Se você estiver interessado em aprender mais sobre geração de código usando Sympy, sugiro a leitura do tutorial [Automatic Code Generation with SymPy](https://www.sympy.org/scipy-2017-codegen-tutorial/)

# Linguagem de domínio específico (Domain-Specific Language - DSL)

Linguagens de Domínio Específico são linguagens projetadas com uma aplicação específica em mente, como uma maneira de expressar soluções para problemas de maneira mais simples e direta que a combinação de uma linguagem de programação de propósito geral e uma biblioteca.

1. Abordam um domínio de aplicação específico
2. Não obrigam o conhecimento avançado de programação
3. Interface próxima ao domínio.

    Considerando essa simples definição, um exemplo concreto de uma DSL é Devito, uma nova DSL para diferenças finitas que utiliza o pacote SymPy para gerar código C paralelo apartir da definição de um operador simbólico de alto nível. Devito tem como objetivo principal a geração de kernels de propagação de ondas para problemas de inversão sísmica e, portanto, oferece uma hierarquia de abstração que permite não apenas a geração de kernels de estêncil de uma forma rápida, mas também uma variedade de recursos específicos do domínio sísmico, necessários em um contexto científico real

## Diferenças Finitas

Uma técnica para a obtenção da solução numérica de uma equação diferencial parcial,
em que substitui as derivadas contínuas (e condições de fronteira e iniciais) pelas
fórmulas das diferenças que envolvem somente valores discretos associados com
posições da malha. Em toda solução numérica a equação diferencial parcial é
substituída por uma aproximação discreta, ou seja, a solução numérica é conhecida somente para um número finito de pontos no domínio físico, enquanto que a solução
analítica deve satisfazer a equação diferencial parcial em cada ponto da região. A
aproximação discreta resulta em um conjunto de equações algébricas que são
calculadas para valores discretos desconhecidos. O conjunto finito de pontos em que
se considera discretizado o domínio é chamado de malha e os pontos chamados de
pontos nodais.

Atenção! Devito atualmente só soporta o  Esquema explícito. Quando é utilizado o
resultado de uma solução num instante anterior já conhecida para calcular a solução
posterior desejada, o esquema de discretização é dito explícito.

## Convecção 1D

Iremos começar nossa discussão da DSL Devito com a solução numérica de uma Equação Diferencial Parcial (EDP) simples ([baseados no Tutorial da profª Lorena Barba; 12 steps to Navier–Stokes](https://github.com/barbagroup/CFDPython/blob/master/lessons/01_Step_1.ipynb)):

A Equação da convecção (ou advecção) linear unidimensional é o modelo mais simples que pode ser usado para aprendermos alguma coisa a respeito da solução numérica de EDPs.

\begin{equation}
\frac{\partial u}{\partial t} = - c \frac{\partial u}{\partial x}
\end{equation}

sujeito à condição inicial $u(x,0)\equiv u_0(x)$ no domínio $[0,L]$. A solução exata para esta equação é dada por
\begin{equation}
u(x,t) = u_0(x-ct)
\end{equation}

Discretizamos esta equação no espaço e no tempo, usando o esquema de diferenças finitas progressivas (Forward Difference) para a derivada do tempo e
o esquema de diferenças finitas regressivas (Backward Difference) para a derivada espacial. Iremos discretizar a coordenada espacial $x$ em pontos que indexamos de $i = 0$ a $N$, e o passo de tempo terá o tamanho $\Delta t$.

Da definição da derivada (apenas removendo o limite), nós sabemos que para um $\Delta x$
suficientemente pequeno:

\begin{equation}
\frac{\partial u}{\partial x} \approx \frac{u(x+\Delta x)- u(x)}{\Delta x}
\end{equation}

Nossa equação discreta, então, é:

\begin{equation}
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t} + c\frac{u_{i}^{n}- u_{i-1}^{n}}{\Delta x}
\end{equation}

sendo que $n$ e $n+1$ representam dois passos consecutivos no tempo, e $i-1$ e $i$ são dois
pontos vizinhos da coordenada $x$ discretizada. Dadas as condições iniciais, a única incógnita nessa
discretização é $u_{i}^{n+1}$. Nós isolamos essa incógnita na equação para resolver o problema seguindo
uma marcha no tempo:

\begin{equation}
u_{i}^{n+1} = u_{i}^{n}-c\dfrac{\Delta t}{\Delta x}(u_{i}^{n}-u_{i-1}^{n})
\end{equation}

### Vamos lá
Primeiro: devemos carregar nossos bibliotecas para lidar com vetores (arrays) e gráficos (plots).

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
```

Vamos começar definindo uma condição inicial - vamos usar:

\begin{equation}
u(x,0)=\left\{
    \begin{array}{l}
      2 \quad\textrm{para} \quad 0,5 \le x \le 1,\\
      1 \quad\forall x \quad \textrm{em} \quad (0,2)
    \end{array}
  \right.
\end{equation}

No código abaixo, nós definimos uma variável chamada $nx$ que representará o número de nós (ou pontos) na grade espacial, e a variável $dx$ que representará a distância entre cada par de pontos adjacentes. Nós também podemos definir um passo de tempo $dt$, o número de passos de tempo, $nt$, e o valor da velocidade da onda: iremos manter as coisas simples e fazer $c=1.$ Também declararamos nossas condições iniciais.

```python
nx = 81 # tente mudar este número de 41 para 81 e rodar tudo novamente ... o que acontece?
dx = 2/(nx-1)
nt = 25
dt = .02
c = 1 # suponha velocidade da onda c = 1
x = np.linspace(0,2,nx)
u = np.ones(nx)      # numpy function ones()
u[int(.5 / dx):int(1 / dx + 1)] = 2  # setting u = 2 between 0.5 and 1 as per our I.C.s
print(x)
print(u)
plt.plot(x,u,'o-')
plt.grid()
```

    [0.    0.025 0.05  0.075 0.1   0.125 0.15  0.175 0.2   0.225 0.25  0.275
     0.3   0.325 0.35  0.375 0.4   0.425 0.45  0.475 0.5   0.525 0.55  0.575
     0.6   0.625 0.65  0.675 0.7   0.725 0.75  0.775 0.8   0.825 0.85  0.875
     0.9   0.925 0.95  0.975 1.    1.025 1.05  1.075 1.1   1.125 1.15  1.175
     1.2   1.225 1.25  1.275 1.3   1.325 1.35  1.375 1.4   1.425 1.45  1.475
     1.5   1.525 1.55  1.575 1.6   1.625 1.65  1.675 1.7   1.725 1.75  1.775
     1.8   1.825 1.85  1.875 1.9   1.925 1.95  1.975 2.   ]
    [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 1. 1. 1. 1. 1. 1. 1.
     1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
     1. 1. 1. 1. 1. 1. 1. 1. 1.]

```python
un = np.ones(nx) #initialize a temporary array

for n in range(nt):  #loop for values of n from 0 to nt, so it will run nt times
    un = u.copy() ##copy the existing values of u into un
    for i in range(1, nx): ## you can try commenting this line and...
    #for i in range(nx): ## ... uncommenting this line and see what happens!
        u[i] = un[i] - c * dt / dx * (un[i] - un[i-1])
```

```python
plt.plot(x, u, 'o-')
plt.grid()
```

## Implementação com Devito

Agora queremos recriar o exemplo acima por meio de um operador Devito. Para fazer isso, podemos começar
definindo nossa grade computacional e criando uma função `u` como a função simbólica `devito.TimeFunction`.
O ponto central a se notar aqui é que esta é uma das funções simbólicas de Devito, que tem um papel duplo na
criação de solucionadores de diferenças finitas:

1. Elas se comportam simbolicamente como objetos `sympy.Function`, para que possamos construir derivadas e usá-las
em expressões simbólicas, herdando assim todo o poder de manipulação simbólica automatizada que o SymPy fornece.
2. Elas agem como contêineres para dados do usuário, fornecendo uma propriedade `.data` que envolve o espaço
de memória alocado automaticamente em uma NumPy array.

O tipo específico `TimeFunction` que usaremos para declarar nossa variável `u`, está ciente do fato de que queremos implementar um algoritmo de avanço no tempo com ele. Assim, o objeto `u` declarará dois buffers (uma vez que a ordem temporal padrão de uma `TimeFunction` é` 1`) de forma `nx` para nós, conforme definido pelo objeto `Grid`, e os apresentará como `u.data[0]` e `u.data[1]`. Vamos preencher o buffer inicial com alguns dados e dar uma olhada:

```python
#NBVAL_IGNORE_OUTPUT
from examples.cfd import plot_field, init_hat
from devito import Grid, TimeFunction

grid = Grid(shape=(nx,), extent=(2.,))
u = TimeFunction(name='u', grid=grid)

u.data[0][:] = 1.
u.data[0][int(.5 / dx):int(1 / dx + 1)] = 2.
# Note que u.data[0] == u.data[0,:]
print(u.data[0],u.data.shape,u.data.size)
plt.plot(x,u.data[0],'o-')
plt.grid()
```

    [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 1. 1. 1. 1. 1. 1. 1.
     1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
     1. 1. 1. 1. 1. 1. 1. 1. 1.] (2, 81) 162

Legal. Agora podemos ver como derivar nosso estêncil de 3 pontos usando os recursos simbólicos dados à nossa
função `u` por SymPy. Para isso, primeiro construiremos nossos termos com as derivadas no espaço e no tempo. Para
a derivada progressiva no tempo, podemos facilmente usar a notação abreviada de Devito `u.dt` para denotar a
primeira derivada no tempo e `u.dxl`para denotar a derivada espaciail. Observe que o `l` significa que estamos
usando as diferenças finitas à "esquerda" ou regressiva aqui para aderir à discretização usada anteriormente.

A partir dos termos resultantes, podemos criar um objeto `sympy.Equation` que contém a equação totalmente discretizada, mas a partir de uma notação de alto nível, como mostrado abaixo.

```python
from devito import Eq

# Specify the `interior` flag so that the stencil is only
# applied to the interior of the domain.
eq = Eq(u.dt + c*u.dxl, subdomain=grid.interior)

print(eq)
eq
```

    Eq(Derivative(u(t, x), x) + Derivative(u(t, x), t), 0)

$\displaystyle \frac{\partial}{\partial x} u{\left(t,x \right)} + \frac{\partial}{\partial t} u{\left(t,x \right)} = 0$

```python
u.forward
```

$\displaystyle u{\left(t + dt,x \right)}$

A etapa acima resultou em uma versão totalmente discretizada de nossa equação, que inclui símbolos de espaço
reservado (*placeholder*) para o espaçamento no tempo (`dt`) e espaço (`h`). Esses símbolos são baseados em uma
convenção interna e serão substituídos posteriormente quando construirmos um operador.

Mas antes de podermos construir um operador, primeiro precisamos mudar nossa expressão discretizada para que estejamos atualizando o ponto de avanço no estêncil em nosso esquema de *timestepping* - Devito fornece outra notação abreviada para isso: `u.forward`. Para o reordenamento simbólico, SymPy vem ao resgate com o comando `solve` que podemos usar para reorganizar nossa equação

```python
from devito import solve
from sympy import nsimplify, pprint

stencil = solve(eq, u.forward)

pprint(stencil)
```

       ⎛  ∂             u(t, x)⎞
    dt⋅⎜- ──(u(t, x)) + ───────⎟
       ⎝  ∂x               dt  ⎠

Agora podemos usar esta expressão de estêncil para criar um `Operator`. Esse `Operator` basicamente se comportará como uma função Python que podemos chamar para aplicar o estêncil
criado sobre nossos dados associados, contanto que forneçamos todas as incógnitas necessárias. Nesse caso,
precisamos fornecer o número de passos no tempo por meio da palavra-chave `time` e o tamanho do passo de tempo
por meio de `dt` (ambos foram definidos acima):

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our initial condition in both buffers.
# This is required to avoid 0s propagating into
# our solution, which has a background value of 1.
for i in range(u.data.shape[0]):
    u.data[i][:] = 1.
    u.data[i][int(.5 / dx):int(1 / dx + 1)] = 2.

# Create an operator that updates the forward stencil point
op = Operator(Eq(u.forward, stencil, subdomain=grid.interior))

# Apply the operator for a number of timesteps
op.apply(time=nt, dt=dt)
```

    Operator `Kernel` ran in 0.01 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=1e-06, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

```python
plt.plot(np.linspace(0, 2, nx), u.data[0], 'o-')
plt.grid()
```

```python
print(op.ccode)
```

    #define _POSIX_C_SOURCE 200809L
    #define START_TIMER(S) struct timeval start_ ## S , end_ ## S ; gettimeofday(&start_ ## S , NULL);
    #define STOP_TIMER(S,T) gettimeofday(&end_ ## S, NULL); T->S += (double)(end_ ## S .tv_sec-start_ ## S.tv_sec)+(double)(end_ ## S .tv_usec-start_ ## S .tv_usec)/1000000;

    #include "stdlib.h"
    #include "math.h"
    #include "sys/time.h"
    #include "xmmintrin.h"
    #include "pmmintrin.h"

    struct dataobj
    {
      void *restrict data;
      int * size;
      int * npsize;
      int * dsize;
      int * hsize;
      int * hofs;
      int * oofs;
    } ;

    struct profiler
    {
      double section0;
    } ;

    int Kernel(const float dt, const float h_x, struct dataobj *restrict u_vec, const int i0x_ltkn, const int i0x_rtkn, const int time_M, const int time_m, const int x_M, const int x_m, struct profiler * timers)
    {
      float (*restrict u)[u_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[u_vec->size[1]]) u_vec->data;

      /* Flush denormal numbers to zero in hardware */
      _MM_SET_DENORMALS_ZERO_MODE(_MM_DENORMALS_ZERO_ON);
      _MM_SET_FLUSH_ZERO_MODE(_MM_FLUSH_ZERO_ON);

      for (int time = time_m, t0 = (time)%(2), t1 = (time + 1)%(2); time <= time_M; time += 1, t0 = (time)%(2), t1 = (time + 1)%(2))
      {
        /* Begin section0 */
        START_TIMER(section0)
        for (int i0x = i0x_ltkn + x_m; i0x <= -i0x_rtkn + x_M; i0x += 1)
        {
          u[t1][i0x + 1] = dt*((u[t0][i0x] - u[t0][i0x + 1])/h_x + u[t0][i0x + 1]/dt);
        }
        STOP_TIMER(section0,timers)
        /* End section0 */
      }

      return 0;
    }

### Outra condição inicial

Agora vamos usar outra condição inicial $u(x,0)\equiv u_0(x)=sin(\omega x)$

```python
nx = 41
L = 1.0
dx = L/(nx-1)
print(dx)
x = np.linspace(0,L,nx)
u = np.sin(2.0*np.pi*x)
plt.plot(x,u,'-o')
plt.grid()
```

    0.025

```python
c = 1.0 # m/s - wave speed
dt = dx/c  # run just at the CFL condition
cfl = c*dt/dx
print(dt, int(1./dt))

un = u.copy() ##copy the existing values of u into un
u[1:-1] = un[1:-1] - cfl * (un[1:-1] - un[:-2])
u[-1] = un[-1] - cfl*(un[-1] - un[-2]) # compute last point on the right using periodicity
u[0] = u[-1] # set periodic boundary on the left
print(u.size,u.shape)
plt.plot(x,un,'-o',x,u,'--o')
plt.grid()
```

    0.025 40
    41 (41,)

```python
c = 1.0 # m/s - wave speed
dt = dx/c  # run just at the CFL condition
cfl = c*dt/dx
print(dt, int(1./dt))
u = np.sin(2.0*np.pi*x)

for n in range(nt):  #loop for values of n from 0 to nt, so it will run nt times
    un = u.copy() ##copy the existing values of u into un
    u[1:-1] = un[1:-1] - cfl * (un[1:-1] - un[:-2])
    u[-1] = un[-1] - cfl*(un[-1] - un[-2]) # compute last point on the right using periodicity
    u[0] = u[-1] # set periodic boundary on the left
print(u.size,u.shape)
```

    0.025 40
    41 (41,)

```python
plt.plot(np.linspace(0,L,nx),u,'-o')
plt.grid()
```

```python
grid = Grid(shape=(nx,), extent=(1.,))
u = TimeFunction(name='u', grid=grid)

u.data[0][:] = np.sin(2.0*np.pi*x)
# Note que u.data[0] == u.data[0,:]
print(u.data[0],u.data.shape,u.data.size)
plt.plot(x,u.data[0],'o-')
plt.grid()
```

    [ 0.0000000e+00  1.5643446e-01  3.0901700e-01  4.5399049e-01
      5.8778524e-01  7.0710677e-01  8.0901700e-01  8.9100653e-01
      9.5105654e-01  9.8768836e-01  1.0000000e+00  9.8768836e-01
      9.5105654e-01  8.9100653e-01  8.0901700e-01  7.0710677e-01
      5.8778524e-01  4.5399049e-01  3.0901700e-01  1.5643446e-01
      1.2246469e-16 -1.5643446e-01 -3.0901700e-01 -4.5399049e-01
     -5.8778524e-01 -7.0710677e-01 -8.0901700e-01 -8.9100653e-01
     -9.5105654e-01 -9.8768836e-01 -1.0000000e+00 -9.8768836e-01
     -9.5105654e-01 -8.9100653e-01 -8.0901700e-01 -7.0710677e-01
     -5.8778524e-01 -4.5399049e-01 -3.0901700e-01 -1.5643446e-01
     -2.4492937e-16] (2, 41) 82

```python
eq = Eq(u.dt + c*u.dxl, subdomain=grid.interior)

stencil = solve(eq, u.forward)
update = Eq(u.forward, stencil)#, subdomain=grid.interior)
update

```

$\displaystyle u{\left(t + dt,x \right)} = dt \left(- 1.0 \frac{\partial}{\partial x} u{\left(t,x \right)} + \frac{u{\left(t,x \right)}}{dt}\right)$

```python
x = grid.dimensions
t = grid.stepping_dim
bc_left = Eq(u[t + 1, 0], u[t + 1, nx-1])
```

```python
for i in range(u.data.shape[0]):
    u.data[i][:] = np.sin(2.0*np.pi*np.linspace(0,L,nx))

#op2 = Operator([update], opt='noop')
op2 = Operator([update]+[bc_left], opt='noop')
op2(time=0, dt=dt)
```

    Operator `Kernel` ran in 0.01 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=1e-06, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

```python
plt.plot(np.linspace(0,L,nx),u.data[0],'o-',np.linspace(0,L,nx),u.data[1],'--o')
plt.grid()
```

```python
print(op2.ccode)
```

    #define _POSIX_C_SOURCE 200809L
    #define START_TIMER(S) struct timeval start_ ## S , end_ ## S ; gettimeofday(&start_ ## S , NULL);
    #define STOP_TIMER(S,T) gettimeofday(&end_ ## S, NULL); T->S += (double)(end_ ## S .tv_sec-start_ ## S.tv_sec)+(double)(end_ ## S .tv_usec-start_ ## S .tv_usec)/1000000;

    #include "stdlib.h"
    #include "math.h"
    #include "sys/time.h"

    struct dataobj
    {
      void *restrict data;
      int * size;
      int * npsize;
      int * dsize;
      int * hsize;
      int * hofs;
      int * oofs;
    } ;

    struct profiler
    {
      double section0;
    } ;

    int Kernel(const float dt, const float h_x, struct dataobj *restrict u_vec, const int time_M, const int time_m, const int x_M, const int x_m, struct profiler * timers)
    {
      float (*restrict u)[u_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[u_vec->size[1]]) u_vec->data;

      for (int time = time_m, t0 = (time)%(2), t1 = (time + 1)%(2); time <= time_M; time += 1, t0 = (time)%(2), t1 = (time + 1)%(2))
      {
        /* Begin section0 */
        START_TIMER(section0)
        for (int x = x_m; x <= x_M; x += 1)
        {
          u[t1][x + 1] = dt*(1.0F*u[t0][x]/h_x - 1.0F*u[t0][x + 1]/h_x + u[t0][x + 1]/dt);
        }
        STOP_TIMER(section0,timers)
        /* End section0 */
        u[t1][1] = u[t1][41];
      }

      return 0;
    }

```python
for i in range(u.data.shape[0]):
    u.data[i][:] = np.sin(2.0*np.pi*np.linspace(0,L,nx))

op2(time=nt, dt=dt)
```

    Operator `Kernel` ran in 0.01 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=4.9999999999999996e-06, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

```python
plt.plot(np.linspace(0,L,nx),u.data[1],'o-')
plt.grid()
```

# Modelagem sísmica

Os estudos de modelagem são de grande relevância na investigação e prospecção de petróleo. Um dos
processos de investigação do meio geológico é o da utilização de um modelo físico (acústico, anisotropico, elastico etc) na implementação numérica da equação da onda e assim
obter informações de subsuperfície (snapshots, sismogramas).

O método de diferenças finitas soluciona a equação da onda (uma EDP), substituindo as derivadas parciais no espaço e no tempo por suas aproximações de diferenças finitas em uma malha retangular discretizada.

Iremos estudar a **Solução da equação da onda acústica pelo método das diferenças finitas** através de um [Jupyter Notebook](https://github.com/devitocodes/devito/blob/master/examples/seismic/tutorials/01_modelling.ipynb) do repositorio oficial do [Devito](https://github.com/devitocodes/devito).


# Practical 2a: Deep Feed-forward Networks

© Deep Learning Indaba. Apache License 2.0.

## Introduction
In this practical, we implement and train a feed-forward neural network (also known as an "MLP" for "multi-layer perceptron") on a dataset called "Fashion MNIST", consisting of small greyscale images of items of fashion. We consider the practical issues around generalisation to out-of-sample data and introduce some important techniques for addressing this.

## Learning Objectives
* Understand how to use **Tensorflow Eager** and **Keras Layers** to build a neural network architecture
* Understand how a model is trained and evaluated
* Be able to explain why there is a difference between **in-sample** and **out-of-sample** model performance
* Understand the concept of **train/validation/test split** and why it's useful
* Research at least 1 technique that can be used to improve model **generalisation**

```
#@title Imports (RUN ME!) { display-mode: "form" }

!pip install tensorflow-gpu==2.0.0-beta0 > /dev/null 2>&1

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

print("TensorFlow executing eagerly: {}".format(tf.executing_eagerly()))
```

## The Data

In this practical, we use the Fashion MNIST dataset consisting of 70,000 greyscale images and their labels. The dataset is divided
 into 60,000 training images and 10,000 test images. The idea is to train a **classifier** to identify the class value (what type of fashion item it is) given the image. We train and *tune* a model on the 60,000 training images and then evaluate how well it classifies the 10,000 test images that the model did not see during training. This task is an example of a **supervised learning** problem, where we are given both input and labels (targets) to learn from. This is in contrast to **unsupervised learning** where we only have inputs from which to learn patterns or **reinforcement learning** where an agent learns how to maximise a reward signal through interaction with its environment.

```
# Tensorflow has convenient modules for loading a number of standard datasets
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_and_validation_images, train_and_validation_labels), (test_images, test_labels) = fashion_mnist.load_data()

text_labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
```

###Optional extra reading: Train/Validation/Test Split

When we build machine learning models, the goal is to build a model that will perform well on *future* data that we have not seen yet. We say that we want our models to be able to **generalise** well from whatever training data we can collect and do have available, to whatever data we will be applying them to in future. To do this, we split whatever data we have available into a **training set, a validation set and a test set**. The idea is that we train our model and use the performance on the validation set to make any adjustments to the model and its hyperparameters, but then we report the final accuracy on the test set. The test set (which we never train on), therefore acts as a proxy for our future data.

### Construct a validation set
Next, we remove 10,000 images and labels from the training set to use as a validation set. We will come back to the validation set later!

```
# Construct a validation set from the last 10000 images and labels from
# train_and_validation_images and train_and_validation_labels
validation_images = train_and_validation_images[-10000:, :, :]
validation_labels = train_and_validation_labels[-10000:]

# Construct a training set from the first 50000 images and labels.
train_images = train_and_validation_images[:50000, :, :]
train_labels = train_and_validation_labels[:50000]
```

### What does the data look like?
Each image in the dataset consists of a 28 x 28 matrix of greyscale pixels. The values are between 0 and 255 where 0 represents black, 255 represents white and there are many shades of grey in-between. Each image is assigned a corresponding numerical label, so the image in ```train_images[i]``` has its corresponding label stored in ```train_labels[i]```. We also have a lookup array called ```text_labels``` to associate a text description with each of the numerical labels. For example, the label 1 is associated with the text description "Trouser".

The 2 cells below demonstrate how to visualise the input data. Make sure you understand what's happening, particularly how the indices correspond to individual items in the dataset.

```
# We use the Matplotlib plotting library to visualise an image selected at random from the training set
plt.figure()
random_index = np.random.randint(0, len(train_images))
plt.imshow(train_images[random_index], cmap='gray_r')
plt.colorbar()
numerical_label = train_labels[random_index]
text_description = text_labels[numerical_label]
plt.title('True Class: {} ("{}")'.format(numerical_label, text_description))

plt.gca().grid(False)
```

```
# Another view, showing 25 randomly selected images at a time
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)

    img_index = np.random.randint(0, 50000)
    plt.imshow(train_images[img_index], cmap="gray_r")
    plt.xlabel(text_labels[train_labels[img_index]])
```

### Exploratory Task
Run the above cells multiple times to get a good sense of the dataset. Look at the general structure of the images and their corresponding labels. Can you spot some classes that might be difficult for a classifier to distinguish? If so, why do you think they would be difficult to distinguish? Chat to your neighbour about this.

### Preparing the data with TensorFlow
At the moment, our training data consists of two large tensors. The images are stored in a tensor of shape $[50000, 28, 28]$, consisting of all the $28 \times 28$ images matrices stacked together. The labels are stored in a 1D vector of shape $[50000]$. We wish to train a model using **mini-batch stochastic gradient descent**. In order to do so, we need to shuffle the data and split it into smaller (mini-)batches. We also convert the data from numpy arrays to TensorFlow Tensors.

#### Questions
* Can you explain what the difference is between normal stochastic gradient descent and "batched" stochastic gradient descent? What is the purpose of chunking data into batches during training? Are there any trade-offs? What are they? **HINT**: Speak to your neighbours and tutors about this. You can also read [this blog post](https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/) for more details.
* Why is it important to randomise the data before chunking it into batches?

In order to do this batching (and shuffling) we will use the Tensorflow [Dataset API](https://www.tensorflow.org/api_docs/python/tf/data/Dataset), which is a set of simple reusable components that allow you to build efficient data pipelines. Data is said to "stream" through the pipeline, meaning that when something at the output of the pipeline wants data, the pipeline will provide that data as soon as it has enough, rather than waiting to process all the data. This allows you to easily build pipelines that work on large datasets without having to load it all into memory in one go!

We build this pipeline step-by-step:

We start by defining the ```batch_size``` hyperparameter of our model. This hyperparameter controls the sizes of the mini-batches (chunks) of data that we use to train the model. The value you use will affect the memory usage, speed of convergence and potentially also the performance of the model. It also interacts with the *learning rate* used in gradient descent.

```
batch_size = 128
```

The first component we add will group the image and label tensors together into a tuple and then split them into individual entries - ie. 50000 tuples containing a (28, 28) dimensional image and 1D label associated with that image. The following line adds this splitting component to the pipeline.

```
train_ds = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
```

The next thing we do is apply a map operation. This lets us run an arbitrary function on each element. The function we provide returns the image values divided by 255 and converted ('cast') to a float and the label converted to a 32-bit integer.

**NOTE**: "Lambda" functions are just one-line, anonymous functions. In this case, it defines a function that takes arguments x and y (before the colon), and outputs their manipulated values (after the colon).

```
# Divide image values and cast to float so that they end up as a floating point number between 0 and 1
train_ds = train_ds.map(lambda x, y: (tf.cast(x, tf.float32) / 255.0, tf.cast(y, tf.int32)))
```

Then we add a **shuffle** component. This returns a random element from the pipeline. Can you spot a potential problem here?

**QUESTION**:  What might happen if the shuffle component didn't have a *buffer*?  **HINT**: Refer back to the discussion about the data pipeline and "data streaming through it" above.

```
# Shuffle the examples.
train_ds = train_ds.shuffle(buffer_size=batch_size * 10)
```

The final component in our pipeline is the batch component. This just requests `batch_size` elements from the previous pipeline component, groups them together into a single tensor and returns that.

```
# Now "chunk" the examples into batches
train_ds = train_ds.batch(batch_size)

# The output of this pipeline will be tuples of tensors containing images and labels.
# The images will be of shape (batch_size, 28, 28) and the labels of shape (batch_size, )
```

Finally, we apply the same pre-processing (converting to Tensors and normalising the values to lie between 0 and 1) to the validation set.

```
# Don't worry about this for now, we will use the validation set later
val_ds = tf.data.Dataset.from_tensor_slices((validation_images, validation_labels))
val_ds = val_ds.map(lambda x, y: (tf.cast(x, tf.float32) / 255.0, tf.cast(y, tf.int32)))
```

## Define the model

In this section we'll build a classifier. A **classifier** is a function that takes an object's characteristics (or "features") as inputs and outputs a prediction of the class (or group) that the object belongs to. It may make a single prediction for each input or it may output some score (for example a probability) for each of the possible classes. Specifically, we will build a classifier that takes in (a batch of) 28 x 28 Fashion MNIST images as we've seen above, and outputs predictions about which class the image belongs to.

For each (batch of) input images, we will use a **feed-forward neural network** to compute un-normalised scores (also known as **logits**) for each of the 10 possible classes that the image could belong to. We can then **classify** the image as belonging to the class which receives the highest score, or we can quantify the model's "confidence" about the classifications by converting the scores into a probability distribution.

A feed-forward neural network consisting of $N$ layers, applied to an input vector $\mathbf{x}$ can be defined as:

\begin{equation}
\mathbf{f_0} = \mathbf{x} \\
\mathbf{f_i} = \sigma_i(\mathbf{W_if_{i-1}} + \mathbf{b_i}) \ \ \ i \in [1, N]
\end{equation}

Each layer has a particular number, $m_i$, of neurons. The parameters of a layer consist of a matrix $\mathbf{W_i} \in \mathbb{R}^{m_i \times m_{i-1}}$ and bias vector $\mathbf{b_i} \in \mathbb{R}^{m_i}$. Each layer also has a non-linear activation function $\sigma_i$.

**QUESTION**: Why do you think the activation functions need to be *non-linear*? What would happen if they were *linear*? **HINT**: If you're stuck, consider the very simplest case of an identity activation (which essentially does nothing) and ignore the biases.

### Optional extra reading: Activation functions

Activation functions are a core ingredient in deep neural networks. In fact they are what allows us to make use of multiple layers in a neural network. There are a number of different activation functions, each of which are more or less useful under different circumstances. The four activation functions that you are most likely to encounter are, arguably, [ReLU](https://www.tensorflow.org/api_docs/python/tf/keras/layers/ReLU), [Tanh](https://www.tensorflow.org/api_docs/python/tf/keras/activations/tanh), [Sigmoid](https://www.tensorflow.org/api_docs/python/tf/keras/activations/sigmoid), and [Softmax](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Softmax).

ReLU, has in recent years, become the default choice for use in MLPs and Convolutional Neural Networks (CNNs). ReLU has two advantages over Tanh and Sigmoid: it is computationally much more efficient, and, it allows us to use deeper networks because it does not suffer from [vanishing gradients](https://en.wikipedia.org/wiki/Vanishing_gradient_problem). As a result of their success, a number of ReLU variants, such as [LeakyRelu](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LeakyReLU) and [PReLU](https://www.tensorflow.org/api_docs/python/tf/keras/layers/PReLU), have been developed.

Sigmoid and Softmax activations are often found after the last layer in binary and multi-class classification networks, respectively, as they transform the outputs of the network in such a way that we can interpret them as class probabilities.

Both Tanh and Sigmoid are found in LSTM and GRU recurrent neural networks, which we will find out more about in the coming days. They are also useful in MLPs and CNNs where we want the output to be bounded between -1 and 1 (Tanh) or 0 and 1 (Sigmoid).

Read more about activation functions [here](https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6).

### Configure the feed-forward neural network
We configure the feed-forward neural network part of our classifier using the [Keras Layers API](https://www.tensorflow.org/api_docs/python/tf/keras/layers). This API consists of various reusable building-blocks that allow us to define many different neural network architectures (similar to how we defined a data pipeline earlier!).

Here we use the [Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) component which allows us to wrap together a sequence of layers. An important point to note here is that we are **configuring** our neural network architecture as a pipeline. We can think of the resulting ```model`` variable as a *function* that takes a batch of images as inputs and outputs a batch of logits.

```
# RE-RUN THIS CELL if you want to restart training!
model = tf.keras.Sequential([
    # Convert the 28x28 image into a flat vector of 28x28 = 784 values
    tf.keras.layers.Flatten(input_shape=(28, 28), name='flatten_input'),
    # Create a "hidden" layer with 256 neurons and apply the ReLU non-linearity
    tf.keras.layers.Dense(256, activation=tf.nn.relu, name='input_to_hidden1'),
    # Create another hidden layer with 128 neurons
    tf.keras.layers.Dense(128, activation=tf.nn.relu, name='hidden1_to_hidden2'),
    # Create an "output layer" with 10 neurons
    tf.keras.layers.Dense(10, name='hidden_to_logits'),
])
```

The following summary shows how many parameters each layer is made up of (the number of entries in the weight matrics and bias vectors). Note that a value of ```None``` in a particular dimension of a shape means that the shape will dynamically adapt based on the shape of the inputs. In particular, the output shape of the ```flatten_input``` layer will be $[N, 784]$ when the batch of inputs passed to the model has shape $[N, 28, 28]$

```
model.summary()
```

**QUESTION**: Note that the flattening operation has no parameters. However, the Dense (fully-connected) layer mapping from the 784-D output of the flatten operation to the 256-D hidden layer has 200,960 parameters! Can you explain why the dense layer has 200,960 parameters? **HINT**: Refer back to the equations of a feed-forward network.

### Define the loss
As we did in the previous practical, we need to specify a loss function for our classifier. This tells us how good our model's predictions are compared to the actual labels (the targets), with a lower loss meaning better predictions. The standard loss function to use with a **multi-class classifier** is the **cross-entropy loss** also known as the "negative log likelihood" for a classifier. Suppose we have a classification problem with $C$ classes. A classifier is trained to predict a probability distribution $p(y | X_i)$ for each input $X_i$ from a batch of $N$ examples. The vector $p(y|X_i)$ is $C$ dimensional, sums to one, and we use $p(y|X_i)_c$ to denote the $c$th component of  $p(y|X_i)$. The true class for example $i$ in the batch is $y_i$ and we define the indicator function $\mathbb{1}[y_i=c]$ to be 1 whenever $y_i = c$ and $0$ otherwise. This classifier has a cross-entropy loss of

$- \frac{1}{N}\sum_{i=1}^N \sum_{c=1}^C log( p(y|X_i)_c) \mathbb{1}[y_i=c]$

**NOTE**: The indicator is one for the true class label, and zero everywhere else. So in that sum, the indicator just "lifts out" the $log(p(y|X_i))$ values for all true classes. So the above expression is minimised (note the negative at the front) when the model places all its probability mass on the true labels for all the examples. Remember that  log(1)=0 , thus the closer all probabilities of $y_i = c$ are to one, the lower the loss will be and the better the model will be performing.

**QUESTION**:
* Why do you think this is a *good* loss function?
* Can you think of any potential issues with this loss function?

Fortunately we don't need to write this function ourselves as Tensorflow provides a version called

```tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)```.

**NOTE**: This function actually computes the cross entropy loss directly from the un-normalised logits, rather than from the probability distribution for numerical stability.

By the way, for training data in which the labels are themselves distributions rather than exact values, this definition of cross-entropy still works, where the indicator function is replaced with the corresponding probability of each class for that example. This might be important when we are not sure whether the training data has been labelled correctly, or when the data was labelled by a human who gave their answer along with a degree of confidence that the answer was correct

## Train the model
Now that we have our data, data processing pipeline, our neural network architecture and the corresponding loss that we want to minimise, we need to **train** the model using batched stochastic gradient descent. We do this in multiple **epochs**, which is a single iteration through the entire training dataset. Briefly, in each epoch we loop over all the batches of images and labels, and for each batch we perform the following steps:
* Get the **predictions** of the model on the current batch of images
* Compute the **average loss** values across the batch, telling us how good these predictions are / how close they are to the true targets.
* Compute the **gradient of the average loss** (or the average gradient of the losses in the batch) with respect to each of the model's parameters: This tells us the direction to move in "parameter space" to decrease the loss value
* **Adjust the parameters** by taking a small step in the direction of each component of the gradient (where the learning rate controls the size of the step)

During training we also track some metrics, such as the loss and accuracy to see how well the classifier is doing. Note that the cell below may take a few minutes to run!

###Optional extra reading: Computing gradients in Eager mode

In graph mode, TensorFlow builds up a "computation graph" which captures all operations of the model and their dependencies. For training, the gradient can then be computed by traversing backwards from every node through its dependents and applying the "chain rule" of differentiation.

However, in Eager mode, we don't have the concept of the computation graph anymore. Operations are performed imperatively (in the order in which they were executed). TensorFlow therefore uses a mechanism called the "GradientTape" for computing gradients in Eager mode. Basically, the gradient tape records the order of all operations as they are executed, and can then be "run backwards" (traversed from last to first operation) for computing the gradients.

### Defining the training loop

```
# Choose an optimizer and loss function for training:
# The optimizer is responsible for controlling the learning rate
optimizer = tf.keras.optimizers.Adam()
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# Select metrics to measure the loss and the accuracy of the model.
# These metrics accumulate the values over epochs and then print the overall result.
train_loss = tf.keras.metrics.Mean(name='train_loss')
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')

num_epochs = 50  # The number of epochs to run

# Lists to store the loss and accuracy of every epoch
train_losses = []
train_accuracies = []

# Defined a function to train the model using tf.GradientTape
@tf.function
def train_step(image, label):
  # Initialise a GradientTape to track the operations
  with tf.GradientTape() as tape:
    # Compute the logits (un-normalised scores) of the current batch of examples
    # using the neural network architecture we defined earlier
    logits = model(image)
    loss = loss_object(label, logits)
  gradients = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(gradients, model.trainable_variables))

  # Add current batch loss to our loss metric tracker - note the function call semantics
  train_loss(loss)
  train_accuracy(label, logits)

# Define a function to test the model
# @tf.function
# def val_step(image, label):
  # TODO

for epoch in range(num_epochs):
  # Loop over our data pipeline
  for image, label in train_ds:
    train_step(image, label)

  template = 'Epoch {:03d}, Loss: {:.3f}, Accuracy: {:.3%}'
  print(template.format(epoch+1,
                        train_loss.result(),
                        train_accuracy.result()))

  train_losses.append(train_loss.result())
  train_accuracies.append(train_accuracy.result())
```

```
# Plot the loss for all epochs using Matplotlib
plt.figure()
plt.plot(range(num_epochs), train_losses)
plt.title('Loss vs epochs')

# Plot the accuracy for all epochs using Matplotlib
plt.figure()
plt.plot(range(num_epochs), train_accuracies)
plt.title('Accuracy vs epochs')
```

The above code block shows a typical training loop. There's actually an easier way to do this using Tenosrflow and Keras, which we'll see in the next prac, but for now we expand it so you can see what's going on.

Lets visualise some of the model's prediction on the training set

```
images, labels = next(iter(train_ds))  # Get a batch of images and labels

_logits = model(images, training=False)  # Pass the images to the model function and get its output logits
predicted_labels = tf.argmax(_logits, axis=1, output_type=tf.int32)

img_indexs = np.arange(images.numpy().shape[0])
np.random.shuffle(img_indexs)

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)

    img_index = img_indexs[i]
    predicted_label = int(predicted_labels[img_index])

    plt.imshow(images[img_index], cmap="gray_r")

    actual_label = int(labels[img_index].numpy())
    plt.xlabel("Actual: {} ({})\n Predicted: {} ({})".format(
        actual_label, text_labels[actual_label], predicted_label, text_labels[predicted_label]
    ))

plt.tight_layout()
plt.show()
```

 ### Optional extra reading: Optimisation schemes

You might have noticed that we are using the Adam optimizer to train our neural networks. Adam is a variant of stochastic gradient descent which often performs well in practice. In fact, there is a whole range of variations on stochastic gradient descent that are often used. Here is an illustration showing how a few of these methods perform on a toy problem:

For a detailed description of various optimization methods read [this](http://ruder.io/optimizing-gradient-descent/) article. For a great visual discussion on how these methods work see [this](https://distill.pub/2017/momentum/) article.

## Testing

After 50 epochs of training on our training set, we obtain a loss of around 0.061 and accuracy of 97.7%. That's pretty good, right?

It is important to distinguish between data that is **in-sample** and **out-of-sample**. Our training data is all in-sample and we would expect any resonably powerful model (like an MLP) to be very accurate and have a low loss on this (in fact, with a sufficiently large MLP and enough training epochs, we can get the loss arbitrarily close to zero!). This is not a good thing though, because pushing the loss close to zero may mean that the model has fit the **noise** in the training data rather than the true signal. If this is the case, it will not perform well on out-of-sample data that it has not seen before (which is what we really care about!). To assess this, we evaluate our trained model on the held-out test set (but we don't update the parameters of the model when testing it, that would be cheating!)

```
# We need to apply the same pre-processing to the test set as we did to the training set
# Since we don't need batching or shuffling, we can do this directly instead of
# building a tf.Dataset pipeline

tf_test_images = tf.convert_to_tensor(test_images, dtype=tf.float32) / 255.0
tf_test_labels = tf.convert_to_tensor(test_labels, dtype=tf.int32)
```

```
test_logits = model(tf_test_images, training=False)

# Compute the average cross-entropy loss of the classification over the entire test set
test_loss = tf.keras.metrics.Mean(name='test_loss')
t_loss = loss_object(tf_test_labels, test_logits)
test_loss(t_loss)

# Compare predicted labels to actual labels
test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')
test_accuracy(tf_test_labels, test_logits)

print('Completed testing on', tf_test_images.shape[0], 'examples...')
print('Loss: {:.3f}, Accuracy: {:.3%}'.format(test_loss.result(), test_accuracy.result()))
```

And again we visualise some of the model's predictions:

```
test_predictions = tf.argmax(test_logits, axis=1, output_type=tf.int32)

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)

    img_index = np.random.randint(0, 10000)
    plt.imshow(test_images[img_index], cmap="gray_r")

    actual_label = int(test_labels[img_index])
    predicted_label = int(test_predictions[img_index])

    plt.xlabel("Actual: {} ({})\n Predicted: {} ({})".format(
        actual_label, text_labels[actual_label], predicted_label, text_labels[predicted_label]
    ))

plt.tight_layout()
plt.show()
```

What happened? We got 97.7% accuracy on the training set, but only 88.9% on the test set. This is an example of how a model's in-sample performance differs from its out-of-sample performance. It may even be considered **overfitting** (where the model failing to generalise its in-sample performance to unseen out-of-sample data). There are a number of ways we can address this, but first, it's important to realise that we should try and keep the number of evaluations on the test set to a minimum.

**QUESTION**: Can you think of WHY we should limit the number of evaluations on the test set? This is an important point, so spend a few minutes thinking about it, discuss it with your neighbour and ask the tutors if you get stuck!

The first important method is called **validation**. Using validation, we reserve a portion of our training dataset and call it the validation set. After each epoch of training (or every K epochs where K can be a fraction), we evaluate our model on this validation set (without updating the model parameters). This gives us a good sense of how well we expect our model to do on out-of-sample data. In this way, the validation set acts as a surrogate for the test set which we can safely use during model development and training. We can also use this validation set to tune our model by selecting hyperparameters such as the batch size, sizes of layers or the network architecture.

### Optional extra reading: Underfitting

In addition to the problem of _overfitting_ (in which the model is too complex and as a result is able to memorize the training data rather than learning a general pattern) there is the problem of _underfitting_ which is the opposite. Underfitting occurs when the model does not have enough complexity (also known as capacity) to learn a general pattern. While overfitting is characterised by diverging test/validation and training scores, underfitting is characterised by test/validation and training scores that very slowly continue to improve.

Unlike overfitting which has a number of solutions, such as collecting more data, data augmentation, dropout, and L1/L2 regularization, underfitting has one simple solution: just make your model incrementally more complex until it no longer underfits.

##TASKS:

1. [**ALL**] Implement **Validation** by adding some code to the loop in the "Train the model" section (**TIP**: The code for evaluating on the validation set will be very similar to evaluating on the test set!):
 * Evaluate your model on the validation set at the end of each training epoch and print the loss and accuracy on the validation set. We have already defined a validation set and done the necessary preprocessing for you. What do you notice?
 * Plot the validation loss and accuracy on the same axes as the training loss and accuracy, respectively.
  * Evaluate how closely the loss and accuracy on the validation set matches the eventual performance on the test set.

2. [**ALL**] Add DROPOUT (see [Tensorflow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout) and [research paper](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) ) to improve the model's generalisation.

3. [**ADVANCED**] Add BATCH NORMALISATION ([Tensorflow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization) and [research paper](http://proceedings.mlr.press/v37/ioffe15.pdf)) to improve the model's generalisation.

4. [**OPTIONAL**] Remove the shuffling from the dataset pipeline and compare the generalisation performance of the model with and without shuffling.

5. [**OPTIONAL**] Adjust the model architecture by adding layers and modifying other parameters to improve the performance on your **validation** set. Once you've settled on an architecture and hyperparameters, evaluate your model on the **test** set. See the Tensorflow documentation for examples of parameters you can change in the model.

## Important Tip
If you want to re-train your model, you need to rerun all the code in the "Define the model" section first. This is to ensure that the Tensorflow variables containing the model parameters get reset. If you don't do this, you're effectively running more epochs on an already-trained model.


Table of Contents

1  Probabilty Fundamentals

2  Objectives

3  Set Theory Basics

3.1  Set Operations

4  Probability Theory

4.1  Why Should I Care About Probabilities?

4.2  Probability Terminology

4.2.1  Random Variables

4.3  Probability Using Sets

4.4  Independence

4.4.1  🧠 Knowledge Check

5  Conditional Probability

5.1  Fermat and Pascal: The Unfinished Game

5.2  A New Condition

5.3  Conditional Probability Rules

5.4  Conditional Probability Example

6  The Law of Total Probability

6.1  Aside: Mathematical Notation and Formalism

7  Level Up: More Set Theory

7.1  A Note on Set Notation

7.1.1  Quick Notation Example

7.1.2  🧠 Knowledge Check

7.2  Empty Set

7.3  Subsets

7.4  Joint Probability

7.5  Python & Sets

7.5.1  Empty Set

7.5.2  Iterables

7.5.3  Subsets

7.5.4  Set Operations - Example

8  Level Up: Bayes Theorem

9  Level Up: Permutations & Combinations

9.0.1  Permutations

9.0.1.1  Easiest Scenario

9.0.1.2  Intermediate Scenario

9.0.1.3  Level Up Scenario

9.0.2  Combinations

9.0.2.1  Easier Scenario:

9.0.2.2  Level Up Scenario

10  Level Up: Three Events

# Probabilty Fundamentals

> [xkcd comic 1252](https://xkcd.com/1252/)

# Objectives

- Describe the fundamentals of probability theory
- Describe set theory and its terminology
- Define conditional probability

# Set Theory Basics

In probability theory, a **set** is a well-defined collection of objects (called **elements**). An element is either in a given set or not and order is not considered.

If an **element** $x$ belongs to a set $S$, then you'd write $x \in S$.

On the other hand, if $x$ does not belong to a set $S$, then you'd write $x\notin S$.

> **More Math & Details**
>
> Set theory is a whole branch of mathematics just like calculus or geometry so there's a lot we could talk about! We won't here but you can see some extra information in a [Level Up section](#Level-Up:-More-Set-Theory) towards the end of this notebook.

Although sets can be abstract mathematical object, we'll typically think of sets as a collection of times sharing some relevant attribute(s).

## Set Operations

Let's first imagine two sets $A$ and $B$ with some overlapping elements

The **union** of 2 sets $A$ and $B$ is the set $T$ of elements of either $A$ or $B$, or in both. We denote this with the symbol $\cup$.

$$\large T = A\cup B = B\cup A$$

The **intersection** of two sets $A$ and $B$ is the set $V$ that contains all elements of $A$ that also belong to $B$. We denote this with the symbol $\cap$.

$$\large V = A\cap B = B\cap A$$

# Probability Theory

Probability theory is the study of the frequency of a given event occurring with respect to all possible events. In this section we'll discuss the event space & sets, calculate the probability of events, and discuss the relevance of probability theory in data science.

## Why Should I Care About Probabilities?

Studying probabilities allows us to make better and more informed decisions, based on data previously collected. For example, understanding the fact that it is nearly impossible for us to ever win the lottery from a probabilistic standpoint deters us from using it as a source of income.

Probability theory also lies at the heart of making inferences using our data, which is what statistics is all about!

## Probability Terminology

- An **outcome** is a possible result of a random phenomenon (e.g. a 5 on a six-sided die)
- A **sample space** is the set of all possible outcomes for a random phenomenon (e.g. {1, 2, 3, 4, 5, 6} for rolling a six-sided die)
- An **event** is a specific outcome of a random phenomemon (e.g. actually rolling a six-sided die and getting a 5)

### Random Variables

> [Source: _Dilbert_ comic](https://dilbert.com/strip/2001-10-25)

> A **random variable** has an outcome that can take on different values, resulting from a random phenomenon

- A random variable can either be continuous or discrete
    - **Continuous**: the variable can have any values within a range (e.g. a person's height)
    - **Discrete**: the variable can only have a finite subset of values within a range (e.g. number of cars owned)

This allows us think of events in mathematical terms and use mathematical notation to abstract the ideas.

For example, let's state that $Y$ is the number of tails after $10$ flips.

Thus we can write these examples:

- $P(Y=3)$ is _the probability of having $3$ tails after flipping a coin $10$ times_
- $P(Y\le8)$ is _the probability of having $8$ or fewer tails after flipping a coin $10$ times_
- $P(Y\text{ is even})$ is _the probability that the number of tails is even after flipping a coin $10$ times_

Hopefully, you can now see the power of abstracting these outcomes with notation.

## Probability Using Sets

In its most basic form, the **probability** of an event occurring is the size of the successful outcome set divided by the size of the sample space. In some ways, probability is just about counting outcomes.

**Example**: What is the probability of rolling an even number on a six-sided die?

* Successful outcomes: {2, 4, 6} has size 3
* Sample space: {1, 2, 3, 4, 5, 6} has size 6
* Probability = 3/6 = 1/2 = 50%

We sometimes write probability statements using notation like $P(E) = 0.5$, where $E$ is an event, such as rolling an even number on a six sided die.

## Independence

A special condition is when the outcome of $A$ has no bearing on the outcome of $B$. We say these two events are **independent** (e.g. rolling a die and tossing a coin).

Formally, *$A$ and $B$ are independent* if and only if the probability that *both* $A$ *and* $B$ happen is:

$$P(A \cap B) = P(A) \cdot P(B)$$

Hopefully, we can see where this comes from when two events are independent. If the the event doesn't have any effect on the other, then the chances of them happening is probability of the first event multiplied by the second.

> **NOTE**
>
> In many cases we'll treat events as independent for convenience even if they technically are not independent from one another. Usually you can ignore very small dependence effects.
>
> For example, we would likely describe two coin flips of the same coin (two events) to be independent from one another. But technically speaking, maybe after a coin flip the side that the coin that lands face-down is damaged and slightly affects future coin flips. In this case, we likely would ignore this affect and assume a sequence of coin flips to be independent.

### 🧠 Knowledge Check

**AND Question**: What is the probability of rolling a 5 on a die _and_ getting a tails on a coin toss?

**OR Question**: What is the probability of rolling a 5 on a die _or_ getting a tails on a coin toss?

# Conditional Probability

## Fermat and Pascal: The Unfinished Game

> This example borrowed from Peter Norvig's excellent repo [here](https://github.com/norvig/pytudes).

Pierre de Fermat
1654
Blaise Pascal]
1654

Consider a gambling game consisting of tossing a coin repeatedly. Player H wins the game as soon as a total of 10 heads come up, and T wins if a total of 10 tails come up before H wins. If the game is interrupted when H has 8 heads and T has 7 tails, how should the pot of money (which happens to be 100 Francs) be split?  Here are some proposals, and arguments against them:
- It is uncertain, so just split the pot 50-50.
    - *No, because surely H is more likely to win.*
- In proportion to each player's current score, so H gets a 8/(8+7) share.
    - *No, because if the score was 0 heads to 1 tail, H should get more than 0/1.*
- In proportion to how many tosses the opponent needs to win, so H gets 3/(3+2).
    - *This seems better, but no, if H is 9 away and T is only 1 away from winning, then it seems that giving H a 1/10 share is too much.*

In 1654, Blaise Pascal and Pierre de Fermat corresponded on this problem, with Fermat [writing](http://mathforum.org/isaac/problems/prob1.html):

>Dearest Blaise,
>
>As to the problem of how to divide the 100 Francs, I think I have found a solution that you will find to be fair. Seeing as I needed only two points to win the game, and you needed 3, I think we can establish that after four more tosses of the coin, the game would have been over. For, in those four tosses, if you did not get the necessary 3 points for your victory, this would imply that I had in fact gained the necessary 2 points for my victory. In a similar manner, if I had not achieved the necessary 2 points for my victory, this would imply that you had in fact achieved at least 3 points and had therefore won the game. Thus, I believe the following list of possible endings to the game is exhaustive. I have denoted 'heads' by an 'h', and tails by a 't.' I have starred the outcomes that indicate a win for myself.
>
>       h h h h *       h h h t *       h h t h *       h h t t *
>       h t h h *       h t h t *       h t t h *       h t t t
>       t h h h *       t h h t *       t h t h *       t h t t
>       t t h h *       t t h t         t t t h         t t t t
>
>I think you will agree that all of these outcomes are equally likely. Thus I believe that we should divide the stakes by the ration 11:5 in my favor, that is, I should receive $\frac{11}{16}\cdot100$ = 68.75 Francs, while you should receive 31.25 Francs.
>
>
>I hope all is well in Paris,
>
>Your friend and colleague,
>
>Pierre

Pascal agreed with this solution, and [replied](http://mathforum.org/isaac/problems/prob2.html) with a generalization that made use of his previous invention, Pascal's Triangle. There's even [a book](https://smile.amazon.com/Unfinished-Game-Pascal-Fermat-Seventeenth-Century/dp/0465018963?sa-no-redirect=1) about it.

## A New Condition

It's an obvious point that the probability of either man winning depends on the results of the coin flips so far. One might like to know not just what, say, Fermat's chances of winning are at this point but also what his chances of  winning would be **given that the next flip results in a 'heads'**. This is asking about a ***conditional probability***.

The probability that Fermat wins the game may be 68.75%, but what would it be if the next flip is 'heads'?

Let's define...
* Event $F$ as Fermat winning the game.
* Event $H$ as the next flip turning up heads

We can write $P(F | H)$ as the **conditional probability** that Fermat wins _given_ that the next flip is heads.

If the next flip is heads, this would make the score 9H - 7T, and so the remaining possible outcomes would then be:

- H (50% chance; Fermat wins 10H-7T)
- TH (25% chance; Fermat  wins 10H-8T)
- TTH (12.5% chance; Fermat wins 10H-9T)
- TTT (12.5% chance; Fermat loses 9H-10T)

So the probability that Fermat wins, **given that the next flip is 'heads'**, $P(F | H)$, is 87.5%.

The probability of Fermat winning has increased, which is to say that the events of (a) the next flip being 'heads' and (b) Fermat winning the game are **not independent**: (a) makes (b) more likely.

## Conditional Probability Rules

When events _are_ independent, the rule for probabilistic AND (I'll use '$\cap$' below) is simple:

$$\huge P(a\cap b) = P(a) P(b)$$

But the more general rule, which includes non-independent events, is:

$$\huge P(a\cap b) = P(a | b) P(b)$$.

In fact, this is the definition of conditional probability. Rearranging:

$$\huge P(a | b) = \frac{P(a\cap b)}{P(b)}$$

> Note that, when $a$ and $b$ are independent, then we have:
>
> $$\begin{align}
    P(a | b) &= \frac{P(a\cap b)}{P(b)} \\
             &= \frac{P(a) P(b)}{P(b)} \\
             &= P(a)
\end{align}$$

## Conditional Probability Example

Let's see how we can use these definitions to solve a conditional probability problem.

Suppose we have the following probabilities:

- The probability of rain tomorrow is 0.8;
- The probability that I take my umbrella tomorrow, given that it rains, is 0.9.

What is the probability that *both* (a) it rains tomorrow and (b) I take my umbrella?

```python
prob_rain = 0.8                           # P(r)
prob_umb_if_rain = 0.9                    # P(u | r)

prob_rain_and_umb = 0.8 * 0.9             # P(r AND u)
prob_rain_and_umb
```

> **NOTE**
>
> This is approaches something called ***Bayes's Theorem***. There's a bit more in [this _Level Up_ section](#Level-Up:-Bayes-Theorem). We'll also see more of this in a future lecture.

# The Law of Total Probability

Sometimes we want to calculate an **unconditional** probability by making use of some conditional probabilities.

Suppose we want to calculate the probability of $e$, the event that Two-Face will free his hostages.

Now like any good cartoon villain, Two-Face has the ridiculous quirk that he makes his decisions partially dictated from his coin's flip.

Since Two-Face is a pretty bad dude, he still does the bad thing even if his coin lands "heads". So let's suppose we express this like below:

- $P(e | H) = 0.9$
- $P(e | T) = 0.1$

where $H$ is the event of a particular coin toss coming up "heads" and $T$ is the event of that same coin toss coming up "tails".

Suppose we also know that the coin is weighted unfairly (Two-Face is one bad dude), in such a way that the coin has a $40\%$ chance of coming up "heads" and a $60\%$ chance of coming up "tails":

- $P(H) = 0.4$
- $P(T) = 0. 6$

We can now solve the original question of the probability that Two-Face will free his hostages.

What we really want to know is $P(e)$, the probability that Two-Face will free his hostages. So we calculate:

$$\begin{align}
P(e) &= P(e | H) P(H) + P(e | T) P(T) \\
     &= (0.9)(0.4) + (0.1)(0.6)  \\
     &= 0.36 + 0.06 \\
     &= 0.42
\end{align}$$

Not great; we may need Batman to step in and alter the conditional probabilities

## Aside: Mathematical Notation and Formalism

Notice that $H$ and $T$ together have an interesting relationship. In particular:

- $P(H\cap T) = 0$, i.e. the two events are mutually *exclusive*; and
- $P(H\cup T) = 1$, i.e. the two events are jointly *exhaustive* of all possibilities.

Any set of two or more events with these two features (where, strictly, for no event $z$ do we have $P(z) = 0$) is said to **partition** the space of possibilities. In our case, we have $\Omega = \{H, T\}$.

For an arbitrary partition, $\Omega = \{\xi_1, ..., \xi_n\}$, the ***law of total probability*** tells us the following:

$P(e) = P(e\cap\xi_1) + ... + P(e\cap\xi_n) = P(e | \xi_1)\times P(\xi_1) + ... + P(e | \xi_n)\times P(\xi_n)$.

# Level Up: More Set Theory

## A Note on Set Notation

We usually denote sets with curly braces $\large \{\}$ with elements within them.

For example, the following set $A$ has the numbers $0,3,5,6,7,8,9$ in it:

$$\large \{8,6,7,5,3,0,9\}$$

Sometimes, there are a pattern of elements/values in a set and it'd be tough to write them down. So we use a special notation to denote a set $S$:

$$\large S = \{x \mid x \in T, \text{other conditions} \}$$

where:
- $x$ is an element in $S$
- $|$ is read as "such that" and denotes that $x$ must fulfill the following conditions
- $x\in T$ means $x$ is from some other defined set $T$
- Followed by other conditions on the elements ($x$) of $S$

### Quick Notation Example

If we defined a set to include all numbers less than $3$, it'd be tough to write down all the those numbers. This would not only include integers such as $2,1,0,-1,\cdots)$ (denoted with $\mathbb{Z}$) but also numbers like $\pi, \sqrt{2}, 1.5,  \frac{1}{3},$ and _infinitely_ more numbers. Turns out these are [_real numbers_]() (denoted with $\mathbb{R}$).

So to save on paper, we might instead write:

$$\large \{x \mid x \in \mathbb{R}, x \lt 6 \}$$

### 🧠 Knowledge Check

What numbers would be included in this set?

$$\large \{x \mid x \in \mathbb{Z}, 0 \leq x \leq 20 \}$$

## Empty Set

The **empty set**, written as $\emptyset$ or { }, has no elements.

Weirdly enough, a set could include $\emptyset$:

$$\large \{\emptyset, 1, 2\}$$

The above set would have ***three*** elements in it

> In set theory, turns out having "nothing" is still something.

## Subsets

Set $T$ is a **subset** of set $S$ if every element in set $T$ is also in set $S$. The mathematical notation for a subset is $T \subset S$.

> **NOTE**
>
> $\emptyset$ is a subset of every set. (_Every set has the set of nothing_)

## Joint Probability

Sometimes we want to know the probability of two different events occurring - we refer to this as **joint probability**.

The probability that *either* $A$ *or* $B$ happens is:

$$P(A  \cup  B) = P(A) + P(B) - P(A  \cap  B)$$

## Python & Sets

Python has a [`set` type](https://docs.python.org/3/library/stdtypes.html#set) and can be useful since they follow the mathematical rules of sets!

In Python, if I want to make a set, I can use the base function `set()`, or I can use '{' and '}'.

```python
a = set([1, 2])
```

```python
b = {1, 2}
```

```python
a == b
```

```python
a1 = {2, 1}

a1 == a
```

### Empty Set

The **empty set**, written as $\emptyset$ or { }, has no elements. Just remember that in Python, `{}` will initiate an empty _dictionary_, not a set.

```python
type({})
```

```python
m = set()
```

```python
type(m)
```

```python
print(m)
```

### Iterables

```python
a = set(['greg'])
```

```python
b = {'greg'}
```

```python
c = set('greg')
```

```python
a == b
```

```python
a == c
```

```python
a
```

```python
c
```

### Subsets

```python
c
```

```python
d = set(['g', 'r'])
```

```python
d.issubset(c)
```

```python
m = set()
m.issubset(c)
```

### Set Operations - Example

We are trying to create rooming arrangements based on staff interest for a staff trip.

Who should room with whom based on shared interests?

```python
Robin = {"art", "traveling", "wine", "doodling", "tech", "gadgets"}
Rob = {"rock-climbing", "traveling", "dad jokes", "ice cream"}
Alison = {"wine", "traveling", "Schitts Creek", "dogs"}
Su = {"Schitts Creek", "dogs", "tarot card reading", "croquet", "taxonomy"}
Molly = {"wine", "ice cream", "dogs", "zookeeping", "traveling"}
```

```python
Robin.intersection(Alison)
```

```python
Rob.intersection(Alison)
```

**Prompt**: What would the union of the sets `Rob` and `Alison` tell us?

```python
Rob.union(Alison)
```

# Level Up: Bayes Theorem

Intuitively, we expect that $P(a\cap b) = P(b\cap a)$, so we could equally well write:

- $P(a\cap b) = P(b | a)\times P(a)$.

This equivalence leads to an important theorem of conditional probability:

$P(a | b)\times P(b) = P(b | a)\times P(a)$

Thus:

$\huge P(a | b) = \frac{P(a)\times P(b | a)}{P(b)}$.

This is **Bayes's Theorem**. We'll have more to say about this in a later lesson.

# Level Up: Permutations & Combinations

Permutations and combinations are helpful when we're trying to solve certain counting problems. Formulas help us calculate the total number of possible outcomes in various scenarios.

### Permutations

Use these when ordering matters: How many different arrangements can you get out of a number of elements?

The number of arrangements of $r$ elements out of a total of $n$ elements is given by: $\Large\frac{n!}{(n – r)!}$.

Why is this the right formula?

We have:
- $n$ choices for the item in the first position;
- $n-1$ choices for the item in the second position (since we've reserved one of the original items for the first position);
- ... ;
- and $n-(r-1)=n-r+1$ choices for the item in the last position.

So that's $n!$ without the terms below $n-r+1$, i.e. $\frac{n!}{(n-r)!}$.

```python
from itertools import permutations
l = list(permutations(range(1, 4)))
print(l)
```

#### Easiest Scenario

How many ways are there of arranging the letters in `DAMICO`?

```python
6*5*4*3*2*1
```

```python
from math import factorial

factorial(6)
```

#### Intermediate Scenario

How many ways are there of arranging the letters in `GREG`?

eggr, egrg, ergg
gegr, gerg, gger, ggre, greg, grge
regg, rgeg, rgge

#### Level Up Scenario

You are trying to break the code - to hack into the mainframe, and stop the KGB from launching US missiles remotely.

You know the password is some 5 letter anagram of a subset of the word `pochemuchka`

How many potential passwords are there? i.e., how large is the **set** of password options?

```python
from itertools import permutations
```

```python
l = list(permutations("pochemuchka", 5))
```

```python
len(l)
```

```python
11 * 10 * 9 * 8 * 7
```

```python
len(set(l))
```

### Combinations

Ordering does not matter.

How many different selections can you get out of a number of elements?

The number of selections of $r$ out of a total of $n$ elements is given by: $\Large{n\choose r} = \frac{n!}{r!(n – r)!}$.

Why is this the right formula?

We have:

The count is related to the number of associated permutations of $r$ elements, choosing from a total of $n$, namely $\frac{n!}{(n-r)!}$.

But since order doesn't matter, we need to divide our count by the number of ways of arranging the $r$ objects we've chosen. And we know what that number is: It's simply $r!$.

So our total count will be $\frac{n!}{(n-r)!r!}$.

#### Easier Scenario:

How many ways are there of choosing two students from a group of six?

```python
factorial(6) / (factorial(2) * factorial(4))
```

#### Level Up Scenario

What are the chances of getting exactly 3 "heads" on 5 flips?

$\rightarrow$ We'll look at the binomial distribution soon!

```python
from count_class import Counter
```

```python
c2 = Counter(arr='david')
c2.calc(3)
```

```python
c2.final_list
```

# Level Up: Three Events

What is the probability of having at least one of three events?

$ P(A [or] B [or] C) = P(A) + P(B) + P(C) - P(A [and] B) - P(A [and] C) - P(B [and] C) + P(A [and] B [and] C) $


# SVR

```python
#SUPPORT VECTOR REGRESSION.
#Those who are in Machine Learning or Data Science are quite familiar with the term SVM or Support Vector Machine.
#But SVR is a bit different from SVM.
#SVR is an regression algorithm , so we can use SVR for working with continuous Values instead of Classification which is SVM.
```

SVR por sus siglas en inglés provenientes de Support Vector Regression es un algoritmo de regresión basado en las máquinas de soporte vectorial utilizados para clasificar elementos de diferentes conjuntos. En la siguiente sección se explicara cómo funciona el algoritmo SVR con datos lineales, pero es importante saber que este algoritmo también funciona para datos no lineales.

### Funcionamiento de una SVR

Imaginemos que existe un conjunto de datos como el que se presenta a continuación, donde se puede observar que los datos presentan un comportamiento similar a un modelo lineal.

El primer paso que debemos realizar es encontrar el modelo lineal que mejor describa el comportamiento de este conjunto de datos, en el contexto de las SVR a nuestra línea la llamaremos Hiperplano. En el contexto de problemas donde los datos no son lineales el hiperplano que mejor representa el comportamiento de los datos será más complejo que una sencilla línea, pero la idea central seguirá siendo la misma.

Continuando con nuestro ejemplo la recta que encontremos tendrá una ecuación de la forma:

\begin{equation}
y = m x + b
\end{equation}

Nuestro siguiente paso consiste en encontrar dos bandas paralelas al hiperplano donde es necesario que entre ambas cubran la mayor cantidad de datos. Estas bandas son conocidas como vectores de apoyo o de soporte.

Continuando con nuestro ejemplo las bandas que paralelas a nuestro hiperplano tendrán las siguientes ecuaciones:

\begin{equation}
y = m x + b + \epsilon \\\
y = m x + b - \epsilon
\end{equation}

En nuestra figura es posible observar cómo las bandas propuestas no cubren en la totalidad a los datos y existen dos datos que están fuera. Estos datos son conocidos como errores y son datos que el algoritmo tomará en cuenta para mejorar su desempeño. En nuestro ejemplo donde se presentan datos lineales se utiliza la siguiente formula que controla el comportamiento de la SVR:

En nuestra formula se relacionan muchas variables diferentes, w es la magnitud del hiperplano, C es una constante (Que debe ser mayor a cero) que se encarga de controlar el equilibrio entre la regularidad de la función y cuanto toleraremos las desviaciones fuera de las bandas de soporte y las variables Xi las cuales controlan el error cometido por la función de regresión al aproximar las bandas.

```python
from sklearn.datasets import load_boston

boston_dataset = load_boston()
import pandas as pd
df = pd.DataFrame(boston_dataset.data,columns=boston_dataset.feature_names)
# Agregamos la variable de respuesta
df['MEDV'] = boston_dataset.target[df.index]
df.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      MEDV

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

```python
df.corr()["MEDV"].sort_values()
#para ver que caracteristicas estan mas y menos relacionadas. Buscamos las mas cercanas a +1 y -1
#veremos que LSTAT (indice de pobreza del barrio) es la variable mas relacionada con el precio de la vivienda
```

    LSTAT     -0.737663
    PTRATIO   -0.507787
    INDUS     -0.483725
    TAX       -0.468536
    NOX       -0.427321
    CRIM      -0.388305
    RAD       -0.381626
    AGE       -0.376955
    CHAS       0.175260
    DIS        0.249929
    B          0.333461
    ZN         0.360445
    RM         0.695360
    MEDV       1.000000
    Name: MEDV, dtype: float64

```python
# normalizamos datos para facilitar operacion
X = df["LSTAT"].values.reshape(-1, 1)
y = df["MEDV"].values.reshape(-1, 1)
```

```python
from sklearn.preprocessing import MinMaxScaler
X_scaler = MinMaxScaler()
y_scaler = MinMaxScaler()
X = X_scaler.fit_transform(X)
y = y_scaler.fit_transform(y)
```

```python
# Creamos un split de datos
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=100)
```

## Documentación:
- https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html#sklearn.svm.SVR

## Tips de uso:
- https://scikit-learn.org/stable/modules/svm.html#tips-on-practical-use

```python
from sklearn.svm import SVR
#creamos 3 modelos cada uno con un kernel
svr_rbf = SVR(kernel='rbf', C=100, gamma=0.1, epsilon=.1)
svr_lin = SVR(kernel='linear', C=100, gamma='auto')
svr_poly = SVR(kernel='poly', C=100, gamma='auto', degree=3, epsilon=.1, coef0=1)
```

```python
# Creando el modelo y entrenando
reg = svr_rbf.fit(X_train,y_train.reshape(-1))
# Prediciendo valores de entrenamiento
y_train_hat = svr_rbf.predict(X_train)
# Prediciendo valores de validación
y_test_hat = svr_rbf.predict(X_test)
```

```python
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline # Le decimos a jupyter que grafique en  el cuaderno

plt.scatter(X_train, y_train )# Creamos un scatter plot con los datos de entrenamiento
plt.scatter(X_test, y_test) # Creamos un scatter plot con los datos de validación

X_plot = np.linspace(0,1).reshape(-1, 1) #En X_plot guardamos valores distribuidos entre 0 y 1
# X_plot_poly = poly.fit_transform(X_plot)
y_plot = svr_rbf.predict(X_plot)

plt.plot(X_plot, y_plot,"r--"); #el ; final es para que no muestre el mensaje de output .....
```

```python
from sklearn.metrics import r2_score
# Calculamos el error
print("Entrenamiento", r2_score(y_train, y_train_hat))
print("Prueba", r2_score(y_test, y_test_hat))
```

    Entrenamiento 0.6549670658597944
    Prueba 0.5959466433326677

```python
# Creando el modelo y entrenando
reg = svr_lin.fit(X_train,y_train.reshape(-1))
# Prediciendo valores de entrenamiento
y_train_hat = svr_lin.predict(X_train)
# Prediciendo valores de validación
y_test_hat = svr_lin.predict(X_test)
```

```python
# Creamos un scatter plot con los datos de entrenamiento
plt.scatter(X_train, y_train)
# Creamos un scatter plot con los datos de validación
plt.scatter(X_test, y_test)
# En X_plot guardamos valores distribuidos entre 0 y 40
X_plot = np.linspace(0,1).reshape(-1, 1)
# Con el modelo predecimos X_plot
# X_plot_poly = poly.fit_transform(X_plot)
y_plot = svr_lin.predict(X_plot)
# Graficamos el modelo
plt.plot(X_plot, y_plot,"r--")

# Calculamos el error
print("Entrenamiento", r2_score(y_train, y_train_hat))
print("Prueba", r2_score(y_test, y_test_hat))
```

```python
# Creando el modelo y entrenando
svr_poly.fit(X_train,y_train.reshape(-1))
# Prediciendo valores de entrenamiento
y_train_hat = svr_poly.predict(X_train)
# Prediciendo valores de validación
y_test_hat = svr_poly.predict(X_test)
```

```python
# Creamos un scatter plot con los datos de entrenamiento
plt.scatter(X_train, y_train)
# Creamos un scatter plot con los datos de validación
plt.scatter(X_test, y_test)
# En X_plot guardamos valores distribuidos entre 0 y 40
X_plot = np.linspace(0,1).reshape(-1, 1)
# Con el modelo predecimos X_plot
# X_plot_poly = poly.fit_transform(X_plot)
y_plot = svr_poly.predict(X_plot)
# Graficamos el modelo
plt.plot(X_plot, y_plot,"r--")

# Calculamos el error
print("Entrenamiento", r2_score(y_train, y_train_hat))
print("Prueba", r2_score(y_test, y_test_hat))
```

```python
from sklearn.svm import SVR

svr_rbf = SVR(kernel='rbf', C=100, gamma=0.1, epsilon=.1)
svr_lin = SVR(kernel='linear', C=100, gamma='auto')
svr_poly = SVR(kernel='poly', C=100, gamma='auto', degree=3, epsilon=.1, coef0=1)

# Creamos un scatter plot con los datos de entrenamiento
plt.scatter(X_train, y_train,label="Training",alpha=0.3)
# Creamos un scatter plot con los datos de validación
plt.scatter(X_test, y_test, label="Testing",alpha=0.3)
# En X_plot guardamos valores distribuidos entre 0 y 1
X_plot = np.linspace(0,1).reshape(-1, 1)

for name, svr, s in zip(["RBF","Linear","Poly"],[svr_rbf,svr_lin,svr_poly],["r--","g--","b--"]):
    svr.fit(X_train,y_train.reshape(-1))
    y_train_hat = svr.predict(X_train)
    y_test_hat = svr.predict(X_test)
    # Con el modelo predecimos X_plot
    y_plot = svr.predict(X_plot)
    # Graficamos el modelo
    plt.plot(X_plot, y_plot, s,label=name)
    # Calculamos el error
    print(name, "Entrenamiento", r2_score(y_train, y_train_hat))
    print(name, "Prueba", r2_score(y_test, y_test_hat))

plt.legend()
```

```python
# Preparamos datos
X = df["LSTAT"].values.reshape(-1, 1)
y = df["MEDV"].values.reshape(-1, 1)

from sklearn.preprocessing import MinMaxScaler
X_scaler = MinMaxScaler()
y_scaler = MinMaxScaler()
X = X_scaler.fit_transform(X)
y = y_scaler.fit_transform(y)

# Creamos un split de datos
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=100)

from sklearn.svm import SVR
plt.style.use("seaborn")
plt.figure(figsize=(6,6))

svr_rbf = SVR(kernel='rbf', C=100, gamma=0.1, epsilon=.1)
svr_lin = SVR(kernel='linear', C=100, gamma='auto')
svr_poly = SVR(kernel='poly', C=100, gamma='auto', degree=3, epsilon=.1, coef0=1)

# Creamos un scatter plot con los datos de entrenamiento
plt.scatter(X_scaler.inverse_transform(X_train), y_scaler.inverse_transform(y_train.reshape(-1,1)),label="Training",alpha=0.5)
# Creamos un scatter plot con los datos de validación
plt.scatter(X_scaler.inverse_transform(X_test), y_scaler.inverse_transform(y_test.reshape(-1,1)),label="Testing",alpha=0.5)
# En X_plot guardamos valores distribuidos entre 0 y 1
X_plot = np.linspace(0,1).reshape(-1, 1)

for name, svr, s in zip(["RBF","Linear","Poly"],[svr_rbf,svr_lin,svr_poly],["r--","g--","b--"]):
    svr.fit(X_train,y_train.reshape(-1))
    y_train_hat = svr.predict(X_train)
    y_test_hat = svr.predict(X_test)
    # Con el modelo predecimos X_plot
    y_plot = svr.predict(X_plot)
    # Graficamos el modelo
    plt.plot(X_scaler.inverse_transform(X_plot), y_scaler.inverse_transform(y_plot.reshape(-1,1)), s,label=name)
    # Calculamos el error
    print(name, "Entrenamiento", r2_score(y_train, y_train_hat))
    print(name, "Prueba", r2_score(y_test, y_test_hat))
    print("----")

plt.legend(); #el ; final es para que no muestre el mensaje de output .....
```

```python

```