# Grupo: A1

Brian Luís Coimbra Maia

Gustavo Morais Rodrigues

Matheus dos Reis Casarim

Mathews Edwirds Gomes Almeida

# Responsável pela entrega:
Brian Luís Coimbra Maia

# Data de formação da equipe:
01/07/21

# Condições Gerais

Primeiro trabalho de Introdução a Modelagem Computacional

As condicões abaixo devem ser observadas:

1. Serão formadas equipes e cada uma delas com até e 4 integrantes.

2. A avaliação será realizada por meio da entrega de um vídeo explicativo e de
códigos desenvolvidos até a data estipulada.

3. Os documentos necessários para a entrega do trabalho são (1) um vídeo com
a descricão da solução e (2) os códigos desenvolvidos pela equipe. Os vídeos podem ser separados por questão, porém o tempo somado de todos os vídeos não deve ultrapassar 10 minutos. A equipe deve usar este modelo de notebook para desenvolver os códigos.

4. Um dos integrantes será indentificado como responsável pela entrega do documentos necessários.

5. A distribuição da pontuacão referente a cada ítem aparece no enunciado de
cada proposicão.

6. As equipes devem ser formadas até às **18 horas o dia 09/07/2021** por meio do preenchimento da planilha [[MAC024] Composição das Equipes](https://docs.google.com/spreadsheets/d/1-7Wdno82aFGgCfY0z8oU5SCbwnbN_JV_LqB_ORGKHZQ/edit?usp=sharing).

7. A formação das equipes pode ser acompanhada arquivo [[MAC024] Composição das Equipes](https://docs.google.com/spreadsheets/d/1-7Wdno82aFGgCfY0z8oU5SCbwnbN_JV_LqB_ORGKHZQ/edit?usp=sharing). Cada equipe será indentificada por uma letra em ordem alfabética seguida do número 1 (A1, B1, C1, e assim por diante). O arquivo está aberto para edição e pode ser alterado pelos alunos até a data estipulada.

8. Equipes formadas após a data estabelecida para a formação das equipes terão a nota da avaliação multiplicada por um coeficiente de **0.80**.

9. A equipe depe indicar no arquivo [[MAC024] Composição das Equipes](https://docs.google.com/spreadsheets/d/1-7Wdno82aFGgCfY0z8oU5SCbwnbN_JV_LqB_ORGKHZQ/edit?usp=sharing) um responsável pela entrega do projeto.   Qualquer outro integrante da equipe não poderá entregar o trabalho exceto em casos de força maior.

10. A entrega dos projetos (videos explicativos e códigos) deve ocorrer até às **23:59 do dia 16/07/2021** na plataforma da disciplina pelo responsável pela entrega.

10. Quaisquer dúvidas ou esclarecimentos devem ser encaminhadas pela sala de aula virtual.

# Evolução de um sistema dinâmico discreto (25 pontos)

Evolua o sistema dinâmico abaixo por 300 passos. Pede-se o seguinte:

- apresente um gráfico com a evolução das variáveis $x_i(t)$ e $y_i(t)$ ao longo dos passos de tempo simulados e
-  um gráfico $x\times y$ com os pontos resultantes da simulação realizada.

$$
        \left\{\begin{array}{ccl} x_{i+1} &=&\alpha \left(x_{i} \cos{\left (0.4 - \frac{0.6}{x_{i}^{2} + y_{i}^{2} + 1} \right )} - y_{i} \sin{\left (0.4 - \frac{0.6}{x_{i}^{2} + y_{i}^{2} + 1} \right )}\right) + 1\\y_{i+1} &=&\alpha \left(x_{i} \sin{\left (0.4 - \frac{0.6}{x_{i}^{2} + y_{i}^{2} + 1} \right )} + y_{i} \cos{\left (0.4 - \frac{0.6}{x_{i}^{2} + y_{i}^{2} + 1} \right )}\right)\end{array}\right.,
        \alpha=0.918, (x_0,y_0)=(0,0)
$$

```python
# Importando os pacotes necessários
import numpy as np
import matplotlib.pylab as plt

# Definimos as variáveis iniciais
x, y = [], []
x.append(0)     #x0=0
y.append(0)     #y0=0
alfa = 0.918
passos = 300

# Laço que faz a evolução do sistema dinâmico
for i in range(0, passos-1):
  func_aux = np.deg2rad(0.4 - (0.6/(x[i]**2 + y[i]**2 + 1)))
  equacao1 = alfa * (x[i] * np.cos(func_aux) - y[i] * np.sin(func_aux)) + 1
  equacao2 = alfa * (x[i] * np.sin(func_aux) + y[i] * np.cos(func_aux))
  x.append(equacao1)
  y.append(equacao2)

# Função responsável por plotar o grafico evolutivo com valores de x e y
def plot_results(x, y, label1, label2, labelX, labelY, title):
    intervalo = range(0, passos)

    plt.figure(figsize = (8, 6))
    plt.plot(x,'-', label=label1)
    plt.plot(y,'-', label=label2)
    plt.xlim(np.amin(intervalo), np.amax(intervalo))
    plt.ylim(np.amin(x) -2, np.amax(x) + 2)

    # Personalização do gráfico (label, fontes, titulos e subtitulos)
    plt.title(title, fontsize = 15, pad = 20)
    plt.xlabel(labelX, labelpad = 5, fontsize = 12)
    plt.xticks(fontsize = 12)
    plt.ylabel(labelY, labelpad = 2, fontsize = 12)
    plt.yticks(fontsize = 12)
    plt.legend(loc = "best", fontsize = 12, frameon = True)
    plt.grid(True)

    plt.show()

# Função responsável por plotar o grafico x*y
def plot_results2(x, y, label1, label2, labelX, labelY, title):

    plt.figure(figsize = (8, 6))
    #plt.plot(x, y,'-', label=label2)
    plt.plot(y, x,'-', label=label1)
    plt.xlim(min(y), max(y))
    plt.ylim(min(x), max(x))

    # Personalização do gráfico (label, fontes, titulos e subtitulos)
    plt.title(title, fontsize = 15, pad = 20)
    plt.xlabel(labelX, labelpad = 5, fontsize = 12)
    plt.xticks(fontsize = 12)
    plt.ylabel(labelY, labelpad = 2, fontsize = 12)
    plt.yticks(fontsize = 12)
    plt.legend(loc = "best", fontsize = 12, frameon = True)
    plt.grid(True)

    plt.show()

# Plotamos os resultados
plot_results(x, y, "x", "y", "Passos", "Evolução", "Gráfico da evolução de x e y em 300 passos")
plot_results2(x, y, "X", "Y", "Y", "X", "Gráfico X × Y")
```

#  Modelo Presa-Predador (25 pontos)

Em 1868, a introdução acidental nos Estados Unidos do inseto de algodão (Icerya purchasi) da Austrália ameaçou destruir a indústria cítrica americana. Para remediar esta situação, um predador australiano natural,  besouro-joaninha (Novius cardinalis), foi importado daquele país. Os besouros mantiveram os insetos em um nível relativamente baixo. Quando o DDT (um inseticida) foi descoberto para matar insetos, os agricultores aplicaram-no na esperança de reduzir ainda mais a população de insetos de algodão. No entanto, o DDT também se mostrou fatal para o besouro, e o efeito geral do uso do inseticida foi aumentar o número dos insetos de algodão.

Seja  $C_n$ e $B_n$ os níveis de população de insetos de algodão e besouros-joaninha, respectivamente, após $n$ dias. O modelo pode ser escrito da forma
$$
\left\{
  \begin{matrix}
  C_{n+1} &=& C_{n}+  k_{0}C_{n}  -  k_{1}B_{n} C_{n}    \\
  B_{n+1} &=& B_{n}-  k_{2}B_{n}  + k_{3}B_{n} C_{n}
  \end{matrix}
  \right.
$$
onde $k_i$ são constantes positivas.

- Determine os pontos fixos do sistema  predador-presa. Considere $$(k_0, k_1, k_2, k_3) = (0.6, 0.004, 0.1, 0.001)$$ como parâmetros do modelo.
- Modifique o modelo para refletir um sistema predador-presa em que os agricultores aplicam (com regularidade) um inseticida que destrói tanto o predador do inseto quanto a presa do inseto a uma taxa proporcional aos números presentes. Escolha adequadamente os novos parâmetros do modelo, se assim for necessário.
- Determine os pontos fixos do sistema  predador-presa em que os agricultores aplicam inseticida.

```python
# Importando as bibliotecas
import numpy as np
import matplotlib.pylab as plt
import sympy as sp
sp.init_printing()

# Inicialização das variáveis
k_0, k_1, k_2, k_3 = sp.var('k_0 k_1 k_2 k_3')
B, C = sp.var('B C')
E = sp.var('E')
v = sp.Matrix([C,B])
fixos = {'k_0':0.6, 'k_1':0.004, 'k_2':0.1, 'k_3':0.001 }

# Funções sem a adição de pesticida
F = sp.Matrix([
     [(C + (k_0*C) - (k_1*B*C))],
     [(B - (k_2*B) + (k_3*B*C))],
    ])

print('1ª Parte')
solucao_1 = sp.solve(F - v,v,dict=True) # Determinação dos pontos fixos
for s in solucao_1:
    display(s)
aux = sp.solve(F.subs(fixos) - v.subs(fixos), v, dict=True)
print('\n Pontos Fixos, substituindo o valor de K:')
for s in aux:
    display(s)

# Funções com a adição de pesticida
F_P = sp.Matrix([
     [(C + (k_0*C) - (k_1*B*C))*((100 - E)/100)],
     [(B - (k_2*B) + (k_3*B*C))*((100 - E)/100)],
    ])

print('\n2ª Parte')
solucao_2 = sp.solve(F_P - v,v,dict=True) # Determinação dos pontos fixos
for s in solucao_2:
    display(s)
print('\nObs: O valor de E na função tem a seguinte condição para que seja visto como porcentagem. 0 <= E <= 100 & E ∈ R')
aux2 = []
valores_E = []
ranger = 41 # Valor sempre menor que #41#, tambem é o numero de vezes que sera executado o programa com uma progressão da efetividade do pesticida
for i in range(ranger): # Atribuindo valor progressivo de E ate 40%
  valores_E.append({'E': (i/(ranger-1) * 40)})
for i in range(ranger): # Calculo substituindo os valores 'K's e 'E'
  aux2.append(sp.solve(F_P.subs(fixos).subs(valores_E[i]) - v.subs(fixos).subs(valores_E[i]), v, dict=True))
print('\nPontos Fixos, substituindo os valores de K e E:')
for i in range(ranger):
    print(str(aux2[i]) + '  Valor de ' + str(valores_E[i]) + '% de efetividade')
```

# Pontos fixos de um sistema dinâmico discreto (25 pontos)

Considere o  sistema dinâmicoa discreto abaixo.
$$\left\{\begin{matrix} x_{i+1} &=&- \delta x_{i} y_{i} + r_{1} x_{i} + x_{i}\\ y_{i+1} &=&- r_{2} y_{i} + \frac{\theta x_{i} y_{i}^{2}}{B + y_{i}} + x_{i}\end{matrix}\right.
$$
onde
$r_1= 0.5, r_2= 0.1, \delta= 0.01, \theta= 0.01, B= 4, (x_0,y_0)=(20,16).$

-  Apresente os pontos fixos e justifique as suas respectivas classificações quando a estabilidade
- Evolua  o sitema por 1200 iterações e apresente os resultados em um  gráfico  $x \times y$.

A determinação do ponto fixo ocorre quando $x_{i+i} = x_i$

A sugestão é subistituir essa informação na equação acima e usar ou computação numérica ou simbólica para resolver o problema.

Para resolver o sistema dinâmico e encontrar os pontos fixos irei adotar $x_{i+1} = x_i = x_{eq}$ e $y_{i+1} = y_i = y_{eq}$. Dessa forma, teremos o sistema de equações:

$$\left\{\begin{matrix} x_{eq} &=&- \delta x_{eq} y_{eq} + r_{1} x_{eq} + x_{eq}\\ y_{eq} &=&- r_{2} y_{eq} + \frac{\theta x_{eq} y_{eq}^{2}}{B + y_{eq}} + x_{eq}\end{matrix}\right. \\
$$

Simplificando o sistema para que possa ser utilizada no código:

$$\left\{\begin{matrix} x_{eq} y_{eq} + r_{1} x_{eq}  &=& 0\\ r_{2} y_{eq} + \frac{\theta x_{eq} y_{eq}^{2}}{B + y_{eq}} + x_{eq} - y_{eq}   &=& 0 \end{matrix}\right. \\
$$
Com o sistema simplificado, podemos resolvê-lo através do SymPy e obter os pontos fixos, que são $(0,0)$ e $ (41.0126582278481, 50.0000000000000)$.

```python
# Importação das bibliotecas necessárias
import numpy as np
import matplotlib.pylab as plt
from sympy import *
import sympy as sym

# Número de iterações
n = 1200
intervalo = np.arange(0,n)

# Constantes
r1 = 0.5
r2 = 0.2
delta = 0.01
theta = 0.01
B = 4
x0 = 20
y0 = 16

# Resolução do Sistema Para encontrar o Ponto Fixo
x,y = sym.symbols('x_eq,y_eq')
eq1 = sym.Eq(-delta*x*y + r1*x,0)
eq2 = sym.Eq(-r2*y +(theta*x*y**2)/(B+y)+ x - y,0)
print("Sistema de equações:\n")
display(eq1)
display(eq2)
print("\nPontos Fixos:")

pontosFixos = sym.solve([eq1,eq2],(x,y))
display(pontosFixos)
print("\n")

# Análise de Estabilidade
f = -delta*x*y +r1*x + x
g = -r2*y +(theta*x*y**2)/(B+y)+ x

# Derivadas das funções do sistema
fx = diff(f,x)
fy = diff(f,y)
gx = diff(g,x)
gy = diff(g,y)
# display(fx)
# display(fy)
# display(gx)
# display(gy)

# Matriz Jacobiana
J = Matrix([[fx,fy],[gx,gy]])
# display(J)

# Calculo do traço da Matriz
trJ = sym.trace(J)
# display(trJ)
trJ = abs(trJ)

# Calculo da determinante da Matriz
detJ = sym.det(J)
# display(detJ)

### Calculo do traço e determinante para cada ponto
# Para o primeiro ponto.
print("Análise do primeiro ponto P = ({},{}):".format(pontosFixos[0][0],pontosFixos[0][1]))
# Traço:
print("\nTraço: ")
traco = trJ.subs([(x,pontosFixos[0][0]),(y,pontosFixos[0][1])])
display(traco)
# Determinante:
print("\nDeterminante: ")
determinante = detJ.subs([(x,pontosFixos[0][0]),(y,pontosFixos[0][1])])
# determinante = detJ.subs(])
display(determinante)
print("\nDeterminante é menor que 1, porém não é maior que o traço menos 1. Dessa forma, o primeiro ponto fixo é instável.")

# Para o segundo ponto.
print("\nAnálise do segundo ponto P = ({},{}):".format(pontosFixos[1][0],pontosFixos[1][1]))
# Traço:
print("\nTraço: ")
traco = trJ.subs([(x,pontosFixos[1][0]),(y,pontosFixos[1][1])])
display(traco)
# Determinante:
print("\nDeterminante: ")
determinante = detJ.subs([(x,pontosFixos[1][0]),(y,pontosFixos[1][1])])
display(determinante)
print("\nDeterminante é menor que 1 e também é maior que o traço menos um. Sendo assim, o segundo ponto fixo é estável.\n")

# Cria vetores solução
x = np.zeros(n)
y = np.zeros(n)

# Inicializa vetores solução
x[0] = x0
y[0] = y0

for i in range(1,n):
  x[i] = -delta*x[i-1]*y[i-1] + r1*x[i-1] + x[i-1]
  y[i] = -r2*y[i-1] + ((theta*x[i-1])*y[i-1]**2)/(B + y[i-1]) + x[i-1]

# Gráfico de X e Y
plt.figure(figsize = (8, 6))
plt.plot(x,y)
plt.title("X vs Y", fontsize = 15, pad = 20)
plt.xlabel("$x$", labelpad = 5, fontsize = 12)
plt.xticks(fontsize = 12)
plt.ylabel("$y$", labelpad = 2, fontsize = 12)
plt.yticks(fontsize = 12)
plt.grid(True)
plt.show()
```

# Propagação de um boato em uma empresa (25 pontos)

Considere a divulgação de um boato através de uma empresa de 1500 funcionários, todos trabalhando num
mesmo edifício. Assuma que a propagação do boato é semelhante à disseminação de
uma doença contagiosa, em que o número de pessoas que escuta o boato
 de cada dia é proporcional ao produto do número que ouviu o
boato anteriormente e o número que não ouviu o boato. Isso é dado por
$$
r_{n+1}  = r_{n} +  kr_{n} (1500-r_{n})
$$
onde $k$ é um parâmetro que depende de quão rápido o boato se espalha e $n$ é o número
de dias. Assuma $k =0.0012$ e suponha que quatro pessoas tenham ouvido inicialmente
o boato. Em quanto tempo todos os 1500 funcionários terão ouvido o boato?

Novamente considere a divulgação de um boato, mas agora assuma uma empresa com 5500 funcionários. Considerando o  modelo apresentado acima, construa um modelo para a empresa com as seguintes taxas de crescimento do boato $k=(0.25, 0.025, 0.0025, 0.00025)$  para determinar o número de pessoas que ouviu o boato após 1 semana.

Com base a simulação do modelo anterior, construa uma proposta de como controlar a taxa de crescimento do boato.

```python
# Desenvolva aqui a sua solução. Use mais células se necessário.

# Importando os pacotes necessários
import numpy as np
import matplotlib.pylab as plt

# Função que faz os cálculos para preencher uma lista de resultados de acordo com o modelo
def propagacao_boato(n_steps, p, a0, k):
  a = np.zeros(n_steps)
  a[0]=a0
  for n in range(0,n_steps-1):
    # Cálculo do valor que será adicionado na lista de resultados
    a[n+1] = a[n] + k*a[n]*(p-a[n])
  return abs(a) # Retorna a lista com valores absolutos

# Função para plotar o gráfico com uma única label
def plota_grafico(lista, n_steps, p, label):
  plt.figure(figsize = (8, 8))
  plt.plot(lista,'o-', label=label)
  plt.xlim(0, n_steps)
  plt.ylim(0, p+500)
  # Personalização do gráfico (label, fontes, titulos e subtitulos)
  plt.axhline(y=p, color='k', linestyle='--', label=p)
  plt.title("Propagação de um boato em uma empresa de "+str(p)+" pessoas", fontsize = 15, pad = 20)
  plt.xlabel("Dias", labelpad = 5, fontsize = 12)
  plt.xticks(fontsize = 12)
  plt.ylabel("Pessoas que ouviram o boato", labelpad = 2, fontsize = 12)
  plt.yticks(fontsize = 12)
  plt.legend(loc = "best", fontsize = 12, frameon = True)
  plt.grid(True)
  plt.show()

# Empresa com 1500 funcionários

# Parâmetros do modelo
k = 0.0012 # Taxa de crescimento do boato
p = 1500 # Número de funcionários
a0 = 4 # Pessoas que ouviram inicialmente o boato
n_steps = 8 # Número de passos do modelo, ou seja, número de dias que serão levados em conta para gerar a lista de resultados

# Usando função propagacao_boato para gerar a lista com os resultados do modelo
resultado = propagacao_boato(n_steps,p, a0, k)
plota_grafico(resultado, n_steps,p,"k = " + str(k))

# Empresa com 5500 funcionários

# Parâmetros do modelo
valores_k = [0.25, 0.025, 0.0025, 0.00025] # Lista com taxas de crescimento do boato
p = 5500
a0 = 4
n_steps = 8

# Plota gráfico com labels diferentes
plt.figure(figsize = (8, 8))
# Adiciona linha pontilhada em x = 5500 para destacar a quantidade de pessoas na empresa
plt.axhline(y=p, color='k', linestyle='--', label=str(p))
# Laço para gerar os resultados com os diferentes valores de k e plotar em um mesmo gráfico
for k in valores_k:
  aux = propagacao_boato(n_steps=n_steps, p=p, a0=a0, k=k)
  plt.plot(aux,'o-', label="k = " + str(k))
  plt.xlim(0, n_steps)
  plt.ylim(0, 12000)
  # Personalização do gráfico (label, fontes, titulos e subtitulos)
  plt.title("Propagação de um boato em uma empresa de "+str(p)+" pessoas", fontsize = 15, pad = 20)
  plt.xlabel("Dias", labelpad = 5, fontsize = 12)
  plt.xticks(fontsize = 12)
  plt.ylabel("Pessoas que ouviram o boato", labelpad = 2, fontsize = 12)
  plt.yticks(fontsize = 12)
  plt.legend(loc = "best", fontsize = 12, frameon = True)
  plt.grid(True)

# Proposta de como controlar a taxa de crescimento do boato

# Parâmetros do modelo
k = 0.000025
p = 5500
a0 = 4
n_steps = 1000

# Plota gráfico com labels diferentes
plt.figure(figsize = (12, 16))
# Adiciona linha pontilhada em com diferentes valores de x para destacar as quantidades de pessoas na empresa
plt.axhline(y=5500, color='k', linestyle='--', label="5500")
plt.axhline(y=3700, color='g', linestyle='--', label="3700")
plt.axhline(y=1900, color='r', linestyle='--', label="1900")
plt.axhline(y=100, color='b', linestyle='--', label="100")
# Laço para gerar o gráfico de empresas com diferentes quantidade de pessoas
for i in range(4):
  aux = propagacao_boato(n_steps=n_steps, p=p, a0=a0, k=k)
  plt.plot(aux,'o-', label="k = " + str(k) + " e p = " + str(p))
  plt.xlim(0, n_steps)
  plt.ylim(0, 7000)
  # Personalização do gráfico (label, fontes, titulos e subtitulos)
  plt.title("Propagação de um boato em uma empresa com diferentes valores de p", fontsize = 15, pad = 20)
  plt.xlabel("Dias", labelpad = 5, fontsize = 12)
  plt.xticks(fontsize = 12)
  plt.ylabel("Pessoas que ouviram o boato", labelpad = 2, fontsize = 12)
  plt.yticks(fontsize = 12)
  plt.legend(loc = "best", fontsize = 12, frameon = True)
  plt.grid(True)
  # Atualiza o valor de p (quantidade de pessoas) a cada iteração
  p = p - 1800
```

*Respostas da questão 4*

**Propagação de um boato em uma empresa de 1500 pessoas**

Observando o primeiro gráfico, notamos que em uma empresa de 1500 pessoas e com uma taxa de crescimento do boato de 0.0012, em pouco mais de 6 dias e 12 horas todos os funcionários da empresa terão ouvido o boato.

**Propagação de um boato em uma empresa de 5500 pessoas**

Observando o segundo gráfico, é possível perceber que as curvas variam enormemente de acordo com o valor da taxa de crescimento do boato. Dessa forma, apenas a curva com taxa de crescimento igual a 0.00025 não permite que todos os funcionários da empresa tenham conhecimento do boato após uma semana. Na verdade, essa taxa de 0.00025 é muito interessante, pois após uma semana, somente cerca de 1500 pessoas teriam conhecimento do boato, o que é muito melhor do que foi visto com as outras taxas.

**Com base a simulação do modelo anterior, construa uma proposta de como controlar a taxa de crescimento do boato.**

Nos dois gráficos iniciais, é possível notar que a taxa de crescimento do boato é um valor fundamental para controlar a propagação do boato. Assim, procuramos uma taxa de crescimento o mais próxima de zero possível. Para isso, o terceiro gráfico mostra uma taxa no valor de 0.000025 e curvas para uma empresa com diferentes quantidades de funcionários. Dessa forma, podemos observar que para a empresa com 1900 funcionários, o boato demora um pouco menos de 1 ano para atingir todas as pessoas e para a empresa com 100 funcionários, o boato não atinge todas as pessoas em menos de 1000 dias. Isso nos indica que separar o funcionários em prédios com pequenas quantidades de pessoas é uma boa alternativa para controlar a propagação do boato. Além disso, um valor próximo de zero para a taxa de crescimento do boato é fundamental para esse controle. Por último, um número inicial de pessoas que ouviram o boato que seja o menor possível também pode atrasar a propagação do boato.


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c) 2014 Tingyu Wang.

# Black-Scholes Equation

In the class, we discussed how to use numerical methods to solve wave propagation problems and heat equation, which both represent a physical phenomenon. In fact, this powerful tool is not only widely applied by engineers and physicists, but also by traders in a stock market and economists working for the government. Economists would build their own financial models in PDE to estimate future economic growth for each year, and PDE solvers can give traders a sketch about the price of a stock or an option.

Black-Scholes equation is one of the most important models concerning the option pricing. It was first published by *Fischer Black* and *Myron Scholes* in 1973. I first read this equation through a book called "*17 equations that changed the world*".

##### What is Black-Scholes Equation?

Before we introduce the equation, let us start with some important terminologies behind the equation. **Options** are used on markets and exchanges. From the definition in Wikipedia, an option is a contract which gives the buyer (the owner) *the right*, but not the obligation, to buy or sell *an underlying asset* at *a specified price* on or before *a specified date*. We call the price determined in the contract "**exercise price** (or **strike price**) ($E$), and call the date "**the expiry time** ($T$)". The underlying asset has its own value, and we denote **the current value of underlying asset** as ($S$). If the asset is a car, then $S$ may be equal to the present value of the car, say 20,000 dollars. If the asset is a stock, then $S$ is equal to the current price of the stock (find it in Nasdaq, Dow or S&P).

We can understand the option as a right or a priviledge, so the option itself should also have a price, "**the price of an option** ($V$)", which depends on the current value of underlying asset ($S$) and time ($t$): $V = V(S,t)$. Notice that there is a "*buy* or *sell*" in the definition of option above. If the owner of the option has the right to buy some shares of Apple's stock at 50 dollars per share from the buyer before Feb 1, 2015, then this type of option is called **call option**. We use $C(S,t)$ instead of $V(S,t)$. If the owner of option has the right to "sell", then this type of option is called **put option**. We denote it as $P(S,t)$ instead of $V(S,t)$. Click the short video below for real-world examples of call options and put options.

```
from IPython.display import YouTubeVideo
YouTubeVideo('EfmTWu2yn5Q')
```

Finally, we introduce the *Black-Scholes* equation as follows.

$$\begin{equation}
\frac{\partial V}{\partial t}(S, t) + \frac{1}{2} \sigma^2 S^2 \frac{\partial^2 V}{\partial S^2}(S, t) + rS \frac{\partial V}{\partial S}(S, t) - rV(S, t) = 0
\end{equation}$$

$V$ stands for the price of the option, $S$ denotes current value of underlying assets, $t$ is time, $r$ denotes the no-risk interest rate (*ex*.saving rate in bank), and $\sigma$ denotes the volatility of underlying asset (*ex*.the standard deviation of the stock's returns).

As we can see, the Black-Scholes equation is aimed at calculating the price of an option, so it is the most popular option-pricing method. The only unknown here is the price of option ($V$), depending on $S$ and $t$. In the following part, we will use finite difference methods (both explicit and implicit schemes) to solve this nonlinear second-order parabolic equation.

##### Why we use it?

Buyers and sellers use financial derivatives (*derivative here means a contract of some underlying assets*) to **offset risk** in their portfolios. The key idea behind the model is to hedge the option by buying and selling the underlying asset in just the right way and, as a consequence, to eliminate risk. This type of hedging is the basis of more complicated hedging strategies such as those engaged in by investment banks and hedge funds. If investors want to trade their financial derivatives, this equation gives a view of **the pricing strategy**.

##### Finite difference methods

Now let's solve a practical problem using our numerical skills!

Consider an European call style option is made for a security currently trading at  40 dollar per share with volatility 0.4. The term is 3 months (0.25 year) and the strike price is 10 dollar. The prevailing no-risk interest rate is 10 %. What should the price per share be for the option?

The time interval $[0,T]$ is divided into $N$ equally sized subintervals of length $\Delta t$. The price of the underlying asset will take values in the unbounded interval $[0,\infty)$. However, an artifical limit $S_{max}$ is introduced. The interval $[0, S_{max}]$ is divided into $M$ equally sized subintervals of length $\Delta S$.

```
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

T = 0.25       # expiration time
N = 2000
dt = T/N
S_max = 40     # price of the underlying asset
M = 200
s = np.linspace(0, S_max, M+1)

r = 0.1        # no-risk interest rate
sigma = 0.4    # volatility of underlying asset
E = 10.        # the exercise price
```

In the following code, we are calculating an European call option. "*European*" means options can only be exercised at time $t=T$. So we have the final condition such as $C(S,T) = C_T(S)$. However, in a numerical code, we prefer a initial condition rather than final condition. So we apply the change of variable: $t_{new} = T -t$. Thus, the new equation we are solving should look like this (note the sign changes):

$$\begin{equation}
\frac{\partial C}{\partial t}(S, t) - \frac{1}{2} \sigma^2 S^2 \frac{\partial^2 C}{\partial S^2}(S, t) - rS \frac{\partial C}{\partial S}(S, t) + rC(S, t) = 0
\end{equation}$$

For the explicit method, we use a forward difference scheme in time $t$, and a central difference scheme in value of underlying asset $S$. From the discretized form of equation, we can rearrange the equation as following:

$$\begin{equation}
C_i^{n+1} = \frac{1}{2}(\sigma^2 i^2 \Delta t-ri\Delta t)C_{i-1}^n + (1 - \sigma^2 i^2 \Delta t -r \Delta t) C_i^n + \frac{1}{2}(\sigma^2 i^2 \Delta t+ri\Delta t)C_{i+1}^n
\end{equation}$$

We can find that the price of option at $(n+1)$ time step $C_i^{n+1}$ does not depend on $\Delta S$. And since there is $i$ in the equation, we can no longer use vectorized stencil operation instead of double loops.

The boundary conditions are:

$$\begin{equation}
C(0,t) = 0\\
C(S_{max},t) = S_{max} - E
\end{equation}$$

The initial conditions are:
$$\begin{equation}
C(S,0) = max (S-E,0)
\end{equation}$$

```
# initial condition
C = np.zeros(M+1)
C = s - E
for i in range(M+1):
    if C[i] < 0:
        C[i] = 0
Cn = np.zeros(M+1)
```

```
for n in range(N):
    Cn = C.copy()
    for i in range(1,M):
        C[i] = 0.5 * (sigma**2 * i**2 * dt - r*i*dt) * Cn[i-1] \
             +       (1 - sigma**2* i**2 *dt - r*dt) * Cn[i]   \
             + 0.5 * (sigma**2 * i**2 * dt + r*i*dt) * Cn[i+1]
print 'the price of the call option should be around {}, \
if the price of stock is 20 dollar.'.format(C[M/2])
```

    the price of the call option should be around 10.2470001259, if the price of stock is 20 dollar.

Here we get the solution of the European call option problem. Then we compared the numerical result with the result given by a statistical method:

$$\begin{equation}
C(S,t) = SN(d_1) -Ee^{-rT}N(d_2)
\end{equation}$$

N(x) is the cumulative distribution function for a standardised normal random variable (we use *norm.cdf* to calculate in python), given by

$$\begin{equation}
N(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} {e^{-\frac{1}{2}y^2}dy}
\end{equation}$$

and

$$\begin{equation}
d_1 = \frac{ln(S/E)+(r + \frac{1}{2}\sigma^2)T}{\sigma \sqrt{T}}\\
d_2 = d_1 - \sigma \sqrt{T}
\end{equation}$$

```
from scipy.stats import norm
import math
C_stat = np.zeros(M+1)
for i in range(M+1):
    d1 = (math.log1p(s[i]/E) + (r+0.5*sigma**2)*T) / (sigma * math.sqrt(T))
    d2 = d1 - (sigma * math.sqrt(T))
    C_stat[i] = s[i] * norm.cdf(d1) - E*math.exp(-r*T) * norm.cdf(d2)
```

```
plt.figure(figsize=(8,5), dpi=100)
plt.plot(s,C,color='#003366', ls='--', lw=3, label='Computational');
plt.plot(s,C_stat, label='Statistical');
plt.xlabel('S')
plt.ylabel('C')
plt.legend();
```

We compare the results between numerical method and statistical method. They match well after $S>10$. When $S<10$, the statisical model is not accurate (below zero is meaningless).

##### Reference

1. Stewart, Ian. In Pursuit of the Unknown: 17 Equations That Changed the World Paperback, Basic Books, 2013. Print
2. "Option (finance)." Wikipedia. Wikimedia Foundation, n.d. Web. 13 Jan. 2015.
2. Fischer Black and Myron Scholes (1973). The Pricing of Options and Corporate Liabilities, Journal of Political Economy, Vol. 81, No. 3 (May - Jun., 1973), pp. 637-654
3. Jiang, L. (2005). Mathematical modeling and methods of option pricing. Singapore: World Scientific.

```
from IPython.core.display import HTML
css_file = '../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>

```

```


Some sample codes can be found at another [notebook](http://nbviewer.ipython.org/github/folivetti/LINKEDOPT/blob/master/LineDistanceSrc.ipynb).

Line Distance
==============

The Line Distance was first proposed in [deFranca][1] and measures the likelihood of two points being located in the basis of attraction of the same local optima. It does so by approximating the curve through a line and checking the approximation error.

Given a multimodal function *f(x)* and two points, *x1* and *x2*, de Line Distance *LD(x1,x2)* can be calculated as:

$$ LD(x1,x2) = \|P_{proj}\| $$

with

$$P_{proj} = \overline{P_1P_m} - (\overline{P_1P_m} \cdot v)v$$

and

$$\begin{align}
P_1 &= [x1, f(x1)] \\
v &= \frac{[x2, f(x2)] - P_1}{\|[x2, f(x2)] - P_1\|} \\
P_m &= [\frac{x1+x2}{2}, f(\frac{x1+x2}{2})] \\
\overline{P_1P_m} &= [\frac{x2-x1}{2}, f(\frac{x1+x2}{2})-f(x1)]
\end{align}$$

as the projection of the middle point between *[x1, f(x1)]* and *[x2, f(x2)]* to the line segment defined by those two points.

[1]: http://dl.acm.org/citation.cfm?doid=1068009.1068057  "deFranca"

We can simplify and generalize this equation to a d-dimensional vector. Let us define:

$$\begin{align}
\overline{P_1P_m} &= [\frac{x_1+x_2}{2}-x_1, y_m-y_1] = [\frac{x_2-x_1}{2}, y_m-y_1] \\
\overline{P_1P_2} &= [x_2-x_1, y_2-y_1] \\
\|\overline{P_1P_2}\| &= \sqrt{\sum_{i}{(x_2-x_1)^{2}} + (y_2-y_1)^{2}} \\
\end{align}$$

$LD(P_1,P_2)$ is then:

$$\begin{align}
LD(P_1,P_2) &= \|\overline{P_1P_m} - (\overline{P_1P_m} \cdot \overline{P_1P_2})\frac{\overline{P_1P_2}}{\|\overline{P_1P_2}\|^{2}}\| \\
LD(P_1,P_2) &= \|\frac{\|\overline{P_1P_2}\|^{2}\overline{P_1P_m} - (\overline{P_1P_m} \cdot \overline{P_1P_2})\overline{P_1P_2}}{\|\overline{P_1P_2}\|^{2}}\| \\
\end{align}$$

Given that

$$(\overline{P_1P_m} \cdot \overline{P_1P_2}) = \frac{1}{2}\sum_{i}{(x_2-x_1)^{2}} + (y_m-y_1)(y_2-y_1)$$

The $LD(P_1,P_2)$ becomes

$$LD(P_1,P_2) = \|\frac{\sum_{i}{(x_2-x_1)^{2}}(\overline{P_1P_m} - \frac{1}{2}\overline{P_1P_2}) + (y_2-y_1)( (y_2-y_1)\overline{P_1P_m} - (y_m-y_1)\overline{P_1P_2} )}{\|\overline{P_1P_2}\|^{2}}\|$$

By developing the first term of the sum we have

$$\begin{align}
& \sum_{i}{(x_2-x_1)^{2}}[\frac{x_2-x_1}{2}-\frac{x_2-x_1}{2}, (y_m-y_1)-\frac{1}{2}(y_2-y_1)] \\
&= \sum_{i}{(x_2-x_1)^{2}}[0, y_m - \frac{y_2-y_1}{2}] \\
\end{align}$$

The second term becomes

$$\begin{align}
& (y_2-y_1)[(y_2-y_1)\frac{x_2-x_1}{2} - (y_m-y_1)(x_2-x_1), (y_2-y_1)(y_m-y_1) - (y_m-y_1)(y_2-y_1)] \\
&= (y_2-y_1)[(x_2-x_1)(\frac{y_1+y_2}{2} - y_m), 0]
\end{align}$$

And summing the terms we have:
$$\begin{align}
LD(P_1,P_2) &= \frac{\|[(x_2-x_1)(y_2-y_1)(\frac{y_1+y_2}{2} - y_m), \sum_{i}{(x_2-x_1)^{2}}(\frac{y_1+y_2}{2} - y_m)]\|}{\|\overline{P_1P_2}\|^{2}} \\
LD(P_1,P_2) &= \sqrt{\frac{(y_2-y_1)^{2}(\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2} + (\sum_{i}{(x_2-x_1)^{2}})^{2}(\frac{y_1+y_2}{2} - y_m)^2}{(\sum_{i}{(x_2-x_1)^2)} + (y_2-y_1)^2)^2}} \\
LD(P_1,P_2) &= \sqrt{\frac{((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})(\sum_{i}{(x_2-x_1)^{2}} + (y_2-y_1)^{2})}{(\sum_{i}{(x_2-x_1)^2)} + (y_2-y_1)^2)^2}} \\
LD(P_1,P_2) &= \sqrt{\frac{((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})}{\sum_{i}{(x_2-x_1)^2)} + (y_2-y_1)^2}} \\
\end{align}$$

From this equation we can see that as two points get closer to each other onto the same basis of attraction, the objective-function curve can be approximated by a line and thus the distance becomes closer to zero.

**Hypothesis 1:** If two points *x1* e *x2* are located on the basis of attraction of the same local optima, scaling the function by a factor K will have an insignificant effect on *LD(x1,x2)*.

**Proof:**

From the Line Distance final equation:

$$LD(P_1,P_2) = \sqrt{\frac{((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})}{\sum_{i}{(x_2-x_1)^2)} + (y_2-y_1)^2}}$$

If we scale the function by a factor K we have:

$$\begin{align}
LD(P_1,P_2) &= \sqrt{\frac{K^{2}((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})}{\sum_{i}{(x_2-x_1)^2)} + K^{2}(y_2-y_1)^2}} \\
LD(P_1,P_2) &= \sqrt{\frac{K^{2}((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})}{ K^{2}(\sum_{i}{(\frac{x_2-x_1}{K})^2)} +(y_2-y_1)^2})} \\
LD(P_1,P_2) &= \sqrt{\frac{((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})}{(\sum_{i}{(\frac{x_2-x_1}{K})^2)} +(y_2-y_1)^2})} \\
\end{align}$$

In this situation, the expression $\frac{x_2-x_1}{K}$ tends to zero as $K$ tends to infinity. So, this distance is bounded to a value as the objective-function is scaled to infinity. This allow us to simplify this measure while maintaining the closest points to a distance close to zero while amplifying the farthest points:

$$LD(P_1,P_2) = \sqrt{\frac{((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})}{(y_2-y_1)^2}}$$

Finally, we can work with the squared Line Distance that not only removes the square root from the calculation but also makes the distance always positive, with a value of $0$ whenever $x1=x2$. Also, an $\epsilon$ should be included in the denominator to avoid a discontinuity whenever $y_2=y_1$.

$$LDS(P_1,P_2) = \frac{((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{(x_2-x_1)^2})}{(y_2-y_1)^2 + \epsilon}$$

### Local Optima

The minimum value possible for this distance function is $0$ that is obtained whenever:

- $x_2 = x_1$
- $\frac{y_1+y_2}{2} = y_m$
- $(y_2-y_1)^2 \to \infty$ (like that's gonna happen)

The first case is the expected solution in which both points are the same.

The second is a situation that we should be aware of, if the $x_1$ and $x_2$ are located at different basis of attraction but,  the middle point intersects $(y_1+y_2)/2$, their distance will become $0$ even though they are not at the same basis. This could be alleviated by taking different middle points with different average weightings.

The third case will only happens if the function can tend to inifinity.

In order to study the maximization of this function, let us fix a certain point $x_1$ and a random direction $d$ described by an unit vector. Let us define $x_2 = x_1 + \alpha d$ with $0 < \alpha < r$, and $r$ being a radius that encloses only one peak. Our goal is to find the value of $\alpha$ that maximizes $LDS(x_1,x_2)$.

$$LDS(P_1,P_2) = \alpha^{2}\frac{((\frac{y_1+y_2}{2} - y_m)^{2}\sum_{i}{d_i^2})}{(y_2-y_1)^2 + \epsilon}$$

As $d$ is an unit vector, then $\sum_{i}{d_i^2}=1$:

$$LDS(P_1,P_2) = \alpha^{2}\frac{((\frac{y_1+y_2}{2} - y_m)^{2}}{(y_2-y_1)^2 + \epsilon}$$

From the definition of $LDS()$ we can assert that the maximum possible value is obtained when $y_2 - y_1$ is close to zero and $\frac{y_1+y_2}{2} - y_m$ is maximized. Replacing $y_2$ with $y_1$ and taking the derivative, we have:

$$\frac{d y_1 - y_m}{d\alpha} = 0$$

$$y'_m = 0$$

So, the maximum is obtained whenever $y'_m$ is a local optima.

```

```


```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import sympy as sy
sy.init_printing()
```

# Definition of Linear Independence

If all $c$'s are zero, a set of vectors $\{v_1, v_2,...,v_p\}$ is said to be **linearly independent**, if the equation

$$c_{1} {v}_{1}+c_{2} {v}_{2}+\cdots+c_{p} {v}_{p}=\mathbf{0}$$

holds.

If any of $c_i\neq 0$, the set of vectors is linearly dependent.

##  Example 1

Determine if ${v}_1, {v}_2, {v}_3$ are linearly independent.
$$
{v}_{1}=\left[\begin{array}{l}
1 \\
2 \\
3
\end{array}\right]^T,
{v}_{2}=\left[\begin{array}{l}
4 \\
5 \\
6
\end{array}\right]^T, \text { and } {v}_{3}=\left[\begin{array}{l}
2 \\
1 \\
0
\end{array}\right]^T
$$

The common way of testing linear combination is to construct augmented matrix and calculate the reduced form, for example

```python
A = sy.Matrix([[1,4,2,0],
               [2,5,1,0],
               [3,6,0,0]])
A.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & -2 & 0\\0 & 1 & 1 & 0\\0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1\right)\right)$

The solution tells that $x_3$ is a free variable, so naturally it could be nonzero because $x_3\cdot 0 =0$, therefore the set is linearly dependent.

##  Example 2

Consider a matrix $A$，determine if columns of $A$ are linearly independent.

$$
A=\left[\begin{array}{rrr}
0 & 1 & 4 \\
1 & 2 & -1 \\
5 & 8 & 0
\end{array}\right]
$$

Solve the system via augmented matrix.

```python
A = sy.Matrix([[0,1,4,0],[1,2,-1,0],[5,8,0,0]])
A.rref()
```

$Ax=0$ has only trivial solution, i.e. $(c_1, c_2, c_3)^T = (0, 0, 0)$, so the columns of $A$ are linearly independent.

Linear independence is closly connected with linear combination, in next section we visualize the linear independence.

#  Visualization of Linear Independence

This is a visual example in $\mathbb{R}^2$, showing $(3, 2)^T$, $(-9, -6)^T$, $(6, 4)^T$ are linearly dependent.

```python
fig, ax = plt.subplots(figsize = (8, 8))
#######################Arrows#######################
arrows = np.array([[[0,0,3,2]],
                   [[0,0,-9,-6]],
                   [[0,0,6,4]]])
colors = ['r','b','g']
for i in range(arrows.shape[0]):
    X,Y,U,V = zip(*arrows[i,:,:])
    ax.arrow(X[0], Y[0], U[0],V[0], color = colors[i], width = .18,
             length_includes_head = True,
             head_width = .3, # default: 3*width
             head_length = .6,
             overhang = .4, zorder = -i)

ax.scatter(0, 0, ec = 'red', fc = 'black', zorder = 5)
ax.text(6, 4, '$(6, 4)$')
ax.text(-9, -6.5, '$(-9, -6)$')
ax.text(3, 2, '$(3, 2)$')

ax.grid(True)
ax.set_title('Linear Dependence Visualization')
ax.axis([-10, 10, -10, 10])
ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
plt.show()
```

Simply put, _if one vector is the scalar multiple of the other vector, e.g. $3u = v$, these two vectors are linearly dependent_.

Next, we visualize linear independence in $\mathbb{R}^3$ with vectors $(1,-2,1)^T$, $(2,1,2)^T$, $(-1,2,3)^T$.

The standard procedure is to write down the span of first two vectors, which is a plane. Then we examine whether the third vector is in the plane. If not, this set of vectors is linearly independent.

$$
\left[
\begin{matrix}
x\\
y\\
z
\end{matrix}
\right]=
s\left[
\begin{matrix}
1\\
-2\\
1
\end{matrix}
\right]+
t\left[
\begin{matrix}
2\\
1\\
2
\end{matrix}
\right]=
\left[
\begin{matrix}
s+2t\\
-2s+t\\
s+2t
\end{matrix}
\right]
$$

```python
# %matplotlib notebook, use this only when you are in Jupyter Notebook, it doesn't work in Jupyterlab
fig = plt.figure(figsize = (10,10))
ax = fig.add_subplot(projection='3d')

s = np.linspace(-1, 1, 10)
t = np.linspace(-1, 1, 10)
S, T = np.meshgrid(s, t)

X = S+2*T
Y = -2*S+T
Z = S+2*T
ax.plot_wireframe(X, Y, Z, linewidth = 1.5, color = 'k', alpha = .6)

vec = np.array([[[0, 0, 0, 1, -2, 1]],
               [[0, 0, 0, 2, 1, 2]],
               [[0, 0, 0, -1, 2, 3]]])
colors = ['r','b','g']
for i in range(vec.shape[0]):
    X, Y, Z, U, V, W = zip(*vec[i,:,:])
    ax.quiver(X, Y, Z, U, V, W, length=1, normalize=False, color = colors[i],
              arrow_length_ratio = .08, pivot = 'tail',
              linestyles = 'solid',linewidths = 3, alpha = .6)

ax.set_title('Linear Independence Visualization')

ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
ax.set_zlabel('z-axis', size = 18)

ax.view_init(elev=50., azim=0)
plt.show()
```

Pan around the image (either by setting ```ax.view_init``` or using JupyterLab widget), we can see that the green vector is not in the plane spanned by red and blue vector, thus they are linearly independent.

A Sidenote About Linear Independence

Let $S = \{{v}_1,{v}_2,{v}_3, ..., {v}_n\}$ be a set of vectors in $\mathbb{R}^m$, if $n>m$, then $S$ is always linearly dependent. Simple example is $4$ vectors in $\mathbb{R}^3$, even if $3$ of them are linearly independent, the $4$th one can never be independent of them.

Also if $S = \{{v}_1,{v}_2,{v}_3, ..., {v}_n\}$ contains a zero vector, then the set is always linearly dependent.

```python

```


# Noisy GP Experiments

```python

```

```python
import numpy as np
import numba
import matplotlib as mpl
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
from sklearn.utils import check_random_state, check_array, check_X_y
from sklearn.model_selection import train_test_split
from scipy.spatial.distance import pdist, cdist, squareform
from sklearn.metrics.pairwise import check_pairwise_arrays, euclidean_distances
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import _check_length_scale, WhiteKernel, RBF, ConstantKernel as C
from scipy import interpolate
from sklearn.metrics import mean_absolute_error, mean_squared_error
import warnings
mpl.rcParams['savefig.pad_inches'] = 0

%matplotlib inline
%load_ext Cython
%load_ext autoreload
%autoreload 2
```

    /home/emmanuel/.conda/envs/sci_py36/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
      return f(*args, **kwds)
    /home/emmanuel/.conda/envs/sci_py36/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
      return f(*args, **kwds)
    /home/emmanuel/.conda/envs/sci_py36/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
      return f(*args, **kwds)
    /home/emmanuel/.conda/envs/sci_py36/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
      return f(*args, **kwds)

## 1D Toy Example

```python
figure_location = '/home/emmanuel/projects/2018_igarss/figures/noisy_gp/1d_example/'
figure_location = '/Users/eman/Documents/code_projects/gaussian_processes/pkg_sgpuppy/figures/'
figure_location = '/home/emmanuel/projects/gaussian_processes/pkg_sgpuppy/figures'
```

```python
weights = {}
predictions = {}
variance = {}
scores = {}
```

### Sample Data

```python
seed = 123
rng = np.random.RandomState(seed=seed)

# sample data parameters
n_train, n_test, n_trial = 60, 100, 100
sigma_y = 0.05
x_cov = 0.3
y_scale = .5
x_min, x_max = -10, 10

# real function
f = lambda x: np.sin(1.0 * np.pi / 1.6 * np.cos(5 + y_scale * x))
# f = lambda x: np.sinc(x)
# f = lambda x: np.sin(2. * x) + np.exp(0.2 * x)

# Training add x, y = f(x)
x = np.linspace(x_min, x_max, n_train + n_test + n_trial)

x, xs, = train_test_split(x, train_size=n_train, random_state=seed)

# add noise
y = f(x)
x_train = x + x_cov * rng.randn(n_train)
y_train = f(x) + sigma_y * rng.randn(n_train)

x_train, y_train = x_train[:, np.newaxis], y_train[:, np.newaxis]

# -----------------
# Testing Data
# -----------------
xs, xt = train_test_split(xs, train_size=n_test, random_state=seed)
xt = np.sort(xt)

ys = f(xs)

# Add noise
x_test = xs + x_cov * rng.randn(n_test)
y_test = ys

x_test, y_test = x_test[:, np.newaxis], y_test[:, np.newaxis]

# -------------------
# Plot Points
# -------------------
x_plot = np.linspace(x_min, x_max, n_test)[:, None]
y_plot = f(x_plot)
```

    /home/emmanuel/.conda/envs/sci_py36/lib/python3.6/site-packages/sklearn/model_selection/_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.
      FutureWarning)

```python
print(x.shape, y.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, x_test.shape)
```

    (60,) (60,)
    (60, 1) (60, 1)
    (100, 1) (100, 1)

```python
# plot toy data
fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, y_plot, linewidth=5, color='k', label='Real Function')
ax.scatter(x_train, y_train, color='r', s=100, label='Training Data')
ax.scatter(x_test, y_test, color='g', s=100, label='Testing Data')

# ax.set_title('Training and Testing Data', fontsize=20)
ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.show()

save_name = 'training_data.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Naive GP Implementation (to Find Suitable Hyperparameters)

```python
def fit_gp(x_train, y_train, kernel='ard', scale=None,
           length_scale_bounds = (0.001, 100),
           noise_level_bounds = (1e-4, 10),
           n_restarts=3):

    warnings.simplefilter('ignore')
    x_train, y_train = check_X_y(x_train, y_train)
    warnings.simplefilter('default')
    n_train, d_dims = x_train.shape

    if kernel in ['ard', 'ARD']:
        length_scale_init = np.ones(shape=(d_dims))
    elif kernel in ['rbf', 'RBF']:
        length_scale_init = 1.0
    else:
        raise ValueError('Unrecognized kernel function...')

    noise_level_init = 1.0

    gp_kernel = C() * RBF(length_scale=length_scale_init,
                    length_scale_bounds=length_scale_bounds) + \
                WhiteKernel(noise_level=noise_level_init,
                            noise_level_bounds=noise_level_bounds)

    gpr_model = GaussianProcessRegressor(kernel=gp_kernel, random_state=123,
                                         n_restarts_optimizer=n_restarts)

    # Fit the GP Model
    gpr_model.fit(x_train, y_train)

    # the parameters

    constant = gpr_model.kernel_.get_params()['k1__k1__constant_value']
    length_scale = gpr_model.kernel_.get_params()['k1__k2__length_scale']
    sigma = gpr_model.kernel_.get_params()['k2__noise_level']

    return length_scale, sigma, constant
```

```python
length_scale, sigma_y, constant = fit_gp(x_train, y_train, n_restarts=0)
print('Length Scale: {:.3f}'.format(length_scale))
print('Sigma y: {:.3f}'.format(sigma_y))
print('Likelihood Variance: {:.3f}'.format(constant))
```

    Length Scale: 1.854
    Sigma y: 0.013
    Likelihood Variance: 0.656

## Uncertainty Propagation Function

```cython
%%cython

import numpy as np
cimport numpy as np
from Covariance import tracedot
from numpy.linalg import inv, det
from libc.math cimport exp

DTYPE = np.float64
ctypedef np.float64_t DTYPE_t

#TODO: complete types

cdef class UncertaintyPropagationGA:
	"""
	Superclass for all UncertaintyPropagationGA Classes
	"""
	cdef object gp

	def __init__(self, object gp):
		"""

		:param gp: callable gaussian process that returns mean and variance for a given input vector x
		"""
		self.gp = gp

	cpdef double propagate_mean(self,np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=2] Sigma_x):
		"""
		Propagates the mean using the gaussian approximation from Girard2004

		:param u: vector of means
		:param Sigma_x: covariance Matrix of the input
		:return: mean of the output
		"""
		return 0

	cpdef tuple propagate_GA(self,np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=2] Sigma_x):
		"""
		Propagates the uncertainty.py using the gaussian approximation from Girard2004

		:param u: vector of means
		:param Sigma_x: covariance Matrix of the input
		:return: mean, variance of the output
		"""
		return 0, 0

cdef class UncertaintyPropagationExact(UncertaintyPropagationGA):

	cdef object Sigma_x
	cdef object Winv
	cdef object LambdaInv
	cdef double normalize_C_corr2
	cdef object Deltainv
	cdef double normalize_C_corr
	cdef object C_ux

	cdef _prepare_C_corr(self,int D):
		"""
		Prepares the constants for C_corr

		:param D: Number of dimensions of the input vectors
		"""
		#D = len(xi)
		cdef np.ndarray[DTYPE_t,ndim=2] I = np.eye(D)
		self.Deltainv = self.Winv - np.diag(
			np.array([self.Winv[i][i] / (1 + self.Winv[i][i] * self.Sigma_x[i][i]) for i in range(D)]))
		self.normalize_C_corr = 1 / np.sqrt(det(I + self.Winv * self.Sigma_x))

	cdef double _get_C_corr(self,np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=1] xi):
		"""
		:param u: N-dimensional vector
		:param xi: N-dimensional vector
		:return: covariance correction factor
		"""
		diff = u - xi
		cdef double normalize_C_corr = self.normalize_C_corr
		cdef np.ndarray[DTYPE_t,ndim=2] Deltainv = self.Deltainv
		return  normalize_C_corr * np.exp(0.5 * (np.dot(diff.T, np.dot(Deltainv, diff))))

	cpdef double propagate_mean(self, np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=2] Sigma_x):
		cdef np.ndarray[DTYPE_t,ndim=2] x = self.gp.x
		cdef int N
		cdef int d
		cdef list C_ux_t
		N = x.shape[0]
		d = x.shape[1]
		cdef np.ndarray[DTYPE_t,ndim=1] C_ux = self.C_ux
		if C_ux is None:
			C_ux_t = []
			for i in range(N):
				C_ux_t.append(self.gp._covariance(u,x[i]))
			C_ux = np.array(C_ux_t)

		cdef np.ndarray[DTYPE_t,ndim=1] beta = self.gp._get_beta()
		self.Winv = self.gp._get_W_inv()
		self.Sigma_x = Sigma_x

		assert(N == len(beta))
		self._prepare_C_corr(len(u))

		cdef double sum = 0.0
		for i in range(N):
			sum += beta[i] * C_ux[i] * self._get_C_corr(u, x[i])

		return sum

	cdef _prepare_C_corr2(self, D):
		"""
		Prepares the constants for C_corr2

		:param D: Number of dimensions of the input vectors
		"""

		#D = len(x)
		I = np.eye(D)
		W = np.diag(np.array([1 / self.Winv[i][i] for i in range(D)]))
		self.LambdaInv = 2 * self.Winv - inv(0.5 * W + self.Sigma_x)
		self.normalize_C_corr2 = 1 / np.sqrt(det(2 * self.Winv * self.Sigma_x + I))

	cdef _get_C_corr2(self, u, x):
		"""
		This function is alternatively calculated by weave inline C code

		:param u: N-dimensional vector
		:param xi: N-dimensional vector
		:return: covariance correction factor 2
		"""

		#D = len(x)
		#I = np.eye(D)
		#W = np.diag(np.array([1/self.Winv[i][i] for i in range(D)]))
		#LambdaInv = 2* self.Winv - inv(0.5*W+self.Sigma_x)
		diff = u - x

		return self.normalize_C_corr2 * np.exp(0.5 * (np.dot(diff.T, np.dot(self.LambdaInv, diff))))

	cpdef tuple propagate_GA(self,np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=2] Sigma_x):
		cdef np.ndarray[DTYPE_t,ndim=1] beta = self.gp._get_beta()
		cdef np.ndarray[DTYPE_t,ndim=2] x = self.gp.x
		self.Winv = self.gp._get_W_inv()
		self.Sigma_x = Sigma_x
		cdef np.ndarray[DTYPE_t,ndim=2] Kinv = self.gp._inv_cov_matrix()
		cdef unsigned int N = x.shape[0]
		cdef unsigned int d = x.shape[1]
		assert(N == len(beta))
		cdef unsigned int i,j,i2,j2

		self._prepare_C_corr2(len(u))

		cdef list C_ux_ = []
		for i in range(N):
			C_ux_.append(self.gp._covariance(u,x[i]))
		cdef np.ndarray[DTYPE_t,ndim=1] C_ux = np.array(C_ux_)
		self.C_ux = C_ux
		mu = self.propagate_mean(u, Sigma_x)

		cdef double nc = self.normalize_C_corr2
		cdef np.ndarray[DTYPE_t,ndim=2] L = self.LambdaInv
		cdef double sum = 0.0
		cdef double dot = 0.0
		for i in range(N):
			for j in range(N):
				dot = 0.0
				for i2 in range(d):
					for j2 in range(d):
						dot += (u[i2]-(x[i,i2]+x[j,i2])/2.0) * (u[j2]-(x[i,j2]+x[j,j2])/2.0) *L[i2,j2]
				sum += (Kinv[i,j] - beta[i] * beta[j]) * C_ux[i]*C_ux[j] * nc * exp(0.5 * dot)

		variance = self.gp._covariance(u, u) - sum - mu ** 2

		return mu + self.gp._get_mean_t(), variance

cdef class UncertaintyPropagationApprox(UncertaintyPropagationGA):

	cdef double v
	cdef object Winv
	cdef object u
	cdef object C_ux
	cdef object H_ux
	cdef object J_ux

	def __init__(self, gp):
		"""

		:param gp: callable gaussian process that returns mean and variance for a given input vector x
		"""
		UncertaintyPropagationGA.__init__(self, gp)
		self.v = self.gp._get_v()
		self.Winv = self.gp._get_W_inv()
		self.u = None

	cpdef double propagate_mean(self,np.ndarray[DTYPE_t,ndim=1] u, np.ndarray[DTYPE_t,ndim=2] Sigma_x):
		cdef np.ndarray[DTYPE_t,ndim=1] beta = self.gp._get_beta()
		cdef np.ndarray[DTYPE_t,ndim=2] x = self.gp.x
		cdef int n = len(x)
		cdef double mu = sum([beta[i]*self.C_ux[i] for i in range(n)])
		#print "Approx mean"
		#print mu
		#print 0.5 * sum([beta[i] * np.trace(np.dot(self.get_Hessian(u,x[i]),Sigma_x)) for i in range(n)])

		#Time complexity d**3 because of dot

		return mu + 0.5 * sum([beta[i] * np.trace(np.dot(self.H_ux[i],Sigma_x)) for i in range(n)])

	cdef double _get_sigma2(self,np.ndarray[DTYPE_t,ndim=1] u, np.ndarray[DTYPE_t,ndim=2] Kinv, np.ndarray[DTYPE_t,ndim=2]x,  np.ndarray[DTYPE_t,ndim=1] C_ux, np.ndarray[DTYPE_t,ndim=3]J_ux,  np.ndarray[DTYPE_t,ndim=3] H_ux):
		cdef int n = len(x)
		cdef unsigned int i,j
		cdef DTYPE_t sum_ = 0.0
		for i in range(n):
			for j in range(n):
				sum_ += Kinv[i,j]*C_ux[i]*C_ux[j]

		sigma2 = self.gp._covariance(u,u) - sum_#

		return sigma2

	cdef double _get_variance_rest(self,np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=2] Sigma_x, np.ndarray[DTYPE_t,ndim=2] Kinv, np.ndarray[DTYPE_t,ndim=2] x,np.ndarray[DTYPE_t,ndim=1] beta, np.ndarray[DTYPE_t,ndim=1] C_ux, np.ndarray[DTYPE_t,ndim=3] J_ux, np.ndarray[DTYPE_t,ndim=3] H_ux):
		cdef int n = x.shape[0]
		cdef int d = x.shape[1]
		cdef unsigned int i,j,k
		cdef double trace_ = 0.0
		cdef np.ndarray[DTYPE_t,ndim=2] S = np.atleast_2d(np.diag(Sigma_x).copy()).T
		cdef double sum_ = 0.0
		for i in range(n):
			for j in range(n):
				trace_ = 0.0
				for k in range(d):
					trace_ += J_ux[i,k,0]*J_ux[j,k,0]*S[k,0]
				sum_ += (Kinv[i,j]-beta[i]*beta[j]) * trace_
		variance2 =- sum_ #sum([(Kinv[i][j]-beta[i]*beta[j]) * (J_ux[i]*J_ux[j]*S).sum() for i in range(n) for j in range(n)])

		cdef np.ndarray[DTYPE_t,ndim=1] trace = np.array([tracedot(H_ux[i],Sigma_x) for i in range(n)])

		sum_ = 0.0
		for i in range(n):
			for j in range(n):
				sum_ += Kinv[i,j]*(C_ux[i]*trace[j]+C_ux[j]*trace[i])
		variance3 = - 0.5* sum_ #sum([Kinv[i][j]*(C_ux[i]*trace[j]+C_ux[j]*trace[i]) for i in range(n) for j in range(n)])

		return variance2+variance3

	cdef tuple _get_sigma2_and_variance_rest(self,np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=2] Sigma_x, np.ndarray[DTYPE_t,ndim=2] Kinv, np.ndarray[DTYPE_t,ndim=2] x, np.ndarray[DTYPE_t,ndim=1] beta):
		n = len(x)
		cdef double sigma2 = self._get_sigma2(u,Kinv,x,self.C_ux,self.J_ux,self.H_ux)
		cdef double variance_rest = self._get_variance_rest(u,Sigma_x,Kinv,x,beta,self.C_ux,self.J_ux,self.H_ux)

		return sigma2, variance_rest

	cpdef tuple propagate_GA(self,np.ndarray[DTYPE_t,ndim=1] u,np.ndarray[DTYPE_t,ndim=2] Sigma_x):
		beta = self.gp._get_beta()
		x = self.gp.x
		n = len(x)
		Kinv = self.gp._inv_cov_matrix()
		#Kinv = inv(self.gp.cov_matrix())

		#We store the values because we can reuse them
		if self.u is None or (self.u != u).any():

			self.u = u
			self.C_ux = []
			self.J_ux = []
			self.H_ux = []
			for i in range(n):
				self.C_ux.append(self.gp._covariance(u,x[i]))
				self.J_ux.append(self.gp._get_Jacobian(u,x[i]))
				self.H_ux.append(self.gp._get_Hessian(u,x[i]))
			self.C_ux = np.array(self.C_ux)
			self.J_ux = np.array(self.J_ux)
			self.H_ux = np.array(self.H_ux)
		print(np.ndim(u), np.ndim(Sigma_x))
		mean = self.propagate_mean(u,Sigma_x)
		sigma2, variance_rest = self._get_sigma2_and_variance_rest(u,Sigma_x,Kinv,x,beta)

		#TODO!!!: Only if C''(u,u) == 0
		variance = sigma2 + variance_rest

		return mean + self.gp._get_mean_t(), variance

	cpdef double _getFactor(self,u,Sigma_x,v):
		"""

		:param u: the mean
		:param Sigma_x: The weight_vector based cov matrix
		:param v: the limiting output variance
		:return: the lambda to multiply Sigma_x with
		"""
		#TODO!!!: consider predefined variances and divide into block matrices

		beta = self.gp._get_beta()
		x = self.gp.x
		n = len(x)

		Kinv = self.gp._inv_cov_matrix() #inv(self.gp.cov_matrix())
		#We store the values because we can reuse them
		if self.u is None or (self.u != u).any():

			self.u = u
			self.C_ux = []
			self.J_ux = []
			self.H_ux = []
			for i in range(n):
				self.C_ux.append(self.gp._covariance(u,x[i]))
				self.J_ux.append(self.gp._get_Jacobian(u,x[i]))
				self.H_ux.append(self.gp._get_Hessian(u,x[i]))
			self.C_ux = np.array(self.C_ux)
			self.J_ux = np.array(self.J_ux)
			self.H_ux = np.array(self.H_ux)

		sigma2, variance_rest = self._get_sigma2_and_variance_rest(u,Sigma_x,Kinv,x,beta)

		return (v-sigma2)/(variance_rest)

	cpdef double _get_variance_dv_h(self,np.ndarray[DTYPE_t,ndim=1] u,unsigned int h):
		cdef np.ndarray[DTYPE_t,ndim=1] beta = self.gp._get_beta()
		cdef np.ndarray[DTYPE_t,ndim=2] Kinv = self.gp._inv_cov_matrix()
		cdef np.ndarray[DTYPE_t,ndim=2] x = self.gp.x
		cdef unsigned int n = x.shape[0]
		cdef unsigned int d = x.shape[1]

		#We store the values because we can reuse them
		if self.u is None or (self.u != u).any():

			self.u = u
			self.C_ux = []
			self.J_ux = []
			self.H_ux = []
			for i in range(n):
				self.C_ux.append(self.gp._covariance(u,x[i]))
				self.J_ux.append(self.gp._get_Jacobian(u,x[i]))
				self.H_ux.append(self.gp._get_Hessian(u,x[i]))
			self.C_ux = np.array(self.C_ux)
			self.J_ux = np.array(self.J_ux)
			self.H_ux = np.array(self.H_ux)

		cdef np.ndarray[DTYPE_t,ndim=1] C_ux = self.C_ux
		cdef np.ndarray[DTYPE_t,ndim=3] H_ux = self.H_ux
		cdef np.ndarray[DTYPE_t,ndim=3] J_ux = self.J_ux

		cdef double sum_ = 0.0
		for i in range(n):
			for j in range(n):
				sum_ += (Kinv[i,j]-beta[i]*beta[j]) * J_ux[i,h,0]*J_ux[j,h,0]
		variance2 = - sum_ #sum([(Kinv[i,j]-beta[i]*beta[j]) * self.J_ux[i,h,0]*self.J_ux[j,h,0] for i in range(n) for j in range(n)])

		sum_ = 0.0
		for i in range(n):
			for j in range(n):
				sum_ += Kinv[i,j]*(C_ux[i]*H_ux[j,h,h] +C_ux[j]*H_ux[i,h,h])
		variance3 = - 0.5* sum_ #sum([Kinv[i,j]*(C_ux[i]*H_ux[j,h,h] +C_ux[j]*H_ux[i,h,h]) for i in range(n) for j in range(n)])

		return variance2+variance3

```

## Gaussian Covariance Function (ARD/RBF Kernel)

\begin{equation}
    K(x, y) = \nu \cdot \text{exp} \left[ -\frac{1}{2} (x - y)^{\top} \cdot W^{-1} \cdot (x - y) \right]
\end{equation}

* $W^{-1}= \text{diag}\left[ w_1, w_2, \ldots, w_{D} \right]$
* $w_d = \frac{1}{\lambda_d^2}$
* $\nu$ is the signal variance parameter

### GP/KRR Manually

```python
from GaussianProcess import GaussianProcess
from Covariance import GaussianCovariance
```

```python
w = np.array([length_scale])
v = constant
vt = sigma_y
Sigma = np.array([0.8])
if np.ndim(Sigma) == 1:
    Sigma = Sigma[:, np.newaxis]

# Preparing the parameter vector
theta = np.zeros(2+len(w))
theta[0] = np.log(v)  # We actually use the log of the parameters as it is easier to optimize (no > 0 constraint etc.)
theta[1] = np.log(vt)
theta[2:2+len(w)] = np.log(w)

# Simulating simulation data by drawing data from a random Gaussian process
t = GaussianProcess.get_realisation(x_train, GaussianCovariance(),theta)

# The regression step is pretty easy:
# Input data x (list of input vectors)
# Corresponding simulation output t (just a list of floats of the same length as x)
# Covariance function of your choice (only GaussianCovariance can be used for uncertainty propagation at the moment)
gp_est = GaussianProcess(x_train, y_train,GaussianCovariance());
```

    [-0.44196361 -1.82825797 -4.62977644]
    l_bfgs_b 	 0.08210444450378418 	 [[-14.67484715]] 	 [-0.37193111 -4.30782621 -1.25336646]
    [-0.37193111 -4.30782621 -1.25336646]

```python
%%time
# Initialize the GP and Fit the model
gp_model = GaussianProcess(x_train, y_train,GaussianCovariance());

# Predict
predictions['simple'], var = gp_model.estimate_many(x_test);

scores['simple'] = mean_absolute_error(predictions['simple'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['simple']))
print(var.min(), var.max())
```

    [-0.44196361 -1.82825797 -4.62977644]
    l_bfgs_b 	 0.09357762336730957 	 [[-14.67484715]] 	 [-0.37193111 -4.30782621 -1.25336646]
    [-0.37193111 -4.30782621 -1.25336646]

    MAE Score: 0.11215

    0.015185502005075069 0.035744759522648994
    CPU times: user 386 ms, sys: 2.84 s, total: 3.23 s
    Wall time: 104 ms

#### Plot Simple GP Results

```python
# Get Figure for x_plot
predictions['simple'], variance['simple'] = gp_model.estimate_many(x_plot)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['simple'], linewidth=5, color='k', label='Predictions (GP)')
ax.fill_between(x_plot.squeeze(),
                predictions['simple'].squeeze() - 2*np.sqrt(variance['simple']).squeeze(),
                predictions['simple'].squeeze() + 2*np.sqrt(variance['simple']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_simple.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Uncertainty Propagation (Approximate)

```python
# from UncertaintyPropagation import UncertaintyPropagationApprox
```

```python
%%time

# Initialize GP
gp_approx = UncertaintyPropagationApprox(gp_model)

# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_test)
predictions['approx'] = np.zeros(n_samples)
var = np.zeros(n_samples)

for i in range(n_samples):
    print(x_test[i].shape)
    print(x_test[i].ndim, Sigma.ndim)
#     predictions['approx'][i], var[i] = gp_approx.propagate_GA(x_test[i],Sigma)
    _, _ = gp_approx.propagate_GA(x_test[i],Sigma)

    break
scores['approx'] = mean_absolute_error(predictions['approx'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['approx']))
print(var.min(), var.max())
```

##### Plot Derivative Term Results

```python
# Get Figure for x_plot
# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_plot)
predictions['approx'] = np.zeros(n_samples)
variance['approx'] = np.zeros(n_samples)

for i in range(n_samples):

    predictions['approx'][i], variance['approx'][i] = gp_approx.propagate_GA(x_plot[i],Sigma)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['approx'], linewidth=5, color='k', label='Predictions (GP + Error)')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')
ax.fill_between(x_plot.squeeze(),
                predictions['approx'].squeeze() - 2*np.sqrt(variance['approx']).squeeze(),
                predictions['approx'].squeeze() + 2*np.sqrt(variance['approx']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_approx.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Exact Uncertainty

```python
from UncertaintyPropagation import UncertaintyPropagationExact
```

```python
%%time
# Initialize GP
gp_exact = UncertaintyPropagationExact(gp_model)

# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_test)
predictions['exact'] = np.zeros(n_samples)
var = np.zeros(n_samples)

for i in range(n_samples):

    predictions['exact'][i], var[i] = gp_exact.propagate_GA(x_test[i],Sigma)

scores['exact'] = mean_absolute_error(predictions['exact'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['exact']))
print(var.min(), var.max())
```

    MAE Score: 0.20656

    0.04141351054176534 0.3892824696322148
    CPU times: user 4.66 s, sys: 71.2 ms, total: 4.73 s
    Wall time: 4.65 s

```python
# Get Figure for x_plot
# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_plot)
predictions['exact'] = np.zeros(n_samples)
variance['exact'] = np.zeros(n_samples)

for i in range(n_samples):

    predictions['exact'][i], variance['exact'][i] = gp_exact.propagate_GA(x_plot[i],Sigma)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['exact'], linewidth=5, color='k', label='Predictions (GP + Error)')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')
ax.fill_between(x_plot.squeeze(),
                predictions['exact'].squeeze() - 2*np.sqrt(variance['exact']).squeeze(),
                predictions['exact'].squeeze() + 2*np.sqrt(variance['exact']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_exact.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Exact Numerical Uncertainty MCMC

```python
from UncertaintyPropagation import UncertaintyPropagationMC
```

```python
%%time
# Initialize GP
gp_stat = UncertaintyPropagationMC(gp_model)

# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_test)
predictions['stat'] = np.zeros(n_samples)
var = np.zeros(n_samples)

for i in range(n_samples):

    predictions['stat'][i], var[i] = gp_stat.propagate_GA(x_test[i],Sigma)

scores['stat'] = mean_absolute_error(predictions['stat'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['stat']))
print(var.min(), var.max())
```

    MAE Score: 0.15482

    0.03645299933786017 0.3896251934569724
    CPU times: user 32.7 s, sys: 1.78 ms, total: 32.7 s
    Wall time: 32.7 s

```python
# Get Figure for x_plot
# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_plot)
predictions['stat'] = np.zeros(n_samples)
variance['stat'] = np.zeros(n_samples)

for i in range(n_samples):

    predictions['stat'][i], variance['stat'][i] = gp_stat.propagate_GA(x_plot[i],Sigma)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['stat'], linewidth=5, color='k', label='Predictions (GP + Error)')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')
ax.fill_between(x_plot.squeeze(),
                predictions['stat'].squeeze() - 2*np.sqrt(variance['stat']).squeeze(),
                predictions['stat'].squeeze() + 2*np.sqrt(variance['stat']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_stat.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Numerical

```python
from UncertaintyPropagation import UncertaintyPropagationNumericalHG
```

```python
%%time
# Initialize GP
gp_numerical = UncertaintyPropagationNumericalHG(gp_model)

# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_test)
predictions['numerical'] = np.zeros(n_samples)
var = np.zeros(n_samples)

for i in range(n_samples):

    predictions['numerical'][i], var[i] = gp_numerical.propagate_GA(x_test[i],Sigma)

scores['numerical'] = mean_absolute_error(predictions['numerical'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['numerical']))
print(var.min(), var.max())
```

    MAE Score: 0.20668

    0.03997094081396435 0.40155955288059003
    CPU times: user 254 ms, sys: 1.82 ms, total: 256 ms
    Wall time: 254 ms

```python
# Get Figure for x_plot
# print(x_new.shape, mean.shape, Sigma.shape)
n_samples = len(x_plot)
predictions['numerical'] = np.zeros(n_samples)
variance['numerical'] = np.zeros(n_samples)

for i in range(n_samples):

    predictions['numerical'][i], variance['numerical'][i] = gp_numerical.propagate_GA(x_plot[i],Sigma)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['numerical'], linewidth=5, color='k', label='Predictions (GP + Error)')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')
ax.fill_between(x_plot.squeeze(),
                predictions['numerical'].squeeze() - 2*np.sqrt(variance['numerical']).squeeze(),
                predictions['numerical'].squeeze() + 2*np.sqrt(variance['numerical']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_numerical.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Empirical Variance Comparison

### Calculate the Empirical Variance (Get Scores)

```python
%%time

from sklearn.metrics import mean_absolute_error, mean_squared_error
rng = np.random.RandomState(None)
n_trials = 20

mae_score = {
    'simple': list(),
    'approx': list(),
    'exact': list(),
    'stat': list(),
    'numerical': list()
}

mse_score = {
    'simple': list(),
    'approx': list(),
    'exact': list(),
    'stat': list(),
    'numerical': list()
}

abs_error = {
    'simple': list(),
    'approx': list(),
    'exact': list(),
    'stat': list(),
    'numerical': list()
}
squared_error = {
    'simple': list(),
    'approx': list(),
    'exact': list(),
    'stat': list(),
    'numerical': list()
}

# testing set (noise-less)
ys = f(xt)
y_test = ys.squeeze()
y_test = np.atleast_2d(y_test).T

# fig, ax = plt.subplots()

for itrial in range(n_trials):

    # Generate x samples with random error
    x_test = xt + 0.3 * rng.randn(n_trial)
    x_test = x_test[:, np.newaxis]
    n_samples = len(x_test)
#     ax.scatter(x_test, y_test)

    # -------------------
    # Model I - Simple GP
    # -------------------
    mean, _ = gp_model.estimate_many(x_test)

    abs_error['simple'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['simple'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['simple'].append(mean_absolute_error(mean.squeeze(), y_test.squeeze()))
    mse_score['simple'].append(mean_squared_error(mean.squeeze(), y_test.squeeze()))

    # -------------------
    # Model II - Approx
    # -------------------
    mean = np.zeros(n_samples)
    for i in range(n_samples):

        mean[i], _ = gp_approx.propagate_GA(x_test[i],Sigma)

    abs_error['approx'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['approx'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['approx'].append(mean_absolute_error(mean.squeeze(), y_test.squeeze()))
    mse_score['approx'].append(mean_squared_error(mean.squeeze(), y_test.squeeze()))

    # -------------------
    # Model III - Exact
    # -------------------
    mean = np.zeros(n_samples)
    for i in range(n_samples):

        mean[i], _ = gp_exact.propagate_GA(x_test[i],Sigma)

    abs_error['exact'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['exact'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['exact'].append(mean_absolute_error(mean.squeeze(), y_test.squeeze()))
    mse_score['exact'].append(mean_squared_error(mean.squeeze(), y_test.squeeze()))

    # -------------------
    # Model IV - Numerical
    # -------------------
    mean = np.zeros(n_samples)
    for i in range(n_samples):

        mean[i], _ = gp_stat.propagate_GA(x_test[i],Sigma)

    abs_error['stat'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['stat'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['stat'].append(mean_absolute_error(mean.squeeze(), y_test.squeeze()))
    mse_score['stat'].append(mean_squared_error(mean.squeeze(), y_test.squeeze()))

    # -------------------
    # Model IV - Numerical
    # -------------------
    mean = np.zeros(n_samples)
    for i in range(n_samples):

        mean[i], _ = gp_numerical.propagate_GA(x_test[i],Sigma)

    abs_error['numerical'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['numerical'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['numerical'].append(mean_absolute_error(mean.squeeze(), y_test.squeeze()))
    mse_score['numerical'].append(mean_squared_error(mean.squeeze(), y_test.squeeze()))

```

```python
abs_error['simple'] = np.mean(np.array(abs_error['simple']).squeeze(), axis=0)
squared_error['simple'] = np.mean(np.array(squared_error['simple']).squeeze(), axis=0)
mae_score['simple'] = np.mean(np.array(mae_score['simple']))
mse_score['simple'] = np.mean(np.array(mse_score['simple']))

abs_error['approx'] = np.mean(np.array(abs_error['approx']).squeeze(), axis=0)
squared_error['approx'] = np.mean(np.array(squared_error['approx']).squeeze(), axis=0)
mae_score['approx'] = np.mean(np.array(mae_score['approx']))
mse_score['approx'] = np.mean(np.array(mse_score['approx']))

abs_error['exact'] = np.mean(np.array(abs_error['exact']).squeeze(), axis=0)
squared_error['exact'] = np.mean(np.array(squared_error['exact']).squeeze(), axis=0)
mae_score['exact'] = np.mean(np.array(mae_score['exact']))
mse_score['exact'] = np.mean(np.array(mse_score['exact']))

abs_error['stat'] = np.mean(np.array(abs_error['stat']).squeeze(), axis=0)
squared_error['stat'] = np.mean(np.array(squared_error['stat']).squeeze(), axis=0)
mae_score['stat'] = np.mean(np.array(mae_score['stat']))
mse_score['stat'] = np.mean(np.array(mse_score['stat']))

abs_error['numerical'] = np.mean(np.array(abs_error['numerical']).squeeze(), axis=0)
squared_error['numerical'] = np.mean(np.array(squared_error['numerical']).squeeze(), axis=0)
mae_score['numerical'] = np.mean(np.array(mae_score['numerical']))
mse_score['numerical'] = np.mean(np.array(mse_score['numerical']))
```

```python
fig, ax = plt.subplots()

ax.scatter(x_train, y_train, label='training samples')
ax.plot(x_plot, f(x_plot), color='black', label='real function')
ax.legend()
plt.show()
# ax.scatter(xs, abs_error['simple'], color='green', label='Empirical Absolute Error (Points)')
```

### No Uncertainty

```python
xnew = np.linspace(xt.min(), xt.max(), 1000)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['simple'], kind='slinear')(xnew)

print('MSE: {:.5f}'.format(mse_score['simple']))
print('MAE: {:.5f}'.format(mae_score['simple']))

pred, var = gp_model.estimate_many(xt[:, np.newaxis])

# pred, var = gp_simple.predict(xt[:, np.newaxis], return_variance=True)

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Variance')

# ax.set_title('Squared Error, Variance')
ax.grid(True)
ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.show()

save_name = 'exp_empirical_simple_var.png'
fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)

```

## Approximate Uncertainty

```python
xnew = np.linspace(xt.min(), xt.max(), 1000)

abs_error_line = interpolate.interp1d(xt, abs_error['approx'], kind='quadratic')(xnew)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['approx'], kind='quadratic')(xnew)

n_samples = len(xt)
pred = np.zeros(n_samples)
var = np.zeros(n_samples)
xt_ = xt[:, np.newaxis]

for i in range(n_samples):
    pred[i], var[i] = gp_approx.propagate_GA(xt_[i], Sigma)
# pred, var = gp_derivative_full.predict(xt[:, np.newaxis], return_variance=True)

print('MSE: {:.5f}'.format(mse_score['approx']))
print('MAE: {:.5f}'.format(mae_score['approx']))

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Error Variance')

# ax.set_title('Squared Error, Variance')
ax.legend(fontsize=14)
# ax.get_xaxis().set_visible(False)
# ax.get_yaxis().set_visible(False)
ax.grid(True)
plt.show()

save_name = 'exp_empirical_approx_var.png'
fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)

```

### Exact Uncertainty

```python
xnew = np.linspace(xt.min(), xt.max(), 1000)

abs_error_line = interpolate.interp1d(xt, abs_error['exact'], kind='quadratic')(xnew)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['exact'], kind='quadratic')(xnew)

n_samples = len(xt)
pred = np.zeros(n_samples)
var = np.zeros(n_samples)
xt_ = xt[:, np.newaxis]

for i in range(n_samples):
    pred[i], var[i] = gp_exact.propagate_GA(xt_[i], Sigma)
# pred, var = gp_derivative_full.predict(xt[:, np.newaxis], return_variance=True)

print('MSE: {:.5f}'.format(mse_score['exact']))
print('MAE: {:.5f}'.format(mae_score['exact']))

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Error Variance')

# ax.set_title('Squared Error, Variance')
ax.legend(fontsize=14)
# ax.get_xaxis().set_visible(False)
# ax.get_yaxis().set_visible(False)
ax.grid(True)
plt.show()

save_name = 'exp_empirical_exact_var.png'
fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)

```

## Numcerical Uncertainty (MC)

```python
xnew = np.linspace(xt.min(), xt.max(), 1000)

abs_error_line = interpolate.interp1d(xt, abs_error['stat'], kind='quadratic')(xnew)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['stat'], kind='quadratic')(xnew)

n_samples = len(xt)
pred = np.zeros(n_samples)
var = np.zeros(n_samples)
xt_ = xt[:, np.newaxis]

for i in range(n_samples):
    pred[i], var[i] = gp_stat.propagate_GA(xt_[i], Sigma)
# pred, var = gp_derivative_full.predict(xt[:, np.newaxis], return_variance=True)

print('MSE: {:.5f}'.format(mse_score['stat']))
print('MAE: {:.5f}'.format(mae_score['stat']))

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Error Variance')

# ax.set_title('Squared Error, Variance')
ax.legend(fontsize=14)
# ax.get_xaxis().set_visible(False)
# ax.get_yaxis().set_visible(False)
ax.grid(True)
plt.show()

save_name = 'exp_empirical_stat_var.png'
fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)

```

### Numerical Uncertainty

```python
xnew = np.linspace(xt.min(), xt.max(), 1000)

abs_error_line = interpolate.interp1d(xt, abs_error['numerical'], kind='quadratic')(xnew)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['numerical'], kind='quadratic')(xnew)

n_samples = len(xt)
pred = np.zeros(n_samples)
var = np.zeros(n_samples)
xt_ = xt[:, np.newaxis]

for i in range(n_samples):
    pred[i], var[i] = gp_numerical.propagate_GA(xt_[i], Sigma)
# pred, var = gp_derivative_full.predict(xt[:, np.newaxis], return_variance=True)

print('MSE: {:.5f}'.format(mse_score['numerical']))
print('MAE: {:.5f}'.format(mae_score['numerical']))

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Error Variance')

# ax.set_title('Squared Error, Variance')
ax.legend(fontsize=14)
# ax.get_xaxis().set_visible(False)
# ax.get_yaxis().set_visible(False)
ax.grid(True)
plt.show()

save_name = 'exp_empirical_numerical_var.png'
fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)

```

### Q Matrix Optimization

```python
xtrain = np.array([1, 5, 3])[:, np.newaxis]
xtest = np.array([1])[:, np.newaxis]
print(xtrain.shape, xtest.shape)
n_samples = xtrain.shape[0]
d_dimensions = xtrain.shape[1]
```

    (3, 1) (1, 1)

```python
xrow = xtrain[:, np.newaxis, :]
xrow = np.tile(xrow, [1, n_samples, d_dimensions])
print(xrow)
print(xrow.shape)
```

    [[[1]
      [1]
      [1]]

     [[5]
      [5]
      [5]]

     [[3]
      [3]
      [3]]]
    (3, 3, 1)

```python
def zterm_matrix(xtrain):
    n_samples, d_dimensions = xtrain.shape

    xrow = xtrain[:, np.newaxis, :]
    xrow = np.tile(xrow, [1, n_samples, d_dimensions])
    zterm = 0.5 * (xrow + np.transpose(xrow, axes=(1,0, 2)))
    return zterm
```

```python
xcol = xrow.copy() #xtrain[:, np.newaxis, :]
xcol = np.transpose(xcol, axes=(1,0, 2))
print(xcol)
print(xcol.shape)

```

    [[[1]
      [5]
      [3]]

     [[1]
      [5]
      [3]]

     [[1]
      [5]
      [3]]]
    (3, 3, 1)

```python
zterm = 0.5 * (xrow + xcol)
zterm = zterm_matrix(xtrain)
print(zterm == zterm2)
print(zterm.shape)
```

```python
zterm = zterm_matrix(xtrain)
zterm = zterm - xtest
print(zterm.shape, zterm.T.shape)
```

    (3, 3, 1) (1, 3, 3)

```python
diag_term = np.array([1.0])
print(zterm.shape, diag_term.shape)
exp_term = zterm.dot(diag_term).dot(zterm.T)
print(exp_term.shape)
```

    (3, 3, 1) (1,)
    (3, 1, 3)

```python

```


```python
%display typeset
```

# Mapas de Dimensão finita
Também conhecidos como equações de diferença são sistemas dinâmicos sobre tempo discreto. Neste Notebook iremos explorar alguns resultados básicos sobre mapas discretos e mapas clássicos em uma e duas dimensões.

Vamos começar com um exemplo simples:

$$U_{n+1}=a sin(U_n), \,\,U_0=U$$

A partir da expressão acima vemos que Mapas são funções de $\mathbb{R}^p \rightarrow \mathbb{R}^p$ que produzem sequências de números da forma $\{U_n\}_{n=0}^{\infty}$. No caso de mapas unidimensionais, $p=1$.

```python
@interact
def oneDmap(a=slider(0.5,3,.1,1.1, label='$a$'), u0=slider(0,1,.1,.1, label='$U_0$'), n=slider(10,300,1,100, label='Iterações:')):
    pts = [(0,a*sin(u0))]
    for i in range(1,n+1):
        nu = a*sin(pts[-1][1])
        pts.append((i,nu))
    p = points(pts,axes_labels=['$n$', '$U_n$'])
    show("Uf= ",pts[-1][1])
    show(p)

```

    Interactive function <function oneDmap at 0x7fa450a5ac10> with 3 widgets
      a: TransformFloatSlider(value=1.1, …

### Mapa quadrático
Como um segundo exemplo, vamos considerar o mapa quadrático:
$$U_{n+1}=a U_n (1-U_n), \,\,\, U_0=U$$

```python
qmap = lambda u,a: a*u*(1-u)
@interact
def quad_map(a=slider(0.5,4.5,.1,1.1, label='$a$'), u0=slider(0,1,.1,.1, label='$U_0$'), n=slider(10,300,1,100, label='Iterações:')):
    pts = [(u0, qmap(u0,a))]
    for i in range(1,n+1):
        nu = qmap(pts[-1][1],a)
        pts.append((i,nu))
    p = points(pts,axes_labels=['$n$', '$U_n$'])
    show("Uf= ",pts[-1][1])
    show(p)
```

    Interactive function <function quad_map at 0x7fa45067e1f0> with 3 widgets
      a: TransformFloatSlider(value=1.1,…

## Diagramas Cobweb (teia de aranha)

Também conhecidos como diagramas de Verhulst, diagramas cobweb nos permitem explorar a dinâmica de mapas discretos. vejamos o exemplo abaixo com um mapa quadrático também conhecido como mapa logístico.

$$U_{n+1} = a U_n (1-U_n)$$

Estes diagramas podem ser construídos utilizado-se os seguintes passos:
1. Encontre o ponto $(U_0, U_1)$;
1. Trace uma linha horizontal deste ponto até encontrar a diagonal no ponto $(U_1,U_1)$;
1. Trace uma linha vertical deste último ponto na diagonal até encontrar a função no ponto $(U_1, U_2)$;
1. Repita os passos 2 e 3 incrementando os indices.

No diagrama de Teia de aranha, um *ponto fixo* aparece como uma **espiral convergente**, Um *ponto instável* como uma **espiral divergente**, Uma *órbita de período 2* aparece como um **retângulo**, *períodos maiores* aparecem como **múltiplos retângulos**. *Atratores caóticos* aparecem como **inúmeros retângulos**.

```python
def cobweb(a_function, start, mask = 0, iterations = 20, xmin = 0, xmax = 1):
    '''
    Returns a graphics object of a plot of the function and a cobweb trajectory starting from the value start.

    INPUT:
        a_function: a function of one variable
        start: the starting value of the iteration
        mask: (optional) the number of initial iterates to ignore
        iterations: (optional) the number of iterations to draw, following the masked iterations
        xmin: (optional) the lower end of the plotted interval
        xmax: (optional) the upper end of the plotted interval

    EXAMPLES:
        sage: f = lambda x: 3.9*x*(1-x)
        sage: show(cobweb(f,.01,iterations=200), xmin = 0, xmax = 1, ymin=0)

    '''
    basic_plot = plot(a_function, xmin = xmin, xmax = xmax, axes_labels=['$U_n$', '$U_{n+1}$'])
    iv = point((start,a_function(start)),color='green', size=30)
    id_plot = plot(lambda x: x, xmin = xmin, xmax = xmax)
    iter_list = []
    current = start
    for i in range(mask):
        current = a_function(current)
    for i in range(iterations):
        iter_list.append([current,a_function(current)])
        current = a_function(current)
        iter_list.append([current,current])
    series = list_plot([[i,pt[0]] for i,pt in enumerate(iter_list)], color='red', legend_label='$U_n$')
    cobweb = line(iter_list, rgbcolor = (1,0,0))
    ga = graphics_array([
        [basic_plot +iv + id_plot + cobweb],
        [series]
    ],)
    ga.show(figsize=[8,10])#basic_plot +iv + id_plot + cobweb
var('x')
@interact
def cobwebber(f_text = input_box(default = "a*x*(1-x)",label = "mapa", type=str),
              a = slider(1,4,0.1,3.8, label='$a$'),
              start_val = slider(0,1,.01,.5,label = 'valor inicial'),
              its = slider([2^i for i in range(0,12)],default = 16, label="iterações")):
    def f(x):
        return eval(f_text.replace('a',str(a)))
    show(cobweb(f, start_val, iterations = its))
```

    Interactive function <function cobwebber at 0x7fa4509b53a0> with 4 widgets
      f_text: TransformText(value='a*x*…

### Diagrama orbital
Vimos que o número de orbitas distintas no modelo logístico varia com o parâmetro $a$. Podemos construir outro diagrama interessante representando o número de órbitas ao longo de um intervalo do parâmetro.

```python
import numpy as np
logmap = lambda a,x: a*x*(1-x)
@interact
def orb_diagram(mask=50, start=0.5, a_min=slider(2.9,4.0,.01,3.4), a_max=slider(3.5,4.0,.01,4),
                ymax = slider(0,1,0.01,1),
               its = slider([2^i for i in range(0,12)],default = 16, label="iterações")):
    func = logmap
    states = []
    for a in np.linspace(a_min,a_max,150, dtype=float):
        iter_list = []
        current = start
        for i in range(mask):
            current = func(a,current)
        for i in range(its):
            iter_list.append([a,func(a, current)])
            current = func(a,current)
#             iter_list.append([a,current])
        states.extend(iter_list)
    p = list_plot(states)
    p.show(figsize=[8,8],ymax=ymax)

orb_diagram(logmap)
```

```python
a_min=slider(3.5,4,.01,3.4)
a_min.value
```

    3.5

## Mapa de Hénon, um mapa bi-dimensional
Stuart e Humphries apresentam no capítulo 1, equação 1.1.3 o mapa de Henón:

\begin{align}
X_{n+1} &= a -bY_n -X^2_n\\
Y_{n+1} &= X_n
\end{align}

```python
@interact
def Henon_map(x0=slider(-0.4,.4,.1,.1, label='$X_0$'),
              y0=slider(-.4,.4,.1,.1, label='$Y_0$'),
              a=slider(0,1,.1,0,label='$a$'),
              b=slider(0,1,.1,1,label='$b$'),
              n=slider(10,1000,50,500, label='Iterações:')):
    pts = [(a-b*y0-x0**2,x0)]
    for i in range(1,n+1):
        xi = a-b*pts[-1][1]-pts[-1][0]**2
        yi = pts[-1][0]
        pts.append((xi,yi))
    p = points(pts, axes_labels=['$X_n$', '$Y_n$'])
    xp = list_plot([[i,pt[0]] for i,pt in enumerate(pts)], color='red', legend_label='$X_n$')
    yp = list_plot([[i,pt[1]] for i,pt in enumerate(pts)], color='green', legend_label='$Y_n$', alpha=.6)
    ga = graphics_array((p,xp+yp))
    ga.show(figsize=[10,5])
```

    Interactive function <function Henon_map at 0x7f4c8e6f6268> with 5 widgets
      x0: TransformFloatSlider(value=0.…

Entretando o Mapa de Hénon [mais conhecido](https://en.wikipedia.org/wiki/H%C3%A9non_map) tem uma formulação ligeiramente diferente:

\begin{align}
X_{n+1} &= 1 -a X^2_n + Y_n \\
Y_{n+1} &= b X_n
\end{align}

É importante mencionar que o sistema acima é a forma como o mapa aparece no [artigo original de Hénon](https://venturi.soe.ucsc.edu/sites/default/files/Henon1976.pdf) de 1976.

```python
@interact
def Henon_map_2(x0=slider(-0.4,.4,.1,0, label='$X_0$'),
              y0=slider(-.4,.4,.1,0, label='$Y_0$'),
              a=slider(0,2,.1,1.2,label='$a$'),
              b=slider(0,1,.1,0.3,label='$b$'),
              n=slider(10,1000,50,500, label='Iterações:')):
    pts = [(a-b*y0-x0**2,x0)]
    for i in range(1,n+1):
        xi = 1-a* pts[-1][0]**2 + pts[-1][1]
        yi = b*pts[-1][0]
        pts.append((xi,yi))
    p = points(pts, axes_labels=['$X_n$', '$Y_n$'])
    xp = list_plot([[i,pt[0]] for i,pt in enumerate(pts)], color='red', legend_label='$X_n$')
    yp = list_plot([[i,pt[1]] for i,pt in enumerate(pts)], color='green', legend_label='$Y_n$', alpha=.3)
    ga = graphics_array((p,xp+yp))
    ga.show(figsize=[10,5])
```

    Interactive function <function Henon_map_2 at 0x7f4c8e28a7b8> with 5 widgets
      x0: TransformFloatSlider(value=…

## O Semigrupo $S^n$.
Vamos introduzir o subgrupo $S^n$ como uma notação para a realização de n passos(iterações) do mapa.

Dado um sistema dinâmico sobre um conjunto $E$. Podemos definir o semigrupo de evolução $S^n:E \rightarrow E$ como sendo o operador tal que $U_n=S^n U_0$. Este operador tem as seguintes propriedades:
1. $U_{n+m}=S^nU_m=S^mU_n=S^{n+m}U_0$, para todo $n,m \geq 0$;
1. $S^0= I $, a identidade.

o semigrupo $S^n$ é comumente chamado de *mapa de evolução*.

## Conjuntos Limite (Limit sets)
As seguintes perguntas são relevantes para sistemas dinâmicos de qualquer natureza:
 - Quais os seus equilíbrios?
 Pontos de equilíbrio são estados de onde o sistema não sairá a menos que perturbado. No caso dos mapas discretos estes equilíbrios são chamados de conjuntos limites, pois podem ser definidos por um conjunto de pontos ao invés de um único.
 - Quais condições iniciais levam a que conjuntos limites?
 ou seja, que conjuntos de pontos constituem sua **bacia de atração**?

Considere o seguinte mapa: $$U_{n+1}=U_n^2,\,\,U_0=U$$

Este é um sistema dinâmico em $\mathbb{R}^+$. É óbvio que se $U<1$, $U_n \rightarrow 0$ quando $n\rightarrow \infty$. Logo, $0$ é um ponto fixo. O conjunto $[0,1)$ é a bacia de atração de 0. Se $U=1$ então $U_n=1, \forall n$. Logo $1$ também é um ponto fixo. Por fim, se $U>1$ então $U_n \rightarrow \infty$ quando $n\rightarrow \infty$. Portanto $\infty$ é um equilíbrio do sistema, e o conjunto $(1,\infty]$ é sua bacia de atração.

Encontrar os equilíbrios de uma Mapa nem sempre é tão simples. Consider este outro mapa:
$$U_{n+1}=a U -U^3, \, U_0=U$$

```python
@interact
def cobwebber(f_text = input_box(default = "a*x - x**3",label = "mapa", type=str),
              a = slider(1,4,0.1,3, label='$a$'),
              start_val = slider(0,2,.01,n(sqrt(3)),label = 'valor inicial'),
              its = slider([2^i for i in range(0,12)],default = 16, label="iterações")):
    def f(x):
        return eval(f_text.replace('a',str(a)))
    show(cobweb(f, start_val, iterations = its,xmin=-2,xmax=2))
```

    Interactive function <function cobwebber at 0x7f4c8e6f6a60> with 4 widgets
      f_text: TransformText(value='a*x …

## Estabilidade
Pontos fixos por definição são valores de $U$ para os quais $U_{n+1}=U_n$. Logo quando existirem sempre pertencerão à diagonal do gráfico de cobweb.

Para uma definição mais formal, considere o seguinte mapa linear:
$$U_{n+1}=AU_n.$$

$A$ tem autovalores $\{\lambda_i\}_{i=1}^l$, onde $l\leq p$. Logo,
1. A origem é assintoticamente estável se e somente se $|\lambda_i|<1$. para todos os $i$.
2. Se $|\lambda_i|\leq1$ para todo $i$ e os autovalores iguais a um são "não-defeituosos", então a origem é estável.

Para mapas não lineares pode-se aplicar técnicas de linearização local nos pontos fixos, e aplicar os critérios acima.

### Analisando o Mapa Logístico
Vamos revisitar o Mapa logístico e analisar seus pontos fixos. Vamos denotar por $x^*$ os pontos fixos.
$$x_{n+1}=rx_n(1-x_n)$$

Os pontos fixos satisfazem $x^* = f(x^*)=rx^*(1-x^*)$. Logo $x^*=0$ ou $x^*=1-\frac{1}{r}$

```python
var('r x')
solve(x==r*x*(1-x),x)
```

<html></html>

A estabilidade dos pontos fixos depende da derivada de $f(x^*)$, $f'(x^*) = r - 2rx^*$. NO caso de $x^*=0$, $f'(0)=r$ este ponto fixo é estável se $r<1$ e instável se $r>1$. Para o segundo ponto fixo, $f'(x^*)=r-2r(1-1/r)=2-r$. Logo o segundo ponto fixo é estável quando $1<r<3$ e instável quando $r>3$.

```python

```


# From Oliver Durr

## Variational Autoencoder (VAE)
A tutorial with code for a VAE as described in [Kingma and Welling, 2013](http://arxiv.org/abs/1312.6114). A talk with more details was given at the [DataLab Brown Bag Seminar](https://home.zhaw.ch/~dueo/bbs/files/vae.pdf).

Much of the code was taken, from https://jmetzen.github.io/2015-11-27/vae.html. However, I tried to focus more on the mathematical understanding, not so much on design of the algorithm.

### Some theoretical considerations

#### Outline
Situation: $x$ is from a high-dimensional space and $z$ is from a low-dimensional (latent) space, from which we like to reconstruct $p(x)$.

We consider a parameterized model $p_\theta(x|z)$ (with parameter $\theta$), to construct x for a given value of $z$. We build this model:

* $p_\theta(x | z)$ with a neural network determening the parameters $\mu, \Sigma$ of a Gaussian (or as done here with a Bernoulli-Density).

#### Inverting $p_\theta(x | z)$

The inversion is not possible, we therefore approximate $p(z|x)$ by $q_\phi (z|x)$ again a combination of a NN determening the parameters of a Gaussian

* $q_\phi(z | x)$ with a neural network + Gaussian

#### Training

We train the network treating it as an autoencoder.

#### Lower bound of the Log-Likelihood
The likelihood cannot be determined analytically. Therefore, in a first step we derive a lower (variational) bound $L^{v}$ of the log likelihood, for a given image. Technically we assume a discrete latent space. For a continous case simply replace the sum by the appropriate integral over the respective densities. We replace the inaccessible conditional propability $p(z|x)$ with an approximation $q(z|x)$ for which we later use a neural network topped by a Gaussian.

\begin{align}
L & = \log\left(p(x)\right) &\\
  & = \sum_z q(z|x) \; \log\left(p(x)\right) &\text{multiplied with 1 }\\
  & = \sum_z q(z|x) \; \log\left(\frac{p(z,x)}{p(z|x)}\right) &\\
  & = \sum_z q(z|x) \; \log\left(\frac{p(z,x)}{q(z|x)} \frac{q(z|x)}{p(z|x)}\right) &\\
  & = \sum_z q(z|x) \; \log\left(\frac{p(z,x)}{q(z|x)}\right) + \sum_z q(z|x) \; \log\left(\frac{q(z|x)}{p(z|x)}\right) &\\
  & = L^{\tt{v}} + D_{\tt{KL}} \left( q(z|x) || p(z|x) \right) &\\
  & \ge L^{\tt{v}} \\
\end{align}

The KL-Divergence $D_{\tt{KL}}$ is always positive, and the smaller the better $q(z|x)$ approximates $p(z|x)$

### Rewritting  $L^\tt{v}$
We split $L^\tt{v}$ into two parts.

\begin{align}
L^{\tt{v}} & = \sum_z q(z|x) \; \log\left(\frac{p(z,x)}{q(z|x)}\right)  & \text{with} \;\;p(z,x) = p(x|z) \,p(z)\\
  & =  \sum_z q(z|x) \; \log\left(\frac{p(x|z) p(z)}{q(z|x)}\right)  &\\
  & =  \sum_z q(z|x) \; \log\left(\frac{p(z)}{q(z|x)}\right)  + \sum_z q(z|x) \; \log\left(p(x|z)\right) &\\
  & =  -D_{\tt{KL}} \left( q(z|x) || p(z) \right)  +  \mathbb{E}_{q(z|x)}\left( \log\left(p(x|z)\right)\right) &\text{putting in } x^{(i)} \text{ for } x\\
  & =  -D_{\tt{KL}} \left( q(z|x^{(i)}) || p(z) \right)  +  \mathbb{E}_{q(z|x^{(i)})}\left( \log\left(p(x^{(i)}|z)\right)\right) &\\
\end{align}

Approximating $\mathbb{E}_{q(z|x^{(i)})}$ with sampling form the distribution $q(z|x^{(i)})$

#### Sampling
With $z^{(i,l)}$ $l = 1,2,\ldots L$ sampled from $z^{(i,l)} \thicksim q(z|x^{(i)})$
\begin{align}
L^{\tt{v}} & = -D_{\tt{KL}} \left( q(z|x^{(i)}) || p(z) \right)
+  \mathbb{E}_{q(z|x^{(i)})}\left( \log\left(p(x^{(i)}|z)\right)\right) &\\
L^{\tt{v}} & \approx -D_{\tt{KL}} \left( q(z|x^{(i)}) || p(z) \right)
+  \frac{1}{L} \sum_{i=1}^L \log\left(p(x^{(i)}|z^{(i,l)})\right) &\\
\end{align}

#### Calculation of $D_{\tt{KL}} \left( q(z|x^{(i)}) || p(z) \right)$
TODO

```python

```