# Introduçao a Python

### Variáveis e Estruturas Básicas

> Material didático baseado no livro:  *Python Programming and Numerical Methods - A Guide for Engineers and Scientists by Qingkai Kong Timmy Siauw and Alexandre Bayen. Imprint: Academic Press*.

Variáveis são estruturas usadas em Python para armazenar dados.  No entanto, os dados podem assumir várias formas.  Por exemplo, os dados podem ser números, palavras ou ter uma estrutura mais complicada.  É natural que o Python tenha diferentes tipos de variáveis para armazenar diferentes tipos de dados.  Nesta seção, você aprenderá como criar e manipular os tipos de variáveis mais comuns do Python.

De maneira geral, uma *variável* é uma sequência de caracteres ou números associados a uma informação. Veja alguns exemplos para o caso numérico:

```python
x=2
```

```python
y=3.5
```

```python
x+y
```

    5.5

Para obter propriedades de váriaveis use o comando abaixo:

```python
%whos
```

    Variable   Type     Data/Info
    -----------------------------
    x          int      2
    y          float    3.5

As variáveis podem conter caracteres, sendo do tipo *String*.

```python
s='Física'
w = ' de Plasmas'
```

```python
p = s+w

print(p)
```

    Física de Plasmas

Strings também têm índices para indicar a localização de cada caractere, para que possamos encontrar facilmente algum caractere. O índice da posição começa com 0, conforme mostrado na imagem a seguir.

```python
w = "Hello World"

```

```python
type(w)

```

    str

```python
len(w)
```

    11

```python
w[:-2]
```

    'Hello Wor'

Em Python, um objeto possui vários métodos que podem ser usados para manipulá-lo (falaremos mais sobre programação orientada a objetos posteriormente). A maneira de obter acesso aos vários métodos é usar este padrão “string.method_name”.

```python
w.upper()

```

    'HELLO WORLD'

```python
w.count("l")

```

    3

```python
w.replace("World", "Brasília")

```

    'Hello Brasília'

Existem diferentes maneiras de pré-formatar uma string. Aqui, apresentamos duas maneiras de fazer isso. Por exemplo, se tivermos duas variáveis nome e país e quisermos imprimi-las em uma frase, podemos fazer o seguinte:

```python
materia = "Física de plasmas"
adjetivo = 'legal'

print("%s é muito %s!"%(materia, adjetivo))
```

    Física de plasmas é muito legal!

```python
print(f"{materia} é muito {adjetivo}!")

```

    Física de plasmas é muito legal!

## Listas

Agora, vamos ver uma estrutura de dados sequenciais mais versátil em Python - as Listas. A maneira de defini-la é usar um par de colchetes [], e os elementos dentro dela são separados por vírgulas. Uma lista pode conter qualquer tipo de dados: numéricos, strings ou outros tipos. Por exemplo:

```python
list_1 = [1, 2, 3]
list_1
```

    [1, 2, 3]

```python
list_2 = ['Hello', 'World']
list_2
```

    ['Hello', 'World']

```python
list_3 = [1, 2, 3, 'Apple', 'orange']
list_3
```

    [1, 2, 3, 'Apple', 'orange']

```python
list_4 = [list_1, list_2]
list_4
```

    [[1, 2, 3], ['Hello', 'World']]

A maneira de obter o elemento na lista é semelhante às strings, veja a figura a seguir para o índice

```python
list_3[2]
```

    3

```python
list_3[:3]

```

    [1, 2, 3]

```python
list_3[-1]

```

    'orange'

```python
list_4[0]

```

    [1, 2, 3]

```python
list_5 = []
list_5.append(5)
list_5
```

    [5]

```python
5 in list_5

```

    True

```python
list('Hello World')

```

    ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd']

```python

```

## Tuplas

Vamos aprender mais uma estrutura de dados de sequência diferente em Python - as Tuplas. Geralmente uma tupla é definida usando um par de parênteses (), e seus elementos são separados por vírgulas. Por exemplo:

```python
tuple_1 = (1, 2, 3, 2)
tuple_1
```

    (1, 2, 3, 2)

```python
len(tuple_1)
```

    4

```python
tuple_1[1:4]

```

    (2, 3, 2)

```python
tuple_1.count(2)

```

    2

Você pode perguntar: qual é a diferença entre listas e tuplas? Se eles são semelhantes entre si, por que precisamos de outra estrutura de dados de sequência?

As tuplas são criadas por uma razão. Segundo a documentação do Python:

> Embora as tuplas possam parecer semelhantes a listas, elas são freqüentemente usadas em diferentes situações e para diferentes propósitos. As tuplas são imutáveis e geralmente contêm uma sequência heterogênea de elementos que são acessados via desempacotamento ou indexação. As listas são mutáveis e seus elementos geralmente são homogêneos e são acessados pela iteração da lista

```python
list_1 = [1, 2, 3]
list_1[2] = 1
list_1
```

    [1, 2, 1]

```python
tuple_1[2] = 1

```

O que significa heterogêneo? As tuplas geralmente contêm uma sequência heterogênea de elementos, enquanto as listas geralmente contêm uma sequência homogênea. Vamos ver um exemplo: temos uma lista que contém frutas diferentes. Normalmente o nome dos frutos pode ser armazenado em uma lista, uma vez que são homogêneos. Agora queremos ter uma estrutura de dados para armazenar quantas frutas temos para cada tipo, normalmente é aqui que as tuplas entram, já que o nome da fruta e o número são heterogêneos. Como (‘maçã’, 3), o que significa que temos 3 maçãs.

```python
# a fruit list
['apple', 'banana', 'orange', 'pear']
```

    ['apple', 'banana', 'orange', 'pear']

```python
# a list of (fruit, number) pairs
[('apple', 3), ('banana', 4) , ('orange', 1), ('pear', 4)]
```

    [('apple', 3), ('banana', 4), ('orange', 1), ('pear', 4)]

As tuplas podem ser acessadas descompactando. Neste caso, é necessário que o número de variáveis no lado esquerdo seja igual ao número de elementos no lado direito.

```python
a, b, c = list_1
print(a, b, c)
```

    1 2 1

## Sets (conjuntos)

Outro tipo de dados em Python são conjuntos (sets). É um tipo de variável que pode armazenar uma coleção não ordenada sem elementos duplicados. É também suporte para as operações matemáticas como união, interseção, diferença e diferença simétrica. É definido usando um par de chaves {} e seus elementos são separados por vírgulas.

```python
{3, 3, 2, 3, 1, 4, 5, 6, 4, 2}
```

    {1, 2, 3, 4, 5, 6}

Um uso rápido disso é descobrir os elementos únicos em uma string, lista ou tupla.

**Exemplo**: Encontre os elementos únicos na lista [1, 2, 2, 3, 2, 1, 2].

```python
set_1 = set([1, 2, 2, 3, 2, 1, 2])
set_1
```

    {1, 2, 3}

```python
set_2 = set((2, 4, 6, 5, 2))
set_2
```

    {2, 4, 5, 6}

```python
set('Banana')

```

    {'B', 'a', 'n'}

**Exemplo**: Encontre a união entre os conjuntos set_1 e set_2:

```python
print(set_1)
print(set_2)
```

    {1, 2, 3}
    {2, 4, 5, 6}

```python
set_1.union(set_2)
```

    {1, 2, 3, 4, 5, 6}

**Exemplo**: Encontre a intersecção entre os conjuntos set_1 e set_2:

```python
set_1.intersection(set_2)
```

    {2}

**Exemplo:** O conjunto set_1 é um subconjunto de  {1, 2, 3, 3, 4, 5}?

```python
set_1.issubset({1, 2, 3, 3, 4, 5})
```

    True

## Dicionários (dictionaries)

Introduzimos vários tipos de dados sequenciais nas seções anteriores. Agora vamos apresentar a você um tipo novo e útil - os **Dicionários**. É um tipo de mapeamento, o que o torna diferente das variáveis que falamos antes. Em vez de usar uma sequência de números para indexar os elementos (como listas ou tuplas), os dicionários são indexados por chaves, que podem ser uma string, um número ou mesmo uma tupla (mas não uma lista). Um dicionário é um par de valores-chave e cada chave é mapeada para um valor correspondente. É definido usando um par de chaves {}, enquanto os elementos são uma lista de pares chave: valor separados por vírgulas (observe o par chave: valor é separado por dois pontos, com a chave na frente e o valor no final).

```python
dict_1 = {'apple':3, 'oragne':4, 'pear':2}
dict_1
```

    {'apple': 3, 'oragne': 4, 'pear': 2}

Dentro de um dicionário, os elementos são armazenados sem ordem, portanto, você não pode acessar um dicionário baseado em uma sequência de números de índice. Para obter acesso a um dicionário, precisamos usar a chave do elemento - dicionário [chave].

```python
dict_1['apple']
```

    3

Podemos obter todas as chaves de um dicionário usando o método *keys* ou todos os valores usando o método *values*.

```python
dict_1.keys()
```

    dict_keys(['apple', 'oragne', 'pear'])

```python
dict_1.values()
```

    dict_values([3, 4, 2])

```python
len(dict_1)
```

    3

Podemos definir um dicionário vazio e preencher o elemento mais tarde. Ou podemos transformar uma lista de tuplas com pares (chave, valor) em um dicionário.

```python
school_dict = {}
school_dict['UC Berkeley'] = 'USA'
school_dict
```

    {'UC Berkeley': 'USA'}

```python
school_dict['Oxford'] = 'UK'
school_dict
```

    {'Oxford': 'UK', 'UC Berkeley': 'USA'}

```python
dict([("UC Berkeley", "USA"), ('Oxford', 'UK')])
```

    {'Oxford': 'UK', 'UC Berkeley': 'USA'}

```python
"UC Berkeley" in school_dict
```

    True

```python
"Harvard" not in school_dict
```

    True

```python
list(school_dict)
```

    ['UC Berkeley', 'Oxford']

```python

```

# Numpy Arrays

Agora, vamos apresentar a maneira mais comum de lidar com matrizes em Python usando o *módulo Numpy*. O Numpy é provavelmente o módulo de computação numérica mais fundamental do Python.

Para usar o módulo Numpy, precisamos importá-lo primeiro. Uma forma convencional de importá-lo é usar “np” como um nome abreviado.

```python
import numpy as np
```

Para definir uma matriz em Python, você pode usar a função *np.array* para converter uma lista.

**Exemplo:**
Construa as seguintes matrizes usando o numpy
\begin{equation}
x = \begin{pmatrix}
1 & 4 & 3 \\
\end{pmatrix}
\end{equation}

\begin{equation}
y = \begin{pmatrix}
1 & 4 & 3 \\
9 & 2 & 7 \\
\end{pmatrix}
\end{equation}

```python
x = np.array([1, 4, 3])
x
```

    array([1, 4, 3])

```python
y = np.array([[1, 4, 3], [9, 2, 7]])
y
```

    array([[1, 4, 3],
           [9, 2, 7]])

Uma matriz 2-D pode usar listas aninhadas para representar, com a lista interna representando cada linha.

Muitas vezes é útil saber o tamanho ou comprimento de um array. O atributo de forma da matriz é chamado em uma matriz M e retorna uma matriz 2 × 3 onde o primeiro elemento é o número de linhas na matriz M e o segundo elemento é o número de colunas em M. Observe que a saída do atributo de forma é uma tupla. O atributo de tamanho é chamado em uma matriz M e retorna o número total de elementos na matriz M.

Encontre as linhas, colunas e o tamanho total da matriz y.

```python
y.shape
```

    (2, 3)

```python
y.size
```

    6

Você pode notar a diferença de que usamos apenas *y.shape* em vez de *y.shape()*, porque a forma é um atributo e não um método neste objeto de matriz. Apresentaremos mais sobre a programação orientada a objetos posteriormente. Por enquanto, você precisa lembrar que quando chamamos um método em um objeto, precisamos usar os parênteses, enquanto o atributo não.

Muitas vezes, queremos gerar arrays que possuam uma estrutura ou padrão. Por exemplo, podemos desejar criar a matriz $z = [1 2 3… 2000]$. Seria muito complicado digitar toda a descrição de $z$ no Python. Para gerar matrizes ordenadas e espaçadas uniformemente, é útil usar a função *arange* no Numpy.

**Exemplo:** Crie uma matriz $z$ de 1 a 2000 com um incremento 1

```python
z = np.arange(1, 2000, 1)
z
```

    array([   1,    2,    3, ..., 1997, 1998, 1999])

Os primeiros dois números de $z$ são o início e o fim da sequência e o último é o incremento. Como é muito comum ter um incremento de 1, se um incremento não for especificado, o Python usará um valor padrão de 1. Portanto, *np.arange(1, 2000)* terá o mesmo resultado que *np.arange(1,2000,1)*. Incrementos negativos ou não inteiros também podem ser usados. Se o incremento “perder” o último valor, ele somente se estenderá até o valor imediatamente anterior ao valor final. Por exemplo, $x = np.arange(1,8,2)$ seria $[1, 3, 5, 7]$.

**Exemplo:** Gere uma matriz com [0.5, 1, 1.5, 2, 2.5].

```python
np.arange(0.5, 3, 0.5)
```

    array([0.5, 1. , 1.5, 2. , 2.5])

Às vezes, queremos garantir um ponto inicial e final para uma matriz, mas ainda ter elementos espaçados uniformemente. Por exemplo, queremos um array que comece em 1, termine em 8 e tenha exatamente 10 elementos. Para este propósito, você pode usar a função *np.linspace*. O *linspace* aceita três valores de entrada separados por vírgulas. Portanto, *A = linspace (a, b, n)* gera uma matriz de *n* elementos igualmente espaçados começando em *a* e terminando em *b*.

**Exemplo:** Use o *linspace* para gerar uma matriz começando em 3, terminando em 9 e contendo 10 elementos

```python
np.linspace(3, 9, 10)
```

    array([3.        , 3.66666667, 4.33333333, 5.        , 5.66666667,
           6.33333333, 7.        , 7.66666667, 8.33333333, 9.        ])

Obter acesso ao array numpy 1D é semelhante ao que descrevemos para listas ou tuplas, tem um índice para indicar a localização. Por exemplo:

```python
# get the 2nd element of x
x[1]
```

    4

```python
# get all the element after the 2nd element of x
x[1:]
```

    array([4, 3])

```python
# get the last element of x
x[-1]
```

    3

Para arrays em 2D, é um pouco diferente, já que temos linhas e colunas. Para obter acesso aos dados em um array 2D M, precisamos usar $M[r, c]$, onde a linha *r* e a coluna *c* são separadas por vírgula. Essa é a indexação da array. O *r* e *c* podem ser um único número, uma lista e assim por diante. Se você pensar apenas no índice da linha ou no índice da coluna, ele será semelhante a uma array 1D. Vamos usar $y = \begin{pmatrix}
1 & 4 & 3 \\
9 & 2 & 7 \\
\end{pmatrix}
$ como exemplo.

**Exemplo:** Obtenha o elemento na primeira linha e na 2ª coluna da matriz y.

```python
y[0,1]
```

    4

```python
# última coluna:
y[:, -1]
```

    array([3, 7])

```python
# terceira coluna
y[:, [0, 2]]
```

    array([[1, 3],
           [9, 7]])

Existem algumas arrays predefinidas que são úteis. Por exemplo, o *np.zeros*, *np.ones* e *np.empty* são 3 funções úteis. Vamos ver os exemplos.

```python
np.zeros((3, 5))
```

    array([[0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.]])

```python
np.ones((5, 3))
```

    array([[1., 1., 1.],
           [1., 1., 1.],
           [1., 1., 1.],
           [1., 1., 1.],
           [1., 1., 1.]])

A forma da matriz é definida em uma tupla com a linha como o primeiro item e a coluna como o segundo. Se você só precisa de uma matriz 1D,use apenas um número como entrada: $np.ones(5)$.

**Exemplo:** Gere uma array vazia 1D com 3 elementos.

```python
np.empty(3)
```

    array([0.75, 0.75, 0.  ])

A matriz vazia não está realmente vazia, ela é preenchida com números pequenos aleatórios.

Você pode reatribuir um valor de uma matriz usando a indexação de matriz e o operador de atribuição. Você pode reatribuir vários elementos a um único número usando a indexação de matriz no lado esquerdo. Você também pode reatribuir vários elementos de uma matriz, desde que o número de elementos sendo atribuídos e o número de elementos atribuídos sejam os mesmos. Você pode criar uma matriz usando a indexação de matriz.

**Exemplo:** Seja a = [1, 2, 3, 4, 5, 6]. Reatribua o quarto elemento de A para 7. Reatribua o primeiro, o segundo e o terceiro elemento para 1. Reatribua o segundo, terceiro e quarto elementos para 9, 8 e 7.

```python
a = np.arange(1, 7)
a
```

    array([1, 2, 3, 4, 5, 6])

```python
a[3] = 7
a
```

    array([1, 2, 3, 7, 5, 6])

```python
a[:3] = 1
a
```

    array([1, 1, 1, 7, 5, 6])

```python
a[1:4] = [9, 8, 7]
a
```

    array([1, 9, 8, 7, 5, 6])

**Exemplo:** Crie uma matriz de zeros com a forma 2 por 2 e defina $b = \begin{pmatrix}
1 & 2 \\
3 & 4  \\
\end{pmatrix}$ usando a indexação de matriz.

```python
b = np.zeros((2, 2))
b[0, 0] = 1
b[0, 1] = 2
b[1, 0] = 3
b[1, 1] = 4
b
```

    array([[1., 2.],
           [3., 4.]])

Embora você possa criar uma matriz do zero usando a indexação, não é aconselhável. Isso pode confundir você e os erros serão mais difíceis de encontrar em seu código posteriormente. Por exemplo, b[1, 1] = 1 dará o resultado $b = \begin{pmatrix}
0 & 0 \\
0 & 1  \\
\end{pmatrix}$, o que é estranho porque b[0, 0], b[0, 1] e b[1, 0] nunca foram especificados.

A aritmética básica é definida para matrizes. No entanto, existem operações entre um escalar (um único número) e uma matriz e operações entre duas matrizes. Começaremos com operações entre um escalar e um array. Para ilustrar, seja *c* um escalar e *b* uma matriz.

$b + c$, $b - c$, $b * c$ e $b/c$ adiciona *a* a cada elemento de *b*, subtrai *c* de cada elemento de *b*, multiplica cada elemento de *b* por *c* e divide cada elemento de *b* por *c*, respectivamente.

**Exemplo** Seja $b = \begin{pmatrix}
1 & 2 \\
3 & 4  \\
\end{pmatrix}$. Adicione e subtraia 2 de *b*. Multiplique e divida *b* por 2. Eleve ao quadrado todos os elementos de *b*. Seja *c* um escalar. Por conta própria, verifique a reflexividade da adição e multiplicação escalar: $b+c=c+b$ e $cb=bc$.

```python
b + 2
```

    array([[3., 4.],
           [5., 6.]])

```python
b - 2
```

    array([[-1.,  0.],
           [ 1.,  2.]])

```python
2 * b
```

    array([[2., 4.],
           [6., 8.]])

```python
b**2
```

    array([[ 1.,  4.],
           [ 9., 16.]])

Descrever operações entre duas matrizes é mais complicado. Sejam *b* e *d* duas matrizes do mesmo tamanho. A operação $b - d$ pega cada elemento de *b* e subtrai o elemento correspondente de *d*. Da mesma forma, $b + d$ adiciona cada elemento de $d$ ao elemento correspondente de $b$.

**Exemplo:** Seja $b = \begin{pmatrix}
1 & 2 \\
3 & 4  \\
\end{pmatrix}$ e $d = \begin{pmatrix}
3 & 4 \\
5 & 6  \\
\end{pmatrix}$. Calcule $b + d$ e $b - d$.

```python
b = np.array([[1, 2], [3, 4]])
d = np.array([[3, 4], [5, 6]])
```

```python
b + d
```

    array([[ 4,  6],
           [ 8, 10]])

```python
b + d
```

    array([[ 4,  6],
           [ 8, 10]])

```python
b - d
```

    array([[-2, -2],
           [-2, -2]])

Existem dois tipos diferentes de multiplicação (e divisão) de matrizes. Há multiplicação de matriz elemento por elemento e multiplicação de matriz padrão. Por enquanto, mostraremos apenas como a multiplicação e a divisão da matriz elemento por elemento funcionam. A multiplicação de matrizes padrão será descrita posteriormente.  Para as matrizes b e d do mesmo tamanho, b * d pega cada elemento de *b* e os multiplica pelo elemento correspondente de *d*. O mesmo é válido para / e **.

**Exemplo** Calcule $b * d$, $b / d$ e $b^d$ [use ($b**d$) para o último].

```python
b * d
```

    array([[ 3,  8],
           [15, 24]])

```python
b / d
```

    array([[0.33333333, 0.5       ],
           [0.6       , 0.66666667]])

```python
b**d
```

    array([[   1,   16],
           [ 243, 4096]])

A transposta de uma matriz, b, é uma matriz, d, onde b [i, j] = d [j, i]. Em outras palavras, a transposta troca as linhas e colunas de b. Você pode transpor uma matriz em Python usando o método de matriz T.

**Exemplo:** Calcule a transposição da matriz b.

```python
b.T
```

    array([[1, 3],
           [2, 4]])

O Numpy tem muitas funções aritméticas, como sin, cos, etc., que podem usar arrays como argumentos de entrada. A saída é a função avaliada para cada elemento da matriz de entrada. Uma função que recebe uma matriz como entrada e executa a função nela é considerada vetorizada.

**Exemplo:** Calcule *np.sqrt* para x = [1, 4, 9, 16].

```python
x = [1, 4, 9, 16]
np.sqrt(x)
```

    array([1., 2., 3., 4.])

As operações lógicas são definidas apenas entre um escalar e uma array e entre duas arrays do mesmo tamanho. Entre um escalar e um array, a operação lógica é conduzida entre o escalar e cada elemento da array. Entre duas arrays, a operação lógica é conduzida elemento por elemento.

**Exemplo:** Verifique quais elementos da matriz x = [1, 2, 4, 5, 9, 3] são maiores que 3. Verifique quais elementos em x são maiores do que o elemento correspondente em y = [0, 2, 3, 1, 2 , 3].

```python
x = np.array([1, 2, 4, 5, 9, 3])
y = np.array([0, 2, 3, 1, 2, 3])
```

```python
x > 3
```

    array([False, False,  True,  True,  True, False])

```python
x > y
```

    array([ True, False,  True,  True,  True, False])

Python pode indexar elementos de uma matriz que satisfaça uma expressão lógica.

**Exemplo:** Seja *x* o mesmo array do exemplo anterior. Crie uma variável *y* que contenha todos os elementos de *x* estritamente maiores que 3. Atribua todos os valores de *x* maiores que 3, o valor 0.

```python
y = x[x > 3]
y
```

    array([4, 5, 9])

```python
x[x > 3] = 0
x
```

    array([1, 2, 0, 0, 0, 3])

# Sumário

1. Armazenar, recuperar e manipular informações e dados é importante em qualquer campo da ciência e da engenharia.

2. Variáveis são uma ferramenta importante para lidar com valores de dados.

3. Existem diferentes tipos de dados para armazenar informações em Python: int, float, boolean para valores únicos e strings, listas, tuplas, sets, dicionários para dados sequenciais.

4. O array Numpy é um objeto poderoso muito usado na computação científica.

```python

```


```python
%load_ext autoreload
%autoreload
from simba import transfer_function_to_graph, tf2rss, adiabatically_eliminate
from sympy import symbols, simplify, Matrix, sqrt, conjugate, lambdify
```

```python
from simba.utils import construct_permutation_matrix
```

```python
construct_permutation_matrix(6)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 1 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0\\0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1\end{matrix}\right]$

Tuned cavity

```python
s = symbols('s')
gamma_f = symbols('gamma_f', real=True, positive=True)
tf = (s + gamma_f) / (s - gamma_f)
split_network = tf2rss(tf).to_slh().split()
gamma, = split_network.aux_coupling_constants
```

    0

```python
split_network.state_vector
```

$\displaystyle \left[\begin{matrix}a\\\overline{a}\\a'\\\overline{a'}\\ain\\\overline{ain}\\aout\\\overline{aout}\end{matrix}\right]$

```python
split_network.dynamical_matrix.eqns
```

$\displaystyle \left[\begin{matrix}- \frac{\sqrt{\gamma} \sqrt{\gamma_{f}} a'}{s}\\- \frac{\sqrt{\gamma} \sqrt{\gamma_{f}} \overline{a'}}{s}\\\frac{\sqrt{\gamma} \sqrt{\gamma_{f}} a}{s} - \frac{\sqrt{2} \sqrt{\gamma} ain}{s} + \frac{\gamma a'}{s}\\\frac{\sqrt{\gamma} \sqrt{\gamma_{f}} \overline{a}}{s} - \frac{\sqrt{2} \sqrt{\gamma} \overline{ain}}{s} + \frac{\gamma \overline{a'}}{s}\\0\\0\\- \sqrt{2} \sqrt{\gamma} a' + ain\\- \sqrt{2} \sqrt{\gamma} \overline{a'} + \overline{ain}\end{matrix}\right]$

```python
tf = split_network.tfm.open_loop('ain', 'aout')
tf
```

$\displaystyle \frac{\gamma \gamma_{f} + \gamma s + s^{2}}{\gamma \gamma_{f} - \gamma s + s^{2}}$

```python
adiabatically_eliminate(tf, gamma).simplify()
```

$\displaystyle \frac{\gamma_{f} + s}{\gamma_{f} - s}$

```python
tf = split_network.tfm.open_loop('a', 'aout').simplify()
tf
```

$\displaystyle \frac{\sqrt{2} \gamma \sqrt{\gamma_{f}}}{\gamma - s}$

```python
adiabatically_eliminate(tf, gamma).simplify()
```

$\displaystyle \sqrt{2} \sqrt{\gamma_{f}}$

```python
tf = split_network.tfm.open_loop('ain', 'a').simplify()
tf
```

$\displaystyle \frac{\sqrt{2} \gamma \sqrt{\gamma_{f}}}{\gamma \gamma_{f} - \gamma s + s^{2}}$

```python
adiabatically_eliminate(tf, gamma).simplify()
```

$\displaystyle \frac{\sqrt{2} \sqrt{\gamma_{f}}}{\gamma_{f} - s}$

```python
split_network.interaction_hamiltonian.h
```

$\displaystyle \left[\begin{matrix}0 & 0 & i \sqrt{\gamma} \sqrt{\gamma_{f}} & 0\\0 & 0 & 0 & - i \sqrt{\gamma} \sqrt{\gamma_{f}}\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

First looking at passive realisation of coupled cavity setup with coupling constant $g = 0$

```python
s = symbols('s')
gamma_f, omega_s = symbols('gamma_f omega_s', real=True, positive=True)
tf = (s**2 + s * gamma_f + omega_s**2) / (s**2 - s * gamma_f + omega_s**2)

transfer_function_to_graph(tf, 'passive_coupled_cavity.png', layout='dot')
```

    0
    wrote passive_coupled_cavity.png

```python
split_network = tf2rss(tf).to_slh().split()
```

    0

```python
h_int = split_network.interaction_hamiltonian
h_int.expr.simplify()
```

$\displaystyle i \left(- \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} a_{1} \overline{a'_{1}} + \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} \overline{a_{1}} a'_{1} + \omega_{s} a_{1} \overline{a_{2}} - \omega_{s} \overline{a_{1}} a_{2}\right)$

```python
split_network.interaction_hamiltonian.h
```

$\displaystyle \left[\begin{matrix}0 & 0 & - i \omega_{s} & 0 & i \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0 & 0\\0 & 0 & 0 & i \omega_{s} & 0 & - i \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right]$

```python
h_int.states
```

$\displaystyle \left[\begin{matrix}a_{1}\\\overline{a_{1}}\\a_{2}\\\overline{a_{2}}\\a'_{1}\\\overline{a'_{1}}\\a'_{2}\\\overline{a'_{2}}\end{matrix}\right]$

```python
simplify(h_int.dynamical_matrix)
```

$\displaystyle \left[\begin{matrix}0 & 0 & - \omega_{s} & 0 & \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0 & 0\\0 & 0 & 0 & - \omega_{s} & 0 & \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0\\\omega_{s} & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & \omega_{s} & 0 & 0 & 0 & 0 & 0 & 0\\- \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & - \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right]$

Looking at adiabatic elimination of $a_1'$

$\dot{a}_1' = -\gamma_1 a_1' - \sqrt{\gamma_1 \gamma_f} a_1 + \sqrt{2 \gamma_1} a_\text{in}$

adiabatic elimination: $\dot{a}_1' = 0$

$a_1' = \sqrt{\frac{\gamma_f}{\gamma_1}} a_1 - \sqrt{\frac{2}{\gamma_1}} a_\text{in}$

$H_\text{int} = i \sqrt{2\gamma_f}(a_\text{in}^\dagger a_1 - a_\text{in} a_1^\dagger)$

```python
split_network.dynamical_matrix.states.states
```

$\displaystyle \left[\begin{matrix}a_{1}\\\overline{a_{1}}\\a_{2}\\\overline{a_{2}}\\a'_{1}\\\overline{a'_{1}}\\a'_{2}\\\overline{a'_{2}}\\ain_{1}\\\overline{ain_{1}}\\aout_{1}\\\overline{aout_{1}}\\ain_{2}\\\overline{ain_{2}}\\aout_{2}\\\overline{aout_{2}}\end{matrix}\right]$

```python
split_network.dynamical_matrix.eqns
```

$\displaystyle \left[\begin{matrix}- \frac{\sqrt{\gamma_{1}} \sqrt{\gamma_{f}} a'_{1}}{s} + \frac{\omega_{s} a_{2}}{s}\\- \frac{\sqrt{\gamma_{1}} \sqrt{\gamma_{f}} \overline{a'_{1}}}{s} + \frac{\omega_{s} \overline{a_{2}}}{s}\\- \frac{\omega_{s} a_{1}}{s}\\- \frac{\omega_{s} \overline{a_{1}}}{s}\\\frac{\sqrt{\gamma_{1}} \sqrt{\gamma_{f}} a_{1}}{s} - \frac{\sqrt{2} \sqrt{\gamma_{1}} ain_{1}}{s} + \frac{\gamma_{1} a'_{1}}{s}\\\frac{\sqrt{\gamma_{1}} \sqrt{\gamma_{f}} \overline{a_{1}}}{s} - \frac{\sqrt{2} \sqrt{\gamma_{1}} \overline{ain_{1}}}{s} + \frac{\gamma_{1} \overline{a'_{1}}}{s}\\0\\0\\0\\0\\ain_{2}\\\overline{ain_{2}}\\- \sqrt{2} \sqrt{\gamma_{1}} a'_{1} + ain_{1}\\\overline{ain_{1}}\\\overline{ain_{1}}\\aout_{1}\end{matrix}\right]$

```python
# Calculating the input-output transfer function
tfm = split_network.tfm

tf = tfm.open_loop('ain_1', 'aout_1').simplify()
gamma_1, _ = split_network.aux_coupling_constants
adiabatically_eliminate(tf, gamma_1)
```

$\displaystyle \frac{\gamma_{f} s + \omega_{s}^{2} + s^{2}}{\gamma_{f} s - \omega_{s}^{2} - s^{2}}$

```python
tf = tfm.open_loop('a_1', 'aout_1').simplify()
gamma_1, _ = split_network.aux_coupling_constants
adiabatically_eliminate(tf, gamma_1)
```

$\displaystyle \sqrt{2} \sqrt{\gamma_{f}}$

Now looking at the active realisation ($g \neq 0$)

```python
# parameterise with lambda = g**2 - omega_s**2 > 0
lmbda = symbols('lambda', real=True, positive=True)
tf = (s**2 + s * gamma_f - lmbda) / (s**2 - s * gamma_f - lmbda)

transfer_function_to_graph(tf, 'active_coupled_cavity.pdf', layout='dot')
```

    0
    wrote active_coupled_cavity.pdf

```python
split_network = tf2rss(tf).to_slh().split()
h_int = split_network.interaction_hamiltonian
h_int.expr.simplify()
```

    0

$\displaystyle i \left(- \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} a_{1} \overline{a'_{1}} + \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} \overline{a_{1}} a'_{1} - \sqrt{\lambda} a_{1} a_{2} + \sqrt{\lambda} \overline{a_{1}} \overline{a_{2}}\right)$

```python
simplify(h_int.dynamical_matrix)
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & \sqrt{\lambda} & \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0 & 0\\0 & 0 & \sqrt{\lambda} & 0 & 0 & \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0\\0 & \sqrt{\lambda} & 0 & 0 & 0 & 0 & 0 & 0\\\sqrt{\lambda} & 0 & 0 & 0 & 0 & 0 & 0 & 0\\- \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & - \sqrt{\gamma_{1}} \sqrt{\gamma_{f}} & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right]$

```python
split_network.frequency_domain_eqns
```

```python
# Calculating the input-output transfer function
tfm = split_network.tfm

tf = tfm.open_loop('ain_1', 'aout_1').simplify()
gamma_1, _ = split_network.aux_coupling_constants
adiabatically_eliminate(tf, gamma_1)
```

```python
(s**2 + s * gamma_f - lmbda) / (s**2 - s * gamma_f - lmbda)
```

Differs by phase shift of $\pi$

Now let's look at the transfer function from $a_1$ to $aout_1$, expect it to be frequency independent

```python
tf = tfm.open_loop('a_1', 'aout_1').simplify()
gamma_1, _ = split_network.aux_coupling_constants
adiabatically_eliminate(tf, gamma_1)
```


# Gaussian-Naive-Bayes Classifier
## Definition
> Prerequisites to increase your understanding
> - Programming knowledge with Python
> - Naive-Bayes knowledge ([More information](Naive-Bayes-Classifier.ipynb))
> - Jupyter notebook knowledge

Naive Bayes classification method can be extended for real-valued features. If the values in our dataset have a continuous pattern, it may be chosen Gaussian Naive Bayes classification. There are other types of funtions but Gaussian (or normal distribution) is the simplest one because you only need to compute the mean and standard deviation from your training dataset.

The Gaussian distribution is denoted by;

\begin{equation}
X \sim \mathcal{N}(\mu, \sigma^2)
\end{equation}

* $X$ is a random variable.
* $\sim$ is tilda that means "distributed as".
* $\mu$ is mean which is the center of the curve.
* $\sigma^2$ is the standard deviation squared that is also called **variance** (more technically it indicates risk).

For example, suppose we have a parameter distribution like below

\begin{equation}
\{x_1, x_2, x_3, \dots , x_m \} \in \mathbf{R}
\end{equation}

It is not hard to draw the chart below intuitively

We can simply write the equations below;

\begin{align}
\mu = \frac{1}{m} \sum_{i = 1}^m x_i
\end{align}

\begin{align}
\sigma^2 = \frac{1}{m} \sum_{i = 1}^m (x_i - \mu)^2
\end{align}

> - It is obvious why we calculate $\sigma^2$ as a variance because $\sigma$ becomes negative in the left side of the chart above.
> - The main characteristic of Gaussian distribution is that the area under the bell-curve is equal to 1.

## Gaussian Formulation

The formula of the Gaussian distribution (above) is formulated like:

\begin{equation}
y = \frac{1}{\sigma \sqrt{2\pi}} e^-\frac{\left(x-\mu \right)}{2\sigma^2}
\end{equation}

* $\mu$ is mean which is the center of the curve.
* $\sigma$ is stardard deviation which describes how the data spreads out.

This formula is also called as **the probability density function**.

With the m-featured training set, we can compute the possibilities of each sample $x$ like

\begin{align}
& Training Set \in \{x_1, x_2, x_3, \dots , x_m \} \\
& x \in R^n \\
& P(x) = P(x_1 \mid \mu_{1}, \sigma_{1}^2) P(x_2 \mid \mu_{2}, \sigma_{2}^2) \dots P(x_n \mid \mu_{n}, \sigma_{n}^2) \\
& P(x) = \Pi_{i=1}^n P(x_i \mid \mu_{i}, \sigma_{i}^2) \\
& P(x) = \Pi_{i=1}^n \left( \frac{1}{\sigma_{i} \sqrt{2\pi}} e^-\frac{\left(x_i-\mu_{i} \right)}{2\sigma_{i}^2} \right)
\end{align}

```python
# libraries
import pandas as pd
import numpy as np

dataset = pd.read_csv('iris.txt',sep=' ')

# if dataset.index % test_indis == 0
# then it is going to be used as test dataset
# they will not be attended into the train dataset
test_indis = 11

test_dataset = dataset[dataset.index % test_indis == 0]
train_dataset = dataset[dataset.index % test_indis != 0]

# total count of sample space
total = float(len(train_dataset))

train_dataset.describe()
```

      sl
      sw
      pl
      pw
      c

      count
      136.000000
      136.000000
      136.000000
      136.000000
      136.000000

      mean
      5.858824
      3.036765
      3.770588
      1.198529
      2.007353

      std
      0.841821
      0.431845
      1.770211
      0.761479
      0.820987

      min
      4.300000
      2.000000
      1.100000
      0.100000
      1.000000

      25%
      5.100000
      2.800000
      1.575000
      0.300000
      1.000000

      50%
      5.800000
      3.000000
      4.350000
      1.300000
      2.000000

      75%
      6.400000
      3.300000
      5.100000
      1.800000
      3.000000

      max
      7.900000
      4.400000
      6.900000
      2.500000
      3.000000

```python
# libraries
import matplotlib.pylab as plt
import matplotlib.mlab as mlab
import math

def pdf(x, mean, std):
    std = std**2
    return (np.exp(-np.power(x - mean, 2.) / (2 * np.power(std, 2.)))) / (std * np.power(2 * math.pi, 1/2))
# end of def pdf

# loop in columns of dataset
for clm in train_dataset.columns:
    if clm != "c":
        mean, std = train_dataset[clm].mean(), train_dataset[clm].std()
        gaussian = pdf(train_dataset[clm], mean, std)
        plt.plot(train_dataset[clm], gaussian, ".")
# end of for loop

# plotting
plt.title("Normal distributions of train dataset")
plt.legend()
plt.show()
```

```python
train_dataset_1 = train_dataset.loc[[idx for idx, item in train_dataset.iterrows() if item.c == 1]]
train_dataset_2 = train_dataset.loc[[idx for idx, item in train_dataset.iterrows() if item.c == 2]]
train_dataset_3 = train_dataset.loc[[idx for idx, item in train_dataset.iterrows() if item.c == 3]]

pc_1 = len(train_dataset_1) / (total)
pc_2 = len(train_dataset_2) / (total)
pc_3 = len(train_dataset_3) / (total)

columns = ["sl", "sw", "pl", "pw"]

mean_sl1 = train_dataset_1[columns[0]].mean()
mean_sl2 = train_dataset_2[columns[0]].mean()
mean_sl3 = train_dataset_3[columns[0]].mean()
mean_sw1 = train_dataset_1[columns[1]].mean()
mean_sw2 = train_dataset_2[columns[1]].mean()
mean_sw3 = train_dataset_3[columns[1]].mean()
mean_pl1 = train_dataset_1[columns[2]].mean()
mean_pl2 = train_dataset_2[columns[2]].mean()
mean_pl3 = train_dataset_3[columns[2]].mean()
mean_pw1 = train_dataset_1[columns[3]].mean()
mean_pw2 = train_dataset_2[columns[3]].mean()
mean_pw3 = train_dataset_3[columns[3]].mean()

std_sl1 = train_dataset_1[columns[0]].std()
std_sl2 = train_dataset_2[columns[0]].std()
std_sl3 = train_dataset_3[columns[0]].std()
std_sw1 = train_dataset_1[columns[1]].std()
std_sw2 = train_dataset_2[columns[1]].std()
std_sw3 = train_dataset_3[columns[1]].std()
std_pl1 = train_dataset_1[columns[2]].std()
std_pl2 = train_dataset_2[columns[2]].std()
std_pl3 = train_dataset_3[columns[2]].std()
std_pw1 = train_dataset_1[columns[3]].std()
std_pw2 = train_dataset_2[columns[3]].std()
std_pw3 = train_dataset_3[columns[3]].std()

gaussian_sl1 = pdf(test_dataset[columns[0]], mean_sl1, std_sl1)
gaussian_sl2 = pdf(test_dataset[columns[0]], mean_sl2, std_sl2)
gaussian_sl3 = pdf(test_dataset[columns[0]], mean_sl3, std_sl3)
gaussian_sw1 = pdf(test_dataset[columns[1]], mean_sw1, std_sw1)
gaussian_sw2 = pdf(test_dataset[columns[1]], mean_sw2, std_sw2)
gaussian_sw3 = pdf(test_dataset[columns[1]], mean_sw3, std_sw3)
gaussian_pl1 = pdf(test_dataset[columns[2]], mean_pl1, std_pl1)
gaussian_pl2 = pdf(test_dataset[columns[2]], mean_pl2, std_pl2)
gaussian_pl3 = pdf(test_dataset[columns[2]], mean_pl3, std_pl3)
gaussian_pw1 = pdf(test_dataset[columns[3]], mean_pw1, std_pw1)
gaussian_pw2 = pdf(test_dataset[columns[3]], mean_pw2, std_pw2)
gaussian_pw3 = pdf(test_dataset[columns[3]], mean_pw3, std_pw3)

p_1 = gaussian_sl1 * gaussian_sw1 * gaussian_pl1 * gaussian_pw1 * pc_1
p_2 = gaussian_sl2 * gaussian_sw2 * gaussian_pl2 * gaussian_pw2 * pc_2
p_3 = gaussian_sl3 * gaussian_sw3 * gaussian_pl3 * gaussian_pw3 * pc_3

print("test_dataset  :", np.array(test_dataset[["c"]]).T[0])

p_arr = np.array([p_1, p_2, p_3])
p = p_arr / np.sum(p_arr, axis=0)

print("classification:", np.array([j + 1 for idx in p.T for j in range(len(idx)) if idx[j] == np.max(idx)]))
```

    test_dataset  : [1 1 1 1 1 2 2 2 2 2 3 3 3 3]
    classification: [1 1 1 1 1 2 2 3 2 2 3 3 3 3]

## References
* Scikit-learn, Naive Bayes, http://scikit-learn.org/stable/modules/naive_bayes.html (30/10/2017)
* Machine Learing Mastery, https://machinelearningmastery.com/naive-bayes-for-machine-learning/ (30/10/2017)
* Normal distribution - Wikipedia, https://en.wikipedia.org/wiki/Normal_distribution#Properties (08/11/2017)


# Data-driven distributionally robust optimization ([Mohajerin Esfahani and Kuhn, 2018](https://rdcu.be/cbBoC))

This notebook illustrates parameterization in MOSEK's [Fusion API for python](https://docs.mosek.com/9.2/pythonfusion/intro_info.html), by implementing the numerical experiments presented in a paper titled  "Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations" ([Mohajerin Esfahani and Kuhn](https://rdcu.be/cbBoC) hereafter), authored by [Peyman Mohajerin Esfahani](https://scholar.google.co.in/citations?user=ZTan-7YAAAAJ&hl=en) and [Daniel Kuhn](https://scholar.google.com/citations?user=RqnXytkAAAAJ&hl=en).

#### Contents:
1. [Data-driven stochastic programming (Paper summary)](#Data-driven-stochastic-programming-(Paper-summary))
   - [Choice of the ambiguity set](#Choice-of-the-ambiguity-set-$\widehat{\mathcal{P}}_N$)
   - [Convex reduction of the worst-case expectation problem](#Convex-reduction-of-the-worst-case-expectation-problem)

2. [Mean risk portfolio optimization with Fusion API](#Mean-risk-Portfolio-Optimization-(Section-7.1-in-paper))
   - [Parameterized Fusion model](#Parameterized-Fusion-model)
   - [Market setup](#Market-setup)

3. [Simulations and results](#Simulations-and-results)

   - [Impact of the Wasserstein radius](#Impact-of-the-Wasserstein-radius)
   - [Portflios driven by out-of-sample performance](#Portfolios-driven-by-out-of-sample-performance)
       - [Holdout method](#Holdout-method)
       - [k-fold cross validation method](#k-fold-cross-validation-method)
   - [Portfolios driven by reliability](#Portfolios-driven-by-reliability)
   - [Impact of sample size on Wasserstein radius](#Impact-of-sample-size-on-Wasserstein-radius)

# Data-driven stochastic programming (Paper summary)

Consider a loss function $h(x,\xi)$ which depends on the decision variables $x$ and uncertain parameters $\xi$. The goal is to minimize the expected value of $h(x,\xi)$ for all $x\in \mathbb{X}$, where $\mathbb{X} \subset \mathbb{R}^n$. The expected value is calculated over $\mathbb{P}(\xi)$, which is the probability distribution of the random vector $\xi \in \mathbb{R}^m$. This distribution is supported on the uncertainty set $\Xi$. The problem statement is written as in $\ref{eq:main_prob}$.

$$
\begin{equation}
J^\star := \underset{x\in \mathbb{X}}{\inf} \bigg\{ \mathbb{E}^{\mathbb{P}}\big[ h(x,\xi) \big] = \int_{\Xi} h(x,\xi)\mathbb{P}(d\xi)\bigg\}
\label{eq:main_prob} \tag{1}
\end{equation}
$$

In practice, however, the probability distribution is often unknown. Instead, one usually has access to a set $\widehat{\Xi}_N := \{\widehat{\xi}_i \}_{i\leq N}$, that contains $N$ historic realizations of the $\xi$ vector. A data-driven solution $\widehat{x}_N$ can be constructed by using some approximation of the true probability distribution. A natural choice for such an approximation is based on $\widehat{\Xi}_N$, and it is given by the discrete empirical probability distribution, as shown in $\ref{eq:empirical_dist}$.

$$
\begin{equation}
\mathbb{\widehat{P}}_N := \frac{1}{N}\sum^{N}_{i = 1} \delta_{\widehat{\xi}_i}
\label{eq:empirical_dist} \tag{2}
\end{equation}
$$

The sample-average-approximation, which is obtained by replacing $\mathbb{P}$ with $\mathbb{\widehat{P}}_N$ in $\ref{eq:main_prob}$, is usually easier to solve but shows poor out-of-sample performance. Out-of-sample performance of a data-driven solution $\widehat{x}_N$ is the expected loss for that decision under the true probability distribution. To obtain better out-of-sample performance, [Mohajerin Esfahani and Kuhn](https://rdcu.be/cbBoC) solve the robust optimization problem stated in $\ref{eq:dist_rob}$.

$$
\begin{equation}
\widehat{J}_N := \underset{x\in \mathbb{X}}{\inf} \underset{\mathbb{Q}\in \mathcal{\widehat{P}}_N}\sup \mathbb{E}^\mathbb{Q}\big[ h(x, \xi)\big]
\label{eq:dist_rob} \tag{3}
\end{equation}
$$

The  distributionally robust problem stated in $\ref{eq:dist_rob}$ is conceptually similar to $\ref{eq:main_prob}$, but it compensates for the unknown $\mathbb{P}$ by maximizing the expected value over all probability distributions $\mathbb{Q}$ inside an ambiguity set $\widehat{\mathcal{P}}_N$. Maximization is done in order to obtain the worst-case estimate.

If $\widehat{x}_N$ is an optimal decision vector for the distributionally robust program, then the authors associate performance guarantees to $\widehat{x}_N$ of the type shown in $\ref{eq:perf_guarantee}$.

$$
\begin{equation}
\mathbb{P}^N \bigg\{ \widehat{\Xi}_N : \mathbb{E}^\mathbb{P}\big[ h(\widehat{x}_N, \xi) \big] \leq \widehat{J}_N \bigg \} \geq 1 - \beta
\label{eq:perf_guarantee} \tag{4}
\end{equation}
$$

Here, $\mathbb{P}^N$ is a probability distribution that governs the realization of the set $\widehat{\Xi}_N$. The expression in $\ref{eq:perf_guarantee}$ states that $\widehat{J}_N$ provides an upper bound to the out-of-sample performance of $\widehat{x}_N$ with at least $1-\beta$ probability. Thus, $\widehat{J}_N$ is a certificate of out-of-sample performance and the probability on the left side is the reliability of this certificate.

### Choice of the ambiguity set $\widehat{\mathcal{P}}_N$

The ambiguity set resides within a space of probability distributions and is chosen to be a ball of radius $\epsilon$ centered at $\widehat{\mathbb{P}}_N$. The metric of choice is the [Wasserstein metric](http://nbviewer.jupyter.org/github/MOSEK/Tutorials/blob/master/wasserstein/wasserstein-bary.ipynb). Mathematically, the ambiguity set is defined as shown in $\ref{eq:wass_ball}$.

$$
\begin{equation}
\mathbb{B}_\epsilon(\mathbb{\widehat{P}}_N) := \big\{ \mathbb{Q} \in \mathcal{M}(\Xi) : d_W (\mathbb{\widehat{P}}_N, \mathbb{Q}) \leq \epsilon \big\}
\label{eq:wass_ball} \tag{5}
\end{equation}
$$

As proved in section 3 of the paper, solving $\ref{eq:dist_rob}$ with $\mathcal{\widehat{P}}_N$ set to $\mathbb{B}_\epsilon(\mathbb{\widehat{P}}_N)$ will provide finite sample performance guarantees of the type seen in $\ref{eq:perf_guarantee}$. Furthermore, given certain assumptions (see paper), as $N\to\infty$, the certificate $\widehat{J}_N$ approaches $J^\star$ (optimal value of $\ref{eq:main_prob}$) and the data-driven decision $\widehat{x}_N$ approaches the true optimal decision $x^\star$.

### Convex reduction of the worst-case expectation problem

The infinite dimensional maximization problem within $\ref{eq:dist_rob}$ can be reduced to a computationally tractable finite-dimensional convex program, given the following requirements are met:

1.) The uncertainty set $\Xi$ is convex and closed.

2.) The loss function is a point-wise maximum of more elementary functions, i.e. $l(\xi) := \underset{k\leq K}{\max} l_k(\xi)$, where $-l_k(\xi)$ are proper, convex, and lower semi-continuous for all $k\leq K$. Note the suppression of the loss function's dependence on $x$.

For problems that satisfy these requirements, the convex reduction is given in $\ref{eq:convex_red}$.
$$
\begin{align}
& \underset{\lambda, s_i, z_{ik}, \nu_{ik}}{\inf} & & \lambda \epsilon + \frac{1}{N}\sum_{i=1}^{N}s_i & \nonumber \\
& \text{s.t.} & &[-l_k]^*(z_{ik} - \nu_{ik}) + \sigma_{\Xi}(\nu_{ik}) - \langle z_{ik},\widehat{\xi}_i\rangle \leq s_i & & \forall i\leq N , \forall k\leq K \label{eq:convex_red} \tag{6}\ \\
& & &\|z_{ik}\|_* \leq \lambda & & \forall i\leq N, \forall k\leq K \nonumber \\
\end{align}
$$

where $[-l_k]^*$ denotes the conjugate of the negative $k^{\text{th}}$ component of the loss function, $\|z_{ik}\|_*$ is the dual norm corresponding to the norm used in defining the Wasserstein ball and $\sigma_\Xi$ is the support function for the uncertainty set.

# Mean risk Portfolio Optimization (Section 7.1 in paper)

Consider a market that forbids short-selling and has $m$ assets. Yearly returns of these assets are given by the random vector $\xi = [\xi_1, ..., \xi_m]^T$. The percentage weights (of the total capital) invested in each asset are given by the decision vector $x = [x_1, ..., x_m]^T$ where $x$ belongs to $\mathbb{X} = \{x \in \mathbb{R}_+^m : \sum^{m}_{i=1}x_i = 1\}$. We have to solve the problem stated in $\ref{eq:mean_risk_port}$.

$$
\begin{equation}
J^\star = \underset{x\in \mathbb{X}}{\inf} \big\{ \mathbb{E}^{\mathbb{P}}[-\langle x, \xi \rangle] + \rho \mathbb{P}\text{-CVaR}_\alpha \big(-\langle x, \xi\rangle\big)\big\}
\label{eq:mean_risk_port} \tag{7}
\end{equation}
$$

which amounts to minimizing a weighted sum of the expected value and the conditional value-at-risk (see also: [Fusion notebook on stochastic risk measures](https://nbviewer.jupyter.org/github/MOSEK/Tutorials/blob/master/stochastic-risk/stochastic-risk-measures.ipynb)) of the portfolio loss, $- \langle x, \xi \rangle$. Using the definition of conditional value-at-risk ([Optimization of conditional value-at-risk](https://www.ise.ufl.edu/uryasev/files/2011/11/CVaR1_JOR.pdf) by R. T. Rockafellar and S. Uryasev), $\ref{eq:mean_risk_port}$ can be written as a maximum of $K=2$ affine loss functions ($\ref{eq:cvar_prob}$).

$$
\begin{equation}
J^\star = \underset{x \in \mathbb{X}, \tau \in \mathbb{R}}{\inf} \mathbb{E}^{\mathbb{P}}\big[\underset{k\leq K}{\max} a_k \langle x, \xi \rangle + b_k \tau \big]
\label{eq:cvar_prob} \tag{8}
\end{equation}
$$
$$
\text{where}\,\, a =\big(-1, -1-\frac{\rho}{\alpha}\big)\, ,\,\, b =\big(\rho, \rho-\frac{\rho}{\alpha}\big)
$$

If the uncertainty set is a polytope of the form $\Xi = \{ \xi \in \mathbb{R}^m : C\xi \leq d \}$, then the distributionally robust counterpart of the program in $\ref{eq:cvar_prob}$ will have a convex reduction as shown in $\ref{eq:convex_red_port}$.

$$
\begin{align}
& \underset{x,\tau,\lambda,s_i, \gamma_{ik}}{\inf} & & \lambda \epsilon + \frac{1}{N}\sum^{N}_{i=1}s_i & \nonumber \\
&\text{s.t.} & & b_k \tau + a_k \langle x, \widehat{\xi}_i \rangle + \langle \gamma_{ik}, d - C\widehat{\xi}_i\rangle \leq s_i & \forall i\leq N, \forall k\leq K \label{eq:convex_red_port} \tag{9}\\
& & &\| C^T \gamma_{ik} - a_k x\|_* \leq \lambda & \forall i\leq N, \forall k\leq K &\nonumber \\
& & & \gamma_{ik} \geq 0 & \forall i \leq N, \forall k \leq K & \nonumber \\
& & & x \in \mathbb{X} & \nonumber \\
\end{align}
$$

### Parameterized Fusion model

Finally, we make a [Fusion](https://docs.mosek.com/9.2/pythonfusion/api-reference.html) model for the program in $\ref{eq:convex_red_port}$. The following code cell defines `DistributionallyRobustPortfolio`, a wrapper class  (around our fusion model) with some convenience methods. The key steps in the Fusion model implementation are (see `portfolio_model` method in class):

- `M = Model('DistRobust_m{0}_N{1}'.format(m, N))`: instantiate a Fusion `Model` object.

- `dat = M.parameter('TrainData', [N, m])` and `eps = M.parameter('WasRadius')`: declare a Fusion `parameter` object named "TrainData" and a parameter for the Wasserstein Radius, called "WasRadius". "TrainData" represents the past `N` realizations of the `m` sized random vector, $\xi$.

- `a_k = [-1, -51]` and `b_k = [10, -40]`: lists that are defined as in $\ref{eq:cvar_prob}$. All experiments that follow have the same $a_k$ and $b_k$, therefore we use lists. However, with minor modifications these lists can be replaced with Fusion parameters.

- `x = M.variable('Weights', m, Domain.greaterThan(0.0))`: declare Fusion variables for all the optimization variables involved in $\ref{eq:convex_red_port}$, except $\gamma$, which is excluded as $\Xi \in \mathbb{R}^m$ ([see the market setup below](#Market-setup)).

- `M.objective('J_N(e)', ObjectiveSense.Minimize, certificate)`: declare the objective sense and the expression of the objective. Here `certificate` is an `Expr` object that evaluates to the objective in $\ref{eq:convex_red_port}$.

- `self.M.constraint('C1',Expr.add([e1, e2, Expr.neg(Expr.repeat(s, 2, 1))]), Domain.lessThan(0.0))`: defining the constraints. Here `e1` and `e2` are `Expr` objects that evaluate to the relevant terms in the first constraint in $\ref{eq:convex_red_port}$. Constraints "C2_pos" and "C2_neg" correspond to the second constraint and finally "C3" ensures that $x \in \mathbb{X}$.

- `self.M.setSolverParam('optimizer','freeSimplex')`: use the [simplex optimizer](https://docs.mosek.com/9.2/pythonfusion/solving-linear.html#the-simplex-optimizer). By default, MOSEK will use its interior point optmizer. However, we plan to re-solve a linear model many times over and it is therefore imperative to exploit the warm-start capabilities of the simplex optimizer.

Note: More details regarding the code can be found within comments.

```python
class DistributionallyRobustPortfolio:

    def __init__(self, m, N):
        self.m, self.N = m, N
        self.M = self.portfolio_model(m, N)
        self.x = self.M.getVariable('Weights')
        self.t = self.M.getVariable('Tau')
        self.dat = self.M.getParameter('TrainData')
        self.eps = self.M.getParameter('WasRadius')
        self.sol_time = []

    def portfolio_model(self, m, N):
        '''
        Parameterized Fusion model for program in 9.
        '''
        M = Model('DistRobust_m{0}_N{1}'.format(m, N))
        ##### PARAMETERS #####
        dat = M.parameter('TrainData', [N, m])
        eps = M.parameter('WasRadius')
        a_k = [-1, -51]  # Alternative: Fusion parameters.
        b_k = [10, -40]  # Alternative: Fusion parameters.
        ##### VARIABLES #####
        x = M.variable('Weights', m, Domain.greaterThan(0.0))
        s = M.variable('s_i', N)
        l = M.variable('Lambda')
        t = M.variable('Tau')
        ##### OBJECTIVE #####
        # certificate = lamda*epsilon + sum(s)/N
        certificate = Expr.add(Expr.mul(eps, l),
                               Expr.mul(1/N, Expr.sum(s)))
        M.objective('J_N(e)', ObjectiveSense.Minimize, certificate)
        ##### CONSTRAINTS #####
        # b_k*t
        e1 = Expr.transpose(Expr.repeat(Expr.mul(t, b_k), N, 1))
        # a_k*<x,xi>
        e2 = Expr.hstack([Expr.mul(a_k[i], Expr.mul(dat, x))
                          for i in range(2)])
        # b_k*t + a_k*<x,xi> <= s
        M.constraint('C1', Expr.add(
            [e1, e2, Expr.neg(Expr.repeat(s, 2, 1))]), Domain.lessThan(0.0))
        # a_k*x
        e3 = Expr.hstack([Expr.mul(a_k[i], x) for i in range(2)])
        e4 = Expr.repeat(Expr.repeat(l, m, 0), 2, 1)
        # ||a_k*x||_infty <= lambda
        M.constraint('C2_pos', Expr.sub(e3, e4), Domain.lessThan(0.0))
        M.constraint('C2_neg', Expr.add(e3, e4), Domain.greaterThan(0.0))
        # x \in X
        M.constraint('C3', Expr.sum(x), Domain.equalsTo(1.0))
        # Use the simplex optimizer
        M.setSolverParam('optimizer', 'freeSimplex')
        return M

    def sample_average(self, x, t, data):
        '''
        Calculate the sample average approximation for given x and tau.
        '''
        l = np.matmul(data, x)
        return np.mean(np.maximum(-l + 10*t, -51*l - 40*t))

    def iter_data(self, data_sets):
        '''
        Generator method for iterating through values for the
        TrainData parameter.
        '''
        for data in data_sets:
            yield self.simulate(data)

    def iter_radius(self, epsilon_range):
        '''
        Generator for iterating through values for the WasRadius
        parameter.
        '''
        for epsilon in epsilon_range:
            yield self.solve(epsilon)

    def simulate(self, data):
        '''
        Tentative goal of this method is to set a value for the
        data parameter, and solve the model for the same.

        Define in child classes.
        '''
        pass

    def solve(self, epsilon):
        '''
        Tentative goal of this method is to set a value for the
        radius parameter, and solve the model for the same.

        Define in child classes.
        '''
        pass
```

### Market setup

The simulation results presented in the paper are based on a market with $m = 10$ assets. Each asset's return is expressed as $\xi_i = \psi + \zeta_i$, where $\psi \sim \mathcal{N}(0, 0.02)$ is the systematic risk factor and $\zeta_i \sim \mathcal{N}(0.03\times i, 0.025\times i)$ is the idiosyncratic risk factor. By construction, assets with higher indices have higher risk as well as higher returns. The uncertainty set is $\Xi \in \mathbb{R}^m$, thus $C$ and $d$ are set to zero. $1$-norm is used to define the Wasserstein ball, therefore the dual norm will be the $\infty$-norm. Lastly, $\alpha = 20\%$ and $\rho = 10$ are used within the loss function.

```python
# Function to generate N samples of the m-shaped random vector xi.
def normal_returns(m, N):
    R = np.vstack([np.random.normal(
        i*0.03, np.sqrt((0.02**2+(i*0.025)**2)), N) for i in range(1, m+1)])
    return (R.transpose())
```

# Simulations and results

The concluding section of this notebook reproduces the results provided in sections 7.2 of [Mohajerin Esfahani and Kuhn](https://rdcu.be/cbBoC). We do not pursue details of the design of these simulations, instead we focus primarily on implementing them with the models constructed above. The reader will find details/hints about the code within the in-code comments.

## Impact of the Wasserstein radius

```python
import numpy as np
from mosek.fusion import Model, Expr, Domain, ObjectiveSense
import matplotlib.pyplot as plt
from scipy.special import erfinv
from sklearn.model_selection import train_test_split
from sklearn.utils import resample
```

```python
# Inherits from the portfolio class defined above.
class SimSet1(DistributionallyRobustPortfolio):
    # Market setup
    m = 10
    mu = np.arange(1, 11)*0.03
    var = 0.02 + (np.arange(1, 11)*0.025)

    # Constants for CVaR calculation.
    beta, rho = 0.8, 10
    c2_beta = 1/(np.sqrt(2*np.pi)*(np.exp(erfinv(2*beta - 1))**2)*(1-beta))

    def __init__(self, N, eps_range):
        self.eps_range = eps_range
        # Fusion model instance
        super().__init__(SimSet1.m, N)

    def simulate(self, data):
        '''
        This method is called within the generator method called iter_data, which
        was defined in the parent class.

        Returns
        wts: optimal asset weights for each radius
        perf: analytic out-of-sample performance for each radius
        rel: reliability for each radius
        '''
        # Set TrainData parameter
        self.dat.setValue(data)
        # Iterate through range of Wasserstein radii (see solve method below)
        wts, perf, rel = zip(*[(_w, _p, _r)
                               for _w, _p, _r in self.iter_radius(self.eps_range)])
        return wts, perf, rel

    def solve(self, epsilon):
        '''
        This method is used within the generator method called iter_radius, which
        was defined in the parent class.

        Returns:
        x_sol: asset weights
        out_perf: analytica out-of-sample performance
        rel: reliability for the certificate
        '''
        # Set WasRadius parameter (TrainData is already set)
        self.eps.setValue(epsilon)
        # Solve the Fusion model
        self.M.solve()
        self.sol_time.append(self.M.getSolverDoubleInfo('optimizerTime'))
        # Portfolio weights
        x_sol = self.x.level()
        # Analytical out-of-sample performance
        out_perf = self.analytic_out_perf(x_sol)
        return x_sol, out_perf, (out_perf <= self.M.primalObjValue())

    def analytic_out_perf(self, x_sol):
        '''
        Method to calculate the analytical value for the out-of-sample performance.
        [see Rockafellar and Uryasev]
        '''
        mean_loss = -np.dot(x_sol, SimSet1.mu)
        sd_loss = np.sqrt(np.dot(x_sol**2, SimSet1.var))
        cVaR = mean_loss + (sd_loss*SimSet1.c2_beta)
        return mean_loss + (SimSet1.rho*cVaR)

    def run_sim(self, data_sets):
        '''
        Method to iterate over several datasets and record the results.
        '''
        wts, perf, rel = zip(*self.iter_data(data_sets))
        self.weights = np.mean(wts, axis=0)
        self.perf_mu = np.mean(perf, axis=0)
        self.perf_20 = np.quantile(perf, 0.2, axis=0)
        self.perf_80 = np.quantile(perf, 0.8, axis=0)
        self.reliability = np.mean(rel, axis=0)
```

```python
# Cardinality
N = 300
# Number of simulations
num_sim = 200

# Range of Wasserstein radii to consider
epsilon_range = np.append(np.concatenate(
    [np.arange(1, 10)*10.0**(i) for i in range(-4, 0)]), 1.0)

# Making 200 independent datasets
sim_data = [normal_returns(10, N) for i in range(num_sim)]

# Instantiate SimSet1 class
sim1 = SimSet1(N, epsilon_range)

# 200 simulations...
sim1.run_sim(sim_data)

print("Time taken in initial solve of model with N={0}: {1:.4f} s".format(
    N, sim1.sol_time[0]))
print("Mean time to solve a model with N={0}: {1:.4f} s".format(
    N, np.mean(sim1.sol_time)))
print("Total time to solve 200 models with N={0}: {1:.4f} s".format(
    N, np.sum(sim1.sol_time)))
```

    Time taken in initial solve of model with N=300: 0.0064 s
    Mean time to solve a model with N=300: 0.0018 s
    Total time to solve 200 models with N=300: 13.0416 s

```python
fig, ax = plt.subplots(ncols=2, figsize=(10, 4), dpi=150, gridspec_kw={'wspace': 0.3})
fig.suptitle('N = {}'.format(N))

# Portfolio weight vs Wasserstein radius
for i in range(1, 11):
    ax[0].fill_between(epsilon_range, np.sum(sim1.weights[:, :i-1], axis=1),
                       np.sum(sim1.weights[:, :i], axis=1),color=plt.cm.RdYlBu(1 - i/11))
    ax[0].plot(epsilon_range, np.sum(sim1.weights[:, :i], axis=1),
               color='black', linewidth=1.0)
ax[0].set_xscale('log')
ax[0].set_xlabel('$\epsilon$')
ax[0].set_xlim(10**-3, 1)
ax[0].set_ylim(0, 1)
ax[0].set_ylabel('Mean portfolio weights')

# Evolution of certificate and reliability
ax[1].plot(epsilon_range, sim1.perf_mu, color='blue')
ax[1].fill_between(epsilon_range, sim1.perf_20, sim1.perf_80, color='blue', alpha=0.4)
ax[1].set_xscale('log')
ax[1].set_xlabel('$\epsilon$')
ax[1].set_xlim(10**-4, 1.0)
ax[1].set_ylabel('Out-of-sample performance', color='blue')
ax[1].set_yticks(np.arange(-1.4, -0.8, 0.1))
ax[1].grid(which='major',alpha=0.4)
ax[1].grid(which='minor',alpha=0.3)

ax2 = ax[1].twinx()

ax2.set_ylabel('Reliability', color='red')
ax2.plot(epsilon_range, sim1.reliability, color='red')
ax2.set_yticks(np.arange(0, 1.2, 0.2))
ax2.set_ylim(0, 1)

plt.savefig('sim1_N{}.jpg'.format(N), dpi = 150, bbox_inches='tight')
plt.show()
plt.show()
```

## Portfolios driven by out-of-sample performance

### Holdout method

```python
class SimSet2_Holdout(DistributionallyRobustPortfolio):
    # Market setup
    m = 10

    # Radius range (see page 156 in Mohajerin Esfahani and Kuhn)
    eps_range = np.concatenate([np.arange(1, 10)*10.0**(i)
                                for i in range(-3, 0)])

    # Validation data set of 2*10**5 samples (page 158 in paper)
    valids = normal_returns(10, 2*10**5)

    def __init__(self, N, k=5):
        # 1/k sized split for the test data.
        self.k = k
        # Fusion model for (train) data size N*(k-1)/k (see holdout method)
        super().__init__(SimSet2_Holdout.m, np.rint(N*(k-1)/k).astype(np.int32))

    def validate(self, data_sets):
        '''
        Method to iterate over a list of independent datasets via the iter_data
        generator method so as to apply the holdout technique to each dataset
        and then save the results.
        '''
        self.perf, self.cert, radii = zip(*self.iter_data(data_sets))
        self.rel = np.mean(np.array(self.perf) <= np.array(self.cert), axis=0)
        self.radius = np.mean(radii, axis=0)

    def simulate(self, data):
        '''
        Method called within the iter_data generator.

        Returns
        out_perf: out-of-sample performance calculated with validation data
        cert: performance certificate (optimal objective for M)
        eps_holdout: radius selected from holdout method
        '''
        # Split data into test and train
        train, self.test = train_test_split(data, test_size=1/self.k)
        # Set the TrainData parameter to train data
        self.dat.setValue(train)
        # Iterate through a range of Wasserstein radii
        out_perf_test, x, t, J_N = zip(
            *self.iter_radius(SimSet2_Holdout.eps_range))
        # Index of eps_holdout in the eps_range.
        min_arg = np.argmin(out_perf_test)
        # Out-of-sample performance for x_N(eps_holdout)
        out_perf = self.sample_average(
            x[min_arg], t[min_arg], SimSet2_Holdout.valids)
        # J_N(eps_holdout)
        cert = J_N[min_arg]
        return out_perf, cert, SimSet2_Holdout.eps_range[min_arg]

    def solve(self, epsilon):
        '''
        Method called within the iter_radius generator.

        Returns
        out_perf: SA-approx of out-of-sample performance using test data
        x: Portfolio weights
        t: Tau
        self.M.primalObjValue(): performance certificate
        '''
        # Set the WasRadius parameter
        self.eps.setValue(epsilon)
        # Solve the Fusion model
        self.M.solve()
        self.sol_time.append(self.M.getSolverDoubleInfo('optimizerTime'))
        # Weights and Tau optimal values
        x, t = self.x.level(), self.t.level()
        # SAA of out-of-sample performance based on test data
        out_perf = self.sample_average(x, t, self.test)
        return out_perf, x, t, self.M.primalObjValue()
```

```python
# Range of cardinality values to consider
N_range = np.append(np.concatenate([np.arange(1,10)*10**(i) for i in range(1,3)]),1000)

holdout_results = []
for N in N_range:
    # List of 200 independent datasets of cardinality N
    n_data = [normal_returns(10, N) for i in range(200)]
    # Instance of SimSet2_Holdout class with cardinality N
    hld = SimSet2_Holdout(N)
    # Running the holdout method over each dataset
    hld.validate(n_data)
    # Save the object
    holdout_results.append(hld)
```

```python
def N_plot(results, title, save_name, y_l = [[-1.5, 0.5], [-2.5, 10], [0, 1]]):
    N_range = np.append(np.concatenate(
        [np.arange(1, 10)*10**(i) for i in range(1, 3)]), 1000)

    fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(13, 3), dpi=150)
    fig.suptitle(title)
    # Out-of-sample performance
    ax[0].fill_between(N_range, [np.quantile(r.perf, 0.2, axis=0) for r in results], [
                       np.quantile(r.perf, 0.8, axis=0) for r in results], alpha=0.4)
    ax[0].plot(N_range, [np.mean(r.perf, axis=0) for r in results], marker='>')
    # Certificate
    ax[1].fill_between(N_range, [np.quantile(r.cert, 0.2, axis=0) for r in results], [
                       np.quantile(r.cert, 0.8, axis=0) for r in results], alpha=0.4)
    ax[1].plot(N_range, [np.mean(r.cert, axis=0) for r in results], marker='>')
    # Reliability
    ax[2].plot(N_range, [r.rel for r in results], marker='>')

    y_n = ["Out-of-sample performance", "Certificate", "Reliability"]
    for i in range(3):
        ax[i].set_ylabel(y_n[i])
        ax[i].set_ylim(y_l[i])
        ax[i].grid(which='major',alpha=0.4)
        ax[i].grid(which='minor',alpha=0.3)
        ax[i].set_xscale('log')
        ax[i].set_xlabel('N')
        ax[i].set_xlim([10, 1000])
    ax[0].set_yticks([-1.5, -1, -0.5, 0, 0.5])
    plt.savefig(save_name, dpi = 150, bbox_inches='tight')
    plt.show()
```

```python
N_plot(holdout_results, 'Holdout Method', 'sim2_holdout.jpg')
```

### k-fold cross validation method

```python
# IMPORTANT: sub-class of the SimSet2_Holdout class!
class SimSet2_kFold(SimSet2_Holdout):

    def __init__(self, N, k=5):
        self.k = k
        # Object for holdout method (k-holdouts)
        super().__init__(N, k=k)
        # Fusion model for N-size dataset (results)
        self.M_N = self.portfolio_model(SimSet2_Holdout.m, N)
        self.dat_N = self.M_N.getParameter('TrainData')
        self.eps_N = self.M_N.getParameter('WasRadius')
        self.x_N = self.M_N.getVariable('Weights')
        self.t_N = self.M_N.getVariable('Tau')

    def simulate(self, data):
        '''
        Method called within the iter_data generator. This method overwrites
        the one defined in the SimSet2_Holdout class.

        Returns
        out_perf: out-of-sample performance calculated with validation data
        cert: performance certificate (optimal objective for M_N)
        eps_kFold: radius selected from k-Fold method
        '''
        # Set TrainData paremeter for M_N to data
        self.dat_N.setValue(data)
        # Perform the holdout method k times and calculate eps_kFold
        eps_kFold = np.mean([self._simulate(data) for i in range(self.k)])
        # Set WasRadius to mean from k holdout runs
        self.eps_N.setValue(eps_kFold)
        # Solve the M_N model.
        self.M_N.solve()
        # Out-of-sample performance for x_N(eps_kFold)
        out_perf = self.sample_average(
            self.x_N.level(), self.t_N.level(), SimSet2_Holdout.valids)
        # J_N(eps_kFold)
        cert = self.M_N.primalObjValue()
        return out_perf, cert, eps_kFold

    def _simulate(self, data):
        '''
        Method to perform the holdout technique for a given dataset. This
        is called k times within each call to the simulate method. Works
        analogously to the simulate method of SimSet2_Holdout class.

        Returns:
        eps_holdout: WasRadius selected in one holdout run
        '''
        # Split data into test and train
        train, self.test = train_test_split(data, test_size=1/self.k)
        # Set TrainData parameter for the N*(k-1)/k model
        self.dat.setValue(train)
        # Solve N*(k-1)/k model iteratively for a range of radii
        saa, x, t, J_N = zip(*self.iter_radius(SimSet2_Holdout.eps_range))
        # Select Wasserstein radius that minimizes out-of-sample perf
        min_arg = np.argmin(saa)
        return SimSet2_Holdout.eps_range[min_arg]
```

```python
# Range of cardinality values to consider
N_range = np.append(np.concatenate([np.arange(1,10)*10**(i) for i in range(1,3)]),1000)

kFold_results = []

for N in N_range:
    # List of 200 independent datdasets of cardinality N
    n_data = [normal_returns(10,N) for i in range(200)]
    # Instance of SimSet2_kFold class with cardinality N
    kFld = SimSet2_kFold(N)
    # Running the k-fold cross validation over each dataset
    kFld.validate(n_data)
    kFold_results.append(kFld)
```

```python
N_plot(kFold_results, 'k-fold cross validation method', 'sim2_kcross.jpg')
```

## Portfolios driven by reliability

```python
class SimSet3(DistributionallyRobustPortfolio):
    m = 10
    eps_range = np.concatenate([np.arange(1, 10)*10.0**(i)
                                for i in range(-3, 0)])
    valids = normal_returns(10, 2*10**5)

    def __init__(self, beta, N, k=50):
        # Number of resamples
        self.k = k
        # Reliability threshold
        self.beta = beta
        # Instantiate Fusion model
        super().__init__(SimSet3.m, N)

    def bootstrap(self, data_sets):
        '''
        Method to iterate over a list of independent datasets via the iter_data
        generator method so as to apply the bootstrap technique to each dataset
        and then save the results.
        '''
        self.perf, self.cert, radii = zip(*self.iter_data(data_sets))
        self.rel = np.mean(np.array(self.perf) <= np.array(self.cert), axis=0)
        self.radii = np.mean(radii, axis=0)

    def simulate(self, data):
        '''
        Method called within the iter_data generator.

        Returns
        out_perf: out-of-sample performance calculated with validation data
        cert: performance certificate (optimal objective for M)
        eps_btstrp: radius selected from holdout method
        '''
        # List to store reliability
        rel = []
        # Perform k resamples
        for i in range(self.k):
            # Split data into test and train
            train, self.test = train_test_split(data, test_size=1/3)
            # Resample train data up-to size N
            train = resample(train, n_samples=self.N)
            # Set TrainData parameter to train
            self.dat.setValue(train)
            # Iterate through a range of Wasserstein radii
            rel.append(list(self.iter_radius(SimSet3.eps_range)))
        # Sum reliability over all resamples (for each epsilon)
        rel = np.sum(rel, axis=0)
        # Smallest radius that has reliability over 1-beta
        _id = next(i for i, r in enumerate(rel) if r >= self.k*(1-self.beta))
        eps_btstrp = SimSet3.eps_range[_id]
        # Set TrainData parameter to data
        self.dat.setValue(data)
        # Set WasRadius parameter to eps_btstrp
        self.eps.setValue(eps_btstrp)
        self.M.solve()
        # Out-of-sample performance for x_N(eps_btstrp)
        out_perf = self.sample_average(
            self.x.level(), self.t.level(), SimSet3.valids)
        cert = self.M.primalObjValue()
        return out_perf, cert, eps_btstrp

    def solve(self, epsilon):
        '''
        Method called within the iter_radius generator.

        Returns
        reliability: SAA of out-of-sample performance <= certificate(epsilon)
        '''
        # Set WasRadius parameter to epsilon and solve
        self.eps.setValue(epsilon)
        self.M.solve()
        # Calculate out-of-sample performance SAA estimator using test
        saa = self.sample_average(self.x.level(), self.t.level(), self.test)
        # Boolean to state if the certificate is greater than SAA estimate
        return saa <= self.M.primalObjValue()
```

```python
# Range of cardinality values to consider
N_range = np.append(np.concatenate(
        [np.arange(1, 10)*10**(i) for i in range(1, 3)]), 1000)
# Reliability threshold: 1 - beta = 0.9
beta = 0.1

bootstrap_results = []
for N in N_range:
    n_data = [normal_returns(10,N) for i in range(200)]
    # Instance of SimSet3 class with cardinality N
    btstrp = SimSet3(beta,N)
    # Running the bootstrap method for each dataset
    btstrp.bootstrap(n_data)
    bootstrap_results.append(btstrp)
```

```python
N_plot(bootstrap_results, "1 - $\\beta$ = {}".format(1-beta), 'sim3_90.jpg',
       y_l = [[-1.5, 1.5], [-2.5, 10], [0, 1]])
```

## Impact of sample size on Wasserstein radius

```python
fig, ax = plt.subplots(figsize=(5, 4), dpi=150)
fig.suptitle('Impact of sample size on Wasserstein radius')

# Wasserstein radius vs sample size N
# HoldouN_rangeethod
ax.plot(N_range, [h.radius for h in holdout_results], color='blue',
        linestyle='--', label='$\widehat{\epsilon}^{hm}_N$ Holdout')
# k-fold cross validation method
ax.plot(N_range, [k.radius for k in kFold_results], color='green',
        marker='>', label='$\widehat{\epsilon}^{cv}_N$ k-fold')
# Bootstrap method (reliability driven portfolio)
ax.plot(N_range, [b.radii for b in bootstrap_results], color='tomato',
        marker='<', label=r'$\widehat{\epsilon}^{\beta}_N$ $1 - \beta = 0.9$')

ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel('N')
ax.set_xlim(10, 1000)
ax.set_ylim(0.005, 0.5)
ax.set_ylabel('Average Wasserstein radii')
ax.grid(which='major',alpha=0.4)
ax.grid(which='minor',alpha=0.3)
ax.legend()
plt.savefig('radius_v_samplesize.jpg', dpi = 150, bbox_inches='tight')
plt.show()
```

The results provided in this notebook were computed on a laptop with Intel® Core™ i7-10875H processor and 32 GB RAM.

This work is licensed under a Creative Commons Attribution 4.0 International License. The **MOSEK** logo and name are trademarks of Mosek ApS. The code is provided as-is. Compatibility with future release of **MOSEK** or the `Fusion API` are not guaranteed. For more information contact our [support](mailto:support@mosek.com).


# Long Short Term Memory Neural Network

We're going to build a Long Short Term Memory Neural Network from scratch and look at the math behind it.

## Imports

```python
import numpy as np
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from IPython import display
import tensorflow as tf
plt.style.use('seaborn-white')
import re
```

    C:\Users\mvp\AppData\Roaming\Python\Python37\site-packages\requests\__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.8) or chardet (3.0.4) doesn't match a supported version!
      RequestsDependencyWarning)

## Downloading Dataset

We will be using the Shakespeare dataset to train an LSTM to generate similar texts.

```python
path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')
data = open(path_to_file, 'rb').read().decode(encoding='utf-8')
```

## One-Hot Encoding

Once we have the dataset we need to one-hot encode by character. To do that later, we will create a dictionary for character to index and vice-versa

```python
chars = sorted(set(data))
data_size, X_size = len(data), len(chars)
print("data has %d characters, %d unique" % (data_size, X_size))
char_to_idx = {ch:i for i,ch in enumerate(chars)}
idx_to_char = {i:ch for i,ch in enumerate(chars)}
```

    data has 1115394 characters, 65 unique

Great! Now that we have our one-hot encoding dictionary in place, we can move on to the LSTM!

# Introduction to the Long Short-Term Memory (LSTM) Cell

A vanilla RNN suffers from [the vanishing gradients problem](http://neuralnetworksanddeeplearning.com/chap5.html#the_vanishing_gradient_problem) which gives challenges in saving memory over longer sequences. To combat these issues the gated hidden units were created. The two most prominent gated hidden units are the Long Short-Term Memory (LSTM) cell and the Gated Recurrent Unit (GRU), both of which have shown increased performance in saving and reusing memory in later timesteps. In this exercise, we will focus on LSTM but you would easily be able to go ahead and implement the GRU as well based on the principles that you learn here.

Below is a figure of the LSTM cell:

Source: https://arxiv.org/abs/1412.7828

The LSTM cell contains three gates, input, forget, output gates and a memory cell.
The output of the LSTM unit is computed with the following functions, where $\sigma = \mathrm{softmax}$.
We have input gate $i$, forget gate $f$, and output gate $o$ defines as

- $i = \sigma ( W^i [h_{t-1}, x_t])$

- $f = \sigma ( W^f [h_{t-1},x_t])$

- $o = \sigma ( W^o [h_{t-1},x_t])$

where $W^i, W^f, W^o$ are weight matrices applied to a concatenated $h_{t-1}$ (hidden state vector) and $x_t$ (input vector)  for each respective gate.

$h_{t-1}$, from the previous time step along with the current input $x_t$ are used to compute the a candidate $g$

- $g = \mathrm{tanh}( W^g [h_{t-1}, x_t])$

The value of the cell's memory, $c_t$, is updated as

- $c_t = c_{t-1} \circ f + g \circ i$

where $c_{t-1}$ is the previous memory, and $\circ$ refers to element-wise multiplication.

The output, $h_t$, is computed as

- $h_t = \mathrm{tanh}(c_t) \circ o$

and it is used for both the timestep's output and the next timestep, whereas $c_t$ is exclusively sent to the next timestep.
This makes $c_t$ a memory feature, and is not used directly to compute the output of the timestep.

## Activation Functions and Derivatives

Let us define activation functions and its derivative for future use

#### Sigmoid

\begin{align}
\sigma(x) &= \frac{1}{1 + e^{-x}}\\
\frac{d\sigma(x)}{dx} &= \sigma(x) \cdot (1 - \sigma(x))
\end{align}

#### Tanh

\begin{align}
\frac{d\text{tanh}(x)}{dx} &= 1 - \text{tanh}^2(x)
\end{align}

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def dsigmoid(y):
    return y * (1 - y)

def tanh(x):
    return np.tanh(x)

def dtanh(y):
    return 1 - y * y
```

## Hyperparameters and Parameters

Let us define the hyperparameters fo the network along with a parameter class to declare the weights of the network

```python
H_size = 100 # Size of the hidden layer
T_steps = 25 # Number of time steps (length of the sequence) used for training
learning_rate = 1e-1 # Learning rate
weight_sd = 0.1 # Standard deviation of weights for initialization
z_size = H_size + X_size # Size of concatenate(H, X) vector
```

Each Param will have a value, its derivative and its momentum (for use in Adagrad later)

```python
class Param:
    def __init__(self, name, value):
        self.name = name
        self.v = value #parameter value
        self.d = np.zeros_like(value) #derivative
        self.m = np.zeros_like(value) #momentum for AdaGrad
```

We use random weights with normal distribution (`0`, `weight_sd`) for $tanh$ activation function and (`0.5`, `weight_sd`) for $sigmoid$ activation function.

Biases are initialized to zeros.

```python
class Parameters:
    def __init__(self):
        self.W_f = Param('W_f',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_f = Param('b_f',
                         np.zeros((H_size, 1)))

        self.W_i = Param('W_i',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_i = Param('b_i',
                         np.zeros((H_size, 1)))

        self.W_C = Param('W_C',
                         np.random.randn(H_size, z_size) * weight_sd)
        self.b_C = Param('b_C',
                         np.zeros((H_size, 1)))

        self.W_o = Param('W_o',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_o = Param('b_o',
                         np.zeros((H_size, 1)))

        #For final layer to predict the next character
        self.W_v = Param('W_v',
                         np.random.randn(X_size, H_size) * weight_sd)
        self.b_v = Param('b_v',
                         np.zeros((X_size, 1)))

    def all(self):
        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,
               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]

parameters = Parameters()
```

## Initialiation of a LSTM network

We will implement the forward pass, backward pass, optimization and training loop, now for an LSTM in numpy.

## Forward pass

*Operation $z$ is the concatenation of $x$ and $h_{t-1}$*

### Concatenation of $h_{t-1}$ and $x_t$
\begin{align}
z & = [h_{t-1}, x_t] \\
\end{align}

### LSTM functions
\begin{align}
f_t & = \sigma(W_f \cdot z + b_f) \\
i_t & = \sigma(W_i \cdot z + b_i) \\
\bar{C}_t & = tanh(W_C \cdot z + b_C) \\
C_t & = f_t * C_{t-1} + i_t * \bar{C}_t \\
o_t & = \sigma(W_o \cdot z + b_t) \\
h_t &= o_t * tanh(C_t) \\
\end{align}

### Logits
\begin{align}
v_t &= W_v \cdot h_t + b_v \\
\end{align}

### Softmax
\begin{align}
\hat{y_t} &= \text{softmax}(v_t)
\end{align}

$\hat{y_t}$ is `y` in code and $y_t$ is `targets`.

```python
def forward(x, h_prev, C_prev, p = parameters):
    assert x.shape == (X_size, 1)
    assert h_prev.shape == (H_size, 1)
    assert C_prev.shape == (H_size, 1)

    z = np.row_stack((h_prev, x))
    f = sigmoid(np.dot(p.W_f.v, z) + p.b_f.v)
    i = sigmoid(np.dot(p.W_i.v, z) + p.b_i.v)
    C_bar = tanh(np.dot(p.W_C.v, z) + p.b_C.v)

    C = f * C_prev + i * C_bar
    o = sigmoid(np.dot(p.W_o.v, z) + p.b_o.v)
    h = o * tanh(C)

    v = np.dot(p.W_v.v, h) + p.b_v.v
    y = np.exp(v) / np.sum(np.exp(v)) #softmax

    return z, f, i, C_bar, C, o, h, v, y
```

## Backward pass

### Loss

\begin{align}
L_k &= -\sum_{t=k}^T\sum_j y_{t,j} log \hat{y_{t,j}} \\
L &= L_1 \\
\end{align}

### Gradients

\begin{align}
dv_t &= \hat{y_t} - y_t \\
dh_t &= dh'_t + W_y^T \cdot dv_t \\
do_t &= dh_t * \text{tanh}(C_t) \\
dC_t &= dC'_t + dh_t * o_t * (1 - \text{tanh}^2(C_t))\\
d\bar{C}_t &= dC_t * i_t \\
di_t &= dC_t * \bar{C}_t \\
df_t &= dC_t * C_{t-1} \\
\\
df'_t &= f_t * (1 - f_t) * df_t \\
di'_t &= i_t * (1 - i_t) * di_t \\
d\bar{C}'_{t-1} &= (1 - \bar{C}_t^2) * d\bar{C}_t \\
do'_t &= o_t * (1 - o_t) * do_t \\
dz_t &= W_f^T \cdot df'_t \\
     &+ W_i^T \cdot di_t \\
     &+ W_C^T \cdot d\bar{C}_t \\
     &+ W_o^T \cdot do_t \\
\\
[dh'_{t-1}, dx_t] &= dz_t \\
dC'_t &= f_t * dC_t
\end{align}

* $dC'_t = \frac{\partial L_{t+1}}{\partial C_t}$ and $dh'_t = \frac{\partial L_{t+1}}{\partial h_t}$
* $dC_t = \frac{\partial L}{\partial C_t} = \frac{\partial L_t}{\partial C_t}$ and $dh_t = \frac{\partial L}{\partial h_t} = \frac{\partial L_{t}}{\partial h_t}$
* All other derivatives are of $L$
* `target` is target character index $y_t$
* `dh_next` is $dh'_{t}$ (size H x 1)
* `dC_next` is $dC'_{t}$ (size H x 1)
* `C_prev` is $C_{t-1}$ (size H x 1)
* $df'_t$, $di'_t$, $d\bar{C}'_t$, and $do'_t$ are *also* assigned to `df`, `di`, `dC_bar`, and `do` in the **code**.
* *Returns* $dh_t$ and $dC_t$

### Model parameter gradients

\begin{align}
dW_v &= dv_t \cdot h_t^T \\
db_v &= dv_t \\
\\
dW_f &= df'_t \cdot z^T \\
db_f &= df'_t \\
\\
dW_i &= di'_t \cdot z^T \\
db_i &= di'_t \\
\\
dW_C &= d\bar{C}'_t \cdot z^T \\
db_C &= d\bar{C}'_t \\
\\
dW_o &= do'_t \cdot z^T \\
db_o &= do'_t \\
\\
\end{align}

```python
def clear_gradients(params = parameters):
    for p in params.all():
        p.d.fill(0)

def clip_gradients(params = parameters):
    for p in params.all():
        np.clip(p.d, -1, 1, out=p.d)

def backward(target, dh_next, dC_next, C_prev,
             z, f, i, C_bar, C, o, h, v, y,
             p = parameters):

    assert z.shape == (X_size + H_size, 1)
    assert v.shape == (X_size, 1)
    assert y.shape == (X_size, 1)

    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:
        assert param.shape == (H_size, 1)

    dv = np.copy(y)
    dv[target] -= 1

    p.W_v.d += np.dot(dv, h.T)
    p.b_v.d += dv

    dh = np.dot(p.W_v.v.T, dv)
    dh += dh_next
    do = dh * tanh(C)
    do = dsigmoid(o) * do
    p.W_o.d += np.dot(do, z.T)
    p.b_o.d += do

    dC = np.copy(dC_next)
    dC += dh * o * dtanh(tanh(C))
    dC_bar = dC * i
    dC_bar = dtanh(C_bar) * dC_bar
    p.W_C.d += np.dot(dC_bar, z.T)
    p.b_C.d += dC_bar

    di = dC * C_bar
    di = dsigmoid(i) * di
    p.W_i.d += np.dot(di, z.T)
    p.b_i.d += di

    df = dC * C_prev
    df = dsigmoid(f) * df
    p.W_f.d += np.dot(df, z.T)
    p.b_f.d += df

    dz = (np.dot(p.W_f.v.T, df)
         + np.dot(p.W_i.v.T, di)
         + np.dot(p.W_C.v.T, dC_bar)
         + np.dot(p.W_o.v.T, do))
    dh_prev = dz[:H_size, :]
    dC_prev = f * dC

    return dh_prev, dC_prev
```

Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.

* `input`, `target` are list of integers, with character indexes.
* `h_prev` is the array of initial `h` at $h_{-1}$ (size H x 1)
* `C_prev` is the array of initial `C` at $C_{-1}$ (size H x 1)
* *Returns* loss, final $h_T$ and $C_T$

```python
def forward_backward(inputs, targets, h_prev, C_prev):
    global paramters

    # To store the values for each time step
    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}
    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}
    v_s, y_s =  {}, {}

    # Values at t - 1
    h_s[-1] = np.copy(h_prev)
    C_s[-1] = np.copy(C_prev)

    loss = 0
    # Loop through time steps
    assert len(inputs) == T_steps
    for t in range(len(inputs)):
        x_s[t] = np.zeros((X_size, 1))
        x_s[t][inputs[t]] = 1 # Input character

        (z_s[t], f_s[t], i_s[t],
        C_bar_s[t], C_s[t], o_s[t], h_s[t],
        v_s[t], y_s[t]) = \
            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass

        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t

    clear_gradients()

    dh_next = np.zeros_like(h_s[0]) #dh from the next character
    dC_next = np.zeros_like(C_s[0]) #dh from the next character

    for t in reversed(range(len(inputs))):
        # Backward pass
        dh_next, dC_next = \
            backward(target = targets[t], dh_next = dh_next,
                     dC_next = dC_next, C_prev = C_s[t-1],
                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],
                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],
                     y = y_s[t])

    clip_gradients()

    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]
```

### Sample the next character

Lets also define a function to sample the next character for generation

```python
def sample(h_prev, C_prev, first_char_idx, sentence_length):
    x = np.zeros((X_size, 1))
    x[first_char_idx] = 1

    h = h_prev
    C = C_prev

    indexes = []

    for t in range(sentence_length):
        _, _, _, _, C, _, h, _, p = forward(x, h, C)
        idx = np.random.choice(range(X_size), p=p.ravel())
        x = np.zeros((X_size, 1))
        x[idx] = 1
        indexes.append(idx)

    return indexes
```

### Optimization (Adagrad)

Update the graph and display a sample output

```python
def update_status(inputs, h_prev, C_prev):
    #initialized later
    global plot_iter, plot_loss
    global smooth_loss

    # Get predictions for 200 letters with current model

    sample_idx = sample(h_prev, C_prev, inputs[0], 200)
    txt = ''.join(idx_to_char[idx] for idx in sample_idx)

    # Clear and plot
    plt.plot(plot_iter, plot_loss)
    display.clear_output(wait=True)
    plt.show()

    #Print prediction and loss
    print("----\n %s \n----" % (txt, ))
    print("iter %d, loss %f" % (iteration, smooth_loss))
```

Update parameters

\begin{align}
\theta_i &= \theta_i - \eta\frac{d\theta_i}{\sum dw_{\tau}^2} \\
d\theta_i &= \frac{\partial L}{\partial \theta_i}
\end{align}

```python
def update_paramters(params = parameters):
    for p in params.all():
        p.m += p.d * p.d # Calculate sum of gradients
        #print(learning_rate * dparam)
        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))
```

### Training Loop

We need to define a full training loop with a forward pass, backward pass, optimization step and validation.

Before we do that let us define a class to ensure the training proccess isn't stopped in between epochs.

```python
import signal

class DelayedKeyboardInterrupt(object):
    def __enter__(self):
        self.signal_received = False
        self.old_handler = signal.signal(signal.SIGINT, self.handler)

    def handler(self, sig, frame):
        self.signal_received = (sig, frame)
        print('SIGINT received. Delaying KeyboardInterrupt.')

    def __exit__(self, type, value, traceback):
        signal.signal(signal.SIGINT, self.old_handler)
        if self.signal_received:
            self.old_handler(*self.signal_received)
```

Lets also store the loss to graph it

```python
# Exponential average of loss
# Initialize to a error of a random model
smooth_loss = -np.log(1.0 / X_size) * T_steps

iteration, pointer = 0, 0

# For the graph
plot_iter = np.zeros((0))
plot_loss = np.zeros((0))
```

```python
while True:
    try:
        with DelayedKeyboardInterrupt():
            # Reset
            if pointer + T_steps >= len(data) or iteration == 0:
                g_h_prev = np.zeros((H_size, 1))
                g_C_prev = np.zeros((H_size, 1))
                pointer = 0

            inputs = ([char_to_idx[ch]
                       for ch in data[pointer: pointer + T_steps]])
            targets = ([char_to_idx[ch]
                        for ch in data[pointer + 1: pointer + T_steps + 1]])

            loss, g_h_prev, g_C_prev = \
                forward_backward(inputs, targets, g_h_prev, g_C_prev)
            smooth_loss = smooth_loss * 0.999 + loss * 0.001

            # Print every hundred steps
            if iteration % 100 == 0:
                update_status(inputs, g_h_prev, g_C_prev)

            update_paramters()

            plot_iter = np.append(plot_iter, [iteration])
            plot_loss = np.append(plot_loss, [loss])

            pointer += T_steps
            iteration += 1
    except KeyboardInterrupt:
        update_status(inputs, g_h_prev, g_C_prev)
        break
```

The generated text does resemble shakespeare's texts in some ways. It has also learned the words that it can make up with the available charachters. So we can say the LSTM has captured the features that make up a shakespearean tale.

```python

```


+ This notebook is part of lecture 4 *Factorization into LU* in the OCW MIT course 18.06 [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
import numpy as np
from sympy import *
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import Image
from warnings import filterwarnings

init_printing(use_latex = 'mathjax')
%matplotlib inline
filterwarnings('ignore')
```

# LU decomposition of a matrix A

* We will decompose the matrix A into and upper and lower triangular matrix, such that multiplying these will result back into A
$$ A = LU $$

## Turning the matrix of coefficients into Upper triangular form

* Consider the following matrix of coefficients
$$ \begin{bmatrix} 1 & -2 & 1 \\ 3 & 2 & -2 \\ 6 & -1 & -1 \end{bmatrix} $$
* Successive elementary row operation follow
    * Which is nothing other than matrix multiplication of the elementary matrices
    * An elementary matrix is an identity matrix on which one elementary row operation was performed

```python
A = Matrix([[1, -2, 1], [3, 2, -2], [6, -1, -1]])
A
```

```python
eye(3)
```

* We have to get a -3 in the first **pivot** (the 1 in row 1, column 1) to get rid of the 3 in position row 2, column 1 (we call the resulting matrix E21, referring to the row 2, column 1)
* Then we add the new row 1 to row two
    Row one of the identity matrix is then (-3,0,0) (but we leave it (1,0,0) in E21) and adding this to row 2 leaves (-3,1,0)

```python
E21 = Matrix([[1, 0, 0], [-3, 1, 0], [0, 0, 1]])
E21
```

```python
E21 * A # The resulting matrix after multiplication by E21
```

* We do the same to get rid of the 6 in row 3, column 1
    * Multiplying row 1 (of the identity matrix) by -6 and adding this new row to row 3 (but again leaving row 1 as (1,0,0) in E31)

```python
E31 = Matrix([[1, 0, 0], [0, 1, 0], [-6, 0, 1]])
E31
```

```python
E31 * E21 * A # This got rid of the leading 6 in row 3
```

* Now the 8 in row 2, column 2 is the **pivot** and we need to get rid of the 11 in row 3, column 2
* Unfortunately we have an 8 and an 11 to deal with
* We will have to do two elementary row operations
    * -11 times row 2 of the identity matrix (0,-11,0)
    * Added to 8 times row 3 (0,0,8) ∴ (0,-11,8)

```python
E32 = Matrix([[1, 0 , 0], [0, 1, 0], [0, -11, 8]])
E32
```

```python
U = E32 * E31 * E21 * A
U # We call is U for upper triangular
```

* The matrix is now in upper triangular form
$$ \left( { E }_{ 32 } \right) \left( { E }_{ 31 } \right) \left( { E }_{ 21 } \right) A=U $$

## Calculating the Lower triangular from

* Note, to reverse this process we would have to do the following:
$$ { \left( { E }_{ 21 } \right)  }^{ -1 }{ \left( { E }_{ 31 } \right)  }^{ -1 }{ \left( { E }_{ 32 } \right)  }^{ -1 }\left( { E }_{ 32 } \right) \left( { E }_{ 31 } \right) \left( { E }_{ 21 } \right) A=A $$
* The inverse of a matrix can be calculated using the sympy method .*inv*()

* We can check this with a Boolean request

```python
E21.inv() * E31.inv() * E32.inv() * E32 * E31 * E21 * A == A # The Boolean double equal signs asks: Is the
# left-hand side equal to the right-hand side?
```

* Indeed, we will be back with the identity matrix just multiplying the inverse elementary matrices and the elementary matrices

```python
E21.inv() * E31.inv() * E32.inv() * E32 * E31 * E21
```

* Multiplying the inverse elementary matrices on the left, must also have it happen on the right
$$ { \left( { E }_{ 21 } \right)  }^{ -1 }{ \left( { E }_{ 31 } \right)  }^{ -1 }{ \left( { E }_{ 32 } \right)  }^{ -1 }\left( { E }_{ 32 } \right) \left( { E }_{ 31 } \right) \left( { E }_{ 21 } \right) A={ \left( { E }_{ 21 } \right)  }^{ -1 }{ \left( { E }_{ 31 } \right)  }^{ -1 }{ \left( { E }_{ 32 } \right)  }^{ -1 }U $$

* The multiplication of these inverse elementary matrices is lower triangular
* We can call in L
$$ A=LU $$

```python
L = E21.inv() * E31.inv() * E32.inv()
L
```

```python
A == L * U # Checking this with a Boolean question
```

```python
A, L * U # They are identical
```

## Doing this in one go using sympy

```python
L, U, _ = A.LUdecomposition()
```

```python
L
```

```python
U # Note the difference from the U above
```

```python
L * U # Back to A
```

## What's special about L?

* This only works when no row interchange happens
* It also actually only works when doing the conventional subtracting the scalar multiplication of a row from another row, leaving the positive scalar as opposed to the negatives I use, allowing me to add the two rows (as opposed to subtraction)
* Note the 3 (in row 2, column 1) and the 6 (in row 3, column 1)
* They are the row multiplications we have to do for E21 and E31
* The ¹¹ / ₈ is what we did for E32 (we just did it in two steps so as not to use fractions)

## Row exchanges

* We have to allow row exchanges if the pivot contains a zero

* For an example, from a 3×3 identity matrix we could have:

```python
eye(3)
```

* Exchanging rows one and two would be:

```python
Matrix([[0, 1, 0], [1, 0, 0], [0, 0, 1]])
```

```python
A, Matrix([[0, 1, 0], [1, 0, 0], [0, 0, 1]]) * A # Showing row exchange
```

* How many permutations of row exchanges are there?
$$ {n!} $$

* In a 3×3 matrix there are 3! = 6 permutations
    * Multiplying any of them will result in one of the 6
    * They are inverses of each other
    * The inverse are the transposes
* For 4×4 there are 4! = 24

## Example problems

### Example problem 01

* Perform LU decomposition of:
$$  \begin{bmatrix} 1 & 0 & 1 \\ a & a & a \\ b & b & a \end{bmatrix} $$
* For which values of *a* and *b* does L and U exist?

#### Solution

```python
a, b = symbols('a b')
```

```python
A = Matrix([[1, 0, 1], [a, a, a], [b, b, a]])
A
```

```python
L,U, _ = A.LUdecomposition()
```

```python
L, U
```

* Checking

```python
L * U == A
```

* For existence:
    * *a* ≠ 0
* It's easy to see why, since if a equals zero, we will have a zero in a pivot position and we will have to do row exchange, which is not allowed for LU-decomposition

## Hints and tips

```python
E21, E21.inv() # To take the inverse of an elementary matrix, simply change the sign of the off-diagonal elements and
# multiply each element by 1 over the determinant
# The determinant is easy to do for these *n* = 3 square matrices, since the top row is (1,0,0)
```

```python
E31, E31.inv()
```

```python
E32, E32.inv()
```

* By keeping track of the elementary matrices it is easy to get L and U
* It's easy to get the inverses of L and U
* This means it is easy to calculate **x**

$$ Ax=LUx=b\\ Ux={ L }^{ -1 }b\\ x={ U }^{ -1 }{ L }^{ -1 }b $$

```python

```