# Neutralising and Equalising Word Embeddings

This notebook shows how non-gender specific words can have the gender part neutralised to avoid bias in word embeddings. In addition to that, it also depicts the process of equalisation, where words that are gender-specific can be equalised towards words that are non-gender specific.

#### Import dependencies

```python
import numpy as np
```

#### Load the GloVe dataset

The GloVe dataset is not part of the repository due to the size of the file. However, feel free to download it from here: https://nlp.stanford.edu/projects/glove/

```python
def read_glove_vecs(glove_file):
    with open(glove_file, 'r') as f:
        words = set()
        word_to_vec_map = {}

        for line in f:
            line = line.strip().split()
            curr_word = line[0]
            words.add(curr_word)
            word_to_vec_map[curr_word] = np.array(line[1:], dtype=np.float64)

    return words, word_to_vec_map
```

```python
words, word_to_vec_map = read_glove_vecs('glove/glove.6B.50d.txt')
```

* words: set of words in the vocabulary.
* word_to_vec_map: dictionary mapping words to their GloVe vector representation.

```python
def cosine_similarity(u, v):
    """
    Cosine similarity reflects the degree of similariy between u and v

    Arguments:
        u -- a word vector of shape (n,)
        v -- a word vector of shape (n,)

    Returns:
        cosine_similarity -- the cosine similarity between u and v defined by the formula above.
    """

    dot = np.dot(u, v)
    norm_u = np.sqrt(np.sum(u**2))
    norm_v = np.sqrt(np.sum(v**2))
    cosine_similarity = dot / (norm_u * norm_v)

    return cosine_similarity
```

```python
father = word_to_vec_map["father"]
mother = word_to_vec_map["mother"]
ball = word_to_vec_map["ball"]
crocodile = word_to_vec_map["crocodile"]

france = word_to_vec_map["france"]
italy = word_to_vec_map["italy"]
paris = word_to_vec_map["paris"]
rome = word_to_vec_map["rome"]

print("cosine_similarity(father, mother) = ", cosine_similarity(father, mother))
print("cosine_similarity(ball, crocodile) = ", cosine_similarity(ball, crocodile))

# This should not show similarity as in the first vector we subtract the city representation
# and in the second we subtract the country representation.
print("cosine_similarity(france - paris, rome - italy) = ", cosine_similarity(france - paris, rome - italy))

# This one, on the other hand, should show similarity as we are checking for the similarity between 2 cities.
print("cosine_similarity(paris - france, rome - italy) = ", cosine_similarity(paris - france, rome - italy))
```

    cosine_similarity(father, mother) =  0.8909038442893615
    cosine_similarity(ball, crocodile) =  0.2743924626137942
    cosine_similarity(france - paris, rome - italy) =  -0.6751479308174201
    cosine_similarity(paris - france, rome - italy) =  0.6751479308174201

## Debiasing word vectors

We start by identifying the gender by subtracting the `man` vector representation from the `woman` vector representation.

```python
gender = word_to_vec_map['woman'] - word_to_vec_map['man']
```

#### Similarity between gender and names

Negative similarities mean that the name is more related to the `female` gender.

```python
name_list = ['john', 'marie', 'sophie',
             'ronaldo', 'priya', 'rahul',
             'danielle', 'reza', 'katy',
             'yasmin', 'sam', 'carolina',
             'logan']

for w in name_list:
    print (w, cosine_similarity(word_to_vec_map[w], gender))
```

    john -0.23163356145973724
    marie 0.315597935396073
    sophie 0.31868789859418784
    ronaldo -0.31244796850329437
    priya 0.17632041839009402
    rahul -0.16915471039231716
    danielle 0.24393299216283895
    reza -0.07930429672199553
    katy 0.2831068659572615
    yasmin 0.23313857767928758
    sam -0.33642281213435427
    carolina 0.0938795106708001
    logan -0.16937077820548485

Unfortunately, non-gender specific words contain bias and hence need some extra treatment. Below a list of common words that look pretty biased.

```python
word_list = ['lipstick', 'driver', 'science', 'arts',
             'literature', 'warrior','doctor', 'librarian',
             'receptionist', 'technology',  'fashion', 'teacher',
             'engineer', 'pilot', 'computer', 'singer',
             'model', 'mechanic', 'babysitter']
for w in word_list:
    print (w, cosine_similarity(word_to_vec_map[w], gender))
```

    lipstick 0.2769191625638267
    driver -0.010681433817247916
    science -0.06082906540929701
    arts 0.008189312385880337
    literature 0.06472504433459932
    warrior -0.20920164641125288
    doctor 0.11895289410935041
    librarian 0.23302221769690296
    receptionist 0.33077941750593737
    technology -0.13193732447554302
    fashion 0.03563894625772699
    teacher 0.17920923431825664
    engineer -0.0803928049452407
    pilot 0.0010764498991916937
    computer -0.10330358873850498
    singer 0.1850051813649629
    model 0.0343357596036095
    mechanic -0.0035264430229621927
    babysitter 0.2797785047879521

Again, positive similarities relates to women whislt negative similaties don't. It's shocking to see that `computer`, `technology` and `engineer` do not relate to women.

### Neutralise bias for non-gender specific words

The formula below is used to compute the debiased version of a given vector representation.

\begin{align}
v^{bias\_component} = \frac{v \cdot gender}{\|gender\|_2^2}gender \\
\\
v^{debiased} = v - v^{bias\_component} \\
\end{align}

You can find the implementation of the `neutralise` formula below:

```python
def neutralize(word, gender, word_to_vec_map):
    """
    Removes the bias of "word" by projecting it on the space orthogonal to the bias axis.
    This function ensures that gender neutral words are zero in the gender subspace.

    Arguments:
        word -- string indicating the word to debias
        gender -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)
        word_to_vec_map -- dictionary mapping words to their corresponding vectors.

    Returns:
        v_debiased -- neutralised word vector representation of the input "word"
    """

    v = word_to_vec_map[word]

    v_biascomponent = (np.dot(v, gender) / np.sqrt(np.sum(gender**2))**2) * gender
    v_debiased = v - v_biascomponent

    return v_debiased
```

```python
w = "receptionist"
print("cosine similarity between " + w + " and gender, before neutralizing: ", cosine_similarity(word_to_vec_map["receptionist"], gender))

v_debiased = neutralize("receptionist", gender, word_to_vec_map)
print("cosine similarity between " + w + " and gender, after neutralizing: ", cosine_similarity(v_debiased, gender))
```

    cosine similarity between receptionist and gender, before neutralizing:  0.33077941750593737
    cosine similarity between receptionist and gender, after neutralizing:  -2.099120994400013e-17

The neutralised result is essentially 0, up to numerical roundof (on the order of  $10^{−17}$).

```python

```


## 1) Numpy Arrays vs Lists

```python
import numpy as np
```

```python
L = [1,2,3]
A = np.array([1,2,3])
L, A
```

    ([1, 2, 3], array([1, 2, 3]))

```python
L + [5]
```

    [1, 2, 3, 5]

```python
L
```

    [1, 2, 3]

```python
A + [5]
```

    array([6, 7, 8])

O Operador + funciona de maneira diferente em listas e numpy arrays.

Enquanto nas listas ele faz um append, no numpy arrays é tratado como soma vetorial.

```python
L * 2
```

    [1, 2, 3, 1, 2, 3]

```python
A * 2
```

    array([2, 4, 6])

O operador * funciona de maneira diferente em listas e numpy arrays.

Enquanto ele repete uma lista, em numpy arrays é tratado como multiplicação elemento a elemento entre vetores

O produto escalar é dado por:
$$\begin{align}
a \cdot b &= a^{T}b \\
&= \sum\limits_{i=0}^{n} a_ib_i\\
&= |a||b|\cos{\theta}
\end{align}
$$

```python
dot_list = 0
for e in L:
    dot_list += e*e
dot_list
```

    14

```python
(A*A).sum(), np.dot(A,A)
```

    (14, 14)

```python
a = np.array([1,3,4])
b = np.array([2,4,1])
```

$$
\cos{\theta} = \frac{a\cdot b}{|a||b|}
$$

```python
cos = a.dot(b)/(np.linalg.norm(a)*np.linalg.norm(b))
cos, np.arccos(cos)
```

    (0.7703288865196434, 0.6914395540229785)

### Speed Comparison: Dot Product

Comparar a velocidade do cálculo do produto escalar utilizando `for loop` e a função `numpy.dot()`

São criados 2 vetores 1x1000 para o teste.

```python
from datetime import datetime
```

```python
a = np.random.randn(100)
b = np.random.randn(100)
T = 1000
```

```python
def for_dot(a,b):
    result = 0
    for e,f in zip(a,b):
        result += e*f
    return result

t0 = datetime.now()
for t in range(T):
    for_dot(a,b)
dt1 = datetime.now() - t0

t0 = datetime.now()
for t in range(T):
    a.dot(b)
dt2 = datetime.now() - t0

dt1,dt2,dt1.total_seconds()/dt2.total_seconds()
```

    (datetime.timedelta(microseconds=79336),
     datetime.timedelta(microseconds=1879),
     42.222458754656735)

### Creating Arrays

```python
Z = np.zeros(10)
Z
```

    array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

```python
np.zeros((10,10))
```

    array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

```python
np.ones((10,10))
```

    array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
           [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])

```python
np.random.random((10,10))
# Números aleatórios baseado em probabilidade 0<x<1
```

    array([[0.8727353 , 0.7101418 , 0.99712466, 0.6913042 , 0.18928091,
            0.3129924 , 0.8707541 , 0.76127674, 0.04484837, 0.20726868],
           [0.58037897, 0.85199027, 0.12931742, 0.69984037, 0.54857979,
            0.19323108, 0.03747739, 0.28693611, 0.13446493, 0.92038251],
           [0.76115359, 0.82189855, 0.59756655, 0.59055865, 0.98912974,
            0.53747975, 0.14158241, 0.48285143, 0.36049548, 0.74296014],
           [0.24990797, 0.54769394, 0.61527197, 0.10531474, 0.71674935,
            0.00848703, 0.63823206, 0.51600374, 0.45159856, 0.12775354],
           [0.12094621, 0.26349439, 0.0460745 , 0.02141862, 0.83742369,
            0.77048361, 0.84818114, 0.58709173, 0.03627597, 0.48886898],
           [0.90964074, 0.94232793, 0.66774458, 0.86564113, 0.25928235,
            0.53727432, 0.50713657, 0.13698081, 0.94057636, 0.12699759],
           [0.85851277, 0.9387563 , 0.20228787, 0.3772616 , 0.04864407,
            0.73258449, 0.06612805, 0.80031866, 0.10438979, 0.18995695],
           [0.80844273, 0.23237812, 0.28653701, 0.25980382, 0.13690201,
            0.15392922, 0.10574834, 0.48174979, 0.47251255, 0.73375288],
           [0.00802248, 0.66516244, 0.53422215, 0.2853337 , 0.75394833,
            0.83927944, 0.082604  , 0.2291872 , 0.68355877, 0.53528026],
           [0.55539544, 0.17080853, 0.61780461, 0.95634259, 0.57328085,
            0.64070114, 0.44743496, 0.71544752, 0.44835268, 0.7934082 ]])

```python
G = np.random.randn(10,10)
G
# Número aleatórios de distribuição gaussiana
```

    array([[-0.29325655,  0.6514455 , -0.59313231,  0.45996943,  1.3397887 ,
             0.1356255 ,  1.33221726,  0.36584929,  0.40187569,  1.30479922],
           [ 0.69069344,  0.16975026, -0.14295021, -1.62121735,  1.43983921,
            -0.74673648, -0.70051634, -0.19065805,  0.99060591,  1.67003266],
           [-1.85002381, -0.79447583, -1.47101947, -0.77519736,  0.60512255,
             1.69136405, -0.37414819, -0.41072259,  1.37359439,  0.34048061],
           [ 0.07398756, -0.68716856,  0.7761272 ,  1.70610709, -0.09549806,
            -0.4563812 ,  0.6250297 ,  0.76567729,  0.65995112, -1.46688799],
           [-1.15924409,  0.91724841, -0.34138872, -2.58894929, -0.23099132,
             0.3138421 ,  0.49306887, -0.94478341, -0.6026703 ,  0.73182024],
           [-0.35001544, -1.2157117 , -0.52556314,  0.14471904,  0.30952623,
             0.84092323,  1.36910066,  0.95006108,  1.30409573,  1.00748436],
           [-0.457104  ,  0.10644546,  1.2413489 ,  1.57673014,  0.92612354,
             1.21189605, -1.28566399,  0.46028265,  0.32939659, -0.75769981],
           [ 1.20889008, -0.74653187, -2.11225921,  1.06619329,  0.58199165,
             1.46652055,  0.51299538, -0.22632982, -0.70686765, -0.35628585],
           [-0.57797704, -0.60259817,  1.3371527 , -0.70509169,  1.16501866,
             0.38331481, -0.69265983, -0.27892637, -0.89107526,  0.18796167],
           [-1.39126114,  1.09006032,  0.36083266,  0.04547842,  0.51360868,
             0.71687125,  0.26616739,  0.18450924, -0.97733634, -0.14369357]])

```python
G.mean()
## Média
```

    0.11352944229300384

```python
G.var()
## Variância
```

    0.8612373657553634

```python
np.linalg.inv(G)
## Inversa da Matriz
```

    array([[-0.12888897,  0.20176253, -0.37353808,  0.0865699 ,  0.40538869,
             0.15471414,  0.36799712,  0.2736015 , -0.16607605, -0.28100721],
           [ 0.05492664,  0.15958189, -0.1694259 ,  0.03874689,  0.31843647,
            -0.08811236,  0.33860869,  0.02826085, -0.42919972,  0.14344601],
           [ 0.22426181, -0.23724135, -0.2569901 ,  0.03846767,  0.45572269,
             0.09262413,  0.45431058, -0.13187685,  0.19296182, -0.56831628],
           [ 0.56757278, -0.4976866 ,  0.12779579, -0.29184054, -0.31054629,
            -0.27194361,  0.03941485, -0.17247039,  0.13287871, -0.44301028],
           [ 0.08329942,  0.38087176,  0.06966686,  0.40665812,  0.01732694,
            -0.15537987, -0.06283071,  0.20244253,  0.17223829,  0.25016165],
           [-0.08987573, -0.09772335, -0.21197512, -0.11253102,  0.5388706 ,
             0.31441403,  0.60904867,  0.20640591, -0.12924144, -0.24885146],
           [ 0.34369076, -0.18141415, -0.16873184,  0.3443168 ,  0.53783069,
             0.07429966,  0.10817637,  0.09360505,  0.060279  , -0.36590588],
           [-1.0628979 ,  0.78289266, -0.17901836,  0.0330127 , -0.6220006 ,
             0.68497583, -0.38634435,  0.10424462, -0.39669954,  1.58077643],
           [ 0.16104689,  0.0660416 ,  0.05603093,  0.25848534,  0.41338857,
            -0.01003739,  0.39605931, -0.06990624, -0.254451  , -0.48900586],
           [ 0.43319019, -0.40228234, -0.00552417, -0.72580933, -0.32709565,
             0.07877201, -0.01662383, -0.28436166,  0.18246469, -0.38697404]])

```python
## Inversa * Matriz = I
G.dot(np.linalg.inv(G))
```

    array([[ 1.00000000e+00, -1.11022302e-16,  1.73472348e-16,
             0.00000000e+00,  3.33066907e-16, -3.05311332e-16,
             7.97972799e-17,  5.55111512e-17,  1.11022302e-16,
            -3.33066907e-16],
           [-1.11022302e-16,  1.00000000e+00, -3.81639165e-17,
             0.00000000e+00,  2.22044605e-16,  1.38777878e-16,
             1.63064007e-16,  5.55111512e-17,  0.00000000e+00,
            -4.44089210e-16],
           [-2.77555756e-17, -1.11022302e-16,  1.00000000e+00,
             0.00000000e+00,  2.63677968e-16,  4.85722573e-17,
             3.55618313e-17, -5.55111512e-17, -4.16333634e-17,
             1.38777878e-16],
           [-2.22044605e-16, -4.44089210e-16,  1.17961196e-16,
             1.00000000e+00,  0.00000000e+00, -2.63677968e-16,
             1.42247325e-16, -1.11022302e-16,  0.00000000e+00,
            -6.66133815e-16],
           [ 3.88578059e-16, -3.33066907e-16,  1.45716772e-16,
            -1.11022302e-16,  1.00000000e+00, -2.42861287e-16,
             1.56125113e-16, -5.55111512e-17,  1.66533454e-16,
            -6.10622664e-16],
           [ 1.66533454e-16, -3.33066907e-16,  9.28077060e-17,
             2.22044605e-16,  1.11022302e-16,  1.00000000e+00,
            -1.49186219e-16, -5.55111512e-17,  1.94289029e-16,
            -4.44089210e-16],
           [-2.22044605e-16, -3.33066907e-16, -6.41847686e-17,
            -1.11022302e-16, -2.22044605e-16,  1.24900090e-16,
             1.00000000e+00, -8.32667268e-17,  3.05311332e-16,
             1.11022302e-15],
           [-5.55111512e-17,  0.00000000e+00, -1.66533454e-16,
             0.00000000e+00, -2.22044605e-16,  4.16333634e-17,
            -2.44596010e-16,  1.00000000e+00,  1.38777878e-16,
             1.66533454e-16],
           [ 6.93889390e-17, -1.80411242e-16, -6.24500451e-17,
            -2.77555756e-17,  8.32667268e-17,  9.88792381e-17,
             8.80372164e-17, -1.31838984e-16,  1.00000000e+00,
            -1.11022302e-16],
           [ 0.00000000e+00, -3.46944695e-17,  1.91903785e-17,
             5.55111512e-17,  2.98372438e-16, -1.71737624e-16,
             1.60895602e-16, -9.02056208e-17, -1.76941795e-16,
             1.00000000e+00]])

Produto externo

```python
a = np.array([1,3,6])
b = np.array([3,1,9])
c = np.outer(a,b)
c
```

    array([[ 3,  1,  9],
           [ 9,  3, 27],
           [18,  6, 54]])

```python
np.trace(G)
```

    -1.8252494059901754

```python
## Calculando covariância de um dataset fake, 3 features e 100 samples
X = np.random.randn(100,3)
```

```python
cov = np.cov(X.T)
```

```python
cov
```

    array([[ 0.9277445 ,  0.03838728, -0.06852152],
           [ 0.03838728,  1.03476599, -0.02533857],
           [-0.06852152, -0.02533857,  1.09690852]])

```python
np.linalg.eig(cov)
```

    (array([0.89761064, 1.13507008, 1.02673828]),
     array([[-0.93302255,  0.35190946,  0.07502439],
            [ 0.20672837,  0.35361487,  0.91226088],
            [-0.29450349, -0.86666964,  0.40268031]]))

```python
a = np.array([[1,3,4],[2,2,5],[5,5,4]])
b = np.array([3,np.pi,np.e])
```

```python
x = np.linalg.inv(a).dot(b)
x
```

    array([-0.20115547,  0.26145185,  0.60419998])

```python
np.linalg.solve(a,b)
```

    array([-0.20115547,  0.26145185,  0.60419998])

#### Resolvendo um problema
O preço do ingresso em uma feira é R\\$1,50 para crianças e R\\$4,00 para adultos. Em um certo dia 2200 pessoas passaram pela feira e a venda total de ingressos foi de R\\$5050,00. Quantas crianças e quantos adultos passaram pela feira?
Seja $x_1$ a quantidade de crianças e $x_2$ a quantidade de adultos:

$$
\begin{cases}
x_1 + x_2 &= 2200 \\ 1.5 x_1 + 4x_2 &= 5050
\end{cases}
$$

$$Ax = b$$

$$
\begin{pmatrix}
1 & 1 \\
1.5 & 4
\end{pmatrix}
\cdot
\begin{pmatrix}
x_1 \\ x_2
\end{pmatrix}
=
\begin{pmatrix}
2200 \\ 5050
\end{pmatrix}
$$

```python
A = np.array([[1,1],[1.5,4]])
b = np.array([2200,5050])
x = np.linalg.solve(A,b)
x[0],x[1]
```

    (1500.0, 700.0)

```python

```


# Elastic wave equation implementation on a staggered grid

This is a first attempt at implemenenting the elastic wave equation as described in:

[1] Jean Virieux (1986). ”P-SV wave propagation in heterogeneous media: Velocity‐stress finite‐difference method.” GEOPHYSICS, 51(4), 889-901. https://doi.org/10.1190/1.1442147

The current version actually attempts to mirror the FDELMODC implementation by Jan Thorbecke:

[2] https://janth.home.xs4all.nl/Software/fdelmodcManual.pdf

## Explosive source

We will first attempt to replicate the explosive source test case described in [1], Figure 4. We start by defining the source signature $g(t)$, the derivative of a Gaussian pulse, given by Eq 4:

$$g(t) = -2 \alpha(t - t_0)e^{-\alpha(t-t_0)^2}$$

```python
from devito import *
from examples.seismic.source import WaveletSource, RickerSource, GaborSource, TimeAxis
from examples.seismic import plot_image
import numpy as np

from sympy import init_printing, latex
init_printing(use_latex=True)
```

```python
# Initial grid: 1km x 1km, with spacing 100m
extent = (2000., 2000.)
shape = (201, 201)
x = SpaceDimension(name='x', spacing=Constant(name='h_x', value=extent[0]/(shape[0]-1)))
z = SpaceDimension(name='z', spacing=Constant(name='h_z', value=extent[1]/(shape[1]-1)))
grid = Grid(extent=extent, shape=shape, dimensions=(x, z))
```

```python
class DGaussSource(WaveletSource):

    def wavelet(self, f0, t):
        a = 0.004
        return -2.*a*(t - 1/f0) * np.exp(-a * (t - 1/f0)**2)

# Timestep size from Eq. 7 with V_p=6000. and dx=100
t0, tn = 0., 600.
dt = (10. / np.sqrt(2.)) / 6.
time_range = TimeAxis(start=t0, stop=tn, step=dt)

src = RickerSource(name='src', grid=grid, f0=0.01, time_range=time_range)
src.coordinates.data[:] = [1000., 1000.]
src.show()
```

```python
# Now we create the velocity and pressure fields
vx= TimeFunction(name='vx', grid=grid, staggered=(0, 1, 0))
vz = TimeFunction(name='vz', grid=grid, staggered=(0, 0, 1))
txx = TimeFunction(name='txx', grid=grid)
tzz = TimeFunction(name='tzz', grid=grid)
txz = TimeFunction(name='txz', grid=grid, staggered=(0, 1, 1))

# And finally we create the discretized/indexed symbols
VX = vx.indexed
VZ = vz.indexed
TXX = txx.indexed
TZZ = tzz.indexed
TXZ = txz.indexed
```

```python
def derivative(f, dim, order, stagger=centered):
    """
    Utility function to generate staggered derivatives
    """
    diff = dim.spacing
    if stagger == left:
        off = -.5
    elif stagger == right:
        off = .5
    else:
        off = 0.
    idx = [(dim + int(i+.5+off)*diff) for i in range(-int(order / 2), int(order / 2))]
    return f.diff(dim).as_finite_difference(idx, x0=dim + off*dim.spacing)
```

```python
# Now let's try and create the staggered updates
t = grid.stepping_dim
time = grid.time_dim

# We need some initial conditions
V_p = 4.0
V_s = 1.0
density = 3.

# The source injection term
src_xx = src.inject(field=txx.forward, expr=src)
src_zz = src.inject(field=tzz.forward, expr=src)

#c1 = 9.0/8.0;
#c2 = -1.0/24.0;

# Thorbecke's parameter notation
cp2 = V_p*V_p
cs2 = V_s*V_s
ro = 1/density

mu = cs2*ro
l = (cp2*ro - 2*mu)

# fdelmodc reference implementation
u_vx = Eq(vx.forward, vx - dt*ro*(derivative(txx, dim=x, order=4, stagger=left)
                                + derivative(txz, dim=z, order=4, stagger=right)))

u_vz = Eq(vz.forward, vz - ro*dt*(derivative(txz, dim=x, order=4, stagger=right)
                                + derivative(tzz, dim=z, order=4, stagger=left)))

u_txx = Eq(txx.forward, txx - (l+2*mu)*dt * derivative(vx.forward, dim=x, order=4, stagger=right)
                            -  l*dt       * derivative(vz.forward, dim=z, order=4, stagger=right))
u_tzz = Eq(tzz.forward, tzz - (l+2*mu)*dt * derivative(vz.forward, dim=z, order=4, stagger=right)
                            -  l*dt       * derivative(vx.forward, dim=x, order=4, stagger=right))

u_txz = Eq(txz.forward, txz - mu*dt * (derivative(vx.forward, dim=z, order=4, stagger=left)
                                     + derivative(vz.forward, dim=x, order=4, stagger=left)))
```

```python
op = Operator([u_vx, u_vz, u_txx, u_tzz, u_txz] + src_xx + src_zz)

```

```python
# Reset the fields
vx.data[:] = 0.
vz.data[:] = 0.
txx.data[:] = 0.
tzz.data[:] = 0.
txz.data[:] = 0.

op()
```

    GNUCompiler: cache hit `/tmp/devito-jitcache-uid944902/7180c0ae6fa2af6526472c104fe69c17e8f77f94.c` [0.01 s]
    =========================================================================================
    section0<<510,197,197>,<510,197,197>> with OI=2.32 computed in 0.077 s [24.15 GFlops/s, 1.28 GPts/s]
    section1<510,1> with OI=2.54 computed in 0.001 s [0.05 GFlops/s, 0.00 GPts/s]
    =========================================================================================

```python
# Let's see what we got....
plot_image(vx.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(vz.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(txx.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(tzz.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(txz.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
```

```python
# Now that looks pretty! But let's do it again with a higher order...
order = 12

u_vx = Eq(vx.forward, vx - dt*ro*(derivative(txx, dim=x, order=order, stagger=left)
                                + derivative(txz, dim=z, order=order, stagger=right)))
u_vz = Eq(vz.forward, vz - ro*dt*(derivative(txz, dim=x, order=order, stagger=right)
                                + derivative(tzz, dim=z, order=order, stagger=left)))
u_txx = Eq(txx.forward, txx - (l+2*mu)*dt * derivative(vx.forward, dim=x, order=order, stagger=right)
                            -  l*dt       * derivative(vz.forward, dim=z, order=order, stagger=right))
u_tzz = Eq(tzz.forward, tzz - (l+2*mu)*dt * derivative(vz.forward, dim=z, order=order, stagger=right)
                            -  l*dt       * derivative(vx.forward, dim=x, order=order, stagger=right))
u_txz = Eq(txz.forward, txz - mu*dt * (derivative(vx.forward, dim=z, order=order, stagger=left)
                                     + derivative(vz.forward, dim=x, order=order, stagger=left)))

op = Operator([u_vx, u_vz, u_txx, u_tzz, u_txz] + src_xx + src_zz)

# Reset the fields
vx.data[:] = 0.
vz.data[:] = 0.
txx.data[:] = 0.
tzz.data[:] = 0.
txz.data[:] = 0.

op()

plot_image(vx.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(vz.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(txx.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(tzz.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
plot_image(txz.data[0], vmin=-.5*1e-2, vmax=.5*1e-2, cmap="seismic")
```

```python

```


# Visualizing Graphical Models

Here we use [daft](https://docs.daft-pgm.org/en/latest/) by David S. Fulford, Dan Foreman-Mackey and David W. Hogg to visualize probabilistic graphical models .

For more on graphical models:

 1. [Foundations of Graphical Models by David Blei](http://www.cs.columbia.edu/~blei/fogm/2016F/) -- see [Basics of Graphical Models](http://www.cs.columbia.edu/~blei/fogm/2016F/doc/graphical-models.pdf)
    1. see also [a video on d-separation by Pieter Abbeel](https://www.youtube.com/watch?v=yDs_q6jKHb0)
    1. semantics of graphical models (called "Boiler plate diagrams" in the tech report) and an extended visual language [Directed Factor Graph Notation for Generative Models
Laura Dietz](https://github.com/jluttine/tikz-bayesnet/blob/master/dietz-techreport.pdf), which is the basis of the `tikz-bayesnet` package

```python
import daft
from matplotlib import rc

rc("font", family="serif", size=12)
rc("text", usetex=True)
```

## Conditional independence

Random variables $A$ and $B$ are conditionally independent given $C$ if and only if

\begin{equation}
P (A|B, C) = P (A|C) ,
\end{equation}

This is often written $ A \perp\kern-5pt\perp B \mid C$. The graphical model for this situation looks like this:

```python
pgm = daft.PGM()
pgm.add_node("A", r"$A$", -1, 0)
pgm.add_node("B", r"$B$", 1, 0)
pgm.add_node("C", r"$C$", 0, 1)
pgm.add_edge("C", "A")
pgm.add_edge("C", "B")
pgm.render(dpi=150)
pgm.savefig("../assets/AperpBmidC.png", dpi=150)
```

## An weak lensing example

[Origin of Weak Lensing Example](https://docs.daft-pgm.org/en/latest/examples/weaklensing/)

```python
pgm = daft.PGM()
pgm.add_node("Omega", r"$\Omega$", -1, 4)
pgm.add_node("gamma", r"$\gamma$", 0, 4)
pgm.add_node("obs", r"$\epsilon^{\mathrm{obs}}_n$", 1, 4, observed=True)
pgm.add_node("alpha", r"$\alpha$", 3, 4)
pgm.add_node("true", r"$\epsilon^{\mathrm{true}}_n$", 2, 4)
pgm.add_node("sigma", r"$\sigma_n$", 1, 3)
pgm.add_node("Sigma", r"$\Sigma$", 0, 3)
pgm.add_node("x", r"$x_n$", 2, 3, observed=True)
pgm.add_plate([0.5, 2.25, 2, 2.25], label=r"galaxies $n$")
pgm.add_edge("Omega", "gamma")
pgm.add_edge("gamma", "obs")
pgm.add_edge("alpha", "true")
pgm.add_edge("true", "obs")
pgm.add_edge("x", "obs")
pgm.add_edge("Sigma", "sigma")
pgm.add_edge("sigma", "obs")

pgm.render(dpi=150)
#pgm.savefig("weaklensing.pdf")
#pgm.savefig("weaklensing.png", dpi=150)
```

This graphical model is equivalent to factorizing the joint distribution according to this equation:

$$
p(\Omega, \Sigma, \gamma, \alpha, \{\epsilon_n^\textrm{obs},\epsilon_n^\textrm{true} \sigma_n, x_n\} ) =
\left[ \prod_n  p(\epsilon_n^\textrm{obs} | x_n, \sigma_n, \epsilon_n^\textrm{true}, \gamma) p(\epsilon_n^\textrm{true}|\alpha) p(x_n) p(\sigma_n | \Sigma) \right ] p(\alpha) p(\Sigma) p(\gamma | \Omega) p(\Omega)
$$

## An exoplanets example

[Origin of exoplanet example](https://docs.daft-pgm.org/en/latest/examples/exoplanets/)

```python
# Colors.
p_color = {"ec": "#46a546"}
s_color = {"ec": "#f89406"}

pgm = daft.PGM()

n = daft.Node("phi", r"$\phi$", 1, 3, plot_params=s_color)
n.va = "baseline"
pgm.add_node(n)
pgm.add_node("speckle_coeff", r"$z_i$", 2, 3, plot_params=s_color)
pgm.add_node("speckle_img", r"$x_i$", 2, 2, plot_params=s_color)

pgm.add_node("spec", r"$s$", 4, 3, plot_params=p_color)
pgm.add_node("shape", r"$g$", 4, 2, plot_params=p_color)
pgm.add_node("planet_pos", r"$\mu_i$", 3, 3, plot_params=p_color)
pgm.add_node("planet_img", r"$p_i$", 3, 2, plot_params=p_color)

pgm.add_node("pixels", r"$y_i ^j$", 2.5, 1, observed=True)

# Edges.
pgm.add_edge("phi", "speckle_coeff")
pgm.add_edge("speckle_coeff", "speckle_img")
pgm.add_edge("speckle_img", "pixels")

pgm.add_edge("spec", "planet_img")
pgm.add_edge("shape", "planet_img")
pgm.add_edge("planet_pos", "planet_img")
pgm.add_edge("planet_img", "pixels")

# And a plate.
pgm.add_plate([1.5, 0.2, 2, 3.2], label=r"exposure $i$", shift=-0.1)
pgm.add_plate([2, 0.5, 1, 1], label=r"pixel $j$", shift=-0.1)

# Render and save.
pgm.render(dpi=150)
#pgm.savefig("exoplanets.pdf")
#pgm.savefig("exoplanets.png", dpi=150)
```


#   Initial Boundary Value PDEs

# Learning Objectives

- Be able to recognize Laplace & Poisson equations
- Describe the basic steps involved in the Method of Lines (MOL)
- Apply knowledge of how discretize interior nodes
- Apply knowledge of how to discretize exterior nodes for each boundary condition (Neumann, Robin, Dirichlet)
- Formulate the system of ODEs created with MOL by interlacing variables on a rectangular grid.

# Simulating a graded catalyst bed

Process engineers are often engaged in tasks centered around increasing the economic productivity of process equipment while ensuring safe operation. For the commodity chemical industry, increased conversion of reactant to product remains a desired outcome. One of the most common used continuously operating reactor in this sector is that of the catalytic packed bed reactor. In this configuration, a tubular reactor is employed. This tube is packed with catalyst covered particles and the reactant flows over the catalyst bed. In many cases, a large number of tubes will be used in parallel cooled by the same jacket. We'll just consider the single tube variant here. A simplifed depiction is given below.

One important industrial chemical is that ortho-xylene which is used to produce phthalic anhydride a common plastizer.
The desired reaction is:

However, two underdesirable side reactions are also known to occur:

The reactor is typically operated such at a hotspot occurs in the reactor, the reactor is stable (and in turn safe), and a high degree of selectivity and conversion is achieved. A number of process parameters including pressure, temperature, residence time, cooling rate, and feed composition may be manipulated to improve reactor productivity, selectivity, conversion, or overall profitability. We'll build a simulation (based on the model in this [paper](https://pubs.acs.org/doi/abs/10.1021/ie4005699).) to investigate the how using a graded catalyst may impact reactor performance. That is to say a reactor in which differing zones are loaded with catalyst of differing activity levels. This can allow the operator to raise the effective reactor temperature while preventing thermal runaway and maintaining adequate selectivity.

## A basic descriptive model

A two-dimensional steady-state pseudohomogeneous packed bed reactor equations consist of a mass balance and an energy balance:

\\[u_s\frac{\partial \mathbf{c}}{\partial z} = \frac{D}{R}\frac{\partial}{\partial R}\left(R\frac{\partial \mathbf{c}}{\partial R}\right) + \sum_{i}{\nu_{ij}r_i(\mathbf{c},T)}  \\]

\\[u_s \rho_{f} \left(\sum_{i}{C_{p,i}c_i}\right) \frac{\partial T}{\partial z} = \frac{\Lambda}{R}\frac{\partial}{\partial R}\left(R\frac{\partial T}{\partial R}\right) + \sum_{i}{(-\Delta H_{i})r_i(\mathbf{c},T)}  \\]

were OX is denotes ortho-xylene, PA denotes phthalic anhydride and

\\[j \in \{c_{OX,1}, c_{PA,1}, c_{H_2 O,1}, c_{O_2,1}, c_{CO_2,1}, c_{N_2}\}\\].

For a full review of different packed bed reactor models, the reader is directed to [this reference](https://ris.utwente.nl/ws/portalfiles/portal/6073612/t0000040.pdf).

Form of the above equations: The above equations are coupled convection-diffusion equations. One of the most commonly encountered forms in fluid mechanics.

Limiting cases: Consider briefly under what conditions we might assume that these equations reduce to the Poisson or Laplace equations below.

1. **Poisson Equation**: \\[\nabla^2 x = f(x)\\]
2. **Laplace Equation**: \\[\nabla^2 x = 0\\]

Next, we assume a sufficient cooling flow exists to hold the reactor wall temperature constant at a fixed value of $T^c$. Then we can write a symmetry and cooling boundary condition:

\\[\left.\frac{dT}{dR}\right\vert_{R=0} = 0 \qquad \qquad \left.\frac{dT}{dR}\right\vert_{R=R_t} = -Bi(T - T^c) \\]

Note: The wall boundary condition ($R = R_t$) is of the Robin type and the symmetry condition ($R = 0$) is of the Neumann type.

We further assume that the reactor wall is impermeable to all species (and note that similar symmetry conditions should also hold):

\\[\left.\frac{d\mathbf{c}}{dR}\right\vert_{R=0} = 0 \qquad \qquad \left.\frac{d\mathbf{c}}{dR}\right\vert_{R=R_t} = 0 \\]

## Further model development

- Assume the inlet stream is well-mixed.
\\[\begin{align} \mathbf{c}(z=0) &= \mathbf{c}_{in} \\
     T(z=0) &= T_{in} \end{align}\\]

Note: The inlet conditions are examples of Dirichlet boundary conditions.

Activity! If the type of the boundary condition isn't immediate obvious take a moment and see if you can rearrange these equations into one of the below forms.

1. **Dirichlet Boundary Condition**: \\[x = a\\]
2. **Neumann Boundary Condition**: \\[\hat{n}\cdot\nabla x = f(x)\\]
3. **Robin Boundary Condition**: \\[\hat{n}\cdot\nabla x + \alpha x = f(x)\\]

- All reactions are treated as pseudo-first-order. Each reaction rate may be written with relation to it's partial pressure which in turn is related to the concentration assuming an ideal gas equaton of state.

\\[r_1 = (\sigma P_{O_2}\rho_s) k_1 P_{OX} = k_1 (\sigma \rho_s)/(R_g T)^2 c_{O_2} c_{OX} \\]
\\[r_2 = (\sigma P_{O_2}\rho_s) k_2 P_{PA} = k_2 (\sigma \rho_s)/(R_g T)^2 c_{O_2} c_{PA} \\]
\\[r_3 = (\sigma P_{O_2}\rho_s) k_3 P_{OX} = k_3 (\sigma \rho_s)/(R_g T)^2 c_{O_2} c_{OX} \\]

The rate constant can be calculated using the Arrhenius relationship as such

\\[k_i = k_i^r \exp{\frac{E^r_i(T - T^r)}{T R_g T^r}}, \qquad i = \{1, 2, 3\} \\]

Individual component rates of change may then be written as:

\\[\begin{align} r_{OX} &= -r_1 + r_3 \\
r_{PA} &= r_1 - r_2 \\
r_{H_2 O} &= 3r_1 + 2r_2 + 5r_3 \\
r_{O_2} &= -3r_1 - 7.5r_2 + 10.5r_3 \\
r_{CO_2} &= 8r_2 + 8r_3 \\
r_{N_2} &= 0
\end{align}\\]

We omit a further discussion of model parameters in favor of including stating there values in the included code.

## Method of Lines Derivation

We now make use of second-order approximations for the derivatives in the radial direction

\\[\left.\frac{\partial u}{\partial R}\right\vert_{r=R_j} = \frac{u_{j+1} - u_{j-1}}{2\Delta R} \\]
\\[\left.\frac{\partial^2 u}{\partial R^2}\right\vert_{r=R_j} = \frac{u_{j+1} - 2u_{j} + u_{j-1}}{\Delta R^2} \\]

where \\[\Delta R_{j} = R_{j+1} - R_{j}\\]

### Method of Lines Derivation - Interior
Applying these rules to the mass balance for the interior nodes, we have

\\[ \left.\frac{\partial \mathbf{c}}{\partial z}\right\vert_{R=R_j} = u_s^{-1}\left(\frac{D}{R}\frac{\partial \mathbf{c}}{\partial R} + \frac{\partial^2 \mathbf{c}}{\partial R^2} + \sum_{i}{\nu_{ij}r_i(\mathbf{c},T)}\right)  \qquad j = 2, \ldots, N_{r-1}\\]

\\[ \left.\frac{d \mathbf{c}_j}{d z}\right\vert_{R=R_j} = u_s^{-1}\left(\frac{D}{R}\frac{\mathbf{c}_{j+1} - \mathbf{c}_{j-1}}{2\Delta R} + D\frac{\mathbf{c}_{j+1} - 2\mathbf{c}_{j} + \mathbf{c}_{j-1}}{\Delta R^2} + \sum_{i}{\nu_{ij}r_i(\mathbf{c},T)}\right)  \qquad j = 2, \ldots, N_{r-1}\\]

Accordlingly, the temperature balance becomes

\\[\left.\frac{\partial T}{\partial z}\right\vert_{R=R_j} = \left(u_s\sum_{i}{f_{i}c_{pj}}\right)^{-1}\left(\frac{\Lambda}{R}\frac{\partial}{\partial R}\left(R\frac{\partial T}{\partial R}\right) + \sum_{i}{(-\Delta H_{i})r_i(\mathbf{c},T)}\right) \qquad j = 2, \ldots, N_{r-1}   \\]

\\[ \left.\frac{d T}{d z}\right\vert_{R=R_j} = \left(u_s\sum_{i}{f_{i}c_{pj}}\right)^{-1}\left(\frac{D}{R}\frac{T_{j+1} - T_{j-1}}{2\Delta R} + D\frac{T_{j+1} - 2T_{j} + T_{j-1}}{\Delta R^2} + \sum_{i}{\nu_{ij}r_i(\mathbf{c},T)}\right)  \qquad j = 2, \ldots, N_{r-1}\\]

### Method of Lines Derivation - Center
The discretization at the center node yeilds:
\\[ \frac{d \mathbf{c}_1}{d z} = u_s^{-1}\left(\frac{D}{R}\frac{\mathbf{c}_{2} - \mathbf{c}_{0}}{2\Delta R} + D\frac{\mathbf{c}_{2} - 2\mathbf{c}_{1} + \mathbf{c}_{0}}{\Delta R^2} + \sum_{i}{\nu_{ij}r_i(c_1,T_1)}\right)\\]

from the boundary condition we have \\[\frac{d \mathbf{c}_1}{d R} = 0 \rightarrow \frac{\mathbf{c}_{2} - \mathbf{c}_{0}}{2\Delta R} = 0 \rightarrow  \mathbf{c}_{0} = \mathbf{c}_{2}\\]. Substituting this in yeilds:

\\[ \frac{d \mathbf{c}_1}{d z} = u_s^{-1}\left(\frac{2D}{\Delta R^2}(\mathbf{c}_{2} - \mathbf{c}_{1}) + \sum_{i}{\nu_{ij}r_i(\mathbf{c}_1,T_1)}\right)\\]

Accordingly for the energy balance we have:

\\[ \frac{d T_1}{d z} = \left(u_s\sum_{i}{f_{i}c_{pj}}\right)^{-1}\left(\frac{\Lambda}{R}\frac{T_{2} - T_{0}}{2\Delta R} + \Lambda \frac{T_{2} - 2T_{1} + T_{0}}{\Delta R^2} + \sum_{i}{(-\Delta H_{i})r_i(\mathbf{c}_1,T_1)}\right)\\]

from the boundary condition we have \\[\frac{d T_1}{d R} = 0 \rightarrow \frac{T_{2} - T_{0}}{2\Delta R} = 0 \rightarrow  T_{0} = T_{2}\\]. Substituting this in yeilds:

\\[ \frac{d T_1}{d z} = \left(u_s\sum_{i}{f_{i}c_{pj}}\right)^{-1}\left(\frac{2\Lambda}{\Delta R^2}(T_{2} - T_{1}) + \sum_{i}{(-\Delta H_{i})r_i(\mathbf{c}_1,T_1)}\right)\\]

Activity! This derivation proceeded by solving obtaining algebraic equations from the boundary condition, solving them analytically for the fictive node value, and then substituting this expression in. We could have instead solved a system of coupled differential and algebraic equations (referred to as a differential-algebraic system of equations or DAEs). In many cases, the algebraic equations formed may not have an analytic (closed form) solution and we must resort to solving DAEs. What type of boundary condition(s) may lead algebraic equations with no analytic solution?

### Method of Lines Derivation - Wall
The discretization at the wall node yeilds:
\\[ \frac{d \mathbf{c}_R}{d z} = u_s^{-1}\left(\frac{D}{R}\frac{\mathbf{c}_{R+1} - \mathbf{c}_{R-1}}{2\Delta R} + D\frac{\mathbf{c}_{R+1} - 2\mathbf{c}_{R} + \mathbf{c}_{R-1}}{\Delta R^2} + \sum_{i}{\nu_{ij}r_i(\mathbf{c}_R,T_R)}\right)\\]

from the boundary condition we have

\\[\frac{d \mathbf{c}_R}{d R} = 0 \rightarrow \frac{\mathbf{c}_{R+1} - \mathbf{c}_{R-1}}{2\Delta R} = 0 \rightarrow  \mathbf{c}_{R+1} = \mathbf{c}_{R-1}.\\]

Substituting this in yeilds:

\\[ \frac{d \mathbf{c}_R}{d z} = u_s^{-1}\left(\frac{2D}{\Delta R^2}(\mathbf{c}_{R-1} - \mathbf{c}_{R}) + \sum_{i}{\nu_{ij}r_i(\mathbf{c}_R,T_R)}\right)\\]

Accordingly for the energy balance we have:

\\[ \frac{d T_R}{d z} = \left(u_s\sum_{i}{f_{i}\mathbf{c}_{pj}}\right)^{-1}\left(\frac{\Lambda}{R}\frac{T_{R+1} - T_{R-1}}{2\Delta R} + \Lambda \frac{T_{R+1} - 2T_{R} + T_{R-1}}{\Delta R^2} + \sum_{i}{(-\Delta H_{i})r_i(\mathbf{c}_R,T_R)}\right)\\]

from the boundary condition we have

\\[\frac{d T_R}{d R} = 0 \rightarrow \frac{T_{R+1} - T_{R-1}}{2\Delta R} = -Bi(T_R - T^c) \rightarrow  T_{R+1} = T_{R-1} -2Bi\Delta R (T_R - T^c).\\]

Substituting this in yeilds:

\\[ \frac{d T_R}{d z} = \left(u_s\sum_{i}{f_{i}c_{pj}}\right)^{-1}\left(\frac{Bi\Lambda}{R}(T_{C} - T_{R}) + \frac{2\Lambda}{\Delta R^2}(T_{R-1} - T_{R} + Bi\Delta R (T_{C} - T_{R})) + \sum_{i}{(-\Delta H_{i})r_i(\mathbf{c}_R,T_R)}\right)\\]

We now need to interlace the variables letting \\[ y = \{T_1, c_{OX,1}, c_{PA,1}, c_{H_2 O,1}, c_{O_2,1}, c_{CO_2,1}, c_{N_2}, \ldots, T_{N_R}, c_{OX,N_R}, c_{PA,N_R}, c_{H_2 O,N_R}, c_{O_2,N_R}, c_{CO_2,N_R}, c_{N_2,N_R}\}\\]

We'll now use a nonlinear equation solving package in Julia to write a simple a fixed-step size implicit Euler integration method (progressively, solving the below equation for $y_{z+1}$)

\\[ y_{z+1} - y_z - \Delta z f(z_{k+1}) = 0 \\]

Activity! Take some time to look over the code for the residual. Where does the right-hand side of the ODE system appear? Note how the physical property calculations are implemented.

For simplicity sake, we'll assume the follow physical constants are valid over the range of interest (constant heat capacities, enthalpy of reaction, diffusion, and thermal heat dispersion coefficients but we'll allow reaction rate to vary with temperature.

```julia
#=
Define rhs of implicit Euler step.
=#
function residual!(out, y1, y2::Vector{W}, a, params) where W

    # unpack parameters
    nr, nz, delR, delZ, catalyst_density, reactor_diam, biot_number,
    Cp, delH, Tc, diff_coeff, heat_coeff, Rg, k_ref, E_ref, T_ref, us = params

    # compute step size
    dr = 1/(nr - 1)

    # preallocate storage
    r_rxn = zeros(W,3)
    r_prod = zeros(W,6)
    k = zeros(W,3)

    # pollulate the rhs of the ODEs for the interior nodes
    for i = 2:(nr-1)

        # get node radial position and temperature
        r = (i-1)*nr
        T = y2[7*(i-1) + 1]

        # compute reaction rate constant
        k[1] = k_ref[1]*exp(E_ref[1]*(T - T_ref)/(T*Rg*T_ref))
        k[2] = k_ref[2]*exp(E_ref[2]*(T - T_ref)/(T*Rg*T_ref))
        k[3] = k_ref[3]*exp(E_ref[3]*(T - T_ref)/(T*Rg*T_ref))

        # compute reaction rate
        r_rxn[1] = a*catalyst_density*k[1]*y2[7*(i-1) + 2]*y2[7*(i-1) + 5]
        r_rxn[2] = a*catalyst_density*k[2]*y2[7*(i-1) + 3]*y2[7*(i-1) + 5]
        r_rxn[3] = a*catalyst_density*k[3]*y2[7*(i-1) + 2]*y2[7*(i-1) + 5]

        # compute production rate
        r_prod[1] = -r_rxn[1] - r_rxn[3]
        r_prod[2] = r_rxn[1] - r_rxn[2]
        r_prod[3] = 3.0*r_rxn[1] + 2.0*r_rxn[2] + 5.0*r_rxn[3]
        r_prod[4] = -3.0*r_rxn[1] - 7.5*r_rxn[2] - 10.5*r_rxn[3]
        r_prod[5] = 8.0*r_rxn[2] + 8.0*r_rxn[3]
        r_prod[6] = 0.0

        # energy balance discretized equation
        prefT = delZ*Cp*us
        sourceT = zero(T)
        for j = 1:3
            sourceT += (-delH[j])*r_rxn[j]
        end
        Tp1 = y2[7*(i-1) + 8]
        Tm1 = y2[7*(i-1) - 6]
        term1 = heat_coeff*(Tp1 - Tm1)/(2*dr*r)
        term2 = heat_coeff*(Tp1 - 2*T + Tm1)/dr^2
        out[7*(i-1) + 1] = y2[7*(i-1) + 1] - y1[7*(i-1) + 1] + prefT*(term1 + term2 + sourceT)

        # mass balance discretized equations
        prefC = delZ*us
        for j = 0:5
            Cp1 = y2[7*(i-1) + 9 + j]
            C1 = y2[7*(i-1) + 2 + j]
            Cm1 = y2[7*(i-1) - 5 + j]
            term1 = diff_coeff*(Tp1 - Tm1)/(2*dr*r)
            term2 = diff_coeff*(Tp1 - 2*T + Tm1)/dr^2
            out[7*(i-1) + 2 + j] = y2[7*(i-1) + 2 + j] - y1[7*(i-1) + 2 + j]  + prefC*(term1 + term2 + r_prod[6 - j])
        end
    end

    # pollulate the rhs of the ODEs for the center nodes
    T = y2[1]

    # compute reaction rate constant
    k[1] = k_ref[1]*exp(E_ref[1]*(T - T_ref)/(T*Rg*T_ref))
    k[2] = k_ref[2]*exp(E_ref[2]*(T - T_ref)/(T*Rg*T_ref))
    k[3] = k_ref[3]*exp(E_ref[3]*(T - T_ref)/(T*Rg*T_ref))

    # compute reaction rate
    r_rxn[1] = a*catalyst_density*k[1]*y2[2]*y2[5]
    r_rxn[2] = a*catalyst_density*k[2]*y2[3]*y2[5]
    r_rxn[3] = a*catalyst_density*k[3]*y2[2]*y2[5]

    # compute production rate
    r_prod[1] = -r_rxn[1] - r_rxn[3]
    r_prod[2] = r_rxn[1] - r_rxn[2]
    r_prod[3] = 3.0*r_rxn[1] + 2.0*r_rxn[2] + 5.0*r_rxn[3]
    r_prod[4] = -3.0*r_rxn[1] - 7.5*r_rxn[2] - 10.5*r_rxn[3]
    r_prod[5] = 8.0*r_rxn[2] + 8.0*r_rxn[3]
    r_prod[6] = 0.0

    # energy balance discretized equation
    prefT = delZ*Cp*us
    sourceT = zero(T)
    for i = 1:3
        sourceT += (-delH[i])*r_rxn[i]
    end
    out[1] = y2[1] - y1[1] + prefT*((2*heat_coeff/dr^2)*(y2[8] - T) + sourceT)

    # mass balance discretized equations
    prefC = delZ*us
    for i = 0:5
        CR1 = y2[i + 9]
        CR = y2[i + 2]
        out[i + 2] = y2[i + 2] - y1[i + 2]  + prefC*((2*diff_coeff/dr^2)*(CR1 - CR) + r_prod[6 - i])
    end

    # pollulate the rhs of the ODEs for the walls nodes
    T = y2[7*nr - 6]

    # compute reaction rate constant
    k[1] = k_ref[1]*exp(E_ref[1]*(T - T_ref)/(T*Rg*T_ref))
    k[2] = k_ref[2]*exp(E_ref[2]*(T - T_ref)/(T*Rg*T_ref))
    k[3] = k_ref[3]*exp(E_ref[3]*(T - T_ref)/(T*Rg*T_ref))

    # compute reaction rate
    r_rxn[1] = a*catalyst_density*k[1]*y2[1]*y2[4]
    r_rxn[2] = a*catalyst_density*k[2]*y2[2]*y2[4]
    r_rxn[3] = a*catalyst_density*k[3]*y2[1]*y2[4]

    # compute production rate
    r_prod[1] = -r_rxn[1] - r_rxn[3]
    r_prod[2] = r_rxn[1] - r_rxn[2]
    r_prod[3] = 3.0*r_rxn[1] + 2.0*r_rxn[2] + 5.0*r_rxn[3]
    r_prod[4] = -3.0*r_rxn[1] - 7.5*r_rxn[2] - 10.5*r_rxn[3]
    r_prod[5] = 8.0*r_rxn[2] + 8.0*r_rxn[3]
    r_prod[6] = 0.0

    # energy balance discretized equation
    prefT = delZ*Cp*us
    sourceT = zero(T)
    for i = 1:3
        sourceT += (-delH[i])*r_rxn[i]
    end
    TR_m_1 = y2[7*nr - 13]
    term1 = (biot_number*heat_coeff/reactor_diam)*(Tc - T)
    term2 = (2*heat_coeff/dr^2)*(TR_m_1 - T + biot_number*dr*(Tc - T))
    out[7*nr - 6] = y2[7*nr - 6] - y1[7*nr - 6] + prefT*(term1 + term2 + sourceT)

    # mass balance discretized equations
    prefC = delZ*us
    for i = 0:5
        CR1 = y2[7*nr - i]
        CR = y2[7*(nr-1) - i]
        out[7*nr - i] = y2[7*nr - i] - y1[7*nr - i] + prefC*((2*diff_coeff/dr^2)*(CR1 - CR) + r_prod[6 - i])
    end

    return nothing
end

println("residual function defined")
```

    residual function defined

We'll use an off the shelf implementation of Newton's method through the package NLSolve.jl.

```julia
import Pkg; Pkg.add("NLsolve")
using NLsolve

function activity_profile(v)
    if v < 0.25
        return 0.9  # keep between 0.9 and 1.0
    elseif v < 1.0
        return 0.7
    elseif v < 1.5
        return 1.0
    end
    return 1.0
end

function nonlinear_step_solve!(out, indx, z, initial_y, params)
    a = activity_profile(z)
    residual_func! = (out, y2) -> residual!(out, initial_y, y2, a, params)
    result = nlsolve(residual_func!, initial_y, autodiff = :forward, method = :newton)
    out[indx,:] = result.zero
    return nothing
end

function integrate_ODEs!(out, initial_y, params)
    nr, nz, delR, delZ, catalyst_density, reactor_diam, biot_number,
    Cp, delH, Tc, diff_coeff, heat_coeff, Rg, k_ref, E_ref, T_ref, us = params
    z = 0.0
    out[1,:] = initial_y                                            # store initial condition
    for indx = 2:nz
        z += delZ                                                   # advance axial dimension value
        nonlinear_step_solve!(out, indx, z, out[indx-1,:], params)  # perform a nonlinear solve for next step
    end
    return nothing
end

println("defined integration scheme")
```

       Updating registry at `C:\Users\wilhe\.julia\registries\General`
      Resolving package versions...
    No Changes to `C:\Users\wilhe\Project.toml`
    No Changes to `C:\Users\wilhe\Manifest.toml`

    defined integration scheme

```julia
# set up discretization points
nr = 20 # number of discretization points in radial direction
nz = 80 # number of discretization points in axial direction

# set reactor design parameters
reactor_length = 3      # length of reactor
reactor_diam = 0.0254   # diameter of reactor

delR = reactor_diam/(nr - 1)
delZ = reactor_length/(nz - 1)

# Input model parameters
k_ref = [6.519E-2; 5.698E-3; 6.442E-3]                      # Reaction rate at refence temperature
E_ref = [113.57; 129.71; 119.68]                            # Activation energy at reference temperature
T_ref = 600                                                 # Reference temperature
Rg = 8.3144*1000                                            # ideal gas constant

catalyst_density = 1300                                     # Density of packed catalytic material
diff_coeff = 0.5
heat_coeff = 7.3871                                         # heat dispersion coefficient

Tc = 323.15                                                 # 50C water used for cooling
T0 = 625
biot_number = 1.6                                           # Biot number, Bi

us = 0.8
Cp = 0.992                                                  # approximate
delH = [-1.361; -14.329; -18.809]*1000                      # heat of reaction

# define initial condition
feed_rate = 0.174                                           # total molar feed rate
feed_conc_split = [0.011; 0.208; 0.781; 0.0; 0.0; 0.0]      # mole fraction

initial_y = zeros(7*nr)                                     # create vector of zeros
for i = 1:nr
    initial_y[7*(i-1) + 1] = T0
    initial_y[7*(i-1) + 2] = feed_conc_split[1]*feed_rate
    initial_y[7*(i-1) + 3] = feed_conc_split[2]*feed_rate
    initial_y[7*(i-1) + 4] = feed_conc_split[3]*feed_rate   # only assign nonzero values
end

# specify parameters
params = nr, nz, delR, delZ, catalyst_density, reactor_diam, biot_number,
Cp, delH, Tc, diff_coeff, heat_coeff, Rg, k_ref, E_ref, T_ref, us

println("parameter defined")

# preallocate storage
out = zeros(nz, 7*nr)

# solve the system
integrate_ODEs!(out, initial_y, params)

println("ODE integrated")
```

    parameter defined
    ODE integrated

Extract coupled values and generate 2D plot

```julia
function extract_values(out, nr, nz)
    T = zeros(nr,nz)
    cOX = zeros(nr,nz)
    cPA = zeros(nr,nz)
    cH2O = zeros(nr,nz)
    cO2 = zeros(nr,nz)
    cCO2 = zeros(nr,nz)
    cN2 = zeros(nr,nz)
    for i = 1:nr
        for j = 1:nz
            T[i,j] = out[j, 7*(i-1) + 1]
            cOX[i,j] = out[j, 7*(i-1) + 2]
            cPA[i,j] = out[j, 7*(i-1) + 3]
            cH2O[i,j] = out[j, 7*(i-1) + 4]
            cO2[i,j] = out[j, 7*(i-1) + 5]
            cCO2[i,j] = out[j, 7*(i-1) + 6]
            cN2[i,j] = out[j, 7*(i-1) + 7]
        end
    end
    return T, cOX, cPA, cH2O, cO2, cCO2, cN2
end

T, cOX, cPA, cH2O, cO2, cCO2, cN2 = extract_values(out, nr, nz)

println("properties extracted")
```

    properties extracted

We now generate a plot of the centerline temperature versus position. If Plots.jl hasn't been installed yet this may take some time.

```julia
import Pkg; Pkg.add("Plots")
using Plots

Tcenter = [T[1,i] for i in 1:nz]
zpnt = [i for i in range(0.0, reactor_length, length = nz)]
plot(zpnt, Tcenter, label="", linealpha = 0.5, marker = (:hexagon, 2, 0.6, :green))
```

      Resolving package versions...
    No Changes to `C:\Users\wilhe\Project.toml`
    No Changes to `C:\Users\wilhe\Manifest.toml`

Activity! In the below box, compute the average temperature, concentrations, and then conversion at each spatial position. Next try, to modifying the activity_profile and re-running the example. Can you pick out can clear trend in the behavior of the system with respect to temperature/conversion with respect to activity profile?

```julia

```

If you manipulated this example substantially, you may have discovered that some of the simulation results you reached were not physically meaningful. Large oscillations may be observed or even negative values. This is actually expected behavior for many PDE systems which may require specialized discretization methods to be dealt with (such as those encountered in [this reference]((https://ris.utwente.nl/ws/portalfiles/portal/6073612/t0000040.pdf))). Alternatively, one can formulate this simulation as an optimization problem (which can accept constraints such as nonnegative concentration). This later approach can be of particular interest when we are attempting to choose design/process parameters from a broad set of possibility in which many options are invalid.

# Questions for reflection
- For a simple, what factors influence whether a system is numerically stable?
- For IVP-PDEs, the problem size can grow quite rapidly. For a 3D systems, using 20 discretization points in each dimension leads to an 8000-by-8000 system while 256 points leads to a ~16 million by ~16 million system. As illustrated by [Jaroudi, et al.](https://www.tandfonline.com/doi/full/10.1080/00207160.2019.1613526?af=R), a large number of such systems can be solved in a few hours on standard desktop. In light of this fact, how do you think the resulting linear systems formulated and solved?
- What are some other questions that you'd expect to be important when design a chemical looping combustion system? Can you provide some ideas of how numerical methods could be used to help answer these?

tutorial/julia/Tutorial 5 -  Partial Differential Equations/.ipynb_checkpoints/Initial Value PDEs-checkpoint.ipynb


# Estadística II

# 1. Parámetros de estimación e hipótesis.

Cuando tenemos una población cuya distribución es conocida pero se desconoce algún parámetro, es posible realizar la estimación de este parámetro si se tiene una muestra representativa.

Un estimador de un parámetro es un valor que puede calcularse a partir de los datos muestrales y que proporciona información sobre el valor del parámetro. Por ejemplo la media muestral es un estimador de la media poblacional, la proporción en la muestra es un estimador de la proporción en la población.

Una estimación es puntual cuando se obtiene un sólo valor para el parámetro.

### 1. Intervalo de confianza para la media.

A partir de una población, se desconoce la media $\mu$, sin embargo, deseamos estimarla a partir de la media de una muestra $\bar{x}$ obtenida en una muestra de tamaño $N$. De esta población se conoce la desviación $\sigma$.

En este caso, se sabe que la distribución muestral de medias es $N(\mu,\sigma/\sqrt{N})$, por lo tanto si se fija una probabilidad 1-$\alpha$ para aceptar lo dicho, se sabe que

\begin{equation}
P\bigg(\mu-z_{\alpha/2}\frac{\sigma}{\sqrt{n}}<\bar{x}-\mu+z_{\alpha/2}\frac{\sigma}{\sqrt{N}}\bigg)=1-\alpha
\end{equation}

esto significa que si el $(1-\alpha)$ de las $\bar{x}$ está a una distancia de $\mu$ inferior a $z_{\alpha/2}\frac{\sigma}{\sqrt{N}}$:

- Entonces para un nivel de confianza de $1-\alpha$, $\mu$ pertenece al intervalo $\bigg( \bar{x}-z_{\alpha/2} \frac{\sigma}{\sqrt{N}} , \bar{x}+z_{\alpha/2} \frac{\sigma}{\sqrt{N}}\bigg)$

Si la desviación típica de la población es desconocida, se aproximar a la de la muestra siempre y cuando $N$ sea muy grande

# 2. Pruebas de hipótesis:

Les pruebas de hipótesis son métodos estadísticos utilizados para tomar decisiones basados en datos. En las pruebas de hipótesis, en general, hacemos suposiciones acerca de los parámetros de la población y buscamos buscar conclusiones matemáticas sobre nuestras suposiciones.

Una prueba de hipótesis evalúa dos afirmaciones mutuamente excluyentes acerca de una población para determinar qué afirmación está mejor soportada por los datos de la muestra.

### Parámetros de hipótesis.
##### Hipótesis nula:
En estadística inferencial, la hipótesis nula es una afirmación general acerca de que no existe relación entre fenómenos medidos. Es la suposición básica basada en el conocimiento del problema. Se prueba la similaridad de esta afirmación de ser verdadera en orden de decidir aceptar o rechazar la hipótesis alternativa.

##### Hipótesis alternativa:
La hipótesis alternativa es contraria a la hipótesis nula. Se determina si rechazar o no esta afirmación basados en la similaridad (likelihood) de la hipótesis nula de ser verdadera.

### Nivel de significancia

El nivel de significancia hace alusión al grado de significancia con la cuál se acepta o rechaza la hipótesis nula. El nivel de significancia usualmente se denota con $\alpha$ y es generalmente de $0.05$

#### Error tipo I:
Se tiene cuando se rechaza una hipótesis nula, a pesar que la hipótesis sea verdadera. Se denota por $\alpha$.

#### Error tipo II:
Se tiene cuando se acepta una hipótesis nula, pero es falsa. Los errores tipo II son denotados por $\beta$. En pruebas de hipótesis, las curvas normales que muestran la región de aceptancia se llaman regiones $\beta$.

#### Pruebas de una cola:
Es un tipo de test de hipótesis estadísticas, donde la región de rechazo es sólo de un lado de la distribución muestreada.

#### Pruebas de dos colas:
Una prueba de dos colas es una prueba estadística en la cual el área crítica de una distrbución es de dos lados y prueba si una muestra es más grande o menos grande que ciertos intervalos de valores. Si la muestra siendo muestreada cae en las áreas críticas, la hipótesis alternativa es aceptada en lugar de la nula.

# 3. P-Value
El valor P, o probabilidad calculada, es la probabilidad de encontrar lo observado, resulta cuando la hipótesis nula de un estudio es verdadera. Si el valor P es mayor que la significancia escogida, significa que se debe rechazar la hipótesis nula, es decir, aceptar que la muestra da la suficiente evidencia de apoyar la hipótesis alternativa. Esto no implica que exista realmente una diferencia significativa o importante, por lo que el criterio de la persona que interpreta la prueba de hipótesis es importante para considerar la relevancia del resultado.

# Extra: Bootstraping

Permite hacer uso de datos remuestreados para realizar inferencia estadística. Una réplica bootstrap

Ejemplo:
Es posible crear una función para crear réplicas en bootstrap utilizando np.random.choice(datos, funcion). En esta ocasión realizaremos un bootstrap de la media, tomaremos los datos de michelson_speed_of_light para realizar replicas de bootstrap en 1 dimensión utilizando choice y luego retornaremos una histograma de frecuencias para la media.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

data_michelson = pd.read_csv('datasets/michelson_speed_of_light.csv')

msl=np.array(data_michelson['velocity of light in air (km/s)'])

def bootstrap_replicate_1d(data, func):

    bs_sample = np.random.choice(data, len(data))
    return func(bs_sample)

bootstrap_replicate_1d(msl, np.mean)

replicas = np.empty(10000)

for i in range(10000):
    replicas[i] = bootstrap_replicate_1d(msl, np.mean)

plt.hist(replicas, bins=30, density=True)
plt.xlabel('Velocidad media de la luz (km/s)')
plt.ylabel('PDF')
plt.show()

```

    <Figure size 640x480 with 1 Axes>

# Intervalo de confianza

Ahora, para mostrar visualmente la representación de un intervalo de confianza, obtengamos el intervalo de confianza de los datos previamente obtenidos utilizando np.percentile para obtener los valores del intervalo del 95%.

```python
conf_int = np.percentile(replicas, [2.5, 97.5])
print(conf_int)

plt.hist(replicas, bins=30, density=True)
plt.xlabel('Velocidad media de la luz (km/s)')
plt.ylabel('PDF')
plt.fill_between(x=conf_int,y1=[0,0],y2=[0.05,0.05],alpha=0.4,label='P(x)')
plt.show()

```

```python
df = pd.read_csv('datasets/2008_all_states.csv')
df_swing = pd.read_csv('datasets/2008_swing_states.csv')
iris = pd.read_csv('datasets/iris.csv')

sns.jointplot(x='total_votes', y='dem_share', data=df_swing, kind="reg");
```

Se pueden realizar permutaciones con np.random.permutation para reordenar las entradas en un array.

```python
dem_share = df[['dem_share', 'state']]
dem_share_PA = dem_share.loc[dem_share['state'] == 'PA']
dem_share_OH = dem_share.loc[dem_share['state'] == 'UT']

def diff_frac(data_A, data_B):
    frac_A = np.sum(data_A) / len(data_A)
    frac_B = np.sum(data_B) / len(data_B)
    return frac_B - frac_A

dem_share_both = np.concatenate((dem_share_PA, dem_share_OH))
dem_share_perm = np.random.permutation(dem_share_both)

perm_sample_PA = dem_share_perm[:len(dem_share_PA)]
perm_sample_OH = dem_share_perm[len(dem_share_PA):]
```

Ejercicio 1: Realice las CDFs de PA y OH según el dem_share.

Ejercicio 2: Realice el estimador de la media de la diferencia de votos porcentuales entre PA-OH, realice el histograma de la PDF estimada.

```python

```