$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
$$

# EECS 545:  Machine Learning
## Lecture 04:  Linear Regression I
* Instructor:  **Jacob Abernethy, Benjamin Bray, Jia Deng and Chansoo Lee**
* Date: 9/21/2016

## Notation

- In this lecture, we will use
    - Let vector $\vec{x}_n \in \R^D$ denote the $n\text{th}$ data. $D$ denotes number of attributes in dataset.
    - Let vector $\phi(\vec{x}_n) \in \R^M$ denote features for data $\vec{x}_n$. $\phi_j(\vec{x}_n)$ denotes the $j\text{th}$ feature for data $x_n$.
    - Feature $\phi(\vec{x}_n)$ is the *artificial* features which represents the preprocessing step. $\phi(\vec{x}_n)$ is usually some combination of transformations of $\vec{x}_n$. For example, $\phi(\vec{x})$ could be vector constructed by $[\vec{x}_n^\top, \cos(\vec{x}_n)^\top, \exp(\vec{x}_n)^\top]^\top$. If we do nothing to $\vec{x}_n$, then $\phi(\vec{x}_n)=\vec{x}_n$.
    - Continuous-valued label vector $t \in \R^D$ (target values). $t_n \in \R$ denotes the target value for $i\text{th}$ data.

## Linear Regression

### Linear Regression (General Case)
- The function $y(\vec{x}_n, \vec{w})$ is linear in parameters $\vec{w}$.
    - **Goal:** Find the best value for the weights $\vec{w}$.
    - For simplicity, add a **bias term** $\phi_0(\vec{x}_n) = 1$.
$$
\begin{align}
y(\vec{x}_n, \vec{w})
&= w_0 \phi_0(\vec{x}_n)+w_1 \phi_1(\vec{x}_n)+ w_2 \phi_2(\vec{x}_n)+\dots +w_{M-1} \phi_{M-1}(\vec{x}_n) \\
&= \sum_{j=0}^{M-1} w_j \phi_j(\vec{x}_n) \\
&= \vec{w}^\top \phi(\vec{x}_n)
\end{align}
$$
of which $\phi(\vec{x}_n) = [\phi_0(\vec{x}_n),\phi_1(\vec{x}_n),\phi_2(\vec{x}_n), \dots, \phi_{M-1}(\vec{x}_n)]^\top$

#### Method I: Batch Gradient Descent
- To minimize the objective function, take derivative w.r.t coefficient vector $\vec{w}$ and descend: initialize $\vec{w}^0$ to be any vector, and at each step $s$,
$$
\vec{w}^{s+1} \gets \vec{w}^{s} - \nabla_{\vec{w}}E(\vec{w}^s)
$$

Exercise: Compute the partial derivative
$$
(\nabla_{\vec{w}}E)_j = \frac{\partial E}{\partial w_j}
$$
where
$$
E(\vec{w}) = \frac{1}{2} \sum_{n=1}^N \sum_{i=1}^{M} \left( w_i \phi_i(\vec{x}_n) - t_n \right)^2
$$

#### Solution
In the summation over $i$, only $i = j$ term is a function of $w_j$. So,
$$
\frac{\partial E}{\partial w_j}
= \frac{1}{2} \sum_{n=1}^N \frac{\partial}{\partial w_j} \left( w_j \phi_j(\vec{x}_n) - t_n \right)^2
= \sum_{n=1}^{N} (w_j \phi_j(\vec{x}_n) - t_n)
$$

*Tip*: If you find subscript notations confusing, just plug in $j = 1$, differentiate, and get $\sum_{n=1}^{N} (w_1 \phi_1(\vec{x}_n) - t_n)$.

### Linear Regression: Matrix Notations
The matrix $\Phi \in \R^{N \times M}$ is called **design matrix**. Each row represents one sample. Each column represents one feature
$$\Phi = \begin{bmatrix}
\phi(\vec{x}_1)^\top\\
\phi(\vec{x}_2)^\top\\
\vdots\\
\phi(\vec{x}_N)^\top
\end{bmatrix}
= \begin{bmatrix}
\phi_0(\vec{x}_1) & \phi_1(\vec{x}_1) & \cdots & \phi_{M-1}(\vec{x}_1) \\
\phi_0(\vec{x}_2) & \phi_1(\vec{x}_2) & \cdots & \phi_{M-1}(\vec{x}_2) \\
\vdots & \vdots & \ddots & \vdots \\
\phi_0(\vec{x}_N) & \phi_1(\vec{x}_N) & \cdots & \phi_{M-1}(\vec{x}_N) \\
\end{bmatrix}
$$

Target value vector is $\vec{t} \in \mathbb{R}^M$.

$$
E(\vec{w})
= \frac{1}{2} \sum_{n=1}^N (y(\vec{x}_n, \vec{w}) - t_n)^2
= \frac{1}{2} \sum_{n=1}^N \left( \sum_{j=0}^{M-1} w_j\phi_j(\vec{x}_n) - t_n \right)^2
= \frac{1}{2} \sum_{n=1}^N \left( \vec{w}^\top \phi(\vec{x}_n) - t_n \right)^2
$$

#### Batch Gradient Descent with Matrix Calculus
Write the objective function in matrix-vector form:
$
\begin{align*}
E(\vec{w}) &=  \frac{1}{2} \sum_{n=1}^N \sum_{i=1}^{M} \left( w_i \phi_i(\vec{x}_n) - t_n \right)^2 \\
&= \frac{1}{2} \sum_{n=1}^N \left(  \phi(\vec{x}_n)^\top \vec{w} - t_n \right)^2 = \frac{1}{2} \|\Phi \vec{w} - \vec{t}\|_2^2
\end{align*}
$

Rewrite $E$ as a sum of three matrix-vector products. Hints:
* $ \vec{x}^\top \vec{x} = (x_1,\ldots,x_M)^\top (x_1,\ldots,x_M) =  x_1^2 + \cdots + x_M^2 = \left(\sqrt{x_1^2 + \cdots + x_M^2}\right)^2 = \|\vec{x}\|_2^2$

* Distributive law: $(\vec{a} + \vec{b})^\top(\vec{c} + \vec{d}) = \vec{a}^\top \vec{c} + \vec{a}^\top \vec{d} + \vec{b}^\top \vec{c} + \vec{b}^\top\vec{d}$

* Transpose of a product: $(AB)^\top = B^\top A^\top$ for matrix-vector multiplication.

#### Batch Gradient Descent with Matrix Calculus
Treat $\Phi \vec{w}$ as a vector and we get from the distributive law
$$
E(\vec{w}) = \frac{1}{2} \|\Phi \vec{w} - \vec{t}\|_2^2 = \frac{1}{2} \left(\vec{w}^\top \Phi^\top \Phi \vec{w} - \vec{w}^\top \Phi^\top \vec{t} - \vec{t}^\top \Phi \vec{w} + \vec{t}^\top \vec{t}\right).
$$

Note that $\vec{w}^\top (\Phi^\top \vec{t}) = (\Phi^\top\vec{t})^\top \vec{w} = \vec{t}^\top \Phi \vec{w}$. So, the above simplifies to
$$ \frac{1}{2} \left(\vec{w}^\top \Phi^\top \Phi \vec{w} - 2\vec{w}^\top \Phi^\top \vec{t} + \vec{t}^\top \vec{t}\right).$$

#### Batch Gradient Descent with Matrix Calculus
Write the objective function in matrix-vector form:
$$
E(\vec{w}) = \frac{1}{2} \left(\vec{w}^\top \Phi^\top \Phi \vec{w} - 2\vec{w}^\top \Phi^\top \vec{t} + \vec{t}^\top \vec{t}\right).
$$

Compute the gradient $\nabla_\vec{w} E(\vec{w})$ with matrix calculus. Hints:
* $\nabla_\vec{x} (\vec{x}^\top A \vec{x}) = (A + A^\top) \vec{x}$   (Challenge: prove this!)
* $\nabla_\vec{x} (\vec{x}^\top \vec{y}) = \nabla_\vec{x} (\vec{y}^\top\vec{x}) = \vec{y}$

* $\Phi^\top \Phi$ has a special property.

#### Batch Gradient Descent with Matrix Calculus

* Since $\Phi^\top\Phi$ is symmetric, $\nabla_{\vec{w}} \vec{w}^\top(\Phi^\top\Phi)\vec{w} = 2(\Phi^\top\Phi)\vec{w}$.
* Treating $\Phi^\top t$ as a vector, $\nabla_{\vec{w}} \vec{w}^\top(\Phi^\top t) = \Phi^\top t$.
* Finally, $t^\top t$ is constant with respect to $\vec{w}$.

So,
$\nabla_{\vec{w}} E(\vec{w}) = (\Phi^\top\Phi)\vec{w} - \Phi^\top t$

## Method I-2: Gradient Descent—Stochastic Gradient Descent
**Main Idea:**  Instead of computing batch gradient (over entire training data), just compute gradient for individual (or a small subset of) training sample and update.

 Exercise : How do you implement the update rule for a minibatch gradient descent (of size, let's say, 5% of the whole dataset)?

You randomly choose 5% of indices between 0 and $M$. Take the corresponding rows of $\Phi$ and $t$. Compute the gradient on this subset of data and desend along.

## Method II: Closed-Form solution, invertible case

**Main Idea**, also **Exercise:**  Solve $\nabla_\vec{w} E(\vec{w}) = 0$, assuming $\Phi^\top\Phi$ is invertible. Discuss why it is sufficent to solve this equation to find optimal $\vec{w}$.

*Answer*: It is sufficent to find a local minimum because $E(\vec{w})$ is convex. The solution is $\vec{w} = (\Phi^\top\Phi)^{-1}\Phi^\top t$.

 Exercise : Show that $\Phi^\top \Phi$ is invertible if $\Phi$ has *linearly independent columns*. Interpret its implications about our features.

*Answer*: It implies that our features are linearly dependent

 Challenge:  Similarly, we can show $\Phi\Phi^\top$ is invertible if $\Phi$ has linearly independent rows. Why do we care/not care about this case?

 Challenge:  Show that $\vec{b}$ is in the column space of $A$ if and only if there exists a vector $\vec{x}$ such that $A\vec{x} = \vec{b}$.

#### Digression: Moore-Penrose Pseudoinverse
- When we have a matrix $A$ that is non-invertible or *not even square*, we might want to invert anyway
- For these situations we use $A^\dagger$, the **Moore-Penrose Pseudoinverse** of $A$
- In general, we can get $A^\dagger$ by SVD: if we write $A \in \R^{m \times n} = U_{m \times m} \Sigma_{m \times n} V_{n \times n}^\top$ then $A^\dagger \in \R^{n \times m} = V \Sigma^\dagger U^\top$, where $\Sigma^\dagger \in \R^{n \times m}$ is obtained by taking reciprocals of *non-zero entries* of $\Sigma^\top$.
- Particularly, when $A$ has linearly independent columns then $A^\dagger = (A^\top A)^{-1} A^\top$. When $A$ is invertible, then $A^\dagger = A^{-1}$.

** Exercise **: One property of Psuedoinverse is that $A A^\dagger A = A$.
Show that $$(A^{\top} A)^{-1}A^\top$$ satisfies this property (assuming linearly independent columns of $A$)

*Challenge: * Show that $$\hat{\vec{w}} = (\Phi^\top\Phi)^\dagger \Phi^\top \vec{t} = \Phi^\dagger \vec{t}$$
satisfies $\nabla_\vec{w} E(\vec{w}) = \Phi^\top\Phi \vec{w} - \Phi^\top \vec{t} = 0$.

** Discuss **: What are the advantages and disadvtanges of each method we learned today (stochastic gradient descent, batch gradient descent, and closed-form solution)?

*Answer*: There are no right answers, but you can say things like: matrix inversion is a cubic-time operation (technically $O(n^{2.37...})$). Performing better on your training data $\Phi$ doesn't necessarily mean performing better on unseen test data.


## The network SEI8R model

The dynamical equations are given as

\begin{align}
\dot{S_{in}}	    &=-\lambda_{in}(t)S_{in}+\sigma_{in},\\
\dot{E}_{in}	    &=\lambda_{i}(t)S_{in}-\gamma_{E}E_{in},\\
\dot{I}_{in}^{a}	&=\alpha\gamma_{E}E_{in}-\gamma_{I^{a}}I_{in}^{a},\\
\dot{I}_{in}^{s}	&=\bar{\alpha}\gamma_{E}E_{in}-\gamma_{I^{s}}I_{in}^{s},\\
\dot{I}_{in}^{s'}	&=\bar{h_{i}}\gamma_{I^{s}}I_{in}^{s}-\gamma_{I^{s'}}I_{in}^{s'},\\
\dot{I}_{in}^{h}	&=h_{i}\gamma_{I^{s}}I_{in}^{s}-\gamma_{I^{h}}I_{in}^{h},\\
\dot{I}_{in}^{h'}	&=\bar{c_{i}}\gamma_{I^{h}}I_{in}^{h}-\gamma_{I^{s'}}I_{in}^{h'},\\
\dot{I}_{in}^{c}	&=c_{i}\gamma_{I^{h}}I_{in}^{h}-\gamma_{I^{c}}I_{in}^{c},\\
\dot{I}_{in}^{c'}	&=\bar{m_{i}}\gamma_{I^{c}}I_{in}^{c}-\gamma_{I^{c'}}I_{in}^{c'},\\
\dot{I}_{in}^{m}	&=m_{i}\gamma_{I^{c}}I_{in}^{c},\\
\dot{N}^{H}_{in}	    &=\sigma_{in}-m_{in}\gamma_{I^{c}}I_{in}^{c}
\end{align}

The key idea is to note that the rate of infection of a susceptible individual in age group `i` at the node `n` is given as

\begin{align}
\lambda_{in}(t)=\lambda^0_{in}(t)=\beta\sum_{j=1}^{M}\left(C_{ij}\frac{I_{jn}^{a}}{N^{H}_{jn}}+f_{sa}C_{ij}\frac{I_{jn}^{s}}{N^{H}_{jn}} + f_{h} C_{ij}\frac{I_{jn}^{h}}{N^{H}_{jn}}\right),
\end{align}

Commute can then be modeled by changing this to

\begin{align}
\lambda_{in}^{T}(t)=\beta r^{T}\sum_{j=1}^{M}\sum_{m=1}^{N_{d}} \sum_{<k,l>} C_{ij} \frac{D_{j,kl}}{N^{H}_{j,l}} \frac{I_{jl}^{a} + f_{sa} I_{jl}^{s} + f_{h} I_{jl}^{h}}{N^{T}_{j,mn}} \frac{D_{i,mn}}{N^{H}_{i,n}},
\end{align}

while workplace interaction can be modeled as

\begin{align}
\lambda_{in}^{W}(t)=\beta r^{W}\sum_{j=1}^{M}\sum_{m,k=1}^{N_{d}} C_{ij} \frac{D_{j,mk}}{N^{H}_{jk}} \frac{I_{jk}^{a} + f_{sa} I_{jk}^{s} + f_{h} I_{jk}^{h}}{N^{W}_{jm}}  \frac{D_{i,mn}}{N^{H}_{in}}.
\end{align}

The derivation of all equations is wrriten in [the documentation](https://github.com/lukastk/PyRossGeo/tree/master/docs/mft-model.pdf).
The simulation then runs by choosing appropriate $\lambda$ as a function of time. The constants $r_T$ and $r_W$ control the strength of interactions. In this model, we assume that each route between the nodes connects the neighbour nodes only.

The defiintion of the variables are in following.
$D_{i,nm}$ is the number of people working at node `n` and livingin `m` in the age group `i`, $N^{H}_{in}=\sum_{m} D_{i,mn}$ is the total number of residence in node `n` and $N^{W}_{in}=\sum_{m} D_{i,nm}$ is the total number of people working at node `n`.
$N_{i,nm}^{T}$ is the effective number of the commuter in the age group `i` using the route between `n` and `m`:

\begin{align}
N^{T}_{i,nm}=\sum_{<kl>}R_{i,kl}\frac{d_{kl}}{d_{nm}},
\end{align}

where $<kl>$ indicate summation of all pair of neighbour node $k$ and $l$ included in the shortest path between node
`n` and `m` and $d_{nm}$ is the distance between node `n` and `m` along the shortest path. $R_{i,nm}=\sum_{<kl>}D_{i,kl}$ is the number of the commuter in the age group `i` using the route between the neighbor node `n` and `m`. The node `n` and `m` are included in the shortest path between node pair
$(k,l)$ took up all combinations satisfying the conditions.
Hence, we only need $D_{i,nm}$ and $d_{nm}$ in order to construct this network model. $D_{i,nm}$ can be obtained census datasets and $d_{nm}$ can be determined by the geographical locations. For the calculation of London including 983 nodes, $D_{i,nm}$ and $d_{nm}$ are stored in census.txt and distnaces.txt, respectively.

```python
%%capture
## compile for this notebook
import os
owd = os.getcwd()
os.chdir('../')
%run setup.py install
os.chdir(owd)
```

Loading $D_{i,nm}$ and $d_{nm}$ from census.txt and distances.txt

```python
import numpy as np
import pyrossgeo
import matplotlib.pyplot as plt
import math
import os

M = 7 #Aged Group

cutoff = 0
Restricted = [] #the restirected area
if os.path.exists('census.txt') and os.path.exists('distances.txt'):
    f_census = open('census.txt')
    f_distances = open('distances.txt')
    c_lines = f_census.readlines()
    d_lines = f_distances.readlines()
    if len(c_lines) != len(d_lines)*M:
        print('Error. Wrong file Status\n')
        exit()

    Nnode=int(len(c_lines)/M)
    census_matrix = np.zeros((M + 1,Nnode, Nnode))
    distances=np.zeros((Nnode, Nnode))

    count = 0
    aged_id = 0
    for line in c_lines:
        parameters = line.split()
        for i in range(len(parameters)):
            census_matrix[aged_id,count,i] = float(parameters[i])
        count += 1
        if count == Nnode:
            count = 0
            aged_id += 1

    #Restricted Area
    for alp in range(M):
        for i in Restricted:
            for j in range(Nnode):
                cij = census_matrix[alp,i,j]
                census_matrix[alp,i,j] = 0.0
                census_matrix[alp,j,j] += cij

                cij = census_matrix[alp,j,i]
                census_matrix[alp,j,i] = 0.0
                census_matrix[alp,i,i] += cij

    #Cutoff
    nonzero_element = 0
    cutoff_total = 0
    c_matrix = census_matrix.copy()
    for alp in range(M):
        for i in range(Nnode):
            for j in range(Nnode):
                cij = c_matrix[alp,i,j]
                if cij > cutoff:
                    nonzero_element += 1
                    cutoff_total += c_matrix[alp,i,j]
                else:
                    census_matrix[alp,i,j] = 0.0
                    #max_wid = np.argsort(census_matrix[:,j])[Nnode- 1]
                    census_matrix[alp,j,j] += cij
    print("The number of nonzero element " + str(nonzero_element) + '/' + str(M*Nnode**2) + ' ' + str(cutoff_total))

    #Make Total
    for i in range(Nnode):
        for j in range(Nnode):
            for alp in range(M):
                census_matrix[M,i,j] += census_matrix[alp,i,j]

    Nh = np.zeros((M + 1,Nnode)) # The population of ususal residence
    Nw = np.zeros((M + 1,Nnode)) # The number of people working in specific node.
    for alp in range(M):
        for i in range(Nnode):
            Nh[alp,i] = 0.
            Nw[alp,i] = 0.
            for j in range(Nnode):
                Nh[alp,i] += census_matrix[alp,j,i]
                Nw[alp,i] += census_matrix[alp,i,j]
    for alp in range(M):
        for i in range(Nnode):
            Nh[M,i] += Nh[alp,i] ## N^{H}_{i} residence in Node i
            Nw[M,i] += Nw[alp,i] ## N^{W}_{i} working in Node i

    count = 0
    for line in d_lines:
        parameters = line.split()
        for i in range(len(parameters)):
            distances[count,i] = float(parameters[i])
        count += 1

# set the matarix D_{nm}
Dnm = census_matrix.copy()

# set the matarix d_{nm}
dnm = distances.copy()

max_wid = np.argsort(Nw[M])[Nnode- 1]
print("Maxmimum City at Working", max_wid, Nw[M,max_wid])
max_rid = np.argsort(Nh[M])[Nnode- 1]
print("Maxmimum City of Residence", max_rid, Nh[M,max_rid])
```

    The number of nonzero element 749693/6764023 4760944.0
    Maxmimum City at Working 0 245633.0
    Maxmimum City of Residence 859 9752.0

To start network model calculation, we must set $C_{ij}$, model parameters and the initial state. $C_{ij}$ and model parameters can be inferred if we obtain the reasonable initial state. In this tutorial, we set the preliminary values.

```python
# set the contact structure
C = np.identity(M)
for i in range(M - 1):
    C[i + 1, i] = 1.0
    C[i, i + 1] = 1.0
C *= 1.0
#print(C)
CH = C
CT = C
CW = C

# parameters
alpha = 0.5             # fraction of asymptomatic infectives
beta  = 0.2/24          # infection rate
gE    = 0.6/24          # recovery rate of exposed
gIa   = 0.1/24          # recovery rate of asymptomatic infectives
gIs   = 0.1/24          # recovery rate of symptomatic infectives
gIsd  = 0.2/24          # recovery rate of symptomatic infectives dash
gIh   = 0.1/24          # recovery rate of hospitalized infectives
gIhd  = 0.2/24          # recovery rate of hospitalized infectives dash
gIc   = 0.1/24          # recovery rate of ICU infectives
gIcd  = 0.2/24          # recovery rate of ICU infectives dash
fsa   = 1.0             # the self-isolation parameter
fh    = 0.0             # the hospitalized-isolation parameter
rW    = 1.0             # the interaction controle in WORK (Maybe 1 is a correct value)
rT    = 1.0             # the interaction controle in TRANS (Maybe 1 is a correct value)
hh    = 0.4             # fraction of hospitalized infectives
cc    = 0.4             # fraction of ICU infectives
mm    = 0.4             # fraction of Im infectives

# option
cutoff    = 4.0        # We ignore the number of people moving to other node less than this number.
trv_restriction = 0.0   # travel restriction: 1.0 means everyone stay home

max_wid = np.argsort(Nw[M])[Nnode- 1]
#print("Maxmimum City at Working", max_wid)
max_rid = np.argsort(Nh[M])[Nnode- 1]
#print("Maxmimum City of Residence", max_rid)
min_wid = np.argsort(Nw[M])[0]
#print("Minimum City at Working", min_wid)
min_rid = np.argsort(Nh[M])[0]
#print("Minimum City of Residence", min_rid)

# Initial state
M1 = Nnode*M
E_0   = np.zeros((M1))
Ia_0  = np.zeros((M1))
Is_0  = np.zeros((M1))
Ih_0  = np.zeros((M1))
Ic_0  = np.zeros((M1))
Isd_0 = np.zeros((M1))
Ihd_0 = np.zeros((M1))
Icd_0 = np.zeros((M1))
Im_0  = np.zeros((M1))

Is_0[1*Nnode + max_rid] += 10

R_0 = np.zeros((M1))
N_0 = np.zeros((M1))
for alp in range(M):
    for i in range(Nnode):
        N_0[Nnode*alp + i] = Nh[alp,i]

S_0  = N_0 - (Ia_0 + Is_0 + Ih_0 + Ic_0 + Isd_0 + Ihd_0 + Icd_0 + Im_0 + R_0)

# the contact matrix
def contactMatrix(t):
    t_p_24 = t%24
    if t_p_24 < 8.0 or t_p_24 > 18.0:
        return CH
    elif t_p_24 > 9.0 and t_p_24 < 17.0:
        return CW
    else:
        return CT

# set the matarix D_{nm}
Dnm = census_matrix.copy()

# set the matarix d_{nm}
dnm = distances.copy()

# instantiate model
parameters = {'alpha':alpha, 'beta':beta, 'gE':gE, 'gIa':gIa, 'gIs':gIs, 'gIsd':gIsd, 'gIh':gIh, 'gIhd':gIhd, 'gIc':gIc, 'gIcd':gIcd, 'fsa':fsa, 'fh':fh, 'hh':hh, 'cc':cc, 'mm':mm, 'rW':rW, 'rT':rT}
model = pyrossgeo.mft.SEI8R(parameters, M, Nnode, Dnm, dnm, trv_restriction, cutoff)
```

    #Start finding the shortest path between each node
    #Start to calculate fixed variables
    Nonzero element 129731/47348161 3757347.0
    Max route number 0 41
    Total index in all route 975761 1.009802450405624
    #Finish calculating fixed variables

After setting the parameters, the calculation of network model will be started. **The modern laptop can finish this calculation about 1 minute.** We can improve this computational rate.

```python
rT=1.0
# intialize the model
parameters = {'alpha':alpha, 'beta':beta, 'gE':gE, 'gIa':gIa, 'gIs':gIs, 'gIsd':gIsd, 'gIh':gIh, 'gIhd':gIhd, 'gIc':gIc, 'gIcd':gIcd, 'fsa':fsa, 'fh':fh, 'hh':hh, 'cc':cc, 'mm':mm, 'rW':rW, 'rT':rT}
model.initialize(parameters, trv_restriction)

# duration of simulation
Tf=24*100; Nf=Tf # Maximum time step equals as Tf/Nf
# simulate model
data=model.simulate(S_0, E_0, Ia_0, Is_0, Isd_0, Ih_0, Ihd_0, Ic_0, Icd_0, Im_0, N_0, contactMatrix, Tf, Nf, Ti=0, highSpeed=0)
```

    #Start to calculate fixed variables
    Nonzero element 129731/47348161 3757347.0
    #Finish calculating fixed variables
    travel restriction 0.0
    cutoff 4.0
    highspeed 0
    #Calculation Start
    Time 100.06157837283018
    Time 200.06157837283018
    Time 300.0615783728302
    Time 400.0615783728302
    Time 500.0615783728302
    Time 600.0615783728301
    Time 700.0615783728301
    Time 800.0615783728301
    Time 900.0615783728301
    Time 1000.0615783728301
    Time 1100.0615783728301
    Time 1200.0615783728301
    Time 1300.0615783728301
    Time 1400.0615783728301
    Time 1500.0615783728301
    Time 1600.0615783728301
    Time 1700.0615783728301
    RM reset 1717.0615783728301
    Time 1800.0615783728301
    Time 1900.0615783728301
    Time 2000.0615783728301
    Time 2100.06157837283
    Time 2200.06157837283
    Time 2300.06157837283
    Time 2400.0

Loading the simulated data

```python
S  = []
E  = []
Ia = []
Is = []
Isd= []
Ih = []
Ihd= []
Ic = []
Icd= []
Im = []
N  = []

for i in range(Nnode*M):
    S.append(data.get('X')[i].flatten())
    E.append(data.get('X')[i + M1].flatten())
    Ia.append(data.get('X')[i + 2*M1].flatten())
    Is.append(data.get('X')[i + 3*M1].flatten())
    Isd.append(data.get('X')[i + 4*M1].flatten())
    Ih.append(data.get('X')[i + 5*M1].flatten())
    Ihd.append(data.get('X')[i + 6*M1].flatten())
    Ic.append(data.get('X')[i + 7*M1].flatten())
    Icd.append(data.get('X')[i + 8*M1].flatten())
    Im.append(data.get('X')[i + 9*M1].flatten())
    N.append(data.get('X')[i + 10*M1].flatten())
t = data['t']

# In 1 dimension, the data index in age group i and node j can be expressed as i*N_{d} + j.
#| Age group 0 | Age group 1 | Age group 2 | Age group 3 | Age group 4 | Age group 5 | Age group 6 |
#|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|
```

Plotting the results

```python
fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

total_w = 0.0
total_r = 0.0
for i in range(Nnode):
    total_w += Nw[M][i]
    total_r += Nh[M][i]
#print('Toatl', total_w, total_r)

S_total = []
E_total = []
I_total = []
Im_total = []
PR_total = []
N_total = []
for i in range(len(Ia[0])):
    S_total.append(0.0)
    E_total.append(0.0)
    I_total.append(0.0)
    Im_total.append(0.0)
    PR_total.append(0.0)
    N_total.append(0.0)
for i in range(M1):
    S_total  += S[i]
    E_total  += E[i]
    I_total  += Ia[i] + Is[i] + Ih[i] + Ic[i] + Im[i]
    PR_total += Isd[i] + Ihd[i] + Icd[i]
    Im_total += Im[i]
    N_total  += N[i] + Im[i]
plt.plot(t/24, S_total, '-', label='$S$', lw=4)
plt.plot(t/24, E_total, '-', label='$E$', lw=4)
plt.plot(t/24, I_total, '-', label='$I$', lw=4)
plt.plot(t/24, PR_total, '-', label='$I\'$', lw=4)
plt.plot(t/24, Im_total, '-', label='$Im$', lw=4)
plt.plot(t/24, N_total, '-', label='$N+I_{m}$', lw=4)
plt.plot(t/24, total_r - S_total - I_total, '-', label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
#plt.autoscale(enable=True, axis='x', tight=True)

ax=plt.subplot()
#ax.grid(which="both")
#ax.set_xlabel("x")
#ax.set_ylabel("y")

ax.set_xlim([0,100])
#ax.set_ylim([0,2.0*10**1])

f_data = open('network-SEI8R.txt', mode='w')
for i in range(len(t)):
    f_data.write(str(t[i]/24) + ' ' + str(S_total[i]) + ' ' + str(I_total[i]) + '\n')
f_data.close()
```

```python

```


# Test of Selection mask

Assuming an additive term due to imaging systematics

$$
n_{g} = \bar{n} (1+\delta) + G(s) \\
n_{\rm pred} = \bar{n} + \hat{G}(s)
$$

I. Similar to Delubac et al.  (https://arxiv.org/pdf/1611.06934.pdf)

\begin{align}
w_{\rm sys} &= 1 + \frac{\bar{n} - n_{\rm pred}}{n_{g}}  \\
n_{g}*w_{\rm sys} &= \bar{n} + n_{g} - n_{\rm pred} \\
&= \bar{n}(1+\delta) + [G(s)-\hat{G}(s)] \\
&= \bar{n}(1+\delta)
\end{align}

II. Similar to Bautista et al. 2018
\begin{align}
w_{\rm sys} &= \frac{\bar{n}}{n_{\rm pred}} \\
n_{g}*w_{\rm sys} &= \frac{\bar{n} (1+\delta) + G(s)}{1 + \hat{G}(s)/\bar{n}}
\end{align}
assuming |$\hat{G}(s)/\bar{n}$| is < 1, we obtain
\begin{align}
n_{g}*w_{\rm sys} &= (\bar{n} (1+\delta) + G(s))~[1 - \hat{G}(s)/\bar{n} + (\hat{G}(s)/\bar{n})^{2}+...] \\
&= \bar{n}(1+\delta) - \hat{G}(s)(1+\delta) + G(s) + O(\hat{G}^{2})\\
&= \bar{n}(1+\delta) + [G(s) - \hat{G}(s) - \hat{G}(s)\delta] + O(\hat{G}^{2})
\end{align}
if we ignore the $O(\hat{G}^{2})$ terms, we obtain
$$
n_{g}*w_{\rm sys} = \bar{n}(1+\delta)
$$

```python
''' import the required modules
'''
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import healpy as hp
import fitsio as ft

import sys
# sys.path.insert(0, '/Users/rezaie/github/SYSNet/src')
sys.path.insert(0, '/Users/rezaie/github/LSSutils')

from scipy.stats import binned_statistic
from LSSutils.lab import AnaFast, MeanDensity, overdensity
from LSSutils.utils import histogram
from LSSutils.stats.cl import get_shotnoise

''' read the observed galaxy count, random count (aka fracgood),
    predicted galaxy count (from NN)
    boolean mask for the pixels on the footprint,
    and a fits table for imaging attributes.
    we only take the EBV map
'''
path = '/Volumes/TimeMachine/data/DR7/'
sysmaps = ft.read(f'{path}eBOSS.ELG.NGC.DR7.table.fits')
nran = hp.read_map(f'{path}frac.hp.256.fits', verbose=False)
ngal = hp.read_map(f'{path}eBOSS.ELG.NGC.DR7.cut.hp256.fits', verbose=False)
wnn = hp.read_map(f'{path}results/regression/nn_p/nn-weights.hp256.fits', verbose=False)
mask = hp.read_map(f'{path}mask.cut.hp.256.fits', verbose=False) > 0

ebv = np.zeros_like(ngal) # EBV map
ebv[sysmaps['hpind']] = sysmaps['features'][:, 0]

# original mask was with frac > 0.2
# t = ngal / (nran x ngal_tot / nran_tot)
t = overdensity(ngal, nran, nran>0.2, nnbar=True)
wnn2 = np.ones_like(ngal)
m = t > 0
wnn2[m] = 1 + (1-wnn[m])/t[m]

wnn1 = 1./wnn
```

    /Users/rezaie/anaconda3/envs/py3p6/lib/python3.6/site-packages/ipykernel_launcher.py:38: RuntimeWarning: invalid value encountered in greater
    /Users/rezaie/anaconda3/envs/py3p6/lib/python3.6/site-packages/ipykernel_launcher.py:41: RuntimeWarning: divide by zero encountered in true_divide

histograms of the weights

```python
mask = nran > 0.2
for wnni in [wnn1[mask], wnn2[mask]]:
    print(np.percentile(wnni, [0, 1, 99, 100]))
```

    [-1.42854275e+03  8.55955625e-01  1.55310544e+00  2.17629945e+04]
    [-1.18168799  0.80400009  1.4564536  24.97700296]

```python
plt.hist(wnn2[mask], bins=60, alpha=0.5, label='additive')
plt.hist(wnn1[mask], bins=60, alpha=0.5, label='multiplicative')

plt.legend()
plt.yscale('log')
plt.xlabel(r'$w_{\rm sys}$', fontsize=20)
```

```python
# initially, mask = nran > 0.2, in another block
mask1 = mask & (wnn1 > 0.5) & (wnn1 < 2.0)
mask2 = mask & (wnn2 > 0.5) & (wnn2 < 2.0)
```

```python
mask1.sum(), mask2.sum()
```

    (185914, 186495)

```python
fpdf = lambda x, m, s: (1./np.sqrt(2*np.pi*s*s)) * np.exp(-1*((x-m)/(s))**2)

xg = np.linspace(0.5, 1.6)

_, b, _ = plt.hist(wnn2[mask2], bins=100, alpha=0.5, label='additive', align='mid')
plt.hist(wnn1[mask1], bins=b, alpha=0.5, label='multiplicative', align='mid')
plt.plot(xg, 2.5e3*fpdf(xg, 1, np.std(wnn1[mask1])), 'k-')
# plt.plot(xg, 3.8e3*fpdf(xg, 0.98, np.std(wnn1[mask])), 'k--')
plt.legend()
# plt.yscale('log')
# plt.ylim(1., 1.2e4)
plt.xlabel(r'$w_{\rm sys}$', fontsize=20)
```

Apply the weights on the galaxy count map, compute density contrast $\delta$, $C_{\ell}$

```python
ngal1 = ngal * wnn1 # multiplicative
ngal2 = ngal * wnn2 # additive

# compute the density contrast
delta01 = overdensity(ngal, nran, mask1)
delta02 = overdensity(ngal, nran, mask2)

delta1 = overdensity(ngal1, nran, mask1)
delta2 = overdensity(ngal2, nran, mask2)

# compute mean density vs ebv
nbar = MeanDensity(ngal, nran, mask, ebv)
nbar.run()
nbar1 = MeanDensity(ngal1, nran, mask1, ebv)
nbar1.run()
nbar2 = MeanDensity(ngal2, nran, mask2, ebv)
nbar2.run()

# compute angular C_ell
af = AnaFast()
cl01 = af(delta01, nran, mask1)
cl02 = af(delta02, nran, mask2)
cl1 = af(delta1, nran, mask1)
cl2 = af(delta2, nran, mask2)
```

    /Users/rezaie/anaconda3/envs/py3p6/lib/python3.6/site-packages/ipykernel_launcher.py:1: RuntimeWarning: invalid value encountered in multiply
      """Entry point for launching an IPython kernel.

```python
shotnoise = []
for ngal_i, mask_i in zip([ngal, ngal, ngal1, ngal2],
                          [mask1, mask2, mask1, mask2]):
    shotnoise.append(get_shotnoise(ngal_i, nran, mask_i))
```

```python
shotnoise
```

    [8.539519975252201e-07,
     8.55658387227114e-07,
     8.619420678921783e-07,
     8.642252887803628e-07]

left: Mean density vs EBV, right: angular power spectrum

```python
fig, ax = plt.subplots(ncols=2, figsize=(12, 4))

mk=['o', 's', 'x']
for i, nbar_i in enumerate([nbar, nbar1, nbar2]):
    ax[0].plot(nbar_i.output['bin_avg'],
               nbar_i.output['nnbar'],
               marker=mk[i])

ax[0].set(xlabel='EBV', ylabel='N/Nbar', xscale='log')

names = ['NWM', 'NWA', 'Multip.', 'Addit.']
for i, cli in enumerate([cl01, cl02, cl1, cl2]):
    ax[1].loglog(cli['l'], cli['cl'], label=names[i])
ax[1].set_ylim(1.0e-7, 1.0e-2)
ax[1].legend()
ax[1].set(ylabel=r'$C_{\ell}$', xlabel=r'$\ell$')
```

ratio of the Cls

```python
plt.rc('font', size=15)
```

```python
fig, ax = plt.subplots(figsize=(10, 4))

bins = np.linspace(1, 770, num=20)

elb0, clb01,_ = histogram(cl01['l'], cl01['cl']-shotnoise[0], bins=bins)
elb0, clb02,_ = histogram(cl02['l'], cl02['cl']-shotnoise[1], bins=bins)
elb1, clb1,_ = histogram(cl1['l'], cl1['cl']-shotnoise[2], bins=bins)
elb2, clb2,_ = histogram(cl2['l'], cl2['cl']-shotnoise[3], bins=bins)

assert np.array_equal(elb0, elb1)

ax.plot(elb0, clb1/clb01, marker='o', label='Mult.', mfc='w')
ax.plot(elb0, clb2/clb02, marker='s', label='Add.')

ax.legend(title='shotnoise subtracted')
ax.grid(True, ls=':', alpha=0.5)
ax.set(ylim=(0.85, 1.1), xlabel=r'$\ell$',
       ylabel=r'C$_{\ell}^{\rm X}$/C$_{\ell}^{\rm before}$')
```

```python
np.mean(wnn1[mask1]), np.mean(wnn1[mask]), np.mean(wnn2[mask2]), np.mean(wnn2[mask])
```

    (1.0025714485961408, 1.1429712921570436, 1.0002123152337312, 1.010540288417593)

```python

```


```python
import numpy as np
import sympy as sym
import pydae.build as db
```

```python
params_dict = {'L':5.21,'G':9.81,'M':10.0,'K_d':1e-6}

u_ini_dict = {'theta':np.deg2rad(5.0)}  # for the initialization problem
u_run_dict = {'f_x':10}  # for the running problem (here initialization and running problem are the same)

#u_ini_dict = {'theta':10}  # for the initialization problem
u_run_dict = {'f_x':10}  # for the running problem (here initialization and running problem are the same)

x_list = ['pos_x','pos_y','v_x','v_y']    # [inductor current, PI integrator]
y_ini_list = ['lam','f_x'] # for the initialization problem
y_run_list = ['lam','theta'] # for the running problem (here initialization and running problem are the same)

sys_vars = {'params':params_dict,
            'u_list':u_run_dict,
            'x_list':x_list,
            'y_list':y_run_list}

exec(db.sym_gen_str())  # exec to generate the required symbolic varables and constants
```

```python
dpos_x = v_x
dpos_y = v_y
dv_x = (-2*pos_x*lam + f_x - K_d*v_x)/M
dv_y = (-M*G - 2*pos_y*lam - K_d*v_y)/M

g_1 = pos_x**2 + pos_y**2 - L**2 -lam*1e-6
g_2 = -theta + sym.atan2(pos_x,-pos_y)
```

```python
sys = {'name':'pendulum',
       'params_dict':params_dict,
       'f_list':[dpos_x,dpos_y,dv_x,dv_y],
       'g_list':[g_1,g_2],
       'x_list':x_list,
       'y_ini_list':y_ini_list,
       'y_run_list':y_run_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_ini_dict,
       'h_dict':{'g_1':g_1,'PE':M*G*pos_y,'KE':0.5*M*(v_x**2+v_y**2),'theta':theta}}

sys = db.system(sys)
db.sys2num(sys)
```

```python
g = -I + Y @ V
g_p = -p + i*v
g_q = -q + i*v
y = V

G_y = jacobiano(g,y)
```

```python

```


# Symbolic Tensor (Quaternion) Rotation

## Author: Ken Sible

## The following module demonstrates symbolic vector or tensor rotation using SymPy.

### NRPy+ Source Code for this module:
1. [tensor_rotation.py](../edit/tensor_rotation.py); [\[**tutorial**\]](Tutorial-Symbolic_Tensor_Rotation.ipynb) The tensor_rotation.py script will perform symbolic tensor rotation using the following function: rotate(tensor, axis, angle).

# Table of Contents
$$\label{toc}$$

0. [Step 0](#prelim): Derivation of quaternion rotation from matrix rotation using [linear algebra](https://en.wikipedia.org/wiki/Linear_algebra) and [ring theory](https://en.wikipedia.org/wiki/Ring_theory)
1. [Step 1](#algorithm): Discussion of the tensor rotation algorithm using the [SymPy](https://www.sympy.org) package for symbolic manipulation </header_section>
1. [Step 2](#validation): Validation and demonstration of the tensor rotation algorithm, including common subexpression elimination
    1. [Step 2.a](#axisrotation): Verification of three-dimensional rotation about a coordinate axis using an equivalent rotation matrix
    1. [Step 2.b](#symtensor): Verification of symbolic tensor rotation and the invariance of argument permutation for a [symmetric tensor](https://en.wikipedia.org/wiki/Symmetric_tensor)
    1. [Step 2.c](#cse): Demonstration of [common subexpression elimination](https://en.wikipedia.org/wiki/Common_subexpression_elimination) applied to a rotated symbolic matrix
1. [Step 3](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Mathematical Background (Optional): Quaternion Rotation \[Back to [top](#toc)\]
$$\label{prelim}$$

Let $\vec{v}$ denote a vector in $\mathbb{R}^2$. We recall from linear algebra that $\vec{v}$ rotated through an angle $\theta$ about the x-axis, denoted $\vec{v}'$, has the following matrix formula

$$\vec{v}'=\begin{bmatrix}\cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}\vec{v}.$$

Let $z=a+bi\in\mathbb{C}$ for some $a,b\in\mathbb{R}$. Consider the corresponding (or [isomorphic](https://en.wikipedia.org/wiki/Isomorphism)) vector $\vec{z}=(a,b)\in\mathbb{R}^2$. We observe from the rotation formula that $\vec{v}'=a(\cos\theta,\sin\theta)+b(-\sin\theta,\cos\theta)$ after expanding the matrix product. Let $w=c+di\in\mathbb{C}$ for some $c,d\in\mathbb{R}$. We recall the definition of the complex product as $zw=(ac-bd)+i(ad+bc)$ where $i^2=-1$. Hence, $z'=(a+bi)(\cos\theta+i\sin\theta)=(a+bi)e^{i\theta}$ after comparing the rotated vector $\vec{v}'$ with the complex product as defined above. Therefore, the following compact formula arises for vector rotation (given the correspondence between $\mathbb{C}$ and $\mathbb{R}^2$):

$$z'=e^{i\theta}z.$$

However, for vector rotation in three-dimensional space, we curiously require a four-dimensional, non-commutative extension of the complex number system. The following discussion will provide an overview of the connection with vector rotation. For further reading, we suggest the document ([quaternion_rotation](http://graphics.stanford.edu/courses/cs348a-17-winter/Papers/quaternion.pdf)).

$$\mathcal{H}=\{a+bi+cj+dk:a,b,c,d\in\mathbb{R}\text{ and }i^2=j^2=k^2=ijk=-1\}$$

Consider the special case of rotating a vector $\vec{v}\in\mathbb{R}^3$ through an angle $\theta$ about a normalized rotation axis $\vec{n}$ perpendicular to the vector $\vec{v}$. We typically decompose a quaternion into a scalar and vector component whenever performing vector rotation, such as the decomposition of $q=a+bi+cj+dk$ into $q=(a,\vec{w})$ where $\vec{w}=(b,c,d)$. For future convenience, we define the following quaternions for vector rotation: $v=(0,\vec{v})$, $v'=(0,\vec{v}')$, and $n=(0,\vec{n})$.

From the fundemental formula of quaternion algebra, $i^2=j^2=k^2=ijk=-1$, we could derive quaternion multiplication, known as the [Hamilton product](https://en.wikipedia.org/wiki/Quaternion#Hamilton_product). Let $q_1,q_2\in\mathcal{H}$ where both $q_1$ and $q_2$ have zero scalar component. The Hamilton product of $q_1$ and $q_2$, after some straightforward verification, has the form $q_1q_2=(-\vec{q}_1\cdot\vec{q}_2,\vec{q}_1\times\vec{q}_2)$. Hence, $nv=(0,\vec{n}\times\vec{v})$ since $\vec{n}$ and $\vec{v}$ are orthogonal. We observe that the projection of the rotated vector $\vec{v}'$ onto $\vec{v}$ is $\cos\theta\,\vec{v}$ and the projection of $\vec{v}'$ onto $\vec{n}\times\vec{v}$ is $\sin\theta\,(\vec{n}\times\vec{v})$, and hence $\vec{v}'=\cos\theta\,\vec{v}+\sin\theta\,(\vec{n}\times\vec{v})$. We define the quaternion exponential as $e^{n\theta}=\cos\theta+n\sin\theta$, analogous to the complex exponential. Finally, we identify the formula for vector rotation in $\mathbb{R}^3$ by comparing the rotated vector $\vec{v}'$ with the Hamilton product:

$$v'=(\cos\theta+n\sin\theta)v=e^{n\theta}v$$

The tensor rotation algorithm defined by the function, rotate(tensor, axis, angle), in tensor_rotation.py does general vector and tensor rotation in $\mathbb{R}^3$ about an arbitrary rotation axis, not necessarily orthogonal to the original vector or tensor. For general three-dimensional rotation, we define the rotation quaternion as $q=e^{n(\theta/2)}$ and the conjugate of $q$ as $q^*=e^{-n(\theta/2)}$. The quaternion rotation operator has the following form for general vector and tensor rotation:

$$\mathcal{L}[v]=qvq^*,\,\,\mathcal{L}[M]=(q(qMq^*)^\text{T}q^*)^\text{T}$$

We should remark that quaternion-matrix multiplication is defined as column-wise quaternion multiplication [(source)](https://people.dsv.su.se/~miko1432/rb/Rotations%20of%20Tensors%20using%20Quaternions%20v0.3.pdf). Furthermore, we claim that $vq^*=qv$ whenever the rotation axis $\vec{n}$ and the vector $\vec{v}$ are perpendicular (straightforward verification), which will recover the rotation formula for the special case.

# Step 1: The Tensor Rotation Algorithm \[Back to [top](#toc)\]
$$\label{algorithm}$$

### Algorithm Pseudocode
```
function rotate(tensor, axis, angle):
    initialize quaternion q from rotation axis and angle
    if tensor is a nested list (i.e. matrix)
        convert tensor to a symbolic matrix
        if not (size of symbolic matrix is (3, 3))
            throw error for invalid matrix size
        end if
        initialize empty vector M
        for column in tensor
            append quaternion(column) onto M
        end for
        M = q * M *(conjugate of q) // rotate each column
        replace each column of tensor with the vector part
            of the associated column quaternion in M
        initialize empty vector M
        for row in tensor
            append quaternion(row) onto M
        end for
        M = q * M *(conjugate of q) // rotate each row
        replace each row of tensor with the vector part
            of the associated row quaternion in M
        convert tensor to a nested list
        return tensor
    else if tensor is a list (i.e. vector)
        if not (length of vector is 3):
            throw error for invalid vector length
        v = q * quaternion(tensor) * (conjugate of q)
        replace each element of tensor with the vector part
            of the associated vector quaternion v
        return tensor
    end if
    throw error for unsupported tensor type
end function
```

```python
""" Symbolic Tensor (Quaternion) Rotation

The following script will perform symbolic tensor rotation using quaternions.
"""

from sympy import Quaternion as quat
from sympy import Matrix
from sympy.functions import transpose

# Input:  tensor = 3-vector or (3x3)-matrix
#         axis   = rotation axis (normal 3-vector)
#         angle  = rotation angle (in radians)
# Output: rotated tensor (of original type)
def rotate(tensor, axis, angle):
    # Quaternion-Matrix Multiplication
    def mul(*args):
        if isinstance(args[0], list):
            q, M = args[1], args[0]
            for i, col in enumerate(M):
                M[i] = col * q
        else:
            q, M = args[0], args[1]
            for i, col in enumerate(M):
                M[i] = q * col
        return M
    # Rotation Quaternion (Axis, Angle)
    q = quat.from_axis_angle(axis, angle)
    if isinstance(tensor[0], list):
        tensor = Matrix(tensor)
        if tensor.shape != (3, 3):
            raise Exception('Invalid Matrix Size')
        # Rotation Formula: M' = (q.(q.M.q*)^T.q*)^T
        M = [quat(0, *tensor[:, i]) for i in range(tensor.shape[1])]
        M = mul(q, mul(M, q.conjugate()))
        for i in range(tensor.shape[1]):
            tensor[:, i] = [M[i].b, M[i].c, M[i].d]
        M = [quat(0, *tensor[i, :]) for i in range(tensor.shape[0])]
        M = mul(q, mul(M, q.conjugate()))
        for i in range(tensor.shape[0]):
            tensor[i, :] = [[M[i].b, M[i].c, M[i].d]]
        return tensor.tolist()
    if isinstance(tensor, list):
        if len(tensor) != 3:
            raise Exception('Invalid Vector Length')
        # Rotation Formula: v' = q.v.q*
        v = q * quat(0, *tensor) * q.conjugate()
        return [v.b, v.c, v.d]
    raise Exception('Unsupported Tensor Type')
```

# Step 2: Validation and Demonstration \[Back to [top](#toc)\]
$$\label{validation}$$

In the following section, we demonstrate the rotation algorithm, specifically for symbolic tensor rotation, and validate that the numeric or symbolic output from the algorithm does agree with a known result, usually obtained from a rotation matrix. The format of each code cell has the structure: generate expected result from a rotation matrix or NRPy+, generate recieved result from the rotation algorithm, assert that these are both equivalent.

## Step 2.a: Verifying Coordinate Axis Rotation \[Back to [top](#toc)\]
$$\label{axisrotation}$$

We recall that any general three-dimensional rotation can be expressed as the composition of a rotation about each coordinate axis (see [rotation theorem](https://en.wikipedia.org/wiki/Euler%27s_rotation_theorem)). Therefore, we validate our rotation algorithm using only rotations about each coordinate axis rather than about an arbitrary axis in three-dimensional space. Consider the following vector $\vec{v}$ and matrix $M$ defined below using the SymPy package.

```python
from sympy.matrices import rot_axis1, rot_axis2, rot_axis3
from sympy import latex, pi
from IPython.display import Math

v, angle = [1, 0, 1], pi/2
M = [[1, 2, 1], [0, 1, 0], [2, 1, 2]]
Math('\\vec{v}=%s,\\,M=%s' % (latex(Matrix(v)), latex(Matrix(M))))
```

$\displaystyle \vec{v}=\left[\begin{matrix}1\\0\\1\end{matrix}\right],\,M=\left[\begin{matrix}1 & 2 & 1\\0 & 1 & 0\\2 & 1 & 2\end{matrix}\right]$

We further recall that for any rotation matrix $R$ and vector $\vec{v}$, the rotated vector $\vec{v}'$ has the formula $\vec{v}'=R\vec{v}$ and the rotated matrix $M'$ has the formula $M'=RMR^\text{-1}$, where $R^{-1}=R^\text{T}$ since every rotation matrix is an [orthogonal matrix](https://en.wikipedia.org/wiki/Orthogonal_matrix).

```python
# vector rotation about x-axis
expected = rot_axis1(-angle) * Matrix(v)
received = Matrix(rotate(v, [1, 0, 0], angle))
assert expected == received; v_ = received

# matrix rotation about x-axis
expected = rot_axis1(-angle) * Matrix(M) * transpose(rot_axis1(-angle))
received = Matrix(rotate(M, [1, 0, 0], angle))
assert expected == received; M_ = received

Math('\\vec{v}\'=%s,\\,M\'=%s' % (latex(Matrix(v_)), latex(Matrix(M_))))
```

$\displaystyle \vec{v}'=\left[\begin{matrix}1\\-1\\0\end{matrix}\right],\,M'=\left[\begin{matrix}1 & -1 & 2\\-2 & 2 & -1\\0 & 0 & 1\end{matrix}\right]$

```python
# vector rotation about y-axis
expected = rot_axis2(-angle) * Matrix(v)
received = Matrix(rotate(v, [0, 1, 0], angle))
assert expected == received; v_ = received

# matrix rotation about y-axis
expected = rot_axis2(-angle) * Matrix(M) * transpose(rot_axis2(-angle))
received = Matrix(rotate(M, [0, 1, 0], angle))
assert expected == received; M_ = received

Math('\\vec{v}\'=%s,\\,M\'=%s' % (latex(Matrix(v_)), latex(Matrix(M_))))
```

$\displaystyle \vec{v}'=\left[\begin{matrix}1\\0\\-1\end{matrix}\right],\,M'=\left[\begin{matrix}2 & 1 & -2\\0 & 1 & 0\\-1 & -2 & 1\end{matrix}\right]$

```python
# vector rotation about z-axis
expected = rot_axis3(-angle) * Matrix(v)
received = Matrix(rotate(v, [0, 0, 1], angle))
assert expected == received; v_ = received

# matrix rotation about z-axis
expected = rot_axis3(-angle) * Matrix(M) * transpose(rot_axis3(-angle))
received = Matrix(rotate(M, [0, 0, 1], angle))
assert expected == received; M_ = received

Math('\\vec{v}\'=%s,\\,M\'=%s' % (latex(Matrix(v_)), latex(Matrix(M_))))
```

$\displaystyle \vec{v}'=\left[\begin{matrix}0\\1\\1\end{matrix}\right],\,M'=\left[\begin{matrix}1 & 0 & 0\\-2 & 1 & 1\\-1 & 2 & 2\end{matrix}\right]$

## Step 2.b: Verifying Symbolic Tensor Rotation \[Back to [top](#toc)\]
$$\label{symtensor}$$

The rotation algorithm does support symbolic rotation, as shown below with the second rank, symmetric tensor $h^{\mu\nu}$ rotated about the 4-vector $v^\mu$.

```python
from sympy import symbols, simplify, cse
import indexedexp as ixp # NRPy+: symbolic indexed expressions

angle = symbols('theta', real=True)
vU    = ixp.declarerank1("vU")
hUU   = ixp.declarerank2("hUU", "sym01")
rotatedhUU, N = rotate(hUU, vU, angle), len(hUU)

Math('hUU=%s' % latex(Matrix(hUU)))
```

$\displaystyle hUU=\left[\begin{matrix}hUU_{00} & hUU_{01} & hUU_{02}\\hUU_{01} & hUU_{11} & hUU_{12}\\hUU_{02} & hUU_{12} & hUU_{22}\end{matrix}\right]$

We demonstrate that a completely symbolic rotation applied to the tensor $h^{\mu\nu}$ does preserve the index symmetry ($h^{\mu\nu}=h^{\nu\mu}$).

```python
for i in range(N):
    for j in range(N):
        if j >= i: continue
        assert simplify(rotatedhUU[i][j] - rotatedhUU[j][i]) == 0
        print('Assertion Passed: rotatedhUU[{i}][{j}] == rotatedhUU[{j}][{i}]'.format(i=i, j=j))
```

    Assertion Passed: rotatedhUU[1][0] == rotatedhUU[0][1]
    Assertion Passed: rotatedhUU[2][0] == rotatedhUU[0][2]
    Assertion Passed: rotatedhUU[2][1] == rotatedhUU[1][2]

## Step 2.c: Common Subexpression Elimination \[Back to [top](#toc)\]
$$\label{cse}$$

If the rotation algorithm is given any symbolic input, then the resulting expression will support common subexpression elimination.

```python
cse_rotatedhUU = cse(Matrix(rotatedhUU))[1][0]
Math('\\text{cse}(hUU)=%s' % latex(Matrix(cse_rotatedhUU)))
```

$\displaystyle \text{cse}(hUU)=\left[\begin{matrix}x_{1} x_{37} + x_{10} x_{41} + x_{3} x_{38} - x_{40} x_{5} & x_{1} x_{40} - x_{10} x_{38} + x_{3} x_{41} + x_{37} x_{5} & x_{1} x_{38} + x_{10} x_{40} - x_{3} x_{37} + x_{41} x_{5}\\x_{1} x_{49} + x_{10} x_{53} + x_{3} x_{50} - x_{5} x_{52} & x_{1} x_{52} - x_{10} x_{50} + x_{3} x_{53} + x_{49} x_{5} & x_{1} x_{50} + x_{10} x_{52} - x_{3} x_{49} + x_{5} x_{53}\\x_{1} x_{61} + x_{10} x_{65} + x_{3} x_{62} - x_{5} x_{64} & x_{1} x_{64} - x_{10} x_{62} + x_{3} x_{65} + x_{5} x_{61} & x_{1} x_{62} + x_{10} x_{64} - x_{3} x_{61} + x_{5} x_{65}\end{matrix}\right]$

# Step 3: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Symbolic_Tensor_Rotation.pdf](Tutorial-Symbolic_Tensor_Rotation.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-Symbolic_Tensor_Rotation")
```

    Created Tutorial-Symbolic_Tensor_Rotation.tex, and compiled LaTeX file to
        PDF file Tutorial-Symbolic_Tensor_Rotation.pdf


# Stroop effect investigation
[Cédric Campguilhem](https://github.com/ccampguilhem/Udacity-DataAnalyst), September 2017

## Table of contents

- [Introduction](#Introduction)
- [Stroop effect experiment](#Stroop effect experiment)
- [Descriptive statistics](#Descriptive statistics)
- [Inferential statistics](#Inferential statistics)
    - [Hypothesis](#Hypothesis)
    - [Checking assumptions](#Checking assumptions)
    - [Critical t-value](#Critical t-value)
    - [t-statistic and decision](#t-statistic)
    - [Additional information](#Additional information)
- [Conclusion](#Conclusion)
- [Appendix](#Appendix)

## Introduction

This project is related to Inferential Statistics course for Udacity Data Analyst Nanodegree program. The purpose of this project is to investigate a phenomenon from the experimental psychology called [Stroop effect](https://en.wikipedia.org/wiki/Stroop_effect).

The project aims at investigating experimental results on a sample. We will use both descriptive and inferential statistics (hypothesis formulation and decision).

The project makes use of of Python language with [pandas](http://pandas.pydata.org/) for calculation parts and [seaborn](https://seaborn.pydata.org/) library for plotting.

## Stroop effect experiment *[top](#Top)*

The experiment consists in saying out loud the ink color in which a words in a list are printed. The participants are given two different lists of words naming colors. A **congruent** one in which words match the ink color and a **incongruent** list where words are different from ink color:

        Congruent list

        Incongruent list

        Source: faculty.washington.edu

The time (in seconds) it takes to each participant to enumerate ink colors is recorded for each list. The type of list (congruent or incongruent) is the **independent** variable in the experiment. The time it takes to go through the list is the **dependent** variable.

This experiment is a **dependent** sample experiment: the same participants are given both lists.

```python
#Import required libraries for project
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
#Read dataset
df = pd.read_csv('./dataset/stroopdata.csv')
```

As an example, here are the first ten participants recorded times for each list:

```python
df[:10].transpose().head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9

      Congruent
      12.079
      16.791
      9.564
      8.630
      14.669
      12.238
      14.692
      8.987
      9.401
      14.480

      Incongruent
      19.278
      18.741
      21.214
      15.687
      22.803
      20.878
      24.572
      17.394
      20.762
      26.282

## Descriptive statistics *[top](#Top)*

We first make a descriptive analysis of our groups. The following box plot shows distribution of recorded time for each list:

```python
#We re-organize table
df2 = df.stack().reset_index(level=1, name='time').rename(columns={"level_1": "list"})
df2 = df2.reset_index().rename(columns={"index": "participant"})
```

```python
figure = plt.figure(figsize=(12, 6))
ax = figure.add_subplot(111)
ax = sns.boxplot(x="list", y="time", data=df2, ax=ax);
texts = ax.set(xlabel='Type of list', ylabel='Time (seconds)', title='Distribution of recorded times for each list')
texts[0].set_fontsize(14)
texts[1].set_fontsize(14)
texts[2].set_fontsize(20)

```

Each box represents three quartiles (25%, mean and 75%) in addition to 1.5 interquartile range past the low and high quartiles. Participant time out of the interquartile range are reported as outliers (diamond markers).

In average, time it takes to go through seems higher in average for incongruent list (below 15 seconds in average for congruent list and above 20 seconds for non-congruent list). At this time, there is no evidence that this observation is statistically significant. To do that, we need to formulate an hypothesis. This will be the object of [next](#Inferential statistics) section.

For incongruent list, two recorded times are above the 1.5 interquartile range and are reported as outliers.

We can have a closer look at distributions:

```python
figure = plt.figure(figsize=(12, 6))
ax = figure.add_subplot(111)
ax = sns.distplot(df2[df2["list"] == "Congruent"]["time"], label="Congruent list", ax=ax);
ax = sns.distplot(df2[df2["list"] == "Incongruent"]["time"], label="Incongruent list", ax=ax);
ax.legend()
texts = ax.set(xlabel='Time (seconds)', ylabel='Proportion', title='Distribution of recorded times for each list')
texts[0].set_fontsize(14)
texts[1].set_fontsize(14)
texts[2].set_fontsize(20)
```

If we refer to kernel density estimators, we can see that distribution of recorded times for congruent list looks like a normal distribution. On the contrary, results for incongruent list differs from a normal distribution due to the outliers who have response time for experiment between 32 and 35.

The actual values for mean, standard deviation and quartiles are reported in the table below:

```python
df2.groupby("list").describe()["time"]
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      list

      Congruent
      24.0
      14.051125
      3.559358
      8.630
      11.89525
      14.3565
      16.20075
      22.328

      Incongruent
      24.0
      22.015917
      4.797057
      15.687
      18.71675
      21.0175
      24.05150
      35.255

In the above table, the standard deviation reported is a sample standard deviation taking into account [Bessel's correction](https://en.wikipedia.org/wiki/Bessel%27s_correction):

\begin{align}
\sigma = \sqrt{\frac{\sum_{i=1}^n (x_i - \bar{x})^2}{n - 1}}
\end{align}

## Inferential statistics *[top](#Top)*

### Hypothesis *[inferential](#Inferential statistics)*

In [previous](#Descriptive statistics) section, we have observed that average response time for incongruent list seems higher than the one recoded for congruent list. In this section, we want to know if this observation is statistically significant.

Before doint a t-test, we need to take several assumptions ([source](https://statistics.laerd.com/stata-tutorials/paired-t-test-using-stata.php)):
- The dependent variable should be continuous, which is the case in our case because we use time as dependent variable.
- The independent variable should consist of two categorical, related groups or matched pairs. In our case we have related groups: each participant is given the two different lists.
- There should be no significant outliers in the difference between the two related groups.
- The distribution of the differences in the dependent variable between the two related groups should be approximately normally distributed.

The last two assumptions need to be checked.

Let's formulate the problem this way. Our **null hypothesis** is that population mean response times for congruent ($\mu_{congruent}$) and incongruent ($\mu_{incongruent}$) lists are the same. Our **alternative** is that population mean reponse time for incongruent list is higher.

For this one-tailed, dependant sample [t-test](https://en.wikipedia.org/wiki/Student%27s_t-test), we choose to use an alpha level of 0.05:

\begin{equation}
\mathtt{H}_{0}: \mathtt{\mu}_{congruent} = \mathtt{\mu}_{incongruent} \\
\mathtt{H}_{A}: \mathtt{\mu}_{congruent} < \mathtt{\mu}_{incongruent} \\
\alpha = 0.05
\end{equation}

Let's set $\mathtt{\mu}_D = \mathtt{\mu}_{incongruent} - \mathtt{\mu}_{congruent}$ . The hypotheses may be re-written as follow:

\begin{equation}
\mathtt{H}_{0}: \mathtt{\mu}_{D} = 0 \\
\mathtt{H}_{A}: \mathtt{\mu}_{D} > 0 \\
\alpha = 0.05
\end{equation}

The problem is then a one-tailed t-test in positive direction.

In this problem, the degree of freedom is $n - 1$:

\begin{equation}
dof = n - 1 = 23
\end{equation}

where $n$ is the sample size (24).

### Checking assumptions *[inferential](#Inferential statistics)*

Before performing t-test, we need to check the following two remaining assumptions:

- There should be no significant outliers in the differences between the two related groups.
- The distribution of the differences in the dependent variable between the two groups should be approximately normally distributed.

```python
#We first calculate the difference:
df['difference'] = df["Incongruent"] - df["Congruent"]
df.head()
```

      Congruent
      Incongruent
      difference

      0
      12.079
      19.278
      7.199

      1
      16.791
      18.741
      1.950

      2
      9.564
      21.214
      11.650

      3
      8.630
      15.687
      7.057

      4
      14.669
      22.803
      8.134

```python
#Now we can make a boxplot to find outliers and a distplot
figure = plt.figure(figsize=(12, 6))
figure.suptitle("Assumption check with differences of response time (incongruent - congruent)", fontsize=20)
ax = figure.add_subplot(121)
ax = sns.boxplot(x="difference", data=df, ax=ax);
texts = ax.set(xlabel='Difference of response time (seconds)', title='Outliers in difference')
ax = figure.add_subplot(122)
ax = sns.distplot(df["difference"], label="Response time (seconds)", ax=ax);
texts = ax.set(xlabel='Difference of response time (seconds)', ylabel='Proportion',
               title='Distribution of response time differences')
```

There is one difference of response time identified as an outlier (above the 1.5 interquartile limit past the high quartile). In the right-hand picture, we can also see that the distribution differs from a uniform distribution because of a bump around 20 seconds. We cannot formally validate all assumptions with this dataset.

### Critical t-value *[inferential](#Inferential statistics)*

From [t-table](https://s3.amazonaws.com/udacity-hosted-downloads/t-table.jpg) we can get the critical value for $dof = 23$ and $\alpha = 0.05$ for a one-tailed t-test:

\begin{equation}
t_{critical} = 1.714
\end{equation}

If the t-statistic of our sample is greater than this value, then we may reject the null hypothesis.

## t-statistic and decision *[inferential](#Inferential statistics)*

t-statistic may be calculated this way:

\begin{equation}
t = \frac{\bar{x} - \bar{\mu}_{E}}{\frac{s}{\sqrt{n}}}
\end{equation}

Where $\bar{x}$ is the difference of the sample means, $s$ is the standard deviation of the samples mean, $n$ is the sample size and $\bar{\mu}_E$ is the expected difference of mean from null hypothesis. In our case we have $\bar{\mu}_E = 0$.

The standard deviation of the sample means is calculated with:

\begin{equation}
s = \sqrt{\frac{\sum_{i=1}^n (x_i - \bar{x})^2}{n-1}}
\end{equation}

```python
df["difference"].describe()
```

    count    24.000000
    mean      7.964792
    std       4.864827
    min       1.950000
    25%       3.645500
    50%       7.666500
    75%      10.258500
    max      21.919000
    Name: difference, dtype: float64

```python
import math
t = df["difference"].mean() / (df["difference"].std() / math.sqrt(24))
print t
```

    8.02070694411

Our sample has the following statitics $M = 7.96$, $SD = 4.86$ (rounded two decimal places).

The t-statistic is $t(23) = 8.02, p < .0001$ one-tailed positive direction.

As p-value is below the selected alpha level of 0.05, we **reject the null hypothesis**. This means that mean of response times with incongruent list is **	extremely significantly** higher than the mean of response times with congruent list.

As this is an experiment, we can also state that incongruent list **causes** higher response times.

### Additional information *[inferential](#Inferential statistics)*

In this section, we provide a 95% confidence interval for mean of differences of response time, and effect size parameters: Cohen's $d$ and $r^2$.

Cohen's d is the ratio of mean differences of response time and standard deviation of sample means:

\begin{equation}
d = \frac{\bar{x}}{s}
\end{equation}

$r^2$ parameter is:

\begin{equation}
r^2 = \frac{t^2}{t^2 + dof}
\end{equation}

Where $dof$ is the degree of freedom. In our case we have $d = 1.64, r^2 = .74$. This means that incongruent list explains 74% of differences of response time.

The margin of error is calculated with:

\begin{equation}
margin = t_{critical}.s
\end{equation}

Where $s$ is the standard deviation of differences of response time and $t_{critical}$ is the t value for $\alpha = 0.05$ two-tailed (95% confidence interval): 2.069.

Margin of error is 10.07 seconds. The 95% confidence interval:

\begin{equation}
95\% CI = \bar{x} \pm margin
\end{equation}

where $\bar{x}$ is the mean of differences of response time, is then:

95% CI = (-2.10, 18.03)

This means that if poupaltion takes the Stroop test then we expect to have an average of differences of response time between incongruent and congruent list between -2.10 and 18.03 seconds.

```python
#Cohen's d
d = df["difference"].mean() / (df["difference"].std())
print "d = ", d

#r2
r2 = t**2 / (t**2 + 23)
print "r2 = ", r2

#Margin of error
margin = 2.069 * df["difference"].std()
print "margin = ", margin

#95% confidence interval
CI = (df["difference"].mean() - margin, df["difference"].mean() + margin)
print "95% CI = ({:.2f}, {:.2f})".format(CI[0], CI[1])
```

    d =  1.63721994912
    r2 =  0.736636416145
    margin =  10.0653268775
    95% CI = (-2.10, 18.03)

## Conclusion *[top](#Top)*

In this project, we have conducted a t-test with a paired (dependent) sample. As this was an Stroop effect experiment, we have concluded that using an incrogruent list causes slower response time compared to a congruent list.

However, two assumptions regarding our sample (distribution should be like a normal distribution and presence of outliers) have been violated. Outlier presence affects both sample mean and sample standard deviation and t-statistic as a consequence. In this case a nonparametric test may be [prefered](http://www.basic.northwestern.edu/statguidefiles/ttest_unpaired_ass_viol.html).

The Stroop test highlights how humain brain processes differently words and colors. In his experiment, John Ridley Stroop also had a third neutral list where words naming different things than colors (animals for example). This third list showed faster response time than the incongruent list, proving the reduced level of interference.

The Stroop test has multiple applications, one of which is to gauge children brain development (it has been shown that interferences reduce from childhood to adulthood). Higher level of interferences are often associated with brain troubles such as Attention-Deficit Hyperactivity Disorder.

Sources: (https://en.wikipedia.org/wiki/Stroop_effect), (https://powersthatbeat.wordpress.com/2012/09/16/what-are-the-different-tests-for-the-stroop-effect-autismaid/), (https://imotions.com/blog/the-stroop-effect/).

## Appendix *[top](#Top)*

### References

Re-organization of panda dataframe on [StackOverflow](https://stackoverflow.com/questions/38241933/how-to-convert-column-names-into-column-values-in-pandas-python).

Paired t-test [assumptions](https://statistics.laerd.com/stata-tutorials/paired-t-test-using-stata.php).

Hypothesis testing on [Stat Trek](http://stattrek.com/hypothesis-test/hypothesis-testing.aspx).

One-tail and two-tailed tests().

Violation of [t-test assumptions](http://www.basic.northwestern.edu/statguidefiles/ttest_unpaired_ass_viol.html).

Statistical calculations with [GraphPad](http://www.graphpad.com/quickcalcs/).

```python
#Export to html
!jupyter nbconvert --to html --template html_minimal.tpl inferential_statistics.ipynb
```

    [NbConvertApp] Converting notebook inferential_statistics.ipynb to html
    [NbConvertApp] Writing 373752 bytes to inferential_statistics.html


Text provided under a Creative Commons Attribution license, CC-BY.  All code is made available under the FSF-approved BSD-3 license.  (c) Lorena A. Barba, Gilbert F. Forsyth 2017. Thanks to NSF for support via CAREER award #1149784.
[@LorenaABarba](https://twitter.com/LorenaABarba)

12 steps to Navier–Stokes
======
***

You should have completed Steps [1](./01_Step_1.ipynb) and [2](./02_Step_2.ipynb) before continuing. This Jupyter notebook continues the presentation of the **12 steps to Navier–Stokes**, the practical module taught in the interactive CFD class of [Prof. Lorena Barba](http://lorenabarba.com).

Step 3: Diffusion Equation in 1-D
-----
***

The one-dimensional diffusion equation is:

$$\frac{\partial u}{\partial t}= \nu \frac{\partial^2 u}{\partial x^2}$$

The first thing you should notice is that —unlike the previous two simple equations we have studied— this equation has a second-order derivative. We first need to learn what to do with it!

### Discretizing $\frac{\partial ^2 u}{\partial x^2}$

The second-order derivative can be represented geometrically as the line tangent to the curve given by the first derivative.  We will discretize the second-order derivative with a Central Difference scheme: a combination of Forward Difference and Backward Difference of the first derivative.  Consider the Taylor expansion of $u_{i+1}$ and $u_{i-1}$ around $u_i$:

$u_{i+1} = u_i + \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i + \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

$u_{i-1} = u_i - \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i - \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

If we add these two expansions, you can see that the odd-numbered derivative terms will cancel each other out.  If we neglect any terms of $O(\Delta x^4)$ or higher (and really, those are very small), then we can rearrange the sum of these two expansions to solve for our second-derivative.

$u_{i+1} + u_{i-1} = 2u_i+\Delta x^2 \frac{\partial ^2 u}{\partial x^2}\bigg|_i + O(\Delta x^4)$

Then rearrange to solve for $\frac{\partial ^2 u}{\partial x^2}\bigg|_i$ and the result is:

$$\frac{\partial ^2 u}{\partial x^2}=\frac{u_{i+1}-2u_{i}+u_{i-1}}{\Delta x^2} + O(\Delta x^4)$$

### Back to Step 3

We can now write the discretized version of the diffusion equation in 1D:

$$\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\nu\frac{u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n}}{\Delta x^2}$$

As before, we notice that once we have an initial condition, the only unknown is $u_{i}^{n+1}$, so we re-arrange the equation solving for our unknown:

$$u_{i}^{n+1}=u_{i}^{n}+\frac{\nu\Delta t}{\Delta x^2}(u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n})$$

The above discrete equation allows us to write a program to advance a solution in time. But we need an initial condition. Let's continue using our favorite: the hat function. So, at $t=0$, $u=2$ in the interval $0.5\le x\le 1$ and $u=1$ everywhere else. We are ready to number-crunch!

```python
import numpy                     #loading our favorite library
from matplotlib import pyplot    #and the useful plotting library
%matplotlib inline

nx = 128
domain_length = 64
dx = domain_length / (nx-1)
xspace = numpy.linspace(0, domain_length, nx)

nt = 100  #the number of timesteps we want to calculate
nu = 5   #the value of viscosity
sigma = .2 #sigma is a parameter, we'll learn more about it later
dt = sigma * dx**2 / nu #dt is defined using sigma ... more later!

u_IC = numpy.ones(nx)      #numpy function ones()
u_IC[int((nx-1)/4):int(nx/2 + 1)] = 2 #setting u = 2 between 0.5 and 1 as per our I.C.s

```

```python

def calc_diffusion(IC,nx,nt,nu,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time

    for n in range(nt):  #iterate through time
        un = u.copy() ##copy the existing values of u into un
        for i in range(1, nx - 1):
            u[i] = un[i] + nu * dt / dx**2 * (un[i+1] - 2 * un[i] + un[i-1])
    return u

u_FD = calc_diffusion(u_IC,nx,nt,nu,dt)
pyplot.plot(xspace, u_FD)

```

# Advection-Diffusion of a Gaussian Hill

In case of an isotropic diffusion, the analytical solution describing evolution of a Gaussian Hill can be expressed as

$$
C(\boldsymbol{x}, t)=\frac{\left(2\pi\sigma_{0}^{2}\right)^{D/2} }{\left(2\pi(\sigma_{0}^{2} + 2 k t)\right)^{D/2}}
C_0 \exp \left(-\frac{\left(\boldsymbol{x}-\boldsymbol{x}_{0}-\boldsymbol{u} t\right)^{2}}{2\left(\sigma_{0}^{2}+ 2 k t\right)}\right)
$$

where:

* $C_0$ - initial concentration,
* $D$ - number of dimensions,
* $t$ - time,
* $k$ - conductivity,
* $\boldsymbol{u}$ - velocity of advection
* $\sigma_{0}$ the initial variance of the distribution.

## Task

1) Implement the `GaussianHillAnal` class.
It shall have a method `get_concentration_ND(self, X, t)`, which will return the concentration at given time and space.

2) Benchmark the FD code against analytical solution.

```python
from sympy.matrices import Matrix
import numpy as np
import sympy as sp

class GaussianHillAnal:
    def __init__(self, C0, X0, Sigma2_0, k, U, D):
        """
        :param C0: initial concentration
        :param X0: initial position of the hill's centre = Matrix([x0, y0])
        :param U:  velocity = Matrix([ux, uy])
        :param Sigma2_0: initial width of the Gaussian Hill
        :param k: conductivity
        :param dimenions: number of dimensions
        """
        self.C0 = C0
        self.X0 = X0
        self.U = U
        self.Sigma2_0 = Sigma2_0
        self.k = k
        self.dim = D

    def get_concentration_ND(self, X, t):
        decay = 2.*self.k*t
        L = X - self.X0 - self.U*t
        C = self.C0
        C *= pow(2. * np.pi * self.Sigma2_0, self.dim / 2.)
        C /= pow(2. * np.pi * (self.Sigma2_0 + decay), self.dim / 2.)
        C *= sp.exp(-(L.dot(L)) / (2.*(self.Sigma2_0 + decay)))
        return C

```

```python
conductivity = nu

nt = 200  #the number of timesteps we want to calculate
nu = 5   #the value of viscosity
sigma = .2 #sigma is a parameter, we'll learn more about it later
dt = sigma * dx**2 / conductivity #dt is defined using sigma ... more later!

time_0    = dt*nt/2     # initial contidion for FD
time_spot = dt*nt       # time to be simulated (by FD and analytically)

X0 = Matrix([domain_length/2.]) # center of the hill
C0 = 1.                 # concentration
variance = 30           # initial variance
reference_level = 1

T_0 = np.zeros(nx)
T_anal = np.zeros(nx)

gha = GaussianHillAnal(C0, X0, variance, conductivity, Matrix([0]), D=1)

for i in range(nx):
    T_0[i] = reference_level + gha.get_concentration_ND(Matrix([xspace[i]]), time_0)
    T_anal[i] = reference_level + gha.get_concentration_ND(Matrix([xspace[i]]), time_spot)

T_FD = calc_diffusion(T_0,nx,nt,nu,dt)

import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 16})
figure, axis = plt.subplots(1, 1, figsize=(8, 6))
plt.subplots_adjust(hspace=1)
axis.set_title('Diffusion of a Gaussian Hill')
axis.plot(xspace, T_0, label=r'$T_{0}$')
axis.plot(xspace, T_anal, label=r'$T_{anal}$')
axis.plot(xspace, T_FD, label=r'$T_{FD}$', marker='x', linestyle="", markevery=1)
axis.set_xlabel('x')
axis.set_ylabel('Concentration')
axis.legend(loc="upper right")

```

## Questions:

* How do you find the FD solution compared to analytical one? Experiment with different dx, dt.

* How would you asses that your mesh is fine enought in a real CFD simulation (without analytical solution)?

## Answers

* Mesh convergence study

# Balance of Enthalpy

The simplified balance of Enthalpy $H=\rho c_p T$ is known as:
\begin{align}
\int \frac{\partial }{\partial t} (\rho c_p T ) dV +  \oint  ( \rho c_p T \boldsymbol{u} ) \cdot \boldsymbol{n} dS &= \oint \boldsymbol{n} \cdot \boldsymbol{q} dS + \int \dot{q} dV \\
\boldsymbol{q} &= k \nabla T
\end{align}
The heat flux $ \boldsymbol{q} $ is related to the thermal conductivity coefficient $k [ W/mK] $.

Applying the GGO theorem, we obtain:

$$
\frac{\partial }{\partial t} (\rho c_p T ) + \nabla \cdot (\boldsymbol{u} \rho c_p T ) = \nabla \cdot (k \nabla T)  + \dot{q}
$$

## Conservative and non-conservative scheme

Consider a 1D, unsteady heat transfer equation with variable conduction coefficient.
For simplicity, assume that the heat capacity $ \rho c_p = 1 $.

The equation be expressed in two mathematically equivalent forms:
$$
\frac{\partial u}{\partial t}=\frac{\partial}{\partial x}\left(k(x) \frac{\partial u}{\partial x}\right) \\
$$
or
$$
\frac{\partial u}{\partial t}=\frac{\partial u}{\partial x} \frac{\partial k}{\partial x}+k(x) \frac{\partial^{2} u}{\partial x^{2}}
$$

Both of these continous forms can be discretized.

First form leads to the *conservative scheme*

$$
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\frac{1}{\Delta x^{2}}\left(k_{i+\frac{1}{2}}\left(u_{i+1}^{n}-u_{i}^{n}\right)-k_{i-\frac{1}{2}}\left(u_{i}^{n}-u_{i-1}^{n}\right)\right)
$$

while the second form results in a *non-conservative scheme*

$$
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\frac{1}{\Delta x^{2}}\left(\frac{\left(k_{i+1}-k_{i-1}\right)}{2} \frac{\left(u_{i+1}^{n}-u_{i-1}^{n}\right)}{2}+k_{i}\left(u_{i+1}^{n}+u_{i-1}^{n}-2 u_{i}^{n}\right)\right)
$$

## Task

* Implement both the *conservative* and the *non-conservative* schemes.

* Impose the following BC:

```
u[int(nx/4)] = 10.
u[-int(nx/4)] = 1.

u[0] = 5.
u[-1] = 5.
```

* Run the simulation for different $k$ and compare results. Which physical quantity may become *not conserved* ?

```python
# variable conductivity
k = 1.*numpy.ones(nx)                   # be aware that dt = sigma * dx**2 / nu
k[:int(nx/2)] = 0.5                     # try with 0.1

u_IC = 5.*numpy.ones(nx)                 #numpy function ones()
u_IC[int((nx-1)/4):int(nx/2 + 1)] = 10   #setting u = 2 between 0.5 and 1 as per our I.C.s
dt = sigma * dx**2 / max(k)              #dt is defined using sigma

nt = 10000  # the number of timesteps we want to calculate

def calc_diffusion_variable_conductivity(IC,nx,nt,k,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time

    for n in range(nt):  #iterate through time
        u[int(nx/4)] = 10.
        u[-int(nx/4)] = 1.

        u[0] = 5.
        u[-1] = 5.

        un = u.copy() ##copy the existing values of u into un
        for i in range(1, nx - 1):
            # this is obviously bad : du/dt = k (d^2)u/(dx)^2
            # u[i] = un[i] + k[i] * dt / dx**2 * (un[i+1] - 2 * un[i] + un[i-1])

            # non conservative scheme
            u[i] = un[i] +  dt / dx**2 * ((k[i+1]-k[i-1])*(un[i+1]-un[i-1])/4.  + k[i]*(un[i+1] - 2 * un[i] + un[i-1]))

            # conservative scheme
            # kf = (k[i+1]+k[i])/2
            # kb = (k[i]+k[i-1])/2
            # u[i] = un[i] +  dt / dx**2 * (kf*(un[i+1]-un[i]) - kb*(un[i] - un[i-1]))

    return u

u_FD = calc_diffusion_variable_conductivity(u_IC,nx,nt,k,dt)
pyplot.plot(xspace, u_FD)
```

## Learn More

For a careful walk-through of the discretization of the diffusion equation with finite differences (and all steps from 1 to 4), watch **Video Lesson 4** by Prof. Barba on YouTube.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('y2WaK7_iMRI')
```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#CD2305;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #CD2305;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

> (The cell above executes the style for this notebook.)


```python
import sympy as s
import matplotlib

x, x0, y, z = s.symbols('x x0 y z', type=s.S.Reals)
my_func = s.Rational(1, 2) * x ** 3
print(my_func)

integral = s.integrate(my_func, x)
print(integral)
cdf = integral

inv = s.solveset(
    s.Eq(my_func.subs(x, y), x), y, domain=s.S.Reals)

inv_cdf = s.solveset(
    s.Eq(cdf.subs(x, y), x), y, domain=s.S.Reals)

print(inv)
print(inv_cdf)
```

    x**3/2
    x**4/8
    Intersection(FiniteSet(2**(1/3)*Abs(x)**(1/3)*sign(x)), Reals)
    Intersection(FiniteSet(2**(3/4)*x**(1/4), -2**(3/4)*x**(1/4)), Reals)

```python
type(cdf), type(inv_cdf)
```

    (sympy.core.mul.Mul, sympy.sets.sets.Intersection)

```python
m_value = 2

def evaluate_func(value):
    return float(my_func.subs({x: value, m: m_value}).evalf())

def evaluate_inverse(value):
    res = max(a for a in inv_cdf.subs({x: value, m: m_value}).evalf())
    return float(res)
```

```python
def array_map(f, x):
    return np.array(list(map(f, x)))
```

```python
from matplotlib import pyplot as plt
import numpy as np

MIN, MAX = -evaluate_func(0), evaluate_func(10)
print(MIN, MAX)

xs, y1s, y2s = [], [], []

for _ in range(10):
    x0 = MIN + np.random.random() * (MAX - MIN)
    y0 = evaluate_inverse(x0)

    xs.append(x0)
    y2s.append(y0)
    y1s.append(evaluate_func(y0))

def map_func(y2s, y1s):
    x = np.arange(-10, 10, 0.1)
    y= array_map(evaluate_func, x)
    plt.plot(x,y)

    plt.scatter(y2s, y1s)
    plt.show()

def map_inv(xs, y2s):
    x = np.arange(MIN, MAX, 0.01 * (MAX - MIN))
    y = array_map(evaluate_inverse, x)
    plt.plot(x,y)

    plt.scatter(xs, y2s)
    plt.show()

map_func(y2s, y1s)
map_inv(xs, y2s)
```

```python

```