# 2. Decentralized Optimizaiton

## 2.1 Network topologies

To run decentralized algorithms, we have to set up the network topology.

```python
import os
import scipy.io as sio
import networkx as nx   # nx will be used for network topology creating and plotting
from bluefog.common import topology_util
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# You can change NUM_PROC into any value smaller than the number of CPUs you have.
import os
NUM_PROC = 2 if os.getenv("TEST_ENV") else 8
NUM_PROC
```

### 2.1.1 Ring topology

```python
network_size = 8
G = topology_util.RingGraph(network_size)
labels={i: i for i in range(network_size)}
nx.draw_circular(G, labels=labels)
```

### 2.1.2 Mesh Grid topology

```python
network_size = 8
G = topology_util.MeshGrid2DGraph(network_size)
labels={i: i for i in range(network_size)}
nx.draw_spring(G, labels=labels)
```

### 2.1.3 Exponential-two topology

```python
network_size = 8
G = topology_util.ExponentialTwoGraph(network_size)
labels={i: i for i in range(network_size)}
nx.draw_circular(G, labels=labels)
```

## 2.2 Decentralized ADMM

Continue on the centralized ADMM algorithm. We consider the same consensus problem:
$$
    \min_{x} \sum_{i=1}^n h_i(x)
$$
Instead of relaxing $x$ into $\{x_i\}$ and forcing them to be the common value $y$, we can map it into a graph structure. More percisely, assigning $h_i$ into each vertex and relaxing the common $x$ by $\{x_i\}$ owned by each vertex. The constraint of $x_i = x_j$ can be transfered into constrain on the edge:

\begin{align}
    \min_{\{x_i\}_{i\in V},\; \{y_e\}_{e\in E} }&\;\; \sum_{i=1}^n h_i(x_i)\\
    {\rm subject\ to} &\;\; x_i - y_e=0,\;  x_j - y_e=0, \;\;\forall e=\{i,j\} \in E
\end{align}

As long as the graph is connected, it is easy to see it is equivalent to original consenus problem.

To solve this, we consider the augmented Lagrangian and the dual variables $u_{e,i}$ for $x_i - y_e=0$ and $u_{e,j}$ for $x_j - y_e=0$. After applying the ADMM algorithm and simplifying, we obtain

\begin{align}
    x_{i}^{k+1} =& {\rm Prox}_{(\alpha |N_i|)^{-1} h_i} (v_i^k)\\
    a_{i}^{k+1} =& \frac{1}{2|N_i|} \sum_{j\in N_i} x_j^{k+1}\\
    v_{i}^{k+1} =& v_{i}^{k} + a_{i}^{k+1} - (a_{i}^{k} + x_{i}^{k})/2
\end{align}

The code snippet:
```python
bf.set_topology(topology_util.RingGraph(bf.size()))

def DecentralizedADMMStepL2(A, b, x, a, v, n_i, alpha):
    next_x = ProximalStepL2(A, b, v, n_i, alpha)
    # `bf.in_neighbor_ranks()` loads all incoming neighbor ranks of the topology in `bf.set_topology()`
    neighbor_weights = {r: 1.0 / n_i for r in bf.in_neighbor_ranks()}
    next_a = bf.neighbor_allreduce(
        next_x, self_weight=0.0, neighbor_weights=neighbor_weights
    )   # <-- this is the key difference. Changing from allreduce to neighbor_allreduce
    next_v = v + next_a - (x + a) / 2
    return next_x, next_a, next_v
```

```python
! bfrun -np $NUM_PROC python DecentralizedADMM.py
```

    [Decentralized ADMM] Allreduce residue gradient norm: 1.1404968325674104e-15
    [Decentralized ADMM] Rank 1: ADMM residue gradient norm: 1.6168195861154377e-14
    [Decentralized ADMM] Rank 0: ADMM residue gradient norm: 1.6168195861154377e-14
    [Decentralized ADMM] Rank 2: ADMM residue gradient norm: 1.6168195861154377e-14
    [Decentralized ADMM] Rank 3: ADMM residue gradient norm: 1.6168195861154377e-14
    Last three entries of x_ar:
     tensor([[0.6872],
            [0.6552],
            [0.8751]], dtype=torch.float64)
    Last three entries of x_admm:
     tensor([[0.6872],
            [0.6552],
            [0.8751]], dtype=torch.float64)

```python
mse_records_dict = sio.loadmat('results/DecentralizedADMM.mat')

plt.semilogy(mse_records_dict["mse"][0], label="Decentralized ADMM")
plt.title("Decentralized ADMM")
plt.xlabel("iteration")
plt.ylabel("Mean-Square-Error")
plt.grid("on")
plt.legend()
dirname = 'images'
if not os.path.exists(dirname):
    os.makedirs(dirname)
plt.savefig(os.path.join(dirname, 'decentralized_admm.png'))
```

## 2.3 Mixing matrix

For commonly used undirected network, BlueFog provides the mixing matrix generated through the Metropolis–Hastings rule.

> **Metropolis-Hastings rule.** Providing a undirected and connected topology $\mathcal{G}$, we select $w_{ij}$ as
>
>\begin{align}
\hspace{-3mm}	w_{ij}=
	\begin{cases}
		\begin{array}{ll}\displaystyle
			\hspace{-2mm}\frac{1}{1 + \max\{d_i, d_j \}},& \mbox{if $j \in \mathcal{N}(i)$}, \\
			\hspace{-2mm}\displaystyle 1 - \sum_{j\in \mathcal{N}(i)}w_{ij}, & \mbox{if $i = j$},\\
			\hspace{-2mm}0,& \mbox{if $j \notin \mathcal{N}(i)$  and $j\neq i$}.
		\end{array}
	\end{cases}
\end{align}
>
> where $d_i = |\mathcal{N}(i)|$ (the number of incoming neighbors of agent $k$). It is easy to verify such $W$ is always doubly-stochastic.

### 2.3.1 Mixing matrix for the ring

The following code is to fetch mixing matrix provided by BlueFog.

```python
def FetchRingMixingMatrix(network_size):
    G = topology_util.RingGraph(network_size)   # organize nodes into a ring
    W = np.zeros((network_size, network_size))  # genearate an empty mixing matrix

    for rank in range(network_size):
        self_weight, neighbor_weights = topology_util.GetRecvWeights(G, rank)  # fetch weights
        W[rank,rank] = self_weight              # put weights into the mixing matrix
        for r, v in neighbor_weights.items():
            W[rank, r] = v

    return W
```

```python
! bfrun -np $NUM_PROC python Ring_mixingMatrix.py
```

    [[0.333 0.333 0.    0.    0.    0.    0.    0.333]
     [0.333 0.333 0.333 0.    0.    0.    0.    0.   ]
     [0.    0.333 0.333 0.333 0.    0.    0.    0.   ]
     [0.    0.    0.333 0.333 0.333 0.    0.    0.   ]
     [0.    0.    0.    0.333 0.333 0.333 0.    0.   ]
     [0.    0.    0.    0.    0.333 0.333 0.333 0.   ]
     [0.    0.    0.    0.    0.    0.333 0.333 0.333]
     [0.333 0.    0.    0.    0.    0.    0.333 0.333]]

    The sum of each col is: [1. 1. 1. 1. 1. 1. 1. 1.]
    The sum of each row is: [1. 1. 1. 1. 1. 1. 1. 1.]
    W is doubly stochastic.
    The second largest eigenvalue is: 0.804737854124365

### 2.3.2 Mixing matrix for the Mesh

```python
! bfrun -np $NUM_PROC python Mesh_mixingMatrix.py
```

    [[0.417 0.25  0.    0.    0.333 0.    0.    0.   ]
     [0.25  0.25  0.25  0.    0.    0.25  0.    0.   ]
     [0.    0.25  0.25  0.25  0.    0.    0.25  0.   ]
     [0.    0.    0.25  0.417 0.    0.    0.    0.333]
     [0.333 0.    0.    0.    0.417 0.25  0.    0.   ]
     [0.    0.25  0.    0.    0.25  0.25  0.25  0.   ]
     [0.    0.    0.25  0.    0.    0.25  0.25  0.25 ]
     [0.    0.    0.    0.333 0.    0.    0.25  0.417]]

    The sum of each col is: [1. 1. 1. 1. 1. 1. 1. 1.]
    The sum of each row is: [1. 1. 1. 1. 1. 1. 1. 1.]
    W is doubly stochastic.
    The second largest eigenvalue is: 0.8535533905932756

### 2.3.3 Mixing matrix for Exponential-two Graph

In the exponential-two graph, each node has the same incoming degree and outgoing degree. Exponential-two graph is a directed graph, BlueFog provides a mixing matrix following averaging rule for Exponential-two graph.

> **Averaging rule.** Providing an exponential-two topology $\mathcal{G}$, we select $w_{ij}$ as
>
>\begin{align}
\hspace{-3mm}	w_{ij}=
	\begin{cases}
		\begin{array}{ll}\displaystyle
			\hspace{-2mm}\frac{1}{1 + d},& \mbox{if $j \in \mathcal{N}_{\rm in}(i)$ or if $i = j$} , \\
			0,& \mbox{otherwise}.
		\end{array}
	\end{cases}
\end{align}
>
> where $d = |\mathcal{N}_{\rm in}(i)|$ (the number of incoming neighbors of agent $i$). It is easy to verify such $W$ is always doubly-stochastic because incoming degree $d$ of each node is the same.

Since Exponential-two Graph is a regular graph, such mixing matrix is doubly stochastic but not symmetric.

```python
! bfrun -np $NUM_PROC python Exp2_mixingMatrix.py
```

    [[0.25 0.   0.   0.   0.25 0.   0.25 0.25]
     [0.25 0.25 0.   0.   0.   0.25 0.   0.25]
     [0.25 0.25 0.25 0.   0.   0.   0.25 0.  ]
     [0.   0.25 0.25 0.25 0.   0.   0.   0.25]
     [0.25 0.   0.25 0.25 0.25 0.   0.   0.  ]
     [0.   0.25 0.   0.25 0.25 0.25 0.   0.  ]
     [0.   0.   0.25 0.   0.25 0.25 0.25 0.  ]
     [0.   0.   0.   0.25 0.   0.25 0.25 0.25]]

    The sum of each col is: [1. 1. 1. 1. 1. 1. 1. 1.]
    The sum of each row is: [1. 1. 1. 1. 1. 1. 1. 1.]
    W is doubly stochastic.
    The second largest eigenvalue is: 0.5000000000000003

### 2.3.4 Commonly-used topology summary

The following table summarizes the second largest eigenvalue and the maximum degree of some common network topologes. The mixing matrix of the undirected topologies, i.e., the ring, 2D-mesh, and fully-connected topology, are generated through the Matropolis-Hastings  rule, while the mixing matrix of exponential-2 topology (which is a directed network) is genereted through the Averaging rule.

| Network topology | second largest eigenvalue | maximum degree |
| -: | :-: | :-: |
| Undirected Ring | $1-O(\frac{1}{n^2})$ | 2 |
| Undirected 2D-Mesh | $1-O(\frac{1}{n})$ | 4 |
| Exponential-2 | $1-O(\frac{1}{\ln(n)})$ | $\ln(n)$ |
| Fully connected| $0$ |$n-1$ |

## 2.4 Average consensus

Given the mixing matrix, the average consensus algorithm is:

$$x_i^{k+1} = \sum_{j \in \mathcal{N}_i \cup \{i\}} w_{ij} x_j^{k}$$

It is guaranteed that $ \frac{1}{n}\sum_{i=1}^n \|x_i^k - \frac{1}{n}\sum_{i=1}^n x_i^0\|^2 \le C \rho^{2k}$, where $\rho$ is the second largest eigenvalue of the matrix $W$. Apparently, $\rho$ decides the convergence rate of the average consensus algorithm. One step of average consensus can be conducted as

```python
G = topology_util.RingGraph(bf.size())  # set up the topology
bf.set_topology(G, is_weighted=True)    # organize nodes into the topology
# one-step average consensus
def avg_consensus_one_step(x):
    x_next = bf.neighbor_allreduce(x)
    return x_next
```

```python
! bfrun -np $NUM_PROC python AverageConsensus.py
```

    Runing average consensus with topology Ring
    Progress 0/60
    Progress 10/60
    Progress 20/60
    Progress 30/60
    Progress 40/60
    Progress 50/60
    Runing average consensus with topology Mesh
    Progress 0/60
    Progress 10/60
    Progress 20/60
    Progress 30/60
    Progress 40/60
    Progress 50/60
    Runing average consensus with topology Exp2
    Progress 0/60
    Progress 10/60
    Progress 20/60
    Progress 30/60
    Progress 40/60
    Progress 50/60

```python
rel_error_dict = sio.loadmat('results/AverageConsensus.mat')
for graph in ['Ring', 'Mesh', 'Exp2']:
    record = rel_error_dict[graph].reshape(-1)
    plt.semilogy(record)
plt.legend(['Ring', 'Mesh', 'Exp2'], fontsize=16)
plt.xlabel('Iteration', fontsize=16)
plt.ylabel('Relative error', fontsize=16)
dirname = 'images'
if not os.path.exists(dirname):
    os.makedirs(dirname)
plt.savefig(os.path.join(dirname, 'AverageConsensus.png'))
```

The above figure depicts the influence of $\rho$.

## 2.5 Decentralized gradient descent

Recall the optimization problem:

\begin{align*}
\min_{x\in \mathbb{R}^d}\ \sum_{i=1}^n h_i(x)\hspace{1cm}  \mbox{(Opt-Problem)}
\end{align*}

Decentralized gradient descent has two fomulations:

\begin{align}
x_i^{(k+1)} &= \sum_{j\in \mathcal{N}_i \cup \{i\}} w_{ij} \Big(x_j^{(k)} - \alpha \nabla h_j(x_j^{(k)}) \Big)  \hspace{2.1cm} \mbox{(adaptation-then-combination, ATC-DGD)}\\
x_i^{(k+1)} &= \sum_{j\in \mathcal{N}_i \cup \{i\}} w_{ij} x_j^{(k)} - \alpha \nabla h_i(x_i^{(k)})   \hspace{2.6cm} \mbox{(adaptation-with-combination, AWC-DGD)}
\end{align}

The comparison between ATC and AWC DGD is as follows:

- ATC-DGD has smaller limiting bias than AWC-DGD

- ATC-DGD has a wider step-size stability range than AWC-DGD

- AWC-DGD can easily overlap communication and computation, and hence is more efficient per iteration than ATC-DGD

### 2.5.1 Compare limiting bias

In the following example, we will consider the simple scenario in which $h_i(x) = \frac{1}{2}\|A_i x - b_i\|^2$. The main code is as follows:

```Python
G = topology_util.RingGraph(bf.size())  # set up the topology
bf.set_topology(G, is_weighted=True)    # organize nodes into the topology

# one-step AWC-DGD.
def AWC_DGD_one_step(x, x_opt, A, b, alpha=1e-2):

    grad_local = A.t().mm(A.mm(x) - b)                       # compute local grad
    x_next = bf.neighbor_allreduce(x) - alpha*grad_local     # AWC update

    return x_next

# one-step ATC-DGD.
def ATC_DGD_one_step(x, x_opt, A, b, alpha=1e-2):

    grad_local = A.t().mm(A.mm(x) - b)      # compute local grad
    y = x - alpha*grad_local                # adapte
    x_new = bf.neighbor_allreduce(y)        # combination

    return x_next
```

```python
! bfrun -np $NUM_PROC python DGD_limitBias.py
```

    [Distributed Grad Descent] Rank 0: global gradient norm: 3.7252473072675936e-13

    Running ATC:
    Progress 0/200
    Progress 100/200

    Running AWC:
    Progress 0/200
    Progress 100/200

```python
rel_error_dict = sio.loadmat('results/DGD_limitBias.mat')

for method in ['ATC', 'AWC']:
    record = rel_error_dict[method].reshape(-1)
    plt.semilogy(record)

plt.legend(['ATC', 'AWC'], fontsize=16)
plt.xlabel('Iteration', fontsize=16)
plt.ylabel('Relative error', fontsize=16)
plt.title('step-size: 5e-3')

dirname = 'images'
if not os.path.exists(dirname):
    os.makedirs(dirname)
plt.savefig(os.path.join(dirname, 'DGD_limitBias.png'))
```

It is observed in the above figure that with the same step-size, ATC can converge to a better solution than AWC.

### 2.5.2 Compare stability range

```python
! bfrun -np $NUM_PROC python DGD_stability.py
```

    [Distributed Grad Descent] Rank 0: global gradient norm: 3.7252473072675936e-13

    Running ATC:
    Progress 0/200
    Progress 100/200

    Running AWC:
    Progress 0/200
    Progress 100/200

```python
rel_error_dict = sio.loadmat('results/DGD_stability.mat')

for method in ['ATC', 'AWC']:
    record = rel_error_dict[method].reshape(-1)
    plt.semilogy(record)

plt.legend(['ATC', 'AWC'], fontsize=16)
plt.xlabel('Iteration', fontsize=16)
plt.ylabel('Relative error', fontsize=16)
plt.title('step-size: 5e-2')

dirname = 'images'
if not os.path.exists(dirname):
    os.makedirs(dirname)
plt.savefig(os.path.join(dirname, 'DGD_stability.png'))
```

When we enlarge the step-size, it is observed that AWC diverges while ATC can still converge

### 2.5.3 Compare communication and computation overlapping

Computation is mainly executed in the CPUs or GPUs, while communication depends on the network situation, other machine's or core's response, etc.
In other words, they are using complete different resources, which is perfect for overlapping. If it is doing properly, it can save use lots of time.
Adapt-With-Combination is one typical example that can overlap the communication and computation naturally.

To achieve than you need to call the nonblocking version functions, for example:

``` python
def AWC_DGD_one_step(x, x_opt, A, b, alpha=1e-2):
    grad_local = A.t().mm(A.mm(x) - b)                       # compute local grad
    x_new = bf.neighbor_allreduce(x) - alpha*grad_local      # AWC update

def NonBlocking_AWC_DGD_one_step(x, x_opt, A, b, alpha=1e-2):
    x_handle = bf.neighbor_allreduce_nonblocking(x)          # Nonblocking function returns handle instead of tensor.
    grad_local = A.t().mm(A.mm(x) - b)                       # compute local grad
    x_new = bf.wait(x_handle) - alpha*grad_local             # AWC update. Note calling `wait` on the handle.
```

```python
! bfrun -np $NUM_PROC python NonBlocking.py
```

    Running ATC:
    Progress 0/1000
    Progress 200/1000
    Progress 400/1000
    Progress 600/1000
    Progress 800/1000
    ATC finishes in 3.258173704147339 seconds.

    Running AWC:
    Progress 0/1000
    Progress 200/1000
    Progress 400/1000
    Progress 600/1000
    Progress 800/1000
    AWC finishes in 3.0531022548675537 seconds.

    Running NBK-AWC:
    Progress 0/1000
    Progress 200/1000
    Progress 400/1000
    Progress 600/1000
    Progress 800/1000
    NBK-AWC finishes in 1.9056735038757324 seconds.

It is observed that Nonblocking-AWC has the shortest time to finish $1000$ iterations through overlapping of communication and computation.

```python
rel_error_dict = sio.loadmat('results/NonBlocking.mat')
for method in ['ATC', 'AWC', 'NBK-AWC']:
    record = rel_error_dict[method].reshape(-1)
    time = rel_error_dict[method+"_time"].reshape(-1)
    plt.semilogy(time, record)

plt.legend(['ATC', 'AWC', 'NBK-AWC'], fontsize=16)
plt.xlabel('Time', fontsize=16)
plt.ylabel('Relative error', fontsize=16)
plt.title('Running time comparison')

dirname = 'images'
if not os.path.exists(dirname):
    os.makedirs(dirname)
plt.savefig(os.path.join(dirname, 'NonBlockingTime.png'))
```

```python

```


Building and analyzing mathematical models in Jupyter

**Table of Contents**

< [Back to Jupyter intro](Intro.ipynb) | [Jump to Data analysis in Python](Data-Python.ipynb) >

## Introduction

Jupyter is a fully-functional alternative to Mathematica or Maple notebooks for developing and analyzing mathematical models in biology (or any other discipine, for that matter). For this, you will need to use a [Computer Algebra System](https://en.wikipedia.org/wiki/Computer_algebra_system) (CAS). A CAS is software that emulates manual (pen-and-paper) manipulations of mathematical expressions. Yes, it can be done, and very effectively, for a vast array of mathematical problems! A CAS combined with a graphing/plotting package like matplotlib gives you a powerful tool for mathematical modelling using a Jupyter notebook.

We will use Python's [SymPy](http://sympy.org/en/index.html) package, which provides powerful CAS features for most common mathematical modelling problems.

There is also [Sage](http://www.sagemath.org/), a more capable CAS. We will not use it here because is not a regular Python package, but rather, uses Python as its programming language. So unlike SymPy it cannot just be loaded in a Jupyter nb with a Python kernel. Instead, you will need to [install its Jupyter kernel](http://doc.sagemath.org/html/en/reference/repl/sage/repl/ipython_kernel/install.html). You can [install and try sage](http://www.sagemath.org/download.html) outside of Jupyter if you want.

So let's use SymPy in Jupyter. First, some imports (assuming you are using the Python kernel):

```python
%matplotlib inline
import matplotlib.pyplot as plt
from sympy import *
import scipy as sc
init_printing()
```

If you used [Anaconda](http://continuum.io/downloads) to install Jupyter, it should alreasy include SymPy, Matplotlib, IPython, NumPy, and other useful packages for scientific computing. If you don't have SymPy for some other reason, install it in Linux/mac using:
```bash
$ sudo apt-get install python-sympy
```
Otherwise, follow the instructions [here](http://docs.sympy.org/latest/install.html).

## Some preliminaries

Before we get started with our mathematical modelling session in Jupyter, some SymPy preliminaries.

### Symbolic variables

In CAS' like SymPy, we need to create symbolic variables for the mathematical variables we want to work with. A new symbolic variable can be created using `var`. Try this:

```python
x = var('x')
type(x) # check it's class
```

    sympy.core.symbol.Symbol

You can also define multiple symbolic variables at one go:

```python
a, b, c = var("a, b, c")
```

For more info on symbolic variables, [have a look at this](http://docs.sympy.org/latest/gotchas.html#variables).

It is often important to add assumptions (constraints) to our symbolic vars:

```python
x = var('x', real=True)
```

Now check:

```python
x.is_imaginary
```

    False

```python
x = Symbol('x', positive=True)
```

Again, check:

```python
x > 0
```

```python
x < 0
```

### Symbolic equations

We can define the mathematical equations (functions) that we will be using/manipulating as follows:

```python
MyFun = (pi + x)**2; MyFun
```

See the nice $\LaTeX$ - formatted output: this is where `init_printing()` comes handy.

SymPy has predefined expressions for a number of mathematical constants, such as: `pi` ($\pi$), `e` (exponential), `oo` (infinity).

You can also get your equation in latex syntax! Try:

```python
latex(MyFun)
```

    '\\left(x + \\pi\\right)^{2}'

That has extra escape slashes for Python to be able to parse it correctly. To display it in its actual form (that you can directly use in a $\LaTeX$ document), `print` it:

```python
print(latex(MyFun))
```

    \left(x + \pi\right)^{2}

### Numerical evaluation

To evaluate an expression numerically we can use the `evalf` function (or `N`). It takes an argument `n` which specifies the number of significant digits.

```python
pi.evalf(n=100) # pi to a 100 places after decimal!
```

`N()` is shorthand alias for `evalf()`:

```python
N(pi, 50)
```

So let's try evaluating our function:

```python
N(MyFun, 5)
```

When we numerically evaluate algebraic expressions we often want to substitute a symbol with a numerical value. In SymPy we do that using the `subs` function:

```python
MyFun.subs(x, 1.5)
```

Now let's evauate it:

```python
MyFun.subs(x, 1.5)
```

The `subs` function can also be used to substitute mathematical variables or expressions. Let's substitute $x$ with $a+\pi$:

```python
MyFun.subs(x, a+pi)
```

And assign it as a new symbolic equation for using later:

```python
MyFun_new = MyFun.subs(x, a+pi); MyFun_new
```

We can also numerically evaluate the function over a range of values using NumPy arrays:

```python
x_vec = sc.arange(0, 10, 0.1)
```

```python
MyFun_vec = sc.array([N(MyFun.subs(x, xx)) for xx in x_vec]) #Note: using a list comprehension!
```

We can also evaluate the new function `MyFun_new` we created by substitution above:

```python
MyFun_new_vec = sc.array([N((MyFun_new).subs(a, xx)) for xx in x_vec])
```

Now plot the two functions that you evaluated (try adding axes and a legend to these basic plots).

```python
fig, ax = plt.subplots()
ax.plot(x_vec, MyFun_vec)
ax.plot(x_vec, MyFun_new_vec)
```

However, numerical evaluation using `evalf()` can be very slow. There is a much more efficient way to do it by using [`lambdify()`](http://docs.sympy.org/latest/modules/utilities/lambdify.html) to "compile" a Sympy expression into a function that is much more efficient to evaluate numerically:

```python
MyFun_lamb = lambdify([x], MyFun, 'numpy')
```

The first argument is a (python) list of variables that `MyFun_lamb` will be a function of. In this case its only $x$. Now we can directly pass a numpy array and MyFun is evaluated more efficiently:

```python
MyFun_vec = MyFun_lamb(x_vec)
```

The speedup when using "lambdified" functions instead of direct numerical evaluation can be significant, often several orders of magnitude. Even in this simple example we get a significant speed up:

```python
%%timeit #remember this?

MyFun_vec = sc.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])
```

    24.6 ms ± 4.67 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
%%timeit

MyFun_vec = MyFun_lamb(x_vec)
```

    3.66 µs ± 546 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## Fundamental mathematical operations

Let's look at some fundamental methematical operations in SymPy that you will almost certainly use at some point in biological models. You can find a full list and addtional tutorials and examples at [SymPy documentation](http://docs.sympy.org/latest/index.html).

### Expand and factor

```python
expand(MyFun)
```

The `expand` function takes a number of keywords arguments which we can tell the functions what kind of expansions to want to perform; use `help(expand)` (or the [SymPy documentation](http://docs.sympy.org/latest/index.html)) for more info.

You can also orint the result of an manipulation in its raw python form:

```python
print(expand(MyFun))
```

    x**2 + 2*pi*x + pi**2

You can also factor using, well, `factor()`:

```python
factor(x**2 + 2*pi*x + pi**2)
```

### Apart and together

To manipulate symbolic expressions of fractions, you can use the `apart` and `together` functions:

```python
f1 = 1/((a+1)*(a+2)); f1
```

```python
apart(f1)
```

```python
f2 = 1/(a+2) + 1/(a+3); f2
```

```python
together(f2)
```

### Simplification

The `simplify` tries to simplify an expression into a nice looking expression, using various techniques. More specific alternatives to the `simplify` functions also exist: `trigsimp`, `powsimp`, `logcombine`, etc. Applying `simplify` to the above example will give the same result as `together`:

```python
simplify(f2)
```

Note that simplify usually combines fractions but does not factor.

In addition to algebraic manipulations, the other main use of CAS is to do calculus, like derivatives and integrals of algebraic expressions.

### Differentiation

Differentiation is usually simple. Use the `diff` function. The first argument is the expression to take the derivative of, and the second argument is the symbol by which to take the derivative:

```python
diff(MyFun_new, a)
```

For higher order derivatives we can do:

```python
diff(MyFun_new, a, a)
```

```python
diff(MyFun_new**2, a, 2) # same as above
```

You can directly apply another manipulation to the result of a previous operation:

```python
expand(diff(MyFun_new**2, a, 2))
```

Calculate the derivative of a multivariate expression:

```python
x, y, z = var("x,y,z")
```

```python
f = sin(x*y) + cos(y*z)
```

$\frac{d^3f}{dxdy^2}$

```python
diff(f, x, 1, y, 2)
```

### Integration

Integration is done in a similar fashion:

```python
MyFun
```

```python
integrate(MyFun, x)
```

By providing limits for the integration variable we can evaluate definite integrals:

```python
integrate(MyFun, (x, -1, 1))
```

and also improper integrals

```python
integrate(exp(-x**2), (x, -oo, oo))
```

Remember, `oo` is the SymPy notation for inifinity.

### Sums and products

You can evaluate sums and products using `Sum`. Noyte that tjhis function is named `Sum` and not `sum` to avoind namespace conflict.

```python
n = var("n")
```

```python
Sum(1/n**2, (n, 1, 10))
```

```python
Sum(1/n**2, (n,1, 10)).evalf()
```

```python
Sum(1/n**2, (n, 1, oo)).evalf()
```

Products work much the same way:

```python
Product(n, (n, 1, 10)) # 10!
```

### Limits

Limits can be evaluated using the `limit` function. For example,

```python
limit(sin(x)/x, x, 0)
```

We can use 'limit' to check the result of derivation using the `diff` function:

```python
f
```

```python
diff(f, x)
```

$\displaystyle \frac{\mathrm{d}f(x,y)}{\mathrm{d}x} = \frac{f(x+h,y)-f(x,y)}{h}$

```python
h = var("h")
```

```python
limit((f.subs(x, x+h) - f)/h, h, 0)
```

OK!

We can change the direction from which we approach the limiting point using the `dir` keywork argument:

```python
limit(1/x, x, 0, dir="+")
```

```python
limit(1/x, x, 0, dir="-")
```

### Series

Series expansion is also one of the most useful features of a CAS. In SymPy we can perform a series expansion of an expression using the `series` function:

```python
series(exp(x), x) # this is a classic!
```

By default it expands the expression around $x=0$, but we can expand around any value of $x$ by explicitly include a value in the function call:

```python
series(exp(x), x, 1)
```

Or try:

```python
series(log(x), x, 0) # will not work why?
```

```python
series(log(x), x,1) # this will work, however
```

And we can explicitly define to which order the series expansion should be carried out:

```python
series(exp(x), x, 0, 3)
```

Another way to do the same:

```python
exp(x).series(x,0,3)
```

The series expansion includes the order of the approximation, which is very useful for keeping track of the order of validity when we do calculations with series expansions of different orders:

```python
s1 = cos(x).series(x, 0, 5); s1
```

```python
s2 = sin(x).series(x, 0, 2); s2
```

```python
expand(s1 * s2)
```

If we want to get rid of the order information we can use the `removeO` method:

```python
expand(s1.removeO() * s2.removeO())
```

### Matrix algebra

Matrices are defined using the `Matrix` class:

```python
m11, m12, m21, m22 = var("m11, m12, m21, m22")
b1, b2 = var("b1, b2")
```

```python
A = Matrix([[m11, m12],[m21, m22]]) # Again, note: capital M for to avoid namespace conflict
A
```

```python
b = Matrix([[b1], [b2]]); b
```

With `Matrix` class instances we can do the usual matrix algebra operations:

```python
A**2
```

```python
A * b
```

And calculate determinants and inverses, and the like:

```python
A.det()
```

```python
A.inv()
```

### Solving equations

For solving equations and systems of equations we can use the `solve` function:

```python
solve(x**2 - 1, x)
```

```python
solve(x**4 - x**2 - 1, x)
```

System of equations:

```python
solve([x + y - 1, x - y - 1], [x,y])
```

In terms of other symbolic expressions:

```python
solve([x + y - a, x - y - c], [x,y])
```

You can also solve a single, or a system of ordinary differential equations (ODEs) using
[dsolve](http://docs.sympy.org/latest/modules/solvers/ode.html). WE will use this in a couple of the biological examples below.

## Some biological examples

Here are some examples of development and analysis of some fundamental mathematical models in biology.

### One population: Exponential growth

Let's look at how populations grow when there are no environmental constraints. The differential equation model is:

\begin{equation}\label{eq:exp_growth}
	\frac{\text{d}N}{\text{d}t} = r_m N
\end{equation}

where $r_m$ is the intrinsic, constant rate of population gowth (units of 1/time), and $N$ is population size (or biomass abundance). I use the subscript $m$ in $r_m$ to denote both [Malthusian](https://en.wikipedia.org/wiki/Malthusianism) and maximal population growth rate because, in theory, without any constraints, this growth rate is expected to at its theoretical/biological maximum.

Let's solve equation \ref{eq:exp_growth} so that we can calculate population size $N_t$ at any given time $t$ given a starting population size  $N_0$ at time 0.

First assign the symbolic variables:

```python
r_m, N, t = var("r_m N t",real = True) # the real bit is not really necessary here
```

Now tell SymPy that $N$ is a function:

```python
N = Function('N')
```

Define $N$ is a derivative of $t$

```python
dN_dt = Derivative(N(t), t) - r_m*N(t); dN_dt
```

Note that we have simply re-written the condition that LHS = RHS in eqn \ref{eq:exp_growth}. Now that we have the differential equation set up, we can solve it using `dsolve`. Since this is a simple ODE, Sympy can do it on its own with no hints or guesses (unlike more complex ODEs; [see the documentation](http://docs.sympy.org/latest/modules/solvers/ode.html)):

```python
MyEq_sol = dsolve(dN_dt); MyEq_sol
```

If you remember your high-school calculus, you might recall that $C_1$ here is an arbitrary constant. We now need to re-express it in terms of the initial conditions. We can do so by setting $t = 0$, and then setting that equal to $N(0)$ at time 0:

$$C_1 = N_0$$

That is,

\begin{equation}\label{eq:exp_growth_sol}
N{\left (t \right )} = N_0 e^{r_m t}
\end{equation}

We could use Sympy for this last step as well (using `subs()` like you learned above), but that would be just plain silly -- like using a sledge-hammer to drive in a nail!

We can now have a go at plotting the model (eqn. \ref{eq:exp_growth}), and also the solution eqn. \ref{eq:exp_growth_sol}. First, let's get an approximate solution by using numerical integration (which you learnt in the Advanced Python chapter):

```python
from scipy  import integrate

# parameters
r_m = 1.

# initial conditions
N_0 = 0.1

# The time vector
t_vec = sc.arange(0, 10., 0.01)

def exp_pop(N, t, r_m):
    """The right-hand side of the exponential growth ODE"""
    return r_m*N

N_vec = integrate.odeint(exp_pop, N_0, t_vec, args=(r_m,)) # the comma is needed!

# plot the numerical solution
plt.plot(t_vec, N_vec)
plt.xlabel('Time') ; plt.ylabel('$N$')

# plot analytical solution
plt.plot(t_vec, N_0 * sc.exp(r_m * t_vec),'k--')
plt.legend(['numerical approximation', 'analytical solution'], loc='best') # draw legend
```

These look practically identical. But they are not:

```python
N_vec - N_0 * sc.exp(r_m * t_vec)
```

    array([[  0.00000000e+00,  -1.00501671e-03,  -2.02013400e-03, ...,
             -2.13744854e+03,  -2.15893125e+03,  -2.18062988e+03],
           [  1.00502283e-03,   6.11819401e-09,  -1.01511118e-03, ...,
             -2.13744753e+03,  -2.15893025e+03,  -2.18062887e+03],
           [  2.02014197e-03,   1.01512526e-03,   7.96376975e-09, ...,
             -2.13744651e+03,  -2.15892923e+03,  -2.18062786e+03],
           ...,
           [  2.13744963e+03,   2.13744863e+03,   2.13744761e+03, ...,
              1.09563845e-03,  -2.14816243e+01,  -4.31802491e+01],
           [  2.15893236e+03,   2.15893136e+03,   2.15893034e+03, ...,
              2.14838268e+01,   1.10688877e-03,  -2.16975180e+01],
           [  2.18063100e+03,   2.18062999e+03,   2.18062898e+03, ...,
              4.31824631e+01,   2.16997432e+01,   1.11834951e-03]])

### One population: Logistic Population growth

Populations eventually run into contraints, even if they can grow exponentially at the start (eqn \ref{eq:exp_growth}). The classical model for logistic growth in population density ($N$) captures this dynamic:

\begin{equation}\label{eq:logist_growth}
	\frac{\text{d}N}{\text{d}t} = r_m N \left(1-\frac{N}{K}\right)
\end{equation}

where $K$ is the carrying capacity of the environment, while $r_m$ is the same parameter same as above. Let's solve this one as well.

As in the case of the exponential growth above, let's find the solution to this equation for any arbitrary timepoint $t$.

Again, first we define the vars and the fuincton:

```python
r_m, K, N, t = var("r_m K N t",real = True) # the real bit is not really necessary here

N = Function('N')
```

```python
dN_dt = Derivative(N(t), t) - r_m * N(t) * (1 - N(t) / K); dN_dt
```

Again, as in the exponential growth example, we have simply re-written the condition that LHS = RHS in eqn \ref{eq:logist_growth}. Now we can solve the ODE:

```python
MyEq_sol = dsolve(dN_dt); MyEq_sol
```

This is a bit more complicated than the solution for exponential growth above. But we can solve it the same way. First substitute $t = 0$, and then solve the resulting equation for the initital condition of $N_0$, which then gives This the time-dependent solution:
\begin{equation}
    N_t = \frac{N_0 K\mathrm{e}^{r_m t}}{K + N_0(\mathrm{e}^{r_m t}-1)}
\end{equation}

You can do the last steps to obtain this solution using Sympy as well (I leave it to you to try it).

No let's again compare the analytical solution against the numerical one:

```python
from scipy  import integrate

# parameters
r_m = 1.
K = 10.
# initial condition
N_0 = 0.1

#The time vector
t_vec = sc.arange(0, 10., 0.01)

def log_pop(N, t, r_m, K):
    """The right-hand side of the logistic ODE"""
    return r_m*N*(1-N/K)

N_vec = integrate.odeint(log_pop, N_0, t_vec, args=(r_m, K));

plt.plot(t_vec, N_vec) # plot the solution
plt.xlabel('Time') ; plt.ylabel('$N$')

# plot analytical solution
plt.plot(t_vec, K * N_0 * sc.exp(r_m * t_vec)/(K + N_0 * (sc.exp(r_m * t_vec) - 1.)),'k--')
plt.legend(['numerical approximation', 'analytical solution'], loc='best') # draw legend
```

### Two interacting populations: The Lotka-Volterra predator-prey model

Now for the classical Lotka-Volterra model that you encountered in the advanced Python week (without logistic growth for the consumer, $C$).

\begin{align}
\frac{dN}{dt} &= r_m N \left(1-\frac{N}{K}\right) - a N C\\
\frac{dC}{dt} &= e a N C - z C
\end{align}

here $r_m$ and $K$ is the Resource's growth rate and carrying capacity respectively as in the Logistic equation, $a$ is the consumer's search rate for the resource, $e$ is consumer's biomass conversion efficiency, and $z$ is it's mortality rate.

To solve this system of ODEs, we will take a different approach -- We will solve for the equilibrum (steady state for the two species' populations). Again, we start by define the vars:

```python
r_m, a, e, z, K, N, C, t = var("r_m, a, e, z, K, N, C, t",real = True)
```

Now define the sysyem of ODEs for Sympy:

```python
dN_dt = r_m * N *(1-N/K) - a * N * C
dC_dt = e * a * N * C - z * C

dC_dt, dN_dt
```

No define the equilibrium state:

```python
N_eqlb = Eq(dN_dt, 0)
C_eqlb = Eq(dC_dt, 0)
N_eqlb, C_eqlb
```

Solve it:

```python
N_eqlb_sol = solve(N_eqlb, C)
C_eqlb_sol = solve(C_eqlb, N)

N_eqlb_sol, C_eqlb_sol
```

So there is one equilbriumm  solution where both species maintain their non-zero populations. That was easy! Now you don't need to guess what parameter values will give you coexistence (both populations remain > 0). Just substitute parameter combinations that satisfy the conditions that

$$ \frac{r_{m} \left(K - N\right)}{K a} > 0, \textrm{ and } \frac{z}{a e} > 0$$

As an exercise try plotting this exact solution for the steady state along with the one you would obtain (asymptotically) using numerical integration of the system of ODEs. Below is code that does the numercial integration (essenatially, same as the `LV.py` script).

```python
from scipy import integrate

t_vec = sc.arange(0, 100., 0.01)

# parameters
r_m = 1.
a = 1
e = 0.5
z = .5
K =10

# initial condition: this is an array now!
N0C0 = sc.array([1., 1.])

# the function still receives only `x`, but it will be an array, not a number
def LV(NC, t, r_m, K, a, e, z):
    # Unlike the esponental and logistic growth model, we now need to convert
    # the output to a numpy array as it has two populations.
    return sc.array([ r_m * NC[0]*(1-NC[0]/K) - a * NC[0] * NC[1],
                   e * a * NC[0] * NC[1] - z * NC[1] ])

NC_vec = integrate.odeint(LV, N0C0, t_vec, (r_m, K, a, e, z))

# Check NC_vec's dimensions:
# print(NC_vec.shape)

# plot the solution
plt.plot(t_vec, NC_vec)
plt.xlabel('Time'); plt.ylabel('Population size') # and of y-axis
plt.legend(['Resource ($N$)', 'Consumer ($C$)'], loc='best')
```

An useful thing to do here is take a look at the *phase space*, that is, plot only the dependent variables, without respect to time:

```python
plt.plot(NC_vec[0,0], NC_vec[0,1], 'o')
print('Initial condition:', NC_vec[0])

plt.plot(NC_vec[:,0], NC_vec[:,1])

#Another solution with a different initial condition:
#NC_vec2 = odeint(LV, [2., 4.], t_vec, (r_m, K, a, e, z))
#plt.plot(NC_vec2[:,0], NC_vec2[:,1])
#plt.plot(NC_vec2[0,0], NC_vec2[0,1], 'o')
#plt.xlabel('Resource Population size'); plt.ylabel('Consumer Population size') # and of y-axis
```

## Readings abd Resources

* [The SymPy documentation](http://sympy.org/en/index.html)
* [SymPy online](http://live.sympy.org)


# Multiscale Geographically Weighted Regression - Binomial dependent variable

The model has been explored and tested for multiple parameters on real and simulated datasets. The research includes the following outline with separate notebooks for each part.

**Notebook Outline:**

**Introduction Notebook (current)**
- [Introduction](#Introduction)
 - [Introduction to the project](#Introduction-to-the-project)
 - [Statistical Equations](#Statistical-Equations)
- [Local Scoring Algorithm](#Local-Scoring-Algorithm)
- [Notebooks with tests](#Notebooks-with-Tests)
- [References](#References)

[Back to the main page](https://mehak-sachdeva.github.io/MGWR_book/)

---

# Introduction

***

## Introduction to the problem

As prefaced earlier, the Geographically Weighted Regression model in PySAL can currently estimate Gaussian, Poisson and Logistic models though the Multiscale extension of the GWR model is currently limited to only Gaussian models. This part of the project aims to expand the MGWR model to nonlinear local spatial regression modeling techniques where the response outcomes may be binomial (or a Logit model). This will enable a richer and holistic local statistical modeling framework to model multi-scale process heterogeneity for the open source community.

## Statistical Equations

***

A conventional Logistic regression model with $x_1, x_2, ... ,x_k$ as predictors, a binary(Bernoulli) response variable y and l denoting the log-odds of the event that y=1, can be written as:

\begin{align}
l = log_b ( p / (1-p)) = ({\sum} {\beta} & _k x _{k,i}) \\
\end{align}

where  $x_{k,1}$ is the kth explanatory variable in place i, $𝛽_{ks}$ are the parameters and p is the probability such that p = P( Y = 1 ).

By exponentiating the log-odds:

$p / (1-p) = b^ {𝛽_0+𝛽_1 x_1+𝛽_2 x_2} $

It follows from this - the probability that Y = 1 is:

$p = (b^ {𝛽_0 + 𝛽_1 x_1 + 𝛽_2 x_2}) / (b^ {𝛽_0 + 𝛽_1 x_1 + 𝛽_2 x_2} + 1)$ = $1 / (1 + b^ {-𝛽_0 + 𝛽_1 x_1 + 𝛽_2 x_2})$

# Local Scoring Algorithm

***

Following the technique from (Hastie & Tibshirani, 1986), for logisitic generalized additive models the model was estimated using the local scoring algorithm as follows:

1. Initialize the current estimate of the additive predictor $n_i^{old}$:
$n_i^{old} = {\sum} {\beta}_k  X_k$
and the probability such P(Y=1): $p_i^{old} = exp({n_i^{old}})/(1+exp({n_i^{old}}))$

2. Compute the working response:

$z_i = n_i^{old} + (y_i - p_i^{old})/(p_i^{old}(1-p_i^{old}))$

3. compute weights $w_i = p_i^{old} (1-p_i^{old})$

4. obtain $n_i^{new}$ by fitting a weighted additive model to $z_i$. In this the smoothers in the backfitting algorithm incorporate the additional weights and GWR is used for the linear parts.

These steps are repeated until the relative change in the fitted coefficients and the functions is below a tolerance threshold (1e-05 in this case).

Reference for these equations: http://ugrad.stat.ubc.ca/~nancy/526_2003/projects/kazi2.pdf

### Further work required:

The parameters for the estimated model using Monte Carlo tests with simulated data are close to expected. Further exploration is required to theoretically justify the model in the context of spatial data models, especially MGWR.

As an exploration, this work includes results from both adding a stochastic error to the model during calibration and without it. Results for both are shown in the notebooks below.

# Notebooks with Tests

**[Initial module changes and univariate model check ](http://mehak-sachdeva.github.io/MGWR_book/Html/Binomial_MGWR_univariate_check)**
- Setup with libraries
- Fundamental equations for Binomial MGWR
- Example Dataset
- Helper functions
- Univariate example
    - Parameter check
    - Bandwidths check

**[Simulated Data example](http://mehak-sachdeva.github.io/MGWR_book/Html/Simulated_data_example_Binomial-MGWR)**
- Setup with libraries
- Create Simulated Dataset
    - Forming independent variables
    - Creating y variable with Binomial distribution
- Univariate example
    - Bandwidth: Random initialization check
    - Parameters check
- Multivariate example
    - Bandwidths: Random initialization check
    - Parameters check
- Global model parameter check

**[Real Data example](http://mehak-sachdeva.github.io/MGWR_book/Html/Real_data_example_Binomial-MGWR)**

- Setup with libraries
- Landslide Dataset
- Univariate example
    - Bandwidth: Random initialization check
    - Parameter check
- Multivariate example
    - Bandwidths: Random initialization check
- MGWR bandwidths
- AIC, AICc, BIC check

## Monte Carlo Tests

***

### Monte Carlo tests for model estimated with error

**[Monte Carlo Simulation Visualization](http://mehak-sachdeva.github.io/MGWR_book/Html/Binomial_MGWR_MonteCarlo_Results)**

- Setup with libraries
- List bandwidths from pickles
- Parameter functions
- GWR bandwidth
- MGWR bandwidths
- AIC, AICc, BIC check
    - AIC, AICc, BIC Boxplots for comparison
- Parameter comparison from MGWR and GWR

### Monte Carlo tests for model estimated without error

**[Monte Carlo Simulation Visualization](http://mehak-sachdeva.github.io/MGWR_book/Html/Binomial_MGWR_MonteCarlo_Results-we)**

- Setup with libraries
- List bandwidths from pickles
- Parameter functions
- GWR bandwidth
- MGWR bandwidths
- AIC, AICc, BIC check
    - AIC, AICc, BIC Boxplots for comparison
- Parameter comparison from MGWR and GWR

# References:

1. Fotheringham, A. S., Yang, W., & Kang, W. (2017). Multiscale Geographically Weighted Regression (MGWR). Annals of the American Association of Geographers, 107(6), 1247–1265. https://doi.org/10.1080/24694452.2017.1352480

2. Yu, H., Fotheringham, A. S., Li, Z., Oshan, T., Kang, W., & Wolf, L. J. (2019). Inference in Multiscale Geographically Weighted Regression. Geographical Analysis, gean.12189. https://doi.org/10.1111/gean.12189

3. Hastie, T., & Tibshirani, R. (1986). Generalized Additive Models. Statistical Science, 1(3), 297–310. https://doi.org/10.1214/ss/1177013604

4. Wood, S. N. (2006). Generalized additive models : an introduction with R. Chapman & Hall/CRC.

[Back to the main page](https://mehak-sachdeva.github.io/MGWR_book/)


# Lista 3 - ICs e Bootstrap

## Intervalos de Confiança

Intervalos de confiança são intervalos calculados a partir de observações que podem variar de amostra para amostra e que com dada frequência (nível de confiança) inclui o parâmetro de interesse real não observável.

**Por exemplo:** Um intervalo com nível de confiança de 99% para a média de uma variável aleatória significa que ao calcularmos *n* intervalos de confiança tomando como base *n* amostras aleatórias, espera-se que 99% dos intervalos construídos contenham o valor real do parâmetro (média).

Em outras palavras, o nível de confiança seria a proporção de intervalos de confiança construídos em experimentos separados da mesma população e com o mesmo procedimento que contém o parâmetro de interesse real.

Foram ministradas duas maneiras de construírmos intervalos de confiança:

- Probabilisticamente direto dos dados (Forma clássica).
- Via sub-amostragem com reposição (*Bootstrap*).

Para o primeiro caso, lembrando do conceito visto em aula, temos (para um IC com 95% de confiança):

$$\begin{align}
0.95 = P(-z \le Z \le z)=P \left(-1.96 \le \frac {\bar X-\mu}{\sigma/\sqrt{n}} \le 1.96 \right) = P \left( \bar X - 1.96 \frac \sigma {\sqrt{n}} \le \mu \le \bar X + 1.96 \frac \sigma {\sqrt{n}}\right).
\end{align}$$

Vamos colocar na prática!

## Exemplo Inicial

Vamos começar construindo um intervalo de confiança pra a média de uma distribuição Normal (Gaussiana) com média $\mu = 0$ e variância $\sigma² = 1$.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as ss
plt.rcParams["figure.figsize"] = (10,8)

def confidence_interval(loc, scale, alpha=0.99):
  """Essa função cria uma distribuição Normal com base nos parâmetros fornecidos e constrói um intervalo de confiança para sua média.

  Parameters:
  -----------
  loc (float): Média da distribuição desejada.
  scale (float): Desvio padrão da distribuição desejada.
  alpha (float): Nível de confiança do intervalo. Deve ser um valor entre 0 e 1. Default = 0.99, o que corresponde a 99% de confiança.

  Returns:
  -----------
  X (array): Dados gerados segundo uma distribuição Normal(0,1).
  LI (float): Limite inferior do intervalo calculado.
  LS (float): Limite superior do intervalo calculado.

  """

  # Gerando amostra de uma Normal(0,1) de tamanho N
  N = 1000
  X = np.random.normal(loc=loc, scale=scale, size=N)

  # Criando plot da amostra gerada
  plt.xlabel('Valores', fontsize=12)
  plt.ylabel('Frequências', fontsize=12)
  plt.title('Histograma de amostra de uma Distribuição Normal(0,1)', fontsize=16)

  plt.hist(X, color='#A3333D', alpha=0.9, rwidth=0.85, bins=15)
  plt.show()

  # Calculando intervalo de  95% de confiança para a média manualmente
  LI = X.mean() - 1.96 * (X.std(ddof=1) / np.sqrt(N)) # LI = limite inferior
  LS = X.mean() + 1.96 * (X.std(ddof=1) / np.sqrt(N)) # LS = limite superior
  print("INTERVALO DE CONFIANCA (manual) = [{:.4f}, {:.4f}]".format(LI, LS))

  # Utilizando o valor da confiança como base, utilizamos o pacote scipy.stats
  LI = X.mean() - ss.norm.ppf(alpha+(1-alpha)/2).round(2) * (X.std(ddof=1) / np.sqrt(N))
  LS = X.mean() + ss.norm.ppf(alpha+(1-alpha)/2).round(2) * (X.std(ddof=1) / np.sqrt(N))

  # Printando intervalo de confiança
  print("INTERVALO DE CONFIANCA (com scipy)= [{:.4f}, {:.4f}]".format(LI, LS))

  return X, LI, LS

X, LI, LS = confidence_interval(loc=0, scale=1, alpha=0.95)
```

Podemos afirmar que, se pudermos repetir muitas vezes o experimento e coletarmos os dados, aproximadamente em 95% das vezes a média populacional estará no intervalo encontrado.

**Algumas observações interessantes. Note que:**
- A cada vez que executamos o código acima, tanto os intervalos como o histograma dos dados são diferentes. Estamos realizando uma amostra de uma distribuição.
- A medida que o tamanho da amostra (N) cresce, o tamanho do intervalo - para um mesmo nível de confiança - cai. Isso ocorre pois com mais dados temos uma maior certeza de que os valores encontrados de fato representam a população de interesse.
- Os valores dos intervalos de confiança (manual e scipy) só coincidem quando *alpha=0.95*. Essa é a vantagem de se utilizar o pacote. Para valores diferentes de 0.95, deve-se consultar o valor correspondente na distribuição Z.

## Dados ENEM 2015.

Nos exercícios dessa seção vamos trabalhar com os dados do [ENEM 2015](https://raw.githubusercontent.com/pedroharaujo/ICD_Docencia/master/enem2015.csv).

Importando pandas e carregando dados.

```python
import pandas as pd

url = 'https://raw.githubusercontent.com/pedroharaujo/ICD_Docencia/master/enem2015.csv'
data = pd.read_csv(url)
```

Primeiras visualizações do banco de dados do ENEM 2015.

```python
data.head()
```

      DEPENDENCIA_ADMINISTRATIVA
      NOTA_MEDIA_ESCOLA
      TAXA_DE_APROVACAO
      TAXA_DE_PARTICIPACAO

      0
      Federal
      659.258362
      92.2
      96.894409

      1
      Federal
      637.646728
      86.5
      98.717949

      2
      Estadual
      556.710632
      82.5
      74.603180

      3
      Estadual
      542.865967
      70.2
      54.651165

      4
      Estadual
      557.379150
      67.0
      64.864861

```python
data.describe()
```

      NOTA_MEDIA_ESCOLA
      TAXA_DE_APROVACAO
      TAXA_DE_PARTICIPACAO

      count
      1685.000000
      1683.000000
      1685.000000

      mean
      575.391745
      86.044444
      75.263669

      std
      40.796223
      10.334011
      16.886193

      min
      487.007141
      47.700000
      50.000000

      25%
      544.937500
      79.500000
      59.615387

      50%
      561.520752
      87.800000
      72.916672

      75%
      607.530579
      94.550000
      92.857140

      max
      713.629028
      100.000000
      100.000000

## Exercício 01

Altere a função a baixo para retornar o intervalo de confiança para a média da variável 'NOTA_MEDIA_ESCOLA' para escolas com 'DEPENDENCIA_ADMINISTRATIVA' == Estadual.

Nível de confiança: 99%

**Lembrete**: será necessário aplicar os conhecimentos de pandas do módulo anterior para filtrar o DataFrame selecionando apenas os casos de interesse.

```python
def CI(array):
  # Retorne uma tupla com os limites inferior e superior do intervalo de confiança, respectivamente
  # return (LI, LS)
  return None
```

Carregando os módulos de testes!

```python
from numpy.testing import assert_almost_equal
from numpy.testing import assert_equal

from numpy.testing import assert_array_almost_equal
from numpy.testing import assert_array_equal
```

Nosso teste

```python
(LI, LS) = CI(data)

assert_equal(548.13, LI.round(2))
assert_equal(551.09, LS.round(2))
```

## Exercício 02 (Sem correção automática)

Plote uma CDF da coluna 'TAXA_DE_APROVACAO'.

**Use o statsmodels e crie um objeto `ecdf = ECDF(...)`.**

Esta tarefa não tem correção automática, use o gráfico abaixo para saber se acertou ou não.

```python

# codigo para importar a função ECDF
from statsmodels.distributions.empirical_distribution import ECDF
```

    /usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

```python
# YOUR CODE HERE
```

## Bootstrap

Quando falamos em bootstrap deve-se ter em mente que estamos falando de amostragem com reposição.

De maneira simplista, utilizamos bootstrap quando queremos construir um intervalo de confiança para uma variável e dispomos de poucos dados. Ao realizarmos várias sub-amostras **com reposição**, a lei dos grandes números nos garante que estamos aproximando a população de interesse.

Note o destaque para o termo com reposição. É CRUCIAL que as sub-amostras sejam feitas com reposição. Só assim garantimos a aleatoriedade!

Veja o exemplo abaixo.

```python
col = 'TAXA_DE_PARTICIPACAO'
n_sub = 10000 #numero de sub-amostras
size = len(data) #tamanho do dataframe
values = np.zeros(n_sub)

def bootstrap(n_sub, size, col):
  for i in range(n_sub):
    # replace=TRUE garanta amostras com reposição
    # random_state=i garante replicabilidade do experimento
    sample = data.sample(size, replace=True, random_state=i)

    # Lembre que podemos utilizar mediana, média ou qualquer outra estatística agregada
    # values[i] = sample[col].median()
    values[i] = sample[col].mean()

  # Gerando valores inferior e superior para um nível de confiança de 95%
  LI = np.percentile(values, 2.5)
  LS = np.percentile(values, 97.5)
  return values, LI, LS

values, LI, LS = bootstrap(n_sub, size, col)
print('Intervalo de Confianca: [{}, {}]'.format(LI.round(4), LS.round(4)))
```

    Intervalo de Confiança: [74.4523, 76.0718]

## Exercício 03

Realizando um groupy pela coluna 'DEPENDENCIA_ADMINISTRATIVA' conseguimos observar para quais casos vale a pena utilizarmos bootstrap.

```python
data.groupby('DEPENDENCIA_ADMINISTRATIVA').count()
```

      NOTA_MEDIA_ESCOLA
      TAXA_DE_APROVACAO
      TAXA_DE_PARTICIPACAO

      DEPENDENCIA_ADMINISTRATIVA

      Estadual
      1054
      1054
      1054

      Federal
      43
      42
      43

      Municipal
      22
      22
      22

      Privada
      566
      565
      566

**A)** Na função abaixo, retorne o número da opção que indica para quais 'DEPENDENCIAS_ADMINISTRATIVAS' é aconselhado utilizar Bootstrap para construção do intervalo de confiança:

- 1) Estadual e Federal.
- 2) Estadual e Municipal.
- 3) Estadual e Privada.
- 4) Federal e Municipal.
- 5) Federal e Privada.
- 6) Municipal e Privada.

```python
def resposta():
  # Escolha uma das opções acima
  # return 1 ou 2 ou 3 ou 4 ou 5 ou 6
  # Escreva sua resposta abaixo:
  # YOUR CODE HERE
  return None
```

```python
assert_equal(4, resposta())
```

**B)** Construa um intervalo de confiança via Bootstrap para a média da variável 'NOTA_MEDIA_ESCOLA' para escolas de 'DEPENDENCIA_ADMINISTRATIVA' **Federal**. Você deve utilizar 5000 amostras e nível de confiança de 90%.

*Nota*: você deve utilizar o argumento random_state=i na função data.sample, como no exemplo inicial da seção de Bootstrap.

```python
def bootstrap_mean(n_sub, alpha):
  # Retorne um vetor com os valores obtidos.
  # Retorne tambem uma tupla com os limites inferiores e superiores do IC construído.
  # Exemplo:
  # return values, (LI, LS)
  return None
```

```python
values, (LI, LS) = bootstrap_mean(n_sub=5000, alpha=0.9)
assert_equal(71.0697, LI)
assert_equal(79.4893, LS)
```

## Exercício 4

Altere a função abaixo para que retorne a 'DEPENDENCIA_ADMINISTRATIVA' (Federal, Estadual, Municipal ou Privada) cujo intervalo de confiança para *mediana* via *bootstrap* para a variável 'TAXA_DE_PARTICIPACAO' apresente maior amplitude (LS-LI), e qual esse valor.

Utilize:
- 95% como nível de confiança.
- 5000 como número de sub-amostras.

```python
def ci_amplitude():
  # Sua função deve retorar uma tupla como no exemplo
  # exemplo: (dependencia_administrativa, amplitude_do_ic)
  return None
```

```python
(dep, amp) = ci_amplitude()
assert_equal(dep, 'Federal')
assert_equal(amp, 2.1116)
```


# Realization of Non-Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Fast Convolution

The straightforward convolution of two finite-length signals $x[k]$ and $h[k]$ is a numerically complex task. This has led to the development of various techniques with considerably lower complexity. The basic concept of the *fast convolution* is to exploit the correspondence between the convolution and the scalar multiplication in the frequency domain.

### Convolution of Finite-Length Signals

The convolution of a causal signal $x_L[k]$ of length $L$ with a causal impulse response $h_N[k]$ of length $N$ is given as

\begin{equation}
y[k] = x_L[k] * h_N[k] = \sum_{\kappa = 0}^{L-1} x_L[\kappa] \; h_N[k - \kappa] = \sum_{\kappa = 0}^{N-1} h_N[\kappa] \; x_L[k - \kappa]
\end{equation}

where $x_L[k] = 0$ for $k<0 \wedge k \geq L$ and $h_N[k] = 0$ for $k<0 \wedge k \geq N$. The resulting signal $y[k]$ is of finite length $M = N+L-1$. The computation of $y[k]$ for $k=0,1, \dots, M-1$ requires $M \cdot N$ multiplications and $M \cdot (N-1)$ additions. The computational complexity of the convolution is consequently [in the order of](https://en.wikipedia.org/wiki/Big_O_notation) $\mathcal{O}(M \cdot N)$. Discrete-time Fourier transformation (DTFT) of above relation yields

\begin{equation}
Y(e^{j \Omega}) = X_L(e^{j \Omega}) \cdot H_N(e^{j \Omega})
\end{equation}

Discarding the effort of transformation, the computationally complex convolution is replaced by a scalar multiplication with respect to the frequency $\Omega$. However, $\Omega$ is a continuous frequency variable which limits the numerical evaluation of this scalar multiplication. In practice, the DTFT is replaced by the discrete Fourier transformation (DFT). Two aspects have to be considered before a straightforward application of the DFT

1. The DFTs $X_L[\mu]$ and $H_N[\mu]$ are of length $L$ and $N$ respectively and cannot be multiplied straightforwardly

2. For $N = L$, the multiplication of the two spectra $X_L[\mu]$ and $H_L[\mu]$ would result in the [periodic/circular convolution](https://en.wikipedia.org/wiki/Circular_convolution) $x_L[k] \circledast_L h_L[k]$ due to the periodicity of the DFT. Since we aim at realizing the linear convolution $x_L[k] * h_N[k]$ with the DFT, special care has to be taken to avoid cyclic effects.

### Linear Convolution by Periodic Convolution

The periodic convolution of the two signals $x_L[k]$ and $h_N[k]$ is defined as

\begin{equation}
x_L[k] \circledast_M h_N[k] = \sum_{\kappa=0}^{M-1} \tilde{x}_M[k - \kappa] \; \tilde{h}_M[\kappa]
\end{equation}

where the periodic continuations $\tilde{x}_M[k]$ of $x_L[k]$ and $\tilde{h}_M[k]$ of $h_N[k]$ with period $M$ are given as

\begin{align}
\tilde{x}_M[k] &= \sum_{m = -\infty}^{\infty} x_L[m \cdot M + k] \\
\tilde{h}_M[k] &= \sum_{m = -\infty}^{\infty} h_N[m \cdot M + k]
\end{align}

The result of the circular convolution has a periodicity of $M$.

To compute the linear convolution by the periodic convolution one has to take care that the result of the linear convolution fits into one period of the periodic convolution. Hence, the periodicity has to be chosen as $M \geq N+L-1$. This can be achieved by zero-padding of $x_L[k]$ and $h_N[k]$ to a total length of $M$

\begin{align}
x_M[k] &= \begin{cases}
x_L[k] & \mathrm{for} \; k=0, 1, \dots, L-1 \\
0 & \mathrm{for} \; k=L, L+1, \dots, M-1
\end{cases}
\\
h_M[k] &= \begin{cases}
h_N[k] & \mathrm{for} \; k=0, 1, \dots, N-1 \\
0 & \mathrm{for} \; k=N, N+1, \dots, M-1
\end{cases}
\end{align}

This results in the desired equality of linear and periodic convolution

\begin{equation}
x_L[k] * h_N[k] = x_M[k] \circledast_M h_M[k]
\end{equation}

for $k = 0,1,\dots, M-1$ with $M = N+L-1$.

#### Example - Linear by periodic convolution

The following example computes the linear, periodic and linear by periodic convolution of a rectangular signal $x[k] = \text{rect}_L[k]$ of length $L$ with a triangular signal $h[k] = \Lambda_N[k]$ of length $N$.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

L = 32  # length of signal x[k]
N = 16  # length of signal h[k]
M = 16  # periodicity of periodic convolution

def periodic_summation(x, N):
    "Zero-padding to length N or periodic summation with period N."
    M = len(x)
    rows = int(np.ceil(M/N))

    if (M < int(N*rows)):
        x = np.pad(x, (0, int(N*rows-M)), 'constant')

    x = np.reshape(x, (rows, N))

    return np.sum(x, axis=0)

def periodic_convolve(x, y, P):
    "Periodic convolution of two signals x and y with period P."
    x = periodic_summation(x, P)
    h = periodic_summation(y, P)

    return np.array([np.dot(np.roll(x[::-1], k+1), h) for k in range(P)], float)

# generate signals
x = np.ones(L)
h = sig.triang(N)

# linear convolution
y1 = np.convolve(x, h, 'full')
# periodic convolution
y2 = periodic_convolve(x, h, M)
# linear convolution via periodic convolution
xp = np.append(x, np.zeros(N-1))
hp = np.append(h, np.zeros(L-1))
y3 = periodic_convolve(xp, hp, L+N-1)

def plot_signal(x):
    '''Plots the signals in stem plot.'''
    plt.figure(figsize=(10, 3))
    plt.stem(x)
    plt.xlabel(r'$k$')
    plt.ylabel(r'$y[k]$')
    plt.axis([0, N+L, 0, 1.1*x.max()])

# plot results
plot_signal(x)
plt.title('Signal $x[k]$')
plot_signal(y1)
plt.title('Linear convolution')
plot_signal(y2)
plt.title('Periodic convolution with period M = %d' % M)
plot_signal(y3)
plt.title('Linear convolution by periodic convolution');
```

**Exercise**

* Change the lengths `L`, `N` and `M` and check how the results for the different convolutions change

### The Fast Convolution

Using the above derived equality of the linear and periodic convolution one can express the linear convolution $y[k] = x_L[k] * h_N[k]$ by the DFT as

\begin{equation}
y[k] = \text{IDFT}_M \{ \; \text{DFT}_M\{ x_M[k] \} \cdot \text{DFT}_M\{ h_M[k] \} \; \}
\end{equation}

This operation requires three DFTs of length $M$ and $M$ complex multiplications. On first sight this does not seem to be an improvement, since one DFT/IDFT requires $M^2$ complex multiplications and $M \cdot (M-1)$ complex additions. The overall numerical complexity is hence in the order of $\mathcal{O}(M^2)$. The DFT can be realized efficiently by the [fast Fourier transformation](https://en.wikipedia.org/wiki/Fast_Fourier_transform) (FFT), which lowers the computational complexity to $\mathcal{O}(M \log_2 M)$. The resulting algorithm is known as *fast convolution* due to its computational efficiency.

The fast convolution algorithm is composed of the following steps

1. Zero-padding of the two input signals $x_L[k]$ and $h_N[k]$ to at least a total length of $M \geq N+L-1$

2. Computation of the DFTs $X[\mu]$ and $H[\mu]$ using a FFT of length $M$

3. Multiplication of the spectra $Y[\mu] = X[\mu] \cdot H[\mu]$

4. Inverse DFT of $Y[\mu]$ using an inverse FFT of length $M$

The overall complexity depends on the particular implementation of the FFT. Many FFTs are most efficient for lengths which are a power of two. It therefore can make sense, in terms of computational complexity, to choose $M$ as a power of two instead of the shortest possible length $N+L-1$. For real valued signals $x[k] \in \mathbb{R}$ and $h[k] \in \mathbb{R}$ the computational complexity can be reduced significantly by using a real valued FFT.

#### Example - Fast convolution

The implementation of the fast convolution algorithm is straightforward. Most implementations of the FFT include zero-padding to a given length $M$, e.g in `numpy` by `numpy.fft.fft(x, M)`. In the following example an implementation of the fast convolution is shown. For illustration the convolution of a rectangular signal $x[k] = \text{rect}_L[k]$ of length $L$ with a triangular signal $h[k] = \Lambda_N[k]$ of length $N$ is considered.

```python
L = 16  # length of signal x[k]
N = 16  # length of signal h[k]
M = N+L-1

# generate signals
x = np.ones(L)
h = sig.triang(N)

# linear convolution
y1 = np.convolve(x, h, 'full')
# fast convolution
y2 = np.fft.ifft(np.fft.fft(x, M) * np.fft.fft(h, M))

plt.figure(figsize=(10, 6))
plt.subplot(211)
plt.stem(y1)
plt.xlabel(r'$k$')
plt.ylabel(r'$y[k] = x_L[k] * h_N[k]$')
plt.title('Result of linear convolution')

plt.subplot(212)
plt.stem(y1)
plt.xlabel(r'$k$')
plt.ylabel(r'$y[k] = x_L[k] * h_N[k]$')
plt.title('Result of fast convolution')
plt.tight_layout()
```

#### Example - Numerical complexity

It was already argued that the numerical complexity of the fast convolution is considerably lower due to the usage of the FFT. The gain with respect to the convolution is evaluated in the following. In order to measure the execution times for both algorithms the `timeit` module is used. The algorithms are evaluated for the convolution of two random signals $x_L[k]$ and $h_N[k]$ of length $L=N=2^n$ for $n=0, 1, \dots, 16$.

```python
import timeit

n = np.arange(17)  # lengths = 2**n to evaluate
reps = 50  # number of repetitions for timeit

gain = np.zeros(len(n))
for N in n:
    length = 2**N
    # setup environment for timeit
    tsetup = 'import numpy as np; from numpy.fft import rfft, irfft; \
            x=np.random.randn(%d); h=np.random.randn(%d)' % (length, length)
    # direct convolution
    tc = timeit.timeit('np.convolve(x, x, mode="full")', setup=tsetup, number=reps)
    # fast convolution
    tf = timeit.timeit('irfft(rfft(x, %d) * rfft(h, %d))' % (2*length, 2*length), setup=tsetup, number=reps)
    # speedup by using the fast convolution
    gain[N] = tc/tf

# show the results
plt.figure(figsize = (15, 10))
plt.barh(n, gain, log=True)
plt.plot([1, 1], [-1, n[-1]+1], 'r-')
plt.yticks(n, 2**n)
plt.xlabel('Gain of fast convolution')
plt.ylabel('Length of signals')
plt.title('Comparison between direct/fast convolution')
plt.grid()
```

**Exercise**

* When is the fast convolution more efficient/faster than a direct convolution?
* Why is it slower below a given signal length?
* Is the trend of the gain as expected by the numerical complexity of the FFT?

Solution: The gain in execution time of a fast convolution over a direct implementation of the the convolution for different signal lengths depends heavily on the particular implementation and hardware used. The fast convolution in this example is faster for two signals having a length equal or larger than 1024 samples. Discarding the outliers and short lengths, the overall trend in the gain is approximately logarithmic as predicted above.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# 2/ Intro to linear algebra

```python
# setup SymPy
from sympy import *
x, y, z, t = symbols('x y z t')
init_printing()

# setup plotting
%matplotlib notebook
import matplotlib.pyplot as mpl
from util.plot_helpers import plot_vec, plot_vecs, autoscale_arrows
from util.plot_helpers import plot_point, plot_line, plot_plane
from util.plot_helpers import plot_augmat

# define alias Vector for Matrix objects
Vector = Matrix
```

```python

```

## Definitions

```python

```

```python

```

### E2.1

```python

```

```python

```

### E2.2

```python

```

```python

```

### E2.3

```python

```

```python

```

## Vector operations

```python

```

```python

```

### E2.4

```python

```

```python

```

### E2.5

```python

```

```python

```

### E2.6

```python

```

```python

```

## Matrix operations

```python

```

```python

```

### E2.7

```python

```

```python

```

### E2.8

```python

```

```python

```

### E2.9

```python

```

```python

```

## Linearity

```python

```

```python

```

### E2.10

```python

```

```python

```


```python
import cirq
```

The message (quantum-state) can be sent by Alice to Bob just using two bits of classical information. During this process, the message state is "destroyed" and later reconstructed by Bob. Let's see how this works in the following

Reading: Ch. 5, PhysRevLett.70.1895 (free to read)

Alice and Bob will need to share a Bell state; this Bell state can be created with the following circuit.

```python
def make_bell_state(q0, q1, index=(0, 0)):
    if index[0]:
        yield cirq.X(q0)
    if index[1]:
        yield cirq.X(q1)
    yield cirq.H(q0)
    yield cirq.CNOT(q0, q1)
```

```python
q0 = cirq.NamedQubit('Bob')
q1 = cirq.NamedQubit('Alice')
s=cirq.Simulator()
```

Let's see if it all works
We want to output the Bell basis
\begin{equation}
|\beta_{00}\rangle= \frac{1}{\sqrt{2}}\left(|00\rangle +|11\rangle\right)\\
|\beta_{01}\rangle= \frac{1}{\sqrt{2}}\left(|01\rangle +|10\rangle\right)\\
|\beta_{10}\rangle= \frac{1}{\sqrt{2}}\left(|00\rangle -|11\rangle\right)\\
|\beta_{11}\rangle= \frac{1}{\sqrt{2}}\left(|01\rangle -|10\rangle\right)
\end{equation}
For example: $C|00\rangle=  cX (H\otimes I) |00\rangle=  cX \frac{1}{\sqrt{2}}\left(|0\rangle + |1\rangle \right)|0\rangle  =  cX \frac{1}{\sqrt{2}}\left(|00\rangle + |10\rangle \right) = |\beta_{00} \rangle $

```python
for index in [(0,0),(0,1),(1,0),(1,1)]:
    bell_c= cirq.Circuit()
    bell_c.append(make_bell_state(q0,q1,index))
    results=s.simulate(bell_c)
    print(results)
print('Circuit for (1,1)')
print(bell_c)
```

    measurements: (no measurements)
    output vector: 0.707|00⟩ + 0.707|11⟩
    measurements: (no measurements)
    output vector: 0.707|01⟩ + 0.707|10⟩
    measurements: (no measurements)
    output vector: 0.707|00⟩ - 0.707|11⟩
    measurements: (no measurements)
    output vector: -0.707|01⟩ + 0.707|10⟩
    Circuit for (1,1)
    Alice: ─────X───────X───
                        │
    message: ───X───H───@───

Next we create the a circuit to initialise some message

```python
def message(q0,pow1,pow2):
    ##applies power of X and Y gate to creat some state
    yield [cirq.X(q0) ** pow1, cirq.Y(q0) ** pow2]
```

We are now in the position to create the Teleportation circuit, where the main idea is that one can rewrite the state of the system in the following form (verify this)
\begin{equation}
|m\rangle|\beta_{00}\rangle= \frac{1}{2}\left(|\beta_{00}\rangle|m\rangle + |\beta_{01}\rangle X|m\rangle +|\beta_{10}\rangle Z|m\rangle+ |\beta_{11}\rangle XZ|m\rangle\right)
\end{equation}
Now recalling that $X^2 = 1$ and $Z^2=1$, the circuit's architecture should become apparent. Namely, depending on the two bits, Bob receives, he will apply $X$,$Z$ or $ZX$ to recover the message $|m\rangle$

```python
def qt(pow1, pow2, p = True):
    q0, q1, q2 =cirq.LineQubit.range(3)
    c = cirq.Circuit()
    ###create initial Bell state between Bob and Alice
    c.append(make_bell_state(q1,q2))
    ##create some message state
    c.append(message(q0,pow1,pow2))
    ###Bell measurement
    c.append([cirq.CNOT(q0, q1), cirq.H(q0)])
    c.append(cirq.measure(q0, q1))
    ###Bob uses the classical bits to reconstruct the message
    c.append([cirq.CNOT(q1,q2),cirq.CZ(q0,q2)])
    if p:
        print('Teleportation Circuit')
        print(c)
    return c
```

```python
def test(pow1,pow2,p=True):
    ##teleport message
    s=cirq.Simulator()
    tele_c = qt(pow1,pow2,p)
    results=s.simulate(tele_c)
    teleported_message = cirq.bloch_vector_from_state_vector(results.final_state_vector, 2)
    ##check acutal message
    q0 = cirq.NamedQubit('message')
    m = cirq.Circuit()
    m.append(message(q0,pow1,pow2))
    results=s.simulate(m)
    acutal_message = cirq.bloch_vector_from_state_vector(results.final_state_vector, 0)
    return teleported_message, acutal_message
```

```python
test(0.1,0.3)
```

    Teleportation Circuit
    0: ───X^0.1───Y^0.3───@───H───M───────@───
                          │       │       │
    1: ───H───────@───────X───────M───@───┼───
                  │                   │   │
    2: ───────────X───────────────────X───@───

    (array([ 0.7694209 , -0.309017  ,  0.55901694], dtype=float32),
     array([ 0.7694209 , -0.309017  ,  0.55901694], dtype=float32))

Show that it actually works

```python
import random
l = [(random.randrange(0, 1), random.randrange(0, 1)) for i in range(20)]
```

```python
for i in l:
    t,m = test(*i, False)
    assert t.all() == m.all()
```

```python

```


# Bloch Equation Solvers
This notebook will investigate methods to solve the Bloch-Equations. Recall the Bloch Equations are:

$$\frac{\partial M}{\partial t} = \gamma M \times B $$

where $\gamma$ is the gyromagnetic ratio, $M$ is the magnetization, and $B$ is the magnetic field. It is most common to solve everything in the rotating frame. Where the entire corrdinate system rotates at the Larmor frequency ($\gamma B_0$), such that we only care about the frequency difference from that. There are two types of solvers:

*   Standard differential equation solvers. These solvers use Runge-Kutta or some othe solver to integrate the changes. These solvers are great for RF pulses or short blocks of time with large changes.
*   Nutation solvers. These solvers break the problem into periods of rotation and periods of relaxation. This enables much more computationally efficient solutions.

This notebooks will show some ways to set this up, starting with the standard differential equation solver.

```
# This is comment, Pyhton will ignore this line

# Import libraries (load libraries which provide some functions)
import numpy as np # array library
import matplotlib.pyplot as plt # for plotting
import math
import cmath

# Hit the play button to run this cell
```

# 1. Standard Solvers

## Bloch Equation Setup
In these solvers, we will use the matrix form of the Bloch equations

\begin{align}
\frac{\partial}{\partial t} \begin{bmatrix}
  M_x \\ M_y \\ M_z
  \end{bmatrix} = \begin{bmatrix}
  -1/T2 & \gamma B_z & -\gamma B_y  \\
   -\gamma B_z & -1/T2 & \gamma B_x \\
   \gamma B_y & -\gamma B_x & -1/T1
  \end{bmatrix} \begin{bmatrix}M_x \\ M_y \\ M_z \end{bmatrix} +
  \begin{bmatrix}
  0 \\ 0 \\ M_0/T1
  \end{bmatrix}
\end{align}

This is effectively a rotation about each axis at the rate $\gamma B$, a decay of $M_x$ and $M_y$ at a rate $M_{xy}/T2$, and a recovery of $M_z$ at a rate $(M_0-M_z)/T1$. $M_0$ is assumed to be 1 for simplicity.

## Solver
The solver will integrate the above equation. This could be done with a simple stepwise solver:
\begin{align}
M_x(t+\Delta t) = M_x(t) + \Delta \frac{\partial M_x(t)}{\partial t}
\end{align}

Where $\Delta$ is a step size and the temporal resolution of the discrete array. However, this method is not fully stable, especially since the derivative of $M_x$ depends on itself. You can very easily find odd conditions, such as the magnetization growing. Instead, more complex methods are needed, which allow for large steps sizes. This is a general issue with numerical solvers and not unique to MRI.
The below code is a standard Runge-Kutta 4 (RK4) solver for a differential equation. It's essential numerical integration but is much more stable. It achives this stability by evaluating the derivative at the next steps. For more info see [Wikipedia](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods)

```
from scipy import interpolate

def bloch_solver( B, time, freq, T1=2000, T2=2000, M0=[0,0,1], GAM=42.58e6*2*math.pi):
  # This is simple Rk4 solution to the Bloch Equations.
  #
  # Inputs:
  #   B(array)    -- Magentic Field [N x 3] (T)
  #   time(array) -- Time of each point in waveforms (s)
  #   T1          -- Longitudinal relaxation times (s)
  #   T2          -- Transverse relaxation times (s)
  #   Freq Offset -- Off center frequency in Hz
  #   M0          -- Initial state of magnetization (not equilibrium magnetization)
  # Outputs:
  #   MOutput -- Magnetization for each position in time

  # Convert frequency to rads/s
  act_freq = 2*math.pi*freq

  #Convert to roation rates (gamma*B)
  assert B.shape[1] == 3
  Bx =  GAM*B[:,0]
  By =  GAM*B[:,1]
  Bz =  GAM*B[:,2] + act_freq

  #Create a spline for interpolation
  spline_Bx = interpolate.splrep(time, Bx)
  spline_By = interpolate.splrep(time, By)
  spline_Bz = interpolate.splrep(time, Bz)

  #Initialize
  Mag = np.array(M0).reshape(3,1)

  # Output storage
  MOutput = np.zeros_like(B)

  #Runge-Kutta PDE Solution
  dt = time[2] - time[1]
  for count, t1 in enumerate(time):

    m1 = Mag

    bx = interpolate.splev(t1, spline_Bx)
    by = interpolate.splev(t1, spline_By)
    bz = interpolate.splev(t1, spline_Bz)
    k1 = np.array([[  -1/T2,   bz,  -by],
                  [  -bz   ,-1/T2, bx],
                  [   by   , -bx, -1/T1]])@ m1 + np.array([[0],[0],[1/T1]])

    t2 = t1 + dt/2
    bx = interpolate.splev(t2, spline_Bx)
    by = interpolate.splev(t2, spline_By)
    bz = interpolate.splev(t2, spline_Bz)
    m2 = Mag + k1*dt/2
    k2 = np.array([[  -1/T2,   bz,  -by],
                  [  -bz   ,-1/T2, bx],
                  [   by   ,  -bx, -1/T1]])@ m2 + np.array([[0],[0],[1/T1]])

    t3 = t1 + dt/2
    bx = interpolate.splev(t3, spline_Bx)
    by = interpolate.splev(t3, spline_By)
    bz = interpolate.splev(t3, spline_Bz)
    m3 = Mag + k2*dt/2
    k3 = np.array([[  -1/T2,   bz,  -by],
                  [  -bz   ,-1/T2, bx],
                  [   by   ,  -bx, -1/T1]])@m3 + np.array([[0],[0],[1/T1]])

    t4 = t1 + dt
    bx = interpolate.splev(t4, spline_Bx)
    by = interpolate.splev(t4, spline_By)
    bz = interpolate.splev(t4, spline_Bz)
    m4 = Mag + k3*dt
    k4 = np.array([[  -1/T2,   bz,  -by],
                  [  -bz   ,-1/T2, bx],
                  [   by   ,  -bx, -1/T1]])@m4 + np.array([[0],[0],[1/T1]])

    # Runge-Kutta averages the above terms
    Mag = Mag + dt/6*(k1 + 2*k2 + 2*k3 + k4);

    # Save to an array
    MOutput[count,0]= Mag[0]
    MOutput[count,1]= Mag[1]
    MOutput[count,2]= Mag[2]

  return MOutput
```

# Excitation Experiment
This is the most simple experiment with a single excitation pulse followed by recovery. We assume magnetization starts at $M_0 \equiv 1$. We aim to apply an RF pulse to rotate the magnetization into the transverse plane. The amount we rotate the magnetization is the flip angle ($\alpha$). This is equal to:
\begin{equation}
\alpha = \int_{0}^{T_{RF}} \gamma B_1(t) dt
\end{equation}
This assumes the RF pulse is applied along a single axis and more complex relationships do exist for some pulse we won't cover (e.g. adiabatic pulses). The below code will just define a pulse envelope and a phase of the pulse. The phase sets the direction on the $B_1$ field (e.g. along x, y, or mixture).

```
# Simulation settings
dt = 2e-6 # systems run around 2 us resolution in the rotating frame
Tmax = 30e-3 #total time to simulate
gamma_bar = 42.58e6
gamma = math.pi*2.0*gamma_bar

# RF Settings
Trf = 2e-3 # period of the pulse
flip = 90 #degrees
excite_phase = 0 # excite phase in degrees

# Define the time span
time = np.arange(-1e-3,Tmax,dt)

# define the RF pulse shape
RF = np.zeros_like(time, dtype=np.complex)
idx = (time<Trf) & (time>=0)
RF[idx] = 1 # Rect function
RF[idx] = np.exp( -16*(time[idx] - Trf/2)**2 / (Trf**2)) # Gaussian

# Now lets scale the RF pulse amplitude to be correct rotation for on-resonance spins
RF = RF*(flip/360) / np.sum(gamma_bar*RF*dt)

#Rotate to the phase of excitation
RF = RF*np.exp(2j*math.pi*excite_phase/360)

# The RF is complex
B = np.zeros( (len(time),3))
B[:,0] = np.real(RF)
B[:,1] = np.imag(RF)

# Plot
plt.figure(figsize=(8,8))
plt.title('Applied magnetic fields vs. time')
plt.plot(time,B[:,0],label='B_x')
plt.plot(time,B[:,1],label='B_y')
plt.plot(time,B[:,2],label='B_z')
plt.xlabel('Time [s]')
plt.ylabel('B1 [T]')
plt.legend()
plt.show()
```

# Simulate use the above define solvers

Some things to change:

*   Change T2 to some other values (typical range 1e-3 to 1), does the behavior match what you expect?
*   Change the frequency offset to other values. Why might the Larmor frequency and the RF frequency be slightly different?
*   Change T1 to some other values (typical range 100e-3 to 5), does the behavior match what you expect?

```
# B is defined above
# freq = offset frequency from assumed Larmor frequency
# time is defined above
# T1 is the longitudinal relaxation rate
# T2 is the transverse longitudunal rate
Mout = bloch_solver( B, time, freq=100, T1=2000, T2=20e-3)

# Plot
plt.figure(figsize=(8,8))
plt.plot(time,Mout[:,0],label=r'$M_x$')
plt.plot(time,Mout[:,1],label=r'$M_y$')
plt.plot(time,np.sqrt(Mout[:,1]**2+Mout[:,0]**2),'--',label=r'$M_{xy}$')
plt.plot(time,Mout[:,2],label=r'$M_z$')
plt.xlabel('Time [s]')
plt.ylabel('M [a.u.]')
plt.legend()
plt.show()
```

# 2. Nutation Solvers

## Problem Setup
Nutation solvers break the Bloch equations into events. Events would include RF exitations, times of free recovery, and magnetic field changes from gradients (more on this last one later). The premise of this is that we know the solutions for the Bloch equations analytically. Such solvers can also include traditional solvers for events with unknown equations. Some we know include:

* **Transverse magnetization decay wo/Frequency shift**
\begin{equation}
M_{xy}(t) = M_{xy}(t=0)e^{-t/T2}
\end{equation}

* **Longitudinal magnetization recovery**
\begin{equation}
M_{z}(t) = M_0 + ( M_{z}(t=0)-M_0)e^{-t/T1}
\end{equation}

* **RF pulse at exactly the Larmor frequency**
\begin{equation}
M_xy = R_z(-\theta)R_x(\alpha)R_z(\theta)M
\end{equation}
where $R_z$ is a rotation about $z$, $R_x$ is rotation about $x$, $\alpha$ is the flip angle, and $\theta$ is the phase of the RF pulse.

```
class Event:
  def __init__(self, excite_flip=0, excite_phase=0, recovery_time=0, spoil=False):
    self.excite_flip = excite_flip
    self.excite_phase = excite_phase
    self.recovery_time = recovery_time
    self.spoil = spoil

def bloch_nutation_solver( event_list, M0, T1, T2, freq ):

  #  Inputs:
  #    event_list -- Special structure with entries
  #             .excite_flip  flip angle of rotation
  #             .excite_phase phase of excite degrees
  #             .recovery_time time after excite to recover
  #             .spoil (if 'true' this set the Mxy to zero at the recovery)
  #    T1         -- Longitudinal relaxation times (s)
  #    T2         -- Transverse relaxation times (s)
  #    Freq Offset-- Off center frequency in Hz
  #    M0         -- Initial state of magnetization (not equilibrium magnetization)
  #  Outputs:
  #    time -- Magnetization for each position in time
  #    BOutput -- Magnetic field for each position in time (interpolated)

  # Initialize
  count = 0;
  time = [0,];
  Mout = [M0,]

  M=M0;

  # Go through the event_list
  for event in event_list:

    theta = event.excite_phase * math.pi / 180
    alpha =  event.excite_flip * math.pi / 180
    T = event.recovery_time
    spoil = event.spoil

    # Excite
    Rz = np.array([[math.cos(theta), math.sin(theta), 0],
                    [-math.sin(theta), math.cos(theta), 0],
                    [0,  0, 1]])
    Rx = np.array([[1,0, 0],
                    [0, math.cos(alpha), math.sin(alpha)],
                    [0, -math.sin(alpha), math.cos(alpha)]])

    M = np.linalg.inv(Rz)@Rx@Rz@M

    # Relaxation (Transverse)
    if spoil:
        Mxy = 0
    else:
        Mxy = M[0] + 1j*M[1]
        Mxy = Mxy*cmath.exp( 2j*math.pi*freq*T)*math.exp(-T/T2)

    # Relaxation (Longitudinal)
    Mz = M[2]
    Mz = 1 + (Mz - 1)*math.exp(-T/T1);

    # Put back into [Mx; My; Mz] vector
    M = np.array([[Mxy[0].real], [Mxy[0].imag], [Mz[0]]])

    # Store for output
    time.append(time[-1]+T)
    Mout.append(M)

  Mout = np.array(Mout)
  return time, Mout

```

# Example

Here we simulate a situation in which we apply two RF pulses seperated by a gap:

$90 ^{\circ} $ - Delay - $180 ^{\circ}$ - Free recovery

Some experiments to consider:
*   Change the flip angles.
*   Change the delays (modify the range(100) to a desired delay in ms
*   Change T1 and T2

```
# Blank list of events
event_list = []

# Excite 90 degree
event_list.append( Event(excite_flip=90, excite_phase=0))

#Recover (100ms,could be one event but plotting possible here)
for pos in range(100):
    event_list.append( Event(recovery_time=1e-3))

# Excite 180 degree
event_list.append( Event(excite_flip=180, excite_phase=90))

#Recover (500ms,could be one event but plotting possible here)
for pos in range(500):
    event_list.append( Event(recovery_time=1e-3))

# Simulate
M0 = np.array([[0],[0],[1.0]])
T1 = 1
T2 = 1
freq = 3
time,Mout = bloch_nutation_solver( event_list, M0, T1, T2, freq );

plt.figure(figsize=(8,8))
plt.plot(time,Mout[:,0], label=r'$M_x$')
plt.plot(time,Mout[:,1], label=r'$M_y$')
plt.plot(time,np.sqrt(Mout[:,1]**2+Mout[:,0]**2), '--', label=r'$M_y$')
plt.plot(time,Mout[:,2], label=r'$M_z$')
plt.legend()
plt.show()

```


# Comparison of Gaussian and Lorentz profiles

Figure 2.4 from Chapter 2 of *Interstellar and Intergalactic Medium* by Ryden & Pogge, 2021,
Cambridge University Press.

Example of Gaussian and Lorentz profile with the same width and normalized to the same area.

Profiles are computed in situ.

```python
%matplotlib inline

import math
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator, LogLocator, NullFormatter

import warnings
warnings.filterwarnings('ignore',category=UserWarning, append=True)
```

## Standard Plot Format

Setup the standard plotting format and make the plot. Fonts and resolution adopted follow CUP style.

```python
figName = 'Fig2_4'

# graphic aspect ratio = width/height

aspect = 4.0/3.0 # 4:3

# Text width in inches - don't change, this is defined by the print layout

textWidth = 6.0 # inches

# output format and resolution

figFmt = 'png'
dpi = 600

# Graphic dimensions

plotWidth = dpi*textWidth
plotHeight = plotWidth/aspect
axisFontSize = 10
labelFontSize = 6
lwidth = 0.5
axisPad = 5
wInches = textWidth
hInches = wInches/aspect

# Plot filename

plotFile = f'{figName}.{figFmt}'

# LaTeX is used throughout for markup of symbols, Times-Roman serif font

plt.rc('text', usetex=True)
plt.rc('font', **{'family':'serif','serif':['Times-Roman'],'weight':'bold','size':'16'})

# Font and line weight defaults for axes

matplotlib.rc('axes',linewidth=lwidth)
matplotlib.rcParams.update({'font.size':axisFontSize})

# axis and label padding

plt.rcParams['xtick.major.pad'] = f'{axisPad}'
plt.rcParams['ytick.major.pad'] = f'{axisPad}'
plt.rcParams['axes.labelpad'] = f'{axisPad}'
```

## Functions for Gaussian and Lorentz Profiles

Gaussian:
\begin{equation}
  G(x,\sigma) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{x^2}{2\sigma^2}\right)
\end{equation}

Lorentz:
\begin{equation}
  L(x,\gamma) = \frac{4\gamma}{16\pi^2x^2 + \gamma^2}
\end{equation}

The two profiles will be plotted normalized to the same area given the same FWHM:
 * Gaussian: $\sigma = FWHM/ 2\sqrt{2\ln 2}$
 * Lorentz: $\gamma = 2\pi FWHM$

This is done in the code for G(x,FWHM) in the following cell.

```python
# Compute a Gaussian profile

def G(x,fwhm):
    sigma = fwhm/(2.0*math.sqrt(2.0*math.log(2.0)))
    xarg = x/sigma
    G = np.exp(-0.5*xarg*xarg)/np.sqrt(np.pi)*sigma
    return G

# Compute a Lorentz Profile

def L(x,fwhm):
    gamma = 2.0*math.pi*fwhm
    xarg = 4*math.pi*x
    L = 4*gamma/(xarg*xarg + gamma*gamma)
    return L

# Calculation Limits

nuMin = -7.0
nuMax = 7.0
nuStep = 0.01
fwhm = 2.0
yMin = -0.01
yMax = 0.5

# Do the calculation

numPts = int((nuMax-nuMin)/nuStep)+1
nu = np.linspace(nuMin,nuMax,num=numPts)
gauss = G(nu,fwhm)
lorentz = L(nu,fwhm)
```

### Make the Plot

The Lorentz profiles is a solid line and the Gaussian profile is a dashed line.

```python
fig,ax = plt.subplots()

fig.set_dpi(dpi)
fig.set_size_inches(wInches,hInches,forward=True)

ax.tick_params('both',length=6,width=lwidth,which='major',direction='in',top='on',right='on')
ax.tick_params('both',length=3,width=lwidth,which='minor',direction='in',top='on',right='on')

plt.xlim(nuMin,nuMax)
ax.xaxis.set_major_locator(MultipleLocator(5))
ax.xaxis.set_minor_locator(MultipleLocator(1))
plt.xlabel(r'$\nu-\nu_{ul}$',fontsize=axisFontSize)

plt.ylim(yMin,yMax)
ax.yaxis.set_major_locator(MultipleLocator(0.1))
ax.yaxis.set_minor_locator(MultipleLocator(0.02))
plt.ylabel(r'$\Phi_\nu$',fontsize=axisFontSize)

# plot the curves

plt.plot(nu,lorentz,'-',color='black',lw=1,zorder=10)
plt.plot(nu,gauss,'--',color='black',lw=1,zorder=9)

# Labels

plt.text(0.5,0.45,'Gaussian',fontsize=axisFontSize,color='black')
plt.text(2.6,0.05,'Lorentz',fontsize=axisFontSize,color='black')

# plot and file

plt.plot()
plt.savefig(plotFile,bbox_inches='tight',facecolor='white')
```


# Stroop Task

## Background Information
In a Stroop task, participants are presented with a list of words, with each word displayed in a color of ink. The participant’s task is to say out loud the *color of the ink* in which the word is printed. The task has two conditions: a congruent words condition, and an incongruent words condition. In the *congruent words* condition, the words being displayed are color words whose names match the colors in which they are printed: for example RED, BLUE. In the *incongruent words* condition, the words displayed are color words whose names do not match the colors in which they are printed: for example PURPLE, ORANGE. In each case, we measure the time it takes to name the ink colors in equally-sized lists. Each participant will go through and record a time from each condition.

### 1. What is our independent variable? What is our dependent variable?

Our independent variable is the test that participants take under two conditions: congruent and incongruent.

Our dependent variable is the time it takes for a participant to name the ink colors in equally-sized lists.

### 2. What is an appropriate set of hypotheses for this task? What kind of statistical test do you expect to perform? Justify your choices.
By looking at the data provided in the sample dataset, it seems like participants taken from a population are taking longer to identify the names of colors when printed in a different color (incongruent words condition). From this observation, we can come up with the following hypotheses:

Let $\mu_{incongruent}$ be defined as the mean of the population that performs the task under the incongruent condition.

Let $\mu_{congruent}$ be defined as the mean of the population that perform the task under the congruent condition.

**Null Hypothesis**: There is no statistical difference in time measurement between the two population means. It will take the same time for an individuals from the population to perform each task.
$$H_0 :\ \mu_{incongruent} = \mu_{congruent}$$

**Alternative Hypothesis**: Individuals from the population will take longer to complete the task under the incongruent words condition than with the congruent words condition.
$$H_A: \mu_{incongruent} > \mu_{congruent}$$

A **One-tailed dependent t-test** in the positive direction is expected to be performed to see if any hypotheses need to be rejected.
* A t-test is used because:
    * The sample size is below 30.
    * The population standard deviation is not known.
* A dependent t-test is used because the two samples are dependent; the same partipants take the two tests under different conditions.
* Only one direction is tested because we want to see if it takes longer for partipants to perform the task under the incongruent words condition.

Based on the results of our t-test we can make a inference towards how conflicting cues play a role in how fast individuals from the human population can process information. In this particular case, we will find out how a name of a color being displayed with a different color affect how long it takes to be recited.

### 3. Report some descriptive statistics regarding this dataset. Include at least one measure of central tendency and at least one measure of variability.

```python
import math
import pandas as pd
import scipy.stats as st
from IPython.display import Latex
from IPython.display import Math
from IPython.display import display

%matplotlib inline
```

```python
path = r'./stroopdata.csv'

df_stroop = pd.read_csv(path)
df_stroop
```

      Congruent
      Incongruent

      0
      12.079
      19.278

      1
      16.791
      18.741

      2
      9.564
      21.214

      3
      8.630
      15.687

      4
      14.669
      22.803

      5
      12.238
      20.878

      6
      14.692
      24.572

      7
      8.987
      17.394

      8
      9.401
      20.762

      9
      14.480
      26.282

      10
      22.328
      24.524

      11
      15.298
      18.644

      12
      15.073
      17.510

      13
      16.929
      20.330

      14
      18.200
      35.255

      15
      12.130
      22.158

      16
      18.495
      25.139

      17
      10.639
      20.429

      18
      11.344
      17.425

      19
      12.369
      34.288

      20
      12.944
      23.894

      21
      14.233
      17.960

      22
      19.710
      22.058

      23
      16.004
      21.157

```python
mu_congruent = round(df_stroop['Congruent'].mean(),4)
mu_incongruent = round(df_stroop['Incongruent'].mean(),4)

text = r"""
\begin{{align}}
    \mu_{{congruent}}={}\\
    \mu_{{incongruent}}={}
\end{{align}}""".format(mu_congruent, mu_incongruent)

Latex(text)
```

\begin{align}
    \mu_{congruent}=14.0511\\
    \mu_{incongruent}=22.0159
\end{align}

```python
df_stroop.plot(kind="bar")

```

### 5. Now, perform the statistical test and report your results. What is your confidence level and your critical statistic value? Do you reject the null hypothesis or fail to reject it? Come to a conclusion in terms of the experiment task. Did the results match up with your expectations?

#### Descriptive statistics for the difference between two conditions
Let $\mu_D$ be defined as the mean difference between the incongruent set and congruent set.

Let $\sigma_D$ be defined as the standard deviation of the differences between the incongruent and congruent set.

```python
df_diff = df_stroop['Incongruent']-df_stroop['Congruent']
mu_D = round(df_diff.mean(),4)
std_D = round(df_diff.std(),4)

text = r"""
        \begin{{align}}
            \mu_D = {}\\
            \sigma_D = {}\\
        \end{{align}}""".format(mu_D,std_D)

display(Latex(text))
```

        \begin{align}
            \mu_D = 7.9648\\
            \sigma_D = 4.8648\\
        \end{align}

#### Calculate the critical t value:

```python
n = df_diff.count()
df = n-1
alpha = .05
t_critical = round(st.t.ppf(1-alpha,df),3)

text = r"""t_{{critical}}=t_{{{},{}}}={}""".format(alpha,df,t_critical)
display(Math(text))
```

$$t_{critical}=t_{0.05,23}=1.714$$

#### Calculate the t value:

```python
# Calculate t
SEM = round(std_D/math.sqrt(n),4)
t = round(mu_D/SEM,3)

text = r"""t = \frac{{\mu_D}}{{SEM}} = \
           \frac{{{}}}{{{}}} = {}""".format(mu_D,SEM,t)
display(Math(text))
```

$$t = \frac{\mu_D}{SEM} =            \frac{7.9648}{0.993} = 8.021$$

#### Confidence Interval:

```python
two_tailed_t_critical = round(st.t.ppf(alpha/2,df),3)
m_error = -1*round(two_tailed_t_critical * SEM,4)

text = r"""
        \begin{{align}}
            {} \pm {}\\
            [{},{}]\\
        \end{{align}}""".format(mu_D,m_error,mu_D-m_error,mu_D+m_error)

display(Latex(text))
```

        \begin{align}
            7.9648 \pm 2.0545\\
            [5.9103,10.0193]\\
        \end{align}

Our results show that t is within the critical region, meaning:
$t > t_{critical}$. The Stroop Effect is observed with these findings. Partipants are likely to take 6 to 10 seconds longer on average to finishing the incongruent word condition compared with the congruent words condition.

We reject the null hypothesis. Based on our statistical findings we can say that the participants take longer to finish the task under the incongruent words condition. The results match up my expectations.

### 6. Optional: What do you think is responsible for the effects observed? Can you think of an alternative or similar task that would result in a similar effect? Some research about the problem will be helpful for thinking about these two questions!

There is a part of our brain that deal with recalling information given a certain stimuli. There is another part of our brain that deals with processing new information. When the information supplied from both of these parts interfere with another, there is a time delay in regards to processing the correct information.

A real life scenario of this problem is learning a new language which uses the roman alphabet. If someone knows English, but tries to learn French, then initially that person will struggle to attach different sounds to the letters that already have a sound attached to it.

```python
import IPython.display as disp
def css_styling():
    styles = open("../css/custom.css", "r").read()
    return disp.HTML(styles)
css_styling()
```