```python
import numpy as np
```

```python
from IPython.display import display, Math, Latex
```

```python
import print_latex

from print_latex import printLatex
```

```python
import pygamma as pg
```

# 1 Introduction

The Hilbert space module in GAMMA lays down the foundation for performing magnetic reso-
nance simulations using quantum mechanics. At the core are three features. First, a complete set
of single spin operators are provided as small dimensional arrays. Second, the spin system
class(es) allow users to set up a group of spins that define a composite spin Hilbert space. Third,
the module provides a host of spin operators defined in the composite space of a provided spin sys-
tem. These are built up from direct produces of single spin operators and thus general for any spin
systems.

Extending these three abilities, GAMMA provides simple functions for constructing spin base
Hamiltonians and functions for density operator manipulations. Simple pulse, delay, and acquisi-
tion functionality is set up in this module.

# 2 Single Spin Operators

## 2.1 Overview

The GAMMA module ***SpinOpSng*** contains routines to handle spin operators associated with a sin-
gle particle. These serve as a front end to class ***spin_op*** which treats spin operators associated with
a system of particles. Each ***SpinOpSng*** exists in the small Hilbert space of its particle whereas each
***spin_op*** will exist in a larger composite Hilbert space produced from all its particles. The latter is
formed from the tensor product of ***SpinOpSng*** for each of its particles.

Normally GAMMA users do NOT deal with single spin operators. Rather, they deal with spin op-
erators (class ***spin_op***) defined over spin systems. This is much more general and produces equiv-
alent results when the spin system only contains a single spin particle.

## 2.2 Available Functions

| Operator         |   Description |
|------------------|---------------|
| - Ie             |   - Identity single spin operator |
| - Ix             |   - Ix single spin operator       |
| - Iy             |   - Iy single spin operator       |
| - Iz             |   - Iz single spin operator       |
| - Ip             |   - I+ single spin operator       |
| - Im             |   - I- single spin operator       |
| - Raxis          |   - Single spin rotation matrix   |

## 2.3 Basic Functions

### 2.3.1  Ie

#### Usage

```python
Ie( qn)
```

#### Description

The function ***Ie*** is used to create a single spin identity spin operator. The function takes the dimension of the single spin Hilbert space, ***qn***, as input where ***qn = (2I+1)*** and ***I*** is the spin quantum number of the spin for which the operator applies.

#### Retun Value:

An identity matrix is returned.

#### Example

```python
Ie = pg.Ie(2);
printLatex(Ie)
```

\begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \\ \end{bmatrix}

```python
print_latex(pg.Ie(2))
print_latex(pg.Ie(3))
print_latex(pg.Ie(4))
```

#### See Also: Ix, Iy, Iz, Ip, Im

#### Mathematical Basis:

There are four operators which provide a basis for spin angular momentum, the set {$I_e$, $I_x$, $I_y$, $I_y$  }. Although $I_e$ is trivial (being simply the identity matrix) its provision is essential in order to complete the basis mentioned above. The matrix representation of $I_e$ for a single spin is efficiently expressed in the dimension of the Hilbert space of the
spin itself, spanning $2(I_i+1)$. Examples are shown below for particles with spin 1/2, 1, and 3/2 respectively.

\begin{align}
I_e \left ( I = \frac{1}{2} \right ) =
\begin{bmatrix} 1 & 0 \\
                0 & 1 \\
\end{bmatrix} \; \;
I_e \left ( I = 1 \right ) =
\begin{bmatrix} 1 & 0 & 0 \\
             0 & 1 & 0\\
             0 & 0 & 1 \\
\end{bmatrix} \; \; I_e \left ( I = \frac{3}{2} \right ) =
\begin{bmatrix} 1 & 0 & 0 & 0\\
                0 & 1 & 0 & 0 \\
                0 & 0 & 1 & 0 \\
                0 & 0 & 0 & 1 \\
                \end{bmatrix}
\end{align}

The $I_e$ operators are used most frequently in the expansion of other spin operators into their composite Hilbert space forms through use of these direct products.

----

### 2.3.2 Ix

#### Usage:

```python
Ix( qn )
```

#### Description:

The function ***Ix*** is used to create a single spin operator for the x-component of spin angular momentum. The function takes the dimension of the single spin Hilbert space, $qn$, as input where $qn = (2I+1)$ and $I$ is the spin quantumnumber of the spin for which the operator applies.

#### Return Value:

A matrix (h_matrix) is returned.

#### Example

```python
IX = pg.Ix(3)       # The Ix spin operator for a single spin 1 particle
printLatex(IX)
```

\begin{bmatrix} 0.0 & 0.7071 & 0.0 \\ 0.7071 & 0.0 & 0.7071 \\ 0.0 & 0.7071 & 0.0 \\ \end{bmatrix}

#### See Also: Ie, Iy, Iz, Ip, Im

#### Mathematical Basis:

There are four operators which provide a basis for spin angular momentum, the set { $I_e$, $I_x$, $I_y$, $I_z$ }. The matrix representation of $I_x$ for a single spin is efficiently expressed in the dimension of the Hilbert space of the spin itself,
spanning $2(I_i +1)$. Examples are shown below for particles with spin 1/2, 1, and 3/2 respectively

\begin{align}
I_x \left ( I = \frac{1}{2} \right ) =  \frac{1}{2} \begin{bmatrix} 0 & 1 \\
                1 & 0 \\
\end{bmatrix} \;\;
\space I_x \left ( I = 1 \right ) = \frac {1}{\sqrt{2}} \begin{bmatrix} 0 & 1 & 0 \\
                1 & 0 & 1\\
                0 & 1 & 0 \\
\end{bmatrix} \;\;
\space I_x \left ( I = \frac{3}{2} \right ) = \frac{1}{2}
\begin{bmatrix} 0        & \sqrt{3} & 0        & 0         \\
                \sqrt{3} & 0        & 2        & 0         \\
                0        & 2        & 0        & \sqrt{3}  \\
                0        & 0        & \sqrt{3} & 0         \\
                \end{bmatrix}
\end{align}

The $I_x$ operators are used most frequently in the expansion of other spin operators into their composite Hilbert space forms through use of these direct products. In general, the matrix elements of the operator are given by$^1$

\begin{equation}
\left < m_i \right | I_{ix} \left | n_i \right > =
\frac{1}{2}
  \left [ I_i \left ( I_i + 1 \right ) -
  m_i \left ( m_i \pm 1 \right )
  \right ]^{1/2} \delta_{m_i,n_i \pm 1 }
\end{equation}

Other important relationships involving this operator are the following.

\begin{equation*}
I_x = \frac {1}{2} \left ( I_+ + I_- \right ) \; \; \;
\left [ I_x, I_y \right ] = iI_z \; \; \;
\left [ I_x, I_z \right ] = -iI_y
\end{equation*}

1. Goldman, page 73, equation (3.93)

----

### 2.3.3 Iy

#### Usage:

```python
import pygamma as pg

pg.Iy(qn)
```

#### Description:

The function ***Iy*** is used to create a single spin operator for the y-component of spin angular momentum. There function takes the dimension of the single spin Hilbert space, ***qn***, as input where $qn = (2I+1)$ and ***I*** is the spin quantum
number of the spin for which the operator applies.

#### Return Value:

A matrix (h_matrix) is returned.

#### Example:

```python
IY = pg.Iy(2)  # The Iy spin operator for a single spin 1/2 particle
printLatex(IY)
```

\begin{bmatrix} 0.0 & -0.5j \\ 0.5j & 0.0 \\ \end{bmatrix}

#### See Also: Ie, Ix, Iz, Ip, Im

#### mathematical Basis:

There are four operators which provide a basis for spin angular momentum, the set { $I_e$, $I_x$, $I_y$, $I_z$}. The matrix representation of ***Iy*** for a single spin is efficiently expressed in the dimension of the Hilbert space of the spin itself, spanning $2(I_i +1)$. Examples are shown below for particles with spin 1/2, 1, and 3/2 respectively$^1$ .

\begin{align}
I_y \left ( I = \frac{1}{2} \right ) =  \frac{1}{2} \begin{bmatrix} 0 & -1 \\
                1 &  0 \\
\end{bmatrix} \;\;
\space I_y \left ( I = 1 \right ) = \frac {1}{\sqrt{2}} \begin{bmatrix} 0 & -1 &  0 \\
                1 &  0 & -1 \\
                 0 & 1 &  0 \\
\end{bmatrix} \;\;
\space I_y \left ( I = \frac{3}{2} \right ) = \frac{1}{2}
\begin{bmatrix} 0        & -\sqrt{3} & 0        & 0         \\
                \sqrt{3} & 0         & -2       & 0         \\
                0        & 2         & 0        & -\sqrt{3} \\
                0        & 0         & \sqrt{3} & 0         \\
                \end{bmatrix}
\end{align}

The ***Iy*** operators are used most frequently in the expansion of other spin operators into their composite Hilbert space forms through use of these direct products. In general, the matrix elements of the operator are given by$^2$

\begin{equation}
\left < m_i \right | I_{iy} \left | n_i \right > =
\pm \frac{1}{2}
  \left [ I_i \left ( I_i + 1 \right ) -
  m_i \left ( m_i \pm 1 \right )
  \right ]^{1/2} \delta_{m_i,n_i \pm 1 }
\end{equation}

Other important relationships involving this operator are the following.

\begin{equation*}
I_y = \frac {i}{2} \left ( I_+ - I_- \right ) \; \; \;
\left [ I_y, I_x \right ] = -iI_z \; \; \;
\left [ I_y, I_z \right ] = iI_x
\end{equation*}

1. These leave out the units of hbar, see Schiff, page 203.
2. Goldman, page 73, equation (3.93)

-----

### 2.3.4 Iz

#### Usage:

```python
import pygamma as pg
pg.Iz(qn)
```

#### Desciption:

The function ***Iz*** is used to create a single spin operator for the z-component of spin angular momentum. There function takes the dimension of the single spin Hilbert space, ***qn***, as input where $qn = (2I+1)$ and ***I*** is the spin quantum number of the spin for which the operator applies.

#### Return Value:

A diagonal matrix (d_matix) is returned.

##### Example:

```python
IZ = pg.Iz(2)  # The Iz spin operator for a single spin 1/2 particle

printLatex(IZ)
```

\begin{bmatrix} 0.5 & 0.0 \\ 0.0 & -0.5 \\ \end{bmatrix}

#### See Also: Ie, Ix, Iy, Ip, Im

#### Mathematical Basis:

There are four operators which provide a basis for spin angular momentum, the set {$I_e$, $I_x$, $Iy$, $I_z$}. The matrix representation of $I_z$ for a single spin is efficiently expressed in the dimension of the Hilbert space of the spin itself, spanning $2(I_i +1)$. Examples are shown below for particles with spin 1/2, 1, and 3/2 respectively$^1$.

\begin{align}
I_{iz} \left ( I_i = \frac{1}{2} \right ) &=  \frac{1}{2} \begin{bmatrix} 1 &  0 \\
                0 & -1 \\
\end{bmatrix} \;\;
\space I_{iz} \left ( I = 1 \right ) &=
\begin{bmatrix} 1 & 0 &  0 \\
                0 & 0 &  0 \\
                0 & 0 & -1 \\
\end{bmatrix} \;\;
\space I_{iz} \left ( I_i = \frac{3}{2} \right ) &= \frac{1}{2} \begin{bmatrix} 3 & 0 &  0 &  0 \\
                0 & 1 &  0 &  0 \\
                0 & 0 & -1 &  0 \\
                0 & 0 &  0 & -3 \\
\end{bmatrix}
\end{align}

The $I_z$ operators are used most frequently in the expansion of other spin operators into their composite Hilbert space forms through use of these direct products. In general, the matrix elements of the operator are given by$^2$

\begin{equation}
\left < m_i \right | I_{iz} \left | n_i \right > =
m \delta_{m_in_i} =
  \left [ I \left ( 1 - \alpha \right ) \right ] \delta_{m_i,n_i \pm 1 }
\end{equation}

Other important relationships involving this operator are the following.

\begin{equation*}
\left [ I_{iz}, I_{ix} \right ] = iI_{iy} \; \; \;
\left [ I_{iz}, I_{i+} \right ] = iI_{i+} \; \; \;
\left [ I_{iz}, I_{i-} \right ] = iI_{i-}
\end{equation*}

1. These leave out the units of hbar, see Schiff, page 203.
2. Goldman, page 73, equation (3.93).

------

### 2.3.5 Ip

#### Usage:

```python
import pygamma as pg

pg.Ip(qn)
```
#### Description:

The function ***Ip*** is used to create a single spin s raising operator $I_+$ . There function takes the dimension of the single spin Hilbert space, ***qn***, as input where $qn = (2I+1)$ and ***I*** is the spin quantum number of the spin for which the operator applies.

#### Return Value:

A matrix (n_matrix) is returned.

#### Example:

```python
IP = pg.Ip(2)  # The I+ spin operator for a single spin 1/2 particle

printLatex(IP)
```

\begin{bmatrix} 0.0 & 1.0 \\ 0.0 & 0.0 \\ \end{bmatrix}

#### See Also: Ie, Iy, Iz, Ix, Im

#### Mathematical Basis:

The raising operator $I_+$ is defined to work on state $ \left |I,m \right >$ as$^1$

\begin{equation*}
I_+ \left | I,m \right > =
\left [ I \left ( I+1 \right ) - m \left ( m+1 \right ) \right ] ^{1/2} \left | i,m+1 \right >
\end{equation*}

having matrix elements

\begin{equation*}
\left < I,m' \right | I_+ \left | i,m \right > =
\left [ I \left ( I+1 \right ) - m \left ( m+1 \right ) \right ]^{1/2} \delta_{m+1,m'}
\end{equation*}

The matrix representation of $I_+$ for a single spin i, $I_{i+}$, is efficiently expressed in the dimension of the Hilbert space of the spin itself, spanning $2(I_i+1)$. Examples are shown below for a spin 1/2, spin 1  and a spin 3/2 particle

\begin{align}
I_{i+} \left ( I_i = \frac{1}{2} \right ) =
\begin{bmatrix} 0 & 1 \\
                0 & 0 \\
\end{bmatrix} \;\;
\space I_{i+} \left ( I_i = 1 \right ) =
\sqrt{2}
\begin{bmatrix} 0 & 1 & 0 \\
                0 & 0 & 1 \\
                0 & 0 & 0 \\
\end{bmatrix} \;\
\space I_{i+} \left ( I_i = \frac{3}{2} \right ) =
\begin{bmatrix} 0 & \sqrt{3} &  0 &  0        \\
                0 & 0        &  2 &  0        \\
                0 & 0        &  0 &  \sqrt{3} \\
                0 & 0        &  0 & 0         \\
\end{bmatrix}
\end{align}

Other useful relationships involving this operator are the following.

\begin{equation*}
I_{i+} = \left ( I_{ix}+iI_{iy} \right ) \; \; \;
\left [ I_{i+}, I_{iz} \right ] = -I_{i+} \; \; \;
\left [ I_{i+}, I_{i-} \right ] = 2I_{iz} \; \; \;
I_{i+} = \left ( I_{i-} \right ) ^{\dagger}
\end{equation*}

1. Schiff, pg. 202, equation (27.25).

----

### 2.3.6 Im

#### Usage:

```python
import pygamma as pg

pg.Im(qn)
```

#### Description:

The function ***Im*** is used to create a single spin lowering operator $I_-$ . There function takes the dimension of the single spin Hilbert space, ***qn***, as input where ***qn = (2I+1)*** and ***I*** is the spin quantum number of the spin for which the operator applies.

#### Return Value:

A matrix (n_matrix) is returned.

#### Example:

```python
IM = pg.Im(2)  # The I- spin operator for a single spin 1/2 particle

printLatex(IM)
```

\begin{bmatrix} 0.0 & 0.0 \\ 1.0 & 0.0 \\ \end{bmatrix}

#### See Also: Ie, Ix, Iy, Iz, Ip

##### Mathematical Basis:

The lowering operator $I_-$ is defined to work on state ***|I,m>*** as$^1$

\begin{equation*}
I_- \left | I,m \right > = \left [ I \left ( I+1 \right ) - m \left ( m-1 \right ) \right ] ^{1/2} \left | I,m-1 \right >
\end{equation*}

having matrix elements

\begin{equation*}
\left < I,m' \left | I_- \right | I,m \right > =
\left [ I \left ( I+1 \right ) - m \left ( m-1 \right ) \right ]^{1/2} \delta_{m-1,m'}
\end{equation*}

The matrix representation of $I_-$ for a single spin $i$, $I_-$, is efficiently expressed in the dimension of the Hilbert space of the spin itself, spanning $2(I_i +1)$. Examples are shown below for particles with spin angular momentum 1/2, 1,
and 3/2 respectively.

\begin{align}
I_{i-} \left ( I_i = \frac{1}{2} \right ) =
\begin{bmatrix} 0 & 0 \\
                1 & 0 \\
\end{bmatrix} \;\;
\space I_{i-} \left ( I_i = 1 \right ) = \sqrt{2}
\begin{bmatrix} 0 & 0 & 0 \\
                1 & 0 & 0 \\
                0 & 1 & 0 \\
\end{bmatrix} \;\
\space I_{i-} \left ( I_i = \frac{3}{2} \right ) =
\begin{bmatrix} 0        & 0 &  0        &  0 \\
                \sqrt{3} & 0 &  0        &  0 \\
                0        & 2 &  0        &  0 \\
                0        & 0 &  \sqrt{3} &  0 \\
\end{bmatrix}
\end{align}

\begin{equation*}
I_{i-} = \left ( I_{ix}-iI_{iy} \right ) \; \; \;
\left [ I_{i-}, I_{iz} \right ] = I_{i-} \; \; \;
\left [ I_{i+}, I_{i-} \right ] = 2I_{iz} \; \; \;
I_{i-} = \left ( I_{i+} \right ) ^{\dagger}
\end{equation*}

1. Schiff, pg. 202, equation (27.25).

----

### 2.3.7 Raxis

#### Usage:

```python
import pygamma as pg

pg.Raxis( qn, beta, axis )
```

#### Description:

The function ***Raxis*** is used to create a single spin rotation operator, $R_u(\beta)$, for rotating spin angular momentum about the ***u*** -axis by the angle $\beta$. This is defined by the formula

\begin{equation*}
R_u \left ( \beta \right ) = \exp \left ( -i \beta I_u \right )
\end{equation*}

where $I_u$ is an appropriate single spin angular momentum operator and $u$ one of the three Cartesian axes.

#### Return Value:

A matrix is returned.

#### Example:

```python
Rx90 = pg.Raxis(2, 90.0, 'x') # Rotation op. for a spin 1/2 particle, 90 deg. about x

printLatex(Rx90)
```

\begin{bmatrix} 0.7071 & -0.7071j \\ -0.7071j & 0.7071 \\ \end{bmatrix}

#### Mathematical Basis:

The formulae used for a single spint 1/2 particle are$^1$ ( using $C \beta = \cos \left ( \beta / 2 \right )$, $S \beta = \sin \left ( \beta / 2 \right )$ )

\begin{align}
R_{ix}^{(1/2)} \left ( \beta \right ) =
\begin{bmatrix}
  C \beta & -i S \beta \\
  -i S \beta & C \beta \\
\end{bmatrix} \;\;
R_y \left ( \beta \right ) =
\begin{bmatrix}
  C \beta & -S \beta \\
  S \beta &  C \beta \\
\end{bmatrix} \;\;
R_z \left ( \beta \right ) =
\begin{bmatrix}
e^{-i \beta / 2} & 0 \\
0 & e^{i \beta / 2} \\
\end{bmatrix}
\end{align}

Unfortunately, there is currently no similar simple formula for a spin with I > 1/2 and these are computed by actual exponentiation of the operator. A few specific rotation matrices one will obtain from this function are as follows.

```python
print("Rx60 = ")
printLatex(pg.Raxis(2,60., 'x'))
print("Rx90 =" )
printLatex(pg.Raxis(2,90., 'x'))
print("Rx180 =")
printLatex(pg.Raxis(2,180., 'x'))
```

    Rx60 =

\begin{bmatrix} 0.866 & -0.5j \\ -0.5j & 0.866 \\ \end{bmatrix}

    Rx90 =

\begin{bmatrix} 0.7071 & -0.7071j \\ -0.7071j & 0.7071 \\ \end{bmatrix}

    Rx180 =

\begin{bmatrix} 0.0 & -1.0j \\ -1.0j & 0.0 \\ \end{bmatrix}

```python
print("Ry60 = ")
printLatex(pg.Raxis(2,60., 'y'))
print("Ry90 =" )
printLatex(pg.Raxis(2,90., 'y'))
print("Ry180 =")
printLatex(pg.Raxis(2,180., 'y'))
```

    Ry60 =

\begin{bmatrix} 0.866 & -0.5 \\ 0.5 & 0.866 \\ \end{bmatrix}

    Ry90 =

\begin{bmatrix} 0.7071 & -0.7071 \\ 0.7071 & 0.7071 \\ \end{bmatrix}

    Ry180 =

\begin{bmatrix} 0.0 & -1.0 \\ 1.0 & 0.0 \\ \end{bmatrix}

```python
print("Rz60 = ")
printLatex(pg.Raxis(2,60., 'z'))
print("Rz90 =" )
printLatex(pg.Raxis(2,90., 'z'))
print("Rz180 =")
printLatex(pg.Raxis(2,180., 'z'))
```

    Rz60 =

\begin{bmatrix} (0.866-0.5j) & 0.0 \\ 0.0 & (0.866+0.5j) \\ \end{bmatrix}

    Rz90 =

\begin{bmatrix} (0.7071-0.7071j) & 0.0 \\ 0.0 & (0.7071+0.7071j) \\ \end{bmatrix}

    Rz180 =

\begin{bmatrix} -1.0j & 0.0 \\ 0.0 & 1.0j \\ \end{bmatrix}

```python

```


```python
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```python
#export
from exp.nb_06 import *
```

## ConvNet

Let's get the data and training interface from where we left in the last notebook.

```python
x_train,y_train,x_valid,y_valid = get_data()

x_train,x_valid = normalize_to(x_train,x_valid)
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)

nh,bs = 50,512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```python
mnist_view = view_tfm(1,28,28)
cbfs = [Recorder,
        partial(AvgStatsCallback,accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```python
nfs = [8,16,32,64,64]
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
%time run.fit(2, learn)
```

    train: [1.768248125, tensor(0.4032, device='cuda:0')]
    valid: [0.3613622314453125, tensor(0.8944, device='cuda:0')]
    train: [0.25451984375, tensor(0.9232, device='cuda:0')]
    valid: [0.15081968994140624, tensor(0.9545, device='cuda:0')]
    CPU times: user 3.77 s, sys: 1.58 s, total: 5.35 s
    Wall time: 17 s

### Custom

Let's start by building our own `BatchNorm` layer from scratch.

```python
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
#         so we have batch norm paramaters for each filter
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
#         register buffer creates self.means and self.vars and gives us benefits like automatically on gpu
# also causes these items to be saved when we save the model which we need since this is part of the model. need them at inference time
        self.register_buffer('vars',  torch.ones(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
#         print("x!!", x.shape)
#         example result:
#             x!! torch.Size([512, 8, 14, 14])
#             x!! torch.Size([512, 16, 7, 7])
#             x!! torch.Size([512, 32, 4, 4])
#             x!! torch.Size([512, 64, 2, 2])
#             x!! torch.Size([512, 64, 1, 1])
#             x!! torch.Size([512, 8, 14, 14])
#             x!! torch.Size([512, 16, 7, 7])
#             x!! torch.Size([512, 32, 4, 4])
#             x!! torch.Size([512, 64, 2, 2])
#             x!! torch.Size([512, 64, 1, 1])
#         we average over all of the batches and the length and width of the parameters so we are left with a mean for each filter (the 1 indexed item in x)
# keep dim=True means it will leave a 0 in the index for each of 0 and 2 and 3 so will broadcast nicely
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
        if self.training:
#             if we're training we update the mean and the variation like normal
            with torch.no_grad(): m,v = self.update_stats(x)
#         if we're in inference mode we take the weighted moving average of the mean and variation
        else: m,v = self.means,self.vars
#         the normalization is just our value minus the mean divided by the standard variation (square root of the variance)
        x = (x-m) / (v+self.eps).sqrt()
        return x*self.mults + self.adds
```

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
#export
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None: m.bias.data.zero_()
    for l in m.children(): init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None, uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
```

```python
with Hooks(learn.model, append_stats) as hooks:
    run.fit(1, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```python
learn,run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.27602759765625, tensor(0.9148, device='cuda:0')]
    valid: [0.1786552490234375, tensor(0.9439, device='cuda:0')]
    train: [0.0937239453125, tensor(0.9704, device='cuda:0')]
    valid: [0.113109423828125, tensor(0.9649, device='cuda:0')]
    train: [0.06871248046875, tensor(0.9787, device='cuda:0')]
    valid: [0.07223031616210937, tensor(0.9784, device='cuda:0')]
    CPU times: user 4.03 s, sys: 496 ms, total: 4.52 s
    Wall time: 4.54 s

### Builtin batchnorm

```python
#export
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.28875259765625, tensor(0.9129, device='cuda:0')]
    valid: [0.11767066650390624, tensor(0.9637, device='cuda:0')]
    train: [0.0749825634765625, tensor(0.9764, device='cuda:0')]
    valid: [0.08199794921875, tensor(0.9757, device='cuda:0')]
    train: [0.0508402392578125, tensor(0.9845, device='cuda:0')]
    valid: [0.07347352905273437, tensor(0.9779, device='cuda:0')]
    CPU times: user 3.33 s, sys: 604 ms, total: 3.93 s
    Wall time: 3.98 s

### With scheduler

Now let's add the usual warm-up/annealing.

Can go up to a learning rate as high as 2 since using batch norm

```python
sched = combine_scheds([0.3, 0.7], [sched_lin(0.6, 2.), sched_lin(2., 0.1)])
```

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched)])
```

```python
run.fit(8, learn)
```

    train: [0.30782044921875, tensor(0.9081, device='cuda:0')]
    valid: [0.119206640625, tensor(0.9614, device='cuda:0')]
    train: [0.0955680078125, tensor(0.9703, device='cuda:0')]
    valid: [0.10755780029296876, tensor(0.9685, device='cuda:0')]
    train: [0.0605357373046875, tensor(0.9803, device='cuda:0')]
    valid: [0.0718800537109375, tensor(0.9779, device='cuda:0')]
    train: [0.0384215673828125, tensor(0.9878, device='cuda:0')]
    valid: [0.0665466064453125, tensor(0.9794, device='cuda:0')]
    train: [0.02464458251953125, tensor(0.9921, device='cuda:0')]
    valid: [0.046663726806640625, tensor(0.9862, device='cuda:0')]
    train: [0.0163460791015625, tensor(0.9954, device='cuda:0')]
    valid: [0.04146202697753906, tensor(0.9877, device='cuda:0')]
    train: [0.010099387817382812, tensor(0.9977, device='cuda:0')]
    valid: [0.03948231506347656, tensor(0.9879, device='cuda:0')]
    train: [0.006879940795898438, tensor(0.9986, device='cuda:0')]
    valid: [0.03834141540527344, tensor(0.9890, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

```python
class LayerNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))

    def forward(self, x):
#         note, the difference is we are doing 1,2,3 rather than 0,2,3
        m = x.mean((1,2,3), keepdim=True)
        v = x.std ((1,2,3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

```python
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.1216, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 4.33 s, sys: 700 ms, total: 5.03 s
    Wall time: 5.06 s

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

```python
class InstanceNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))

    def forward(self, x):
#         the difference here is doing 2,3 versus 0,2,3 with regular batch norm or 1,2,3 with layer norm
        m = x.mean((2,3), keepdim=True)
        v = x.std ((2,3), keepdim=True)
        res = (x-m) / ((v+self.eps).sqrt())
        return res*self.mults + self.adds
```

```python
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 4.46 s, sys: 718 ms, total: 5.18 s
    Wall time: 5.18 s

*Question*: why can't this classify anything?

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) have you covered:

### Group norm

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

Let's see how this works by creating a databunch with a batch size of 2

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [2.35619296875, tensor(0.1649, device='cuda:0')]
    valid: [2867.7198, tensor(0.2604, device='cuda:0')]
    CPU times: user 1min 32s, sys: 835 ms, total: 1min 33s
    Wall time: 1min 33s

### Running Batch Norm

To solve this problem we introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std.

```python
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step', tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        s = x.sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel()/nc)
        mom1 = 1 - (1-self.mom)/math.sqrt(bs-1)
        self.mom1 = self.dbias.new_tensor(mom1)
        self.sums.lerp_(s, self.mom1)
        self.sqrs.lerp_(ss, self.mom1)
        self.count.lerp_(c, self.mom1)
        self.dbias = self.dbias*(1-self.mom1) + self.mom1
        self.batch += bs
        self.step += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        c = self.count
        if self.step<100:
            sums = sums / self.dbias
            sqrs = sqrs / self.dbias
            c    = c    / self.dbias
        means = sums/c
        vars = (sqrs/c).sub_(means*means)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div_((vars.add_(self.eps)).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```python
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(RunningBatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_rbn, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [0.157932021484375, tensor(0.9511, device='cuda:0')]
    valid: [0.0986408935546875, tensor(0.9730, device='cuda:0')]
    CPU times: user 16.5 s, sys: 1.57 s, total: 18.1 s
    Wall time: 18.1 s

This solves the small batch size issue!

### What can we do in a single epoch?

Now let's see with a decent batch size what result we can get.

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 32), c)
```

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_rbn, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

```python
%time run.fit(1, learn)
```

    train: [0.1573110546875, tensor(0.9521, device='cuda:0')]
    valid: [0.09242745971679688, tensor(0.9818, device='cuda:0')]
    CPU times: user 16.6 s, sys: 1.52 s, total: 18.1 s
    Wall time: 18.2 s

## Export

```python
nb_auto_export()
```

    <IPython.core.display.Javascript object>

```python

```


# Стандартная форма записи уравнения звена

    Рассмотрим систему с одномерным входом и одномерным выходом, которая описывается линейным дифференциальным уравнением. Чтобы облегчить анализ системы, уравнение записывается так, чтобы сигнал на выходе объекта и его производные находились в левой части уравнения, а остальные члены — в правой части. Кроме того, принято, чтобы сам сигнал на выходе имел в уравнении коэффициент равный единице.

    Таким образом, стандартное линейное уравнение звена имеет вид
    $$
        \sum_{i=0}^n T_i \frac{\mathrm{d}^i x}{\mathrm{d} t^i}(t) = \sum_{j=0}^m k_j \frac{\mathrm{d}^j u}{\mathrm{d} u^j}(t) + k_{m+1} f(t),\quad T_0 = 0, \tag{1}
    $$
    где $x$ — выход системы, $u$ — управляющее воздействие, а функция $f(t)$ характеризует возмущения. Коэффициенты $k_0, \ldots, k_{m+1}$, называются коэффициентами передачи, а $T_0, \ldots, T_n$ — постоянными времени.

        Термин "коэффициент передачи" можно пояснить следующим образом. На вход звена подадим сигнал $u = const$ при условии, что $f(t) \equiv 0$. Тогда уравнение $(1)$ имеет решение $x = k_0 u$. Поэтому коэффициент $k_0$ представляет собой отношение сигнала на выходе к сигналу на входе в установившемся режиме.

# Типы звеньев систем управления
 Все звенья делятся на три большие группы:

1. позиционные
2. интегрирующие
2. дифференцирующие

Рассмотрим каждую из групп подробнее.

## Позиционные звенья

Звенья такого типа описываются дифференциальным уравнением $(1)$, в правой части которого $m = 0$. Рассмотрим следующие примеры таких звеньев:

 безинерционное (пропорциональное) звено $(n=0)$
        $$x = ku \tag{2.1}$$

 инерционное (апериодическое) звено $(n=1)$
        $$T \dot{x} +x = ku \tag{2.2}$$

Построим для каждого из этих звеньев передаточную, весовую и переходную функции при помощи библиотеки символьных вычислений Sympy.

### Пропорциональное звено

```python
# импортируем необходимые библиотеки
import sympy
import matplotlib.pyplot as plt
import numpy as np
from IPython.display import display, Math
```

```python
# функции для удобного отображения равенств
def dispStr(s):
    return display(Math("$"+s+"$"))
def dispEq(left, right):
    return display(Math("$"+left+'='+right+"$"))
def makelatex(sym):
    return sympy.latex(sym)
```

```python
# функции для часто повторяющихся действий
# применение преобразования Лапласа к уравнению
def LEq(Left, Right):
    return sympy.Eq(L(Left), L(Right))

# решение полученного уравнения относительно изображения x(t)
def LSolve(LEq):
    return sympy.solve(LEq, L(x))[0]

# передаточная функция
def H(LSol):
    return sympy.simplify(LSol/L(u))

# весовая функция
def h(H):
    return iL(H)

# переходная функция
def pi(H):
    return iL(H/s)
```

```python
def lambdify(expr, subs):
    """Преобразует символьную функцию expr(t) аргумента t в числовую с заменой параметров на соответствующие
    им значения из subs"""

    return np.vectorize(lambda arg : float(expr.evalf(subs=dict(list(zip(subs.keys(), subs.values()))+[(t, arg)]))))
```

```python
# Функция для построения графиков символьных функций с разничными параметрами в одних осях
def plot_sym(expr, subs_list, ax, ls, labels):
    for subs in subs_list:
        ax.plot(ls, lambdify(expr, subs)(ls))
    ax.grid()
    ax.legend(labels)

```

```python
# t, s и k - действительные символьные переменные
t, s, k = sympy.symbols("t, s, k", real=True)

# x(t), u(t) - действительные символьные функции переменой t
x = sympy.Function("x", real=True)(t)
u = sympy.Function("u", real=True)(t)
```

```python
# функции для прямого и обратного преобразования Лапласа
def L(x):
    return sympy.laplace_transform(x, t, s, noconds=True)

def iL(X):
    return sympy.inverse_laplace_transform(X, s, t)
```

Определим уравнение $(2.1)$

```python
# левая часть
Left21 = x
# правая часть
Right21 = k*u
# уравнение
dispEq(makelatex(Left21),makelatex(Right21))
```

$\displaystyle x{\left(t \right)}=k u{\left(t \right)}$

Применим к обеим частям преобразование Лапласа

```python
LEq21 = LEq(Left21, Right21)
LEq21
```

$\displaystyle \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

и решим его относительно $L \left[ x(t) \right]$:

```python
Sol21 = LSolve(LEq21)
dispEq(makelatex(L(x)), makelatex(Sol21))
```

$\displaystyle \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right)=k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

По определению передаточная функция — имеющее наименьший порядок отношение изображений выходной и входной переменных при нулевых начальных условиях:

```python
H21 = H(Sol21)
dispEq("\mathsf{H}_{(2.1)}(s)",makelatex(H21))
```

$\displaystyle \mathsf{H}_{(2.1)}(s)=k$

Весовой функцией $h(t)$ называется отклик системы на единичное импульсное воздействие ($\delta$-функцию) при нулевых начальных условиях.

Найдём весовую функцию пропорционального звена, учитывая, что $\mathcal{L}\left[\delta(t)\right](s) \equiv 1$ и, следовательно, $\mathsf{X}(s) \equiv \mathsf{H}(s)$.

```python
h21 = h(H21)
dispEq("h_{(2.1)}(t)",makelatex(h21))
```

$\displaystyle h_{(2.1)}(t)=k \delta\left(t\right)$

Переходной функцией $\pi(t)$ называется отклик системы на единичное ступенчатое воздействие ($\theta(t)$ - функция Хевисайда) при нулевых начальных условиях.

    Найдём переходную функцию пропорционального звена, учитывая что $\mathcal{L}\left[\theta(t)\right](s) = \frac{1}{s}$.

```python
pi21 = pi(H21)
dispEq("\pi_{(2.1)}(t)",makelatex(pi21))
```

$\displaystyle \pi_{(2.1)}(t)=k \theta\left(t\right)$

Таким образом, откликом пропорционального звена на любой воздействие является сигнал, пропорциональный воздействию с коэффициентом $k$.

#### Физичные примеры пропорционального звена

Примерами пропорционального звена могут служить: рычаг, зубчатая передача, делитель напряжений.

1) рычаг

Рис. 1.1

Перемещение $h_2$ (рис 1.1) зависит от соотношения плеч рычага $l_1$ и $l_2$ и величины входного перемещения $h_1$
$$
h_2(h_1) = \frac{l_2}{l_1} = k_р h_1
$$
$
k_р=\frac{l_2}{l_1}$ — коэффициент передачи.

2) зубчатая передача

Рис. 1.2

Угловая скорость зубчатой передачи $w_2$ (рис. 1.2) зависит от соотношения числа зубьев на малой и большой шестернях $z_1$ и $z_2$ и величины угловой скорости малой шестерни $w_1$
$$
\omega_2 = \frac{z_1}{z_2}\omega_1 =k_п \omega_1
$$
$k_п=\frac{z_2}{z_1}$ — коэффициент передачи.

3) делитель напряжений

Рис. 1.3

Выходное напряжение $U_вых$ (рис. 1.3) зависит от соотношения сопротивлений электрической цепочки $R_1$ и $R_2$ и величины входного напряжения $U_вх$
$$
U_вых =\frac{R_2}{R_1+R_2}U_вх = k_д U_вх
$$
$k_д=\frac{R_2}{R_1+R_2}$ — коэффициент передачи.

### Апериодическое звено

```python
# действительная постоянная символьная переменная T
T = sympy.Symbol('T', real=True, const=True)
```

Определим уравнение (2.2)

```python
# левая часть
Left22 = T*x.diff(t) + x
# правая часть
Right22 = k*u
# уравнение
dispEq(makelatex(Left22), makelatex(Right22))
```

$\displaystyle T \frac{d}{d t} x{\left(t \right)} + x{\left(t \right)}=k u{\left(t \right)}$

Применим к обеим частям преобразование Лапласа

```python
LEq22 = LEq(Left22, Right22)
LEq22
```

$\displaystyle T \mathcal{L}_{t}\left[\frac{d}{d t} x{\left(t \right)}\right]\left(s\right) + \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

По одному из свойств преобразования Лапласа
$$
    \mathcal{L}\left[ \frac{\mathrm{d}^n x}{\mathrm{d}t^n}(t) \right](s) = s^n \mathcal{L}\left[x(t) \right](s) - \sum_{i=1}^{n} s^{i-1} x^{(n-i)}(0) = s^n \mathcal{L}\left[x(t) \right](s) $$
   при  $x^{m}(0)=0; \space m = 1\ldots n-1$

Учтём это, произведя замену в полученных символьных выражениях.

```python
# составляем массив из пар (изображение n-ой производной x(t), s^n*изображение x(t)), то же и для u(t)
subl = [(L(sympy.diff(x,(t,i))), s**i*L(x)) for i in range(3)] + [(L(sympy.diff(u,(t,i))), s**i*L(u)) for i in range(3)]
subl
```

    [(LaplaceTransform(x(t), t, s), LaplaceTransform(x(t), t, s)),
     (LaplaceTransform(Derivative(x(t), t), t, s), s*LaplaceTransform(x(t), t, s)),
     (LaplaceTransform(Derivative(x(t), (t, 2)), t, s),
      s**2*LaplaceTransform(x(t), t, s)),
     (LaplaceTransform(u(t), t, s), LaplaceTransform(u(t), t, s)),
     (LaplaceTransform(Derivative(u(t), t), t, s), s*LaplaceTransform(u(t), t, s)),
     (LaplaceTransform(Derivative(u(t), (t, 2)), t, s),
      s**2*LaplaceTransform(u(t), t, s))]

```python
# функция для замены в соответствии со свойством выше
def LSubs(LEq):
    return sympy.together(LEq.subs(subl))
```

```python
#Произведём замену в уравнении и приведём подобные
LEq22 = LSubs(LEq22)
LEq22
```

$\displaystyle \left(T s + 1\right) \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

Решим уравнение относительно $L \left[ x(t) \right]$:

```python
Sol22 = LSolve(LEq22)
dispEq(makelatex(L(x)), makelatex(Sol22))
```

$\displaystyle \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right)=\frac{k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)}{T s + 1}$

Найдём передаточную функцию апериодического звена $\mathsf{H}_{(2.2)}(s)$:

```python
H22 = H(Sol22)
dispEq('\mathsf{H}_{(2.2)}(s)',makelatex(H22))
```

$\displaystyle \mathsf{H}_{(2.2)}(s)=\frac{k}{T s + 1}$

Найдём весовую функцию, учитывая, что $\mathcal{L}\left[\delta(t)\right](s) \equiv 1$ и, следовательно, $\mathsf{X}(s) \equiv \mathsf{H}(s)$:

```python
h22 = h(H22)
dispEq('h_{(2.2)}(t)', makelatex(h22))
```

$\displaystyle h_{(2.2)}(t)=\frac{k e^{- \frac{t}{T}} \theta\left(t\right)}{T}$

    Построим её график при $k=1$ и целых значениях $T \in [1,5]$:

```python
N = 6
subs_list = [{k:1, T:i} for i in range(1,N)]
labels = ['T = '+str(i) for i in range(1,N)]
ls = np.linspace(0.0001, 10)
fig, ax = plt.subplots()
ax.set_title('$h_{(2.2)}(t)$')
plot_sym(h22, subs_list, ax, ls, labels)

```

    Найдём переходную функцию, учитывая, что $\mathcal{L}\left[\theta(t)\right](s) = \frac{1}{s}$:

```python
pi22 = pi(H22)
dispEq('\pi_{(2.2)}(t)',makelatex(pi22))
```

$\displaystyle \pi_{(2.2)}(t)=k \theta\left(t\right) - k e^{- \frac{t}{T}} \theta\left(t\right)$

    Построим её график при $k=1$ и целых значениях $T \in [1,5]$:

```python
N = 6
subs_list = [{k:1, T:i} for i in range(1,N)]
labels = ['T = '+str(i) for i in range(1,N)]
ls = np.linspace(0.0001, 10)
fig, ax = plt.subplots()
ax.set_title('$\pi_{(2.2)}(t)$')
plot_sym(pi22, subs_list, ax, ls, labels)

```

#### Физичные примеры апериодического звена

Примерами апериодического звена являются пассивные R-L или R−C цепочки:

1) R-C цепочка

Рис. 2.1

    Напряжение на выходе
    $$
    y = \frac{q}{C}
    $$
    где $q$ — заряд на конденсаторе.

Напряжение на входе
$$x = IR + \frac{q}{C} = \dot{q}R + y = RC \dot{y} + y$$

$$ RC \dot{y} +y = x $$

Получено уравнение апериодического звена, где $T = RC,\space k = 1$.

2) L-R цепочка

Рис. 2.2

    Напряжение на выходе
    $$
    y = -IR
    $$
    где $I$ — сила тока в цепи.

Напряжение на входе
$$x = IR +L \dot{I} = \frac{L}{R}\dot{y} + y$$

$$ \frac{L}{R} \dot{y} +y = x $$

Получено уравнение апериодического звена, где $T = \frac{L}{R},\space k = 1$.

## Интегрирующие звенья

Главная особенность таких звеньев состотит в том, что дифференциальное уравнение, описывающее процесс, содержит производные от сигнала на выходе и не содержит самого сигнала:

$$
\sum_{i=1}^{n} T_i \frac{\mathrm{d}^i x}{\mathrm{d} t^i}(t) =
\sum_{j=0}^{m} k_i \frac{\mathrm{d}^i u}{\mathrm{d} t^i}(t) + k_{m+1}f(t)
$$

Простейшим примером интегрирующего звена является идеально интегрирующее звено:

$$
\dot{x} = ku \tag{3}
$$

Построим для этого звена передаточную, весовую и переходную функции.

Определим уравнение (3)

```python
# левая часть
Left3 = x.diff(t)
# правая часть
Right3 = k*u
# уравнение
dispEq(makelatex(Left3), makelatex(Right3))
```

$\displaystyle \frac{d}{d t} x{\left(t \right)}=k u{\left(t \right)}$

Применим к обеим частям преобразование Лапласа

```python
LEq3 = LEq(Left3, Right3)
LEq3
```

$\displaystyle \mathcal{L}_{t}\left[\frac{d}{d t} x{\left(t \right)}\right]\left(s\right) = k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

Произведём  в полученных символьных выражениях замену в соответствии со свойствами преобразования Лапласа.

```python
#Произведём замену в уравнении и приведём подобные
LEq3 = LSubs(LEq3)
LEq3
```

$\displaystyle s \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

Решим уравнение относительно $L \left[ x(t) \right]$:

```python
Sol3 = LSolve(LEq3)
dispEq(makelatex(L(x)), makelatex(Sol3))
```

$\displaystyle \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right)=\frac{k \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)}{s}$

Найдём передаточную функцию идеально интегрирующего звена $\mathsf{H}_{(3)}(s)$:

```python
H3 = H(Sol3)
dispEq('\mathsf{H}_{(3)}(s)',makelatex(H3))
```

$\displaystyle \mathsf{H}_{(3)}(s)=\frac{k}{s}$

Найдём весовую функцию, учитывая, что $\mathcal{L}\left[\delta(t)\right](s) \equiv 1$ и, следовательно, $\mathsf{X}(s) \equiv \mathsf{H}(s)$.
Построим её график.

```python
h3 = h(H3)
dispEq('h_{(3)}(t)', makelatex(h3))
```

$\displaystyle h_{(3)}(t)=k \theta\left(t\right)$

Весовая функция идеально инегрирующего звена представляет собой функцию Хевисайда, умноженную на коэффициент передачи.

    Найдём переходную функцию, учитывая, что $\mathcal{L}\left[\theta(t)\right](s) = \frac{1}{s}$:

```python
pi3 = pi(H3)
dispEq('\pi_{(3)}(t)',makelatex(pi3))
```

$\displaystyle \pi_{(3)}(t)=k t \theta\left(t\right)$

    Построим её график при целых значениях $k\in [1,5]$:

```python
N = 6
subs_list = [{k:i} for i in range(1,N)]
labels = ['k = '+str(i) for i in range(1,N)]
ls = np.linspace(0.0001, 10)
fig, ax = plt.subplots()
ax.set_title('$\pi_{(3.1)}(t)$')
plot_sym(pi3, subs_list, ax, ls, labels)

```

#### Физичные примеры идеально интегрирующего звена

Примерами идеально интегрирующего звена являются интегрирующие R-L или R−C цепочки:

1) интегрирующая R-C цепочка

Рис. 3.1

В этой цепочке входное напряжение подаётся на сопротивление, а выходное снимается с ёмкости.

    Напряжение на выходе
    $$
    y = \frac{q}{C}
    $$
    где $q$ — заряд на конденсаторе.

Напряжение на входе:
$$x = IR = \dot{q}R = RC \dot{y}$$

$$ \dot{y} = \frac{x}{RC} $$

Получено уравнение идеально интегрирующего звена, где $k = \frac{1}{RC}$.

Такой же результат может быть получен при $x = E_{1}$ и $RC\dot{y} \gg y$:

$$ RC \dot{y} +y = x  \Rightarrow RC\dot{y} = x$$

2) интегрирующая L-R цепочка

Рис. 3.2

В этой цепочке входное напряжение подаётся на индуктивность, а выходное снимается с сопротивления.

    Напряжение на выходе
    $$
    y = -IR
    $$
    где $I$ — сила тока в цепи.

напряжение на входе:
$$x = - L \dot{I} = \frac{L}{R}\dot{y}$$

$$ \dot{y}=  \frac{R}{L}x $$

Получено уравнение идеально интегрирующего звена, где $ k = \frac{R}{L}$.

Такой же результат может быть получен при $x = E_{вх}$ и $\frac{L}{R}\dot{y} \gg y$:

$$ \frac{L}{R} \dot{y} +y = x  \Rightarrow \frac{L}{R}\dot{y} = x$$

## Дифференцирующие звенья

Звенья такого типа описываются уравнением вида (1) с $k_0 = 0:$

$$
\sum_{i=0}^{n} T_i \frac{\mathrm{d}^i x}{\mathrm{d} t^i}(t) =
\sum_{j=1}^{m} k_i \frac{\mathrm{d}^i u}{\mathrm{d} t^i}(t) + k_{m+1}f(t)
$$

Простейшим примером дифференцирующего звена является идеально дифференцирующее звено:

$$
x = k\dot{u} \tag{4}
$$

Построим для этого звена передаточную, весовую и переходную функции.

Определим уравнение (4)

```python
# левая часть
Left4 = x
# правая часть
Right4 = k*u.diff(t)
# уравнение
dispEq(makelatex(Left4), makelatex(Right4))
```

$\displaystyle x{\left(t \right)}=k \frac{d}{d t} u{\left(t \right)}$

Применим к обеим частям преобразование Лапласа

```python
LEq4 = LEq(Left4, Right4)
LEq4
```

$\displaystyle \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = k \mathcal{L}_{t}\left[\frac{d}{d t} u{\left(t \right)}\right]\left(s\right)$

Произведём  в полученных символьных выражениях замену в соответствии со свойствами преобразования Лапласа.

```python
#Произведём замену в уравнении и приведём подобные
LEq4 = LSubs(LEq4)
LEq4
```

$\displaystyle \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = k s \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

Решим уравнение относительно $L \left[ x(t) \right]$:

```python
Sol4 = LSolve(LEq4)
dispEq(makelatex(L(x)), makelatex(Sol4))
```

$\displaystyle \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right)=k s \mathcal{L}_{t}\left[u{\left(t \right)}\right]\left(s\right)$

Найдём передаточную функцию идеально дифференцирующего звена $\mathsf{H}_{(4)}(s)$:

```python
H4 = H(Sol4)
dispEq('\mathsf{H}_{(4)}(s)',makelatex(H4))
```

$\displaystyle \mathsf{H}_{(4)}(s)=k s$

Найдём весовую функцию идеально дифференцирующего звена, учитывая, что $\mathcal{L}\left[\delta(t)\right](s) \equiv 1$ и, следовательно, $\mathsf{X}(s) \equiv \mathsf{H}(s)$.

```python
h4 = h(H4)
dispEq('h_{(4)}(t)', makelatex(h4))

```

$\displaystyle h_{(4)}(t)=k \mathcal{L}^{-1}_{s}\left[s\right]\left(t\right)$

Так как $\mathcal{L}^{-1}[s](t) = \delta'(t)$, весовая функция идеально дифференцирующего звена

$$h_{(4.1)} = k \delta'(t)$$

где $\delta'(t-a)$ определяется равенством

$$
\int_{-\infty}^{+\infty} f(t)\delta'(t-a) dt = - \int_{-\infty}^{+\infty} f'(t) \delta(t-a) dt = -f'(a)
$$

    Найдём переходную функцию идеально дифференцирующего звена, учитывая, что $\mathcal{L}\left[\theta(t)\right](s) = \frac{1}{s}$:

```python
pi4 = pi(H4)
dispEq('\pi_{(4)}(t)',makelatex(pi4))
```

$\displaystyle \pi_{(4)}(t)=k \delta\left(t\right)$

    Переходная функция идеально дифференцирующего звена представляет собой дельта-фукнкцию, умноженную на коэффициент передачи $k$.

#### Физичные примеры идеально дифференцирующего звена

Примерами идеально дифференцирующего звена являются дифференцирующие R-L или R−C цепочки:

1) дифференцирующая R-C цепочка

Рис. 4.1

В этой цепочке входное напряжение подаётся на ёмкость, а выходное снимается с сопротивления.

    Напряжение на входе:
$$ x = \frac{q}{C}$$

 где $q$ — заряд конденсатора.

Напряжение на выходе
$$
y = IR = \dot{q}R = RC \dot{x}
$$
где $I$ — сила тока в цепи.
$$ y = RC\dot{x}$$

Получено уравнение идеально дифференцирующего звена, где $k = RC$.

Такой же результат может быть получен при $y = E_{1}$ и $RC\dot{x} \gg x$:

$$ RC \dot{x} +x = y  \Rightarrow RC\dot{x} = y$$

2) дифференцирующая L-R цепочка

Рис. 4.2

В этой цепочке входное напряжение подаётся на сопротивление, а выходное снимается с индуктивности.

    Напряжение на выходе
    $$
    y = -IR
    $$
    где $I$ — сила тока в цепи.

напряжение на индуктивности:
$$x = L \dot{I} = \frac{L}{R}\dot{y}$$

$$ \dot{y}=  \frac{R}{L}x $$

Получено уравнение идеально дифференцирующего звена, где $ k = \frac{R}{L}$.

Такой же результат может быть получен при $x = E_{вх}$ и $\frac{L}{R}\dot{y} \gg y$:

$$ \frac{L}{R} \dot{y} +y = x  \Rightarrow \frac{L}{R}\dot{y} = x$$

Lecture Jupyter Notebooks/Lecture 4 Elementary Blocks of Control Systems.ipynb


# Moran processes - solutions

1. Give the transition probabilities for the Moran process with neutral drift.

   Bookwork: https://vknight.org/gt/chapters/12/#Moran-process-with-neutral-drift

2. Obtain the transition probability matrix for the Moran process with neutral drift with $N=4$ individuals.

$$
M = \begin{pmatrix}
   1&0&0&0&0\\
   3/16&5/8&3/16&0&0\\
   0&1/4&1/2&1/4&0\\
   0&0&3/16&5/8&3/16\\
   0&0&0&0&1\\
\end{pmatrix}
$$

`3`. State and prove the theorem for fixation probabilities in a birth death process.

   Bookwork: https://vknight.org/gt/chapters/12/#Theorem:-Fixation-probabilities-for-the-birth-death-process

`4`. Extend the formulae of question 3 to the case of a Moran process on a game.

   Boookwork: https://vknight.org/gt/chapters/12/#Moran-process-on-a-game

`5`. For the following games, obtain the fixation probabilities for $N=4$:

   1. $A=\begin{pmatrix}1 & 1 \\ 1 & 1\end{pmatrix}$

      This game corresponds to neutral drift, thus:

      $x_1=1/N=1/4$ (for both strategies).

   2. $A=\begin{pmatrix}1 & 2 \\ 3 & 1\end{pmatrix}$

      Assuming $i$ individuals of the first type, for this game we have $N=4$ and $(a, b, c, d)=(1, 2, 3, 1)$ the fitness of both types is given respectively by:

      $$f_{1i}=\frac{a(i-1)+b(N-i)}{N-1}=\frac{7-i}{3}$$
      $$f_{2i}=\frac{c(i)+d(N-i-1)}{N-1}=\frac{2i+3}{3}$$

      which gives:

      $$\gamma_i=\frac{f_{2i}}{f_{1i}}=\frac{2i+3}{7-i}$$

      thus:

      $$
      x_1=\frac{1}{1+\sum_{j=1}^{3}\prod_{k=1}^j\frac{2k+3}{5-k}}=\frac{1}{1+5/6+(5/6)(7/5)+(5/6)(7/5)(9/4)}=\frac{8}{45}
      $$

      Some code to verify this:

```python
import sympy as sym
import numpy as np
def theoretic_fixation(N, game, i=1):
    """
    Calculate x_i as given by the above formula
    """
    f_ones = np.array([(game[0, 0] * (i - 1) + game[0, 1] * (N - i)) / (N - 1) for i in range(1, N)])
    f_twos = np.array([(game[1, 0] * i + game[1, 1] * (N - i - 1)) / (N - 1) for i in range(1, N)])
    gammas = f_twos / f_ones
    return (1 + np.sum(np.cumprod(gammas[:i-1]))) / (1 + np.sum(np.cumprod(gammas)))
```

```python
game = np.array([[sym.S(1), sym.S(2)], [sym.S(3), sym.S(1)]])
theoretic_fixation(N=4, game=game)
```

    8/45

`6`. Consider the game $A=\begin{pmatrix}r & 1 \\ 1 & 1\end{pmatrix}$ for $r>1$, and obtain $x_1$ as a function of $r$ and $N$. How does $r$ effect the chance of fixation?

Assuming $i$ individuals of the first type, for this game we have $N=4$ and $(a, b, c, d)=(r, 1, 1, 1)$ the fitness of both types is given respectively by:

$$f_{1i}=\frac{r(i-1)+(N-i)}{N-1}$$
$$f_{2i}=\frac{(i)+(N-i-1)}{N-1}=1$$

which gives:

$$\gamma_i=\frac{f_{2i}}{f_{1i}}=\frac{N-1}{r(i-1)+(N-i)}$$

thus:

$$
  x_1=\frac{1}{1+\sum_{j=1}^{N-1}\prod_{k=1}^j\frac{N-1}{r(k-1)+(N-k)}}
$$

Note that as $r$ increases the denominator of $x_1$ decreases (each term in the product decreases), thus $x_1$ increases as a function of $r$ (as expected).


# Multi-armed bandits

```python
import numpy as np
np.set_printoptions(precision=3)

import pandas as pd
```

## Multi-armed bandits

The classic multi-armed bandit is illustrated below.
The agent (the octopus) must choose which "arm" to pull.
Each slot machine (aka one-armed bandit) has a different, but unknown, payout rate (reward), denoted $R(a)$.

* In a MAB problem, the goal is to choose a sequence of actions (arm pulls) that maximizes the sum of expected rewards:
$$
a_{1:T} = \arg \max_{a_{1:T}} \sum_{t=}^T E_{a_t \sim \pi}[R(a_t | \theta)]
$$
where $T$ is the **finite horizon**, $\pi()$ is the (stochastic) **policy**, $R(a|\theta)$ is the reward function for action $a$ using model parameters $\theta$.
* In the **infinite horizon** case, we use a **discount factor** $\gamma < 1$ to ensure the sum converges:
$$
J = \sum_{t=1}^\infty \gamma^{t-1} E_{ a_t \sim \pi}[R(a_t | \theta)]
$$
* For binary rewards, we can use a **Bernoulli bandit**
$$
p(R|a) = \text{Ber}(R|\mu_a)
$$
where $\mu_a=E[R|a]=p(R=1|a)$.
* For real-valued rewards, we can use a **Gaussian bandit**
$$
p(R|a) = \text{Gauss}(R|\mu_a, \sigma_a^2)
$$

## Contextual bandits

* In **contextual bandits**, the agent is in a random state $s_t$ at each step. States are drawn iid from some unknown distribution $p(s_t)$. The policy should now depend on the inputs, $a_t \sim \pi(s_t)$.
* The goal is to maximimize
$$
J = \sum_{t=1}^\infty \gamma^{t-1} E_{ s_t \sim p(s), a_t \sim \pi(a|s_t)}[R(s_t, a_t | \theta)]
$$
* There are numerous applications, e.g.
** **Online advertising**, where the action specifies which ad to show, and the state specifies user features (e.g., search history) and/or web page features (e.g., content).
** **Recommender systems**, where the action specifies which (set of) item(s)  to show, and the state specifies user features.
** **Clinical trials**, where the action specifies which drug to give, and the state specifies patient features.
* For binary rewards, we can use logistic regression model
$$
p(R|s,a;\theta) = \text{Ber}(R|\sigma(\theta^T \phi(s,a)))
$$
where $\phi(s,a)$ is a feature vector, possibly computed using a deep neural network, and $\sigma(x)=1/(1+e^{-x})$ is the sigmoid or logistic function.
* For real-valued rewards, we can replace the sigmoid output with a linear output, and use a Gaussian observation model.
* A key problem is that the rewards for each (state,action) pair (represented by model parameters $\theta$) are unknown.
* The agent can infer the parameters in an online (recursive) fashion given the data stream $D_{1:t} = (s_{1:t}, a_{1:t}, r_{1:t})$ using **sequential Bayesian updating**:
$$
\begin{align}
p(\theta|D_{1:t}) &=
p(\theta|D_t, D_{1:t-1})
= \frac{p(D_t|\theta,D_{1:t-1}) p(\theta|D_{1:t-1})}
{p(D_t|D_{1:t-1})}
\propto p(D_t|\theta) p(\theta|D_{1:t-1})
\end{align}
$$
* The likelihood is $p(D_t|\theta)=p(r_t|s_t, a_t,\theta)$ and the prior is the belief state from the previous time step, $p(\theta|D_{1:t-1})$.

## Exploration-exploitation tradeoff

* The agent must try each (state, action) pair (possibly multiple times) in order to collect data, but these experiments incur an opportunity cost. This is called the **exploration-exploitation tradeoff**.
* A common heuristic solution is the **upper confidence bound** method, which selects the action according to
$$
a_t = \arg \max_a E[R(s_t,a;\theta)] + c \;\text{std}[R(s_t,a;\theta)]
$$
* The  constant $c \geq 0$ reflects the amount of optimism about unknown rewards, and hence the amount of exploration.
* Below we illustrate the distribution over possible rewards for  a 3-armed bandit (without contextual features).

## Reinforcement learning

* In **reinforcement learning**, the setup is similar to a contextual bandit, except now the current state depends on the previously visited states, as well as the actions that the agent has taken.
* If we assume a Markov model for the enviroment, we get a **Markov decision process** or **MDP**.
$$
J = \sum_{t=1}^\infty \gamma^{t-1} E_{ s_t \sim p(s|s_{t-1}, a_{t-1}; \theta), a_t \sim \pi(a|s_{1:t})}[R(s_t, a_t | \theta)]
$$
* Solving for the optimal policy when the parameters $\theta$ of the environment are unknown is challenging. This is beyond the scope of this tutorial.

```python

```


This notebook supplements the manuscript "Data-driven modeling reveals a universal dynamic underlying the COVID-19 pandemic under social distancing" by Robert Marsland III and Pankaj Mehta.

In this work, we show that the cumulative fatalities $N(t)$ for every region with more than 500 deaths as of April 15, 2020 is well-described by the function:
\begin{align}
N(t) = N_{\rm max}\Phi\left(\frac{t-t_h}{\sigma}\right),
\end{align}
where
\begin{align}
\Phi(t) &= {1 \over \sqrt{2\pi}} \int_{-\infty}^t e^{-x^2/2}\,dx
\end{align}
is the cumulative normal distribution. The same functional form provides a good fit to the cumulative number of confirmed cases for most countries with sufficient data. This model has two key parameters, the final total number of fatalities (or cases) $N_{\rm max}$, and the infection timescale $\sigma$. The third parameter $t_h$, sets the overall location of the infection curve on the time axis.

The first cell of this notebook loads the current case and fatality data from the [Johns Hopkins github repository](https://github.com/CSSEGISandData/COVID-19) along with the best-fit values and confidence bounds for the three fitting parameters using data through April 15, 2020, and a set of functions for fitting, analyzing and simulating the data. Example code is provided for plotting the predictions, making new predictions and running simulations.

We emphasize that the confidence intervals shown here are lower bounds on the true amount of uncertainty. These intervals are calculated under the assumption that the data is accurately described by the fitting function given above, with no changes in parameter values, and with all deviations from this function due to random multiplicative noise. The confidence intervals do not account for possible future changes in social policy, or for systematic deviations from the fitting function that may arise at late times.

# Plot predictions

To plot the live data with the predictions made on April 15, replace the arguments of the "plot_region" function with the country and province/state of interest. Country-level data is accessed by using 'NaN' for the second argument. Dashed line marks April 15, the date the predictions were made.

```python
from covid_plotting import *
plot_region('Sweden','NaN',forecast_days=30)
```

Changing the horizontal axis to a log scale allows you to see the power law behavior at early times:

```python
plot_region('Sweden','NaN',forecast_days=30,log_scale=True)
```

To access the parameters for a given region, replace the two strings in the cells below with the country and state/province of interest.

For fatalities, use:

```python
predictions_deaths_apr15.loc['Sweden','NaN']
```

    Nmax                      2007.23
    Nmax_low                  1499.11
    Nmax_high                 3433.77
    sigma                     9.88884
    sigma_low                 8.60252
    sigma_high                11.9405
    th            2020-04-12 00:00:00
    th_low        2020-04-09 00:00:00
    th_high       2020-04-18 00:00:00
    Name: (Sweden, NaN), dtype: object

For cases, use:

```python
predictions_cases_apr15.loc['Sweden','NaN']
```

    Nmax                      49328.6
    Nmax_low                  33886.3
    Nmax_high                 85215.3
    sigma                     21.8862
    sigma_low                 19.7903
    sigma_high                24.6654
    th            2020-04-29 00:00:00
    th_low        2020-04-21 00:00:00
    th_high       2020-05-10 00:00:00
    Name: (Sweden, NaN), dtype: object

The complete spreadsheets can also be downloaded from the "output" folder of [the github repository](https://github.com/Emergent-Behaviors-in-Biology/covid19).

# Make new predictions

The cell below allows you to fit the full dataset including the latest data, for all countries and regions. It generates Pandas dataframes called `params_cases` and `params_deaths`, which contain the best-fit parameter values along with the RMS deviation of the data from the fitting function in log space ("score"). These parameters can be used as the starting point for generating confidence intervals below. The parameter `p0` sets a threshold for discarding data from the beginning of the pandemic in each region, in order to focus on the later, more universal part of the curve. All timepoints with fewer cumulative cases/fatalities than `p0` are discarded for the purposes of fitting.

```python
from covid_plotting import *
params_deaths = fit_all(deaths.iloc[:-1],plot=False,p0=50)
params_cases = fit_all(cases.iloc[:-1],plot=False,p0=5e2)
```

We can use the best-fit parameters to perform an updated data collapse:

```python
#params_deaths.loc['China','Hubei'] = params_china #Fitting Hubei fatality data only up through Feb 25 gives better collapse
plot_collapse(params_deaths,params_cases)
```

We can now look at the updated predictions for regions of interest, computing the 95 percent confidence interval just for the region plotted, following the method described in the manuscript. As noted above, the calculation underestimates the true amount of uncertainty, since it assumes that the data is accurately described by the fitting function given above, with no changes in parameter values, and with all deviations from this function due to random multiplicative noise.

```python
country = 'US'
province = 'Massachusetts'

predictions_deaths = predict_all(deaths,params_deaths.loc[[(country,province)]],p0=50)
predictions_cases = predict_all(cases,params_cases.loc[[(country,province)]],p0=500)
plot_region(country,province,forecast_days=30,new_predictions=(predictions_deaths,predictions_cases))
```

The next cell computes 95 percent confidence intervals for all countries and regions. The best-fit parameters and upper/lower bounds on the confidence interval are contained in the Pandas dataframes `predictions_deaths` and `predictions_cases`. Set `verbose` to `False` if you do not want each country's name to be printed as its predictions are generated. Note that this cell takes about 10 minutes to run.

```python
predictions_deaths = predict_all(deaths,params_deaths,p0=50,verbose=True)
predictions_cases = predict_all(cases,params_cases,p0=500,verbose=True)
```

    Sweden, NaN
    Sweden, NaN

# Run simulation

The following cell simulates disease spreading on a NetworkX graph `G` of your choice. When a node is infected, waiting times $T_g$ are randomly drawn for all neighboring nodes, and each neighbor is infected a time $T_g$ after the original node, if it is not already infected. The default distribution for drawing $T_g$ is a Gamma distribution, which is guaranteed to yield positive values. We parameterize the distribution by the mean $\mu_G$ and standard deviation $\sigma_G$. The Gamma distribution reduces to the exponential distribution (i.e., a Markovian spreading model) when these parameters are chosen such that $\mu_G = \sigma_G$.

```python
from covid_plotting import *

#Choose graph parameters
N=1000  #Number of nodes
#gamma = 4 #For scale-free graphs
#k = 1.5 #Mean degree for Erdos-Renyi
R0 = 1.5 #for fully connected graphs, we set the probability of infection to p = R0/N
p = R0/N

#Make the graph
#G = nx.expected_degree_graph(nx.utils.powerlaw_sequence(N, gamma), selfloops=False)
G = nx.gnp_random_graph(N, p)
#G = nx.complete_graph(N)

#Set up waiting time parameters
muG = 4 #Mean waiting time
sigG = 4 #standard deviation of waiting time

#Simulate
t,cum_cases = simulate_pandemic_edges(G,muG,sigG,tmax=80)

#Format like real data, and use same fitting script
t_dt = pd.to_datetime([datetime.today()-timedelta(days=int(np.max(t))-k) for k in range(int(np.max(t))+1)])
sim_data = pd.Series(cum_cases,index=t_dt)
th,logNmax,sigma,score = fit_erf(sim_data,p0=50)[0]

#Process output and plot
tau = ((t_dt-tref)/timedelta(days=1) - th)/sigma
Nmax = np.exp(logNmax)
fig,ax=plt.subplots()
ax.semilogy(t_dt,sim_data.values,'o-',label='Simulation data',ms=4)
ax.semilogy(t_dt,Nmax*norm.cdf(tau),label='Fit')
ax.xaxis.set_minor_locator(mdates.DayLocator())
ax.xaxis.set_major_formatter(mdates.DateFormatter('%m-%d'))
ax.set_xlabel('Date (mm-dd)')
ax.set_ylim((4,N))
ax.set_ylabel('Cumulative Cases')
fig.autofmt_xdate()
ax.legend()
plt.show()

#Also plot on log-log axes
fig,ax=plt.subplots()
ax.loglog(t,sim_data.values,'o-',label='Simulation data',ms=4)
ax.loglog(t,Nmax*norm.cdf(tau),label='Fit')
ax.set_xlabel('Elapsed Time')
ax.set_ylim((4,N))
ax.set_ylabel('Cumulative Cases')
ax.legend()
plt.show()
```

```python
#Also plot on log-log axes
fig,ax=plt.subplots()
ax.loglog(t+15,sim_data.values,'o-',label='Simulation data',ms=4)
ax.loglog(t+15,Nmax*norm.cdf(tau),label='Fit')
ax.set_xlabel('Elapsed Time')
ax.set_ylim((4,N))
ax.set_ylabel('Cumulative Cases')
ax.legend()
plt.show()
```

# Check confidence interval

```python
country = 'US'
region = 'Florida'
p0 = 50

data = deaths[country,region]

#Fit the data
params,parm0,s = fit_erf(data,p0=p0)

train = pd.to_numeric(data.loc[data>p0])
train.index=(train.index-tref)/timedelta(days=1)
sig_xi2 = params[-1]

fig,ax=plt.subplots(figsize=(4,3))
fig.subplots_adjust(bottom=0.2,left=0.2)
c = 0.95
M = len(train)
params_sweep = sweep_sigma(params,data,p0)
sigma, prob,scoremax = get_score_thresh(params_sweep,M,0.95)
ax.plot(sigma,prob,'k')
score = params_sweep[1:,3]
ax.fill_between(sigma[score<=scoremax],prob[score<=scoremax])

sigma = params_sweep[:,2]
dsig = np.diff(sigma)
sigma = sigma[1:]
pcut = np.exp(-scoremax*M/(2*sig_xi2))/(np.exp(-score*M/(2*sig_xi2))*dsig).sum()
ax.plot([sigma[0],sigma[-1]],[pcut,pcut],'k--')

ax.set_xlabel(r'$\sigma$')
ax.set_ylabel(r'$p(\sigma|N_i)$')
ax.set_title(country+', '+region)
plt.show()
```

```python

```


Este trabalho está licenciado sob a Licença Atribuição 4.0 Internacional Creative Commons. Para visualizar uma cópia desta licença, visite http://creativecommons.org/licenses/by/4.0/ ou mande uma carta para Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

Autor/Instituição: Pedro H A Konzen / UFRGS

pedro.konzen@ufrgs.br

Data: Out/2015

# Cálculo com Python

Nesta terceira parte do curso vamos estudar como usar o python para calcular derivadas de funções de uma variável.

## 3 - Derivadas

Aqui você irá aprender a usar a biblioteca [Python][py] de matemática simbólica [Sympy][sy] para calcular:

+ derivadas

+ retas tangentes

+ derivadas de ordem mais altas

+ máximos e mínimos

[py]:https://www.python.org/
[sy]:http://www.sympy.org/en/index.html

Começamos carrendo a biblioteca [Sympy][sy]:
[sy]:http://www.sympy.org/en/index.html

```python
from __future__ import division
from sympy import *
init_printing()
x, y = symbols('x y') #define x e y como variáveis simbólicas.
```

Para fixar as ideias, vamos continuar trabalhando com a função:

$f(x) = (x^3 - 3x + 2)e^{-x/4} - 1$.

```python
def f(x): return (x**3 - 3*x + 2)*exp(-x/4) - 1
f(x)
```

### Derivadas

Vamos ver como podemos usar o Sympy para calcularmos a derivada da função $f(x)$, i.e. $f'(x)$. Para tanto, usamos a função [$\verb+diff+$][diff]:
[diff]:http://docs.sympy.org/latest/tutorial/calculus#Derivatives.html

```python
diff(f(x),x)
```

Para avaliar a derivada em um ponto, por exemplo, para calcular $f'(1)$, digitamos:

```python
diff(f(x),x).subs(x,1)
```

#### Exercício:

Sendo $g(x) = x^2 + \frac{1}{2}$ calcule $g'(1)$.

```python
#digite sua solução aqui!
```

### Retas Tangente

Aqui, vamos ver como calcular a reta tangente ao gráfico da função $f(x)$ no ponto $x_0 = -\frac{1}{2}$. Lembremos que tal reta terá equação:

$y = f'(x_0)(x - x_0) + f(x_0)$.

Desta forma, vamos calcular $f'(x_0)$:

```python
x0 = -1/2
fl = diff(f(x),x).subs(x,-0.5) #f'(x_0)
print("coef. angular = ")
fl
```

Agora, já podemos definir a reta tangente:

```python
def r(x) : return fl*(x-x0) + f(x0)
print("equação da reta tangente y=")
r(x)
```

Vejamos os gráficos de $f(x)$ e da reta tangente calculada.

```python
%matplotlib inline
p1 = plot(f(x),(x,-2,2),show=False,line_color='b')
p2 = plot(r(x),(x,-1.5,1),show=False,line_color='r')
p1.extend(p2)
p1.show()
```

$\blacktriangleleft$

#### Exercício:

Encontre a reta tangente ao gráfico de $y = \frac{1}{x}$ em $x=1$. Faça os esboços dos gráficos da função e da reta tangente em um mesmo gráfico.

```python
#digite a resolução aqui.
```

### Derivadas de ordem mais altas

Similarmente a derivadas de primeira ordem, as derivadas de ordem mais alta podem ser obtidas usando-se a função [$\verb+diff+$][diff]. Por exemplo, para calcularmos:

$\frac{d^2}{d x^2}f(x)$

podemos digitar:

[diff]:http://docs.sympy.org/latest/tutorial/calculus#Derivatives.html

```python
diff(f(x),x,2)
```

#### Exercício

Calcule:

$\frac{d^3}{d x^3}\left(\frac{x^2 - 1 + \text{sen}\,x}{x^3 - 3x + 1}\right)$.

```python
#digite sua resposta aqui!
```

### Máximos e Mínimos Locais

Agora, vamos ver como podemos usar o que aprendemos até aqui para resolver problemas de máximos e mínimos. Nosso objetivo será de encontrar e determinar os pontos de máximos e mínimos locais da função $f(x)$. Para tanto, vamos usar o teste da segunda derivada.

Comecemos lembrando do gráfico de $f(x)$:

```python
plot(f(x),(x,-2,2))
plot(f(x),(x,2,40))
plot(f(x),(x,40,60))
```

Observando os gráficos, vemos que os pontos críticos de $f(x)$ têm máximos locais próximos ao ponto $x=-1$ e $x=10$ e um mínimo local próximo do ponto $x=1$. Logo, podemos usar estes valores como parâmetro da função [$\verb+nsolve+$][nsolve]:
[nsolve]:http://docs.sympy.org/latest/modules/solvers/solvers.html?highlight=nsolve#sympy.solvers.solvers.nsolve

```python
fl = diff(f(x),x) #calcula $f'(x)$
x1 = nsolve(fl,x,-1) #calcula ponto crítico próx. de x=-1
x2 = nsolve(fl,x,1) #calcula ponto crítico próx. de x=1
x3 = nsolve(fl,x,10) #calcula ponto crítico próx. de x=10
print("x1=",x1)
print("x2=",x2)
print("x3=",x3)
```

Ok. Acamos de encontrar os pontos críticos de $f(x)$ (podem haver outros pontos críticos?). Por inspesão gráfica, vemos que $x_1 \approx -1,15$ e $x_3 \approx 12,15$ são os pontos de máximos locais de $f(x)$ e $x_3 = 1,0$ é o ponto de mínimo (global?).

Vamos confirmar isso com o teste da segunda derivada. Observe:

```python
diff(f(x),x,2).subs(x,x1) #calcula f''(x1)
```

```python
diff(f(x),x,2).subs(x,x2) #calcula f''(x2)
```

```python
diff(f(x),x,2).subs(x,x3) #calcula f''(x3)
```

#### Exercício

Referente ao problema de otimização que acabamos de discutir, responda:

(a) Os pontos $x_1$, $x_2$ e $x_3$ calculados acima são os únicos pontos críticos de $f(x)$? Justifique sua resposta.

(b) Quais são os pontos de mínimo e máximo globais de $f(x)$?

(c) Qual é o maior valor que $f(x)$ assume? E qual é o menor?

```python
#digite sua solução aqui!
```

## Referências

+ [1] Python Software Fundation (US): [www.python.org][python]

+ [2] Sympy - Python Library for Symbolic Mathematics: [www.sympy.org][sympy]

## Related links

+ [1] Sympy Gamma: [www.sympygamma.org][sympygamma]
+ [2] Sympy Live: [live.sympy.org][sympylive]

[python]: https://www.python.org/
[sympy]: http://www.sympy.org/en/index.html
[sympygamma]: http://www.sympygamma.com/
[sympylive]: http://live.sympy.org/

#### Próxima parte: 4 - Integral

```python

```


```python
import matplotlib.pyplot as plt
import matplotlib.patches as patches

import oeis

import sys
import sympy
import math
from sympy import *
from sympy.abc import x, n, z, t, k
from sympy.core.cache import *

init_printing() # for nice printing, a-la' TeX

sys.setrecursionlimit(100000)

plt.rcParams['figure.figsize'] = (10.0, 10.0)
```

```python
with assuming(Q.positive(1-x)):
    print(sqrt((1-x)**2))
```

    sqrt((-x + 1)**2)

```python
x = symbols('x', negative=True)
s = sqrt((1-x)**2)
s
```

$$- x + 1$$

---

```python
c = IndexedBase('c')
```

```python
(1*c[n,4]).args
```

$$\left ( c, \quad n, \quad 4\right )$$

```python
eqs=Tuple(#Eq(c[0,0],c[3,3]),
      Eq(c[1,0],c[0,0]),
      Eq(c[1,1],c[0,0]),
      Eq(c[2,0],c[1,0]+c[1,1]),
      Eq(c[2,1],c[1,0]+c[1,1]),
      Eq(c[2,2],c[1,1]),
        Eq(c[3,0],c[2,0]+c[2,1]+c[2,2]),
        Eq(c[3,1],c[2,0]+c[2,1]+c[2,2]),
        Eq(c[3,2],c[2,1]+c[2,2]),
        Eq(c[3,3],c[0,0]),
    )
solve(eqs, eqs.atoms(Indexed) )
      #[c[0,0],c[1,0],c[2,0],c[3,0]], check=True)
      #exclude=[c[3,0],c[3,1],c[3,2],c[3,3]], check=True)
      #[c[0,0],c[1,1],c[2,2],c[3,3]])
      #check=True, implicit=True)
```

```python
solve([#Eq(c[0,0],c[3,3]),
      Eq(c[4,0],c[3,0]+c[3,1]+c[3,2]+c[3,3]),
      Eq(c[4,1],c[3,0]+c[3,1]+c[3,2]+c[3,3]),
      Eq(c[4,2],c[3,1]+c[3,2]+c[3,3]),
      Eq(c[4,3],c[3,2]+c[3,3]),
      Eq(c[4,4],c[3,3]),
    ], check=True
      #[c[0,0],c[1,0],c[2,0],c[3,0]], check=True)
      #exclude=[c[3,0],c[3,1],c[3,2],c[3,3]], check=True)
      #[c[3,0],c[3,1],c[3,2],c[3,3]])
      #check=True, implicit=True)
      )
```

$$\left [ \left \{ c_{3,0} : c_{4,1} - c_{4,2}, \quad c_{3,1} : c_{4,2} - c_{4,3}, \quad c_{3,2} : c_{4,3} - c_{4,4}, \quad c_{3,3} : c_{4,4}, \quad c_{4,0} : c_{4,1}\right \}\right ]$$

```python
solve([#Eq(c[0,0],c[3,3]),
      Eq(c[1,0],c[0,0]),
      Eq(c[1,1],c[0,0])
    ], check=True
      #[c[0,0],c[1,0],c[2,0],c[3,0]], check=True)
      #exclude=[c[3,0],c[3,1],c[3,2],c[3,3]], check=True)
      #[c[3,0],c[3,1],c[3,2],c[3,3]])
      #check=True, implicit=True)
      )
```

$$\left [ \left \{ c_{0,0} : c_{1,1}, \quad c_{1,0} : c_{1,1}\right \}\right ]$$

```python
def catalan_gf(t): return (1-sqrt(1-4*t))/(2*t)
```

```python
catalan_series = catalan_gf(t).series(t, n=15)
```

```python
[catalan_series.coeff(t,n=i) for i in range(15)]
```

$$\left [ 1, \quad 1, \quad 2, \quad 5, \quad 14, \quad 42, \quad 132, \quad 429, \quad 1430, \quad 4862, \quad 16796, \quad 58786, \quad 208012, \quad 742900, \quad 2674440\right ]$$

---

```python
from contextlib import contextmanager

@contextmanager
def bind_Mul_indexed(term, indexed, forbidden_terms=[]):
    coeff_w, ind_w = Wild('coeff', exclude=[indexed] + forbidden_terms), Wild('ind')
    matched = term.match(coeff_w * ind_w)
    # if no indexing happen then isinstance(matched[ind_w], IndexedBase) holds
    if matched and ind_w in matched and isinstance(matched[ind_w], Indexed):
        _, *subscripts = matched[ind_w].args
        yield matched[coeff_w], subscripts
    else:
        raise Exception()

f, n, k = IndexedBase('f'), *symbols('n k')
term = 3 * f[n,k]
try:
    with bind_Mul_indexed(term, f) as (coeff, subscripts):
        print('{} * {}'.format(coeff, subscripts))
except Exception:
    print('something else')

term
```

    3 * [n, k]

$$3 f_{n,k}$$

```python

```

```python
n, k = symbols('n k')
i = IndexedBase('i')
```

```python
a, *b = i.args
a,b
```

$$\left ( i, \quad \left [ \right ]\right )$$

```python
i_w = Wild('i_w')
m = i[n,k].match(i_w)
m[i_w]
```

$$i_{n,k}$$

```python
type(i)
```

    sympy.tensor.indexed.IndexedBase

```python
eq = Eq(2**n * k, 111)
k_constraint = Eq(k, 2*t+1)
eq, k_constraint
```

$$\left ( 2^{n} k = 111, \quad k = 2 t + 1\right )$$

```python
solve([eq, k_constraint], [n,k,t])
```

$$\left [ \left \{ k : 2 t + 1, \quad n : \frac{\log{\left (\frac{111}{2 t + 1} \right )}}{\log{\left (2 \right )}}\right \}\right ]$$

```python
n_sol = floor(solve(Eq(2**n, 111), n).pop())
n_sol
```

$$6$$

```python
m=(solve(eq.subs(n, n_sol), k).pop())
m
```

$$\frac{111}{64}$$

```python
m.p, m.q
```

$$\left ( 111, \quad 64\right )$$

```python
Mul(2**n_sol*m.q, m.p, evaluate=False)
```

$$111 \cdot 4096$$

```python
divmod(Integer(83726), 8475)
```

$$\left ( 9, \quad 7451\right )$$

```python
def list_to_frac(count, symbol_chr='a', start_index=0):
    l = symbols('{}{}:{}'.format(symbol_chr, start_index, count+start_index))
    expr = Integer(0)
    for i in reversed(l[1:]):
        expr += i
        expr = 1/expr
    return l[0] + expr, l
four_convergent, syms = list_to_frac(4)
four_convergent
```

$$a_{0} + \frac{1}{a_{1} + \frac{1}{a_{2} + \frac{1}{a_{3}}}}$$

```python
four_convergent.subs(syms[1], 1)
```

$$a_{0} + \frac{1}{1 + \frac{1}{a_{2} + \frac{1}{a_{3}}}}$$

```python
symbols('a_{n+1}')
```

$$a_{n+1}$$

```python
@contextmanager
def split_dict(symbols, subs):
    yield (subs[s] for s in symbols)
```

```python
with split_dict([n,k,t], {k:40, n:39, t:3}) as (x_v, *rest): pass
x_v
```

$$39$$

```python
T = IndexedBase('T')
hanoi_eq = Eq(T[n]+1, 2*T[n-1]+2).factor()
hanoi_eq
```

$$T_{n} + 1 = 2 \left(T_{n - 1} + 1\right)$$

the following shows that `subs` doesn't perform *true* pattern matching; on the other hand, `replace` does it since it is possible to use `Wild` object too.

```python
U = IndexedBase('U')
a = Wild('a')
hanoi_eq.subs(T[n]+1, U[n]), hanoi_eq.replace(T[a]+1, U[a])
```

$$\left ( U_{n} = 2 \left(T_{n - 1} + 1\right), \quad U_{n} = 2 U_{n - 1}\right )$$

```python
n, m = Wild('n'), Wild('m')
r = Integer(8475).replace(2**n * m, m, map=True)
r
```

$$\left ( 8475, \quad \left \{ \right \}\right )$$

```python
solve(Equality(k**2, 2*n*log(1/(1-t))), k)[1].simplify()
```

$$\sqrt{2} \sqrt{n \log{\left (- \frac{1}{t - 1} \right )}}$$

```python
solve?
```

```python
17 >> 1
```

$$8$$

```python
a = IndexedBase('a')
eq1 = Eq(a[2], 2+a[1])
eq2 = Eq(a[3], 1+a[1])
eq3 = Eq(3, a[2]+a[3])
solve([eq1, eq2, eq3], [a[1],a[2],a[3],])
```

$$\left \{ a_{1} : 0, \quad a_{2} : 2, \quad a_{3} : 1\right \}$$

---

```python
x, y = IndexedBase('x'), IndexedBase('y')

eq_4 = Eq(1, x[2]*y[2])
eq_3 = Eq(0, x[2]*y[1] + x[1]*y[2])
eq_2 = Eq(0, x[2] + x[1]*y[1] + y[2])
eq_1 = Eq(0, x[1] + y[1])
eq_0 = Eq(0, x[0] * y[0])

eqs = [eq_4, eq_3, eq_2, eq_1, eq_0]
eqs
```

$$\left [ 1 = x_{2} y_{2}, \quad 0 = x_{1} y_{2} + x_{2} y_{1}, \quad 0 = x_{1} y_{1} + x_{2} + y_{2}, \quad 0 = x_{1} + y_{1}, \quad 0 = x_{0} y_{0}\right ]$$

```python
solve(eqs, [x[2],x[1], x[0],y[2],y[1],y[0]], )
```

$$\left [ \left \{ x_{0} : 0, \quad x_{1} : 0, \quad x_{2} : - i, \quad y_{1} : 0, \quad y_{2} : i\right \}, \quad \left \{ x_{0} : 0, \quad x_{1} : 0, \quad x_{2} : i, \quad y_{1} : 0, \quad y_{2} : - i\right \}, \quad \left \{ x_{0} : 0, \quad x_{1} : - \sqrt{2}, \quad x_{2} : 1, \quad y_{1} : \sqrt{2}, \quad y_{2} : 1\right \}, \quad \left \{ x_{0} : 0, \quad x_{1} : \sqrt{2}, \quad x_{2} : 1, \quad y_{1} : - \sqrt{2}, \quad y_{2} : 1\right \}, \quad \left \{ x_{0} : 0, \quad x_{1} : - \sqrt{2} i, \quad x_{2} : -1, \quad y_{1} : \sqrt{2} i, \quad y_{2} : -1\right \}, \quad \left \{ x_{0} : 0, \quad x_{1} : \sqrt{2} i, \quad x_{2} : -1, \quad y_{1} : - \sqrt{2} i, \quad y_{2} : -1\right \}, \quad \left \{ x_{1} : 0, \quad x_{2} : - i, \quad y_{0} : 0, \quad y_{1} : 0, \quad y_{2} : i\right \}, \quad \left \{ x_{1} : 0, \quad x_{2} : i, \quad y_{0} : 0, \quad y_{1} : 0, \quad y_{2} : - i\right \}, \quad \left \{ x_{1} : - \sqrt{2}, \quad x_{2} : 1, \quad y_{0} : 0, \quad y_{1} : \sqrt{2}, \quad y_{2} : 1\right \}, \quad \left \{ x_{1} : \sqrt{2}, \quad x_{2} : 1, \quad y_{0} : 0, \quad y_{1} : - \sqrt{2}, \quad y_{2} : 1\right \}, \quad \left \{ x_{1} : - \sqrt{2} i, \quad x_{2} : -1, \quad y_{0} : 0, \quad y_{1} : \sqrt{2} i, \quad y_{2} : -1\right \}, \quad \left \{ x_{1} : \sqrt{2} i, \quad x_{2} : -1, \quad y_{0} : 0, \quad y_{1} : - \sqrt{2} i, \quad y_{2} : -1\right \}\right ]$$

```python

```


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 9

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Mar. 9 at **2pm.**

## Exercise: New-Keynesian Model Stochastic Simulation with Different Monetary Policies

You will compute two stochastic simulations of the new-Keynesian model. In the first, you will set $\phi_{\pi}=1.5$. In the second, you will set $\phi_{\pi}=100$. For both simulations, you will compute standard deviations and correlation coefficients for the simulated variables. Then you will answer questions about the two simulations.

### Equilibrium Conditions and Paramter Values

The most basic version of the New-Keynesian Model can be expressed as:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
i_{t} & = \bar{r} + \pi^T + \phi_{\pi}\big(\pi_t - \pi^T\big) + \phi_{y}\big(y_t - \bar{y}\big)\\
\pi_t -\pi^T & = \beta \left( E_t\pi_{t+1} - \pi^T\right)  + \kappa (y_t -\bar{y})+ u_t,
\end{align}

where: $y_t$ is (log) output, $r_t$ is the real interest rate, $i_t$ is the nominal interest rate, $\pi_t$ is the rate of inflation between periods $t-1$ and $t$, $\bar{r}$ is the long-run average real interest rate or the *natural rate of interest*, $\beta$ is the household's subjective discount factor, and $\pi^T$ is the central bank's inflation target. The coeffieints $\phi_{\pi}$ and $\phi_{y}$ reflect the degree of intensity to which the central bank *endogenously* adjusts the nominal interest rate in response to movements in inflation and output.

The variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}
\end{align}

Note that the model *does not* contain a monetary policy shock process.

### Parameter Values:

Use the following parameter values to input the model. Note that the value for $\phi_{\pi}$ is empty for now.

| $$\bar{y}$$ | $$\beta$$ | $$\bar{r}$$    | $$\kappa$$ | $$\pi^T$$ | $$\phi_{\pi}$$ | $$\phi_y$$ | $$\rho_g$$ | $$\sigma_g$$    | $$\rho_u$$ | $$\sigma_u$$    |
|-------------|-----------|----------------|------------|-----------|----------------|------------|------------|-------------------|------------|-------------------|
| 0           | 0.995     | $$-\log\beta$$ | 0.25        | 0.02/4    | -            | 0.5/4      | 0.5        | 0.015 | 0.5        | 0.003 |

### Input Model and Solve

Refer to Week 9 lecture and discussion Notebooks for examples. Note that the model *does not* contain a monetary policy shock process.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series.
parameters = pd.Series(dtype=float)
parameters['y_bar'] = 0
parameters['beta'] = 0.995
parameters['r_bar'] = -np.log(parameters.beta)
parameters['kappa'] = 0.25
parameters['pi_T'] = 0.02/4
parameters['phi_y'] = 0.5/4
parameters['rho_g'] = 0.5
parameters['rho_u'] = 0.5
parameters['sigma_g'] = 0.015
parameters['sigma_u'] = 0.003

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable.
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero.
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters
    p = parameters

    # Current variables
    cur = variables_current

    # Forward variables
    fwd = variables_forward

    # IS equation
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy
    monetary_policy = p.r_bar + p.pi_T + p.phi_pi*(cur.pi - p.pi_T) + p.phi_y*cur.y - cur.i

    # Phillips curve
    phillips_curve = p.beta*(fwd.pi- p.pi_T) + p.kappa*cur.y + cur.u - (cur.pi-p.pi_T)

    # Demand process
    demand_process = p.rho_g*cur.g - fwd.g

    # Inflation process
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array
    return np.array([
            is_equation,
            fisher_equation,
            monetary_policy,
            phillips_curve,
            demand_process,
            inflation_process
        ])

# Initialize the model into a variable named 'nk_model'.
nk_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters = parameters)
```

### Stochastic Simulation 1: $\phi_{\pi}=1.5$

Construct a stochastic simulation of the New-Keynesian model with the follwoing properties:

1. 401 periods.
2. Seed for random number generator: 126.

Note that the shock covariance matrix is:

\begin{align}
\text{Covariance matrix} & = \left[\begin{array}{cc}\sigma_g^2 & 0\\ 0 & \sigma_u^2\end{array}\right]
\end{align}

Use the following values for $\sigma_g$ and $\sigma_u$ in the simulation:

| $$\sigma_g$$ | $$\sigma_u$$ |
|--------------|----------------|
| 0.015        | 0.003          |

```python
# Set 'phi_pi' to 1.5 in nk_model.parameters
nk_model.parameters['phi_pi'] = 1.5

# Compute the steady state numerically using .compute_ss() method of nk_model
nk_model.compute_ss()

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False
nk_model.approximate_and_solve(log_linear=False)

# Compute the stochastic simulation using the .stoch_sim() method of nk_model. Set seed to 126.
nk_model.stoch_sim(T=401,seed=126,cov_mat=[[parameters['sigma_g']**2,0],[0,parameters['sigma_u']**2]])
```

Construct a plot with simulated output and inflation plotted together. Multiply simulated output by 100 and simulated inflation by 400 since, by convention, we always annualize inflation and interest rates.

```python
# Create a figure with dimensions 12x4. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create the left axis. PROVIDED
ax1 = fig.add_subplot(1,1,1)

# Plot the simulated series for output (times 100) and inflation (times 400)
(nk_model.simulated['y']*100).plot(ax=ax1,lw=3,alpha=0.75,grid=True)
(nk_model.simulated['pi']*400).plot(ax=ax1,lw=3,alpha=0.75,grid=True,ylabel='Percent',title='Simulated NK Model')

# Construct legend. PROVIDED
ax1.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

Compute the standard deviations of inflation $\pi_t$ (times 400), output $y_t$ (times 100), and the nominal interest rate $i_t$ (times 400) for the simulated series.

```python
# Compute standard deviations of output (times 100), inflation (times 400), and the nominal interest rate (times 400)
print('pi:',round(nk_model.simulated['pi'].std()*400,5))
print('y: ',round(nk_model.simulated['y'].std()*100,5))
print('i: ',round(nk_model.simulated['i'].std()*400,5))
```

    pi: 3.51539
    y:  1.73467
    i:  5.89212

Compute the coefficients of correlation of inflation $\pi_t$, output $y_t$, and the nominal interest rate $i_t$ for the simulated series. *You don't need to multiply the variables by anything because multiplication by a constant doeasn't affect correlations.*

```python
# Compute correlation coefficients of output, inflation, and the nominal interest rate
nk_model.simulated[['pi','y','i']].corr()
```

      pi
      y
      i

      pi
      1.000000
      0.673382
      0.994061

      y
      0.673382
      1.000000
      0.749837

      i
      0.994061
      0.749837
      1.000000

### Stochastic Simulation 2: $\phi_{\pi}=100$

Construct another stochastic simulation of the New-Keynesian model with the follwoing properties:

1. 401 periods.
2. Seed for random number generator: 126.

Note that the shock covariance matrix is:

\begin{align}
\text{Covariance matrix} & = \left[\begin{array}{cc}\sigma_g^2 & 0\\ 0 & \sigma_u^2\end{array}\right]
\end{align}

Use the following values for $\sigma_g$ and $\sigma_u$ in the simulation:

| $$\sigma_g$$ | $$\sigma_u$$ |
|--------------|----------------|
| 0.015        | 0.003          |

```python
# Set 'phi_pi' to 100 in nk_model.parameters
nk_model.parameters['phi_pi'] = 100

# Compute the steady state numerically using .compute_ss() method of nk_model
nk_model.compute_ss()

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False
nk_model.approximate_and_solve(log_linear=False)

# Compute the stochastic simulation using the .stoch_sim() method of nk_model. Set seed to 126.
nk_model.stoch_sim(T=401,seed=126,cov_mat=[[parameters['sigma_g']**2,0],[0,parameters['sigma_u']**2]])
```

Construct a plot with simulated output and inflation plotted together. Multiply simulated output by 100 and simulated inflation by 400 since, by convention, we always annualize inflation and interest rates.

```python
# Create a figure with dimensions 12x4. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create the left axis. PROVIDED
ax1 = fig.add_subplot(1,1,1)

# Plot the simulated series for output (times 100) and inflation (times 400)
(nk_model.simulated['y']*100).plot(ax=ax1,lw=3,alpha=0.75,grid=True)
(nk_model.simulated['pi']*400).plot(ax=ax1,lw=3,alpha=0.75,grid=True,ylabel='Percent',title='Simulated NK Model')

# Construct legend. PROVIDED
ax1.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

Compute the standard deviations of inflation $\pi_t$ (times 400), output $y_t$ (times 100), and the nominal interest rate $i_t$ (times 400) for the simulated series.

```python
# Compute standard deviations of output (times 100), inflation (times 400), and the nominal interest rate (times 400)
print('pi:',round(nk_model.simulated['pi'].std()*400,5))
print('y: ',round(nk_model.simulated['y'].std()*100,5))
print('i: ',round(nk_model.simulated['i'].std()*400,5))
```

    pi: 0.07872
    y:  1.3989
    i:  7.61968

Compute the coefficients of correlation of inflation $\pi_t$, output $y_t$, and the nominal interest rate $i_t$ for the simulated series. *You don't need to multiply the variables by anything because multiplication by a constant doesn't affect correlations.*

```python
# Compute correlation coefficients of output, inflation, and the nominal interest rate
nk_model.simulated[['pi','y','i']].corr()
```

      pi
      y
      i

      pi
      1.000000
      -0.399476
      0.996453

      y
      -0.399476
      1.000000
      -0.320912

      i
      0.996453
      -0.320912
      1.000000

**Questions**

1. Based on your results, how did increasing $\phi_{\pi}$ from 1.5 to 100 affect the volatility of output over the business cycle?
2. Based on your results, how did increasing $\phi_{\pi}$ from 1.5 to 100 affect the volatility of inflation over the business cycle?
3. With $\phi_{\pi}=100$, the central bank is commited to raising the nominal interest rate by 100 percentage points for each one percent increase in the inflation rate. But the computed standard deviation of the nominal interest rate in the second simulation is about 7 percent, far less than 100. How do you explain this?

**Answers**

1. Output became *less* volatile when $\phi_{\pi}$ increased from 1.5 to 100. In the first simulation, the standard deviation of output was 1.73467% and it fell to 1.3989% in the second.
2. Inflation became *less* volatile when $\phi_{\pi}$ increased from 1.5 to 100. In the first simulation, the standard deviation was 3.51539% (annualized) and it fell to 0.07872% (annualized) in the second.
3. With $\phi_{\pi}=100$, the public knows the central bank will react aggressively to counteract even small inflationary pressures. This keep inflation expectations close to target and therefore also keeps the actual inflation rate close to target. In a sense, the central bank is threatening to raise the nominal interest rate to fight inflation and since the threat is credible, it doesn't have to in the equilibrium.