 9. Markovske verige

 Andrej Košir, Lucami, FE

 Kontakt: prof. dr. Andrej Košir, andrej.kosir@lucami.fe.uni-lj.si, skype=akosir_sid

 1

 9. Markovske verige

 Cilji

## ■ Cilji

- Cilj:
    - Modeliranje z markovskimi verigami
    - Osnove analize stanj markovske verige

- Slučajni proces na končno stanjih, ki je najbolje poznan in uporaben. Omejimo se na:
    - Diskreten čas
    - Diskreten prostor stanj (končno ali števno neskončno stanj)

- Primer: uporabniki in njihovo obnašanje
- Primeri uporabe
    - Analiza končnega avtomata: analizaobnašanja TK sistema, ...
    - Ekonomski modeli

- Kaj nas zanima:
    - Pogostnost obiskov
    - Stanje po dolgem času
    - Pojavi kot je absorbcija

 2

 9. Markovske verige

 Cilji

## ■ Poglavja

9.1. Definicija, osnovne lastnosti

■ Definicija markovske verige $\large{*}$

■ O prehodni matriki, porazdelitev stanj $\large{*}$

9.2. Klasifikacija stanj

■ Povezanost stanj in perioda $\large{*}$

■ Povrnljivo in minljivo stanje $\large{*}$

■ Ergodijska mar. veriga in časi vračanja v stanje

■ Stacionarna (limitna) porazdelitev $\large{*}$

■ Končna markovska veriga $\large{*}$

9.3. Ocenjevanje prehodne matrike

■ Ocene prehodnih verjetnosti

■ Interval zaupanja in grafikov kvantilov

■ Določane velikosti vzorca

■ Numerični vidik

■ Primer: razredi uporabnikov TK storitev

 3

 9. Markovske verige

 9.1. Definicija, osnovne lastnosti

## 9.1. Definicija, osnovne lastnosti

■ Definicija markovske verige

■ O prehodni matriki, porazdelitev stanj

 4

 9. Markovske verige

 9.1. Definicija, osnovne lastnosti

## ■ Definicija markovske verige

- Prostor stanj $S$, $m=|S|$
    - Končen, neskončen (števen)
- Čas $T$
    - Diskreten, zvezen

- Slučajni proces:
$$ X_n : T \to S $$

- Diskretna Markovska veriga je slučajni proces na diskretnem času in diskretnim prostorom stanj, za katerega velja pravilo prehodne verjetnosti enega koraka
$$ P[X_{n+1}|_{X_n, X_{n-1}, \ldots, X_0}] = P[X_{n+1}|_{X_n}] = P[X_1|_{X_0}] $$

- Prehodna verjetnost enega koraka
$$ p_{ij} = P[X_{n+1}=j|_{X_n=i}] $$

- Končen prostor stanj – matrika prehodnih verjetnosti = prehodna matrija
$$ P = [p_{ij}], \; i,j\in S. $$

Vrstične vsote ima enake 1.

 5

```python
import numpy as np
from sympy import Matrix
from sympy.functions import re

# Case 1
# Transition matrix
P = np.array([[0.7, 0.2, 0.1, 0.0],
              [0.2, 0.5, 0.2, 0.1],
              [0.05, 0.15, 0.6, 0.2],
              [0.0, 0.2, 0.0, 0.8]])

P
```

    array([[0.7 , 0.2 , 0.1 , 0.  ],
           [0.2 , 0.5 , 0.2 , 0.1 ],
           [0.05, 0.15, 0.6 , 0.2 ],
           [0.  , 0.2 , 0.  , 0.8 ]])

 9. Markovske verige

 9.1. Definicija, osnovne lastnosti

## ■ O prehodni matriki, porazdelitev stanj

- Prehodna matrika za $n$ korakov
    - $p_{ij}^{(n)} = P[X_{k+n}=j|_{X_k=i}]$, $P^{(n)}=[p_{ij}^{(n)}]$
    - Formula o popolni verjetnosti
    $$ P^{(n)} = P^n $$

- Porazdelitev stanj:
  $$ \pi_i^n = P[X_n=i], \pi^n = [\pi_1^n, \ldots, \pi_i^n] $$
    - To je delež časa sistema v danem stanju na dolgi rok? Ne vedno.
    - Začetna porazdelitev: $\pi^0$. Če je stanje $X_0=i_0$ znano, je
    $$ \pi^0=\delta_{i_0 i} $$
    - Formula o popolni verjetnosti:
    $$ \pi^n = P^{(n)} \pi^0 $$
    - O limitni porazdelitvi kasneje

 6

 9. Markovske verige

 9.2. Klasifikacija stanj

## 9.2. Klasifikacija stanj

■ Povezanost stanj in perioda

■ Povrnljivo in minljivo stanje

■ Ergodijska mar. veriga in časi vračanja v stanje

■ Stacionarna (limitna) porazdelitev

■ Končna markovska veriga

 7

 9. Markovske verige

 9.2. Klasifikacija stanj

## ■ Povezanost stanj in perioda

- Dosegljivo stanje
    - Stanje $j$ je dosegljivo iz stanja $i$, če $\exists n: p_{ij}^{(n)}>0$
    - Stanji sta povezani, če sta vzajemno dosegljivi, označimo $i\leftrightarrow j$
    - Povezanost je ekvivalenčna relacija

- Markovska veriga je nerazcepna, če ima en sam razred stanj;
    - Prehodna matrika je bločno diagonalna

- Perioda stanja $d(i)$ je največji skupni delitelj vseh $n$, za katere velja $P_{ii}^{(n)} > 0$
    - Če je $i\leftrightarrow j$, potem $d(i) = d(j)$;
    - Stanje je neperiodično, če je $d(i)=1$;

- Stanje je ničelno, če je
$$ \lim_{n\to\infty} P_{ii}^{(n)} = 0. $$

 8

```python
# ---------------------------------------------------------------------------------------------------
# Classification of states
print ('Transition matrix P=\n', P)
print ('\nTransition matrix P^2=\n', np.linalg.matrix_power(P, 2))

# All entries of P^2 are non-zero
# => All atates are connected
```

    Transition matrix P=
     [[0.7  0.2  0.1  0.  ]
     [0.2  0.5  0.2  0.1 ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

    Transition matrix P^2=
     [[0.535 0.255 0.17  0.04 ]
     [0.25  0.34  0.24  0.17 ]
     [0.095 0.215 0.395 0.295]
     [0.04  0.26  0.04  0.66 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Decompose transition matrix
# Use Jordan decomposition
#mP = Matrix(P)
#mB, mJ = mP.jordan_form()
#B, J = np.array(mB), np.array(mJ)
B = np.array([[-0.5,-0.776313,0.659526,-0.413722],
              [-0.5,-0.212847,-0.11514,0.830469],
              [-0.5,0.283275,-0.73692,-0.123153],
              [-0.5,0.521335,0.0933625,-0.352121]])
J = np.array([[1.,0,0,0],
              [0,0.718345,0,0],
              [0,0,0.553349,0],
              [0,0,0,0.328305]])

# Test decomposition
err_mat = P - np.dot(B, np.dot(J, np.linalg.inv(B)))
print ('\nDecomposition error norm = ', np.linalg.norm(err_mat))
```

    Decomposition error norm =  7.249920892047682e-07

 9. Markovske verige

 9.2. Klasifikacija stanj

## ■ Povrnljivo in minljivo stanje

- Verjetnost prve vrnitve po $n$ korakih
$$ f_{ij}^n = P[X_n=j|_{X_{n-1}\ne j, \ldots, X_1\ne j, X_0=i}] $$

- Slučajna spremenljivka števila korakov prehoda iz $i$ v $j$ označimo s $T_{ij}$ in ima porazdelitev
$$ T_{ij} \sim \left(\matrix{0 & 1 & 2 & \ldots \cr f_{ij}^0 & f_{ij}^1 & f_{ij}^2 & \ldots}\right) $$
    - Posebej označimo $T_i = T_{ii}$
    - Označimo: $\sum_{n=0}^\infty f_{ii}^n = f_{ii}^*$. Vrednost
    $$ 1- f_{ii}^* $$
    je verjetnost, da se nikoli več ne vrnemo v stanje $i$ potem ko smo stanje $i$ zapustili.

- Stanje $i$ je **povrnljivo**, če je $f_{ii}^* = 1$. Stanje je **minljivo**, če ni povrnljivo.
    - Obe lastnosti se ohranjata pri povezanosti, torej definirata ekvivalenčne razrede.

- Velja: stanje $i$ povrnljivo je natanko tedaj, ko je
$$ \sum_{n=1}^\infty P_{ii}^{(n)} = \infty $$

- Velja : če $i\leftrightarrow j$, sta obe stanji ali minljivi ali povarnljivi

 9

```python
# Recurent, transient?

# Eigenvalues
eigVals, eigVecs = np.linalg.eig(P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigVa = eigVals[idx]
eigVc = eigVecs[:,idx]

las_bf =  (1./(1-eigVa[1:4]))-1
print ('las_bf', las_bf)

sumJn = np.diag(np.insert(las_bf, 0, np.inf))
print ('\nSum J^n\n', sumJn)

sumPn = np.dot(B, np.dot(sumJn, np.linalg.inv(B)))
print ('\nSum P^n\n', sumPn)

# Since all diagonal elements of sum P^n are infinity
#  => all states are recurrent

```

    las_bf [2.55044912 1.23888596 0.48877142]

    Sum J^n
     [[       inf 0.         0.         0.        ]
     [0.         2.55044912 0.         0.        ]
     [0.         0.         1.23888596 0.        ]
     [0.         0.         0.         0.48877142]]

    Sum P^n
     [[inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]]

```python
def inf_sum(a_lst):
    return [(la/(1.0-la) if la < 1 else np.infty) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Are states recurrent?
# Compute sum of P^n
sumJ = np.diag(inf_sum(np.diag(J)))
sumP = np.dot(B, np.dot(sumJ, np.linalg.inv(B)))

print ('\nSum of P^n = \n', sumP)
# => All states are reccurent
```

    Sum of P^n =
     [[inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]]

 9. Markovske verige

 9.2. Klasifikacija stanj

## ■ Ergodijska mar. veriga in časi vračanja v stanje

- Markovska veriga je Ergodijska, če je
    - Nerazcepna
    - Povrnljiva (vsa stanja so povrnljiva)
    - Neperiodična (vsa stanja so nepriodična)
    - Neničelna (vsa stanja so neničelna)

- Pričakovan čas vračanja:
$$ E(T_i) = \sum_{n=0}^\infty n f_{ii}^{(n)} $$

- Za vsa stanja Ergodijske markovske verige velja
$$ \lim_{n\to\infty} P_{ii}^{(n)} = \frac{1}{E(T_i)} $$
$$ \lim_{n\to\infty} P_{ij}^{(n)} = \lim_{n\to\infty} P_{ii}^{(n)} $$

- Povrnljivo **ničelno** stanje $i$
$$ \lim_{n\to\infty} P_{ii}^{(n)} = 0 \; \Leftrightarrow \;  E(T_i) = \infty  $$

- **Povrnljivo pozitivno** = **ergodijsko stanje** $i$:
$$ \lim_{n\to\infty} P_{ii}^{(n)} > 0 \; \Leftrightarrow \; E(T_i) < \infty  $$

 10

```python
def inf_lim(a_lst):
    return [(0 if la < 1 else 1) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Zero or positive states?
# Compute lim of P^n
limJ = np.diag(inf_lim(np.diag(J)))
limP = np.dot(B, np.dot(limJ, np.linalg.inv(B)))

print ('\nlim of P^n = \n', limP)

# Since all lim P^n are positive
# => all states are positive

# Expected times of visits
ETi = 1.0/np.diag(limP)

print ('\nExpected number of visits py states:\n', ETi)

```

    lim of P^n =
     [[0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]]

    Expected number of visits py states:
     [4.69444311 3.67391519 5.28124903 3.07272668]

 9. Markovske verige

 9.2. Klasifikacija stanj

## ■ Stacionarna (limitna) porazdelitev

- Spomnimo se
$$ \pi_i^n = P[X_n=i], \qquad \pi^n = [\pi_1^n, \ldots, \pi_i^n] $$

- Stacionarna (limitna) porazdelitev:
  $$ \pi = \lim_{n\to\infty} \pi^n $$
    - Enolično določena: vsa stanja so povrnljiva neničelna

- Za ergodijsko markovsko verigo obstaja limitna porazdelitev
- Velja: limitna porazdelitev Ergodijske Markovske verige je levi lastni vektor prehodne matrike:
  $$ \pi^\top = \pi^\top P. $$

- Opazimo: V ergodijsk verigi je limitna porazdelitev neodvisna od začetne porazdelitve $\pi^0$.

 11

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(P.T)
pi = eigVecs[:, 0]/sum(eigVecs[:, 0])

print ('\neigVals=\n', eigVals)
#print '\neigVecs=\n', eigVecs
print ('\nLimit distribution =\n', pi)
```

    eigVals=
     [1.         0.32830522 0.71834549 0.55334929]

    Limit distribution =
     [0.21301775 0.27218935 0.18934911 0.32544379]

 9. Markovske verige

 9.2. Klasifikacija stanj

## ■ Verjetnost absorbcije

- Absorbcija: veriga dane podmnožice stanj $S_0 \subset S$ ne zapusti več
- Verjetnost absorpcije: verjetnost, da se to zgodi
- Če je veriga na začetku v minljivem stanje, se vedno zgodi:
    1. Slučajni proces z verjetnostjo 1 zapusti množico minljivih stanj
    2. Slučajni proces z verjetnostjo 1 ostane v množici povrnljivih stanj, v katera je vstopil;

- Absorbcijsko stanje $i$: edino stanje povrnljivega razreda.
    Velja: $i$ absorbirajoče če in samo če $P_{ii}=1$.

- Naj bo $i\in S$ minljivo stanje, $S_e$ razred povrnljivih stanj:

**Verjetnost absorbcije:** $\pi_i(S_e)$ je verjetnost, da je veriga z $X_0=i$ absorbirana v $S_e$

- Velja – formula za izračun - za vsak $i$
$$ \lim_{n\to\infty} P_{ij}^{(n)} = \pi_i(S_e)\pi_j. $$

 12

 9. Markovske verige

 9.2. Klasifikacija stanj

## ■ Končna markovska veriga

- Ima končno množico stanj $|S|=m < \infty$.
- Vsaka končna Markovska veriga stanja vsebuje tudi povrnljiva stanja.
- Vsa povrnljiva stanja so pozitivna, torej **ergodijska** in imajo končen pričakovan čas vrnitve
$$ \lim_{n\to\infty} P_{ii}^{(n)} > 0. $$

- Klasifikacija stanj: vsa stanja so $S=S_m\cup S_e$, kjer je
    - Minljiva stanja: $S_m$
    - Ergodijska (povrnljiva neničelna): $S_e$

- Bločna razdelitev prehodne matrike: uredimo indekse stanj
    - Ergodisjka stanja: $1, \ldots, m-K$
    - Minljiva stanja: $m-K+1, \ldots, m$
    $$ P = \left[\matrix{S & 0 \cr R & Q}\right] $$

- Fundamentalna matrika:
    $$ N = (I - Q)^{-1}.  $$

 13

```python
import numpy as np
# Case 2
c2P = np.array(
    [[0.7, 0.3, 0.0, 0.0],
     [0.2, 0.8, 0.0, 0.0],
     [0.05,0.15, 0.6, 0.2],
     [0.0, 0.2, 0.0, 0.8]])
print ('Transition matrix P=\n', c2P)

```

    Transition matrix P=
    [[0.7  0.3  0.   0.  ]
     [0.2  0.8  0.   0.  ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Classification of states
print ('Transition matrix P=\n', c2P)
print ('\nTransition matrix P^2=\n', np.linalg.matrix_power(c2P, 2))
print ('\nTransition matrix P^3=\n', np.linalg.matrix_power(c2P, 3))

# Matrix multiplications by blocks
# => 2 x 2 block of zeros stayes the same
# => There are two connected classes of states

```

    Transition matrix P=
    [[0.7  0.3  0.   0.  ]
     [0.2  0.8  0.   0.  ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

    Transition matrix P^2=
    [[0.55  0.45  0.    0.   ]
     [0.3   0.7   0.    0.   ]
     [0.095 0.265 0.36  0.28 ]
     [0.04  0.32  0.    0.64 ]]

    Transition matrix P^3=
    [[0.475  0.525  0.     0.    ]
     [0.35   0.65   0.     0.    ]
     [0.1375 0.3505 0.216  0.296 ]
     [0.092  0.396  0.     0.512 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Decompose transition matrix
# Use Jordan decomposition
#mP = Matrix(c2P)
#mB, mJ = mP.jordan_form()
#B, J = np.array(mB), np.array(mJ)
c2B = np.array([[1.0, 0, 0, 9.0/4],
              [1.0, 0, 0, -3.0/2],
              [1.0, 1.0, 1.0, -7.0/8],
              [1.0, 1.0, 0, 1.0]])
c2J = np.array([[1.0,0,0,0],
              [0,4.0/5,0,0],
              [0,0,3.0/5,0],
              [0,0,0,1.0/2]])

# Test decomposition
err_mat = c2P - np.dot(c2B, np.dot(c2J, np.linalg.inv(c2B)))
print ('\nDecomposition error norm = ', np.linalg.norm(err_mat))
```

    Decomposition error norm =  1.9675159943996247e-16

```python
def prd(a,b):
    if np.isinf(a) and b == 0:
        return 0
    if np.isinf(b) and a == 0:
        return 0
    return a*b

def myDot(A,B):
    nA1, nA2, nB2 = A.shape[0], A.shape[1], B.shape[1]
    AB = np.zeros((nA1,nB2))
    for i in range(nA1):
        for j in range(nB2):
            curr = 0
            for k in range(nA2):
                curr += prd(A[i,k], B[k,j])
            AB[i, j] = curr
    return AB

# Classification of states
# Recurent, transient?

# Eigenvalues
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigVa = eigVals[idx]
eigVc = eigVecs[:,idx]

las_bf =  (1./(1-eigVa[1:4]))-1
sumJn = np.diag(np.insert(las_bf, 0, np.inf))

sumPn = myDot(c2B, myDot(sumJn, np.linalg.inv(c2B)))
print ('\nSum P^n\n', sumPn)

# Since diagonal elements of sum P^n i=1 and i=2 are infinity
#  => states i=1 and i=2 are recurrent
# Since diagonal elements of sum P^n i=3 and i=4 are finite
#  => states i=3 and i=4 are transient

```

    Sum P^n
    [[inf inf 0.  0. ]
     [inf inf 0.  0. ]
     [inf inf 1.5 2.5]
     [inf inf 0.  4. ]]

```python
def inf_lim(a_lst):
    return [(0 if la < 1 else 1) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Zero or positive states?
# Compute lim of P^n
limJ = np.diag(inf_lim(np.diag(c2J)))
limP = np.dot(c2B, np.dot(limJ, np.linalg.inv(c2B)))

print ('\nlim of P^n = \n', limP)

# Since all lim P^n are positive
# => all states are positive

# Expected times of visits
ETi = 1.0/np.diag(limP)

print ('\nExpected number of visits py states:\n', ETi)
```

    lim of P^n =
    [[0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]]

    Expected number of visits py states:
    [2.5        1.66666667        inf        inf]

    /home/nbuser/anaconda2_501/lib/python2.7/site-packages/ipykernel/__main__.py:16: RuntimeWarning: divide by zero encountered in divide

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigenValues = eigVals[idx]
eigenVectors = eigVecs[:,idx]

# Normalize eigenvector
pi = eigenVectors[:, 0]/sum(eigenVectors[:, 0])

print ('\neigVals=\n', eigenValues)
#print '\neigVecs=\n', eigenVectors
print ('\nLimit distribution =', pi)
```

    eigVals=
    [1.  0.8 0.6 0.5]

    Limit distribution = [ 0.4  0.6 -0.  -0. ]

 9. Markovske verige

 9.2. Klasifikacija stanj

## ■ Število obiskov minljivega stanja

- Slučajna spremenljivka $n_i$: število obiskov stanja $i$ v poljubno dolgem času
- Matematično upanje števila obiskov $j$ če je začetno stanje $i$:
 $$ E_i[n_j] < \infty $$

- Velja: če je stanje $i$ minljivo, je
 $$ E_i[n_j] = \sum_{k=1}^\infty P_{ij}^{(k)} $$

- Velja: fundamentalna matrika vsebuje pričakovana števila vračanj:
$$ \left[E_i[n_j]\right] = N $$

 14

```python
# Expected number of visits of recurent states
m = 4 # |S|
K = 2 # number of recurent states

Q = c2P[K:m, K:m]
print ('Part of transition matrix Q = \n', Q)

N = np.linalg.inv(np.eye(m-K)-Q)
print ('\nFundamental matrix N =\n', N)
```

    Part of transition matrix Q =
    [[0.6 0.2]
     [0.  0.8]]

    Fundamental matrix N =
    [[2.5 2.5]
     [0.  5. ]]

 9. Markovske verige

 9.3 Ocenjevanje prehodne matrike

## 9.3 Ocenjevanje prehodne matrike

■ Ocene prehodnih verjetnosti

■ Interval zaupanja in grafikov kvantilov

■ Določane velikosti vzorca

■ Numerični vidik

■ Primer: razredi uporabnikov TK storitev

 15

 9. Markovske verige

 9.3 Ocenjevanje prehodne matrike

## ■ Ocene prehodnih verjetnosti

- Preodna verjetnost
$$ p_{ij} = \frac{n_k}{n} $$
je statistična ocena;

- Ocena parametra (npr. povprečje ali delež):
    - ocenimo in določimo
        - povprečje - pričakovano vrednost: $\overline{x}$
        - standardni odklon: $\sigma$
        - za stopnjo tveganja $\alpha$ določimo $z_\alpha$ (za normalno porazdelitev je $z_{0.05}=1.96$)
    - Interval zaupanja:
    $$ CI = [\overline{x} - z_\alpha\frac{\sigma}{\sqrt{n}}, \overline{x} + z_\alpha\frac{\sigma}{\sqrt{n}}] $$
    - Velikost vzorca: določimo ga glede na predpisano velikost intervala zaupanja
    $$ |CI| = 2 z_\alpha\frac{\sigma}{\sqrt{n}} $$

 16

 9. Markovske verige

 9.3 Ocenjevanje prehodne matrike

## ■ Interval zaupanja in grafikov kvantilov

- Vizualizacija intervala zaupanja

 17

 9. Markovske verige

 9.3 Ocenjevanje prehodne matrike

## ■ Določane velikosti vzorca

- Iz zahteve po širini intervala zaupanja
- Graf velikosti vzorca za oceno deleža $\alpha=0.05$ ($z_\alpha=1.96$)
$$ CI = [p_0 - z_\alpha\sqrt{p_0(1-p_0)/n}, p_0 + z_\alpha\sqrt{p_0(1-p_0)/n}] $$

- Za primer $p=0.65$ in $|CI|=\Delta p$ pri $\alpha=0.05$ velja neenačba
$$ n\geq \frac{4 z_\alpha^2}{\Delta p^2} p_0 (1-p_0) $$

- Od tod dobimo:
    - Za $\Delta p=0.01:$ $n\geq 34959$,
    - Za $\Delta p=0.02:$ $n\geq 175$,
    - Za $\Delta p=0.04:$ $n\geq 88$.

 18

 9. Markovske verige

 9.3 Ocenjevanje prehodne matrike

## ■ Numerični vidik - matrični račun

- Potrebujemo izračun spektra in potenc
$$ P^n = P^{(n)} $$

- Jordanova dekompozicija
$$ P = B J B^{-1},  $$
kjer je $J$ Jordanova kanonična forma.

- Za vsak razred stanj posebej velja
$$ J = \lambda I + R $$

- Velja
$$ P^n = B J^n B^{-1} $$

 19

 9. Markovske verige

 9.3 Ocenjevanje prehodne matrike

## ■ Primer: razredi uporabnikov TK storitev

- Množica uporabnikov TK storitve $U$
- Razdelimo jih po zadovoljstvu: $U=U_1 \cup U_2 \cup U_3 \cup U_4$
    - Zelo zadovoljni: $U_1$
    - Zadovoljni: $U_2$
    - Nezadovoljni: $U_3$
    - Zapuščajo ponudnika: $U_4$

- Začetna porazdelitev: ocena trenutnega stanja uporabnikov
- Rezultati:
    - Povrnljivost - minljivost stanj in razredi teh stanj
    - Limitna porazdelitev

 20

 9. Markovske verige

 9.3 Ocenjevanje prehodne matrike

## ■ Zaključek

- Preprosta in uporabna analiza končnih stanj
- Preverjanje pogojev pri oceni začetne porazdelitve in prehodne matrike

 21


# Eliashberg theory for Holstein-Hubbard model

Author: [Shintaro Hoshino](mailto:shintaro.hoshino@gmail.com) and [Hiroshi Shinaoka](mailto:h.shinaoka@gmail.com)

### Equations

The (multiorbital) Hubbard model coupled to local phonons describes a simplest case where the electrons and phonons are correlated.
Such situation can be realized in a molecular-based systems such as fullerides.
While the DMFT approach to electron-phonon coupled systems is still technically challenging, the Eliashberg approach [1,2] is a powerful method in the weakly correlated regime and provides an intuitive understanding for the underlying physics.

The single-orbital case is known as a Holstein-Hubbard model whose Hamiltonian is given by

$$
\begin{align}
    \mathscr H &= \mathscr H_{e0} +
    \frac{U}{2}\sum_i :n_{i}^2:
    + \omega_0 \sum_i a_i^\dagger a_i
    + g \sum_i (a_i+a_i^\dagger) n_i
\end{align}
$$

where $n_i=\sum_{\sigma} n_{i\sigma} = \sum_{\sigma} c^\dagger_{i\sigma} c_{i\sigma}$ is the local number operator, and the colon represents a normal ordering.

We generalize it so as to deal with the three-orbital case relevant to fulleride superconductors (the Jahn-Teller-Hubbard model).
The details for the three-orbital model are provided in Ref [3], and here we explain the outline of the approach.
We begin with the Hamiltonian $\mathscr H = \mathscr H_{e} + \mathscr H_{ee} + \mathscr H_p + \mathscr H_{ep}$
where

$$
\begin{align}
    \mathscr H_{ee}  &= \sum_{i\eta} I_\eta : T_{i\eta}^2:
    \hspace{10mm}
    \Big( T_{i\eta} = \sum_{\gamma\gamma'\sigma} c^\dagger_{i\gamma\sigma} \lambda^\eta_{\gamma\gamma'} c_{\gamma'\sigma}  \Big)
    \\
    \mathscr H_{p}  &= \sum_{i\eta} \omega_\eta a_{i\eta}^\dagger a_{i\eta} = \sum_{i\eta} \frac{\omega_\eta}{4} (p_{i\eta}^2 + \phi_{i\eta}^2)
    \equiv \mathscr H_{p,{\rm kin}} + \mathscr H_{p,{\rm pot}}
    \\
    \mathscr H_{ep}  &= \sum_{i\eta} g_\eta
    \phi_{i\eta}
    T_{i\eta}
\end{align}
$$

$\gamma$ is an orbital index. The $\eta$ distinguish the types of charge-orbital moments for electrons and vibration modes for phonons, respectively,
where $M=\sum_\gamma 1$, ${\rm Tr}(\hat \lambda^\eta)^2=2$, $\sum_\eta 1 = \frac{M(M+1)}{2}$.
We may use $M=1$ (single-orbital) or $M=3$ (three-orbital).
For examples, the charge operator is given by $T_{i,\eta=0} = \sqrt 2 \sum_\sigma n_{i\sigma}$ for $M=1$ and $T_{i,\eta=0} = \sqrt{\tfrac 2 3} \sum_\sigma (n_{i1\sigma}+n_{i2\sigma}+n_{i3\sigma})$ for $M=3$.

We have introduced displacement and momentum operators by $\phi_{i\eta} = a_{i\eta}+ a_{i\eta}^\dagger$ and $p_{i\eta} = (a_{i\eta} - a_{i\eta}^\dagger)/{\rm i}$, which satisfy the canonical commutation relation $[\phi_{i\eta},p_{j\eta'}] = 2{\rm i} \delta_{ij} \delta_{\eta\eta'}$.
$\mathscr H_{ee}$ is a compact multipole representation for the Hubbard or Slater-Kanamori interactions [4].
More specifically, $I_0 = U/4$ for the single-orbital case, and $I_0 = 3U/4 -J$, $I_{\eta\neq 0} = J/2$ for the three-orbital case with the Hund's coupling $J$ [3].

We assume that there is no $\eta$-dependence, i.e., we write $g_\eta = g_0$ and $\omega_\eta = \omega_0$.
Assuming also that the self-energy is local as in DMFT, we obtain the Eliashberg equations

$$
\begin{align}
    \Sigma(\tau) &= -U_{\rm eff}(\tau) G(\tau)
    \\
    \Delta(\tau) &= U_{\rm eff}(\tau) F(\tau)
    \\
    U_{\rm eff}(\tau) &= (U+2J)\delta(\tau) + (M+1)  g_0^2 D (\tau)
    \\
    \Pi  (\tau) &=  4g_0^2 [ G(\tau) G(-\tau) - F(\tau)^2 ]
\end{align}
$$

where the local Green functions are defined by
$G(\tau) = - \langle \mathcal T c_{i\gamma\sigma}(\tau) c^\dagger_{i\gamma\sigma} \rangle$,
$F(\tau) = - \langle \mathcal T c_{i\gamma\uparrow}(\tau) c_{i\gamma\downarrow} \rangle$
for electrons, and
$D(\tau) = - \langle \mathcal T \phi_{i\eta}(\tau) \phi_{i\eta} \rangle$ for phonons.
We have simplified the equations using the cubic symmetry.
The phase of the superconducting order parameter is fixed as $\Delta\in \mathbb R$.
If we take $M=1$ and correspondingly $J=0$, it reproduces the single-orbital Holstein-Hubbard model.

The internal energy may be evaluated through the single-particle Green functions.
In a manner similar to Ref. [5],
we have the following relations

$$
\begin{align}
    2MT\sum_{n}{\rm i}\omega_n G({\rm i}\omega_n) {\rm e}^{{\rm i} \omega_n 0^+}
    &= \langle \mathscr H_{e0} \rangle + 2\langle \mathscr H_{ee} \rangle
    + \langle \mathscr H_{ep} \rangle
    \\
    \frac{M(M+1)}{2} T\sum_m \frac{({\rm i}\nu_m)^2}{2\omega_0} D({\rm i}\nu_m) {\rm e}^{{\rm i} \nu_m 0^+}
    &= - 2\langle \mathscr H_{p,{\rm pot}}\rangle
    - \langle \mathscr H_{ep}\rangle
    = - 2\langle \mathscr H_{p,{\rm kin}}\rangle
    \\
    2M T\sum_n \Sigma({\rm i}\omega_n) G({\rm i}\omega_n) {\rm e}^{{\rm i} \omega_n 0^+}
    &= 2\langle \mathscr H_{ee} \rangle
    + \langle \mathscr H_{ep} \rangle
    \\
    \frac{M(M+1)}{2} T\sum_m \Pi({\rm i}\nu_m) D({\rm i}\nu_m) {\rm e}^{{\rm i} \nu_m 0^+}&= - \langle \mathscr H_{ep} \rangle
\end{align}
$$

Thus the internal energy $\langle \mathscr H \rangle = \langle \mathscr H_{e0} + \mathscr H_{ee} + \mathscr H_{p,{\rm kin}} + \mathscr H_{p,{\rm pot}} + \mathscr H_{ep} \rangle$ is obtained.
The specific heat is then calculated numerically through differentiating it by temperature.

In addition to the above Eliashberg equations, the Dyson equation completes a closed set of the self-consistent equations.
The Green functions are given in the Fourier domain:

$$
\begin{align}
    G({\rm i}\omega_n) &= \int {\rm d} \varepsilon \rho(\varepsilon)
    \frac{\zeta ({\rm i}\omega_n) + \varepsilon}{\zeta({\rm i}\omega_n)^2- \Delta({\rm i}\omega_n)^2 - \varepsilon^2}
    \\
    F({\rm i}\omega_n) &= \int {\rm d} \varepsilon \rho(\varepsilon)
    \frac{\Delta ({\rm i}\omega_n)}{\zeta({\rm i}\omega_n)^2- \Delta({\rm i}\omega_n)^2 - \varepsilon^2}
    \\
    \zeta({\rm i}\omega_n) &=  {\rm i}\omega_n+\mu-\Sigma({\rm i}\omega_n)
\end{align}
$$

for electrons, and

$$
\begin{align}
    D({\rm i}\nu_m)^{-1} &= D_0({\rm i}\nu_m)^{-1} - \Pi({\rm i}\nu_m)
    \\
    D_0({\rm i}\nu_m) &= \frac{2\omega_0}{({\rm i}\nu_m)^2 - \omega_0^2}
\end{align}
$$

for phonons.
We take the semi-circular density of states $\rho(\varepsilon) = \frac{2}{\pi D^2} \sqrt{D^2-\varepsilon^2}$ for electron parts.

```
## Implementation: Eliashberg solver
```

```
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import sparse_ir

plt.rcParams.update({
    "font.family": "serif",
    "font.size": 16,
})
```

```
from numpy.polynomial.legendre import leggauss

def scale_quad(x, w, xmin, xmax):
    """ Scale weights and notes of quadrature to the interval [xmin, xmax] """
    assert xmin < xmax
    dx = xmax - xmin
    w_ = 0.5 * dx * w
    x_ = (0.5 * dx) * (x + 1) + xmin
    return x_, w_

class Eliashberg:
    def __init__(self, bset: sparse_ir.FiniteTempBasisSet, rho_omega, omega_range, U, J, omega0, g, deg_leggauss=100):
        assert isinstance(omega_range, tuple)
        assert omega_range[0] < omega_range[1]
        self.U = U
        self.J = J
        self.bset = bset
        self.beta = bset.beta
        self.rho_omega = rho_omega
        self.omega0 = omega0
        self.g = g

        x_, w_ = leggauss(deg_leggauss)
        self.quad_rule = scale_quad(x_, w_, omega_range[0], omega_range[1])
        self.omega = self.quad_rule[0]
        self.omega_coeff = rho_omega(self.omega) * self.quad_rule[1]

        # Sparse sampling in Matsubara frequencies
        self.iv_f = self.bset.wn_f * (1j * np.pi/self.beta)
        self.iv_b = self.bset.wn_b * (1j * np.pi/self.beta)

        # Phonon propagator
        self.d0_iv = 2 * omega0/(self.iv_b**2 - omega0**2)

    def xi_iv(self, mu, sigma_iv):
        """
        Compute xi(iv)
        """
        return self.iv_f + mu - sigma_iv

    def g_f_iv(self, mu, sigma_iv, delta_iv):
        """
        Compute G(iv) and F(iv) from sigma(iv), Delta(iv)
        """

        xi_iv = self.xi_iv(mu, sigma_iv)
        denominator = (xi_iv**2 - delta_iv**2)[:, None] - (self.omega**2)[None, :]
        numerator_G = xi_iv[:, None] + self.omega[None, :]
        numerator_F = delta_iv[:, None]
        g_iv = np.einsum('q,wq->w', self.omega_coeff, numerator_G/denominator,
                        optimize=True)
        f_iv = np.einsum('q,wq->w', self.omega_coeff, numerator_F/denominator,
                        optimize=True)
        return g_iv, f_iv

    def pi_tau(self, g_tau, f_tau):
        """
        Compute Pi(tau)
        """
        return - 4 * (self.g**2) * (g_tau * g_tau[::-1] + f_tau**2)

    def d_iv(self, phi_iv):
        """
        Compute D(iv)
        """
        return 1/(1/self.d0_iv - phi_iv)

    def sigma(self, g_tau, d_tau):
        """
        Compute Sigma(tau) and Sigma(iv)
        """
        sigma_tau = - 4 * (self.g**2) * d_tau * g_tau
        sigma_iv = self.to_matsu(sigma_tau, "F")
        return sigma_tau, sigma_iv

    def delta_iv(self, f_tau, d_tau):
        """
        Compute Delta(iv)
        """
        tl_delta_tau = (4 * self.g**2) * d_tau * f_tau
        tl_delta_iv = self.to_matsu(tl_delta_tau, "F")
        return tl_delta_iv + (self.U + 2*self.J) * f_tau[0]

    def _smpl_tau(self, stat):
        return {"F": self.bset.smpl_tau_f, "B": self.bset.smpl_tau_b}[stat]

    def _smpl_wn(self, stat):
        return {"F": self.bset.smpl_wn_f, "B": self.bset.smpl_wn_b}[stat]

    def to_tau(self, obj_iv, stat):
        """
        Transform to tau
        """
        return self._smpl_tau(stat).evaluate(self._smpl_wn(stat).fit(obj_iv))

    def to_matsu(self, obj_tau, stat):
        """
        Transform to Matsubara
        """
        return self._smpl_wn(stat).evaluate(self._smpl_tau(stat).fit(obj_tau))

    def internal_energy(self, sigma_iv=None, delta_iv=None, g_iv=None, d_iv=None, tau=0.0):
        """
        Compute internal energy
        """
        stat_sign = {0.0: 1, self.beta: -1}[tau]
        smpl_tau0 = [sparse_ir.TauSampling(b, [tau]) for b in [self.bset.basis_f, self.bset.basis_b]]

        e1 = stat_sign * smpl_tau0[0].evaluate(self.bset.smpl_wn_f.fit(self.iv_f * g_iv - 1))
        e2 = stat_sign * smpl_tau0[0].evaluate(
                self.bset.smpl_wn_f.fit(
                    g_iv * ((self.iv_f - sigma_iv)**2 - delta_iv**2)/(self.iv_f - sigma_iv) - 1
                )
            )
        f2 = self.bset.smpl_tau_b.evaluate(
            self.bset.smpl_wn_b.fit(
                (self.omega0**-2) * ( (self.iv_b**2) * d_iv - 2 * self.omega0)
            )
        )
        return (3 * (e1 + e2  - self.omega0 * f2))[0]
```

```
def solve(elsh, sigma_iv, delta_iv, niter, mixing, verbose=False, ph=False, atol=1e-10):
    """
    Solve the self-consistent equation

    ph: Force ph symmetry
    """
    sigma_iv_prev = None
    delta_iv_prev = None
    converged = False
    for iter in range(niter):
        # Update G and F
        g_iv, f_iv = elsh.g_f_iv(mu, sigma_iv, delta_iv)
        g_tau = elsh.to_tau(g_iv, "F")
        f_tau = elsh.to_tau(f_iv, "F")
        if ph:
            g_tau = 0.5 * (g_tau + g_tau[::-1])
        g_tau[g_tau > 0] = 0

        # Update Phi
        phi_tau = elsh.pi_tau(g_tau, f_tau)
        phi_iv = elsh.to_matsu(phi_tau, "B")
        phi_iv.imag = 0

        # Update D
        d_iv = elsh.d_iv(phi_iv)
        d_tau = elsh.to_tau(d_iv, "B")

        # Update Sigma
        sigma_tau, sigma_iv_new = elsh.sigma(g_tau, d_tau)
        sigma_iv_prev = sigma_iv.copy()
        sigma_iv = (1-mixing) * sigma_iv + mixing * sigma_iv_new

        # Update Delta
        delta_iv_new = elsh.delta_iv(f_tau, d_tau)
        delta_iv_prev = delta_iv.copy()
        delta_iv = (1-mixing) * delta_iv + mixing * delta_iv_new
        delta_iv.imag = 0.0
        delta_iv = 0.5 * (delta_iv + delta_iv[::-1])

        diff_sigma = np.abs(sigma_iv_new - sigma_iv_prev).max()
        diff_delta = np.abs(delta_iv_new - delta_iv_prev).max()
        if verbose and iter % 100 == 0:
            print(f"iter= {iter} : diff_sigma= {diff_sigma}, diff_delta={diff_delta}")

        #print(max(diff_sigma, diff_delta), atol)
        if atol is not None and max(diff_sigma, diff_delta) < atol:
            converged = True
            break

    if not converged:
        print("Not converged!")

    # Internal energy
    u = elsh.internal_energy(
        sigma_iv=sigma_iv,
        delta_iv=delta_iv,
        g_iv=g_iv, d_iv=d_iv,
        tau=0.0)

    others = {
        'sigma_tau': sigma_tau,
        'phi_iv': phi_iv,
        'g_iv': g_iv,
        'f_iv': f_iv,
        'd_iv': d_iv,
        'd_tau': d_tau,
        'f_tau': f_tau,
        'g_tau': g_tau,
        'u' : u
        }
    return sigma_iv, delta_iv, others
```

```
def add_noise(arr, noise):
    """
    Add Gaussian noise to an array
    """
    arr += noise*np.random.randn(*arr.shape)
    arr += noise*1j*np.random.randn(*arr.shape)
    return arr
```

## Self-consistent calculation

### Paramaters

We now reproduce the results for $\lambda_0 = 0.125$ shown in Fig. 1 of Ref. [3].
The parameter $g_0$ is related to $\lambda_0$ as

$$
g_0 = \sqrt{\frac{3 \lambda_0 \omega_0}{4}}.
$$

(In the code, we drop the suffix 0 for simplicity.)
We consider a semicircular density of state with a half bandwidth of $1/2$, $T=0.002$, $U=2$, $J/U = 0.03$ and half filling.

```
beta = 500.0

D = 0.5
rho_omega = lambda omega: np.sqrt(D**2 - omega**2) / (0.5 * D**2 * np.pi)

U = 2.0
J = 0.03 * U
omega0 = 0.15
lambda0 = 0.125
mu = 0.0
```

### Setup IR basis

```
eps = 1e-7
wmax = 10*D
bset = sparse_ir.FiniteTempBasisSet(beta, wmax, eps)
```

### Solve the equation

```
# Number of fermionic sampling frequencies
nw_f = bset.wn_f.size

# Initial guess
noise = 1e-2
sigma_iv0 = add_noise(np.zeros(nw_f, dtype=np.complex128), noise)
delta_iv0 = np.full(nw_f, 1.0, dtype=np.complex128)

max_niter = 10000
mixing = 0.1
deg_leggauss = 100 # Degree of Gauss quadrature for DOS integration

# Construct a solver
g = np.sqrt(3 * lambda0 * omega0/4)
elsh = Eliashberg(bset, rho_omega, (-D,D), U, J, omega0, g, deg_leggauss=deg_leggauss)

# Solve the equation
sigma_iv, delta_iv, others = solve(elsh, sigma_iv0, delta_iv0, max_niter, mixing, verbose=True, ph=True)

# Result
res = {"bset": bset, "sigma_iv": sigma_iv, "delta_iv": delta_iv, **others}
```

```
# Let us check `F` is represented compactly in IR
f_l = bset.smpl_wn_f.fit(res['f_iv'])
plt.semilogy(np.abs(f_l), label=r"$|F_l|$", marker="o", ls="")
plt.semilogy(bset.basis_f.s/bset.basis_f.s[0], label=r"$S_l/S_0$")
plt.ylim([1e-8, 10])
plt.legend(frameon=False)
plt.show()
```

### Plot results

```
def plot_res(res):
    """ For plotting results """
    beta = res["bset"].beta
    iv_f = res["bset"].wn_f * (1j*np.pi/beta)
    iv_b = res["bset"].wn_b * (1j*np.pi/beta)

    plt.plot(iv_f.imag, res['sigma_iv'].imag, marker="x")
    plt.xlabel(r"$\omega_n$")
    plt.ylabel(r"Im $\Sigma(\mathrm{i}\omega_n)$")
    plt.xlim([-0.8, 0.8])
    plt.xticks([-0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8])
    plt.grid()
    plt.show()

    plt.plot(iv_b.imag, res['d_iv'].real, marker="x")
    plt.xlabel(r"$\nu_m$")
    plt.ylabel(r"Re $D(\mathrm{i}\nu_m)$")
    plt.xlim([-0.8, 0.8])
    plt.xticks([-0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8])
    #plt.ylim([-0.1, 0.1])
    plt.grid()
    plt.show()

    plt.plot(bset.tau, res['d_tau'].real, marker="x")
    plt.xlabel(r"$\tau$")
    plt.ylabel(r"Re $D(\tau)$")
    plt.grid()
    plt.show()

    plt.plot(iv_f.imag, res['delta_iv'].real, marker="x")
    plt.xlabel(r"$\nu_m$")
    plt.ylabel(r"Re $\Delta(\mathrm{i}\nu_m)$")
    plt.xlim([-0.8, 0.8])
    plt.xticks([-0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8])
    plt.ylim([-0.05, 0.3])
    plt.grid()
    plt.show()

    plt.plot(iv_f.imag, res['f_iv'].real, marker="x")
    plt.xlabel(r"$\omega_n$")
    plt.ylabel(r"Re $F(\mathrm{i}\omega_n)$")
    plt.xlim([-0.8, 0.8])
    plt.xticks([-0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8])
    plt.grid()
    plt.show()

    plt.plot(elsh.iv_b.imag, res['phi_iv'].real, marker="x")
    plt.xlabel(r"$\nu_m$")
    plt.ylabel(r"Re $\Pi(\mathrm{i}\nu_m)$")
    plt.xlim([-0.8, 0.8])
    plt.grid()
    plt.show()

plot_res(res)
```

## Second calculation on temperature dependence

We now compute the temperature dependence of the specific heat for $\lambda_0=0.175$ shown in Fig. 5 of Ref. [1].

```
lambda0 = 0.175
g = np.sqrt(3 * lambda0 * omega0/4)
```

We now compute solutions by changing the temperature gradually. To use the same number of IR functions all the temperatures, we fix the UV cutoff $\Lambda$ to a common value.

```
res_temp = {}
temps = np.linspace(0.009, 0.013, 10)

# Set Lambda to a large enough value for the lowest temperature
Lambda_common = (10 * D)/temps.min()

# Shift the mesh points by dt, which doubles the mesh size,
# to compute the specific heat
dt = 1e-5
temps_all = np.unique(np.hstack((temps, temps + dt)))

bset = None
sigma_iv0 = None
elta_iv0  = None

for temp in temps_all:
    beta = 1/temp

    if bset is None:
        bset = sparse_ir.FiniteTempBasisSet(beta, Lambda_common/beta, eps)
    else:
        # Reuse the SVE results
        bset = sparse_ir.FiniteTempBasisSet(beta, Lambda_common/beta, eps, sve_result=bset.sve_result)

    elsh = Eliashberg(bset, rho_omega, (-D, D), U, J, omega0, g, deg_leggauss=deg_leggauss)

    # Initial guess
    if sigma_iv0 is None:
        noise = 1e-5
        sigma_iv0 = add_noise(np.zeros(bset.wn_f.size, dtype=np.complex128), noise)
        delta_iv0 = np.full(bset.wn_f.size, 1.0, dtype=np.complex128)

    # Solve!
    #sigma_iv, delta_iv, others = solve(elsh, sigma_iv0, delta_iv0, niter, mixing, verbose=True, ph=True)
    max_iter = 1000000
    sigma_iv, delta_iv, others = solve(elsh, sigma_iv0, delta_iv0, max_iter, mixing, verbose=False, ph=True)
    res = {"sigma_iv": sigma_iv, "delta_iv": delta_iv}
    for k, v in others.items():
        res[k] = v
    res_temp[temp] = res

    # Use the converged result for an initial guess for the next temperature
    sigma_iv0 = res["sigma_iv"].copy()
    delta_iv0 = res["delta_iv"].copy()
```

```
u_temps = np.array([res_temp[temp]["u"].real for temp in temps_all])
plt.plot(temps_all, u_temps, marker="x")
plt.xlabel(r"$T$")
plt.ylabel(r"$E(T)$")
plt.show()
```

```
u_dict = {temp: res_temp[temp]["u"].real for temp in temps_all}
specific_heat = np.array([u_dict[temp+dt] - u_dict[temp] for temp in temps])/dt
plt.plot(temps, specific_heat/temps, marker="o")
plt.ylim([0, 1200])
plt.xlabel(r"$T$")
plt.ylabel(r"$C(T)$")
plt.show()
```

<!-- [1] Y. Kaga, P. Werner and S. Hoshino arXiv:2203.06946. -->

[1] G. M. Eliashberg, Zh. Eksperim. Teor. Fiz. **38**, 966
(1960) [Sov. Phys. JETP **11**, 696 (1960)].

[2] G. M. Eliashberg, Zh. Eksperim. Teor. Fiz. **43**, 1005
(1962) [Sov. Phys. JETP **16**, 780 (1963)].

[3] Y. Kaga, P. Werner, and S. Hoshino,  arXiv:2203.06946 (2022).

[4] S. Iimura, M. Hirayama, and S. Hoshino, Phys. Rev. B **104**, L081108 (2021).

[5] Y. Wada, Phys. Rev. **135**, A1481 (1964).


# Tutorial 1: Biological vs. Artificial Neural Networks

**Week 1, Day 3: Multi Layer Perceptrons**

**By Neuromatch Academy**

__Content creators:__ Arash Ash, Surya Ganguli

__Content reviewers:__ Saeed Salehi, Felix Bartsch, Yu-Fang Yang, Antoine De Comite, Melvin Selim Atay, Kelson Shilling-Scrivo

__Content editors:__ Gagana B, Kelson Shilling-Scrivo, Spiros Chavlis

__Production editors:__ Anoop Kulkarni, Kelson Shilling-Scrivo, Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial objectives

In this tutorial, we will explore the Multi-layer Perceptrons (MLPs). MLPs are arguably one of the most tractable models (due to their flexibility) that we can use to study deep learning fundamentals. Here we will learn why they are:

* similar to biological networks
* good at function approximation
* implemented the way they are in PyTorch

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/4ye56/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

This is a GPU free notebook!

```python
# @title Install dependencies
!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet
from evaltools.airtable import AirtableForm

# generate airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W1D3_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')
```

```python
# Imports
import random

import torch
import numpy as np
import matplotlib.pyplot as plt

import torch.nn as nn
import torch.optim as optim
from tqdm.auto import tqdm
from IPython.display import display
from torch.utils.data import DataLoader, TensorDataset
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Plotting functions

def imshow(img):
  img = img / 2 + 0.5     # unnormalize
  npimg = img.numpy()
  plt.imshow(np.transpose(npimg, (1, 2, 0)))
  plt.axis(False)
  plt.show()

def plot_function_approximation(x, relu_acts, y_hat):
  fig, axes = plt.subplots(2, 1)

  # Plot ReLU Activations
  axes[0].plot(x, relu_acts.T);
  axes[0].set(xlabel='x',
              ylabel='Activation',
              title='ReLU Activations - Basis Functions')
  labels = [f"ReLU {i + 1}" for i in range(relu_acts.shape[0])]
  axes[0].legend(labels, ncol = 2)

  # Plot function approximation
  axes[1].plot(x, torch.sin(x), label='truth')
  axes[1].plot(x, y_hat, label='estimated')
  axes[1].legend()
  axes[1].set(xlabel='x',
              ylabel='y(x)',
              title='Function Approximation')

  plt.tight_layout()
  plt.show()
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  device = "cpu"
  if device != "cuda":
    print("GPU is not enabled in this notebook. \n"
          "If you want to enable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `GPU` from the dropdown menu")
  else:
    print("GPU is enabled in this notebook. \n"
          "If you want to disable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `None` from the dropdown menu")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    GPU is not enabled in this notebook.
    If you want to enable it, in the menu under `Runtime` ->
    `Hardware accelerator.` and select `GPU` from the dropdown menu

---
# Section 0: Introduction to MLPs

```python
# @title Video 0: Introduction
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1E3411r7TL", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"Gh0KYl7ViAc", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 0: Introduction')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

---
# Section 1: The Need for MLPs

*Time estimate: ~35 mins*

```python
# @title Video 1: Universal Approximation Theorem
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1SP4y147Uv", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"tg8HHKo1aH4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 1: Universal Approximation Theorem')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Coding Exercise 1: Function approximation with ReLU
We learned that one hidden layer MLPs are enough to approximate any smooth function! Now let's manually fit a sine function using ReLU activation.

We will approximate the sine function using a linear combination (a weighted sum) of ReLUs with slope 1. We need to determine the bias terms (which determines where the ReLU inflection point from 0 to linear occurs) and how to weight each ReLU. The idea is to set the weights iteratively so that the slope changes in the new sample's direction.

First, we generate our "training data" from a sine function using `torch.sine` function.

```python
>>> import torch
>>> torch.manual_seed(2021)
<torch._C.Generator object at 0x7f8734c83830>
>>> a = torch.randn(5)
>>> print(a)
tensor([ 2.2871,  0.6413, -0.8615, -0.3649, -0.6931])
>>> torch.sin(a)
tensor([ 0.7542,  0.5983, -0.7588, -0.3569, -0.6389])
```

These are the points we will use to learn how to approximate the function. We have 10 training data points so we will have 9 ReLUs (we don't need a ReLU for the last data point as we don't have anything to the right of it to model).

We first need to figure out the bias term for each ReLU and compute the activation of each ReLU where:

\begin{equation}
y(x) = \text{max}(0,x+b)
\end{equation}

We then need to figure out the correct weights on each ReLU so the linear combination approximates the desired function.

```python
def approximate_function(x_train, y_train):

  ####################################################################
  # Fill in missing code below (...),
  # then remove or comment the line below to test your function
  #raise NotImplementedError("Complete approximate_function!")
  ####################################################################

  # Number of relus
  n_relus = x_train.shape[0] - 1

  # x axis points (more than x train)
  x = torch.linspace(torch.min(x_train), torch.max(x_train), 1000)

  ## COMPUTE RELU ACTIVATIONS

  # First determine what bias terms should be for each of `n_relus` ReLUs
  b = -x_train[:-1]
  #b[0] = y_train[0] - x_train[0]

  # Compute ReLU activations for each point along the x axis (x)
  relu_acts = torch.zeros((n_relus, x.shape[0]))

  for i_relu in range(n_relus):
    relu_acts[i_relu, :] = torch.relu(x + b[i_relu])

  ## COMBINE RELU ACTIVATIONS

  # Set up weights for weighted sum of ReLUs
  combination_weights = torch.zeros((n_relus, ))

  # Figure out weights on each ReLU
  prev_slope = 1
  for i in range(1, n_relus):
    delta_x = x_train[i+1] - x_train[i]
    slope = (y_train[i+1] - y_train[i]) / delta_x
    combination_weights[i] = slope - prev_slope
    prev_slope = slope

  # Get output of weighted sum of ReLU activations for every point along x axis
  y_hat = combination_weights @ relu_acts

  return y_hat, relu_acts, x

# add event to airtable
atform.add_event('Coding Exercise 1: Function approximation with ReLU')

# Make training data from sine function
N_train = 10
x_train = torch.linspace(0, 2*np.pi, N_train).view(-1, 1)
y_train = torch.sin(x_train)

## Uncomment the lines below to test your function approximation
y_hat, relu_acts, x = approximate_function(x_train, y_train)
plot_function_approximation(x, relu_acts, y_hat)
```

```python

```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_e09a57e7.py)

*Example output:*

As you see in the top panel, we obtain 10 shifted ReLUs with the same slope. These are the basis functions that MLP uses to span the functional space, i.e., MLP finds a linear combination of these ReLUs.

---
# Section 2: MLPs in Pytorch

*Time estimate: ~1hr and 20 mins*

```python
# @title Video 2: Building MLPs in PyTorch
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1zh411z7LY", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"XtwLnaYJ7uc", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 2: Building MLPs in PyTorch')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

In the previous segment, we implemented a function to approximate any smooth function using MLPs. We saw that using Lipschitz continuity; we can prove that our approximation is mathematically correct. MLPs are fascinating, but before we get into the details on designing them, let's familiarize ourselves with some basic terminology of MLPs- layer, neuron, depth, width, weight, bias, and activation function. Armed with these ideas, we can now design an MLP given its input, hidden layers, and output size.

## Coding Exercise 2: Implement a general-purpose MLP in Pytorch
The objective is to design an MLP with these properties:
* works with any input (1D, 2D, etc.)
* construct any number of given hidden layers using `nn.Sequential()` and `add_module()` function
* use the same given activation function (i.e., [Leaky ReLU](https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html)) in all hidden layers.

**Leaky ReLU** is described by the following mathematical formula:

\begin{equation}
\text{LeakyReLU}(x) = \text{max}(0,x) + \text{negative_slope} \cdot \text{min}(0, x) =
\left\{
  \begin{array}{ll}
  x & ,\; \text{if} \; x \ge 0 \\
  \text{negative_slope} \cdot x & ,\;  \text{otherwise}
  \end{array}
\right.
\end{equation}

```python
class Net(nn.Module):
  def __init__(self, actv, input_feature_num, hidden_unit_nums, output_feature_num):
    super(Net, self).__init__()
    self.input_feature_num = input_feature_num # save the input size for reshapinng later
    self.mlp = nn.Sequential() # Initialize layers of MLP

    in_num = input_feature_num # initialize the temporary input feature to each layer
    for i in range(len(hidden_unit_nums)): # Loop over layers and create each one

      ####################################################################
      # Fill in missing code below (...),
      # then remove or comment the line below to test your function
      #raise NotImplementedError("Create MLP Layer")
      ####################################################################

      out_num = hidden_unit_nums[i] # assign the current layer hidden unit from list
      layer = nn.Linear(in_num, out_num) # use nn.Linear to define the layer
      in_num = out_num # assign next layer input using current layer output
      self.mlp.add_module('Linear_%d'%i, layer) # append layer to the model with a name

      actv_layer = eval('nn.%s'%actv) # Assign activation function (eval allows us to instantiate object from string)
      self.mlp.add_module('Activation_%d'%i, actv_layer) # append activation to the model with a name

    out_layer = nn.Linear(in_num, output_feature_num) # Create final layer
    self.mlp.add_module('Output_Linear', out_layer) # append the final layer

  def forward(self, x):
    # reshape inputs to (batch_size, input_feature_num)
    # just in case the input vector is not 2D, like an image!
    x = x.view(-1, self.input_feature_num)

    ####################################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your function
    #raise NotImplementedError("Run MLP model")
    ####################################################################

    logits = self.mlp(x) # forward pass of MLP
    return logits

# add event to airtable
atform.add_event('Coding Exercise 2: Implement a general-purpose MLP in Pytorch')

input = torch.zeros((100, 2))
## Uncomment below to create network and test it on input
net = Net(actv='LeakyReLU(0.1)', input_feature_num=2, hidden_unit_nums=[100, 10, 5], output_feature_num=1).to(DEVICE)
y = net(input.to(DEVICE))
print(f'The output shape is {y.shape} for an input of shape {input.shape}')
```

    The output shape is torch.Size([100, 1]) for an input of shape torch.Size([100, 2])

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_b0675c60.py)

```
The output shape is torch.Size([100, 1]) for an input of shape torch.Size([100, 2])
```

## Section 2.1: Classification with MLPs

```python
# @title Video 3: Cross Entropy
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Ag41177mB", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"N8pVCbTlves", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 3: Cross Entropy')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

The main loss function we could use out of the box for multi-class classification for `N` samples and `C` number of classes is:
* CrossEntropyLoss:
This criterion expects a batch of predictions `x` with shape `(N, C)` and class index in the range $[0, C-1]$ as the target (label) for each `N` samples, hence a batch of `labels` with shape `(N, )`. There are other optional parameters like class weights and class ignores. Feel free to check the PyTorch documentation [here](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) for more detail. Additionally, [here](https://sparrow.dev/cross-entropy-loss-in-pytorch/) you can learn where is appropriate to use the CrossEntropyLoss.

To get CrossEntropyLoss of a sample $i$, we could first calculate $-\log(\text{softmax(x}))$ and then take the element corresponding to $\text { labels }_i$ as the loss. However, due to numerical stability, we implement this more stable equivalent form,

\begin{equation}
\operatorname{loss}(x_i, \text { labels }_i)=-\log \left(\frac{\exp (x[\text { labels }_i])}{\sum_{j} \exp (x[j])}\right)=-x_i[\text { labels }_i]+\log \left(\sum_{j=1}^C \exp (x_i[j])\right)
\end{equation}

### Coding Exercise 2.1: Implement Batch Cross Entropy Loss

To recap, since we will be doing batch learning, we'd like a loss function that given:
* a batch of predictions `x` with shape `(N, C)`
* a batch of `labels` with shape `(N, )` that ranges from `0` to `C-1`

returns the average loss $L$ calculated according to:

\begin{align}
loss(x_i, \text { labels }_i) &= -x_i[\text { labels }_i]+\log \left(\sum_{j=1}^C \exp (x_i[j])\right) \\
L &= \frac{1}{N} \sum_{i=1}^{N}{loss(x_i, \text { labels }_i)}
\end{align}

Steps:

1.   Use indexing operation to get predictions of class corresponding to the labels (i.e., $x_i[\text { labels }_i]$)
2.   Compute $loss(x_i, \text { labels }_i)$ vector (`losses`) using `torch.log()` and `torch.exp()` without Loops!
3. Return the average of the loss vector

```python
def cross_entropy_loss(x, labels):
  # x is the model predictions we'd like to evaluate using lables
  x_of_labels = torch.zeros(len(labels))
  ####################################################################
  # Fill in missing code below (...),
  # then remove or comment the line below to test your function
  #raise NotImplementedError("Cross Entropy Loss")
  ####################################################################
  # 1. prediction for each class corresponding to the label
  for i, label in enumerate(labels):
    x_of_labels[i] = x[i, label]
  # 2. loss vector for the batch
  losses = -x_of_labels + torch.log(torch.sum(torch.exp(x), axis=1))
  # 3. Return the average of the loss vector
  avg_loss = torch.mean(losses)

  return avg_loss

# add event to airtable
atform.add_event('Coding Exercise 2.1: Implement Batch Cross Entropy Loss')

labels = torch.tensor([0, 1])
x = torch.tensor([[10.0, 1.0, -1.0, -20.0],  # correctly classified
                  [10.0, 10.0, 2.0, -10.0]])  # Not correctly classified
CE = nn.CrossEntropyLoss()
pytorch_loss = CE(x, labels).item()
## Uncomment below to test your function
our_loss = cross_entropy_loss(x, labels).item()
print(f'Our CE loss: {our_loss:0.8f}, Pytorch CE loss: {pytorch_loss:0.8f}')
print(f'Difference: {np.abs(our_loss - pytorch_loss):0.8f}')
```

    Our CE loss: 0.34672737, Pytorch CE loss: 0.34672749
    Difference: 0.00000012

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_f3376d8f.py)

```
Our CE loss: 0.34672737, Pytorch CE loss: 0.34672749
Difference: 0.00000012
```

## Section 2.2: Spiral classification dataset
Before we could start optimizing these loss functions, we need a dataset!

Let's turn this fancy-looking equation into a classification dataset

\begin{equation}
\begin{array}{c}
X_{k}(t)=t\left(\begin{array}{c}
\sin \left[\frac{2 \pi}{K}\left(2 t+k-1\right)\right]+\mathcal{N}\left(0, \sigma\right) \\
\cos \left[\frac{2 \pi}{K}\left(2 t+k-1\right)\right]+\mathcal{N}\left(0, \sigma\right)
\end{array}\right)
\end{array}, \quad 0 \leq t \leq 1, \quad k=1, \ldots, K
\end{equation}

```python
def create_spiral_dataset(K, sigma, N):

  # Initialize t, X, y
  t = torch.linspace(0, 1, N)
  X = torch.zeros(K*N, 2)
  y = torch.zeros(K*N)

  # Create data
  for k in range(K):
    X[k*N:(k+1)*N, 0] = t*(torch.sin(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))
    X[k*N:(k+1)*N, 1] = t*(torch.cos(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))
    y[k*N:(k+1)*N] = k

  return X, y

# Set parameters
K = 4
sigma = 0.16
N = 1000

set_seed(seed=SEED)
X, y = create_spiral_dataset(K, sigma, N)
plt.scatter(X[:, 0], X[:, 1], c = y)
plt.show()
```

## Section 2.3: Training and Evaluation

```python
# @title Video 4: Training and Evaluating an MLP
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1QV411p7mF", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"DfXZhRfBEqQ", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 4: Training and Evaluating an MLP')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

### Coding Exercise 2.3: Implement it for a classfication task
Now that we have the Spiral dataset and a loss function, it's your turn to implement a simple train/test split for training and validation.

Steps to follow:
  * Dataset shuffle
  * Train/Test split (20% for test)
  * Dataloader definition
  * Training and Evaluation

```python
def shuffle_and_split_data(X, y, seed):
  # set seed for reproducibility
  torch.manual_seed(seed)
  # Number of samples
  N = X.shape[0]
  ####################################################################
  # Fill in missing code below (...),
  # then remove or comment the line below to test your function
  #raise NotImplementedError("Shuffle & split data")
  ####################################################################
  # Shuffle data
  shuffled_indices = torch.randperm(N)   # get indices to shuffle data, could use torch.randperm
  X = X[shuffled_indices]
  y = y[shuffled_indices]

  # Split data into train/test
  test_size = int(0.2 * N)    # assign test datset size using 20% of samples
  X_test = X[:test_size]
  y_test = y[:test_size]
  X_train = X[test_size:]
  y_train = y[test_size:]

  return X_test, y_test, X_train, y_train

# add event to airtable
atform.add_event('Coding Exercise 2.3: Implement for a classfication task')

## Uncomment below to test your function
X_test, y_test, X_train, y_train = shuffle_and_split_data(X, y, seed=SEED)
plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test)
plt.title('Test data')
plt.show()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_083a4d77.py)

*Example output:*

And we need to make a Pytorch data loader out of it. Data loading in PyTorch can be separated in 2 parts:
* Data must be wrapped on a Dataset parent class where the methods __getitem__ and __len__ must be overrided. Note that at this point the data is not loaded on memory. PyTorch will only load what is needed to the memory. Here `TensorDataset` does this for us directly.
* Use a Dataloader that will actually read the data in batches and put into memory. Also, the option of `num_workers > 0` allows multithreading, which prepares multiple batches in the queue to speed things up.

```python
g_seed = torch.Generator()
g_seed.manual_seed(SEED)

batch_size = 128
test_data = TensorDataset(X_test, y_test)
test_loader = DataLoader(test_data, batch_size=batch_size,
                         shuffle=False, num_workers=2,
                         worker_init_fn=seed_worker,
                         generator=g_seed)

train_data = TensorDataset(X_train, y_train)
train_loader = DataLoader(train_data, batch_size=batch_size, drop_last=True,
                        shuffle=True, num_workers=2,
                         worker_init_fn=seed_worker,
                         generator=g_seed)
```

Let's write a general-purpose training and evaluation code and keep it in our pocket for next tutorial as well. So make sure you review it to see what it does.

Note that `model.train()` tells your model that you are training the model. So layers like dropout, batchnorm etc. which behave different on the train and test procedures know what is going on and hence can behave accordingly. And to turn off training mode we set `model.eval()`

```python
def train_test_classification(net, criterion, optimizer, train_loader,
                              test_loader, num_epochs=1, verbose=True,
                              training_plot=False, device='cpu'):

  net.train()
  training_losses = []
  for epoch in tqdm(range(num_epochs)):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
      # get the inputs; data is a list of [inputs, labels]
      inputs, labels = data
      inputs = inputs.to(device).float()
      labels = labels.to(device).long()

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = net(inputs)

      loss = criterion(outputs, labels)
      loss.backward()
      optimizer.step()

      # print statistics
      if verbose:
        training_losses += [loss.item()]

  net.eval()
  def test(data_loader):
    correct = 0
    total = 0
    for data in data_loader:
      inputs, labels = data
      inputs = inputs.to(device).float()
      labels = labels.to(device).long()

      outputs = net(inputs)
      _, predicted = torch.max(outputs, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum().item()

    acc = 100 * correct / total
    return total, acc

  train_total, train_acc = test(train_loader)
  test_total, test_acc = test(test_loader)

  if verbose:
    print(f"Accuracy on the {train_total} training samples: {train_acc:0.2f}")
    print(f"Accuracy on the {test_total} testing samples: {test_acc:0.2f}")

  if training_plot:
    plt.plot(training_losses)
    plt.xlabel('Batch')
    plt.ylabel('Training loss')
    plt.show()

  return train_acc, test_acc
```

### Think! 2.3.1: What's the point of .eval() and .train()?

Is it necessary to use `net.train()` and `net.eval()` for our MLP model? why?

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q1', text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_51988471.py)

Now let's put everything together and train your first deep-ish model!

```python
set_seed(SEED)
net = Net('ReLU()', X_train.shape[1], [128], K).to(DEVICE)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=1e-3)
num_epochs = 100

_, _ = train_test_classification(net, criterion, optimizer, train_loader,
                                 test_loader, num_epochs=num_epochs,
                                 training_plot=True, device=DEVICE)
```

And finally, let's visualize the learned decision-map. We know you're probably running out of time, so we won't make you write code now! But make sure you have reviewed it since we'll start with another visualization technique next time.

```python
def sample_grid(M=500, x_max=2.0):
  ii, jj = torch.meshgrid(torch.linspace(-x_max, x_max, M),
                          torch.linspace(-x_max, x_max, M))
  X_all = torch.cat([ii.unsqueeze(-1),
                     jj.unsqueeze(-1)],
                     dim=-1).view(-1, 2)
  return X_all

def plot_decision_map(X_all, y_pred, X_test, y_test,
                      M=500, x_max=2.0, eps=1e-3):
  decision_map = torch.argmax(y_pred, dim=1)

  for i in range(len(X_test)):
    indices = (X_all[:, 0] - X_test[i, 0])**2 + (X_all[:, 1] - X_test[i, 1])**2 < eps
    decision_map[indices] = (K + y_test[i]).long()

  decision_map = decision_map.view(M, M)
  plt.imshow(decision_map, extent=[-x_max, x_max, -x_max, x_max], cmap='jet')
  plt.show()
```

```python
X_all = sample_grid()
y_pred = net(X_all)
plot_decision_map(X_all, y_pred, X_test, y_test)
```

### Think! 2.3.2: Does it generalize well?
Do you think this model is performing well outside its training distribution? Why?

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q2' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_d90d9e4b.py)

What would be your suggestions to increase models ability to generalize? Think about it and discuss with your pod.

---
# Summary

In this tutorial we have explored the Multi-leayer Perceptrons (MLPs). More specifically, we have discuss the similarities of artificial and biological neural networks (for more information see the Bonus section), we have learned the Universal Approximation Theorem, and we have implemented MLPs in PyTorch.

```python
# @title Airtable Submission Link
from IPython import display as IPydisplay
IPydisplay.HTML(
   f"""
 <div>
   <a href= "{atform.url()}" target="_blank">
   </a>
   </div>""" )
```

  <a href= "https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303?data=eyJmb3JtX2lkIjogImFwcG43VmRQUnNlU29NWEVHIiwgInRhYmxlX25hbWUiOiAiVzFEM19UMSIsICJhbnN3ZXJzIjoge30sICJldmVudHMiOiBbeyJldmVudCI6ICJpbml0IiwgInRzIjogMTYyODEwNjMxMC40MzU5OTIyfSwgeyJldmVudCI6ICJWaWRlbyAwOiBJbnRyb2R1Y3Rpb24iLCAidHMiOiAxNjI4MTA2MzExLjI1Njg1OTV9LCB7ImV2ZW50IjogIlZpZGVvIDE6IFVuaXZlcnNhbCBBcHByb3hpbWF0aW9uIFRoZW9yZW0iLCAidHMiOiAxNjI4MTA2MzExLjMyNDg2NH0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDE6IEZ1bmN0aW9uIGFwcHJveGltYXRpb24gd2l0aCBSZUxVIiwgInRzIjogMTYyODEwNjMxMS4zMzE4ODQ5fSwgeyJldmVudCI6ICJWaWRlbyAyOiBCdWlsZGluZyBNTFBzIGluIFB5VG9yY2giLCAidHMiOiAxNjI4MTA2MzExLjc0ODY2MDh9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAyOiBJbXBsZW1lbnQgYSBnZW5lcmFsLXB1cnBvc2UgTUxQIGluIFB5dG9yY2giLCAidHMiOiAxNjI4MTA2MzExLjc1NTc0MTR9LCB7ImV2ZW50IjogIlZpZGVvIDM6IENyb3NzIEVudHJvcHkiLCAidHMiOiAxNjI4MTA2MzExLjg5NDE3MzF9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAyLjE6IEltcGxlbWVudCBCYXRjaCBDcm9zcyBFbnRyb3B5IExvc3MiLCAidHMiOiAxNjI4MTA2MzExLjkwMDMzODZ9LCB7ImV2ZW50IjogIlZpZGVvIDQ6IFRyYWluaW5nIGFuZCBFdmFsdWF0aW5nIGFuIE1MUCIsICJ0cyI6IDE2MjgxMDYzMTIuMTM2NTF9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSAyLjM6IEltcGxlbWVudCBmb3IgYSBjbGFzc2ZpY2F0aW9uIHRhc2siLCAidHMiOiAxNjI4MTA2MzEyLjE0MjMxfSwgeyJldmVudCI6ICJ1cmwgZ2VuZXJhdGVkIiwgInRzIjogMTYyODEwNjMyMC43Nzk0Njc4fV19" target="_blank">

---
# Bonus: Neuron Physiology and Motivation to Deep Learning

This section will motivate one of the most popular nonlinearities in deep learning, the ReLU nonlinearity, by starting from the biophysics of neurons and obtaining the ReLU nonlinearity through a sequence of approximations. We will also show that neuronal biophysics sets a time scale for signal propagation speed through the brain. This time scale implies that neural circuits underlying fast perceptual and motor processing in the brain may not be excessively deep.

This biological motivation for deep learning is not strictly necessary to follow the rest of this course, so this section can be safely skipped and **is a bonus section**.

```python
# @title Video 5: Biological to Artificial Neurons
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1mf4y157vf", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"ELAbflymSLo", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 5: Biological to Artificial Neurons')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Leaky Integrate-and-fire (LIF)
The basic idea of LIF neuron was proposed in 1907 by Louis Édouard Lapicque, long before we understood the electrophysiology of a neuron (see a translation of [Lapicque's paper](https://pubmed.ncbi.nlm.nih.gov/17968583/) ). More details of the model can be found in the book [**Theoretical neuroscience**](http://www.gatsby.ucl.ac.uk/~dayan/book/) by Peter Dayan and Laurence F. Abbott.

The model dynamics is defined with the following formula,

\begin{equation}
\frac{d V}{d t}=\left\{\begin{array}{cc}
\frac{1}{C}\left(-\frac{V}{R}+I \right) & t>t_{r e s t} \\
0 & \text { otherwise }
\end{array}\right.
\end{equation}

Note that $V$, $C$, and $R$ are the membrane voltage, capacitance, and resitance of the neuron respectively and $-\frac{V}{R}$ is the leakage current. When $I$ is sufficiently strong such that $V$ reaches a certain threshold value $V_{\rm th}$, it momentarily spikes and then $V$ is reset to $V_{\rm reset}< V_{\rm th}$, and voltage stays at $V_{\rm reset}$ for $\tau_{\rm ref}$ ms, mimicking the refractoriness of the neuron during an action potential (note that $V_{\rm reset}$ and $\tau_{\rm ref}$ is assumed to be zero in the lecture):

\begin{eqnarray}
V(t)=V_{\rm reset} \text{  for } t\in(t_{\text{sp}}, t_{\text{sp}} + \tau_{\text{ref}}]
\end{eqnarray}

where $t_{\rm sp}$ is the spike time when $V(t)$ just exceeded $V_{\rm th}$.

Thus, the LIF model captures the facts that a neuron:
- performs spatial and temporal integration of synaptic inputs
- generates a spike when the voltage reaches a certain threshold
- goes refractory during the action potential
- has a leaky membrane

For in-depth content on computational models of neurons, follow the [NMA](https://www.neuromatchacademy.org/) tutorial 1 of *Biological Neuron Models*. Specifically, for NMA-CN 2021 follow this [Tutorial](https://github.com/NeuromatchAcademy/course-content/blob/master/tutorials/W2D3_BiologicalNeuronModels/W2D3_Tutorial1.ipynb).

## Simulating an LIF Neuron

In the cell below is given a function for LIF neuron model with it's arguments described.

Note that we will use Euler's method to make a numerical approximation to a derivative. Hence we will use the following implementation of the model dynamics,

\begin{equation}
V_n=\left\{\begin{array}{cc}
V_{n-1} + \frac{1}{C}\left(-\frac{V}{R}+I \right) \Delta t & t>t_{r e s t} \\
0 & \text { otherwise }
\end{array}\right.
\end{equation}

```python
def run_LIF(I, T=50, dt=0.1, t_ref=10,
            Rm=1, Cm=10, Vth=1, V_spike=0.5):
  """
  Simulate the LIF dynamics with external input current

  Args:
    I          : input current (mA)
    T          : total time to simulate (msec)
    dt         : simulation time step (msec)
    t_ref      : refractory period (msec)
    Rm         : resistance (kOhm)
    Cm         : capacitance (uF)
    Vth        : spike threshold (V)
    V_spike    : spike delta (V)

  Returns:
    time       : time points
    Vm         : membrane potentials
  """

  # Set up array of time steps
  time = torch.arange(0, T+dt, dt)

  # Set up array for tracking Vm
  Vm = torch.zeros(len(time))

  # Iterate over each time step
  t_rest = 0
  for i, t in enumerate(time):

    # If t is after refractory period
    if t > t_rest:
      Vm[i] = Vm[i-1] + 1/Cm*(-Vm[i-1]/Rm + I)  * dt

    # If Vm is over the threshold
    if Vm[i] >= Vth:

      # Increase volatage by change due to spike
      Vm[i] += V_spike

      # Set up new refactory period
      t_rest = t + t_ref

  return time, Vm

sim_time, Vm = run_LIF(1.5)
# Plot the membrane voltage across time
plt.plot(sim_time, Vm)
plt.title('LIF Neuron Output')
plt.ylabel('Membrane Potential (V)')
plt.xlabel('Time (msec)')
plt.show()
```

### Interactive Demo: Neuron's transfer function explorer for different $R_m$ and $t_{ref}$
We know that real neurons communicate by modulating the spike count meaning that more input current causes a neuron to spike more often. Therefore, to find an input-output relationship, it makes sense to characterize their spike count as a function of input current. This is called the neuron's input-output transfer function. Let's plot the neuron's transfer function and see how it changes with respect to the **membrane resistance** and **refractory time**?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!
my_layout = widgets.Layout()

@widgets.interact(Rm=widgets.FloatSlider(1., min=1, max=100.,
                                         step=0.1, layout=my_layout),
                  t_ref=widgets.FloatSlider(1., min=1, max=100.,
                                            step=0.1, layout=my_layout)
                  )

def plot_IF_curve(Rm, t_ref):
  T = 1000 # total time to simulate (msec)
  dt = 1 # simulation time step (msec)
  Vth = 1 # spike threshold (V)
  Is_max = 2
  Is = torch.linspace(0, Is_max, 10)
  spike_counts = []
  for I in Is:
    _, Vm = run_LIF(I, T=T, dt=dt, Vth=Vth, Rm=Rm, t_ref=t_ref)
    spike_counts += [torch.sum(Vm > Vth)]

  plt.plot(Is, spike_counts)
  plt.title('LIF Neuron: Transfer Function')
  plt.ylabel('Spike count')
  plt.xlabel('I (mA)')
  plt.xlim(0, Is_max)
  plt.ylim(0, 80)
  plt.show()
```

    interactive(children=(FloatSlider(value=1.0, description='Rm', min=1.0), FloatSlider(value=1.0, description='t…

### Think!: Real and Artificial neuron similarities

What happens at infinite membrane resistance ($R_m$) and small refactory time ($t_{ref}$)? Why?

Take 10 mins to discuss the similarity between a real neuron and an artificial one with your pod.

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_d58d2933.py)