# **[HW7] Practice_5**
1. Install packages
2. Geometry of eigendecomposition
3. Eigendecomposition

선형대수 실습은, exercise 혹은 coding 문제풀이에 집중하기 보다는, 수업시간에 배웠던 개념들을 visualize 하면서 이를 폭넓게 이해하는 데에 초점을 맞추고 있습니다. 실습에서 사용한 예시 외에도, 다양한 matrix들을 visualize 하면서 개념을 익혀보시기 바랍니다.

# 1. Install packages

> 필요한 package를 설치하고 import합니다

```python
# visualization을 위한 helper code입니다.
from urllib.request import urlretrieve
URL = 'https://go.gwu.edu/engcomp4plot'
urlretrieve(URL, 'plot_helper.py')

import sys
sys.path.append('../scripts/')

# 다음 세 custom function (1)plot_vector, (2)plot_linear_transformation, (3) plot_linear_transformations
# 을 사용할 것입니다.
from plot_helper import *
```

```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import scipy.linalg
import sympy as sy

sy.init_printing()
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

# 2. Geometry of eigendecomposition

> Eigendecomposition은 선형대수에서 가장 중요한 주제 중 하나입니다. 여러 행렬들의 eigenvector들을 구하고 이를 시각화하면서, 해당 성질들을 이해해보겠습니다.

## Eigenvectors of $A = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}$

> 다음과 같은 행렬 $A = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}$로 표현되는 linear transformation을 생각해봅시다. 아래 plot_linear_transformation(A)는 $\begin{bmatrix}1\\0\end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 $A$에 의해 transform 되는 과정을 visualize하는 custom 함수입니다.

```python
A = np.array([[2,1], [1,2]])
print(A)
plot_linear_transformation(A)
```

> $A$는 $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$ 을 $\begin{bmatrix} 2 \\ 1 \end{bmatrix}$로, $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$을 $\begin{bmatrix} 1 \\ 2 \end{bmatrix}$로 옮기는 linear transformation입니다. $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$, $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$ 외에 다른 vector들은 어떻게 옮겨지는 지 visualize 해봅시다.

```python
# 길이(norm)가 1인 vector들
alpha = np.linspace(0, 2*np.pi, 41)  # x축과의 각도 (0~2pi)
vectors = list(zip(np.cos(alpha), np.sin(alpha)))
vectors = np.array(vectors)

newvectors = A @ vectors.T
newvectors = newvectors.T

plot_vector(vectors)  # plot_vector는 주어진 vector들을 plot하는 custom function 입니다.
```

```python
plot_vector(newvectors)
```

> 위 $A$는 원을 타원으로 만드는 linear transformation임을 확인할 수 있습니다. 이 타원의 semi major axis (장축의 절반), semi minor axis (단축의 절반)을 찾기 위해선, transform된 newvectors의 길이가 가장 길고 짧은 두 vector를 찾으면 됩니다.

```python
lengths = np.linalg.norm(newvectors, axis=1)  # newvectos 각각의 norm
```

```python
semi_major_index = np.argmax(lengths)  # norm이 가장 긴 것이 타원의 장축
semi_major_vector = newvectors[semi_major_index]
semi_major_length = lengths[semi_major_index]

print(semi_major_vector)  # 장축 vector
print(semi_major_length)
```

    [-2.121 -2.121]
    3.0000000000000004

```python
semi_minor_index = np.argmin(lengths)  # norm이 가장 짧은 것이 타원의 장축
semi_minor_vector = newvectors[semi_minor_index]
semi_minor_length = lengths[semi_minor_index]

print(semi_minor_vector)  # 단축 vector
print(semi_minor_length)
```

    [-0.707  0.707]
    1.0

> 장축은 $y=x$ 위에 있는 vector, 즉 x축과 45도를 이루는 vector임을 알 수 있습니다. 단축은 $y=-x$ 위에 있는 vector, 즉 x축과 $-45(=\pi/4)$도를 이루는 vector임을 알 수 있습니다. 이 두 vector $\begin{bmatrix} -2.121 \\ -2.121 \end{bmatrix}$, $\begin{bmatrix} -0.707 \\ 0.707 \end{bmatrix}$은 원래 원의 어느 vector가 linear transform된 것일까요?

> 이를 구하기 위해 $A^{-1}$을 곱해서, 원래 vector를 구해봅시다.

```python
A_inv = np.linalg.inv(A)
v1 = A_inv @ semi_major_vector
v2 = A_inv @ semi_minor_vector

print(v1)  # v1 -> A -> 장축
print(v2)  # v2 -> A -> 단축
```

    [-0.707 -0.707]
    [-0.707  0.707]

> $v_1$과 semi major vector, 그리고 $v_2$와 semi minor vector은 각각 방향이 같습니다 ($v_1$과 semi major vector는 y=x, $v_2$와 semi minor vector는 $y=-x$위에 존재). 즉 $Av_1 = 3v_1, Av_2=1v_2$임을 알 수 있습니다. 따라서, 위 $A$는 원을 타원으로 바꾸는 matrix이며, 이때 타원의 장축, 단축이 각각 eigenvector가 됨을 알 수 있습니다. \\

**(주의) 이는 모든 타원에 대해서 성립하는 것이 아닙니다. $A$가 symmetric matrix라는 점이 중요한데, 실습 마지막 부분에서 더 자세히 알아보겠습니다.**

## Scaling과 Rotation으로 A 만들기

```python
plot_vector(vectors)
```

```python
plot_vector(newvectors)
```

> $A$로 만들어진 타원(newvectors)을 보면, scaling과 rotation의 조합으로 만들 수 있는 것처럼 보입니다. 예를 들면, 반지름이 1인 원(vectors)을 (1) 가로로 3배(장축 방향의 eigenvalue가 3이니까) 늘린 다음 (i.e., y축 성분은 그냥 놔두고, x축 성분을 3배), (2) 시계 반대방향으로 45를 돌린다면 위의 타원을 만들 수 있지 않을까요? 한번 이를 visualize 해봅시다.

### Exercise

```python
# answer
S = np.array([[3,0],[0,1]])

ellipse = S@vectors.T
ellipse = ellipse.T
```

```python
# 정답 plot 예시
plot_vector(ellipse)
```

> 이제 이 가로로 3배 늘려진 ellipse를 시계 반대방향으로 45도 돌려봅시다. Rotation matrix는 $\mathbf{i}$는 $(\cos\theta, \sin\theta)$로 옮기고, $\mathbf{j}$는 $(-\sin\theta, \cos\theta)$로 옮기는 matrix입니다.

 출처: Ref. [1]

$$R = \begin{bmatrix} \cos{\theta} & -\sin{\theta} \\ \sin{\theta} & \cos{\theta} \end{bmatrix}$$

> 이제 위의 ellipse를 $R$을 통해 45도 회전시켜보도록 하겠습니다.

### Exercise

```python
# answer
theta = numpy.pi/4
R = numpy.array([[numpy.cos(theta), -numpy.sin(theta)],
                 [numpy.sin(theta), numpy.cos(theta)]])

rotated_ellipse = R@ellipse.T
rotated_ellipse = rotated_ellipse.T
```

```python
# 정답 plot 예시
plot_vector(rotated_ellipse)
```

> 위의 rotated_ellipse는 처음에 $A$에 의해 변환된 아래 그림과 겉으로 보기에 같아 보입니다.

```python
plot_vector(newvectors)
```

> 즉, $R$과 $S$의 조합으로 A를 만드는 데 성공한 것처럼 보입니다. 하지만 vector 하나 하나를 transformation 시켜보면, 저 $R$과 $S$의 조합은 사실 A가 아님을 알 수 있습니다.

```python
plot_linear_transformations(S,R)
```

> 위에서 $v_1$은 matrix $A$의 eigenvector임을 알 수 있었습니다 ($Av_1 = 3v_1$). 하지만, 위 그림에선 변환 전 (Before transformation)의 초록색 vector $\mathbf{i}$가, 변환 후 (After 2 transformations)의 초록색 vector $\begin{bmatrix} \alpha \\ \alpha \end{bmatrix}$가 됨을 알 수 있습니다 ($\alpha$는 상수). 이 말은, $v_1$는 $S$@$R$의 eigenvector가 아니라는 뜻입니다. vector의 방향이 바뀌기 때문입니다.

$$A\begin{bmatrix} \alpha \\ \alpha \end{bmatrix} = 3 \begin{bmatrix} \alpha \\ \alpha \end{bmatrix} \\
RS\begin{bmatrix} \alpha \\ \alpha \end{bmatrix} \neq 3\begin{bmatrix} \alpha \\ \alpha \end{bmatrix}$$

> 그럼 $v_1$은 실제로 어떻게 바뀌고 있을까요?

### Exercise

```python
#answer
v1 = -v1
print(v1)

scaled_v1 = S @ v1
rotated_scaled_v1 = R @ scaled_v1
plot_vector([v1, rotated_scaled_v1])

print(rotated_scaled_v1)
```

```python

```

> 즉 v1이 크기 뿐 아니라 방향마저 바뀌었음을 알 수 있고, 눈으로 언뜻 보기와는 다르게 $A$와 $R$@$S$는 다름을 알 수 있습니다.

$$ A \neq RS$$

## A의 Eigendecomposition

> 그렇다고 해서, rotation과 scaling의 조합으로 $A$를 만들 수 없는 것은 아닙니다. 정답부터 말하자면, (1) rotation을 먼저 시계방향으로 45도로 돌린 후 (2) 이를 scaling하고 (3) 다시 반시계방향으로 45도로 돌리면 정확히 $A$를 만들 수 있습니다.

> 이를 자세히 알아보기전에, 시계방향으로 돌리는 rotation matrix는 $R (=\begin{bmatrix} \cos{\theta} & -\sin{\theta} \\ \sin{\theta} & \cos{\theta} \end{bmatrix})$에서 $\theta$를 $-\theta$로 바꿔주면 됩니다 (위의 rotation matrix에서 설명했던 것의 반대의 작용을 하는 matrix, 즉 $(\cos\theta, \sin\theta)$를  $\mathbf{i}$로 옮기고, $(-\sin\theta, \cos\theta)$를 $\mathbf{j}$로 옮기는 matrix입니다). 즉 $\begin{bmatrix} \cos{\theta} & \sin{\theta} \\ -\sin{\theta} & \cos{\theta} \end{bmatrix}$가 됩니다.

> $\begin{bmatrix} \cos{\theta} & \sin{\theta} \\ -\sin{\theta} & \cos{\theta} \end{bmatrix}$는 반시계방향으로 돌려진 matrix를 원래대로 되돌리는 역할을 하기 때문에, 당연히 $R^{-1}$이 됩니다.

```python
theta_list = [np.pi/8, np.pi/4, np.pi/2, np.pi]

for theta in theta_list:
  R = np.array([[numpy.cos(theta), -numpy.sin(theta)],  # 반시계방향으로 45도 회전하는 matrix
                  [numpy.sin(theta), numpy.cos(theta)]])
  print(R)
  R_inv = np.linalg.inv(R)
  print(R_inv)
  print()
```

    [[ 0.924 -0.383]
     [ 0.383  0.924]]
    [[ 0.924  0.383]
     [-0.383  0.924]]

    [[ 0.707 -0.707]
     [ 0.707  0.707]]
    [[ 0.707  0.707]
     [-0.707  0.707]]

    [[ 0. -1.]
     [ 1.  0.]]
    [[ 0.  1.]
     [-1.  0.]]

    [[-1. -0.]
     [ 0. -1.]]
    [[-1.  0.]
     [-0. -1.]]

> 동시에, $\begin{bmatrix} \cos{\theta} & \sin{\theta} \\ -\sin{\theta} & \cos{\theta} \end{bmatrix}$은 $R^{T}$와 같습니다. 왜냐하면 rotation은 orthogonal matrix이기 때문입니다. Orthogonal matrix란 (i) 모든 column의 norm이 1이고 (ii) 모든 column vector가 서로 orthogonal한 matrix를 말합니다. 이에 관한 자세한 설명은 wiki를 참조하시기 바랍니다 (https://en.wikipedia.org/wiki/Orthogonal_matrix). 결론적으로 다음이 성립하게 됩니다.

$$ \text{시계방향으로 } \theta\text{만큼 회전하는 matrix}  = R^{-1} = R^T$$

```python
for theta in theta_list:
  R = np.array([[numpy.cos(theta), -numpy.sin(theta)],  # 반시계방향으로 45도 회전하는 matrix
                  [numpy.sin(theta), numpy.cos(theta)]])
  R_inv = np.linalg.inv(R)

  print(R.T)
  print(R_inv)
  print()
```

    [[ 0.924  0.383]
     [-0.383  0.924]]
    [[ 0.924  0.383]
     [-0.383  0.924]]

    [[ 0.707  0.707]
     [-0.707  0.707]]
    [[ 0.707  0.707]
     [-0.707  0.707]]

    [[ 0.  1.]
     [-1.  0.]]
    [[ 0.  1.]
     [-1.  0.]]

    [[-1.  0.]
     [-0. -1.]]
    [[-1.  0.]
     [-0. -1.]]

> 다시 원래 내용으로 돌아가서, rotation과 scaling을 조합해 $A$를 만들어 봅시다.

1. 시계방향으로 45도 회전한다. ($R$)
2. scaling 한다. ($S$)
3. 반시계방향으로 45도 회전해서 원래 방향으로 되돌려준다.. ($R^T$)

```python
theta = np.pi/4
R = np.array([[np.cos(theta), -np.sin(theta)],  # 반시계방향으로 theta만큼 회전하는 matrix
              [np.sin(theta), np.cos(theta)]])
plot_linear_transformation(R @ S @ R.T)
```

> 좀 더 자세한 이해를 위해, $R, S, R^T$를 하나씩 visualize 해보겠습니다. $A$의 eigenvector $v_1, v_2$를 세 행렬로 변환시켜봅시다.

```python
v1, v2
```

    (array([0.707, 0.707]), array([-0.707,  0.707]))

```python
plot_linear_transformation(R.T, v1, v2)
```

> $y=x, y=-x$ 위에 있는 $v_1$(파랑), $v_2$(분홍)가 시계방향으로 45도 돌아가서 $\begin{bmatrix}1\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 된 것을 확인할 수 있습니다. 이제 scaling을 적용해 봅시다.

```python
rotated_v1 = R.T @ v1
rotated_v2 = R.T @ v2

scaled_rotated_v1 = S @ rotated_v1
scaled_rotated_v2 = S @ rotated_v2

plot_linear_transformation(S, rotated_v1, rotated_v2)
```

> $\begin{bmatrix}1\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 x축 방향으로 3배 길어져서, $\begin{bmatrix}3\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 된 것을 확인할 수 있습니다. 이제 다시 반시계방향으로 45도를 돌려봅시다.

```python
rotated_scaled_rotated_v1 = R@scaled_rotated_v1
rotated_scaled_rotated_v2 = R@scaled_rotated_v2

plot_linear_transformation(R, scaled_rotated_v1, scaled_rotated_v2)
print(rotated_scaled_rotated_v1, rotated_scaled_rotated_v2)
```

> $\begin{bmatrix}3\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 반시계 방향으로 돌아가서, $\begin{bmatrix}2.121\\2.121 \end{bmatrix}, \begin{bmatrix}-0.707\\0.707\end{bmatrix}$이 된 것을 확인할 수 있습니다. 즉 $v_1 \rightarrow 3v_1$, $v_2 \rightarrow v_2$이 됩니다. 결론적으로

$$ v_1 \xrightarrow{rotate(-\theta)} \xrightarrow{scaling} \xrightarrow{rotate(\theta)} 3v_1 \\
v_2 \xrightarrow{rotate(-\theta)} \xrightarrow{scaling} \xrightarrow{rotate(\theta)} v_2 \\
A = RSR^T$$

> 이 됩니다. $v_1, v_2$가 rotate($-\theta$)로  $\begin{bmatrix}1\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 된 것에서 짐작할 수 있듯, rotate(-$\theta$)은 eigenvector $v_1, v_2$를 one-hot vector로 만들게 됩니다 ($v_1$은 첫 번째, $v_2$는 두 번째 component만 non-zero). 이는 예전에 배웠던 change of basis와 연관되는데, 이를 이용해 eigendecomposition을 정리해보도록 하겠습니다.

# 3. Eigendecomposition 정리

## Symmetric Matrix

> 잠시 eigendecomposition을 정리하기 전에, symmetric matrix에 대해 알아봅시다. 지금까지 우리는 symmetric matrix인 $A$를 다뤘습니다. Symmetric matrix란 다음의 성질을 만족하는 matrix를 말합니다.

$$ A = A^T  $$

> 내일 강의에서 배우겠지만, symmetric matrix는 여러가지 유용한 성질을 갖고 있습니다. 그 중 하나는, $n \times n$ symmetric matrix의 경우 항상 n개의 orthogonal한 eigenvector를 가질 수 있다는 것입니다.

```python
F = np.random.rand(5,5)
F_symmetric = F + F.T  # A + A^T는 항상 symmetric입니다.
F_symmetric
```

    array([[1.021, 0.861, 0.566, 1.146, 1.106],
           [0.861, 0.333, 0.679, 1.051, 1.429],
           [0.566, 0.679, 1.753, 0.893, 0.938],
           [1.146, 1.051, 0.893, 0.131, 0.739],
           [1.106, 1.429, 0.938, 0.739, 1.559]])

```python
eigenvalues, eigenvectors = np.linalg.eig(F_symmetric)
eigenvectors
```

    array([[ 0.435,  0.349,  0.621, -0.251,  0.491],
           [ 0.414,  0.19 , -0.134, -0.604, -0.64 ],
           [ 0.45 , -0.874,  0.017, -0.118,  0.14 ],
           [ 0.372,  0.028,  0.385,  0.692, -0.484],
           [ 0.546,  0.28 , -0.669,  0.283,  0.31 ]])

```python
from itertools import combinations
for i,j in combinations(range(5), 2):
  print(eigenvectors[:,i] @ eigenvectors[:,j])
```

    1.3877787807814457e-17
    -6.938893903907228e-17
    1.1882855810441129e-16
    3.885780586188048e-16
    -1.5265566588595902e-16
    2.914335439641036e-16
    3.0531133177191805e-16
    -6.661338147750939e-16
    0.0
    -5.551115123125783e-17

> 모든 eigenvector의 dot product가 0이 되는 것으로부터, 우리는 symmetric matrix로부터 n개의 orthogonal eigenvector를 얻을 수 있음을 알 수 있습니다.

## Eigendecomposition

> 이번 실습에서 예시로 들었던 $A$는 symmetric matrix입니다. 하지만 (1) matrix $A$를 거쳐서 나온 타원의 장축, 단축이 $A$의 eigenvector가 되는 것, 그리고 (2) 모든 행렬이 rotation과 scaling의 조합으로 쪼갤 수 있는것, 두 성질은 모든 임의의 matrix에 대해 성립하는 것은 아닙니다. 지금부터는 오늘 강의에서 배웠던 것처럼, 보다 일반적으로 eigendecomposition을 유도하도록 하겠습니다.

$$
\begin{align*}
  A \mathbf{v_1} = s_1 \mathbf{v_1} \\
  A \mathbf{v_2} = s_2 \mathbf{v_2}
\end{align*}
$$

> 위 두 식은 matrix의 곱 형태로 나타낼 수 있습니다.

$$ A \begin{bmatrix}
    \mid & \mid \\
    \mathbf{v_1} & \mathbf{v_2}\\
    \mid & \mid
    \end{bmatrix}
    = \begin{bmatrix}
    A\mathbf{v_1} & A\mathbf{v_2}
    \end{bmatrix}
$$

$$ A \begin{bmatrix}
    \mid & \mid \\
    \mathbf{v_1} & \mathbf{v_2} \\
    \mid & \mid \\
    \end{bmatrix}
    = \begin{bmatrix}
    s_1\mathbf{v_1} & s_2\mathbf{v_2}
    \end{bmatrix}
$$

> $\begin{bmatrix} s_1\mathbf{v_1} & s_2\mathbf{v_2} \end{bmatrix}$는 다음과 같이 표현이 가능합니다.

$$ s_1 \mathbf{v_1} = \begin{bmatrix}  \mid & \mid \\  \mathbf{v_1} & \mathbf{v_2} \\ \mid & \mid  \end{bmatrix} \begin{bmatrix} s_1 \\ 0 \end{bmatrix} $$

$$ s_2 \mathbf{v_2} = \begin{bmatrix}  \mid & \mid \\  \mathbf{v_1} & \mathbf{v_2} \\  \mid & \mid \end{bmatrix} \begin{bmatrix} 0 \\ s_2 \end{bmatrix} $$

$$  \begin{bmatrix}s_1\mathbf{v_1} & s_2\mathbf{v_2} \end{bmatrix} =
\begin{bmatrix}  \mid & \mid \\  \mathbf{v_1} & \mathbf{v_2} \\  \mid & \mid \end{bmatrix} \begin{bmatrix} s_1 & 0 \\ 0 & s_2 \end{bmatrix} $$

> $V$를 eigenvector를 column으로 갖는 matrix $\begin{bmatrix} \mid & \mid \\ \mathbf{v_1} & \mathbf{v_2} \\\mid & \mid \\\end{bmatrix}$라고 하고, $\begin{bmatrix} s_1 & 0 \\ 0 & s_2 \end{bmatrix}$를 D라고 합시다. 그렇다면 결론적으로

$$ AV = VD $$

> 를 유도할 수 있습니다. $V$의 역행렬이 존재한다면,

$$ A = VDV^{-1} \\ D = V^{-1}AV $$

> 이 됩니다. 위와 같은 형태로 A가 decompose 될 수 있으면, A를 diagonalizable이라고 합니다. 사실 우리는 이를 이전 실습에서 본 적이 있습니다. [HW5] Practice_3의 change of basis를 공부할 때, **같은 linear transformation을 새로운 basis에서 계산하려고 할 때**, 정확히 위와 같은 형태로 유도되었습니다. 이 부분이 확실하지 않으시다면, 오늘 강의와 해당 실습을 복습하시기 바랍니다.

> $V$를 change of basis로 본다면, $VDV^{-1}$는 어떤 의미가 될까요? 저번 실습에서 배웠던 것을 떠올려 본다면, (1) $V^{-1}$: standard basis ($\{ \mathbf{i}, \mathbf{j}\}$) 에서 eigenbasis ($\{ \mathbf{v_1}, \mathbf{v_2}\}$)로 basis를 change하고, (2) $D=\begin{bmatrix}s_1&0\\0&s_2\end{bmatrix}:$ $\mathbf{v_1}$ 방향의 성분 (eigenbasis에서의 좌표의 첫 번째 성분)에 $s_1$을 곱해주고, $\mathbf{v_2}$ 방향의 성분 (eigenbasis에서의 좌표의 두 번째 성분)에 $s_2$을 곱해준 다음, (i.e., elementwise scaling) (3) 다시 standard basis로 되돌리는 행위가 됩니다.

$$
  A\, \mathbf{x} = V\, D\, V^{-1}\mathbf{x}
$$

> 위와 같은 형태로 나타낼 수 있는 A, D를 **similar** 하다고 합니다. 즉 어떤 두 matrix가 같은 linear transformation을 나타내는데 basis만 다른 경우에, 두 matrix는 similar한 관계에 있다고 합니다.

> 이제 non-symmetric matrix인  $B = \begin{bmatrix} 1 & 0 \\ 1 & 3 \end{bmatrix}$의 eigendecomposition을 visualize 해봅시다.

```python
B = np.array([[1,0], [1,3]])
plot_eigen(B)
```

```python
eigenvalues, eigenvectors = np.linalg.eig(B)
eigenvalues
```

    array([3., 1.])

```python
eigenvectors
```

    array([[ 0.   ,  0.894],
           [ 1.   , -0.447]])

> 갈색, 파랑색 vector는 basis를 나타내고, 왼쪽 위의 빨간색 원은 길이가 1인 vector들의 set입니다 ($x$는 빨간색 원 위의 벡터 하나하나를 나타냅니다).

> 위 설명의 $V$를 $C$라고 하고, $v_1, v_2$는 $\mathbf{a,b}$라고 합시다 ($V=C$, $v_1=\mathbf{a}$, $v_2=\mathbf{b}$).  $C^{-1}x$는 standard basis에서 eigenbasis로 바꾸는 행위입니다. 따라서 basis $\mathbf{i,j}$는 $\mathbf{a,b}$로 바뀌었고, 빨간색 원은 바뀌지 않았습니다. 왜냐하면, change of basis는 vector를 고정시킨 채 축만 바꾸는 것이기 때문입니다.

> $DC^{-1}x$는 $C^{-1}$에 $D$로 scaling한 결과입니다. 축은 바뀌지 않았기 때문에 $\mathbf{a,b}$는 유지되고, 빨간색 원 vector의 $\mathbf{a}$ 성분은 3배, $\mathbf{b}$ 성분은 1배 (eigenvalue) 가 됩니다. 강의에서 배웠듯, matrix 계산에서 scalar 계산으로 바뀌기 때문에 (4번 계산할 것을 2번만 계산해도 됨), computational advantage가 생기게 됩니다.

> $CDC^{-1}x$는 $DC^{-1}x$의 basis를 다시 standard basis로 되돌리는 행위가 됩니다. 마찬가지로 change of basis는 vector를 고정시킨 채 축만 바꾸기 때문에 빨간색 원을 바뀌지 않고, basis만 $\mathbf{a,b}$는 $\mathbf{i,j}$로 바뀌게 됩니다.

> Eigendecomposition은 computational advantage를 비롯해, 다양한 application에 사용됩니다.

# Reference

1. https://github.com/jclosure/EngComp4_landlinear


 ILI285 - Computación Científica I  / INF285 - Computación Científica

 Interpolation: Splines

  [S]cientific [C]omputing [T]eam

 Version: 1.22

## Table of Contents
* [Introduction](#intro)
* [Splines](#sp)
* [Properties](#pr)
* [Solving](#so)
* [The additional Property](#ad)
* [Exercises](#ex)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import scipy as sp
from scipy import interpolate
import ipywidgets as widgets
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
from scipy.interpolate import CubicSpline
M=8
```

## Introduction

Previously in our jupyter notebooks, we learn about interpolation. Methods like Newton's Divided Difference, Lagrange, among others. Other alternative for interpolate a set of data points is using ** Cubic Splines**.
This technique, avoids the Runge's Phenomenon and creates a 3-degree polynomial easily.

## Splines

The most common spline is the linear spline. Given a set of points $(x_{1},y_{1}), (x_{2},y_{2}),...,(x_{n},y_{n}) $, this spline connects each point creating a non-smooth curve. However, this polynomial haves a problem. It's no smooth curve! For to avoid this problem, **the cubic splines creates a set of 3-degree polynomial (specifically n-1 polynomials)**... a much better curve.

```python
# Code based on Example from: https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.CubicSpline.html#scipy.interpolate.CubicSpline

# The data
x = np.linspace(0,2*np.pi,12)
y = np.sin(x)*x

# Building interpolation object
cs = CubicSpline(x, y)

# Defining a finer mesh to plot the function
xx = np.linspace(0,2*np.pi,1000)
yyo = np.sin(xx)*xx
yyo1 = np.cos(xx)*xx+np.sin(xx)
yyo2 = -np.sin(xx)*xx+2*np.cos(xx)
yyo3 = -np.cos(xx)*xx-3*np.sin(xx)
yyo4 = np.sin(xx)*xx-4*np.cos(xx)

#Interpolating the date with the spline
yy = cs(xx)
yy1 = cs(xx, 1)
yy2 = cs(xx, 2)
yy3 = cs(xx, 3)
yy4 = cs(xx, 4)

# Plotting the splines and its derivatives
plt.figure(figsize=(M,M))
plt.plot(x,y,'k.',markersize=20,label=r'Data Points')
plt.plot(xx,yy, linewidth=4, label=r'S$(x)$')
plt.plot(xx,yy1, linewidth=4, label=r'$\frac{d}{dx}$S$(x)$')
plt.plot(xx,yy2, linewidth=4, label=r'$\frac{d^2}{dx^2}$S$(x)$')
plt.plot(xx,yy3, linewidth=4, label=r'$\frac{d^3}{dx^3}$S$(x)$')
plt.plot(xx,yy4, linewidth=4, label=r'$\frac{d^4}{dx^4}$S$(x)$')

plt.plot(xx,yyo4,'k--',linewidth=4, label='test', alpha=0.4)

plt.plot(x,y,'k.',markersize=20)

plt.title(r'Cubic Spline is defined as S$(x)$')
plt.axis('tight')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.grid(True)
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.show()

```

```python
plt.figure()
plt.semilogy(np.abs(yy-yyo))
plt.grid(True)
```

The orange curve is generated with cubic splines (using the scipy implementation). The other colors are the **derivatives** of the Cubic Spline as indicated in the legend.
However, if we thinks about this curve, we can say that exists an **infinitely many** quantity of polynomials such that meets all the points. Our goal is to create an unique polynomial. Given this condition, there is 4 properties that defines the cubic spline we are looking for.

## Properties of Splines

When we want creates a spline of **n** data points, we obtains a set of **n-1** 3-degree polynomials. For example:
Given a set of points $(x_{1},y_{1}), (x_{2},y_{2}),...,(x_{n},y_{n})$, the splines is:

\begin{equation} S_{1}(x) = y_{1} + b_{1}(x-x_{1}) + c_{1}(x-x_{1})^{2} + d_{1}(x-x_{1})^{3} \\
                 S_{2}(x) = y_{2} + b_{2}(x-x_{2}) + c_{2}(x-x_{2})^{2} + d_{2}(x-x_{2})^{3} \\
                 ... \\
                 ... \\
                 ... \\
                 S_{n-1}(x) = y_{n-1} + b_{n-1}(x-x_{n-1}) + c_{n-1}(x-x_{n-1})^{2} + d_{n-1}(x-x_{n-1})^{3}
\end{equation}

Thus, our goal is obtains the $y, b, c$ and $d$ coefficients. With this values, we are creating the spline $S(x)$ that meets all the data points. This spline have the next properties:

## Property 1 (Are the points connected?)

The first property checks if each x-coordinate reachs the correspondant y-coordinate in the spline $S(x)$. Thus, checks if the spline meets the data points.

$$S_{i}(x_{i}) = y_{i}$$ $$ S_{i}(x_{i+1}) = y_{i+1}$$

$$i \in [1,n-1]$$

## Property 2 (Slope Continuity)

The second property ensures that the slopes of the splines at the points' neighborhood to be equals. This guarantees the smoothly of $S(x)$

$$S'_{i-1}(x_{i}) = S'_{i}(x_{i})$$

$$i \in [2,n-1]$$

## Property 3 (Curvature Continuity)

This property ensure that the curvature between differents polynomials will be equal at the neighborhood of points, avoiding abrupt changes of the curve at the data points.

$$S''_{i-1}(x_{i}) = S''_{i}(x_{i})$$

$$i \in [2,n-1]$$

## Solving the system

If we have **n points**, we know that our splines will be composed of **n-1 curves** $S_{i}(x)$. We have too, **(3n-3) unknowns variables** ($b_{i}, c_{i}, d_{i}$ for each spline). However, we can build a system of equations for find this variables. How can i do this? Easy.. Using the previous properties!

Using the previously defined splines for n points:
\begin{equation} S_{1}(x) = y_{1} + b_{1}(x-x_{1}) + c_{1}(x-x_{1})^{2} + d_{1}(x-x_{1})^{3} \\
                 S_{2}(x) = y_{2} + b_{2}(x-x_{2}) + c_{2}(x-x_{2})^{2} + d_{2}(x-x_{2})^{3} \\
                 \vdots \\
                 S_{n-1}(x) = y_{n-1} + b_{n-1}(x-x_{n-1}) + c_{n-1}(x-x_{n-1})^{2} + d_{n-1}(x-x_{n-1})^{3}
\end{equation}

We need too, the first derivatives of this curves:
\begin{equation} S'_{1}(x) = b_{1} + 2c_{1}(x-x_{1}) + 3d_{1}(x-x_{1})^{2} \\
                 S'_{1}(x) = b_{2} + 2c_{2}(x-x_{2}) + 3d_{2}(x-x_{2})^{2} \\
                 \vdots \\
                 S'_{n-1}(x) = b_{n-1} + 2c_{n-1}(x-x_{n-1}) + 3d_{n-1}(x-x_{n-1})^{2} \\
\end{equation}

And its second derivatives:
\begin{equation} S''_{1}(x) = 2c_{1} + 6d_{1}(x-x_{1}) \\
                 S''_{2}(x) = 2c_{2} + 6d_{2}(x-x_{2}) \\
                 \vdots \\
                 S''_{n-1}(x) = 2c_{n-1} + 6d_{n-1}(x-x_{n-1}) \\
\end{equation}

Using the first property, we get **(n-1) equations**:

\begin{equation} b_{1}(x_{2}-x_{1}) + c_{1}(x_{2}-x_{1})^2 + d_{1}(x_{2}-x_{1})^3 = y_{2} - y_{1} \hspace{1cm}(1)\\
                 b_{2}(x_{3}-x_{2}) + c_{2}(x_{3}-x_{2})^2 + d_{2}(x_{3}-x_{2})^3 = y_{3} - y_{2} \hspace{1cm}(2)\\
                 \vdots\\
                 b_{n-1}(x_{n}-x_{n-1}) + c_{n-1}(x_{n}-x_{n-1})^2 + d_{n-1}(x_{n}-x_{n-1})^3 = y_{n} - y_{n-1} \hspace{1cm}(n-1)
\end{equation}

Using the second property, we get **(n-2) equations**:

\begin{equation} b_{1}+2c_{1}(x_{2}-x_{1}) + 3d_{1}(x_{2}-x_{1})^2 - b_{2}= 0 \hspace{1cm}(1)\\
                 b_{2}+2c_{2}(x_{3}-x_{2}) + 3d_{2}(x_{3}-x_{2})^2 - b_{3}=  0 \hspace{1cm}(2)\\
                 \vdots\\
                 b_{n-2}+2c_{n-2}(x_{n-1}-x_{n-2}) + 3d_{n-2}(x_{n-1}-x_{n-2})^2 -b_{n-1}=0  \hspace{1cm}(n-2)\\
\end{equation}

Using the third property, we get **(n-2) equations**:

\begin{equation} 2c_{1}+6d_{1}(x_{2}-x_{1}) - 2c_{2} = 0 \hspace{1cm}(1)\\
                 2c_{2}+6d_{2}(x_{3}-x_{2}) - 2c_{3}=0 \hspace{1cm}(2)\\
                 \vdots\\
                 2c_{n-2}+6d_{n-2}(x_{n-1}-x_{n-2}) - 2c_{n-1} = 0 \hspace{1cm}(n-2)\\
\end{equation}

If we adds all our equations, we obtains **(3n-5) equations**. Clearly, the matrix in that system is not square (we need 2 equations more). For this, **we have another property**, that defines the edges conditions of the splines

# Splines Ending options

For this special property, we have the following 5 properties:

## Natural Spline:
This property create a spline with zero curvature, thus:
\begin{align*}
S''_{1}(x_{1}) &= 2c_{1} = 0\\
S''_{n-1}(x_{n}) &= 2c_{n-1}+6d_{n-1}(x_{n}-x_{n-1}) = 0
\end{align*}

## Adjusted curvature:
This property create a spline which curvature is equal to a parameter previously defined, not necessarily zero.
\begin{align*}
S''_{1}(x_{1}) &= 2c_{1} = \kappa_{1}\\
S''_{n-1}(x_{n}) &= 2c_{n-1}+6d_{n-1}(x_{n}-x_{n-1}) = \kappa_{2}
\end{align*}

## Clamped cubic spline:
This property adjust the slopes at the edges of splines to a value previously defined
\begin{align*}
S'_{1}(x_{1}) & = b_{1} = p_{1} \\
S'_{n-1}(x_{n}) & = b_{n-1}+2c_{n-1}(x_{n}-x_{n-1}) + 3d_{n-1}(x_{n}-x_{n-1})^2  = p_{2}
\end{align*}

## Ended Parabolically
With this property, the edges of the splines are 2-degree polynomials. Hence the coefficients:
\begin{align*}
d_{1} &= 0 \\
d_{n-1} &= 0
\end{align*}

- Why does this property does not work for $n\le 3$?

## Not-a-Knot cubic spline
This condition, checks the continuity at the edges, for the third derivative:
\begin{align*}
S'''_{1}(x_{2}) &= S'''_{2}(x_{2})\\
6d_{1}&=6d_{2}\\
S'''_{n-2}(x_{n-1}) &= S'''_{n-1}(x_{n-1})\\
6d_{n-2}&=6d_{n-1}
\end{align*}

- Why does this property does not work for $n\le 4$?

Each property give us the 2 equations needed. Thanks to this, we have **(3n-3) unknowns and equations**
Finally, for to find the coefficients of the spline, we''ll build the system of equations.

## Example:
- If we have the following 3 points: $(x_1,y_1),(x_1,y_1),(x_3,y_3)$
- We will get 6 unknowns
- So, to build a Natural spline, we need to solve the following linear system of equation:

$$ \begin{bmatrix} (x_2-x_1) & (x_2-x_1)^2 & (x_2-x_1)^3 & 0 & 0 & 0 \\
                  0 & 0 & 0 &(x_3-x_2) & (x_3-x_2)^2 & (x_3-x_2)^3  \\
                  1 & 2(x_2-x_1) & 3(x_2-x_1)^2 & -1 & 0 & 0 \\
                  0 & 2 & 6(x_2-x_1) & 0 & -2 & 0 \\
                  0 & 2 & 0 & 0 & 0 & 0 \\
                  0 & 0 & 0 & 0 & 2 & 6(x_3-x_2) \\
                  \end{bmatrix}
                  \left[ \begin{array}{c} b_1 \\ c_1 \\ d_1 \\ b_2 \\ c_2 \\ d_2 \end{array} \right] =
                  \left[ \begin{array}{c} y_2-y_1 \\ y_3-y_2 \\ 0 \\ 0 \\ 0 \\ 0  \end{array} \right]
$$

Finally, find the coefficients of splines is reduced to solve an equation system, and we already know this from previous notebooks!

Now the code:

```python
def cubic_spline(x, y, end=None, k1=0, k2=0, p1=0, p2=0):
    #x: x-coordinates of points
    #y: y-coordinates of points
    #end: Natural, Adjusted, Clamped, Parabolically, NaK

    n = len(x)
    A = np.zeros((3*n-3, 3*n-3))
    b = np.zeros(3*n-3)

    delta_x=np.diff(x)

    #Building the linear system of equations

    #1st property
    for i in np.arange(n-1):
        b[i]= y[i+1]-y[i]
        A[i,3*i:3*(i+1)] = [delta_x[i],delta_x[i]**2,delta_x[i]**3]
    #2nd property
    for i in np.arange(n-2):
        A[(n-1)+i,3*i:3*(i+1)+1]=[1, 2*delta_x[i], 3*delta_x[i]**2, -1]
    #3rd property
    for i in np.arange(n-2):
        A[(n-1)+(n-2)+i,3*i:3*(i+1)+2] = [0, 2, 6*delta_x[i], 0, -2]

    #Ending conditions (4th property)
    if end =='Natural':
        A[-2,1]= 2
        A[-1,-2] = 2
        A[-1,-1] = 6*delta_x[-1]

    elif end == 'Adjusted':
        A[-2,1]= 2
        A[-1,-2] = 2
        A[-1,-1] = 6*delta_x[-1]
        b[-2:] = [k1,k2]
        print('Adjusted',b[-2:])

    elif end == 'Clamped':
        A[-2,0]=1
        A[-1,-3:] = [1,2*delta_x[-1],3*delta_x[-1]**2]
        b[-2:] = [p1,p2]

    elif end == 'Parabolically':
        A[-2,2]=1
        A[-1,-1]=1

    elif end == 'NaK':
        A[-2,2:6]=[6,0,0,-6]
        A[-1,-4:]=[6,0,0,-6]

    #Solving the system
    sol = np.linalg.solve(A,b)
    S = {'b':sol[::3],
         'c':sol[1::3],
         'd':sol[2::3],
         'x':x,
         'y':y
        }
    return S

# 'der' computes the 'der'-derivative of the Spline,
# but it has not been implemented. Can you do it? Please do it!
def cubic_spline_eval(xx,S,der=0):
    x=S['x']
    y=S['y']
    b=S['b']
    c=S['c']
    d=S['d']
    n=len(x)
    yy=np.zeros_like(xx)
    for i in np.arange(n-1):
        jj = np.where(np.logical_and(x[i]<=xx,xx<=x[i+1]))
        yy[jj]=y[i]+b[i]*(xx[jj]-x[i])+c[i]*(xx[jj]-x[i])**2+d[i]*(xx[jj]-x[i])**3
    return yy
```

## Example 1 - Hand made interpolation

```python
x = np.array([1,2,4,5])
y = np.array([2,1,4,3])
S = cubic_spline(x,y,end='Natural')

x1 = np.linspace(1,2,200)
x2 = np.linspace(2,4,200)
x3 = np.linspace(4,5,200)

S1 = y[0]+S['b'][0]*(x1-x[0])+S['c'][0]*(x1-x[0])**2+S['d'][0]*(x1-x[0])**3
S2 = y[1]+S['b'][1]*(x2-x[1])+S['c'][1]*(x2-x[1])**2+S['d'][1]*(x2-x[1])**3
S3 = y[2]+S['b'][2]*(x3-x[2])+S['c'][2]*(x3-x[2])**2+S['d'][2]*(x3-x[2])**3

plt.figure(figsize=(M,M))
plt.plot(x,y,'k.',markersize=20,label='Data Points')
plt.plot(x1,S1,'b',linewidth=5,label=r'S$1(x)$')
plt.plot(x2,S2,'g',linewidth=5,label=r'S$2(x)$')
plt.plot(x3,S3,'r',linewidth=5,label=r'S$2(x)$')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.grid(True)
plt.show()
```

## Example 2

```python
def show_spline(type_ending='Natural',k1=0, k2=0, p1=0, p2=0):
    x = np.array([1,2,4,5,7,9])
    y = np.array([2,1,4,3,3,4])
    xx=np.linspace(np.min(x),np.max(x),1000)
    S = cubic_spline(x,y,end=type_ending, k1=k1, k2=k2, p1=p1, p2=p2)
    plt.figure(figsize=(M,M))
    plt.plot(xx,cubic_spline_eval(xx,S),'-',linewidth=5,label=r'S$(x)$')
    plt.plot(x,y,'k.',markersize=20,label='Data Points')
    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    plt.grid(True)
    plt.show()
widgets.interact(show_spline, type_ending=['Natural','Adjusted','Clamped','Parabolically','NaK'],
                k1=(-20,20,1),k2=(-20,20,1),p1=(-2,2,0.2),p2=(-2,2,0.2))
```

    interactive(children=(Dropdown(description='type_ending', options=('Natural', 'Adjusted', 'Clamped', 'Paraboli…

    <function __main__.show_spline(type_ending='Natural', k1=0, k2=0, p1=0, p2=0)>

## Exercises
Now, a few questions about splines:
- Create the Clamped Cubic Spline given $(0,0), (1,1), (2,2)$. Consider  $S'_1(x_1) = 0$ y $S'_3(x_3) = 1$. Plot it.
- Can you add another point without recalculating all the coefficients again?
- Implement the 'der' input of the 'cubic_spline_eval' function so we can visualize the derivatives of the spline.
- Add the plots of all the derivatives you consider can be computed into Example 2.

## References

### Scipy Cubic Spline Interpolation
- https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.CubicSpline.html#scipy.interpolate.CubicSpline

# Acknowledgements
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistans: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. April 2016._
* _Updated by Claudio Torres_ (`ctorres@inf.utfsm.cl`). _DI UTFSM. June 2017._
* _Updated by Claudio Torres_ (`ctorres@inf.utfsm.cl`). _DI UTFSM. May 2018._
* _Update July 2020 - v1.22 - C.Torres_ : Fixing formatting issues.

```python

```


# Tutorial 2: "How" models
**Week 1, Day 1: Model Types**

**By Neuromatch Academy**

__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom, Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

___
# Tutorial Objectives

*Estimated timing of tutorial: 45 minutes*

This is tutorial 2 of a 3-part series on different flavors of models used to understand neural data. In this tutorial we will explore models that can potentially explain *how* the spiking data we have observed is produced

To understand the mechanisms that give rise to the neural data we save in Tutorial 1,  we will build simple neuronal models and compare their spiking response to real data. We will:
- Write code to simulate a  simple "leaky integrate-and-fire" neuron model
- Make the model more complicated — but also more realistic — by adding more physiologically-inspired details

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/6dxwe/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

```python
# @title Video 1: "How" models
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1yV41167Di", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="PpnagITsb3E", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Plotting Functions

def histogram(counts, bins, vlines=(), ax=None, ax_args=None, **kwargs):
  """Plot a step histogram given counts over bins."""
  if ax is None:
    _, ax = plt.subplots()

  # duplicate the first element of `counts` to match bin edges
  counts = np.insert(counts, 0, counts[0])

  ax.fill_between(bins, counts, step="pre", alpha=0.4, **kwargs)  # area shading
  ax.plot(bins, counts, drawstyle="steps", **kwargs)  # lines

  for x in vlines:
    ax.axvline(x, color='r', linestyle='dotted')  # vertical line

  if ax_args is None:
    ax_args = {}

  # heuristically set max y to leave a bit of room
  ymin, ymax = ax_args.get('ylim', [None, None])
  if ymax is None:
    ymax = np.max(counts)
    if ax_args.get('yscale', 'linear') == 'log':
      ymax *= 1.5
    else:
      ymax *= 1.1
      if ymin is None:
        ymin = 0

  if ymax == ymin:
    ymax = None

  ax_args['ylim'] = [ymin, ymax]

  ax.set(**ax_args)
  ax.autoscale(enable=False, axis='x', tight=True)

def plot_neuron_stats(v, spike_times):
  fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))

  # membrane voltage trace
  ax1.plot(v[0:100])
  ax1.set(xlabel='Time', ylabel='Voltage')
  # plot spike events
  for x in spike_times:
    if x >= 100:
      break
    ax1.axvline(x, color='red')

  # ISI distribution
  if len(spike_times)>1:
    isi = np.diff(spike_times)
    n_bins = np.arange(isi.min(), isi.max() + 2) - .5
    counts, bins = np.histogram(isi, n_bins)
    vlines = []
    if len(isi) > 0:
      vlines = [np.mean(isi)]
    xmax = max(20, int(bins[-1])+5)
    histogram(counts, bins, vlines=vlines, ax=ax2, ax_args={
      'xlabel': 'Inter-spike interval',
      'ylabel': 'Number of intervals',
      'xlim': [0, xmax]
    })
  else:
    ax2.set(xlabel='Inter-spike interval',
           ylabel='Number of intervals')
  plt.show()
```

---
# Section 1: The Linear Integrate-and-Fire Neuron

*Note that we'll be using some math to describe our leaky integrate-and-fire neurons. If you need to quickly check the notation (i.e. what a certain variable name stands for), you can look at the notation section after the summary!*

How does a neuron spike?

A neuron charges and discharges an electric field across its cell membrane. The state of this electric field can be described by the _membrane potential_. The membrane potential rises due to excitation of the neuron, and when it reaches a threshold a spike occurs. The potential resets, and must rise to a threshold again before the next spike occurs.

One of the simplest models of spiking neuron behavior is the linear integrate-and-fire model neuron. In this model, the neuron increases its membrane potential $V_m$ over time in response to excitatory input currents $I$ scaled by some factor $\alpha$:

\begin{align}
  dV_m = {\alpha}I
\end{align}

Once $V_m$ reaches a threshold value a spike is produced, $V_m$ is reset to a starting value, and the process continues.

Here, we will take the starting and threshold potentials as $0$ and $1$, respectively. So, for example, if $\alpha I=0.1$ is constant---that is, the input current is constant---then $dV_m=0.1$, and at each timestep the membrane potential $V_m$ increases by $0.1$ until after $(1-0)/0.1 = 10$ timesteps it reaches the threshold and resets to $V_m=0$, and so on.

Note that we define the membrane potential $V_m$ as a scalar: a single real (or floating point) number. However, a biological neuron's membrane potential will not be exactly constant at all points on its cell membrane at a given time. We could capture this variation with a more complex model (e.g. with more numbers). Do we need to?

The proposed model is a 1D simplification. There are many details we could add to it, to preserve different parts of the complex structure and dynamics of a real neuron. If we were interested in small or local changes in the membrane potential, our 1D simplification could be a problem. However, we'll assume an idealized "point" neuron model for our current purpose.

#### Spiking Inputs

Given our simplified model for the neuron dynamics, we still need to consider what form the input $I$ will take. How should we specify the firing behavior of the presynaptic neuron(s) providing the inputs to our model neuron?

Unlike in the simple example above, where $\alpha I=0.1$, the input current is generally not constant. Physical inputs tend to vary with time. We can describe this variation with a distribution.

We'll assume the input current $I$ over a timestep is due to equal contributions from a non-negative ($\ge 0$) integer number of input spikes arriving in that timestep. Our model neuron might integrate currents from 3 input spikes in one timestep, and 7 spikes in the next timestep. We should see similar behavior when sampling from our distribution.

Given no other information about the input neurons, we will also assume that the distribution has a mean (i.e. mean rate, or number of spikes received per timestep), and that the spiking events of the input neuron(s) are independent in time. Are these reasonable assumptions in the context of real neurons?

A suitable distribution given these assumptions is the Poisson distribution, which we'll use to model $I$:

\begin{align}
  I \sim \mathrm{Poisson}(\lambda)
\end{align}

where $\lambda$ is the mean of the distribution: the average rate of spikes received per timestep.

## Coding Exercise 1: Compute $dV_m$

For your first exercise, you will write the code to compute the change in voltage $dV_m$ (per timestep) of the linear integrate-and-fire model neuron. The rest of the code to handle numerical integration is provided for you, so you just need to fill in a definition for `dv` in the `lif_neuron` function below. The value of $\lambda$ for the Poisson random variable is given by the function argument `rate`.

The [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/stats.html) package is a great resource for working with and sampling from various probability distributions. We will use the `scipy.stats.poisson` class and its method `rvs` to produce Poisson-distributed random samples. In this tutorial, we have imported this package with the alias `stats`, so you should refer to it in your code as `stats.poisson`.

```python
def lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  """ Simulate a linear integrate-and-fire neuron.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    rate (int): The mean rate of incoming spikes

  """
  # Precompute Poisson samples for speed
  exc = stats.poisson(rate).rvs(n_steps)

  # Initialize voltage and spike storage
  v = np.zeros(n_steps)
  spike_times = []

  ################################################################################
  # Students: compute dv, then comment out or remove the next line
  #raise NotImplementedError("Excercise: compute the change in membrane potential")
  ################################################################################

  # Loop over steps
  for i in range(1, n_steps):

    # Update v
    dv = alpha * exc[i]
    v[i] = v[i-1] + dv

    # If spike happens, reset voltage and record
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

# Set random seed (for reproducibility)
np.random.seed(12)

# Model LIF neuron
v, spike_times = lif_neuron()

# Visualize
plot_neuron_stats(v, spike_times)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial2_Solution_9e5a4843.py)

*Example output:*

## Interactive Demo 1: Linear-IF neuron
Like last time, you can now explore how various parametes of the LIF model influence the ISI distribution. Specifically, you can vary `alpha`, which is input scaling factor, and `rate`, which is the mean rate of incoming spikes.

1.   What is the spiking pattern of this model?
2.   What effect does raising or lowering alpha have?
3.   What effect does raising or lowering the rate have?
4.   Does the distribution of ISIs ever look like what you observed in the data in Tutorial 1?

```python
# @markdown You don't need to worry about how the code works – but you do need to **run the cell** to enable the sliders.

def _lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  exc = stats.poisson(rate).rvs(n_steps)
  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = alpha * exc[i]
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0
  return v, spike_times

@widgets.interact(
  alpha=widgets.FloatLogSlider(0.01, min=-2, max=-1),
  rate=widgets.IntSlider(10, min=5, max=20)
)
def plot_lif_neuron(alpha=0.01, rate=10):
  v, spike_times = _lif_neuron(2000, alpha, rate)

  plot_neuron_stats(v, spike_times)
```

    interactive(children=(FloatLogSlider(value=0.01, description='alpha', max=-1.0, min=-2.0), IntSlider(value=10,…

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial2_Solution_2af538db.py)

```python
# @title Video 2: Linear-IF models
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1iZ4y1u7en", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="QBD7kulhg4U", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

---
# Section 2: Inhibitory signals

*Estimated timing to here from start of tutorial: 20 min*

Our linear integrate-and-fire neuron from the previous section was indeed able to produce spikes. However, our ISI histogram doesn't look much like empirical ISI histograms seen in Tutorial 1, which had an exponential-like shape. What is our model neuron missing, given that it doesn't behave like a real neuron?

In the previous model we only considered excitatory behavior -- the only way the membrane potential could decrease was upon a spike event. We know, however, that there are other factors that can drive $V_m$ down. First is the natural tendency of the neuron to return to some steady state or resting potential. We can update our previous model as follows:

\begin{align}
  dV_m = -{\beta}V_m + {\alpha}I
\end{align}

where $V_m$ is the current membrane potential and $\beta$ is some leakage factor. This is a basic form of the popular Leaky Integrate-and-Fire model neuron (for a more detailed discussion of the LIF Neuron, see Bonus Section 2 and the Biological Neuron Models day later in this course).

We also know that in addition to excitatory presynaptic neurons, we can have inhibitory presynaptic neurons as well. We can model these inhibitory neurons with another Poisson random variable:

\begin{align}
I = I_{\mathrm{exc}} - I_{\mathrm{inh}} \\
I_{\mathrm{exc}} \sim \mathrm{Poisson}(\lambda_{\mathrm{exc}}) \\
I_{\mathrm{inh}} \sim \mathrm{Poisson}(\lambda_{\mathrm{inh}})
\end{align}

where $\lambda_{\mathrm{exc}}$ and $\lambda_{\mathrm{inh}}$ are the average spike rates (per timestep) of the excitatory and inhibitory presynaptic neurons, respectively.

### Coding Exercise 2: Compute $dV_m$ with inhibitory signals

For your second exercise, you will again write the code to compute the change in voltage $dV_m$, though now of the LIF model neuron described above. Like last time, the rest of the code needed to handle the neuron dynamics are provided for you, so you just need to fill in a definition for `dv` below.

```python
def lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  """ Simulate a simplified leaky integrate-and-fire neuron with both excitatory
  and inhibitory inputs.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    beta (float): The membrane potential leakage factor
    exc_rate (int): The mean rate of the incoming excitatory spikes
    inh_rate (int): The mean rate of the incoming inhibitory spikes
  """

  # precompute Poisson samples for speed
  exc = stats.poisson(exc_rate).rvs(n_steps)
  inh = stats.poisson(inh_rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []

  ###############################################################################
  # Students: compute dv, then comment out or remove the next line
  #raise NotImplementedError("Excercise: compute the change in membrane potential")
  ################################################################################

  for i in range(1, n_steps):

    dv = -beta * v[i-1] + (alpha * (exc[i] - inh[i]))

    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

# Set random seed (for reproducibility)
np.random.seed(12)

# Model LIF neuron
v, spike_times = lif_neuron_inh()

# Visualize
plot_neuron_stats(v, spike_times)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial2_Solution_6bd84e18.py)

*Example output:*

## Interactive Demo 2: LIF + inhibition neuron

As in Interactive Demo 1, you can play with the parameters of the input to our LIF neuron and visualize what happens. Here, in addition to controlling `alpha`, which scales the input, you can also control `beta`, which is a leakage factor on the voltage, `exc_rate`, which is the mean rate of the excitatory presynaptic neurons, and `inh_rate`, which is the mean rate of the inhibitory presynaptic neurons.

1.   What effect does raising the excitatory rate have?
2.   What effect does raising the inhibitory rate have?
3.   What if you raise both the excitatory and inhibitory rate?
4.   Does the distribution of ISIs ever look like what you observed in the data in Tutorial 1?

```python
#@title
#@markdown **Run the cell** to enable the sliders.
def _lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  """ Simulate a simplified leaky integrate-and-fire neuron with both excitatory
  and inhibitory inputs.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    beta (float): The membrane potential leakage factor
    exc_rate (int): The mean rate of the incoming excitatory spikes
    inh_rate (int): The mean rate of the incoming inhibitory spikes
  """
  # precompute Poisson samples for speed
  exc = stats.poisson(exc_rate).rvs(n_steps)
  inh = stats.poisson(inh_rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = -beta * v[i-1] + alpha * (exc[i] - inh[i])
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

@widgets.interact(alpha=widgets.FloatLogSlider(0.5, min=-1, max=1),
                  beta=widgets.FloatLogSlider(0.1, min=-1, max=0),
                  exc_rate=widgets.IntSlider(12, min=10, max=20),
                  inh_rate=widgets.IntSlider(12, min=10, max=20))
def plot_lif_neuron(alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  v, spike_times = _lif_neuron_inh(2000, alpha, beta, exc_rate, inh_rate)
  plot_neuron_stats(v, spike_times)
```

    interactive(children=(FloatLogSlider(value=0.5, description='alpha', max=1.0, min=-1.0), FloatLogSlider(value=…

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial2_Solution_570e738e.py)

```python
# @title Video 3: LIF + inhibition
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1nV41167mS", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="Aq7JrxRkn2w", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

---
# Section 3: Reflecting on how models

*Estimated timing to here from start of tutorial: 35 min*

## Think! 3: Reflecting on how models

Please discuss the following questions for around 10 minutes with your group:

- Have you seen how models before?
- Have you ever done one?
- Why are how models useful?
- When are they possible? Does your field have how models?
- What do we learn from constructing them?

# Summary

*Estimated timing of tutorial: 45 minutes*

In this tutorial we gained some intuition for the mechanisms that produce the observed behavior in our real neural data. First, we built a simple neuron model with excitatory input and saw that it's behavior, measured using the ISI distribution, did not match our real neurons. We then improved our model by adding leakiness and inhibitory input. The behavior of this balanced model was much closer to the real neural data.

---
# Notation

\begin{align}
V_m &\quad \text{membrane potential} \\
dV_m &\quad \text{change in membrane potential}\\
C_m &\quad \text{membrane capacitance}\\
I &\quad \text{input current}\\
R_m &\quad \text{membrane resistance}:\\
V_\mathrm{rest} &\quad \text{resting potential}\\
\alpha &\quad \text{scaling factor for input current}\\
\beta &\quad \text{leakage factor}\\
\lambda &\quad \text{average spike rate}\\
\lambda_\mathrm{exc} &\quad \text{average spike rate for excitatory neurons}\\
\lambda_\mathrm{inh} &\quad \text{average spike rate for inhibitory neurons}\\
\end{align}

---
# Bonus

---
## Bonus Section 1: Why do neurons spike?

A neuron stores energy in an electric field across its cell membrane, by controlling the distribution of charges (ions) on either side of the membrane. This energy is rapidly discharged to generate a spike when the field potential (or membrane potential) crosses a threshold. The membrane potential may be driven toward or away from this threshold, depending on inputs from other neurons: excitatory or inhibitory, respectively. The membrane potential tends to revert to a resting potential, for example due to the leakage of ions across the membrane, so that reaching the spiking threshold depends not only on the amount of input ever received following the last spike, but also the timing of the inputs.

The storage of energy by maintaining a field potential across an insulating membrane can be modeled by a capacitor. The leakage of charge across the membrane can be modeled by a resistor. This is the basis for the leaky integrate-and-fire neuron model.

---
## Bonus Section 2: The LIF Model Neuron

The full equation for the LIF neuron is

\begin{align}
C_{m}\frac{dV_m}{dt} = -(V_m - V_{\mathrm{rest}})/R_{m} + I
\end{align}

where $C_m$ is the membrane capacitance, $R_m$ is the membrane resistance, $V_{\mathrm{rest}}$ is the resting potential, and $I$ is some input current (from other neurons, an electrode, ...).

In our above examples we set many of these parameters to convenient values ($C_m = R_m = dt = 1$, $V_{\mathrm{rest}} = 0$) to focus more on the general behavior of the model. However, these too can be manipulated to achieve different dynamics, or to ensure the dimensions of the problem are preserved between simulation units and experimental units (e.g. with $V_m$ given in millivolts, $R_m$ in megaohms, $t$ in milliseconds).