# **[HW3] Practice_1**
1. Install packages
2. Matrix operations
3. Linear system
4. Inverse matrix
5. Linear combination

선형대수 실습은, exercise 혹은 coding 문제풀이에 집중하기 보다는, 수업시간에 배웠던 개념들을 visualize 하면서 이를 폭넓게 이해하는 데에 초점을 맞추고 있습니다. 실습에서 사용한 예시 외에도, 다양한 matrix들을 visualize 하면서 개념을 익혀보시기 바랍니다.

# 1. Install packages

> 필요한 package를 설치하고 import합니다

```python
# visualization을 위한 helper code입니다.
if 'google.colab' in str(get_ipython()):
    print('Downloading plot_helpers.py to util/ (only neded for colab')
    !mkdir util; wget https://raw.githubusercontent.com/minireference/noBSLAnotebooks/master/util/plot_helpers.py -P util
```

```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import scipy.linalg
import sympy as sy
from util.plot_helpers import plot_vec, plot_vecs, autoscale_arrows

sy.init_printing()
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

## Numpy, matplotlib

> 선형대수 및 앞으로 공부할 과목들 (머신러닝, 딥러닝, 자연어처리)에서 끊임없이 사용하게 될 두 가지 package가 있는데, (1) Numpy, (2) Matplotlib 입니다. 본 실습은 선형대수 실습시간인 만큼, python package에 대해 자세히 설명하지는 않을 것입니다. 따라서 이에 대해 전혀 모르시는 경우, 다음 tutorial의 Numpy와 Matplotlib 부분을 먼저 따라 치면서 익히고 오시는 것을 추천합니다 (http://aikorea.org/cs231n/python-numpy-tutorial/).

> (주의 1) 해당 tutorial은 python 2 기준으로 작성되었기 때문에, google colab에서 오류가 날 수 있습니다 (e.g., print 다음 괄호가 없기 때문에 오류가 남). 그럴 경우엔 다음 링크 (https://cs231n.github.io/python-numpy-tutorial/#numpy)를 보고 수정하시면 됩니다 (같은 파일인데, 영어로 되어있습니다. 영어도 괜찮으시다면, 이 링크를 바로 보시는게 편합니다).

> (주의 2) Matplotlib의 맨 마지막 예시 (imshow 사용)는 건너뛰시면 됩니다.

# 2. Matrix operations

강의에서 배운 matrix의 기본 성질에 대해 복습해봅시다.

> Matrix addition $\cdot$ matrix multiplication은 Matrix $A,B,C$, 그리고 constant $c$에 대해 다음 식들을 만족합니다.

> Addition
1.   $A+ B= B+ A$
2.   $(A+B)+ C=A+(B+C)$
3.   $c(A+B)=cA+cB$
4.   $(c+d)A=cA+c{D}$
5.   $c(dA)=(cd)A$
6.   $A+{0}=A$, where ${0}$ is the zero matrix
7.   For any $A$, there exists an $- A$, such that $ A+(- A)=0$.

> Multiplication
1.   $ A({BC})=({AB}) C$
2.   $c({AB})=(cA)B=A(cB)$
3.   $A(B+ C)={AB}+{AC}$
4.   $(B+C)A={BA}+{CA}$

> Multiplication의 경우 (1) elementwise multiplication과 (2) matrix multiplication이 있는데, 양쪽 다 많이 쓰이는 만큼 잘 구분해야 합니다.

```python
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
```

```python
A*B # elementwise product
```

    array([[ 5, 12],
           [21, 32]])

```python
A@B # matrix product
```

    array([[19, 22],
           [43, 50]])

> Matrix multiplication은 교환법칙이 성립하지 않습니다 ($AB \neq BA$). 아래 sympy는 $x_1$, $x_2$ 등의 기호(symbol)을 사용하게 해주는 라이브러리입니다.

```python
import sympy as sy

A = sy.Matrix([[3, 4], [7, 8]])
B = sy.Matrix([[5, 3], [2, 1]])

A@B
```

$\displaystyle \left[\begin{matrix}23 & 13\\51 & 29\end{matrix}\right]$

```python
B@A
```

$\displaystyle \left[\begin{matrix}36 & 44\\13 & 16\end{matrix}\right]$

> 하지만 vector multiplication은 교환법칙이 성립합니다 ($x^Ty = y^Tx$).

```python
# 계산에 사용할 기호(symbol)들을 아래와 같이 선언합니다.
x1, x2, y1, y2 = sy.symbols('x1, x2, y1, y2', real = True)

# 위에서 선언한 기호를 활용하여, np.array와 유사한 방식으로 벡터를 선업합니다.
x = sy.Matrix([x1, x2])
y = sy.Matrix([y1, y2])

x.T@y
```

$\displaystyle \left[\begin{matrix}x_{1} y_{1} + x_{2} y_{2}\end{matrix}\right]$

```python
y.T@x
```

$\displaystyle \left[\begin{matrix}x_{1} y_{1} + x_{2} y_{2}\end{matrix}\right]$

> Matrix transpose는 Matrix $A,B$, 그리고 constant $c$에 대해 다음 식들을 만족합니다.

1. $(A^T)^T$
2. $(A+B)^T=A^T+B^T$
3. $(cA)^T=cA^T$
4. $(AB)^T=B^TA^T$

4번이 성립하는지 실제로 알아봅시다.

```python
a, b, c, d, e, f, g, h, i, j, k, l = sy.symbols('a, b, c, d, e, f, g, h, i, j, k, l', real = True)

A = sy.Matrix([[a, b], [c, d], [e, f]])
B = sy.Matrix([[g, h, i], [j, k, l]])
AB = A@B
AB_tr = AB.transpose()
AB_tr  # 4번의 좌항
```

$\displaystyle \left[\begin{matrix}a g + b j & c g + d j & e g + f j\\a h + b k & c h + d k & e h + f k\\a i + b l & c i + d l & e i + f l\end{matrix}\right]$

```python
B_tr_A_tr = B.transpose()@A.transpose()
B_tr_A_tr  # 4번의 우항
```

$\displaystyle \left[\begin{matrix}a g + b j & c g + d j & e g + f j\\a h + b k & c h + d k & e h + f k\\a i + b l & c i + d l & e i + f l\end{matrix}\right]$

```python
AB_tr == B_tr_A_tr
```

    True

> 따라서 $(AB)^T$ 는 $B^TA^T$와 같음을 알 수 있습니다.

# 3. Linear system

> Linear system은 n(>=1)개의 linear equation(일차 방정식)들의 collection을 뜻합니다. 동일한 n개의 variable을 가진 m개의 일차 방정식들은, 다음과 같이 표현될 수 있습니다

$$ a_{11}x_{1}+a_{12}x_{2}+a_{13}x_{3}+\cdots +a_{1n}x_{n}=b_{1} \\
a_{21}x_{1}+a_{22}x_{2}+a_{13}x_{3}+\cdots +a_{2n}x_{n}=b_{2} \\
\cdots \\
a_{m1}x_{1}+a_{m2}x_{2}+a_{m3}x_{3}+\cdots +a_{mn}x_{n}=b_{m}
$$

### n=2일 때의 linear system
> 다음과 같은 예시를 생각해 봅시다.

$$ x_1 + x_2 = 6 \\
x_1 - x_2 = -4 $$.

> 두 linear equation을 plot한다면

```python
x1 = np.linspace(-5, 5, 100)
x2_1 = -x1 + 6
x2_2 = x1 + 4

fig, ax = plt.subplots(figsize = (12, 7))
ax.scatter(1, 5, s = 200, zorder=5, color = 'r', alpha = .8)

ax.plot(x1, x2_1, lw =3, label = '$x_1+x_2=6$')
ax.plot(x1, x2_2, lw =3, label = '$x_1-x_2=-4$')
ax.plot([1, 1], [0, 5], ls = '--', color = 'b', alpha = .5)
ax.plot([-5, 1], [5, 5], ls = '--', color = 'b', alpha = .5)
ax.set_xlim([-5, 5])
ax.set_ylim([0, 12])

ax.legend()
s = '$(1,5)$'
ax.text(1, 5.5, s, fontsize = 20)
ax.set_title('Solution of $x_1+x_2=6$, $x_1-x_2=-4$', size = 22)
ax.grid()
```

> 이 나오게 됩니다. 따라서 두 직선이 만나는 교점, 즉 $x_1=1, x_2=5$가 주어진 두 linear equation을 만족하는 해가 됨을 알 수 있습니다. 이는 굳이 plot을 그리지 않고도, 연립방정식을 사용하여 쉽게 구할 수 있습니다.

### n=3일 때의 linear system
> 다음과 같은 예시를 생각해 봅시다.

$$ x_1 + 2x_2 + 3x_3 = 6 \\
2x_1 + 5x_2 + 2x_3 = 4 \\
6x_1 - 3x_2 + x_3 = 2 $$.

> 세 linear equation을 plot한다면

```python
x1 = np.linspace(-10, 10, 20)
x2 = np.linspace(-10, 10, 20)
X1, X2 = np.meshgrid(x1, x2)

fig = plt.figure(figsize = (9, 9))
ax = fig.add_subplot(111, projection = '3d')

X3_1 = (6 - 2*X2 - X1) * (1/3)
ax.plot_surface(X1, X2, X3_1, cmap ='viridis', alpha = 1)

X3_2 = (4 - 5*X2 - 2*X1) * (1/2)
ax.plot_surface(X1, X2, X3_2, cmap ='summer', alpha = 1)

X3_3 = 2  + 3*X2 - 6*X1
ax.plot_surface(X1, X2, X3_3, cmap ='spring', alpha = 1)

ax.set_xlabel('$x_1$-axis')
ax.set_ylabel('$x_2$-axis')
ax.set_zlabel('$x_3$-axis')

plt.show()
```

> 이 나오게 됩니다. 따라서 n=2인 첫 번째 예시와 마찬가지로, 세 평면이 만나는 교점을 찾으면 주어진 세 linear equation을 만족하는 해가 됩니다. n=2일 경우보다 그림을 보고 직관적으로 해를 알기 어렵고, 연립방정식으로 해를 구하는 것도 상대적으로 복잡해졌음을 알 수 있습니다.

> 주어진 예시들은

$$ Ax = b $$
$$ \begin{bmatrix}
1 & 2 & 3\\
2 & 5 & 2\\
6 & -3 & 1
\end{bmatrix} \begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix} = \begin{bmatrix}
6\\
4\\
2
\end{bmatrix} $$

> 의 꼴로 생각할 수 있는데, 위의 두 plot은 이를 "row" (linear equation)를 중심으로 이들의 교점을 구하는 방식으로 접근했습니다. 이 방식과는 다르게, "column"들을 중심으로 생각해봅시다. 각 column을 vector로 생각하면 위 linear system은

$$ x_1\begin{bmatrix}
1\\
2\\
6
\end{bmatrix} + x_2\begin{bmatrix}
2\\
5\\
-3
\end{bmatrix} + x_3\begin{bmatrix}
3\\
2\\
1
\end{bmatrix} =  \begin{bmatrix}
6\\
4\\
2
\end{bmatrix}$$

> 로 분해할 수 있습니다 (Matrix multiplications as column combinations). 즉 이 경우 $x_1, x_2, x_3$은 각 column vector의 계수가 됩니다. 이를 plot한다면

```python
A = sy.Matrix([[1, 2, 3], [2, 5, 2], [6, -3, 1]])
x1, x2, x3 = sy.symbols('x1 x2 x3')
x = sy.Matrix([x1,x2,x3])

print('column vectors:', A.col(0), A.col(1), A.col(2))

fig = plt.figure(figsize = (9,9))
ax = fig.add_subplot(111, projection = '3d')
ax.scatter(6, 4, 2, s = 200, color = 'red')

plot_vecs(A.col(0), A.col(1), A.col(2))
autoscale_arrows()
```

> 위와 같이 나오게 됩니다. 즉, 세 column vector ([1,2,6], [3,2,1],[2,5,-3])를 linear combination 해서 b ([6,4,2]) 를 만드는 계수 $x_1,x_2,x_3$을 찾는 문제로 바뀌게 됩니다.

*   Row picture: 각 row (linear equation; $a_{i1}x_{1}+a_{i2}x_{2}+a_{i3}x_{3}+\cdots +a_{in}x_{n}=b_{i}$) 들의 교점을 찾는다.
*   Column picture: 각 column (vector; $[a_{1i}, a_{2i}, \cdots, a_{mi}]^T$) 들의 linear combination이 b가 되도록 하는 계수 $x_1, x_2, x_3$을 찾는다.

> 오늘 강의에서 $Ax=b$의 해를 찾는 방법을 자세히 배우지는 않았지만, 위 예시의 경우 우연히도 2*[3,2,1] = [6,4,2]이기 때문에, 해가 $[x_1, x_2, x_3] = [0,0,2]$임을 쉽게 알 수 있습니다. sympy를 사용해서 위 $x$가 해가 맞는지 확인해봅시다.

```python
from sympy.solvers.solveset import linsolve  # linsolve 함수로 해를 구할 것입니다

x1, x2, x3 = sy.symbols('x1 x2 x3')
A = sy.Matrix(((1,2,3),(2,5,2),(6,-3,1)))
b = sy.Matrix((6,4,2))

system = A,b  # 이와 같이 system을 정의합니다.
linsolve(system, x1, x2, x3)  # system, 그리고 x의 원소를 차례로 넣습니다.
```

$\displaystyle \left\{\left( 0, \  0, \  2\right)\right\}$

>$x$의 해가 [0,0,2]가 맞음을 sympy로도 확인할 수 있습니다.이처럼, 앞으로 배우게 될 강의에서도 주로 column picture의 관점에서 문제들을 생각하게 될 것입니다.

### 해가 없는 경우의 linear system
> 위의 두 예시는 해가 1개만 존재하는 linear system 이었습니다. 오늘 강의에서 배웠듯, $A$의 inverse가 존재하기 때문에, 해 $x$는 unique하게 한 개만 존재하는 경우입니다. 이젠 다음과 같은 예시를 생각해 봅시다.

$$ x_1 + x_2 + x_3 = 1 \\
x_1 - x_2 - 2x_3 = 2 \\
2x_1 - x_3 = 1 $$.

> 위 system을 행렬로 표현해보면

$$ Ax = b $$
$$ \begin{bmatrix}
1 & 1 & 1\\
1 & -1 & -2\\
2 & 0 & -1
\end{bmatrix} \begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix} = \begin{bmatrix}
1\\
2\\
1
\end{bmatrix} $$

> 이 됩니다. 이를 column picture로 plot해 본다면

```python
A = sy.Matrix([[1, 1, 1], [1, -1, -2], [2, 0, -1]])
x1, x2, x3 = sy.symbols('x1 x2 x3')
x = sy.Matrix([x1,x2,x3])

fig = plt.figure(figsize = (9,9))
ax = fig.add_subplot(111, projection = '3d')
ax.scatter(1, 2, 1, s = 200, color = 'red')

plot_vecs(A.col(0), A.col(1), A.col(2))
autoscale_arrows()
```

> 세 column vector의 linear combination으로 b (빨간색 점) 를 만들 수 있으면 해가 1개 이상 존재하고, 만드는 것이 불가능 하다면 해가 존재하지 않습니다. A의 해를 확인해 봅시다.

```python
x1, x2, x3 = sy.symbols('x1 x2 x3')
A = sy.Matrix(((1,1,1),(1,-1,-2),(2,0,-1)))
b = sy.Matrix((1,2,1))

system = A,b
linsolve(system, x1, x2, x3)
```

> 해가 없음을 알 수 있습니다. 만약 $A$가 invertible 하다면 (즉 A의 determinant가 0이라면), 해 $x$는 반드시 존재하며, 1개로 unique하게 얻어져야 합니다. 따라서, 이 경우엔 해가 없기 때문에, *det A* 가 0이 되는지 확인해봅시다.

```python
A.det()
```

### 해가 무수히 많은 경우의 linear system
> 다음과 같은 예시를 생각해 봅시다.

$$ x_2 - x_3 = 4 \\
2x_1 + x_2 + 2x_3 = 4 \\
2x_1 + 2x_2 + x_3 = 8 $$.

> 위 system을 행렬로 표현해보면

$$ Ax = b $$
$$ \begin{bmatrix}
0 & 1 & -1\\
2 & 1 & 2\\
2 & 2 & 1
\end{bmatrix} \begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix} = \begin{bmatrix}
4\\
4\\
8
\end{bmatrix} $$

> 이 됩니다. 이를 column picture로 plot해 본다면

```python
A = sy.Matrix([[0, 1, -1], [2, 1, 2], [2, 2, 1]])
x1, x2, x3 = sy.symbols('x1 x2 x3')
x = sy.Matrix([x1,x2,x3])

fig = plt.figure(figsize = (9,9))
ax = fig.add_subplot(111, projection = '3d')

plot_vecs(A.col(0), A.col(1), A.col(2))
autoscale_arrows()

ax.scatter(4,4,8, s = 200, color = 'red')
ax.set(xlim = [0.,5.], ylim = [0.,5.,], zlim = [0., 10.])

plt.show()
```

> 이 됩니다. 세 column vector의 linear combination으로 b (빨간색 점) 를 만들 수 있으면 해가 1개 이상 존재하고, 만드는 것이 불가능 하다면 해가 존재하지 않습니다. A의 해를 확인해봅시다.

```python
A = sy.Matrix(((0,1,-1),(2,1,2),(2,2,1)))
b = sy.Matrix((4,4,8))
system = A,b
linsolve(system, x1, x2, x3)
```

> 위 결과가 의미하는 것은, 어떤 $x_3$의 값에 대해서도 위와 같은 형태의 vector는 해가 된다는 것입니다. 즉 ..., (-3/2, 5, 1) ($x_3$이 1일 경우), (-3, 6, 2) ($x_3$이 2일 경우), ..., (-3n/2, n+4, n), ... 모두가 해가 될 수 있다는 것이므로, 해가 무수히 많이 존재하는 경우입니다. 이때도 마찬가지로 해 $x$가 unique하게 결정되지 않는 case이므로, A의 determinant가 0인지 확인해봅시다.

```python
A.det()
```

### Exercise

다음 linear system들의 (i) 해가 무수히 많이 존재하는 경우, (ii) 해가 한 개 존재하는 경우, (iii) 해가 존재하지 않는 경우로 나누고 존재할 경우 해를 구하시오. 또한 각각에 대해 A의 inverse가 존재하는지 설명하시오. (Hint: 위 예시의 linsolve 함수를 사용)

$ Ax = b $ \\
(a) $ \begin{bmatrix}
0 & 1 & 4\\
1 & 3 & 5\\
3 & 7 & 7
\end{bmatrix} \begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix} = \begin{bmatrix}
-5\\
-2\\
6
\end{bmatrix} $ \\
(b) $ \begin{bmatrix}
1 & 0 & -3\\
2 & 2 & 9\\
0 & 1 & 5
\end{bmatrix} \begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix} = \begin{bmatrix}
8\\
7\\
-2
\end{bmatrix} $ \\
(c) $ \begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 9\\
9 & 12 & 21
\end{bmatrix} \begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix} = \begin{bmatrix}
5\\
2\\
9
\end{bmatrix} $

Answer:

```python
# answer: 해가 존재하지 않음
A = sy.Matrix([[0,1,4],[1,3,5],[3,7,7]])
b = sy.Matrix([-5,-2,6])
system = A,b
linsolve(system, x1, x2, x3)
```

```python
# answer: 해가 unique하게 한개만 존재
A = sy.Matrix([[1,0,-3],[2,2,9],[0,1,5]])
b = sy.Matrix([8,7,-2])
system = A,b
linsolve(system, x1, x2, x3)
```

$\displaystyle \left\{\left( 5, \  3, \  -1\right)\right\}$

```python
# answer: 해가 무수히 많이 존재
A = sy.Matrix([[1,2,3],[4,5,9],[9,12,21]])
b = sy.Matrix([5,2,9])
system = A,b
linsolve(system, x1, x2, x3)
```

# 4. Inverse matrix

> Square matrix $A$의 역행렬 $A^{-1}$은 다음과 같이 정의됩니다.

$$ A^{-1}A = AA^{-1} = I_n$$

> 역행렬이 존재한다는 의미는, input vector $x$와 output vector $Ax$ 사이에 일대일대응 관계가 성립한다는 의미입니다.

 출처: Ref. [1]

> 즉, $Ax$로부터 $x$를 복원할 수 있다는 말이고, 이를 복원하는 행렬이 $A^{-1}$이 됩니다.

> 만약 $n \times n$ 행렬 $A$의 역행렬이 존재한다면, $Ax = b$는 $x = A^{-1}b$로 해를 구할 수 있습니다.

```python
A = np.array([[1, 2, 3], [2, 5, 2], [6, -3, 1]])
b = np.array([6,4,2])

A_inv = np.linalg.inv(A)  # np.linalg.inv를 활용하면 A의 inverse를 구할 수 있습니다
A_inv_b = A_inv @ b  # A^{-1}b
x = A_inv_b  # x = A^{-1}b
print(x)

# 검산해봅시다 (구한 x를 대입)
print(A@x)
print(b)
```

    [0. 0. 2.]
    [6. 4. 2.]
    [6 4 2]

> Ax와 b가 같음을 알 수 있습니다.

# 5. Linear combination

두 vector $ \begin{bmatrix} 4 \\ 2 \end{bmatrix}   $, $ \begin{bmatrix} -2 \\ 2 \end{bmatrix} $의 linear combination은 다음과 같이 나타낼 수 있습니다.

$$ c_1\begin{bmatrix} 4 \\ 2 \end{bmatrix} + c_2\begin{bmatrix} -2 \\ 2 \end{bmatrix} $$

이를 plot해 봅시다.

```python
fig, ax = plt.subplots(figsize=(8, 8))

vec = np.array([[[0,0,4,2]],
                 [[0,0,-2,2]],
                 [[0,0,2,10]],
                 [[0,0,8,4]],
                 [[0,0,-6,6]]])
colors = ['b','b','r','b','b']

# tail이 origin, head가 (4,2), (-2,2), (2,10), (8,4), (-6,6)인 vector plot
for i in range(vec.shape[0]):
    X,Y,U,V = zip(*vec[i,:,:])
    ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', color = colors[i], scale=1, alpha = .6)
    ax.text(x = vec[i,0,2], y = vec[i,0,3], s = '(%.0d, %.0d)' %(vec[i,0,2],vec[i,0,3]), fontsize = 16)

# tail이 (8,4), head가 (2,10)인 vector plot
points12 = np.array([[8,4],[2,10]])
ax.plot(points12[:,0], points12[:,1], c = 'b', lw = 3.5,alpha =0.5, ls = '--')

# tail이 (-6,6), head가 (2,10)인 vector plot
points34 = np.array([[-6, 6],[2,10]])
ax.plot(points34[:,0], points34[:,1], c = 'b', lw = 3.5,alpha =0.5, ls = '--')

ax.set_xlim([-10, 10])
ax.set_ylim([0, 10.5])
ax.set_xlabel('x-axis', fontsize =16)
ax.set_ylabel('y-axis', fontsize =16)
ax.grid()

######################################Basis########################################
a = np.arange(-11, 20, 1)
x = np.arange(-11, 20, 1)

# 붉은색 격자 plot
for i in a:
    y1 = i + 0.5*x  # 0.5(기울기) = 2/4
    ax.plot(x, y1, ls = '--', color = 'pink', lw = 2)
    y2 = i - x  # -1(기울기) = 2/(-2)
    ax.plot(x, y2, ls = '--', color = 'pink', lw = 2)

ax.set_title('Linear Combination of Two Vectors in $\mathbf{R}^2$', size = 22, x =0.5, y = 1.01)
plt.show()
```

> 위의 그림은 두 벡터의 linear combination으로 $ \begin{bmatrix} 2 \\ 10 \end{bmatrix}$을 만드는 그립입니다.

$$ \begin{bmatrix} 2 \\ 10 \end{bmatrix} =  2\begin{bmatrix} 4 \\ 2 \end{bmatrix} + 3\begin{bmatrix} -2 \\ 2 \end{bmatrix} $$

> $\begin{bmatrix} 4 \\ 2 \end{bmatrix}$에 3을 곱하고, $\begin{bmatrix} -2 \\ 2 \end{bmatrix}$에 2를 곱한 뒤 더하면 $\begin{bmatrix} 2 \\ 10 \end{bmatrix}$이 나옴을 알 수 있습니다. 붉은 격자는 두 벡터의 linear combination으로 도달할 수 있는 공간을 의미합니다. 그림에서 해당 붉은 격자가 $R^2$ 전체를 커버함을 확인할 수 있습니다.

# Reference

1. https://en.wikipedia.org/wiki/Bijection,_injection_and_surjection
2. http://aikorea.org/cs231n/python-numpy-tutorial/#matplotlib
3. https://github.com/MacroAnalyst/Linear_Algebra_With_Python


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Toggle cell visibility <a href="javascript:code_toggle()">here</a>.''')
display(tag)
```

Toggle cell visibility here.

```python
%matplotlib notebook
import numpy as np
import control as control
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from ipywidgets import widgets
from ipywidgets import interact
import scipy.signal as signal
import sympy as sym

```

## Mechanical systems

#### General mass-spring-damper model
> The mass-spring-damper model consists of discrete mass nodes distributed throughout an object and interconnected via a network of springs and dampers. This model is well-suited for modelling object with complex material properties such as nonlinearity and viscoelasticity. (source: [Wikipedia](https://en.wikipedia.org/wiki/Mass-spring-damper_model "Mass-spring-model"))
#### 1/4 car model
> 1/4 car model is used to analyze the ride quality of automotive suspension systems. Mass $m_1$ is the "sprung mass", which is one-quarter of the vehicle mass that is supported by the suspension system. Mass $m_2$ is the "unsprung mass", which is lumped mass composed of one wheel and half-axle assembly, plus the shock absorber and suspensison springs. The stiffness and damping of the suspension system are modeled by the ideal spring constant $k_1$ and friction coefficient $B$, respecitvely. Tire stifness is modeled by spring constant $k_2$. (source: [Chegg Study](https://www.chegg.com/homework-help/questions-and-answers/figure-p230-shows-1-4-car-model-used-analyze-ride-quality-automotive-suspension-systems-ma-q26244005 "1/4 car model"))

---

### How to use this notebook?
1. Toggle between *mass-spring-damper* and *1/4 car model* system by clicking on a corresponding button.
2. Toggle betweeen *step function*, *impulse function*, *ramp function*, and *sine function* to select the function of the force $F$.
3. Move the sliders to change the values of the mass ($m$; $m_1$ and $m_2$), spring coefficients ($k$; $k_1$ and $k_2$), damping constant ($B$), input signal amplification and initial conditions ($x_0$, $\dot{x}_0$, $y_0$, $\dot{y}_0$).

        Mass-spring-damper
        1/4 car model

```python
# create figure
fig = plt.figure(figsize=(9.8, 4),num='Mechanical systems')

# add sublot
ax = fig.add_subplot(111)
ax.set_title('Time Response')
ax.set_ylabel('input, output')
ax.set_xlabel('$t$ [s]')

ax.grid(which='both', axis='both', color='lightgray')

inputf, = ax.plot([], [])
responsef, = ax.plot([], [])
responsef2, = ax.plot([], [])
arrowf, = ax.plot([],[])

style = {'description_width': 'initial'}

selectSystem=widgets.ToggleButtons(
    options=[('mass-spring-damper',0),('1/4 car model',1)],
    description='Select system: ', style=style) # define toggle buttons
selectForce = widgets.ToggleButtons(
    options=[('step function', 0), ('impulse function', 1), ('ramp function', 2), ('sine function', 3)],
    description='Select $F$ function: ', style=style)
display(selectSystem)
display(selectForce)

def build_model(M,K,B,M1,M2,B1,K1,K2,amp,x0,xpika0,select_System,index):

    num_of_samples = 1000
    total_time = 25
    t = np.linspace(0, total_time, num_of_samples) # time for which response is calculated (start, stop, step)

    global inputf, responsef, responsef2, arrowf

    if select_System==0:

        system0 = control.TransferFunction([1], [M, B, K])

        if index==0:
            inputfunc = np.ones(len(t))*amp
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0*M])
            inputfunc[0]=0

        elif index==1:
            inputfunc=signal.unit_impulse(1000, 0)*amp
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0*M])

        elif index==2:
            inputfunc=t;
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0*M])

        elif index==3:
            inputfunc=np.sin(t)*amp
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0*M])

    elif select_System==1:

        system1 = control.TransferFunction([B1*K2, K1*K2],[M1*M2, M1*B1+M2*B1, M1*K1+M1*K2+M2*K1, B1*K2, K1*K2])
        system2 = control.TransferFunction([M1*K2, B1*K2, K1*K2],[M1*M2, M1*B1+M2*B1, M1*K1+M1*K2+M2*K1, B1*K2, K1*K2])

        if index==0:
            inputfunc = np.ones(len(t))*amp

            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,0,0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0])
            inputfunc[0]=0

        elif index==1:
            inputfunc=signal.unit_impulse(1000, 0)*amp
            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,0,0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0])

        elif index==2:
            inputfunc=t;
            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,0,0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0])

        elif index==3:
            inputfunc=np.sin(t)*amp
            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,0,0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0])

    ax.lines.remove(responsef)
    ax.lines.remove(inputf)
    ax.lines.remove(responsef2)
    ax.lines.remove(arrowf)

    inputf, = ax.plot(t,inputfunc,label='$F$',color='C0')
    responsef, = ax.plot(time, response,label='$x$',color='C3')

    if select_System==1:
        responsef2, = ax.plot(time, response2,label='$y$',color='C2')
    elif select_System==0:
        responsef2, = ax.plot([],[])

    if index==1:
        if amp>0:
            arrowf, = ax.plot([-0.1,0,0.1],[amp-((amp*0.05)/2),amp,amp-((amp*0.05)/2)],color='C0',linewidth=4)
        elif amp==0:
            arrowf, = ax.plot([],[])
        elif amp<0:
            arrowf, = ax.plot([-0.1,0,0.1],[amp-((amp*0.05)/2),amp,amp-(amp*(0.05)/2)],color='C0',linewidth=4)
    else:
        arrowf, = ax.plot([],[])

    ax.relim()
    ax.autoscale_view()

    ax.legend()

def update_sliders(index):
    global m1_slider, b1_slider, k1_slider, m21_slider, m22_slider, b2_slider, k21_slider, k22_slider
    global x0_slider, xpika0_slider

    m1val = [0.1,0.1,0.1,0.1]
    k1val = [1,1,1,1]
    b1val = [0.1,0.1,0.1,0.1]
    m21val = [0.1,0.1,0.1,0.1]
    m22val = [0.1,0.1,0.1,0.1]
    b2val = [0.1,0.1,0.1,0.1]
    k21val = [1,1,1,1]
    k22val = [1,1,1,1]
    x0val = [0,0,0,0]
    xpika0val = [0,0,0,0]
    y0val = [0,0,0,0]
    ypika0val = [0,0,0,0]

    m1_slider.value = m1val[index]
    k1_slider.value = k1val[index]
    b1_slider.value = b1val[index]
    m21_slider.value = m21val[index]
    m22_slider.value = m22val[index]
    b2_slider.value = b2val[index]
    k21_slider.value = k21val[index]
    k22_slider.value = k22val[index]
    x0_slider.value = x0val[index]
    xpika0_slider.value = xpika0val[index]

def draw_controllers(type_select,index):

    global m1_slider, b1_slider, k1_slider, m21_slider, m22_slider, b2_slider, k21_slider, k22_slider
    global x0_slider, xpika0_slider

    if type_select==0:

        amp_slider = widgets.FloatSlider(value=1.,min=-2.,max=2.,step=0.1,
            description='Input signal amplification:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',style=style)

        m1_slider = widgets.FloatSlider(value=.1, min=.01, max=1., step=.01,
                                        description='$m$ [kg]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        k1_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
                                       description='$k$ [N/m]:',disabled=False,continuous_update=False,
                                       orientation='horizontal',readout=True,readout_format='.1f',)
        b1_slider = widgets.FloatSlider(value=.1,min=0.0,max=0.5,step=.01,
                                        description='$B$ [Ns/m]:',disabled=False,continuous_update=False,
                                        rientation='horizontal',readout=True,readout_format='.2f',)
        m21_slider = widgets.FloatSlider(value=.1,min=.01,max=1.,step=.01,
            description='$m_1$ [kg]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        m22_slider = widgets.FloatSlider(value=.1,min=.0,max=1.,step=.01,
            description='$m_2$ [kg]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        b2_slider = widgets.FloatSlider(value=.1,min=0.0,max=2,step=.01,
            description='$B$ [Ns/m]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        k21_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_1$ [N/m]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )
        k22_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_2$ [N/m]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )

        x0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='$x_0$ [dm]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        xpika0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='${\dot{x}}_0$ [dm/s]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)

        #y0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
        #                                description='$y_0$ [dm]:',disabled=True,continuous_update=False,
        #                                orientation='horizontal',readout=True,readout_format='.2f',)
        #ypika0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
        #                                description='${\dot{y}}_0$ [dm/s]:',disabled=True,continuous_update=False,
        #                                orientation='horizontal',readout=True,readout_format='.2f',)
    elif type_select==1:

        amp_slider = widgets.FloatSlider(value=1.,min=-2.,max=2.,step=0.1,
            description='Input signal amplification:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',style=style)

        m1_slider = widgets.FloatSlider(value=.1, min=.01, max=1., step=.01,
                                        description='$m$ [kg]:',disabled=True,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        k1_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
                                       description='$k$ [N/m]:',disabled=True,continuous_update=False,
                                       orientation='horizontal',readout=True,readout_format='.1f',)
        b1_slider = widgets.FloatSlider(value=.1,min=0.0,max=0.5,step=.01,
                                        description='$B$ [Ns/m]:',disabled=True,continuous_update=False,
                                        rientation='horizontal',readout=True,readout_format='.2f',)
        m21_slider = widgets.FloatSlider(value=.1,min=.01,max=1.,step=.01,
            description='$m_1$ [kg]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        m22_slider = widgets.FloatSlider(value=.1,min=.0,max=1.,step=.01,
            description='$m_2$ [kg]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        b2_slider = widgets.FloatSlider(value=.1,min=0.0,max=2,step=.01,
            description='$B$ [Ns/m]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        k21_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_1$ [N/m]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )
        k22_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_2$ [N/m]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )

        x0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='$x_0$ [dm]:',disabled=True,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        xpika0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='${\dot{x}}_0$ [dm/s]:',disabled=True,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)

        #y0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
        #                                description='$y_0$ [dm]:',disabled=True,continuous_update=False,
        #                                orientation='horizontal',readout=True,readout_format='.2f',)
        #ypika0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
        #                                description='${\dot{y}}_0$ [dm/s]:',disabled=True,continuous_update=False,
        #                                orientation='horizontal',readout=True,readout_format='.2f',)
    input_data = widgets.interactive_output(build_model, {'M':m1_slider, 'K':k1_slider, 'B':b1_slider, 'M1':m21_slider,
                                                    'M2':m22_slider, 'B1':b2_slider, 'K1':k21_slider, 'K2':k22_slider, 'amp':amp_slider,
                                                    'x0':x0_slider,'xpika0':xpika0_slider,
                                                    'select_System':selectSystem,'index':selectForce})

    input_data2 = widgets.interactive_output(update_sliders, {'index':selectForce})

    box_layout = widgets.Layout(border='1px solid black',
                    width='auto',
                    height='',
                    flex_flow='row',
                    display='flex')

    buttons1=widgets.HBox([widgets.VBox([amp_slider],layout=widgets.Layout(width='auto')),
                         widgets.VBox([x0_slider,xpika0_slider])],layout=box_layout)
    display(widgets.VBox([widgets.Label('Select the values of the input signal amplification and intial conditions:'), buttons1]))
    display(widgets.HBox([widgets.VBox([m1_slider,k1_slider,b1_slider], layout=widgets.Layout(width='45%')),
                          widgets.VBox([m21_slider,m22_slider,k21_slider,k22_slider,b2_slider], layout=widgets.Layout(width='45%'))]), input_data)

widgets.interactive_output(draw_controllers, {'type_select':selectSystem,'index':selectForce})
```

    <IPython.core.display.Javascript object>

    ToggleButtons(description='Select system: ', options=(('mass-spring-damper', 0), ('1/4 car model', 1)), style=…

    ToggleButtons(description='Select $F$ function: ', options=(('step function', 0), ('impulse function', 1), ('r…

    Output()


```python
%matplotlib inline
```

Sparse Regression
=================

We demonstrate how to do (fully Bayesian) sparse linear regression using the
approach described in [1]. This approach is particularly suitable for situations
with many feature dimensions (large P) but not too many datapoints (small N).
In particular we consider a quadratic regressor of the form:

\begin{align}f(X) = \text{constant} + \sum_i \theta_i X_i + \sum_{i<j} \theta_{ij} X_i X_j + \text{observation noise}\end{align}

**References:**

    1. Raj Agrawal, Jonathan H. Huggins, Brian Trippe, Tamara Broderick (2019),
       "The Kernel Interaction Trick: Fast Bayesian Discovery of Pairwise Interactions in High Dimensions",
       (https://arxiv.org/abs/1905.06501)

```python
import argparse
import itertools
import os
import time

import numpy as onp

import jax
from jax import vmap
import jax.numpy as np
import jax.random as random

import numpyro
import numpyro.distributions as dist
from numpyro.infer import MCMC, NUTS

def dot(X, Z):
    return np.dot(X, Z[..., None])[..., 0]

# The kernel that corresponds to our quadratic regressor.
def kernel(X, Z, eta1, eta2, c, jitter=1.0e-6):
    eta1sq = np.square(eta1)
    eta2sq = np.square(eta2)
    k1 = 0.5 * eta2sq * np.square(1.0 + dot(X, Z))
    k2 = -0.5 * eta2sq * dot(np.square(X), np.square(Z))
    k3 = (eta1sq - eta2sq) * dot(X, Z)
    k4 = np.square(c) - 0.5 * eta2sq
    if X.shape == Z.shape:
        k4 += jitter * np.eye(X.shape[0])
    return k1 + k2 + k3 + k4

# Most of the model code is concerned with constructing the sparsity inducing prior.
def model(X, Y, hypers):
    S, P, N = hypers['expected_sparsity'], X.shape[1], X.shape[0]

    sigma = numpyro.sample("sigma", dist.HalfNormal(hypers['alpha3']))
    phi = sigma * (S / np.sqrt(N)) / (P - S)
    eta1 = numpyro.sample("eta1", dist.HalfCauchy(phi))

    msq = numpyro.sample("msq", dist.InverseGamma(hypers['alpha1'], hypers['beta1']))
    xisq = numpyro.sample("xisq", dist.InverseGamma(hypers['alpha2'], hypers['beta2']))

    eta2 = np.square(eta1) * np.sqrt(xisq) / msq

    lam = numpyro.sample("lambda", dist.HalfCauchy(np.ones(P)))
    kappa = np.sqrt(msq) * lam / np.sqrt(msq + np.square(eta1 * lam))

    # sample observation noise
    var_obs = numpyro.sample("var_obs", dist.InverseGamma(hypers['alpha_obs'], hypers['beta_obs']))

    # compute kernel
    kX = kappa * X
    k = kernel(kX, kX, eta1, eta2, hypers['c']) + var_obs * np.eye(N)
    assert k.shape == (N, N)

    # sample Y according to the standard gaussian process formula
    numpyro.sample("Y", dist.MultivariateNormal(loc=np.zeros(X.shape[0]), covariance_matrix=k),
                   obs=Y)

# Compute the mean and variance of coefficient theta_i (where i = dimension) for a
# MCMC sample of the kernel hyperparameters (eta1, xisq, ...).
# Compare to theorem 5.1 in reference [1].
def compute_singleton_mean_variance(X, Y, dimension, msq, lam, eta1, xisq, c, var_obs):
    P, N = X.shape[1], X.shape[0]

    probe = np.zeros((2, P))
    probe = jax.ops.index_update(probe, jax.ops.index[:, dimension], np.array([1.0, -1.0]))

    eta2 = np.square(eta1) * np.sqrt(xisq) / msq
    kappa = np.sqrt(msq) * lam / np.sqrt(msq + np.square(eta1 * lam))

    kX = kappa * X
    kprobe = kappa * probe

    k_xx = kernel(kX, kX, eta1, eta2, c) + var_obs * np.eye(N)
    k_xx_inv = np.linalg.inv(k_xx)
    k_probeX = kernel(kprobe, kX, eta1, eta2, c)
    k_prbprb = kernel(kprobe, kprobe, eta1, eta2, c)

    vec = np.array([0.50, -0.50])
    mu = np.matmul(k_probeX, np.matmul(k_xx_inv, Y))
    mu = np.dot(mu, vec)

    var = k_prbprb - np.matmul(k_probeX, np.matmul(k_xx_inv, np.transpose(k_probeX)))
    var = np.matmul(var, vec)
    var = np.dot(var, vec)

    return mu, var

# Compute the mean and variance of coefficient theta_ij for a MCMC sample of the
# kernel hyperparameters (eta1, xisq, ...). Compare to theorem 5.1 in reference [1].
def compute_pairwise_mean_variance(X, Y, dim1, dim2, msq, lam, eta1, xisq, c, var_obs):
    P, N = X.shape[1], X.shape[0]

    probe = np.zeros((4, P))
    probe = jax.ops.index_update(probe, jax.ops.index[:, dim1], np.array([1.0, 1.0, -1.0, -1.0]))
    probe = jax.ops.index_update(probe, jax.ops.index[:, dim2], np.array([1.0, -1.0, 1.0, -1.0]))

    eta2 = np.square(eta1) * np.sqrt(xisq) / msq
    kappa = np.sqrt(msq) * lam / np.sqrt(msq + np.square(eta1 * lam))

    kX = kappa * X
    kprobe = kappa * probe

    k_xx = kernel(kX, kX, eta1, eta2, c) + var_obs * np.eye(N)
    k_xx_inv = np.linalg.inv(k_xx)
    k_probeX = kernel(kprobe, kX, eta1, eta2, c)
    k_prbprb = kernel(kprobe, kprobe, eta1, eta2, c)

    vec = np.array([0.25, -0.25, -0.25, 0.25])
    mu = np.matmul(k_probeX, np.matmul(k_xx_inv, Y))
    mu = np.dot(mu, vec)

    var = k_prbprb - np.matmul(k_probeX, np.matmul(k_xx_inv, np.transpose(k_probeX)))
    var = np.matmul(var, vec)
    var = np.dot(var, vec)

    return mu, var

# Sample coefficients theta from the posterior for a given MCMC sample.
# The first P returned values are {theta_1, theta_2, ...., theta_P}, while
# the remaining values are {theta_ij} for i,j in the list `active_dims`,
# sorted so that i < j.
def sample_theta_space(X, Y, active_dims, msq, lam, eta1, xisq, c, var_obs):
    P, N, M = X.shape[1], X.shape[0], len(active_dims)
    # the total number of coefficients we return
    num_coefficients = P + M * (M - 1) // 2

    probe = np.zeros((2 * P + 2 * M * (M - 1), P))
    vec = np.zeros((num_coefficients, 2 * P + 2 * M * (M - 1)))
    start1 = 0
    start2 = 0

    for dim in range(P):
        probe = jax.ops.index_update(probe, jax.ops.index[start1:start1 + 2, dim], np.array([1.0, -1.0]))
        vec = jax.ops.index_update(vec, jax.ops.index[start2, start1:start1 + 2], np.array([0.5, -0.5]))
        start1 += 2
        start2 += 1

    for dim1 in active_dims:
        for dim2 in active_dims:
            if dim1 >= dim2:
                continue
            probe = jax.ops.index_update(probe, jax.ops.index[start1:start1 + 4, dim1],
                                         np.array([1.0, 1.0, -1.0, -1.0]))
            probe = jax.ops.index_update(probe, jax.ops.index[start1:start1 + 4, dim2],
                                         np.array([1.0, -1.0, 1.0, -1.0]))
            vec = jax.ops.index_update(vec, jax.ops.index[start2, start1:start1 + 4],
                                       np.array([0.25, -0.25, -0.25, 0.25]))
            start1 += 4
            start2 += 1

    eta2 = np.square(eta1) * np.sqrt(xisq) / msq
    kappa = np.sqrt(msq) * lam / np.sqrt(msq + np.square(eta1 * lam))

    kX = kappa * X
    kprobe = kappa * probe

    k_xx = kernel(kX, kX, eta1, eta2, c) + var_obs * np.eye(N)
    k_xx_inv = np.linalg.inv(k_xx)
    k_probeX = kernel(kprobe, kX, eta1, eta2, c)
    k_prbprb = kernel(kprobe, kprobe, eta1, eta2, c)

    mu = np.matmul(k_probeX, np.matmul(k_xx_inv, Y))
    mu = np.sum(mu * vec, axis=-1)

    covar = k_prbprb - np.matmul(k_probeX, np.matmul(k_xx_inv, np.transpose(k_probeX)))
    covar = np.matmul(vec, np.matmul(covar, np.transpose(vec)))
    L = np.linalg.cholesky(covar)

    # sample from N(mu, covar)
    sample = mu + np.matmul(L, onp.random.randn(num_coefficients))

    return sample

# Helper function for doing HMC inference
def run_inference(model, args, rng_key, X, Y, hypers):
    start = time.time()
    kernel = NUTS(model)
    mcmc = MCMC(kernel, args.num_warmup, args.num_samples, num_chains=args.num_chains,
                progress_bar=False if "NUMPYRO_SPHINXBUILD" in os.environ else True)
    mcmc.run(rng_key, X, Y, hypers)
    mcmc.print_summary()
    print('\nMCMC elapsed time:', time.time() - start)
    return mcmc.get_samples()

# Get the mean and variance of a gaussian mixture
def gaussian_mixture_stats(mus, variances):
    mean_mu = np.mean(mus)
    mean_var = np.mean(variances) + np.mean(np.square(mus)) - np.square(mean_mu)
    return mean_mu, mean_var

# Create artificial regression dataset where only S out of P feature
# dimensions contain signal and where there is a single pairwise interaction
# between the first and second dimensions.
def get_data(N=20, S=2, P=10, sigma_obs=0.05):
    assert S < P and P > 1 and S > 0
    onp.random.seed(0)

    X = onp.random.randn(N, P)
    # generate S coefficients with non-negligible magnitude
    W = 0.5 + 2.5 * onp.random.rand(S)
    # generate data using the S coefficients and a single pairwise interaction
    Y = onp.sum(X[:, 0:S] * W, axis=-1) + X[:, 0] * X[:, 1] + sigma_obs * onp.random.randn(N)
    Y -= np.mean(Y)
    Y_std = np.std(Y)

    assert X.shape == (N, P)
    assert Y.shape == (N,)

    return X, Y / Y_std, W / Y_std, 1.0 / Y_std

# Helper function for analyzing the posterior statistics for coefficient theta_i
def analyze_dimension(samples, X, Y, dimension, hypers):
    vmap_args = (samples['msq'], samples['lambda'], samples['eta1'], samples['xisq'], samples['var_obs'])
    mus, variances = vmap(lambda msq, lam, eta1, xisq, var_obs:
                          compute_singleton_mean_variance(X, Y, dimension, msq, lam,
                                                          eta1, xisq, hypers['c'], var_obs))(*vmap_args)
    mean, variance = gaussian_mixture_stats(mus, variances)
    std = np.sqrt(variance)
    return mean, std

# Helper function for analyzing the posterior statistics for coefficient theta_ij
def analyze_pair_of_dimensions(samples, X, Y, dim1, dim2, hypers):
    vmap_args = (samples['msq'], samples['lambda'], samples['eta1'], samples['xisq'], samples['var_obs'])
    mus, variances = vmap(lambda msq, lam, eta1, xisq, var_obs:
                          compute_pairwise_mean_variance(X, Y, dim1, dim2, msq, lam,
                                                         eta1, xisq, hypers['c'], var_obs))(*vmap_args)
    mean, variance = gaussian_mixture_stats(mus, variances)
    std = np.sqrt(variance)
    return mean, std

def main(args):
    X, Y, expected_thetas, expected_pairwise = get_data(N=args.num_data, P=args.num_dimensions,
                                                        S=args.active_dimensions)

    # setup hyperparameters
    hypers = {'expected_sparsity': max(1.0, args.num_dimensions / 10),
              'alpha1': 3.0, 'beta1': 1.0,
              'alpha2': 3.0, 'beta2': 1.0,
              'alpha3': 1.0, 'c': 1.0,
              'alpha_obs': 3.0, 'beta_obs': 1.0}

    # do inference
    rng_key = random.PRNGKey(0)
    samples = run_inference(model, args, rng_key, X, Y, hypers)

    # compute the mean and square root variance of each coefficient theta_i
    means, stds = vmap(lambda dim: analyze_dimension(samples, X, Y, dim, hypers))(np.arange(args.num_dimensions))

    print("Coefficients theta_1 to theta_%d used to generate the data:" % args.active_dimensions, expected_thetas)
    print("The single quadratic coefficient theta_{1,2} used to generate the data:", expected_pairwise)
    active_dimensions = []

    for dim, (mean, std) in enumerate(zip(means, stds)):
        # we mark the dimension as inactive if the interval [mean - 3 * std, mean + 3 * std] contains zero
        lower, upper = mean - 3.0 * std, mean + 3.0 * std
        inactive = "inactive" if lower < 0.0 and upper > 0.0 else "active"
        if inactive == "active":
            active_dimensions.append(dim)
        print("[dimension %02d/%02d]  %s:\t%.2e +- %.2e" % (dim + 1, args.num_dimensions, inactive, mean, std))

    print("Identified a total of %d active dimensions; expected %d." % (len(active_dimensions),
                                                                        args.active_dimensions))

    # Compute the mean and square root variance of coefficients theta_ij for i,j active dimensions.
    # Note that the resulting numbers are only meaningful for i != j.
    if len(active_dimensions) > 0:
        dim_pairs = np.array(list(itertools.product(active_dimensions, active_dimensions)))
        means, stds = vmap(lambda dim_pair: analyze_pair_of_dimensions(samples, X, Y,
                                                                       dim_pair[0], dim_pair[1], hypers))(dim_pairs)
        for dim_pair, mean, std in zip(dim_pairs, means, stds):
            dim1, dim2 = dim_pair
            if dim1 >= dim2:
                continue
            lower, upper = mean - 3.0 * std, mean + 3.0 * std
            if not (lower < 0.0 and upper > 0.0):
                format_str = "Identified pairwise interaction between dimensions %d and %d: %.2e +- %.2e"
                print(format_str % (dim1 + 1, dim2 + 1, mean, std))

        # Draw a single sample of coefficients theta from the posterior, where we return all singleton
        # coefficients theta_i and pairwise coefficients theta_ij for i, j active dimensions. We use the
        # final MCMC sample obtained from the HMC sampler.
        thetas = sample_theta_space(X, Y, active_dimensions, samples['msq'][-1], samples['lambda'][-1],
                                    samples['eta1'][-1], samples['xisq'][-1], hypers['c'], samples['var_obs'][-1])
        print("Single posterior sample theta:\n", thetas)

if __name__ == "__main__":
    assert numpyro.__version__.startswith('0.2.4')
    parser = argparse.ArgumentParser(description="Gaussian Process example")
    parser.add_argument("-n", "--num-samples", nargs="?", default=1000, type=int)
    parser.add_argument("--num-warmup", nargs='?', default=500, type=int)
    parser.add_argument("--num-chains", nargs='?', default=1, type=int)
    parser.add_argument("--num-data", nargs='?', default=100, type=int)
    parser.add_argument("--num-dimensions", nargs='?', default=20, type=int)
    parser.add_argument("--active-dimensions", nargs='?', default=3, type=int)
    parser.add_argument("--device", default='cpu', type=str, help='use "cpu" or "gpu".')
    args = parser.parse_args()

    numpyro.set_platform(args.device)
    numpyro.set_host_device_count(args.num_chains)

    main(args)
```


```python
%matplotlib inline
%config InlineBackend.figure_format = 'svg'

import scqubits as qubit
import scqubits.utils.sweep_plotting as splot
from scqubits import HilbertSpace, InteractionTerm, ParameterSweep

import numpy as np
import qutip as qt
```

# Composite Hilbert Spaces, QuTiP Interface

The `HilbertSpace` class provides data structures and methods for handling composite Hilbert spaces which may consist of multiple qubits or qubits and oscillators coupled to each other. To harness the power of QuTiP, a toolbox for studying stationary and dynamical properties of closed and open quantum systems (and much more), `HilbertSpace` provides a convenient interface: it generates `qutip.qobj` objects which are then directly handled by QuTiP.

## Example: two transmons coupled to a harmonic mode

Transmon qubits can be capacitively coupled to a common harmonic mode, realized by an LC oscillator or a transmission-line resonator. The Hamiltonian describing such a composite system is given by:
\begin{equation}
H=H_\text{tmon,1} + H_\text{tmon,2} + \omega_r a^\dagger a + \sum_{j=1,2}g_j n_j(a+a^\dagger),
\end{equation}
where $j=1,2$ enumerates the two transmon qubits, $\omega_r$ is the (angular) frequency of the resonator. Furthermore, $n_j$ is the charge number operator for qubit $j$, and $g_j$ is the coupling strength between qubit $j$ and the resonator.

### Create Hilbert space components

The first step consists of creating the objects describing the individual building blocks of the full Hilbert space. Here, these will be the two transmons and one oscillator:

```python
tmon1 = qubit.Transmon(
    EJ=40.0,
    EC=0.2,
    ng=0.3,
    ncut=40,
    truncated_dim=4     # after diagonalization, we will keep 3 levels
)

tmon2 = qubit.Transmon(
    EJ=15.0,
    EC=0.15,
    ng=0.0,
    ncut=30,
    truncated_dim=4
)

resonator = qubit.Oscillator(
    E_osc=4.5,
    truncated_dim=4  # up to 3 photons (0,1,2,3)
)
```

The system objects are next grouped into a Python list, and in this form used for the initialization of a `HilbertSpace` object. Once created, a print call to this object outputs a summary of the composite Hilbert space.

```python
hilbertspace = qubit.HilbertSpace([tmon1, tmon2, resonator])
print(hilbertspace)
```

    ====== HilbertSpace object ======

    TRANSMON
     ———— PARAMETERS ————
    EJ	: 40.0
    EC	: 0.2
    ng	: 0.3
    ncut	: 40
    truncated_dim	: 4
    Hilbert space dimension	: 81

    TRANSMON
     ———— PARAMETERS ————
    EJ	: 15.0
    EC	: 0.15
    ng	: 0.0
    ncut	: 30
    truncated_dim	: 4
    Hilbert space dimension	: 61

    OSCILLATOR
     ———— PARAMETERS ————
    E_osc	: 4.5
    truncated_dim	: 4
    Hilbert space dimension	: 4

One useful method of the `HilbertSpace` class is `.bare_hamiltonian()`. This yields the bare Hamiltonian of the non-interacting subsystems, expressed as a `qutip.Qobj`:

```python
bare_hamiltonian = hilbertspace.bare_hamiltonian()
bare_hamiltonian
```

Quantum object: dims = [[4, 4, 4], [4, 4, 4]], shape = (64, 64), type = oper, isherm = True\begin{equation*}\left(\begin{array}{*{11}c}-48.968 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & -44.468 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & -39.968 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & -35.468 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & -44.881 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\\vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & -4.723 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & -14.477 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & -9.977 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & -5.477 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & -0.977\\\end{array}\right)\end{equation*}

### Set up the interaction between subsystems

The pairwise interactions between subsystems are assumed to have the general form

$V=\sum_{i\not= j} g_{ij} A_i B_j$,

where $g_{ij}$ parametrizes the interaction strength between subsystems $i$ and $j$. The operator content of the coupling is given by the two coupling operators $A_i$, $B_j$, which are operators in the two respective subsystems.
This structure is captured by setting up an `InteractionTerm` object:

```python
g1 = 0.1  # coupling resonator-CPB1 (without charge matrix elements)
g2 = 0.2  # coupling resonator-CPB2 (without charge matrix elements)

interaction1 = InteractionTerm(
    hilbertspace = hilbertspace,
    g_strength = g1,
    op1 = tmon1.n_operator(),
    subsys1 = tmon1,
    op2 = resonator.creation_operator() + resonator.annihilation_operator(),
    subsys2 =resonator
)

interaction2 = InteractionTerm(
    hilbertspace = hilbertspace,
    g_strength = g2,
    op1 = tmon2.n_operator(),
    subsys1 = tmon2,
    op2 = resonator.creation_operator() + resonator.annihilation_operator(),
    subsys2 =resonator
)
```

Each `InteractionTerm` object is initialized by specifying
1. the Hilbert space object to which it will belong
2. the interaction strength coefficient $g_{ij}$
3. `op1`, `op2`: the subsystem operators $A_i$, $B_j$ (these should be operators within the subsystems' respective Hilbert spaces only)
4. `subsys1`: the subsystem objects to which `op1` and `op2` belong

Now, collect all interaction terms in a list, and insert into the HilbertSpace object.

```python
interaction_list = [interaction1, interaction2]
hilbertspace.interaction_list = interaction_list
```

With the interactions specified, the full Hamiltonian of the coupled system can be obtained via the method `.hamiltonian()`. Again, this conveniently results in a `qubit.Qobj` operator:

```python
dressed_hamiltonian = hilbertspace.hamiltonian()
dressed_hamiltonian
```

Quantum object: dims = [[4, 4, 4], [4, 4, 4]], shape = (64, 64), type = oper, isherm = True\begin{equation*}\left(\begin{array}{*{11}c}-48.968 & 0.030 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.030 & -44.468 & 0.042 & 0.0 & 0.261 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.042 & -39.968 & 0.052 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.052 & -35.468 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.261 & 0.0 & 0.0 & -44.881 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\\vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & -4.723 & 0.0 & 0.0 & -0.749 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & -14.477 & 0.030 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.030 & -9.977 & 0.042 & 0.0\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & -0.749 & 0.0 & 0.042 & -5.477 & 0.052\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.052 & -0.977\\\end{array}\right)\end{equation*}

### Obtaining the eigenspectrum via QuTiP

Since the Hamiltonian obtained this way is a proper `qutip.qobj`, all QuTiP routines are now available. For instance, here is how to obtain eigenenergies and eigenstates:

```python
evals, evecs = hilbertspace.eigensys(evals_count=4)
print(evals)
```

    [-48.97770317 -45.02707241 -44.36656205 -41.18438832]

## Sweeping over an external parameter

scqubits provides the class `ParameterSweep` to facilitate computation of spectra as function of an external parameter. For efficiency in creating different plots, the computed bare and dressed spectral data are stored internally. A sweep object is initialized as follows.

Suppose both `tmon1` and `tmon2` subsystems are flux-tunable split-junction transmons with different loop areas.
We consider sweeping a global external flux through the loops. To create the necessary `ParameterSweep` object, we specify:
1. `param_name`: the name of the sweep parameter (below expressed as the flux in units of the flux quantum)
2. `param_vals`: an array with the flux values used in the sweep
3. `subsys_update_list`: a list containing all Hilbert space subsystems that change as the flux is varied
4. `update_hilbertspace(param_val)`: a function that shows how a change in the sweep parameter affects the Hilbert space; here only the `.flux` attributed of the fluxonium qubit needs to be changed

These ingredients all make it into the initialization of the `ParameterSweep` object. Once initialized, spectral data is generated and stored.

```python
param_name = 'flux'                          # name of varying external parameter
param_vals = np.linspace(0.0, 2.0, 300)     # parameter values

subsys_update_list = [tmon1, tmon2]          # list of HilbertSpace subsystems which are affected by parameter changes

def update_hilbertspace(param_val):          # function that shows how Hilbert space components are updated
    tmon1.EJ = 20*np.abs(np.cos(np.pi * param_val))
    tmon2.EJ = 15*np.abs(np.cos(np.pi * param_val * 0.7))

sweep = ParameterSweep(
    param_name=param_name,
    param_vals=param_vals,
    evals_count=20,
    hilbertspace=hilbertspace,
    subsys_update_list=subsys_update_list,
    update_hilbertspace=update_hilbertspace,
)
```

    HBox(children=(FloatProgress(value=0.0, description='Bare spectra', max=300.0, style=ProgressStyle(description…

    HBox(children=(FloatProgress(value=0.0, description='Dressed spectrum', max=300.0, style=ProgressStyle(descrip…

Once the data has been generated, the spectrum can be plotted as a function of the parameter value:

```python
splot.dressed_spectrum(sweep, ymax=10);
```

```python

```


Copyright statement?

# A New 'Transition'
Welcome back! At this point in the course you are all likley aces at numerical analysis, expecially after 5 modules ranging from Diffusion, to Convection, to Conduction, in 1 or 2D, with various Neumann and Dirichlet Boundary conditions!

But now it's time for a new "phase". In this lesson we begin looking at what happens when your nice rigid boundary conditions decide to start moving on you, essentially creating a moving boundary interface!

A moving boundary interface is represented by numerous physical behaviors in real-world applications, from the polar ice caps melting, to the phase transformations of metal alloys, to the varying oxygen content of  muscles near a clotted bloodvessel [2].

 #### Real World Applications of a Moving Boundary Interface

### The Stefan Problem
This new type of problem is known as the "Stefan Problem" as it was first studied by Slovene physicist Jozef Stefan around 1890 [3]. Though his focus was primarily on the analysis of ice formations, nowadays his name is synonymous with the particular type of boundary value problem for PDEs where the boundary can move with time.  Since the classic Stefan problem concentrated on the temperature distribution of a homogeneous material undergoing a phase transition, one of the most commonly studied Stefan problems today is the melting of Ice to Water!

 #### Jozef Stefan pioneered work into phase transitions of materials (ie Ice)

## A Review: 1D Heat Conduction

Recall from both Modules 2 and 4 we took a loook at the Diffusion equation in 1D:

$$\begin{equation}
\frac{\partial U}{\partial t} = \alpha \frac{\partial^2U}{\partial x^2}
\end{equation}$$

Where we have the temperature distribution $U(x,t)$ and the thermal diffusivity $\alpha$. While before we looked at the conduction of heat through a graphite rod of length 1 meter, in this scenario we will analyze heat conduction though a 1D rod of ice.  Let's first list some basic coefficients associated with the new material:

##### Thermal Properties of Ice at ~$0^{\circ}C$:
Thermal Conductivity: $k = 2.22 \frac{W}{mK}$

Density: $\rho \approx 1000 \frac{kg}{m^3}$

Specific Heat: $c_{p}= 2.05x10^{-3} \frac{J}{kgK}$

and lastly, Thermal Diffusivity: $\alpha = \frac{k_{ice}}{\rho_{ice}c_{p_{ice}}} = 1.083x10^{-6} \frac{m^2}{sec}$

Melting Temperature of Ice is: $T_{melt}=0^{\circ}C$

Okay! With that out of the way, let's look at the temperature distribution across the a rod of ice of length 1m with some basic initial and boundary conditions. For this first scenario, we will not take into account phase transition if temperatures hit $T_{melt}$, and will therefore assume a static boundary condition.

##### Problem Setup
Governing Equation: $$\begin{equation}
\frac{\partial U}{\partial t} = \alpha \frac{\partial^2U}{\partial x^2}
\end{equation}$$

Boundary Conditions:

LHS: $\frac{\partial U}{\partial x} = -e^{\beta t}$,   @x=0, t>0, time-dependant (increasing) heat flux in

RHS: $\frac{\partial U}{\partial x} = 0$,  @x=L, t>0,   Insulated end, no heat flux in/out.

Initial Conditions:

$U(x,t) = -10^{\circ}C$, for 0<x<L, t=0

Lets start coding!!

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family']= 'serif'
rcParams['font.size']=16
from IPython.display import Image
```

Now let's define a function to run the governing equations using the Forward-Time/Central-Difference Method discretization

```python
def FTCS(U, nt, dt, dx, alpha, beta):

    for n in range(nt):

        Un=U.copy()
        U[1:-1] = Un[1:-1] + alpha*(dt/dx**2)*(Un[2:]-2*Un[1:-1]+Un[0:-2])

        #Boundary Conditions

        U[-1]=U[-2]                  #RHS Insulated BC

        t=n*dt                                    #Increasing time = n*timestep(dt)
        U[0]=numpy.exp(beta*t)     #LHS Time dependant heat input in, BC

    return U
```

```python
#Basic Parameters and Initialization

# Temperature scale: Celsius
# Length scale: meters
# Mass scale: kg
# Time scale: seconds
# Energy scale: Joules
# Power scale: Watts

L = 1                                                         # Length of my ice rod
nt = 40000                                                  # Number of timesteps
nx = 51                                                     # Number of grid space steps
alpha = 1.083e-6                                    # Thermal Diffusivity
dx = L/(nx-1)                                           # grid spacing in "x"
Ui = numpy.ones(nx)*(-10)                    # initialized Temperature array
beta = 0.001                                               #Growth Factor of my Temperature input

sigma = 1/2                                             # Stability*
dt=0.1                                                      # Timestep chosen to be 0.1 seconds*
```

A word on Stability:

Recall from module 2 we had the Courant-Friedrichs-Lewy (CFL) Stability condition for the FTCS Diffusion equation in the form:
$$\sigma = \alpha \frac{\Delta t}{\Delta x^{2}} \leq \frac{1}{2} $$

re-arranging to determine the upper limit for a time-step (dt) we have:
$$\Delta t \leq \frac{\sigma \Delta x^{2}}{\alpha} \approx 923 seconds$$

As you can see, chosing a time-step (dt) equal to $0.1$ seconds, we more than satisfy this CFL Stability condition.

```python
x = numpy.linspace(0, 1, nx)
print('Initial ice rod temperature distribution')
pyplot.plot(x, Ui, color = '#003366', ls = '-', lw =4)
pyplot.ylim(-20, 60)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')

```

```python
U = FTCS(Ui.copy(), nt, dt, dx, alpha, beta)
print('Total elapsed time is ', nt*dt, 'seconds, or', nt*dt/60, 'minutes' )
pyplot.plot(x, U, color = '#003366', ls = '-', lw =3)
pyplot.ylim(-20, 60)
pyplot.xlim(0, 1)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
```

The above figure shows us the temperature distribution of the ice rod at a point in time approximately 1 hour into applying our heat source. Now this would be all well and good if ice didn't melt (and therefore change phase) at zero degrees. But it does!!

Let's build a rudimentary function to see what portion of the rod should be water by now, and which part should still be ice:

```python
def Phase_graph(U, x, nx):
    phase=numpy.ones(nx)*(-100)
    for n in range(nx):
        if U[n]>0:
            phase[n]=100
    return phase
```

```python
pyplot.plot(x, U, color = '#003366')
pyplot.ylabel('Temperature', color ='#003366')
Phase1=numpy.ones(nx)
Phase=Phase_graph(U, x, nx)
pyplot.plot(x, Phase, color = '#654321', ls = '-', lw =3)
pyplot.ylim(-12, 10)
pyplot.xlim(0, 0.3)
pyplot.xlabel('Approx Water-Ice Interface', color = '#654321')
```

As you can see, the ice SHOULD have melted about 0.07 meters  (or 2.75 inches) into our rod. In reality, our boundary interface has moved from x=0 to the right as time elapsed.  Not only should our temperature distribution profile change due to the differences in properties ($\rho, k, c_{p}$), but also the feedback from the moving boundary condition.

## Solutions to the Dimensionless Stefan problem:
### Analytical Solution:

Before we continue we need to make some simplfications:

1) No convection, heat transfer is limited to conduction,

2) Pressure is constant,

3) Density does not change between the solid and liquid phase (Ice/Water), ie $\rho_{ice}=\rho_{water}\approx 1000 \frac{kg}{m^3}$,

4) The phase change interface at $s(t)$ has no thickness

Looking closer at the problem at hand, we see that our temperature distribution must conform to the below figure:

 #### Phase Change Domain

  As you can see, our temperature distribution experiences a discontinuity at our solid-liquid interface at x=s(t). Furthermore, this boundary moves to the right as time elapses. If we wish to analyze the distribution in one region or the other, we need to take into account a growing or shrinking domain, and therefore, the boundary of the domain has to be found as part of the solution [2].  In the Melting Problem depicted above, a moving interface separates the liquid and solid phases. The displacement of the boundary interface (denoted as $\dot{s}$) is driven by the heat transport through it [2]. The relationship  between the moving interface, s(t), and the temperature distribution through it was developed by and is known as the Stefan Condition (or Stefan Equation) and takes the form of the following boundary conditions:

when x =s(t),
######  [1, 3, 2]
$$\dot{s} = \frac{ds}{dt} = -\frac{\partial U}{\partial x}$$
$$U=0$$

There are many solution methods for the Stefan problem, for this lesson we will focus on finding the temperature distribution in the liquid region only (ie 0<x<s(t)). Critical parameters to track are both temperature, U(x,t), and the position of the interface, s(t). Let's first solve this problem analytically:

To make our lives easier we want a simplified version of the Stefan Problem, In his paper, Crowley [4] presented that both Oleinik and Jerome demonstrated how using an appropriate dimensionless model, and appropriate boundary conditions, and initial conditions, not only could one determine a general solution for the diffusion equation with the Stefan condition, but also that an explicit finite difference scheme converges to this solution [4]. For our dimensionless model we will set $\alpha=1, \beta=1$ to get the following governing equations, and boundary conditions:

#### Dimensionless Stefan Problem Equations

$$\frac{\partial U}{\partial t} = \frac{\partial^{2} U}{\partial x^{2}}$$

$$s(t)$$

(1) $\frac{\partial U(x=0, t)}{\partial x} = -e^{t}$, LHS BC (Heat input into the system)

(2) $\frac{\partial U(x=s(t), t)}{\partial x} = -\frac{d s(t)}{d t}$, RHS BC, Stefan Condition

(3) $U(x=s(t), t) = T_{melt}=0$, By definition of the Melting interface

(4) s(t=0)=0, initial condition

These equations set up the new below figure:

Because we want to be able to judge the accuracy of our numerical analysis (and thanks to Crowley, Oleinik, and Jerome!) we next find the general solution of this problem by means of Separation of Variables:

SOV: $U(x,t)=X(x)T(t)$, gives us a general solution of the form:

$$U(x,t) = c_{1}e^{c_{2}^{2}t-c_{2}x} +c_{3}$$

from BC(1) we get $c_{2}=1, c_{1}=1, yielding: U(x,t) = e^{t-x} +c_{3}$

from BC(2) we get:  $\dot{s}=\frac{\partial s(t)}{\partial t} = -\frac{\partial U(x=s(t), t)}{\partial x}\rightarrow -e^{t-s(t)}= \frac{ds(t)}{d t}$, this yields that the only solution fo s(t) is: $$s(t)=t$$

from BC(3) we get: $U(s(t),t) + c_{3} = 0, \rightarrow e^{t-t} +c_{3}=0, \rightarrow c_{3}=-1$,

Finally we get an exact solution for the temperature distribution:

$$U(x,t) = e^{t-x}-1$$,    $$s(t) = t$$

and this satisfies the initial condition IC(4) that s(t=0)=0!!

And now you see why we chose the heat input function to be a time-dependant exponential, if you don't believe me that the analytical solution would be much nastier if you had chosesn say, a constant heat input, give SOV a try and let me know how it goes! Let's graph this solution and see if it makes sense.

```python
def Exact_Stefan( nt, dt, x):
    U=numpy.ones(nx)
    for n in range(nt):
        U = numpy.exp(n*dt-x)-1
    return U
```

```python
dt_A=0.002
nt_A=500
ExactS=Exact_Stefan(nt_A, dt_A, x)
Max_tempA=max(ExactS)

pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
print('Analytically, this is our temperature profile after:', nt_A*dt_A,'seconds')
print('The temperature at our LHS boundary is:', Max_tempA, 'degrees')
```

Does the solution look right? YES! Remember, this temperature profile is "Dimensionless" so we can't compare it to our previous example (not only is the diffusivity constant vastly different, but we are looking 1 second into the diffusion vs 1 hr!). Also it accounts ONLY for the temperature distribution in the liquid, assuming an ever increasing domain due to an ever expanding RHS boundary. Not only did we expect temperature to be highest at the input side, but our moving boundary interface, s(t), moves to the right with time and always hits our melting temperature ($0^{\circ} c$) at x=1, just as one would expect.

### Numerical Solution: The Variable Grid Method

What makes this problem unique is that we must now track the time-dependant moving boundary. Now, for previous numerical analysis in a 1D domain, you had a constant number of spatial intervals nx (or N) between your two boundaries x=0, and x=L. Thus your spatial grid steps dx were constant and defined as $dx = L/(nx-1)$. But now you have one fixed bounary and one moving boundary, and your domain increases as time passes. [1]

One of the key tenements of the Variable Grid Method for solving the Stefan probem is that you keep the NUMBER of grid points (nx) fixed, thus your grid SIZE (dx) will increase as the domain increases. Your grid size now varies dependant on the location of the interaface front, dx = s(t)/N.

$$dx = \frac{L}{N-1}     \longrightarrow      dx = \frac{s(t)}{N}$$

Now, while one might be tempted to view the figure above as the new "Variable Grid" stencil, one must remember that the "Y"-axis is time, and therefore as you move up you are moving forward in time, therefore a more accurate depiction of the FTCS method in stencil form for the Variable Grid method would be:

and so it becomes clear - that the our spatial step ($dx$) will be depentant on our time step ($m$)!!

####  Derivation of the new Governing Equation

   Now let's set up the new governing equations to be discretized for our code. We know that the 1D Diffusion equation must be valid for all points on our spatial grid so we can re-write the LHS of equation:
   $$\frac{\partial U}{\partial t} = \frac{\partial^{2} U}{\partial x^{2}}$$
   to be $$ \frac{\partial U_{i}}{\partial t} = \frac{\partial U_{@t}}{\partial x}\frac{dx_{i}}{dt} + \frac{\partial U_{@x}}{\partial t}$$

  we can use the expression: $$\frac{dx_{i}}{dt} = \frac{x_{i}}{s(t)}\frac{ds}{dt}$$   to track the movement of the node $i$.

  Substituting these into the diffusion equation (and droping the i,t, and x indices since they are constant) we obtain a new governing equation for diffusion:

  $$\frac{\partial U}{\partial t} = \frac{x_{i}}{s}\frac{ds}{dt}\frac{\partial U}{\partial x} + \frac{\partial^{2}U}{\partial x^{2}}$$

  This is subject to the boundary and initial conditions (BC1-BC3, IC4) as stated for the 1D dimensionless Stefan problem above.

#### Discretization

and now we seek to discretize the new governing equations. For this code we will implement an explicit, FTCS scheme, with parameters taking a Taylor expansion centered about the node ($x_{i}^{m}$), and time ($t_{m}$) just like before. In the above equations we discritized the time derivatives of U using forward time, and the spatial derivitaives using centered space. We re-write ds/dt as $\dot{s}$ and leave it a variable for now. (EXERCISE: See if you can discretize this equation form memory!!)

Taylor Expand and Discretize: (moving from left to right)

$\frac{\partial U}{\partial t} \longrightarrow  \frac{U_{i}^{m+1}-U_{i}^{m}}{\Delta t}$,

$\frac{x_{i}}{s}  \longrightarrow  \frac{x_{i}^{m}}{s_{m}}$,

$\frac{ds}{dt} \longrightarrow \dot{s_{m}}$,

$\frac{\partial U }{\partial x} \longrightarrow \frac{U_{i+1}^{m}-U_{i-1}^{m}}{2\Delta x^{m}}$,

$\frac{\partial^{2}U}{\partial x^{2}} \longrightarrow \frac{U_{i+1}^{m}-2U_{i}^{m}+U_{i-1}^{m}}{(\Delta x^{m})^{2}}$

And now we substitute, rearrange and solve for $U_{i}^{m+1}$ to get:

$U_{i}^{m+1}=U_{i}^{m}+\frac{\Delta t x_{i}^{m} \dot{s_{m}}}{2 \Delta x^{m}s_{m}}(U_{i+1}^{m}-U_{i-1}^{m}) + \frac{\Delta t}{(\Delta x^{m})^{2}}(U_{i+1}^{m}-2U_{i}^{m}+U_{i-1}^{m})$

Great!! We're almost ready to start coding, but before we begin, do you notice a problem with this expression? What about if i=0? Plug it in and you will see that we have expressions of the form $U_{-1}^{m}$ in both right hand terms, but that can't be right, looking at our stencil we see that $i=-1$ is off our grid! This is where the boundary conditions come in!

Generate discretization expressions for the boundary conditions at x=0 (LHS) and x=s(t) (RHS). For the RHS and the temperature gradient across the moving boundary interface we will use a  three-term backward difference scheme:

LHS:  $\frac{\partial U(x=0, t)}{\partial x} = -e^{t} \longrightarrow    \frac{U_{i+1}^{m}(0,t)-U_{i-1}^{m}(0,t)}{2\Delta x^{m}} = e^{t_{m}}$, and

RHS: $\frac{\partial U(x=s(t), t)}{\partial x} = -\frac{d s(t)}{d t} \longrightarrow \frac{\partial U(x=s(t),t)}{\partial x} = \frac{3U_{N}^{m}-4U_{N-1}^{m}+U_{N-2}^{m}}{2\Delta x^{m}}  $

from our LHS boundary condition expression, if we set $i=0$ and solve for $U_{-1}^{m}$ we get: $$U_{-1}^{m}=U_{1}^{m} + 2\Delta x^{m} e^{t_{m}}$$

We can now combine this expression and substitute into the governing equation to get expressions for diffusion at  $i=0, i=1$ to $(N-1)$, and $i=N$:

$U_{i}^{m+1}=(1-2\frac{\Delta t}{(\Delta x^{m})^{2}})U_{i}^{m} + 2\frac{\Delta t}{(\Delta x^{m})^{2}}U_{i+1}^{m} + (2\frac{\Delta t}{\Delta x^{m}}-\frac{\Delta t x_{i}^{m}\dot{s_{m}}}{s_{m}})e^{t_{m}}$

$U_{i}^{m+1}=U_{i}^{m}+\frac{\Delta t x_{i}^{m} \dot{s_{m}}}{2 \Delta x^{m}s_{m}}(U_{i+1}^{m}-U_{i-1}^{m}) + \frac{\Delta t}{(\Delta x^{m})^{2}}(U_{i+1}^{m}-2U_{i}^{m}+U_{i-1}^{m})$

$U_{i}^{m}=0$, $i=N$

Phew!! Now that is one impressive looking set of discretization expressions. But, sorry, we aren't done yet..... In the above expressions what are we supposed to do about $\dot{s_{m}}$ and $s_{m}$?

Well luckily, that is where the Stefan condition comes in, remember:

$\dot{s_{m}} = \frac{ds}{dt}= -\frac{\partial U_{@x=s}}{\partial x} = -\frac{3U_{N}^{m}-4U_{N-1}^{m}+U_{N-2}^{m}}{2\Delta x^{m}} $, and as for $s_{m}$ that is just a Heat Balance equation:

$s_{m+1} = s_{m}+\dot{s_{m}}\Delta t = s_{m} - \frac{\Delta t}{2\Delta x^{m}}(3U_{N}^{m}-4U_{N-1}^{m}+U_{N-2}^{m})$, where $s_{0}=0$

To make our lives easier, instead of inserting these expressions for $\dot{s_{m}}$ and $s_{m}$ into our governing equations, lets keep them as coupled expressions to be calculated during the time loops and calculated at every grid point $i$.

lastly, lets not forget that the updated interface location $s_{m+1}$ and grid size $\Delta x^{m}$ are calculated at every timestep $m$, and have the relationship:

$$\Delta x^{m+1} = \frac{s_{m+1}}{N}$$

Don't worry, we are almost ready to code, I promise! There is just one more expression we need to consider, and that relates to Stability! Following the calculations of Caldwell and Savovic [5], for now, we will use something simple in order to limit the size of the timestep ($\Delta t$), that is: $$\frac{\Delta t}{(\Delta x^{m})^{2}} \leq 1$$, or

$$\Delta t \leq (\Delta x^{m})^{2}$$

## Let's Code!!

To summarize our governing equations we need to code these 7 coupled equations:

(A) $u_{i}^{m+1}=(1-2\frac{\Delta t}{(\Delta x^{m})^{2}})u_{i}^{m} + 2\frac{\Delta t}{(\Delta x^{m})^{2}}u_{i+1}^{m} + (2\frac{\Delta t}{\Delta x^{m}}-\frac{\Delta t x_{i}^{m}\dot{s_{m}}}{s_{m}})e^{t_{m}}$

(B) $u_{i}^{m+1}=u_{i}^{m}+\frac{\Delta t x_{i}^{m} \dot{s_{m}}}{2 \Delta x^{m}s_{m}}(u_{i+1}^{m}-u_{i-1}^{m}) + \frac{\Delta t}{(\Delta x^{m})^{2}}(u_{i+1}^{m}-2u_{i}^{m}+u_{i-1}^{m})$

(C) $u_{i}^{m}=0$, $i=N$

(D) $\dot{s_{m}} = \frac{ds}{dt}= -\frac{\partial u_{@x=s}}{\partial x} = -\frac{3u_{N}^{m}-4u_{N-1}^{m}+u_{N-2}^{m}}{2\Delta x^{m}} $

(E) $s_{m+1} = s_{m}+\dot{s_{m}}\Delta t = s_{m} - \frac{\Delta t}{2\Delta x^{m}}(3u_{N}^{m}-4u_{N-1}^{m}+u_{N-2}^{m}), s_{0}=0$

(F) $\Delta x^{m+1} = \frac{s_{m+1}}{N}$

(G) $\Delta t \leq (\Delta x^{m})^{2}$

```python
# First we set up some new initial parameters.These values differ from our first
#example since we are now dealing with a new non-dimensionalized heat equation,
#and new governing equations.

#1.0 Numerical Solution Parameters:
dt=2.0e-6              # Size of our Time-step, Set constant, Chosen in accordance
                       # with Caldwell and Savovic [5]
nt=500000              # Number of time steps, chosen so that elapsed solution will
                       # determine heat diffusion after
                       # (nt*dt= 1.0 seconds) Just like in our Analytical Solution.
N = 10                 # Number of spatial grid points, Chosen, for now, in
                       #accordance with Caldwell and Savovic [5]

#2.0 Array Creation and Initial Values:
Uo=0                          #Initial Temperature input
U0 = numpy.ones(N)*(Uo)
s0=0.02                      #Initial Interface Position
                             #(cannot chose zero or our expressions would blow up!)
s=numpy.ones(nt)*(s0)
dx=numpy.ones(nt)*(s[1]/N)
sdot=(s[1]-s[0])/dt

s[1]= s[0] - (dt/(2*dx[0]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])
dx[0]=(s[1]/N)

print('Initial Temperature Array:', U0)
print('Initial Position of the interface (So):', s[0])
print('Initial X_step(dx):', dx[0])
print('Time Step (dt):', dt)
print('Initial speed of our moving interface (Sdot):', sdot)
```

    Initial Temperature Array: [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]
    Initial Position of the interface (So): 0.02
    Initial X_step(dx): 0.002
    Time Step (dt): 2e-06
    Initial speed of our moving interface (Sdot): 0.0

```python
# 3.0 Application of the Stefan Problem Governing Equations

for m in range(0, nt-1):

    for i in range(0, N-1):

        #LHS BC (x=0, i=0)
         if(i==0):
                U0[i]= (1-2*(dt/dx[m]**2))*U0[i]+2*(dt/dx[m]**2)*U0[i+1] + \
                       (2*(dt/dx[m])-dt*(dx[m]*i)*sdot/s[m])*numpy.exp(m*dt)

        #Governing Equation (B)
         else:
                U0[i]= U0[i]+((dt*(dx[m]*i)*sdot)/(2*dx[m]*s[m]))*(U0[i+1]-U0[i-1]) \
                            +(dt/(dx[m]**2))*(U0[i+1]-2*U0[i]+U0[i-1])

         #RHS BC (x=L, L=dx{m*N})
         U0[N-1]=0

    #Heat Balance Equation, (E)

    s[m+1]= s[m] - (dt/(2*dx[m]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])

    sdot=(s[m+1]-s[m])/dt       #Updating Speed of Interface
    dx[m+1]=(s[m+1]/N)          #Updating dx for each time step

    if (U0[i]>0):
        Location=s[m]

```

### Results and Discussions

Ok! Now lets see what we get, and more importantly, if it makes any sense:

```python
XX = numpy.linspace(0, 1, N)
Max_tempVGM=max(U0)

pyplot.plot(XX,U0, color = '#654322', ls = '--', lw =4)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)

print('This is our VGM temperature profile after:', nt*dt, 'seconds')
print('The temperature at our LHS boundary is:', Max_tempVGM, 'degrees')
print('Final position of our interface is:', s[m])
print('The final speed of our interface is:', sdot)
print('Our grid spaceing (dx) after', nt*dt, 'seconds is:', dx[m] )
```

Hey now! That looks pretty good! Let's compare this result to our earlier Analytical (exact) solution to the 1D Dimensionless problem, for 1 second into the diffusion:

```python
pyplot.figure(figsize=(12,10))
pyplot.plot(XX,U0, color = '#654322', ls = '--',  lw =4, label = 'Variable Grid Method')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical Solution')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(0, 1.8)
pyplot.xlim(0, 1.0)
pyplot.legend();

print('Max error (@x=0) is:',abs(Max_tempVGM - Max_tempA)*(100/Max_tempA),'percent')
```

Under 2% error is pretty good! What else should we verify? How about the change in the interface location s(t) and the size of our spatial grid (dx) over time?

```python
Time=numpy.linspace(0,dt*nt,nt)
pyplot.figure(figsize=(10,6))

pyplot.plot(Time,dx, color = '#660033', ls = '--', lw =2, label='Spatial Grid Size')
pyplot.plot(Time,s,color = '#222222', ls = '-', lw =4, label='Interface Position, s(t)')
pyplot.xlabel('Time (Seconds)')
pyplot.ylabel('Length')
pyplot.legend();
```

and so we see that both the spatial grid size, dx, and the interface location, s(t), increase with time over the course of 1 second. This is exactly what we would expect! Infact, remember back to our anayltical solution, we solved the interface function  to be: $s(t)=t$, which is exactly the graph that our numerical solution has given us!

Well, then! That's a wrap! Looks like everything is all accounted for, right?. . . . . . . . . . . .  . .well, not exactly. There are still quite a few questions left unanswered, and now we must discuss the limitations in the Variable Grid Method of numerical implementation.

### Limitations of the Variable Grid Method

A number of questions should be coming to mind, among those are:

1) Why do we only analyze for 1 second of diffusion?

2) Why is your VGM timestep (dt) so small at 2.0e-6?

3) Why is your number of spatial grid points (N) only 10?

These are all VERY good questions, do you already know the answer? Here is a hint: it all has to do with stability!

You see, we have a relatively benign stability statement: $\Delta t \leq (\Delta x^{m})^{2}$, but still, it is a crutial aspect of our numerical analysis. This statement limits the size of our timestep, dt, and is the key reason why we chose our initial, constant, dt to be 2.0e-6 (lets not also forget the fact that this dt and N were also chosen by Caldwell and Savovic [5]) . Remember, because we are using a Central Difference scheme, our stability criteria essentially comes from the CFL condition:

$$\frac{\Delta t}{(\Delta x^{m})^{2}} \leq \sigma = 1$$.

The parameters dt and dx are not just arbitrarilty chosen. They determine the speed of your numerical solution. At all times you need the speed at which your solution progresses to be faster than the speed at which the problem (in this case, thermal diffusion) propagates. Afterall, how can you calculate the solution numerically when the solution is faster than the numerical analysis?!

Here is a pertinent question for you: Which do you determine first: dt, nt, N, or L? Does it matter which one you chose first?
The answer is normally, yes, we chose N and L which gives us dx, we then use a stability statement with dx (and $\alpha$) to limit dt, we then choose our nt to determine the elapsed time into our solution in the form of (nt*dt) seconds, easy right?

Well this is where we get to limitation #1 with the Variable Grid Method:

Because the end of our domain "L" is in actuality the position of our interface, s(t), and our interface at time t=0 is approximately at the origin (x=0.02), then if we want a constant number of spatial grids N (say N=10), our initial dx is extremely small!. Remember governing equation (F) says: $\Delta x^{m+1} = \frac{s_{m+1}}{N}$, well at t=0 (m=0) s[0] is 0.02, meaning our initial dx, dx[0] is: 0.002. Plug this back into our stability statement and you see that dt must be smaller than or equal to 4e-6! That is VERY harsh criterion for a timestep dt. In essence, with SUCH a small time step, we would need hundreds of thousands of time loops, nt, in order to get just 1 second of data. Now you see why nt is chosen to be 500,000 in order to get just 1 second of data. If I had wanted more time elapsed, we would have needed iterations in the millions. (This is also why the analytical solution was calculated only for 1 second, we wanted to be able to compare results at the end!)

Now limitation #2 of the Variable Grid Method: Because the interface starts at the origin, the initial spatial step is very small, and forces a very very small timestep, dt. Thus a very very large number of iterations is needed to get into the "Seconds" scale for simulations, this is a slow numerical process...

Not only this, but we have limitation #3 of the Variable Grid Method: If we want a longer dimensional domain, we need to choose a larger N. This will give us a final domain of $L=N*dx_{final}$, but if N is larger, then going back to our governing equation (F),  $\Delta x^{m+1} = \frac{s_{m+1}}{N}$ this means that dx[0] is even smaller, which continues to force a much smaller dt. Thus for the Variable Grid Method, if you want a large domain, you need an even smaller timestep, which forces more time in your numerical calculation.

Let's test this out for ourselves, lets redo the VGM calculations but increase N to say, 14:
(The cell below has the nt set to 500 since we didn't want it to slow you down the first time you ran this program. Notice the N is now 14, go ahead and run this below cellwith nt=500000 and see what happens)

```python
dt=2.0e-6
nt=500
#nt=500000

N = 14

Uo=0
U0 = numpy.ones(N)*(Uo)
s0=0.02
s=numpy.ones(nt)*(s0)
dx=numpy.ones(nt)*(s[1]/N)
sdot=(s[1]-s[0])/dt

s[1]= s[0] - (dt/(2*dx[0]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])
dx[0]=(s[1]/N)

for m in range(0, nt-1):
     for i in range(0, N-1):
            if(i==0):
                U0[i]= (1-2*(dt/dx[m]**2))*U0[i]+2*(dt/dx[m]**2)*U0[i+1] + \
                        (2*(dt/dx[m])-dt*(dx[m]*i)*sdot/s[m])*numpy.exp(m*dt)
            else:
                U0[i]= U0[i]+ ((dt*(dx[m]*i)*sdot)/(2*dx[m]*s[m]))*(U0[i+1]-U0[i-1])\
                             +(dt/(dx[m]**2))*(U0[i+1]-2*U0[i]+U0[i-1])

            U0[N-1]=0

     s[m+1]= s[m] - (dt/(2*dx[m]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])
     sdot=(s[m+1]-s[m])/dt
     dx[m+1]=(s[m+1]/N)

XX = numpy.linspace(0, 1, N)
Max_tempVGM=max(U0)
pyplot.plot(XX,U0, color = '#654322', ls = '--', lw =4, label = 'VGM')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)
pyplot.legend();
print('Time Step dt is:', dt)
print('Initial spatial step, dx[0]^2 is:', dx[0]**2 )
print('Max error (@x=0) is:',abs(Max_tempVGM-Max_tempA)*(100/Max_tempA),'Percent')
```

As you can see, our stability criteria is still met, since $\Delta t \leq (\Delta x_{0})^{2}$. Infact, with a 40% larger N we see that we have less error as well! But what if we set N=15?
(The cell below has the nt set to 500 since we didn't want it to slow you down the first time you ran this program. Notice the N is now 15, go ahead and run this below cellwith nt=500000 and see what happens THIS time...)

```python
dt=2.0e-6
nt=500
#nt=500000

N = 15

Uo=0
U0 = numpy.ones(N)*(Uo)
s0=0.02
s=numpy.ones(nt)*(s0)
dx=numpy.ones(nt)*(s[1]/N)
sdot=(s[1]-s[0])/dt

s[1]= s[0] - (dt/(2*dx[0]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])
dx[0]=(s[1]/N)

for m in range(0, nt-1):
     for i in range(0, N-1):
            if(i==0):U0[i]= (1-2*(dt/dx[m]**2))*U0[i]+2*(dt/dx[m]**2)*U0[i+1]\
                            +(2*(dt/dx[m])-dt*(dx[m]*i)*sdot/s[m])*numpy.exp(m*dt)
            else: U0[i]= U0[i]+((dt*(dx[m]*i)*sdot)/(2*dx[m]*s[m]))*(U0[i+1]-U0[i-1])\
                              +(dt/(dx[m]**2))*(U0[i+1]-2*U0[i]+U0[i-1])

            U0[N-1]=0

     s[m+1]= s[m] - (dt/(2*dx[m]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])
     sdot=(s[m+1]-s[m])/dt
     dx[m+1]=(s[m+1]/N)

XX = numpy.linspace(0, 1, N)
Max_tempVGM=max(U0)
pyplot.plot(XX,U0, color = '#654322', ls = '--', lw =4, label = 'VGM')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)
pyplot.legend();
print('Time Step dt is:', dt)
print('Initial spatial step, dx[0]^2 is:', dx[0]**2 )
print('Max error (@x=0) is:',abs(Max_tempVGM-Max_tempA)*(100/Max_tempA),'Percent')
```

Wow! That blew up! Just as we expect it would now that with N=15 our time step is larger than our initial spatial step!

#### Final Thoughts

And now we know, that the Stefan Problem, a boundary value PDE with  a time-dependant moving boundary, CAN be solved. We have just demonstrated the solution for a 1D, Dimensionless Stefan Problem with specifically chosen (time-dependant) input heat flux in order to give us a simplified exact solution to compare with. You have also seen implementation of the Variable Grid Method, one of many ways inwhich one can numerical simulate the Stefan problem. But we have also seen the limitations, namely in small time-steps ($dt$), smaller numerical domain ($N\cdot dx_{f}$), and large number of iterations ($nt$) for just a small amount of analytical time (t=$nt \cdot dt$ seconds). Perhaps when it comes time for you to model the melting of the Polar Ice caps, you'll choose a more "expedient" method? Just make sure you get the answer BEFORE the caps melt...)

EXERCISE #1: Can you implement the Variable Grid Method discretization governing equations to give us a solution faser? Can it handle millions of time iterations without putting us to sleep? (Hint: While governing equations (B) and (C) might be easy to re-write in array form, LHS BC and Moving Interface equations (A), (D), and (E) will not be!)

EXERCISE #2: Can you write a stability condition statement that will maximize our time-step (dt) for a given N and still keep it constant?

# References

1. Kutluay S., The numerical solution of one-phase classical Stefan problem, Journal of Computational and Applied Mathematics 81 (1997) 135-144

2. Javierre, E., A Comparison of Numeical Models for one-dimensional Stefan problems, Journal of Compuational and Applied Mathematics 192 (2006) 445-459

3. Vuik, C., "Some historical notes about the Stefan problem". Nieuw Archief voor Wiskunde, 4e serie 11 (2): 157-167 (1993)

4. Crowley, A. B., Numerical Solution of Stefan Problems, Brunel University, Department of Mathematics, TR/69 December 1976

5. Caldwell, J., Numerical Solution of Stefan Problem By Variable Space Grid Method and Boundary Immodbilisation Method, Jour. of Mathematical Sciences, Vol. 13 No.1 (2002) 67-79.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file ='numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

</style>

Chris Tiu Project/Chris Tiu - Stefan Problem.ipynb


```python
import math
import numpy as np
import matplotlib.pyplot as plt
import argparse
import logging
import sys
import pandas as pd

from scipy import integrate
from src.precession.config import Config
from src.precession.planet import Planet

from src.precession.precession import solve
from sklearn.ensemble import RandomForestRegressor as rf
from scipy import fft
from sympy import *
from einsteinpy import constant
from sympy.physics.vector import dynamicsymbols
from sympy.solvers.ode.ode import *
from sympy.solvers.ode.single import *
from sympy.solvers.solvers import *
from sympy.simplify.simplify import *
from sympy.assumptions import *
from sympy.physics.mechanics import *

init_printing()

```

```python
m_M, m_P, m_S, G = symbols("m_M m_P m_S G", nonzero=True)
t = symbols("t")
r_M, r_P, theta_M, theta_P = dynamicsymbols("r_M r_P theta_M theta_P")
r_M_dot, r_P_dot, theta_M_dot, theta_P_dot = dynamicsymbols("r_M r_P theta_M theta_P", level=1)
V_M, Omega_M = dynamicsymbols("V_M \\Omega_M")
V_P, Omega_P = dynamicsymbols("V_P \\Omega_P")

substitutions = [ (r_M_dot, V_M), (r_P_dot, V_P), (theta_M_dot, Omega_M), (theta_P_dot, Omega_P)]

L = Symbol("\mathscr{L}")

def substitute_lm(eq, substitutions, index=None):
    for idx, sub in enumerate(substitutions):
        if index is not None and index == idx:
            continue
        eq = eq.subs(sub[0], sub[1])
    return eq
```

## Three body central force problem Sun-Mercury-Jupiter system

*Note* Assume the Sun is stationary at origin

Terms for Mercury, Jupiter and the Sun have subscripts M, K and S respectively

```python
r_PM = sqrt(r_P**2 + r_M**2 - 2*r_P*r_M*cos(theta_M - theta_P))
Lagrangian = G*m_S*m_M/r_M +  G*m_S*m_P/r_P + G*m_P*m_M/r_PM + m_M*r_M_dot**2/2 + m_M*r_M**2*theta_M_dot**2/2 + m_P*r_P_dot**2/2 + m_P*r_P**2*theta_P_dot**2/2
Eq(L, substitute_lm(Lagrangian, substitutions))
```

Compute the Euler Lagrange equations. Each element of the resultant array is the rhs of time gradient of the generalized coordinates.

```python
LM = LagrangesMethod(Lagrangian, [r_M, r_P, theta_M, theta_P])
LM.form_lagranges_equations()
LM.rhs()
```

$\displaystyle \left[\begin{matrix}\frac{d}{d t} \operatorname{r_{M}}{\left(t \right)}\\\frac{d}{d t} \operatorname{r_{P}}{\left(t \right)}\\\frac{d}{d t} \theta_{M}{\left(t \right)}\\\frac{d}{d t} \theta_{P}{\left(t \right)}\\\frac{\frac{G m_{M} m_{P} \left(- \operatorname{r_{M}}{\left(t \right)} + \operatorname{r_{P}}{\left(t \right)} \cos{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)}\right)}{\left(\operatorname{r_{M}}^{2}{\left(t \right)} - 2 \operatorname{r_{M}}{\left(t \right)} \operatorname{r_{P}}{\left(t \right)} \cos{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)} + \operatorname{r_{P}}^{2}{\left(t \right)}\right)^{\frac{3}{2}}} - \frac{G m_{M} m_{S}}{\operatorname{r_{M}}^{2}{\left(t \right)}} + m_{M} \operatorname{r_{M}}{\left(t \right)} \left(\frac{d}{d t} \theta_{M}{\left(t \right)}\right)^{2}}{m_{M}}\\\frac{\frac{G m_{M} m_{P} \left(\operatorname{r_{M}}{\left(t \right)} \cos{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)} - \operatorname{r_{P}}{\left(t \right)}\right)}{\left(\operatorname{r_{M}}^{2}{\left(t \right)} - 2 \operatorname{r_{M}}{\left(t \right)} \operatorname{r_{P}}{\left(t \right)} \cos{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)} + \operatorname{r_{P}}^{2}{\left(t \right)}\right)^{\frac{3}{2}}} - \frac{G m_{P} m_{S}}{\operatorname{r_{P}}^{2}{\left(t \right)}} + m_{P} \operatorname{r_{P}}{\left(t \right)} \left(\frac{d}{d t} \theta_{P}{\left(t \right)}\right)^{2}}{m_{P}}\\\frac{- \frac{G m_{M} m_{P} \operatorname{r_{M}}{\left(t \right)} \operatorname{r_{P}}{\left(t \right)} \sin{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)}}{\left(\operatorname{r_{M}}^{2}{\left(t \right)} - 2 \operatorname{r_{M}}{\left(t \right)} \operatorname{r_{P}}{\left(t \right)} \cos{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)} + \operatorname{r_{P}}^{2}{\left(t \right)}\right)^{\frac{3}{2}}} - 2 m_{M} \operatorname{r_{M}}{\left(t \right)} \frac{d}{d t} \operatorname{r_{M}}{\left(t \right)} \frac{d}{d t} \theta_{M}{\left(t \right)}}{m_{M} \operatorname{r_{M}}^{2}{\left(t \right)}}\\\frac{\frac{G m_{M} m_{P} \operatorname{r_{M}}{\left(t \right)} \operatorname{r_{P}}{\left(t \right)} \sin{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)}}{\left(\operatorname{r_{M}}^{2}{\left(t \right)} - 2 \operatorname{r_{M}}{\left(t \right)} \operatorname{r_{P}}{\left(t \right)} \cos{\left(\theta_{M}{\left(t \right)} - \theta_{P}{\left(t \right)} \right)} + \operatorname{r_{P}}^{2}{\left(t \right)}\right)^{\frac{3}{2}}} - 2 m_{P} \operatorname{r_{P}}{\left(t \right)} \frac{d}{d t} \operatorname{r_{P}}{\left(t \right)} \frac{d}{d t} \theta_{P}{\left(t \right)}}{m_{P} \operatorname{r_{P}}^{2}{\left(t \right)}}\end{matrix}\right]$

Substituting generalized coordinates

```python
Eq(V_M, substitute_lm(LM.rhs()[0], substitutions, 0))
```

```python
Eq(V_P, substitute_lm(LM.rhs()[1], substitutions, 1))
```

```python
Eq(Omega_P, substitute_lm(LM.rhs()[2], substitutions, 2))
```

```python
Eq(Omega_M, substitute_lm(LM.rhs()[3], substitutions, 3))
```

```python
Eq(V_P.diff(), substitute_lm(LM.rhs()[4], substitutions).simplify())
```

```python
Eq(V_M.diff(), substitute_lm(LM.rhs()[5], substitutions).simplify())
```

```python
Eq(Omega_P.diff(), substitute_lm(LM.rhs()[6], substitutions).simplify())
```

### Load planet data

```python
config = Config.load("src/precession/config.yml")
planet = Planet.load(config, "src/precession/jupiter.yml")
mercury = Planet.load(config, "src/precession/mercury.yml")
```

### Use scipy solver to solve the 8 equations

```python
df = solve(config, planet, mercury)
```

### Finding the period of precession

For precession to occur, the period between r_m maxima will differ from the orbital period. Thus a fourier transform of the r_m should give us this period of r_m maxima. The difference between this period and the period of the orbit should yield the angle of precession.

For this to be computationally possible we will have to compute the DFT of r_m. But we the r_m computed by the solving the 8 equations is not uniformly sampled. We will use sklearn to fit a curve over the values of r_m and then compute r_m at regular samples of time.

Fit a curve

```python
model = rf()
model.fit(df["time"].to_numpy().reshape(-1, 1), df["r_m"].to_numpy())
model_theta = rf()
model_theta.fit(df["time"].to_numpy().reshape(-1, 1), df["theta_m"].to_numpy())
```

    RandomForestRegressor()

### Compute r_m at regular samples of time

```python
t_start = df["time"].iloc[0]
t_end = df["time"].iloc[-1]

t_sample = np.arange(t_start, t_end, (t_end - t_start)/100000)
r_m_fit = model.predict(np.array(t_sample).reshape(-1, 1))
theta_m_fit = model_theta.predict(np.array(t_sample).reshape(-1, 1))
```

### Sanity check r_m_fit and theta_m_fit

```python
fig, ax = plt.subplots(subplot_kw={'projection': 'polar'})
ax.plot(df["theta_m"], df["r_m"])
plt.show()
```

### Sanity check r_j

```python
fig, ax = plt.subplots(subplot_kw={'projection': 'polar'})
ax.plot(df["theta_p"], df["r_p"])
plt.show()
```

### Compute fft

```python
frequencies = np.array(fft.rfft(r_m_fit))
```

### Sanity check the frequencies

```python
plt.plot(frequencies.real[1:250])
plt.plot(frequencies.imag[1:250])
plt.show()
```

### Compute the precession angle per year

```python
coeff_max_r = np.argmax(frequencies.real[1:])
total_time = t_sample[-1] - t_sample[0]
# Time period between two maxima of r_m
r_m_period = total_time / (coeff_max_r)
# Difference in angle for an r_m time period
# Pick a time when r_m is max.
max_r_m_index = np.argmax(r_m_fit)

start_time = t_sample[max_r_m_index]
end_time = start_time + r_m_period

[start_theta, end_theta] = model_theta.predict([[start_time], [end_time]])
[r_m_1, r_m_2] = model.predict([[start_time], [end_time]])
print(f"{r_m_1} {r_m_2}")
print(f"{r_m_period} {end_theta - start_theta - 2 * np.pi}")
```

    0.46672126121078883 0.46670216731357206
    0.24096144578313253 0.01946519141447567

```python
x = sympy.symbols('x')
sympy.simplify(sympy.cos(2*x))
```

$\displaystyle \cos{\left(2 x \right)}$