2019年度 線形代数　I　講座資料

資料番号　No.5-1

## [5-1] 　連立１次方程式

##  掃き出し法

連立方程式を変更を繰り返しながら，解を求める

例
$
\begin{eqnarray}  \left\{    \begin{array}{l}      x + 2 y = 1 \\      2x + 7y = -1    \end{array}  \right.\end{eqnarray}
$
を解く

変形しながら

$ \Downarrow $

$
\begin{eqnarray}\left(  \begin{array}{cc:c}    1 & 0 & \diamondsuit \\    0 & 1 & \spadesuit \\  \end{array}\right)\end{eqnarray}
$
に変形して，$x=\diamondsuit $  , $y=\spadesuit $ として求める

以下に変形を示す．

$
\begin{eqnarray}  \left\{    \begin{array}{l}      x + 2 y = 1 \\      2x + 7y = -1    \end{array}  \right.\end{eqnarray}
$　　　　$
\begin{eqnarray}\left(  \begin{array}{cc:c}    1 & 2 & 1 \\    2 & 7 & -1 \\  \end{array}\right)\end{eqnarray}
$

2式に１式の(ー２)倍を加える

$
\begin{eqnarray}  \left\{    \begin{array}{l}      x + 2 y = 1 \\     3y = -3    \end{array}  \right.\end{eqnarray}
$　　　　$
\begin{eqnarray}\left(  \begin{array}{cc:c}    1 & 2 & 1 \\    0 & 3 & -3 \\  \end{array}\right)\end{eqnarray}
$

2式を$\frac{ 1 }{ 3 }$倍する

$
\begin{eqnarray}  \left\{    \begin{array}{l}      x + 2 y = 1 \\     y = -1    \end{array}  \right.\end{eqnarray}
$　　　　$
\begin{eqnarray}\left(  \begin{array}{cc:c}    1 & 2 & 1 \\    0 & 1 & -1 \\  \end{array}\right)\end{eqnarray}
$

1式に2式の(ー２)倍を加える．

$
\begin{eqnarray}  \left\{    \begin{array}{l}      x = 3 \\     y = -1    \end{array}  \right.\end{eqnarray}
$　　　　$
\begin{eqnarray}\left(  \begin{array}{cc:c}    1 & 0 & 3 \\    0 & 1 & -1 \\  \end{array}\right)\end{eqnarray}
$

### 拡大係数行列での行基本変形

連立方程式は以下のようにあらわされると

$
  \left\{
  \begin{array}{l}
      a_{11}x_1+a_{12}x_2+a_{13}x_3 + \cdots + a_{1n}x_n & = b_1\\
      a_{21}x_1+a_{22}x_2+a_{23}x_3 + \cdots + a_{2n}x_n & = b_2\\
        \vdots                                           &  \vdots \\
      a_{n1}x_1+a_{n2}x_2+a_{n3}x_3 + \cdots + a_{nn}x_n & = b_n\\
  \end{array}
  \right.
$

行列を用いて
$A$ $\boldsymbol{x}$=$\boldsymbol{b}$
と表され，係数拡大行列は

$
\begin{eqnarray}
(Ab) = \left(
  \begin{array}{cccc}
    a_{ 11 } & a_{ 12 } & \ldots & a_{ 1n }& \color{red}{b_1} \\
    a_{ 21 } & a_{ 22 } & \ldots & a_{ 2n }&\color{red}{b_2}\\
    \vdots & \vdots & \ddots & \vdots \\
    a_{ n1 } & a_{ n2 } & \ldots & a_{ nn }&\color{red}{b_n}\\
  \end{array}
\right)
\end{eqnarray}
$

となる，これに以下の行基本変形を繰り返す

## 【行基本変形】

〇　i行にｊ行のｋ倍を加える

〇　i行をｋ倍する

〇　i行とｊ行を入れ替える

### 【変形のコツ】

１）左から変形すること（対角線上のみに数を残しながら，他を０にする）

２）できるだけ大きい数を小さくする（大きい数は１へ，１は０へ）

３）分数はできるだけしない（するとのは最後にするだけ）

行基本変形を繰り返して，拡大係数行列が

$
\begin{eqnarray}
 \left(
  \begin{array}{cccc}
    1 & 0 & \ldots & 0& \color{red}{s_1} \\
    0 & 1 & \ldots & 0&\color{red}{s_2}\\
    \vdots & \vdots & \ddots & \vdots \\
        0 & 0 & \ldots & 1&\boxed{\color{red}{s_n}}\\
  \end{array}
\right)
\end{eqnarray}
$

となれば，これが連立1次方程式の解，

$x_1=s_1 , x_2=s_2 ,\ldots, x_n= s_n  $

対応している．

これが掃き出し法と呼ばれる．

解は，解あり，解なし，解が無数存在する，の場合がある

### 解なしのときの例は

$
\begin{eqnarray}
 \left(
  \begin{array}{cccc}
    1 & 0 & \ldots & 0& \color{red}{s_1} \\
    0 & 1 & \ldots & 0&\color{red}{s_2}\\
    \vdots & \vdots & \ddots & \vdots \\
    \boxed{\color{blue}{0 }}&\boxed{\color{blue}{0 }}& \ldots &\boxed{\color{blue}{0 }}&\boxed{\color{red}{s_n}}\\
  \end{array}
\right)
\end{eqnarray}
$

で，$\color{red}{s_n \neq 0}$ のときなどがある，

### 解が無数あるのときの例は

$
\begin{eqnarray}
 \left(
  \begin{array}{cccc}
    1 & 0 & \ldots & 0& \color{red}{s_1} \\
    0 & 1 & \ldots & 0&\color{red}{s_2}\\
    \vdots & \vdots & \ddots & \vdots \\
        \boxed{\color{blue}{0 }}&\boxed{\color{blue}{0 }}& \ldots &\boxed{\color{blue}{0 }}&\boxed{\color{red}{0}}\\
  \end{array}
\right)
\end{eqnarray}
$

で，$\color{red}{s_n = 0}$ のときなどがある，

問２．１８（２）

$
\begin{eqnarray}
\left\{    \begin{array}{r}
3x - 2y +5z= 11 \\
x+3y-7z=1\\
-4x + y +6z= 4
\end{array}
\right.\end{eqnarray}
$

$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
3 & -2 &5 & 11 \\
1 & 3 & -7 & 1\\
-4& 1& 6 & 4
\end{array}\right)\end{eqnarray}
$

1行と2行を交換した
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 3 & -7 & 1\\
3 & -2 &5 & 11 \\
-4& 1& 6 & 4
\end{array}\right)\end{eqnarray}
$

3行に2行の（1）倍を加える
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 3 & -7 & 1\\
3 & -2 &5 & 11 \\
-1& -1& 11 & 15
\end{array}\right)\end{eqnarray}
$

3行に1行の（1）倍を加える，
2行に1行の（－３）倍を加える
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 3 & -7 & 1\\
0& -11 &26 & 8 \\
0& 2& 4 & 16
\end{array}\right)\end{eqnarray}
$

3行を(1/2)倍する
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 3 & -7 & 1\\
0& -11 &26 & 8 \\
0& 1& 2 & 8
\end{array}\right)\end{eqnarray}
$

3行と2行を交換する
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 3 & -7 & 1\\
0& 1& 2 & 8 \\
0& -11 &26 & 8
\end{array}\right)\end{eqnarray}
$

3行に2行の（１１）倍を加える，1行に2行の（－３）倍を加える
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 0 & -13 & -23\\
0& 1& 2 & 8 \\
0& 0 &48 & 96
\end{array}\right)\end{eqnarray}
$

3行を(1/48)倍する
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 0 & -13 & -23\\
0& 1& 2 & 8 \\
0& 0 & 1 & 2
\end{array}\right)\end{eqnarray}
$

1行に3行の（１３）倍を加える，2行に3行の（－２）倍を加える
$
\dashrightarrow
$
$
\begin{eqnarray}\left(  \begin{array}{ccc:c}
1 & 0 & 0 & 3\\
0& 1& 0 & 4 \\
0& 0 & 1 & 2
\end{array}\right)\end{eqnarray}
$

したがって
$
x=3,y=4,z=2
$
となる

```python
# 初めに1回だけ実行させてください．
from sympy import *
init_printing()
# init_session()
```

```python
# 拡大係数行列を作り,連立方程式を解く方法
# 解があるときの
#例題 2.2

x, y,z = symbols('x, y,z')
A = Matrix([[1, 3, 5], [3, 7,11],[-1,4,7]])
b = Matrix([6,16,-3])
print("A=")
display(A)
print("b=")
display(b)
print("(A b)=")
display((A,b))

linsolve((A, b), x, y, z)

```

```python
# 拡大係数行列を作り,連立方程式を解く方法
# 解がないときの例

x, y,z = symbols('x, y,z')
A = Matrix([[1, 1,1], [1, 2,3],[1,3,5]])
b = Matrix([2,5,7])

print("A=")
display(A)
print("b=")
display(b)
print("(A b)=")
display((A,b))

linsolve((A, b), x, y, z)
```

```python
# 拡大係数行列を作り,連立方程式を解く方法
# 解が無数の存在するとき

x, y,z = symbols('x, y,z')
A = Matrix([[1, 1,1], [1, 2,3],[1,3,5]])
b = Matrix([2,5,8])
print("A=")
display(A)
print("b=")
display(b)
print("(A b)=")
display((A,b))
linsolve((A, b), x, y, z)
```

```python
# 例題　2.7　の別解　linsolveを使う時
# 拡大係数行列を作り,連立方程式を解く方法
# 解が無数の存在するとき

x, y,z = symbols('x, y,z')
A = Matrix([[1, -2,1], [1, 2,-2]])
b = Matrix([-3,5])

print("A=")
display(A)
print("b=")
display(b)
print("(A b)=")
display((A,b))

linsolve((A, b), x, y, z)
```

```python
# 例題　2.7　の別解　solveを使う時
x,y ,z= symbols('x,y,z')
equation1 = x - 2*y +z+ 3
equation2 = x + 2*y -2*z-5

print(equation1,"=0")
print(equation2,"=0")

solve([equation1, equation2])
```

```python
# 例題　２．２　解が１つある時の図

from sympy import *
init_printing()

%matplotlib notebook
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import cm

x = np.arange(-50, 50, 5)
y = np.arange(-50, 50, 5)
X, Y = np.meshgrid(x, y)

# 3つの平面の式より
Z1 = (-X-3*Y+6)/5
Z2 = (-3*X -7*Y+16)/11
Z3 = (X-4*Y-3)/7

fig = plt.figure()
ax = Axes3D(fig)

ax.plot_wireframe(X, Y, Z1, rstride=1, cstride=1,  color='blue')
ax.plot_wireframe(X, Y, Z2, rstride=1, cstride=1, color='red')
ax.plot_wireframe(X, Y, Z3, rstride=1, cstride=1, color='green')
plt.show()

```

    <IPython.core.display.Javascript object>

```python
# 例題　２．４　　解が存在しないときの図

from sympy import *
init_printing()

%matplotlib notebook
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import cm

x = np.arange(0, 1, 0.05)
y = np.arange(0, 1, 0.05)
X, Y = np.meshgrid(x, y)

# 3つの平面の式より
Z1 = -X-Y+2
Z2 = (-X -2*Y+5)/3
Z3 = (-X-3*Y+7)/5

fig = plt.figure()
ax = Axes3D(fig)

ax.plot_wireframe(X, Y, Z1, rstride=1, cstride=1,  color='blue')
ax.plot_wireframe(X, Y, Z2, rstride=1, cstride=1, color='red')
ax.plot_wireframe(X, Y, Z3, rstride=1, cstride=1, color='black')
plt.show()

```

    <IPython.core.display.Javascript object>

```python
# 例題　２．５　　解が無数に存在するときの図

from sympy import *
init_printing()

%matplotlib notebook
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import cm

x = np.arange(-1, 1, 0.1)
y = np.arange(-1, 1, 0.1)
X, Y = np.meshgrid(x, y)

# 3つの平面の式より
Z1 = -X-Y+2
Z2 = (-X -2*Y+5)/3
Z3 = (-X-3*Y+8)/5

fig = plt.figure()
ax = Axes3D(fig)

ax.plot_wireframe(X, Y, Z1, rstride=1, cstride=1,  color='blue')
ax.plot_wireframe(X, Y, Z2, rstride=1, cstride=1, color='red')
ax.plot_wireframe(X, Y, Z3, rstride=1, cstride=1, color='black')
plt.show()

```

    <IPython.core.display.Javascript object>

```python
# 拡大係数行列を作り,連立方程式を解く方法
# 解があるときの
#例題 2.2

x, y,z = symbols('x, y,z')
A = Matrix([[2, 4, 6], [3, 5,3],[5,3,8]])
b = Matrix([5,6,8])
print("A=")
display(A)
print("b=")
display(b)
print("(A b)=")
display((A,b))

linsolve((A, b), x, y, z)

```

```python

```


# Bayesian Statistics for Physicists

## Overview

A brief, hands-on introduction to the basics of Bayesian statistics in a manner adapted to the general intuition and experience of physicists.  We use a Jupyter notebook with Python (scipy, numpy, mathplotlib) to allow for active visualization of examples, hands-on follow-ups, and readily extended content. You can find the notebook and associated files at
https://github.com/furnstahl/Bayes_for_physicists.

This is not an exhaustive guide to Bayesian statistics, but a selected sampling of topics that come up regularly in physics applications, with links to more information.
Most of the examples were adapted from code found on the web.
Please contribute suggestions, comments, links, code, ...

Last revised: 08-Oct-2018 by Dick Furnstahl [furnstahl.1@osu.edu].

## Contents

Overview

Python set up

Bayesian basics
        [Rules]
        [Networks]
        [Model discrepancy]

Choosing priors
        [Maximum entropy]
        [Conjugate priors]

Bayesian updating examples

Sampling
        [Multivariate gaussian]
        [MCMC]

Model selection: Bayes ratio and evidence

Gaussian processes

Appendices:
         [References]
         [Vocabulary]
         [Notation]

## Python/Jupyter set up

We recommend installing the standard Anaconda Python3 package (from https://www.anaconda.com/download), which is available for Windows, Mac OS X, and Linux.  Anaconda will provide scipy, numpy, matplotlib, and jupyter notebooks (and more!).

You can start this notebook from the Anaconda Navigator or from the command line (go to the directory with this notebook and type: jupyter notebook).

It is convenient to use the Jupyter notebook extensions "Code Folding" and "Collapsible Headings", which can be turned on using the jupyter_nbextensions_configurator extension after installing via

conda install -c conda-forge jupyter_contrib_nbextensions

 (see https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/ for a complete list and full instructions).

Other packages you will need to install for this notebook:

pymc3 [conda install -c conda-forge pymc3]

emcee [conda install -c astropy emcee]

corner [conda install -c astropy corner]

```python
# set up for plots in this notebook using matplotlib (there are other plotting choices)
%matplotlib inline
```

```python
import numpy as np

import scipy.stats as stats
from scipy.stats import norm, uniform

import matplotlib.pyplot as plt
#plt.style.use('seaborn') # pretty matplotlib plots

import corner
import pymc3 as pm

```

    /Users/furnstah/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

```python
# make font adjustments
#plt.rcParams['font.size'] = 12
#plt.rcParams['legend.fontsize'] = 'medium'
#plt.rcParams['figure.titlesize'] = 'medium'
plt.rcdefaults()  # revert to defaults for now
```

```python
%%html
<!-- Use html cell magic to add css styling -->
<style>
  em {
      color: red;
  }
  dd {
      margin-left: 15px;
  }
  .red{color: red}
  .blue{color: blue}
</style>
```

```python
#%%javascript
#IPython.OutputArea.auto_scroll_threshold = 9999;
```

## Bayesian basics

### Why should physicists use Bayesian statistics?

cf. Why isn't every physicist a Bayesian? from 1995.

Includes conventional physics statistics (e.g., for parameter estimation)

Calculate what you really want, e.g., probability for some parameter vs. frequentist confidence interval

Assumptions are made explicit (in the form of priors)

Allows us to account for "nuisance parameters"

Well suited for theory errors, which are generally systematic

Clear prescription for combining different types of errors

Model selection: compare different theoretical models (or combine!)

Model checking: we can test if our UQ model works and study sensitivities

Statistics as diagnostic and discovery tools for physics

 **[add your own favorite reasons]**

### Everything is a pdf (probability density function)

Physicists are used to multidimensional normalized pdfs as wave functions squared, e.g. probability density for particle 1 at $x_1$ and particle 2 at $x_2$:

 $$
 |\Psi(x_1, x_2)|^2 \Longrightarrow p(x_1,x_2) \equiv p(\textbf{x})
   \quad \mbox{with}\quad \textbf{x}
 \equiv \{x_1,x_2\}
 $$

(Other notation for generic pdfs: $p(\textbf{x}) = P(\textbf{x}) = \textrm{pr}(\textbf{x}) = \textrm{prob}(\textbf{x}) = \ldots$ )

$p(x_1,x_2)$ is the joint probability density of $x_1$ and $x_2$.

What is the probability to find particle 1 at $x_1$ and particle 2 anywhere?  $\int\! |\Psi(x_1,x_2)|^2 dx_2$

The marginal probability density of $x_1$ is:
$\color{blue}{p(x_1) = \int\! p(x_1,x_2)\,dx_2}$.

"Marginalizing" = "integrating out" (eliminates "nuisance parameters" from posterior).

Just as with "Lagrangian", we will not always be careful about saying probability vs. probability density.

In Bayesian statistics there are pdfs (or pmfs if discrete) for data, experimental and theoretical uncertainties, fit parameters, hyperparameters (what?), events (Will it rain tomorrow?), etc.  Even if definite $x_0$, we can use $p(x) = \delta(x-x_0)$.

### Visualization of pdfs

#### Matplotlib plotting definitions

```python
def dist_stuff(dist):
    """
    Find the median, mean, and 68%/95% credible intervals
     for the given 1-d distribution (from stats).
    """
    median = [dist.median(), dist.pdf(dist.median())]
    mean = [dist.mean(), dist.pdf(dist.mean())]
    cred68 = dist.interval(0.68)
    cred95 = dist.interval(0.95)
    return median, mean, cred68, cred95

def dist_mode(dist, x):
    """
    Find the mode (maximum) of the 1-d distribution.
    """
    x_max_index = dist.pdf(x).argmax()
    mode = [x[x_max_index], dist.pdf(x[x_max_index])]
    return mode

def dist_plot(dist_label, x_dist, dist, plot_num):
    """
    Plot the distribution, indicating median, mean, mode
    and 68%/95% probability intervals.
    """
    colors = ('blue', 'blue', 'blue')
    median, mean, cred68, cred95 = dist_stuff(dist)
    mode = dist_mode(dist, x_dist)

    plt.subplot(1,3,plot_num)
    plt.plot(x_dist,dist.pdf(x_dist), label=dist_label, color=colors[plot_num-1])
    text_x = 0.2*(x_dist[-1]-x_dist[0])
    text_x_mid = (x_dist[-1]+x_dist[0])/2
    text_y = mode[1]*1.15
    plt.annotate('median', xy=median, xytext=(text_x_mid+text_x, text_y),
            arrowprops=dict(facecolor='black', shrink=0.05))
    plt.annotate('mode', xy=mode, xytext=(text_x_mid-text_x, text_y),
            arrowprops=dict(facecolor='red', shrink=0.05))
    plt.annotate('mean', xy=mean, xytext=(text_x_mid, text_y),
            arrowprops=dict(facecolor='blue', shrink=0.05))
    plt.xlabel('x')
    plt.ylabel('p(x)')
    plt.fill_between(x_dist, 0, dist.pdf(x_dist),
                 where=((x_dist > cred68[0]) & (x_dist < cred68[1])),
                 facecolor='blue', alpha=0.2)
    plt.fill_between(x_dist, 0, dist.pdf(x_dist),
                 where=((x_dist > cred95[0]) & (x_dist < cred95[1])),
                 facecolor='blue', alpha=0.1)
    plt.legend();

```

#### Some standard pdfs: normal and beta distributions

```python
%matplotlib inline

# Make some standard plots
plt.figure(figsize=(15,5))

# Standard normal distribution
x_norm = np.linspace(-4, 4, 500)
mu = 0     # mean
sigma = 1.0  # standard deviation
norm_dist = stats.norm(mu, sigma) # the normal distribution
norm_label='normal pdf' + '\n' + r'$\mu=${:1.1f},'.format(mu) \
             + '\n' + r'$\sigma=${:1.1f}'.format(sigma)
dist_plot(norm_label, x_norm, norm_dist, 1)

# beta distribution
x_beta = np.linspace(-0.1, 1.1, 500)
a1 = 1
b1 = 5
beta_dist = stats.beta(a1, b1)
beta1_label='beta pdf' + '\n' + r'$a=${:1.1f}'.format(a1) \
              + ',\n$b=${:1.1f}'.format(b1)
dist_plot(beta1_label, x_beta, beta_dist, 2)

# another beta distribution
#x_beta = np.linspace(-0.1, 1.1, 500)
a2 = 10
b2 = 10
beta2_dist = stats.beta(a2, b2)
beta2_label='beta pdf' + '\n' + r'$a=${:1.1f}'.format(a2) \
              + ',\n$b=${:1.1f}'.format(b2)
dist_plot(beta2_label, x_beta, beta2_dist, 3)

```

The 68%/98% probability regions are shown in dark/light shading.  When applied to posteriors, these are known as credible intervals or DoBs (degree of belief intervals) or Bayesian confidence intervals. The horizontal extent on the $x$-axis translates into the vertical extent of the error bar or error band for $x$.

#### More standard pdfs:  Student t

```python
%matplotlib inline

# Make some plots of the Student t distribution
plt.figure(figsize=(15,5))

x_t = np.linspace(-5, 5, 500)

nu1 = 1
t1_dist = stats.t(nu1) # the Student t distribution
t1_label='t pdf' + '\n' + r'$\nu=${:1.1f}'.format(nu1)
dist_plot(t1_label, x_t, t1_dist, 1)

nu2 = 3
t2_dist = stats.t(nu2) # the Student t distribution
t2_label='t pdf' + '\n' + r'$\nu=${:1.1f}'.format(nu2)
dist_plot(t2_label, x_t, t2_dist, 2)

nu3 = 100
t3_dist = stats.t(nu3) # the Student t distribution
t3_label='t pdf' + '\n' + r'$\nu=${:1.1f}'.format(nu3)
dist_plot(t3_label, x_t, t3_dist, 3)

```

Note the "heavy tails" in the t distribution as $\nu$ gets small.  As $\nu$ gets large, the distribution approaches a standard normal (Gaussian) distribution.

#### Projected posterior plots

Here we use the [corner package](https://corner.readthedocs.io/en/latest/api.html) to make some projected posterior plots.

```python
%matplotlib inline
# examples of corner plots
ndim, nsamples = 2, 100000
#np.random.seed(42)
# generate some fake data from a normal distribution
norm_samples = stats.norm.rvs(size=ndim * nsamples).reshape([nsamples, ndim])
#figure = corner.corner(norm_samples)
figure1 = corner.corner(norm_samples, labels=[r"$x$", r"$y$", r"$\log \alpha$", r"$\Gamma \, [\mathrm{parsec}]$"],
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True, title_kwargs={"fontsize": 12})
ax = figure1.get_axes()
figure1.set_size_inches(5,5)

ndim, nsamples = 2, 100000
#np.random.seed(42)
# generate some fake data from a beta distribution
a = 4
b = 20
beta_samples = stats.beta(a,b).rvs(size=ndim * nsamples).reshape([nsamples, ndim])
#figure = corner.corner(beta_samples)
figure2 = corner.corner(beta_samples, labels=[r"$x$", r"$y$", r"$\log \alpha$", r"$\Gamma \, [\mathrm{parsec}]$"],
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True, title_kwargs={"fontsize": 12})
figure2.set_size_inches(5,5)

```

```python
%matplotlib inline
# now more than one mode (all random)
ndim, nsamples = 4, 50000
np.random.seed(1234)
data1 = np.random.randn(ndim * 4 * nsamples // 5).reshape([4 * nsamples // 5, ndim])
mean = 4*np.random.rand(ndim)
data2 = (mean[None, :] + np.random.randn(ndim * nsamples // 5).reshape([nsamples // 5, ndim]))
samples = np.vstack([data1, data2])

#figure = corner.corner(samples)
figure = corner.corner(samples, labels=[r"$x$", r"$y$", r"$\log \alpha$", r"$\Gamma \, [\mathrm{parsec}]$"],
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True, title_kwargs={"fontsize": 12})

figure.set_size_inches(5,5)

```

### Sampling of 1d pdfs in Python

```python
%matplotlib inline

def plot_hist(name, x_dist, dist, num_samples, num_bins, plot_num):
    plt.subplot(1,3,plot_num)
    samples = dist.rvs(size=num_samples)
    count, bins, ignored = plt.hist(samples, num_bins, density=True,
                                     color='blue', alpha=0.7)
    plt.plot(x_dist,dist.pdf(x_dist), linewidth=2, color='r')
    title_string = name + '  samples = {:d}'.format(num_samples)
    plt.title(title_string)

mu, sigma = 0, 1.0 # mean and standard deviation
x_dist = np.linspace(-4, 4, 500)
name = r'normal $\mu=${:1.1f}, $\sigma=${:1.1f}'.format(mu,sigma)
plt.figure(figsize=(15,5))
num_bins = 50

num_samples = 100
norm_dist = stats.norm(mu, sigma)
plot_hist(name, x_dist, norm_dist, num_samples, num_bins, 1)

num_samples = 1000
norm_dist = stats.norm(mu, sigma)
plot_hist(name, x_dist, norm_dist, num_samples, num_bins, 2)

num_samples = 10000
norm_dist = stats.norm(mu, sigma)
plot_hist(name, x_dist, norm_dist, num_samples, num_bins, 3)

```

### Bayes' Rule: Interaction of prior and likelihood

$A$ and $B$ are generic propositions and $I$ is "information" (things we know).  $p(A \mid B)$ means the probability of $A$ given $B$ (or contingent or conditional on $B$).

A particular case is a vector of parameters $\textbf{a} = \{a_1, a_2, \cdots\}$ for a theoretical model and some data it describes. Here Bayes' Rule is being used for parameter estimation.

$$
  \newcommand{\avec}{\textbf{a}}
  p(A \mid B,I) =
        \frac{p(B \mid A,I)\, p(A \mid I)}{p(B \mid I)}
  \ \Longrightarrow\
   \overbrace{p(\avec \mid \textrm{data},I)}^{\textrm{posterior}} =
  \frac{\color{red}{\overbrace{p(\textrm{data} \mid \avec,I)}^{\textrm{likelihood}}} \times
   \color{blue}{\overbrace{p(\avec \mid I)}^{\textrm{prior}}}}
   {\color{darkgreen}{\underbrace{p(\textrm{data} \mid I)}_{\textrm{evidence}}}}
$$

Common notation in statistics: $\boldsymbol{\theta}$ for parameters, $\mathcal{L}$ or $L$ for the likelihood, $\pi(\boldsymbol{\theta})$ for the prior.

$$\overbrace{p(\avec \mid \textrm{data},I)}^{\textrm{posterior}} \propto \color{red}{\overbrace{p(\textrm{data} \mid \avec,I)}^{\textrm{likelihood}}} \times
   \color{blue}{\overbrace{p(\avec \mid I)}^{\textrm{prior}}}$$

Left: likelihood overwhelms prior.  Right: prior is returned (restricts domain)

Note: these are one-dimensional projections of multi-dimensional pdfs.

Here we don't need to calculate the evidence separately; just normalize the numerator.

### Bayesian rules of probability as principles of logic

Notation: $p(x \mid I)$ is the probability (or pdf) of $x$ being true
given information $I$

 Sum rule: If set $\{x_i\}$ is exhaustive and exclusive,
  $$ \sum_i p(x_i  \mid  I) = 1   \quad \longrightarrow \quad       \color{red}{\int\!dx\, p(x \mid I) = 1}
  $$

 cf. complete and orthonormal

 implies marginalization (cf. inserting complete set of states or integrating out variables)
  $$
   p(x \mid  I) = \sum_j p(x,y_j \mid I)
    \quad \longrightarrow \quad
   \color{red}{p(x \mid I) = \int\!dy\, p(x,y \mid I)}
  $$

 Product rule: expanding a joint probability of $x$ and $y$
        $$
         \color{red}{ p(x,y \mid I) = p(x \mid y,I)\,p(y \mid I)
              = p(y \mid x,I)\,p(x \mid I)}
        $$

 If $x$ and $y$ are mutually independent:  $p(x \mid y,I)
      = p(x \mid I)$, then
    $$
       p(x,y \mid I) \longrightarrow p(x \mid I)\,p(y \mid I)
    $$

 Rearranging the second equality yields  Bayes' Rule (or Theorem)
     $$
      \color{blue}{p(x  \mid y,I) = \frac{p(y \mid x,I)\,
       p(x \mid I)}{p(y \mid I)}}
     $$

See Cox for the proof.

### Bayesian model checking: one example

How can you evaluate whether your Bayesian predictions are working?
Cf. checking whether a least-squares fit to data with  Gaussian noise misses about 1/3 of your 1-$\sigma$ error bars.
More generally: are the residuals normally distributed?

Are your Bayesian credible intervals consistent with observed successes? Check with a calibration or empirical coverage or credible interval diagnostic plot.

To be discussed: other ways to do Bayesian model checking.

### Networks

A Bayesian network is a graphical model that makes conditional dependence explicit through the edges in a directed graph.  (More on this soon!)

### Model discrepancy

$\newcommand{\yexp}{\textbf{y}_{\rm exp}}$
$\newcommand{\yth}{\textbf{y}_{\rm th}}$
$\newcommand{\ytrue}{\textbf{y}_{\rm true}}$
The main goal of Bayesian parameter estimation is the calculation of a joint posterior pdf for the model parameters given a set of experimental data and any other information we have.  This task begins with a statistical model:
$$ \yexp = \yth + \Delta \yth + \Delta \yexp $$
where $\yexp$ are the experimental measurements of an observable $y$
and $\yth$ are the corresponding theoretical (model) calculations.
In terms of the true results $\ytrue$, we can decompose this as
$$ \yexp = \ytrue + \Delta\yexp\;; \qquad
   \ytrue = \yth + \Delta\yth  \;.
$$
The model for the experimental uncertainty $\Delta\yexp$ is usually stochastic noise
$$ \Delta\yexp \sim \mathcal{N}(0,\Sigma_{\rm exp}) $$
with zero mean and $\Sigma_{\rm exp}$ typically uncorrelated (so the covariance matrix is diagonal).  Systematic uncertainties can also be incorporated.

The "new" feature here is $\Delta\yth$, which is the model discrepancy or model defect.  Its role is to account statistically for the deficiencies of the model.  It could be a truncation error from an expansion or a model of the observed residuals in a set of training data.
By including $\Delta\yth$, we can suppress overfitting and deal with underfitting.

[Return to Contents]

## Choosing priors

A key feature of Bayesian statistics is the choice of the prior probability distribution.  We should incorporate all of the information we have, but no more. What are the options for choosing priors?  What are the subtleties?  Places to use an informative prior:

your dataset is small, but there is related information available from other systems;

your model is very flexible but you want to prevent overfitting (so use priors that prefer
      values close to zero);

you want to stay away from particular regions of parameter space: maybe strictly unphysical such as
      negative cross sections or acausal, or maybe
      counter to well-motivated assumptions such as parameters are of order unity;

the posterior from a previous experiment can become a prior ==> Bayesian updating;

other examples??

### Subtlety with uniform prior

An example from the 2016 workshop on [Bayesian Methods in Astronomy](https://github.com/jakevdp/BayesianAstronomy/blob/master/Index.ipynb) considers fitting a straight line $y = mx+b$ to noisy data. If you take the prior on the slope $m$ to be uniform (flat), thinking that this is non-informative, then look at this plot, which samples lines with uniformly distributed slopes:

```python
%matplotlib inline
xx = np.linspace(-1, 1)
for slope in np.linspace(0, 20, 100):
    plt.plot(xx, slope * xx, '-k', linewidth=1)
plt.axis([-1, 1, -1, 1], aspect='equal');
```

The density of the lines indicates the relative probability of different slopes. Summary point: flat priors are not necessarily minimally informative.  For the slope we probably want a prior that does not artificially over-weight large slopes; see http://arxiv.org/abs/1411.5018 for some discussion. For example, we might use a flat prior on the *angle* the line makes with the x-axis, which implies
$$
P(m) \propto (1 + m^2)^{-3/2}
$$

### Maximum entropy for priors (appealing to physicists!)

 Basic idea: identify least informative $p(x)$ from maximizing entropy:
   $$
     S[p(x)] = -\int\!dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
   $$
   subject to constraints from the prior information.

 $m(x)$ is an appropriate measure (often uniform, but see Sivia)

 Use Lagrange multiplier(s) to maximize

 One constraint is always normalization: $\int\!dx\, p(x) = 1$
     $\Longrightarrow$ alone it leads to uniform $p(x)$ (actually to $m(x)$)

   If the mean $\mu$ and variance $\sigma^2$ are known, then maximize (may be clearer to do with discretized variables)

   \begin{align}
    Q[p(x)] &= -\int\! dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
    + \lambda_0 \left[ 1 - \int\! dx\, p(x) \right] \\
     & \quad + \lambda_1 \left[\sigma^2 -  \int\! dx\, (x-\mu)^2 p(x)  \right]
   \end{align}

   Then
   $$
    \frac{\delta Q}{\delta p(x)} = 0   \quad\mbox{and}\quad
     m(x) = \mbox{const.}
     \ \Longrightarrow\
     p(x \mid \mu,\sigma) = \frac{1}{\sigma\sqrt{2 \pi}}
     e^{-(x-\mu)^2/2\sigma^2}
   $$

    For $\textbf{x} = \{x_1, \cdots, x_N\}$, we find $p(\textbf{x})$
    is the familiar least-squares likelihood function.  See Sivia for many more details.

If you want a "scale invariant" prior, this often means a flat prior on the *logarithm* of the parameter.  This is an example of a [Jeffreys Prior](https://en.wikipedia.org/wiki/Jeffreys_prior).

[Return to Contents]

### Conjugate priors

Bayes theorem tells us that the posterior follows from multiplying the prior by the likelihood and normalizing:

$$
  p(\theta\,|\,x) = \frac{p(x\,|\,\theta)\, p(\theta)}
                          {\int\!d\theta'\, p(x\,|\,\theta')\, p(\theta')}
$$

If the resulting $p(\theta\,|\,x)$ is in the same family of pdfs as $p(\theta)$, then $p(\theta)$ is said to be a conjugate prior for the likelihood $p(x\,|\,\theta)$.

The likelihood is often a fixed form. If there is freedom to choose a conjugate prior, the Bayesian updating of the posterior is given in closed form.

Comments:

Suppose we are flipping a coin and seek the posterior for the probability of heads $\theta \in [0,1]$.  Let $x$ be the number of successes (heads) in $N$ trials. The likelihood is binomial:
   $$ p(x\mid\theta) = \binom{N}{x}
       \theta^{x} (1-\theta)^{N-x} \;.
   $$
If we choose the prior to be a beta distribution with hyperparameters $a$ and $b$ (which are parameters of the prior as opposed to the model),
   $$ p(\theta\mid a,b) = \textrm{Beta}(\theta\mid a,b)
      = \frac{\theta^{a-1}(1-\theta)^{b-1}}{B(a,b)}
   $$
with beta function $B(a,b)$, then the posterior is of the same form ($\bar x \equiv N - x$):
   $$ p(\theta\mid x,a,b) = \frac{\theta^{x+a-1}(1-\theta)^{\bar x+b-1}}{B(x+a,\bar x+b)} = \textrm{Beta}(\theta\mid x+a,\bar x +b)
   $$
This posterior can be used as the prior for more samples; the hyperparameters just add additional information (check that it doesn't matter if you analyze the data all at once or sequentially because the tosses are independent).

See the [Wikipedia article](https://en.wikipedia.org/wiki/Conjugate_prior) for a big table of conjugate prior pairs.

If the likelihood is a normal distribution with known variance, the conjugate prior is also normal.  If it is a normal distribution with known mean, the inverse gamma distribution is a conjugate prior.

[Return to Contents]

## Bayesian updating examples

### Determining the bias of a coin

```python
%matplotlib inline
# adapted from  https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 15)

import scipy.stats as stats

# If the coin is fair, prob_heads = 0.5 but you can set it to what you want.
prob_heads = 0.7

# hyperparameters for several different priors
# prior 1 is uniform in [0,1]
alpha_1 = 1
beta_1 = 1
# prior 2 is concentrated near 0.5 with very small tails
alpha_2 = 30
beta_2 = 30
# prior 3 is peaked at ends, but allows for probability everywhere
alpha_3 = .2
beta_3 = .2

# Calculate Bayesian updating using the conjugate prior for binomial, which is a beta distribution
dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500, 1000, 2000]
data = stats.bernoulli.rvs(prob_heads, size=n_trials[-1])  # heads or tails, 1 or 0
x = np.linspace(0, 1, 301) # mesh for posterior plots

for k, N in enumerate(n_trials):  # enumerate creates a tuple with a counter for each n_trials entry
    heads = data[:N].sum()   # add up the number of 1s = number of heads
    # update using the conjugate prior
    y_1 = dist.pdf(x, alpha_1 + heads, beta_1 + N - heads)   # beta(x,alpha+heads,beta+(N-heads))
    y_2 = dist.pdf(x, alpha_2 + heads, beta_2 + N - heads)
    y_3 = dist.pdf(x, alpha_3 + heads, beta_3 + N - heads)
    y_max = np.max([y_1.max(), y_2.max()])

    # now make the plots!
    sx = plt.subplot(len(n_trials)/2, 2, k+1)
    plt.xlabel("$p$, probability of heads")
    plt.setp(sx.get_yticklabels(), visible=False)
    plt.yticks([])
    plt.plot(x, y_1, label="uniform prior")
    plt.fill_between(x, 0, y_1, color="blue", alpha=0.1)
    plt.plot(x, y_2, label="informative prior", color="r")
    plt.fill_between(x, 0, y_2, color="red", alpha=0.1)
    plt.plot(x, y_3, label="anti prior", color="g")
    plt.fill_between(x, 0, y_3, color="green", alpha=0.1)
    plt.vlines(prob_heads, 0, 1.1*y_max, color="k", linestyles="--", lw=2)
    plt.annotate("observe {:d} tosses,\n {:d} heads".format(N, heads), xy=(0.05,0.35),
                 xycoords='axes fraction', horizontalalignment='left',verticalalignment='top')
    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

figure_title = "Bayesian updating of posterior probabilities for biased coin with actual p(heads) = %1.2f" % prob_heads
plt.suptitle(figure_title,
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

## Sampling

### Sampling from multivariate normal distributions

Suppose we have a univariate normal distribution
$$
  x \sim \mathcal{N}(\mu,\sigma^2)
  \ \Longleftrightarrow\
  p(x\mid\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}
     e^{-(x-\mu)^2/2\sigma^2}
$$
If we have a way to generate standard normals $\mathcal{N}(0,1)$, then we can sample $x$ from
$$
    x \sim \mu + \sigma \mathcal{N}(0,1)  \;.
$$
So there is a simple shift by $\mu$ and then we scale the normal draw by $\sigma$, the square root of the variance.

$\newcommand{\xvec}{\textbf{x}}$
$\newcommand{\muvec}{\boldsymbol{\mu}}$
The general multivariate Gaussian distribution is
$$
   \xvec \sim \mathcal{N}(\muvec,\Sigma)
  \ \Longleftrightarrow\
 p(\xvec\mid \muvec,\Sigma) = \frac{1}{\sqrt{\det(2\pi\Sigma)}} e^{-\frac12(\xvec-\muvec)^{\rm T}\Sigma^{-1}(\xvec-\muvec)}
$$
The generalization to sample it will be to shift by $\muvec$ and scale by some square root of the covariance matrix $\Sigma$:
$$
    \xvec \sim \muvec + B \mathcal{N}(0,I) \;,
$$
where $I$ is the identity matrix and $B B^\intercal = \Sigma$ (e.g., a Cholesky decomposition).

[Return to Contents]

### MCMC sampling

Great examples of MCMC sampling are at http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/.

Here are the individual simplified simulations:

Metropolis-Hastings 2D Gaussian
     and donut

Hamiltonian Monte Carlo 2D Gaussian
     and donut

Hamiltonian Monte Carlo 2D Gaussian with
      U-turn

NUTS sampler 2D Gaussian
     and multi-modal

The detailed simulator can be found at https://chi-feng.github.io/mcmc-demo/, which also links to the github repository with the javascript source.

```javascript
%%javascript
IPython.OutputArea.auto_scroll_threshold = 9999;
```

    <IPython.core.display.Javascript object>

```python
# At present this adversely affects the menu bar
from IPython.display import display,HTML

display(HTML(filename="./MCMC_javascript_visualizations.html"))

```

<!DOCTYPE html>

	<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Markov Chains: Why Walk When You Can Flow? | Elements of Evolutionary Anthropology</title>
    <link rel="profile" href="http://gmpg.org/xfn/11">
    <link rel="pingback" href="http://elevanth.org/blog/xmlrpc.php">
    <link rel="dns-prefetch" href="http://s0.wp.com/">
<link rel="dns-prefetch" href="http://fonts.googleapis.com/">
<link rel="dns-prefetch" href="http://s.w.org/">
<link rel="alternate" type="application/rss+xml" title="Elements of Evolutionary Anthropology » Feed" href="http://elevanth.org/blog/feed/">
<link rel="alternate" type="application/rss+xml" title="Elements of Evolutionary Anthropology » Comments Feed" href="http://elevanth.org/blog/comments/feed/">

<link rel="stylesheet" id="wp-quicklatex-format-css" href="./MCMC_javascript_visualizations_files/quicklatex-format.css" type="text/css" media="all">
<link rel="stylesheet" id="tt-easy-google-fonts-css" href="./MCMC_javascript_visualizations_files/css" type="text/css" media="all">
<link rel="stylesheet" id="dashicons-css" href="./MCMC_javascript_visualizations_files/dashicons.min.css" type="text/css" media="all">
<link rel="stylesheet" id="crawford_oswald-css" href="./MCMC_javascript_visualizations_files/css(1)" type="text/css" media="all">
<link rel="stylesheet" id="crawford_domine-css" href="./MCMC_javascript_visualizations_files/css(2)" type="text/css" media="all">
<link rel="stylesheet" id="crawford_bootstrap_css-css" href="./MCMC_javascript_visualizations_files/bootstrap.min.css" type="text/css" media="all">
<link rel="stylesheet" id="crawford_style-css" href="./MCMC_javascript_visualizations_files/style.css" type="text/css" media="all">
<link rel="stylesheet" id="jetpack-widget-social-icons-styles-css" href="./MCMC_javascript_visualizations_files/social-icons.css" type="text/css" media="all">
<link rel="stylesheet" id="social-logos-css" href="./MCMC_javascript_visualizations_files/social-logos.min.css" type="text/css" media="all">
<link rel="stylesheet" id="jetpack_css-css" href="./MCMC_javascript_visualizations_files/jetpack.css" type="text/css" media="all">

<link rel="https://api.w.org/" href="http://elevanth.org/blog/wp-json/">
<link rel="EditURI" type="application/rsd+xml" title="RSD" href="http://elevanth.org/blog/xmlrpc.php?rsd">
<link rel="wlwmanifest" type="application/wlwmanifest+xml" href="http://elevanth.org/blog/wp-includes/wlwmanifest.xml">
<link rel="prev" title="Journal Clubbing: Accurate Age Estimation" href="http://elevanth.org/blog/2017/09/09/journal-clubbing-accurate-age-estimation/">
<link rel="next" title="Algebra and the Missing Oxen" href="http://elevanth.org/blog/2018/01/29/algebra-and-missingness/">
<meta name="generator" content="WordPress 4.9.8">
<link rel="canonical" href="http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/">
<link rel="shortlink" href="https://wp.me/p6D80w-bM">
<link rel="alternate" type="application/json+oembed" href="http://elevanth.org/blog/wp-json/oembed/1.0/embed?url=http%3A%2F%2Felevanth.org%2Fblog%2F2017%2F11%2F28%2Fbuild-a-better-markov-chain%2F">
<link rel="alternate" type="text/xml+oembed" href="http://elevanth.org/blog/wp-json/oembed/1.0/embed?url=http%3A%2F%2Felevanth.org%2Fblog%2F2017%2F11%2F28%2Fbuild-a-better-markov-chain%2F&format=xml">

<link rel="dns-prefetch" href="http://v0.wordpress.com/">

<meta property="og:type" content="article">
<meta property="og:title" content="Markov Chains: Why Walk When You Can Flow?">
<meta property="og:url" content="http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/">
<meta property="og:description" content="Abstract: If you are still using a Gibbs sampler, you are working too hard for too little result. Newer, better algorithms trade random walks for frictionless flow. In 1989, Depeche Mode was popula…">
<meta property="article:published_time" content="2017-11-28T20:50:01+00:00">
<meta property="article:modified_time" content="2018-04-26T08:15:56+00:00">
<meta property="og:site_name" content="Elements of Evolutionary Anthropology">
<meta property="og:image" content="http://elevanth.org/blog/wp-content/uploads/2017/11/KRDx8SR-1.gif">
<meta property="og:image:width" content="500">
<meta property="og:image:height" content="381">
<meta property="og:locale" content="en_US">
<meta name="twitter:site" content="@rlmcelreath">
<meta name="twitter:text:title" content="Markov Chains: Why Walk When You Can Flow?">
<meta name="twitter:card" content="summary">

<link rel="icon" href="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-150x150.png" sizes="32x32">
<link rel="icon" href="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-300x300.png" sizes="192x192">
<link rel="apple-touch-icon-precomposed" href="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-300x300.png">
<meta name="msapplication-TileImage" content="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-300x300.png">

Markov Chains: Comparing different samplers

Abstract: If you are still using a Gibbs sampler, you are working too hard for too little result. Newer, better algorithms trade random walks for frictionless flow.
            [Adapted from a blog entry
            by Richard McElreath (@rlmcelreath).]

Metropolis, Hastings, and the Random Walk

The simplest and least reliable way of building a Markov chain is the Metropolis-Hastings algorithm.
Below are embedded the MCMC simulations written by Chi Feng, found here. The target distribution is a benign two-dimensional Gaussian—a nice Gaussian hill. You are looking down on it, with its peak in the center. The Markov chain wanders around this hill, making random proposals to move away from its current position. These proposals are represented by the arrows. Green arrows are accepted proposals. The chain moves to the new location. Red arrows are rejections.

<object data="./MCMC_javascript_visualizations_files/applet.html#RandomWalkMH,standard" type="text/html" width="600" height="400"></object>

As samples build up, the target distribution takes shape, as seen in the histograms on the bottom and left margins. The algorithm works, even though it blindly stumbles around the target, doing a kind of random walk.

And that is exactly the problem. A major problem is that Metropolis-Hastings is a bit too random. So it spends a lot of time re-exploring the same parts of the target, and unless it is tuned just right, it will also reject a lot of proposals (the red arrows), wasting computer time.

Here’s another simulation, this time with a harder target: a donut. The donut target might look weird, but it represents a very common target, by analogy. In high dimensions—once there are many parameters—the target distribution occupies a narrow ring (in high-dimensional space). Most of the probability mass is not near the center. Now look what happens to Metropolis-Hastings:

<object data="./MCMC_javascript_visualizations_files/applet.html#RandomWalkMH,donut" type="text/html" width="600" height="400"></object>

Notice how the Markov chain tends to get stuck in specific regions of the donut. Not only that, but it rejects a lot of proposals (the red arrows), so it wastes a lot of computer time doing nothing. Given enough time, it can explore the entire target. But it might take a very long time indeed. When we don’t know what the target looks like in the first place, this kind of behavior is not only annoying, but also hazardous to inference.

The fundamental problem with Metropolis-Hastings, and with Gibbs-Sampling as a special case, is that it is just too random. In simple targets, that isn’t so bad. But in even moderately complex targets, it means inference often isn’t reliable. It tends to get stuck in narrow regions of the target. There must be a better way.

Better Living Through Physics

Can your Markov chain do this?

If there’s a random way to do something, there’s usually a less random way that is both better and requires more thought. Instead of making random proposals, suppose instead that you run a physics simulation. Your vector of parameters is now a particle in n-dimensional space. The surface in this space is a giant n-dimensional bowl. The shape of the bowl is determined by the shape of the logarithm of the target distribution. If the target is a nice Gaussian, for example, then the log-Gaussian is a smooth parabolic bowl like this (in one-dimension):

To make things a little crazier, suppose that this surface is frictionless. Now what we do is flick the particle in a random direction. It will frictionlessly flow across the bowl, eventually turning around. If we take samples of the particle’s position along its path, before flicking it off in another random trajectory, then we can learn about the shape of the surface.

This is the principle behind Hamiltonian Monte Carlo. It will be easier to see it in action. Here is another simulation, this time using Hamiltonian Monte Carlo, again on the two-dimensional Gaussian target. The paths are flicks of the particle, and the green arrows again represent accepted proposals.

<object data="./MCMC_javascript_visualizations_files/applet.html#HamiltonianMC,standard" type="text/html" width="600" height="400"></object>

Now the proposals are both within the high-probability region of the target—so many fewer proposals are rejected—and the proposals can get far away from their starting point, so that the chain efficiently explores the whole shape of the target in less time. Effectively, it flows across the target and maps out its whole shape much faster.

The cost of all this elegance is needing more information about the target. Hamiltonian Monte Carlo does a lot more calculation. But it also needs fewer samples to get a good image of the target. Where this really counts is with the donut. Let’s revisit it:

<object data="./MCMC_javascript_visualizations_files/applet.html#HamiltonianMC,donut" type="text/html" width="600" height="400"></object>

Now instead of getting stuck, the chain sweeps around the target. Even though all the chain knows at any moment in time is the local shape of the target—it can’t see the whole distribution like you can here—it still manages to glide smoothly around it. This shouldn’t be so amazing—a ball doesn’t know the shape of the surface it rolls on, yet its path is governed by it.

Stan is NUTS

There are still some improvements to be had. Hamiltonian Monte Carlo needs to be told how many steps to take in its simulated paths. The step number determines how long the path continues before a new flick is made in a new random direction. If it takes too few steps, then it ends up with samples too similar to one another. If it takes too many, it can also end up with samples too similar to one another. Why? Because the path eventually makes a U-turn. Here’s an example:

<object data="./MCMC_javascript_visualizations_files/applet.html#HamiltonianMC,standard,65" type="text/html" width="600" height="400"></object>

The path goes on long enough that it often eats its own tail—it makes an unfortunate U-turn. The algorithm still works, but it isn’t very efficient, because it again explores in local spaces. We can of course tune the number of steps by hand, but that is not so easy when the target distribution is complex.

The No U-Turn Sampler (NUTS) is an approach for adaptively finding a good number of steps. The NUTS algorithm tries to figure out when the path starts to turn around. In order to do this efficiently, it needs to simulate the path in both directions. It looks pretty weird:

<object data="./MCMC_javascript_visualizations_files/applet.html#NaiveNUTS,standard" type="text/html" width="600" height="400"></object>

Notice how the path grows in both directions. This is the algorithm figuring out when the path turns around. When the path starts to turn around, NUTS stops the simulation and takes a sample. Then it flicks the particle in another random direction and starts another simulation. There are lots of little adaptive nudges in these algorithms that help them explore the target more efficiently. An implementation like Stan (mc-stan.org) uses an advanced version of NUTS that is even slicker.

Problems Remain

Hamiltonian algorithms still have limitations. Some targets are still hard to explore. Here’s a example: a multimodal target. While the paths do a good job exploring each lump of probability mass, they have trouble transitioning among them.

<object data="./MCMC_javascript_visualizations_files/applet.html#NaiveNUTS,multimodal" type="text/html" width="600" height="400"></object>

Targets like this are not so unusual. They arise in many classification and latent variable models. With some clever coding, you can collapse some modes together. But you have to realize what is going on, first. Other issues arise with models that contain very steep changes in log-probability. Wizards are working on solutions to these problems. But even before solutions are found, Hamiltonian samplers are typically much better than Gibbs zombies.

Read More

The BUGS project’s history is summarized in: Lunn, Spiegelhalter, and Best. (2009). “The BUGS project: Evolution, critique and future directions.” doi:10.1002/sim.3680

Hamiltonian Monte Carlo was originally called “Hybrid Monte Carlo”: Duane, Kennedy, Pendleton, and Roweth (1987) “Hybrid Monte Carlo”. doi:10.1016/0370-2693(87)91197-X

The No U-Turn Sampler (NUTS) was first described by Hoffman and Gelman (2011) “The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo.” arxiv.org/abs/1111.4246

Michael Betancourt’s “Conceptual Introduction to Hamiltonian Monte Carlo” is only slightly technical and worth your time. arxiv.org/abs/1701.02434

		<form action="http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/" method="post">
			Send to Email Address
			<input type="email" name="target_email" id="target_email" value="">

				Your Name
				<input type="text" name="source_name" id="source_name" value="">

				Your Email Address
				<input type="email" name="source_email" id="source_email" value="">

						<input type="text" id="jetpack-source_f_name" name="source_f_name" class="input" value="" size="25" autocomplete="off" title="This field is for validation and should not be changed">

			<input type="submit" value="Send Email" class="sharing_send">
			Cancel

				Post was not sent - check your email addresses!

				Email check failed, please try again

				Sorry, your blog cannot share posts by email.

		</form>

[Return to Contents]

## Model selection and evidence

  $\newcommand{\thetavec}{\boldsymbol{\theta}}$
  Determine the evidence for different models $M_1$ and $M_2$ via marginalization by integrating over all possible sets of parameters ${\thetavec}$ in different models, with the same data $D$ and information $I$.

The evidence ratio for two different models:
$$
   \frac{p(M_1\mid D, I)}{p(M_2\mid D, I)}
   = \frac{p(D\mid M_1, I)\,p(M_1,I)}{p(D\mid M_2, I)\,p(M_2,I)}
$$

The Bayes Ratio (implements Occam’s Razor):
$$
\frac{p(D\mid M_1, I)}{p(D\mid M_2, I)}
  = \frac{\int\!d\thetavec_1\, p(D\mid\thetavec_1,M_1,I)
             \,p(\thetavec_1\mid M_1,I)}
         {\int\!d\thetavec_2\, p(D\mid\thetavec_2,M_2,I)
             \,p(\thetavec_2\mid M_2,I)}
$$

Example: what order polynomial underlies the noisy data?

[Return to Contents]

## Gaussian processes (GPs)

### Overview of GPs

GP: the natural generalization of multivariate Gaussian random variables to infinite (countably or continuous) index sets.  They look like random functions, but with characteristic degrees of smoothness, correlation lengths, and range.  Here are some examples with different "kernels" and different parameters that dictate those features (figure from J. Melendez):

#### Explanations of GPs from the web

The following is adapted from a blog entry from Kathleen Bailey at http://katbailey.github.io/post/gaussian-processes-for-dummies/.

"Here’s how Kevin Murphy explains it in the excellent textbook Machine Learning: A Probabilistic Perspective:"
'A GP defines a prior over functions, which can be converted into a posterior over functions once we have seen some data. Although it might seem difficult to represent a distribution over a function, it turns out that we only need to be able to define a distribution over the function’s values at a finite, but arbitrary, set of points, say $x_1, \ldots, x_N$.  A GP assumes that $p(f(x_1),\ldots,f(x_N))$ is jointly Gaussian, with some mean $\mu(x)$ and covariance $\Sigma(x)$ given by $\kappa_{ij} = \Sigma(x_i,x_j)$, where $\kappa$ is a positive definite kernel function. The key idea is that if $x_i$ and $x_j$ are deemed by the kernel to be similar, then we expect the output of the function at those points to be similar, too.'

So it is important to stress that we are really only dealing with a discrete set of points. Thus the physicist-friendly idea of a continuum limit of masses on springs may be preferred to more abstract notions in function space.

It should also be sufficient to consider the bivariate case, because the generalization from one to two variables is really where the new feature of correlation comes in.  Generalizing further really doesn't introduce anything new.

### Bivariate normal case

$\newcommand{\xvec}{\textbf{x}}$
$\newcommand{\muvec}{\boldsymbol{\mu}}$
The general multivariate Gaussian distribution is
$$
 p(\xvec\mid \muvec,\Sigma) = \frac{1}{\sqrt{\det(2\pi\Sigma)}} e^{-\frac12(\xvec-\muvec)^{\rm T}\Sigma^{-1}(\xvec-\muvec)}
$$

For the bivariate case we can parameterize the mean vector and covariance matrix as
$$
  \muvec = \left( \begin{array}{c}
                     \mu_x \\ \mu_y
                  \end{array}       \right)
    \;, \qquad
  \Sigma = \left( \begin{array}{cc}
                    \sigma_x^2 & \rho\sigma_x\sigma_y \\
                    \rho\sigma_x\sigma_y & \sigma_y^2
                  \end{array}
           \right)
$$
The covariance matrix must be positive definite, which implies $\color{red}{0\lt\rho^2\lt 1}$.

If take $\mu_x = \mu_y = 0$ and $\sigma_x = \sigma_y = \sigma$ for clarity,
so that
$$
  \Sigma = \sigma^2 \left(\begin{array}{cc}
                       1 & \rho \\
                       \rho & 1
                       \end{array}
                    \right)
$$
and
$$
  p(x,y\mid \sigma,\rho) = \frac{1}{2\pi\sigma^2}
        \exp\left(-\frac{x^2 + y^2 - 2\rho x y }{2\sigma^2\sqrt{1-\rho^2}}
          \right)
          \;.
$$
It's clear that contours of equal probability have $x^2 + y^2 - 2\rho xy = \mbox{constant}$, so they are ellipses.  The value of $\rho$ determines the eccentricity of the ellipse.
If $\rho=0$, $x$ and $y$ are independent (uncorrelated) and we have a circle.  As $\rho$ approaches $+1$, $x$ and $y$ are increasingly correlated (toward straight line at $45^\circ$), while for $\rho$ approaching $-1$ they become increasingly anti-correlated (toward straight line at $-45^\circ$).

For reference, the Cholesky decomposition of $\Sigma$ is
$$
  \Sigma = \sigma^2\left( \begin{array}{cc}
                    1 & \rho \\
                    \rho & 1
                  \end{array}
           \right)
           =
           \sigma^2\left( \begin{array}{cc}
                    1 & 0 \\
                    \rho &  \sqrt{1-\rho^2}
                  \end{array}
           \right)
           \left( \begin{array}{cc}
                    1 & \rho \\
                    0 &  \sqrt{1-\rho^2}
                  \end{array}
           \right)
$$

### Example code for generating and plotting GPs

The following code is adapted from a blog post by Katherine Bailey entitled Gaussian Processes for Dummies.  First we generate several instances of draws from a Gaussian process with a squared exponential kernel function, which is the covariance between $x$ and $x'$:
$$  \kappa_{\rm SE}(x,x') = \sigma^2 e^{-(x-x')^2/2l^2} $$

So we can see that $\sigma$ controls the vertical extent of the functions while $l$ controls
how rapidly they wiggle.  Comparing to our expression above for the bivariate normal case,
we see that $\rho$ is given by $e^{-(x-x')^2/2l^2}$.  So when $x$ and $x'$ are close,
$\rho \approx 1$ and the value of the function is highly correlated.  When $x$ and $x'$ are far apart, $\rho \rightarrow 0$, and they become independent (thus $l$ plays the role of a correlation length).

Let's generate some GPs with this kernel! For the function $f(x)$ we write draws as
$$
  f(x) \sim \mathcal{GP[\mu(x),\kappa(x,x')]}
$$
where $\mu(x)$ is the mean at each $x$ and $\kappa(x,x')$ is the covariance between $x$ and $x'$.  In practice we have a finite set of $N$ points $\textbf{x} = \{x_i\}_{i=1}^{N}$ with corresponding function values $\textbf{f}=\{f(x_i)\}_{i=1}^{N}$.
We form the mean vector $\boldsymbol{\mu} = m(\textbf{x})$ and the covariance matrix $K_{ij} = \kappa(x_i,x_j)$. Then
$$  \textbf{f} \mid \textbf{x} \sim \mathcal{N}(\boldsymbol{\mu},K)
$$
are draws from a multivariate normal distribution.  Try it:

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

# Define the squared exponential kernel function for the covariance
#   We take the variance to be 1.
def sqr_exp_kernel(a, b, length_param):
    sqdist = np.sum(a**2,1).reshape(-1,1) + np.sum(b**2,1) - 2*np.dot(a, b.T)
    return np.exp(-sqdist / (2*length_param**2))

# Grid of x points
npts = 500
x_min = -5
x_max = +5
Xtest = np.linspace(x_min, x_max, npts).reshape(-1,1)

length_param = 1.  # this is "l" (correlation length)
K_ss = sqr_exp_kernel(Xtest, Xtest, length_param)

# Get Cholesky decomposition (square root) of the covariance matrix
nugget = 1e-12   # size of nugget will depend on how many points are used
                 #  1e-12 for 500; 1e-13 for 100; 1e-15 for 50
L = np.linalg.cholesky(K_ss + nugget*np.eye(npts))

# Sample 3 sets of standard normals for our test points,
# multiply them by the square root of the covariance matrix
# Note: mean mu = 0 here implicitly.
f_prior = np.dot(L, np.random.normal(size=(npts,3)))

# Now let's plot the 3 sampled functions.
plt.plot(Xtest, f_prior)
plt.axis([-5, 5, -3, 3])
plt.title('Three samples from the GP prior with l = {:1.1f}'.format(length_param))
plt.show()
```

Now we train it on some data (see references for details):

```python
# Noiseless training data
Xtrain = np.array([-4, -3, -2, -1, 1]).reshape(5,1)
ytrain = np.sin(Xtrain)

# Apply the same kernel function to our training points
nugget_train = 5e-5
K = sqr_exp_kernel(Xtrain, Xtrain, length_param)
L = np.linalg.cholesky(K + nugget_train*np.eye(len(Xtrain)))

# Compute the mean at our test points.
K_s = sqr_exp_kernel(Xtrain, Xtest, length_param)
Lk = np.linalg.solve(L, K_s)
mu = np.dot(Lk.T, np.linalg.solve(L, ytrain)).reshape((npts,))

# Compute the standard deviation so we can plot it
s2 = np.diag(K_ss) - np.sum(Lk**2, axis=0)
stdv = np.sqrt(s2)

# Draw samples from the posterior at our test points.
nugget_test = 1e-6
L = np.linalg.cholesky(K_ss + nugget_test*np.eye(npts) - np.dot(Lk.T, Lk))
f_post = mu.reshape(-1,1) + np.dot(L, np.random.normal(size=(npts,3)))

plt.plot(Xtrain, ytrain, 'bs', ms=8)
plt.plot(Xtest, f_post)
plt.gca().fill_between(Xtest.flat, mu-2*stdv, mu+2*stdv, color="#dddddd")
plt.plot(Xtest, mu, 'r--', lw=2)
plt.axis([-5, 5, -3, 3])
plt.title('Three samples from the GP posterior')
plt.show()
```

### Other demos for Gaussian Processes (and other regression)

Gaussian process regression, where you can add data points, play with the hyperparameters, and then see the inference for the curve. It’s by Tomi Peltola:
http://www.tmpl.fi/gp/

This simulation shows how a GP prior is a distribution over functions, and how observing data conditions the prior to obtain the GP posterior.
http://rpradeep.webhop.net/gpr/

[Return to Contents]

## Appendices

### References

Please suggest additional references (with links).

### Physics-oriented pedagogical articles and texts

R. Trotta,
        Bayes in the sky: Bayesian inference and model selection in cosmology, Contemp. Phys. 49, 71 (2008)
        [arXiv:0803.4089].

D.S. Sivia and J. Skilling,
       Data Analysis: A Bayesian Tutorial, 2nd edition, (Oxford University Press, 2006).

P. Gregory,
     Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica® Support, (Cambridge University Press, 2010).

### Standard statistics references

A. Gelman et al.,
        Bayesian Data Analysis, 3rd edition, (Chapman and Hall/CRC, 2013).

### BUQEYE references

R.J. Furnstahl, D.R. Phillips and S. Wesolowski,
      A recipe for EFT uncertainty quantification in nuclear physics,
      J. Phys. G 42, 034028 (2015), [arXiv:1407.0657].

 R.J. Furnstahl, N. Klco, D.R. Phillips and S.Wesolowski,
    Quantifying truncation errors in effective field theory,
    Phys. Rev. C 92, 024005 (2015)
    [arXiv:1506.01343].

S. Wesolowski, N. Klco, R.J. Furnstahl, D.R. Phillips and A. Thapaliya,
  Bayesian parameter estimation for effective field theories,
    J. Phys. G 43, 074001 (2016)
    [arXiv:1511.03618].

  J.A. Melendez, S. Wesolowski and R.J. Furnstahl,
    Bayesian truncation errors in chiral effective field theory: nucleon-nucleon observables,
  Phys. Rev. C 96, 024003 (2017)
  [arXiv:1704.03308].

  S. Wesolowski, R.J. Furnstahl, J.A. Melendez and D.R. Phillips,
  Exploring Bayesian parameter estimation for chiral effective field theory using nucleon-nucleon phase shifts,
  [arXiv:1808.08211].

### Github repositories

Please suggest more!

https://github.com/jakevdp/BayesianAstronomy Materials for the Bayesian Methods in Astronomy workshop at the 227th American Astronomical Society meeting.  Includes Jupyter notebooks and useful exercises.

http://people.duke.edu/~ccc14/sta-663-2018/ STA 663: Computational Statistics and Statistical Computing (2018) at Duke University.  Lots of good things here!

### Vocabulary

Plan: build up a good set of definitions with appropriate links.  Please add more words/phrases!

conjugate prior

If the probability distribution family (e.g., beta distributions) for the posterior pdf is the same as for the prior pdf, the latter is said to be a conjugate prior. This means that the updating by Bayes' rule can be carried out analytically.   Some Bayesian practitioners are strongly opposed to the use of conjugate priors (see  comments here).

credible vs. confidence interval

This is a contrast between Bayesian and frequentist statistics.  For a frequentist, a parameter has a true value, which is fixed and not a distribution.  A 95% confidence interval mean that with a large number of repeated trials, 95% of the calculated confidence intervals would include the true value. This is clearly hard to think about!  A Bayesian 95% credible interval is the range of the posterior for the parameter (which is treated as a random variable) that has 95% of the probability.  So there is a 95% probability that the parameter is in that interval.

evidence

In the standard context of inferring parameters $\boldsymbol{\theta}$ given data $D$ and information $I$, the evidence is $p(D\mid I) = \int\! d\boldsymbol{\theta}\, p(D \mid \boldsymbol{\theta},I)\,p(\boldsymbol{\theta},I)$.  This is also called the Fully Marginalized Likelihood or FML. The expression shows that it is the integral over all $\boldsymbol{\theta}$ weighted by the likelihood.  This is typically an expensive integral to do.  In the context of model fitting (i.e., parameter estimation), it acts as a normalization constant and in most cases can be ignored because the normalization can be found directly (or only relative probabilities are needed).

gaussian process

From [Wikipedia](https://en.wikipedia.org/wiki/Gaussian_process): "In probability theory and statistics, a Gaussian process is a stochastic process (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a multivariate normal distribution, i.e. every finite linear combination of them is normally distributed. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space."

hierarchical model

A model with hyperparameters. See [Wikipedia](https://en.wikipedia.org/wiki/Bayesian_hierarchical_modeling).

hyperparameter

A parameter of a prior distribution.

iid (independently and identically distributed)

A set of random variables is iid (or i.i.d. or IID) if each random variable has the same probability distribution and all are mutually independent.

likelihood

Usually in the form $p(D\mid \boldsymbol{\theta},I)$, where $\boldsymbol{\theta}$ are the parameters of our model, $D$ is the data, and $I$ is any other information we use.  This is the probability of observing our actual data given the model (with the particular parameters $\boldsymbol{\theta}$). It is the same quantity that is maximized in frequentist maximum-likelihood approaches.

MAP estimate

Maximum a posteriori estimate. This is the mode (maximum) of the posterior distribution for the quantity of interest. If the prior is uniform, the MAP estimate equals the maximum likelihood estimate.

maximum entropy

A method used to determine priors.

MCMC

Markov-chain Monte Carlo. A generic name for stochastic sampling methods.

model selection and model averaging

nugget

For Gaussian process (GP) calculations or any sampling of a multivariate normal distribution, one typically needs to find the Cholesky decomposition of the covariance matrix.  However, this matrix can become ill-conditioned (too-small or negative eigenvalues).   A standard solution is to add a small number, called a nugget, to the diagonal of the covariance matrix. For GP regression, this is equivalent to adding (or increasing, if already present) the data noise.

nuisance parameter

A nuisance parameter is a parameter in your model whose value you don't care about for the posterior. So you integrate it out (marginalize).

overfitting and underfitting

This example from http://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html of fitting polynomials to nonlinear functions illustrates overfitting and underfitting.  The true function is a cosine with noise added.  A polynomial of degree 1 is an inadequate model for this data; this is underfitting.  The polynomial of degree 15 tries to fit the noise; this is overfitting.

point estimate (cf. interval estimate)

A point estimate is a single value to characterize a
posterior.  It could be the mode, mean, median or something
else.  An interval estimate is more natural in Bayesian statistics, because the full posterior is the real target.  Giving a series of credible intervals often conveys much of the information about the posterior.

posterior

This is the quantity of the left side of Bayes' rule, the thing we want to compute.  Often in the form $p(\boldsymbol{\theta}\mid D,I)$, where $\boldsymbol{\theta}$ are the parameters of our model, $D$ is the data, and $I$ is any other information we use. It is our knowledge of the model given the data and any relevant background knowledge (which include the choice of model).

prior

A pdf that encodes what is known about the answer (e.g., parameters) before any data is used. The notation consistent with our definitions of posterior and likelihood is $p(\boldsymbol{\theta}\mid I)$, where $\boldsymbol{\theta}$ are the parameters of our model and $I$ is any other information we use (e.g., some of the parameters must be positive or less than a known magnitude because of physics reasons).
     See also conjugate prior and maximum entropy.

residual

The difference of theory prediction and experimental data.

### Notation    [still coming . . .]

Plan: build up a dictionary of notation with appropriate links and examples (with code).

univariate normal distribution
$$\mathcal{N}(\mu,\sigma^2)$$

[Return to Contents]

```python

```


```python
"""Header cell, contains modules and functions to make the whole notebook experience better"""
%matplotlib inline
# plots graphs within the notebook

from IPython.display import display,Image, Latex
from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')
from IPython.display import clear_output

import time

from IPython.display import display,Image, Latex

from IPython.display import clear_output

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

font = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 12,
        }
fontlabel = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 12,
        }

from matplotlib.ticker import FormatStrFormatter
plt.rc('font', **font)

class PDF(object):
  def __init__(self, pdf, size=(200,200)):
    self.pdf = pdf
    self.size = size

  def _repr_html_(self):
    return ''.format(self.pdf, self.size)

  def _repr_latex_(self):
    return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

```

```python
import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

from math import sin, cos

class Robot(object):
    """Defines basic mobile robot properties"""
    def __init__(self):
        self.pos_x  = 0.0
        self.pos_y  = 0.0
        self.angle  = 0.0
        self.plot   = False
        self._delta = 0.01
        self.step_plot = int(5)
        self.mag_plot = 1.

    # Movement
    def step(self):
        """ updates the x,y and angle """
        self.deltax()
        self.deltay()
        self.deltaa()

    def move(self, seconds):
        """ Moves the robot for an 's' amount of seconds"""
        for i in range(int(seconds/self._delta)):
            self.step()
            if i % self.step_plot == 0 and self.plot: # plot path every 3 steps
                self.plot_xya()

    # Printing-and-plotting:
    def print_xya(self):
        """ prints the x,y position and angle """
        print ("x = " + str(self.pos_x) +" "+ "y = " + str(self.pos_y))
        print ("a = " + str(self.angle))

    def plot_robot(self):
        """ plots a representation of the robot """
        plt.arrow(self.pos_x, self.pos_y, 0.001
                  * cos(self.angle), 0.001 * sin(self.angle),
                  head_width=self.mag_plot*self.length, head_length=self.mag_plot*self.length,
                  fc='k', ec='k')

    def plot_xya(self):
        """ plots a dot in the position of the robot """
        plt.scatter(self.pos_x, self.pos_y, c='r', edgecolors='r')

class DDRobot(Robot):
    """Defines a differential drive robot"""

    def __init__(self):
        Robot.__init__(self)
        self.radius = 0.1
        self.length = 0.4

        self.rt_spd_left = 0.0
        self.rt_spd_right = 0.0

    def deltax(self):
        """ update x depending on l and r angular speeds """
        self.pos_x += self._delta * (self.radius*0.5) \
        * (self.rt_spd_right + self.rt_spd_left)*cos(self.angle)

    def deltay(self):
        """ update y depending on l and r angular speeds """
        self.pos_y += self._delta * (self.radius*0.5) \
        * (self.rt_spd_right + self.rt_spd_left)*sin(self.angle)

    def deltaa(self):
        """ update z depending on l and r angular speeds """
        self.angle += self._delta * (self.radius/self.length) \
        * (self.rt_spd_right - self.rt_spd_left)
# function to convert degrees to radians
def D2R(a):
    return math.pi*a/180
def R2D(a):
    return 180*a/math.pi
# mybot = DDRobot()            # robot called 'enesbot'

# mybot.angle = 3.1416/4        # 45 degrees
# mybot.plot = True             # plot the robot!
# mybot.plot_robot()

# mybot.rt_spd_left = 10
# mybot.rt_spd_right = 10       # straight line
# mybot.move(2)                 # move for 2 seconds

# mybot.rt_spd_left = 12.5664
# mybot.rt_spd_right = 18.8496  # (2m diameter circle)
# mybot.move(1)                 # move for 1 second

# mybot.rt_spd_left = 18.8496
# mybot.rt_spd_right = 12.5664  # (2m diameter circle)
# mybot.move(2.5)               # move for 2.5 second

# mybot.rt_spd_left = 12.5664
# mybot.rt_spd_right = 18.8496  # (2m diameter circle)
# mybot.move(3.5)               # move for 2.5 second

# mybot.plot_robot()

# plt.xlim([-1, 6])               # axis limits
# plt.ylim([-1, 6])

# plt.show()
```

```python
PDF('bot-sketch.pdf',size = (550,400))
```

# Differential Drive Robot - 2: Calibration

Alphabot 2 is a differential drive robot: Two independent motors drive the left and right wheels and the differential speed between the two wheels control the speed and direction of the robot. The following code is copied from http://enesbot.me/kinematic-model-of-a-differential-drive-robot.html

## Objective
The objective is to investigate methods of calibration of the robot.

## Quick Theoretical Background

The robot parameters are:

 Wheel radius: $r$.

 Length between wheels: $L$

 Angular velocity of the left and right wheels: $\omega_L$, $\omega_R$, respectively.

 Angle from horizontal: $\alpha$.

 Position vector of the robot: $(x,y)$.

 Velocity vector of the robot: $V=(\dot{x},\dot{y})$.

Hereafter $\dot{a}$ refers to the time derivative of variable $a$:

$$
\dot{a}=\frac{da}{dt}
$$

The velocities of the wheels are therefore defined as:

$$
V_L=\omega_Lr\text{ and }V_R=\omega_Rr.
$$

The velocity of the robot, taken at the center of the wheels, is simply:

$$
\vec{V}=\frac{V_R+V_L}{2}(\cos(\alpha)\vec{e}_x+\sin(\alpha\vec{e}_y),
$$

yielding the following equations of motions:

$$
\dot{x}=r\frac{\omega_R+\omega_L}{2}\cos{\alpha}
$$

$$
\dot{y}=r\frac{\omega_R+\omega_L}{2}\sin{\alpha}
$$

and

$$
\dot{\alpha}=\frac{r}{L}(\omega_R-\omega_L)
$$

In the previous assignment, an equation was derived to determine the time $T$ needed to turn the robot an angle $\alpha_o$ at given rotation speed $\omega_L$ and $\omega_R$ and robot dimensions $r$ and $L$. This equation started by intergrating the robot angle equation above:

$$
\dot{\alpha}=\frac{d\alpha}{dt}=\frac{r}{L}(\omega_R-\omega_L)
$$

from $0$ to a time $T$ yields

$$
\int_0^T\frac{d\alpha}{dt}dt=\int_0^T\frac{r}{L}(\omega_R-\omega_L)dt
$$

Since the left hand side function is constant with respect to time, the result is simply:

$$
\alpha(T) - \alpha(0) =Tr\frac{\omega_R-\omega_L}{L}
$$

Therefore the time to execute a rotation of angle $\alpha_o=\alpha(T) - \alpha(0)$ is

$$
T = \frac{\alpha_oL}{r(\omega_R-\omega_L)}.\;\;\;\mathrm{Eq. 1}
$$

## Position of the Problem
In a perfect world if your code sets the same rotation speed for both wheels, the robot travels on a straight line. In the present case of independent motors, this outcome is very unlikely. Actually one should not expect the robot to drive straight. Below is a model of drift. Here the drift has two components: a constant drift (it could be a signal offset issue), and complex mechanical friction modeled by a quadratic function of the rotation speed. You will assume that you don't know the equation of the drift.

## Goal
You will derive a calibration curve using polynomial fit and test your calibration on a specific motion.

## Example of drift

Below is a code showing the trajectory of a robot for a series of 5 runs at different rotation speeds $\omega=1, 2.5, 5, 7.5, 10$. The robot starts from the same initial conditions, $x=0,y=0,\alpha=0$, has no drift (perfect robot) and moves for 10 s for all runs.

```python
T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_R = omega
omega_L = omega
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

Now we introduce a constant random drift $\delta$ and see its effect on the different trajectories. The function numpy.random.rand returns a random number between 0 and 1.

```python
delta = np.random.rand(1)
T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_R = omega + delta[0]
omega_L = omega
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

The plot below shows the angle at the end of each run. Based on Eq. 1 above, does this plot make sense?

```python
plt.plot(omega,R2D(end_angle),'o-',label="experiment")
plt.xlabel(r"$\omega$")
plt.ylabel(r"$\alpha\;(^\circ)$ ")
plt.legend(loc=3, bbox_to_anchor=[0, 1.], ncol=2, shadow=False, fancybox=True)
plt.show()
```

The correction is straightforward. For each run, the rotation speed of the right wheel must be corrected by $-\delta$.

```python

T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] - delta[0]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

## Proposed calibration

You will assume that the drift $\delta$ is caused by the right motor and additive:

$$
\omega_L = \omega,\;\omega_R = \omega+\delta(\omega).
$$

If $\delta(\omega)$ is positive, the robot, when programmed to go straight, the robot will actually turn left.

The proposed calibration consists of programming the robot for a series of straight runs for various $\omega$ and constant $T$, $r$ and $L$. Once the robot stops, the deviation from the straight motion is measured by the angle of the robot at time $T$. This angle is a function of $\omega$, $\alpha(\omega)$ if the drift is a function of the rotation speed. Use Eq. 1 above to determine the drift $\delta(\omega)$ as a function of $\alpha(\omega)$, $L$, $T$ and $r$.

```python
""" random quadratic drift equation. Note that every time
you run this cell, the trajectories will change"""
a = np.random.random(2)# random drift amplitude
print(a)
def drift(omega):
    f = a[0] + a[1] * (omega/5)**2
    return f
```

    [0.03285105 0.15091591]

### Uncalibrated robot

```python
T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_R = omega + drift(omega)
omega_L = omega
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

Note in the code above, the angle at the end of the motion is stored in the array end_angle. As shown below, the drift appears to follow a parabola.

```python
plt.plot(omega,R2D(end_angle),'o',label="experiment")
plt.xlabel(r"$\omega$")
plt.ylabel(r"$\alpha$")
plt.legend(loc=3, bbox_to_anchor=[0, 1.], ncol=2, shadow=False, fancybox=True)
plt.show()
```

Let's now fit a quadratic polynomial to these points. The creation of a polynomial in python is shown below using numpy polyfit function. p2_coef are the coefficients $p_i$ of the polynomial $p(x)=p_0x^2+p_1x+p_2$ (see https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.polyfit.html).

```python
p2_coef = np.polyfit(omega,end_angle,2)
print(p2_coef)
```

    [ 1.50915907e-02 -4.78006585e-15  8.21276361e-02]

To generate a function, use the numpy function poly1d:

```python
p2 = np.poly1d(p2_coef)
p2 = np.poly1d(np.polyfit(omega,end_angle,2)) #equivalent compact form
p2
```

    poly1d([ 1.50915907e-02, -4.78006585e-15,  8.21276361e-02])

You can now generate a fit over a range of your choice, here $0\leq\omega\leq10$

```python
N = 51
om = np.linspace(0,10,N)

plt.plot(omega,R2D(end_angle),'o',label="experiment")
plt.plot(om,R2D(p2(om)),label='fit')
plt.xlabel(r"$\omega$")
plt.ylabel(r"$\alpha$")
plt.legend(loc=3, bbox_to_anchor=[0, 1.], ncol=2, shadow=False, fancybox=True)
plt.show()
```

In Eq. 1, introduce the drift at $\delta$, i.e. $\omega_L=\omega$ and $\omega_R=\omega+\delta$, and derive a correction for the drift at as a function of $\alpha(\omega)$, which here is the array end_angle or the polynomial p2. Using the knowledge of the drift for all 5 rotation speed in the array omega, create a polynomial to compute the drift for $\omega\in[0,10]$ and introduce this polynomial into the code below to correct for the drift.

```python
delta = mybot.length/(mybot.radius*T)*end_angle
print(end_angle)
print(mybot.length/(mybot.radius*T))
print(delta)
print(R2D(p2(2)))
```

    [ 0.34574835  0.9517579   4.88811036  5.34589337  7.02843155]
    [ 2.          1.33333333  1.          2.          1.33333333]
    [  0.69149669   1.26901054   4.88811036  10.69178675   9.37124206]
    107.370677846

```python
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_L = omega
omega_R = omega + drift(omega) # DO NOT MODIFY
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] - delta[i] #write your correction here

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(10) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-1,11)
plt.ylim(-1,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

Now apply your correction to the following example. In this example the robot experiences 5 step motions of different durations and speed. The robot is supposed to move in a straight line. Print the angle of the robot at the end of each step. Does your correction work? What would it take the correction to work for positive AND negative rotation speed?

Write your answers here.

```python
omega = np.array([1.5, 2.4, 10, 3 , 7])
T = np.array([2, 3, 4, 2, 3])
end_angle = np.zeros(5)
omega_L = omega
omega_R = omega + drift(omega)
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations
mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = D2R(0) #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r
for i in range(5):

    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] # write your correction here

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T[i]) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

```python
print(R2D(end_angle))
```

Create a complex motion (with rotations and speed variations). Show the result with and without correction. At minimum the motion should include one rotation and two different wheel rotation speed. See example below

```python
Nmoves = 2 # make sure to increase this number if you increase the number of moves
omega = np.array([6, 9])
T = np.array([12,8])
rotation_angle = np.array([90, 90])
rotation_angle = D2R(rotation_angle)
end_angle = np.zeros(Nmoves)
omega_L = omega
omega_R = omega + drift(omega)
t_rotation = rotation_angle*mybot.length/(mybot.radius*(omega_R+omega_L))
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations
mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = D2R(0) #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r
mybot.plot = True #True if you want to plot the robot's trajectory
mybot.plot_robot() #draw an arrow for the location of the robot at t=0
for i in range(Nmoves):

    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] # write your correction here

    mybot.move(T[i]) #move the robot
    # rotation
    mybot.rt_spd_left = -omega_L[i]
    mybot.rt_spd_right = omega_R[i] # write your correction here
    mybot.move(t_rotation[i]) #move the robot
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

```python

```


**Nguyễn Tiến Dũng**

KSTN Toán Tin - K62

*20170062*

Đại học Bách khoa Hà Nội

```python
import processviz as pvz
# import sympy as sy

# sy.init_printing()
```

**Câu 1:**

*a.* Tính $h_1, h_2$

Trước hết ta có $h_{00} = 1$

Gọi $h_{i0}$ là xác suất xuất phát từ trạng thái $i$ có thể đến được trạng thái $0$. Khi đó, xuất phát từ trạng thái $1$, nhận thấy rằng có $p_{1i}$ khả năng trạng thái 1 chuyển đến trạng thái $i$, và từ trạng thái $i$ đó chuyển đến trạng thái 0. Do đó ta có:

$$
h_{10} = 0.2h_{00} + 0.4h_{10}+0.2h_{20} + 0.2h_{30}
$$

Tương tự, ta có:

$$
\left\{\begin{matrix}
h_{10} = 0.2h_{00} + 0.4h_{10}+0.2h_{20} + 0.2h_{30} \\
h_{20} = 0.3h_{00} + 0.3h_{10} + 0.2h_{20} + 0.2h_{30} \\
h_{30} = 0.5h_{00} + 0.3h_{20} + 0.2h_{30} \\
h_{00} = 1
\end{matrix}\right.
$$

Giải hệ trên ta thu được:

$$
\left\{ \begin{matrix}
h_{10} = 1\\
h_{20} = 1 \\
h_{30} = 1
\end{matrix}\right.
$$

```python
G = pvz.MarkovChain()
G.from_file('./ass5/input_1.csv')
G.generate_graph(1)
```

*b.* Tính $k_{10}, k_{20}$

Dễ thấy $k_{00} = 0$

Kí hiệu $k_{i0}$ là số bước trung bình để đi từ trạng thái $i$ đến trạng thái 0. Thấy rằng từ trạng thái $i$, ta có thể đến trạng thái $j$ mất 1 bước, rồi từ trạng thái $j$ mất trung bình $k_{j0}$ bước để đến trạng thái 0. Từ đó ta có:

$$
k_{10} = 1+0.4k_{10} + 0.2k_{20} + 0.2k_{30} + 0.2k_{00}
$$

Tương tự ta có:

$$
\left\{\begin{matrix}
k_{00} = 0 \\
k_{10} = 1+0.4k_{10} + 0.2k_{20} + 0.2k_{30} + 0.2k_{00} \\
k_{20} = 1 + 0.3k_{00} + 0.3k_{10} + 0.2k_{20} + 0.2k_{30} \\
k_{30} = 1 + 0.5k_{00} + 0.3k_{20} + 0.2k_{30}
\end{matrix}\right.
$$

Giải hệ trên ta thu được

$$
\left\{\begin{matrix}
k_{00} = 0\\
k_{10} = \frac{500}{141} \sim 3.55 \\
k_{20} = \frac{150}{47} \sim 3.19 \\
k_{30} = \frac{115}{47} \sim 2.45
\end{matrix}\right.
$$

**Câu 2:**
*a.* Tư tưởng chính là xây dựng bài toán sử dụng công thức truy hồi. Đầu tiên, ta sẽ tính $P_2$.

Giả sử ma trận $P$ có kích thước $(m+n) \times (m+n)$, trong đó tương ứng các thành phần có kích thước như sau:

- Ma trận đơn vị $I$: $m \times m$
- Ma trận O: $m \times n$
- Ma trận R: $n \times m$
- Ma trận Q: $n \times n$

Ma trận $P^2$ hiển nhiên cũng có thể chia ra thành các phần như trên. Giả sử

$$
P^2 = \begin{pmatrix}
I_2 & |& O_2 \\
 R_2 & |& Q_2
\end{pmatrix}
$$

Ta sẽ chứng minh $I_2 = I$ và $O_2 = O$.

Thật vâỵ, với $P_{ij}^{2} \in I_2$ ta có:

$$
P_{ij}^2 = \sum_{k=1}^{m+n}P_{ik}P_{kj} = \sum_{k=1}^{m}P_{ik}P_{kj} + \sum_{k=m+1}^{m+n}P_{ik}P_{kj}
$$

Do $(i, j)$ nằm trong $I_2$ nên $1 \le i, j \le m$. Thấy rằng:

- Nếu $i=k$ thì $P_{ik} = 1$, mọi i
- Nếu $i \ne k$ thì $P_{ik} = 0$, mọi i

Do đó $\sum_{k=1}^{m}P_{ik}P_{kj} = 1$ nếu $i=j$ và bằng 0 nếu $i \ne j$

Xét $\sum_{k=m+1}^{m+n}P_{ik}P_{kj}$, dễ thấy tổng này bằng 0 do $P_{ik}$ lúc này thuộc phần ma trận $O$.

Từ 2 điều trên, suy ra $I_2 = I$.

Chứng minh hoàn toàn tương tự ta cũng có $O_2 = O$.

---

> Gọi $X_{c(k)}$ là cột thứ $k$ của ma trận $X$; $X_{r(k)}$ là hàng thứ $k$ của ma trận $X$.

---

Ta tính các vị trí $R_2$.

Ta có:

$$
P_{ij}^2 = \sum_{k=1}^{m+n}P_{ik}P_{kj} = \sum_{k=1}^{m}P_{ik}P_{kj} + \sum_{k=m+1}^{m+n}P_{ik}P_{kj}
$$

Xét $(i, j) \in R_2$.

Xét tổng $\sum_{k=1}^{m}P_{ik}P_{kj}$. Dễ thấy với $k = j$ thì $P_{kj} = 1$ và $P_{kj} = 0$ nếu $k \ne j$. Do đó $\sum_{k=1}^{m}P_{ik}P_{kj} = P_{ij}$.

Xét $\sum_{k=m+1}^{m+n}P_{ik}P_{kj} = R_{c(k)}.Q_{r(k)}$. Điều này chứng tỏ với $P_{ij}^2 \in R_2$ thì $P_{ij}^2 = P_{ij} + R_{c(j)}Q_{r(i)}$

Do đó: $R_2 = R + Q.R$

---

Tính $Q_2$:

Ta có:

$$
Q_{ij}^2 = \sum_{k=1}^{m+n}P_{ik}P_{kj} = \sum_{k=1}^{m}P_{ik}P_{kj} + \sum_{k=m+1}^{m+n}P_{ik}P_{kj}
$$

Xét $(i, j) \in Q_2$.

Xét tổng $\sum_{k=1}^{m}P_{ik}P_{kj}$. Dễ thấy với $P_{kj} = 0$ với mọi $k$. Do đó $\sum_{k=1}^{m}P_{ik}P_{kj} = 0$.

Xét $\sum_{k=m+1}^{m+n}P_{ik}P_{kj} = Q.Q = Q^2$. Điều này chứng tỏ với $P_{ij}^2 \in Q_2$ thì $Q_{ij}^2 = Q_{r(i)}Q_{c(j)}$, hay $Q_2 = Q^2$

---

Lập luận hoàn toàn tương tự, ta có thể chứng minh được ma trận $P^n$ như sau:

$$
P^n = \begin{pmatrix}
I & |& O \\
 R_n & |& Q_n
\end{pmatrix}
$$

trong đó:
- $Q_n = Q^n$
- $R_n = R + Q^{n-1}*R$

---

**Remark:** Đến đây thì em có thể lập trình được chương trình tính ma trận $P^n$ có dạng này.

---

**Câu 3:** Mê cung có dạng như sau:

|   |   |   |
|---|---|---|
| 1 | 2 | 3 |
| 4 | 5 | 6 |
| 7 | 8 | 9 |

---

*a.* Xét $$P(X_{n+1} = i_{n+1}|X_n = i_n,...,X_0 = i_0) = \frac{P(X_{n+1} = i_{n+1},X_n = i_n,...,X_0 = i_0)}{P(X_n = i_n,...,X_0 = i_0)} = \frac{P(X_{n+1} = i_{n+1},X_n = i_n|X_{n-1} = i_{n-1},...,X_0 = i_0)}{P(X_n = i_n|X_{n-1}=i_{n-1}...,X_0 = i_0)}$$
$$
= \frac{P(X_{n+1} = i_{n+1}|X_n = i_n)P(X_n = i_n|X_{n-1} = i_{n-1},...,X_0 = i_0)}{P(X_n = i_n|X_{n-1}=i_{n-1}...,X_0 = i_0)} = P(X_{n+1} = i_{n+1}|X_n = i_n)
$$

chứng tỏ $(X_n)$ là xích Markov.

*b.* Với sơ đồ mê cung như trên ta có ma trận xác suất chuyển như sau

$$
P = \left(\begin{matrix}
0 & 0.5 & 0 &0.5&0&0&0&0&0 \\
\frac{1}{3} & 0& \frac{1}{3} & 0 &\frac{1}{3}&0&0&0& 0\\
0 & 0.5 & 0 & 0 & 0 & 0.5&0&0&0 \\
\frac{1}{3}&	0	&0	&0&	\frac{1}{3}&	0	&\frac{1}{3}&	0	&0 \\
0	&0.25&	0	&0.25&	0	&0.25&	0 &	0.25&	0 \\
0&	0&	\frac{1}{3}&	0&	\frac{1}{3}&	0&	0&	0&	\frac{1}{3} \\
0&	0&	0&	0.5&	0&	0&	0&	0.5&	0 \\
0&	0&	0	&0&	\frac{1}{3}&	0&	\frac{1}{3}&	0&	\frac{1}{3} \\
0&	0&	0&	0&	0&	0&	0&	0&	1
\end{matrix}\right)
$$

---

*c.* Gọi $k_i$ là thời gian trung bình con chuột rơi vào bẫy. Ta cần tính $k_1$.

Từ trạng thái 1, con chuột có thể tới được các trạng thái $2, 4$ sau 1 bước rồi từ đó tốn tương ứng trung bình $k_2, k_4$ bước để đến trạng thái $9$.

Ta có quan hệ $k_1 = 1 + 0.5(k_2 + k_4)$. Lập luận tương tự ta thu được:

$$
\left\{\begin{matrix}
k_1 = 1 + 0.5(k_2 + k_4) \\
k_2 = 1 + \frac{1}{3}(k_5  + k_1 + k_3) \\
k_3 = 1 + 0.5(k_2 + k_6) \\
k_4 = 1 + \frac{1}{3}(k_1 + k_5 + k_7) \\
k_5 = 1 + \frac{1}{4}(k_2 + k_4 + k_6 + k_8) \\
k_6 = 1 + \frac{1}{3}(k_3 + k_5 + k_9) \\
k_7 = 1 + 0.5(k_4 + k_8)\\
k_8 = 1 + \frac{1}{3}(k_5 + k_7 + k_9) \\
k_9 = 0
\end{matrix}\right.
$$

Hệ phương trình trên có thể viết lại thành:

$$
Ak = b
$$

trong đó $A = I - P, k = (k_1, k_2,...,k_8,0)^T, b=(1,1,1,1,1,1,1,1,0)^T$

Giải hệ trên ta được $k = (17.88, 16.88, 14.90, 16.87, 14.90,
       10.92, 14.90, 10.92, 0)^T$

```python
G3 = pvz.MarkovChain()
G3.from_file('./ass5/input_2.csv')
G3.generate_graph()
```

**Câu 4:** Xét $(X_n)$ với $X_n$ là số điểm ở lần gieo thứ $n$. Dễ thấy $(X_n)$ là xích Markov.

Ta tính xác suất gieo được $k$ điểm, $2\le k \le 12$.

Gọi số điểm trên con thứ nhất và thứ hai lần lượt là $x_1, x_2$. Ta có

$$
x_1 + x_2 = k
$$

Không gian mẫu $6 \times 6 = 36$. Không gian trạng thái $I = \{2,3,4,...,12\}$

Từ đó ta có bảng phân phối xác suất như sau:

---

```python
G2 = pvz.MarkovChain()
G2.from_file('./ass5/input_3.csv')
G2.generate_graph()
G2.data
```

```python
G2.get_mean_time(['10'])
```

    {'Mean time spent of 2': 12.0,
     'Mean time spent of 3': 12.0,
     'Mean time spent of 4': 12.0,
     'Mean time spent of 5': 12.0,
     'Mean time spent of 6': 12.0,
     'Mean time spent of 7': 12.0,
     'Mean time spent of 8': 12.0,
     'Mean time spent of 9': 12.0,
     'Mean time spent of 11': 12.0,
     'Mean time spent of 12': 12.0}

**Câu 5:**

Xét ma trận xác suất chuyển $P$, không gian trạng thái $I$ và tập các trạng thái hút $S$. Ta có:

$$
P_{ii} = 1, \forall i \in S
$$

Thời gian trung bình được xác định như sau;

$$
\left\{\begin{matrix}
k_i = 0, \forall i \in S\\
k_i = 1+\sum_{j \in I / S}{P_{ij}k_j}
\end{matrix}\right.
$$

Chuyển vế ta được

$$
\left\{\begin{matrix}
k_i = 0, \forall i \in S\\
(1-P_{ii})k_i - \sum_{j \ne i \in I / S}{P_{ij}k_j} = 1
\end{matrix}\right.
$$

Hay ta được hệ phương trình viết lại dưới dạng ma trận như sau:

$$
A_1k = b_1
$$

trong đó $A_1 = I - P, b_1 = (b_{1i})$ thỏa mãn $b_{1i} = 0$ nếu $i \in S$ và $b_{1i} = 1$ nếu $i \notin S$.

Dễ thấy rằng $A_1$ là một `singular matrix`, do đó ta sẽ loại bớt tất cả những hàng $i$ có giá trị bằng 0 và cột $i$ tương ứng để được ma trận $A$.

Thực hiện tương tự với $b_1$, ta xóa hết tất cả các hàng có số $0$.

```python
def get_mean_time(self, source=None, target=None, type='transient'):
    try:
        state = mt.get_transient_state(self.state, self.P)
        matrix = mt.get_mean_time_transient(self.state, self.P)
        if type == 'absoring':
            return state, (mt.get_mean_time_absoring(self.state, self.P)).tolist()
        elif type == 'transient':
            if source == None and target == None:
                return state, matrix
            elif source == None:
                return state, (np.transpose(matrix)).tolist()[state.index(target)]
            elif target == None:
                return state, (matrix[state.index(source)]).tolist()
            else:
                return state, matrix[state.index(source)][state.index(target)]
    except:
        return "Invalid"
```

Thực hiện giải hệ phương trình trên bằng cách nhân với nghịch đảo của ma trận $A$. Kết quả trả về là dictionary tương ứng giữa `mean time spent` và trạng thái.

---

Thực hiện chạy với ví dụ của **câu 3**. Ta có:

```python
G3.get_mean_time(['9'])
```

    {'Mean time spent of 1': 17.87,
     'Mean time spent of 2': 16.87,
     'Mean time spent of 3': 14.9,
     'Mean time spent of 4': 16.87,
     'Mean time spent of 5': 14.9,
     'Mean time spent of 6': 10.92,
     'Mean time spent of 7': 14.9,
     'Mean time spent of 8': 10.92}

**Remark:** Tạm thời em đang để làm tròn đến 2 chữ số thập phân.