```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 15: Prescott's Real Business Cycle Model II

In this notebook, we continue to examine the centralized version of the model from pages 11-17 in Edward Prescott's article "Theory Ahead of Business Cycle Measurement in the Fall 1986 of the Federal Reserve Bank of Minneapolis' *Quarterly Review* (link to article: [https://www.minneapolisfed.org/research/qr/qr1042.pdf](https://www.minneapolisfed.org/research/qr/qr1042.pdf)). In this notebook, we:

1. Look at the effect of changing $\rho$ (the autoregressive coefficient on log TFP) on the simulated impulse responses of model variables to a TFP shock
2. Compute a stochastic simulation of the model and compute summary statistics.

## Example: Effect of Changing $\rho$ on Impulse Responses

Recall that the equilibrium conditions for Prescott's RBC model are:

\begin{align}
\frac{1}{C_t} & = \beta E_t \left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1}L_{t+1}^{1-\alpha} +1-\delta }{C_{t+1}}\right]\\
\frac{\varphi}{1-L_t} & = \frac{(1-\alpha)A_tK_t^{\alpha}L_t^{-\alpha}}{C_t} \\
Y_t & = A_t K_t^{\alpha}L_t^{1-\alpha}\\
K_{t+1} & = I_t + (1-\delta) K_t\\
Y_t & = C_t + I_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$.

The objective is use `linearsolve` to simulate impulse responses to a TFP shock using the following parameter values for the simulation for $\rho = 0.5,0.75,0.9,0.99$. Other parameter values are given in the table below:

| $$\sigma$$ | $$\beta$$ | $$\varphi$$ | $$\alpha$$ | $$\delta $$ |
|------------|-----------|-------------|------------|-------------|
| 0.006      | 0.99      | 1.7317      | 0.35       |  0.025      |

## Model Preparation

As usual, we recast the model in the form required for `linearsolve`. Write the model with all variables moved to the left-hand side of the equations and dropping the expecations operator $E_t$ and the exogenous shock $\epsilon_{t+1}$:

\begin{align}
0 & = \beta\left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1}L_{t+1}^{1-\alpha} +1-\delta }{C_{t+1}}\right] - \frac{1}{C_t}\\
0 & = \frac{(1-\alpha)A_tK_t^{\alpha}L_t^{-\alpha}}{C_t} - \frac{\varphi}{1-L_t}\\
0 & = A_t K_t^{\alpha}L_t^{1-\alpha} - Y_t\\
0 & = I_t + (1-\delta) K_t - K_{t+1}\\
0 & = C_t + I_t - Y_t\\
0 & = \rho \log A_t - \log A_{t+1}
\end{align}

Remember, capital and TFP are called *state variables* because they're $t+1$ values are predetermined. Output, consumption, and investment are called a *costate* or *control* variables. Note that the model as 5 equations in 5 endogenous variables.

## Initialization, Approximation, and Solution

The next several cells initialize the model in `linearsolve` and then approximate and solve it.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series. CELL PROVIDED
# Note that a value for rho is absent
parameters = pd.Series(dtype=float)

parameters['beta'] = 0.99
parameters['phi'] = 1.7317
parameters['alpha'] = 0.35
parameters['delta'] = 0.025

# Print the model's parameters
print(parameters)
```

    beta     0.9900
    phi      1.7317
    alpha    0.3500
    delta    0.0250
    dtype: float64

```python
# Create a variable called 'sigma' that stores the value of sigma
sigma = 0.006
```

```python
# Create variable called 'varNames' that stores the variable names in a list with state variables ordered first. CELL PROVIDED
var_names = ['a','k','y','c','i','l']

# Create variable called 'shockNames' that stores an exogenous shock name for each state variable. CELL PROVIDED
shock_names = ['e_a','e_k']
```

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. CELL PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Euler equation
    mpk = p.alpha*fwd.a*fwd.k**(p.alpha-1)*fwd.l**(1-p.alpha)
    euler_equation = p.beta*(mpk+1-p.delta)/fwd.c - 1/cur.c

    # Labor-labor choise
    mpl = (1-p.alpha)*cur.a*cur.k**p.alpha*cur.l**(-p.alpha)
    labor_leisure = mpl/cur.c - p.phi/(1-cur.l)

    # Production function
    production_function = cur.a*cur.k**p.alpha*cur.l**(1-p.alpha) - cur.y

    # Capital evolution
    capital_evolution = cur.i + (1 - p.delta)*cur.k - fwd.k

    # Market clearing
    market_clearing = cur.c+cur.i - cur.y

    # Exogenous tfp
    tfp_process = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        euler_equation,
        labor_leisure,
        production_function,
        capital_evolution,
        market_clearing,
        tfp_process
        ])
```

Next, initialize the model using `ls.model` which takes the following required arguments:

* `equations`
* `n_states`
* `var_names`
* `shock_names`
* `parameters`

```python
# Initialize the model into a variable named 'rbc_model'. CELL PROVIDED.
rbc_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters=parameters)
```

### SImulation and Plotting

The objective is to create a $2\times 2$ grid of plots containing the impulse responses of TFP, output, labor, and consumption to a one percent shock to TFP for each of the values for $\rho$: 0.5,0.75,0.9,0.99. Here are the steps that we'll take:

1. Initilize figure and axes for plotting.
2. Iterate over each desired value for $\rho$.
    1. Set `rbc_model.parameters['rho']` equal to current value of $\rho$.
    2. Use `rbc_model.compute_ss()` to compute the steady state with `guess` equal to `[1,4,1,1,1,0.5]`.
    3. Use `rbc_model.approximate_and_solve()` to approximate and solve the model with the current value of $\rho$.
    4. Use `rbc_model.impulse()` to compute the 51 period impulse response to a 0.01 unit shock to TFP in period 5.
    5. Add the computed impulse responses to the axes.

```python
# Create a 12x8 figure
fig = plt.figure(figsize=(12,8))

# Create four axis variables: 'ax1', 'ax2', 'ax3', 'ax4'
ax1 = fig.add_subplot(2,2,1)
ax2 = fig.add_subplot(2,2,2)
ax3 = fig.add_subplot(2,2,3)
ax4 = fig.add_subplot(2,2,4)

# Create an axis equal to the size of the figure. PROVIDED
ax0 = fig.add_subplot(1,1,1)
# Turn off the axis so that the underlying axes are visible. PROVIDED
ax0.set_frame_on(False)
# Hide the x-axis. PROVIDED
ax0.get_xaxis().set_visible(False)
# Hide the y-axis. PROVIDED
ax0.get_yaxis().set_visible(False)

# Create variable called 'rho_values' that stores the desired values of rho
rho_values = [0.5,0.75,0.9,0.99]

# Iterate over the elements of rho_values
for rho in rho_values:

    # Update the value of rho in rbc_model.parameters
    rbc_model.parameters['rho'] = rho

    # Compute the steady state with initial guess equal to [1,4,1,1,1,0.5]
    rbc_model.compute_ss([1,4,1,1,1,0.5])

    # Approximate the model and solve
    rbc_model.approximate_and_solve()

    # Compute the impulse responses to a 0.01 unit shock to TFP
    rbc_model.impulse(T=51,t0=5,shocks=[0.01,0])

    # Add plots of TFP, output, labor, and consumption to ax1, ax2, ax3, and ax4
    ax1.plot(rbc_model.irs['e_a']['a']*100,lw=3,alpha=0.75)
    ax2.plot(rbc_model.irs['e_a']['y']*100,lw=3,alpha=0.75)
    ax3.plot(rbc_model.irs['e_a']['l']*100,lw=3,alpha=0.75)
    ax4.plot(rbc_model.irs['e_a']['c']*100,lw=3,alpha=0.75)

    # Plot the point 0,0 on ax0 with the same line properties used for the other plotted lines and provide a label
    ax0.plot(0,0,lw=3,alpha=0.75,label='$\\rho='+str(rho)+'$')

# Set axis 1 title. PROVIDED
ax1.set_title('TFP')

# Set axis 2 title. PROVIDED
ax2.set_title('Output')

# Set axis 3 title. PROVIDED
ax3.set_title('Labor')

# Set axis 4 title. PROVIDED
ax4.set_title('Consumption')

# Add grid to the axis 1. PROVIDED
ax1.grid()

# Add grid to the axis 2. PROVIDED
ax2.grid()

# Add grid to the axis 3. PROVIDED
ax3.grid()

# Add grid to the axis 4. PROVIDED
ax4.grid()

# Set ax1 y-axis limits to [0,2]. PROVIDED
ax1.set_ylim([0,2])
# Set ax2 y-axis limits to [0,2]. PROVIDED
ax2.set_ylim([0,2])
# Set ax3 y-axis limits to [-0.5,1.25]. PROVIDED
ax3.set_ylim([-0.5,1.5])
# Set ax4 y-axis limits to [-0.5,1.5]. PROVIDED
ax4.set_ylim([-0.5,1.5])

# Add legend below the figure. PROVIDED
legend = ax0.legend(loc='upper center',bbox_to_anchor=(0.5,-0.075), ncol=4,fontsize=15)
```

## Example: Stochastic Simulation

Compute a 401 period stochastic simulation of the model. Set $\rho=0.75$ to match US business cycle data. And set the seed for the simulation to 126.

```python
# Set the value of rho in rbc_model.parameters to 0.75
rbc_model.parameters['rho'] = 0.75

# Compute the steady state with initial guess equal to [1,4,1,1,1,0.5]
rbc_model.compute_ss([1,4,1,1,1,0.5])

# Approximate the model and solve
rbc_model.approximate_and_solve()
```

```python
# Compute the stochastic simulation using the .stoch_sim() method of rbc_model
rbc_model.stoch_sim(T=401,cov_mat=[[sigma**2,0],[0,0]],seed=126)
```

On a single axis, plot the simulated values for output, consumption, investment, and labor.

```python
# Create 12x4 figure. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output, consumption, investment, labor
rbc_model.simulated[['y','c','i','l']].plot(ax=ax,lw=3,alpha=0.75)

# Add legend outside plot. PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.grid()
```

```python
# Print the standard deviations (times 100) of output, consumption, investment, and labor in the simulated data
rbc_model.simulated[['y','c','i','l']].std()*100
```

    y    1.629103
    c    0.646561
    i    5.555957
    l    0.888973
    dtype: float64

```python
# Print the correlations of output, consumption, investment, and labor in the simulated data
rbc_model.simulated[['y','c','i','l']].corr()
```

      y
      c
      i
      l

      y
      1.000000
      0.614215
      0.960996
      0.923899

      c
      0.614215
      1.000000
      0.372012
      0.265520

      i
      0.960996
      0.372012
      1.000000
      0.993686

      l
      0.923899
      0.265520
      0.993686
      1.000000


# Section 0 问题描述与完成项目流程

## 1. 问题描述

</img>

在该项目中，你将使用强化学习算法，实现一个自动走迷宫机器人。

1. 如上图所示，智能机器人显示在右上角。在我们的迷宫中，有陷阱（红色炸弹）及终点（蓝色的目标点）两种情景。机器人要尽量避开陷阱、尽快到达目的地。
2. 小车可执行的动作包括：向上走 `u`、向右走 `r`、向下走 `d`、向左走 `l`。
3. 执行不同的动作后，根据不同的情况会获得不同的奖励，具体而言，有以下几种情况。
    - 撞到墙壁：-10
    - 走到终点：50
    - 走到陷阱：-30
    - 其余情况：-0.1
4. 我们需要通过修改 `robot.py` 中的代码，来实现一个 Q Learning 机器人，实现上述的目标。

## 2. 完成项目流程

1. 配置环境，使用 `envirnment.yml` 文件配置名为 `robot-env` 的 conda 环境，具体而言，你只需转到当前的目录，在命令行/终端中运行如下代码，稍作等待即可。
```
conda env create -f envirnment.yml
```
安装完毕后，在命令行/终端中运行 `source activate robot-env`（Mac/Linux 系统）或 `activate robot-env`（Windows 系统）激活该环境。

2. 阅读 `main.ipynb` 中的指导完成项目，并根据指导修改对应的代码，生成、观察结果。
3. 导出代码与报告，上传文件，提交审阅并优化。
---

---

# Section 1 算法理解

## 1. 1 强化学习总览

强化学习作为机器学习算法的一种，其模式也是让智能体在“训练”中学到“经验”，以实现给定的任务。但不同于监督学习与非监督学习，在强化学习的框架中，我们更侧重通过智能体与环境的**交互**来学习。通常在监督学习和非监督学习任务中，智能体往往需要通过给定的训练集，辅之以既定的训练目标（如最小化损失函数），通过给定的学习算法来实现这一目标。然而在强化学习中，智能体则是通过其与环境交互得到的奖励进行学习。这个环境可以是虚拟的（如虚拟的迷宫），也可以是真实的（自动驾驶汽车在真实道路上收集数据）。

在强化学习中有五个核心组成部分，它们分别是：**环境（Environment）**、**智能体（Agent）**、**状态（State）**、**动作（Action）**和**奖励（Reward）**。在某一时间节点 $t$：

- 智能体在从环境中感知其所处的状态 $s_t$
- 智能体根据某些准则选择动作 $a_t$
- 环境根据智能体选择的动作，向智能体反馈奖励 $r_{t+1}$

通过合理的学习算法，智能体将在这样的问题设置下，成功学到一个在状态 $s_t$ 选择动作 $a_t$ 的策略 $\pi (s_t) = a_t$。

---

**问题 1**：请参照如上的定义，描述出 “机器人走迷宫这个问题” 中强化学习五个组成部分对应的实际对象：

- **环境** : 迷宫环境
- **状态** : 撞到墙壁，踩到陷阱，到达终点，其余情况

- **动作** : 前后左右
- **奖励** : -30，-10，-0.1，+50

$$T(s^{'}, a, s) = P(s^{'}|a,s)$$

---

## 1.2 计算 Q 值

在我们的项目中，我们要实现基于 Q-Learning 的强化学习算法。Q-Learning 是一个值迭代（Value Iteration）算法。与策略迭代（Policy Iteration）算法不同，值迭代算法会计算每个”状态“或是”状态-动作“的值（Value）或是效用（Utility），然后在执行动作的时候，会设法最大化这个值。因此，对每个状态值的准确估计，是我们值迭代算法的核心。通常我们会考虑**最大化动作的长期奖励**，即不仅考虑当前动作带来的奖励，还会考虑动作长远的奖励。

在 Q-Learning 算法中，我们把这个长期奖励记为 Q 值，我们会考虑每个 ”状态-动作“ 的 Q 值，具体而言，它的计算公式为：

$$
q(s_{t},a) = R_{t+1} + \gamma \times\max_a q(a,s_{t+1})
$$

也就是对于当前的“状态-动作” $(s_{t},a)$，我们考虑执行动作 $a$ 后环境给我们的奖励 $R_{t+1}$，以及执行动作 $a$ 到达 $s_{t+1}$后，执行任意动作能够获得的最大的Q值 $\max_a q(a,s_{t+1})$，$\gamma$ 为折扣因子。

不过一般地，我们使用更为保守地更新 Q 表的方法，即引入松弛变量 $alpha$，按如下的公式进行更新，使得 Q 表的迭代变化更为平缓。

$$
q(s_{t},a) = (1-\alpha) \times q(s_{t},a) + \alpha \times(R_{t+1} + \gamma \times\max_a q(a,s_{t+1}))
$$

---
</img>

**问题 2**：根据已知条件求 $q(s_{t},a)$，在如下模板代码中的空格填入对应的数字即可。

已知：如上图，机器人位于 $s_1$，行动为 `u`，行动获得的奖励与题目的默认设置相同。在 $s_2$ 中执行各动作的 Q 值为：`u`: -24，`r`: -13，`d`: -0.29、`l`: +40，$\gamma$ 取0.9。

$$
\begin{align}
q(s_{t},a) & = R_{t+1} + \gamma \times\max_a q(a,s_{t+1}) \\
 & =(0.1) + (0.9)*(40) \\
 & =(36.1)
\end{align}
$$

---

## 1.3 如何选择动作

在强化学习中，「探索-利用」问题是非常重要的问题。具体来说，根据上面的定义，我们会尽可能地让机器人在每次选择最优的决策，来最大化长期奖励。但是这样做有如下的弊端：
1. 在初步的学习中，我们的 Q 值会不准确，如果在这个时候都按照 Q 值来选择，那么会造成错误。
2. 学习一段时间后，机器人的路线会相对固定，则机器人无法对环境进行有效的探索。

因此我们需要一种办法，来解决如上的问题，增加机器人的探索。由此我们考虑使用 epsilon-greedy 算法，即在小车选择动作的时候，以一部分的概率随机选择动作，以一部分的概率按照最优的 Q 值选择动作。同时，这个选择随机动作的概率应当随着训练的过程逐步减小。

---

**问题 3**：在如下的代码块中，实现 epsilon-greedy 算法的逻辑，并运行测试代码。

```python
import random
import numpy as np

actions = ['u','r','d','l']
qline = {'u':1.2, 'r':-2.1, 'd':-24.5, 'l':27}
epsilon = 0.3 # 以0.3的概率进行随机选择

def choose_action(actions,qline,epsilon):

    action = None
    if random.random()<epsilon: # 以某一概率
        action=np.random.choice(actions) # 实现对动作的随机选择
    else:
        action=max(qline,key=qline.get) # 否则选择具有最大 Q 值的动作
    return action
```

```python
action=choose_action(actions,qline,epsilon)
print(action)
```

    l

---

---

# Section 2 代码实现

## 2.1. `Maze` 类理解

我们首先引入了迷宫类 `Maze`，这是一个非常强大的函数，它能够根据你的要求随机创建一个迷宫，或者根据指定的文件，读入一个迷宫地图信息。

1. 使用 `Maze("file_name")` 根据指定文件创建迷宫，或者使用 `Maze(maze_size=(height,width))` 来随机生成一个迷宫。
2. 使用 `trap_number` 参数，在创建迷宫的时候，设定迷宫中陷阱的数量。
3. 直接键入迷宫变量的名字按回车，展示迷宫图像（如 `g=Maze("xx.txt")`，那么直接输入 `g` 即可。
4. 建议生成的迷宫尺寸，长在 6~12 之间，宽在 10～12 之间。

---

**问题 4**：在如下的代码块中，创建你的迷宫并展示。

```python
from Maze import Maze
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

## todo: 创建迷宫并展示
mario=Maze(maze_size=(8,12),trap_number=2)
mario
```

---

你可能已经注意到，在迷宫中我们已经默认放置了一个机器人。实际上，我们为迷宫配置了相应的 API，来帮助机器人的移动与感知。其中你随后会使用的两个 API 为 `maze.sense_robot()` 及 `maze.move_robot()`。

1. `maze.sense_robot()` 为一个无参数的函数，输出机器人在迷宫中目前的位置。
2. `maze.move_robot(direction)` 对输入的移动方向，移动机器人，并返回对应动作的奖励值。

---

**问题 5**：随机移动机器人，并记录下获得的奖励，展示出机器人最后的位置。

```python
rewards = []

## 循环、随机移动机器人10次，记录下奖励
for i in range(10):
    action=np.random.choice(actions)
    rewards.append(mario.move_robot(action))

## 输出机器人最后的位置
print(mario.sense_robot())
print(rewards)

## 打印迷宫，观察机器人位置
```

    (0, 10)
    [-0.1, -10.0, -0.1, -10.0, -0.1, -0.1, -0.1, -10.0, -0.1, -0.1]

## 2.2. `Robot` 类实现

`Robot` 类是我们需要重点实现的部分。在这个类中，我们需要实现诸多功能，以使得我们成功实现一个强化学习智能体。总体来说，之前我们是人为地在环境中移动了机器人，但是现在通过实现 `Robot` 这个类，机器人将会自己移动。通过实现学习函数，`Robot` 类将会学习到如何选择最优的动作，并且更新强化学习中对应的参数。

首先 `Robot` 有多个输入，其中 `alpha=0.5, gamma=0.9, epsilon0=0.5` 表征强化学习相关的各个参数的默认值，这些在之前你已经了解到，`Maze` 应为机器人所在迷宫对象。

随后观察 `Robot.update` 函数，它指明了在每次执行动作时，`Robot` 需要执行的程序。按照这些程序，各个函数的功能也就明了了。

最后你需要实现 `Robot.py` 代码中的8段代码，他们都在代码中以 `#TODO` 进行标注，你能轻松地找到他们。

---

**问题 6**：实现 `Robot.py` 中的8段代码，并运行如下代码检查效果（记得将 `maze` 变量修改为你创建迷宫的变量名）。

```python
from Robot import Robot
robot = Robot(mario) # 记得将 maze 变量修改为你创建迷宫的变量名
robot.set_status(learning=True,testing=False)
print(robot.update())

mario
```

---

## 2.3 用 `Runner` 类训练 Robot

在实现了上述内容之后，我们就可以开始对我们 `Robot` 进行训练并调参了。我们为你准备了又一个非常棒的类 `Runner`，来实现整个训练过程及可视化。使用如下的代码，你可以成功对机器人进行训练。并且你会在当前文件夹中生成一个名为 `filename` 的视频，记录了整个训练的过程。通过观察该视频，你能够发现训练过程中的问题，并且优化你的代码及参数。

---

**问题 7**：尝试利用下列代码训练机器人，并进行调参。可选的参数包括：

- 训练参数
    - 训练次数 `epoch`
- 机器人参数：
    - `epsilon0` (epsilon 初值)
    - `epsilon`衰减（可以是线性、指数衰减，可以调整衰减的速度），你需要在 Robot.py 中调整
    - `alpha`
    - `gamma`
- 迷宫参数:
    - 迷宫大小
    - 迷宫中陷阱的数量

```python
## 可选的参数：
epoch = 100

epsilon0 = 0.05
alpha = 0.5
gamma = 0.9

maze_size = (6,6)
trap_number = 1
```

```python
from Runner import Runner

g = Maze(maze_size=maze_size,trap_number=trap_number)
r = Robot(g,alpha=alpha, epsilon0=epsilon0, gamma=gamma)
r.set_status(learning=True)

runner = Runner(r, g)
runner.run_training(epoch, display_direction=True)
#runner.generate_movie(filename = "final1.mp4") # 你可以注释该行代码，加快运行速度，不过你就无法观察到视频了。
```

---

使用 `runner.plot_results()` 函数，能够打印机器人在训练过程中的一些参数信息。

- Success Times 代表机器人在训练过程中成功的累计次数，这应当是一个累积递增的图像。
- Accumulated Rewards 代表机器人在每次训练 epoch 中，获得的累积奖励的值，这应当是一个逐步递增的图像。
- Running Times per Epoch 代表在每次训练 epoch 中，小车训练的次数（到达终点就会停止该 epoch 转入下次训练），这应当是一个逐步递减的图像。

---

**问题 8**：使用 `runner.plot_results()` 输出训练结果，根据该结果对你的机器人进行分析。

- 指出你选用的参数如何，选用参数的原因。
- 建议你比较不同参数下机器人的训练的情况。
- 训练的结果是否满意，有何改进的计划。

```python
runner.plot_results()
```

(回答区)

---

**问题 9：** 请将如下的文件打包，提交文件给审阅者。

- `robot.py`
- `robot_maze.ipynb`
- 由 `robot_maze.ipynb` 导出的 `robot_maze.html`


# **[HW8] Practice_6**
1. Install packages
2. PageRank

선형대수 실습은, exercise 혹은 coding 문제풀이에 집중하기 보다는, 수업시간에 배웠던 개념들을 visualize 하면서 이를 폭넓게 이해하는 데에 초점을 맞추고 있습니다. 실습에서 사용한 예시 외에도, 다양한 matrix들을 visualize 하면서 개념을 익혀보시기 바랍니다.

# 1. Install packages

> 필요한 package를 설치하고 import합니다

```python
# visualization을 위한 helper code입니다.
from urllib.request import urlretrieve
URL = 'https://go.gwu.edu/engcomp4plot'
urlretrieve(URL, 'plot_helper.py')

import sys
sys.path.append('../scripts/')

# 다음 세 custom function (1)plot_vector, (2)plot_linear_transformation, (3) plot_linear_transformations
# 을 사용할 것입니다.
from plot_helper import *
```

```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import scipy.linalg
import sympy as sy

sy.init_printing()
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

# 2. PageRank

> 이번 실습에선 Google의 PageRank 알고리즘을 구현하면서, eigenvector가 실제로 어떻게 사용되는지 알아보도록 하겠습니다.

> PageRank는 Google의 검색 알고리즘 중 가장 먼저 사용되었던 알고리즘입니다. PageRank는 web**Page**의 **rank**를 매기기 위해, 해당 webpage와 연결되어 있는 다른 webpage들의 개수가 몇 개인지,  (ii) 연결된 webpage들은 각각 얼마나 중요한 webpage인지, 두 가지를 고려하게 됩니다. PageRank에 깔려있는 가정은, **더 중요한 webpage 일수록, 다른 중요한 webpage들로부터 link가 걸려있을 가능성이 높다**는 것입니다 (https://en.wikipedia.org/wiki/PageRank).

> 어떤 webpage의 점수는, 해당 webpage를 link 건 다른 webpage들의 중요도의 weighted average로 계산합니다. 다음 예시를 통해서 알아봅시다.

Example network of 4 webpages, 출처: Ref. [1]

> 위 그림에선, 4개의 wegbpage가 서로 link를 걸고 있음을 알 수 있습니다 (화살표가 링크를 걸었다는 의미입니다). D의 rank(중요도)는 몇일까요? 위 그림에선, D를 link 건 webpage가 A, C입니다. 따라서 A와 C는 D에게 본인의 rank를 나눠주게 됩니다. 이때 A는 나머지 3개 webpage를 전부 link걸었기 때문에, B,C,D 세 webpage에 1/3씩 본인의 rank를 나눠주게 됩니다. B는 B,C 두 webpage를 link걸었기 때문에, B,C 두 webpage에 1/2씩 본인의 rank를 나눠주게 됩니다.

* Page A : B,C,D에게 본인 rank의 1/3씩 나눠주게 됨
* Page B : C,D에게 본인 rank의 1/2씩 나눠주게 됨
* Page C : Link가 없음
* Page D : A,C에게 본인 rank의 1/2씩 나눠주게 됨

> 위 내용을 행렬로 정리하면, 다음과 같이 표현할 수 있습니다.

$$M = \begin{bmatrix} 0   & 0   & 0 & 1/2 \\
                       1/3 & 0   & 0 & 0  \\
                       1/3 & 1/2 & 0 & 1/2 \\
                       1/3 & 1/2 & 0 & 0 \end{bmatrix}
$$

> 그럼 최종적으로 A,B,C,D의 rank는 어떻게 계산될까요? 최종 rank는 모든 webpage들의 rank가 변하지 않을 때까지 나눠주는 행위를 반복한 다음, 수렴된 값을 최종 rank로 생각하게 됩니다.

$$
\begin{bmatrix} 1/4 \\ 1/4 \\ 1/4 \\ 1/4 \end{bmatrix}
\xrightarrow{M} Mx=\begin{bmatrix} 1/8 \\ 1/12 \\ 1/3 \\ 5/24 \end{bmatrix}\xrightarrow{M} M^2x = \begin{bmatrix} 5/48 \\ 1/24 \\ 3/16 \\ 1/12 \end{bmatrix} \xrightarrow{M} \cdots \xrightarrow{M} M^{n}x \xrightarrow{M} M^{n+1}x
$$

> $M^nx = M^{n+1}x$가 된다면, 더이상 값이 바뀌지 않는다는 의미이고, 그 값이 A,B,C,D 각각의 최종 rank가 됩니다.

> 그런데 문제점이 있습니다. Webpage C는 다른 webpage로 link를 걸지 않기 때문에, rank를 나눠주지 않고 webpage C의 rank는 소멸되어버리기 때문에, 최종적으로 모든 webpage의 rank가 0으로 되어버리게 됩니다 (위의 예시에서 모든 rank의 합이 1, 0.75, 0.4167, ... 으로 점점 주는 것을 알 수 있습니다).

```python
M = np.array([[0,0,0,1/2],[1/3,0,0,0],[1/3,1/2,0,1/2],[1/3,1/2,0,0]])
x = np.array([1/4,1/4,1/4,1/4])
```

```python
print(M@x)
print(M@M@M@M@M@M@M@M@M@M@M@M@M@M@x)
```

    [0.125 0.083 0.333 0.208]
    [0. 0. 0. 0.]

> 따라서, 이 문제를 해결하기 위해 $M$을 약간 수정해야 합니다. 가장 단순한 해결책은, M의 zero-column을 $\begin{bmatrix} 1/4 \\ 1/4 \\ 1/4 \\ 1/4 \end{bmatrix}$로 바꿔버리는 것입니다. 이를 modified matrix, $M'$이라고 합시다. 최종적으로는, 이 $M'$과 모든 entry가 1/4인 행렬과의 가중 평균을 사용하게 됩니다. 이 모든 entry가 1/4인 행렬을 더해주게 되면, 모든 entry가 0 이상이 되고, 이는 어떤 임의의 webpage에서 본인을 포함한 다른 webpage로 향하는 모든 link를 만들어줬다고 생각할 수 있습니다.

$$
  M' = \begin{bmatrix} 0   & 0   & 1/4 & 1/2 \\
                      1/3 & 0   & 1/4 & 0  \\
                      1/3 & 1/2 & 1/4 & 1/2 \\
                      1/3 & 1/2 & 1/4 & 0 \end{bmatrix}
\\
  B = n \times n \text{ matrix with 1/n in every entry} \\
  G = (1-p)\, M^{\prime} + p\, B
$$

> $p$는 *damping factor*인데, 둘 간의 가중치를 정하기 위해 우리가 지정하는 hyper-parameter입니다. 결론적으로는, $G$는 $M$과 다르게 $x$과 non-zero vector로 수렴하게 됩니다. 이를 좀 더 엄밀하게 알고 싶으신 분은, 다음 강의를 참조하시기 바랍니다 (https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/lecture-24-markov-matrices-fourier-series/.)

### Exercise

```python
p = 0.15

modified_M = np.array([[0,0,1/4,1/2],[1/3,0,1/4,0],[1/3,1/2,1/4,1/2],[1/3,1/2,1/4,0]]) #TODO  (M')
B = 1/4*(modified_M) #TODO 4 x 4 matrix with 1/4 in every entry
G = (1-p)*modified_M + p*B #TODO
print(G)
print(x)
```

    [[0.    0.    0.222 0.444]
     [0.296 0.    0.222 0.   ]
     [0.296 0.444 0.222 0.444]
     [0.296 0.444 0.222 0.   ]]
    [0.25 0.25 0.25 0.25]

> 위 내용을 참조하여, PageRank algorithm을 구현해 봅시다. Parameter로 $M'$, damping factor $p$, tolerance $\epsilon$를 받아서, 최종 수렴된 vector를 return하면 됩니다. 위 설명처럼 $G^nx=G^{n+1}x$가 됐다면 rank vector가 수렴되었음을 의미하는데, 둘의 차가 정확히는 같지 않더라도 $\epsilon$ 이하라면 수렴되었다고 간주합니다.

$$ ||G^{n+1}x - G^nx||^2_2 < \epsilon \rightarrow \text{ rank vector is converged to }G^nx $$

```python
def pagerank(modified_M,G,x, p=0.15, eps=1e-3):
    '''
    G를 수렴될 때 까지 여러번 곱해서, 수렴된 vector v를 구합니다.
    Returns: v (normalized) numpy array, PageRank vector
    '''
    # TODO

    A = G @ x
    B = G @ A
    C = G @ B
    print(C)

    # return G, v/np.sum(v)
pagerank(modified_M,G,x)
```

    [0.147 0.119 0.258 0.176]

```python
pagerank(modified_M)
```

> 실제로 G를 여러번 곱했을 때, 해당 값으로 수렴함을 확인할 수 있습니다.

```python
G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@x
```

    array([0.02 , 0.015, 0.034, 0.023])

> 즉 세 번쨰, C가 가장 높은 rank를 획득했음을 알 수 있습니다. B는 가장 낮은 rank를 획득하였습니다. 각 rank는 임의의 user가 각 page에 도달할 확률이라고 해석할 수 있습니다.

### Exercise

> 다음 network에 pagerank 알고리즘을 적용하여 최종 rank vector를 구하시오 (damping factor $p$=0.15).

Example network of 8 webpages, 출처: Ref. [2]

```python
p = 0.15

new_M = np.array([[0,0,0,0,0,0,1/3,0],
                  [1/2,0,1/2,1/2,0,0,0,0],
                  [1/2,0,0,0,0,0,0,0],
                  [0,1,0,0,0,0,0,0],
                  [0,0,1/2,0,0,0,1/3,0],
                  [0,0,0,1/2,1/3,0,0,1/2],
                  [0,0,0,0,1/3,0,0,1/2],
                  [0,0,0,0,1/3,1,1/3,0]])

x = np.array([1/8,1/8,1/8,1/8,1/8,1/8,1/8,1/8])

B = 1/8 * (new_M) # 8 x 8 matrix with 1/8 in every entry
G = (1-p)*new_M + p*B #TODO
rank_vector = G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@x

print('최종 rank vector : ', rank_vector)
```

    최종 rank vector :  [0.003 0.005 0.002 0.005 0.004 0.013 0.01  0.018]

> 위와 같이 수렴된 vector는, 사실 eigenvector를 이용하면 쉽게 풀 수 있습니다. 첫 번쨰 예시 G에서는 $G^nx = G^{n+1}x$을 만족하는 $G^nx$를 찾으면 됐습니다. 이때 $G^nx = v$라고 하면, 이는 $v = Gv$가 됩니다. 즉, $v$는 $G$의 eigenvalue가 1인 eigenvector가 됩니다.

```python
eigenvalues, eigenvectors = np.linalg.eig(G)
eigenvalues
```

    array([ 0.887+0.j   , -0.28 +0.244j, -0.28 -0.244j, -0.106+0.j   ])

> 이 중 eigenvalue가 1인, 첫번째 eigenvector는 위에서 구한 rank vector와 같음을 알 수 있습니다.

```python
eigenvectors
```

    array([[ 0.414+0.j   ,  0.598+0.j   ,  0.598-0.j   ,  0.598+0.j   ],
           [ 0.316+0.j   , -0.262-0.5j  , -0.262+0.5j  , -0.075+0.j   ],
           [ 0.71 +0.j   ,  0.082+0.344j,  0.082-0.344j, -0.762+0.j   ],
           [ 0.474+0.j   , -0.418+0.156j, -0.418-0.156j,  0.238+0.j   ]])

```python
rank_vector = G @ eigenvectors[:,0]
rank_vector = rank_vector / rank_vector.sum()
rank_vector
```

    array([0.216+0.j, 0.165+0.j, 0.371+0.j, 0.247+0.j])

> j는 허수 부분을 나타내는데, 계수가 0이므로 무시하면 됩니다. 즉, $G$의 eigenvalue가 1인 eigenvector는 $G$를 여러번 곱해서 얻어진 rank_vector와 같음을 알 수 있습니다.

# References

1. Dan Margalit and Joseph Rabinoff (2018), Interactive Linear Algebra, an open book under the GNU Free Documentation License, Version 1.2.
2. David Austin (2019), Understanding Linear Algebra, an open book under CC-By license.
3. https://github.com/jclosure/EngComp4_landlinear

02_Linear_Alegebra/my_sol/[HW8] LCS Practice 6 answer.ipynb


## Reinforcement Learning Tutorial -3: DDPG

### MD Muhaimin Rahman
contact: sezan92[at]gmail[dot]com

In the last tutorial, I tried to Explain DQN. DQN solves one problem, that is it can deal with continuous state space. But it cannot output continuous action. To solve that problem, here comes DDPG! It means Deep Deterministic Policy Gradient

- [Importing Libraries](#libraries)
- [Algorithm](#algorithm)
- [Model Definition](#model)
- [Replay Buffer](#buffer)
- [Noise Class](#noise)
- [Training](#training)

### Importing Libraries

```python
from __future__ import print_function,division
import gym
import keras
from keras import layers
from keras import backend as K
from collections import deque
from tqdm import tqdm
import random
import numpy as np
import copy
SEED =123
np.random.seed(SEED)
```

    /home/sezan92/anaconda/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters
    Using TensorFlow backend.

Important constants

```python
num_episodes = 100
steps_per_episode=500
BATCH_SIZE=256
TAU=0.001
GAMMA=0.95
actor_lr=0.0001
critic_lr=0.001
SHOW= False
```

```python
from keras.models import Model
```

### Algorithm

The actual algorithm was developed by Timothy lilicap et al. The algorithm is an actor-critic based algorithm.. Which means, it has two networks to train- an actor network, which predicts action based on the current state. The other networ- known as Critic network- evaluates the state and action. This is the case for all actor-critic networks. The critic network is updated using Bellman Equation like DQN. The difference is the training of actor network. In DDPG , we train Actor network by trying to get the maximum value of gradient of $Q(s,a)$ for given action $a$ in a state $s$. In a normal machine learning classification and regression algorithm, our target is to get the value with minimum loss. Then we train the network by gradient descent technique using the gradient of Loss .

\begin{equation}
\theta \gets \theta - \alpha \frac{\partial L}{\partial \theta}
\end{equation}

Here, $\theta$ is the weight parameter of the network, and $L$ is loss

But in our case, we have to get the maximize the $Q$ value. So we have to set the weight parameters such that we get the maximum $Q$ value. This technique is known as Gradient Ascent, as it does the exact opposite of Gradient Descent

\begin{equation}
\theta_a \gets \theta_a - \alpha (-\frac{\partial Q(s,a) }{\partial \theta_a})
\end{equation}

The above equation looks like the actual Gradient Descent equation. Only difference is , the minus sign. It makes the equation to minimize the negative value of $Q$ , which in turn maximizes $Q$ value.

So the training is as following

- 1) Define Actor network $actor$ and Critic Network $critic$
- 2) Define Target Actor and Critic Networks - $actor_{target}$ and $critic_{target}$ with exact same weights
- 3) Initialize Replay Buffer
- 4) Get the initial state , $state$
- 5) Get the action $a$ from , $a \gets actor(state)$ + Noise .[Here Noise is given to make the process stochastic and not so deterministic. The paper uses ornstein uhlenbeck noise process , so we will as well]
- 6) Get Next state $state_{next}$ , Reward $r$ , Terminal from agent for given $state$ and $action$
- 7) Add the experience , $state$,$action$,$reward$,$state_{next}$,$terminal$ to replay buffer
- 8) Get sample minibatch from Replay buffer
- 9) Train Critic Network Using Bellman Equation. Like DQN
- 10) Train Actor Network using Gradient Ascent with gradients of $Q$ . $\theta_a \gets \theta_a - \alpha (-\frac{\partial Q(s,a) }{\partial \theta_a}) $
- 11) Update weights of $actor_{target}$ and $critic_{target}$ using the equation $ \theta \gets \tau \theta + (1-\tau)\theta_{target}$

### Model Definition

After some trials and errors, I have selected this network. The Actor Network is 3 layer MLP with 320 hidden nodes in each layer. The critic network is also a 3 layer MLP with 640 hidden nodes in each layer.Notice that the return arguments of function ```create_critic_network```.

```python
def create_actor_network(state_shape,action_shape):
    in1=layers.Input(shape=state_shape,name="state")
    l1 =layers.Dense(320,activation="relu")(in1)
    l2 =layers.Dense(320,activation="relu")(l1)
    l3 =layers.Dense(320,activation="relu")(l2)
    action =layers.Dense(action_shape,activation="tanh")(l3)
    actor= Model(in1,action)
    return actor

def create_critic_network(state_shape,action_shape):
    in1 = layers.Input(shape=state_shape,name="state")
    in2 = layers.Input(shape=action_shape,name="action")
    l1 = layers.concatenate([in1,in2])
    l2 = layers.Dense(640,activation="relu")(l1)
    l3 = layers.Dense(640,activation="relu")(l2)
    l4 = layers.Dense(640,activation="relu")(l3)
    value = layers.Dense(1)(l4)
    critic = Model(inputs=[in1,in2],outputs=value)
    return critic,in1,in2
```

I am chosing ```MountainCarContinuous-v0``` game. Mainly because my GPU is not that good to work on higher dimensional state space

```python
env = gym.make("MountainCarContinuous-v0")
```

    WARN: gym.spaces.Box autodetected dtype as <type 'numpy.float32'>. Please provide explicit dtype.
    WARN: gym.spaces.Box autodetected dtype as <type 'numpy.float32'>. Please provide explicit dtype.

```python
state_shape= env.observation_space.sample().shape
```

```python
action_shape=env.action_space.sample().shape
```

```python
actor = create_actor_network(state_shape,action_shape[0])
critic,state_tensor,action_tensor = create_critic_network(state_shape,action_shape)
target_actor=create_actor_network(state_shape,action_shape[0])
target_critic,_,_ = create_critic_network(state_shape,action_shape)
target_actor.set_weights(actor.get_weights())
target_critic.set_weights(critic.get_weights())
```

I have chosen ```RMSProp``` optimizer, due to more stability compared to Adam . I found it after trials and errors, no theoritical background on chosing this optimizer

```python
actor_optimizer = keras.optimizers.RMSprop(actor_lr)

critic_optimizer = keras.optimizers.RMSprop(critic_lr)
```

```python
critic.compile(loss="mse",optimizer=critic_optimizer)
```

    WARNING:tensorflow:From /home/sezan92/anaconda/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
    Instructions for updating:
    keep_dims is deprecated, use keepdims instead

#### Actor training

I think this is the most critical part of ddpg in keras. The object ```critic``` and ```actor``` has a ```__call__``` method inside it, which will give output tensor if you give input a tensor. So to get the tensor object of ```Q``` we will use this functionality.

```python
CriticValues = critic([state_tensor,actor(state_tensor)])
```

Now it is time to get the gradient value of $-\frac{\partial Q(s,a)}{\theta_a}$

```python
updates = actor_optimizer.get_updates(
    params=actor.trainable_weights,loss=-K.mean(CriticValues))
```

Now we will create a function which will train the actor network.

```python
actor_train = K.function(inputs=[state_tensor],outputs=[actor(state_tensor),CriticValues],
                   updates=updates)

```

### Replay Buffer

```python
memory = deque(maxlen=10000)
```

```python
state = env.reset()
state = state.reshape(-1,)
for _ in tqdm(range(memory.maxlen)):
    action = env.action_space.sample()
    next_state,reward,terminal,_=env.step(action)

    state=next_state
    if terminal:
        reward=-100
        state= env.reset()
        state = state.reshape(-1,)
    memory.append((state,action,reward,next_state,terminal))
```

    100%|██████████| 10000/10000 [00:00<00:00, 21304.88it/s]

```python
next_state
```

    array([-0.49533114, -0.00344986])

### Noise class

The use of Noise is to make the process Stochastic and to help the agent explore different actions. The paper used Orstein Uhlenbeck Noise class

```python
class OrnsteinUhlenbeckProcess(object):
    def __init__(self, theta, mu=0, sigma=1, x0=0, dt=1e-2, n_steps_annealing=10, size=1):
        self.theta = theta
        self.sigma = sigma
        self.n_steps_annealing = n_steps_annealing
        self.sigma_step = - self.sigma / float(self.n_steps_annealing)
        self.x0 = x0
        self.mu = mu
        self.dt = dt
        self.size = size
    def restart(self):
        self.x0=copy.copy(self.mu)
    def generate(self, step):
        #sigma = max(0, self.sigma_step * step + self.sigma)
        x = self.x0 + self.theta * (self.mu - self.x0) * self.dt + self.sigma * np.sqrt(self.dt) * np.random.normal(size=self.size)
        self.x0 = x
        return x
```

### Training

```python
steps_per_episodes=5000
ou = OrnsteinUhlenbeckProcess(theta=0.35,mu=0.8,sigma=0.4,n_steps_annealing=10)
max_total_reward=0
for episode in range(num_episodes):
    state= env.reset()
    state = state.reshape(-1,)
    total_reward=0
    ou.restart()
    for step in range(steps_per_episodes):
        action= actor.predict(state.reshape(1,-1))+ou.generate(episode)
        next_state,reward,done,_ = env.step(action)
        total_reward=total_reward+reward
        #random minibatch from buffer

        batches=random.sample(memory,BATCH_SIZE)
        states= np.array([batch[0].reshape((-1,)) for batch in batches])
        actions= np.array([batch[1] for batch in batches])
        actions=actions.reshape(-1,1)
        rewards=np.array([batch[2] for batch in batches])
        rewards = rewards.reshape((-1,1))
        new_states=np.array([batch[3].reshape((-1,)) for batch in batches])
        terminals=np.array([batch[4] for batch in batches])
        terminals = terminals.reshape((-1,1))
        #training

        target_actions = target_actor.predict(new_states)
        target_Qs = target_critic.predict([new_states,target_actions])

        new_Qs = rewards+GAMMA*target_Qs*terminals
        critic.fit([states,actions],new_Qs,verbose=False)
        _,critic_values=actor_train(inputs=[states])
        target_critic_weights=[TAU*weight+(1-TAU)*target_weight for weight,target_weight in zip(critic.get_weights(),target_critic.get_weights())]
        target_actor_weights=[TAU*weight+(1-TAU)*target_weight for weight,target_weight in zip(actor.get_weights(),target_actor.get_weights())]
        target_critic.set_weights(target_critic_weights)
        target_actor.set_weights(target_actor_weights)
        print("Total Reward %f"%total_reward,end="\r")
        if SHOW:
            env.render()
        if done or step==(steps_per_episodes-1):

            if total_reward<0:
                print("Failed!",end=" ")
                reward=-100
            elif total_reward>0:
                print("Passed!",end=" ")
                reward=100
            memory.append((state,action,reward,next_state,done))
            break

        memory.append((state,action,reward,next_state,done))
        state=next_state
    if total_reward>max_total_reward:
        actor.save_weights("MC_DDPG_Weights/Actor_Best_weights episode %d_GAMMA_%f_TAU%f_lr_%f.h5"%(episode,GAMMA,TAU,actor_lr))
        critic.save_weights("MC_DDPG_Weights/Critic_Best_weights episode %d_GAMMA_%f_TAU%f_lr_%f.h5"%(episode,GAMMA,TAU,critic_lr))
        max_total_reward=total_reward
    print("Episode %d Total Reward %f"%(episode,total_reward))
```

    Failed! Episode 0 Total Reward -121.829750
    Failed! Episode 1 Total Reward -118.666569
    Failed! Episode 2 Total Reward -162.041720
    Failed! Episode 3 Total Reward -87.619343
    Failed! Episode 4 Total Reward -96.425173
    Failed! Episode 5 Total Reward -143.686660
    Failed! Episode 6 Total Reward -38.429055
    Passed! Episode 7 Total Reward 76.095012
    Passed! Episode 8 Total Reward 78.345259
    Failed! Episode 9 Total Reward -0.019448
    Passed! Episode 10 Total Reward 48.135478
    Passed! Episode 11 Total Reward 54.750747
    Passed! Episode 12 Total Reward 79.410100
    Passed! ward 83.0033361Episode 13 Total Reward 83.003336
    Passed! Episode 14 Total Reward 83.174427
    Failed! Episode 15 Total Reward -48.217783
    Passed! Episode 16 Total Reward 87.284775
    Passed! Episode 17 Total Reward 32.635075
    Passed! Episode 18 Total Reward 84.165725
    Passed! Episode 19 Total Reward 89.819676
    Passed! Episode 20 Total Reward 76.334761
    Passed! Episode 21 Total Reward 58.299554
    Passed! Episode 22 Total Reward 81.035049
    Failed! Episode 23 Total Reward -39.691471
    Failed! Episode 24 Total Reward -161.331777
    Passed! Episode 25 Total Reward 86.132543
    Failed! Episode 26 Total Reward -104.653697
    Passed! Episode 27 Total Reward 85.415800
    Failed! Episode 28 Total Reward -74.066867
    Failed! Episode 29 Total Reward -29.747882
    Passed! Episode 30 Total Reward 59.700877
    Passed! Episode 31 Total Reward 64.003951
    Failed! Episode 32 Total Reward -43.807664
    Passed! Episode 33 Total Reward 81.749323
    Passed! Episode 34 Total Reward 66.484130
    Failed! Episode 35 Total Reward -120.525069
    Failed! Episode 36 Total Reward -24.451582
    Failed! Episode 37 Total Reward -207.239527
    Failed! Episode 38 Total Reward -43.106546
    Passed! Episode 39 Total Reward 34.880121
    Passed! Episode 40 Total Reward 66.188821
    Passed! Episode 41 Total Reward 29.939728
    Failed! Episode 42 Total Reward -86.516192
    Failed! Episode 43 Total Reward -225.143733
    Passed! Episode 44 Total Reward 7.568837
    Passed! Episode 45 Total Reward 84.050707
    Failed! Episode 46 Total Reward -86.705966
    Passed! Episode 47 Total Reward 51.108435
    Failed! Episode 48 Total Reward -88.531423
    Passed! Episode 49 Total Reward 79.210735
    Failed! Episode 50 Total Reward -64.247977
    Failed! Episode 51 Total Reward -28.334167
    Failed! Episode 52 Total Reward -53.942703
    Passed! Episode 53 Total Reward 56.541394
    Failed! Episode 54 Total Reward -30.573394
    Failed! Episode 55 Total Reward -43.900060
    Failed! Episode 56 Total Reward -31.793103
    Passed! Episode 57 Total Reward 49.367744
    Failed! Episode 58 Total Reward -59.277307
    Failed! Episode 59 Total Reward -91.342287
    Passed! Episode 60 Total Reward 72.884442
    Passed! Episode 61 Total Reward 81.967582
    Failed! Episode 62 Total Reward -69.682571
    Failed! Episode 63 Total Reward -21.655243
    Failed! Episode 64 Total Reward -102.427871
    Failed! Episode 65 Total Reward -18.727795
    Failed! Episode 66 Total Reward -106.180673
    Passed! Episode 67 Total Reward 66.997363
    Failed! Episode 68 Total Reward -98.338890
    Failed! Episode 69 Total Reward -17.488416
    Failed! Episode 70 Total Reward -126.062955
    Passed! Episode 71 Total Reward 91.446696
    Failed! Episode 72 Total Reward -153.244886
    Passed! Episode 73 Total Reward 32.589752
    Failed! Episode 74 Total Reward -106.034746
    Failed! Episode 75 Total Reward -46.197340
    Failed! Episode 76 Total Reward -64.983000
    Failed! Episode 77 Total Reward -50.082150
    Failed! Episode 78 Total Reward -44.506584
    Failed! Episode 79 Total Reward -130.709335
    Passed! Episode 80 Total Reward 65.267968
    Failed! Episode 81 Total Reward -32.770421
    Passed! Episode 82 Total Reward 68.083454
    Failed! Episode 83 Total Reward -81.669658
    Failed! Episode 84 Total Reward -40.234104
    Passed! Episode 85 Total Reward 89.699236
    Passed! Episode 86 Total Reward 76.233450
    Failed! Episode 87 Total Reward -60.663254
    Passed! Episode 88 Total Reward 75.066797
    Failed! Episode 89 Total Reward -46.776051
    Failed! Episode 90 Total Reward -25.376468
    Passed! Episode 91 Total Reward 81.885317
    Passed! Episode 92 Total Reward 61.040792
    Failed! Episode 93 Total Reward -59.524774
    Passed! Episode 94 Total Reward 80.259871
    Passed! Episode 95 Total Reward 71.395997
    Passed! Episode 96 Total Reward 70.046112
    Passed! Episode 97 Total Reward 87.850163
    Passed! Episode 98 Total Reward 86.638371
    Failed! Episode 99 Total Reward -37.902022

### Video

Please watch at 2x speed. I changed some simple mistakes after the video so the rewards are not exactly the same

[](http://www.youtube.com/watch?v=9Fe_n-ovIaA "Keras tutorial DDPG")

```python

```


```python
from pycalphad import Database, variables as v
from pycalphad.variables import Species

dbf = Database('AlCoCrNi-2016Liu-minified.TDB')
```

```python
import itertools
import string
import sympy
import numpy as np

def genericize(mapping, const_array):
    new_array = []
    for subl in const_array:
        new_array.append(tuple(mapping[x] for x in subl))
    return tuple(new_array)
def sigfigs(x, n):
    """Round x to n significant digits"""
    if x != 0:
        return np.around(x, -(np.floor(np.log10(np.abs(x)))).astype(np.int) + (n - 1))
    else:
        return x
def average_piecewise(pw):
    return sum(x.expr for x in pw.args)/len(pw.args)

def categorize_phase(phase_name, constituents, site_ratios):
    if phase_name == 'LIQUID' and len(constituents) == 1:
        return 'liquid'
    elif phase_name == 'GAS' and len(constituents) == 1:
        return 'gas'
    elif len(constituents) == 1 or \
        (len(constituents) == 2 and len(constituents[1]) == 1 and list(constituents[1])[0] == Species('VA')):
        return 'solid'
    elif all(len(x) == 1 for x in constituents):
        return 'stoichiometric'
    else:
        return 'complex'

def extract_coefficients(expr, polys):
    result = {}
    for poly in polys:
        if poly == 1:
            continue
        else:
            coef = float(expr.coeff(poly))
        result[poly] = coef
        expr = expr - poly*coef
    expr = expr.n()
    if isinstance(expr, (sympy.Float, sympy.Integer)):
        result[1] = float(expr)
        expr = sympy.S.Zero
    if expr is not sympy.S.Zero:
        raise ValueError(expr)
    return tuple(result[p] for p in polys)

total_results = []

for combo in itertools.combinations(dbf.elements - {'VA', '/-'}, 2):
    combo_str = sorted(set(combo) | {'VA'})
    combo = [Species(x) for x in combo_str]
    combo_dict = dict(zip([c for c in combo if c.number_of_atoms > 0], string.ascii_uppercase))
    combo_dict[Species('VA')] = 'VA'
    allowed_phases = set()
    # TODO: Categorize each phase? solid solution, intermetallic, complex intermetallic, liquid, etc.
    for phase_name, phase_obj in dbf.phases.items():
        include_phase = True
        for sublattice in phase_obj.constituents:
            if len(set(sublattice).intersection(combo)) == 0:
                # None of the components in a sublattice are active
                # We cannot build a model of this phase
                include_phase = False
                break
        if include_phase:
            allowed_phases |= {phase_name}
    for param in dbf._parameters.all():
        if param['phase_name'] in allowed_phases and param['parameter_type'] in ['G', 'L']:
            active_pure_elements = [list(x.constituents.keys()) for subl in param['constituent_array'] for x in subl]
            active_pure_elements = [el.upper() for constituents in active_pure_elements for el in constituents]
            active_pure_elements = set(active_pure_elements)
            if not active_pure_elements.issubset(combo_str):
                continue
            generic_combo = [Species(x) for x in sorted(active_pure_elements)]
            gen_combo_dict = dict(zip([c for c in generic_combo if c.number_of_atoms > 0], string.ascii_uppercase))
            gen_combo_dict[Species('VA')] = 'VA'
            gen_combo_dict[Species('*')] = '*'
            ratio_sum = sum(dbf.phases[param['phase_name']].sublattices)
            reduced_ratios = tuple(sigfigs(c, 3) for c in dbf.phases[param['phase_name']].sublattices)
            param_val = param['parameter'].subs(dbf.symbols).subs(dbf.symbols).subs(dbf.symbols).subs(dbf.symbols)
            # If parameter is Piecewise, average it (for now)
            param_val = sympy.piecewise_fold(param_val)
            pw_atoms = param_val.atoms(sympy.Piecewise)
            pw_dict = {key: average_piecewise(key) for key in pw_atoms}
            param_val = param_val.subs(pw_dict)
            # Cross terms must be listed first
            polys = [v.T*sympy.log(1.0e-5*v.P)] + [v.T*sympy.log(v.T), sympy.log(v.T)] + [v.T**n for n in range(-30,0)] + [v.T**n for n in range(1,30)] + [1]
            columns = ['system', 'phase_name', 'phase_type', 'site_ratios',
                       'parameter_order', 'constituent_array', 'parameter_value'] + \
                        [str(x) for x in polys]
            try:
                id_tuple = (tuple(c for c in combo_str if c != 'VA'), param['phase_name'],
                            categorize_phase(param['phase_name'], dbf.phases[param['phase_name']].constituents, reduced_ratios),
                            reduced_ratios, param['parameter_order'],
                            genericize(gen_combo_dict, param['constituent_array']),
                            param_val) + extract_coefficients(param_val, polys)
                total_results.append(id_tuple)
            except ValueError as e:
                print(e)
```

    102862.903225806*((4.65e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 1115.9095
    172788.461538462*((2.6e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 3645.714
    172788.461538462*((2.6e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 7287.714
    102862.903225806*((4.65e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 1115.9095
    172788.461538462*((2.6e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 3645.714
    172788.461538462*((2.6e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 7287.714
    102862.903225806*((4.65e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 1115.9095
    172788.461538462*((2.6e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 3645.714
    172788.461538462*((2.6e-11*P + 1.0)**0.8 - 1.0)*exp(9.2e-9*T**2 + 1.7e-5*T) - 7287.714

```python
import pandas as pd
from IPython.display import display

df = pd.DataFrame(total_results, columns=columns)
for p in polys:
    df[str(p)] = pd.to_numeric(df[str(p)])
multi_df = df.set_index(['phase_type', 'site_ratios', 'constituent_array', 'parameter_order'])
display(multi_df)
```

      system
      phase_name
      parameter_value
      T*log(1.0e-5*P)
      T*log(T)
      log(T)
      T**(-30)
      T**(-29)
      T**(-28)
      T**(-27)
      ...
      T**21
      T**22
      T**23
      T**24
      T**25
      T**26
      T**27
      T**28
      T**29
      1

      phase_type
      site_ratios
      constituent_array
      parameter_order

      solid
      (1.0, 3.0)
      ((VA,), (VA,))
      0
      (CO, NI)
      BCC_A2
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      liquid
      (1.0,)
      ((A,),)
      0
      (CO, NI)
      LIQUID
      -7.3267e-22*T**7 - 5.78266666666667e-8*T**3 - ...
      0.0
      -21.862033
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4849.556000

      solid
      (1.0, 1.0)
      ((A,), (VA,))
      0
      (CO, NI)
      FCC_A1
      -2.89133333333333e-8*T**3 - 0.0004424565*T**2 ...
      0.0
      -10.931017
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -2600.775833

      (1.0, 3.0)
      ((A,), (VA,))
      0
      (CO, NI)
      BCC_A2
      -2.89133333333333e-8*T**3 - 0.0004424565*T**2 ...
      0.0
      -10.931017
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -1345.570833

      ((A, VA), (VA,))
      0
      (CO, NI)
      BCC_A2
      -35.045*T + 96765
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      96765.000000

      (1.0, 0.5)
      ((A,), (VA,))
      0
      (CO, NI)
      HCP_A3
      -2.89133333333333e-8*T**3 - 0.0004424565*T**2 ...
      0.0
      -10.931017
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -2814.570833

      complex
      (0.5, 0.5, 3.0)
      ((A,), (VA,), (VA,))
      0
      (CO, NI)
      BCC_B2
      -0.2875*T + 25867.5
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      25867.500000

      ((VA,), (A,), (VA,))
      0
      (CO, NI)
      BCC_B2
      -0.2875*T + 25867.5
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      25867.500000

      liquid
      (1.0,)
      ((A,),)
      0
      (CO, NI)
      LIQUID
      -1.27439333333333e-21*T**7 - 0.000537855555555...
      0.0
      -21.610667
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -1380.564556

      solid
      (1.0, 1.0)
      ((A,), (VA,))
      0
      (CO, NI)
      FCC_A1
      -0.000806783333333333*T**2 - 10.866*T*log(T) +...
      0.0
      -10.866000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -5503.302333

      (1.0, 3.0)
      ((A,), (VA,))
      0
      (CO, NI)
      BCC_A2
      -0.000806783333333333*T**2 - 10.866*T*log(T) +...
      0.0
      -10.866000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -1145.760333

      ((A, VA), (VA,))
      0
      (CO, NI)
      BCC_A2
      -0.22789*T + 24593.23
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      24593.230000

      (1.0, 0.5)
      ((A,), (VA,))
      0
      (CO, NI)
      HCP_A3
      -0.000806783333333333*T**2 - 10.866*T*log(T) +...
      0.0
      -10.866000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -4980.302333

      complex
      (0.5, 0.5, 3.0)
      ((VA,), (A,), (VA,))
      0
      (CO, NI)
      BCC_B2
      -6.7374925*T + 28302.71
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      28302.710000

      ((A,), (VA,), (VA,))
      0
      (CO, NI)
      BCC_B2
      -6.7374925*T + 28302.71
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      28302.710000

      liquid
      (1.0,)
      ((A, B),)
      0
      (CO, NI)
      LIQUID
      1331/2
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      665.500000

      solid
      (1.0, 1.0)
      ((A, B), (VA,))
      0
      (CO, NI)
      FCC_A1
      0.63145*T - 400
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -400.000000

      (1.0, 3.0)
      ((A, B), (VA,))
      0
      (CO, NI)
      BCC_A2
      1000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1000.000000

      (1.0, 0.5)
      ((A, B), (VA,))
      0
      (CO, NI)
      HCP_A3
      -0.1925*T - 810
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -810.000000

      complex
      (0.5, 0.5, 3.0)
      ((A,), (B,), (VA,))
      0
      (CO, NI)
      BCC_B2
      -0.832*T + 1932.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1932.000000

      ((B,), (A,), (VA,))
      0
      (CO, NI)
      BCC_B2
      -0.832*T + 1932.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1932.000000

      (0.75, 0.25, 1.0)
      ((B,), (A,), (VA,))
      0
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A,), (B,), (VA,))
      0
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A, B), (A,), (VA,))
      0
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A, B), (B,), (VA,))
      0
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A, B), (A,), (VA,))
      1
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A, B), (B,), (VA,))
      1
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((*,), (A, B), (VA,))
      0
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      1
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A, B), (*,), (VA,))
      0
      (CO, NI)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      solid
      (1.0, 0.5)
      ((A,), (VA,))
      0
      (AL, CR)
      HCP_A3
      -2.46201666666667e-7*T**3 + 0.000315725*T**2 -...
      0.0
      -12.818000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -5068.714000

      complex
      (0.5, 0.5, 3.0)
      ((A,), (VA,), (VA,))
      0
      (AL, CR)
      BCC_B2
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((VA,), (A,), (VA,))
      0
      (AL, CR)
      BCC_B2
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      liquid
      (1.0,)
      ((A, B),)
      0
      (AL, CR)
      LIQUID
      -14500
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -14500.000000

      1
      (AL, CR)
      LIQUID
      -5500
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -5500.000000

      solid
      (1.0, 1.0)
      ((A, B), (VA,))
      0
      (AL, CR)
      FCC_A1
      3*T - 22950
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -22950.000000

      (1.0, 3.0)
      ((A, B), (VA,))
      0
      (AL, CR)
      BCC_A2
      5*T - 27450
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -27450.000000

      complex
      (0.5, 0.5, 3.0)
      ((A,), (B,), (VA,))
      0
      (AL, CR)
      BCC_B2
      -1000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -1000.000000

      ((B,), (A,), (VA,))
      0
      (AL, CR)
      BCC_B2
      -1000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -1000.000000

      (0.75, 0.25, 1.0)
      ((B,), (A,), (VA,))
      0
      (AL, CR)
      L12_FCC
      -1245
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -1245.000000

      ((A,), (B,), (VA,))
      0
      (AL, CR)
      L12_FCC
      -1245
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -1245.000000

      ((A, B), (A,), (VA,))
      0
      (AL, CR)
      L12_FCC
      -2490.00000000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -2490.000000

      ((A, B), (B,), (VA,))
      0
      (AL, CR)
      L12_FCC
      -2490.00000000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -2490.000000

      ((A, B), (A,), (VA,))
      1
      (AL, CR)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A, B), (B,), (VA,))
      1
      (AL, CR)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((*,), (A, B), (VA,))
      0
      (AL, CR)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      1
      (AL, CR)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      ((A, B), (*,), (VA,))
      0
      (AL, CR)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      1
      (AL, CR)
      L12_FCC
      0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000

      stoichiometric
      (10.0, 1.0, 2.0)
      ((A,), (A,), (B,))
      0
      (AL, CR)
      AL11CR2
      -9.62500345833333e-6*T**3 + 0.023521515*T**2 -...
      0.0
      -155.848251
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -144305.234000

      (13.0, 2.0)
      ((A,), (B,))
      0
      (AL, CR)
      AL13CR2
      -1.12854762083333e-5*T**3 + 0.027683345*T**2 -...
      0.0
      -179.523206
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -151390.426000

      (4.0, 1.0)
      ((A,), (B,))
      0
      (AL, CR)
      AL4CR
      -3.56714716666667e-6*T**3 + 0.008639385*T**2 -...
      0.0
      -60.167910
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -67065.598000

      (8.0, 5.0)
      ((A,), (B,))
      0
      (AL, CR)
      AL8CR5_H
      -7.87289933333333e-6*T**3 + 0.018225945*T**2 -...
      0.0
      -158.789819
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -140835.338000

      0
      (AL, CR)
      AL8CR5_L
      -7.87289933333333e-6*T**3 + 0.018225945*T**2 -...
      0.0
      -158.789819
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -181726.838000

      (9.0, 4.0)
      ((A,), (B,))
      0
      (AL, CR)
      AL9CR4_H
      -8.45693404166667e-6*T**3 + 0.019991135*T**2 -...
      0.0
      -157.809297
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -130714.470000

      0
      (AL, CR)
      AL9CR4_L
      -8.45693404166667e-6*T**3 + 0.019991135*T**2 -...
      0.0
      -157.809297
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -178872.970000

      (1.0, 2.0)
      ((A,), (B,))
      0
      (AL, CR)
      ALCR2
      -1.32263970833333e-6*T**3 + 0.002712365*T**2 -...
      0.0
      -37.473477
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -34741.774000

      complex
      (8.0, 18.0, 4.0)
      ((A,), (A,), (B,))
      0
      (AL, CR)
      SIGMA_SGTE
      -1.50988250416667e-5*T**3 + 0.036638455*T**2 -...
      0.0
      -252.509116
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      31918.762000

      ((A,), (B,), (B,))
      0
      (AL, CR)
      SIGMA_SGTE
      -1.20583276666667e-5*T**3 + 0.02359327*T**2 - ...
      0.0
      -376.695819
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      -166917.476000

      (3.0, 2.0, 1.0)
      ((A,), (A,), (VA,))
      0
      (AL, CR)
      AL3NI2
      -4.151181875e-6*T**3 + 0.010404575*T**2 - 59.1...
      0.0
      -59.187387
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      21125.770000

218 rows × 66 columns

```python
for system_name, system_df in df.groupby('system'):
    system_df = system_df.drop_duplicates(subset='phase_name')
    print('-')
    print('-'.join(system_name))
    print(system_df['phase_type'].value_counts().to_string())
```

    -
    AL-CO
    stoichiometric    4
    solid             3
    complex           3
    liquid            1
    -
    AL-CR
    stoichiometric    8
    complex           4
    solid             3
    liquid            1
    -
    AL-NI
    solid             3
    complex           3
    stoichiometric    2
    liquid            1
    -
    CO-CR
    complex    3
    solid      3
    liquid     1
    -
    CO-NI
    solid      3
    complex    2
    liquid     1
    -
    CR-NI
    complex    3
    solid      3
    liquid     1

```python
import numpy as np

def frac_nonzero(x):
    return np.count_nonzero(x)/np.size(x)
def fnz_var(x):
    fnz = frac_nonzero(x)
    return fnz*(1-fnz)

mdc = (multi_df.drop(columns=['system', 'phase_name', 'parameter_value'])
                .groupby(level=[0,1,2,3])
                .agg([frac_nonzero, fnz_var])
    )
#display(mdc)
mdl = (multi_df.drop(columns=['system', 'phase_name', 'parameter_value'])
                .groupby(level=[0,1,2,3])
                .apply(np.size, axis=0)
    )
display(mdl)
```

    phase_type      site_ratios        constituent_array      parameter_order
    complex         (0.5, 0.5, 3.0)    ((A,), (B,), (VA,))    0                   6
                                       ((A,), (VA,), (VA,))   0                  12
                                       ((B,), (A,), (VA,))    0                   6
                                       ((VA,), (A,), (VA,))   0                  12
                    (0.75, 0.25, 1.0)  ((*,), (A, B), (VA,))  0                   6
                                                              1                   6
                                       ((A,), (B,), (VA,))    0                   6
                                       ((A, B), (*,), (VA,))  0                   6
                                                              1                   6
                                       ((A, B), (A,), (VA,))  0                   6
                                                              1                   6
                                       ((A, B), (B,), (VA,))  0                   6
                                                              1                   6
                                       ((B,), (A,), (VA,))    0                   6
                    (3.0, 2.0, 1.0)    ((A,), (*,), (B, VA))  0                   1
                                       ((A,), (A,), (B,))     0                   1
                                       ((A,), (A,), (VA,))    0                   3
                                       ((A,), (A, B), (*,))   0                   1
                                       ((A,), (B,), (B,))     0                   1
                                       ((A,), (B,), (VA,))    0                   1
                    (8.0, 18.0, 4.0)   ((A,), (A,), (B,))     0                   2
                                       ((A,), (B,), (B,))     0                   2
                                       ((B,), (A,), (A,))     0                   1
                                       ((B,), (B,), (A,))     0                   1
    liquid          (1.0,)             ((A,),)                0                   9
                                       ((A, B),)              0                   6
                                                              1                   5
                                                              2                   2
                                                              3                   2
                                                              4                   1
    solid           (1.0, 0.5)         ((A,), (VA,))          0                  12
                                       ((A, B), (VA,))        0                   2
                    (1.0, 1.0)         ((A,), (VA,))          0                   9
                                       ((A, B), (VA,))        0                   6
                                                              1                   2
                                                              2                   2
                                                              3                   1
                    (1.0, 3.0)         ((A,), (VA,))          0                   9
                                       ((A, B), (VA,))        0                   6
                                                              1                   2
                                       ((A, VA), (VA,))       0                  12
                                       ((VA,), (VA,))         0                   6
    stoichiometric  (0.375, 0.625)     ((A,), (B,))           0                   1
                    (0.75, 0.25)       ((A,), (B,))           0                   1
                    (1.0, 2.0)         ((A,), (B,))           0                   1
                    (3.0, 1.0)         ((A,), (B,))           0                   1
                    (4.0, 1.0)         ((A,), (B,))           0                   1
                    (5.0, 2.0)         ((A,), (B,))           0                   1
                    (8.0, 5.0)         ((A,), (B,))           0                   2
                    (9.0, 2.0)         ((A,), (B,))           0                   1
                    (9.0, 4.0)         ((A,), (B,))           0                   2
                    (10.0, 1.0, 2.0)   ((A,), (A,), (B,))     0                   1
                    (13.0, 2.0)        ((A,), (B,))           0                   1
                    (13.0, 4.0)        ((A,), (B,))           0                   1
    dtype: int64

```python
df.drop_duplicates(subset=['phase_name', 'system'])['phase_type'].value_counts()
```

    complex           18
    solid             18
    stoichiometric    14
    liquid             6
    Name: phase_type, dtype: int64

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sy
import simtk.unit as unit
from simtk import openmm as mm
from simtk.openmm import app
import skopt as skopt
from tqdm import tqdm
```

# 3D Liquid Argon model

```python
mass = 39.948 * unit.amu
sigma = 3.404 * unit.angstroms
epsilon = 0.238 * unit.kilocalories_per_mole
charge = 0.0 * unit.elementary_charge

n_particles = 1000
reduced_density = 0.85
l_box = None # 40.0 * unit.angstroms
```

```python
temperature = 300.00 * unit.kelvin
integration_timestep = 0.002 * unit.picoseconds
collisions_rate = 1.0 / unit.picoseconds

equilibration_time = 1.0 * unit.nanoseconds
production_time = 10.0* unit.nanoseconds
saving_time = 20.0 * unit.picoseconds

equilibration_n_steps = round(equilibration_time/integration_timestep)
```

```python
radius = 2**(-5.0/6.0)*sigma

if l_box is None:
    volume_particles = n_particles*(4.0/3.0)*np.pi*radius**3
    volume = volume_particles/reduced_density
    l_box = volume**(1/3)
    print('Side of the box: {}'.format(l_box))
else:
    volume_particles = n_particles*(4.0/3.0)*np.pi*radius**3
    volume = l_box**3
    reduced_density = volume_particles/volume
    print('Reduced density: {}'.format(reduced_density))
```

    Side of the box: 32.51031041310817 A

```python
space = skopt.Space([[0.0, l_box._value], [0.0, l_box._value], [0.0, l_box._value]])
generator = skopt.sampler.Lhs(criterion="maximin", iterations=1000)
positions_3d = generator.generate(space.dimensions, n_particles)
positions_3d = np.array(positions_3d)*unit.angstroms
```

```python
system = mm.System()
```

```python
v1 = np.zeros(3) * unit.angstroms
v2 = np.zeros(3) * unit.angstroms
v3 = np.zeros(3) * unit.angstroms

v1[0] = l_box
v2[1] = l_box
v3[2] = l_box

system.setDefaultPeriodicBoxVectors(v1, v2, v3)
```

```python
non_bonded_force = mm.NonbondedForce()
non_bonded_force.setNonbondedMethod(mm.NonbondedForce.CutoffPeriodic)
non_bonded_force.setCutoffDistance(3.0*sigma)
non_bonded_force.setUseSwitchingFunction(True)
non_bonded_force.setSwitchingDistance(2.0*sigma)
non_bonded_force.setUseDispersionCorrection(True)
```

```python
for _ in range(n_particles):
    system.addParticle(mass)
    non_bonded_force.addParticle(charge, sigma, epsilon)
```

```python
_ = system.addForce(non_bonded_force)
```

```python
integrator = mm.LangevinIntegrator(temperature, collisions_rate, integration_timestep)
platform = mm.Platform.getPlatformByName('CUDA')
context = mm.Context(system, integrator, platform)
```

```python

```

```python
context.setPositions(initial_positions)
context.setVelocitiesToTemperature(temperature)
```

```python
state=context.getState(getEnergy=True)
print("Before minimization: {}".format(state.getPotentialEnergy()))
mm.LocalEnergyMinimizer_minimize(context)
state=context.getState(getEnergy=True)
print("After minimization: {}".format(state.getPotentialEnergy()))
```

```python
equilibration_n_steps = round(equilibration_time/integration_timestep)
integrator.step(equilibration_n_steps)
context.setTime(0.0*unit.picoseconds)
```

```python
production_n_steps = int(production_time/integration_timestep)
saving_n_steps = int(saving_time/integration_timestep)
n_saving_periods = int(production_n_steps/saving_n_steps)

time = np.zeros([n_saving_periods]) * unit.nanoseconds
trajectory = np.zeros([n_saving_periods, n_particles, 3]) * unit.angstroms
potential_energy = np.zeros([n_saving_periods]) * unit.kilocalories_per_mole

for ii in tqdm(range(n_saving_periods)):
    integrator.step(saving_n_steps)
    #remove enforcePeriodicBox to get unwrapped trajectories
    state = context.getState(getPositions=True, getEnergy=True, enforcePeriodicBox = False)
    time[ii] = state.getTime()
    trajectory[ii,:,:] = state.getPositions(asNumpy=True)
    potential_energy = state.getPotentialEnergy()
```

```python
trajectory_mem = trajectory.size * trajectory.itemsize * unit.bytes
print('Trajectory size: {} MB'.format(trajectory_mem._value/(1024*1024)))
```

```python
trajectory.max()
```

```python
plt.plot(time, trajectory[:,0,0])
plt.plot(time, trajectory[:,0,1])
plt.plot(time, trajectory[:,0,2])
plt.plot(time, trajectory[:,100,0])
plt.plot(time, trajectory[:,100,1])
plt.plot(time, trajectory[:,100,2])
```

- Calcular difusion de las partículas
- Calcular RDF
- Energy
- Sensibilidad frente a integration_timestep
- Cuando converge
- Empezar con una grid (muy fuera del equilibrio)... cuando tengo una estructura más equilibrada?
- Empezar con otro tipo de distribución aleatoria o quasi-aleatoria y ver si converge más rapidamente.
- Aplicar barostato
- Cambiar el integrador, de langevin a verlet por ejemplo, la termodinámica converge antes?
- Sacar magnitudes de equilibrio haciendo montecarlo

```python

```


```python
import holoviews as hv
hv.extension('bokeh')
```

```python
import numpy as np
import scipy.signal
```

# Parametric modeling

## Introduction

Modeling is a key step in the pipeline of statistical inference. In this series of lectures we will learn how to fit parametric and non-parametric models to our data.

- In the parametric case we will focus on the classical frequentist method to fit models: maximum likelihood estimation (MLE)
- Then we will review non-parametric modeling techniques such as the Histogram and Kernel density estimation
- After that we will go back to parametric modeling and extend MLE using priors: Maximum a posteriori
- Finally we will present key ideas on bayesian modeling that are to be further developed in a future series of lectures

### What is statistical inference?

Inference is:

> To draw conclusions from facts through a scientific premise

In the particular case of **Statistical inference** we have

- Facts: Observed data
- Premise: Probabilistic model
- Conclusion: An unobserved quantity of interest
- Objective: Quantify the uncertainty of the conclusion given the data and the premise

The following are examples examples of statistical inference tasks:

- **Parameter estimation:** What is the best estimate of a model parameter based on the observed data?
- **Confidence estimation:** How trustworthy is our point estimate?
- **Hypothesis testing:** Is the data consistent with a given hypothesis or model?

### The difference between parametric and non-parametric models

To conduct inference we start by defining a statistical model

Models can be broadly classified as:

**Parametric:**

- It corresponds to an analytical function  (distribution) with free parameters
- Has an *a-priori* fixed number of parameters
- In general: Stronger assumptions, easier to interpret, faster to use

**Non-parametric:**

- Distribution-free model but they do have parameters and assumptions (e.g. dependence)
- The number of parameters depends on the amount of training data
- In general: More flexible, harder to train

### The difference between Frequentist and Bayesian inference

There are two main paradigms or perspectives for statistical inference: Frequentist (F) or classical and Bayesian (B). There are conceptual differences between these paradigms, for example

**Definition of probability:**

- F: Relative frequency of an event. An objective property of the real world
- B: Degree of subjective belief. Probability statements can be made not only on data but also on parameters and models themselves

**Interpretation of parameters:**

- F: They are unknown and fixed constants
- B: They have distributions that quantify the uncertainty of our knowledge about them. We can compute expected values of the parameters

In this lecture we will focus on the frequentist approach to modeling. We will review the bayesian perspective in a future lesson

## Frequentist approach on parametric modeling

In the case of **parametric inference** we assume that observations follow a certain distribution, *i.e.* observations are a realization of a random process (sampling)

The conceptual (iterative) steps of parametric inference are:

1. **Model fitting:** Find parameters by fitting data to the current model
1. **Model proposition:** Propose a new model that accommodates important features of the data better than the previous one

In the frequentist approach step 1 is typically solved using **Maximum Likelihood Estimation (MLE)**. Other frequentist alternatives are the Method of Moments (MoM) and the M-estimator. Only MLE is covered in this lecture

### The likelihood

The likelihood function is a quantitative description of our experiment (measuring process), and it is the  starting point for **parametric modeling** for both the frequentist and bayesian paradigms. In simple terms the likelihood tells us how good our model is with respect to the **observed data**

Let's now give a mathematical description of the likelihood

- Suppose we have an experiment that we model as a set of R.Vs $X_1, X_2, \ldots, X_N$
- We also have observations/realizations from our R.Vs $\{x_i\} = x_1, x_2, \ldots, x_N$
- We assume that the R.Vs follow a certain joint probability distribution $f(x_1, x_2, \ldots, x_N | \theta)$ with parameters $\theta$

The likelihood function is then defined as

$$
\begin{align}
\mathcal{L}(\theta) &= P(X_1=x_1, X_2=x_2, \ldots, X_N=x_n) \nonumber \\
&= f(x_1, x_2, \ldots, x_N | \theta) \nonumber
\end{align}
$$

which is a function of the parameters $\theta$

For the examples in this lecture we will additionally assumme that our observations are **independent and identically distributed** (iid). With this we can simplify the previous expression as

$$
\begin{align}
\mathcal{L}(\theta) &= f(x_1| \theta) \cdot f(x_2| \theta) \cdot \ldots \cdot f(x_N| \theta) \nonumber \\
&= \prod_{i=1}^N f(x_i| \theta) \nonumber
\end{align}
$$

The value of the likelihood itself does not hold much meaning, but it can be used to make comparisons between different values of the parameter vector $\theta$. **The larger the likelihood the better the model**, i.e. likelihood maximization allows us to find the best $\theta$ for our data

Before continuing consider the following

:::{warning}

**Likelihood is not probability**

:::

- The likelihood of a set of RVs does not integrate (or sum in the discrete case) to unity, *i.e.* in general the likelihood is not a valid probability density function.
- The likelihood by itself cannot be interpreted as a probability of $\theta$, it only tells us how likely is that $\{x_i\}$ was generated by the distribution $f$ with parameter $\theta$

## Maximum Likelihood Estimation (MLE)

In parametric modeling we are interested in finding $\theta$ that best fit our observations.

One method to do this is **MLE**:

- **1** Select a distribution (model) for the observations and formulate the likelihood $\mathcal{L}(\theta)$
- **2** Search for $\theta$ that maximizes $\mathcal{L}(\theta)$ given the data, *i.e.*

$$
\hat \theta = \text{arg} \max_\theta \mathcal{L}(\theta),
$$

where the point estimate $\hat \theta$  is called the **maximum likelihood estimator** of $\theta$

After this we can

- **3** Determine the confidence region of $\hat \theta$ either analytically or numerically (bootstrap, cross-validation, etc)
- **4** Make conclusions about your model (hypothesis test)

:::{important}

A wrong assumption in step 1 can ruin your inference. Evaluate how appropriate your model is, compare with other models and suggest incremental improvements

:::

###  Example: MLE  for the mean of a Gaussian distribution

Let us consider a set of N measurements $\{x_i\}_{i=1,\ldots, N}$ obtained from a sensor and that all were obtained under the same conditions

The sensor used to measure the data has an error that follows a Gaussian distribution with known variance $\sigma^2$

If the conditions are the same then the measurements can be viewed as noisy realizations of the true value $\mu$

$$
x_i = \mu + \epsilon_i, \quad \epsilon_i \sim \mathcal{N}(0, \sigma^2),
$$

and we can write the distribution of $x_i$ as

$$
f(x_i) = \mathcal{N}(x_i |\mu,\sigma^2) \quad \forall i
$$

Finally, the likelihood of $\mu$ given the measurements and the variance $\sigma^2$ is

$$
\mathcal{L}(\mu) = f(\{x_i\}| \mu, \sigma^2) = \prod_{i=1}^N f(x_i| \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \prod_{i=1}^N  \exp  \left( -\frac{(x_i-\mu)^2}{2\sigma^2} \right)
$$

We can find $\mu$ by maximizing the likelihood given $\{x_i\}$

:::{tip}

In many cases it is more practical to find the maximum of the **logarithm of the likelihood** (e.g. distributions from the exponential family). Logarithm is a monotonic function and its maximum is the same as its argument.

:::

In this case the log likelihood is

$$
\begin{align}
\log \mathcal{L} (\mu) &= \log \prod_{i=1}^N f(x_i|\mu, \sigma^2) \nonumber \\
&= \sum_{i=1}^N \log f(x_i|\mu, \sigma^2) \nonumber \\
&= - \frac{1}{2} \sum_{i=1}^N \log 2\pi\sigma^2 - \frac{1}{2} \sum_{i=1}^N  \frac{(x_i-\mu)^2}{\sigma^2}  \nonumber  \\
&=  - \frac{N}{2} \log 2\pi\sigma^2 - \frac{1}{2\sigma^{2}}   \sum_{i=1}^N (x_i-\mu)^2 \nonumber
\end{align}
$$

We maximize by making the derivative of the log likelihood equal to zero

$$
\frac{d  \log \mathcal{L} (\mu)}{d\mu} =  \frac{1}{\sigma^{2}}  \sum_{i=1}^N (x_i-\mu) =0
$$

Finally the MLE of $\mu$ is

$$
\hat \mu = \frac{1}{N} \sum_{i=1}^N x_i, \quad \sigma >0,
$$

The [sample mean](https://en.wikipedia.org/wiki/Sample_mean_and_covariance#Definition_of_the_sample_mean) is the MLE estimator of the mean for a Gaussian likelihood

### Example: MLE for the variance of a Gaussian distribution

Let's say now that we don't know the variance of the noise of the sensor

The MLE estimator of the variance can be obtained using the same procedure:

$$
\log \mathcal{L} (\mu, \sigma^2) =  - \frac{N}{2} \log 2\pi\sigma^2 - \frac{1}{2\sigma^{2}}   \sum_{i=1}^N (x_i-\mu)^2
$$

$$
\frac{d  \log \mathcal{L} (\mu, \sigma^2)}{d\sigma^2} =  - \frac{N}{2} \frac{1}{\sigma^2} + \frac{1}{2\sigma^{4}}\sum_{i=1}^N (x_i-\mu)^2 =0
$$

$$
\hat \sigma^2 = \frac{1}{N} \sum_{i=1}^N (x_i- \hat\mu)^2
$$

:::{note}

If the true mean is not known then this is a biased estimator of the true variance. MLE does not guarantee unbiased estimators

:::

The following code example shows how the value of the MLEs of these parameters evolve as more data is observed

```python
# data from the sensor
np.random.seed(1234)
x = 80 + np.random.randn(10000)
#x = 80 + 2*np.random.rand(1000)  # What happens if the data is not normal

# Computing the MLE
Ns = np.round(np.logspace(0, 4, num=16)).astype('int')
hat_mu = np.array([np.mean(x[:N]) for N in Ns])
hat_s2 = np.array([np.mean((x[:N]-mu)**2) for mu, N in zip(hat_mu, Ns)])
```

```python
mu_plot = hv.Curve((Ns, hat_mu), 'Number of samples', 'hat mu')*hv.HLine(80).opts(color='k')
s2_plot = hv.Curve((Ns, hat_s2), 'Number of samples', 'hat s2')*hv.HLine(1).opts(color='k')
(mu_plot + s2_plot).opts(hv.opts.Curve(logx=True, width=350), hv.opts.HLine(line_dash='dashed'))
```

### A note on biased and unbiased estimators

For a parameter $\theta$ and an estimator $\hat \theta$, if

$$
\mathbb{E}[\hat \theta] = \theta,
$$

then $\hat \theta$ is an unbiased estimator of $\theta$

**Is the MLE of $\mu$ unbiased?**

$$
\begin{align}
\mathbb{E}[\hat \mu] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N x_i \right]  \nonumber \\
&= \frac{1}{N} \sum_{i=1}^N \mathbb{E}[x_i] = \frac{1}{N} \sum_{i=1}^N \mu = \mu  \nonumber
\end{align}
$$

> The answer is YES

**Is the MLE of $\sigma^2$ unbiased?**

First lets expand the expression of the MLE of the variance

$$
\begin{align}
\hat \sigma^2 &= \frac{1}{N} \sum_{i=1}^N \left(x_i- \frac{1}{N}\sum_{j=1}^N x_j \right)^2 \nonumber \\
&= \frac{1}{N} \sum_{i=1}^N x_i^2 - \frac{1}{N^2} \sum_{i=1}^N \sum_{j=1}^N x_i  x_j \nonumber \\
&= \frac{1}{N} \sum_{i=1}^N x_i^2 - \frac{1}{N^2} \sum_{i=1}^N \sum_{j\neq i} x_i x_j - \frac{1}{N^2} \sum_{i=1}^N x_i^2 \nonumber  \\
&= \frac{N-1}{N^2} \sum_{i=1}^N x_i^2 - \frac{1}{N^2} \sum_{i=1}^N \sum_{j \neq i} x_i x_j  \nonumber
\end{align}
$$

Then applying the expected value operator we get

$$
\begin{align}
\mathbb{E}[\hat \sigma^2] &= \frac{N-1}{N^2} \sum_{i=1}^N \mathbb{E} [x_i^2] - \frac{1}{N^2} \sum_{i=1}^N \sum_{j \neq i} \mathbb{E} [x_i] \mathbb{E} [x_j] \nonumber  \\
&= \frac{N-1}{N} (\sigma^2 + \mu^2) - \frac{N-1}{N} \mu^2 \nonumber \\
&= \frac{N-1}{N} \sigma^2 \neq \sigma^2  \nonumber
\end{align}
$$

> The answer is NO

**Can we correct for the bias?**

> In this case, YES!

If we multiply it by $\frac{N}{N-1}$ we obtain the well known [unbiased estimator of the variance](https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation)

$$
\hat \sigma_{u}^2 = \frac{N}{N-1} \hat \sigma^2 = \frac{1}{N-1} \sum_{i=1}^N (x_i- \hat\mu)^2
$$

### Example: MLE of a Gaussian mixture

Let's imagine that our *iid* data come from a mixture of Gaussians with K components

$$
f(x_i|\pi,\mu,\sigma^2) = \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k, \sigma_k^2),
$$

where $\sum_{k=1}^K \pi_k = 1$ and $\pi_k \in [0, 1] ~~ \forall k$

We can write the log likelihood as

$$
\log \mathcal{L}(\pi,\mu,\sigma^2) = \sum_{i=1}^N \log \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k, \sigma_k^2)
$$

:::{note}

In this case we cannot obtain analytical expressions for the parameters by setting the derivative to zero. We have to resort to iterative methods/optimizers, e.g. gradient descent, expectation maximization

:::

We will come back to this in a future class on expectation maximization and Gaussian mixture models

## Optimality properties and uncertainty of MLEs

Assuming that the data truly comes from the specified model the MLE is

**Consistent:** The estimate converge to the true parameter as data points increase

$$
\lim_{N\to \infty} \hat \theta = \theta
$$

**Asymptotically normal:** The distribution of the estimate approaches a normal centered at the true parameter.

$$
\lim_{N\to \infty} p(\hat \theta) = \mathcal{N}(\hat \theta | \theta, \sigma_\theta^2),
$$

which is a consequence of the **central limit theorem**

For *i.i.d.* $\{X_i\}, i=1,\ldots,N$ with $\mathbb{E}[X] < \infty$ and $\text{Var}[X] < \infty$ then

$$
\lim_{N\to\infty} \sqrt{N} (\bar X - \mathbb{E}[X]) = \mathcal{N}(0, \sigma^2)
$$

Because MLE have asymptotically normal distributions the log likelihood ratio have asymptotically a *chi-square* distributions (more about this later)

**Minimum variance:** The estimate achieve the theoretical minimal variance given by the **Cramer-Rao bound**. This bound is the inverse of the expected Fisher information, *i.e* the second derivative of $- \log L$ with respect to $\theta$

$$
\sigma_{nm}^2 =  \left (- \frac{d^2 \log \mathcal{L} (\theta)}{d\theta_n \theta_m} \bigg\rvert_{\theta = \hat\theta}\right)^{-1}
$$

:::{note}

- $\sigma_{nm}^2$ is the minimum variance achieved by an unbiased estimator.
- $\sigma_{nn}^2$ gives the marginal error bars
- If $\sigma_{nm} \neq 0 ~~ n\neq m$, then errors are correlated, *i.e* some combinations of parameters might be better determined than others

:::

### Example: Cramer-Rao bound for the MLE of $\mu$

Considering a Gaussian likelihood from the previous example

$$
\log \mathcal{L} (\mu, \sigma^2) =  - \frac{N}{2} \log 2\pi\sigma^2 - \frac{1}{2\sigma^{2}}   \sum_{i=1}^N (x_i-\mu)^2
$$

What is the uncertainty of the MLE of $\mu$? In this case the Cramer-rao bound

$$
\begin{align}
\sigma_{\hat\mu}^2  &= \left (- \frac{d^2 \log \mathcal{L}(\mu, \sigma^2)}{d\mu^2} \bigg\rvert_{\mu=\hat\mu}\right)^{-1}  \nonumber \\
&=  \left (- \frac{1}{\sigma^2} \frac{d}{d\mu}  \sum_{i=1}^N (x-\mu) \bigg\rvert_{\mu=\hat\mu}\right)^{-1}  \nonumber \\
&=  \left ( \frac{N}{\sigma^2}  \bigg\rvert_{\mu=\hat\mu}\right)^{-1} = \frac{\sigma^2}{N}  \nonumber
\end{align}
$$

an expression that is known as the [standard error of the mean](https://en.wikipedia.org/wiki/Standard_error)

Then we have

$$
p(\hat \mu) \to \mathcal{N}(\hat \mu| \mu, \sigma^2/N)
$$

Let's see how the MLE and its uncertainty changes as more data is observed

```python
# Generate data
np.random.seed(12345)
mu_real, s_real = 2.23142, 1.124123
x = mu_real + s_real*np.random.randn(10000)
# MLE and its standard error
hat_mu = np.array([np.mean(x[:n]) for n in range(1, len(x))])
standard_error = s_real/np.sqrt(np.arange(1, len(x)))
```

```python
mu_plot = hv.Curve((range(1, len(x)), hat_mu),
                   kdims='Number of samples', vdims='mu', label='Estimated mu')
se_plot = hv.Spread((range(1, len(x)), hat_mu, standard_error),
                    kdims='Number of samples', label='Standard error')
mu_real_plot = hv.HLine(mu_real).opts(line_dash='dashed', color='k', alpha=0.5)
(mu_plot*se_plot*mu_real_plot).opts(hv.opts.Curve(logx=True, width=500))
```

## Hypothesis tests based on the likelihood

Considering the asymptotic distributions shown before we can formulate a hypothesis test for the MLE of $\theta$

We will present the Wald-test and the Wilks test

###  Wald-test

Suppose we wish to test

$$
\mathcal{H}_0: \theta = \theta_0
$$
$$
\mathcal{H}_A: \theta \neq \theta_0
$$

Under the null we can write

$$
W = \frac{(\hat \theta - \theta_0)^2}{\left (- \frac{d^2 \log \mathcal{L} (\theta)}{d\theta^2} \bigg\rvert_{\theta = \hat\theta}\right)^{-1}} = (\hat \theta - \theta_0)^2 \sigma_{\hat \theta}^2 \to \chi^2_1
$$

The test statistic have a $\chi^2$ distribution with one degree of freedom

If $W$ is greather than the $(1-\alpha)100\%$ quantile of $\chi^2_1$ we reject the null hypothesis

### Log-likelihood ratio test or Wilks test

Suppose we wish to test

$$
\mathcal{H}_0: \theta = \theta_0
$$
$$
\mathcal{H}_A: \theta =\theta_1
$$

We can write a ratio between likelihoods

$$
\lambda(\mathcal{D}) = \frac{\mathcal{L}(\theta_0|\mathcal{D})}{\mathcal{L}(\theta_1|\mathcal{D})}
$$

Asymptotically, under the null, we have

$$
-2 \log \lambda(\mathcal{D}) \to \chi^2_1
$$

If $-2 \log \lambda(\mathcal{D})$ is greather than the $(1-\alpha)100\%$ quantile of $\chi^2_1$ we reject the null

## Criteria for model comparison

How to compare models with different number of parameters? In general the more number of parameters the better the fit (overfitting). The likelihood does not take into account the complexity (number of parameters) of the model

> How to score models taking into account their complexity?

One option is to use the Akaike information criterion (AIC). For a model with $k$ parameters and N data points the AIC is

$$
\text{AIC} = -2 \log \mathcal{L}(\hat \theta) + 2k + \frac{2k(k+1)}{N-k-1},
$$

which one seeks to minimize. The AIC combines the likelihood (score) of the model and its complexity. The AIC is based on an asumptotic approximation which we will review in the future

This is also related to the idea of regularization, which will also be reviewed in future lectures

```python

```